@andrewchen

Get the newsletter · Featured · Recent

Benefit-Driven Metrics: Measure the lives you save, not the life preservers you sell

Measuring value created rather than optimizing for yourself
In my last blog post, I talked about the idea that value creation generates revenue, traffic, and other metrics, not the other way around. This is a particularly interesting idea to implement because it goes against much of the standard analytics reports that are out there.

The reason is that ultimately, most metrics tend to focus inwards, on self-interested gain, rather than outwards on the value you’re creating for your customers. Let’s take a couple examples of inward-focused metrics that people often cite:

  • account registrations
  • pageviews
  • unique visitors per month
  • revenues

I’m sure you measure many of the above, as I do as well. It’s OK to measure this stuff, but if you start to optimize for it, you are starting to focus on the business of value extraction, not value creation.

Measuring # of life preservers sold versus the # of lives saved
Thus we come back to the title of the post. Most people are using standard analytics packages that are commoditized to focus inwards on metrics like pageviews and revenue. To use an analogy, that’s akin to the idea of measuring the # of life preservers that you sell, and trying to optimize for that, rather than optimizing for the benefit, which is the # of lives saved.

If you focus primarily on selling life preservers, then you’ll tend to do all sorts of stuff like:

  • making them cheaper to build
  • adding doo-dads to them that are flashy to customers
  • using aggressive sales tactics
  • etc.

These things might generate revenue in the short to medium run, but if you prioritize this at the expense of actually delivering on the product benefit, then that’s a bad optimization in the long term.

Now contrast that to the idea of trying to save as many lives as possible. You might still want to make them as cheap as possible, so that every ship in the world can have as many of them as needed. You might still want to add upgrades to them, but only if they help save lives. etc. While these changes may be similar in execution, they are different in spirit than the changes you’d make when optimizing for sales.

Introducing “Benefit-Driven Metrics”
So ultimately to start this exercise, you should throw out all the standard metrics (conversion rates, pageviews, etc.) and just focus on one thing:

What are your customers measuring?

By looking at how they define value, then you get yourself aligned to them as closely as possible. Answering this question sets your company up for value creation, which then unlocks the ability to gain something from that value, then you have to start here.

I’ll deem these quantitative measurements as “Benefit-driven metrics.”

How do you measure it?
Here’s the interesting part – everyone’s benefit-driven metrics will be completely different, because most people’s customers and value proposition and product are ultimately very different. Unfortunately, you don’t have the crutch of standardized numbers like pageviews or uniques to lean on.

But let me give you some examples for reference:

For dating sites:
Why do customers join dating sites? To find their soulmates. Thus, measure the quantity of successful matches you make, not the lifetime value of the customer. Focusing on LTV can easily lead you to do things like creating fake accounts to make people come back, or optimizing it so that they find their best matches several months down the line, or trying to get everyone to pre-pay for the service rather than making the product experience awesome.

For marketplaces:
Why do customers sell on a marketplace? To make money and get rid of their stuff. Why do customers buy on a marketplace? So that they can get things cheaply and quickly, and are happy with their purchase. Thus, measure the quantity of how much your sellers take home, and how many buyers are happy with their experiences. Contrast this with overfocusing on listing fee revenues, which might get you into a spiral of raising prices rather than creating the best commerce experience.

For social networks:
Why do customers use social networks? To “connect” with their friends – let’s boil that down to communicating (though it’s obviously much richer than that). Then ideally, you might want to focus on the number of messages/comments/posts that end up getting replies from their friends. If you overfocus on something like user registrations, then you might get a ton of users, but maybe they won’t be getting to experience the benefits of the product.

For online publishers who sell to advertisers:
Why do advertisers buy ads on websites? To generate traffic to their own sites, which in turn leads to revenue. In this case, you should quantify the amount of revenue you generate for your advertiser customers, or at least the number of conversions they receive. Contrast this with the approach of measuring and optimizing your own CPMs as a publisher, which results in potentially delivering a lot of crappy traffic to advertisers who will drop their payments in the long term.

Special note for ad-driven startups :-)
Now ad-supported startups have a particularly interesting issue in this, because I keep using the words “benefits” and “customers” and perhaps it’d be easy to think this refers to the users of the product. But maybe not, as I’ve outlined before in Your ad-supported Web 2.0 site is actually a B2B enterprise in disguise. The reason is that your customer may actually be the advertiser on the site, not your user!

And in fact, if you overfocus on pleasing your users to the detriment of your advertiser customers, which is very easy – then that leads to very bad things.

Start this benefits-driven approach now, not later, so you can learn the right things
Finally, I want to emphasize that I believe it’s important to start thinking about these benefit-driven metrics from the beginning of your business, not later. The reason is that every learning that a startup makes is often hugely applicable to a specific context, but not at all applicable to other variations.

If you’re going to start a website that churns users like crazy but hits massive user goals, you will build an entire organization to optimize for those metrics. And once you’ve gone far on this, it’s not clear that you’ll have the DNA, the technology, the ideas, or the willpower to execute in a different direction.

Want more?
If you liked this post, please subscribe or follow me on Twitter. You can also find more essays here.

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

Creating value versus optimizing revenue


Revenue stems from value, not the other way around
One of the big thematic issues that has been referenced numerous times by myself, Eric Ries, Mike Speiser, and others is the limitations of quantitative testing in building a business. In particular, several objections have been mentioned:

  • Over-optimizing leads to local maxima, particularly in product design
  • Focusing too much on pageviews/uniques ignores actual product/market fit
  • Relying on quantitative models leads to anti-innovative behavior
  • etc.

For all the data geeks reading my blog – my opinion on all of this is, these are absolutely all true, and are all very important and relevant conversations that every data-driven startup needs to be having. Are you having them?

All of these have gotten me focused on one of the core questions of any business: What value are you actually creating?

Distribution-led approaches can lead to local maxima on value creation
Many new companies in this age of quantitative virality easily fall into hitting local maxima on value creation, all for very good reasons. By focusing on viral invites, addressbook scraping, A/B testing, and other techniques, you end up getting a big inflow of traffic and your question becomes, “What is the best product I can make to keep all these users around?”

There become three major temptations:

  1. First, there’s a huge desire to build as efficiently as possible. That is, build in just enough to satisfy the user, but don’t overpolish
  2. Similarly, there’s a big temptation to build for the lowest common denominator, because you’re trying to appeal to a huge audience. As a result, a lot of designs err towards persistent low-brow internet “recipes” – like quizzes, polls, forums, and other mechanics
  3. In addition, your product veers towards a portfolio of experiments, rather than one cohesive experience. After all, you’re still trying stuff out, and it’s a lot easier to add a new feature or use a crazy headline to get people to your site, rather than really going through the difficult synthesis process that’s at the heart of every design discussion

As a result of all of this, it’s very easy to build a shitty product that generates small to medium value, but doesn’t do something amazing.

I won’t go too much into the solution of how to solve this, but I think the key thing to think about is that the quantitative lean philosophy doesn’t allow you to skip the difficult process of coming up with a hugely value-creating product. You still have to do it, but you have a framework in which to think about the process.

Maximizing the source rather than your share
Another issue in all of this is that the focus for quantitatively driven companies ends up being on outputs rather than inputs. For example, it’s easy to start to optimize traffic as an entity in itself, rather than thinking about the fact that traffic comes out of product/market fit. Or similarly, you can optimize revenue, but I think it’s misguided to do it without considering the fact that you have to be creating value for whoever is paying you.

Thus, one can argue that “value creation” is the ultimate source for all of these secondary variables like revenue, traffic, etc. And you can make the decision to focus on extracting as much as possible from the secondary variables, but you become fundamentally limited by the primary value creation process within your product.

Another way to think of this is that ultimately, every product creates a bunch of “value” (however you want to define it) and then you end up taking some % of that value back as revenue. Abstractly, this is true regardless of whether your product is ad-based, freemium, or otherwise. If you think about things this way, the following two approaches are fundamentally different strategies:

  1. Create a massive amount of value, and capture a small amount
  2. Create a moderate amount of value, and totally dominate the economics

(and obviously this is a spectrum as well)

I would put companies like Wikipedia, Craigslist, Open Source, and others as extreme examples of #1. And unfortunately, I think a lot of short-lived apps on Facebook are really more or less examples of #2.

I think this is why, for people who question the value of internet companies like Facebook and Twitter, the natural thing to ask is, are these companies generating real value? If they are, I think the process of turning that into cash is much easier than the process of creating the huge value in the first place!

The biggest value drivers are qualitative
So the question then becomes, how do you systematically create value? I think that this is a very hard question, and one that it may be difficult to use quantitative tools to define, because the biggest value drivers are often qualitative. They are things like:

  • What’s does your product do?
  • Who’s your customer?
  • Why do people give you money?
  • etc.

Now, a lot of these you can turn from qualitative to quantitative. After all, after you build your product, you can generate hypotheses around how people ought to use it and make it better in the most common flows, by optimizing the page flows. Similarly, you can figure out how much money you should charge for something.

Yet simultaneously, the process of figuring out the core product requires the entrepreneur to have an opinion, perhaps one that is difficult to test or takes many years to test. And whether you do this quantitatively is its own thing – after all, companies like IDEO have a very evidence-driven design process, but is one that uses qualitative evidence gathering to generate the product prototypes.

Looking at landing page optimization as a value creator
I think that this entire perspective about maximizing value creation rather than optimizing outputs leads to a lot of interesting, subtle changes in how you approach things. Let’s take landing page optimization as an example of this.

Typically, the entire discussion around landing page optimization is just one about conversion rates, and all the different possible candidates to get to a conversion. Instead of this perspective, you might ask: What value does an optimized landing page generate in the first place? Ultimately, I think this optimization makes it so that people can grok what they’re signing up for better. It helps them scan the page better for relevant pieces of information. And it could make them less confused about the page they’re on.

Compare that line of thought with, “hey, let’s make a lot of random headlines and see what people react to” as two different ways to approach the same problem, with one prioritizing value creation and the other prioritizing the output (conversion rates in this case).

Looking at viral loops as a value creator
Same for viral loops, and the process of getting people to invite their friends to a site. If you are sincerely value-oriented, then the entire question is:

  • why do people WANT to invite their friends to the site?
  • how does having your friends on the site make the product a better experience?
  • what conveniences can you build in to make people expose their friends to the process?
  • etc.

Contrast this to a perspective that an outcome like the viral factor is all you care about optimizing, and however you can get that number >1.0, then the better off you are. I think that this numbers-centric model absolutely can lead to viral websites and apps, but also sucks at actually creating a huge base of value that you can recoup later.

Conclusion
My point with all of the above is simple: No matter what your product is, the only way to make money long-term is to make a lot of people happy, and then getting some % of the value you created back, in return. The right strategy to build a long-term sustainable business is to build long-term sustainable value. No amount of viral tricks or optimization will allow you to escape that truth!

Want more?
If you liked this post, please subscribe or follow me on Twitter. You can also find more essays here.

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

Twitter Weekly Updates for 2009-06-08

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

Twitter Weekly Updates for 2009-06-01

Twitter Weekly Updates for 2009-05-25

How do you do concrete interviews for non-technical people?

Twitter Weekly Updates for 2009-05-18

I want more tools to reach my readers, not monetize them!

Dear readers, I need your help!

Talk to your target customer in 4 easy steps

Bay Area investors I’m following on Twitter

3 key ideas from a recent Freemium dinner conversation

“Stealing MySpace” and my personal experience monetizing MySpace ads

Designing and Testing an Ad Product: 5 lessons learned form imeem’s audio ads (Guest Post)

Will social payment platforms really work long-term? (Guest post by Jay Weintraub)

Video: Panel on “Monetization and Business Models for Flash Games”

Friends versus Followers: Twitter’s elegant design for grouping contacts

Random links for week of March 16th

App monetization: Gambit launches, funnel metrics, and ARPU vs “CPM”

Free to Freemium: 5 lessons learned from YouSendIt.com

Twitter links for March 4, 2009

Bay Area entrepreneurs that I’m following on Twitter

5 warning signs: Does A/B testing lead to crappy products?

Which startup’s collapse will end the Web 2.0 era?

10% off for Flash Gaming Summit, March 22nd in San Francisco

Warren Buffett’s bio “The Snowball” and lessons for startups

Returning from my blogging vacation + Twitter links

Great iPhone preso on AppStore retention curves, pricing strategies, engagement metrics, etc.

How to create a profitable Freemium startup (spreadsheet model included!)

Twitter links for Jan 19, 2009

2009 conference schedule for the digital media industry

Will be in Seattle on Jan 8th and 9th

Freemium business model case study: AdultFriendFinder ARPU, churn, and conversion rates

Twitter links (while still on blog vacation!)

Taking a quick blog vacation…

Video up from Virtual Goods Summit, Metrics for Virtual Goods Businesses: The Whirled Case Study

How to calculate cost-per-acquisition for startups relying on freemium, subscription, or virtual items biz models

Amazon Associates stats for this blog

The Secret History of Silicon Valley (next Thurs, Nov 20)

List of essays from this blog is now up to date!