@andrewchen

Get the newsletter · 2018 essays (PDF) · Featured · Recent

Know the difference between data-informed and versus data-driven

Metrics are merely a reflection of the product strategy that you have in place
Data is powerful because it is concrete. For many entrepreneurs, particularly with technical backgrounds, empirical data can trump everything else – best practices, guys with fancy educations and job titles – and for good reason. It’s really the skeptic’s best weapon, and it’s been an important tool in helping startups solve problems in new and innovative ways.

It’s easy to go too far – and that’s the distinction made between “data-informed” versus “data-driven,” which I originally heard at a Facebook talk in 2010 (included underneath the post). Ultimately, metrics are merely a reflection of the product strategy that you already have in place and are limited because they’re based on what you’ve already built, which is based on your current audience and how your current product behaves. Being data-informed means that you acknowledge the fact that you only have a small subset of the information that you need to build a successful product. After all, your product could target other audiences, or have a completely different set of features. Data is generated based on a snapshot based on what you’ve already built, and generally you can change a few variables at a time, but it’s limited.

This means you often know how to iterate towards the local maximum, but you don’t have enough data to understand how to get to the best outcome in the biggest market.

This is a messy problem, don’t let data falsely simplify it
So the difference between data-informed versus data-driven, in my mind, is that you weigh the data as one piece of a messy problem you’re solving with thousands of constantly changing variables. While data is concrete, it is often systematically biased. It’s also not the right tool, because not everything is an optimization problem. And delegating your decision-making to only what you can measure right now often de-prioritizes more important macro aspects of the problem.

Let’s examine a couple ways in which a data-driven approach can lead to weak decision-making.

Data is often systematically biased in ways that are too expensive to fix
The first problem with being data-driven is that the data you can collect is often systematically biased in unfixable ways.

It’s easy to collect data when the following conditions are met:

  • You have a lot of traffic/users to collect the data
  • You can collect the data quickly
  • There are clear metrics for what’s good versus bad
  • You can collect data with the product you have (not the one you wish you had)
  • It doesn’t cost anything

This type of data is good for stuff like, say, signup %s on homepages. They are often the most trafficked parts of the site, and there’s a clear metric, so you can run an experiment in a few days and get your data back quickly.

In contrast, if you are looking to measure long-retention rates, that’s much more difficult. Or long-term perceptions of your user experience, or trying to measure the impact of an important but niche feature (like account deletion). These are all super difficult because they take a long time, or are expensive, or are impossible datapoints to collect – people don’t want to wait around for a month to see what their +1 month retention looks like.

And yet, oftentimes these metrics are exactly the most important ones to solve.

Worse yet, consider the cases where you take a “data-driven” mindset and try to trade off the metrics between concrete datapoints like signup %s versus long-term retention rates. It’s difficult for retention to ever win out, unless you take a more macro and enlightened perspective on the role of data. Short- vs long-term tradeoffs require deep thinking, not shallow data!

Not everything is an optimization problem
At a more macro level, it’s also important to note that the most important strategic issues are not optimization problems. Let’s start at the beginning, when you’re picking out your product. You could, for example, build a great business targeting consumers or enterprises or SMBs. Similarly, you can build businesses that are web-first (Pinterest!) or mobile-first (Instagram!) and both be successful. These are things where it might be nice to have a feel for some of the general parameters, like market size or mobile growth, but ultimately they are such large markets that it’s important to make the decision where you feel good about it. In these cases, you’re forced to be data-informed but it’s hard to be data-driven.

These types are strategy questions are especially important when the industry is undergoing a disruptive innovation, as discussed in Innovator’s Dilemma. In the book, Clayton Christensen discusses the pattern of companies who are successful and build a big revenue base in one area. They find that it’s almost always easier to increase their core business by 10% than it is to create a new business to do the same, but this thinking eventually leads to their demise. This happened in the tech industry from mainframes vs PCs, hardware vs software, desktop vs web, and web vs mobile now. The incumbents are doing what they think is right- listening to their current customer base, improving revenues from a % basis, and in general trying to do the most data-driven thing. But without a vision for how the industry will evolve and improve, the big guys are eventually disrupted.

Leverage data in the right way
It’s important to leverage data the same way, whether it’s a strategic or tactical issue: Have a vision for what you are trying to do. Use data to validate and help you navigate that vision, and map it down into small enough pieces where you can begin to execute in a data-informed way. Don’t let shallow analysis of data that happens to be cheap/easy/fast to collect nudge you off-course in your entrepreneurial pursuits.

Facebook on data-informed versus data-driven
I leave you with the Facebook video that inspired this post in the first place – presented by Adam Mosseri. He uses the example of multiple photo uploads, and how they use metrics to optimize the workflow. Watch the video embed below or go to YouTube.

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

Views expressed in “content” (including posts, podcasts, videos) linked on this website or posted in social media and other platforms (collectively, “content distribution outlets”) are my own and are not the views of AH Capital Management, L.L.C. (“a16z”) or its respective affiliates. AH Capital Management is an investment adviser registered with the Securities and Exchange Commission. Registration as an investment adviser does not imply any special skill or training. The posts are not directed to any investors or potential investors, and do not constitute an offer to sell -- or a solicitation of an offer to buy -- any securities, and may not be used or relied upon in evaluating the merits of any investment.

The content should not be construed as or relied upon in any manner as investment, legal, tax, or other advice. You should consult your own advisers as to legal, business, tax, and other related matters concerning any investment. Any projections, estimates, forecasts, targets, prospects and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Any charts provided here are for informational purposes only, and should not be relied upon when making any investment decision. Certain information contained in here has been obtained from third-party sources. While taken from sources believed to be reliable, I have not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation. The content speaks only as of the date indicated.

Under no circumstances should any posts or other information provided on this website -- or on associated content distribution outlets -- be construed as an offer soliciting the purchase or sale of any security or interest in any pooled investment vehicle sponsored, discussed, or mentioned by a16z personnel. Nor should it be construed as an offer to provide investment advisory services; an offer to invest in an a16z-managed pooled investment vehicle will be made separately and only by means of the confidential offering documents of the specific pooled investment vehicles -- which should be read in their entirety, and only to those who, among other requirements, meet certain qualifications under federal securities laws. Such investors, defined as accredited investors and qualified purchasers, are generally deemed capable of evaluating the merits and risks of prospective investments and financial matters. There can be no assurances that a16z’s investment objectives will be achieved or investment strategies will be successful. Any investment in a vehicle managed by a16z involves a high degree of risk including the risk that the entire amount invested is lost. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by a16z is available at https://a16z.com/investments/. Excluded from this list are investments for which the issuer has not provided permission for a16z to disclose publicly as well as unannounced investments in publicly traded digital assets. Past results of Andreessen Horowitz’s investments, pooled investment vehicles, or investment strategies are not necessarily indicative of future results. Please see https://a16z.com/disclosures for additional important information.