@andrewchen

Subscribe · Featured · Recent · The Cold Start Problem 📘

Why are we so bad at predicting startup success?

Startups and bad predictions
One of my favorite reads this year was Nate Silver’s The Signal and the Noise which has the subtitle “Why so many predictions fail, but some don’t.” It covers a ton of different topics, from weather to politics to gambling, and I couldn’t help but read it with a startup/tech point of view.

After all, the industry of technology startups is all about prediction- we try to predict what will be a good market, what will be a good product, as we “iterate” and “pivot” on our predictions. And of course the business of venture capital is even more directly about knowing how to pick winners- especially the seed and Series A investments.

And yet, we’re all so bad at predicting what will work and what won’t. I’ve written about my embarrassing skepticism about Facebook, but hey, I’m just a random tech guy. For the folks whose job it is to professionally pick winners, the venture capitalists, they aren’t doing very well either. It’s been widely noted that the venture capital asset class, after fees, has lagged the public markets- you’d be better off buying some index funds.

Startup exceptionalism = sparse data sets = shitty prediction models
One of the most challenging aspects of predicting the next breakout startup is that there’s so few of them. It’s been widely discussed that 10-15 startups a year generate 97% of the returns in tech, and each one seems like a crazy exception. And as an industry we get myopically focused on each one of them.

Watch Ben Horowitz elaborate on the sobering stats, starting at the 38:00 minute mark:

With these kinds of odds, our brains go crazy with pattern-matching. When a once-in-a-generation startup like Google comes around, for the next few years after that, we all ask, “OK, but do you have any PhDs on the team? What’s the ‘PageRank’ of your product?” And now that we have AirBnb, we’ve gone from being skeptical of designer-led companies to being huge fans of them. With so few datapoints, the prediction models we generate as a community aren’t great- they’re simplistic and are amplified with the swirl of attention-grabbing headlines and soundbites.

These simplistic models result in generic startup advice. As I wrote about earlier, there’s a whole ecosystem of vendors, press, consultants, and advisors who go on advice autopilot and give the same advice regardless of situation. Invest in great UX, charge users right away, iterate quickly, measure everything, launch earlier, work long hours, raise more money, raise less money – all of these ideas are helpful to complete newbies but dangerous when applied recklessly to every situation.

We all know how to parrot this common wisdom, but how do we know when we’re hearing good versus bad advice? If you think about the idea that there’s 10-15 companies every year who are breakouts, how many people really have first-hand experience making the right decisions to start and build breakout companies?

Hedgehogs and pundits
I was reminded for my dislike of generic startup advice when in his book, Nate Silver writes about hedgehogs versus foxes and their approaches towards generating predictions – here’s the Wikipedia definition on the concept:

[There are] two categories: hedgehogs, who view the world through the lens of a single defining idea and foxes who draw on a wide variety of experiences and for whom the world cannot be boiled down to a single idea.

Silver clearly identifies as a fox, and contrasted his approach to the talking head pundits that dominate political talk shows on TV and radio. For the pundits, the more aggressive, contrarian, and certain they seem, the more attention-grabbing they are. Rather similar to what we see in the blogosphere, where people are rewarded for writing headlines like “10 reasons why [hot company] will be killed by [new product].” Or “Every startup should care about [metric X]” or whatever.

This hedgehog-like behavior is amplified by the fact that there’s always pressure to articulate a thesis on what’s going on in the market. People in the press are always trying to spot trends or boil down complex ideas, and investors are constantly asked, “What kinds of startups are you investing in? Why?” And entrepreneurs are always forced to fit their businesses into the broader trends of the market, to find sexy competitors, all in the change to find a simple narrative that describes what’s going on.

The solution to all of this isn’t easy- to be a fox means to draw from a much broader set of data, to look at the problem from multiple perspectives, and to reach a conclusion that combines all of those datapoints. There’s been some great work on the science of forecasting by Philip Tetlock of UPenn, who’s set up an open contest to study good forecasting here. There’s an interview of him Edge.org here and a video describing some of his academic research below:

Worth watching.

My personal experience  
Over my 5 years in Silicon Valley, the biggest lesson I’ve learned from trying to predict startups is calibration. They talk about it in the video above, but the short way to describe it is to be careful with what you think you know versus what you don’t. I’ve found that my area of expertise where I can make good decisions is actually pretty narrow- I’ve done a bunch of work in online ads, analytics, consumer communication/publishing, and I think my judgement is pretty good there, but it’s much shakier outside of that area.

When I do an analysis, I try to match my delivery with how much I think I know- and these days, it means that they sound a lot more tentative than the younger, brasher version of myself when I first came to SF. I’ve also tried to be diligent in my employment of “advice autopilot” – if I meet with entrepreneurs and find myself saying the same thing multiple times, then I try to refine the idea to take into account the specifics and nuances of that product. It’s easier, lazier, but less helpful to just say the same thing over and over again.

Be the fox, not the hedgehog.

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

Views expressed in “content” (including posts, podcasts, videos) linked on this website or posted in social media and other platforms (collectively, “content distribution outlets”) are my own and are not the views of AH Capital Management, L.L.C. (“a16z”) or its respective affiliates. AH Capital Management is an investment adviser registered with the Securities and Exchange Commission. Registration as an investment adviser does not imply any special skill or training. The posts are not directed to any investors or potential investors, and do not constitute an offer to sell -- or a solicitation of an offer to buy -- any securities, and may not be used or relied upon in evaluating the merits of any investment.

The content should not be construed as or relied upon in any manner as investment, legal, tax, or other advice. You should consult your own advisers as to legal, business, tax, and other related matters concerning any investment. Any projections, estimates, forecasts, targets, prospects and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Any charts provided here are for informational purposes only, and should not be relied upon when making any investment decision. Certain information contained in here has been obtained from third-party sources. While taken from sources believed to be reliable, I have not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation. The content speaks only as of the date indicated.

Under no circumstances should any posts or other information provided on this website -- or on associated content distribution outlets -- be construed as an offer soliciting the purchase or sale of any security or interest in any pooled investment vehicle sponsored, discussed, or mentioned by a16z personnel. Nor should it be construed as an offer to provide investment advisory services; an offer to invest in an a16z-managed pooled investment vehicle will be made separately and only by means of the confidential offering documents of the specific pooled investment vehicles -- which should be read in their entirety, and only to those who, among other requirements, meet certain qualifications under federal securities laws. Such investors, defined as accredited investors and qualified purchasers, are generally deemed capable of evaluating the merits and risks of prospective investments and financial matters. There can be no assurances that a16z’s investment objectives will be achieved or investment strategies will be successful. Any investment in a vehicle managed by a16z involves a high degree of risk including the risk that the entire amount invested is lost. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by a16z is available at https://a16z.com/investments/. Excluded from this list are investments for which the issuer has not provided permission for a16z to disclose publicly as well as unannounced investments in publicly traded digital assets. Past results of Andreessen Horowitz’s investments, pooled investment vehicles, or investment strategies are not necessarily indicative of future results. Please see https://a16z.com/disclosures for additional important information.