@andrewchen

Get the newsletter · 2018 essays (PDF) · Featured · Recent

omg I’m just a startup, I can’t do those fancy analytics!

Reader comments on user retention post
In my previous post on user retention strategies from the catalog marketing world, I received a number of comments commenting on the difficulty of implementing analytics systems when you don’t have the resources of a F500 company.

Another Andrew Chen writes:

I agree with the general sentiment of the post, but how would you suggest a startup deal with the problem? Without the marketing budgets of larger companies and without the time to do detailed analysis of its customers, how can a startup find the best messaging and segmentation?

 

Similarly, Rachanap writes:

RFM is a good idea in theory, however, even in traditional marketing world, like in CPG, companies don’t have the bandwidth to adopt it, let alone successfully implement it. Implementing such a truly effective and scalable marketing system needs a tool in place for segmenting the data, an then group the users (often needing expensive software) as well as additional infrastructure to capture that data, neither of which is an easy task. For startups this would be difficult, given their low bandwidth.

In essence, the question is, how do you take advantage of all of these different analytics systems given the constrained resources of a startup. Good question. Let’s tackle this below.

The role of analytics in startup decision-making
In general, a philosophy on the role of analytics within a startup is:

If you’re not going to do something about it, it may not be worth measuring.

(Similarly, if you want to act to improve something, you’ll want to measure it)

Don’t build metrics that aren’t going to be part of your day-to-day operations or don’t have potential to be incorporated as such. Building reports that no one looks at is just activity without accomplishment, and is a waste of time.

So instead, I recommend a “layers of onion” approach on figuring out what analytics you require. Early on, your goal may be to focus on creating a solid product that people like and stay engaged on. Obviously you could slap in Google analytics, but at the very least I’d recommend cohort-based analysis to get an actual feel for how well your products are retaining people. Similarly, you might reach a point where you want to be focused on traffic, in which case you’ll want to make sure you properly instrument to capture your viral loop. Later on, you’ll want to do the same for your ad inventory, to figure out what segments of audience and what sections of your site monetize the best.

The point is, develop the necessary metrics alongside whatever feature development that makes sense, don’t do any more than you need. Now when you are far enough along that segmenting your users based on behavior matters – likely this only because relevant once you hit a certain user acquisition threshold – then it may be important to implement RFM-based segmentation. Or not. It just depends on your product goals.

Metrics as a “product tax”
In fact, one way to view analytics is that they are a double-digit “tax” on your product development process because of a couple things:

  • It takes engineers lots of time and development effort
  • It produces numbers that people argue about
  • It requires machines, serious infrastructure, its own software, etc
  • Fundamentally, it slows down your feature development

As a rough estimate, I’ve found that it takes between 25-40% of your resources to do analytics REALLY well. So for every 3 engineers working on product features, you’d want to put 1 just on analytics. This may seem like a ton (and it is), but it throws off indispensible knowledge that you can’t get elsewhere, like:

  • Validating your assumptions
  • Pinpointing bottlenecks and key problems
  • Creating the ability to predict/model your business to make future decisions
  • It tells you which features actually are good and what features don’t matter

Question: Is it better to build 10 features where you don’t know what worked and what didn’t, or is it better to build LESS features but have a clear sense for what and why something worked? In my opinion, you want to learn as much as you can so you can “run up the score” on the features that work.

Prioritizing metrics development
The key to this philosophy is figuring out how to prioritze the metrics that you build relative to your features. Again, you want to only develop the analytics that you need to build out your product correctly – no more, no less.

In fact, it’s often wise to make a distinction between Operational reports versus Investigative reports,
each of which have their own goals and structure. The operational reports should reflect all the issues you care about in your business, and the investigative stuff generally satisfies curiosity (which is always a good thing).

So for folks with startups, I’d encourage you to ask: What are the outcomes that you care about? What are the assumptions you are making? If you’re a startup that’s spending copious amounts of time building features X, Y, and Z, what are you making the assumption of? That these features will make people want to extend their engagement times on the site? That these features will build word-of-mouth? What are the metrics that would prove you right or wrong in your assumptions?

Let’s discuss an example of a sample roadmap for a online photo-site.

Example product roadmap
I’ll close out this post with an example roadmap for a product that roughly had the same featureset as Imageshack.us. You may end up making these product decision out-of-order of what I have them, but the philosophy remains the same: Build analytics only when you need them, and align them to the key efforts behind your product development process.

Here’s an example sequencing of product development:

1) Core product and features

  • Obviously, you want people to be able to upload a photo and give a URL for it
  • These photos then can be displayed on other sites

1a) Analytics for core product

  • First add Google analytics since it’s free :-)
  • Build out a cohort matrix that tells you how many users join each week, how many come back, and what they do during their visits
  • Where are these images being uploaded to? A breakdown of image displays by the domain they were displayed on would be nice

2) Working on growing traffic

  • Once you have the core product done, perhaps you want to make some whizbang features to grow traffic
  • You might make a photo slideshow, or some awesome photo effects, etc.
  • Also, you might want to make one-click integration with Typepad/MySpace/etc via something like Gigya

2a) Analytics for traffic growth features

  • One key issue to measure is how much additional traffic is gained by these different methods. You may want to start annotating visits that come to your site based on what kind of widget or photo created it. Do sparkly photo galleries generate the most visits? Why? Make a report
  • Second feature might be an A/B testing framework. If you divide two groups, A and B, and show them different featuresets to see how the traffic grows for each

 

etc.

UPDATE: Thanks to Jeremy for the link and additional thoughts on spec’ing out metrics in the product process. Check them out.

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

Views expressed in “content” (including posts, podcasts, videos) linked on this website or posted in social media and other platforms (collectively, “content distribution outlets”) are my own and are not the views of AH Capital Management, L.L.C. (“a16z”) or its respective affiliates. AH Capital Management is an investment adviser registered with the Securities and Exchange Commission. Registration as an investment adviser does not imply any special skill or training. The posts are not directed to any investors or potential investors, and do not constitute an offer to sell -- or a solicitation of an offer to buy -- any securities, and may not be used or relied upon in evaluating the merits of any investment.

The content should not be construed as or relied upon in any manner as investment, legal, tax, or other advice. You should consult your own advisers as to legal, business, tax, and other related matters concerning any investment. Any projections, estimates, forecasts, targets, prospects and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Any charts provided here are for informational purposes only, and should not be relied upon when making any investment decision. Certain information contained in here has been obtained from third-party sources. While taken from sources believed to be reliable, I have not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation. The content speaks only as of the date indicated.

Under no circumstances should any posts or other information provided on this website -- or on associated content distribution outlets -- be construed as an offer soliciting the purchase or sale of any security or interest in any pooled investment vehicle sponsored, discussed, or mentioned by a16z personnel. Nor should it be construed as an offer to provide investment advisory services; an offer to invest in an a16z-managed pooled investment vehicle will be made separately and only by means of the confidential offering documents of the specific pooled investment vehicles -- which should be read in their entirety, and only to those who, among other requirements, meet certain qualifications under federal securities laws. Such investors, defined as accredited investors and qualified purchasers, are generally deemed capable of evaluating the merits and risks of prospective investments and financial matters. There can be no assurances that a16z’s investment objectives will be achieved or investment strategies will be successful. Any investment in a vehicle managed by a16z involves a high degree of risk including the risk that the entire amount invested is lost. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by a16z is available at https://a16z.com/investments/. Excluded from this list are investments for which the issuer has not provided permission for a16z to disclose publicly as well as unannounced investments in publicly traded digital assets. Past results of Andreessen Horowitz’s investments, pooled investment vehicles, or investment strategies are not necessarily indicative of future results. Please see https://a16z.com/disclosures for additional important information.