@andrewchen

Get the newsletter · 2018 essays (PDF) · Featured · Recent

How to generate awesome test candidates for A/B testing

A/B testing is fun ;-)
Generating candidates for A/B testing can be a great source of entertainment for your team. It’s easy and fun to generate dozens of potential candidates for a new headline, subtitle, picture, or other element of an important page. It’s also a great exercise in both the qualitative, consumer psychology skillsets required, as well as the quantitative set.

I’ve gather a couple rules of thumb in helping you generate good candidates for A/B testing below:

  • Brainstorm the RIGHT way
  • Dive down into potential customer motivation
  • Go for high variariance approaches
  • Test big things first, smaller things later

Let’s dive into each of these…

Brainstorm the RIGHT way
First off, not all brainstorming is created equal – you want to make sure you are going for lots of quantity, that the most senior person in the room doesn’t “run” the whiteboard, and a bunch of other guidelines that you can find in this article on IDEO’s brainstorming techniques. I generally find that after brainstorming individually on dozens of candidates, you can build on very interesting themes and start to coalesce the entire process.

Dive down into potential customer motivation
One important issue is that every product and every page within your product likely caters to multiple needs. Influence, the classic book on persuasion by Robert Cialdini, enumerates many of them. Is it:

  • Reciprocation
  • Commitment and consistency
  • Social proof
  • Liking
  • Authority
  • Scarcity
  • etc?

Or alternatively, you may have specific ideas about value propositions or user emotions – for example, a social network like MySpace could be marketed around:

  • Customizing profiles
  • Socializing and friends
  • Media content
  • Photos and blogs
  • etc.

Who knows which feature is king? The point is, each one of these potential customer motivations and values probably deserves at least one, if not several, test candidates in your A/B test. The fundamental emotions driving your product have a huge likelihood chance of altering the outcomes of your split tests.

Go for high variariance approaches
Similarly, life is too short for the safe stuff. Because of the fact that you throw away all the bad candidates and keep the good ones, it’s in your best interest to try to make the good ones as good as possible! As a result, make sure you try to go for extremely polarizing, high-variance approaches.

For example, make sure you try candidates that:

  • … are aggressive and in your face
  • … use different graphical elements like videos versus text versus audio
  • … are varied in length, like very very long or very short
  • … may offend certain subsets of your audience
  • … are commanding and direct the user

Typically, in an A/B test I will usually have a control, then a candidate that incrementally improves on the control, and then a couple candidates from left-field. As you try out more candidates and learn from the process, then often times you will start going with more incremental stuff to finish your optimization.

But early on in your experimentation process, remember to go wild!

Test big things first, smaller things later
Similarly, make sure that you prioritize the your tests so that you aren’t testing subtitles and paragraph copy when you could be trying out even more extreme stuff. Things like the user flow, the layout, “hero shots,” and other factors are usually much more important than smaller things like icons or specific sub-labels for forms.

As a result, oftentimes the best thing to do is to rush out some forms to test, then make things prettier and more finalized from there.

Anyway, I hope this was helpful, and if you have more ideas, please comment and suggest more ideas!

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

Views expressed in “content” (including posts, podcasts, videos) linked on this website or posted in social media and other platforms (collectively, “content distribution outlets”) are my own and are not the views of AH Capital Management, L.L.C. (“a16z”) or its respective affiliates. AH Capital Management is an investment adviser registered with the Securities and Exchange Commission. Registration as an investment adviser does not imply any special skill or training. The posts are not directed to any investors or potential investors, and do not constitute an offer to sell -- or a solicitation of an offer to buy -- any securities, and may not be used or relied upon in evaluating the merits of any investment.

The content should not be construed as or relied upon in any manner as investment, legal, tax, or other advice. You should consult your own advisers as to legal, business, tax, and other related matters concerning any investment. Any projections, estimates, forecasts, targets, prospects and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Any charts provided here are for informational purposes only, and should not be relied upon when making any investment decision. Certain information contained in here has been obtained from third-party sources. While taken from sources believed to be reliable, I have not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation. The content speaks only as of the date indicated.

Under no circumstances should any posts or other information provided on this website -- or on associated content distribution outlets -- be construed as an offer soliciting the purchase or sale of any security or interest in any pooled investment vehicle sponsored, discussed, or mentioned by a16z personnel. Nor should it be construed as an offer to provide investment advisory services; an offer to invest in an a16z-managed pooled investment vehicle will be made separately and only by means of the confidential offering documents of the specific pooled investment vehicles -- which should be read in their entirety, and only to those who, among other requirements, meet certain qualifications under federal securities laws. Such investors, defined as accredited investors and qualified purchasers, are generally deemed capable of evaluating the merits and risks of prospective investments and financial matters. There can be no assurances that a16z’s investment objectives will be achieved or investment strategies will be successful. Any investment in a vehicle managed by a16z involves a high degree of risk including the risk that the entire amount invested is lost. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by a16z is available at https://a16z.com/investments/. Excluded from this list are investments for which the issuer has not provided permission for a16z to disclose publicly as well as unannounced investments in publicly traded digital assets. Past results of Andreessen Horowitz’s investments, pooled investment vehicles, or investment strategies are not necessarily indicative of future results. Please see https://a16z.com/disclosures for additional important information.