@andrewchen

Get the newsletter · Featured · Recent

A Practitioner’s Guide to Net Promoter Score

[Dear readers, this essay is about the practical aspects of measuring Net Promoter Score, an important metric that often correlates strongly with word-of-mouth virality. Sachin Rekhi, the author, has a blog and can be found on Twitter at @sachinrekhi. He’s was most recently Director of Product at Linkedin, leading the Sales Navigator product, and previously, an Entrepreneur-in-Residence at Trinity Ventures. Most importantly, Sachin married my sister Ada after meeting her in college at UPenn :) -Andrew]

Sachin Rekhi (ex-LinkedIn):
A Practitioner’s Guide to Net Promoter Score

18853007256_1c7ee52ee5_z

Over the past year at LinkedIn I developed a strong appreciation for using Net Promoter Score (NPS) as a key performance indicator (KPI) to understand customer loyalty. In addition to the standard repertoire of acquisition, engagement, and monetization KPIs, NPS has become a great additional measure for understanding customer loyalty and ultimately an actionable metric for enhancing your product experience to deliver delight.

I wanted to share the best practices I’ve learned for implementing an NPS program within an organization to get the most out of this KPI for driving more delightful product experiences.

The Origin of NPS
Net Promoter Score (NPS) is a measure of your customer’s loyalty, devised by Fred Reichheld at Bain & Company in 2003. He introduced it in a seminal HBR article entitled The One Number You Need to Grow, which I highly recommend anyone serious about NPS to read in detail. Fred found NPS to be a strong alternative to long customer satisfaction surveys as it was such a simple single question to administer and was able to show correlation between NPS and long-term company growth.

How NPS is Calculated
NPS is calculated by surveying your customers and asking them a very simple question: “How likely is it that you would recommend our company to a friend or colleague?” Based on their responses on a 0 – 10 scale, group your customers into Promoters (9-10 score), Passives (7-8 score), and Detractors (0-6 score). Then subtract the percentage of detractors from the percentage of promoters and you have your NPS score. The score ranges from -100 (all detractors) to +100 (all promoters). An NPS score that is greater than 0 is considered good and a score of +50 is excellent.

Additional NPS Questions
In addition to asking the likelihood to recommend, it’s essential to also ask the open-ended question: “Why did you give our company a rating of [customer’s score]?” This is critical because it’s what turns the score from simply a past performance measure to an actionable metric to improve future performance.

It’s also helpful to ask how likely they are to recommend your competitor products or alternatives, so you can establish a benchmark for how your NPS score compares to others in your industry as there are substantial differences in scores by product category. Keep in mind though that these results are biased since you are sampling your own customers for these benchmarks instead of a random cross-section of potential customers, including those who have chosen competitive solutions.

Many ask additional questions to understand additional drivers of the customer’s score. These are optional as while they add value in understanding the results, they add complexity which reduces the response rate, so you need to consider the trade-off of doing so.

Collection Methods
NPS scores for online products are typically collected by sending the survey via email to your customers or through an in-product prompt to answer the survey. To maximize response rates, it’s important to offer the survey across both your desktop & mobile experiences. While you could create such a collection tool in-house, I encourage folks to use one of the NPS survey solutions out there that support collection and analysis across a variety of channels and interfaces, such as one offered by my wife Ada’s employer SurveyMonkey.

One challenge with both email and in-product based survey methodologies is they tend to bias responses to more engaged customers as less engaged users are likely not coming back to the product nor answering your company’s emails as frequently. We’ll talk about potentially addressing this below.

Sample Selection
It’s important to survey a random representative sample of your customers each NPS survey. While that may sound easy, we found cases in which the responses weren’t in fact random and it became important to control for this in sampling or analysis. For example, we found strong correlation between engagement and NPS results. Therefore it was important to ensure your sample in fact reflects the engagement levels of your actual overall user base. Similarly, we found a correlation between customer tenure and NPS results as well, thus another key factor to ensure the customer tenure in the sample similarly matches that of your overall user base.

Survey Frequency
When thinking about how frequently to administer an NPS survey, there are several key considerations. The first is the size of your customer base. The smaller your customer base, the larger sample you need to survey each time or even wait longer for more responses to achieve a higher response rate, which limits how frequently you can administer future surveys. The second consideration is associated with your product development cycle. Product enhancements end up being one way to drive increases in scores and therefore the frequency of score changes depends on how quickly you are iterating on your product to drive such increases. NPS tends to be a lagging indicator so it takes time even after you’ve implemented changes to the customer’s experience for them to internalize the changes and then reflect such changes in their scores. On my team at LinkedIn we found it best to administer our NPS survey quarterly, which aligned with our quarterly product planning cycle. This enabled us to have the most recent scores before going into quarterly planning and enabled us to react to any meaningful observations from the survey in our upcoming roadmap.

Analysis Team
If your goal is to use NPS to drive more delightful product experiences, it’s important that you have all the key stakeholders involved in product development as part of the NPS analysis team. Without this, the NPS survey rarely get’s used as a meaningful part of the product development lifecycle. For us at LinkedIn, this meant including product managers, product marketing, market research, and business operations in the core NPS team. We also broadly share the findings with the entire R&D team each quarter. While it will certainly depend on your own development process, it’s critical to ensure the right stakeholders are involved right from the beginning.

Verbatim Analysis
The most actionable part of the NPS survey is the categorization of the open-ended verbatim comments from promoters & detractors. Each survey we would analyze the promoter comments and categorize each comment into primary promoter benefit categories as well as similarly categorize each detractor comment into primary detractor issue categories. The categories were initially deduced by reading every single comment and coming up with the large themes across them. We conducted this analysis every quarter so we could see quarter-over-quarter trends in the results. This categorization became the basis of how we came up with roadmap suggestions to address detractor pain points and improve their overall experience. While it can be daunting to read every comment, there is no substitute for the product team digging in and really listening directly to the voice of the customer and how they articulate their experience with your product.

Promoter Drivers
While oftentimes folks spend a lot of time looking at NPS detractors and how to address their concerns, we found it equally helpful to spend time on promoters and understanding what was different about their experiences to make them successful. We correlated specific behavior within the product to NPS results (logins, searches, profile views, and more) and found a strong correlation between certain product actions and a higher NPS. This can help deduce what your product’s “magic moment” is when your users are truly activated and likely to derive delight from your product. Then you can focus on product optimizations to get more of your customer base to this point. The best way to get to these correlations is simply to look at every major action in your product and see if there are any clear correlations with NPS scores. It’s easy to just graph and see if this is the case.

Methodology Sensitivities
We found NPS to be sensitive to methodology changes in the questions being asked. So it’s incredibly important to be as consistent in your methodology across surveys. Only with a fully consistent methodology can you consider results comparable across surveys. The ordering of the questions matters. The list of competitors that you include in the survey matters. The sampling approach matters. Change the methodology as infrequently as possible.

Seasonality
We found that there may be some seasonality at play in certain quarters that effect NPS results, correlating with engagement seasonality. We’ve heard that this is even truer for other businesses. So it may end up being more important to compare year-over-year changes as opposed to quarter-over-quarter changes to ensure the effects of seasonality are minimized. While this may not be possible, it’s at least important to realize how this could be effecting your scores.

Limitations of NPS
While NPS is an effective measure for understanding customer loyalty and developing concrete action plans to drive it up, it does have it’s limitations that are important to understand:

1. The infrequency of NPS results make it a poor operational metric for running your day-to-day business. Continue to leverage your existing acquisition, engagement, and monetization dashboards for tracking regular performance as well as for conducting A/B tests and other optimizations.

2. Margin of error with the NPS results depend on your sample size. It can often be prohibitive to get large enough of a sample to significantly reduce the margin of error. So it’s important to be aware of this and not sweat small changes in NPS results between surveys. More classic measures like engagement that don’t require sampling have a far lower margin of error.

3. NPS analysis is not a replacement for product strategy. It’s simply a tool for understanding how your customers are perceiving your execution against your product strategy as well as provides concrete optimizations you can make to better achieve your already defined strategy.

[This essay was first published here. Sachin Rekhi, the author, has a blog and can be found on Twitter at @sachinrekhi. He’s was most recently Director of Product at Linkedin, leading the Sales Navigator product, and previously, an Entrepreneur-in-Residence at Trinity Ventures. ]

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

Quick announcement: The Backstory, a private discussion forum for tech, marketing, growth

News

Dear readers,

Some quick news- I’ve started some private discussion forums, as a complement to my writing. It’s easier to talk than write, so I figure it’s a good way to stay in touch while I’m between essays.

sf-big-v4

And surprise: It’s actually a year old, and has thousands of posts/discussions/users. I was drip sending invites to my mailing list for most of 2015, to build up content for an unveiling.

So try it out and let me know what you think.

Here’s the link to sign up.

Thanks,
Andrew
San Francisco, CA

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

Uber’s virtuous cycle. Geographic density, hyperlocal marketplaces, and why drivers are key

BpfMqKzCUAAL7Xw

Uber’s virtuous cycle
Back in 2014, David Sacks (ex-Paypal, Yammer, Zenefits) tweeted the above diagram to explain why Uber’s geographic density is the new network effect. It’s an insightful diagram that’s been built upon by Bill Gurley (Benchmark Capital and on Uber’s board) in his essay How to Miss By a Mile: An Alternative Look at Uber’s Potential Market Size.

Bill Gurley sums up Uber’s network effect as three major drivers:

  1. Pick-up times. As Uber expands in a market, and as demand and supply both grow, pickup times fall. Residents of San Francisco have seen this play out over many years. Shorter pickup times mean more reliability and more potential use cases. The more people that use Uber, the shorter the pick up times in each region.
  2. Coverage Density. As Uber grows in a city, the outer geographic range of supplier liquidity increases and increases. Once again, Uber started in San Francisco proper. Today there is coverage from South San Jose all the way up to Napa. The more people that use Uber, the greater the coverage.
  3. Utilization. As Uber grows in any given city, utilization increases. Basically, the time that a driver has a paying ride per hour is constantly rising. This is simply a math problem – more demand and more supply make the economical traveling-salesman type problem easier to solve. Uber then uses the increased utilization to lower rates – which results in lower prices which once again leads to more use cases. The more people that use Uber, the lower the overall price will be for the consumer.

Ben Thompson says it differently, with a competitive lens, in his essay Why Uber Fights, which is also a great compliment to Gurley’s piece.

The point to the above articles is super interesting. From a UX experience, Uber is “hit a button and a car comes,” but from a business standpoint, it’s a vast collection of hundreds of hyperlocal marketplaces in nearly 70 countries. Each marketplace is 2-sided, with riders and drivers, has its own network effects driven by pickup times, coverage density, and utilization.

Understanding the above has been one of my biggest lessons since joining Uber. There’s a lot of nuances in the business that come out of deeply grokking this perspective, and the run-on implications – especially the importance of drivers – are fundamental in understanding Uber and on-demand companies in general.

“More Drivers”
If I were to simplify my role at Uber, it’s pretty simple – in the diagram above, it’s figuring out how to get More Drivers. This is one of the foundational elements of Uber’s business, because as I mentioned before, the company is a collection of hundreds of local 2-sided marketplaces. And while most in the tech scene have a pretty good understanding of how you might go about getting more people to install the Uber rider app, it’s harder to imagine what it takes to get more drivers onto the Uber platform. I know I certainly didn’t know much about it before starting to work at the company!

Uber’s platform has 1M+ drivers
So let’s dive into this topic, and we’ll start with a quote about why Uber’s platform is so important for drivers, using a quote from David Plouffe, who’s on the board of Uber and also ran Obama’s 2008 campaign.

In his essay Uber and the American Worker, he writes:

The Bureau of Labor Statistics estimates that 20 million Americans are forced to work part-time for “non-economic reasons” like child care or education. And 47 percent of people in the U.S. say they would struggle to handle an unexpected $400 bill, and a third of those said they would have to borrow to pay it.

In other words, tens of millions of people in America need work. The Uber platform has a lot of drivers on the platform – over a million – and we hope to get more. That’s real scale, and something that inspires me every day. Plouffe continues in his essay with some interesting statistics:

Uber currently has 1.1 million active drivers on the platform globally. Here in the U.S., there are more than 400,000 active drivers taking at least four trips a month. Many more take only a trip or two to earn a little extra cash. It adds up: in 2015, drivers have earned over $3.5 billion. And by the way, only about 40 percent of drivers are still active a year after taking their first trip.

You can see from the above why driver growth is a key to Uber’s success – you’re convincing a ton of people to drive who have often never driven before, and many try it out and leave. Or they are part-time.

If you want to read more about drivers, their demographics, growth rate, etc., here’s a great 30-page paper called An Analysis of the Labor Market for Driver-Partners in the United States. Great reading.

Surge pricing and lowering fares: Keeping the marketplace in balance
The lens of Uber as hundreds of 2-sided local marketplaces also helps explains the importance of compromises like surge pricing and fare cuts. These mechanisms keep the marketplace in balance, and help grow the network effects that Gurley and Sacks recognize in Uber. Without them, one side of the marketplace might outstrip the other, causing a downward spiral. So even though neither side is happy with all of the marketplace balance tools that Uber puts to use, it’s ultimately a foundational tool in Uber’s business.

Take surge pricing, for instance. It’s easy to hate it, as a rider, and there are legitimate cases where it should be turned off. But think about it from the driver’s point of view- it gives them a huge incentive to get out onto the road, and to come to the exact area in the city where they are most needed. In fact, surge is done on a hyperlocal basis- just check out the screenshot of the driver/partner app to get a sense for how tightly drivers are directed to come to high-demand spots.

surge

Each colored hexagon above is a different level of surge. If you want to go in-depth on surge pricing, there’s a medium-length case study here: The Effects of Uber’s Surge Pricing.

(You can see more about the driver/partner app here and a Wired article on its development).

The flip side of surge prices, which raise fares for consumers, are lowering fares. Uber has recently cut fares in about 100 markets. Like surge prices, these cuts are a marketplace balancing mechanism to increase demand and ultimately increase driver earnings.

This is done in a way described within Sack’s diagram above, where the “less downtime” arrow is the key. When drivers aren’t sitting and waiting for their next trip, they are more efficiently utilized, which increases their earnings. If you can get earnings-per-trip and trips-per-hour to go the right way, you can increase earnings-per-hour.

Uber has released some directional charts showing this as positive for drivers, as part of Price cuts for riders and guaranteed earnings for drivers. The essay describes an approach of lowering fares to boost demand, and pairing that with guarantees while the rider side of the market figures this out. In tandem, good things happen, such as these graphs of earnings in some of Uber’s largest markets:

uber_earnings

As you can see, the earnings numbers are moving up and to the right. Not bad.

In closing, a fun video
Ultimately, Uber is providing an important platform for both riders and drivers to interact, across hundreds of hyperlocal marketplaces around the world. When you start to think of it this way, and especially from the driver’s POV, rather than the rider, you’ll start to 10X your understanding of Uber’s business.

If you want to learn more about roles at Uber, here’s a link to get in touch or just look at the careers page.

And finally, I want to leave y’all with a fun video featuring Jonathan driving an Uber and singing Roses (The Chainsmokers) with his riders. Enjoy.

PS. Get new updates/analysis on tech and startups

I write a high-quality, weekly newsletter covering what's happening in Silicon Valley, focused on startups, marketing, and mobile.

My top 2015 essays on Uber, online dating, push notifications, Apple Watch, and more

Personal update- I’m joining Uber! Here’s why

This is the Product Death Cycle. Why it happens, and how to break out of it

Quick update: Quoted in WSJ on dating apps, recent podcast interview, plus recent essays

New data shows losing 80% of mobile users is normal, and why the best apps do better

Photos of the women who programmed the ENIAC, wrote the code for Apollo 11, and designed the Mac

The Next Feature Fallacy: The fallacy that the next new feature will suddenly make people use your product

Why investors don’t fund dating

Why we should aim to build a forever company, not just a unicorn

Ten classic books that define tech

How I first met Eric Ries and also why I’ve ordered his new Kickstarter-exclusive book The Leader’s Guide

This is what free, ad-supported Uber rides might look like. Mockups, economics, and analysis.

Meet me and Eric Ries at a private event on March 21st – here’s how to attend

Personal update: I’ve moved to Oakland! Here’s why.

The most common mistake when forecasting growth for new products (and how to fix it)

The race for Apple Watch’s killer app

My top essays in 2014 about mobile, growth, and tech

Why messaging apps are so addictive (Guest Post)

IAC’s HowAboutWe co-founder: How to Avoid Delusional Thinking in Start-up Growth Strategy (Guest Post)

Mobile retention benchmarks for 2014 vs 2013 show a 50% drop in D1 retention (Guest post)

New data on push notifications show up to 40% CTRs, the best perform 4X better than the worst (Guest post)

Why Android desperately needs a billion dollar success story: The best new apps are all going iPhone-first

Early Traction: How to go from zero to 150,000 email subscribers (Guest Post)

New data shows up to 60% of users opt-out of push notifications (Guest Post)

Why aren’t App Constellations working? (Guest Post)

There’s only a few ways to scale user growth, and here’s the list

Lessons learned adding messaging to a notes app (Guest Post)

Retention is King (Guest Post)

Why consumer product metrics are all terrible

How to solve the cold-start problem for social products

How to design successful social products with 3 habit-forming feedback loops

Congrats to my sis Ada Chen, who’s joining SurveyMonkey as VP Marketing

How to make content creation easy: Short-form, ephemeral, mobile, and now, anonymous

My 2013 essays on mobile, startups, and tech

When a great product hits the funding crunch

A clever way to buy Facebook ads based on what your users like (Guest post)

Use this spreadsheet for churn, MRR, and cohort analysis (Guest Post)