Measuring the value your community is providing to members is incredibly important. Without value, people won’t stick around, contribute, or spread the word.
So how do you measure this?
No One Metric to Rule Them All
First, a caveat: you should never rely on a single metric for anything. Single metrics give you one angle of a situation that is complex. You should always compliment your primary metric with other viewpoints.
For example, if someone says that the community provides them value but you see incredibly low contributor rates, that could mean your community is riskily balanced on the backs of a very small number of contributors. Or you see great contributor rates, but when you review the quality, 90% are spam.
Choose the most useful primary metric, then supplement with other viewpoints.
The Case Against NetPromoter Score
NetPromoter Score (or NPS) asks what seems like a simple question:
“How likely is it that you would recommend [Organization/Product/Service/Community] to a friend or colleague?”
Members then answer by choosing a number between 0 (not at all likely) and 10 (extremely likely).
You then break members into different groups based on their score:
- 0-6: Detractors
- 7-8: Passives
- 9-10: Promoters
You then subtract the percentage of Detractors from the percentage of Promoters to get your score.
NPS is rampant across the business world. It’s ubiquity suggests that it simply must be a high quality, useful metric. And although I’ve used it before, I have come to the conclusion that it’s generally problematic, and specifically a bad fit for communities.
0-10 Scales Are Confusing
Quick, tell me what a 6 out of 10 likelihood to recommend to a friend means!
Hard, right? The 0-10 scale for NPS is overwhelming for both customer and analyst. Research validates this: one study showed that a 5- or 7-point scale was better.
(They also found that there wasn’t a single best predictor, and that different question response combos resulted in different results – see my point above about looking at more than one metric.)
It’s Not Entirely Clear There Is Sound Logic Behind the Calculation
This piece nicely illustrates exactly how wacky this calculation can be:
“Let’s say we’re having a bad day and 10 respondents give us all zeroes: 0, 0, 0, 0, 0, 0, 0, 0, 0, and 0.
The average of these ten numbers is a 0. (Makes sense.)
NPS is -100. That’s the worst it can get.
Now, let’s say the team works really hard. They make the product so much better.
After all this hard work, we get all sixes: 6, 6, 6, 6, 6, 6, 6, 6, 6, and 6.
The average of these ten numbers is 6.
But NPS is still -100.”
This is not only depressing, not only confusing, but also doesn’t give you good directional feedback. You just increased likelihood to recommend by a massive amount…yet your score says you made no progress. This could lead your team to abandon the work that got you those 6’s and go back to the drawing board, even though you were on the right track.
There Are Reasons Someone Might Not Recommend A Community They Love
This last point comes from personal experience. I’ve used NPS many times in my career. And every time I dig into the details of the responses, I see oddities based on the exact phrasing of the question. Common ones:
- “I wouldn’t recommend this to my friends because I have no friends.”
- “I wouldn’t recommend this I don’t ever recommend products.”
- “My friends don’t want me to bug them with product recommendations.”
With communities, it becomes even more interesting:
- “I wouldn’t recommend this community to my friends because I don’t want them to know what I’m posting here.”
- “I wouldn’t recommend this community because my friends aren’t into this topic.”
So your member could be experiencing your community as the brightest point in their life, the reason they get out of bed in the morning, and yet they’d give you a bad NPS. Again – this is not helpful directionally.
The Case for Product-Market Fit
The Product-Market Fit (PMF) survey asks:
“How would you feel if you could no longer use [Organization/Product/Service/Community]? A) Very disappointed B) Somewhat disappointed C) Not disappointed”
Then you simply report on the % of members that fall into each category.
Now, to be clear, I’m not 100% sure PMF is the right solution, and I have limited personal data points. But on the surface, it feels like a better fit.
It’s Simpler
Three options. Worded in a human way. Far easier for the member to determine, and far easier for you to interpret; “Hey, most folks would not be disappointed if we shut this community down” is a pretty definitive statement.
It Focuses on the Member
Rather than focusing on recommendations, which I’ve already demonstrated are highly complex things, this question focuses purely on the member’s experience with your community.
It’s Encompasses Value and Emotion
Especially with communities, you want to provide value but ALSO be eliciting a strong emotion. Someone might find a community useful but not be passionate enough about it to contribute. This question nicely straddles the usefulness of the community and how strongly the member feels about it.
It’s a Familiar Term for Most Startup Leaders
The hardest thing about moving away from NPS is that everyone seems to use it, so you’ll get skepticism when you say you want to try something new.
Thankfully, entrepreneurs talk about product-market fit constantly – so you’ll still be speaking their language.
The Magic is in the Responses
Regardless of which survey you use, be sure to include a “Why did you give that score?” write-in question after the numerical rating. Yes, having a score and being able to look at which way you’re trending is valuable. But more valuable is knowing WHY people feel this way. This is what you can spin into gold. This is what is really bugging or delighting people.
So even if you’re stuck with NPS, make sure you get those write-ins. Ultimately, what your members say is more valuable than any number.
Keep Experimenting
My goal here isn’t to get you to adopt PMF, but rather to carefully consider the value of NPS for your community and to explore alternatives.
As mentioned above, no one metric will serve all your needs.
No community is the same, and you may have unique aspects that you should measure. Don’t get bullied into using others’ metrics unless they’re actually valuable for you.
Try different things. Try validating them – do you actually see recommendation actions from high scorers? Increased contributions? Or are people saying one thing and doing another?
And lastly, please share – we’re all traversing these challenges together, so I’d love to hear from you about what worked for you!
Stars photo by Towfiqu barbhuiya
Frustrated man photo by Oladimeji Ajegbile
Apple photo by Andres Ayrton