From Idea to Impact: How to Measure the New Product Features’ Success

Launching new product features may be extremely rewarding. However, it is product analytics that helps us decide whether the idea is worth developing further or if we had better move on to the next one.

I'm Alex Sokolnikov, Product Manager at Semrush. I’ve spent a large chunk of my career launching new products—constantly diving into fresh research, testing hypotheses, and proving product viability in uncertain conditions.  

Currently, together with my team, we work on the ContentShake AI tool—an assistant for content marketing. Our focus is on generating SEO-optimized content to drive traffic from search engines, a niche we've carved out to help marketers achieve better results.

There are many truths I’ve learned over six years of my career in product development, but one has proven viable repeatedly: before introducing new features, consider whether refining what you already have can help better achieve your goals.

Let's explore why new features might—or might not—be the answer.

Running Fast Just to Keep Up

In a fast-growing market, ideas for new features are everywhere—from user feedback to analytics showing product gaps, while competitors constantly release updates. New companies are popping up all over, claiming to have the best solutions. With so many options, it’s harder than ever for your customers to know where to start.

So, let’s imagine: you and your team use a framework like ICE (Impact, Confidence, Effort) to decide what to focus on for the next three months. But then, you wonder: should we maybe improve existing features instead? Or even shut them down and build something new?

Here’s an example from my experience: In one version of our product, we targeted the small business segment. Small businesses are known as a backbone of a country’s economy. One of our clients organizes safaris in South Africa: they invested money in their own hotel, rented cameras and safari vehicles, but they can’t compete with large hotel chains when it comes to ad budgets. They need a marketing strategy that’s affordable, requires no deep learning curve (since they’re already running an entire business), and can be managed in the little free time they have.

We suggested they generate traffic from search engines, paired with running engaging social media pages featuring striking photos of African animals.

Through interviews, we found that while these clients had no trouble managing their social media, they didn’t have the time or expertise to figure out SEO, write huge volumes of blog articles, or manage Google Analytics. At the time, we were able to generate a variety of content types, while some of our competitors offered even more options for AI-generated content. So, we decided to focus on generating SEO articles—and it paid off.

What data led us to that decision? Well, there’s no quick answer. Let’s break it down together.

Three (or More) Keys to Success

Before rolling out a major feature, I create a one-pager (though it's usually closer to eight :)). One of the key sections asks: ‘How will we measure success?’

It’s worth having some sort of a numeric metric that gives us a clear answer—something that leaves no room for data interpretation. Of course, the best case is when you already have the data from previous launches that can help you improve the accuracy of forecasts. In case you don’t, you can just look up benchmarks across industry, ask other product owners in your company, or just take a wild guess.

There are three main metrics categories that are worth being considered and can offer us useful insights—some more clear, some less. Let’s take a closer look at them to see how we appropriated those categories to measure the success of our own ContentShake AI.

Revenue

In an ideal world, we'd measure everything in relation to revenue, but it is a lagging metric. If a customer doesn’t like something, we’ll only find out a month later—when they may decide to cancel. It’s probably a good thing we haven’t introduced annual subscriptions yet, or we wouldn’t know until the following year.

And as much as we'd love to link everything to revenue, not every feature has a clear financial impact. For example, how do we measure the value of an 'auto-publish to WordPress' feature? Still, when we can tie a feature to revenue, it’s incredibly satisfying!

Activation

Since the revenue metric takes time to show results, let's track feature clicks instead. The most-clicked feature must be the most valuable, right? For more accuracy, we’ll filter out free users, sort the paying ones by “Lifetime Value”, and see which features they use most. Additionally, we can isolate a cohort of new users, or even users who came from a specific ad campaign, and track their behavior.

You open up the analytics system and check what percentage of the user base is using a particular feature; let’s say 30%. This number is high enough for us to consider a feature successful. What can possibly go wrong here?

Well, the problem with this approach is that it’s incredibly easy to manipulate. Want your feature to ‘win’? Just place it at the top left. Want to boost it more? Highlight it with a bright color. Still not enough? Slap a ‘new’ badge on it.

Retention

You can calculate retention for each feature, but it’s a bit more complex and takes longer. A ‘good’ feature is one that keeps users coming back. It becomes a habit: Whether they love it or simply stick with it because there’s no better alternative, it's part of their routine.

To properly commit to this metric, you need to understand how customers use your product.

In my experience, people create social media posts daily, so you'd track daily retention. But for something like SEO articles, written once a week, you'd want to measure weekly or monthly retention instead.

If paying users love a feature, that's the one we’ll focus on developing. And we’ll measure the effectiveness of everything else based on the retention rate of our most successful feature. Sounds good?

Close, but still not quite: If we went that route, we'd likely end up continuously refining niche features, like automatic publishing to WordPress, which are only used by a small but highly engaged group of customers.

Typically, each product has a hidden gem: a niche feature with good retention of a small number of customers. You can find it by checking the retention rate.

Activate, Retain, and Consider

As we just demonstrated, the Activation metric gauges user interest in a feature, but it's easy to manipulate. The Retention metric, on the other hand, measures how well a feature is executed, but it can lead to a narrow focus—potentially locking you into a small, niche audience, even if the feature is executed flawlessly.

So why not combine them since they complement each other so well?

That’s exactly what we did for our product ContentShake AI. Now, before launching any new feature, we set target goals for both activation and retention metrics. Separately, they don’t tell you much, but together, they provide a roadmap for what to do with the feature next.

When considering these two metrics together, you may face several scenarios, from worst-case to best-case. Let’s explore all the causal possibilities.

Low Activation, Low Retention

Something went wrong. You’ve released a feature that’s not interesting to users, and even the curious few who discovered it don’t want to use it. The silver lining? Not many people saw it, so you can quietly tuck it away in the interface or shut it down entirely. If you still believe in it, feel free to give it another shot—but in my experience, it might be better to focus on something more exciting.

High Activation, Low Retention

The feature isn’t meeting user expectations, even though it’s in high demand from a marketing standpoint. Coordinate with the marketing team to boost its promotion while the demand is still high. At the same time, gather user feedback through interviews and address any implementation issues swiftly before the major ad campaign launches. Your competitors are watching!

Low Activation, High Retention

The feature is a bit too niche, but it’s very well-executed. You need to dig into which specific segment is loving it and, most importantly, figure out why others aren’t. Maybe it’s time to increase visibility, add it to the onboarding process, or slap a ‘new’ badge on it to draw attention.

High Activation, High Retention

Congrats, you’re awesome! You’ve found a feature that appeals to a wide audience and executed it in a way that users actually love. Now’s the time to expand on this success—whether by presenting it at a product meetup, spotlighting it in your performance review, or even turning it into a premium offering to upsell your clients.

Conclusion

As you can see, it is absolutely crucial to properly evaluate new product features before and after its development—so that your product doesn’t end up like a zoo full of empty cages. Fortunately, using the Activation and Retention metrics at the same time provides us a clear picture of how valuable a feature really is and helps to align with user needs. This focus allows your team to direct efforts where they’ll achieve the best results.

I truly believe that data and analytics are the cornerstone tools for any product manager. If you can’t measure something, you can’t control it. Hopefully, after reading this article, you’ll feel more confident in answering that all-important question: ‘Do I really need this new feature?’

If you have more questions, feel free to reach out on LinkedIn. I often share quick insights about product management and am always up for chats and discussions.