Product metrics - we track too much

In this post I'll be discussing the importance of focus and how to track product metrics that really matter

Let's just track every possible user action, then we can figure out what metrics we want to report on later.

Who has heard this sentence before? 🖐🏼

Continuing on with my series on product analytics, I want to talk about my favourite analytics pitfall today.

I recently highlighted the steep learning curve especially for new Product Managers to gain product analytics skills. Unfortunately this is also true for our stakeholders who don’t have enough knowledge on which product KPIs really matter to their business.

I’ve seen it often enough that I know how this usually plays out in the end. Tracking analytics events for every possible button and interaction in your product, and figuring out what metrics you want to report on to the business at a later stage usually results in two things:

1️⃣ the team ends up with an overwhelming amount of data and events that in fact slows down analysis and decision making and
2️⃣ the product team ends up scraping for something interesting to pull out of that data pool later on, only to realise that they haven’t spent enough time testing the events that were truly important, which then leads to lack of trust in the data from the team and stakeholders.

Rather than relying on input from stakeholders, I strongly believe Product Managers should introduce standardised analytics systems that they can use to create alignment on key metrics, before implementing the first analytics event.

I know it may sound counterintuitive, but here’s why it’s better to start small and stop tracking too many events:

1. You’re creating less focus

While I admire your aspiration to implement 300+ events for your product, chances are that you won’t really be using all of them to create meaningful insights.

Do you really need to know when a user changed every single field in their profile? Do you need screen view events for every single part of your product or do you actually just care about the key actions the user performed?

Usually us Product Managers don’t find nearly enough time in our day to even focus on the most important key metrics, so why should we cram in even more information if we don’t end up looking at it?

If you ask a Product Manager about their current key metrics like onboarding conversion or customer churn, can they quickly and reliably answer this question? The answer is often no. A study from Mixpanel and Product School showed that only 10% of product teams are able to validate their most important product decisions with data.

The point I’m trying to make is that we typically lack focus, and having even more events in your data warehouse won’t help with that.

2. You’re making your life harder

Having too many events can make it a lot harder for you to build out funnels, remember which event was fired where, and it generally creates a lot more room for error.

If you’ve worked with product analytics before then you know how painful it is to test and refine your analytics events. We typically have to spend a lot of combined effort together with our engineers and QA testers to check our events and event triggers whenever those reports don’t seem to add up.

I have also learnt how important it is to insist on testing all event triggers as part of your QA testing process before you release a new product or feature, so you really want to make sure that you spend enough time on testing the important events properly. Like with your product features, prioritise on the ones that will bring the most value, and get those right.

3. It could get pricy

A lot of product analytics platforms like Segment or Amplitude charge based on monthly data volume, so the more events you send to your tracking tools the more expensive it gets.

I’ve always been a strong advocate for investing in a proper setup and tools for measuring your product usage early on, but cost is still something you want to keep in mind. Especially if your landing page gets a fair bit of traffic from your marketing campaigns you might be adding up a lot of costs for data that might not bring any valuable insights to the product team.

How to do it better

The goal is to create less clutter of events, and with that more clarity and focus on your most important metrics. The product analytics tool Amplitude recommends setting up between 20 and 200 events maximum for a product. This will allow you to:

1️⃣ test the events you implement more thoroughly
2️⃣ reduce the costs for your analytics tools as well as
3️⃣ decrease the mental load on you and the team to keep track of all of the events.

I hear from most Product Managers that they aren’t confident in the data of even the most basic numbers like their product reach and retention. Very rarely do I hear someone say that they wish they would have put in more events - unless they have chosen to implement any analytics at a later stage which is never a good idea.

Here are the two key steps to create more focused tracking:

1. Standardise your key metrics

If you’re launching a completely new product with a small feature set, I recommend to keep it simple and start small with a set of around 20 events that will tell you about your

• Reach - how many people created validated or complete accounts (this is essentially measuring your true acquisition and onboarding conversion)

• Activation - how many users perform a key action in the first x days of using your product

• Engagement - how often users get value out of your product like using one of your key features

• Retention - how many users are still engaged or churn after x weeks or months

If you’re adding new features to your product, you want to then layer on basic events that track whether your users are successfully using these new features. You should also setup a goal during discovery of those new features on the expected impact on your four key metrics.

Yes, you need to set those goals and expected impact on your numbers before launching new features 😃 For example you might have added new prompts for your users to perform a key action in the first week of using your product, with the predefined goal to increase your activation ratio to x.

2. Link your metrics to your product north star

I often see people staring at their metrics trying to figure out how those numbers will translate into business outcomes. New sign ups don’t necessarily translate into long term paying customers as I had to experience painfully when we tried to grow a visitor management SaaS product.

The mysterious gap between your product metrics and the business goal

Your product goal or product north star will help you fill this gap. While your business goal is typically a commercial goal like generating a certain amount of revenue for the business, you don’t want to wait for those numbers from finance at the end of the year to understand whether your product or new feature was successful.

Rather than looking at a lagging revenue number (the business goals), you want to find a leading indicator in your product (the product north star) that will drive revenue in the future.

As an example, you might want your users to use feature x at least once a week as this means they will be getting the most value out of your product, and will more likely become long term customers. Amplitude has created a lot of fantastic content on how to define a good product north star.

Once you have your product goal or north star defined, you can now go back to your key metrics again (reach, activation, engagement and retention) and define how those key metrics will impact your overall product north star.

To sum it up

Make sure you have your most important events firing properly to be able to give you accurate insights on the four key metrics. Defining your clear product north star will help you create focus and communicate how those metrics will lead to overall product and business success.

So next time someone tells you to just track everything, to then desperately try to find useful bits in your data later, I hope you can now argue for a more focused step by step approach. Get your analytics basics right and expand on your events list over time once you learn more about which areas of your product you need deeper insights into.

---

Want to read more on this topic?

I am proud to share I have launched my first ebook to help you with your product analytics and data practises! The Insights Driven Product Manager is your essential guide on how to track less and get more insights from your data to make better product decisions. It includes practical frameworks and worksheets that you can apply to your products straight away.