Should you redo your product analytics from scratch?

In this post I’ll be sharing a list of factors every product team should consider when facing this big decision

Last week we hosted a meetup on how we as Product Managers can become better at product analytics. I presented the common analytics pitfalls I’ve observed or fallen into myself over the past few years, and I saw a lot of nodding heads when I pointed out the gap of analytical skills in the product community.

After my talk we ran the usual round of Q&A and I received some awesome questions about naming conventions, tools, and where product analytics fits in with other metrics like performance or marketing reporting. One question from the crowd however particularly stuck with me and I kept thinking about it over the weekend. The question was:

When should you completely redo your product analytics events, versus trying to refine the existing setup? 

I recently made the decision on a B2B SaaS product I look after to do a complete redo, so I had some relevant experience to share on this. However the more I thought about this question, the more I felt it needed an even deeper discussion than what I was able to get into during the Q&A, so I consolidated a list of four key factors that product teams should consider when facing this big decision. 

1. What is the current state of your analytics?

First we need to ask ourselves how useful our current data is. Do we get enough actionable insights from it and most importantly, can we trust our data? I’m assuming one of those things is an issue if you’re already here reading this article 😁

Assess what is currently broken - is it a specific platform that seems less reliable, does the overall structure of your events need to change or can your analytics just do with a more focused effort on testing?

2. How big is the history of your data?

I made the decision to do a complete overhaul of my product analytics when we’ve already been in market for months. Since it’s very hard to not break your reports when redoing your analytics, I had to be comfortable with the thought of not being able to create most of my insights back in time. 

On a side note, it is possible to continue your current reports if you have data science resources who can map all old events to the new ones, but this can be a very manual and time consuming task so it’s often not an option.

A great tip I’ve read in Segment’s article on how to do analytics cleanups is to pick a time for the switchover which will be the least disruptive to the business. For example you could turn on your new analytics events at the start of the new reporting period like your financial year.

3. How painful will it be to fix the existing events? 

Now that you have a clear picture of the issues and what needs to be done, you have to assess the level of pain that you will have to go through to fix your current analytics. If you’ve worked with events before, you know how much effort it can take to find those weird little spelling mistakes, or having to figure out where and why certain events are not firing properly.

If you’re just hoping to update your naming conventions, this can easily be done without a complete overhaul and you can keep the history of your events. However if you want to change the overall structure of your events it may be easier to start from scratch. 

In my case I wanted to move away from a flat events structure where every event was recorded as a separate event per platform. Instead I wanted to make our events a little bit more sophisticated and roll them up into one event per action that I can then break down by platform through properties. This would have been an actual nightmare if I had to wedge the existing events into this format one by one. 

4. Will my stakeholders be on board? 

The longer you have been in market with your product, the more analytics history you might loose when redoing your analytics. As described in point #2, unless you’re investing a lot of time manually mapping old events to new events, your reports will need to record from scratch again. 

For me it was a fairly clear decision as I’d rather loose six months of history than having messy data for the next few years. However if you have been in market for longer and have a huge user base, the loss of continuous reports may scare some of your stakeholders and you have to do a lot more convincing work. You should also consider if any other departments like finance may be affected by this and include them in your decision process.

Final 2 cents

Especially if you can get your key insights without having to look at years of historical data and your current analytics are not very useful anyway, a clean up can sometimes be more efficient than trying to debug and fix a long list of broken events. Not to mention how frustrating this process can be.

The idea of losing continuity in your reports may sound scary to you and your stakeholders though, and it is a big decision to make - so it’s important to weigh up the efforts of either approach based on the factors above.

I do strongly believe though that no matter which path you decide on, you should always ask yourself: how would I go about setting up my analytics if I were to start from scratch? Then define the events as if you were in fact starting from scratch. Otherwise it it very easy to just stay within the past limitations and you will not make any significant progress.

If any of you have gone through the process of redoing or refining your analytics, I would love to hear about your experiences and if there are any other tips you can share!

---

Want to read more about product analytics?

I am proud to share I have launched my first ebook to help you with your product analytics and data practises! The Insights Driven Product Manager is your essential guide on how to track less and get more insights from your data to make better product decisions. It includes practical frameworks and worksheets that you can apply to your products straight away.

You can also check out my previous articles on analytics:

The big gap between product and analytics

Why we need to stop lying to ourselves

We need to stop tracking too much

And lastly, subscribe below if you would like to get my upcoming articles sent straight into your inbox! ✌🏼