How Not to Use Data 101

When I was asked to lead the growth team at Tailscale, my first instinct was to instrument everything and dive into the data to find correlations I could leverage for growth. That playbook made sense — I’d seen it at the bigger companies I worked at: everything is tracked, you have mature tools and frameworks for experimentation, and you spend your time testing growth ideas. So I spent six months trying to replicate that approach — and got nowhere. That (re)taught me a thing or two about data that I’d like to share in this post.

Data Without Context Is Misleading

It’s tempting to instrument everything — track feature usage, user counts, linger time — and look for correlations with business outcomes. After all, you know the product, you know what’s valuable, and you just need the data to prove what you already believe.

But your customers aren’t you. They don’t all behave the same way, use the same features, or find value at the same time. One user might start using a high-value feature immediately, another might get there six months in, and a third might find value in something you didn’t anticipate at all.

Without understanding why users behave the way they do, your data will likely lead you to the wrong conclusions. That context comes from talking to customers. Talk to 10–20 of them — more if you can. These conversations help you segment customers and uncover the reasoning behind their decisions. You should aim to learn:

• What problems are you trying to solve with the product?
• How are you solving those problems today?
• What’s wrong with your current solution?
• How did you discover the features that helped?

This isn’t an exhaustive list, but it’s a solid start. If you correlate what you hear in these conversations with detailed usage data for those same users, you’ll start to uncover meaningful patterns. For example: maybe people solving problem X tend to use feature Y — and those users are also the ones upgrading to higher-tier plans.

At this stage, you’re looking for early, low-fidelity signals. You’re not trying to be scientifically accurate — just directionally correct.

Mind the Time

People tend to focus on the metric itself — subscription, retention — but most meaningful metrics have a time dimension baked in:

• Customer used feature X — by when?
• Customer converted to a paid plan — how soon?
• Customer was retained — for how long?

When defining metrics, be clear about the timeframe you’re measuring over. Are you looking for actions in the first day? Week? Month? Customer conversations can help validate both when a behavior tends to happen and how often it repeats (e.g., feature X is used weekly vs. monthly).

Keep your timeframes as tight as possible. There’s a tradeoff between timeframe and signal: the longer you wait, the fewer customers remain in your sample. Unless you’re operating at Google or Meta scale, most high-signal metrics show up within the first few days — or at most, the first couple of weeks. I ended up limiting my analysis window to 1–4 weeks post-signup for both output metrics (like retention or subscription) and input metrics (like feature usage).

Get Specific About Metrics

Once you have more than a thousand records and dozens of columns, data gets dizzying fast. If you just dive in (or feed it to a model), you’ll likely end up with spurious correlations or misleading results. That’s why it’s critical to get specific — about what you’re measuring, and why.

A good place to start is by distinguishing between input and output metrics.

Output metrics are the outcomes you care about — like retention or subscription — but you don’t directly control them.

Input metrics are behaviors you can influence — like feature usage or onboarding completion. The goal is to identify which inputs correlate with the outputs you’re trying to drive.

Once you’ve defined your inputs, outputs, and timeframes, you can center your measurement around specific hypotheses. For example, if customer conversations suggest that using Feature X leads to faster subscription, test it. If you have enough data (say, 10K users over 1–4 weeks), an A/B experiment is ideal.

If not, you can still ship the change and compare outcomes against historical data. It’s less rigorous — but in my experience, meaningful changes show up clearly in the data. Everything else is just noise.

Final Thoughts

These are just a few lessons I learned the hard way. There are plenty of great resources out there that go deeper into Product-Led Growth and data-informed product management. My favorite growth practitioner is Elena Verna, and I’ve also heard good things about the Reforge Growth Series — though I haven’t taken the course myself.