The Metric Everyone Defined Differently

This post started as a note to my future self. Every few months I relearn the same lessons about incremental models, so this time I'm writing them down.

The setup was ordinary: Airflow at the edges, pandas doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

Some background

When we finally measured, the numbers were worse than anyone guessed. Around 43% of the time spent on a typical change went into working around incremental models rather than building anything.

Most of the complexity was accidental. Over the years we'd layered three different approaches to incremental models on top of each other, and each one made sense at the time.

The fix

Rather than a big-bang rewrite, we migrated one module at a time over about 8 weeks, keeping both paths alive behind a flag.

Here's a trimmed-down version of what we ended up with:

import polars as pl

daily = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event") == "signup")
    .group_by(pl.col("ts").dt.date().alias("day"))
    .agg(pl.len().alias("signups"))
    .sort("day")
    .collect()
)

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

Delete dead code aggressively; it's the cheapest refactor there is.
Make the default the safe choice, not just the documented one.
Write the migration guide before writing the migration.
Pair on the first few changes so the knowledge doesn't live in one head.

Where this breaks down

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

We still haven't solved backfills. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

Wrapping up

I don't think there's a universal answer here, but I hope this gives you a few ideas to try on your own project. If you do, I'd love to hear how it goes.