Incremental Models That Don't Lie to You

A colleague asked me last week why our team cares so much about incremental models. I gave a rambling answer, so here is the longer, more organized version.

Most of the complexity was accidental. Over the years we'd layered three different approaches to incremental models on top of each other, and each one made sense at the time.

What was going wrong

When we finally measured, the numbers were worse than anyone guessed. Around 39% of the time spent on a typical change went into working around incremental models rather than building anything.

We had good intentions. We even had a document describing how incremental models should work. The trouble was that the document and the code had quietly gone their separate ways.

Trying something different

The fix turned out to be smaller than expected. Instead of adding another layer, we removed one and leaned on what pandas already gives you out of the box.

Stripped of the project-specific bits, it looks like this:

import polars as pl

daily = (
    pl.scan_parquet("events/*.parquet")
    .filter(pl.col("event") == "signup")
    .group_by(pl.col("ts").dt.date().alias("day"))
    .agg(pl.len().alias("signups"))
    .sort("day")
    .collect()
)

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

Make the default the safe choice, not just the documented one.
Pair on the first few changes so the knowledge doesn't live in one head.
Agree on what "done" means before the first pull request.

Caveats

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

The biggest cost was social, not technical. Changing how people think about incremental models takes much longer than changing the code.

Wrapping up

That's it for now. Next time I'll dig into data quality checks, which turned out to be closely related.