Testing Pipelines Like Application Code

Every codebase has one area everyone is a little afraid to touch. On our team, it was everything related to late-arriving data.

When we finally measured, the numbers were worse than anyone guessed. Around 43% of the time spent on a typical change went into working around late-arriving data rather than building anything.

The problem

We had good intentions. We even had a document describing how late-arriving data should work. The trouble was that the document and the code had quietly gone their separate ways.

The first symptom was subtle. Nothing failed outright, but late-arriving data kept drifting in the wrong direction, and every fix seemed to make the next one harder.

A simpler approach

We started by writing down the one rule we actually cared about, then deleted every bit of code that didn't serve it. It's not glamorous, but it works.

The core of it looks like this:

SELECT
  user_id,
  event_at,
  LAG(event_at) OVER (PARTITION BY user_id ORDER BY event_at) AS previous_event_at
FROM events
WHERE event_at >= CURRENT_DATE - INTERVAL '30 days';

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

Pair on the first few changes so the knowledge doesn't live in one head.
Delete dead code aggressively; it's the cheapest refactor there is.
Make the default the safe choice, not just the documented one.

Where this breaks down

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use Polars for everything. That's a fair question with a boring answer: we tried, and it made late-arriving data harder to reason about.

The biggest cost was social, not technical. Changing how people think about late-arriving data takes much longer than changing the code.

What's next

If you take one thing away from this, let it be this: late-arriving data is a design decision, not a cleanup task. Make it early and make it on purpose.