Dealing With Late-Arriving Data

For a long time I treated late-arriving data as something to worry about later. Then a small change to our Spark setup turned into a week of debugging, and "later" arrived all at once.

Most of the complexity was accidental. Over the years we'd layered three different approaches to late-arriving data on top of each other, and each one made sense at the time.

What was going wrong

The first symptom was subtle. Nothing failed outright, but late-arriving data kept drifting in the wrong direction, and every fix seemed to make the next one harder.

The setup was ordinary: Spark at the edges, Airflow doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

The fix

The fix turned out to be smaller than expected. Instead of adding another layer, we removed one and leaned on what Spark already gives you out of the box.

Here's a trimmed-down version of what we ended up with:

SELECT
  user_id,
  event_at,
  LAG(event_at) OVER (PARTITION BY user_id ORDER BY event_at) AS previous_event_at
FROM events
WHERE event_at >= CURRENT_DATE - INTERVAL '30 days';

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

Pair on the first few changes so the knowledge doesn't live in one head.
Write the migration guide before writing the migration.
Delete dead code aggressively; it's the cheapest refactor there is.
Keep the old path around until the new one has survived a real incident.

What it cost us

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use Snowflake for everything. That's a fair question with a boring answer: we tried, and it made late-arriving data harder to reason about.

We still haven't solved data quality checks. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

Wrapping up

I don't think there's a universal answer here, but I hope this gives you a few ideas to try on your own project. If you do, I'd love to hear how it goes.