Polars vs. pandas for Everyday Work

For a long time I treated backfills as something to worry about later. Then a small change to our pandas setup turned into a week of debugging, and "later" arrived all at once.

The first symptom was subtle. Nothing failed outright, but backfills kept drifting in the wrong direction, and every fix seemed to make the next one harder.

What was going wrong

When we finally measured, the numbers were worse than anyone guessed. Around 44% of the time spent on a typical change went into working around backfills rather than building anything.

Most of the complexity was accidental. Over the years we'd layered three different approaches to backfills on top of each other, and each one made sense at the time.

How it works

The fix turned out to be smaller than expected. Instead of adding another layer, we removed one and leaned on what pandas already gives you out of the box.

Stripped of the project-specific bits, it looks like this:

SELECT
  user_id,
  event_at,
  LAG(event_at) OVER (PARTITION BY user_id ORDER BY event_at) AS previous_event_at
FROM events
WHERE event_at >= CURRENT_DATE - INTERVAL '30 days';

Nothing about this is clever, and that's the point. Anyone on the team can read it in a minute and know exactly what it does.

A few things that helped along the way:

Pair on the first few changes so the knowledge doesn't live in one head.
Agree on what "done" means before the first pull request.
Make the default the safe choice, not just the documented one.

Trade-offs

We still haven't solved partitioning. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

The biggest cost was social, not technical. Changing how people think about backfills takes much longer than changing the code.

Wrapping up

That's it for now. Next time I'll dig into data quality checks, which turned out to be closely related.