Sofia's Notes

data engineering

Sofia's Notes

Hi, I'm Sofia. Essays and field notes on data engineering. I write about backfills, data contracts, partitioning and whatever else I'm learning this month.

About meSubscribe via RSS

Latest writing

  • · 3 min read

    Small Data Is Underrated

    I've been working with Python for about 9 years now, and my opinions on backfills have changed more than I expected.

    #dbt#pipelines#python
  • · 3 min read

    Data Contracts Without the Hype

    There is no shortage of articles about pandas. Most of them cover the happy path. This one is about what happened after the happy path ended.

    #warehouse#pipelines
  • · 3 min read

    The Metric Everyone Defined Differently

    This post started as a note to my future self. Every few months I relearn the same lessons about incremental models, so this time I'm writing them down.

    #sql#pipelines#data-quality
  • · 3 min read

    Polars vs. pandas for Everyday Work

    For a long time I treated backfills as something to worry about later. Then a small change to our pandas setup turned into a week of debugging, and "later" arrived all at once.

    #python#sql
  • · 3 min read

    Incremental Models That Don't Lie to You

    A colleague asked me last week why our team cares so much about incremental models. I gave a rambling answer, so here is the longer, more organized version.

    #warehouse#data-quality#analytics
  • · 3 min read

    Testing Pipelines Like Application Code

    Every codebase has one area everyone is a little afraid to touch. On our team, it was everything related to late-arriving data.

    #dbt#sql#duckdb
  • · 3 min read

    DuckDB Replaced Half of Our Data Stack

    For a long time I treated data quality checks as something to worry about later. Then a small change to our DuckDB setup turned into a week of debugging, and "later" arrived all at once.

    #data-quality#analytics#python
  • · 3 min read

    Partitioning Strategies I Wish I'd Known Earlier

    There is no shortage of articles about DuckDB. Most of them cover the happy path. This one is about what happened after the happy path ended.

    #duckdb#warehouse
  • · 3 min read

    Dealing With Late-Arriving Data

    For a long time I treated late-arriving data as something to worry about later. Then a small change to our Spark setup turned into a week of debugging, and "later" arrived all at once.

    #dbt#analytics#data-quality
  • · 3 min read

    Backfills Are a Product Feature

    I've been working with Snowflake for about 7 years now, and my opinions on backfills have changed more than I expected.

    #dbt#pipelines