Tagged “data-engineering”
6 posts
Data freshness and schema-drift alerting
A pipeline can run successfully and still be wrong. The two failures that matter most are silent by nature — the data stopped arriving, or it quietly changed shape — and neither shows up in a green DAG. Here's what to measure, and how lineage turns an alarm into an assignment.
|7 min readSessionization with SQL window functions
A stream of events isn't sessions until you decide where one visit ends and the next begins. You can do that in three window functions — no self-joins, no Python — and the whole thing hinges on the same frame-clause detail that trips people up on running totals.
|6 min readdlt REST API sources: declarative pagination without the glue
The schema glue is gone, but most teams still hand-write the other half of every API pipeline — pagination loops, auth headers, and "what changed since last time." dlt's REST API source turns all of it into configuration. Here's the declarative shape, end to end.
|7 min readData pipelines with dlt and DuckDB
Most pipeline code is glue nobody wants to maintain. dlt and DuckDB let you skip the glue and keep the parts that matter — schema inference, incremental loading, and contracts that fail loudly instead of silently corrupting your warehouse.
|6 min readColumn-level data lineage in practice
Table-level lineage tells you two models are connected. Column-level lineage tells you that changing one field breaks exactly these three downstream columns and nothing else. That precision is the difference between a confident change and a prayer.
|5 min readSQL window functions: a practical guide
Window functions are the single highest-leverage SQL feature for analytics work, and the frame clause is the part everyone skips — right up until their running total is silently wrong. Here's the whole thing, practically.
|5 min read