Tagged “data-engineering”
9 posts
Semantic dbt tests on Databricks: 50,000 real reviews through a Unity Catalog function
I took four dbt tests written as English sentences off my laptop and onto Databricks. The schema.yml didn't change. Everything underneath it did, and the 50,000-review production run failed its own gate by one point of recall, which turned out to be the most interesting part.
|13 min readTyped judgments inside data tools: five Jev demos
Most LLM features in data tools generate text you then have to parse. I built five demos around a model that returns typed judgments instead (probabilities, choices, scores), so English becomes a column you can filter, test and sort.
|9 min readSemantic dbt tests: data quality checks in plain English
not_null and accepted_values will happily pass a customer called Mickey Mouse, a return coded late whose comment says the box arrived crushed, and a ticket with someone's IBAN in it. So I wrote four dbt tests as English sentences and scored them against an answer key.
|12 min readData freshness and schema-drift alerting
A pipeline can run successfully and still be wrong. The two failures that matter most are silent by nature — the data stopped arriving, or it quietly changed shape — and neither shows up in a green DAG. Here's what to measure, and how lineage turns an alarm into an assignment.
|7 min readSessionization with SQL window functions
A stream of events isn't sessions until you decide where one visit ends and the next begins. You can do that in three window functions — no self-joins, no Python — and the whole thing hinges on the same frame-clause detail that trips people up on running totals.
|6 min readdlt REST API sources: declarative pagination without the glue
The schema glue is gone, but most teams still hand-write the other half of every API pipeline — pagination loops, auth headers, and "what changed since last time." dlt's REST API source turns all of it into configuration. Here's the declarative shape, end to end.
|7 min readData pipelines with dlt and DuckDB
Most pipeline code is glue nobody wants to maintain. dlt and DuckDB let you skip the glue and keep the parts that matter — schema inference, incremental loading, and contracts that fail loudly instead of silently corrupting your warehouse.
|6 min readColumn-level data lineage in practice
Table-level lineage tells you two models are connected. Column-level lineage tells you that changing one field breaks exactly these three downstream columns and nothing else. That precision is the difference between a confident change and a prayer.
|5 min readSQL window functions: a practical guide
Window functions are the single highest-leverage SQL feature for analytics work, and the frame clause is the part everyone skips — right up until their running total is silently wrong. Here's the whole thing, practically.
|5 min read