Tagged “data-quality”
4 posts
Semantic dbt tests on Databricks: 50,000 real reviews through a Unity Catalog function
I took four dbt tests written as English sentences off my laptop and onto Databricks. The schema.yml didn't change. Everything underneath it did, and the 50,000-review production run failed its own gate by one point of recall, which turned out to be the most interesting part.
|13 min readSemantic dbt tests: data quality checks in plain English
not_null and accepted_values will happily pass a customer called Mickey Mouse, a return coded late whose comment says the box arrived crushed, and a ticket with someone's IBAN in it. So I wrote four dbt tests as English sentences and scored them against an answer key.
|12 min readData freshness and schema-drift alerting
A pipeline can run successfully and still be wrong. The two failures that matter most are silent by nature — the data stopped arriving, or it quietly changed shape — and neither shows up in a green DAG. Here's what to measure, and how lineage turns an alarm into an assignment.
|7 min readColumn-level data lineage in practice
Table-level lineage tells you two models are connected. Column-level lineage tells you that changing one field breaks exactly these three downstream columns and nothing else. That precision is the difference between a confident change and a prayer.
|5 min read