At the end of the column-level lineage post I made a claim in passing: a drift alert that can name the downstream columns at risk is worth far more than an undifferentiated alarm. This is the worked version of that claim — what to actually monitor, and how the lineage graph turns a monitoring signal into something a person can act on.
The premise is that orchestration status is a weak proxy for correctness. A green run means the job exited zero. It does not mean rows arrived, that they arrived in the shape you expected, or that the numbers downstream are still true. The two failure modes that most reliably cause a wrong dashboard are both invisible to the scheduler: the data stopped arriving, and the data changed shape. Freshness monitoring catches the first. Schema-drift detection catches the second.
Freshness: pick the right clock#
Freshness sounds like one number and is actually two, and conflating them is the most common mistake in this area.
Load time is when a row landed in your warehouse. Event time is when the thing the row describes actually happened. A pipeline that runs hourly and loads zero rows has perfect load-time freshness on the pipeline metadata and completely stale data. An upstream system that backfills a week of history has excellent event-time coverage and a load timestamp that says everything is minutes old.
You want both, and they answer different questions. Load time answers is the pipeline moving? Event time answers is the world represented here current? Alert on load time and you catch broken plumbing; alert on event time and you catch an upstream system that has quietly stopped producing.
In dbt this is expressed as source freshness — you name the column that carries the timestamp and give it thresholds:
sources:
- name: raw_events
loaded_at_field: _loaded_at
freshness:
warn_after: {count: 6, period: hour}
error_after: {count: 24, period: hour}
tables:
- name: orders
- name: sessionsdbt source freshness then computes the age of the newest row per table and fails on the thresholds. The mechanism is simple enough that if you're not using dbt you can reproduce it in a few lines: a table of (dataset, last_loaded_at, last_event_at, checked_at) updated at the end of every load, and a query that compares those timestamps to now.
The subtlety is what "now" should be compared against. A wall-clock threshold — error if older than 24 hours — generates a false alarm on every table whose source only produces on weekdays, and misses a table that should update every fifteen minutes but is allowed a full day of slack. Freshness thresholds belong per dataset, derived from its expected cadence, not set globally. The threshold I default to is roughly two missed intervals: enough to absorb one late run without paging anyone, tight enough that a genuinely dead source surfaces the same day.
Freshness monitoring is a dead-man's switch, and dead-man's switches have to run independently of the thing they watch. If the freshness check is a task inside the pipeline DAG, then a pipeline that never starts also never checks itself — the one failure you most needed to catch is the one that disables the detector. Run it on its own schedule, against the warehouse.
Row count is the companion signal and costs almost nothing to add. A load that arrived on time with three percent of its usual volume is a failure that freshness alone scores as healthy. Comparing each load's row count to a trailing median for the same weekday catches partial extracts, a truncated API page, and a filter someone tightened upstream.
Drift: detect it at the boundary, not downstream#
Schema drift is the second failure, and the useful move is to catch it at the ingestion boundary rather than three transformations later when a metric looks odd.
That's exactly what schema contracts are for. In the dlt and DuckDB post I argued for evolving schemas in development and freezing them in production, across the levels dlt exposes — tables, columns, and data types. Freezing converts a silent change into a loud, located error: the pipeline stops and tells you which field moved. That failure is itself your drift detector, and it's a better one than anything you'd bolt on afterwards, because it fires before the bad data lands.
Not all drift is structural, though, and the residual cases are worth watching explicitly:
- A new column appears. Usually benign, occasionally a signal that upstream split a field in two and the old one is now half-populated.
- A type changes. An integer arrives as a string, or a timestamp loses its timezone. Contracts catch this cleanly.
- A column disappears or goes all-null. Structurally the table is fine. Every metric derived from that column is now silently zero.
- Cardinality or nullability shifts. The column is present and correctly typed, and its distribution moved — an enum gained a value nobody's
CASEstatement handles.
The last two don't trip a schema contract at all, which is why null-rate and distinct-count checks per column belong alongside it.
Where lineage earns its keep#
Freshness and drift detection are individually straightforward. The reason most teams still ignore their alerts is that the alerts are undifferentiated: every one arrives with the same urgency, most of them are about datasets nobody depends on, and the signal drowns.
Column-level lineage fixes the differentiation problem, because it can answer the only question that determines how much an alert matters — what depends on this? Given the column graph, a drift event on orders.amount resolves to a concrete downstream set: revenue.gross, finance.mrr, and the two dashboards built on them. That gives you three things a bare alert can't:
Severity by blast radius. An alert on a column with no downstream consumers is a ticket. An alert on a column feeding the revenue model is a page. The graph computes that distinction for you rather than asking a human to recall it at 2am.
Routing. The owner of the downstream model is usually a different person from the owner of the ingestion job, and they're the one who needs to know a number is about to be wrong. Lineage supplies that mapping.
Suppression. When a source goes stale, every model downstream of it also goes stale, and naive freshness monitoring emits one alert per affected dataset. Walking the graph lets you fire once at the root and mark the rest as consequences.
Loading artifact: pipeline-dashboard...
The pipeline dashboard artifact is the shape of this made concrete: freshness per pipeline, row counts as sparklines so the trend is visible rather than a single threshold, and drift flagged inline with the run it came from. Click into a pipeline to see the individual runs.
What I'd actually do#
Start with a freshness table in the warehouse, written at the end of every load and checked on an independent schedule, with per-dataset thresholds set to about two missed intervals. Add trailing row-count comparison in the same query — it's one more column and it catches a whole class of partial-load failures. Freeze schema contracts on anything feeding production so drift fails at the boundary. Then, once that's running, join the alerts to the column graph you're already generating in CI, and use the blast radius to decide severity and recipient.
The order matters: detection first, because an alert nobody can prioritise is still better than not knowing; lineage second, because it's what makes the alerts survive contact with a busy team. That sequencing is the same instinct I describe in how I build AI-native — put the effort into the systems that keep being right, rather than the heroics of noticing manually.
Green pipelines are not the goal. Correct data is, and the gap between the two is exactly what these two signals cover.