T3.2Interactive artifact · P3 · Data quality & lineage

Pipeline health dashboard: freshness, SLAs and schema drift

A simulated monitoring sheet for ten data pipelines: how fresh each one is against its SLA, how its row counts are trending, and whether its schema contract still holds. Click a pipeline for its run history.

Title blockT3.2
Type
Data tool
Drawn
Runs
In your browser, nothing to install
Tags
data-engineering · data-quality · monitoring · dashboard

Detail A · Reading the drawing

What it shows

Most data incidents aren't crashes. They're a table that quietly stopped updating, a load that brought in half its usual rows, or a source that renamed a column. This dashboard puts those three signals side by side for every pipeline: freshness against its SLA, the row-count trend, and schema drift.

The simulation includes a few deliberate incidents among healthy pipelines: a streaming job with repeated failures, a dbt model past its freshness SLA, a vendor API that added columns, and one that removed a column outright. Faults are marked in redline; everything else stays in ink.

Detail B · Procedure

How to use it

  1. Scan the summary strip for failing pipelines and the average SLA.
  2. Read the active alerts, newest first: SLA breaches, failed runs and schema changes.
  3. Filter the status table to failing or warning pipelines.
  4. Click a pipeline, or focus it and press Enter, to open its row-count chart and last ten runs.
  5. Switch the time range to redraw the sheet.

Detail C · Glossary

Key concepts

Freshness
Time since a pipeline last delivered data, measured against the SLA it has to meet.
Freshness SLA
The agreed maximum staleness for a dataset, e.g. orders are never more than 20 minutes old.
Row-count anomaly
A load far above or below its usual volume, often the first sign of a broken upstream.
Schema drift
Columns added, removed or retyped at the source. Additions are a warning; removals and type changes break consumers.
Data contract
An explicit agreement on schema and freshness between a producer and its consumers, checked on every load.

Detail D · Cited by

Related writing

Posts that use this drawing as a worked example.

  • D3.2
    Data freshness and schema-drift alerting

    A pipeline can run successfully and still be wrong. The two failures that matter most are silent by nature — the data stopped arriving, or it quietly changed shape — and neither shows up in a green DAG. Here's what to measure, and how lineage turns an alarm into an assignment.

  • D3.1
    Column-level data lineage in practice

    Table-level lineage tells you two models are connected. Column-level lineage tells you that changing one field breaks exactly these three downstream columns and nothing else. That precision is the difference between a confident change and a prayer.

All artifacts →