T2.1Interactive artifact · P2 · Data pipelines / ingestion

dlt pipeline sandbox: dlt → DuckDB in the browser

A simulator for the parts of dlt that decide whether a pipeline is safe to re-run: write dispositions, incremental cursors and schema contracts. Run it a few times and watch the consequences land in DuckDB tables.

Title blockT2.1
Type
Data tool
Drawn
Runs
In your browser, nothing to install
Tags
data-engineering · simulation · interactive · developer-tool

Detail A · Reading the drawing

What it shows

dlt (data load tool) is an open-source Python library that extracts, normalizes and loads data declaratively. DuckDB is an embedded analytical database. The sandbox models the dlt behaviour that catches people out in production: what happens on the second run, and what happens when the source changes shape.

The destination keeps state between runs, so merge, append and replace diverge exactly as they would for real: upserts by primary key, silent duplicates, or a truncate and reload. Nested JSON from the REST API source is unnested into a child table linked by _dlt_parent_id, the way dlt's normalizer does it.

Detail B · Procedure

How to use it

  1. Pick a preset, or choose a source (REST API, CSV files, Postgres), a write disposition and a schema contract.
  2. Run the pipeline two or three times and compare the row counts in shop.orders and shop.orders__items.
  3. Toggle the incremental cursor to see stale re-deliveries skipped, or loaded again.
  4. Arm inject schema drift, run again, and see how evolve, freeze, discard_value and discard_row each respond.
  5. Copy the generated dlt Python code, or the prompt for Claude.

Detail C · Glossary

Key concepts

write_disposition
How a load lands: append adds rows, replace truncates and reloads, merge upserts on the primary key.
Incremental cursor
dlt.sources.incremental remembers the highest updated_at it has seen, so the next run only extracts newer records.
Schema contract
The rule for unexpected columns: evolve adds them, freeze fails the load, discard_value strips them, discard_row drops the record.
Normalization
Nested lists become child tables (orders__items) linked to their parent row by _dlt_parent_id.
_dlt_id
The row identifier dlt adds to every loaded record; child rows point back to it.

Detail D · Cited by

Related writing

Posts that use this drawing as a worked example.

  • D2.2
    dlt REST API sources: declarative pagination without the glue

    The schema glue is gone, but most teams still hand-write the other half of every API pipeline — pagination loops, auth headers, and "what changed since last time." dlt's REST API source turns all of it into configuration. Here's the declarative shape, end to end.

  • D2.1
    Data pipelines with dlt and DuckDB

    Most pipeline code is glue nobody wants to maintain. dlt and DuckDB let you skip the glue and keep the parts that matter — schema inference, incremental loading, and contracts that fail loudly instead of silently corrupting your warehouse.

All artifacts →