Every report is only as right as the tables under it, and tables go wrong quietly: a load that stopped at 60%, a renamed column arriving empty, a day loaded twice, a feed that landed at ten past nine. This engine is the WATCH stage for those tables: six checks every morning with printed thresholds, a quarantine that makes a report use yesterday's numbers rather than today's wrong ones, and a fever chart so the week can be read at a glance. It is running below, in your browser, on 60 days you can make as bad as you like.
A wrong report is rarely wrong because the SQL is wrong. It is wrong because the table it read was: a partition that landed half-full, a schema change upstream that turned a key column into nulls, a re-run that loaded Tuesday twice, a pricing change that shifted every amount by a third and made last week's baseline a lie. None of these announce themselves. The report runs, the dashboard refreshes, and someone makes a decision on it. In the arrears work, where a wrong number became a letter to a customer in debt, the answer was a tie-out that refused to publish; in the Databricks rebuild it was reconciling every figure to the old stack before anyone switched. The Data Sentinel is that habit written as rules and run every morning: a table that reports its own fever before a report can catch it.
Nothing in the engine is clever. Each check is one sentence with two numbers in it: a WARN threshold and a CRITICAL one. The thresholds are printed, the baselines are printed, and the distribution check uses ten buckets whose edges are printed too. What makes it trustworthy is the policy, not the statistics: a CRITICAL does not raise a ticket, it stops the table being read. Downstream reports do not fail; they run on yesterday's snapshot with a flag on them, which is the difference between a report that is a day old and a report that is wrong.
| CHECK | RULE | WARN → CRITICAL |
|---|---|---|
| C1 · volume drift | Today's rows against the median of the last 7 days (seeded with the printed daily volume: 2,000 / 45,000 / 30,000 / 850,000) | |Δ| > 15% (slider) → |Δ| > 35% |
| C2 · null rate | Each key column against its printed baseline (email 4%, gstin 12% · po_ref 6%, due_date 0.5% · invoice_id 1%, utr 3% · segment 2%, balance 0%); the worst column decides | +5 pp → +15 pp |
| C3 · duplicate keys | Share of rows whose key already exists in the partition | > 0.1% → > 1% |
| C4 · referential integrity | Payments whose invoice_id matches no invoice (payments only) | > 0.2% → > 2% |
| C5 · freshness | When the partition landed (normally 05:45 ± 30 min) | after 07:00 → missing at 09:00 |
| C6 · distribution shift | PSI of today's 400 sampled amounts against a 30-day baseline, frozen before day 0, over 10 printed buckets (invoices, payments, arrears_snapshot) | > 0.10 → > 0.25 |
| Escalation | A WARN on a check that also warned within the previous two days becomes a CRITICAL | two WARNs in 3 days → CRITICAL |
| Policy | WARN → a line in the 09:05 digest. CRITICAL → the table is QUARANTINED and its owner is paged; the six reports that read it use yesterday's snapshot, flagged. A quarantined table is cleared by its owner next morning with p 0.75 (drawn); until then it stays out. | OK / WARN / CRITICAL / QUARANTINED |
| Incidents | Injected per day with the slider probabilities, one draw per type, then one draw for the table: partial load (rows × 0.6), schema change (a key column arrives 10–100% null), re-run (rows × 2, half the keys duplicated), outage (lands 07:00–11:00), pricing shift (log-mean +0.15 to +0.60), orphan payments (0.3–5.3%) | caught = the check meant for it fired |
# the escalation and the policy — the sentences a controller can disagree with (engine.py) for c in range(6): if sev[c] == 1: # a WARN today... if last_warn[ti][c] >= day - ESCALATE_WINDOW: # ...that also warned in the last 2 days sev[c] = 2; escalations += 1 # becomes CRITICAL last_warn[ti][c] = day if max(sev) == 2: status = "CRITICAL"; quarantined[ti] = True; pages += 1 # reports use yesterday's snapshot elif quarantined[ti]: status = "QUARANTINED" # not yet cleared (owner clears with p 0.75) elif max(sev) == 1: status = "WARN"; digest += 1 # a line in the 09:05 digest
Run the sixty days to see what the sentinel stops.
Two things to try. Drag the volume WARN threshold down to 5%: the daily jitter is 3%, so the sentinel starts warning on ordinary mornings, the false-alarm count climbs, and the two-WARNs-in-three-days rule begins turning noise into CRITICALs and quarantining healthy tables. A threshold is a choice about how many pages the owner will tolerate, which is why it is a slider. Then tick the bad fortnight: the fever line in 05 spikes for two weeks, and the number that matters is not how many alerts fired but how many reports ran on a trusted snapshot instead of a wrong one.
| WHEN | WHAT HAPPENS | WHO SEES IT |
|---|---|---|
| 05:45 daily | The four partitions land from their sources. C5 starts its clock: after 07:00 the table is WARN, missing at 09:00 it is CRITICAL. | Nobody yet. The clock is the check. |
| 09:00 | Six checks per table against printed baselines; two WARNs on one check in three days escalate. Any CRITICAL quarantines the table and pages its owner; the six reports that read it run on yesterday's snapshot, flagged. | The table owner (a page). Report readers see a flag, never a wrong number. |
| 09:05 | The digest: every WARN with its value and threshold, every table in quarantine and for how long, every report running on a snapshot. | The data team, over coffee. Nothing in it is a surprise by 09:06. |
| Next morning | The owner backfills and clears the quarantine; the checks run again on the new partition. A table that fails again is paged again. | The owner. Quarantine is a state, not a ticket. |
| Weekly | The fever line and the false-alarm count on one page. Thresholds move only when that page says the sentinel is crying wolf or sleeping through fires. | Leadership — the WATCH stage of the Leak Ledger. |
A data-quality check that only raises a ticket is a diary. The check that matters is the one wired to a consequence: the table stops being read, the owner is paged, and the reports keep running on numbers that were true yesterday rather than numbers that are wrong today. Print the thresholds, freeze the baseline on purpose, count the false alarms as honestly as the catches, and the tables under your reports will tell you they are ill before your readers do.
Reports nobody quite trusts? A data sentinel is a two-to-three-week SEAL: your tables, your thresholds, your owners — six checks every morning with a quarantine that stops a wrong number ever reaching a report.
Start with a TRACE → engine.py README results.json The case it came from →