# The Data Sentinel — the table that reports its own fever

Part of **The Engine Room** on simranjaiswal.in: automation engines that stop money going missing
between systems. This one is the WATCH stage for the tables under every report: six checks on every
table every morning, with printed thresholds, a quarantine that stops a wrong number reaching a
report, and a fever chart so the week can be read at a glance.

Live case, with the engine running in the browser: https://simranjaiswal.in/work/data-sentinel

## What it does

Four tables land a new partition every morning (customers 2,000 rows, invoices 45,000, payments
30,000, arrears_snapshot 850,000; ± 3% daily jitter). Six checks run on each, in order:

1. **C1 volume drift** — today's rows against the median of the last 7 days (seeded with the printed
   daily volume): |Δ| > 15% WARN (a slider), > 35% CRITICAL.
2. **C2 null rate** — each key column against its printed baseline (email 4%, gstin 12%; po_ref 6%,
   due_date 0.5%; invoice_id 1%, utr 3%; segment 2%, balance 0%); the worst column decides:
   +5 pp WARN, +15 pp CRITICAL.
3. **C3 duplicate keys** — > 0.1% WARN, > 1% CRITICAL.
4. **C4 referential integrity** — payments whose invoice_id matches no invoice: > 0.2% WARN, > 2%
   CRITICAL (payments only).
5. **C5 freshness** — the partition normally lands 05:45 ± 30 min; after 07:00 WARN, missing at 09:00
   CRITICAL.
6. **C6 distribution shift** — PSI of 400 sampled amounts against a 30-day baseline frozen before day
   0, over 10 printed buckets (decile edges of the table's lognormal): > 0.10 WARN, > 0.25 CRITICAL.
   Applies to invoices, payments and arrears_snapshot.

**Escalation.** A WARN on a check that also warned within the previous two days becomes CRITICAL.

**Policy.** WARN → a line in the 09:05 digest. CRITICAL → the table is QUARANTINED and its owner is
paged; the six downstream reports that read it (each report's tables are printed) run on yesterday's
snapshot, flagged. A quarantined table is cleared by its owner next morning with p 0.75 (one draw per
quarantined table per morning); until then its status is QUARANTINED. Every table-day is therefore
exactly one of OK / WARN / CRITICAL / QUARANTINED, and the tie-out (240 = the sum) refuses to publish
if it is not.

"Wrong reports prevented" = report-days on which a report read a trusted snapshot because a table it
reads was in quarantine. "Caught" = the check meant for an incident fired on the day it was injected
(same day by construction — the checks run on the partition the morning it lands, so the mean
time-to-detect is 0 and is not a result). "False alarm" = an alert on a table-day with no injected
incident.

## The synthetic sixty days

There is no client data here. Incidents are injected per day with the slider probabilities, one draw
per type in this order, then one draw for the table it hits: partial load (rows × 0.6, p 0.05),
schema change nulling a key column (10–100% of rows, drawn; p 0.04), re-run loading a day twice
(rows × 2, half the keys duplicated; p 0.03), upstream outage (lands 07:00–11:00, drawn; p 0.06),
FX / pricing shift (log-mean +0.15 to +0.60, drawn; p 0.04, amount tables only), orphan payments from
a late invoice feed (0.3–5.3%, drawn; p 0.05, payments only). `--storm 1` is "the bad fortnight":
days 20–33 with every probability × 4. Draw order within a table-day is volume, nulls per column,
duplicates, orphans, arrival time, then the 400 amounts.

Randomness is mulberry32, a 32-bit generator the browser port implements bit-for-bit, so the page's
reference run reproduces `results.json` exactly (the page's status line says so when it does).
Rounding for printed values is half-up via floor(x·10ᵈ + 0.5) on both sides.

## Run it

```
python3 engine.py                          # reference: seed 42, probabilities 5/4/3/6/4/5%, volume WARN 15%
python3 engine.py --storm 1                # the bad fortnight
python3 engine.py --vol-warn 5             # a tight threshold: watch the false alarms and escalations
python3 engine.py --partial .01 --schema .01 --rerun .01 --outage .01 --shift .01 --orphans .01   # quiet
node parity.mjs                            # proves the browser port reproduces results.json
```

Stdlib only. `results.json` carries the printed tables (baselines, bucket edges, baseline shares), the
60 × 4 status grid with the six severities per cell, every alert with its value and detail, every
incident with whether it was caught, the fever series, PSI by day per amount table, alerts by check,
incidents by type, the per-table and per-report tallies, and a log sample.

## Reference run (seed 42)

See `results.json`. Headline: 240 table-days; 18 alerts (5 WARN, 13 CRITICAL); 12 pages; 15
quarantine table-days; 18 incidents injected, 17 caught the same day (one small pricing shift, PSI
below 0.10, missed); 0 false alarms at the 15% volume threshold; 54 of 360 report-runs kept on a
trusted snapshot.

## Where the shape comes from

The checks Simran ran on a UK energy supplier's arrears path (8.5M-row tables and a tie-out that
refused to publish; https://simranjaiswal.in/work/arrears-path) and the figure-by-figure
reconciliation behind a DREQ rebuilt on Databricks (https://simranjaiswal.in/work/dreq-databricks).
Outcome figures in those cases are as reported there; nothing in this repository is a client's data.

MIT.