# The invoice that comes back — a billing-reversal model on real wholesale transactions

A machine-learning project by Simran Jaiswal on the billing leak itself: money that was invoiced and
later credited back. It ties every cancellation to the invoice it reverses, predicts at invoice time
which invoices will come back, and turns that into a **check-before-dispatch list with an economic
cut-off** — the TRACE and SEAL stages of the Leak Ledger, as a model.

Live case, with an in-browser scorer: https://simranjaiswal.in/work/billing-reversals

## Data

UCI Machine Learning Repository, *Online Retail II* (Chen, 2019): 1,067,371 invoice lines from a UK
online wholesaler of giftware, December 2009 – December 2011, 5,942 customers in 43 countries.
Cancellations are recorded as their own invoices (numbers prefixed "C") with negative quantities.
Public, real, anonymised (customer ids only).

After cleaning (guest sales without a customer id, pseudo stock codes such as POST / DOT / M / BANK
CHARGES, zero prices): **36,594 invoices for 5,852 customers, £17.4M billed; 6,505 invoices (17.8%)
come back at least in part, £672k credited (3.9%)**. Most reversals are partial — the median reversed
invoice loses 4% of its value; 4% are reversed in full.

## Pipeline (`train.py`)

1. **Reconciliation** — each cancellation line is matched to the most recent sale of the same product
   to the same customer within 120 days (`merge_asof`). 85.5% of cancellation lines match (93.4% by
   value); median lag 9 days, 90% within 51 days. The unmatched rest is guest sales, pseudo codes and
   returns older than the window.
2. **Features known when the invoice is raised** — value, lines, distinct products, quantity, the
   largest line's share, the share of products new to this customer, price against the product's
   usual price (from the training period only), country, hour, weekday, month — and the customer's
   history: prior invoices, prior reversal rate and credited share, whether the last invoice came
   back, days since the last order, tenure. **No lookahead:** history uses only earlier invoices, and
   a prior invoice counts as reversed only if its cancellation had already happened by then.
3. **Time split** — train on invoices before 2011-05-01 (24,012); test on May–August 2011 (5,522), leaving
   reversals three months to arrive before the data ends on 9 December 2011.
4. **Models** — logistic regression (scaled) and gradient boosting (250 trees, depth 3).
   Baseline: the customer's own prior reversal rate.
5. **Evaluation** — AUC / PR-AUC / Brier, calibration by decile, lift, AUC by country, history depth
   and invoice value; the credited value captured by decile.
6. **Economics** — rank by P(reversed) × value; check the top *k*% before dispatch at a cost per
   check; a check on an invoice that would have come back recovers a share of its credit. The page
   lets you move both.
7. **Export** — trees to `model.json`; the page reproduces sklearn to < 1e-6 and explains each score
   with path contributions.

## Results (5,522 held-out invoices, May–Aug 2011, base rate 18.4%)

| model | AUC | PR-AUC | Brier |
|---|---|---|---|
| customer's prior reversal rate (the baseline) | 0.677 | 0.319 | 0.153 |
| logistic regression | 0.732 | 0.399 | 0.134 |
| gradient boosting | **0.735** | **0.429** | **0.131** |

Top decile: 51% of invoices come back (2.8× the base) and it holds a quarter of all credited value in
the test period. Calibrated within a few points in every decile. AUC by segment 0.70–0.77, except
invoices under £100 (0.55 — almost nothing under £100 comes back, so there is nothing to rank) and a
customer's first invoice (0.64 — no history to learn from).

What carries the signal: invoice value (29% of split gain — bigger invoices come back more: 31% above
£3,000 vs 7% below £100) and the customer's prior reversal rate (28%), then prior reversal count,
distinct products, largest-line share.

## Reproduce

```bash
python -m venv .venv && ./.venv/bin/pip install -r requirements.txt openpyxl
# download online+retail+ii.zip from UCI (dataset 502), export the two sheets to data/online_retail_ii.csv
# with columns Invoice, StockCode, Description, Quantity, InvoiceDate, Price, Customer ID, Country, cancel, value
./.venv/bin/python train.py
```

## Limitations, stated plainly

- One wholesaler, two years, giftware. A "reversal" here mixes genuine billing errors with customers
  editing orders and returning stock; the data cannot tell them apart, and neither can the model.
- One line — 80,995 units of "PAPER CRAFT, LITTLE BIRDIE", £168k, cancelled the same day — is a
  quarter of all matched credit. It is real and it stays in; the model sees invoice value on a log scale.
- Matching is by customer + product + recency; a cancellation of a product bought on two recent
  invoices is tied to the latest one.
- The economics are illustrative: cost per check and recovery share are inputs, not findings.

## Files

- `train.py` — reconciliation + pipeline, one file, deterministic (`SEED = 42`)
- `model.json` — exported trees for in-browser scoring
- `evaluation.json` — results, hygiene, reconciliation, curves, segments, economics, tables, presets, a 1,000-invoice reference sample
- `requirements.txt`

MIT for the code. Dataset: Chen, D. (2019). *Online Retail II*. UCI Machine Learning Repository.
https://doi.org/10.24432/C5CG6D