A billing-reversal model on 1.07 million real wholesale invoice lines. Every cancellation tied back to the invoice it reverses; a model that says, as the invoice is raised, how likely it is to come back; and a check-before-dispatch list with a cut-off where checking stops paying. Judged against the customer's own record, built with no lookahead, exported to plain JavaScript and scoring live on this page.
A credit note is the purest billing leak: the sale happened, the invoice went out, and then the ledger took it back. Some of that is the customer changing their mind; some is the wrong quantity, the wrong price, the wrong product — the discrepancies that case 01 cut by 30% by fixing the ledger. Either way it costs twice: once to raise and ship, once to credit and handle. This model asks the question the day the invoice is raised: how likely is this one to come back, and is it worth a look before it goes out the door?
The UCI Online Retail II dataset (Chen, 2019): 1,067,371 invoice lines, December 2009 to December 2011, 5,942 customers in 43 countries. Cancellations carry an invoice number prefixed "C" and negative quantities — which means the dataset records the reversal but not what it reverses. That link had to be rebuilt, and it is the first thing the page does.
| FIELD | WHAT IT IS | USED AS |
|---|---|---|
| Invoice | Invoice number; "C…" for a cancellation | The unit of the model; the C-prefix marks a reversal |
| StockCode · Description · Quantity · Price | Product, its name, units and unit price | Lines, products, value, the largest line, price against the product's usual price |
| InvoiceDate | Timestamp to the minute | Time order for history; hour, weekday, month |
| Customer ID · Country | Masked customer; country of the customer | History per customer; UK / EIRE / elsewhere |
| Target | Did any line of this invoice get credited back later? | reversed = 1 · 17.8% of invoices · plus the credited value and the lag |
Hygiene. 6,093 lines carry pseudo stock codes — POST, DOT, M (manual), BANK CHARGES, ADJUST — and are dropped; they are the ledger's plumbing, not products. 22.8% of lines have no customer id (guest and cash sales) and cannot carry history, so they stay out of the model and are declared here. Zero-price lines (0.6%) go. And one line — 80,995 units of "Paper Craft, Little Birdie", £168,470, cancelled the same afternoon — is a quarter of all matched credit on its own. It is real and it stays; the model sees invoice value on a log scale so one order does not own the fit.
The reconciliation. Each cancellation line is matched to the most recent sale of the same product to the same customer within 120 days. 85.5% of cancellation lines match, 93.4% by value; the rest are guest sales, pseudo codes and older returns. The lag between invoice and reversal is the chart below — a median of nine days, a tenth on the same day, one in ten past seven weeks.
No lookahead. The strongest feature is the customer's record: how often their invoices have come back. Built carelessly it leaks twice — by using invoices raised after this one, and by counting a reversal that had not yet happened. So history uses only earlier invoices, and an earlier invoice counts as reversed only if its cancellation date is before this invoice's date. The protocol: train on invoices raised before 1 May 2011, test on May to August, and leave September to December for the test invoices' reversals to arrive.
# tie each cancellation to the sale it reverses (train.py) m = pd.merge_asof(cancellations.sort_values("date"), sales.sort_values("date"), left_on="date", right_on="sdate", by=["cust", "stock"], direction="backward", tolerance=pd.Timedelta(days=120)) # a prior invoice counts as reversed only if the cancellation had already happened by "today" known = prior & ~isnat(first_cancel) & (first_cancel < t) f["prior_reversed_rate"] = known.sum() / prior.sum()
| PRODUCT | CREDITED | CANCELLATION LINES |
|---|
| MODEL | AUC | PR-AUC | BRIER | NOTE |
|---|
| SEGMENT | N | AUC | ACTUAL RATE | MEAN SCORE |
|---|
Two honest gaps. Under £100 the AUC is 0.55: almost nothing that small comes back, so there is little to rank — and little to lose. A customer's first invoice scores 0.64: no record yet, only the invoice itself. Everywhere else the model sits between 0.70 and 0.77, and its mean score tracks each segment's actual rate.
Rank invoices by probability × value, check the top k% before dispatch, and every check costs something while only some would-have-come-back invoices are fixed because someone looked. Move the two levers; the model's job is to make the best k visible.
| CARD | |
|---|---|
| Intended use | Decide which invoices get a human look before dispatch, and where that looking stops paying. A prioritisation tool; it never blocks an order on its own. |
| Training data | Online Retail II, one UK wholesaler, invoices with a customer id raised Dec 2009 – Apr 2011 (24,012); tested on May – Aug 2011 (5,522). |
| Not valid for | Any other ledger. A "reversal" here mixes billing errors with order edits and stock returns — the data cannot tell them apart, and so the model predicts "comes back", not "was wrong". Re-fit, and label the reason, before trusting a number. |
| Known failure modes | First invoices (AUC 0.64 — nothing to learn from yet). Tiny invoices (nothing to rank). One £168k same-day cancellation is a quarter of matched credit; the fit is protected by the log scale, the economics are not. Matching by customer + product + recency can tie a return to the wrong of two recent invoices. |
| Monitoring | Monthly, once reversals have had 90 days to arrive: calibration by decile, credit captured in the top two deciles, matching rate of new cancellations (a fall means a process change), PSI on prior_reversed_rate and log_value. |
| Explainability | Per-invoice path contributions (shown above), which sum exactly to the model's output; global importance by split gain; the baseline published beside every metric. |
The cheapest credit note is the one you never raise. One invoice in six on this ledger comes back; the money is concentrated in big orders from customers who have done it before, and a model that reads the invoice on the day it is raised puts half of the reversals — and a quarter of the credited value — in its top decile. It cannot say whether the reversal will be the customer's fault or the ledger's; that is the label a real engagement adds first. What it can do is make sure that the ten invoices worth a second look are the ten someone actually looks at.
Raising credit notes faster than you can explain them? A reversal model on your own invoices — with the reconciliation that ties every credit to its cause — is a Seal-sized piece of work: two to four weeks, fixed fee, check-list and model card included. It starts with a TRACE of where the credits come from.
Start with a TRACE → train.py README model.json