Tuning Anomaly Radar to under 2% false positives: what we changed

The first version of Anomaly Radar, built in late 2024, used a single global threshold: flag any transaction more than three standard deviations from the mean transaction size for its category. It was simple, it was fast to build, and within three weeks of our first pilot customer going live, the finance lead asked us to turn it off.

The problem wasn't that it was wrong exactly — most of what it flagged was, technically, statistically unusual. The problem was that it flagged so much unremarkable-but-unusual activity that nobody could tell the difference between a real problem and noise. A quarterly software renewal, a one-off equipment purchase, a slightly larger-than-usual client refund — all statistically outlying, none of them actually a problem, and all mixed in the same undifferentiated alert feed as the handful of things that genuinely mattered. Within three weeks, the finance lead had learned to ignore the alerts entirely, which is the worst possible outcome for a fraud-detection tool: not a false negative on one transaction, but a trained-in blindness to the entire feed.

The fix wasn't a better threshold, it was a different unit of comparison

A global, category-wide threshold treats a €2,000 invoice from a vendor that always bills €2,000 the same as a €2,000 invoice from a vendor that has never billed more than €400. Both might sit at the same absolute distance from the category mean, but only one of them is actually unusual for the specific relationship it's part of. The rebuild replaced the single global model with a baseline computed per vendor (or per customer, for inbound anomalies), built from that specific counterparty's own history — median amount, typical frequency, day-of-month pattern, and how those have trended over the trailing two quarters.

That single change — comparing a transaction to its own counterparty's history instead of to a category-wide distribution — did more to cut false positives than any amount of threshold tuning had. It also meant the system needed roughly three payment cycles with a new vendor before it could say anything confident about drift, which we made explicit in the product rather than pretending day-one detection was as strong as month-three detection.

Splitting one model into three narrower ones

The second change was giving up on a single unified "anomaly score" in favor of three separate, narrower detectors, each tuned against a specific failure mode rather than anomalies-in-general:

• Duplicate-payment matching — amount, payee, and a timing window, tolerant of small reference-number differences, tuned specifically against the double-payment and near-duplicate-invoice pattern.
• Vendor-drift detection — changes to bank details, a first payment at a new amount tier, or a frequency change for an established vendor, tuned against the shape business email compromise and invoice fraud actually take.
• Category-spend outliers — a category's total spend deviating from its own trailing pattern, tuned for the subscription-creep and budget-overrun cases rather than fraud specifically.

Each detector only fires within its own narrow definition of unusual, which means each one can be validated and tuned independently against labeled outcomes, instead of trying to find one threshold that works for three different underlying problems at once.

What we gave up to get there

It would be dishonest to present this as a pure win. Tightening the model to per-vendor baselines and narrower detectors reduced recall on genuinely novel anomaly patterns — the kind of one-off, never-seen-before fraud that doesn't fit any of the three shapes above. We accept that trade-off deliberately: a system that catches 100% of anomalies in theory but gets ignored in practice catches roughly 0% of them in reality. A narrower system that finance teams actually read every flag from catches meaningfully more, even though its theoretical coverage is smaller.

We validate the false-positive rate against labeled outcomes from live customer accounts — every flag gets marked resolved, expected, or false-positive by an actual finance user, and we track the trailing 90-day false-positive rate against that labeled set rather than against a synthetic test suite. That number is currently under 2% across live customer transaction volume, and it's the metric we watch most closely before shipping any change to the models, because it's the one number that predicts whether a customer keeps the alerts turned on six months in.