What the global track record was hiding

On 2026-05-21 we shipped an isotonic refit that collapsed the forecast's headline calibration drift from +56.45pp to ~0pp. That looks great on the global track record at /v1/atlas/prediction-track-record — but “global ~0pp” is fully consistent with individual countries still drifting in opposite directions and silently cancelling each other out in the aggregate.

Iran can be well-calibrated while Egypt's slice is +20pp under-predicting and Venezuela's is −15pp over-predicting. Aggregate to a single number and the picture looks healthy. Slice it per country and we see the truth.

What this monitor does

Every day at 05:00 UTC, scripts/build-per-country-calibration-drift.py walks the top-50 most-active forecast countries and computes:

• Mean predicted probability over the trailing 30 days
• Empirical positive rate (censorship/mixed incidents only — IODA disruption rows excluded, matching the forecast model's post-2026-05-21 labeling rule)
• Calibration drift in percentage points
• Per-country Brier

Any country with drift exceeding ±15pp (and at least 15 predictions in the window) gets flagged, the sidecar at /opt/voidly-ai/ml-deploy/calibration_drift_by_country.json is updated, and a calibration_drift event fires into the existing CenAlerts pipeline with 24h dedup.

Endpoints

• GET /v1/sentinel/calibration-drift — full table, sorted by absolute drift desc. Supports ?status=drifting|ok|insufficient_data and ?min_n=N filters.
• GET /v1/sentinel/calibration-drift/{cc} — single-country slice.
• GET /v1/sentinel/calibration-drift/info — methodology + honest caveats.

How it reads

Each per-country row carries the rebuilt observed labels (so we don't ding the model for IODA disruption rows it wasn't trained to predict), the median operational threshold actually emitted in the window, and a flag that toggles to drift_alert: true when the ±15pp gate is crossed. Operators can subscribe to the calibration_drift event in CenAlerts and get pinged the moment a country starts diverging.

Honest caveats

• 30-day windows are narrow. Countries with fewer than 15 predictions in the window get status: insufficient_data and don't fire alerts.
• High-frequency-flip countries (Venezuela, Myanmar) produce noisy drift estimates — abs_drift_pp tends to overstate true drift when the country is bouncing between blocked and unblocked.
• Drift > 0 ≠ broken model. A real distribution shift in the country (election, mass protest, fresh shutdown campaign) will legitimately have empirical positive rate above mean predicted until the next weekly retrain catches up. This monitor surfaces the symptom; root-cause analysis is human.
• The calibrator is applied with the same WATCHED_FOR_CALIBRATION + CALIBRATOR_FLOOR gate as the live API, so drift here is the literal drift on the probability that would have been served at request time.

Why this matters

A forecast model that is honest globally but silently broken on Egypt is worse than a model that is openly broken everywhere — the former gets cited as authoritative, while the actual users in Egypt see wildly wrong numbers. Daily per-country auditing makes that failure mode visible and actionable: when a country crosses ±15pp, the alert fires, the sidecar updates, and the endpoint can be cited as proof that we're watching.