The Voidly censorship classifier (v3.3, a GradientBoosting model over 16
country-day features) is the workhorse behind incident labeling and the
live /v1/classifier/score endpoints. Its public model card
reports three numbers: stratified 5-fold F1 0.729 / AUC 0.899
and leave-one-country-out (LOCO) median F1 0.870. This
finding asks the question those splits don’t: when you train on the
past and classify the future — which is what
actually happens in deployment — how much skill survives?
The audit
We reproduced the published split first, as a control: a fresh GradientBoosting (n_estimators=200, max_depth=6, lr=0.1) over the exact v3.3 feature set scores stratified 5-fold AUC 0.895 / F1 0.725 — matching the model card, so the comparison is apples to apples. Then we re-split the same 4,237 country-day samples by time: train on the oldest 70% of distinct days, test on the newest 30% (a strict past→future holdout, no row from the test window seen in training).
- Forward-temporal AUC: 0.669 (down from 0.895 — a 0.226 drop).
- Forward-temporal F1: 0.474 (down from 0.725).
- At the 0.5 threshold: recall 0.80, precision 0.34 — it still catches most forward incidents, but with many more false positives than the in-distribution number implies.
- PR-AUC 0.52 at a 0.27 base rate = 1.9× lift over chance, out-of-time.
What this means — carefully
The classifier is not broken. Forward in time it still beats chance by ~2× and recovers 80% of incidents. But the random-split and LOCO numbers measure different kinds of generalization than deployment needs:
- Stratified 5-fold (AUC 0.90) shuffles rows, so days from the same week land in both train and test. It answers “can the model recognize patterns it has near-duplicates of?” — the easiest question.
- LOCO (F1 0.87) holds out whole countries. It answers “does the model generalize to a country it has never seen?” — a genuinely hard and meaningful test, and v3.3 passes it well. This remains the honest headline for cross-country generalization.
- Forward-temporal (AUC 0.67) holds out the future. It answers “train today, classify tomorrow’s data” — the deployment question. Here v3.3 degrades, because what an “incident” looks like drifts over time (new blocking techniques, shifting country mix, evolving label-minting).
So the honest one-liner is: v3.3 generalizes well across countries but degrades across time. That is precisely why the model is retrained weekly — a stale classifier loses ~0.23 AUC. The retrain cadence isn’t incidental hygiene; it is load-bearing, and the published AUC should not be read as forward-deployment accuracy.
Contrast with the shutdown predictor
This is a milder result than the shutdown-risk forward audit published the same week. There, the within-country day-timing signal fell below chance forward (AUC ~0.36) — because that was a 7-day forecast, the hardest task. The classifier is same-day detection, so it holds up far better forward (AUC 0.67, real lift). Same audit method, two honestly different outcomes.
What we changed
No model change — v3.3 stays live, and its LOCO F1 0.87 is a real,
defensible number for cross-country generalization. What changed is
disclosure. The live /v1/classifier/info evaluation now carries
the forward-temporal block and a note stating plainly that the random-split
number overstates forward-deployment skill and that the weekly retrain is
what keeps the model current. Reproduce it yourself with
scripts/audit-classifier-v3.3-temporal.py.
A number is only honest if you say which split produced it. v3.3 is good at the job it does — labeling censorship across 130+ countries — and now its model card says exactly how far that skill reaches in time.