Voidly
All findingsPublished research

The censorship classifier generalizes across countries but degrades across time — a forward-temporal audit

The production censorship classifier v3.3 (GradientBoosting, 16 country-day features) reports stratified 5-fold F1 0.729 / AUC 0.899 and leave-one-country-out (LOCO) median F1 0.870. We reproduced the stratified number exactly (AUC 0.895 / F1 0.725 with a fresh GB on the same features), then…

Read the full published summary

The production censorship classifier v3.3 (GradientBoosting, 16 country-day features) reports stratified 5-fold F1 0.729 / AUC 0.899 and leave-one-country-out (LOCO) median F1 0.870. We reproduced the stratified number exactly (AUC 0.895 / F1 0.725 with a fresh GB on the same features), then re-split the 4,237 samples by TIME — train on the oldest 70% of distinct days, test on the newest 30% (strict past→future). Forward-temporal skill drops materially: AUC 0.669 (−0.226), F1 0.474, precision 0.34 / recall 0.80 at threshold 0.5, PR-AUC 0.52 at a 0.27 base rate (1.9x lift). So the classifier is NOT broken — forward in time it still beats chance ~2x and recovers 80% of incidents — but the random-split AUC overstates forward-deployment accuracy. The three splits answer different questions: stratified 5-fold (shuffled rows, near-duplicates in both folds) is the easiest; LOCO (hold out whole countries) is a genuinely hard cross-country test that v3.3 passes well and remains the honest headline for country generalization; forward-temporal (hold out the future) is the deployment question, and there v3.3 degrades because what an incident looks like drifts over time. Honest one-liner: v3.3 generalizes across COUNTRIES but degrades across TIME — which is exactly why it is retrained weekly (the cadence is load-bearing, not hygiene). Milder than the same-week shutdown-risk forward audit (whose within-country 7-day forecast fell BELOW chance at ~0.36) because this is same-day detection, not forecasting. No model change: v3.3 stays live and its LOCO F1 0.87 is real. What changed is disclosure — the live /v1/classifier/info evaluation now carries the forward-temporal block + a note that the random-split number is not forward-deployment accuracy. Reproduce with scripts/audit-classifier-v3.3-temporal.py.

The Voidly censorship classifier (v3.3, a GradientBoosting model over 16 country-day features) is the workhorse behind incident labeling and the live /v1/classifier/score endpoints. Its public model card reports three numbers: stratified 5-fold F1 0.729 / AUC 0.899 and leave-one-country-out (LOCO) median F1 0.870. This finding asks the question those splits don’t: when you train on the past and classify the future — which is what actually happens in deployment — how much skill survives?

The audit

We reproduced the published split first, as a control: a fresh GradientBoosting (n_estimators=200, max_depth=6, lr=0.1) over the exact v3.3 feature set scores stratified 5-fold AUC 0.895 / F1 0.725 — matching the model card, so the comparison is apples to apples. Then we re-split the same 4,237 country-day samples by time: train on the oldest 70% of distinct days, test on the newest 30% (a strict past→future holdout, no row from the test window seen in training).

  • Forward-temporal AUC: 0.669 (down from 0.895 — a 0.226 drop).
  • Forward-temporal F1: 0.474 (down from 0.725).
  • At the 0.5 threshold: recall 0.80, precision 0.34 — it still catches most forward incidents, but with many more false positives than the in-distribution number implies.
  • PR-AUC 0.52 at a 0.27 base rate = 1.9× lift over chance, out-of-time.

What this means — carefully

The classifier is not broken. Forward in time it still beats chance by ~2× and recovers 80% of incidents. But the random-split and LOCO numbers measure different kinds of generalization than deployment needs:

  • Stratified 5-fold (AUC 0.90) shuffles rows, so days from the same week land in both train and test. It answers “can the model recognize patterns it has near-duplicates of?” — the easiest question.
  • LOCO (F1 0.87) holds out whole countries. It answers “does the model generalize to a country it has never seen?” — a genuinely hard and meaningful test, and v3.3 passes it well. This remains the honest headline for cross-country generalization.
  • Forward-temporal (AUC 0.67) holds out the future. It answers “train today, classify tomorrow’s data” — the deployment question. Here v3.3 degrades, because what an “incident” looks like drifts over time (new blocking techniques, shifting country mix, evolving label-minting).

So the honest one-liner is: v3.3 generalizes well across countries but degrades across time. That is precisely why the model is retrained weekly — a stale classifier loses ~0.23 AUC. The retrain cadence isn’t incidental hygiene; it is load-bearing, and the published AUC should not be read as forward-deployment accuracy.

Contrast with the shutdown predictor

This is a milder result than the shutdown-risk forward audit published the same week. There, the within-country day-timing signal fell below chance forward (AUC ~0.36) — because that was a 7-day forecast, the hardest task. The classifier is same-day detection, so it holds up far better forward (AUC 0.67, real lift). Same audit method, two honestly different outcomes.

What we changed

No model change — v3.3 stays live, and its LOCO F1 0.87 is a real, defensible number for cross-country generalization. What changed is disclosure. The live /v1/classifier/info evaluation now carries the forward-temporal block and a note stating plainly that the random-split number overstates forward-deployment skill and that the weekly retrain is what keeps the model current. Reproduce it yourself with scripts/audit-classifier-v3.3-temporal.py.

A number is only honest if you say which split produced it. v3.3 is good at the job it does — labeling censorship across 130+ countries — and now its model card says exactly how far that skill reaches in time.

The source record.

Original source links and labels. Linked responses may have changed since publication.