Voidly Atlas ships a 7-day internet-shutdown predictor at
/v1/shutdown-risk/{cc}, trained and validated against
Access Now’s journalist-verified KeepItOn shutdown record. Its public
model card has long reported two headline AUCs: a cross-country
full-panel AUC ~0.88–0.90 (“which country is at
risk”) and a within-country median AUC ~0.73
(“which day inside that country”). This finding is the honest
audit of those two numbers — one survives scrutiny and is genuinely
strong; the other does not mean what it appears to mean.
Two upgrades, then one hard question
We first improved the model honestly. The live ensemble (v5) combined two
base signals — a structural country-risk model (v4, KeepItOn-derived)
and an OONI measurement-trajectory model (honest_forecast_v1) — by
multiplying them: p = sr × hf. A product requires
both signals to be high, so whenever the OONI trajectory was near
zero or anti-correlated for a country, the product annihilated an otherwise
good structural signal. Oman’s within-country AUC collapsed from 0.498
to 0.040; Sudan, Ethiopia, Indonesia and India lost ground
the same way.
v9 replaces the product with a logit-space blend —
p = isotonic(sigmoid(0.6·logit(sr) + 0.4·logit(hf)))
— which can never let one factor zero out the other. The 0.60 weight
was not hand-picked on the evaluation set: it was selected by
leave-one-country-out, and all 23 folds independently chose 0.60.
The held-out within-country median rose from 0.7291 to 0.7386 (+0.0095, with
18 of 23 countries improving) and the cross-country full-panel AUC rose from
0.8848 to 0.8979. A logistic stacker fit on the two base signals confirmed
the imbalance the product had baked in — it put 0.99 weight on
structural risk versus 0.056 on the OONI trajectory. v9 is now live.
But a higher in-sample number is not the question that matters to someone deciding whether to trust an alert. The question is: does the model predict the future? So we ran the test the rest of our honesty work demands — a strict past→future temporal holdout. Fit the calibrator only on data up to the 75th-percentile date; score the future; measure skill on the held-out tail.
The within-country “which day” signal does not survive
Forward in time, the within-country day-ranking AUC is ~0.36 — below chance, for both v5 (0.3835) and v9 (0.3555). The published within-country median of ~0.73 is cross-sectional: it is measured with the isotonic calibrator fit on the entire panel, all dates mixed together. It tells you the model can rank the riskier days within a country once it has already seen the whole period. It is not a measure of forward skill, and we should never have let it read like one.
Why does it fall apart forward? Non-stationarity. Internet shutdowns cluster in time — a country that was quiet during the training window and then erupts in the test window inverts the very structural signal the model leans on. Country-level aggregates also miss the roughly half of KeepItOn events that are sub-national (a single region, a single city), which no country-day score can resolve. Predicting the day a shutdown begins inside a given country, from country-level data, going forward, is a problem this model does not solve. We now say so on the model card and the public page.
The cross-country “which country” signal is real — and forward-validated
The good news is the part that matters most. On the same strict past→future holdout, the cross-country ranking holds:
- Forward full-panel AUC 0.8653 (versus 0.8912 in-sample — it barely degrades).
- Forward country-ranking AUC 0.9015 across 60 countries — ranking which countries see any shutdown in the future.
- Forward PR-AUC lift 7.84× over the base rate, out-of-time.
In plain terms: the model reliably tells you which countries are heading into danger, and that skill is validated forward in time, not just in-sample. That is the actual product, and it is strong.
What changed, and what we now claim
v9 stays live — it is the better model at the thing that works
(cross-country ranking, full-panel AUC 0.8979 versus v5’s 0.8848). This
is not a rollback; it is a correction to how the model is described. The live
/v1/shutdown-risk/info now reports all four numbers side by side
— cross-sectional within-country (0.74), forward within-country (0.36),
forward cross-country (0.87) and forward country-ranking (0.90) — and
the lead caveat states plainly that the within-country figure is
cross-sectional and that the model is a country risk ranking, not a
within-country calendar oracle. The public /shutdown-risk page
carries the same framing.
A research organization’s credibility is its product. A model that is strong at what it claims and explicit about what it cannot do is worth more than a bigger number with an asterisk no one reads. Use Voidly shutdown-risk to know which countries to watch — that signal is real, and it holds going forward. Do not use it to bet on which day.