Voidly
All findingsPublished research

Shutdown-risk predicts WHICH country, not WHICH day — a forward-validated honest scoping (v9)

Voidly's 7-day shutdown predictor (/v1/shutdown-risk) reported two AUCs: cross-country full-panel ~0.88-0.90 and within-country median ~0.73. This week we shipped v9 (a logit-blend combiner that replaced v5's multiplicative product — the product collapsed when the OONI trajectory was near-zero,…

Read the full published summary

Voidly's 7-day shutdown predictor (/v1/shutdown-risk) reported two AUCs: cross-country full-panel ~0.88-0.90 and within-country median ~0.73. This week we shipped v9 (a logit-blend combiner that replaced v5's multiplicative product — the product collapsed when the OONI trajectory was near-zero, e.g. Oman within-AUC 0.498→0.040; alpha=0.60 chosen by leave-one-country-out, all 23 folds agreed; within-country median 0.7291→0.7386, full-panel 0.8848→0.8979). Then we ran the test that actually matters: a strict past→future temporal holdout. The within-country day-ranking AUC is ~0.36 forward — BELOW chance — for both v5 and v9. The published ~0.73 is cross-sectional (isotonic fit on the whole panel, all dates mixed); it measures day-ranking once the model has seen the period, not forward skill, and shutdowns cluster in time so a country quiet in train then active in test inverts the structural signal. BUT the cross-country ranking — the real product — holds forward: full-panel AUC 0.8653 out-of-time (vs 0.8912 in-sample), country-ranking AUC 0.9015 across 60 countries, 7.84x PR-lift over base rate. Verdict: v9 stays live (better at the cross-country ranking that works); the within-country claim was overstated and is now corrected. The live /v1/shutdown-risk/info reports all four numbers (cross-sectional within 0.74, forward within 0.36, forward cross-country 0.87, forward country-ranking 0.90) with a lead caveat; the public page now frames it as a forward-validated country risk ranking, not a within-country calendar oracle. Use it to know WHICH countries to watch — that signal is real and holds going forward — not WHICH day.

Voidly Atlas ships a 7-day internet-shutdown predictor at /v1/shutdown-risk/{cc}, trained and validated against Access Now’s journalist-verified KeepItOn shutdown record. Its public model card has long reported two headline AUCs: a cross-country full-panel AUC ~0.88–0.90 (“which country is at risk”) and a within-country median AUC ~0.73 (“which day inside that country”). This finding is the honest audit of those two numbers — one survives scrutiny and is genuinely strong; the other does not mean what it appears to mean.

Two upgrades, then one hard question

We first improved the model honestly. The live ensemble (v5) combined two base signals — a structural country-risk model (v4, KeepItOn-derived) and an OONI measurement-trajectory model (honest_forecast_v1) — by multiplying them: p = sr × hf. A product requires both signals to be high, so whenever the OONI trajectory was near zero or anti-correlated for a country, the product annihilated an otherwise good structural signal. Oman’s within-country AUC collapsed from 0.498 to 0.040; Sudan, Ethiopia, Indonesia and India lost ground the same way.

v9 replaces the product with a logit-space blend — p = isotonic(sigmoid(0.6·logit(sr) + 0.4·logit(hf))) — which can never let one factor zero out the other. The 0.60 weight was not hand-picked on the evaluation set: it was selected by leave-one-country-out, and all 23 folds independently chose 0.60. The held-out within-country median rose from 0.7291 to 0.7386 (+0.0095, with 18 of 23 countries improving) and the cross-country full-panel AUC rose from 0.8848 to 0.8979. A logistic stacker fit on the two base signals confirmed the imbalance the product had baked in — it put 0.99 weight on structural risk versus 0.056 on the OONI trajectory. v9 is now live.

But a higher in-sample number is not the question that matters to someone deciding whether to trust an alert. The question is: does the model predict the future? So we ran the test the rest of our honesty work demands — a strict past→future temporal holdout. Fit the calibrator only on data up to the 75th-percentile date; score the future; measure skill on the held-out tail.

The within-country “which day” signal does not survive

Forward in time, the within-country day-ranking AUC is ~0.36 — below chance, for both v5 (0.3835) and v9 (0.3555). The published within-country median of ~0.73 is cross-sectional: it is measured with the isotonic calibrator fit on the entire panel, all dates mixed together. It tells you the model can rank the riskier days within a country once it has already seen the whole period. It is not a measure of forward skill, and we should never have let it read like one.

Why does it fall apart forward? Non-stationarity. Internet shutdowns cluster in time — a country that was quiet during the training window and then erupts in the test window inverts the very structural signal the model leans on. Country-level aggregates also miss the roughly half of KeepItOn events that are sub-national (a single region, a single city), which no country-day score can resolve. Predicting the day a shutdown begins inside a given country, from country-level data, going forward, is a problem this model does not solve. We now say so on the model card and the public page.

The cross-country “which country” signal is real — and forward-validated

The good news is the part that matters most. On the same strict past→future holdout, the cross-country ranking holds:

  • Forward full-panel AUC 0.8653 (versus 0.8912 in-sample — it barely degrades).
  • Forward country-ranking AUC 0.9015 across 60 countries — ranking which countries see any shutdown in the future.
  • Forward PR-AUC lift 7.84× over the base rate, out-of-time.

In plain terms: the model reliably tells you which countries are heading into danger, and that skill is validated forward in time, not just in-sample. That is the actual product, and it is strong.

What changed, and what we now claim

v9 stays live — it is the better model at the thing that works (cross-country ranking, full-panel AUC 0.8979 versus v5’s 0.8848). This is not a rollback; it is a correction to how the model is described. The live /v1/shutdown-risk/info now reports all four numbers side by side — cross-sectional within-country (0.74), forward within-country (0.36), forward cross-country (0.87) and forward country-ranking (0.90) — and the lead caveat states plainly that the within-country figure is cross-sectional and that the model is a country risk ranking, not a within-country calendar oracle. The public /shutdown-risk page carries the same framing.

A research organization’s credibility is its product. A model that is strong at what it claims and explicit about what it cannot do is worth more than a bigger number with an asterisk no one reads. Use Voidly shutdown-risk to know which countries to watch — that signal is real, and it holds going forward. Do not use it to bet on which day.

The source record.

Original source links and labels. Linked responses may have changed since publication.