Voidly
All findingsPublished research

Alert lead-time retrospective: did Sentinel actually warn early? (the accountability number)

Voidly Sentinel fires a forecast_threshold alert when a country's 7-day censorship-risk forecast crosses the alert threshold, and the pitch is "early warning." This finding is the honest audit of that pitch. For every forecast-threshold alert that fired in the last 90 days — 154 alerts across 30…

Read the full published summary

Voidly Sentinel fires a forecast_threshold alert when a country's 7-day censorship-risk forecast crosses the alert threshold, and the pitch is "early warning." This finding is the honest audit of that pitch. For every forecast-threshold alert that fired in the last 90 days — 154 alerts across 30 countries — it measures whether a confirmed censorship/mixed incident actually followed within 14 days (a true positive), or never came (a false alarm), and if it did, how many days of lead time the alert gave (incident first_seen minus alert issued_at). The headline numbers, deliberately unsmoothed: 30 true positives (true-positive rate 19.5%), 122 false alarms (false-alarm rate 79.2%), 2 lagging alerts where the forecast reacted to a shutdown already underway. For the 30 true positives the lead-time distribution is median 4.2 days, mean 5.8 days, IQR 2.2-8.9 days, full range 0.9-13.9 days. The 79.2% false-alarm rate is high and the endpoint says so prominently in a headline_warning field rather than burying it — a single Sentinel alert is a watch signal, not a prediction; the genuine early-warning value is in the aggregate lead-time distribution. The per-country split is the real story: where Sentinel works it works (Egypt 3/3 true positives, 100%, median lead 10.9 days; Uzbekistan 7/7, 100%, 1.9 days; Pakistan 5/6, 83%, 2.9 days) and where it does not it does not (Iran 0/7, 100% false alarms — the worst country; Turkmenistan, Brazil, Cuba, Myanmar each 0-for-7). Honest caveats baked into every response: (1) "lead time" is alert-issued vs incident-DETECTION, not alert vs the real-world shutdown start — incident detection itself lags (OONI/IODA ingest + the 30-min incident builder) so a positive lead time is a LOWER bound on the true early-warning margin, it does not over-state; (2) the matched incident is a TEMPORAL match (next confirmed incident within the horizon), not a causal one — the alert did not necessarily predict that specific incident; (3) the 14-day horizon means a real incident the alert correctly anticipated but that arrived on day 16 is scored as a false alarm here, so 79.2% is an UPPER bound on true mis-fires; (4) only confirmed censorship/mixed incidents count — IODA disruption rows (real outages never confirmed as censorship) are excluded, which raises the measured false-alarm rate; (5) lagging alerts are reported separately and excluded from the median-lead-time headline; (6) countries with <2 alerts are excluded from best/worst ranking. Live at GET /v1/sentinel/alert-lead-time + /v1/sentinel/alert-lead-time/{cc} + /v1/sentinel/alert-lead-time/info. Rebuilt daily 04:50 UTC. Paired with /v1/sentinel/accuracy — accuracy grades every forecast, this grades every alert that actually fired. Implementation: scripts/build-alert-lead-time-retrospective.py + patch-alert-lead-time-endpoint.py.

Voidly Sentinel sends a forecast_threshold alert when a country's 7-day censorship-risk forecast crosses the alert threshold. The pitch is "early warning." This page is the honest audit of that pitch: for every forecast-threshold alert that fired in the last 90 days, did a confirmed censorship incident actually follow — and if so, by how many days did the alert beat it?

We built this because an early-warning claim is only worth anything if it is measured against real history. It is easy to publish a forecast; it is harder to grade the forecast after the fact and publish the grade whether it flatters us or not. This is the accountability number. It is deliberately not smoothed.

The headline numbers

  • 154 forecast-threshold alerts analyzed across 30 countries (window: 2026-02-20 to 2026-05-21).
  • 30 true positives — an alert followed within 14 days by a confirmed censorship/mixed incident in the same country. That is a true-positive rate of 19.5%.
  • 122 false alarms — no confirmed incident followed within the horizon. That is a false-alarm rate of 79.2%.
  • 2 lagging alerts — the forecast reacted to a shutdown that had already started. These are not early warnings and are excluded from the lead-time headline.
  • For the 30 true positives, the lead-time distribution is: median 4.2 days, mean 5.8 days, interquartile range 2.2 to 8.9 days, full range 0.9 to 13.9 days.

Say the uncomfortable part plainly

Roughly four out of five forecast-threshold alerts in the last 90 days were not followed by a confirmed censorship incident within two weeks. That is a high false-alarm rate, and the endpoint surfaces it in a headline_warning field rather than burying it. A single Sentinel alert should be read as a watch signal — a prompt to look harder at a country — not as a prediction that a shutdown is coming. The genuine early-warning value lives in the aggregate: when an alert does precede an incident, it does so by a useful margin (a median of four-plus days, sometimes nearly two weeks).

Two structural reasons the false-alarm rate is partly an over-count, stated honestly so the number is not gamed:

  • The horizon is a fixed 14 days. A real incident the alert correctly anticipated, but which arrived on day 16, is scored here as a false alarm. So 79.2% is an upper bound on true mis-fires, not a floor.
  • An alert is matched only against confirmed censorship/mixed incidents. IODA disruption rows — real network outages that were never confirmed as state censorship — are excluded. Some of those disruptions in alerting countries may well have been undeclared shutdowns; counting them would lower the false-alarm rate, but we do not, because they are not confirmed.

The spread between countries is the real story

The 19.5% aggregate hides a wide split. Where Sentinel works, it works:

  • Egypt — 3 alerts, 3 true positives (100%), median lead time 10.9 days. The best country in the retrospective.
  • Uzbekistan — 7 alerts, 7 true positives (100%), median lead time 1.9 days. High hit rate, short lead.
  • Pakistan — 6 alerts, 5 true positives (83%), median lead time 2.9 days.

And where it does not, it does not:

  • Iran — 7 alerts, 0 true positives, a 100% false-alarm rate over the window. The worst country in the retrospective. Iran is chronically censored, so the forecast keeps crossing the threshold, but no new confirmed incident landed in the matching windows — partly a real forecast weakness, partly an incident-coverage gap.
  • Turkmenistan, Brazil, Cuba, and Myanmar also each ran 0-for-7 in the window.

That split is itself useful: it tells a newsroom which country's Sentinel alerts have earned trust (Egypt, Uzbekistan, Pakistan) and which to treat as background noise until the model improves.

What "lead time" does and does not mean

Lead time here is alert-issued vs incident-detection, not alert vs the real-world start of the shutdown. Voidly's incident detection itself lags: OONI and IODA measurements take time to ingest, and the incident builder runs on a 30-minute cycle. So a positive lead time is a lower bound on the true early-warning margin against the actual event — the alert beat our own detection by the reported number of days, and almost certainly beat the real shutdown start by more. The number does not over-state.

One more honest caveat: the matched incident is the next confirmed incident in the same country within the horizon. That is a temporal match, not a causal one. We are not claiming the alert predicted that specific incident — only that an alert and an incident occurred in that order, in that country, within two weeks.

How to read it

Use GET /v1/sentinel/alert-lead-time for the full retrospective — the lead-time distribution, the true-positive and false-alarm rates, the per-country breakdown, and the per-alert detail. Use GET /v1/sentinel/alert-lead-time/{cc} to pull one country's record before deciding whether to trust its alerts. The retrospective rebuilds daily, so the numbers move as new alerts fire and new incidents are confirmed. It is paired with /v1/sentinel/accuracy (the rolling forecast confusion matrix); accuracy grades every forecast, this grades every alert that actually fired.

The source record.

Original source links and labels. Linked responses may have changed since publication.