Voidly
All findingsPublished research

Synthetic baseline benchmark: every Atlas ML model vs predict_yesterday, base-rate, and four other trivial baselines

It is easy to claim "F1 0.87" for a censorship-forecast model. It is much harder to answer "is the model adding value over predict_yesterday?" honestly. This finding ships the synthetic-baseline benchmark suite: every Voidly Atlas ML model evaluated against six trivial baselines (always_zero,…

Read the full published summary

It is easy to claim "F1 0.87" for a censorship-forecast model. It is much harder to answer "is the model adding value over predict_yesterday?" honestly. This finding ships the synthetic-baseline benchmark suite: every Voidly Atlas ML model evaluated against six trivial baselines (always_zero, always_one, base_rate_constant, predict_yesterday, country_base_rate, random_with_base_rate), with the lift surfaced inline as a single number per row. Live at GET /v1/atlas/baseline-benchmark; weekly cron Thursday 04:45 UTC. First run benchmarked 23 models. 7 flagged barely_beats_baseline=true (F1/AUC lift < 5pp vs predict_yesterday) — including forecast_7day (F1 lift +2.29pp), forecast_1d/7d/30d (multi-horizon, all F1-negative vs persistence), classifier_v3.3 (F1 lift -21pp on rolling holdout — different task framing, documented honestly), and trajectory_d7/d30 (AUC lift -21 to -23pp because persistence dominates on 30-day horizons). Best by AUC lift is per_domain_pornhub.com at +50.74pp, but every AUC=1.000 row gets an explicit "code smell" honest_caveat — the model is likely reconstructing the labeling rule from a leaked feature, not discovering signal. Worst is trajectory_d30 at -22.78pp AUC. Brier sign-flipped so positive=better. Filters: family, barely, min_lift_pp. Six baselines documented in /info. The point is accountability: a model that barely beats predict_yesterday is on a list the user can see, not buried under a nice-looking F1.

It is easy to claim "F1 0.87" for a censorship-forecast model. It is much harder to answer the next question honestly: is the model adding value, or is it just memorizing the prior? An F1 of 0.87 on a target with an 80% positive rate is barely beating "always predict positive". On a target with a 5% positive rate, that same F1 is genuine signal. Without a baseline, the headline number is uninterpretable.

This finding ships the synthetic-baseline benchmark suite: every Voidly Atlas ML model evaluated against six trivial baselines on the same data, with the lift surfaced inline as a single number per row. Live at GET /v1/atlas/baseline-benchmark; weekly cron Thursday 04:45 UTC.

The six baselines

  • always_zero — predict 0 for everything. F1 = 0 by definition.
  • always_one — predict 1 for everything. Recall = 1, precision = base rate, F1 = 2·p/(1+p).
  • base_rate_constant — predict every row's probability as the training-set positive rate. Well-calibrated by definition, but provides no resolution.
  • predict_yesterday — lag-1 of the label within each country group. For autocorrelated targets like "will country X be in shutdown next week", this is a brutal baseline: if the country was in shutdown yesterday, it almost certainly will be tomorrow.
  • country_base_rate — predict each country's historical positive rate. No temporal signal at all.
  • random_with_base_rate — Bernoulli sample with p = base rate. The sanity floor.

Lift is reported in percentage points of F1 (or AUC, for AUC-only models) against the predict_yesterday baseline — the toughest one of the six.

The honest headline numbers

  • 23 models benchmarked. Forecast (7-day), forecast multi-horizon (1/7/30d), classifier v3.3, trajectory v1 (1/7/30d), contagion-chain 7d, duration RSF, 5 sampled per-platform, 5 sampled per-domain, 4 per-method.
  • 7 models flagged barely_beats_baseline=true — lift of less than 5pp vs predict_yesterday. That list includes: forecast_7day (F1 lift +2.29pp), forecast_1d/7d/30d (multi-horizon, all F1-negative), classifier_v3.3 (F1 lift −21.04pp), and trajectory_d7 / d30 (AUC lift −21 to −23pp).
  • Best by AUC lift: per_domain_pornhub.com at +50.74pp. Other per-domain and per-platform models cluster near 1.000 AUC. This is a code smell, not a triumph: AUC at 0.999+ usually means the model is reconstructing the labeling rule from a leaked feature. Every such row gets an honest_caveats entry inline.
  • Worst by AUC lift: trajectory_d30 at −22.78pp. Trajectory models lose to predict_yesterday on long horizons because, once a country is in shutdown today, it usually stays that way for the next 30 days. Persistence beats trajectory at that horizon.

Why forecast_7day only beats predict_yesterday by 2.29pp F1

This is the point of the whole exercise. forecast_7day has a headline F1 of 0.937 on the temporal holdout. Looks great. But predict_yesterday — literally just "the country's label yesterday" — scores F1 0.914 on the same holdout. The model adds 2.29 percentage points of F1 over a one-line baseline. On AUC the lift is 4.51pp, which is more meaningful (AUC is threshold-free).

This historical comparison found only a small lift over persistence on the May holdout. It did not establish reliable prediction of new shutdown events. SHAP attributions can explain model outputs, but current calibration and 90% interval coverage remain unvalidated; public forecast intervals are withheld. The headline F1 alone overstates lift.

Why classifier_v3.3 shows a negative F1 lift

The classifier reports an F1 of 0.742 from 5-fold stratified CV. The predict_yesterday baseline on the rolling holdout scores 0.953. That's an F1 lift of −21pp.

Honest reading: the classifier is solving a different task. It scores country-days that already crossed an evidence threshold — mostly positive examples — whereas predict_yesterday is trivially perfect on day-to-day continuity within a country. Comparing the two as F1 is approximate, not apples-to-apples. The per-row source field documents this; readers who want a fair comparison should look at the classifier's stratified CV against a CV baseline computed on the same folds. The benchmark surfaces the ugly-but-honest number rather than hiding it.

What this is NOT

  • Not a full LOCO re-evaluation of every model. Some rows pull training-time sidecar metrics and compute the baselines on a separate last-60d temporal holdout. Lift is approximate in those cases — check the per-row source field.
  • Not a verdict on the models. barely_beats_baseline=true is a yellow flag, not red. The point of this endpoint is to make every shipped model's value claim auditable in one query.
  • Not a static benchmark. The cron regenerates the sidecar every Thursday at 04:45 UTC; the numbers move as we ship new model versions and as the holdout window slides forward.

How to read the endpoint

Every row in the response has the same shape:

  • model_id, family, horizon
  • model_metrics: F1, AUC, Brier, n, pos_rate on this evaluation
  • baseline_metrics: the six baselines' F1/AUC/Brier on the same data
  • lift_vs_predict_yesterday: signed pp delta for F1, AUC, Brier (Brier is sign-flipped so positive = better)
  • barely_beats_baseline: boolean flag
  • barely_beats_metric: which metric the flag is on (F1 or AUC)
  • source: live re-eval vs. training-time sidecar
  • honest_caveats: model-specific gotchas, including the "AUC at 0.999+ is a code smell" flag

Query filters: family=<name>, barely=true|false, min_lift_pp=<float>. The /info endpoint exposes the methodology and the per-row caveat policy without hitting the full payload.

Methodology footnote: why predict_yesterday is the right toughest baseline

Internet shutdowns are extremely autocorrelated: if a country was in shutdown yesterday, the prior on shutdown today is over 95%. On those targets, “persistence” is the hardest one-line baseline to beat. We pick it deliberately. If a model can't beat predict_yesterday at some metric by at least 5pp, we want it on a list the user can see — not buried.

This pattern is borrowed from the time-series forecasting literature (Hyndman & Athanasopoulos, Forecasting: Principles and Practice) and from the GluonTS benchmarking conventions. It is also consistent with how Cloudflare and NetBlocks report their own shutdown detection — usually as deltas from a 7-day baseline rather than as absolute counts.

The source record.

Original source links and labels. Linked responses may have changed since publication.