It is easy to claim "F1 0.87" for a censorship-forecast model. It is much harder to answer the next question honestly: is the model adding value, or is it just memorizing the prior? An F1 of 0.87 on a target with an 80% positive rate is barely beating "always predict positive". On a target with a 5% positive rate, that same F1 is genuine signal. Without a baseline, the headline number is uninterpretable.
This finding ships the synthetic-baseline benchmark suite: every Voidly
Atlas ML model evaluated against six trivial baselines on the same data,
with the lift surfaced inline as a single number per row. Live at
GET /v1/atlas/baseline-benchmark; weekly cron Thursday
04:45 UTC.
The six baselines
- always_zero — predict 0 for everything. F1 = 0 by definition.
- always_one — predict 1 for everything. Recall = 1, precision = base rate, F1 = 2·p/(1+p).
- base_rate_constant — predict every row's probability as the training-set positive rate. Well-calibrated by definition, but provides no resolution.
- predict_yesterday — lag-1 of the label within each country group. For autocorrelated targets like "will country X be in shutdown next week", this is a brutal baseline: if the country was in shutdown yesterday, it almost certainly will be tomorrow.
- country_base_rate — predict each country's historical positive rate. No temporal signal at all.
- random_with_base_rate — Bernoulli sample with p = base rate. The sanity floor.
Lift is reported in percentage points of F1 (or AUC, for AUC-only
models) against the predict_yesterday baseline — the
toughest one of the six.
The honest headline numbers
- 23 models benchmarked. Forecast (7-day), forecast multi-horizon (1/7/30d), classifier v3.3, trajectory v1 (1/7/30d), contagion-chain 7d, duration RSF, 5 sampled per-platform, 5 sampled per-domain, 4 per-method.
- 7 models flagged
barely_beats_baseline=true— lift of less than 5pp vspredict_yesterday. That list includes:forecast_7day(F1 lift +2.29pp),forecast_1d/7d/30d(multi-horizon, all F1-negative),classifier_v3.3(F1 lift −21.04pp), andtrajectory_d7/d30(AUC lift −21 to −23pp). - Best by AUC lift: per_domain_pornhub.com at +50.74pp. Other per-domain and per-platform models cluster near 1.000 AUC. This is a code smell, not a triumph: AUC at 0.999+ usually means the model is reconstructing the labeling rule from a leaked feature. Every such row gets an honest_caveats entry inline.
- Worst by AUC lift: trajectory_d30 at −22.78pp.
Trajectory models lose to
predict_yesterdayon long horizons because, once a country is in shutdown today, it usually stays that way for the next 30 days. Persistence beats trajectory at that horizon.
Why forecast_7day only beats predict_yesterday by 2.29pp F1
This is the point of the whole exercise. forecast_7day has a
headline F1 of 0.937 on the temporal holdout. Looks great. But
predict_yesterday — literally just "the country's
label yesterday" — scores F1 0.914 on the same holdout. The model
adds 2.29 percentage points of F1 over a one-line baseline. On AUC the
lift is 4.51pp, which is more meaningful (AUC is threshold-free).
This historical comparison found only a small lift over persistence on the May holdout. It did not establish reliable prediction of new shutdown events. SHAP attributions can explain model outputs, but current calibration and 90% interval coverage remain unvalidated; public forecast intervals are withheld. The headline F1 alone overstates lift.
Why classifier_v3.3 shows a negative F1 lift
The classifier reports an F1 of 0.742 from 5-fold stratified CV. The
predict_yesterday baseline on the rolling holdout scores
0.953. That's an F1 lift of −21pp.
Honest reading: the classifier is solving a different task. It scores
country-days that already crossed an evidence threshold — mostly
positive examples — whereas predict_yesterday is
trivially perfect on day-to-day continuity within a country. Comparing
the two as F1 is approximate, not apples-to-apples. The per-row
source field documents this; readers who want a fair
comparison should look at the classifier's stratified CV against a CV
baseline computed on the same folds. The benchmark surfaces the
ugly-but-honest number rather than hiding it.
What this is NOT
- Not a full LOCO re-evaluation of every model. Some rows pull
training-time sidecar metrics and compute the baselines on a separate
last-60d temporal holdout. Lift is approximate in those cases —
check the per-row
sourcefield. - Not a verdict on the models.
barely_beats_baseline=trueis a yellow flag, not red. The point of this endpoint is to make every shipped model's value claim auditable in one query. - Not a static benchmark. The cron regenerates the sidecar every Thursday at 04:45 UTC; the numbers move as we ship new model versions and as the holdout window slides forward.
How to read the endpoint
Every row in the response has the same shape:
model_id,family,horizonmodel_metrics: F1, AUC, Brier, n, pos_rate on this evaluationbaseline_metrics: the six baselines' F1/AUC/Brier on the same datalift_vs_predict_yesterday: signed pp delta for F1, AUC, Brier (Brier is sign-flipped so positive = better)barely_beats_baseline: boolean flagbarely_beats_metric: which metric the flag is on (F1 or AUC)source: live re-eval vs. training-time sidecarhonest_caveats: model-specific gotchas, including the "AUC at 0.999+ is a code smell" flag
Query filters: family=<name>,
barely=true|false, min_lift_pp=<float>.
The /info endpoint exposes the methodology and the per-row
caveat policy without hitting the full payload.
Methodology footnote: why predict_yesterday is the right toughest baseline
Internet shutdowns are extremely autocorrelated: if a country was in
shutdown yesterday, the prior on shutdown today is over 95%. On those
targets, “persistence” is the hardest one-line baseline to
beat. We pick it deliberately. If a model can't beat
predict_yesterday at some metric by at least 5pp,
we want it on a list the user can see — not buried.
This pattern is borrowed from the time-series forecasting literature (Hyndman & Athanasopoulos, Forecasting: Principles and Practice) and from the GluonTS benchmarking conventions. It is also consistent with how Cloudflare and NetBlocks report their own shutdown detection — usually as deltas from a 7-day baseline rather than as absolute counts.