We have four base learners for country-day censorship detection on the Voidly Atlas, each trained or tuned on a different lens:
- v3.3 GradientBoosting — the promoted supervised classifier. 16 features (anomaly rate, measurement count, spike, probe agreement, regime-weighted neighbor contagion). LOCO median F1 0.870. The strongest single signal we have.
- DBSCAN anomaly v1 — CenDTect-style rolling-45d per-country unsupervised. AUC 0.6506 vs labels. Surfaces shape- anomalous days the supervised model shrugs at.
- Bayesian corroboration v1 — naive-Bayes posterior P(censorship | which-sources-signaled). AUC 0.916. Independent evidence: cross-source agreement among OONI / IODA / CensoredPlanet / Voidly probes.
- Per-measurement classifier v1 — XGBoost on 84K
evidence rows. Aggregated to per-country-day mean. Caveat: AUC ~1.0
because the model reconstructs the labeling rule from
signal_value+sourcepatterns rather than discovering novel signal.
The build
- Training set: 4,237 country-day samples / 1,116 positive / 131 countries (same labels v3.3 uses).
- Feature vector: 5 base-learner scores per row —
p_classifier(out-of-fold from v3.3),dbscan_score,dbscan_is_anom,p_corroboration,p_measurement. - p_classifier uses 5-fold out-of-fold predictions from the v3.3 architecture — the textbook stacking move that prevents the base learner's in-sample optimism from leaking into the meta-learner.
- Meta-learner: head-to-head between logistic regression (class-weight balanced) and a shallow MLP (16 → 8 → 1, ReLU, early-stopping). Both standardized via per-fold scalers inside a Pipeline.
- Promote criteria: stratified-CV F1 ≥ v3.3 baseline + 0.02 AND LOCO median F1 ≥ 0.85. Honest deltas only.
Results
- v3.3 alone (out-of-fold, p_classifier-as-probability): F1 0.7424, AUC ≈ 0.91.
- LogisticRegression meta: F1 0.7534, AUC 0.9033 — winner by 0.07 F1 over MLP.
- MLP(16,8) meta: F1 0.7527, AUC 0.8977 — lower-recall, higher-precision (0.84 vs 0.77).
- LOCO median F1: 0.8974 (mean 0.8299, 97 of 131 countries evaluated; 34 skipped for degenerate splits).
- Stratified F1 delta vs baseline: +0.0110 (needs +0.0200 to clear gate).
Promote decision: NOT promoted
- Stratified F1 gate: +0.0110 < +0.0200 required → FAIL
- LOCO median gate: 0.8974 ≥ 0.85 required → PASS
- One out of two gates passed. We do not promote on a single-gate pass; the F1 lift is real but inside the noise band of the v3.3 baseline (std ≈ 0.017).
What the coefficients show
The LR fit on the full training set produces these standardized weights:
p_classifier= 1.948 (dominant)dbscan_is_anom= 0.324 (modest)dbscan_score= 0.136p_measurement= 0.122 (smaller than expected given AUC ~1.0; the OOF v3.3 has already absorbed most of the per-measurement signal)p_corroboration= 0.0007 (effectively zero)- Intercept: −0.359
The Bayes posterior adds almost nothing on top of v3.3 — the cross-source agreement signal it captures is already implicit in v3.3's anomaly rate × measurement count interaction. DBSCAN is the only meaningful diversifying signal, adding ~+0.3 log-odds when its binary flag fires.
Honest caveats
- Base learners share training labels. All four base learners ultimately train against the same incident labels. True meta-learning gains require base learners with structurally different label sources — that's a longer-term roadmap item.
- Per-measurement classifier inflates by reconstructing the labeling rule. Its mean-per-day score should not be read as independent evidence.
- DBSCAN scores only cover 92.6% of training rows (rolling window needs warm-up); missing rows fall back to 0.
- Small absolute lift. +1.1pp F1 is real but inside the per-fold std (≈ 0.017). One bad fold could flip the sign.
Why ship it anyway
Even though we did not promote, the endpoint is live as a transparency tool: every base learner's score for a given country-day, plus the meta-learner's combined view, plus the per-feature log-odds contribution. When v3.3 and DBSCAN disagree, the meta-learner's confidence calibrates the disagreement — that's a useful diagnostic for human review even when it's not a leaderboard win.
Live at
GET /v1/classifier/stacking/info — model meta + gate result + coefficients
GET /v1/classifier/stacking/{cc}?date=YYYY-MM-DD — meta-learner score + per-base contributions
Example: GET /v1/classifier/stacking/IR
returns meta_learner_probability with all four base scores broken
out, plus the log-odds contribution from each.
Negative result number 4
This is our fourth honest negative — TabPFN v2 (frozen), v3.4 regime-finetune (rolled back), per-ASN forecast (not viable yet), and now v3.7 stacking ensemble. We surface every one of them rather than burying them, because tracking which ideas don't work is how the platform stays calibrated.