Voidly
All findingsPublished research

v3.7 stacking ensemble over 4 base learners — failed F1 gate (+1.1pp vs needed +2.0pp), shipped as transparency endpoint

Stacked v3.3 GradientBoosting (OOF), DBSCAN unsupervised anomaly v1, Bayesian corroboration v1, and per-measurement classifier v1 into a meta-learner. Logistic regression won head-to-head vs MLP (16,8): stratified 5-fold F1 0.7534 vs v3.3 OOF baseline 0.7424 (+1.1pp), AUC 0.9033, LOCO median F1…

Read the full published summary

Stacked v3.3 GradientBoosting (OOF), DBSCAN unsupervised anomaly v1, Bayesian corroboration v1, and per-measurement classifier v1 into a meta-learner. Logistic regression won head-to-head vs MLP (16,8): stratified 5-fold F1 0.7534 vs v3.3 OOF baseline 0.7424 (+1.1pp), AUC 0.9033, LOCO median F1 0.8974 across 97 countries. Promote criteria required +2.0pp stratified F1 AND >=0.85 LOCO median — only LOCO gate passed. Endpoint shipped live at /v1/classifier/stacking/{cc} with passed_promote_gates=false for transparency. Coefficient analysis: v3.3 dominates (1.95), DBSCAN flag adds 0.32, per-measurement adds 0.12, Bayes posterior adds ~0 (already implicit in v3.3). Fourth honest negative result.

We have four base learners for country-day censorship detection on the Voidly Atlas, each trained or tuned on a different lens:

  • v3.3 GradientBoosting — the promoted supervised classifier. 16 features (anomaly rate, measurement count, spike, probe agreement, regime-weighted neighbor contagion). LOCO median F1 0.870. The strongest single signal we have.
  • DBSCAN anomaly v1 — CenDTect-style rolling-45d per-country unsupervised. AUC 0.6506 vs labels. Surfaces shape- anomalous days the supervised model shrugs at.
  • Bayesian corroboration v1 — naive-Bayes posterior P(censorship | which-sources-signaled). AUC 0.916. Independent evidence: cross-source agreement among OONI / IODA / CensoredPlanet / Voidly probes.
  • Per-measurement classifier v1 — XGBoost on 84K evidence rows. Aggregated to per-country-day mean. Caveat: AUC ~1.0 because the model reconstructs the labeling rule from signal_value + source patterns rather than discovering novel signal.

The build

  • Training set: 4,237 country-day samples / 1,116 positive / 131 countries (same labels v3.3 uses).
  • Feature vector: 5 base-learner scores per row — p_classifier (out-of-fold from v3.3), dbscan_score, dbscan_is_anom, p_corroboration, p_measurement.
  • p_classifier uses 5-fold out-of-fold predictions from the v3.3 architecture — the textbook stacking move that prevents the base learner's in-sample optimism from leaking into the meta-learner.
  • Meta-learner: head-to-head between logistic regression (class-weight balanced) and a shallow MLP (16 → 8 → 1, ReLU, early-stopping). Both standardized via per-fold scalers inside a Pipeline.
  • Promote criteria: stratified-CV F1 ≥ v3.3 baseline + 0.02 AND LOCO median F1 ≥ 0.85. Honest deltas only.

Results

  • v3.3 alone (out-of-fold, p_classifier-as-probability): F1 0.7424, AUC ≈ 0.91.
  • LogisticRegression meta: F1 0.7534, AUC 0.9033 — winner by 0.07 F1 over MLP.
  • MLP(16,8) meta: F1 0.7527, AUC 0.8977 — lower-recall, higher-precision (0.84 vs 0.77).
  • LOCO median F1: 0.8974 (mean 0.8299, 97 of 131 countries evaluated; 34 skipped for degenerate splits).
  • Stratified F1 delta vs baseline: +0.0110 (needs +0.0200 to clear gate).

Promote decision: NOT promoted

  • Stratified F1 gate: +0.0110 < +0.0200 required → FAIL
  • LOCO median gate: 0.8974 ≥ 0.85 required → PASS
  • One out of two gates passed. We do not promote on a single-gate pass; the F1 lift is real but inside the noise band of the v3.3 baseline (std ≈ 0.017).

What the coefficients show

The LR fit on the full training set produces these standardized weights:

  • p_classifier = 1.948 (dominant)
  • dbscan_is_anom = 0.324 (modest)
  • dbscan_score = 0.136
  • p_measurement = 0.122 (smaller than expected given AUC ~1.0; the OOF v3.3 has already absorbed most of the per-measurement signal)
  • p_corroboration = 0.0007 (effectively zero)
  • Intercept: −0.359

The Bayes posterior adds almost nothing on top of v3.3 — the cross-source agreement signal it captures is already implicit in v3.3's anomaly rate × measurement count interaction. DBSCAN is the only meaningful diversifying signal, adding ~+0.3 log-odds when its binary flag fires.

Honest caveats

  • Base learners share training labels. All four base learners ultimately train against the same incident labels. True meta-learning gains require base learners with structurally different label sources — that's a longer-term roadmap item.
  • Per-measurement classifier inflates by reconstructing the labeling rule. Its mean-per-day score should not be read as independent evidence.
  • DBSCAN scores only cover 92.6% of training rows (rolling window needs warm-up); missing rows fall back to 0.
  • Small absolute lift. +1.1pp F1 is real but inside the per-fold std (≈ 0.017). One bad fold could flip the sign.

Why ship it anyway

Even though we did not promote, the endpoint is live as a transparency tool: every base learner's score for a given country-day, plus the meta-learner's combined view, plus the per-feature log-odds contribution. When v3.3 and DBSCAN disagree, the meta-learner's confidence calibrates the disagreement — that's a useful diagnostic for human review even when it's not a leaderboard win.

Live at

GET /v1/classifier/stacking/info — model meta + gate result + coefficients
GET /v1/classifier/stacking/{cc}?date=YYYY-MM-DD — meta-learner score + per-base contributions

Example: GET /v1/classifier/stacking/IR returns meta_learner_probability with all four base scores broken out, plus the log-odds contribution from each.

Negative result number 4

This is our fourth honest negative — TabPFN v2 (frozen), v3.4 regime-finetune (rolled back), per-ASN forecast (not viable yet), and now v3.7 stacking ensemble. We surface every one of them rather than burying them, because tracking which ideas don't work is how the platform stays calibrated.

The source record.

Original source links and labels. Linked responses may have changed since publication.