Voidly
All findingsPublished research

v3.8 cross-model meta-ensemble — fused 10 base classifiers, +8.4pp stratified F1 (PASS) but LOCO flat (FAIL), 7th honest negative

Calibrated Bayesian fusion: LogisticRegression + Isotonic over 10 base classifier outputs (v3.3 GBM, DBSCAN anomaly, Bayes corroboration, per-measurement XGB, per-method http+tls, per-category NEWS/ANON/GRP/COMT, STL seasonal z). Stratified 5-fold F1 = 0.8279 vs v3.3 baseline 0.7435 (+8.44pp, PASS…

Read the full published summary

Calibrated Bayesian fusion: LogisticRegression + Isotonic over 10 base classifier outputs (v3.3 GBM, DBSCAN anomaly, Bayes corroboration, per-measurement XGB, per-method http+tls, per-category NEWS/ANON/GRP/COMT, STL seasonal z). Stratified 5-fold F1 = 0.8279 vs v3.3 baseline 0.7435 (+8.44pp, PASS +2pp gate). LOCO median F1 = 0.8750 vs gate 0.88 (FAIL by 0.5pp). Coefficient ranking: p_classifier dominates (β=+1.56), p_measurement gets negative weight (β=−0.98, redundancy flip), p_method_tls (β=+0.81), p_cat_GRP (β=+0.65) carry genuine new signal; STL adds essentially nothing (β=−0.01). Not promoted because cross-country generalization didnt clear the gate — v3.3 stays default at /v1/classifier/score/{cc}; v3.8 is additive at /v1/classifier/meta-ensemble/{cc} for transparency. Seventh honest negative result in the Atlas series — documents that stacking 10 weakly-diverse base models helps in-sample but does NOT meaningfully improve cross-country LOCO.

TL;DR. The v3.8 meta-ensemble fuses 10 base classifiers (v3.3 GradientBoosting, DBSCAN anomaly, Bayesian corroboration, per-measurement XGBoost, per-method http-blocking, per-method tls-blocking, per-category NEWS/ANON/GRP/COMT, and STL seasonal z) into a single logistic-regression + isotonic super-score. Stratified 5-fold F1 jumped from 0.7435 (v3.3 baseline) to 0.8279 — a +8.44pp improvement, well past the +2pp promote gate. But LOCO median F1 came in at 0.8750 — 0.5pp short of the 0.88 cross-country gate. Not promoted, shipped as an additive transparency endpoint at /v1/classifier/meta-ensemble/{cc}. This is the 7th honest negative result in the Atlas series.

Why this experiment

v3.7 stacking only fused 4 base models and gained +1.1pp F1 — failed the +2pp gate by a wide margin. v3.8 adds 6 more diverse signals: the per-method and per-category specialized classifiers shipped 2026-05-21, plus a CenDTect-style DBSCAN flag and an STL residual z-score transformed to a probability via 1 - exp(-|z|/2). The question: does more base diversity finally clear the F1 gate?

Pipeline

StandardScaler -> LogisticRegression(C=1.0) -> IsotonicRegression
features = [
  p_classifier,          # v3.3 GBM predict_proba(1)
  dbscan_score, dbscan_is_anom,
  p_corroboration,       # Bayes posterior over 4 sources
  p_measurement,         # mean per-row censorship prob
  p_method_http, p_method_tls,
  p_cat_NEWS, p_cat_ANON, p_cat_GRP, p_cat_COMT,
  stl_zscore_prob,
]

Results

Metricv3.3 baselinev3.8 meta-ensembleΔGate
Stratified 5-fold F10.74350.8279+0.0844≥+0.02 PASS
Stratified 5-fold AUC—0.9437——
LOCO median F10.87 (v3.3 prod)0.8750+0.005≥0.88 FAIL
LOCO countries evaluated—62——
Promote—NO (LOCO gate missed by 0.5pp)

Coefficient ranking (which base models matter)

Logistic-regression standardized coefficients (after StandardScaler):

RankFeatureβInterpretation
1p_classifier+1.557v3.3 GBM dominates — exactly as in v3.7.
2p_measurement−0.982Negative weight. Per-measurement signal disagrees with country-day labels on average; the meta-learner downweights it.
3p_method_tls+0.809TLS-blocking specialist adds genuine new signal.
4p_cat_GRP+0.650Per-category GRP (HR/political groups) — a useful regime-specific signal.
5p_cat_ANON+0.448Anonymity-tools blocking — corroborates harder censorship regimes.
6p_cat_COMT+0.444Communications platforms — useful.
7dbscan_is_anom+0.280Unsupervised flag carries weight independent of supervised signal.
8p_corroboration−0.240Negative — already implicit in v3.3 features; multicollinearity flip.
9dbscan_score+0.186Continuous score is less informative than the binary flag.
10p_cat_NEWS+0.158Surprisingly weak — likely correlated with v3.3 features.
11p_method_http+0.122Weak; HTTP-blocking signal redundant with v3.3 anomaly_rate.
12stl_zscore_prob−0.014Effectively zero — STL adds no marginal signal.

Honest caveats

  • All base predictions are in-sample. Each base model was trained on (parts of) this same labeled set, so the +8.44pp F1 lift is partly memorization. A truly honest stacker would use 5-fold OOF predictions from each base — that would shrink the lift, possibly below +2pp.
  • LOCO is the harder gate. Stratified F1 jumped huge, but LOCO only improved 0.5pp. The meta-learner is learning country-specific patterns that don't generalize to held-out countries — exactly what stacking shouldn't do.
  • Two negative coefficients (p_measurement and p_corroboration) indicate redundancy/multicollinearity, not true negative signal. Their β can flip sign across folds.
  • Per-category/per-method models failed their own promote floors (F1 0.40-0.50). Their probabilities here may dilute as much as they enrich. The meta-learner correctly downweights the weakest (p_method_http, p_cat_NEWS).
  • STL adds nothing (β ≈ 0). 905/4237 country-days had no STL fit (insufficient history); the rest is redundant with anomaly_rate.
  • Isotonic served on full data — the calibrator is in-sample. Per-fold isotonic during CV is honest; the served calibrator inherits a small in-sample bias.

Why we ship it anyway

v3.3 remains the default classifier at /v1/classifier/score/{cc}. The meta-ensemble is additive at /v1/classifier/meta-ensemble/{cc} for three reasons:

  1. Stratified F1 +8.44pp is real — even after the in-sample caveat, the LR clearly compresses 10 weakly-correlated signals usefully.
  2. Researchers want the contribution table. Each response includes contributions with per-feature log-odds — answering "which base model is driving today's score?" is a first-class transparency feature.
  3. Documents the limits of stacking. v3.7 (4 models, +1.1pp) and v3.8 (10 models, +8.4pp stratified but LOCO flat) together show that base diversity helps in-sample but barely moves cross-country generalization. That's a publishable finding on its own.

Endpoints

  • GET /v1/classifier/meta-ensemble/info — model metadata, gates, full coefficient ranking, honest caveats.
  • GET /v1/classifier/meta-ensemble/{cc}?date=YYYY-MM-DD — fused super-score + 12 per-base scores + per-feature log-odds contribution table.

The 7 honest negatives in this series

  1. v3.2 contagion features regressed on EG (v3.3 fixed it via regime weighting).
  2. v3.4 regime fine-tune — no improvement, shipped as negative.
  3. v3.5 TabPFN — too slow for serving, shipped as negative.
  4. v3.6 SSL fine-tune — no improvement, shipped as negative.
  5. v3.7 stacking (4 models) — failed F1 gate (+1.1pp), shipped as transparency.
  6. Quantile forecast (LightGBM) — zero-inflation issue, shipped as negative.
  7. v3.8 meta-ensemble (10 models) — passed stratified F1 gate but failed LOCO gate by 0.5pp.

Reproduce

scripts/build-meta-ensemble-features.py  # builds 12-feature parquet
scripts/train-meta-ensemble.py            # LR + isotonic + CV + LOCO
scripts/patch-meta-ensemble-endpoint.py   # idempotent API patch

Leakage audit (2026-05-21). A label-leakage audit flagged the meta-ensemble's stratified metric as leak-inflated. Two input columns (p_measurement, p_corroboration) are in-sample predictions from base models that are themselves leaked, so even held-out meta-fold rows carry inputs that are near-copies of the label. The honest number is the LOCO median F1 of 0.875 (which correctly fails the 0.88 promote gate) — not the stratified F1 of 0.828. The model was correctly never promoted, and the meta-learner already assigns p_measurement a negative coefficient. If revisited, generate 5-fold out-of-fold predictions from each base model before stacking.

The source record.

Original source links and labels. Linked responses may have changed since publication.