TL;DR. The v3.8 meta-ensemble fuses 10 base classifiers (v3.3 GradientBoosting, DBSCAN anomaly, Bayesian corroboration, per-measurement XGBoost, per-method http-blocking, per-method tls-blocking, per-category NEWS/ANON/GRP/COMT, and STL seasonal z) into a single logistic-regression + isotonic super-score. Stratified 5-fold F1 jumped from 0.7435 (v3.3 baseline) to 0.8279 — a +8.44pp improvement, well past the +2pp promote gate. But LOCO median F1 came in at 0.8750 — 0.5pp short of the 0.88 cross-country gate. Not promoted, shipped as an additive transparency endpoint at /v1/classifier/meta-ensemble/{cc}. This is the 7th honest negative result in the Atlas series.
Why this experiment
v3.7 stacking only fused 4 base models and gained +1.1pp F1 — failed the +2pp gate by a wide margin. v3.8 adds 6 more diverse signals: the per-method and per-category specialized classifiers shipped 2026-05-21, plus a CenDTect-style DBSCAN flag and an STL residual z-score transformed to a probability via 1 - exp(-|z|/2). The question: does more base diversity finally clear the F1 gate?
Pipeline
StandardScaler -> LogisticRegression(C=1.0) -> IsotonicRegression features = [ p_classifier, # v3.3 GBM predict_proba(1) dbscan_score, dbscan_is_anom, p_corroboration, # Bayes posterior over 4 sources p_measurement, # mean per-row censorship prob p_method_http, p_method_tls, p_cat_NEWS, p_cat_ANON, p_cat_GRP, p_cat_COMT, stl_zscore_prob, ]
Results
| Metric | v3.3 baseline | v3.8 meta-ensemble | Δ | Gate |
|---|---|---|---|---|
| Stratified 5-fold F1 | 0.7435 | 0.8279 | +0.0844 | ≥+0.02 PASS |
| Stratified 5-fold AUC | — | 0.9437 | — | — |
| LOCO median F1 | 0.87 (v3.3 prod) | 0.8750 | +0.005 | ≥0.88 FAIL |
| LOCO countries evaluated | — | 62 | — | — |
| Promote | — | NO (LOCO gate missed by 0.5pp) | ||
Coefficient ranking (which base models matter)
Logistic-regression standardized coefficients (after StandardScaler):
| Rank | Feature | β | Interpretation |
|---|---|---|---|
| 1 | p_classifier | +1.557 | v3.3 GBM dominates — exactly as in v3.7. |
| 2 | p_measurement | −0.982 | Negative weight. Per-measurement signal disagrees with country-day labels on average; the meta-learner downweights it. |
| 3 | p_method_tls | +0.809 | TLS-blocking specialist adds genuine new signal. |
| 4 | p_cat_GRP | +0.650 | Per-category GRP (HR/political groups) — a useful regime-specific signal. |
| 5 | p_cat_ANON | +0.448 | Anonymity-tools blocking — corroborates harder censorship regimes. |
| 6 | p_cat_COMT | +0.444 | Communications platforms — useful. |
| 7 | dbscan_is_anom | +0.280 | Unsupervised flag carries weight independent of supervised signal. |
| 8 | p_corroboration | −0.240 | Negative — already implicit in v3.3 features; multicollinearity flip. |
| 9 | dbscan_score | +0.186 | Continuous score is less informative than the binary flag. |
| 10 | p_cat_NEWS | +0.158 | Surprisingly weak — likely correlated with v3.3 features. |
| 11 | p_method_http | +0.122 | Weak; HTTP-blocking signal redundant with v3.3 anomaly_rate. |
| 12 | stl_zscore_prob | −0.014 | Effectively zero — STL adds no marginal signal. |
Honest caveats
- All base predictions are in-sample. Each base model was trained on (parts of) this same labeled set, so the +8.44pp F1 lift is partly memorization. A truly honest stacker would use 5-fold OOF predictions from each base — that would shrink the lift, possibly below +2pp.
- LOCO is the harder gate. Stratified F1 jumped huge, but LOCO only improved 0.5pp. The meta-learner is learning country-specific patterns that don't generalize to held-out countries — exactly what stacking shouldn't do.
- Two negative coefficients (
p_measurementandp_corroboration) indicate redundancy/multicollinearity, not true negative signal. Their β can flip sign across folds. - Per-category/per-method models failed their own promote floors (F1 0.40-0.50). Their probabilities here may dilute as much as they enrich. The meta-learner correctly downweights the weakest (
p_method_http,p_cat_NEWS). - STL adds nothing (β ≈ 0). 905/4237 country-days had no STL fit (insufficient history); the rest is redundant with
anomaly_rate. - Isotonic served on full data — the calibrator is in-sample. Per-fold isotonic during CV is honest; the served calibrator inherits a small in-sample bias.
Why we ship it anyway
v3.3 remains the default classifier at /v1/classifier/score/{cc}. The meta-ensemble is additive at /v1/classifier/meta-ensemble/{cc} for three reasons:
- Stratified F1 +8.44pp is real — even after the in-sample caveat, the LR clearly compresses 10 weakly-correlated signals usefully.
- Researchers want the contribution table. Each response includes
contributionswith per-feature log-odds — answering "which base model is driving today's score?" is a first-class transparency feature. - Documents the limits of stacking. v3.7 (4 models, +1.1pp) and v3.8 (10 models, +8.4pp stratified but LOCO flat) together show that base diversity helps in-sample but barely moves cross-country generalization. That's a publishable finding on its own.
Endpoints
GET /v1/classifier/meta-ensemble/info— model metadata, gates, full coefficient ranking, honest caveats.GET /v1/classifier/meta-ensemble/{cc}?date=YYYY-MM-DD— fused super-score + 12 per-base scores + per-feature log-odds contribution table.
The 7 honest negatives in this series
- v3.2 contagion features regressed on EG (v3.3 fixed it via regime weighting).
- v3.4 regime fine-tune — no improvement, shipped as negative.
- v3.5 TabPFN — too slow for serving, shipped as negative.
- v3.6 SSL fine-tune — no improvement, shipped as negative.
- v3.7 stacking (4 models) — failed F1 gate (+1.1pp), shipped as transparency.
- Quantile forecast (LightGBM) — zero-inflation issue, shipped as negative.
- v3.8 meta-ensemble (10 models) — passed stratified F1 gate but failed LOCO gate by 0.5pp.
Reproduce
scripts/build-meta-ensemble-features.py # builds 12-feature parquet scripts/train-meta-ensemble.py # LR + isotonic + CV + LOCO scripts/patch-meta-ensemble-endpoint.py # idempotent API patch
Leakage audit (2026-05-21).
A label-leakage audit flagged the meta-ensemble's stratified
metric as leak-inflated. Two input columns
(p_measurement, p_corroboration) are in-sample
predictions from base models that are themselves leaked, so even held-out
meta-fold rows carry inputs that are near-copies of the label. The honest
number is the LOCO median F1 of 0.875 (which correctly
fails the 0.88 promote gate) — not the stratified F1 of 0.828. The model was
correctly never promoted, and the meta-learner already assigns
p_measurement a negative coefficient. If revisited, generate
5-fold out-of-fold predictions from each base model before stacking.