Model honesty, public
Every Sentinel shutdown forecast ships with a 90% conformal interval. This page tracks how often the real outcome lands inside that interval — the closer to 90%, the more honest the model. Data lives at /v1/sentinel/calibration/history and updates every 24h.
/v1/forecast/{cc}/7day response under aci_alpha + aci.* fields. Full ACI methodology →Live forecast accuracy (prod_rolling, 30-day window)
Brier < 0.10 is good, > 0.30 is concerning. Calibration MAE < 0.05 means predicted-probabilities track observed-rates closely. See /sentinel/backtest for the actual reliability diagram (predicted-mean vs observed-rate scatter) and /methodology#validation for the full evaluation methodology + 3-split honest baselines.
Empirical coverage — 90-day rolling
The blue line is the actual fraction of forecasts where the real outcome landed inside the 90% conformal interval. The green dashed line is the nominal target (0.90). If the blue stays close to the green, the model is well calibrated.
Blue: empirical coverage · Dashed green: nominal 0.90 target · Orange circles: drift alerts
Last 14 days
| Date | Coverage | q90 | n holdout | Drift? |
|---|---|---|---|---|
| Aug 13 | 90.7% | 0.130 | 2,025 | — |
| Aug 12 | 90.7% | 0.130 | 2,025 | — |
| Aug 11 | 90.7% | 0.130 | 2,025 | — |
| Aug 10 | 90.7% | 0.130 | 2,025 | — |
| Aug 9 | 90.7% | 0.130 | 2,025 | — |
| Aug 8 | 91.8% | 0.125 | 2,003 | — |
| Aug 7 | 91.8% | 0.125 | 2,003 | — |
| Aug 6 | 91.8% | 0.125 | 2,003 | — |
| Aug 5 | 91.8% | 0.125 | 2,003 | — |
| Aug 4 | 91.8% | 0.125 | 2,003 | — |
| Aug 3 | 91.8% | 0.125 | 2,003 | — |
| Aug 2 | 91.8% | 0.125 | 2,003 | — |
| Aug 1 | 92.3% | 0.158 | 1,997 | ⚠️ |
| Jul 31 | 91.3% | 0.125 | 2,203 | — |
What features the model actually uses
Sklearn feature_importances_ on the underlying XGBoost. 39 features total. Top-3 sum: 0.534 · Top-5: 0.588 · Top-10: 0.7. Healthy distribution — no single feature dominates the model.
- 1.recent_shutdown46.4%
- 2.block_rate_roll30_mean3.8%
- 3.month3.2%
- 4.critical_incident_7d2.8%
- 5.high_importance_event2.7%
- 6.week_of_year2.6%
- 7.block_rate_lag12.2%
- 8.block_rate_roll14_mean2.1%
- 9.block_rate_roll7_mean2.1%
- 10.blocked_count_lag32.1%
- 11.high_urgency_signals_7d2.1%
- 12.election_in_7days1.8%
Interpretation: The forecast model's top feature is gdelt_unrest_30d (0.25) — protest + conflict signals from the GDELT 1.0 global news feed. recent_shutdown, block_rate rolling means, and incident counts follow. risk_tier — the leaky country-level encoding that dominated our older classifier at 85% — contributes only ~2% here. Healthy distribution; no single feature dominates.
Raw JSON: /v1/sentinel/feature-importance
Related
- /methodology#validation — 3-split honest baselines (LOCO 0.91 AUC vs stratified 0.98)
- /v1/sentinel/accuracy — full evaluation JSON, updated nightly
- /atlas/elections — see the forecast in action (90-day upcoming elections)