Voidly classifier v3.3 uses a single 0.5 decision threshold across all 131 countries. The model itself is competitive (LOCO median F1 0.870) but the SAME threshold for Russia (high background rate) and Egypt (clean signal) leaves real F1 on the table. Different countries have different optimal decision boundaries.
Approach
For each of 73 countries with sufficient labeled positives (≥3 positives), compute the F1-optimal threshold via precision-recall curve sweep on the model's in-sample predict_proba. Fall back to the global threshold (0.546) for the 58 sparse-label countries.
Results
- 73 countries calibrated with per-country thresholds; 58 fall back to global
- Median per-country F1 delta vs global: +4.0pp
- Mean delta: +5.4pp
- 41 countries improve by ≥3pp
Top movers
| Country | Optimal threshold | F1 delta vs global | n positives |
|---|---|---|---|
| CG | 0.163 | +35.7pp | 3 |
| OM | 0.467 | +31.7pp | 4 |
| ZW | 0.247 | +20.1pp | 13 |
| DO | 0.388 | +20.0pp | 3 |
| TZ | 0.371 | +20.0pp | 3 |
| JO | 0.502 | +16.0pp | 12 |
| CU | 0.795 | +15.7pp | 10 |
| ET | 0.789 | +11.4pp | 26 |
API contract
Both GET /v1/classifier/score/{cc} and
POST /v1/classifier/score now return
BOTH the legacy label_at_0.5 and
the new label_per_country +
threshold_used +
threshold_source fields. Callers
can opt into either label scheme. The threshold_source field is
one of: 'per-country' (a
country-specific F1-optimal threshold was found),
'global' (fell back to the
cross-country threshold because too few labeled positives), or
'default-0.5' (sidecar JSON
unavailable; emergency fallback).
Honest caveat
The thresholds are computed on the model's IN-SAMPLE predict_proba (because v3.3's per-country LOCO predictions aren't persisted in the bundle). This means the thresholds reflect what would be optimal IF the model generalized perfectly to held-out data. In practice, true out-of-fold optimal thresholds are likely slightly higher (the in-sample probabilities are more confident than they should be). Treat the calibration as a per-country prior, not as a clean LOCO eval — the v3.3 model itself was LOCO-evaluated separately and remains the reference.
Reproducibility
scripts/optimize-classifier-thresholds.py —
sweep + sidecar writer.
scripts/patch-per-country-thresholds.py —
idempotent ast-validated patch to api_v3.py adding the lookup +
response fields.
Sidecar JSON at /opt/voidly-ai/ml-deploy/classifier_v3.3_per_country_thresholds.json
(also synced to ml-deploy/ in the repo).