Voidly
All findingsPublished research

Per-country F1-optimal thresholds — +4pp median F1 without retraining v3.3

v3.3 uses a single 0.5 decision threshold across 131 countries. Computing per-country F1-optimal thresholds via precision-recall sweep lifts median F1 +4pp (mean +5.4pp) for 73 countries with sufficient labels. 41 countries improve by ≥3pp. CG (+35.7pp), OM (+31.7pp), ZW (+20.1pp) are the biggest…

Read the full published summary

v3.3 uses a single 0.5 decision threshold across 131 countries. Computing per-country F1-optimal thresholds via precision-recall sweep lifts median F1 +4pp (mean +5.4pp) for 73 countries with sufficient labels. 41 countries improve by ≥3pp. CG (+35.7pp), OM (+31.7pp), ZW (+20.1pp) are the biggest movers. Live in /v1/classifier/score/{cc} as label_per_country + threshold_used + threshold_source fields.

Voidly classifier v3.3 uses a single 0.5 decision threshold across all 131 countries. The model itself is competitive (LOCO median F1 0.870) but the SAME threshold for Russia (high background rate) and Egypt (clean signal) leaves real F1 on the table. Different countries have different optimal decision boundaries.

Approach

For each of 73 countries with sufficient labeled positives (≥3 positives), compute the F1-optimal threshold via precision-recall curve sweep on the model's in-sample predict_proba. Fall back to the global threshold (0.546) for the 58 sparse-label countries.

Results

  • 73 countries calibrated with per-country thresholds; 58 fall back to global
  • Median per-country F1 delta vs global: +4.0pp
  • Mean delta: +5.4pp
  • 41 countries improve by ≥3pp

Top movers

Country Optimal threshold F1 delta vs global n positives
CG0.163+35.7pp3
OM0.467+31.7pp4
ZW0.247+20.1pp13
DO0.388+20.0pp3
TZ0.371+20.0pp3
JO0.502+16.0pp12
CU0.795+15.7pp10
ET0.789+11.4pp26

API contract

Both GET /v1/classifier/score/{cc} and POST /v1/classifier/score now return BOTH the legacy label_at_0.5 and the new label_per_country + threshold_used + threshold_source fields. Callers can opt into either label scheme. The threshold_source field is one of: 'per-country' (a country-specific F1-optimal threshold was found), 'global' (fell back to the cross-country threshold because too few labeled positives), or 'default-0.5' (sidecar JSON unavailable; emergency fallback).

Honest caveat

The thresholds are computed on the model's IN-SAMPLE predict_proba (because v3.3's per-country LOCO predictions aren't persisted in the bundle). This means the thresholds reflect what would be optimal IF the model generalized perfectly to held-out data. In practice, true out-of-fold optimal thresholds are likely slightly higher (the in-sample probabilities are more confident than they should be). Treat the calibration as a per-country prior, not as a clean LOCO eval — the v3.3 model itself was LOCO-evaluated separately and remains the reference.

Reproducibility

scripts/optimize-classifier-thresholds.py — sweep + sidecar writer.

scripts/patch-per-country-thresholds.py — idempotent ast-validated patch to api_v3.py adding the lookup + response fields.

Sidecar JSON at /opt/voidly-ai/ml-deploy/classifier_v3.3_per_country_thresholds.json (also synced to ml-deploy/ in the repo).

The source record.

Original source links and labels. Linked responses may have changed since publication.