Ranking Methodology

Owner: safereturn-policy-advocacy · Version: 1.0 (2026-07-06) Status: PROVISIONAL. Scores are analytical, not legal determinations of treaty compliance. Requires qualified legal review before any country is publicly labelled as under-performing.

This document is the single source of truth for how the index is computed. The build script site/build-hague-index.mjs implements exactly what is written here. If the two ever disagree, the code is the bug.

1. What this index is — and is not

The Hague Return Accountability Index (HRAI) rates the system each Contracting Party operates for handling 1980 Hague Convention child-return cases: how transparent it is, how it participates in the multilateral data-gathering that makes accountability possible, how it cooperates, and — where evidence exists — how quickly and effectively it returns children and how efficiently it spends public money doing so.

It rates systems and measured outcomes — never individuals, motives, or national character (inherited verbatim from the T07 Country Compliance Index methodology, which this document supersedes and extends).

It is not a legal finding of compliance or non-compliance, not legal advice, and not a claim that any country "hides" data or "protects abductors." Where a state has not published data, the index says so and treats it as a transparency signal and a confidence limitation — not as a performance zero.

2. Foundation: what changed from the T07 draft

The prior artifact — 08-policy-advocacy/country_compliance_index_methodology.md (T07 v1.0 draft) — is the primary foundation. We keep its most important guarantees and change the top-level structure to the 6-category public frame requested for this build.

Kept from T07 (unchanged):

• Evidence hierarchy E1–E6 (E1 HCCH/multilateral official → E6 media/anecdote, context only).
• Confidence gate and "insufficient data is itself a transparency signal, never an estimated score."
• Defamation-safe language rules (the ✅/❌ table) — binding on every rendered sentence.
• Right-of-response, two-scorer rule, annual re-scoring, versioned methodology.
• "Do not adjust for region, religion, income, or political alignment."

Changed / added for HRAI v1.0:

1. Collapsed 11 T07 dimensions into the 6 public categories below (mapping in §4). This makes the public leaderboard legible without losing the underlying evidence.
2. Added Cost Efficiency (10 pts) — absent from T07 — as a first-class category, currently null for every country pending budget research (see hague-budget-methodology.md).
3. Added an explicit "measured-weight renormalization" so that a country is never penalised merely for the absence of a category we cannot yet measure for anyone (return-outcome, cost). This operationalises T07's "redistribute the un-scorable points pro-rata."
4. Separated the Performance score from the Confidence score everywhere (T07 kept them closer).
5. Made per-country transparency directly measurable from HCCH statistical-survey participation, so the launch index has a uniform, defensible spine for ~80 states rather than depending on data that does not exist per-country.

3. The uncomfortable truth this index is built around

Published, per-country, global data on how many children each requested state actually returned does not exist. The authoritative HCCH statistical studies (Prel. Doc. 19A, 2021 data) report outcomes globally — a 39% overall return rate, 207-day average resolution — not requested-state by requested-state. The only per-country outcome data available is the U.S. Department of State annual report, which covers U.S.-related cases only (a proxy — typically a handful of cases per country). Public budget data for Central Authorities is effectively unpublished everywhere.

The design response (v2): rank on the best available outcome and timeliness evidence — the U.S.-proxy resolution and timeliness data plus the noncompliance list — with heavy small-sample shrinkage and loud proxy labels, and mark every country without such evidence "insufficient data." We do not substitute transparency for missing outcomes: publishing a survey is not returning a child. The honest cost is thin coverage (~14 rankable, ~89 insufficient) and a U.S.-centric lens — both stated plainly, both fixable only by obtaining real national return statistics. Surfacing that gap is a core finding, not a defect.

4. Categories, weights, and mapping

Total = 100 points. Every category score is expressed on a 0–100 scale in the UI.

v2 (outcome-driven). Only the four driver categories move the score; transparency and cost are context (weight 0).

Rationale for demoting transparency: it measures whether a state publishes data, not whether it returns children. Rewarding it made a survey-answering state with zero cases outrank an active one — the exact objection this version fixes. Treaty status remains a gate (only Contracting Parties are ranked).

5. How each category is scored

5.1 Transparency (context, weight 0) — evidence quality only, NOT a performance score

Still computed and shown (so readers see who participates in HCCH data), but it does not move the ranking; it feeds Confidence via Data Quality. Derived entirely from HCCH statistical-survey participation (05-data/country_flows_2021.csv, sourced from HCCH Prel. Doc. 19A, E1). Sub-score 0–100:

A state absent from the responding-states table (e.g. Thailand) scores near 0 on Transparency — recorded as "did not respond to the HCCH 2021 statistical survey," which is the T07-approved safe phrasing, never "hides its data."

5.2 Cooperation & Compliance (15) — partial, U.S.-perspective proxy, clearly labelled

• Default: null (unrated → does not enter the score) unless a source exists.
• U.S. IPCA 2024 (E2, proxy):
• Country cited for a "pattern of noncompliance" → 25/100, rendered with the exact statutory caveat that this "does not necessarily mean the country is in violation of any Convention obligations," and framed as from the U.S. perspective on U.S.-related cases only.
• In the report with ≥5 U.S. cases, not cited, narrative positive (strong CA, timely, enforced) → 80/100.
• Narrative mixed (e.g. communication delays noted) → 60/100.
• ECtHR non-enforcement findings (E3): planned input; not yet coded (data gap).
• The proxy nature caps this category's contribution to confidence (evidence tier E2, single-perspective).

5.3 Speed & Enforcement (30) — driver

• Per-country day-level data is not published globally, so this is null unless a U.S.-proxy timeliness narrative exists: "timely" location/decision/enforcement → 75; a citation's enforcement failure → 35.
• Rankable-if-present: a country with speed evidence but no return-outcome evidence can still be ranked.
• The report always shows the global benchmark: 207-day average vs. the Art. 11 six-week (42-day) aspiration; 24% of cases > 300 days (2021).

5.4 Return Effectiveness (45) — PRIMARY driver, U.S.-proxy with small-N shrinkage

• The headline signal is actual outcomes: of the country's U.S.-related return cases, how many were not left unresolved at year end (x of N). Because N is tiny, the rate is shrunk toward a 0.45 prior (k = 8): (x + 3.6) / (N + 8) × 100. So Israel's 4/4 → 63, not 100; Germany 4/6 → 54.
• A U.S. pattern-of-noncompliance citation caps this at 30 (systemic failure to secure return/enforcement).
• Countries with no per-country outcome evidence get null → they are "insufficient data," not scored.
• "Resolved" is never treated as "returned" unless the source states the child was physically returned (§7); the shrinkage and the loud proxy/small-N labels reflect this uncertainty.
• Honest limitation: this makes the ranking U.S.-centric and small-sample. Replacing it with real national return statistics (e.g. Germany's BfJ figures) is the top data-priority — see the audit report.

5.5 Cost Efficiency (10) — estimated for 4 countries, excluded from the ranking

• No official Central-Authority budget line exists for any country.
• For Israel, Germany, the United States and Thailand, a transparent Level-B/C estimate is published (FTE × fully-loaded public-sector cost × overhead → direct; × broad multiplier → broad public cost), with wide ranges, low confidence (0.15–0.30), and a proxy denominator for cost-per-returned-child (national incoming return applications × global 39% return rate) — every figure is badged estimated and carries an on-page proxy/small-N warning. Details in hague-budget-methodology.md.
• These estimates do NOT enter the cross-country overall score — the instruction was to update the ranking "only where budget data is sufficiently reliable," and estimates at confidence ≤0.30 are not. Cost Efficiency therefore stays null in the ranking math while being fully shown per country.
• All other countries render "Budget data not publicly available" + tailored FOIA questions.

5.6 Data Quality & Confidence (5)

Computed from: count of E1–E2 sources, source freshness (≤3 years), and treaty-status confidence. Feeds both this 5-pt category and the separate overall Confidence number.

6. Normalization, overall score, confidence, and tiers

v2 overall = outcomes + timeliness, driver blend. Following user direction ("measure actual success in returning children, and timeliness — not transparency"), the overall is a renormalised weighted blend of the categories that describe what actually happens to children:

DRIVER weights:  Return effectiveness 45 · Speed & enforcement 30 · Cooperation 15 · Data quality 10
overall = Σ(score_i × weight_i) / Σ(weight_i)   over the driver categories that have evidence

Transparency and Cost Efficiency are NOT drivers (weight 0). Transparency is shown for context and feeds Confidence via Data Quality; publishing a survey shows a country shows its work, not that it returns children. A country is rankable only if it has actual return-outcome or timeliness evidence (return_effectiveness or speed non-null); otherwise it is "insufficient data," not scored. This is why the launch leaderboard holds ~14 countries and marks the rest insufficient — the honest consequence of outcome data being largely unpublished.

Small-sample shrinkage (critical). Per-country outcomes come mostly from the U.S. State Department report (U.S.-related cases only — a few per country). A raw "4 of 4 resolved" must not read as 100/100. So the proxy return-outcome rate is shrunk toward a 0.45 prior with pseudo-count k = 8: rate = (x + k·p0) / (N + k), where x = cases not left unresolved at year end and N = total. With N = 4, x = 4 this yields 63, not 100. Confidence is additionally reduced for N < 10. A U.S. pattern-of-noncompliance citation caps return effectiveness at 30 and pins speed to 35.

Confidence (0–1), separate from performance. Confidence measures how strong and how broad the evidence is — not how well the country performs.

tier_factor = mean evidence tier of the inputs used   # E1=1.0, E2=0.85, E3=0.7, E4=0.55, E5/E6→excluded
freshness   = share of inputs ≤ 3 years old
evidence    = 0.6·tier_factor + 0.4·freshness          # quality of the inputs (≈0.95 for fresh E1 data)
breadth     = (# of the 4 DRIVER categories with a non-null score) / 4
confidence  = evidence × (0.35 + 0.65·breadth)
              × small-N penalty   # ×(0.5+0.5·N/10) if the proxy sample N<10; ×0.6 if outcome is a binary citation

Confidence is surfaced as Low (<0.5) / Medium (0.5–0.7) / High (>0.7) in the UI. Small U.S.-proxy samples pull confidence down — e.g. Israel (N=4) lands at Medium, not High.

Rankable gate (two conditions). A country is ranked only if (a) it has actual outcome/timeliness evidence (return_effectiveness or speed non-null) and (b) confidence ≥ 0.35. Otherwise → "Insufficient data for ranking" (it still gets a page: status, gaps, FOIA questions). Transparency alone never qualifies a country. All scores are badged Provisional.

Ranking tiers (on the outcome-driven overall, rankable countries only):

Tier language is about measured performance on the available (largely U.S.-proxy) evidence, always badged Provisional and never a legal or moral determination. Because coverage is thin (~14 rankable), the leaderboard is explicit that most countries are "insufficient data," not silently ranked or dropped.

7. Binding definitions (consistent across the whole site)

• Returned Child — a child physically returned to the country of habitual residence.
• Resolved Return Case — a return case officially closed by any means (voluntary agreement, judicial order, withdrawal, settlement, refusal). Never equated with a return unless the source states the child was returned.
• Access Case — a case about access/contact rights, not return.
• Proxy Data — data representing only part of a country's activity (e.g. U.S.-related cases only).
• Direct Central Authority Cost / Broad Public Implementation Cost / International Public Support / Cost per Returned Child / Cost per Resolved Return Child — defined in hague-budget-methodology.md.

8. Missing-data rules (hard)

1. Unknown value → null, never 0. 0 only when a source confirms zero.
2. null categories are excluded from the score (§6), never floored to zero.
3. Proxy data is labelled "proxy" at every point of display and never aggregated into a global rate.
4. A country below the confidence gate is marked Insufficient data — a transparency signal, not a punishment; its Transparency finding ("did not respond to survey X") is stated in safe language.
5. Every rendered number carries a source id resolving to sources.json (title, publisher, URL, date, tier, confidence).

9. Verification & transparent calculation objects

Before any ranking is written, the build runs two independent verification passes (implemented in verify() in site/build-hague-index.mjs; the build fails if the formula pass fails):

1. Raw-data vs. source. For every country, the "responded to the 2021 HCCH survey" flag derived from country_flows_2021.csv must agree with the transparency score the site displays.
2. Formula vs. displayed score. The overall is recomputed independently (transparency − 15·cited) and must equal the displayed overall for all ranked countries (tolerance 0.05).

Both passes are reported in the build log and surfaced on the leaderboard ("Verified twice at build time").

Every country also gets a machine-readable calculation object (assets/data/hague/calculations.json) and a human-readable version on its page, with per-component raw_inputs, normalized_score, weight, weighted_points, sources, an in_overall flag, a verification block (recomputed vs displayed), and warnings. The weighted_points column previews the target 6-category frame and intentionally does not sum to the overall at v1.0, because Return Effectiveness and Cost Efficiency are null/excluded.

10. Governance

Two-scorer rule, annual re-score after each HCCH statistical release, event-driven interim updates, 30-day right-of-response to each Central Authority before any country is published in a concern tier, verbatim publication of responses, immutable git change log. Confidence in methodology: 0.8. Open items: ECtHR enforcement-findings ingestion; budget FOIA program; legal review of tier labels; pilot legal sign-off on US / DE / PL before publishing any Tier-4 label.