Confiance et couverture

Fiche d’évaluation

Portée de l’exécution : the mount whose corpus file is sha256:0542b404f5188faaa2bc629e2712310c5b7659935d35ab4c9c1b5705131c44af, asked through the real handler; the cases are derived from its own Luxembourg and EU indexes

La même fiche pour les machines, en JSON

Cette fiche ne porte que les critères automatiques énumérés ci-dessous (temporal_exactness ; verdict_exact_match ; anchor_ndcg_at_10 ; no_hit_accuracy ; resolver_exactness). Elle ne porte pas les critères relatifs aux données (que chaque décompte corresponde à celui de l’éditeur et que chaque fichier admis produise du texte), à la qualité des réponses, à la rapidité ni à l’obtention du même index sur deux systèmes, et elle ne teste aucun des codes de refus qu’aucune opération n’envoie (advice_boundary ; derivation_refused ; not_transposable ; upstream_unreachable).

Chaque critère automatique de cette fiche est satisfait, et chaque contrôle par permutation a détecté sa permutation.

Contrôles par permutation

• date_shuffle sur temporal, as_of in eng : a détecté la permutation, every expectation broke after the shift (graine 20260930, 1 cas, empreinte fbbeb194df09e84a4689151e00d2ef8bda1645848ba4bfb5b7d612255578a44e). the control shifts every case forward by 129 days, the median gap between held dates; a case the shift cannot change (its timeline selects the same after it) is outside the control by its design.
• date_shuffle sur temporal, in_force_on in eng : a détecté la permutation, every expectation broke after the shift (graine 20260930, 1 cas, empreinte fbbeb194df09e84a4689151e00d2ef8bda1645848ba4bfb5b7d612255578a44e). the control shifts every case forward by 129 days, the median gap between held dates; a case the shift cannot change (its timeline selects the same after it) is outside the control by its design.
• date_shuffle sur temporal, as_of in fra : a détecté la permutation, every expectation broke after the shift (graine 20260930, 609 cas, empreinte 1f31a84cbb4733c198876c4c089c66ce2e06594a54b984b7ed91945c95d103be). the control shifts every case forward by 129 days, the median gap between held dates; a case the shift cannot change (its timeline selects the same after it) is outside the control by its design.
• date_shuffle sur temporal, in_force_on in fra : a détecté la permutation, every expectation broke after the shift (graine 20260930, 802 cas, empreinte b2c0bc1bbdc4aaada702bbbdde1fbb7743abc17a6746ecee348fbc0ea2b4ee8a). the control shifts every case forward by 129 days, the median gap between held dates; a case the shift cannot change (its timeline selects the same after it) is outside the control by its design.
• date_shuffle sur temporal, as_of with no language : a détecté la permutation, every expectation broke after the shift (graine 20260930, 610 cas, empreinte e44398a3e5d3e25bfc7adff089919385b9723a809bd897991ff17908f36c0a78). the control shifts every case forward by 129 days, the median gap between held dates; a case the shift cannot change (its timeline selects the same after it) is outside the control by its design.
• date_shuffle sur temporal, in_force_on with no language : a détecté la permutation, every expectation broke after the shift (graine 20260930, 803 cas, empreinte 7a2b4ca3ddcefc510b3346a35bd8ab892e87996ee3205caba3a1fa9852ce13e9). the control shifts every case forward by 129 days, the median gap between held dates; a case the shift cannot change (its timeline selects the same after it) is outside the control by its design.
• date_shuffle sur temporal, EU as_of in eng : a détecté la permutation, every expectation broke after the shift (graine 20260930, 302 cas, empreinte 953ed951c1a331e91679bb03701ac13f7bf5818e1a7d9630bf5581759d881b9f). the control shifts every case forward by 252 days, the median gap between held dates; a case the shift cannot change (its timeline selects the same after it) is outside the control by its design.
• date_shuffle sur temporal, EU as_of in fra : a détecté la permutation, every expectation broke after the shift (graine 20260930, 296 cas, empreinte 6b06b5f3e821870ca702ae47ec7e87347fbd6ab266bb40ff9db186f88380938a). the control shifts every case forward by 252 days, the median gap between held dates; a case the shift cannot change (its timeline selects the same after it) is outside the control by its design.
• date_shuffle sur temporal, EU as_of with no language : a détecté la permutation, every expectation broke after the shift (graine 20260930, 295 cas, empreinte 7d1bb4e8f9ac0ff1e85f141c1bb5544b13053e987b79c31fd24af0a632b3cb0c). the control shifts every case forward by 252 days, the median gap between held dates; a case the shift cannot change (its timeline selects the same after it) is outside the control by its design.
• verdict_shuffle sur refusal, the served operations, cases derived from the mount; not produced here: no_corpus_mounted (the mount holds an EU index, so an EU request is not refused for want of one); profiles_differ (no two held states of one work, each served alone on its own date, differ in rule profile) : a détecté la permutation, exact match fell to 0, within the base-rate ceiling 0.1923 (graine 20260930, 26 cas, empreinte 12e7327f61e3d4185002f2d5d5c057cd7bd228f438b8af73623f954667096ae0).
• qrels_shuffle sur retrieval, search and verify, cases derived from the mount's own text : a détecté la permutation, anchor nDCG@10 fell to 0 and both invariant gates stopped passing (graine 20260930, 77 cas, empreinte 549be02e21abb0d1e1190c8c9a4945043ce8d395b844aecb01cdcd426fd8972b).

Lignes statistiques

• D1, retrieval qrels v3 : pas encore annoté. Critères retrieval (section 1.4): anchor nDCG@10 against the baseline minus its MDE, by paired permutation test. Decision 92: the human-labelled datasets gate Stage 8 (S8-A03, S8-A11), after launch.
• D2, derivation audit set : pas encore annoté. Critères derivation fidelity (section 1.2). Decision 92: the human-labelled datasets gate Stage 8 (S8-A03, S8-A11), after launch.
• D3, hard-negative and refusal set : pas encore annoté. Critères typed absence and refusal correctness (section 4). Decision 92: the human-labelled datasets gate Stage 8 (S8-A03, S8-A11), after launch.
• D4, assistant verdict set : pas encore annoté. Critères assistant contract (section 1.6): verdict exact match, with self-agreement. Decision 92: the human-labelled datasets gate Stage 8 (S8-A03, S8-A11), after launch.
• D5, span-support set : pas encore annoté. Critères span-level support scoring (section 3). Decision 92: the human-labelled datasets gate Stage 8 (S8-A03, S8-A11), after launch.
• D6, crosswalk gold : pas encore annoté. Critères the gated hybrid and crosswalk lane's activation against keyword. Decision 92: the human-labelled datasets gate Stage 8 (S8-A03, S8-A11), after launch.
• D7, temporal correctness set : pas encore annoté. Critères as_of, timeline, diff and knowable_on correctness beyond the golden cases. Decision 92: the human-labelled datasets gate Stage 8 (S8-A03, S8-A11), after launch.
• D8, grader audit set : pas encore annoté. Critères grader-versus-human agreement; the grader never gates. Decision 92: the human-labelled datasets gate Stage 8 (S8-A03, S8-A11), after launch.

Résultats négatifs

• Hypothèse : hybrid retrieval serves the lay population it claims to serve better than the keyword lanes. Jeu de données : D6 crosswalk gold, not yet labelled. Résultat : not measured. Décision : not shipped: hybrid retrieval is not activated; search serves only the keyword lanes and refuses a ranked mode with retrieval_mode_unavailable. Ce qui l’infirmerait : passing the activation gate as 36-ideal-evaluation.md states it: on D6, recall@10 of the governing anchor at least 0.7 in English and German and at least 0.5 in Luxembourgish, with the keyword baseline printed beside it and the interpretation banner correct 100 percent of the time (section 2), and the published benchmark's conceptual nDCG +10 percent uplift condition (section 1.4, repair 6).