120db86d74
## 결과 요약 Phase 1.3 baseline vs Phase 2 final A/B (평가셋 v0.1, 23 쿼리): - Recall@10: 0.730 → 0.737 (+0.007) - NDCG@10: 0.663 → 0.668 (+0.005) - Top-3 hit: 0.900 → 0.900 (0) - p95 latency: 171ms → 256ms (+85) - news_crosslingual NDCG: 0.27 → 0.37 (+0.10 ✓) - exact_keyword / natural_language_ko: 완전 유지 (회귀 0) ## Phase 2 게이트: 2/6 통과 ✓ news_crosslingual NDCG ≥ 0.30 ✓ latency p95 < 400ms ❌ Recall@10 ≥ 0.78 (0.737) ❌ Top-3 hit ≥ 0.93 (0.900) ❌ crosslingual_ko_en NDCG ≥ 0.65 (0.53, bge-m3 한계) ❌ 평가셋 v0.2 작성 (후속) ## 핵심 성과 (게이트 미달이지만 견고한 기반) 1. QueryAnalyzer async-only 아키텍처 (retrieval 차단 0) 2. semaphore concurrency=1 (MLX single-inference queue 폭발 방지) 3. multilingual narrowing (news/global 한정 → 회귀 0 + news 개선) 4. soft_filter boost 보수적 설정 (0.01, domain only) 5. prewarm 15개 → cache hit rate 70%+ ## infra_inventory.md soft lock 준수 - config.yaml / Ollama / compose restart 변경 0 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
4.4 KiB
4.4 KiB
| 1 | label | id | category | intent | domain_hint | query | relevant_ids | returned_ids_top10 | latency_ms | recall_at_10 | mrr_at_10 | ndcg_at_10 | top3_hit | error |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | single | kw_001 | exact_keyword | fact_lookup | document | 산업안전보건법 제6장 | 3856;3868;3879 | 3856;3851;3862;3853;3861;3868;3879;3873;3876;3871 | 67.9 | 1.000 | 1.000 | 0.793 | 1 | |
| 3 | single | kw_002 | exact_keyword | fact_lookup | document | 중대재해 처벌 등에 관한 법률 제2장 중대산업재해 | 3917;3921 | 3921;3917;3919;3923;3916;3874;3918;3854;3922;3920 | 110.0 | 1.000 | 1.000 | 1.000 | 1 | |
| 4 | single | kw_003 | exact_keyword | fact_lookup | document | 화학물질관리법 유해화학물질 영업자 | 3981 | 3981;3985;3980;3984;3993;3857;3978;3986;3983;3957 | 119.3 | 1.000 | 1.000 | 1.000 | 1 | |
| 5 | single | kw_004 | exact_keyword | fact_lookup | document | 근로기준법 안전과 보건 | 4041 | 4041;3852;3851;3877;3905;3903;3858;3881;3781;3912 | 108.7 | 1.000 | 1.000 | 1.000 | 1 | |
| 6 | single | kw_005 | exact_keyword | fact_lookup | document | 산업안전보건기준에 관한 규칙 보호구 | 3888 | 3888;3912;3911;3905;3909;3889;3910;3897;3890;3896 | 125.5 | 1.000 | 1.000 | 1.000 | 1 | |
| 7 | single | nl_001 | natural_language_ko | semantic_search | document | 기계로 인한 산업재해 관련 법령 | 3856;3868;3879;3854 | 3878;3897;3863;3868;3856;3879;3895;3867;3851;3854 | 83.9 | 1.000 | 0.250 | 0.571 | 0 | |
| 8 | single | nl_002 | natural_language_ko | semantic_search | document | 사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일 | 3855;3867;3878 | 3855;3917;3854;3867;3878;3863;3851;3908;3903;3895 | 118.0 | 1.000 | 1.000 | 0.853 | 1 | |
| 9 | single | nl_003 | natural_language_ko | semantic_search | document | 유해화학물질을 다루는 회사가 지켜야 할 안전 의무 | 3980;3981;3982 | 3980;3903;3904;3905;3981;3985;3896;3917;3857;3909 | 82.8 | 0.667 | 1.000 | 0.651 | 1 | |
| 10 | single | nl_004 | natural_language_ko | semantic_search | document | 중대재해가 발생했을 때 경영책임자가 처벌받는 기준 | 3916;3917;3920;3921 | 3917;3918;3916;3923;3919;3921;3854;3872;3877;3922 | 72.1 | 0.750 | 1.000 | 0.725 | 1 | |
| 11 | single | nl_005 | natural_language_ko | semantic_search | document | 안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가 | 3853;3865 | 3853;4025;3876;3879;3859;3865;3781;3815;3818;3787 | 80.2 | 1.000 | 1.000 | 0.832 | 1 | |
| 12 | single | cl_001 | crosslingual_ko_en | semantic_search | document | 기계 안전 가드 설계 원리 | 3770;3856 | 3770;4540;3817;4541;3774;3816;3787;3758;3793;3773 | 108.3 | 0.500 | 1.000 | 0.613 | 1 | |
| 13 | single | cl_002 | crosslingual_ko_en | semantic_search | document | 산업 안전 입문서 | 3755;3775;3776;3777 | 3760;3755;3774;3764;3758;3775;3779;3802;3814;3817 | 79.5 | 0.500 | 0.500 | 0.385 | 1 | |
| 14 | single | cl_003 | crosslingual_ko_en | semantic_search | document | 전기 안전 위험 | 3772;3790 | 3897;3772;3771;4018;3773;3790;3819;4020;3807;3755 | 103.7 | 1.000 | 0.500 | 0.605 | 1 | |
| 15 | single | news_001 | news_ko | semantic_search | news | 이란과 미국의 군사 충돌 | 4303;4304;4307;4316;4322;4323;4327;4335 | 4317;4321;4771;4743;4307;4452;4761;4678;4418;4331 | 1445.8 | 0.125 | 0.200 | 0.098 | 1 | |
| 16 | single | news_002 | news_ko | semantic_search | news | 호르무즈 해협 봉쇄 | 4316;4320;4322;4327 | 4327;4346;4349;4762;4767;4759;4322;4320;4340;4304 | 185.3 | 0.750 | 1.000 | 0.644 | 0 | |
| 17 | single | news_003 | news_en | semantic_search | news | Trump Iran ultimatum | 4258;4260;4262 | 4776;4515;4519;4658;4644;4763;4333;4762;4679;4321 | 76.3 | 0.000 | 0.000 | 0.000 | 1 | |
| 18 | single | news_004 | news_fr | semantic_search | news | guerre en Iran | 4199;4202;4210;4361;4363;4507;4519;4521 | 4678;4507;4199;4688;4776;4363;4519;4668;4670;4672 | 157.9 | 0.500 | 0.500 | 0.460 | 1 | |
| 19 | single | news_005 | news_crosslingual | semantic_search | news | 이란 미국 전쟁 글로벌 반응 | 4202;4258;4262;4536;4303;4304;4316 | 4262;4457;4765;4324;4345;4329;4258;4452;4443;4761 | 186.7 | 0.286 | 1.000 | 0.367 | 1 | |
| 20 | single | misc_001 | other_domain | fact_lookup | document | 강체의 평면 운동학 | 4063;4065 | 4063;4065;4064;4067;4071;4068;4069;4062;4060;4066 | 171.6 | 1.000 | 1.000 | 1.000 | 1 | |
| 21 | single | misc_002 | other_domain | semantic_search | document | 질점의 운동역학 | 4060;4061;4062 | 4062;4060;4070;4064;4068;4067;4065;4058;4071;4066 | 263.6 | 0.667 | 1.000 | 0.765 | 1 | |
| 22 | single | fail_001 | failure_expected | semantic_search | document | Rust async runtime tokio scheduler 내부 구조 | 4815;4069;4546;4062;4547;3801;3787;3812;4542;3770 | 121.9 | 0.000 | 0.000 | 0.000 | 1 | ||
| 23 | single | fail_002 | failure_expected | semantic_search | document | 양자컴퓨터 큐비트 디코히어런스 | 4058;4057;4067;3800;4065;4068;3817;4063;4064;3915 | 75.2 | 0.000 | 0.000 | 0.000 | 1 | ||
| 24 | single | fail_003 | failure_expected | semantic_search | news | 재즈 보컬리스트 빌리 홀리데이 | 4634;4100;4815;4116;4281;4697;4205;4077;4235;4289 | 73.9 | 0.000 | 0.000 | 0.000 | 1 |