8021a1debd
23개 평가셋 × 3 전략(legacy/rrf/rrf_boost) 측정 + 분석. 핵심 발견: - 전체 NDCG: legacy 0.705 → rrf 0.699 → rrf_boost 0.700 (미세 차이) - RRF가 약간 나쁜 이유: kw_001(산업안전보건법 제6장)에서 RRF가 4041 (근로기준법 안전과 보건)을 false positive로 promotion. NDCG 1.000→0.906. - boost가 가치 입증한 사례: news_004(guerre en Iran)에서 RRF의 미스를 완벽 보정해 legacy NDCG 복원. - RRF의 진짜 가치는 Phase 1+ 다중 신호(trigram, reranker, multi-query) 통합 시 발휘됨. 현 평가셋은 너무 단순해서 차이가 noise에 묻힘. 결정: rrf_boost를 default로 유지. Phase 1 후 재측정. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
25 lines
4.4 KiB
CSV
25 lines
4.4 KiB
CSV
label,id,category,intent,domain_hint,query,relevant_ids,returned_ids_top10,latency_ms,recall_at_10,mrr_at_10,ndcg_at_10,top3_hit,error
|
|
single,kw_001,exact_keyword,fact_lookup,document,산업안전보건법 제6장,3856;3868;3879,3856;4041;3868;3879;3912;3950;3908;3915;3911;3858,542.0,1.000,1.000,0.906,1,
|
|
single,kw_002,exact_keyword,fact_lookup,document,중대재해 처벌 등에 관한 법률 제2장 중대산업재해,3917;3921,3917;3921;3916;3922;3920;3918;3919;3923;3874;3946,526.3,1.000,1.000,1.000,1,
|
|
single,kw_003,exact_keyword,fact_lookup,document,화학물질관리법 유해화학물질 영업자,3981,3981;3980;3857;3988;3869;3880;3978;3986;3985;3979,533.4,1.000,1.000,1.000,1,
|
|
single,kw_004,exact_keyword,fact_lookup,document,근로기준법 안전과 보건,4041,4041;3853;3860;3883;3858;3865;4036;3885;3901;3888,522.5,1.000,1.000,1.000,1,
|
|
single,kw_005,exact_keyword,fact_lookup,document,산업안전보건기준에 관한 규칙 보호구,3888,3888;3905;3908;3909;3897;3885;3884;3890;3911;3901,528.3,1.000,1.000,1.000,1,
|
|
single,nl_001,natural_language_ko,semantic_search,document,기계로 인한 산업재해 관련 법령,3856;3868;3879;3854,3868;3856;3879;3895;3915;3872;3851;3867;3897;3863,530.3,0.750,1.000,0.832,1,
|
|
single,nl_002,natural_language_ko,semantic_search,document,사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일,3855;3867;3878,3855;3867;3878;3863;3917;3872;3854;3896;3861;3886,536.2,1.000,1.000,1.000,1,
|
|
single,nl_003,natural_language_ko,semantic_search,document,유해화학물질을 다루는 회사가 지켜야 할 안전 의무,3980;3981;3982,3857;3869;3980;3880;3896;3903;3854;3981;3909;3904,529.5,0.667,0.333,0.383,1,
|
|
single,nl_004,natural_language_ko,semantic_search,document,중대재해가 발생했을 때 경영책임자가 처벌받는 기준,3916;3917;3920;3921,3918;3917;3919;3921;3916;3923;3867;3922;3877;3984,538.3,0.750,0.500,0.565,1,
|
|
single,nl_005,natural_language_ko,semantic_search,document,안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가,3853;3865,3853;3876;3965;3871;3958;3875;3861;3866;3877;3856,533.4,0.500,1.000,0.613,1,
|
|
single,cl_001,crosslingual_ko_en,semantic_search,document,기계 안전 가드 설계 원리,3770;3856,3895;3770;3762;3773;3879;3856;3767;3788;3868;3897,529.9,1.000,0.500,0.605,1,
|
|
single,cl_002,crosslingual_ko_en,semantic_search,document,산업 안전 입문서,3755;3775;3776;3777,3755;3816;3775;3851;3896;3853;3876;3871;3776;3863,538.3,0.750,1.000,0.703,1,
|
|
single,cl_003,crosslingual_ko_en,semantic_search,document,전기 안전 위험,3772;3790,3772;3897;3790;4024;4018;4020;4023;4022;4013;4019,529.2,1.000,1.000,0.920,1,
|
|
single,news_001,news_ko,semantic_search,news,이란과 미국의 군사 충돌,4303;4304;4307;4316;4322;4323;4327;4335,4452;4307;4317;4321;4339;4331;4329;4418;4446;4459,530.3,0.125,0.500,0.160,1,
|
|
single,news_002,news_ko,semantic_search,news,호르무즈 해협 봉쇄,4316;4320;4322;4327,4349;4346;4199;4320;4322;4327;4340;4304;4316;4260,523.8,1.000,0.250,0.576,0,
|
|
single,news_003,news_en,semantic_search,news,Trump Iran ultimatum,4258;4260;4262,4199;4519;4202;4258;4321;4333;4515;4313;4445;4418,531.1,0.333,0.250,0.202,1,
|
|
single,news_004,news_fr,semantic_search,news,guerre en Iran,4199;4202;4210;4361;4363;4507;4519;4521,4199;4507;4521;4363;4519;4211;4258;4324;4210;4536,526.3,0.750,1.000,0.822,1,
|
|
single,news_005,news_crosslingual,semantic_search,news,이란 미국 전쟁 글로벌 반응,4202;4258;4262;4536;4303;4304;4316,4329;4457;4307;4345;4324;4452;4443;4444;4450;4262,539.1,0.143,0.100,0.079,1,
|
|
single,misc_001,other_domain,fact_lookup,document,강체의 평면 운동학,4063;4065,4071;4063;4066;4064;4065;4067;4069;4058;4068;4060,565.1,1.000,0.500,0.624,1,
|
|
single,misc_002,other_domain,semantic_search,document,질점의 운동역학,4060;4061;4062,4062;4060;4061;4070;4066;4064;4068;4058;4065;4059,555.3,1.000,1.000,1.000,1,
|
|
single,fail_001,failure_expected,semantic_search,document,Rust async runtime tokio scheduler 내부 구조,,4069;3789;4067;4070;4060;4061;4071;4062;3807;4433,533.4,0.000,0.000,0.000,1,
|
|
single,fail_002,failure_expected,semantic_search,document,양자컴퓨터 큐비트 디코히어런스,,4068;4058;4064;4060;4065;4063;4061;3899;4067;4196,537.1,0.000,0.000,0.000,1,
|
|
single,fail_003,failure_expected,semantic_search,news,재즈 보컬리스트 빌리 홀리데이,,4289;4281;4205;4116;4100;4077;4316;4343;4235;4504,529.7,0.000,0.000,0.000,1,
|