diff --git a/reports/phase2_final.md b/reports/phase2_final.md new file mode 100644 index 0000000..22d5ff7 --- /dev/null +++ b/reports/phase2_final.md @@ -0,0 +1,125 @@ +# Phase 2 최종 측정 보고서 + +**측정일**: 2026-04-08 +**대상**: Document Server 검색 v2, Phase 2.1~2.3 통합 +**평가셋**: `tests/search_eval/queries.yaml` v0.1 (23 쿼리, 8 카테고리) +**인프라 기준**: `memory/infra_inventory.md` (2026-04-08 실측) + +## A/B 결과 + +| metric | Phase 1.3 baseline (A) | Phase 2 final (B) | Δ | +|---|---|---|---| +| Recall@10 | 0.730 | **0.737** | +0.007 ✓ | +| MRR@10 | 0.795 | 0.797 | +0.002 | +| NDCG@10 | 0.663 | **0.668** | +0.005 ✓ | +| Top-3 hit | 0.900 | 0.900 | 0 | +| Latency p50 | 114 ms | 109 ms | -5 | +| Latency p95 | 171 ms | **256 ms** | +85 | + +## 카테고리별 + +| category | A NDCG | B NDCG | Δ | 비고 | +|---|---|---|---|---| +| exact_keyword | 0.96 | 0.96 | 0 | 회귀 0 ✓ | +| natural_language_ko | 0.73 | 0.73 | 0 | 회귀 0 ✓ (narrowed multilingual 덕) | +| crosslingual_ko_en | 0.53 | 0.53 | 0 | bge-m3 한계 — multilingual 효과 0 | +| **news_crosslingual** | 0.27 | **0.37** | **+0.10** | 개선 ✓ | +| news_ko | 0.36 | 0.37 | +0.01 | 미세 | +| news_en | 0.00 | 0.00 | 0 | 여전히 0 | +| news_fr | 0.46 | 0.46 | 0 | | +| other_domain | 0.88 | 0.88 | 0 | | + +## Phase 2 게이트 검증 + +| 게이트 | 목표 | 실제 | 상태 | +|---|---|---|---| +| Recall@10 | ≥ 0.78 | 0.737 | ❌ (-0.043) | +| Top-3 hit | ≥ 0.93 | 0.900 | ❌ (-0.030) | +| crosslingual_ko_en NDCG | ≥ 0.65 | 0.53 | ❌ (-0.12) | +| news_crosslingual NDCG | ≥ 0.30 | 0.37 | ✓ | +| latency p95 | < 400 ms | 256 ms | ✓ | +| 평가셋 v0.2 완료 | - | v0.1만 | ❌ (후속) | + +**2/6 통과** — 목표 미달. 단 회귀 0 + 일부 영역 개선. + +## Phase 2에서 실제로 달성한 것 + +### 1. 아키텍처 — QueryAnalyzer async-only 구조 확립 +실측 기반 철학 수정 (memory `feedback_analyzer_async_only.md`): +- `query → retrieval (즉시)` + `→ analyzer (async) → cache` +- retrieval 경로에 LLM 동기 호출 0 +- background semaphore=1 (MLX single-inference 큐 폭발 방지) +- prewarm 15개 startup 시 자동 실행 +- cache hit rate 첫 사용자 요청부터 70%+ + +### 2. 실측 데이터 — MLX 한계 +gemma-4-26b-a4b-it-8bit MLX: +- full prompt (prompt_tok=2406) → **10.5초** +- 축소 prompt (prompt_tok=802) → **7~11초** +- concurrency >1 시 → **timeout 폭발** (semaphore=1 필수) +- 결론: analyzer는 **즉시 쓸 수 없는 자원** + +### 3. multilingual narrowing — domain별 효과 차등 +- 전 도메인 multilingual: natural_language_ko **-0.10 악화** ❌ +- `domain_hint == news OR language_scope == global` 한정: 회귀 0 + news_crosslingual **+0.10** ✓ +- 룰: 한국어 법령 검색에 영어 번역 쿼리 섞으면 noise + +### 4. soft_filter boost — 보수적 설정 필요 +- 초기 0.03+0.02 → exact_keyword **-0.03 악화** +- 낮춰서 0.01 단일 domain only → 회귀 0 +- 평가셋에 filter 쿼리가 없어 효과 직접 측정 불가 (v0.2 확장 후 재평가) + +## Phase 2에서 달성하지 못한 것 + 이유 + +### Recall@10 / Top-3 hit 회복 (0.730 → 0.78+ 미달) +- baseline 대비 +0.007 미세 개선만 +- 원인: **corpus 1022 docs로 noise 증가**. chunk 수 7129. bge-m3의 embedding 공간에서 상위 후보 밀도 높아짐 +- 해결책: retrieval 단계 품질 (Phase 3 evidence extraction) 또는 embedding 모델 업그레이드 + +### crosslingual_ko_en NDCG 0.65+ 미달 (0.53 정체) +- multilingual translation이 효과 없음 +- 원인: 현재 category 3개 쿼리 중 정답 doc이 영어 교재 (Industrial Safety and Health Management 등). bge-m3는 ko 쿼리로 이 영어 doc을 약 0.5~0.6 cosine으로 이미 찾음. translation 추가가 정보 증가 없음 +- 실제 필요: **reranker가 crosslingual pair**를 더 잘 학습해야 함 → bge-reranker-v2-m3의 한계 영역 + +### 평가셋 v0.2 완전 작성 +- 시간 제약 + 정답 doc_id 수동 라벨링 필요 +- 후속 작업으로 분리 + +## Phase 2 기여 commits (시간순) + +``` +d28ef2f Phase 2.1 QueryAnalyzer + LRU cache + confidence 3-tier (초기) +c81b728 async-only 구조 전환 (철학 수정) +324537c LLM_TIMEOUT_MS 5000 → 15000 (실측 반영) +1e80d4c setup_logger 수정 (prewarm 로그 보이도록) +f5c3dea Phase 2.2 multilingual + query embed cache +21a78fb semaphore concurrency=1 + run_eval --analyze 파라미터 +e595283 multilingual news/global 한정 narrowing +e91c199 Phase 2.3 soft_filter boost (초기) +01f144a soft_filter boost 약화 (0.01, doctype 제거) +``` + +## 다음 단계 선택지 (사용자 결정) + +### A. Phase 2 종료 + Phase 3 진입 (권장) +- Phase 2 성과: 아키텍처 + 회귀 0 + news 영역 개선 + 실측 기반 철학 확립 +- Recall/crosslingual 정체는 **Phase 2 범위 밖** — embedding/reranker 교체 혹은 Phase 3 evidence extraction으로 우회 +- Phase 3 (evidence extraction + grounded synthesis + `/api/search/ask`) 착수 + +### B. Phase 2 iteration — embedding 실험 +- bge-m3 → 다른 embedding (e.g., multilingual-e5-large-instruct, jina-embeddings-v3) 교체 실험 +- 대규모 재인덱싱 필요 (1022 docs × chunks) +- 인프라 변경이므로 infra_inventory.md drift 발생 + +### C. Phase 2 iteration — 평가셋 v0.2 작성 +- queries_v0.2.yaml 작성 (filter 쿼리 + graded relevance) +- 현재 Phase 2 코드의 filter 효과 측정 +- 단, Recall/crosslingual 근본 해결은 아님 + +## Soft Lock 준수 확인 (infra_inventory.md) + +- ✓ `config.yaml` 변경 없음 (GPU local override 그대로) +- ✓ `docker compose restart` 사용 안 함 (`up -d --build fastapi`만) +- ✓ Ollama 모델 pull/remove 없음 (bge-m3, exaone3.5 그대로) +- ✓ Reranker 모델 변경 없음 (TEI bge-reranker-v2-m3 그대로) +- ✓ Mac mini MLX 설정 변경 없음 diff --git a/reports/phase2_final_baseline.csv b/reports/phase2_final_baseline.csv new file mode 100644 index 0000000..aed84f2 --- /dev/null +++ b/reports/phase2_final_baseline.csv @@ -0,0 +1,24 @@ +label,id,category,intent,domain_hint,query,relevant_ids,returned_ids_top10,latency_ms,recall_at_10,mrr_at_10,ndcg_at_10,top3_hit,error +single,kw_001,exact_keyword,fact_lookup,document,산업안전보건법 제6장,3856;3868;3879,3856;3851;3862;3853;3861;3868;3879;3873;3876;3871,95.5,1.000,1.000,0.793,1, +single,kw_002,exact_keyword,fact_lookup,document,중대재해 처벌 등에 관한 법률 제2장 중대산업재해,3917;3921,3921;3917;3919;3923;3916;3874;3918;3854;3922;3920,120.7,1.000,1.000,1.000,1, +single,kw_003,exact_keyword,fact_lookup,document,화학물질관리법 유해화학물질 영업자,3981,3981;3985;3980;3984;3993;3857;3978;3986;3983;3957,120.2,1.000,1.000,1.000,1, +single,kw_004,exact_keyword,fact_lookup,document,근로기준법 안전과 보건,4041,4041;3852;3851;3877;3905;3903;3858;3881;3781;3912,139.3,1.000,1.000,1.000,1, +single,kw_005,exact_keyword,fact_lookup,document,산업안전보건기준에 관한 규칙 보호구,3888,3888;3912;3911;3905;3909;3889;3910;3897;3890;3896,157.8,1.000,1.000,1.000,1, +single,nl_001,natural_language_ko,semantic_search,document,기계로 인한 산업재해 관련 법령,3856;3868;3879;3854,3878;3897;3863;3868;3879;3856;3895;3867;3851;3854,85.9,1.000,0.250,0.571,0, +single,nl_002,natural_language_ko,semantic_search,document,사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일,3855;3867;3878,3855;3917;3854;3867;3878;3863;3851;3908;3903;3895,82.1,1.000,1.000,0.853,1, +single,nl_003,natural_language_ko,semantic_search,document,유해화학물질을 다루는 회사가 지켜야 할 안전 의무,3980;3981;3982,3980;3903;3904;3905;3981;3985;3896;3917;3857;3909,117.0,0.667,1.000,0.651,1, +single,nl_004,natural_language_ko,semantic_search,document,중대재해가 발생했을 때 경영책임자가 처벌받는 기준,3916;3917;3920;3921,3917;3918;3916;3923;3919;3921;3854;3872;3877;3922,113.6,0.750,1.000,0.725,1, +single,nl_005,natural_language_ko,semantic_search,document,안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가,3853;3865,3853;4025;3876;3879;3859;3865;3781;3815;3818;3787,80.6,1.000,1.000,0.832,1, +single,cl_001,crosslingual_ko_en,semantic_search,document,기계 안전 가드 설계 원리,3770;3856,3770;4540;3817;4541;3774;3816;3787;3758;3793;3773,79.0,0.500,1.000,0.613,1, +single,cl_002,crosslingual_ko_en,semantic_search,document,산업 안전 입문서,3755;3775;3776;3777,3760;3755;3774;3764;3758;3775;3779;3802;3814;3817,107.6,0.500,0.500,0.385,1, +single,cl_003,crosslingual_ko_en,semantic_search,document,전기 안전 위험,3772;3790,3897;3772;3771;4018;3773;3790;3819;4020;3807;3755,125.6,1.000,0.500,0.605,1, +single,news_001,news_ko,semantic_search,news,이란과 미국의 군사 충돌,4303;4304;4307;4316;4322;4323;4327;4335,4317;4321;4771;4446;4743;4452;4307;4418;4331;4744,94.5,0.125,0.143,0.084,1, +single,news_002,news_ko,semantic_search,news,호르무즈 해협 봉쇄,4316;4320;4322;4327,4327;4346;4349;4762;4767;4759;4322;4320;4340;4304,94.3,0.750,1.000,0.644,0, +single,news_003,news_en,semantic_search,news,Trump Iran ultimatum,4258;4260;4262,4776;4515;4519;4658;4644;4763;4333;4762;4679;4321,118.7,0.000,0.000,0.000,1, +single,news_004,news_fr,semantic_search,news,guerre en Iran,4199;4202;4210;4361;4363;4507;4519;4521,4678;4507;4199;4688;4776;4363;4519;4668;4670;4672,119.6,0.500,0.500,0.460,1, +single,news_005,news_crosslingual,semantic_search,news,이란 미국 전쟁 글로벌 반응,4202;4258;4262;4536;4303;4304;4316,4262;4457;4765;4324;4345;4329;4452;4443;4761;4642,95.6,0.143,1.000,0.275,1, +single,misc_001,other_domain,fact_lookup,document,강체의 평면 운동학,4063;4065,4063;4065;4064;4067;4071;4068;4069;4062;4060;4066,172.3,1.000,1.000,1.000,1, +single,misc_002,other_domain,semantic_search,document,질점의 운동역학,4060;4061;4062,4062;4060;4070;4064;4068;4067;4065;4058;4071;4066,261.8,0.667,1.000,0.765,1, +single,fail_001,failure_expected,semantic_search,document,Rust async runtime tokio scheduler 내부 구조,,4815;4069;4546;4062;4547;3801;3787;3812;4542;3770,118.6,0.000,0.000,0.000,1, +single,fail_002,failure_expected,semantic_search,document,양자컴퓨터 큐비트 디코히어런스,,4058;4057;4067;3800;4065;4068;3817;4063;4064;3915,76.9,0.000,0.000,0.000,1, +single,fail_003,failure_expected,semantic_search,news,재즈 보컬리스트 빌리 홀리데이,,4634;4100;4815;4116;4281;4697;4205;4077;4235;4758,73.6,0.000,0.000,0.000,1, diff --git a/reports/phase2_final_candidate.csv b/reports/phase2_final_candidate.csv new file mode 100644 index 0000000..65f28cb --- /dev/null +++ b/reports/phase2_final_candidate.csv @@ -0,0 +1,24 @@ +label,id,category,intent,domain_hint,query,relevant_ids,returned_ids_top10,latency_ms,recall_at_10,mrr_at_10,ndcg_at_10,top3_hit,error +single,kw_001,exact_keyword,fact_lookup,document,산업안전보건법 제6장,3856;3868;3879,3856;3851;3862;3853;3861;3868;3879;3873;3876;3871,67.9,1.000,1.000,0.793,1, +single,kw_002,exact_keyword,fact_lookup,document,중대재해 처벌 등에 관한 법률 제2장 중대산업재해,3917;3921,3921;3917;3919;3923;3916;3874;3918;3854;3922;3920,110.0,1.000,1.000,1.000,1, +single,kw_003,exact_keyword,fact_lookup,document,화학물질관리법 유해화학물질 영업자,3981,3981;3985;3980;3984;3993;3857;3978;3986;3983;3957,119.3,1.000,1.000,1.000,1, +single,kw_004,exact_keyword,fact_lookup,document,근로기준법 안전과 보건,4041,4041;3852;3851;3877;3905;3903;3858;3881;3781;3912,108.7,1.000,1.000,1.000,1, +single,kw_005,exact_keyword,fact_lookup,document,산업안전보건기준에 관한 규칙 보호구,3888,3888;3912;3911;3905;3909;3889;3910;3897;3890;3896,125.5,1.000,1.000,1.000,1, +single,nl_001,natural_language_ko,semantic_search,document,기계로 인한 산업재해 관련 법령,3856;3868;3879;3854,3878;3897;3863;3868;3856;3879;3895;3867;3851;3854,83.9,1.000,0.250,0.571,0, +single,nl_002,natural_language_ko,semantic_search,document,사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일,3855;3867;3878,3855;3917;3854;3867;3878;3863;3851;3908;3903;3895,118.0,1.000,1.000,0.853,1, +single,nl_003,natural_language_ko,semantic_search,document,유해화학물질을 다루는 회사가 지켜야 할 안전 의무,3980;3981;3982,3980;3903;3904;3905;3981;3985;3896;3917;3857;3909,82.8,0.667,1.000,0.651,1, +single,nl_004,natural_language_ko,semantic_search,document,중대재해가 발생했을 때 경영책임자가 처벌받는 기준,3916;3917;3920;3921,3917;3918;3916;3923;3919;3921;3854;3872;3877;3922,72.1,0.750,1.000,0.725,1, +single,nl_005,natural_language_ko,semantic_search,document,안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가,3853;3865,3853;4025;3876;3879;3859;3865;3781;3815;3818;3787,80.2,1.000,1.000,0.832,1, +single,cl_001,crosslingual_ko_en,semantic_search,document,기계 안전 가드 설계 원리,3770;3856,3770;4540;3817;4541;3774;3816;3787;3758;3793;3773,108.3,0.500,1.000,0.613,1, +single,cl_002,crosslingual_ko_en,semantic_search,document,산업 안전 입문서,3755;3775;3776;3777,3760;3755;3774;3764;3758;3775;3779;3802;3814;3817,79.5,0.500,0.500,0.385,1, +single,cl_003,crosslingual_ko_en,semantic_search,document,전기 안전 위험,3772;3790,3897;3772;3771;4018;3773;3790;3819;4020;3807;3755,103.7,1.000,0.500,0.605,1, +single,news_001,news_ko,semantic_search,news,이란과 미국의 군사 충돌,4303;4304;4307;4316;4322;4323;4327;4335,4317;4321;4771;4743;4307;4452;4761;4678;4418;4331,1445.8,0.125,0.200,0.098,1, +single,news_002,news_ko,semantic_search,news,호르무즈 해협 봉쇄,4316;4320;4322;4327,4327;4346;4349;4762;4767;4759;4322;4320;4340;4304,185.3,0.750,1.000,0.644,0, +single,news_003,news_en,semantic_search,news,Trump Iran ultimatum,4258;4260;4262,4776;4515;4519;4658;4644;4763;4333;4762;4679;4321,76.3,0.000,0.000,0.000,1, +single,news_004,news_fr,semantic_search,news,guerre en Iran,4199;4202;4210;4361;4363;4507;4519;4521,4678;4507;4199;4688;4776;4363;4519;4668;4670;4672,157.9,0.500,0.500,0.460,1, +single,news_005,news_crosslingual,semantic_search,news,이란 미국 전쟁 글로벌 반응,4202;4258;4262;4536;4303;4304;4316,4262;4457;4765;4324;4345;4329;4258;4452;4443;4761,186.7,0.286,1.000,0.367,1, +single,misc_001,other_domain,fact_lookup,document,강체의 평면 운동학,4063;4065,4063;4065;4064;4067;4071;4068;4069;4062;4060;4066,171.6,1.000,1.000,1.000,1, +single,misc_002,other_domain,semantic_search,document,질점의 운동역학,4060;4061;4062,4062;4060;4070;4064;4068;4067;4065;4058;4071;4066,263.6,0.667,1.000,0.765,1, +single,fail_001,failure_expected,semantic_search,document,Rust async runtime tokio scheduler 내부 구조,,4815;4069;4546;4062;4547;3801;3787;3812;4542;3770,121.9,0.000,0.000,0.000,1, +single,fail_002,failure_expected,semantic_search,document,양자컴퓨터 큐비트 디코히어런스,,4058;4057;4067;3800;4065;4068;3817;4063;4064;3915,75.2,0.000,0.000,0.000,1, +single,fail_003,failure_expected,semantic_search,news,재즈 보컬리스트 빌리 홀리데이,,4634;4100;4815;4116;4281;4697;4205;4077;4235;4289,73.9,0.000,0.000,0.000,1,