hyungi b00d9f5e15 docs(eval): Phase 2Q Category-Analysis — standards/exam 회귀 진단 (inflation 정정) ...

Apply rollout 후속 read-only 진단. Phase 3 측정 (commit a41adb6) 의 NDCG 0.927 + standards 1.441 + exam 1.109 = **측정 artifact (top-N doc 중복 박제 → graded NDCG inflation)**.

진단 path:
- script category_analysis_phase2q.py (csv parse + queries.yaml graded lookup + standards/exam 18 case 3-way top-5 박제)
- 회귀 큰 case top: kw_004/kw_009/kw_010 = Phase 3 inflation 1.631 → Rerank-Fix 정상 1.000 (baseline 동일, 회귀 0)
- kw_001/exam_004 = Rerank-Fix 가 baseline 대비도 회귀 (reranker chunk-level relevance 우선 → doc grade 3 가 rank 5 밀림)

정정값 박제:
- Phase 3 NDCG 0.927 → **Rerank-Fix 0.876 (정확값)**
- Δ vs baseline: +0.268 (inflated) → **+0.217 (실제 multi-query 효과)**
- standards 1.441 → 1.157 (vs baseline 0.873, +0.284)
- exam 1.109 → 0.918 (vs baseline 0.738, +0.180)

결론:
- **Apply rollout 결정 = 정정값 기준 invariant 유지** — +0.217 vs baseline = 유의미 net 개선
- standards -0.28 / exam -0.19 회귀 = false alarm (inflation 정정)
- 실제 회귀 case (kw_001/exam_004) = Apply 후 telemetry 박제 항목

산출물:
- tests/search_eval/baselines/v0_2_phase2q_category_analysis_2026-05-24.md (180+ lines, §1~8)
- tests/search_eval/scripts/category_analysis_phase2q.py (read-only csv parse script, reproducibility)

신규 feedback memory: graded-ndcg-dedup-invariant (NDCG > 1.0 = inflation 의심 invariant + dedup audit 필수)

후속 별 chore 후보:
- PR-Eval-GradedNDCG-Dedup — run_eval.py 의 graded NDCG 계산 dedup + NDCG > 1.0 warning
- PR-2Q-Search-Result-Dedup — _rrf_fuse_variants 의 representative doc_id 중복 audit

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-24 04:23:58 +00:00

..

api

feat(search): /ask/react endpoint with Qwen native tool calling ReAct loop

2026-05-22 13:43:47 +00:00

calibrate_fixtures

feat(scripts): Phase 3.5 — calibrate_ask.py CLI (Q0~Q8 + render + FP CSV)

2026-04-17 08:11:06 +09:00

fixtures

feat(eval): Phase 2Q Diagnose Phase 1A — fixture (4 카테고리 × 2 LLM) + prompt v1

2026-05-23 22:09:29 +00:00

load/fixtures

ops(gpu-health): GPU 서비스 health/smoke 표준화 + synthetic VRAM 피크 가드

2026-05-14 09:42:07 +09:00

phase3_5_eval

feat(ask): Phase 3.5a guardrails (classifier + refusal gate + grounding + partial)

2026-04-10 08:49:11 +09:00

policy

fix(policy): use container-compatible imports (drop app. prefix)

2026-04-24 09:42:24 +09:00

scripts

feat(chunk): Phase 1.2-E reindex 스크립트 추가

2026-04-08 12:31:29 +09:00

search_eval

docs(eval): Phase 2Q Category-Analysis — standards/exam 회귀 진단 (inflation 정정)

2026-05-24 04:23:58 +00:00

services

feat(search): /ask/react endpoint with Qwen native tool calling ReAct loop

2026-05-22 13:43:47 +00:00

__init__.py

feat: scaffold v2 project structure with Docker, FastAPI, and config

2026-04-02 10:20:15 +09:00

_worker_pool_helpers.py

refactor(worker-pool): Registry-1B test fixture — NullPool helper standalone

2026-05-19 12:43:53 +09:00

conftest.py

feat: scaffold v2 project structure with Docker, FastAPI, and config

2026-04-02 10:20:15 +09:00

test_ask_eval_auth.py

feat(ask): Phase 3.5 A0 — ask_events source/eval_case_id + eval auth boundary

2026-04-17 08:11:06 +09:00

test_briefing_historical.py

fix(briefing): backfill country_perspectives[].article_ids from cluster members

2026-05-12 13:15:26 +09:00

test_explanation_cap.py

fix(tests): explanation cap test setup — 한글 chunk 길이 부족 보정

2026-05-02 08:35:34 +09:00

test_freshness_decay.py

feat(search): PR-RAG-Time-1 freshness decay (news/law_monitor)

2026-05-03 08:38:09 +09:00

test_grounding_fabricated_number.py

feat(grounding): Phase 3.5 B1 — unit-aware fabricated_number + bound semantics

2026-04-17 08:11:06 +09:00

test_internal_worker_authz.py

refactor(worker-pool): Registry-1B test fixture — NullPool helper standalone

2026-05-19 12:43:53 +09:00

test_internal_worker_endpoints.py

refactor(worker-pool): Registry-1B test fixture — NullPool helper standalone

2026-05-19 12:43:53 +09:00

test_laptop_worker_bot_auth.py

feat(worker-pool): Registry-1B Pull 활성화 (auth + worker_jobs + 5 endpoint)

2026-05-19 08:54:07 +09:00

test_marker_image_persist.py

feat(markdown): persist extracted images with auth routes

2026-05-10 14:05:41 +09:00

test_priority_gate.py

feat(search): MLX priority gate (B-1, Priority.FOREGROUND vs BACKGROUND)

2026-05-17 08:42:58 +09:00

test_query_rewriter.py

fix(search): Phase 2Q rerank payload — chunk_id dedup + cap 60 + TEI batch 64 (Apply prereq)

2026-05-24 03:54:59 +00:00

test_session_summary_guard_pattern.py

feat(study): Phase 4-D 운영 관찰 + confidence calibration

2026-05-02 07:33:57 +09:00

test_synthesis_failure_regate.py

fix(search): re-gate Tier 0 — synthesis self-refuse / timeout / empty answer 일관 처리

2026-04-17 08:29:49 +09:00

test_verifier_numeric_promote.py

feat(verifier): Phase 3.5 B2 — numeric_conflict promote (env flag) + Tier 4

2026-04-17 08:11:06 +09:00

test_worker_jobs_ownership.py

refactor(worker-pool): Registry-1B test fixture — NullPool helper standalone

2026-05-19 12:43:53 +09:00

test_worker_jobs_retry.py

refactor(worker-pool): Registry-1B test fixture — NullPool helper standalone

2026-05-19 12:43:53 +09:00

test_worker_jobs_skip_locked.py

refactor(worker-pool): Registry-1B test fixture — NullPool helper standalone

2026-05-19 12:43:53 +09:00

test_worker_jobs_smoke.py

refactor(worker-pool): Registry-1B test fixture — NullPool helper standalone

2026-05-19 12:43:53 +09:00

test_worker_recap_compaction.py

feat(worker-pool): Registry-1C cap 1MB + deterministic compaction

2026-05-19 12:55:51 +09:00

test_worker_recap_endpoint.py

feat(worker-pool): Registry-1C cap 1MB + deterministic compaction

2026-05-19 12:55:51 +09:00