hyungi/hyungi_document_server
1
1
Fork 0
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity

feat(eval): v0.2 graded relevance schema + harness #24

Merged
hyungi merged 1 commits from feat/eval-v0-2-graded-relevance into main 2026-05-23 13:03:13 +09:00
Conversation 0 Commits 1 Files Changed 3
+742 -121
hyungi commented 2026-05-23 13:03:07 +09:00
Owner
Copy Link
Copy Source

queries.yaml v0.1 23 case → v0.2 schema swap:

  • 7 카테고리 (standards / korean_only / english_only / mixed / exam /
    ocr_derived / failure_expected)
  • language / ocr_derived / failure_expected / graded_relevance 컬럼 추가
  • v0.1 호환 보존 (legacy_category + relevant_ids + top3_ids)
  • 신규 28 case (50+ 목표) 는 후속 PR-Eval-V0_2-Baseline-Analysis

run_eval.py 확장:

  • graded_ndcg_at_k / graded_recall_at_k 함수 추가
  • Query / QueryResult dataclass 확장 (v0.2 컬럼)
  • load_queries v0.1 fallback (top3 → grade 3, 나머지 → grade 2)
  • --eval-version v0.1/v0.2/both flag (default both)
  • print_summary 의 by_language / by_ocr_derived 집계 추가
  • write_csv 의 graded 컬럼 추가

README.md 신규:

  • graded 등급 정의 (0~3) + 카테고리 정의 (7개)
  • v0.2 schema 컬럼 + 신규 case 작성 가이드
  • v0.1 호환성 + CLI 사용 예 + baseline 박제 정책

Phase 1 plan: ~/.claude/plans/phase-1-graded-eval-v0-2.md
Parent: ~/.claude/plans/peppy-hugging-nest.md § Phase 1

본 PR closure: schema + harness + README. 신규 28 case + baseline 박제 +
약점 분석 (embedding-sensitive failure pattern 4 카테고리 식별) 은 후속 PR.

Co-Authored-By: Claude Opus 4.7 (1M context) noreply@anthropic.com

queries.yaml v0.1 23 case → v0.2 schema swap: - 7 카테고리 (standards / korean_only / english_only / mixed / exam / ocr_derived / failure_expected) - language / ocr_derived / failure_expected / graded_relevance 컬럼 추가 - v0.1 호환 보존 (legacy_category + relevant_ids + top3_ids) - 신규 28 case (50+ 목표) 는 후속 PR-Eval-V0_2-Baseline-Analysis run_eval.py 확장: - graded_ndcg_at_k / graded_recall_at_k 함수 추가 - Query / QueryResult dataclass 확장 (v0.2 컬럼) - load_queries v0.1 fallback (top3 → grade 3, 나머지 → grade 2) - --eval-version v0.1/v0.2/both flag (default both) - print_summary 의 by_language / by_ocr_derived 집계 추가 - write_csv 의 graded 컬럼 추가 README.md 신규: - graded 등급 정의 (0~3) + 카테고리 정의 (7개) - v0.2 schema 컬럼 + 신규 case 작성 가이드 - v0.1 호환성 + CLI 사용 예 + baseline 박제 정책 Phase 1 plan: ~/.claude/plans/phase-1-graded-eval-v0-2.md Parent: ~/.claude/plans/peppy-hugging-nest.md § Phase 1 본 PR closure: schema + harness + README. 신규 28 case + baseline 박제 + 약점 분석 (embedding-sensitive failure pattern 4 카테고리 식별) 은 후속 PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hyungi added 1 commit 2026-05-23 13:03:08 +09:00
feat(eval): v0.2 graded relevance schema + harness 725a4e1f1d 
queries.yaml v0.1 23 case → v0.2 schema swap:
- 7 카테고리 (standards / korean_only / english_only / mixed / exam /
  ocr_derived / failure_expected)
- language / ocr_derived / failure_expected / graded_relevance 컬럼 추가
- v0.1 호환 보존 (legacy_category + relevant_ids + top3_ids)
- 신규 28 case (50+ 목표) 는 후속 PR-Eval-V0_2-Baseline-Analysis

run_eval.py 확장:
- graded_ndcg_at_k / graded_recall_at_k 함수 추가
- Query / QueryResult dataclass 확장 (v0.2 컬럼)
- load_queries v0.1 fallback (top3 → grade 3, 나머지 → grade 2)
- --eval-version v0.1/v0.2/both flag (default both)
- print_summary 의 by_language / by_ocr_derived 집계 추가
- write_csv 의 graded 컬럼 추가

README.md 신규:
- graded 등급 정의 (0~3) + 카테고리 정의 (7개)
- v0.2 schema 컬럼 + 신규 case 작성 가이드
- v0.1 호환성 + CLI 사용 예 + baseline 박제 정책

Phase 1 plan: ~/.claude/plans/phase-1-graded-eval-v0-2.md
Parent: ~/.claude/plans/peppy-hugging-nest.md § Phase 1

본 PR closure: schema + harness + README. 신규 28 case + baseline 박제 +
약점 분석 (embedding-sensitive failure pattern 4 카테고리 식별) 은 후속 PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hyungi merged commit 3f6314494e into main 2026-05-23 13:03:13 +09:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
hyungi
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: hyungi/hyungi_document_server#24
Delete Branch "feat/eval-v0-2-graded-relevance"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?