hyungi/hyungi_document_server
1
1
Fork 0
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
chore/dead-code-cleanup
hyungi_document_server/tests/search_eval/queries.yaml
T
hyungi 4d14ab69d9 feat(eval): v0.2 28 신규 case + 2026-05-23 baseline + analysis 
PR-1 (725a4e1) v0.2 schema + harness 위에 신규 28 case 추가 → 51 case
완성 + 현재 모델로 baseline 박제 + 약점 카테고리 analysis md.

신규 28 case 분포 (계획 +28 = standards +6 / english_only +8 / mixed +5
/ exam +7 / failure_expected +2 / ocr_derived 0):
- standards 5 → 11 (KGS FP111/FU551 + 산안기준 후반 편 + 고압가스법)
- english_only 1 → 9 (Pressure Vessel Design Manual + ASME VIII/IX +
  Hydrogen ASME + Industrial Safety 영문 교재 + Structural Analysis)
- mixed 5 → 10 (한↔영 ASME / KGS-영문 / 양언어 압력용기)
- exam 0 → 7 (가스기사 study_questions → library 개념 docs 매핑)
- failure_expected 3 → 5 (KGS AC999 / 초전도 안전 관리법)
- ocr_derived 0 (TBD-O FAILED: extract_meta NULL 21385, chunks.source
  = RSS feed 명. OCR 식별 컬럼 부재 → +4 case 재배분, analysis 명시)

baseline 측정 결과 (corpus 21,385, hybrid mode, bge-m3 + bge-reranker-v2-m3):
- v0.1 Recall@10 0.646, MRR 0.724, NDCG 0.606, Top-3 0.891
- v0.2 graded NDCG 0.659, Recall@10 g≥2 0.695, g≥3 0.761
- latency p50 528ms / p95 1,664ms
- failure precision 0/5 (DS confidence threshold 미적용)

약점 top 3 (analysis md):
- mixed crosslingual 0.39 graded NDCG — TOP weakness, bge-m3
  multilingual 한계 추정
- korean_only natural language 0.51 — query rewrite 부재 추정
- failure_expected 0/5 — confidence cutoff 부재

Phase 2 dispatch 권고 (analysis md):
- 2A Embedding bge-m3 — 즉시 진입 (mixed/korean 동시 타격)
- 2B Reranker — M (2A 이후)
- 2C OCR-Marker — 선행 chore (OCR 식별 컬럼 추가) 필요
- 2D STT — 본 평가셋 외 (별 평가셋 필요)

Query rewrite 는 Phase 2Q/Search-PR 로 별도 분리.

영향 받는 파일:
- tests/search_eval/queries.yaml: 23 → 51 case (기존 23 변경 0, append only)
- tests/search_eval/baselines/v0_2_baseline_2026-05-23.json: 신규
- tests/search_eval/baselines/v0_2_baseline_2026-05-23_analysis.md: 신규

PR plan: ~/.claude/plans/pr-2-serialized-hummingbird.md
Phase 1 plan: ~/.claude/plans/phase-1-graded-eval-v0-2.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-23 03:32:55 +00:00

955 lines
32 KiB
YAML
Raw Permalink Blame History

# Document Server 검색 평가셋 v0.2 — graded relevance baseline
#
# Phase 1 산출물 (peppy-hugging-nest.md Phase 1) — Phase 2 모델 점검과 Phase 3
# Markdown 전환의 효과를 객관 측정할 도구 마련. graded relevance 1년 미작성
# (document-server-2026-05-21.md:24) gap 해소.
#
# v0.2 schema 변경:
# - category : 사용자 권장 7 카테고리
# (standards / korean_only / english_only / mixed /
# exam / ocr_derived / failure_expected)
# - legacy_category : v0.1 카테고리 보존 (호환)
# - language : ko / en / mixed
# - ocr_derived : marker/OCR pipeline 통한 chunk 포함 여부
# - graded_relevance : {doc_id: grade}
# (0=irrelevant / 1=marginal / 2=relevant / 3=highly)
# - failure_expected : 결과 0건 expected (별도 flag)
#
# v0.1 호환:
# - relevant_ids / top3_ids 는 deprecated 가 아니라 호환용 (--mode v0.1 재현)
# - run_eval.py --mode v0.1 에서 옛 점수 ±0.001 재현 가능
#
# graded 등급 기준:
# - 3 : highly relevant (top-3 강제, direct match — 본법/본 chapter/본 article)
# - 2 : relevant (정답 후보, 보조 정답, top-10 within OK)
# - 1 : marginal (약한 관련, top-10 within = 가점, 밖 = 무감점)
# - 0 : irrelevant (검색 결과 잡히면 감점 후보, 현재는 표시만)
version: "0.2"
created_at: "2026-05-23"
corpus_size: 753 # v0.1 기준. PR-Eval-V0_2-Baseline-Analysis 에서 갱신
notes: |
Phase 1 plan: ~/.claude/plans/phase-1-graded-eval-v0-2.md
본 PR (PR-Eval-V0_2-Schema-Harness) = v0.1 23 case → v0.2 23 case schema swap
+ run_eval graded 함수 + README. 신규 28 case + baseline 박제는 후속 PR.
queries:
# ─────────────────────────────────────────────────────────
# 1. standards (법령/규칙/기준) — v0.1 exact_keyword 흡수
# ─────────────────────────────────────────────────────────
- id: kw_001
query: "산업안전보건법 제6장"
category: standards
legacy_category: exact_keyword
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
3856: 3
3868: 2
3879: 2
relevant_ids: [3856, 3868, 3879]
top3_ids: [3856]
notes: |
"유해·위험 기계 등에 대한 조치"가 들어있는 6장.
Act(3856) = grade 3 (본법, top-3 강제).
Decree(3868)/Rule(3879) = grade 2 (보조 정답, 본법 우선).
- id: kw_002
query: "중대재해 처벌 등에 관한 법률 제2장 중대산업재해"
category: standards
legacy_category: exact_keyword
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
3917: 3
3921: 2
relevant_ids: [3917, 3921]
top3_ids: [3917]
notes: |
본법 제2장(3917) = grade 3.
시행령 동일 장(3921) = grade 2.
- id: kw_003
query: "화학물질관리법 유해화학물질 영업자"
category: standards
legacy_category: exact_keyword
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
3981: 3
relevant_ids: [3981]
top3_ids: [3981]
notes: 화학물질관리법 제4장 = 유해화학물질 영업자 (3981, grade 3).
- id: kw_004
query: "근로기준법 안전과 보건"
category: standards
legacy_category: exact_keyword
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
4041: 3
relevant_ids: [4041]
top3_ids: [4041]
notes: 근로기준법 제6장 = 안전과 보건 (4041, grade 3).
- id: kw_005
query: "산업안전보건기준에 관한 규칙 보호구"
category: standards
legacy_category: exact_keyword
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
3888: 3
relevant_ids: [3888]
top3_ids: [3888]
notes: 산업안전보건기준 규칙 제4장 = 보호구 (3888, grade 3).
# ─────────────────────────────────────────────────────────
# 2. korean_only — v0.1 natural_language_ko + news_ko + other_domain 흡수
# ─────────────────────────────────────────────────────────
- id: nl_001
query: "기계로 인한 산업재해 관련 법령"
category: korean_only
legacy_category: natural_language_ko
intent: semantic_search
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
3856: 3
3868: 2
3879: 2
3854: 1
relevant_ids: [3856, 3868, 3879, 3854]
top3_ids: [3856]
notes: |
플랜의 대표 예시 쿼리. 기계 안전 = 산안법 6장(3856, grade 3).
Decree(3868)/Rule(3879) = grade 2.
4장 유해·위험 방지 조치(3854) = grade 1 (의미상 약한 관련).
- id: nl_002
query: "사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일"
category: korean_only
legacy_category: natural_language_ko
intent: semantic_search
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
3855: 3
3867: 2
3878: 2
relevant_ids: [3855, 3867, 3878]
top3_ids: [3855]
notes: 산안법 제5장 도급 시 산업재해 예방. Act(3855) = grade 3, Decree(3867)/Rule(3878) = grade 2.
- id: nl_003
query: "유해화학물질을 다루는 회사가 지켜야 할 안전 의무"
category: korean_only
legacy_category: natural_language_ko
intent: semantic_search
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
3980: 2
3981: 2
3982: 2
relevant_ids: [3980, 3981, 3982]
notes: |
화관법 제3-5장(유해화학물질 관리/영업자/사고 대응). 셋 다 동등하게 정답 후보.
특정 chapter 강제 없음 → grade 2 셋 (top3_ids 없음).
- id: nl_004
query: "중대재해가 발생했을 때 경영책임자가 처벌받는 기준"
category: korean_only
legacy_category: natural_language_ko
intent: semantic_search
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
3917: 3
3916: 2
3920: 2
3921: 2
relevant_ids: [3916, 3917, 3920, 3921]
top3_ids: [3917]
notes: 중대재해처벌법 본법+시행령 제1-2장. 제2장(3917) = grade 3, 나머지 = grade 2.
- id: nl_005
query: "안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가"
category: korean_only
legacy_category: natural_language_ko
intent: semantic_search
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
3853: 3
3865: 2
relevant_ids: [3853, 3865]
top3_ids: [3853]
notes: 산안법 제3장 안전보건교육 (3853, grade 3) + 시행령 제3장 (3865, grade 2).
- id: news_001
query: "이란과 미국의 군사 충돌"
category: korean_only
legacy_category: news_ko
intent: semantic_search
domain_hint: news
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
4303: 2
4304: 2
4307: 2
4316: 2
4322: 2
4323: 2
4327: 2
4335: 2
relevant_ids: [4303, 4304, 4307, 4316, 4322, 4323, 4327, 4335]
notes: |
경향신문의 이란-미국 전쟁 보도. recall 위주 평가, 동등 정답 8건 (grade 2).
diversity 제약 적용 후에도 최소 5건은 top-10에 들어와야 함.
- id: news_002
query: "호르무즈 해협 봉쇄"
category: korean_only
legacy_category: news_ko
intent: semantic_search
domain_hint: news
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
4316: 3
4320: 2
4322: 2
4327: 2
relevant_ids: [4316, 4320, 4322, 4327]
top3_ids: [4316]
notes: 호르무즈 해협 직접 언급 기사. 4316 = grade 3 (direct), 나머지 = grade 2.
- id: misc_001
query: "강체의 평면 운동학"
category: korean_only
legacy_category: other_domain
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
4063: 3
4065: 2
relevant_ids: [4063, 4065]
top3_ids: [4063]
notes: 공업역학 동역학 ch16(4063, grade 3), ch18(4065, grade 2).
- id: misc_002
query: "질점의 운동역학"
category: korean_only
legacy_category: other_domain
intent: semantic_search
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
4060: 2
4061: 2
4062: 2
relevant_ids: [4060, 4061, 4062]
notes: 공업역학 동역학 ch13~15 (질점 운동역학). 동등 정답 3건 (grade 2).
# ─────────────────────────────────────────────────────────
# 3. english_only — v0.1 news_en 흡수
# ─────────────────────────────────────────────────────────
- id: news_003
query: "Trump Iran ultimatum"
category: english_only
legacy_category: news_en
intent: semantic_search
domain_hint: news
language: en
ocr_derived: false
failure_expected: false
graded_relevance:
4258: 2
4260: 2
4262: 2
relevant_ids: [4258, 4260, 4262]
notes: Der Spiegel 영어판 Iran 관련 기사. 동등 정답 3건 (grade 2).
# ─────────────────────────────────────────────────────────
# 4. mixed — v0.1 crosslingual_ko_en + news_fr + news_crosslingual 흡수
# ─────────────────────────────────────────────────────────
- id: cl_001
query: "기계 안전 가드 설계 원리"
category: mixed
legacy_category: crosslingual_ko_en
intent: semantic_search
domain_hint: document
language: mixed
ocr_derived: false
failure_expected: false
graded_relevance:
3770: 3
3856: 2
relevant_ids: [3770, 3856]
top3_ids: [3770]
notes: |
Industrial Safety and Health Management(7-ED) Ch15 Machine Guarding(3770, grade 3)이
한국어 쿼리로 검색되어야 함. 한국 산안법 6장(3856) = grade 2.
- id: cl_002
query: "산업 안전 입문서"
category: mixed
legacy_category: crosslingual_ko_en
intent: semantic_search
domain_hint: document
language: mixed
ocr_derived: false
failure_expected: false
graded_relevance:
3755: 2
3775: 2
3776: 2
3777: 2
relevant_ids: [3755, 3775, 3776, 3777]
notes: |
Safety and Health for Engineers / Industrial Safety and Health Management
영문 교재 입문 챕터들이 한국어 쿼리로 검색되어야 함. 동등 정답 4건 (grade 2).
- id: cl_003
query: "전기 안전 위험"
category: mixed
legacy_category: crosslingual_ko_en
intent: semantic_search
domain_hint: document
language: mixed
ocr_derived: false
failure_expected: false
graded_relevance:
3772: 2
3790: 2
relevant_ids: [3772, 3790]
notes: |
Electrical Hazards(3772), Electrical Safety(3790) 영문 챕터.
한국어 안전기준 규칙 중 전기 관련 장도 있을 수 있음(보수적으로 영문만 정답).
- id: news_004
query: "guerre en Iran"
category: mixed
legacy_category: news_fr
intent: semantic_search
domain_hint: news
language: mixed # 정확히는 fr 이지만 v0.2 enum (ko/en/mixed) 에서 mixed 매핑
ocr_derived: false
failure_expected: false
graded_relevance:
4199: 2
4202: 2
4210: 2
4361: 2
4363: 2
4507: 2
4519: 2
4521: 2
relevant_ids: [4199, 4202, 4210, 4361, 4363, 4507, 4519, 4521]
notes: |
Le Monde 불어 Iran 전쟁 보도. 동등 정답 8건 (grade 2).
불어 query 라 v0.2 의 ko/en/mixed enum 에서는 mixed 매핑 (정확히는 fr).
후속 PR 에서 language enum 확장 검토.
- id: news_005
query: "이란 미국 전쟁 글로벌 반응"
category: mixed
legacy_category: news_crosslingual
intent: semantic_search
domain_hint: news
language: mixed
ocr_derived: false
failure_expected: false
graded_relevance:
4202: 2
4258: 2
4262: 2
4536: 2
4303: 2
4304: 2
4316: 2
relevant_ids: [4202, 4258, 4262, 4536, 4303, 4304, 4316]
notes: |
한국어 쿼리로 한/영/불/독 뉴스가 골고루 검색되어야 함. 동등 정답 7건 (grade 2).
Phase 1 domain-aware retrieval + multilingual embedding 효과 측정용.
diversity 제약(국가당 max 2)이 동작하면 최소 4개국 이상 노출.
# ─────────────────────────────────────────────────────────
# 5. failure_expected — 결과 0건 expected
# ─────────────────────────────────────────────────────────
- id: fail_001
query: "Rust async runtime tokio scheduler 내부 구조"
category: failure_expected
legacy_category: failure_expected
intent: semantic_search
domain_hint: document
language: mixed # 한국어 + 영문 keyword 혼합
ocr_derived: false
failure_expected: true
graded_relevance: {}
relevant_ids: []
notes: |
코퍼스에 Rust/프로그래밍 문서 없음. 결과 0건 = correct.
Phase 0.3 search_failure_logs로 자동 수집되어야 함.
Phase 1+에서 confidence 점수 < 0.5로 분류되는지 확인.
- id: fail_002
query: "양자컴퓨터 큐비트 디코히어런스"
category: failure_expected
legacy_category: failure_expected
intent: semantic_search
domain_hint: document
language: ko
ocr_derived: false
failure_expected: true
graded_relevance: {}
relevant_ids: []
notes: 코퍼스에 양자물리 문서 없음. 결과 0건 = correct.
- id: fail_003
query: "재즈 보컬리스트 빌리 홀리데이"
category: failure_expected
legacy_category: failure_expected
intent: semantic_search
domain_hint: news
language: ko
ocr_derived: false
failure_expected: true
graded_relevance: {}
relevant_ids: []
notes: 코퍼스에 음악/재즈 문서 없음. 결과 0건 = correct.
# ═════════════════════════════════════════════════════════
# PR-Eval-V0_2-Baseline-Analysis 신규 28 case (2026-05-23)
# 분배 (계획 plan 의 +28):
# standards +6 / english_only +8 / mixed +5 / exam +7 / failure_expected +2
# ocr_derived 0 — TBD-O FAILED (chunks.source = RSS feed 명,
# extract_meta / md_extraction_engine 전부 NULL → OCR 식별 불가).
# 재배분: standards +1 / english_only +2 / mixed +1 → analysis md 명시.
# ═════════════════════════════════════════════════════════
# ─────────────────────────────────────────────────────────
# 1b. standards 신규 6 (한국 KGS code + 산안기준 후반 편 + 고압가스법)
# ─────────────────────────────────────────────────────────
- id: kw_006
query: "산업안전보건기준에 관한 규칙 작업장 통로"
category: standards
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
3886: 3
3887: 2
relevant_ids: [3886, 3887]
top3_ids: [3886]
notes: 산안기준 규칙 1편 제2장 작업장(3886, grade 3) + 제3장 통로(3887, grade 2).
- id: kw_007
query: "산업안전보건기준 폭발 화재 위험물 누출 방지"
category: standards
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
3896: 3
3766: 1
relevant_ids: [3896, 3766]
top3_ids: [3896]
notes: |
산안기준 규칙 2편 제2장 폭발ㆍ화재(3896) = grade 3.
Industrial Safety and Health Management ch11 Flammable and Explosive(3766)
= grade 1 (영문 도서, 주제 일치 but 한국 법령 query → cross-lingual marginal).
- id: kw_008
query: "고압가스 안전관리법 전문"
category: standards
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
4025: 3
4026: 2
relevant_ids: [4025, 4026]
top3_ids: [4025]
notes: 고압가스법 본법(4025, grade 3) + 시행령(4026, grade 2). "전문" 키워드 매칭.
- id: kw_009
query: "KGS FP111 가스설비 배관설비 기준"
category: standards
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
13305: 3
relevant_ids: [13305]
top3_ids: [13305]
notes: KGS FP111 § 2.4~2.5 가스설비·배관설비(13305) = grade 3. KGS code exact lookup.
- id: kw_010
query: "KGS FU551 가스설비 압력조정기 가스계량기"
category: standards
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
13652: 3
relevant_ids: [13652]
top3_ids: [13652]
notes: KGS FU551 § 2.4 가스설비 기준(13652) = grade 3. KGS code exact lookup.
- id: kw_011
query: "산업안전보건기준 전기로 인한 위험 방지"
category: standards
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
3897: 3
3772: 1
relevant_ids: [3897, 3772]
top3_ids: [3897]
notes: |
산안기준 규칙 2편 제3장 전기 위험 방지(3897) = grade 3.
Industrial Safety and Health Management ch17 Electrical Hazards(3772) = grade 1
(영문, 주제 일치).
# ─────────────────────────────────────────────────────────
# 3b. english_only 신규 8 (Pressure Vessel Design Manual + ASME VIII/IX
# + Hydrogen ASME + Industrial Safety 영문 교재 + Structural Analysis)
# ─────────────────────────────────────────────────────────
- id: en_001
query: "pressure vessel flange design"
category: english_only
intent: semantic_search
domain_hint: document
language: en
ocr_derived: false
failure_expected: false
graded_relevance:
5144: 3
5136: 2
relevant_ids: [5144, 5136]
top3_ids: [5144]
notes: |
Pressure Vessel Design Manual ch03 Flange Design(5144) = grade 3.
Manual_00 contents(5136) = grade 2 (catalog).
- id: en_002
query: "ASME Section VIII introduction"
category: english_only
intent: semantic_search
domain_hint: document
language: en
ocr_derived: false
failure_expected: false
graded_relevance:
5204: 3
5206: 2
relevant_ids: [5204, 5206]
top3_ids: [5204]
notes: ASME VIII Introduction(5204) = grade 3, Design_Basic_R1(5206) = grade 2.
- id: en_003
query: "impact test requirements ASME"
category: english_only
intent: semantic_search
domain_hint: document
language: en
ocr_derived: false
failure_expected: false
graded_relevance:
5205: 3
5148: 1
relevant_ids: [5205, 5148]
top3_ids: [5205]
notes: |
ASME VIII Impact Test(5205) = grade 3. Structural Analysis Materials of
Construction(5148) = grade 1 (재료 관련 약한 연관).
- id: en_004
query: "design of vessel supports"
category: english_only
intent: semantic_search
domain_hint: document
language: en
ocr_derived: false
failure_expected: false
graded_relevance:
5149: 3
relevant_ids: [5149]
top3_ids: [5149]
notes: Pressure Vessel Design Manual ch04 Design of Vessel Supports(5149) = grade 3.
- id: en_005
query: "hydrogen piping ASME code"
category: english_only
intent: semantic_search
domain_hint: document
language: en
ocr_derived: false
failure_expected: false
graded_relevance:
5178: 3
relevant_ids: [5178]
top3_ids: [5178]
notes: Hydrogen_Piping_and_Pipelines_ASME_Code(5178) = grade 3. exact title match.
- id: en_006
query: "ASME welding qualification requirements"
category: english_only
intent: semantic_search
domain_hint: document
language: en
ocr_derived: false
failure_expected: false
graded_relevance:
5209: 3
3771: 1
relevant_ids: [5209, 3771]
top3_ids: [5209]
notes: |
ASME IX Welding(5209) = grade 3.
Industrial Safety and Health Management ch16 Welding(3771) = grade 1
(안전 관점 영문 도서, qualification 주제 약한 연관).
- id: en_007
query: "pressure vessel fabrication and inspection"
category: english_only
intent: semantic_search
domain_hint: document
language: en
ocr_derived: false
failure_expected: false
graded_relevance:
5208: 3
5145: 2
relevant_ids: [5208, 5145]
top3_ids: [5208]
notes: |
ASME VIII Fabrication-Inspection_R1(5208) = grade 3.
Structural Analysis ch02 Selection of Vessel/Specifications(5145) = grade 2.
- id: en_008
query: "Industrial Safety and Health Management ergonomics"
category: english_only
intent: semantic_search
domain_hint: document
language: en
ocr_derived: false
failure_expected: false
graded_relevance:
3763: 3
3755: 1
relevant_ids: [3763, 3755]
top3_ids: [3763]
notes: |
Industrial Safety and Health Management(7-ED) ch08 Ergonomics(3763) = grade 3.
Contents(3755) = grade 1 (목차 매칭, 본 챕터 우선).
# ─────────────────────────────────────────────────────────
# 4b. mixed 신규 5 (양언어 ASME / 한국 KGS ↔ 영문 ASME / 영문 query → 한국 doc)
# ─────────────────────────────────────────────────────────
- id: cl_004
query: "ASME 압력용기 설계 실무"
category: mixed
intent: semantic_search
domain_hint: document
language: mixed
ocr_derived: false
failure_expected: false
graded_relevance:
5207: 3
5210: 3
5139: 2
5135: 2
relevant_ids: [5207, 5210, 5139, 5135]
top3_ids: [5207]
notes: |
한국어 query → 영문/혼합 ASME 도서.
ASME VIII Design_Practical_R1(5207, en) = grade 3 (실무 정확 일치).
ASME B-PV CODE SECTION Ⅷ Div.1 압력용기 코드 2007(5210, mixed title) = grade 3.
압력용기 핸드북_설계와 제작(5139, ko) = grade 2.
압력용기 핸드북_기하학과 배치도(5135, ko) = grade 2.
- id: cl_005
query: "ASME 용접 코드 해설"
category: mixed
intent: semantic_search
domain_hint: document
language: mixed
ocr_derived: false
failure_expected: false
graded_relevance:
5224: 3
5209: 2
relevant_ids: [5224, 5209]
top3_ids: [5224]
notes: |
asme_code해설[용접](5224, ko 해설) = grade 3 (직접 일치).
ASME IX Welding(5209, en) = grade 2 (원본 코드).
- id: cl_006
query: "pressure vessel Korean industrial safety regulation"
category: mixed
intent: semantic_search
domain_hint: document
language: mixed
ocr_derived: false
failure_expected: false
graded_relevance:
4025: 2
3856: 2
5136: 1
relevant_ids: [4025, 3856, 5136]
notes: |
영문 query → 한국 법령. 고압가스 안전관리법(4025, ko) = grade 2,
산안법 6장(3856, ko) = grade 2, PV Design Manual(5136, en) = grade 1
(보조 영문 자료). 동등 정답 2건, top3 강제 없음.
- id: cl_007
query: "KGS 코드 LPG safety standard"
category: mixed
intent: semantic_search
domain_hint: document
language: mixed
ocr_derived: false
failure_expected: false
graded_relevance:
11647: 3
11689: 2
11645: 2
4025: 1
relevant_ids: [11647, 11689, 11645, 4025]
top3_ids: [11647]
notes: |
혼합 keyword (KGS + LPG + safety standard).
04_KGS_Code(11647, ko 개요) = grade 3.
02_KGS_FU_사용(11689) / 02_LPG법(11645) = grade 2.
고압가스 안전관리법(4025) = grade 1 (관련 법령).
- id: cl_008
query: "수소 가스 안전 기준 hydrogen safety"
category: mixed
intent: semantic_search
domain_hint: document
language: mixed
ocr_derived: false
failure_expected: false
graded_relevance:
5178: 3
5169: 2
relevant_ids: [5178, 5169]
top3_ids: [5178]
notes: |
Hydrogen_Piping_and_Pipelines_ASME_Code(5178, en) = grade 3.
수소인프라 이슈 및 과제(5169, ko) = grade 2.
한국 KGS 수소 관련 별도 doc 부재 → 영문 우선 확정.
# ─────────────────────────────────────────────────────────
# 6. exam 신규 7 (가스기사 study_questions → library 개념 docs)
# ─────────────────────────────────────────────────────────
- id: exam_001
query: "레이놀즈수 정의 공식"
category: exam
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
11504: 3
11505: 2
relevant_ids: [11504, 11505]
top3_ids: [11504]
notes: |
study_questions 2019-1회 #2 "레이놀즈수를 옳게 나타낸 것은?" 매핑.
01_레이놀즈수(11504) = grade 3 (직접 정답 개념).
02_층류와난류(11505) = grade 2 (연관 개념).
- id: exam_002
query: "탱크 바닥 구멍 유체 유속 토리첼리"
category: exam
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
11500: 3
11495: 2
11496: 2
relevant_ids: [11500, 11495, 11496]
top3_ids: [11500]
notes: |
study_questions 2019-1회 #1 "수면의 높이가 10m로 일정한 탱크 바닥 5mm 구멍 유속"
매핑. 01_토리첼리(11500) = grade 3. 베르누이 개념/식(11495/11496) = grade 2.
- id: exam_003
query: "이상기체 음속 마하수"
category: exam
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
11514: 3
11515: 2
relevant_ids: [11514, 11515]
top3_ids: [11514]
notes: |
study_questions 2019-1회 #4 "이상기체 속에서의 음속을 옳게 나타낸 식" 매핑.
01_음속과마하수(11514) = grade 3. 02_등엔트로피유동(11515) = grade 2.
- id: exam_004
query: "고압가스 용기 내압시험 영구증가량"
category: exam
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
11591: 3
11644: 2
11691: 2
relevant_ids: [11591, 11644, 11691]
top3_ids: [11591]
notes: |
study_questions 2019-1회 #58 "고압가스 용접용기 내압검사 전증가량 250 mL
영구증가량" 매핑. 02_고압가스용기(11591) = grade 3.
01_고압가스안전관리법(11644) / 04_KGS_AC_용기(11691) = grade 2 (법령/용기 코드).
- id: exam_005
query: "도시가스 배관 매설 이격거리"
category: exam
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
11627: 3
11625: 2
11646: 1
relevant_ids: [11627, 11625, 11646]
top3_ids: [11627]
notes: |
study_questions 2019-1회 #76 "고압가스일반제조 사업소 밖 배관 매몰 설치
매설물 최소 이격거리" 매핑. 06_도시가스배관매설(11627) = grade 3.
04_도시가스배관(11625) = grade 2. 03_도시가스사업법(11646) = grade 1.
- id: exam_006
query: "LPG 저장탱크 안전거리 분말소화기"
category: exam
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
11617: 3
11669: 2
11620: 1
relevant_ids: [11617, 11669, 11620]
top3_ids: [11617]
notes: |
study_questions 2019-1회 #69 (LPG지상저장탱크 거리) + #80 (소형저장탱크
분말소화기) 매핑. 03_LPG저장설비(11617) = grade 3.
01_LPG안전(11669) = grade 2. 06_LPG사용설비(11620) = grade 1.
- id: exam_007
query: "오리피스 차압식 유량계"
category: exam
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: false
graded_relevance:
11712: 3
11711: 2
11503: 2
relevant_ids: [11712, 11711, 11503]
top3_ids: [11712]
notes: |
가스기사 계측 분야 (study_questions 회차 빈출). 02_오리피스유량계(11712) =
grade 3. 01_차압식유량계(11711) = grade 2. 04_오리피스(11503, 유체역학)
= grade 2 (원리 설명).
# ─────────────────────────────────────────────────────────
# 5b. failure_expected 신규 2 (의도적 no-result)
# ─────────────────────────────────────────────────────────
- id: fail_004
query: "KGS AC999 임의 가스 코드"
category: failure_expected
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: true
graded_relevance: {}
relevant_ids: []
notes: |
KGS AC 시리즈 (용기) 에 999 번호 부재. 의도적 no-result.
AC101/AC201/AC213 등 실재 번호와 혼동 회피 위해 999 선택.
- id: fail_005
query: "초전도 안전 관리법 시행규칙"
category: failure_expected
intent: fact_lookup
domain_hint: document
language: ko
ocr_derived: false
failure_expected: true
graded_relevance: {}
relevant_ids: []
notes: 존재하지 않는 가짜 법령명. 결과 0건 = correct.
Reference in New Issue View Git Blame Copy Permalink