Compare commits

...

44 Commits

Author SHA1 Message Date
hyungi 540bc00dba feat(docpage): D3 절 구조 탐색기 — 슬림 인사이트 레일 + 절 트리 (frontend only)
문서 상세 /documents/[id] 재구성 (BE 무변경):
- 우측 탭(정보/AI/관리) → 슬림 전역 인사이트 레일: 요약·심층·불일치를 탭
  게이트 없이 상시 노출(details open, 모바일은 접기 가능), 정보/관리는 접이.
  → 가공 자료가 탭/온디맨드에 묻히던 IA 문제(G1) 해소.
- SectionOutline 절 목차 레벨 기반 들여쓰기(평탄→트리 모양).
- 모바일: 본문 메인 + 절목차/인사이트/정보/관리 접이 + 절 탭 본문 이동(기존 구조 활용).
관련 문서(See Also)는 v1 제외(자리만 유지). 심화 목업 = comparisons/2026-06-13-ds-docpage-d3-deepened.html.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 15:18:15 +09:00
hyungi 30c235e4c1 Merge feat/safety-library-a1 (C-1 후속 version_status+facets) into ds-board-merged
검색 결과 wrapper decoration: 법령 version_status + facets 집계(ranking 무관·additive).
2026-06-13 15:08:24 +09:00
hyungi 8a3bea6b31 feat(safety): C-1 후속 — version_status decorate + facets 집계
검색 엔드포인트 wrapper decoration(run_search 코어 무접촉·ranking 무관):
- version_status: 법령 결과(material_type=law)에 legal_meta.version_status 부착
  (decorate_version_status, law 무결과 시 query skip). SearchResult.version_status 신설.
- facets=true: top-K 결과 분류 축(material_type/jurisdiction/version_status) 분포 라벨
  (compute_facets). 미요청=None(byte 불변). SearchResponse.facets 신설.
- result_decorate.py 신설. 단위 4건.
freshness incident 변경(law_365d 제거+흡수)=ranking 변경이라 별 슬라이스 defer.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 15:07:57 +09:00
hyungi cd439b0ff4 Merge feat/safety-library-a1 (B-4 licensed_restricted + watch 폴더 license) into ds-board-merged
B-4 PR①②: licensed_restricted 단일 술어(retrieval 3-leg/digest/briefing/study 풀이 공유)
+ file_watcher Books/Manuals/Papers_Purchased license 주입. prod 통합 브랜치 배포용.
2026-06-13 14:53:34 +09:00
hyungi a6db6c999b fix(safety): B-4 리뷰 반영 — 단일 술어 중앙화 + study/briefing 경로 커버
적대 리뷰(10에이전트) 확정 반영:
- license_filter.py 신설 — restricted_exclude_sql(raw)/restricted_exclude_orm(ORM)
  단일 정의. retrieval _license_sql·digest·briefing·study 풀이가 공유(드리프트 방지).
- major: explanation_rag(study 문제 AI 풀이 RAG)에 술어 누락 → doc_meta 쿼리에 ORM
  적용(valid_doc_ids 경유로 청크도 차단). briefing/loader 2쿼리에 누락 → digest 와
  동일 술어 추가(news restricted 부재=방어적·경로 일관성).
- blocker(low-impact): file_watcher changed-doc 경로 material/license 보정(merge 주입·
  license 부재 시만 — extract_meta clobber 회피, pre-B-4 적재분 동기화).
- 테스트: 단일-source 검증 + ORM 구성 스모크 2건 추가.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 14:52:04 +09:00
hyungi ed7740beee feat(safety): B-4 PR①② — licensed_restricted 차단 술어 + watch 폴더 license 주입
PR① licensed_restricted 단일 술어(_license_sql) — retrieval 3-leg(text/vec-doc/
vec-chunk) + digest loader 공유. a안(U-2①): 색인 허용·구매자료 verbatim 을 RAG 증거/
digest 발행에서 구조적 제외. 술어=COALESCE(extract_meta->'license'->>'restricted',
'false')<>'true' (restricted 부재/false 미제외 → 기존 코퍼스 결과 불변). 개인 파일
열람 미차단. chunk leg 는 outer 의 documents JOIN(항상) 활용 post-rank(restricted 소수).
PR② file_watcher _TARGET_AXIS 확장 — Books/Papers_Purchased=restricted / Manuals=
non-restricted(사용자 결정) / KGS=law·KR·kogl. ingest 시 extract_meta.license
deterministic 주입(classify material IS NULL 일 때만 제안·meta 미기록=보존).
PR③(KGS 버전 flip)=별 슬라이스 deferred(파일 포맷 조사 선행).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 14:34:42 +09:00
hyungi 595f4b7d5e feat(board): 통합 보드 v3 — 머신 레인 + 정직 번다운/ETA (B-1·2·3·5)
ProcessingFlowBoard 를 통합안으로 재작성:
- 머신 3레인(GPU/맥미니/맥북) = "누가 일하나" + 요약 오프로드 가시화
  (요약 칩 분담 막대 맥미니 vs 맥북 + 맥북 레인 '요약 합류' 칩, summarize_by_machine 소비)
- 지배 백로그 스트립 + 정직 ETA(summarize_eta, 유입 차감 / null=소진 불가)
- 24h 번다운 SVG(유입 vs 소화) + 맥북 합류 변곡점 + 단계별 정직 ETA 미니리스트
- 신선도 '갱신 N초 전' + stale 경고(queueUpdatedAt, B-4)
- 실패 드로어 + 노드 상세 패널은 v2 자산 그대로 재사용
- 레인 stack + 칩 wrap 으로 모바일 반응형

svelte-check: 변경 파일 에러 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 14:01:50 +09:00
hyungi b630c31077 feat(board): expose summarize_by_machine for offload visibility (A-1)
요약 풀의 머신별 완료 실적(맥미니 vs 맥북)을 /api/queue/overview 응답에
summarize_by_machine 로 노출. rows_to_summarize_split 이 이미 계산하던 값의
additive 투영 — 신규 수집 SQL/마이그 0. 통합 보드 레인의 오프로드 가시화
(맥북이 요약 86% 처리) 재료. + FE 타입 동기 + store 신선도 timestamp(B-4).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 13:54:39 +09:00
hyungi 235aa648ad feat(safety): B-2 KOSHA 사망사고 속보 수집기 (callApiId=1040)
data.go.kr 15119137 활용신청 전파 완료 → news_api02/getNews_api02 라이브.
collect_fatal_accidents: arno dedup(kosha-fatal|{arno}) + material_type=incident/
jurisdiction=KR + license=kogl. contents=HTML → _clean_html, published_date =
arno 접두 8자리(YYYYMMDD 등록일, 2019~ 라이브 전수 동형 검증). 첨부 API·business
필드 없는 별 채널(1040). run() 일일 잡(06:40 KST) 튜플 합류 — 소스별 실패 격리 유지.
순수 헬퍼 _fatal_fields + fixture 테스트(tests/test_kosha_fatal.py).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 13:42:12 +09:00
hyungi 60cb48bbe4 fix(safety): C-1 fusion 재구성 시 분류 축 메타 전파 — 3 SearchResult 재생성 지점
fusion legacy(line 66)·RRF(122)·multi_query rewrite(pipeline 456)가 명시 필드 나열로
SearchResult 재구성 → material_type/jurisdiction/published_date 누락(필터는 정확, D-1
유형 표시만 None). 세 지점 동기화. 흉터: SearchResult 필드 추가 시 재구성 지점 전수 동기 필요.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 13:03:27 +09:00
hyungi 79deae0644 feat(safety): C-1 검색 명시 필터 — material_type/jurisdiction/year 3-leg 동등 + documents exclude 해제
plan safety-library-1 C-1 (검색 핵심 경로 — byte 불변 invariant):
- AxisFilter + _axis_sql 헬퍼: 미지정 시 모든 SQL 절 빈 문자열(run_eval 회귀 0 보장)
- 3 leg 동등 적용: search_text(JOIN 후 WHERE) / _search_vector_docs(prod+cand) /
  _search_vector_chunks(★inner topk JOIN — R6 결정: outer post-filter면 ANN top-k 후
  좁은 필터 후보 붕괴. 미지정 시 JOIN 없음=byte 불변)
- SearchResult + material_type/jurisdiction/published_date (3 leg SELECT additive)
- year = COALESCE(published_date, created_at) (freshness 동일 사상)
- GET /documents/: material_type 지정 시 기본 exclude(news·law_monitor·note) 해제
- _axis_sql 단위 테스트 PASS (미지정=빈문자열+param0 / active 4절 / alias 분기)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 12:59:18 +09:00
hyungi 9a7e231dcc fix(safety): verify_statute_chain sys.path — /app 루트 자동 탐지 (workers import)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 12:44:58 +09:00
hyungi 1646617a31 feat(safety): B-1 PR③ — 법령 체인 검증 3술어 스크립트 (read-only 진단)
plan safety-library-1 B-1 PR③. E-1 법령 게이트 도구 겸용 (반복 실행 안전):
- ① 존재성: watch family 각 primary current 정확 1건 + annex 시리즈당 ≤1
- ② 노출 유일성: primary current 보유 family당 노출 1건 (③a에 흡수)
- ③ 고아 그물: 정규화 동등 매핑 — flip 누락(current family 노출 레거시)·무매핑(매핑 구멍) 0
- repealed family ①② 면제. 종료코드 0/1 (관찰 게이트용)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 12:44:25 +09:00
hyungi bacb36924b feat(safety): B-1 PR② — fetch_version(payload 리스트) + ingest 4축 + 생애주기 잡 통째 + 부트스트랩
plan safety-library-1 B-1 PR② (R8-B1: 승격·supersede·스윕·repeal = 잡 코드 통째 배포):
- kr.fetch_version: 전문 1콜 → primary+annex payload 리스트 (R4-M4)
  ★fixture 가 잡은 결함 2: 별표구분(별표/서식) 차원 누락 시 (번호,가지) 4건 충돌
  → version_key='MST|{구분}{번호}-{가지}' / 삭제 tombstone 3건(별표10·서식1·2) skip
  — KR 별표 삭제 = absence 아닌 명시 tombstone (R7-M3 absence 추론 불요 확정)
- ingest: 전 버전 pending 적재 + 4축(law/KR/COALESCE날짜/public_domain) + backfill 마커
- 생애주기 잡: 버전 시리즈 단위 승격·supersede(R7-B1) + 상태 기반 레거시 스윕(primary
  current 보유 한정) + repeal(레거시 매핑분 포함, R7-M2) — 단일 트랜잭션·KST
- 법령명 매핑: 정규화 동등 비교(prefix 금지 — 시행령 오폭 차단), 가운뎃점·공백 흡수
- 워터마크 = 파싱 검증 통과 후에만 / 스케줄 daily 07:00 KST (law_monitor 슬롯 승계)
- 테스트 14/14 (매핑 표본·시리즈 키·payload fixture)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 09:37:51 +09:00
hyungi a28f12b12e feat(safety): B-1 PR① — law_monitor 스케줄 제거 + statute KR poll_changes + fixture 박제 (mig 356)
plan safety-library-1 B-1 PR① (fixture-first):
- law.go.kr 라이브 fixture 5종 박제 (OC 새니타이즈 검증 — 응답 법령상세링크에 키 포함 함정)
- R7-M3 판정: 전문 1콜 XML = 조문 853+별표 23 전체 스냅샷(부분 실패 개념 없음)
  + 별표번호/가지번호 = 구조화 필드 — 조문 취득 = 전문 1콜+로컬 파싱 확정(R2-m1)
- legal_acts KR 시드 26행(법령ID 라이브 실측, watch=26 전부, FK 계열 9그룹)
  ★ '유해ㆍ위험작업...' 정식명 = 가운뎃점 — law_monitor 하드코딩(점 없음)은 영구 미매칭 잠복
- statute_adapters/kr.py: poll_changes(lawSearch MST diff) — 순수 파서 분리, fixture 테스트 8/8
- statute_collector.py: 관찰 전용 코어(워터마크 영속 0 — ingest=PR②), 스케줄 미등록(R8-B1)
- main.py: law_monitor 스케줄 제거 — 버전 체인 밖 레거시 매일 증식의 유일 경로 차단

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 09:01:21 +09:00
hyungi 0c8fb41366 fix(safety): backfill text() 콜론 bind 오인 — exec_driver_sql 로 교체
정규식 '(?:' 의 콜론을 text() 가 bind param 으로 해석 (migration 러너 동일 함정).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 06:49:58 +09:00
hyungi e5ddd0e4d6 feat(safety): A-3 backfill 스크립트 — 기존 코퍼스 분류 축 소급 (교정 술어)
plan safety-library-1 A-3. prod 실측 반영:
- KGS frontmatter = 'code' 키 확정(117/118, kgs_code 0) → 경로 술어
- 레거시 law 243건 — extract_meta 빈값, title '(YYYYMMDD)' 공포일 추출
- GUIDE ofancYmd = 'YYYY-MM-DD' 실측
- KOSHA 본문 = source_id JOIN (kind='case' 부재 — R2 blocker 교정 그대로)
- dry-run = 트랜잭션 ROLLBACK 방식 (정확 rowcount + 검증표, 변경 0)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 06:48:30 +09:00
hyungi 3feddd012b feat(safety): A-2 수집기 ingest 시점 분류 축 부여 — 레지스트리 전파 + 승인 가드 (mig 352~355)
plan safety-library-1 A-2 (classify-skip 경로 전수 커버):
- news_sources 에 material_type/license_scheme/license_redistribute + 안전·공학 12행 시드
- news_collector: 레지스트리 → documents 전파 (_material_axis — paper 는 jurisdiction NULL 강제)
- kosha(사례·첨부=incident, GUIDE=guide)/csb(incident·US)/api_std(standard·US)/law_monitor(law·KR)
  /file_watcher(KGS=law·KR 타깃 매핑) deterministic 부여 + extract_meta.license 주입
- published_date: 소스별 가용 날짜 (GUIDE 공표일·CSB lastmod·API 공지일·법령 공포일·뉴스 발행일)
- classify_worker: document_type→material_type 결정적 매핑 제안 (자동 전이 금지)
- accept-suggestion: material 제안 적용 + law=jurisdiction 필수(기본값 없음) + 청크 미러 1문 동기화
- chunk_worker: 비뉴스 문서 country=jurisdiction 미러 (R3-m3: 검색측 country 소비자 0 실측)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 06:23:22 +09:00
hyungi 5da94213ec feat(safety): 분류 축 A-1 — material_type/jurisdiction/published_date + legal_acts/legal_meta (mig 340~351)
안전 자료실 plan safety-library-1 A-1 (r3 계약 반영):
- documents 3컬럼 (TEXT+CHECK, nullable additive) + law→jurisdiction NOT NULL 구조 강제
- legal_acts 단일 레지스트리(워치리스트 겸, watermark·repeal_detected_at 포함)
- legal_meta 최소형 (version_key 합성형 UNIQUE, 전 버전 pending 적재 계약)
- partial 인덱스 2 + family 인덱스 + paper DOI partial UNIQUE (doi=서지 단일 보유 계약)
- ephemeral PG16 스모크: 12파일 적용 + CHECK/UNIQUE 계약 6종 검증 PASS

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 21:25:04 +09:00
hyungi 85304878f4 test(eval): Phase 2A eval 산출물 — baseline exact 0.6315 vs qwen06/-0.041 qwen4/-0.032 qwen4m/-0.033 (no-go)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 05:53:20 +00:00
hyungi adce639445 ops(eval): Phase 2A eval 종료 — embed/chunk hold 복원 (E-4 = no-go, bge-m3 유지)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 12:03:39 +09:00
hyungi d05e41128a fix(board): 실패 뱃지 잘림(스크롤 컨테이너 헤드룸) + 구 단계별 현황 섹션 제거 + ETA 48h+ 일 표기
- 흐름 컨테이너 pt/px 헤드룸 — -top/-right 돌출 뱃지가 overflow-x-auto 에 잘리던 문제
- 단계별 현황 details = 흐름 보드가 대체(R2 통합안 의도) — 전용 파생값/헬퍼/chevron 동반 제거
- etaShort: 48시간 이상은 일 단위 (약 131시간 → 약 5.5일)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 03:02:04 +00:00
hyungi 2bbdf63d86 ops(eval): Phase 2A eval 동결 — embed/chunk 일시 hold (eval 후 복원)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 11:17:42 +09:00
hyungi 5581d3f1ce feat(board): 처리 보드 v2 — 파이프라인 흐름 뷰·엔진 구분·실패 재시도/건너뛰기 (ds-board-engines-1)
- 흐름 뷰 메인: 좌→우 노드(머신·엔진 태그, 유입 우세 amber, 실패 뱃지) + 머신 스트립(모델 표기) + trend_24h 스파크라인 첫 렌더
- 노드 클릭 상세 패널: KV 4칸 + 다중 stage 행 + 지금 처리 중
- 실패 처리 드로어: 에러 패턴 그룹 + 재시도/건너뛰기 (영구 실패의 첫 사용자 조치 경로)
- API: stages[].done_1h/created_1h 노출 + GET /api/queue/failed + POST /api/queue/retry|/skip (uq_queue_active 충돌 skip, 건너뛰기는 enqueue_next_stage 미호출)
- 엔진/모델 표기 = queueDisplay.ts 정적 맵 단일 지점 (모델 교체 시 1곳)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 01:05:04 +00:00
hyungi 8ac1dbf4a8 test(eval): Phase 2A E-4 비교기 — per-query win/loss/tie(ε)·부트스트랩 CI·카테고리 분해
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 08:34:18 +09:00
hyungi c3d237766d feat(search): Phase 2A E-1 — Qwen 후보 3종 백필 CLI + eval 디스패처 확장 (마이그 328~333)
- 후보 섀도 테이블 6종(전부 vector 타입 — eval=exact scan 이라 인덱스 불요, halfvec 은 C-1 소관)
- workers/phase2a_cand_backfill: resumable(NOT EXISTS)·배치 커밋·동결셋 한정(--doc/chunk-id-max),
  문서/청크 입력 = production 경로 동일 구성 + plain
- CANDIDATE_BACKEND_MAP += cand_qwen06/qwen4/qwen4m (embed_kind=ollama, 쿼리측 instruct prefix
  G-1 핀 문자열, qwen4m = dimensions 1024 MRL)
- qwen4m 적재는 qwen4 에서 SQL 파생(subvector+l2_normalize) — 본 CLI 비대상

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 08:29:53 +09:00
hyungi 5bc68c95f6 test(eval): Phase 2A G-1 — Qwen3-Embedding 서빙 fixture 박제 (Ollama 0.20.0, /api/embed)
0.6b=1024d/4b=2560d 정규화 출력, MRL dimensions 옵션 지원(재정규화 포함),
비대칭 instruct prefix 효과 실측(+0.016), instruct 문자열 핀.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 08:14:24 +09:00
hyungi 5dca5b5d28 ops(pipeline): embed/chunk 고속 컨슈머 분리 + 배치 1→10 — LLM 사이클 인질 해소
진단(2026-06-12 용량 평가): 단일 루프에서 classify(~190s×3)가 사이클을 점유,
건당 <1s 인 embed/chunk 가 사이클당 1건 캡 → 실효 ~580/일 vs 수요 최대 2,700/일,
적체 3,570 + 신규 문서 벡터 미적재(RAG 검색 누락). 4070 가동률 0% = 순수 구조 캡.
수리 = markdown 분리(05-01) 선례: consume_fast_queue 1분 잡 + 배치 10(GPU 공유 보수값,
캡 ~14,400/일). 세 컨슈머 stage 집합 disjoint(stale reset 이중 복구 방지). retrieval
로직·임베딩 모델 무접촉.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 07:50:07 +09:00
hyungi 9c9ff6eeba test(drain): classify 합류 반영 — 거부 케이스를 extract 로 교체
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 07:22:47 +09:00
hyungi d667545185 fix(classify): 적대 리뷰 반영 — use_deep 스레딩(B1)·StageDeferred 전파(B2)·legacy 호출 deep 경유(M3)
- _run_tier_triage(use_deep) 스레딩 — 미배선 NameError(전 classify 파괴) fix
- process 의 triage try 에 except StageDeferred: raise 선행 (drain 보류 시멘틱 복구)
- legacy classify()/summarize() 에 cfg 파라미터 — use_deep 시 deep 슬롯 경유 +
  is_deferrable_error → StageDeferred 변환(첫 호출 = 최저비용 지점에서 보류, doc 쓰기 0)
- ai_model_version = 실제 처리 경로 모델 (drain=qwen-macbook 귀속)
- analyze_event model_name 스레딩 + deep triage cfg 에 top_p 동승

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 07:12:40 +09:00
hyungi 235bbf9881 ops(pipeline): fair-share 번들 — drain classify 합류 + deep 맥미니 폴백 + mlx 게이트 동시 2
사용자 '공평하게 동일한 작업' 지적의 비대칭 잔재 2건 + 예고된 배칭 레버:
- queue_drain --stage classify (use_deep: deep 슬롯 endpoint + triage sampling,
  완료 시 enqueue_next_stage 로 embed/chunk/markdown 연쇄 — DAG 단절 방지)
- deep_summary consumer = 맥북 우선, 불가 시 맥미니 primary 즉시 처리(동일 모델 —
  강등 아님). drain 은 defer_on_deep_unavailable=True 로 기존 보류-종료 유지
- llm_gate capacity 일반화 (config pipeline.mlx_gate_concurrency, 기본 1, 운영 2) —
  'MLX_CONCURRENCY=1 고정' 영구 룰의 전제(single-inference 서버) 소멸을 docstring 에 개정 박제
- analyze_events FK(users) CLI 컨텍스트 INSERT 실패 fix (models.user 명시 import)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 06:56:02 +09:00
hyungi 30200a4e49 ops(ai): deep 슬롯 재도입 — 맥북 야간 night-drain 레버 (Qwen3.6-27B-6bit)
사용자 지시: 자기 전 night-drain 한 번 실행 → 맥북이 밤새 summarize/deep_summary 분담.
보류 시멘틱(StageDeferred)·drain CLI·라우터 wake preflight = 기존 검증 자산 재사용.
맥북 측 = RunAtLoad=false 수동 기동 + 서버 수명 한정 caffeinate + idle-watch 자동 종료.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 21:49:12 +09:00
hyungi eff2c3b7d3 ops(search): Qwen 27B 속도 반영 — synthesis 30s→120s, classifier 슬롯 모델 동승 교체
- config classifier 모델 gemma 잔존 = mlx 서버 Gemma 재로드(이중 적재) 위험 → Qwen 6bit 로 동승 교체
- synthesis 는 timeout 시 graceful skip 이 없는 답변 본체라 단독 상향 (classifier/query_analyzer/
  rewriter 의 30s/15s 캡은 초과 시 skip·원쿼리 폴백으로 degrade — 관찰 후 별도 튜닝)
- ask.backend.timeout_read_s 30→120 align

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 17:31:26 +09:00
hyungi 3d79002dfa ops(ai): Qwen 27B 프리필 실측(~112 tok/s) 반영 timeout 상향 — triage 480 / primary 900
장문(context_char_limit 상한급) 프리필이 수 분 걸려 기존 120/300s 로는 timeout 실패 churn.
단일 코루틴 컨슈머라 장문 1건이 사이클을 수 분 점유하는 것은 수용(관찰 후 배칭/컨텍스트 튜닝 PR).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 17:29:45 +09:00
hyungi 3d60008965 ops(ai)!: 맥미니 생성 모델 Qwen3.6-27B-6bit 전환 + 생성 LLM 홀드 해제
B안(사용자 2026-06-11): Gemma 26B-A4B → Qwen3.6-27B-6bit 풀교체.
- config.yaml triage/primary model 교체 + dense 감속 반영 timeout 상향(30→120/180→300)
- held_stages [] (홀드 해제 — 적체 자연 드레인, deep_summary 는 primary 복귀)
- eid deep 모드 = mac-mini-default 재지정(맥북 백지화). llm_gate '예외 없이 gate' invariant 에
  따라 deep 도 alias 조건으로 자동 게이트 (구 무게이트 = 맥북 별 endpoint 예외였음)
- deep probe 실패 reason = router_unreachable 로 정정 + 테스트 동기화
잔여(별 PR): ask 표면 qwen-macbook 옵션/백엔드 클래스/처리보드 맥북 카드 정리

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 17:19:35 +09:00
hyungi cd0040925a ops(pipeline): 생성 LLM 홀드 게이트 held_stages — 맥미니 모델 확정까지 보류
맥북 LLM 백지화 + 맥미니 모델 재결정에 따라 DS 의 생성 LLM 소비를 일괄 보류.
held = classify/summarize/deep_summary(큐, claim 미발생·attempts 미소모) +
digest(04:00)/briefing(05:10) cron + study explanation/session_analysis/memo_card 컨슈머.
GPU 특화 스테이지·수집기·인터랙티브(ask/eid chat)는 무영향. 기본값 [] = 무동작.
/api/digest/regenerate 는 홀드 중 409 명시. 해제 = config held_stages 비우고 fastapi 재기동.
exec plan: ~/.claude/plans/ds-llm-hold-exec-20260611.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 16:52:46 +09:00
hyungi fdac449a48 Merge pull request 'Feat/eid chat' (#35) from feat/eid-chat into main
Reviewed-on: #35
2026-06-11 15:14:43 +09:00
hyungi 40f5b5fe9e Merge pull request 'Feat/ds processing board' (#33) from feat/ds-processing-board into main
Reviewed-on: #33
2026-06-11 15:14:24 +09:00
hyungi 250896cdfa feat(eid): deep 모드 = ReAct 자동검색 + 근거 카드 (ds-eid-ask-absorb P1)
- deep 분기 _eid_chat_deep: 비생성 probe → phase:searching → agentic_ask_loop
  (tool_choice=auto 가 검색 여부 자율 판단, 검색 불요는 early-exit 대화) → final_answer
  + eid_sources envelope → DONE. heartbeat {phase:ping}(~10s, 프록시 idle timeout 차단)
  · mid-stream BackendUnavailable → in-stream error envelope · disconnect 시 task.cancel()
  + await(고아화·27B 점유 방지).
- daily = call_stream 무변경(맥미니 대화). deep = 맥북 27B ReAct (tool calling 27B 전용,
  맥미니 26B token-leak 미검증). 멀티턴 = 메시지 단독 처리(agentic_ask_loop query: str,
  history 2단계 백로그).
- EidEvidenceCard.svelte 접이식 근거 카드(sources 순서번호·제목·점수) + 프론트 SSE 파서
  확장(ping/searching/error/eid_sources) + 검색 중 표시 + 이력 보존.
- 테스트: deep 4건(검색성/대화성/probe-503/mid-stream-error) + 기존 call_stream 회귀 daily
  로 이전 = 29 passed.
- 동반(이전 eid-chat 세션 미커밋): /api/eid/status endpoint + llm_gate.gate_status +
  test_eid_status (채팅 대기 UI 의 '대기 vs 고장' 구분용, 5 passed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 14:51:00 +09:00
hyungi a410f5b65c fix(ui): 머신 state 우선순위 — 가동 > 보류 (일하는 중엔 백오프 잔여여도 가동)
실측: 맥북이 드레인 처리 중인데도 백오프 잔여 때문에 카드 전체가 '보류'로 표시.
보류 칩은 일이 멈춰 있고 백오프만 쌓인 상태(sleep/불가 지속) 한정으로 강등,
보류 건수 자체는 카드의 deferred_pending 라인이 계속 표시.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 14:36:10 +09:00
hyungi 7031439364 feat(ui): 단계별 현황 재설계 — 완료 가시화 + 빈 단계 숨김 (사용자 피드백)
'대기만 보이고 성공은 안 보인다' 피드백 반영:
- overview 에 stages[] 노출 (stage 별 done_today + oldest_pending_age, SQL 1필드 추가)
- 게이지 의미 전환: 단계 간 대기량 비교(amber) → 단계 내 오늘 진척(완료=green 비율,
  가득 찬 초록 = 다 끝남) + 처리 중 pulse dot
- 움직임 없는 단계는 행 제거, 하단 '비어 있음: ...' 한 줄로
- 라벨 누수 fix: details 가 구 STAGE_LABEL 을 쓰던 것 → queueStageLabel 통일
  (deep_summary/markdown/summarize/chunk/fulltext 한글화)
- 헤더: 오늘 N 완료(성공 가시화) · 실패(error) · 대기. 데이터 소스 = overview 단일화

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 14:26:27 +09:00
hyungi 468804494d feat(ui): 처리 머신 보드 — 누가 일하나 (안2) + ETA·전 페이지 스트립/드로어 (안5/6 라이트)
plan ds-processing-ui-6an (시안 choice 채택: 안2 1차 + 안5/6 지원):
- GET /api/queue/overview — 머신(GPU/맥미니/맥북) 귀속 라이브 집계 5쿼리, 마이그레이션 0.
  summarize 풀 완료 실적은 documents.ai_model_version 조인으로 맥북/맥미니 분리,
  보류(deferred_until)=맥북 카드 귀속, state=active/deferred/idle. raw 모델명 비노출
- 홈: 처리 머신 보드(3열 카드 + 지금 처리 중 제목) + ETA 라인(유입 우세 시 null 명시),
  기존 stage 테이블은 details 접힘으로 강등 (구조 개편)
- 전 페이지: 상태 스트립(처리중·대기·실패·맥북 칩) + 우측 드로어(QueueDrawer,
  dialog a11y) — 공유 60s 폴링 store, 경량 fetch(401 강제 logout 부수효과 회피)
- tests: 판정부 30건 (귀속/풀 분리/state 9케이스/ETA 경계/trend 버킷/계약 shape)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-11 14:13:35 +09:00
hyungi 5e8b998a11 feat(documents): hier analyze 서브커맨드 — 재분해와 독립한 절분석 self-heal (g3-t3 갭)
re-decompose 의 char_start 완료마커는 'jump-target char_start 보유'라 컨테이너 recreate/deadline 으로
analyze 가 잘린 doc(char_start 있으나 일부 leaf 미분석)을 재선별 못 함 → rail summary 영구 미수렴 갭.
`analyze` 가 LEAF_SQL(미분석 leaf 보유) 기준 독립 선별로 수렴(멱등, --doc 제한 가능, jump 무관).
sweep 로그도 `analyze` 커맨드 안내로 갱신. (2026-06-10 백필서 recreate 로 잘린 5 doc·53 leaf 수동 처리한 케이스 항구화.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 06:32:10 +09:00
hyungi 53999b2825 fix(documents): g-measure junk 검출 all-caps 과탐 제거 + verdict=coarse 스크린 명시
전부-대문자 휴리스틱이 기술문서 정상 heading(GENERAL REQUIREMENTS/WELDING) 130건 과탐 →
windowed/clean doc 거짓 A_better 강등. 회사-접미사(INC./LLC…)만, cover 영역(앞 4노드)+미stored 게이트.
verdict 는 coarse 스크린(감사용)이고 실집행 결정 = 결정적 partition + 적대 워크플로임을 docstring 박제.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-09 12:58:36 +09:00
110 changed files with 6598 additions and 248 deletions
+22 -8
View File
@@ -150,15 +150,26 @@ def is_deferrable_error(exc: Exception) -> bool:
return isinstance(exc, httpx.TransportError)
async def call_deep_or_defer(client: "AIClient", prompt: str, system: str | None = None) -> str:
async def call_deep_or_defer(
client: "AIClient",
prompt: str,
system: str | None = None,
cfg: "AIModelConfig | None" = None,
) -> str:
"""call_deep + 보류 변환 — 맥북 불가(503/연결/절단)는 StageDeferred 로 raise.
deep_summary_worker / summarize_worker(drain) 가 공유. StageDeferred 는 queue_consumer/
queue_drain 이 attempts 미소모 + deferred_until 백오프로 처리한다 (sleep-안전 불변식).
deep_summary_worker / summarize_worker(drain) / classify_worker(drain) 가 공유.
StageDeferred 는 queue_consumer/queue_drain 이 attempts 미소모 + deferred_until
백오프로 처리한다 (sleep-안전 불변식).
cfg: 지정 시 deep 슬롯 대신 이 config 로 호출 (classify drain — deep 슬롯의
endpoint 는 쓰되 triage 의 temperature/max_tokens 를 적용한 변형).
"""
from models.queue import StageDeferred
try:
if cfg is not None:
return await client._request(cfg, prompt, system=system)
return await client.call_deep(prompt, system=system)
except Exception as exc:
if is_deferrable_error(exc):
@@ -231,20 +242,23 @@ class AIClient:
# ─── Legacy API (classify_worker 교체 시 제거 예정) ───────────────────
async def classify(self, text: str) -> dict:
async def classify(self, text: str, cfg=None) -> dict:
"""[DEPRECATED] 기존 classify_worker 전용. B-1 에서 summary_triage 로 대체.
호출부 정리 전 존속. 신규 코드는 call_triage + prompt_render 를 쓸 것.
cfg (2026-06-12 fair-share): 지정 시 primary 대신 해당 config 로 호출 —
drain classify 가 deep 슬롯(맥북) 경유에 사용. cfg != ai.primary 라
_call_chat 의 primary→fallback 자동 전환은 발동하지 않는다 (에러 raw 전파).
"""
prompt = CLASSIFY_PROMPT.replace("{document_text}", text)
response = await self._call_chat(self.ai.primary, prompt)
response = await self._call_chat(cfg or self.ai.primary, prompt)
return response
async def summarize(self, text: str, force_premium: bool = False) -> str:
"""[DEPRECATED] 기존 호출부용. B-1 에서 summary_triage 가 tldr 대체."""
async def summarize(self, text: str, force_premium: bool = False, cfg=None) -> str:
"""[DEPRECATED] 기존 호출부용. B-1 에서 summary_triage 가 tldr 대체. cfg = classify() 와 동일."""
if force_premium:
return await self._call_chat(self.ai.premium, f"다음 문서를 500자 이내로 요약해주세요:\n\n{text}")
return await self._call_chat(self.ai.primary, f"다음 문서를 500자 이내로 요약해주세요:\n\n{text}")
return await self._call_chat(cfg or self.ai.primary, f"다음 문서를 500자 이내로 요약해주세요:\n\n{text}")
async def embed(self, text: str) -> list[float]:
"""벡터 임베딩 — GPU 서버 전용"""
+8
View File
@@ -244,7 +244,15 @@ async def regenerate(
user: Annotated[User, Depends(require_admin)],
):
"""수동 트리거 — 백그라운드 태스크로 워커 실행 (admin 필요)."""
from core.config import settings
from workers.digest_worker import run
# 홀드 중 silent no-op 방지 — 워커 게이트와 동일 조건을 표면에서 명시.
if "digest" in settings.pipeline_held_stages:
raise HTTPException(
status_code=409,
detail="global_digest 보류 중 (config.yaml pipeline.held_stages) — 해제 후 재시도",
)
asyncio.create_task(run())
return {"status": "started", "message": "global_digest 워커 백그라운드 실행 시작"}
+57 -4
View File
@@ -210,8 +210,14 @@ class DocumentDetailResponse(DocumentResponse):
class AcceptSuggestionRequest(BaseModel):
"""§1 accept-suggestion 요청 body — stale payload / doc 수정 검출."""
"""§1 accept-suggestion 요청 body — stale payload / doc 수정 검출.
jurisdiction: 안전 자료실 A-2 — material_type 제안 승인 시 사용자가 지정하는 관할.
law 승인은 필수 (기본값 없음 — KR 자동 부여 시 외국 자료가 KR 법령으로 오염되는
경로를 차단, plan A-2 계약).
"""
expected_source_updated_at: datetime
jurisdiction: str | None = None
class DocumentUpdate(BaseModel):
@@ -537,6 +543,8 @@ async def list_documents(
category: str | None = Query(None, description="doc_category enum — 지정 시 기본 news/memo 제외 해제"),
has_suggestion: bool | None = Query(None, description="true: ai_suggestion IS NOT NULL"),
proposed_category: str | None = Query(None, description="ai_suggestion.proposed_category 필터"),
material_type: str | None = Query(None, description="안전 자료실 C-1: 자료유형. 지정 시 기본 exclude 해제"),
jurisdiction: str | None = Query(None, description="안전 자료실 C-1: 관할 (KR/US/...)"),
):
"""문서 목록 조회 (페이지네이션 + 필터).
@@ -550,6 +558,10 @@ async def list_documents(
if category:
# 명시적 카테고리 필터 — 기본 exclude 해제
query = query.where(Document.category == category)
elif material_type:
# 안전 자료실 C-1: material_type 지정 = 기본 exclude(news·law_monitor·note) 해제.
# 안전 코퍼스 본체(KOSHA 사례·CSB·법령 등)가 전부 note/crawl 채널이라 exclude 면 빈 화면.
query = query.where(Document.material_type == material_type)
else:
# 기본 목록: 뉴스/메모/법령 제외 (문서함 용도)
query = query.where(
@@ -558,6 +570,9 @@ async def list_documents(
Document.file_type != "note",
)
if jurisdiction:
query = query.where(Document.jurisdiction == jurisdiction)
if has_suggestion is True:
query = query.where(Document.ai_suggestion.isnot(None))
elif has_suggestion is False:
@@ -1244,11 +1259,49 @@ async def accept_suggestion(
# payload 적용
proposed_category = doc.ai_suggestion.get("proposed_category")
proposed_path = doc.ai_suggestion.get("proposed_path")
# 안전 자료실 A-2 — material_type 제안 (classify 의 document_type 결정적 매핑)
proposed_material = doc.ai_suggestion.get("proposed_material_type")
if not proposed_category:
raise HTTPException(status_code=422, detail="proposed_category 누락된 suggestion")
if not proposed_category and not proposed_material:
raise HTTPException(
status_code=422,
detail="proposed_category/proposed_material_type 둘 다 누락된 suggestion",
)
doc.category = proposed_category
if proposed_category:
doc.category = proposed_category
if proposed_material:
_MATERIAL_TYPES = {"law", "paper", "book", "incident", "manual", "standard", "guide"}
_JURISDICTIONS = {"KR", "US", "EU", "JP", "GB", "INT"}
if proposed_material not in _MATERIAL_TYPES:
raise HTTPException(
status_code=422, detail=f"허용 밖 material_type: {proposed_material}"
)
jur = body.jurisdiction or doc.ai_suggestion.get("proposed_jurisdiction")
if jur is not None and jur not in _JURISDICTIONS:
raise HTTPException(status_code=422, detail=f"허용 밖 jurisdiction: {jur}")
# law = 국가 필수 입력, 기본값 없음 (plan A-2 — KR 자동 부여 시 외국 법령 오염.
# DB CHECK(chk_documents_law_jurisdiction) 도 거부하지만 422 로 명시 안내).
if proposed_material == "law" and not jur:
raise HTTPException(
status_code=422,
detail="법령(law) 승인은 jurisdiction 필수 — body.jurisdiction 으로 국가를 지정하세요 (기본값 없음)",
)
doc.material_type = proposed_material
doc.jurisdiction = jur
# 미러 동기화 1문 — jurisdiction 부여/정정 시 청크 country 동반 UPDATE
# (leg 간 국가 불일치 방지, plan A-2 계약. 단일 지점 = 본 승인 경로).
if jur:
from sqlalchemy import update as sa_update
from models.chunk import DocumentChunk
await session.execute(
sa_update(DocumentChunk)
.where(DocumentChunk.doc_id == doc.id)
.values(country=jur)
)
# user_tags append (중복 방지, normalize + dedup 통과)
if proposed_path:
+160 -6
View File
@@ -2,8 +2,9 @@
확정 결정:
- D-1 경로 = /api/eid/chat (main.py prefix=/api/eid + 본 라우터 POST /chat)
- D-2 mode 닫힌 어휘: daily(mac-mini-default) / deep(qwen-macbook). 클라는 mode 만 보냄 —
claude-cloud / auto 금지 (Literal 로 422 차단). 심층(deep) 모드 무게이트.
- D-2 mode 닫힌 어휘: daily / deep — 둘 다 mac-mini-default (맥북 백지화 2026-06-11,
맥미니 Qwen 27B 단일 호스트. deep = ReAct 자동검색 모드 구분). 클라는 mode 만 보냄 —
claude-cloud / auto 금지 (Literal 로 422 차단). 게이트 = alias 기준 자동 적용(무게이트 폐지).
- D-3 독립 /chat 라우트 (frontend) — 본 모듈은 백엔드 API 만.
- D-5 LLM 호출 = EidAIClient.call_stream 한 곳 (이드 egress 봉쇄 불변식 #5,
RouterBackend 직접 호출 금지).
@@ -18,24 +19,58 @@ backend 실패는 /api/search/ask 와 동일 shape 의 503 + error_reason 매핑
from __future__ import annotations
import asyncio
import json
from collections.abc import AsyncIterator
from typing import Annotated, Literal
import httpx
from fastapi import APIRouter, Depends
from fastapi.responses import JSONResponse, StreamingResponse
from pydantic import BaseModel, Field, field_validator, model_validator
from sqlalchemy.ext.asyncio import AsyncSession
from core.auth import get_current_user
from core.database import get_session
from core.utils import setup_logger
from eid import compose as eid_compose
from eid.ai import EidAIClient
from models.user import User
from services.llm.backends import BackendUnavailable
from services.llm.backends import BackendUnavailable, _router_url, get_backend
from services.search import llm_gate
from services.search.react_loop import agentic_ask_loop
logger = setup_logger("eid_chat")
router = APIRouter()
# ── ds-eid-ask-absorb P1: deep 모드 = ReAct 자동검색 (맥미니 Qwen 27B, 2026-06-11~) ──
# 비생성 reachability probe — router 도달만 확인(coarse). 27B(맥북) 자체 미가용은
# 첫 generate_with_tools 호출의 BackendUnavailable → mid-stream error envelope 로 커버
# (plan: probe 정밀도 불필요, TOCTOU 는 in-stream error 가 처리). ~2s 타임아웃·생성 슬롯 비점유.
_DEEP_PROBE_TIMEOUT = httpx.Timeout(connect=2.0, read=2.0, write=2.0, pool=2.0)
# heartbeat: ReAct 다회 tool call 시 수십초 무출력 → 프록시 idle timeout 차단.
# `{"phase":"ping"}` no-op 이벤트 (프론트 envelope 파서가 자연 스킵 — `: ping` comment 는
# POST SSE fetch 파서가 처리 보장 안 됨).
_HEARTBEAT_INTERVAL_S = 10.0
async def _probe_router_reachable() -> bool:
"""router(:8890) /v1/models GET — 도달 확인(비생성). 실패/비200 = 미가용."""
url = f"{_router_url().rstrip('/')}/v1/models"
try:
async with httpx.AsyncClient(timeout=_DEEP_PROBE_TIMEOUT) as client:
resp = await client.get(url)
return resp.status_code == 200
except Exception:
return False
def _sse(obj: dict) -> bytes:
"""SSE 이벤트 1건 — data: <json>\\n\\n. final_answer 는 OpenAI 호환 choices.delta.content
로, sources/phase 는 별 envelope 키로(프론트가 분기). model/usage 머신 메타 미포함."""
return b"data: " + json.dumps(obj, ensure_ascii=False).encode("utf-8") + b"\n\n"
class ChatMessage(BaseModel):
"""채팅 턴 1건. role=system 은 Literal 밖 → 422 (system 합본은 서버 compose 만 주입)."""
@@ -71,16 +106,130 @@ class ChatRequest(BaseModel):
return self
@router.get("/status")
async def eid_status(
user: Annotated[User, Depends(get_current_user)],
):
"""이드 backend 점유 상태 스냅샷 — GET /api/eid/status (UI 의 "대기 vs 고장" 구분용).
daily(맥미니 MLX) 의 DS 프로세스 내부 llm_gate 점유만 본다 — 외부 소비자
(맥미니 자체 derived-worker·Hermes 등)의 endpoint 점유는 미포착.
따라서 busy=true 는 확실(지금 줄이 있다), false 는 근사(외부 점유 가능성 잔존).
가벼움 보장: DB 0 / LLM 0 / 본문 로깅 0 — 폴링 대상으로 안전.
자동 fallback 판단 근거로 쓰지 않는다 (모드 전환 = 명시 버튼만, 정책).
"""
snap = llm_gate.gate_status()
inflight = bool(snap["inflight"])
waiters = int(snap["waiters"])
return {
"daily": {
"busy": inflight or waiters > 0,
"inflight": inflight,
"waiters": waiters,
}
}
def _backend_unavailable_response(body: ChatRequest, reason: str, backend_name: str) -> JSONResponse:
"""스트림 시작 전 27B 미가용 → ask 컨벤션과 동일 shape 503 (자동 fallback 0)."""
logger.warning(
"eid_chat backend_unavailable mode=%s turns=%d status=503 reason=%s",
body.mode, len(body.messages), reason,
)
return JSONResponse(
status_code=503,
content={
"error": "backend_unavailable",
"error_reason": reason,
"backend_requested": backend_name,
"detail": (
"심층 엔진(검색)이 일시적으로 응답할 수 없습니다. "
"잠시 후 다시 시도하거나 일상 모드로 물어보세요."
),
},
)
async def _eid_chat_deep(body: ChatRequest, session: AsyncSession) -> StreamingResponse | JSONResponse:
"""deep 모드 = ReAct 자동검색. ReAct(`tool_choice=auto`)가 검색 여부를 LLM 자율 판단 —
검색 불요 질문은 early-exit 으로 대화 답변. substrate(persona+rules+react_ask task)는
agentic_ask_loop 내부 compose("react_ask") 가 주입(evidence-first 자동 상속).
멀티턴 = 1단계는 마지막 user 메시지 단독 처리(agentic_ask_loop 가 query: str — history
미지원). 후속 질문 대명사 해소는 2단계 백로그.
"""
# ① 첫 SSE 바이트(=HTTP 200 확정) 전 비생성 probe — router 도달 실패 시 503 (재매핑 가능 구간)
if not await _probe_router_reachable():
return _backend_unavailable_response(body, "router_unreachable", "mac-mini-default")
query = body.messages[-1].content # 메시지 단독 처리 (마지막 user 턴)
backend = get_backend("mac-mini-default")
async def _stream() -> AsyncIterator[bytes]:
# ② phase:searching 방출 = HTTP 200 확정. 이후 미가용은 503 불가 → in-stream error.
yield _sse({"phase": "searching"})
task = asyncio.create_task(agentic_ask_loop(session, query, backend=backend))
try:
# heartbeat: task 미완 동안 ~10s 마다 ping (shield 로 wait_for 취소가 task 안 죽임)
while not task.done():
try:
await asyncio.wait_for(asyncio.shield(task), timeout=_HEARTBEAT_INTERVAL_S)
except asyncio.TimeoutError:
yield _sse({"phase": "ping"})
result = task.result() # BackendUnavailable 은 여기서 raise (mid-stream)
# final_answer = OpenAI 호환 1청크(프론트 기존 content 누적 경로 재사용)
yield _sse({"choices": [{"delta": {"content": result.final_answer}}]})
# 근거 = 별 envelope (citation 번호 없음 — 프론트가 순서 기반). partial = 근거 부족 표식
yield _sse({"eid_sources": result.sources, "partial": result.partial})
yield b"data: [DONE]\n\n"
logger.info(
"eid_chat deep ok turns=%d sources=%d partial=%s iters=%d",
len(body.messages), len(result.sources), result.partial, result.iterations,
)
except BackendUnavailable as exc:
# mid-stream 미가용(검색 중 AC 분리·뚜껑 닫힘) — 200 이미 송신, in-stream error envelope.
# error 뒤 [DONE] = 프론트 sawDone 로 '중단' 오경보 방지(명시 error notice 유지).
logger.warning(
"eid_chat deep mid-stream unavailable turns=%d reason=%s",
len(body.messages), exc.reason,
)
yield _sse({"phase": "error", "error_reason": exc.reason})
yield b"data: [DONE]\n\n"
except asyncio.CancelledError:
raise # 클라 disconnect — finally 가 task 정리
except Exception:
logger.exception("eid_chat deep stream failed turns=%d", len(body.messages))
yield _sse({"phase": "error", "error_reason": "deep_failed"})
yield b"data: [DONE]\n\n"
finally:
# 클라 disconnect 시 ReAct task 고아화 방지 — cancel + await(전파 완료 보장).
# 안 하면 27B 가 닫힌 연결 위해 수분 점유, router 동시성상 다음 검색 대기.
if not task.done():
task.cancel()
try:
await task
except (asyncio.CancelledError, Exception):
pass
return StreamingResponse(
_stream(),
media_type="text/event-stream",
headers={"Cache-Control": "no-store", "X-Accel-Buffering": "no"},
)
@router.post("/chat")
async def eid_chat(
body: ChatRequest,
user: Annotated[User, Depends(get_current_user)],
session: Annotated[AsyncSession, Depends(get_session)],
):
"""이드 채팅 — router SSE 스트리밍 pass-through.
"""이드 채팅 — daily = router SSE pass-through(대화) / deep = ReAct 자동검색(근거).
503 경로 (둘 다 자동 fallback 없음):
503 경로 (모두 자동 fallback 없음):
- substrate_degraded: rules.md 부재 (D-6 fail-closed, 채팅 진행 거부)
- backend_unavailable: 스트림 시작 전 backend 실패 (ask 컨벤션과 동일 shape)
- backend_unavailable: 스트림 시작 전 backend 실패 (daily/deep 공통, ask 컨벤션 shape)
"""
# D-6: rules 부재 = fail-closed. 채팅은 안전·정책 가드 없이 진행하지 않는다(배너 X).
if not eid_compose.rules_present():
@@ -99,6 +248,11 @@ async def eid_chat(
},
)
# deep = ReAct 자동검색 (별 흐름 — probe + 동기 ReAct → SSE 변환)
if body.mode == "deep":
return await _eid_chat_deep(body, session)
# daily = 순수 대화 SSE pass-through (기존)
system = eid_compose.compose("eid_chat", task="")
client = EidAIClient()
stream = client.call_stream(
+192
View File
@@ -0,0 +1,192 @@
"""처리 머신 보드 API — /api/queue/* (plan ds-processing-ui-6an → ds-board-engines-1).
- GET /overview: 홈 stage 평면 테이블을 "머신 관점 보드(누가 일하나)"로 — 집계
로직은 services/queue_overview.py (순수 판정부 분리). 응답 스키마는 FE 와
계약 고정. 응답에 raw 모델명 노출 금지 — 머신 label 만 (엔진/모델 표기는
FE 정적 맵 책임).
- GET /failed + POST /retry|/skip: 실패 처리 (ds-board-engines-1) — 영구 실패
(자동 재시도 3회 소진)의 유일한 사용자 조치 경로. 일괄 조치는 FE 가 그룹의
id 목록을 모아 보낸다 (서버측 패턴 매칭 없음 — raw 식별자/패턴 미수신).
"""
from datetime import datetime
from typing import Annotated, Literal
from fastapi import APIRouter, Depends
from pydantic import BaseModel, Field
from sqlalchemy.ext.asyncio import AsyncSession
from core.auth import get_current_user
from core.database import get_session
from models.user import User
from services.queue_overview import (
build_overview,
fetch_failed_items,
retry_failed,
skip_failed,
)
router = APIRouter()
class CurrentItem(BaseModel):
"""머신이 지금 처리 중인 문서 (최대 2건)."""
document_id: int
title: str
stage: str
class MachineCard(BaseModel):
"""머신 카드 — stage 귀속 합산 + 완료 실적(summarize 는 풀 분리) + state."""
key: Literal["gpu", "macmini", "macbook"]
label: str
state: Literal["active", "deferred", "idle"]
stages: list[str]
pending: int
processing: int
failed: int
done_1h: int
done_today: int
deferred_pending: int
current: list[CurrentItem]
class SummarizeEta(BaseModel):
"""summarize 풀 ETA — done > inflow 일 때만 eta_minutes 산출."""
pending: int
done_rate_1h: int
inflow_rate_1h: int
eta_minutes: int | None
class MachineDone(BaseModel):
"""머신 1대의 summarize 완료 실적 (분담 표시용)."""
done_1h: int
done_today: int
class SummarizeByMachine(BaseModel):
"""summarize 풀의 머신별 완료 실적 분담 — 보드 레인의 '맥미니 vs 맥북'
오프로드 가시화용. rows_to_summarize_split 이 이미 계산하던 값의 노출
(ds-board-merged A-1, 신규 수집 SQL 0)."""
macmini: MachineDone
macbook: MachineDone
class TrendBucket(BaseModel):
"""summarize 24h 추이 버킷 — hour 는 KST "HH:00" 라벨."""
hour: str
inflow: int
done: int
class Totals(BaseModel):
"""전 stage 합계."""
pending: int
processing: int
failed: int
class StageRow(BaseModel):
"""단계별 현황 행 — 흐름 노드/상세 패널용.
done_1h/created_1h = 처리율·유입률 (유입 우세 판정 + ETA 의 FE 재료,
ds-board-engines-1 추가 — 수집 SQL 에 이미 있던 값의 노출).
"""
stage: str
pending: int
processing: int
failed: int
done_1h: int
created_1h: int
done_today: int
oldest_pending_age_sec: int | None
class QueueOverviewResponse(BaseModel):
machines: list[MachineCard]
stages: list[StageRow]
summarize_eta: SummarizeEta
summarize_by_machine: SummarizeByMachine
trend_24h: list[TrendBucket]
totals: Totals
class FailedItem(BaseModel):
"""영구 실패 행 — 실패 드로어 표시 단위."""
id: int
stage: str
document_id: int
title: str
attempts: int
max_attempts: int
error_message: str | None
failed_at: datetime | None
class FailedListResponse(BaseModel):
items: list[FailedItem]
total: int
class QueueActionRequest(BaseModel):
"""재시도/건너뛰기 대상 — 실패 행 id 목록 (FE 가 그룹핑 후 전달)."""
ids: list[int] = Field(min_length=1, max_length=300)
class RetryResponse(BaseModel):
requested: int
retried: int
not_retried: int
class SkipResponse(BaseModel):
requested: int
skipped: int
not_skipped: int
@router.get("/overview", response_model=QueueOverviewResponse)
async def get_queue_overview(
user: Annotated[User, Depends(get_current_user)],
session: Annotated[AsyncSession, Depends(get_session)],
):
"""머신 관점 처리 보드 + summarize ETA 집계 (라이브 계산, 신규 테이블 0)"""
return QueueOverviewResponse.model_validate(await build_overview(session))
@router.get("/failed", response_model=FailedListResponse)
async def get_failed_items(
user: Annotated[User, Depends(get_current_user)],
session: Annotated[AsyncSession, Depends(get_session)],
):
"""영구 실패 행 목록 (문서 제목 포함, 최대 300건)"""
items = await fetch_failed_items(session)
return FailedListResponse(
items=[FailedItem.model_validate(i) for i in items],
total=len(items),
)
@router.post("/retry", response_model=RetryResponse)
async def retry_failed_items(
body: QueueActionRequest,
user: Annotated[User, Depends(get_current_user)],
session: Annotated[AsyncSession, Depends(get_session)],
):
"""실패 행 재시도 — attempts 리셋 + pending 복귀.
not_retried = 같은 (문서, 단계) 의 active 행 충돌(uq_queue_active) 또는
이미 failed 가 아닌 행 (중복 클릭 등) — 건드리지 않고 건수만 보고.
"""
return RetryResponse.model_validate(await retry_failed(session, body.ids))
@router.post("/skip", response_model=SkipResponse)
async def skip_failed_items(
body: QueueActionRequest,
user: Annotated[User, Depends(get_current_user)],
session: Annotated[AsyncSession, Depends(get_session)],
):
"""실패 행 건너뛰기 — completed 마킹(payload.skipped_by_user) + 연쇄 없음"""
return SkipResponse.model_validate(await skip_failed(session, body.ids))
+34
View File
@@ -12,6 +12,7 @@
import asyncio
import hmac
import time
from datetime import date
from typing import Annotated, Literal
from fastapi import APIRouter, BackgroundTasks, Depends, Header, Query
@@ -31,6 +32,8 @@ from services.search.fusion_service import DEFAULT_FUSION
from services.search.grounding_check import check as grounding_check
from services.search.refusal_gate import RefusalDecision, decide as refusal_decide
from services.search import query_rewriter
from services.search.retrieval_service import AxisFilter
from services.search.result_decorate import compute_facets, decorate_version_status
from services.search.search_pipeline import PipelineResult, run_search
from services.search.synthesis_service import SynthesisResult, synthesize
from services.search.verifier_service import VerifierResult, verify
@@ -70,6 +73,14 @@ class SearchResult(BaseModel):
# PR-RAG-Time-1: freshness decay 디버그 메타. apply_freshness_decay 가 채움.
# 비적용 row 도 채워짐(freshness_policy=None). base_score 는 항상 보존.
freshness_debug: dict | None = None
# 안전 자료실 C-1: 분류 축 메타 (3 leg SELECT 에서 채움 — additive, ranking 무관).
# D-1 UI 결과 카드 유형별 렌더 + 해외 법령(B-5) 가동 시 국가 무표지 혼재 차단의 선행 조건.
material_type: str | None = None
jurisdiction: str | None = None
published_date: date | None = None
# 안전 자료실 C-1 후속: 법령 버전 상태(legal_meta.version_status) — wrapper 1회 decorate.
# law 결과만 채워짐(legal_meta 위성), 그 외/무매핑 law = None. D-1 버전 뱃지 선행.
version_status: str | None = None
# ─── Phase 0.4: 디버그 응답 스키마 ─────────────────────────
@@ -101,6 +112,9 @@ class SearchResponse(BaseModel):
query: str
mode: str
debug: SearchDebug | None = None
# 안전 자료실 C-1 후속: facets=true 일 때만 채워짐(미요청=None, byte 불변).
# top-K 결과 내 분류 축 분포 라벨 {axis: {label: count}}.
facets: dict[str, dict[str, int]] | None = None
def _to_debug_candidates(rows: list[SearchResult], n: int = 20) -> list[DebugCandidate]:
@@ -205,9 +219,23 @@ async def search(
"분리용. production 검색에는 사용 금지 (latency 큼)."
),
),
material_type: str | None = Query(
None, description="안전 자료실 C-1: 자료유형 필터 CSV (law,paper,incident,...). material_type = ANY"),
jurisdiction: str | None = Query(
None, description="안전 자료실 C-1: 관할 필터 (KR/US/EU/JP/GB/INT)"),
year_from: int | None = Query(None, ge=1900, le=2100, description="published_date 연도 하한 (NULL=created_at fallback)"),
year_to: int | None = Query(None, ge=1900, le=2100, description="published_date 연도 상한"),
facets: bool = Query(False, description="안전 자료실 C-1 후속: top-K 결과 분류 축 분포(material_type/jurisdiction/version_status)를 응답 facets 에 집계. 미지정=계산/노출 0"),
):
"""문서 검색 — FTS + ILIKE + 벡터 결합 (Phase 3.1 이후 run_search wrapper)"""
try:
axis = AxisFilter(
material_types=[m.strip() for m in material_type.split(",") if m.strip()]
if material_type else None,
jurisdiction=jurisdiction,
year_from=year_from,
year_to=year_to,
)
pr = await run_search(
session,
q,
@@ -223,6 +251,7 @@ async def search(
rewrite_backend=rewrite_backend,
corpus_variant=corpus_variant,
exact_knn=exact_knn,
axis=axis,
)
except ValueError as e:
# _resolve_backend / _resolve_reranker / _resolve_rewrite_backend / _resolve_corpus_variant unknown slug → HTTP 400
@@ -313,12 +342,17 @@ async def search(
debug_obj = _build_search_debug(pr) if debug else None
# 안전 자료실 C-1 후속 — wrapper decoration (검색 코어 무접촉, ranking 무관)
await decorate_version_status(session, pr.results) # 법령 결과에 version_status
facets_obj = compute_facets(pr.results) if facets else None
return SearchResponse(
results=pr.results,
total=len(pr.results),
query=q,
mode=pr.mode,
debug=debug_obj,
facets=facets_obj,
)
+28
View File
@@ -158,6 +158,17 @@ class Settings(BaseModel):
# 업로드 한도 (authoritative policy)
upload: UploadConfig = UploadConfig()
# 생성 LLM 홀드 (2026-06-11): config.yaml pipeline.held_stages 에 든 이름의
# 컨슈머/워커는 claim 자체를 하지 않는다 (attempts 미소모, pending 적체 = 의도).
# 유효 키 = 큐 stage 명(classify/summarize/deep_summary) + cron/컨슈머 키(digest,
# briefing, study_explanation, study_session_analysis, study_memo_card).
# 빈 리스트 = 무동작 (기존 동작 그대로).
pipeline_held_stages: list[str] = []
# mlx gate 동시 실행 상한 (2026-06-12, config.yaml pipeline.mlx_gate_concurrency).
# 1 = 구 single-inference 동작. 2 = continuous batching 활용 (llm_gate docstring 참조).
mlx_gate_concurrency: int = 1
# PR-MacMini-Derived-Worker-1: study explanation owner = Mac mini
# GPU 측은 false 로 설정 (.env), explanation 분기 skip guard 트리거.
study_explanation_enabled: bool = True
@@ -244,6 +255,21 @@ def load_settings() -> Settings:
)
)
pipeline_held_stages: list[str] = []
mlx_gate_concurrency = 1
if config_path.exists() and raw and "pipeline" in raw:
held_raw = (raw.get("pipeline") or {}).get("held_stages") or []
# 스칼라(문자열) 오기입 시 char-split 방지 — 단일 항목 리스트로 수용.
if not isinstance(held_raw, (list, tuple)):
held_raw = [held_raw]
pipeline_held_stages = [str(s) for s in held_raw]
try:
mlx_gate_concurrency = max(
1, int((raw.get("pipeline") or {}).get("mlx_gate_concurrency", 1))
)
except (TypeError, ValueError):
mlx_gate_concurrency = 1
taxonomy = raw.get("taxonomy", {}) if config_path.exists() and raw else {}
document_types = raw.get("document_types", []) if config_path.exists() and raw else []
upload_cfg = (
@@ -272,6 +298,8 @@ def load_settings() -> Settings:
study_explanation_enabled=study_explanation_enabled,
study_card_extract_enabled=study_card_extract_enabled,
internal_worker_token=internal_worker_token,
pipeline_held_stages=pipeline_held_stages,
mlx_gate_concurrency=mlx_gate_concurrency,
)
+10 -7
View File
@@ -29,16 +29,19 @@ import httpx
from ai.client import AIClient
from services.llm.backends import (
MAC_MINI_DEFAULT,
QWEN_MACBOOK,
BackendUnavailable,
_router_url, # router URL 단일 출처 재사용 (settings → env LLM_ROUTER_URL → MVP default)
)
from services.search.llm_gate import Priority, acquire_mlx_gate
# 이드 채팅 mode → router alias 닫힌 매핑 (D-2). 클라는 mode 만 보냄 — claude-cloud/auto 금지.
# 2026-06-11 맥북 백지화: deep 도 mac-mini-default (맥미니 Qwen 27B 단일 호스트).
# mode 구분은 유지 — deep = ReAct 자동검색 경로(모델이 아니라 동작이 다름).
# 게이트는 alias==MAC_MINI_DEFAULT 조건이라 deep 도 자동으로 mlx gate 적용
# (llm_gate "예외 없이 gate 획득 필수" invariant 충족 — 구 무게이트는 맥북 예외였음).
_CHAT_ALIAS: dict[str, str] = {
"daily": MAC_MINI_DEFAULT, # router tier_b → Mac mini :8801 gemma-4-26b
"deep": QWEN_MACBOOK, # router named upstream → M5 Max Qwen3.6-27B (무게이트, D-2)
"daily": MAC_MINI_DEFAULT, # router tier_b → Mac mini :8801
"deep": MAC_MINI_DEFAULT, # 맥북 폐기로 동일 upstream — ReAct 검색 모드 구분만 유지
}
# read 는 per-chunk 적용이라 MacBook wake(24s)+토큰 생성 간격 커버. connect 는 내부 router 라 짧게.
@@ -161,10 +164,10 @@ class EidAIClient(AIClient):
_rewrite_sse_line 으로 model 치환(mode 어휘)·usage 제거만 하고 프레이밍은 보존.
취소/disconnect 시 AsyncExitStack 이 response·client 정리(upstream 닫힘 보장).
daily(mac-mini-default)는 Mac mini MLX 단일 inference 영구 룰(llm_gate docstring
"예외 없이 gate 획득 필수")에 따라 acquire_mlx_gate(FOREGROUND) 안에서 스트리밍 —
RouterBackend 의 requires_gate=True 와 동일한 client-side mutex 효과.
deep(qwen-macbook)은 별 endpoint 라 무게이트 (D-2, RouterBackend 동형).
daily/deep 모두 mac-mini-default(2026-06-11 맥북 백지화) → Mac mini MLX 단일
inference 영구 룰(llm_gate docstring "예외 없이 gate 획득 필수")에 따라
acquire_mlx_gate(FOREGROUND) 안에서 스트리밍 — 게이트 조건이 alias 기준이라
deep 도 자동 적용 (구 무게이트는 맥북 별 endpoint 시절 예외였음).
중계 전체(업스트림 진입~종료)는 asyncio.timeout(_STREAM_DEADLINE_S) wall-clock
deadline 안 — llm_gate 계약 "timeout 은 gate 안쪽" 준수(gate 대기엔 미적용).
+11 -3
View File
@@ -22,6 +22,7 @@ from api.events import router as events_router
from api.library import router as library_router
from api.memos import router as memos_router
from api.news import router as news_router
from api.queue_overview import router as queue_overview_router
from api.search import router as search_router
from api.setup import router as setup_router
from api.study_question_progress import router as study_question_progress_router
@@ -52,15 +53,15 @@ async def lifespan(app: FastAPI):
from workers.dedup_reconcile import run as dedup_reconcile_run
from workers.digest_worker import run as global_digest_run
from workers.file_watcher import watch_inbox
from workers.law_monitor import run as law_monitor_run
from workers.mailplus_archive import run as mailplus_run
from workers.statute_collector import run as statute_run
from workers.news_collector import run as news_collector_run
from workers.fulltext_worker import reconcile_unresolved as fulltext_reconcile_run
from workers.kosha_collector import run as kosha_collector_run
from workers.csb_collector import run as csb_collector_run
from workers.api_standards_collector import run as api_standards_run
from workers.ccps_collector import run as ccps_collector_run
from workers.queue_consumer import consume_queue, consume_markdown_queue
from workers.queue_consumer import consume_queue, consume_fast_queue, consume_markdown_queue
from workers.study_queue_consumer import consume_study_queue
from workers.study_session_queue_consumer import consume_study_session_queue
from workers.study_memo_card_jobs_consumer import consume_study_memo_card_queue
@@ -94,6 +95,9 @@ async def lifespan(app: FastAPI):
# 대형 PDF split 변환(수십 분)이 메인 consume_queue 를 점유해 전 파이프라인을
# stall 시키던 문제 제거. max_instances=1(기본) 으로 동시 marker 변환 2건은 방지.
scheduler.add_job(consume_markdown_queue, "interval", minutes=1, id="markdown_consumer")
# 2026-06-12 fast-consumer split: embed/chunk(건당 <1s)를 LLM 사이클에서 분리 —
# classify(~190s×3)가 사이클을 점유해 벡터 적재가 굶던 구조 캡 해소 (markdown 선례).
scheduler.add_job(consume_fast_queue, "interval", minutes=1, id="fast_queue_consumer")
scheduler.add_job(watch_inbox, "interval", minutes=5, id="file_watcher")
scheduler.add_job(cleanup_orphan_uploads, "interval", minutes=10, id="upload_cleanup")
# PR-4: study_questions 자동 임베딩 (status='none/failed/stale' 행을 batch=10 처리).
@@ -116,7 +120,9 @@ async def lifespan(app: FastAPI):
# safety > law > manual 우선순위로 25건씩. 6720 레거시 → 야간당 ~150건 → 약 45일 소화.
scheduler.add_job(tier_backfill_run, "interval", minutes=30, id="tier_backfill")
# 일일 스케줄 (KST)
scheduler.add_job(law_monitor_run, CronTrigger(hour=7, timezone=KST), id="law_monitor")
# statute_collector = 구 law_monitor 대체 (safety-library-1 B-1 PR②) — poll→ingest→
# 생애주기 잡(버전 시리즈 승격·supersede·레거시 스윕·repeal) 통째 (R8-B1).
scheduler.add_job(statute_run, CronTrigger(hour=7, timezone=KST), id="statute_collector")
scheduler.add_job(mailplus_run, CronTrigger(hour=7, timezone=KST), id="mailplus_morning")
scheduler.add_job(mailplus_run, CronTrigger(hour=18, timezone=KST), id="mailplus_evening")
scheduler.add_job(daily_digest_run, CronTrigger(hour=20, timezone=KST), id="daily_digest")
@@ -183,6 +189,8 @@ app.include_router(events_router, prefix="/api/events", tags=["events"])
app.include_router(dashboard_router, prefix="/api/dashboard", tags=["dashboard"])
app.include_router(library_router, prefix="/api/library", tags=["library"])
app.include_router(news_router, prefix="/api/news", tags=["news"])
# 처리 머신 보드 (plan ds-processing-ui-6an) — GET /api/queue/overview
app.include_router(queue_overview_router, prefix="/api/queue", tags=["queue"])
app.include_router(digest_router, prefix="/api/digest", tags=["digest"])
app.include_router(briefing_router, prefix="/api/briefing", tags=["briefing"])
app.include_router(audio_router, prefix="/api/audio", tags=["audio"])
+5
View File
@@ -14,6 +14,11 @@ from sqlalchemy.orm import Mapped, mapped_column
from core.database import Base
# FK("users.id") 해석에 users 테이블 메타데이터 필요 — fastapi 앱은 어차피 전 모델을
# import 하지만, CLI 단독 실행(queue_drain 등)은 본 모듈만 끌어와 INSERT 시
# "could not find table 'users'" 로 실패했다 (2026-06-12 drain 로그 실측). 명시 import.
from models.user import User # noqa: F401
class AnalyzeEvent(Base):
__tablename__ = "analyze_events"
+12 -2
View File
@@ -1,9 +1,9 @@
"""documents 테이블 ORM"""
from datetime import datetime
from datetime import date, datetime
from pgvector.sqlalchemy import Vector
from sqlalchemy import BigInteger, Boolean, DateTime, Enum, ForeignKey, Integer, String, Text
from sqlalchemy import BigInteger, Boolean, Date, DateTime, Enum, ForeignKey, Integer, String, Text
from sqlalchemy.dialects.postgresql import JSONB
from sqlalchemy.orm import Mapped, mapped_column
@@ -146,6 +146,16 @@ class Document(Base):
# /accept-suggestion 승인 시에만 category / user_tags 반영 (자동 전이 금지)
ai_suggestion: Mapped[dict | None] = mapped_column(JSONB)
# === 안전 자료실 분류 축 (plan safety-library-1, migrations 340~345) ===
# 자료유형 — law/paper/book/incident/manual/standard/guide (TEXT+CHECK, enum 아님).
# 수집기 ingest 시점 deterministic 부여 (classify-skip 경로 다수 — classify_worker 의존 금지).
# AI 라우팅(subject_domain) 매칭 키 사용 금지 (axis separation — category 와 동일 불변식).
material_type: Mapped[str | None] = mapped_column(Text)
# 관할 — KR/US/EU/JP/GB/INT. law 는 CHECK 로 jurisdiction NOT NULL 구조 강제 (migration 344).
jurisdiction: Mapped[str | None] = mapped_column(Text)
# 유형별 대표 날짜 — 법령=COALESCE(시행일, 공포일) / 논문=발행일 / 재해=발생일
published_date: Mapped[date | None] = mapped_column(Date)
# PR-B B-1: summary_triage (4B, 상시) / summary_deep (26B, 에스컬레이션) 분할 산출
ai_tldr: Mapped[str | None] = mapped_column(Text) # ≤60자 TL;DR
ai_bullets: Mapped[list | None] = mapped_column(JSONB) # 3~5개 핵심 bullets
+73
View File
@@ -0,0 +1,73 @@
"""legal_acts / legal_meta 테이블 ORM — 법령 레지스트리(워치리스트 겸) + 버전 위성
plan: safety-library-1 (migrations 346~347).
- legal_acts = 폴링 순회 대상 목록이 테이블 (news_sources 패턴의 법령판).
KOSHA GUIDE(비법령)·KGS Code(watch-폴더 단독 트랙) 비대상.
- legal_meta = 법령 문서 1버전(또는 별표·해석례 1) 1, documents 1:0..1 위성.
version_status 전이는 statute_collector 일일 잡이 유일한 코드 지점
( 버전 pending 적재 잡이 승격·supersede·repeal 트랜잭션 처리).
"""
from datetime import date, datetime
from sqlalchemy import BigInteger, Boolean, Date, DateTime, ForeignKey, Text, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column
from core.database import Base
class LegalAct(Base):
__tablename__ = "legal_acts"
# 'kr-law:{법령ID}' / 'us-cfr:29-1910' 형식. KGS 는 시드 비대상 (R3-M5).
family_id: Mapped[str] = mapped_column(Text, primary_key=True)
# 어댑터 상수 고정값 — 파싱 결과에서 추론 금지 (코어가 적재 직전 assert)
jurisdiction: Mapped[str] = mapped_column(Text, nullable=False)
# statute(법률) / decree(시행령) / rule(시행규칙·부령) / admin_rule(고시·예규) / code(법정 위임 상세기준)
law_level: Mapped[str] = mapped_column(Text, nullable=False)
title: Mapped[str] = mapped_column(Text, nullable=False)
title_ko: Mapped[str | None] = mapped_column(Text)
# 법률 → 시행령 → 시행규칙 계층
parent_family_id: Mapped[str | None] = mapped_column(ForeignKey("legal_acts.family_id"))
# 법령ID / CFR part / CELEX / e-Gov law_id 등 소스 고유 식별자
native_id: Mapped[str] = mapped_column(Text, nullable=False)
# 'law.go.kr' / 'ecfr' / 'cellar' / 'egov_v2' / 'leg_gov_uk'
source_api: Mapped[str] = mapped_column(Text, nullable=False)
# 시드 26개 전부 true — '우선순위'는 정렬일 뿐 watch 제외 아님 (R3-B1)
watch: Mapped[bool] = mapped_column(Boolean, nullable=False, default=True)
poll_cycle: Mapped[str] = mapped_column(Text, nullable=False, default="daily")
# 변경이력 폴링 워터마크 — 파싱 검증 통과 후에만 영속
watermark: Mapped[str | None] = mapped_column(Text)
# 어댑터는 폐지 감지 마킹만, repealed 전이는 일일 잡 (R3-M3)
repeal_detected_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), default=datetime.now
)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), default=datetime.now, onupdate=datetime.now
)
class LegalMeta(Base):
__tablename__ = "legal_meta"
__table_args__ = (
# 버전 dedup 구조 강제 — annex 는 version_key='MST|별표N' 합성형 (R3-M4)
UniqueConstraint("family_id", "law_doc_kind", "version_key", name="uq_legal_meta_version"),
)
document_id: Mapped[int] = mapped_column(
BigInteger, ForeignKey("documents.id", ondelete="CASCADE"), primary_key=True
)
family_id: Mapped[str] = mapped_column(
ForeignKey("legal_acts.family_id"), nullable=False
)
# primary(본문) / annex(별표·서식) / interpretation(해석례)
law_doc_kind: Mapped[str] = mapped_column(Text, nullable=False, default="primary")
version_key: Mapped[str] = mapped_column(Text, nullable=False)
promulgation_date: Mapped[date | None] = mapped_column(Date)
effective_date: Mapped[date | None] = mapped_column(Date)
# pending → current → superseded / repealed. 전이는 일일 잡 단일 지점, KST 기준.
version_status: Mapped[str] = mapped_column(Text, nullable=False, default="pending")
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), default=datetime.now
)
+9
View File
@@ -53,3 +53,12 @@ class NewsSource(Base):
name="source_channel"),
default="news",
)
# ── 안전 자료실 분류 축 (plan safety-library-1 A-2, migrations 352~355) ──
# 자료유형 기본값 — documents.material_type 으로 ingest 시점 전파 (NULL=비대상).
# jurisdiction 은 별도 컬럼 없이 country 전파, 단 paper 는 코드에서 NULL 강제.
material_type: Mapped[str | None] = mapped_column(Text)
# extract_meta.license 주입용 — kogl/ogl/public_domain/proprietary/unknown.
# 미확정 = 보수적(unknown + redistribute=false), 근거 확보 시 완화.
license_scheme: Mapped[str | None] = mapped_column(Text)
license_redistribute: Mapped[bool | None] = mapped_column(Boolean)
+7 -2
View File
@@ -15,11 +15,12 @@ from sqlalchemy import text
from core.database import async_session
from core.utils import setup_logger
from services.search.license_filter import restricted_exclude_sql
logger = setup_logger("briefing_loader")
_NEWS_WINDOW_SQL = text("""
_NEWS_WINDOW_SQL = text(f"""
SELECT
d.id,
d.title,
@@ -41,6 +42,8 @@ _NEWS_WINDOW_SQL = text("""
AND d.created_at < :window_end
AND d.embedding IS NOT NULL
AND d.ai_summary IS NOT NULL
-- 안전 자료실 B-4: licensed_restricted 발행 차단 (digest 동일 공유 술어, 경로 일관성)
AND {restricted_exclude_sql("d")}
""")
@@ -49,7 +52,7 @@ _SOURCE_COUNTRY_SQL = text("""
""")
_HISTORICAL_CANDIDATES_SQL = text("""
_HISTORICAL_CANDIDATES_SQL = text(f"""
SELECT
d.id,
d.title,
@@ -63,6 +66,8 @@ _HISTORICAL_CANDIDATES_SQL = text("""
AND d.created_at < :hist_end
AND d.embedding IS NOT NULL
AND d.ai_summary IS NOT NULL
-- 안전 자료실 B-4: licensed_restricted 발행 차단 (공유 술어)
AND {restricted_exclude_sql("d")}
""")
+5 -1
View File
@@ -15,11 +15,12 @@ from sqlalchemy import text
from core.database import async_session
from core.utils import setup_logger
from services.search.license_filter import restricted_exclude_sql
logger = setup_logger("digest_loader")
_NEWS_WINDOW_SQL = text("""
_NEWS_WINDOW_SQL = text(f"""
SELECT
d.id,
d.title,
@@ -41,6 +42,9 @@ _NEWS_WINDOW_SQL = text("""
AND d.created_at < :window_end
AND d.embedding IS NOT NULL
AND d.ai_summary IS NOT NULL
-- 안전 자료실 B-4: licensed_restricted 발행 차단 (모든 경로 공유 술어 = license_filter).
-- news 채널엔 현재 restricted 부재 = 방어적 게이트(미래 유료 news 소스 대비, 경로 누락 방지).
AND {restricted_exclude_sql("d")}
""")
+524
View File
@@ -0,0 +1,524 @@
"""처리 머신 보드 + ETA 집계 (plan ds-processing-ui-6an, 안2+안5/6).
GET /api/queue/overview 집계 로직. 모든 수치는 기존 processing_queue /
documents 컬럼에서 라이브 계산 신규 테이블/마이그레이션 0 (HARD 제약).
구조: SQL 수집부(build_overview 내부 5쿼리) 판정부(순수 함수) 분리.
판정부(rows_to_* / build_machines / build_summarize_eta / build_trend /
build_totals / compute_eta_minutes) DB 없이 단위테스트 가능.
귀속 규칙 (단일 진실):
- stagemachine 정적 : gpu = extract/embed/chunk/markdown/preview/thumbnail/
fulltext/stt · macmini = classify/summarize · macbook = deep_summary
(, settings.ai.deep 부재 deep_summary macmini 귀속).
- summarize (pool): pending/processing/failed macmini 귀속이되, 완료
실적(done_*) documents.ai_model_version 조인으로 분리 'qwen-macbook'
이면 macbook 실적, 아니면 macmini 실적.
- deferred_pending(payload.deferred_until 미래) macbook 카드 귀속
(보류 = 맥북 불가 신호).
"""
from datetime import datetime, timedelta
from posixpath import basename
from zoneinfo import ZoneInfo
from sqlalchemy import bindparam, text
from sqlalchemy.ext.asyncio import AsyncSession
from core.config import settings
KST = ZoneInfo("Asia/Seoul")
# 내부 판별용 alias — 응답에 raw 모델명 노출 금지, 머신 label 만 노출.
_MACBOOK_MODEL_ALIAS = "qwen-macbook"
# stage→machine 정적 맵 재료 (선언 순서 = 카드 stages 표시 순서)
_GPU_STAGES = (
"extract", "embed", "chunk", "markdown",
"preview", "thumbnail", "fulltext", "stt",
)
_MACMINI_STAGES = ("classify", "summarize")
_MACBOOK_STAGES = ("deep_summary",)
_STAGE_ORDER = _GPU_STAGES + _MACMINI_STAGES + _MACBOOK_STAGES
_MACHINE_KEYS = ("gpu", "macmini", "macbook")
_MACHINE_LABELS = {
"gpu": "GPU 서버",
"macmini": "맥미니",
"macbook": "맥북 M5 Max",
}
# 머신 카드당 current 표시 상한
_CURRENT_LIMIT = 2
def stage_machine_map(deep_enabled: bool) -> dict[str, str]:
"""stage → machine key 맵. deep 슬롯 부재 시 deep_summary 는 macmini 귀속."""
mapping: dict[str, str] = {}
for s in _GPU_STAGES:
mapping[s] = "gpu"
for s in _MACMINI_STAGES:
mapping[s] = "macmini"
for s in _MACBOOK_STAGES:
mapping[s] = "macbook" if deep_enabled else "macmini"
return mapping
def _zero_stage() -> dict:
return {
"pending": 0, "processing": 0, "failed": 0,
"done_1h": 0, "done_today": 0, "done_15m": 0,
"deferred_pending": 0, "created_1h": 0, "oldest_pending_at": None,
}
def rows_to_stage_stats(rows) -> dict[str, dict]:
"""stage×status 집계 쿼리 행 → {stage: {pending, ..., created_1h}} 변환."""
stats: dict[str, dict] = {}
for row in rows:
stats[row[0]] = {
"pending": int(row[1] or 0),
"processing": int(row[2] or 0),
"failed": int(row[3] or 0),
"done_1h": int(row[4] or 0),
"done_today": int(row[5] or 0),
"done_15m": int(row[6] or 0),
"deferred_pending": int(row[7] or 0),
"created_1h": int(row[8] or 0),
"oldest_pending_at": row[9] if len(row) > 9 else None,
}
return stats
def rows_to_summarize_split(rows) -> dict[str, dict]:
"""summarize 완료 실적 분리 쿼리 행 → {"macbook"|"macmini": {done_*}}.
is_macbook = documents.ai_model_version 'qwen-macbook' 인지 (내부 판별 전용).
"""
split = {
"macbook": {"done_1h": 0, "done_today": 0, "done_15m": 0},
"macmini": {"done_1h": 0, "done_today": 0, "done_15m": 0},
}
for row in rows:
key = "macbook" if row[0] else "macmini"
split[key]["done_1h"] += int(row[1] or 0)
split[key]["done_today"] += int(row[2] or 0)
split[key]["done_15m"] += int(row[3] or 0)
return split
def display_title(row: dict) -> str:
"""표시용 제목 — title > original_filename > file_path basename > 문서 id."""
if row.get("title"):
return row["title"]
if row.get("original_filename"):
return row["original_filename"]
if row.get("file_path"):
return basename(row["file_path"].rstrip("/"))
return f"문서 #{row['document_id']}"
def build_machines(
stage_stats: dict[str, dict],
summarize_split: dict[str, dict],
current_rows: list[dict],
*,
deep_enabled: bool,
) -> list[dict]:
"""머신 카드 3장 (gpu / macmini / macbook) 구성 — 귀속 규칙의 판정부."""
smap = stage_machine_map(deep_enabled)
def g(stage: str, field: str) -> int:
return stage_stats.get(stage, {}).get(field, 0)
# current 귀속: processing 행을 머신별 최대 2건 (summarize processing → macmini)
current_by_machine: dict[str, list[dict]] = {k: [] for k in _MACHINE_KEYS}
for row in current_rows:
machine = smap.get(row["stage"])
if machine and len(current_by_machine[machine]) < _CURRENT_LIMIT:
current_by_machine[machine].append({
"document_id": row["document_id"],
"title": display_title(row),
"stage": row["stage"],
})
machines = []
for key in _MACHINE_KEYS:
stages = [s for s in _STAGE_ORDER if smap[s] == key]
pending = sum(g(s, "pending") for s in stages)
processing = sum(g(s, "processing") for s in stages)
failed = sum(g(s, "failed") for s in stages)
# 완료 실적: summarize 는 풀이라 stage 합산에서 제외하고 split 로 귀속
done_1h = sum(g(s, "done_1h") for s in stages if s != "summarize")
done_today = sum(g(s, "done_today") for s in stages if s != "summarize")
done_15m = sum(g(s, "done_15m") for s in stages if s != "summarize")
if key in summarize_split:
done_1h += summarize_split[key]["done_1h"]
done_today += summarize_split[key]["done_today"]
done_15m += summarize_split[key]["done_15m"]
# 보류 백오프 = 맥북 불가 신호 → macbook 카드 귀속 (deep 슬롯 유무 무관)
deferred_pending = (
g("summarize", "deferred_pending") + g("deep_summary", "deferred_pending")
if key == "macbook" else 0
)
# state 판정 — 우선순위: 가동 > 보류 > 대기 (사용자 피드백 2026-06-11).
# 일하고 있으면(처리 중 또는 최근 15분 완료) 백오프 잔여가 있어도 "가동" —
# 보류 건수는 카드의 deferred_pending 라인이 따로 보여준다. "보류" 칩은
# 실제로 일이 멈춰 있고 백오프만 쌓인 상태(sleep/불가 지속)에서만.
if processing > 0 or done_15m > 0:
state = "active"
elif key == "macbook" and deferred_pending > 0:
state = "deferred"
else:
state = "idle"
machines.append({
"key": key,
"label": _MACHINE_LABELS[key],
"state": state,
"stages": stages,
"pending": pending,
"processing": processing,
"failed": failed,
"done_1h": done_1h,
"done_today": done_today,
"deferred_pending": deferred_pending,
"current": current_by_machine[key],
})
return machines
def compute_eta_minutes(pending: int, done_1h: int, inflow_1h: int) -> int | None:
"""ETA(분) = 순소화율 기반. done > inflow 일 때만 산출, 아니면 None (소화 불가)."""
if done_1h > inflow_1h:
return round(pending / (done_1h - inflow_1h) * 60)
return None
def build_summarize_eta(stage_stats: dict[str, dict]) -> dict:
"""summarize 풀 ETA — pending 은 보류(deferred) 포함 총수."""
s = stage_stats.get("summarize", _zero_stage())
pending = s["pending"]
done_rate = s["done_1h"]
inflow_rate = s["created_1h"]
return {
"pending": pending,
"done_rate_1h": done_rate,
"inflow_rate_1h": inflow_rate,
"eta_minutes": compute_eta_minutes(pending, done_rate, inflow_rate),
}
def build_summarize_by_machine(summarize_split: dict[str, dict]) -> dict:
"""summarize 머신별 완료 실적 분담 (macmini vs macbook) — 보드 레인의
오프로드 가시화용. rows_to_summarize_split 이미 만든 값을 응답 형태로
투영(done_1h/done_today , done_15m 내부 state 판정 전용이라 제외)."""
def m(key: str) -> dict:
s = summarize_split.get(key, {})
return {"done_1h": int(s.get("done_1h", 0)), "done_today": int(s.get("done_today", 0))}
return {"macmini": m("macmini"), "macbook": m("macbook")}
def build_trend(
inflow_buckets: dict[str, int],
done_buckets: dict[str, int],
now_kst: datetime,
) -> list[dict]:
"""summarize 24h 추이 — KST 시간 버킷 24개 (오래된 것부터, 빈 버킷 0).
버킷 key = "YYYY-MM-DD HH:00" (KST). SQL to_char 출력과 동일 포맷.
"""
base = now_kst.replace(minute=0, second=0, microsecond=0)
trend = []
for i in range(23, -1, -1):
bucket = base - timedelta(hours=i)
key = bucket.strftime("%Y-%m-%d %H:00")
trend.append({
"hour": bucket.strftime("%H:00"),
"inflow": inflow_buckets.get(key, 0),
"done": done_buckets.get(key, 0),
})
return trend
def build_stages(stage_stats: dict[str, dict], now=None) -> list[dict]:
"""단계별 현황 행 — '단계 상세' 패널용 (2026-06-11 사용자 피드백: 완료가 보여야 한다).
파이프라인 순서 유지, 미지 stage 뒤에. 숨김/강조 판단은 FE 여기선 사실만.
oldest_pending_age_sec = 가장 오래된 pending 경과 (pending 없으면 None).
"""
from datetime import datetime, timezone
now = now or datetime.now(timezone.utc)
extra = [s for s in stage_stats if s not in _STAGE_ORDER]
rows = []
for stage in [*_STAGE_ORDER, *extra]:
st = stage_stats.get(stage) or _zero_stage()
oldest = st.get("oldest_pending_at")
age = None
if oldest is not None:
if oldest.tzinfo is None:
oldest = oldest.replace(tzinfo=timezone.utc)
age = max(0, int((now - oldest).total_seconds()))
rows.append({
"stage": stage,
"pending": st["pending"],
"processing": st["processing"],
"failed": st["failed"],
"done_1h": st["done_1h"],
"created_1h": st["created_1h"],
"done_today": st["done_today"],
"oldest_pending_age_sec": age,
})
return rows
def build_totals(stage_stats: dict[str, dict]) -> dict:
"""전 stage 합계."""
return {
"pending": sum(s["pending"] for s in stage_stats.values()),
"processing": sum(s["processing"] for s in stage_stats.values()),
"failed": sum(s["failed"] for s in stage_stats.values()),
}
def compose_overview(
stage_stats: dict[str, dict],
summarize_split: dict[str, dict],
inflow_buckets: dict[str, int],
done_buckets: dict[str, int],
current_rows: list[dict],
*,
deep_enabled: bool,
now_kst: datetime,
) -> dict:
"""수집된 통계 → 응답 dict (계약 shape). 순수 함수 — DB 불요."""
return {
"machines": build_machines(
stage_stats, summarize_split, current_rows, deep_enabled=deep_enabled
),
"stages": build_stages(stage_stats),
"summarize_eta": build_summarize_eta(stage_stats),
"summarize_by_machine": build_summarize_by_machine(summarize_split),
"trend_24h": build_trend(inflow_buckets, done_buckets, now_kst),
"totals": build_totals(stage_stats),
}
# ─── SQL 수집부 (총 5쿼리) ────────────────────────────────────────────────────
# 1) stage×status 집계 + 시간창 완료/유입 + 보류 (1방)
_STAGE_STATS_SQL = """
SELECT
stage,
COUNT(*) FILTER (WHERE status = 'pending') AS pending,
COUNT(*) FILTER (WHERE status = 'processing') AS processing,
COUNT(*) FILTER (WHERE status = 'failed') AS failed,
COUNT(*) FILTER (WHERE status = 'completed'
AND completed_at > NOW() - INTERVAL '1 hour') AS done_1h,
COUNT(*) FILTER (WHERE status = 'completed'
AND completed_at > :kst_midnight) AS done_today,
COUNT(*) FILTER (WHERE status = 'completed'
AND completed_at > NOW() - INTERVAL '15 minutes') AS done_15m,
COUNT(*) FILTER (WHERE status = 'pending'
AND payload ->> 'deferred_until' IS NOT NULL
AND (payload ->> 'deferred_until')::timestamptz > NOW())
AS deferred_pending,
COUNT(*) FILTER (WHERE created_at > NOW() - INTERVAL '1 hour') AS created_1h,
MIN(created_at) FILTER (WHERE status = 'pending') AS oldest_pending_at
FROM processing_queue
GROUP BY stage
"""
# 2) summarize 풀 완료 실적 분리 (documents.ai_model_version 조인, 1방)
# 스캔 하한 = 오늘 0시(KST)와 1h 전 중 더 이른 시각 (자정 직후 1h 창 보전).
_SUMMARIZE_SPLIT_SQL = """
SELECT
COALESCE(d.ai_model_version = :macbook_alias, false) AS is_macbook,
COUNT(*) FILTER (WHERE q.completed_at > NOW() - INTERVAL '1 hour') AS done_1h,
COUNT(*) FILTER (WHERE q.completed_at > :kst_midnight) AS done_today,
COUNT(*) FILTER (WHERE q.completed_at > NOW() - INTERVAL '15 minutes') AS done_15m
FROM processing_queue q
JOIN documents d ON d.id = q.document_id
WHERE q.stage = 'summarize'
AND q.status = 'completed'
AND q.completed_at > LEAST(:kst_midnight, NOW() - INTERVAL '1 hour')
GROUP BY 1
"""
# 3/4) summarize 24h 추이 — KST 시간 버킷 (inflow/done 각 1방)
_TREND_INFLOW_SQL = """
SELECT to_char(date_trunc('hour', created_at AT TIME ZONE 'Asia/Seoul'),
'YYYY-MM-DD HH24:00') AS bucket,
COUNT(*) AS n
FROM processing_queue
WHERE stage = 'summarize'
AND created_at > NOW() - INTERVAL '24 hours'
GROUP BY 1
"""
_TREND_DONE_SQL = """
SELECT to_char(date_trunc('hour', completed_at AT TIME ZONE 'Asia/Seoul'),
'YYYY-MM-DD HH24:00') AS bucket,
COUNT(*) AS n
FROM processing_queue
WHERE stage = 'summarize'
AND status = 'completed'
AND completed_at > NOW() - INTERVAL '24 hours'
GROUP BY 1
"""
# 5) processing 행 + 표시용 제목 재료 (1방 — 머신별 2건 슬라이스는 판정부에서)
_CURRENT_SQL = """
SELECT q.stage, q.document_id, d.title, d.original_filename, d.file_path
FROM processing_queue q
JOIN documents d ON d.id = q.document_id
WHERE q.status = 'processing'
ORDER BY q.started_at DESC NULLS LAST
LIMIT 50
"""
async def build_overview(session: AsyncSession) -> dict:
"""5쿼리 수집 → compose_overview 판정 → 응답 dict."""
now_kst = datetime.now(KST)
kst_midnight = now_kst.replace(hour=0, minute=0, second=0, microsecond=0)
deep_enabled = settings.ai is not None and settings.ai.deep is not None
stage_rows = (
await session.execute(text(_STAGE_STATS_SQL), {"kst_midnight": kst_midnight})
).all()
split_rows = (
await session.execute(
text(_SUMMARIZE_SPLIT_SQL),
{"kst_midnight": kst_midnight, "macbook_alias": _MACBOOK_MODEL_ALIAS},
)
).all()
inflow_rows = (await session.execute(text(_TREND_INFLOW_SQL))).all()
done_rows = (await session.execute(text(_TREND_DONE_SQL))).all()
current_result = (await session.execute(text(_CURRENT_SQL))).all()
current_rows = [
{
"stage": row[0],
"document_id": row[1],
"title": row[2],
"original_filename": row[3],
"file_path": row[4],
}
for row in current_result
]
return compose_overview(
rows_to_stage_stats(stage_rows),
rows_to_summarize_split(split_rows),
{row[0]: int(row[1]) for row in inflow_rows},
{row[0]: int(row[1]) for row in done_rows},
current_rows,
deep_enabled=deep_enabled,
now_kst=now_kst,
)
# ─── 실패 처리 (plan ds-board-engines-1) ─────────────────────────────────────
# 실패 = 자동 재시도(max_attempts=3) 소진 후 영구 정지 상태. 여기 함수들은
# 사용자 명시 조치 전용 — 자동 호출 경로 없음 (보드 실패 드로어가 유일 호출자).
# 실패 행은 completed_at 이 비어 있을 수 있어(소비자 실패 경로가 미기록)
# started_at 을 시각 fallback 으로 쓴다.
_FAILED_LIST_SQL = """
SELECT q.id, q.stage, q.document_id, q.attempts, q.max_attempts,
q.error_message,
COALESCE(q.completed_at, q.started_at) AS failed_at,
d.title, d.original_filename, d.file_path
FROM processing_queue q
JOIN documents d ON d.id = q.document_id
WHERE q.status = 'failed'
ORDER BY q.stage, COALESCE(q.completed_at, q.started_at) DESC NULLS LAST
LIMIT 300
"""
# 재시도: failed → pending (attempts 리셋 = 자동 재시도 3회 새로 부여).
# error_message 는 감사용으로 보존 — 성공 시 완료 행에 남아도 무해.
# uq_queue_active((doc,stage) pending/processing 부분 유니크)와 충돌하는 행 —
# 같은 문서·단계가 이미 재enqueue 된 경우 — 는 건드리지 않고 건수만 보고.
_RETRY_SQL = """
UPDATE processing_queue q
SET status = 'pending', attempts = 0,
started_at = NULL, completed_at = NULL
WHERE q.id IN :ids
AND q.status = 'failed'
AND NOT EXISTS (
SELECT 1 FROM processing_queue p
WHERE p.document_id = q.document_id
AND p.stage = q.stage
AND p.status IN ('pending', 'processing')
AND p.id <> q.id
)
RETURNING q.id
"""
# 건너뛰기: failed → completed + payload 마킹 (감사 추적).
# enqueue_next_stage 는 의도적으로 호출하지 않는다 — 실패 문서(빈 텍스트 등)가
# 하류 단계로 흘러가는 것 방지. 후속 단계가 필요하면 재시도가 정상 경로.
_SKIP_SQL = """
UPDATE processing_queue
SET status = 'completed', completed_at = NOW(),
payload = COALESCE(payload, '{}'::jsonb)
|| jsonb_build_object('skipped_by_user', true,
'skipped_at', NOW()::text)
WHERE id IN :ids AND status = 'failed'
RETURNING id
"""
async def fetch_failed_items(session: AsyncSession) -> list[dict]:
"""영구 실패 행 목록 (문서 제목 포함, 최대 300건)."""
rows = (await session.execute(text(_FAILED_LIST_SQL))).all()
return [
{
"id": r[0],
"stage": r[1],
"document_id": r[2],
"attempts": int(r[3] or 0),
"max_attempts": int(r[4] or 0),
"error_message": r[5],
"failed_at": r[6],
"title": display_title({
"document_id": r[2],
"title": r[7],
"original_filename": r[8],
"file_path": r[9],
}),
}
for r in rows
]
async def retry_failed(session: AsyncSession, ids: list[int]) -> dict:
"""failed → pending 복귀. not_retried = active 충돌 + 이미 failed 아님."""
unique_ids = list(set(ids))
stmt = text(_RETRY_SQL).bindparams(bindparam("ids", expanding=True))
retried = (await session.execute(stmt, {"ids": unique_ids})).all()
await session.commit()
return {
"requested": len(unique_ids),
"retried": len(retried),
"not_retried": len(unique_ids) - len(retried),
}
async def skip_failed(session: AsyncSession, ids: list[int]) -> dict:
"""failed → completed(건너뛰기 마킹). 후속 단계 연쇄 없음."""
unique_ids = list(set(ids))
stmt = text(_SKIP_SQL).bindparams(bindparam("ids", expanding=True))
skipped = (await session.execute(stmt, {"ids": unique_ids})).all()
await session.commit()
return {
"requested": len(unique_ids),
"skipped": len(skipped),
"not_skipped": len(unique_ids) - len(skipped),
}
+8
View File
@@ -72,6 +72,10 @@ class LegacyWeightedSum(FusionStrategy):
score=existing.score + r.score * 0.5,
snippet=existing.snippet,
match_reason=f"{existing.match_reason}+vector",
# C-1: 분류 축 메타 전파 (재구성 시 누락 = D-1 유형 표시 None)
material_type=existing.material_type,
jurisdiction=existing.jurisdiction,
published_date=existing.published_date,
)
elif r.score > 0.3:
merged[r.id] = r
@@ -128,6 +132,10 @@ class RRFOnly(FusionStrategy):
score=rrf_score,
snippet=base.snippet,
match_reason="+".join(reasons),
# C-1: 분류 축 메타 전파 (재구성 시 누락 = D-1 유형 표시 None)
material_type=base.material_type,
jurisdiction=base.jurisdiction,
published_date=base.published_date,
)
)
return merged[:limit]
+28
View File
@@ -0,0 +1,28 @@
"""안전 자료실 B-4 — licensed_restricted 단일 술어 (a안 U-2①, 모든 경로 공유 정의).
색인은 허용하되 restricted=true(구매 전자책·유료자료) verbatim span RAG 증거·발행물
(검색/ask·digest·morning_briefing·study 풀이) 들어가는 모든 경로를 구조적으로 차단.
경로마다 술어를 복붙하지 않고 정의를 공유 가드 누락/드리프트 방지
([[feedback_structural_integrity_over_path_discipline]]).
개인 파일 열람(GET /documents/{id}?download) a안상 허용 = 미적용.
표현(raw SQL / ORM) 의미 동일: restricted 부재·false·extract_meta NULL = COALESCE
미제외(redistribute=false 여도 restricted 부재면 미제외 redistributerestricted 핵심).
"""
def restricted_exclude_sql(alias: str = "") -> str:
"""raw text() 쿼리용 bare 술어('AND' 미포함). alias='' = 컬럼 직접 참조."""
p = (alias + ".") if alias else ""
return f"COALESCE({p}extract_meta -> 'license' ->> 'restricted', 'false') <> 'true'"
def restricted_exclude_orm():
"""SQLAlchemy ORM .where() 절 — restricted_exclude_sql 과 동일 의미(JSONB extract_meta)."""
from sqlalchemy import func
from models.document import Document
return func.coalesce(
Document.extract_meta["license"]["restricted"].astext, "false"
) != "true"
+40 -12
View File
@@ -26,8 +26,11 @@ PR-MacBook-RAG-Backend-1 부터 `services.llm.QwenMacBookBackend` 는 별 endpoi
- **fallback(Claude Sonnet 4 API) 경로는 gate 제외**. PR #20 이후 fallback = Claude API. 단 현재
구현상 `AIClient._call_chat` 내부에서 primaryfallback 전환이 일어나므로
fallback도 gate 점유 상태로 실행된다. 허용 가능(fallback 빈도 낮음).
- **MLX concurrency는 `MLX_CONCURRENCY = 1` 고정**. 모델이 바뀌어도 single-
inference 특성이 깨지지 않는 값을 올리지 .
- ~~**MLX concurrency는 `MLX_CONCURRENCY = 1` 고정**~~ **2026-06-12 개정**:
룰의 전제(서버 = single-inference) 소멸 mlx_vlm server continuous
batching 으로 동시 스트림 흡수(실측). 상한은 config `pipeline.mlx_gate_concurrency`
(기본 1, 운영 2). **게이트 자체(상한+우선순위 ) 영구 유지** thundering herd
(23 concurrent 22 timeout 사고) 방지는 계속 상한이 담당. 무제한 금지.
## 우선순위 정책 (B-1, 2026-05-17)
@@ -80,8 +83,22 @@ from core.utils import setup_logger
logger = setup_logger("llm_gate")
# MLX primary는 single-inference → 1
MLX_CONCURRENCY = 1
def _capacity() -> int:
"""게이트 동시 실행 상한 — config.yaml `pipeline.mlx_gate_concurrency` (기본 1).
2026-06-12 일반화: "MLX_CONCURRENCY = 1 고정" 영구 룰의 전제( 서버 = single-
inference, 23 concurrent 22 timeout 실측) 소멸 mlx_vlm server
continuous batching 으로 동시 스트림을 흡수(2026-06-11 6~8 concurrent 실측
정상). 게이트 자체(상한 + 우선순위) 유지하고 상한만 config thundering
herd 재발 방지는 상한이 계속 담당한다. 런타임 acquire 조회라
config 변경 + 프로세스 재기동으로 반영, 테스트는 settings monkeypatch.
"""
from core.config import settings
try:
return max(1, int(getattr(settings, "mlx_gate_concurrency", 1)))
except (TypeError, ValueError):
return 1
# Background waiter wait_ms 가 이 값 초과 시 WARN (starvation 신호, aging mitigation 은 Phase 2)
STARVATION_WARN_MS = 300_000 # 5 min
@@ -101,7 +118,7 @@ DEFAULT_PRIORITY: Priority = Priority.BACKGROUND
# Tuple format: (priority: int, seq: int, future: asyncio.Future, enqueue_ts: float)
_waiters: list[tuple[int, int, asyncio.Future, float]] = []
_seq = itertools.count()
_inflight: bool = False
_inflight_n: int = 0 # 동시 실행 수 (구 bool — capacity 일반화로 카운터)
_lock: asyncio.Lock | None = None
@@ -143,7 +160,7 @@ async def acquire_mlx_gate(
`asyncio.timeout` 반드시 gate 안쪽 (Future await ) .
"""
global _inflight, _waiters
global _inflight_n, _waiters
lock = _get_lock()
seq = next(_seq)
@@ -152,9 +169,9 @@ async def acquire_mlx_gate(
fut: asyncio.Future | None = None
async with lock:
if not _inflight and not _waiters:
if _inflight_n < _capacity() and not _waiters:
# fast path — 즉시 inflight 진입, Future 생성 안 함
_inflight = True
_inflight_n += 1
else:
# 대기열 진입
fut = asyncio.get_event_loop().create_future()
@@ -194,8 +211,8 @@ async def acquire_mlx_gate(
async with lock:
next_fut = _dispatch_next_locked()
if next_fut is None:
_inflight = False
# _inflight 는 True 유지 (다음 waiter 가 진입 예정)
_inflight_n = max(0, _inflight_n - 1)
# next_fut 가 있으면 슬롯 handover — 카운트 유지 (다음 waiter 가 진입 예정)
logger.debug(
"mlx_gate release duration_ms=%.0f priority=%s seq=%d",
duration_ms, priority.name, seq,
@@ -222,13 +239,24 @@ def get_mlx_gate():
return acquire_mlx_gate(DEFAULT_PRIORITY)
# ── Read-only status (UI 표시용) ─────────────────────────────────────────────
def gate_status() -> dict:
"""현재 gate 점유 스냅샷 (read-only, lock-free 근사치 — UI 표시용).
inflight = 동시 실행 (int). 기존 소비자(eid status) bool() 캐스팅이라 호환.
"""
return {"inflight": _inflight_n, "waiters": len(_waiters)}
# ── Test helpers (conftest reset) ────────────────────────────────────────────
def _reset_for_test() -> None:
"""테스트 fixture 가 fresh loop 마다 호출. production code 에서 사용 X."""
global _waiters, _inflight, _lock, _seq
global _waiters, _inflight_n, _lock, _seq
_waiters = []
_inflight = False
_inflight_n = 0
_lock = None
_seq = itertools.count()
+55
View File
@@ -0,0 +1,55 @@
"""안전 자료실 C-1 후속 — 검색 결과 wrapper decoration (version_status + facets).
엔드포인트 wrapper 에서 run_search() 결과에 1 적용 검색 코어(run_search) 무접촉(r3).
- version_status: 법령 결과(material_type='law') legal_meta.version_status
(current/superseded/pending/repealed) 부착. legal_meta.document_id 1:0..1 위성
매핑 없는 law(레거시 ) None 유지. law 결과 없으면 query skip.
- facets: top-K 결과 분류 (material_type/jurisdiction/version_status) 분포 라벨(r2-M4).
facets=true 때만 계산(미요청 None = byte 불변·ranking 무관).
"""
from __future__ import annotations
from collections import Counter
from typing import TYPE_CHECKING
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession
if TYPE_CHECKING:
from api.search import SearchResult
async def decorate_version_status(
session: AsyncSession, results: list["SearchResult"]
) -> None:
"""법령 결과에 legal_meta.version_status 부착 (in-place). law 결과 없으면 query skip."""
law_ids = [r.id for r in results if r.material_type == "law" and r.id is not None]
if not law_ids:
return
rows = await session.execute(
text(
"SELECT document_id, version_status FROM legal_meta "
"WHERE document_id = ANY(:ids)"
),
{"ids": law_ids},
)
status_by_id = {row.document_id: row.version_status for row in rows}
for r in results:
if r.id in status_by_id:
r.version_status = status_by_id[r.id]
def compute_facets(results: list["SearchResult"]) -> dict[str, dict[str, int]]:
"""top-K 결과의 분류 축 분포 라벨. None 값은 제외(present 라벨만, 빈 축은 미포함)."""
axes = {
"material_type": [r.material_type for r in results],
"jurisdiction": [r.jurisdiction for r in results],
"version_status": [getattr(r, "version_status", None) for r in results],
}
facets: dict[str, dict[str, int]] = {}
for axis, vals in axes.items():
counter = Counter(v for v in vals if v is not None)
if counter:
facets[axis] = dict(counter.most_common())
return facets
+167 -13
View File
@@ -24,6 +24,7 @@ import asyncio
import hashlib
import re
import time
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any
from sqlalchemy import text
@@ -63,8 +64,98 @@ CANDIDATE_BACKEND_MAP: dict[str, dict[str, str] | None] = {
"chunks_table": "document_chunks_cand_snowflake_l_v2",
"embed_endpoint": "http://embedding-cand-snowflake-l-v2:80/embed",
},
# ─── Phase 2A (embedding-phase2a-1, 2026-06-12): Qwen3-Embedding 후보 3종 ───
# embed_kind="ollama" = /api/embed 호출 + 쿼리측 instruct prefix (비대칭 사용,
# G-1 fixture 실측: prefix 가 관련쌍 cos +0.016). 문서측은 backfill 이 plain 으로 적재.
# qwen4m = 4B 의 MRL 1024d (dimensions 옵션 — Ollama 가 truncate+재정규화 수행, G-1 실측).
"cand_qwen06": {
"docs_table": "documents_cand_qwen06",
"chunks_table": "document_chunks_cand_qwen06",
"embed_endpoint": "http://ollama:11434/api/embed",
"embed_kind": "ollama",
"embed_model": "qwen3-embedding:0.6b",
},
"cand_qwen4": {
"docs_table": "documents_cand_qwen4",
"chunks_table": "document_chunks_cand_qwen4",
"embed_endpoint": "http://ollama:11434/api/embed",
"embed_kind": "ollama",
"embed_model": "qwen3-embedding:4b",
},
"cand_qwen4m": {
"docs_table": "documents_cand_qwen4m",
"chunks_table": "document_chunks_cand_qwen4m",
"embed_endpoint": "http://ollama:11434/api/embed",
"embed_kind": "ollama",
"embed_model": "qwen3-embedding:4b",
"embed_dimensions": 1024,
},
}
# G-1 핀 고정 instruct 문자열 (inventory 2026-06-12-c 기록과 동일해야 함 —
# 문구 변경 = 저장=조회 불변식 위반과 동급. 쿼리 측 전용, 문서 적재는 plain).
QWEN3_QUERY_INSTRUCT = (
"Instruct: Given a web search query, retrieve relevant passages that answer the query"
"\nQuery: "
)
# ─── 안전 자료실 C-1: 분류 축 명시 필터 (3 leg 동등, byte 불변) ───────────────
# 미지정(active=False) 시 모든 SQL 절이 빈 문자열 → 기존 SQL byte 불변(run_eval 회귀 0).
# year 는 published_date NULL fallback created_at (freshness 와 동일 COALESCE 사상).
@dataclass
class AxisFilter:
material_types: list[str] | None = None # CSV → list, material_type = ANY
jurisdiction: str | None = None
year_from: int | None = None
year_to: int | None = None
def active(self) -> bool:
return bool(self.material_types or self.jurisdiction
or self.year_from is not None or self.year_to is not None)
def _axis_sql(alias: str, af: "AxisFilter | None", params: dict) -> str:
"""alias 기준 axis 필터 SQL — 미지정 시 '' (byte 불변). 반환 형태 ' AND ...'.
alias='' 이면 컬럼 직접 참조(단일 테이블 FROM documents 경로). 파라미터는 af_ prefix
호출측 기존 bind 충돌 방지.
"""
if af is None or not af.active():
return ""
p = (alias + ".") if alias else ""
cl: list[str] = []
if af.material_types:
cl.append(f"{p}material_type = ANY(:af_mt)")
params["af_mt"] = af.material_types
if af.jurisdiction:
cl.append(f"{p}jurisdiction = :af_jur")
params["af_jur"] = af.jurisdiction
if af.year_from is not None:
cl.append(f"COALESCE({p}published_date, {p}created_at::date) >= make_date(:af_yf, 1, 1)")
params["af_yf"] = af.year_from
if af.year_to is not None:
cl.append(f"COALESCE({p}published_date, {p}created_at::date) <= make_date(:af_yt, 12, 31)")
params["af_yt"] = af.year_to
return " AND " + " AND ".join(cl)
# ─── 안전 자료실 B-4: licensed_restricted 단일 술어 (a안 U-2① — 항상 적용) ──────
def _license_sql(alias: str) -> str:
"""licensed_restricted(extract_meta.license.restricted=true) 문서를 retrieval 에서 제외.
a안: 색인은 허용하되, 구매 전자책/유료자료의 verbatim span RAG 증거·digest 발행에
들어가는 경로를 구조적으로 차단. 단일 술어를 모든 retrieval leg + digest loader
공유 경로별 가드 누락 방지([[feedback_structural_integrity_over_path_discipline]]).
개인 파일 열람(GET /documents/{id}?download) a안상 허용이라 미적용.
axis 필터(조건부) 달리 항상 적용. restricted 부재/false = COALESCE 미제외
기존 코퍼스(restricted=true 0)에서 결과 불변. 반환 ' AND ...' (alias='' = 컬럼 직접).
술어 정의 = license_filter.restricted_exclude_sql 공유(digest/briefing/study 풀이와 단일 source).
"""
from services.search.license_filter import restricted_exclude_sql
return " AND " + restricted_exclude_sql(alias)
# 2단계 gate (R2-B1) — SQL string interpolation 직전 final allowlist.
_VALID_DOCS_TABLE = re.compile(r"^(documents|documents_cand_[a-z0-9_]+)$")
# corpus_chunks = document_chunks WHERE in_corpus=true 뷰 (Hier-Decomp-1 c2 choke point).
@@ -137,6 +228,34 @@ async def _embed_query_via_tei(endpoint: str, text_: str) -> list[float] | None:
return None
async def _embed_query_via_ollama(cfg: dict, text_: str) -> list[float] | None:
"""Phase 2A 후보 쿼리 임베딩 — Ollama /api/embed + 비대칭 instruct prefix.
쿼리 전용: QWEN3_QUERY_INSTRUCT 선두에 붙인다 (문서 적재 = plain).
embed_dimensions 지정(qwen4m) Ollama dimensions 옵션 = MRL truncate+재정규화
(G-1 fixture: 1024 출력 L2=1.0 실측). cache 미사용 slug 분포 상이.
"""
if not text_:
return None
import httpx
body: dict = {"model": cfg["embed_model"], "input": [QWEN3_QUERY_INSTRUCT + text_]}
if cfg.get("embed_dimensions"):
body["dimensions"] = cfg["embed_dimensions"]
try:
async with httpx.AsyncClient(timeout=60.0) as c:
r = await c.post(cfg["embed_endpoint"], json=body)
r.raise_for_status()
embs = r.json().get("embeddings")
if not isinstance(embs, list) or not embs or not isinstance(embs[0], list):
raise ValueError("unexpected /api/embed shape")
return embs[0]
except Exception as exc:
logger.warning(
"candidate ollama embed failed model=%s err=%r", cfg.get("embed_model"), exc
)
return None
def _query_embed_key(text_: str) -> str:
return hashlib.sha256(f"{text_}|bge-m3".encode("utf-8")).hexdigest()
@@ -174,7 +293,7 @@ def query_embed_cache_stats() -> dict[str, int]:
async def search_text(
session: AsyncSession, query: str, limit: int
session: AsyncSession, query: str, limit: int, *, axis: "AxisFilter | None" = None
) -> list["SearchResult"]:
"""FTS + trigram 필드별 가중치 검색 (Phase 1.2-B UNION 분해).
@@ -205,8 +324,12 @@ async def search_text(
# SQLAlchemy async session 내 두 execute는 같은 connection 사용
await session.execute(text("SELECT set_limit(0.15)"))
_params: dict[str, Any] = {"q": query, "limit": limit}
# license(항상) + axis(조건부). license 가 항상 ' AND ...' 이라 WHERE 는 늘 존재.
_where = _license_sql("d") + _axis_sql("d", axis, _params)
result = await session.execute(
text("""
text(f"""
WITH candidates AS (
-- title trigram (idx_documents_title_trgm)
SELECT id FROM documents
@@ -259,13 +382,15 @@ async def search_text(
WHEN similarity(coalesce(d.ai_summary, ''), :q) >= 0.3 THEN 'summary'
WHEN similarity(coalesce(d.extracted_text, ''), :q) >= 0.3 THEN 'content'
ELSE 'fts'
END AS match_reason
END AS match_reason,
d.material_type, d.jurisdiction, d.published_date
FROM documents d
JOIN candidates c ON d.id = c.id
WHERE{_where[4:]}
ORDER BY score DESC
LIMIT :limit
"""),
{"q": query, "limit": limit},
_params,
)
return [SearchResult(**row._mapping) for row in result]
@@ -280,6 +405,7 @@ async def search_vector(
snapshot_chunk_id_max: int | None = None,
corpus_variant: str | None = None,
exact_knn: bool = False,
axis: "AxisFilter | None" = None,
) -> list["SearchResult"]:
"""Hybrid 벡터 검색 — doc + chunks 동시 retrieval (Phase 1.2-G).
@@ -323,7 +449,10 @@ async def search_vector(
else:
docs_table = cfg["docs_table"]
chunks_table = cfg["chunks_table"]
query_embedding = await _embed_query_via_tei(cfg["embed_endpoint"], query)
if cfg.get("embed_kind") == "ollama":
query_embedding = await _embed_query_via_ollama(cfg, query)
else:
query_embedding = await _embed_query_via_tei(cfg["embed_endpoint"], query)
logger.info(
"[embedding-dispatch] backend=%s docs_table=%s chunks_table=%s snapshot_doc_id_max=%s "
@@ -351,6 +480,7 @@ async def search_vector(
docs_table=docs_table,
snapshot_doc_id_max=snapshot_doc_id_max,
exact_knn=exact_knn,
axis=axis,
)
async def _chunks_call() -> list["SearchResult"]:
@@ -360,6 +490,7 @@ async def search_vector(
chunks_table=chunks_table,
snapshot_chunk_id_max=snapshot_chunk_id_max,
exact_knn=exact_knn,
axis=axis,
)
doc_results, chunk_results = await asyncio.gather(_docs_call(), _chunks_call())
@@ -375,6 +506,7 @@ async def _search_vector_docs(
docs_table: str = "documents",
snapshot_doc_id_max: int | None = None,
exact_knn: bool = False,
axis: "AxisFilter | None" = None,
) -> list["SearchResult"]:
"""documents (또는 documents_cand_<slug>).embedding 직접 검색.
@@ -399,28 +531,34 @@ async def _search_vector_docs(
if snapshot_doc_id_max is not None:
snapshot_clause = " AND id <= :snapshot_doc_id_max"
params["snapshot_doc_id_max"] = snapshot_doc_id_max
axis_clause = _axis_sql("", axis, params) # alias 없음 (단일 FROM documents)
license_clause = _license_sql("") # B-4: restricted 항상 제외
sql = f"""
SELECT id, title, ai_domain, ai_summary, file_format,
(1 - (embedding <=> cast(:embedding AS vector))) AS score,
left(extracted_text, 1200) AS snippet,
'vector_doc' AS match_reason,
NULL::bigint AS chunk_id, NULL::integer AS chunk_index, NULL::text AS section_title
NULL::bigint AS chunk_id, NULL::integer AS chunk_index, NULL::text AS section_title,
material_type, jurisdiction, published_date
FROM documents
WHERE embedding IS NOT NULL AND deleted_at IS NULL{snapshot_clause}
WHERE embedding IS NOT NULL AND deleted_at IS NULL{snapshot_clause}{axis_clause}{license_clause}
ORDER BY embedding <=> cast(:embedding AS vector)
LIMIT :limit
"""
else:
# candidate: docs_table 은 (doc_id, embed_input, embed_input_hash, embedding) 만 보유 → JOIN documents
axis_clause = _axis_sql("d", axis, params)
license_clause = _license_sql("d") # B-4: restricted 항상 제외
sql = f"""
SELECT d.id, d.title, d.ai_domain, d.ai_summary, d.file_format,
(1 - (c.embedding <=> cast(:embedding AS vector))) AS score,
left(d.extracted_text, 1200) AS snippet,
'vector_doc' AS match_reason,
NULL::bigint AS chunk_id, NULL::integer AS chunk_index, NULL::text AS section_title
NULL::bigint AS chunk_id, NULL::integer AS chunk_index, NULL::text AS section_title,
d.material_type, d.jurisdiction, d.published_date
FROM {docs_table} c
JOIN documents d ON d.id = c.doc_id
WHERE d.deleted_at IS NULL
WHERE d.deleted_at IS NULL{axis_clause}{license_clause}
ORDER BY c.embedding <=> cast(:embedding AS vector)
LIMIT :limit
"""
@@ -436,6 +574,7 @@ async def _search_vector_chunks(
chunks_table: str = "document_chunks",
snapshot_chunk_id_max: int | None = None,
exact_knn: bool = False,
axis: "AxisFilter | None" = None,
) -> list["SearchResult"]:
"""document_chunks (또는 document_chunks_cand_<slug>).embedding window partition.
@@ -461,12 +600,25 @@ async def _search_vector_chunks(
snapshot_clause = " AND c.id <= :snapshot_chunk_id_max"
params["snapshot_chunk_id_max"] = snapshot_chunk_id_max
# C-1: axis 필터는 inner topk 에 JOIN (R6 결정 — outer post-filter 면 ANN top-:inner_k
# 후보를 뽑은 뒤 거르므로 좁은 필터(GB 법령 등)에서 후보 붕괴). 미지정 시 JOIN 없음 = byte 불변.
if axis and axis.active():
chunk_join = " JOIN documents df ON df.id = c.doc_id"
chunk_axis = _axis_sql("df", axis, params)
else:
chunk_join = ""
chunk_axis = ""
# B-4: restricted 제외 — outer 가 documents d 를 항상 JOIN 하므로 post-rank 위치.
# restricted 는 소수(구매자료)라 inner topk 후 제외해도 candidate collapse 없음(axis 와 상이).
license_clause = _license_sql("d")
sql = f"""
WITH topk AS (
SELECT c.id AS chunk_id, c.doc_id, c.chunk_index, c.section_title, c.text,
c.embedding <=> cast(:embedding AS vector) AS dist
FROM {chunks_table} c
WHERE c.embedding IS NOT NULL{snapshot_clause}
FROM {chunks_table} c{chunk_join}
WHERE c.embedding IS NOT NULL{snapshot_clause}{chunk_axis}
ORDER BY c.embedding <=> cast(:embedding AS vector)
LIMIT :inner_k
),
@@ -479,10 +631,12 @@ async def _search_vector_chunks(
d.ai_summary AS ai_summary, d.file_format AS file_format,
(1 - r.dist) AS score, left(r.text, 1200) AS snippet,
'vector_chunk' AS match_reason,
r.chunk_id AS chunk_id, r.chunk_index AS chunk_index, r.section_title AS section_title
r.chunk_id AS chunk_id, r.chunk_index AS chunk_index, r.section_title AS section_title,
d.material_type AS material_type, d.jurisdiction AS jurisdiction,
d.published_date AS published_date
FROM ranked r
JOIN documents d ON d.id = r.doc_id
WHERE r.rn <= 2 AND d.deleted_at IS NULL
WHERE r.rn <= 2 AND d.deleted_at IS NULL{license_clause}
ORDER BY r.dist
LIMIT :limit
"""
+9 -1
View File
@@ -47,6 +47,7 @@ from .rerank_service import (
rerank_chunks,
)
from .retrieval_service import (
AxisFilter,
compress_chunks_to_docs,
search_text,
search_vector,
@@ -148,6 +149,7 @@ async def run_search(
rewrite_backend: str | None = None,
corpus_variant: str | None = None,
exact_knn: bool = False,
axis: AxisFilter | None = None,
) -> PipelineResult:
"""검색 파이프라인 실행.
@@ -275,6 +277,7 @@ async def run_search(
snapshot_chunk_id_max=snapshot_chunk_id_max,
corpus_variant=corpus_variant,
exact_knn=exact_knn,
axis=axis,
)
timing["vector_ms"] = (time.perf_counter() - t0) * 1000
if not raw_chunks:
@@ -284,7 +287,7 @@ async def run_search(
results = vector_results
else:
t0 = time.perf_counter()
text_results = await search_text(session, q, limit)
text_results = await search_text(session, q, limit, axis=axis)
timing["text_ms"] = (time.perf_counter() - t0) * 1000
if mode == "hybrid":
@@ -306,6 +309,7 @@ async def run_search(
snapshot_chunk_id_max=snapshot_chunk_id_max,
corpus_variant=corpus_variant,
exact_knn=exact_knn,
axis=axis,
)
timing["vector_ms"] = (time.perf_counter() - t1) * 1000
@@ -458,6 +462,10 @@ def _rrf_fuse_variants(
score=rrf_score,
snippet=doc.snippet,
match_reason=f"{doc.match_reason}+multi_query_rrf",
# C-1: 분류 축 메타 전파 (SearchResult 재구성 지점 — fusion 2곳과 동기)
material_type=doc.material_type,
jurisdiction=doc.jurisdiction,
published_date=doc.published_date,
))
return fused[:limit]
+1 -1
View File
@@ -47,7 +47,7 @@ logger = setup_logger("synthesis")
# ─── 상수 (plan 영구 룰) ─────────────────────────────────
PROMPT_VERSION = "v2"
LLM_TIMEOUT_MS = 30000 # 2026-05-17 B-3: 15s 시 동시 부하 (Mac mini 26B classifier+evidence+synthesis serialized) 빈발 timeout — classifier (30s) 와 align
LLM_TIMEOUT_MS = 120000 # 2026-06-11 Qwen3.6-27B-6bit 전환: 프리필 ~112 tok/s·디코드 ~11.7 tok/s 실측 — 30s 면 synthesis(답변 본체) 상시 timeout. synthesis 는 graceful skip 불가(=답변 실패)라 단독 상향, config ask.backend.timeout_read_s=120 와 align
CACHE_TTL = 3600 # 1h (answer 는 원문 변경에 민감 → query_analyzer 24h 보다 짧게)
CACHE_MAXSIZE = 300
MAX_ANSWER_CHARS = 600
+4
View File
@@ -24,6 +24,7 @@ from models.chunk import DocumentChunk
from models.document import Document
from models.study_question import StudyQuestion
from models.study_topic import StudyTopicDocument
from services.search.license_filter import restricted_exclude_orm
logger = logging.getLogger(__name__)
@@ -124,11 +125,14 @@ async def _gather_document_evidence(
return []
# 매핑된 documents 메타 (제목·요약 표기)
# B-4: licensed_restricted 제외 → valid_doc_ids 에서 빠지므로 아래 청크 쿼리(doc_id IN)도
# 자동 차단. study 풀이 RAG 도 retrieval/digest 와 동일 단일 술어 공유(a안 U-2①).
doc_meta_rows = (
await session.execute(
select(Document.id, Document.title, Document.ai_summary).where(
Document.id.in_(doc_ids),
Document.deleted_at.is_(None),
restricted_exclude_orm(),
)
)
).all()
+6
View File
@@ -175,10 +175,16 @@ async def _ingest_detail(session, source: NewsSource, url: str) -> str:
ai_domain="Engineering",
ai_sub_group=_SOURCE_NAME,
ai_tags=["Engineering/API 표준 공지"],
# 안전 자료실 A-2 — 표준 '공지' = standard (코드 본문 아님 — ASME/API 본문은 paywall)
material_type="standard",
jurisdiction="US",
published_date=pub_dt.date() if pub_dt else None,
extract_meta={
"source_id": source.id,
"source_name": _SOURCE_NAME,
"published_at": pub_dt.isoformat() if pub_dt else None,
"license": {"scheme": "proprietary", "redistribute": False,
"attribution": "American Petroleum Institute"},
"fulltext": {
"status": "api_announcement",
"engine": engine,
+4
View File
@@ -8,6 +8,7 @@
import asyncio
from datetime import date
from core.config import settings
from core.utils import setup_logger
from services.briefing.pipeline import run_briefing_pipeline
@@ -22,6 +23,9 @@ async def run(target_date: date | None = None) -> dict | None:
Args:
target_date: KST 기준 briefing_date (None = 오늘). API regenerate 명시 지정 가능.
"""
if "briefing" in settings.pipeline_held_stages:
logger.info("[briefing] 보류 (pipeline.held_stages) — 이번 실행 skip")
return None
try:
result = await asyncio.wait_for(
run_briefing_pipeline(target_date),
+4
View File
@@ -311,6 +311,10 @@ async def process(document_id: int, session: AsyncSession) -> None:
country, source, src_lang = await _lookup_news_source(session, doc)
if src_lang:
language = src_lang
# 안전 자료실 A-2 — 뉴스 lookup 미해당(crawl/law/업로드) 문서는 jurisdiction 을
# chunk.country 미러로 (leg 간 국가 일치. EU/INT 도 이 경로로 첫 유입 — String(10) 수용).
if country is None and doc.jurisdiction:
country = doc.jurisdiction
domain_category = "news" if doc.source_channel == "news" else "document"
# 기존 chunks 삭제 (재처리)
+90 -14
View File
@@ -31,12 +31,18 @@ from pydantic import BaseModel, Field, ValidationError
from sqlalchemy import text as sql_text
from sqlalchemy.ext.asyncio import AsyncSession
from ai.client import AIClient, parse_json_response, strip_thinking
from ai.client import (
AIClient,
call_deep_or_defer,
is_deferrable_error,
parse_json_response,
strip_thinking,
)
from ai.envelope import EscalationEnvelope
from core.config import settings
from core.utils import setup_logger
from models.document import Document
from models.queue import enqueue_stage
from models.queue import StageDeferred, enqueue_stage
from policy.prompt_render import render_4b, policy_version as compute_policy_version
from policy.routing import decide_routing
from services.document_telemetry import record_analyze_event
@@ -56,6 +62,15 @@ FACET_DOCTYPES = {"발주서", "세금계산서", "명세표", "도면", "증명
# 자료실 자동 분류 제안 대상 (거래 하위)
LIBRARY_SUGGESTION_DOCTYPES = {"발주서", "세금계산서", "명세표"}
# 안전 자료실 A-2 — document_type → material_type 결정적 매핑 (제안 전용, 자동 전이 금지).
# 모호한 doctype(Reference/Report 등)은 매핑하지 않음 — 무리한 전수 분류 시도 금지 (plan 0-1).
_DOCTYPE_TO_MATERIAL = {
"Law_Document": "law",
"Academic_Paper": "paper",
"Manual": "manual",
"Standard": "standard",
}
# PR-B prompt_version task 이름
SUMMARY_TRIAGE_TASK = "p3a_short_summary"
@@ -345,13 +360,20 @@ _FRONTMATTER_PRESERVED_KEYS = {
# ───────────────────────── main process ────────────────────────────────
async def process(document_id: int, session: AsyncSession) -> None:
async def process(
document_id: int, session: AsyncSession, *, use_deep: bool = False
) -> None:
"""문서 분류 + 요약 + tier triage.
1) Legacy: classify() ai_domain/document_type/ai_tags/ai_confidence/ai_suggestion
2) Legacy: summarize() ai_summary
3) PR-B B-1: summary_triage (4B) ai_tldr/ai_bullets/ai_analysis_tier='triage'
use_deep (2026-06-12 fair-share, queue_drain 전용): triage LLM 호출을 deep 슬롯
(맥북, 라우터 경유)으로 보낸다 sampling triage temperature/max_tokens
유지(분류 결정성), endpoint 교체. 맥북 불가 = StageDeferred 전파(drain
보류 처리). False(기본/consumer) = 기존 call_triage(맥미니 직접) 그대로.
예외 source_channel='law_monitor':
법령은 외부 source-of-truth (law.go.kr) 보유 + immutable + 자동 재수집.
AI 분류는 무가치 + 본문 해석 환각 위험. 26B legacy + 4B triage 전부 skip.
@@ -446,10 +468,20 @@ async def process(document_id: int, session: AsyncSession) -> None:
logger.info(f"doc {document_id}: frontmatter 부분 인식 → LLM 으로 미설정 필드 보완")
client = AIClient()
# fair-share (2026-06-12): use_deep 시 legacy classify/summarize 도 deep 슬롯(맥북)
# 경유 — 그래야 drain 의 "맥북 분담" 이 실제로 성립 (triage 만 보내면 50K 요약
# 프리필이 맥미니에 남는다). deep 슬롯 sampling = primary 와 동일(0.3/0.9/8192).
legacy_cfg = settings.ai.deep if (use_deep and settings.ai.deep is not None) else None
try:
# ─── 1. Legacy classify (primary 26B) ───
# ─── 1. Legacy classify (primary 또는 deep) ───
truncated = doc.extracted_text[:MAX_CLASSIFY_TEXT]
raw_response = await client.classify(truncated)
try:
raw_response = await client.classify(truncated, cfg=legacy_cfg)
except Exception as exc:
if legacy_cfg is not None and is_deferrable_error(exc):
# 맥북 불가 — 첫 호출(최저 비용 지점)에서 보류로 전환, doc 쓰기 0
raise StageDeferred(f"macbook_unavailable:{type(exc).__name__}") from exc
raise
parsed = parse_json_response(raw_response)
if not parsed:
@@ -469,6 +501,24 @@ async def process(document_id: int, session: AsyncSession) -> None:
if not doc.document_type:
doc.document_type = doc_type if doc_type in DOCUMENT_TYPES else "Note"
# ─── 안전 자료실 A-2: material_type 제안 (업로드 경로 — LLM 직접 부여 금지) ───
# document_type → material_type 결정적 매핑만 제안으로 적재 (프롬프트 변경 0).
# 승인(accept-suggestion) 시에만 전이 — law 는 국가 필수 입력 (KR 기본값 오염 차단,
# 자동 전이 금지 사상은 category 와 동일). 수집기 deterministic 경로는 이미 채워져
# 있어(material_type IS NOT NULL) 본 제안 비대상. 거래문서 제안(ai_suggestion 점유)과
# 충돌 시 기존 제안 우선 (두 제안이 겹치는 문서는 실무상 없음 — 거래 vs 안전자료).
_mt_prop = _DOCTYPE_TO_MATERIAL.get(doc.document_type or "")
if _mt_prop and doc.material_type is None and doc.ai_suggestion is None:
doc.ai_suggestion = {
"proposed_material_type": _mt_prop,
"proposed_jurisdiction": None,
"confidence": doc.ai_confidence,
"source_updated_at": (
doc.updated_at.isoformat() if doc.updated_at else None
),
"reason": "document_type→material_type 결정적 매핑",
}
# confidence
confidence = parsed.get("confidence", 0.5)
doc.ai_confidence = max(0.0, min(1.0, float(confidence)))
@@ -517,12 +567,17 @@ async def process(document_id: int, session: AsyncSession) -> None:
"reason": "classify pipeline",
}
# ─── 2. Legacy 요약 (primary 26B) ───
summary = await client.summarize(doc.extracted_text[:50000])
# ─── 2. Legacy 요약 (primary 또는 deep) ───
try:
summary = await client.summarize(doc.extracted_text[:50000], cfg=legacy_cfg)
except Exception as exc:
if legacy_cfg is not None and is_deferrable_error(exc):
raise StageDeferred(f"macbook_unavailable:{type(exc).__name__}") from exc
raise
doc.ai_summary = strip_thinking(summary)
# ─── 메타데이터 (legacy 완료) ───
doc.ai_model_version = settings.ai.primary.model
# ─── 메타데이터 (legacy 완료) — 실제 처리 머신 귀속 (drain=qwen-macbook) ───
doc.ai_model_version = (legacy_cfg or settings.ai.primary).model
doc.ai_processed_at = datetime.now(timezone.utc)
logger.info(
@@ -533,7 +588,9 @@ async def process(document_id: int, session: AsyncSession) -> None:
# ─── 3. PR-B B-1 — tier triage (4B, 실패는 legacy 결과 보존) ───
try:
await _run_tier_triage(client, doc, session)
await _run_tier_triage(client, doc, session, use_deep=use_deep)
except StageDeferred:
raise # 보류는 실패가 아님 — drain/consumer 가 attempts 미소모 처리
except Exception as exc:
logger.exception(f"[triage] id={document_id} 전체 실패 — legacy 유지: {exc}")
@@ -541,8 +598,10 @@ async def process(document_id: int, session: AsyncSession) -> None:
await client.close()
async def _run_tier_triage(client: AIClient, doc: Document, session: AsyncSession) -> None:
"""summary_triage (p3a_short_summary) 경로."""
async def _run_tier_triage(
client: AIClient, doc: Document, session: AsyncSession, *, use_deep: bool = False
) -> None:
"""summary_triage (p3a_short_summary) 경로. use_deep = process() 에서 전달 (drain 전용)."""
document_id = doc.id
text = doc.extracted_text or ""
input_chars = len(text)
@@ -550,6 +609,14 @@ async def _run_tier_triage(client: AIClient, doc: Document, session: AsyncSessio
triage_start = time.perf_counter()
parse_error: str | None = None
triage_out = TriageOutput()
# drain 경유 시 triage 도 deep 슬롯(맥북) — sampling 은 triage 것 유지(결정성).
deep_triage_cfg = None
if use_deep and settings.ai.deep is not None:
deep_triage_cfg = settings.ai.deep.model_copy(update={
"temperature": settings.ai.triage.temperature,
"top_p": settings.ai.triage.top_p,
"max_tokens": settings.ai.triage.max_tokens,
})
# 입력이 triage 한도 초과면 호출 생략하고 long_context 로 escalate
if input_chars > TRIAGE_TEXT_LIMIT:
@@ -590,7 +657,14 @@ async def _run_tier_triage(client: AIClient, doc: Document, session: AsyncSessio
prompt = rendered.replace("{extracted_text}", text[:TRIAGE_TEXT_LIMIT])
try:
raw_triage = await client.call_triage(prompt)
if deep_triage_cfg is not None:
# drain 전용 — deep 슬롯 endpoint + triage sampling. 맥북 불가(StageDeferred)
# 는 아래 generic except 에 먹히지 않게 먼저 전파.
raw_triage = await call_deep_or_defer(client, prompt, cfg=deep_triage_cfg)
else:
raw_triage = await client.call_triage(prompt)
except StageDeferred:
raise # drain 이 attempts 미소모 + 백오프로 처리 (sleep-안전)
except Exception as exc:
logger.warning(
"[triage] 4B 호출 실패 id=%s type=%s repr=%r",
@@ -656,6 +730,7 @@ async def _run_tier_triage(client: AIClient, doc: Document, session: AsyncSessio
escalation_reason=escalation_reason,
parse_error=parse_error,
routing_decision=routing_decision,
model_name=(deep_triage_cfg.model if deep_triage_cfg is not None else None),
)
@@ -670,6 +745,7 @@ async def _apply_triage_result(
escalation_reason: str | None,
parse_error: str | None,
routing_decision=None,
model_name: str | None = None, # fair-share: 실제 호출 경로 모델 (None=triage 기본)
) -> None:
"""TriageOutput → Document 필드 + R2 suppression + envelope enqueue + audit.
@@ -760,7 +836,7 @@ async def _apply_triage_result(
layers_returned=["tldr", "bullets"] if not parse_error else [],
cached=False,
latency_ms=latency_ms,
model_name=settings.ai.triage.model,
model_name=(model_name or settings.ai.triage.model),
prompt_version=(f"{SUMMARY_TRIAGE_TASK}@{pv}" if pv else SUMMARY_TRIAGE_TASK),
error_code=parse_error,
source="document_server",
+12 -1
View File
@@ -202,7 +202,12 @@ async def _ingest_pdf(session, page_slug: str, pdf_url: str) -> bool:
import_source="csb_sitemap",
edit_url=pdf_url,
ai_tags=["Safety/CSB/보고서"],
extract_meta={"csb": {"page_slug": page_slug, "kind": "report_pdf"}},
# 안전 자료실 A-2 — ingest 시점 deterministic. CSB = 미 연방기관 = public domain.
material_type="incident",
jurisdiction="US",
extract_meta={"csb": {"page_slug": page_slug, "kind": "report_pdf"},
"license": {"scheme": "public_domain", "redistribute": True,
"attribution": "U.S. Chemical Safety Board"}},
)
session.add(doc)
await session.flush()
@@ -290,10 +295,16 @@ async def _ingest_url(session, source: NewsSource, url: str, lastmod: datetime)
ai_domain="Safety",
ai_sub_group=_SOURCE_NAME,
ai_tags=["Safety/CSB"],
# 안전 자료실 A-2 — ingest 시점 deterministic (classify-skip 경로)
material_type="incident",
jurisdiction="US",
published_date=lastmod.date() if lastmod else None,
extract_meta={
"source_id": source.id,
"source_name": _SOURCE_NAME,
"published_at": lastmod.isoformat(),
"license": {"scheme": "public_domain", "redistribute": True,
"attribution": "U.S. Chemical Safety Board"},
"fulltext": {
"status": "csb_sitemap",
"engine": engine,
+27 -7
View File
@@ -54,8 +54,18 @@ class DeepSummaryOutput(BaseModel):
confidence: float = 0.5
async def process(document_id: int, session: AsyncSession) -> None:
"""deep_summary 큐 pickup → 26B 호출 → 필드 저장."""
async def process(
document_id: int, session: AsyncSession, *, defer_on_deep_unavailable: bool = False
) -> None:
"""deep_summary 큐 pickup → LLM 호출 → 필드 저장.
defer_on_deep_unavailable:
False (기본, consumer 경로) = 맥북(deep 슬롯) 우선 시도, 불가 즉시
맥미니 primary 처리. 2026-06-12 fair-share: 머신이 동일 모델
(Qwen3.6-27B-6bit)이라 폴백 = 품질 강등이 아니라 단순 분배.
True (queue_drain 전용) = 맥북 불가를 StageDeferred 올려 drain
보류 run 멈춘다 (drain = 맥북 분담 전용 레버 시멘틱 유지).
"""
doc = await session.get(Document, document_id)
if not doc:
raise ValueError(f"deep_summary: document id={document_id} 없음")
@@ -111,16 +121,26 @@ async def process(document_id: int, session: AsyncSession) -> None:
try:
start = time.perf_counter()
if deep_cfg is not None:
# 맥북 경유 — 맥미니 mlx gate 미점유(게이트는 맥미니 보호 목적). 맥북 불가
# (503/연결/생성 중 sleep 절단)는 StageDeferred = 보류, 맥미니 강등 없음.
# doc 쓰기는 완주+파싱 후에만 일어나므로 어느 시점에 끊겨도 부분 쓰기 0.
raw = await call_deep_or_defer(client, prompt)
# 맥북 우선 — 맥미니 mlx gate 미점유(별 endpoint). doc 쓰기는 완주+파싱
# 후에만 일어나므로 어느 시점에 끊겨도 부분 쓰기 0.
try:
raw = await call_deep_or_defer(client, prompt)
except StageDeferred:
if defer_on_deep_unavailable:
raise # drain 전용 — 맥북 레버 시멘틱 (보류 후 run 종료)
# consumer 경로: 동일 모델이라 강등 아님 — 맥미니가 즉시 처리 (2026-06-12)
logger.info(
f"[deep] id={document_id} 맥북 불가 → 맥미니 primary 처리 (fair-share)"
)
used_cfg = settings.ai.primary
async with acquire_mlx_gate(Priority.BACKGROUND):
raw = await client.call_primary(prompt)
else:
async with acquire_mlx_gate(Priority.BACKGROUND): # 2026-05-17 B-1: classify-escalate worker
raw = await client.call_primary(prompt)
latency_ms = int((time.perf_counter() - start) * 1000)
except StageDeferred:
# 보류는 실패가 아님 — analyze_event 미기록(가짜 완료 방지), consumer 가 백오프 기록.
# 보류는 실패가 아님 — analyze_event 미기록(가짜 완료 방지), drain 이 백오프 기록.
logger.info(f"[deep] id={document_id} 맥북 일시 불가 — 보류 (deferred)")
raise
except Exception as exc:
+4
View File
@@ -10,6 +10,7 @@ global_digests / digest_topics 테이블에 저장한다.
import asyncio
from core.config import settings
from core.utils import setup_logger
from services.digest.pipeline import run_digest_pipeline
@@ -24,6 +25,9 @@ async def run() -> None:
pipeline 자체는 timeout 으로 감싸지 않음 (per-call timeout summarizer 처리).
여기서는 전체 hard cap 강제.
"""
if "digest" in settings.pipeline_held_stages:
logger.info("[global_digest] 보류 (pipeline.held_stages) — 이번 실행 skip")
return
try:
result = await asyncio.wait_for(
run_digest_pipeline(),
+38
View File
@@ -58,6 +58,23 @@ SCAN_TARGETS: list[tuple[str, str | None]] = [
("Videos", "video"),
]
# 안전 자료실 A-2/B-4 — watch 타깃별 (material_type, jurisdiction, license) deterministic 축.
# 키 = 타깃 경로의 마지막 성분. license = extract_meta.license 에 그대로 주입(None=미주입).
# restricted=true → retrieval_service._license_sql 가 RAG 증거·digest 에서 제외(a안 U-2① —
# 구매자료 verbatim span 차단, 색인 자체는 허용. 개인 파일 열람은 미차단).
# 사용자 결정(2026-06-13): Books/Papers=proprietary+restricted / Manuals=proprietary·restricted=false
# (검색·RAG 활용) / KGS=법정 위임 상세기준 law/KR·KOGL 공공·restricted 아님.
_TARGET_AXIS: dict[str, tuple[str, str | None, dict | None]] = {
"KGS_Code": ("law", "KR", {"scheme": "kogl", "redistribute": True,
"restricted": False, "attribution": "한국가스안전공사(KGS)"}),
"Books": ("book", None, {"scheme": "proprietary", "redistribute": False,
"restricted": True, "attribution": "구매 도서"}),
"Papers_Purchased": ("paper", None, {"scheme": "proprietary", "redistribute": False,
"restricted": True, "attribution": "구매 논문"}),
"Manuals": ("manual", None, {"scheme": "proprietary", "redistribute": False,
"restricted": False, "attribution": "기술 매뉴얼"}),
}
def should_skip(path: Path) -> bool:
if path.name in SKIP_NAMES or path.name.startswith("._"):
@@ -242,6 +259,11 @@ async def watch_inbox():
if not scan_root.exists():
continue
# 안전 자료실 A-2/B-4 — 타깃 폴더 기반 (material, jurisdiction, license)
target_mt, target_jur, target_license = _TARGET_AXIS.get(
Path(sub).name, (None, None, None)
)
for file_path in scan_root.rglob("*"):
if not file_path.is_file() or should_skip(file_path):
continue
@@ -275,7 +297,14 @@ async def watch_inbox():
source_channel="drive_sync",
category=category,
needs_conversion=needs_conversion,
# 안전 자료실 A-2/B-4 — watch 타깃 매핑 (KGS=law/KR 등, 비대상=NULL)
material_type=target_mt,
jurisdiction=target_jur,
)
# B-4 — 타깃 폴더 license 주입(restricted 포함, 비대상=미주입). classify 는
# material_type IS NULL 일 때만 제안 + extract_meta 미기록이라 주입 보존.
if target_license:
doc.extract_meta = {"license": dict(target_license)}
session.add(doc)
await session.flush()
@@ -291,6 +320,15 @@ async def watch_inbox():
existing.category = category
if needs_conversion and not getattr(existing, "needs_conversion", False):
existing.needs_conversion = True
# B-4 — 축/license 보정(B-4 이전 적재분이 재변경 시): material 미설정 시 주입,
# license 부재 시에만 merge 주입(clobber 회피 — 기존 extract_meta 키 보존).
if existing.material_type is None and target_mt is not None:
existing.material_type = target_mt
existing.jurisdiction = target_jur
if target_license and not (existing.extract_meta or {}).get("license"):
meta = dict(existing.extract_meta or {})
meta["license"] = dict(target_license)
existing.extract_meta = meta
if next_stage:
await enqueue_stage(session, existing.id, next_stage)
+141 -4
View File
@@ -1,14 +1,17 @@
"""C-2 KOSHA Open API 수집 워커 (plan crawl-24x7-1).
3 API (2026-06-10 실키 live 검증 + fixture 박제 tests/fixtures/kosha_*_response.json):
4 API (2026-06-10/06-13 실키 live 검증 + fixture 박제 tests/fixtures/kosha_*_response.json):
재해사례 게시판: GET /B552468/disaster_api02/getdisaster_api02 callApiId=1060
재해사례 첨부: GET /B552468/disaster_attach_api02/Disaster_attach_api02 callApiId=1070
KOSHA GUIDE: GET /B552468/koshaguide/getKoshaGuide callApiId=1050
사망사고 속보: GET /B552468/news_api02/getNews_api02 callApiId=1040
daily 스케줄 1 (main.py):
재해사례 = 최근 페이지만 diff (boardno dedup) 사례 본문 Document(텍스트 네이티브)
+ 첨부 PDF/HWP 다운로드 /documents/crawl_raw/kosha/{boardno}/ 저장
파일 Document + extract enqueue (kordoc HWP/PDF 기존 파이프라인 재사용).
사망사고 = 최근 페이지만 diff (arno dedup) 속보 본문 Document(HTML _clean_html).
첨부 API 없음·business 필드 없음. 등록일 = arno 접두 8자리(YYYYMMDD).
GUIDE = 전체 레지스트리 메타 diff (1039, 100/page = 11 call) 신규/개정만,
일일 ingest cap(기본 25) = backlog 자동 점진 백필(~6) + 부하 평탄화.
cap 으로 미처리 잔량은 매회 로그 (silent cap 금지).
@@ -23,7 +26,7 @@ import hashlib
import os
import random
import re
from datetime import datetime, timezone
from datetime import date, datetime, timezone
from pathlib import Path
import httpx
@@ -38,6 +41,7 @@ from models.news_source import NewsSource
from models.queue import enqueue_stage
from workers.news_collector import (
FeedError,
_clean_html,
_get_or_create_health,
_record_failure,
_record_success,
@@ -49,17 +53,36 @@ _BASE = "https://apis.data.go.kr/B552468"
_BOARD_EP = f"{_BASE}/disaster_api02/getdisaster_api02"
_ATTACH_EP = f"{_BASE}/disaster_attach_api02/Disaster_attach_api02"
_GUIDE_EP = f"{_BASE}/koshaguide/getKoshaGuide"
_FATAL_EP = f"{_BASE}/news_api02/getNews_api02"
_CASE_SOURCE = "KOSHA 재해사례"
_GUIDE_SOURCE = "KOSHA GUIDE"
_FATAL_SOURCE = "KOSHA 사망사고"
_CASE_PAGES = 2 # daily diff 범위 (30×2 = 최근 60건 — 등록일 역순 API)
_CASE_ROWS = 30
_FATAL_PAGES = 2 # 사망사고 속보 daily diff (30×2 = 최근 60건 — 등록일 역순)
_FATAL_ROWS = 30
_GUIDE_ROWS = 100
_GUIDE_DAILY_CAP = int(os.getenv("KOSHA_GUIDE_DAILY_CAP", "25"))
_MAX_FILE_BYTES = 50 * 1024 * 1024
_DOWNLOAD_DELAY = (2.0, 5.0) # portal.kosha.or.kr 파일서버 — 연속 다운로드 간격
# 안전 자료실 A-2 — KOSHA 산출물 라이선스 (KOGL 유형 미확정 → 보수적 redistribute=False,
# 근거 확보 시 완화. 0-3 license 메타 deterministic 주입).
_KOSHA_LICENSE = {"scheme": "kogl", "redistribute": False, "attribution": "한국산업안전보건공단(KOSHA)"}
def _ymd_to_date(ymd: str | None) -> date | None:
"""'YYYYMMDD'/'YYYY-MM-DD' → date. 형식 불일치는 None (fail-quiet — 날짜는 보조 축)."""
digits = re.sub(r"\D", "", ymd or "")
if len(digits) != 8:
return None
try:
return date(int(digits[:4]), int(digits[4:6]), int(digits[6:8]))
except ValueError:
return None
def _api_key() -> str:
key = os.getenv("KOSHA_API_KEY", "")
@@ -93,6 +116,29 @@ def _items(payload: dict) -> list[dict]:
return [item] if isinstance(item, dict) else list(item)
def _fatal_fields(item: dict) -> dict | None:
"""사망사고 item(arno/keyword/contents 3필드 고정) → Document 필드 매핑.
순수 함수(httpx/DB 불요 fixture 단위 테스트 대상). 필수 = arno+keyword,
부재 None(skip). 날짜 전용 필드가 없어 등록 식별자 arno 접두에서 유도:
arno = 'YYYYMMDDHHMMSS' + 임의 6 (2019~ 라이브 전수 동형 검증). 접두 8자리=KST
등록일 published_date, 14자리=등록시각 reg_dt(원문 그대로, tz 해석 미주장).
"""
arno = str(item.get("arno") or "").strip()
title = (item.get("keyword") or "").strip()
if not arno or not title:
return None
text = _clean_html(item.get("contents") or "", max_len=None)
reg_dt = arno[:14] if re.fullmatch(r"\d{14}", arno[:14]) else None
return {
"arno": arno,
"title": title,
"text": text,
"published_date": _ymd_to_date(arno[:8]),
"reg_dt": reg_dt,
}
def _safe_filename(name: str) -> str:
"""NAS 파일명 정화 — 경로분리자/제어문자/공백연쇄 제거 (쉘 함정 회피)."""
name = re.sub(r"[/\\\x00-\x1f]", "_", name).strip()
@@ -155,7 +201,11 @@ async def _ingest_attachment(session, boardno: str, filenm: str, filepath: str)
import_source="kosha_api",
edit_url=filepath,
ai_tags=["Safety/KOSHA재해사례/첨부"],
extract_meta={"kosha": {"boardno": boardno, "kind": "case_attachment"}},
# 안전 자료실 A-2 — ingest 시점 deterministic (classify 경유해도 LLM 비의존)
material_type="incident",
jurisdiction="KR",
extract_meta={"kosha": {"boardno": boardno, "kind": "case_attachment"},
"license": dict(_KOSHA_LICENSE)},
)
session.add(doc)
await session.flush()
@@ -213,12 +263,16 @@ async def collect_disaster_cases(session) -> int:
ai_domain="Safety",
ai_sub_group=_CASE_SOURCE,
ai_tags=[f"Safety/KOSHA재해사례/{business or '기타'}"],
# 안전 자료실 A-2 — ingest 시점 deterministic (classify-skip 경로)
material_type="incident",
jurisdiction="KR",
extract_meta={
"source_id": source.id,
"source_name": _CASE_SOURCE,
"published_at": None,
"kosha": {"boardno": boardno, "business": business,
"atcflcnt": item.get("atcflcnt")},
"license": dict(_KOSHA_LICENSE),
},
)
session.add(doc)
@@ -250,6 +304,83 @@ async def collect_disaster_cases(session) -> int:
return new_count
async def collect_fatal_accidents(session) -> int:
"""사망사고 속보 daily diff — 최근 _FATAL_PAGES 페이지, arno dedup.
재해사례(1060) 채널(1040): business 필드·첨부 API 없음, contents=HTML.
본문 = 텍스트 네이티브(_clean_html) md 변환 비대상, summarize/embed/chunk .
"""
key = _api_key()
source = await _get_or_create_source(session, _FATAL_SOURCE, _FATAL_EP)
new_count = 0
for page in range(1, _FATAL_PAGES + 1):
payload = await _api_get(
f"{_FATAL_EP}?serviceKey={key}&callApiId=1040&pageNo={page}&numOfRows={_FATAL_ROWS}"
)
items = _items(payload)
if not items:
break
page_all_dup = True
for item in items:
fields = _fatal_fields(item)
if fields is None:
continue
arno = fields["arno"]
fhash = hashlib.sha256(f"kosha-fatal|{arno}".encode()).hexdigest()[:32]
existing = await session.execute(
select(Document).where(Document.file_hash == fhash).limit(1)
)
if existing.scalars().first():
continue
page_all_dup = False
text = fields["text"]
now = datetime.now(timezone.utc)
doc = Document(
file_path=f"crawl/{_FATAL_SOURCE}/{arno}",
file_hash=fhash,
file_format="article",
file_size=len(text.encode()),
file_type="note",
title=fields["title"],
extracted_text=f"{fields['title']}\n\n{text}",
extracted_at=now,
extractor_version="kosha_api",
md_status="skipped",
md_extraction_error="kosha fatal: 텍스트 네이티브, markdown 변환 비대상",
source_channel="crawl",
data_origin="external",
review_status="approved",
ai_domain="Safety",
ai_sub_group=_FATAL_SOURCE,
ai_tags=["Safety/KOSHA사망사고"],
# 안전 자료실 A-2 — ingest 시점 deterministic (classify-skip 경로)
material_type="incident",
jurisdiction="KR",
published_date=fields["published_date"],
extract_meta={
"source_id": source.id,
"source_name": _FATAL_SOURCE,
"published_at": None,
"kosha": {"arno": arno, "kind": "fatal_accident",
"reg_dt": fields["reg_dt"]},
"license": dict(_KOSHA_LICENSE),
},
)
session.add(doc)
await session.flush()
await enqueue_stage(session, doc.id, "summarize")
await enqueue_stage(session, doc.id, "embed")
await enqueue_stage(session, doc.id, "chunk")
new_count += 1
if page_all_dup:
break # 등록일 역순 — 페이지 전체가 기존이면 이후 페이지도 기존
logger.info(f"[kosha] 사망사고 신규 {new_count}")
return new_count
async def collect_kosha_guide(session, cap: int = _GUIDE_DAILY_CAP) -> int:
"""GUIDE 레지스트리 전체 메타 diff → 신규/개정만 다운로드 (일일 cap 점진 백필)."""
key = _api_key()
@@ -307,8 +438,13 @@ async def collect_kosha_guide(session, cap: int = _GUIDE_DAILY_CAP) -> int:
import_source="kosha_api",
edit_url=spec["url"],
ai_tags=["Safety/KOSHA GUIDE"],
# 안전 자료실 A-2 — GUIDE = 구속력 없는 권고 기술지침 (law 아님, plan 0-1)
material_type="guide",
jurisdiction="KR",
published_date=_ymd_to_date(spec["ymd"]),
extract_meta={"kosha": {"kind": "guide", "techGdlnNo": spec["no"],
"ofancYmd": spec["ymd"]}},
"ofancYmd": spec["ymd"]},
"license": dict(_KOSHA_LICENSE)},
)
session.add(doc)
await session.flush()
@@ -325,6 +461,7 @@ async def run() -> None:
"""daily 1회 — 소스별 실패 격리 (재해사례 실패가 GUIDE 를 막지 않게)."""
now = datetime.now(timezone.utc)
for name, collector in ((_CASE_SOURCE, collect_disaster_cases),
(_FATAL_SOURCE, collect_fatal_accidents),
(_GUIDE_SOURCE, collect_kosha_guide)):
async with async_session() as session:
result = await session.execute(select(NewsSource).where(NewsSource.name == name))
+18 -1
View File
@@ -6,7 +6,7 @@
import os
import re
from datetime import datetime, timezone
from datetime import date, datetime, timezone
from pathlib import Path
from xml.etree import ElementTree as ET
@@ -262,6 +262,16 @@ async def _save_law_split(
f"개정구분: {revision_type}"
)
# 안전 자료실 A-2 — 공포일 파싱 (law published_date = COALESCE(시행일, 공포일) 계약,
# 본 레거시 워커는 공포일만 보유 — 시행일 기반 버전 체인은 B-1 statute_collector 소관)
_digits = re.sub(r"\D", "", str(proclamation_date or ""))
pub_date = None
if len(_digits) == 8:
try:
pub_date = date(int(_digits[:4]), int(_digits[4:6]), int(_digits[6:8]))
except ValueError:
pub_date = None
doc = Document(
file_path=rel_path,
file_hash=file_hash(file_path),
@@ -272,6 +282,13 @@ async def _save_law_split(
source_channel="law_monitor",
data_origin="work",
category="law",
# 안전 자료실 A-2 — ingest 시점 deterministic. 법령 텍스트 = 저작권법 제7조
# 비보호 저작물 (public domain). 본 워커는 휴면(LAW_OC 미설정)이나 코드 경로 유지.
material_type="law",
jurisdiction="KR",
published_date=pub_date,
extract_meta={"license": {"scheme": "public_domain", "redistribute": True,
"attribution": "국가법령정보센터"}},
user_note=note or None,
)
session.add(doc)
+42 -1
View File
@@ -341,11 +341,35 @@ def _entry_body(source: NewsSource, entry, summary: str) -> tuple[str, str]:
def _build_extract_meta(source: NewsSource, pub_dt: datetime) -> dict:
"""fulltext_worker / 패널이 쓰는 출처 메타 (documents 에 source FK 가 없어 여기 기록)."""
return {
meta = {
"source_id": source.id,
"source_name": source.name,
"published_at": pub_dt.isoformat(),
}
# 안전 자료실 A-2: 소스 레지스트리의 라이선스를 deterministic 주입 (0-3 license 메타).
# P3 다이제스트/발행류가 redistribute=false 소스를 구조적으로 제외하는 게이트 입력.
if source.license_scheme:
meta["license"] = {
"scheme": source.license_scheme,
"redistribute": bool(source.license_redistribute),
"attribution": source.name,
}
return meta
def _material_axis(source: NewsSource) -> tuple[str | None, str | None]:
"""안전 자료실 분류 축 (material_type, jurisdiction) — 레지스트리 deterministic.
- material_type = news_sources.material_type (NULL = 비대상, 뉴스/철학 )
- jurisdiction = source.country 전파. paper NULL 강제
(국제 학술지에 관할 개념 부적합 plan 0-1 계약. 레지스트리 country=US 여도 미전파).
"""
mt = source.material_type
if not mt:
return None, None
if mt == "paper":
return mt, None
return mt, source.country
def _doc_identity(source: NewsSource, source_short: str, category: str) -> dict:
@@ -354,17 +378,22 @@ def _doc_identity(source: NewsSource, source_short: str, category: str) -> dict:
file_path 접두사가 채널 디렉토리. ai_domain 다이제스트/검색 필터의 분기 축이라
crawl 채널이 'News' 오염시키지 않게 분리 (0-5 채널 레벨 분리 사상).
"""
material_type, jurisdiction = _material_axis(source)
if source.source_channel == "crawl":
domain = category if category and category != "Other" else "Domain"
return {
"path_prefix": "crawl",
"ai_domain": domain,
"ai_tags": [f"{domain}/{source_short}"],
"material_type": material_type,
"jurisdiction": jurisdiction,
}
return {
"path_prefix": "news",
"ai_domain": "News",
"ai_tags": [f"News/{source_short}/{category}"],
"material_type": material_type,
"jurisdiction": jurisdiction,
}
@@ -528,6 +557,10 @@ async def _fetch_rss(session, source: NewsSource) -> tuple[int, str]:
ai_domain=ident["ai_domain"],
ai_sub_group=source_short,
ai_tags=ident["ai_tags"],
# 안전 자료실 A-2 — 레지스트리 deterministic (classify-skip 경로라 ingest 시점 필수)
material_type=ident["material_type"],
jurisdiction=ident["jurisdiction"],
published_date=pub_dt.date() if pub_dt else None,
extract_meta=_build_extract_meta(source, pub_dt),
)
session.add(doc)
@@ -661,6 +694,10 @@ async def _fetch_api_guardian(session, source: NewsSource) -> tuple[int, str]:
ai_domain=ident["ai_domain"],
ai_sub_group=source_short,
ai_tags=ident["ai_tags"],
# 안전 자료실 A-2 — 레지스트리 deterministic (classify-skip 경로라 ingest 시점 필수)
material_type=ident["material_type"],
jurisdiction=ident["jurisdiction"],
published_date=pub_dt.date() if pub_dt else None,
extract_meta=_build_extract_meta(source, pub_dt),
)
session.add(doc)
@@ -757,6 +794,10 @@ async def _fetch_api_nyt(session, source: NewsSource) -> tuple[int, str]:
ai_domain=ident["ai_domain"],
ai_sub_group=source_short,
ai_tags=ident["ai_tags"],
# 안전 자료실 A-2 — 레지스트리 deterministic (classify-skip 경로라 ingest 시점 필수)
material_type=ident["material_type"],
jurisdiction=ident["jurisdiction"],
published_date=pub_dt.date() if pub_dt else None,
extract_meta=_build_extract_meta(source, pub_dt),
)
session.add(doc)
+142
View File
@@ -0,0 +1,142 @@
"""Phase 2A 후보 임베딩 백필 CLI (embedding-phase2a-1 E-1).
docker compose exec -T fastapi python -m workers.phase2a_cand_backfill \
--target qwen06 --doc-id-max 41944 --chunk-id-max 104140 [--batch 32]
설계 원칙 (plan r3):
- resumable/idempotent: 대상 = NOT EXISTS(후보 테이블) 중단/재실행 이어서.
배치 단위 커밋. C-1 백필 게이트 = "후보 카운트 == 동결셋 카운트".
- 동결셋: id <= *_id_max AND 베이스라인 embedding IS NOT NULL (AND docs.deleted_at IS NULL).
cand 테이블은 동결 범위로만 INSERT (retrieval cand path snapshot filter 타는 전제).
- 문서/청크 입력 = production 경로와 동일 구성(embed_worker._build_embed_input /
chunk_worker [제목][섹션][본문]) + plain (instruct prefix 쿼리 전용 G-1 불변식).
- 임베딩 = Ollama /api/embed 배치 호출 (G-1 fixture: 정규화 출력).
- qwen4m CLI 대상이 아님 qwen4 적재 SQL 파생(subvector+l2_normalize), plan E-1.
"""
import argparse
import asyncio
import hashlib
import time
import httpx
from sqlalchemy import text
from core.database import async_session
from core.utils import setup_logger
from models.document import Document
from workers.embed_worker import _build_embed_input
logger = setup_logger("phase2a_cand_backfill")
OLLAMA_EMBED = "http://ollama:11434/api/embed"
TARGETS = {
"qwen06": {
"model": "qwen3-embedding:0.6b", "dim": 1024,
"docs": "documents_cand_qwen06", "chunks": "document_chunks_cand_qwen06",
},
"qwen4": {
"model": "qwen3-embedding:4b", "dim": 2560,
"docs": "documents_cand_qwen4", "chunks": "document_chunks_cand_qwen4",
},
}
async def _embed_batch(client: httpx.AsyncClient, model: str, texts: list[str]) -> list[list[float]]:
r = await client.post(OLLAMA_EMBED, json={"model": model, "input": texts}, timeout=600)
r.raise_for_status()
embs = r.json()["embeddings"]
if len(embs) != len(texts):
raise RuntimeError(f"embed count mismatch: {len(embs)} != {len(texts)}")
return embs
async def backfill_docs(target: dict, doc_id_max: int, batch: int, http: httpx.AsyncClient) -> int:
total = 0
while True:
async with async_session() as session:
rows = (await session.execute(text(f"""
SELECT d.id FROM documents d
WHERE d.id <= :m AND d.embedding IS NOT NULL AND d.deleted_at IS NULL
AND NOT EXISTS (SELECT 1 FROM {target['docs']} c WHERE c.doc_id = d.id)
ORDER BY d.id LIMIT :b
"""), {"m": doc_id_max, "b": batch})).scalars().all()
if not rows:
break
docs = [(await session.get(Document, i)) for i in rows]
inputs = [_build_embed_input(d) for d in docs]
embs = await _embed_batch(http, target["model"], inputs)
for d, inp, e in zip(docs, inputs, embs):
await session.execute(text(f"""
INSERT INTO {target['docs']} (doc_id, embed_input_hash, embedding)
VALUES (:i, :h, cast(:e AS vector))
ON CONFLICT (doc_id) DO NOTHING
"""), {"i": d.id, "h": hashlib.sha256(inp.encode()).hexdigest()[:16], "e": str(e)})
await session.commit()
total += len(rows)
if total % (batch * 10) < batch:
logger.info(f"[{target['docs']}] +{total} (last id={rows[-1]})")
return total
async def backfill_chunks(target: dict, chunk_id_max: int, batch: int, http: httpx.AsyncClient) -> int:
total = 0
while True:
async with async_session() as session:
rows = (await session.execute(text(f"""
SELECT c.id, c.doc_id, c.chunk_index, c.section_title, c.text, d.title
FROM corpus_chunks c JOIN documents d ON d.id = c.doc_id
WHERE c.id <= :m AND c.embedding IS NOT NULL AND d.deleted_at IS NULL
AND NOT EXISTS (SELECT 1 FROM {target['chunks']} k WHERE k.id = c.id)
ORDER BY c.id LIMIT :b
"""), {"m": chunk_id_max, "b": batch})).all()
if not rows:
break
inputs = [
f"[제목] {r.title or ''}\n[섹션] {r.section_title or ''}\n[본문] {r.text}"
for r in rows
]
embs = await _embed_batch(http, target["model"], inputs)
for r, e in zip(rows, embs):
await session.execute(text(f"""
INSERT INTO {target['chunks']} (id, doc_id, chunk_index, section_title, text, embedding)
VALUES (:i, :d, :x, :s, :t, cast(:e AS vector))
ON CONFLICT (id) DO NOTHING
"""), {"i": r.id, "d": r.doc_id, "x": r.chunk_index,
"s": r.section_title, "t": r.text, "e": str(e)})
await session.commit()
total += len(rows)
if total % (batch * 10) < batch:
logger.info(f"[{target['chunks']}] +{total} (last id={rows[-1]})")
return total
async def run(target_key: str, doc_id_max: int, chunk_id_max: int, batch: int) -> None:
target = TARGETS[target_key]
start = time.monotonic()
async with httpx.AsyncClient() as http:
nd = await backfill_docs(target, doc_id_max, batch, http)
nc = await backfill_chunks(target, chunk_id_max, batch, http)
mins = (time.monotonic() - start) / 60
async with async_session() as session:
cd = (await session.execute(text(f"SELECT count(*) FROM {target['docs']}"))).scalar_one()
cc = (await session.execute(text(f"SELECT count(*) FROM {target['chunks']}"))).scalar_one()
logger.info(
f"[{target_key}] 완료 — 이번 run docs +{nd} chunks +{nc} ({mins:.1f}분) · "
f"누적 docs {cd} / chunks {cc} (동결 게이트 = 베이스라인 동결셋 카운트와 일치 확인)"
)
def main() -> None:
p = argparse.ArgumentParser(description="Phase 2A 후보 임베딩 백필 (resumable)")
p.add_argument("--target", required=True, choices=sorted(TARGETS))
p.add_argument("--doc-id-max", type=int, required=True)
p.add_argument("--chunk-id-max", type=int, required=True)
p.add_argument("--batch", type=int, default=32)
a = p.parse_args()
asyncio.run(run(a.target, a.doc_id_max, a.chunk_id_max, a.batch))
if __name__ == "__main__":
main()
+46 -3
View File
@@ -13,18 +13,25 @@ from sqlalchemy import select, update, delete, exists
from sqlalchemy.exc import IntegrityError, SQLAlchemyError
from sqlalchemy.orm import aliased
from core.config import settings
from core.database import async_session
from core.utils import setup_logger
from models.queue import ProcessingQueue, StageDeferred, enqueue_stage, not_deferred_condition
logger = setup_logger("queue_consumer")
# pipeline.held_stages 안내 로그는 1분 사이클마다 반복하지 않고 최초 1회만.
_hold_logged = False
# stage별 배치 크기
# stt 는 GPU 단일 점유 + 회의 30분짜리도 가능 → 배치 1. thumbnail 은 ffmpeg subprocess 로 가벼움.
# deep_summary (PR-B B-1) 는 MLX 26B 단일 Semaphore(1) 경유 → 배치 1.
# fulltext 는 politeness 지연(같은 도메인 5–15s)이 배치 내 직렬로 걸린다 — 배치 3 이면
# 같은 도메인 최악 ~45s/사이클, 메인 큐 1m 간격(max_instances=1, coalesce)이 흡수.
BATCH_SIZE = {"extract": 5, "classify": 3, "summarize": 3, "embed": 1, "chunk": 1,
# embed/chunk 1→10 (2026-06-12 fast-consumer): 건당 <1s 실측 — Phase 0.1 초기 보수값이
# LLM 사이클에 인질로 잡혀 실효 ~580/일 vs 수요 최대 2,700/일 → 적체 원인이었음.
# 10 = TEI/marker 와 GPU 공유 고려한 보수 상향(전용 1분 잡 기준 캡 ~14,400/일).
BATCH_SIZE = {"extract": 5, "classify": 3, "summarize": 3, "embed": 10, "chunk": 10,
"preview": 2, "stt": 1, "thumbnail": 3, "deep_summary": 1, "markdown": 1,
"fulltext": 3}
STALE_THRESHOLD_MINUTES = 10
@@ -34,14 +41,21 @@ STALE_THRESHOLD_MINUTES = 10
# 따라서 markdown consumer 는 별도의 generous 임계를 쓴다.
MARKDOWN_STALE_THRESHOLD_MINUTES = int(os.getenv("MARKDOWN_STALE_MINUTES", "120"))
# consume_queue(메인) 가 담당하는 stage. markdown 은 consume_markdown_queue 로 분리.
# consume_queue(메인) 가 담당하는 stage. markdown 은 consume_markdown_queue,
# embed/chunk 는 consume_fast_queue (2026-06-12) 로 분리 — 세 집합은 disjoint
# (reset_stale_items 가 자기 집합만 reset, 교차 시 이중 복구 위험).
# STT 도 장기 작업 가능성이 있으나 본 PR 범위 밖 — main 에 유지(follow-up).
MAIN_QUEUE_STAGES = [
"extract", "classify", "summarize", "embed", "chunk",
"extract", "classify", "summarize",
"preview", "stt", "thumbnail", "deep_summary", "fulltext",
]
MARKDOWN_QUEUE_STAGES = ["markdown"]
# 고속(비-LLM·경량 GPU) stage — LLM 사이클(분 단위)에서 분리해 1분 잡 전용 소비.
# embed/chunk 는 건당 <1s 라 main 루프에 두면 classify(~190s×3) 뒤에서 굶는다
# (2026-06-12 실측: 적체 3,570 · 4070 가동률 0%). markdown 분리(05-01)와 동일 패턴.
FAST_QUEUE_STAGES = ["embed", "chunk"]
async def reset_stale_items(stages, threshold_minutes=STALE_THRESHOLD_MINUTES):
"""processing 상태로 오래 방치된 항목 복구 (지정 stage 한정)
@@ -335,14 +349,43 @@ async def _process_stage(stage, worker_fn):
async def consume_queue():
"""메인 큐 소비자 — markdown 제외 전 stage 를 1분 간격으로 처리."""
global _hold_logged
workers = _load_workers()
held = [s for s in MAIN_QUEUE_STAGES if s in settings.pipeline_held_stages]
if held and not _hold_logged:
logger.info(f"pipeline.held_stages 보류 중: {held} — claim 하지 않음 (pending 적체 = 의도)")
_hold_logged = True
try:
await reset_stale_items(MAIN_QUEUE_STAGES, STALE_THRESHOLD_MINUTES)
except Exception:
logger.exception("stale reset failed, but continuing queue consumption")
for stage in MAIN_QUEUE_STAGES:
if stage in settings.pipeline_held_stages:
continue
await _process_stage(stage, workers[stage])
async def consume_fast_queue():
"""embed/chunk 전용 고속 소비자 — LLM 사이클과 완전 디커플 (2026-06-12).
main 루프는 classify/summarize/deep 사이클을 단위로 점유해 건당 <1s 짜리
embed/chunk 사이클당 1번씩만 기회를 얻었다 (실효 ~60/ = 적체 원인).
분리 = 1 × 배치 10 ~600/. APScheduler max_instances=1 이라
배치가 1분을 넘으면 다음 fire coalesce (폭주 방지).
"""
workers = _load_workers()
try:
await reset_stale_items(FAST_QUEUE_STAGES, STALE_THRESHOLD_MINUTES)
except Exception:
logger.exception("fast stale reset failed, but continuing queue consumption")
for stage in FAST_QUEUE_STAGES:
if stage in settings.pipeline_held_stages:
continue
await _process_stage(stage, workers[stage])
+18 -6
View File
@@ -30,9 +30,12 @@ from models.queue import ProcessingQueue, StageDeferred, not_deferred_condition
logger = setup_logger("queue_drain")
# summarize = 맥미니 백로그 본체 / deep_summary = 심층 (consumer 도 deep 슬롯 시 맥북 경유).
# classify 는 triage 경량 호출이라 맥미니 적합 — 대상에서 제외 (plan Q-4).
DRAIN_STAGES = ("summarize", "deep_summary")
# summarize = 맥미니 백로그 본체 / deep_summary = 심층 / classify = triage 분류.
# classify 는 2026-06-12 fair-share 로 합류 — 구 제외 사유(plan Q-4 "triage 경량 = 맥미니
# 적합")는 Gemma a4b(42 tok/s) 전제. Qwen 27B 전환 후 classify 가 장문 프리필로 컨슈머
# 사이클을 점유하는 최대 병목이라, 맥북(프리필 ~5배)이 가장 효과적인 분담처다.
# classify 완료 시 enqueue_next_stage(embed/chunk/markdown) 필수 — 누락 = DAG 단절.
DRAIN_STAGES = ("summarize", "deep_summary", "classify")
async def _claim_one(stage: str) -> tuple[int, int] | None:
@@ -98,14 +101,16 @@ async def _mark_failed(queue_id: int, exc: Exception) -> None:
async def drain(stage: str, limit: int, defer_retries: int = 5, defer_wait: int = 120) -> None:
if stage not in DRAIN_STAGES:
raise SystemExit(f"--stage 는 {DRAIN_STAGES} 만 허용 (classify 등은 맥미니 적합 — plan Q-4)")
raise SystemExit(f"--stage 는 {DRAIN_STAGES} 만 허용")
if settings.ai.deep is None:
raise SystemExit(
"config.yaml ai.models.deep 슬롯 미구성 — drain 은 맥북 분담 전용 레버라 진행하지 않음"
" (맥미니로의 silent 강등 금지)"
)
from workers.classify_worker import process as classify_process
from workers.deep_summary_worker import process as deep_summary_process
from workers.queue_consumer import enqueue_next_stage
from workers.summarize_worker import process as summarize_process
done = failed = 0
@@ -121,11 +126,18 @@ async def drain(stage: str, limit: int, defer_retries: int = 5, defer_wait: int
async with async_session() as worker_session:
if stage == "summarize":
await summarize_process(document_id, worker_session, use_deep=True)
elif stage == "classify":
await classify_process(document_id, worker_session, use_deep=True)
else:
# deep_summary 는 deep 슬롯 구성 시 워커가 자체적으로 맥북 경유
await deep_summary_process(document_id, worker_session)
# deep_summary: drain 은 맥북 전용 레버 — 불가 시 보류(폴백은 consumer 만)
await deep_summary_process(
document_id, worker_session, defer_on_deep_unavailable=True
)
await worker_session.commit()
await _mark_completed(queue_id)
# 다음 stage 연쇄 — classify 는 embed/chunk/markdown enqueue (consumer 와 동형,
# summarize/deep_summary 는 next_stages 미등록이라 no-op)
await enqueue_next_stage(document_id, stage)
done += 1
consecutive_defers = 0
logger.info(f"[drain:{stage}] {done}/{limit} doc={document_id} 완료")
+43
View File
@@ -0,0 +1,43 @@
"""statute_collector 나라별 어댑터 패키지 (plan safety-library-1 B-1).
어댑터 계약 (2함수 + 상수):
JURISDICTION: str 어댑터 상수 고정. 코어가 적재 직전 assert (파싱 결과 추론 금지).
poll_changes(client, watch_rows) -> list[ChangeEvent] 개정 감지만 (경량 호출).
fetch_version(client, act, change) -> list[VersionPayload] PR②.
payload 리스트: primary + annex 각각 자기 version_key (R4-M4).
ChangeEvent.kind: amend / repeal / bootstrap(합성 PR② 부트스트랩이 amend
동일 ingest 경로 재사용, R6-m2).
"""
from dataclasses import dataclass, field
@dataclass
class ChangeEvent:
"""개정 감지 이벤트 — poll_changes 산출물."""
family_id: str
kind: str # amend / repeal / bootstrap
new_version_key: str # KR = MST (법령일련번호)
title: str
promulgation_date: str | None = None # YYYYMMDD
effective_date: str | None = None # YYYYMMDD (목록 시행일자 — 조문별 차등 시행 주의)
revision_type: str | None = None # 제개정구분명
@dataclass
class VersionPayload:
"""fetch_version 산출물 1건 — primary 또는 annex 각자 자기 version_key (R4-M4).
전문 1 스냅샷 의미론(R7-M3 fixture 판정): 응답에서 primary + annex 전부 생성.
annex version_key = 'MST|{별표번호}-{별표가지번호}' (zero-padded 구조화 필드 그대로
suffix 문자열 파싱 아닌 필드 기반, R7-B1 a 업그레이드).
"""
law_doc_kind: str # primary / annex
version_key: str
title: str
content: str # 조문/별표 markdown 텍스트
promulgation_date: str | None = None # YYYYMMDD (본문 기본정보)
effective_date: str | None = None # YYYYMMDD (본문 기본정보 — 목록값과 다를 수 있음)
annex_label: str | None = None # '별표1' / '별표5의2' (표시용)
meta: dict = field(default_factory=dict)
+213
View File
@@ -0,0 +1,213 @@
"""KR 법령 어댑터 — 국가법령정보센터 (law.go.kr DRF) (plan safety-library-1 B-1 PR①).
poll_changes = lawSearch 목록 diff: 워치리스트 행별 정식 법령명 exact 조회
MST(법령일련번호) != watermark 이면 ChangeEvent. law_monitor 검증된 호출 형태 재사용.
fixture (2026-06-13 라이브 박제, tests/fixtures/statute_kr/):
- lawsearch_*.xml 목록 필드: 법령ID(불변)·법령일련번호(MST)·공포일자·시행일자·제개정구분명
- lawservice_*.xml.gz 전문 1 XML: 조문단위 853(산안기준규칙) + 별표단위 23 전부 포함
= 스냅샷 의미론 확정(R7-M3 : annex 부분 fetch 실패 개념 없음 같은 응답에 없는
별표 = 삭제 간주 가능). 별표번호+별표가지번호 = 구조화 필드(R7-M3 suffix 문자열
파싱 불요, version_key 합성은 필드 기반. PR② fetch_version 소관).
- 조문 취득 방식 판정(R2-m1): 전문 1 + 로컬 파싱 확정 lawjosub 단위 호출이면
산안기준규칙(853) 개정당 호출 폭증. lawjosub fixture 보조 박제.
주의: 응답의 '법령상세링크' 필드에 OC 키가 포함됨 fixture/로그에 raw 응답을 남길
새니타이즈 필수 (repo fixture __OC_REDACTED__ 처리됨).
"""
import asyncio
import os
import xml.etree.ElementTree as ET
import httpx
from core.crawl_politeness import CRAWL_UA
from core.utils import setup_logger
from workers.statute_adapters import ChangeEvent, VersionPayload
logger = setup_logger("statute_kr")
JURISDICTION = "KR"
SOURCE_API = "law.go.kr"
LAW_SEARCH_URL = "https://www.law.go.kr/DRF/lawSearch.do"
LAW_SERVICE_URL = "https://www.law.go.kr/DRF/lawService.do"
# 같은 도메인 연속 호출 간격 (일 1회 x 26콜 — 보수적)
_POLL_DELAY_S = 1.5
def _oc() -> str:
oc = os.getenv("LAW_OC", "")
if not oc:
raise RuntimeError("LAW_OC 미설정 — statute KR 어댑터 사용 불가")
return oc
def parse_search_hit(xml_text: str, official_title: str) -> dict | None:
"""lawSearch XML 에서 정식 법령명 exact match 1건 추출 (순수 함수 — fixture 테스트 대상).
정식명 기준 exact match 워치리스트 title 정식명(가운뎃점 포함)이므로 안전.
(law_monitor 하드코딩 '유해위험작업...'( 없음) 영구 미매칭이던 함정의 교훈:
조회 키는 반드시 레지스트리의 정식명을 쓴다.)
"""
root = ET.fromstring(xml_text)
for law in root.findall(".//law"):
if (law.findtext("법령명한글") or "").strip() != official_title:
continue
mst = (law.findtext("법령일련번호") or "").strip()
if not mst:
continue
return {
"mst": mst,
"law_id": (law.findtext("법령ID") or "").strip(),
"promulgation_date": (law.findtext("공포일자") or "").strip() or None,
"effective_date": (law.findtext("시행일자") or "").strip() or None,
"revision_type": (law.findtext("제개정구분명") or "").strip() or None,
"status_code": (law.findtext("현행연혁코드") or "").strip() or None,
}
return None
def detect_change(hit: dict | None, act_family_id: str, act_title: str,
watermark: str | None) -> ChangeEvent | None:
"""목록 hit + 워터마크 → ChangeEvent (순수 함수 — fixture 테스트 대상).
- hit 없음 = 감지 불가 (None 호출측이 fail-loud 로그. 폐지 단정 금지:
검색 누락/표기 변경 가능성과 구분 불가하므로 repeal 제개정구분명 기준만)
- MST == watermark = 변경 없음
- 제개정구분명에 '폐지' = repeal, = amend
"""
if hit is None:
return None
if watermark and hit["mst"] == watermark:
return None
kind = "repeal" if (hit.get("revision_type") or "").find("폐지") >= 0 else "amend"
return ChangeEvent(
family_id=act_family_id,
kind=kind,
new_version_key=hit["mst"],
title=act_title,
promulgation_date=hit.get("promulgation_date"),
effective_date=hit.get("effective_date"),
revision_type=hit.get("revision_type"),
)
def _article_markdown(art: ET.Element) -> str:
"""조문단위 1건 → 텍스트. 조문내용(이미 '제N조(제목) ...' 형태) + 항/호/목 전체.
메타 필드(조문번호/조문여부/조문시행일자 ) 제외 조문내용과 서브트리만.
"""
parts = []
body = (art.findtext("조문내용") or "").strip()
if body:
parts.append(body)
for hang in art.findall(""):
text = "\n".join(t.strip() for t in hang.itertext() if t.strip())
if text:
parts.append(text)
return "\n".join(parts)
def parse_service_payloads(xml_text: str, official_title: str, mst: str) -> list[VersionPayload]:
"""lawService 전문 XML → VersionPayload 리스트 (순수 함수 — fixture 테스트 대상).
스냅샷 의미론: 응답에 있는 별표가 버전의 별표 전체 (R7-M3 fixture 판정).
- primary 1: 조문 markdown (조문여부 != '조문' = / 헤더 '## ' 처리)
- annex N건: 별표단위별 version_key = 'MST|{별표번호}-{가지번호}' (zero-padded 그대로)
"""
root = ET.fromstring(xml_text)
base = root.find(".//기본정보")
prom = (base.findtext("공포일자") or "").strip() or None if base is not None else None
eff = (base.findtext("시행일자") or "").strip() or None if base is not None else None
lines: list[str] = [f"# {official_title}", ""]
for art in root.findall(".//조문단위"):
is_article = (art.findtext("조문여부") or "").strip() == "조문"
text = _article_markdown(art)
if not text:
continue
if is_article:
lines.append(f"### {text}" if not text.startswith("") else text)
else:
lines.append(f"## {text}")
lines.append("")
primary_content = "\n".join(lines).strip()
payloads = [VersionPayload(
law_doc_kind="primary",
version_key=mst,
title=official_title,
content=primary_content,
promulgation_date=prom,
effective_date=eff,
)]
for annex in root.findall(".//별표단위"):
no = (annex.findtext("별표번호") or "").strip()
sub = (annex.findtext("별표가지번호") or "").strip() or "00"
kind = (annex.findtext("별표구분") or "별표").strip() # 별표 / 서식 — 별도 차원!
a_title = (annex.findtext("별표제목") or "").strip()
a_body = (annex.findtext("별표내용") or "").strip()
if not no:
continue
# 삭제 tombstone — KR 은 별표/서식 삭제가 absence 가 아니라 '삭제 <날짜>' 명시 행
# (fixture 실측: 산안기준규칙 서식1·2). 내용 없는 tombstone 은 적재 skip.
# 시리즈의 구버전 current 잔존 처리 = PR③ 관찰 후보 (absence 추론은 불요 확정).
if a_title.startswith("삭제") and len(a_body) < 50:
continue
label = f"{kind}{int(no)}" + (f"{int(sub)}" if sub not in ("", "0", "00") else "")
payloads.append(VersionPayload(
law_doc_kind="annex",
# 구분 차원 포함 — (번호,가지)만으로는 별표1 vs 서식1 충돌 (fixture 실측)
version_key=f"{mst}|{kind}{no}-{sub}",
title=f"{official_title} {label} {a_title}".strip(),
content=f"# {official_title} {label}\n## {a_title}\n\n{a_body}".strip(),
promulgation_date=prom,
effective_date=eff,
annex_label=label,
))
return payloads
async def fetch_version(client: httpx.AsyncClient, act, change: ChangeEvent) -> list[VersionPayload]:
"""전문 1콜 → payload 리스트 (R2-m1 판정: lawjosub 조 단위 호출 안 함 — 853조 폭증 회피)."""
resp = await client.get(
LAW_SERVICE_URL,
params={"OC": _oc(), "target": "law", "MST": change.new_version_key, "type": "XML"},
headers={"User-Agent": CRAWL_UA},
)
resp.raise_for_status()
payloads = parse_service_payloads(resp.text, act.title, change.new_version_key)
if not payloads or len(payloads[0].content) < 200:
# 파싱 검증 floor — 미달 시 예외 = 워터마크 미영속 (재시도 가능 상태 유지)
raise ValueError(f"전문 파싱 결과 빈약 ({act.family_id}): payloads={len(payloads)}")
return payloads
async def poll_changes(client: httpx.AsyncClient, watch_rows: list) -> list[ChangeEvent]:
"""워치리스트 행별 lawSearch diff. 행 단위 실패 격리 (한 법령 실패가 나머지를 막지 않음)."""
oc = _oc()
events: list[ChangeEvent] = []
for act in watch_rows:
try:
resp = await client.get(
LAW_SEARCH_URL,
params={"OC": oc, "target": "law", "type": "XML", "query": act.title},
headers={"User-Agent": CRAWL_UA},
)
resp.raise_for_status()
hit = parse_search_hit(resp.text, act.title)
if hit is None:
# fail-loud: 정식명 미매칭 = 표기 변경/검색 누락 의심 — 침묵 skip 금지
logger.warning(f"[statute-kr] 목록 미매칭: {act.family_id} {act.title!r}")
else:
ev = detect_change(hit, act.family_id, act.title, act.watermark)
if ev:
events.append(ev)
except Exception as e:
logger.error(f"[statute-kr] poll 실패 ({act.family_id}): {type(e).__name__}: {e!r}")
await asyncio.sleep(_POLL_DELAY_S)
return events
+381
View File
@@ -0,0 +1,381 @@
"""statute_collector — 법령 수집 코어 (plan safety-library-1 B-1, PR②).
구성 ( 코드 통째 R8-B1: 승격과 스윕의 PR 분리 = 배포 이중 노출 윈도):
poll_changes(어댑터) fetch_version(전문 1, payload 리스트) ingest( 버전
pending 적재 + 4 주입) 생애주기 (버전 시리즈 단위 승격·supersede + 상태 기반
레거시 스윕 + repeal 단일 트랜잭션, KST 기준).
핵심 계약 (카드 = 스펙):
- 워터마크 영속 = ingest 파싱 검증 통과 후에만 (실패 다음 폴링이 재감지)
- 승격·supersede 단위 = 버전 시리즈 = (family_id, law_doc_kind, annex 식별자)
R7-B1: family 단위 구현 금지 (annex 승격이 primary 소거하는 본문 소실 경로)
- 레거시 스윕 = 상태 기반: 실행, primary 시리즈 current 보유 + repeal 미감지
family 법령명 매핑 레거시(law_monitor 스냅샷) 청크 in_corpus=false (멱등)
- 매핑 = 정확 일치 가정 금지: title '법령명 (YYYYMMDD)' 패턴에서 법령명 추출
정규화(공백·가운뎃점 변형 흡수) **동등** 비교 prefix 비교 금지 ('산업안전보건법'
'산업안전보건법 시행령' 레거시를 오폭하는 경로 차단)
- ingest 4 (R8-M1): material_type='law' / jurisdiction=어댑터 상수 /
published_date=COALESCE(시행일, 공포일) / license=public_domain(저작권법 제7조)
- 부트스트랩(--bootstrap) = kind='bootstrap' 합성 이벤트, amend 동일 경로 +
extract_meta.backfill=true (E-1 게이트 집계 제외 마커)
- 가시성: source_health 성공/실패 기록 (HC.io 2026-05-30 알림 레이어 폐기로 부재
silent-skip 가드 정신은 crawl-health 보드 + health 행으로 대체)
실행:
스케줄 = daily 07:00 KST (main.py law_monitor 슬롯 승계)
수동 = docker compose exec -T fastapi python -m workers.statute_collector [--bootstrap]
"""
import argparse
import asyncio
import hashlib
import re
import unicodedata
from datetime import date, datetime, timezone
from zoneinfo import ZoneInfo
import httpx
from sqlalchemy import select, update
from core.database import async_session
from core.utils import setup_logger
from models.chunk import DocumentChunk
from models.document import Document
from models.legal_act import LegalAct, LegalMeta
from models.news_source import NewsSource
from models.queue import enqueue_stage
from workers.news_collector import _get_or_create_health, _record_failure, _record_success
from workers.statute_adapters import ChangeEvent, VersionPayload
from workers.statute_adapters import kr
logger = setup_logger("statute_collector")
_KST = ZoneInfo("Asia/Seoul")
_SOURCE_NAME = "KR 법령 (law.go.kr)"
_LICENSE = {"scheme": "public_domain", "redistribute": True, "attribution": "국가법령정보센터"}
_FETCH_DELAY_S = 2.5 # lawService 전문(최대 ~1.3MB) 연속 호출 간격
# jurisdiction → 어댑터 모듈 (Phase 1 = KR 단독, 해외는 B-5 게이트 뒤)
_ADAPTERS = {"KR": kr}
# ─── 법령명 매핑 (R8-m1: 정확 일치 가정 금지 — 변형 흡수 정규화 + 동등 비교) ───
_LEGACY_TITLE_RE = re.compile(r"^(.*?)\s*\((\d{8})\)")
def normalize_law_name(name: str) -> str:
"""공백·가운뎃점 변형 흡수 — NFC 정규화 후 공백/ㆍ·・ 제거."""
s = unicodedata.normalize("NFC", name or "")
return re.sub(r"[\sㆍ·・]", "", s)
def legacy_law_name(title: str) -> str | None:
"""레거시 law_monitor title('법령명 (YYYYMMDD) 섹션')에서 법령명 추출."""
m = _LEGACY_TITLE_RE.match(title or "")
return m.group(1).strip() if m else None
def series_suffix(version_key: str) -> str | None:
"""버전 시리즈의 annex 식별자 — version_key 'MST|NNNN-SS''|' 뒤 (primary=None)."""
return version_key.split("|", 1)[1] if "|" in version_key else None
def _to_date(ymd: str | None) -> date | None:
digits = re.sub(r"\D", "", ymd or "")
if len(digits) != 8:
return None
try:
return date(int(digits[:4]), int(digits[4:6]), int(digits[6:8]))
except ValueError:
return None
# ─── ingest (전 버전 pending 적재 — R2-B2/R3 계약) ──────────────────────────────
async def _ingest_payload(session, act: LegalAct, ev: ChangeEvent,
payload: VersionPayload, backfill: bool) -> bool:
"""payload 1건 → Document + legal_meta(pending). 반환 = 신규 여부 (dedup 멱등)."""
fhash = hashlib.sha256(
f"statute|{act.jurisdiction}|{act.native_id}|{payload.version_key}".encode()
).hexdigest()[:32]
existing = await session.execute(
select(Document.id).where(Document.file_hash == fhash).limit(1)
)
if existing.scalars().first():
return False
prom = _to_date(payload.promulgation_date or ev.promulgation_date)
eff = _to_date(payload.effective_date or ev.effective_date)
now = datetime.now(timezone.utc)
extra = {"backfill": True} if backfill else {}
doc = Document(
file_path=f"crawl/statute/{act.family_id}/{payload.version_key.replace('|', '_')}",
file_hash=fhash,
file_format="article",
file_size=len(payload.content.encode()),
file_type="note",
title=f"{payload.title} ({payload.promulgation_date or ev.promulgation_date or ''})".strip(),
extracted_text=payload.content,
extracted_at=now,
extractor_version="statute_kr@law.go.kr",
md_status="skipped",
md_extraction_error="statute: 텍스트 네이티브, markdown 변환 비대상",
source_channel="crawl",
data_origin="external",
review_status="approved",
ai_domain="법령",
ai_sub_group=act.title,
ai_tags=[f"법령/KR/{act.title}"],
# 안전 자료실 ingest 4축 (R8-M1 — classify-skip 경로라 ingest 시점 필수)
material_type="law",
jurisdiction=kr.JURISDICTION,
published_date=eff or prom,
extract_meta={
"statute": {"family_id": act.family_id, "law_id": act.native_id,
"kind": payload.law_doc_kind, "version_key": payload.version_key,
"annex_label": payload.annex_label,
"event_kind": ev.kind, "revision_type": ev.revision_type},
"license": dict(_LICENSE),
**extra,
},
)
session.add(doc)
await session.flush()
session.add(LegalMeta(
document_id=doc.id,
family_id=act.family_id,
law_doc_kind=payload.law_doc_kind,
version_key=payload.version_key,
promulgation_date=prom,
effective_date=eff,
version_status="pending", # 전 버전 pending 적재 — 승격은 생애주기 잡만
))
# summarize 안 함 (조문 자체가 정본 — 맥미니 부하 0), embed+chunk 만
await enqueue_stage(session, doc.id, "embed")
await enqueue_stage(session, doc.id, "chunk")
return True
# ─── 생애주기 잡 (전이·supersede·스윕·repeal 의 유일한 코드 지점) ────────────────
async def _flip_chunks(session, doc_ids: list[int]) -> int:
if not doc_ids:
return 0
result = await session.execute(
update(DocumentChunk)
.where(DocumentChunk.doc_id.in_(doc_ids), DocumentChunk.in_corpus.is_(True))
.values(in_corpus=False)
)
return result.rowcount or 0
async def _legacy_doc_ids(session, act: LegalAct) -> list[int]:
"""법령명 매핑 레거시(law_monitor) 문서 id — 정규화 동등 비교 (prefix 금지)."""
result = await session.execute(
select(Document.id, Document.title).where(
Document.source_channel == "law_monitor",
Document.deleted_at.is_(None),
)
)
want = normalize_law_name(act.title)
ids = []
for doc_id, title in result.all():
name = legacy_law_name(title or "")
if name and normalize_law_name(name) == want:
ids.append(doc_id)
return ids
async def run_lifecycle(session) -> dict:
"""일 1회 생애주기 잡 — 호출측이 단일 트랜잭션 commit. KST 기준, 멱등."""
today = datetime.now(_KST).date()
stats = {"promoted": 0, "superseded": 0, "repealed": 0,
"legacy_flipped_docs": 0, "legacy_flipped_chunks": 0}
acts_result = await session.execute(select(LegalAct).where(LegalAct.watch.is_(True)))
acts = {a.family_id: a for a in acts_result.scalars().all()}
lm_result = await session.execute(
select(LegalMeta).where(LegalMeta.family_id.in_(list(acts.keys())))
)
metas = lm_result.scalars().all()
# 1) repeal — 마킹된 family: current+pending 전부 repealed + 청크 flip + 레거시 flip (R7-M2)
repeal_families = {fid for fid, a in acts.items() if a.repeal_detected_at is not None}
for fid in repeal_families:
rows = [m for m in metas if m.family_id == fid and m.version_status in ("pending", "current")]
for m in rows:
m.version_status = "repealed"
stats["repealed"] += 1
await _flip_chunks(session, [m.document_id for m in rows])
legacy_ids = await _legacy_doc_ids(session, acts[fid])
stats["legacy_flipped_chunks"] += await _flip_chunks(session, legacy_ids)
# 2) 승격 + supersede — 버전 시리즈 단위 (R7-B1 a: family 단위 금지)
series: dict[tuple, list[LegalMeta]] = {}
for m in metas:
if m.family_id in repeal_families:
continue
series.setdefault(
(m.family_id, m.law_doc_kind, series_suffix(m.version_key)), []
).append(m)
for key, rows in series.items():
due = sorted(
(m for m in rows if m.version_status == "pending"
and (m.effective_date or m.promulgation_date)
and (m.effective_date or m.promulgation_date) <= today),
key=lambda m: (m.effective_date or m.promulgation_date),
)
for m in due:
prev = [c for c in rows if c.version_status == "current" and c is not m]
for c in prev:
c.version_status = "superseded"
stats["superseded"] += 1
await _flip_chunks(session, [c.document_id for c in prev])
m.version_status = "current"
stats["promoted"] += 1
# 3) 레거시 스윕 — 상태 기반 (R6-B1 a / R7-B1 b: primary 시리즈 current 보유 한정)
for fid, act in acts.items():
if fid in repeal_families:
continue
has_primary_current = any(
m.family_id == fid and m.law_doc_kind == "primary" and m.version_status == "current"
for m in metas
)
if not has_primary_current:
continue # R3-B1 ② 내장 — fetch 실패 family 의 레거시 보존
legacy_ids = await _legacy_doc_ids(session, act)
flipped = await _flip_chunks(session, legacy_ids)
if flipped:
stats["legacy_flipped_docs"] += len(legacy_ids)
stats["legacy_flipped_chunks"] += flipped
return stats
# ─── 메인 런 ─────────────────────────────────────────────────────────────────────
async def run(bootstrap: bool = False) -> None:
"""poll → fetch → ingest(가족 단위 커밋) → 생애주기 잡. 가족 단위 실패 격리."""
async with async_session() as session:
result = await session.execute(
select(LegalAct).where(LegalAct.watch.is_(True)).order_by(LegalAct.family_id)
)
rows = result.scalars().all()
if not rows:
logger.warning("[statute] 워치리스트 비어 있음 — 시드(migration 356) 미적용?")
return
source = await _get_source(session)
await session.commit()
source_id = source.id
ingested = 0
failed = 0
by_jur: dict[str, list] = {}
for row in rows:
by_jur.setdefault(row.jurisdiction, []).append(row)
async with httpx.AsyncClient(timeout=60) as client:
for jur, acts in by_jur.items():
adapter = _ADAPTERS.get(jur)
if adapter is None:
logger.warning(f"[statute] 어댑터 없는 jurisdiction skip: {jur}")
continue
assert adapter.JURISDICTION == jur, \
f"어댑터/행 jurisdiction 불일치: {adapter.JURISDICTION} != {jur}"
events = await adapter.poll_changes(client, acts)
acts_by_id = {a.family_id: a for a in acts}
for ev in events:
if bootstrap:
ev.kind = "bootstrap" # 합성 이벤트 — amend 와 동일 경로 (R6-m2)
act_ref = acts_by_id[ev.family_id]
try:
payloads = await adapter.fetch_version(client, act_ref, ev)
async with async_session() as session:
act = await session.get(LegalAct, ev.family_id)
new_docs = 0
for p in payloads:
if await _ingest_payload(session, act, ev, p, backfill=bootstrap):
new_docs += 1
# 워터마크 영속 = 파싱 검증(payload floor) 통과 후에만
act.watermark = ev.new_version_key
if ev.kind == "repeal":
act.repeal_detected_at = datetime.now(timezone.utc)
await session.commit()
ingested += new_docs
logger.info(f"[statute] ingest {ev.family_id} ({ev.kind}): "
f"payload {len(payloads)}건 중 신규 {new_docs}")
except Exception as e:
failed += 1
logger.error(f"[statute] ingest 실패 ({ev.family_id}): "
f"{type(e).__name__}: {e!r} — 워터마크 미영속, 다음 폴링 재감지")
await asyncio.sleep(_FETCH_DELAY_S)
# 생애주기 잡 — 수집 사이클 직후, 단일 트랜잭션 (0-2 ②)
async with async_session() as session:
stats = await run_lifecycle(session)
await session.commit()
logger.info(f"[statute] lifecycle: {stats}")
# health — fail-loud 가시성 (HC.io 폐기로 보드/health 행이 1차 관측면)
async with async_session() as session:
h = await _get_or_create_health(session, source_id)
now = datetime.now(timezone.utc)
if failed:
_record_failure(h, f"ingest 실패 {failed}", now)
else:
_record_success(h, ingested, False, now)
await session.commit()
logger.info(f"[statute] run 완료 — 신규 문서 {ingested}건, 실패 {failed}"
+ (" (bootstrap)" if bootstrap else ""))
async def _get_source(session) -> NewsSource:
result = await session.execute(select(NewsSource).where(NewsSource.name == _SOURCE_NAME))
source = result.scalars().first()
if source is None:
source = NewsSource(
name=_SOURCE_NAME, feed_url=kr.LAW_SEARCH_URL, feed_type="rss",
fetch_method="api", fulltext_policy="none", source_channel="crawl",
category="Safety", language="ko", country="KR",
enabled=False, # 6h 뉴스 사이클 비대상 — 본 워커가 daily 폴링
)
session.add(source)
await session.flush()
return source
async def poll_once() -> int:
"""관찰 전용 폴링 (PR① 잔존 CLI — 상태 변경 0)."""
async with async_session() as session:
result = await session.execute(
select(LegalAct).where(LegalAct.watch.is_(True)).order_by(LegalAct.family_id)
)
rows = result.scalars().all()
total = 0
async with httpx.AsyncClient(timeout=30) as client:
events = await kr.poll_changes(client, [r for r in rows if r.jurisdiction == "KR"])
for ev in events:
logger.info(f"[statute] 변경 감지 ({ev.kind}): {ev.family_id} {ev.title} "
f"MST={ev.new_version_key}")
total = len(events)
logger.info(f"[statute] poll 완료 — 변경 {total}건 (관찰 전용)")
return total
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--bootstrap", action="store_true",
help="26 family 현행판 1회 부트스트랩 (backfill 마커, R4-M1)")
parser.add_argument("--poll-only", action="store_true", help="관찰 전용 폴링")
args = parser.parse_args()
if args.poll_only:
asyncio.run(poll_once())
else:
asyncio.run(run(bootstrap=args.bootstrap))
@@ -14,6 +14,7 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import select, update
from sqlalchemy.exc import SQLAlchemyError
from core.config import settings
from core.database import async_session
from core.utils import setup_logger
from models.study_memo_card_job import StudyMemoCardJob
@@ -50,6 +51,10 @@ async def reset_stale_card_jobs() -> None:
async def consume_study_memo_card_queue() -> None:
"""APScheduler 진입점. pending card_extract job 을 BATCH_SIZE 만큼 처리."""
# 생성 LLM 홀드: claim 자체를 하지 않음 (1분 주기라 로그는 debug).
if "study_memo_card" in settings.pipeline_held_stages:
logger.debug("study_memo_card 보류 (pipeline.held_stages)")
return
await reset_stale_card_jobs()
async with async_session() as session:
+5
View File
@@ -59,6 +59,11 @@ async def reset_stale_study_jobs() -> None:
async def consume_study_queue() -> None:
"""APScheduler 진입점. pending job BATCH_SIZE 만큼 처리."""
# 생성 LLM 홀드: env(study_explanation_enabled) 와 별개의 self-contained 게이트.
# pending 은 그대로 유지 (Mac mini derived-worker 흡수 경로도 본 게이트와 무관).
if "study_explanation" in settings.pipeline_held_stages:
logger.debug("study_explanation 보류 (pipeline.held_stages)")
return
await reset_stale_study_jobs()
async with async_session() as session:
@@ -12,6 +12,7 @@ from datetime import datetime, timedelta, timezone
from sqlalchemy import select, update
from sqlalchemy.exc import SQLAlchemyError
from core.config import settings
from core.database import async_session
from core.utils import setup_logger
from models.study_quiz_session_job import StudyQuizSessionJob
@@ -48,6 +49,10 @@ async def reset_stale_session_jobs() -> None:
async def consume_study_session_queue() -> None:
"""APScheduler 진입점. pending session_jobs 를 BATCH_SIZE 만큼 처리."""
# 생성 LLM 홀드: claim 자체를 하지 않음 (1분 주기라 로그는 debug).
if "study_session_analysis" in settings.pipeline_held_stages:
logger.debug("study_session_analysis 보류 (pipeline.held_stages)")
return
await reset_stale_session_jobs()
async with async_session() as session:
+40 -12
View File
@@ -6,25 +6,40 @@ ai:
models:
# ─── 단일 generation 호스트 routing (2026-05-14 GPU LLM 제거) ───
# GPU Ollama gemma4:e4b-it-q8_0 제거. Mac mini 26B-A4B 가 triage + primary + classifier 모두 흡수.
# fallback 은 Claude Sonnet 4 API (Mac mini 다운 시 자동 trigger, premium 과 budget 공유).
# plan: ~/.claude/plans/rosy-launching-otter.md §C/§D/§E
# 2026-06-11 B안: 맥미니 모델 = Gemma 26B-A4B → Qwen3.6-27B-6bit 풀교체 (사용자 결정).
# dense 27B 라 디코드 ~13 tok/s 급 (a4b ~42 대비 감속) → timeout 상향 (triage 30→120, primary 180→300).
# fallback 은 Claude Sonnet 4 API (CLAUDE_API_KEY 미주입 = 비활성).
# plan: ~/.claude/plans/rosy-launching-otter.md §C/§D/§E + project_macmini_model_decision
# triage: 상시 분류·요약·근거 선별. Mac mini 26B (primary 와 동일 endpoint, 짧은 max_tokens).
# triage: 상시 분류·요약·근거 선별. Mac mini Qwen 27B (primary 와 동일 endpoint, 짧은 max_tokens).
triage:
endpoint: "http://100.76.254.116:8801/v1/chat/completions"
model: "mlx-community/gemma-4-26b-a4b-it-8bit"
model: "mlx-community/Qwen3.6-27B-6bit"
max_tokens: 4096
timeout: 30
timeout: 480 # 프리필 실측 ~112 tok/s — 120K자 장문 커버 (2026-06-11)
context_char_limit: 120000
temperature: 0.0
# primary: 에스컬레이션 전용. 26B MLX (맥미니 Semaphore(1) 보호 대상).
# primary: 에스컬레이션 전용. Qwen 27B MLX (맥미니 Semaphore(1) 보호 대상).
primary:
endpoint: "http://100.76.254.116:8801/v1/chat/completions"
model: "mlx-community/gemma-4-26b-a4b-it-8bit"
model: "mlx-community/Qwen3.6-27B-6bit"
max_tokens: 8192
timeout: 180
timeout: 900 # 프리필 실측 ~112 tok/s — 260K자 상한 장문 커버 (2026-06-11)
context_char_limit: 260000
temperature: 0.3
top_p: 0.9
# deep: 야간 night-drain 전용 — 맥북 M5 Max Qwen3.6-27B-6bit (llm-router :8890 경유,
# model=qwen-macbook alias). 2026-06-11 재도입 (사용자: 자기 전 night-drain 으로 백로그 분담).
# 맥북 불가(503/연결/절단) = StageDeferred 보류 — 맥미니/cloud 강등 없음, attempts 미소모.
# consumer 의 deep_summary 도 슬롯 존재 시 맥북 경유 (잠들어 있으면 30분 백오프 보류 = 무해).
# 슬롯 제거 시 deep_summary 는 primary(맥미니) 경로 복귀.
deep:
endpoint: "http://100.76.254.116:8890/v1/chat/completions"
model: "qwen-macbook"
max_tokens: 8192
timeout: 900
context_char_limit: 260000
temperature: 0.3
top_p: 0.9
@@ -58,9 +73,9 @@ ai:
# classifier_service 가 hasattr 체크로 optional 이므로 이 섹션 제거 시 classifier gate 는 자동 skip (score-only).
classifier:
endpoint: "http://100.76.254.116:8801/v1/chat/completions"
model: "mlx-community/gemma-4-26b-a4b-it-8bit"
model: "mlx-community/Qwen3.6-27B-6bit" # 2026-06-11 B안 동승 — gemma id 잔존 시 mlx 서버가 Gemma 를 재로드(이중 적재) 위험
max_tokens: 512
timeout: 30 # 2026-05-17: 15s 도 동시 부하 시 elapsed 14.4s 직전이라 tight — 30s 로 2x 마진 (Mac mini 26B concurrent load). classifier_service.LLM_TIMEOUT_MS=30000 와 align
timeout: 30 # 2026-05-17: 15s 도 동시 부하 시 elapsed 14.4s 직전이라 tight — 30s 로 2x 마진. classifier_service.LLM_TIMEOUT_MS=30000 와 align (초과 = score-only skip, graceful)
# 제거: vision (미사용)
# ─── deep_summary enqueue 폭발 억제 (B-1 R2) ───
@@ -84,7 +99,7 @@ search:
macbook_url: "http://100.118.112.84:8810" # MacBook M5 Max Tailscale interface bind
macbook_model: "mlx-community/Qwen3.6-27B-8bit"
timeout_connect_s: 1 # MacBook sleep/wake 빠른 감지 (자동 fallback 부재 → 빠른 503)
timeout_read_s: 30 # synthesis_service.LLM_TIMEOUT_MS=30000 와 align
timeout_read_s: 120 # 2026-06-11 Qwen 27B(디코드 ~11.7 tok/s) — synthesis_service.LLM_TIMEOUT_MS=120000 와 align
# PR-DocSrv-Ask-ToolCalling-ReAct-1: /api/search/ask/react ReAct loop (qwen-macbook only)
react:
enabled: true
@@ -176,3 +191,16 @@ schedule:
daily_digest: "20:00"
file_watcher_interval_minutes: 5
queue_consumer_interval_minutes: 10
# 생성 LLM 홀드 게이트 (2026-06-11 신설): held_stages 에 든 이름의 컨슈머/워커는 claim 자체를
# 하지 않는다 (attempts 미소모, pending 적체). 유효 키 8 = classify/summarize/deep_summary(큐) +
# digest/briefing(cron) + study_explanation/study_session_analysis/study_memo_card(컨슈머).
# 그 외 문자열은 무동작(오타 주의). 적용/해제 = 리스트 수정 후 fastapi 재기동.
# 이력: 2026-06-11 맥미니 모델 확정까지 8키 홀드 → 同日 Qwen3.6-27B-6bit 전환과 함께 해제([]).
pipeline:
held_stages: []
# mlx gate 동시 실행 상한 (2026-06-12 fair-share): 구 "1 고정" 룰의 전제(single-inference
# 서버)가 소멸 — 현 mlx_vlm 은 continuous batching (2026-06-11 밤 6~8 concurrent 실측 정상).
# 2 = 워커 LLM 호출과 인터랙티브(ask/eid)가 서로 안 막힘 + 집계 throughput ~1.8배.
# 게이트(상한+우선순위)는 유지 — thundering herd 방지. 1 로 되돌리면 구 동작.
mlx_gate_concurrency: 2
@@ -0,0 +1,527 @@
<script lang="ts">
// 처리 머신 보드 v3 — 통합안 (plan ds-board-merged: C2 머신레인 + C3 번다운/정직ETA).
// · 머신 3레인(GPU/맥미니/맥북) = "누가 일하나" + 요약 오프로드(맥북 합류) 가시화
// · 지배 백로그 번다운 패널 = "언제 끝나나" + 유입 차감한 정직 ETA(summarize_eta)
// · 신선도 '갱신 N초 전' + stale 경고 / 실패 드로어·상세 패널은 v2 자산 재사용.
// 데이터 = GET /api/queue/overview (60s 폴링 store) + GET /api/queue/failed (드로어).
import { api } from '$lib/api';
import { refreshQueueOverview, queueUpdatedAt } from '$lib/stores/queueOverview';
import { addToast } from '$lib/stores/toast';
import {
AUX_NODES,
FLOW_NODES,
MACHINE_META,
type FlowNodeDef,
type FlowMachine,
etaShort,
flowStageLabel,
formatAgeSec,
formatRate,
} from '$lib/utils/queueDisplay';
import type {
FailedItem,
FailedListResponse,
MachineCurrentItem,
MachineOverview,
QueueOverview,
QueueStageRow,
RetryResponse,
SkipResponse,
} from '$lib/types/queue';
let { overview }: { overview: QueueOverview } = $props();
// ─── 노드 통계 합성 ───
interface NodeStats {
def: FlowNodeDef;
/** 다중 stage 노드(청크·임베딩)는 같은 문서가 양쪽 큐에 있어 max — 합산 = 이중계산 */
pending: number;
processing: number;
failed: number; // 실패는 행 단위 사실이라 합산
done1h: number;
created1h: number;
doneToday: number;
oldestAgeSec: number | null;
etaMinutes: number | null;
inflowDominant: boolean;
perStage: QueueStageRow[];
}
const stageBy = $derived(new Map(overview.stages.map((s) => [s.stage, s])));
function nodeStats(def: FlowNodeDef): NodeStats {
const rows = def.stages
.map((s) => stageBy.get(s))
.filter((r): r is QueueStageRow => r != null);
const pending = rows.reduce((m, r) => Math.max(m, r.pending), 0);
const done1h = rows.reduce((m, r) => Math.max(m, r.done_1h), 0);
const created1h = rows.reduce((m, r) => Math.max(m, r.created_1h), 0);
const oldest = rows.reduce<number | null>(
(m, r) => (r.oldest_pending_age_sec == null ? m : Math.max(m ?? 0, r.oldest_pending_age_sec)),
null,
);
return {
def,
pending,
processing: rows.reduce((s, r) => s + r.processing, 0),
failed: rows.reduce((s, r) => s + r.failed, 0),
done1h,
created1h,
doneToday: rows.reduce((m, r) => Math.max(m, r.done_today), 0),
oldestAgeSec: oldest,
etaMinutes: pending > 0 && done1h > 0 ? Math.round((pending / done1h) * 60) : null,
inflowDominant: pending > 0 && created1h > done1h,
perStage: rows,
};
}
const mainNodes = $derived(FLOW_NODES.map(nodeStats));
const auxAll = $derived(AUX_NODES.map(nodeStats));
const auxActive = $derived(
auxAll.filter((n) => n.pending + n.processing + n.failed + n.doneToday > 0),
);
const auxIdle = $derived(
auxAll.filter((n) => n.pending + n.processing + n.failed + n.doneToday === 0),
);
const totalFailed = $derived(overview.totals.failed);
// ─── 선택 상태 (노드 상세 / 실패 드로어 — 동시에 하나만) ───
let selected = $state<string | null>(null);
let failOpen = $state(false);
function toggleNode(key: string) {
selected = selected === key ? null : key;
if (selected) failOpen = false;
}
const selectedNode = $derived(
[...mainNodes, ...auxAll].find((n) => n.def.key === selected) ?? null,
);
function nodeCurrent(def: FlowNodeDef): MachineCurrentItem[] {
return overview.machines.flatMap((m) => m.current.filter((c) => def.stages.includes(c.stage)));
}
// ─── 실패 드로어 ───
let failItems = $state<FailedItem[]>([]);
let failLoading = $state(false);
let busy = $state(false);
let expanded = $state<Record<string, boolean>>({});
async function openFailures() {
failOpen = true;
selected = null;
await loadFailures();
}
async function loadFailures() {
failLoading = true;
try {
const r = await api<FailedListResponse>('/queue/failed');
failItems = r.items;
} catch {
addToast('error', '실패 목록을 불러오지 못했습니다');
} finally {
failLoading = false;
}
}
interface FailGroup {
key: string;
stage: string;
pattern: string;
items: FailedItem[];
}
// 그룹핑 = stage + 에러 메시지 prefix(36자) — 같은 원인(ReadTimeout 등) 묶음
const failGroups = $derived.by(() => {
const map = new Map<string, FailGroup>();
for (const it of failItems) {
const pattern = (it.error_message ?? '(메시지 없음)').slice(0, 36);
const key = `${it.stage}::${pattern}`;
const g = map.get(key);
if (g) g.items.push(it);
else map.set(key, { key, stage: it.stage, pattern, items: [it] });
}
return [...map.values()].sort(
(a, b) => a.stage.localeCompare(b.stage) || b.items.length - a.items.length,
);
});
async function retryIds(ids: number[]) {
if (busy || ids.length === 0) return;
busy = true;
try {
const r = await api<RetryResponse>('/queue/retry', {
method: 'POST',
body: JSON.stringify({ ids }),
});
addToast(
'success',
`재시도 ${r.retried}건 큐 재진입${r.not_retried > 0 ? ` (${r.not_retried} 제외 이미 활성/처리됨)` : ''}`,
);
await afterAction();
} catch {
addToast('error', '재시도 요청 실패');
} finally {
busy = false;
}
}
async function skipIds(ids: number[]) {
if (busy || ids.length === 0) return;
busy = true;
try {
const r = await api<SkipResponse>('/queue/skip', {
method: 'POST',
body: JSON.stringify({ ids }),
});
addToast('success', `건너뛰기 ${r.skipped}건 처리 (해당 단계 제외)`);
await afterAction();
} catch {
addToast('error', '건너뛰기 요청 실패');
} finally {
busy = false;
}
}
async function afterAction() {
await Promise.all([loadFailures(), refreshQueueOverview()]);
}
// ─── 머신 레인 (C2) — mainNodes 를 머신별로 그룹 + 머신 카드(state/처리율) 결합 ───
const machineByKey = $derived(
new Map<FlowMachine, MachineOverview>(overview.machines.map((m) => [m.key as FlowMachine, m])),
);
const LANE_ORDER: FlowMachine[] = ['gpu', 'macmini', 'macbook'];
const lanes = $derived(
LANE_ORDER.map((key) => ({
key,
meta: MACHINE_META[key],
card: machineByKey.get(key) ?? null,
nodes: mainNodes.filter((n) => n.def.machine === key),
})),
);
// 요약 오프로드 분담 — 맥미니 vs 맥북 (A-1 summarize_by_machine)
const split = $derived(overview.summarize_by_machine);
const splitTotal1h = $derived(Math.max(1, split.macmini.done_1h + split.macbook.done_1h));
const macbookSharePct = $derived(Math.round((split.macbook.done_1h / splitTotal1h) * 100));
// 맥북이 요약을 실제로 가져가는 중인가 (합류 표식 게이트)
const offloadActive = $derived(split.macbook.done_1h > 0);
// ─── 지배 백로그 = 요약. 정직 ETA(유입 차감) — summarize_eta ───
const eta = $derived(overview.summarize_eta);
// 정직 ETA 라벨: eta_minutes null = 유입이 소화를 앞섬(소진 불가)
const honestEtaLabel = $derived(
eta.pending === 0
? '비어 있음'
: eta.eta_minutes != null
? etaShort(eta.eta_minutes)
: '소진 불가',
);
const honestEtaWarn = $derived(eta.pending > 0 && eta.eta_minutes == null);
/** 단계별 정직 ETA(순소화율) — 노드용. 유입>소화면 null(소진 불가) */
function netEtaLabel(n: NodeStats): string | null {
if (n.pending === 0) return '한가';
const net = n.done1h - n.created1h;
if (net > 0) return etaShort(Math.round((n.pending / net) * 60));
if (n.created1h > n.done1h) return '유입 우세';
return null;
}
// ─── 신선도 (B-4) — '갱신 N초 전' + stale 경고 (폴링 60s) ───
let now = $state(Date.now());
$effect(() => {
const id = setInterval(() => (now = Date.now()), 1000);
return () => clearInterval(id);
});
const ageSec = $derived(
$queueUpdatedAt != null ? Math.max(0, Math.round((now - $queueUpdatedAt) / 1000)) : null,
);
const stale = $derived(ageSec != null && ageSec > 90);
const freshLabel = $derived(
ageSec == null
? '갱신 대기'
: ageSec < 60
? `갱신 ${ageSec}초 전`
: `갱신 ${Math.round(ageSec / 60)}분 전`,
);
// ─── 24h 번다운 (C3) — 요약 유입 vs 소화 + 맥북 합류 변곡점 마커 ───
const burn = $derived.by(() => {
const t = overview.trend_24h;
if (!t || t.length === 0) return null;
const max = Math.max(1, ...t.map((b) => Math.max(b.inflow, b.done)));
const w = 300;
const h = 64;
const step = w / Math.max(1, t.length - 1);
const y = (v: number) => (h - (v / max) * (h - 8) + 4).toFixed(1);
const line = (sel: (b: (typeof t)[number]) => number) =>
t.map((b, i) => `${(i * step).toFixed(1)},${y(sel(b))}`).join(' ');
const doneLine = line((b) => b.done);
const area = `0,${h} ${doneLine} ${w.toFixed(1)},${h}`;
// 합류 변곡점 = done 최대 버킷 (맥북 야간 drain 합류 추정)
let mi = 0;
t.forEach((b, i) => {
if (b.done > t[mi].done) mi = i;
});
return {
w,
h,
area,
doneLine,
inflowLine: line((b) => b.inflow),
markX: (mi * step).toFixed(1),
markHour: t[mi].hour,
markDone: t[mi].done,
peak: max,
};
});
// 머신 상태 dot 색 클래스
function dotClass(state: string): string {
return state === 'active' ? 'bg-success' : state === 'deferred' ? 'bg-warning' : 'bg-faint';
}
</script>
<div class="mt-5">
<!-- 헤더: 타이틀 + 신선도 + 실패 합계 -->
<div class="flex items-center justify-between gap-3 mb-3">
<div class="text-[11px] font-bold text-dim uppercase tracking-wider">처리 머신</div>
<div class="flex items-center gap-3">
{#if totalFailed > 0}
<button
class="text-[11px] font-semibold text-error hover:underline cursor-pointer"
onclick={openFailures}
>실패 {totalFailed}건 처리</button>
{/if}
<span class="flex items-center gap-1.5 text-[10px] tabular-nums {stale ? 'text-warning' : 'text-faint'}" title="60초 폴링">
<span class="w-1.5 h-1.5 rounded-full {stale ? 'bg-warning' : 'bg-success'}"></span>
{freshLabel}{#if stale} · 갱신 지연{/if}
</span>
</div>
</div>
<!-- 지배 백로그 스트립 (요약) + 정직 ETA -->
<div class="flex items-center flex-wrap gap-x-3 gap-y-1 bg-surface border border-warning/50 rounded-card px-3.5 py-2 mb-3">
<span class="text-[9px] font-bold text-warning border border-warning/60 rounded-full px-2 py-px">지배 백로그</span>
<span class="text-xs font-bold text-text">요약</span>
<span class="text-[11px] text-dim tabular-nums">대기 <b class="text-text">{eta.pending.toLocaleString()}</b> · 순소화 <b class="text-text">{formatRate(eta.done_rate_1h)}</b>/h · 유입 {formatRate(eta.inflow_rate_1h)}/h</span>
<span class="ml-auto flex items-center gap-1.5 border rounded-full px-2.5 py-0.5 {honestEtaWarn ? 'border-warning text-warning' : 'border-accent text-accent'}">
<span class="text-[10px] font-semibold">정직 ETA</span>
<span class="text-xs font-bold tabular-nums">{honestEtaLabel}</span>
</span>
</div>
<!-- 머신 레인 (누가 일하나 + 요약 오프로드) -->
<div class="grid gap-2 mb-3">
{#each lanes as lane (lane.key)}
<div class="bg-surface border border-default rounded-card px-3.5 py-2.5">
<div class="flex items-center gap-2 flex-wrap mb-2">
<span class="w-2 h-2 rounded-full shrink-0 {dotClass(lane.card?.state ?? 'idle')}"></span>
<span class="text-[9px] font-bold rounded px-1.5 py-px mtag-{lane.key}">{lane.meta.label}</span>
<span class="text-[10px] text-faint font-mono">{lane.meta.model}</span>
<span class="text-[11px] text-dim tabular-nums ml-1">{formatRate(lane.card?.done_1h ?? 0)}/h</span>
{#if lane.key === 'macbook' && (lane.card?.deferred_pending ?? 0) > 0}
<span class="text-[10px] font-semibold text-warning tabular-nums">보류 {lane.card?.deferred_pending}</span>
{/if}
{#if lane.card?.state === 'deferred'}
<span class="text-[9px] text-warning">잠듦 — 요약은 맥미니로 복귀</span>
{/if}
</div>
<div class="flex items-stretch gap-1.5 flex-wrap">
{#each lane.nodes as n (n.def.key)}
{@const idle = n.pending + n.processing + n.doneToday + n.failed === 0}
<button
class="relative text-left rounded-lg border px-2.5 py-1.5 transition-colors cursor-pointer hover:bg-surface-hover min-w-[96px]
{idle ? 'border-dashed border-default opacity-55' : n.inflowDominant ? 'border-warning' : 'border-default'}
{selected === n.def.key ? 'node-sel' : ''}"
onclick={() => toggleNode(n.def.key)}
title="{n.def.label} — 클릭하면 상세"
>
{#if n.failed > 0}
<span class="absolute -top-1.5 -right-1 text-[9px] font-extrabold bg-error text-white rounded-full px-1.5">{n.failed}</span>
{/if}
<div class="flex items-center gap-1 text-[11px] font-semibold text-text whitespace-nowrap">
{n.def.label}
{#if n.processing > 0}<span class="inline-block w-1.5 h-1.5 rounded-full bg-accent animate-pulse"></span>{/if}
</div>
<div class="text-sm font-extrabold tabular-nums leading-tight text-text">{n.pending.toLocaleString()}<span class="text-[9px] text-faint font-normal ml-0.5">대기</span></div>
<div class="text-[9px] text-dim tabular-nums whitespace-nowrap">{formatRate(n.done1h)}/h · 오늘 {n.doneToday.toLocaleString()}</div>
{#if n.def.key === 'summarize'}
<div class="mt-1 h-1 w-full rounded-full overflow-hidden flex" title="맥미니 {split.macmini.done_1h}/h · 맥북 {split.macbook.done_1h}/h">
<span class="block h-full mtag-macmini-bar" style="width:{100 - macbookSharePct}%"></span>
<span class="block h-full mtag-macbook-bar" style="width:{macbookSharePct}%"></span>
</div>
<div class="text-[9px] text-faint tabular-nums whitespace-nowrap mt-0.5">맥미니 {split.macmini.done_1h} · 맥북 {split.macbook.done_1h}/h</div>
{/if}
</button>
{/each}
{#if lane.key === 'macbook' && offloadActive}
<button
class="text-left rounded-lg border border-dashed border-warning/50 px-2.5 py-1.5 cursor-pointer hover:bg-surface-hover min-w-[96px]"
onclick={() => toggleNode('summarize')}
title="맥북이 요약을 맥미니에서 가져와 처리 중"
>
<div class="flex items-center gap-1 text-[11px] font-semibold text-text whitespace-nowrap">요약 합류 <span class="text-[8px] font-bold text-warning">OFFLOAD</span></div>
<div class="text-sm font-extrabold tabular-nums leading-tight text-text">{split.macbook.done_1h}<span class="text-[9px] text-faint font-normal ml-0.5">/h</span></div>
<div class="text-[9px] text-dim tabular-nums whitespace-nowrap">요약의 {macbookSharePct}% 담당</div>
</button>
{/if}
</div>
</div>
{/each}
</div>
<!-- 번다운 / ETA 패널 -->
{#if burn}
<div class="bg-surface border border-default rounded-card px-3.5 py-3 mb-1">
<div class="flex items-center gap-2 mb-2">
<span class="text-[11px] font-bold text-text">요약 백로그 24시간</span>
<span class="text-[9px] text-faint">유입(회색) vs 소화(녹색)</span>
{#if offloadActive}<span class="text-[9px] text-warning ml-auto">맥북 합류 {burn.markHour} — 소화 급증</span>{/if}
</div>
<svg viewBox="0 0 {burn.w} {burn.h}" class="block w-full" style="height:64px" preserveAspectRatio="none" role="img" aria-label="요약 백로그 24시간 번다운">
<polygon points={burn.area} fill="currentColor" class="text-success" opacity="0.12" />
<polyline points={burn.inflowLine} fill="none" stroke="currentColor" stroke-width="1.2" class="text-faint" />
<polyline points={burn.doneLine} fill="none" stroke="currentColor" stroke-width="1.6" class="text-success" />
{#if offloadActive}
<line x1={burn.markX} y1="0" x2={burn.markX} y2={burn.h} stroke="currentColor" stroke-width="1" stroke-dasharray="2 2" class="text-warning" opacity="0.7" />
{/if}
</svg>
<div class="flex flex-wrap gap-x-4 gap-y-1 mt-2 pt-2 border-t border-default text-[10px] text-dim tabular-nums">
{#each mainNodes.filter((n) => n.pending > 0 && n.def.key !== 'summarize') as n (n.def.key)}
<span class="whitespace-nowrap">{n.def.label} 대기 <b class="text-text">{n.pending.toLocaleString()}</b>{#if netEtaLabel(n)} · <span class="text-accent font-semibold">{netEtaLabel(n)}</span>{/if}</span>
{/each}
</div>
</div>
{/if}
<!-- 보조 라인 -->
<p class="text-[10px] text-faint mt-1.5 tabular-nums">
{#each auxActive as n, i (n.def.key)}
{i > 0 ? ' · ' : '보조: '}{n.def.label}({n.def.engine}) 대기 {n.pending.toLocaleString()} · {formatRate(n.done1h)}/h{n.failed > 0 ? ` · 실패 ${n.failed}` : ''}
{/each}
{#if auxIdle.length > 0}
{auxActive.length > 0 ? ' — ' : ''}한가: {auxIdle.map((n) => n.def.label).join(' · ')}
{/if}
— 뉴스 등 일부 소스는 분류/추출을 건너뜀 (흐름 그림은 대표 경로)
</p>
<!-- 상세 패널 (노드 클릭) -->
{#if selectedNode}
<div class="border rounded-card mt-3 overflow-hidden bg-surface detail-frame">
<div class="flex items-center gap-2.5 px-4 py-2.5 text-xs font-bold detail-head">
{selectedNode.def.label}{selectedNode.def.engine}
<span class="text-[10px] font-mono font-medium text-dim bg-surface border border-default rounded px-1.5">{selectedNode.def.sub} · {MACHINE_META[selectedNode.def.machine].label}</span>
<button class="ml-auto text-[11px] text-dim font-normal cursor-pointer hover:text-text" onclick={() => (selected = null)}>닫기</button>
</div>
<div class="px-4 pb-3.5">
<div class="grid grid-cols-2 md:grid-cols-4 gap-2.5 my-2.5">
<div class="bg-bg border border-default rounded-card px-3 py-2">
<div class="text-[9px] text-faint uppercase tracking-wide">대기</div>
<div class="text-lg font-extrabold tabular-nums text-text">{selectedNode.pending.toLocaleString()}</div>
</div>
<div class="bg-bg border border-default rounded-card px-3 py-2">
<div class="text-[9px] text-faint uppercase tracking-wide">처리율 (1h)</div>
<div class="text-lg font-extrabold tabular-nums text-text">{formatRate(selectedNode.done1h)}<span class="text-[11px] text-dim font-semibold">/h</span></div>
</div>
<div class="bg-bg border border-default rounded-card px-3 py-2">
<div class="text-[9px] text-faint uppercase tracking-wide">오늘 완료</div>
<div class="text-lg font-extrabold tabular-nums text-text">{selectedNode.doneToday.toLocaleString()}</div>
</div>
<div class="bg-bg border border-default rounded-card px-3 py-2">
<div class="text-[9px] text-faint uppercase tracking-wide">소진 예상</div>
<div class="text-lg font-extrabold tabular-nums {selectedNode.inflowDominant ? 'text-warning' : 'text-accent'}">
{#if selectedNode.inflowDominant}유입 우세{:else if selectedNode.etaMinutes != null}{etaShort(selectedNode.etaMinutes)}{:else if selectedNode.pending === 0}한가{:else}{/if}
</div>
</div>
</div>
{#if selectedNode.perStage.length > 1}
{#each selectedNode.perStage as row (row.stage)}
<div class="flex items-center gap-2.5 py-1.5 border-t border-default text-xs">
<span class="font-semibold text-text min-w-[72px]">{flowStageLabel(row.stage)}</span>
<span class="ml-auto text-dim tabular-nums">
대기 <strong class="text-text">{row.pending.toLocaleString()}</strong>
· {formatRate(row.done_1h)}/h · 오늘 {row.done_today.toLocaleString()}
{#if row.failed > 0}· <span class="text-error font-semibold">실패 {row.failed}</span>{/if}
</span>
</div>
{/each}
{/if}
<div class="text-[11px] text-dim border-t border-dashed border-default mt-2 pt-2 tabular-nums">
{#if selectedNode.oldestAgeSec != null && selectedNode.oldestAgeSec > 600}
가장 오래 기다린 항목 {formatAgeSec(selectedNode.oldestAgeSec)}
{/if}
{#each nodeCurrent(selectedNode.def) as c, i (c.document_id + c.stage)}
{i === 0 && !(selectedNode.oldestAgeSec != null && selectedNode.oldestAgeSec > 600) ? '' : ' · '}지금: {c.title} ({flowStageLabel(c.stage)})
{/each}
{#if selectedNode.failed > 0}
· <button class="text-error font-semibold cursor-pointer hover:underline" onclick={openFailures}>실패 {selectedNode.failed} 처리</button>
{/if}
</div>
</div>
</div>
{/if}
<!-- 실패 처리 드로어 -->
{#if failOpen}
<div class="border border-error/40 rounded-card mt-3 overflow-hidden bg-surface">
<div class="flex items-center gap-2.5 px-4 py-2.5 bg-error/5 text-xs font-bold text-text">
실패 처리
<span class="text-[10px] font-semibold text-error">영구 실패 {failItems.length}건 — 자동 재시도 3회 소진, 수동 조치 대기</span>
<button class="ml-auto text-[11px] text-dim font-normal cursor-pointer hover:text-text" onclick={() => (failOpen = false)}>닫기</button>
</div>
{#if failLoading}
<p class="text-xs text-dim text-center py-4">불러오는 중…</p>
{:else if failItems.length === 0}
<p class="text-xs text-dim text-center py-4">영구 실패 항목 없음</p>
{:else}
{#each failGroups as g (g.key)}
<div class="px-4 py-2.5 border-t border-default">
<div class="flex items-center gap-2 flex-wrap text-xs font-bold text-text mb-1">
{flowStageLabel(g.stage)} {g.items.length}
<span class="text-[10px] font-mono font-medium text-error bg-error/10 rounded px-1.5 py-px">{g.pattern}{g.items[0]?.error_message && g.items[0].error_message.length > 36 ? '…' : ''}</span>
</div>
{#each expanded[g.key] ? g.items : g.items.slice(0, 4) as it (it.id)}
<div class="flex items-center gap-2.5 py-1 border-t border-dashed border-default/60 text-xs">
<span class="flex-1 min-w-0 truncate text-text" title={it.title}>{it.title}</span>
<span class="text-[10px] font-mono text-faint shrink-0 tabular-nums">시도 {it.attempts}/{it.max_attempts}</span>
<span class="text-[10px] font-mono text-error shrink-0 max-w-[260px] truncate" title={it.error_message ?? ''}>{it.error_message ?? ''}</span>
<button class="text-[10px] font-bold border border-accent text-accent rounded px-2 py-0.5 shrink-0 cursor-pointer hover:bg-accent/10 disabled:opacity-40" disabled={busy} onclick={() => retryIds([it.id])}>재시도</button>
<button class="text-[10px] font-bold border border-default text-faint rounded px-2 py-0.5 shrink-0 cursor-pointer hover:bg-surface-hover disabled:opacity-40" disabled={busy} onclick={() => skipIds([it.id])}>건너뛰기</button>
</div>
{/each}
{#if g.items.length > 4 && !expanded[g.key]}
<button class="text-[10px] text-dim cursor-pointer hover:text-text mt-1" onclick={() => (expanded = { ...expanded, [g.key]: true })}> {g.items.length - 4} 펼치기</button>
{/if}
{#if g.items.length > 1}
<div class="flex gap-2 mt-1.5">
<button class="text-[10px] font-bold border border-accent text-accent rounded px-2.5 py-0.5 cursor-pointer hover:bg-accent/10 disabled:opacity-40" disabled={busy} onclick={() => retryIds(g.items.map((x) => x.id))}>그룹 전체 재시도 ({g.items.length})</button>
<button class="text-[10px] font-bold border border-default text-faint rounded px-2.5 py-0.5 cursor-pointer hover:bg-surface-hover disabled:opacity-40" disabled={busy} onclick={() => skipIds(g.items.map((x) => x.id))}>그룹 전체 건너뛰기</button>
</div>
{/if}
</div>
{/each}
<p class="text-[10px] text-faint px-4 py-2 border-t border-default">
재시도 = 시도 횟수 리셋 후 큐 재진입 (자동 재시도 3회 새로 부여) · 건너뛰기 = 이 단계 완료 처리(후속 단계 연쇄 없음, 감사 마킹) · 같은 오류가 반복되는 항목(빈 텍스트 등)은 건너뛰기 권장
</p>
{/if}
</div>
{/if}
</div>
<style>
/* 머신 색 — 디자인 토큰 외 3색 (gpu 청/macmini 보라/macbook 황) — 이 컴포넌트 한정 */
.mtag-gpu { background: #e7eef6; color: #3b6ea5; }
.mtag-macmini { background: #efe9f7; color: #8a5fbf; }
.mtag-macbook { background: #f7eedd; color: #b07a10; }
/* 요약 오프로드 분담 막대 채움 (맥미니 보라 / 맥북 황) */
.mtag-macmini-bar { background: #8a5fbf; }
.mtag-macbook-bar { background: #b07a10; }
.node-sel { outline: 2px solid #3b6ea5; outline-offset: 1px; }
.detail-frame { border-color: #3b6ea5; }
.detail-head { background: #e7eef6; }
</style>
@@ -0,0 +1,106 @@
<script lang="ts">
// 처리 현황 드로어 (안6 라이트) — 전 페이지 상태 스트립 클릭 시 우측에서 열림.
// 머신 미니카드 3 + ETA 한 줄 + 실패 합계 + 홈 링크 축약본. 상세는 홈 보드가 담당.
// 데이터 = queueOverview store 공유 (60s 폴링, 실패 시 null → 안내문으로 degrade).
// 열림 상태는 uiState 단일 drawer slot('queue') — 사이드바 드로어와 동시 오픈 차단.
import { X } from 'lucide-svelte';
import { ui } from '$lib/stores/uiState.svelte';
import { queueOverview } from '$lib/stores/queueOverview';
import {
MACHINE_STATE_LABEL, machineChipClass, machineDotClass, formatRate, etaPhrase,
} from '$lib/utils/queueDisplay';
import IconButton from '$lib/components/ui/IconButton.svelte';
let open = $derived(ui.isDrawerOpen('queue'));
let data = $derived($queueOverview);
function close() {
ui.closeDrawer();
}
// ESC 닫기 — 레이아웃 전역 핸들러(ui.handleEscape)와 중복돼도 무해(멱등).
// modal stack 이 열려 있으면 modal 우선 (전역 우선순위와 동일).
function onWindowKeydown(e: KeyboardEvent) {
if (e.key === 'Escape' && open && ui.modalStack.length === 0) close();
}
</script>
<svelte:window onkeydown={onWindowKeydown} />
{#if open}
<div class="fixed inset-0 z-drawer">
<!-- 스크림 — 클릭 시 닫기 -->
<button
type="button"
onclick={close}
class="absolute inset-0 bg-scrim transition-opacity"
aria-label="드로어 닫기"
></button>
<!-- 패널 — div + role="dialog" (aside 는 interactive role 불가, a11y 경고) -->
<div
role="dialog"
aria-modal="true"
aria-label="처리 현황"
class="absolute right-0 top-0 bottom-0 w-rail max-w-full bg-sidebar shadow-xl overflow-y-auto"
>
<div class="flex items-center justify-between px-4 h-12 border-b border-default">
<span class="text-sm font-bold text-text">처리 현황</span>
<IconButton icon={X} size="sm" aria-label="닫기" onclick={close} />
</div>
<div class="p-4 space-y-3">
{#if data}
<!-- 머신 미니카드 3 -->
{#each data.machines as m (m.key)}
<div class="bg-surface border border-default rounded-lg px-3.5 py-2.5">
<div class="flex items-center justify-between gap-2">
<span class="flex items-center gap-2 text-[13px] font-semibold text-text min-w-0">
<span class="w-2 h-2 rounded-full shrink-0 {machineDotClass(m.state)}"></span>
<span class="truncate">{m.label}</span>
</span>
<span class="text-[10px] font-bold rounded-full px-2 py-0.5 shrink-0 {machineChipClass(m.state)}">
{MACHINE_STATE_LABEL[m.state]}
</span>
</div>
<div class="text-[11px] text-dim mt-1 tabular-nums">
대기 <strong class="text-text">{m.pending.toLocaleString()}</strong>
· 오늘 <strong class="text-text">{m.done_today.toLocaleString()}</strong>건 처리
</div>
</div>
{/each}
<!-- ETA 한 줄 (안5 라이트 — 추정치) -->
<div
class="text-[11px] text-dim leading-relaxed tabular-nums"
title="현재 페이스 기반 추정치 — 유입 변동 시 달라질 수 있습니다"
>
요약 대기 <strong class="text-text">{data.summarize_eta.pending.toLocaleString()}</strong>
— 소화 {formatRate(data.summarize_eta.done_rate_1h)}/h
· 유입 {formatRate(data.summarize_eta.inflow_rate_1h)}/h
{#if data.summarize_eta.eta_minutes != null}
· <span class="text-accent font-semibold">{etaPhrase(data.summarize_eta.eta_minutes)}</span>
{:else}
· 유입 우세(백필 중)
{/if}
</div>
<!-- 실패 합계 -->
{#if data.totals.failed > 0}
<div class="text-[11px] font-semibold text-error bg-error/10 rounded-md px-2.5 py-1.5 tabular-nums">
실패 {data.totals.failed.toLocaleString()}건 — 확인 필요
</div>
{/if}
{:else}
<p class="text-xs text-dim">처리 현황을 불러오지 못했습니다.</p>
{/if}
<a
href="/"
onclick={close}
class="block text-xs text-accent font-semibold hover:underline pt-1"
>홈에서 자세히 →</a>
</div>
</div>
</div>
{/if}
@@ -43,14 +43,17 @@
{@const open = selectedId === s.chunk_id}
{@const active = activeKey != null && activeKey === s.chunk_id}
{@const typeLabel = sectionTypeLabel(s.section_type)}
{@const depth = Math.max(0, (s.level ?? 1) - 1)}
<li>
<button
type="button"
onclick={() => { toggle(item); onJump?.(s.chunk_id); }}
aria-expanded={open}
aria-current={active ? 'true' : undefined}
style="padding-left:{8 + depth * 13}px"
class={[
'w-full text-left px-2 py-1.5 rounded-md text-xs flex items-start gap-1.5 transition-colors border-l-2',
'w-full text-left pr-2 py-1.5 rounded-md text-xs flex items-start gap-1.5 transition-colors border-l-2',
depth > 0 ? 'text-[11px]' : '',
open ? 'bg-surface-active text-text border-accent' : active ? 'bg-surface text-accent-hover border-accent' : 'text-dim hover:bg-surface hover:text-text border-transparent',
].join(' ')}
>
@@ -0,0 +1,31 @@
<!--
EidEvidenceCard — 이드 채팅 deep(검색) 답변의 근거 카드 (ds-eid-ask-absorb P1).
ReactResult.sources = {id, doc_id, title, score} (citation 번호 n 없음 — /ask 의 Citation 과
다름) → 순서 기반 번호([1],[2]...). 1단계 카드 = 제목·출처·점수 (스니펫은 react_loop
_result_payload items_src 에 없음 — 2단계 후보). 접이식 <details> 로 채팅 흐름 보존.
디자인 토큰만 (CLAUDE.md lint:tokens).
-->
<script lang="ts">
type EidSource = { id?: number; doc_id?: number; title?: string; score?: number };
let { sources, partial = false }: { sources: EidSource[]; partial?: boolean } = $props();
</script>
{#if sources.length}
<details class="mt-2 rounded-lg border border-default bg-surface text-xs max-w-[85%] sm:max-w-[75%]">
<summary class="cursor-pointer px-3 py-2 text-dim hover:text-text select-none font-semibold">
근거 {sources.length}{partial ? ' · 부분 답변 (확정 근거 부족)' : ''}
</summary>
<ul class="px-3 pb-2.5 flex flex-col gap-1.5">
{#each sources as src, i (src.id ?? i)}
<li class="flex items-start gap-2">
<span class="text-accent font-bold shrink-0">[{i + 1}]</span>
<span class="flex-1 min-w-0 text-text break-words">{src.title || `문서 ${src.doc_id ?? '?'}`}</span>
{#if typeof src.score === 'number'}
<span class="text-faint shrink-0 tabular-nums">{src.score.toFixed(2)}</span>
{/if}
</li>
{/each}
</ul>
</details>
{/if}
+70
View File
@@ -0,0 +1,70 @@
// 처리 큐 overview store — GET /api/queue/overview 를 60초 주기로 폴링.
// system.ts 의 dashboardSummary 와 같은 구독 기반 패턴 (첫 subscribe 시 시작).
//
// 의도적으로 api() 헬퍼를 쓰지 않는다 — 폴링 경로의 401 이 refresh 실패 →
// window.location='/login' 강제 logout 부수효과를 일으키면 안 됨 (eid 리뷰
// finding 재발 방지). 백엔드 미배포(404)/401/네트워크 실패 전부 silent 하게
// null 로 수렴하고, 소비자(스트립/보드/드로어)는 null 이면 스스로 숨는다.
import { writable } from 'svelte/store';
import { browser } from '$app/environment';
import { getAccessToken } from '$lib/api';
import type { QueueOverview } from '$lib/types/queue';
const POLL_INTERVAL_MS = 60_000;
let pollHandle: ReturnType<typeof setInterval> | null = null;
let subscriberCount = 0;
let inFlight: Promise<void> | null = null;
// 마지막 성공 갱신 시각(epoch ms) — 보드 신선도 '갱신 N초 전' + stale 경고용
// (ds-board-merged B-4). 실패(null 수렴) 시엔 갱신 안 함 → age 가 늘어 stale 로 드러남.
const updatedAt = writable<number | null>(null);
export const queueUpdatedAt = { subscribe: updatedAt.subscribe };
const internal = writable<QueueOverview | null>(null, (_set) => {
subscriberCount += 1;
if (subscriberCount === 1 && browser) {
void refreshQueueOverview();
pollHandle = setInterval(() => void refreshQueueOverview(), POLL_INTERVAL_MS);
}
return () => {
subscriberCount -= 1;
if (subscriberCount === 0 && pollHandle) {
clearInterval(pollHandle);
pollHandle = null;
}
};
});
export const queueOverview = { subscribe: internal.subscribe };
/** 경량 fetch — 실패는 전부 null (silent 비차단, 강제 logout 경로 없음) */
async function fetchOverview(): Promise<QueueOverview | null> {
try {
const headers: Record<string, string> = {};
const token = getAccessToken();
if (token) headers['Authorization'] = `Bearer ${token}`;
const res = await fetch('/api/queue/overview', { headers, credentials: 'include' });
if (!res.ok) return null;
return (await res.json()) as QueueOverview;
} catch {
return null;
}
}
/** 수동/추가 폴링용 — 홈은 자체 30s interval 로 이 함수를 호출 (동시 fetch 합치기) */
export async function refreshQueueOverview(): Promise<void> {
if (!browser) return;
if (inFlight) return inFlight;
inFlight = (async () => {
try {
const ov = await fetchOverview();
internal.set(ov);
if (ov) updatedAt.set(Date.now()); // 성공 시에만 신선도 갱신 (실패=stale 유지)
} finally {
inFlight = null;
}
})();
return inFlight;
}
+5 -3
View File
@@ -3,7 +3,9 @@
// (toast는 별도 store. drawer가 persistent inline panel(예: xl+ meta rail)일 때는
// 여기 시스템 밖이다 — 그저 레이아웃의 일부.)
type Drawer = { id: 'sidebar' | 'meta' } | null;
// 'queue' = 처리 현황 드로어 (상태 스트립 클릭 시 우측) — 단일 slot 규칙 동일
export type DrawerId = 'sidebar' | 'meta' | 'queue';
type Drawer = { id: DrawerId } | null;
type Modal = { id: string };
class UIState {
@@ -11,14 +13,14 @@ class UIState {
modalStack = $state<Modal[]>([]);
// ── Drawer (단일 slot) ──────────────────────────────
openDrawer(id: 'sidebar' | 'meta') {
openDrawer(id: DrawerId) {
// 새 drawer 열면 이전 drawer는 자동으로 사라진다 (단일 slot)
this.drawer = { id };
}
closeDrawer() {
this.drawer = null;
}
isDrawerOpen(id: 'sidebar' | 'meta') {
isDrawerOpen(id: DrawerId) {
return this.drawer?.id === id;
}
+115
View File
@@ -0,0 +1,115 @@
/**
* GET /api/queue/overview .
*
* Backend (feat/ds-processing-board).
* .
*/
export type MachineKey = 'gpu' | 'macmini' | 'macbook';
/** 머신 상태 — active(가동) / deferred(보류) / idle(대기) */
export type MachineState = 'active' | 'deferred' | 'idle';
/** 머신이 지금 처리 중인 문서 1건 */
export interface MachineCurrentItem {
document_id: number;
title: string;
stage: string;
}
export interface MachineOverview {
key: MachineKey;
label: string;
state: MachineState;
/** 담당 단계 키 목록 (extract/classify/... — 홈 STAGE_LABEL 로 한글화) */
stages: string[];
pending: number;
processing: number;
failed: number;
/** 최근 1시간 완료 건수 (처리율 N/h 표기) */
done_1h: number;
done_today: number;
/** 보류 건수 — 맥북 sleep 등으로 자동 재개 대기 중 */
deferred_pending: number;
current: MachineCurrentItem[];
}
/** 요약 백로그 ETA (안5 라이트) — 추정치, 유입 변동 시 오차 */
export interface SummarizeEta {
pending: number;
done_rate_1h: number;
inflow_rate_1h: number;
/** null = 유입이 소화를 앞섬 (백필 중) — 소진 예상 불가 */
eta_minutes: number | null;
}
/** 시간당 유입 vs 소화 (요약 24h 추이) */
export interface TrendPoint {
hour: string;
inflow: number;
done: number;
}
/** summarize 머신별 완료 실적 분담 (오프로드 가시화 — ds-board-merged A-1) */
export interface SummarizeByMachine {
macmini: { done_1h: number; done_today: number };
macbook: { done_1h: number; done_today: number };
}
export interface QueueTotals {
pending: number;
processing: number;
failed: number;
}
export interface QueueStageRow {
stage: string;
pending: number;
processing: number;
failed: number;
/** 최근 1시간 완료 — 노드 처리율·ETA 재료 (ds-board-engines-1) */
done_1h: number;
/** 최근 1시간 유입 — 유입 우세 판정 재료 (ds-board-engines-1) */
created_1h: number;
done_today: number;
oldest_pending_age_sec: number | null;
}
export interface QueueOverview {
machines: MachineOverview[];
summarize_eta: SummarizeEta;
summarize_by_machine: SummarizeByMachine;
trend_24h: TrendPoint[];
stages: QueueStageRow[];
totals: QueueTotals;
}
/** ─── 실패 처리 (ds-board-engines-1) — GET /api/queue/failed · POST /retry|/skip ─── */
export interface FailedItem {
id: number;
stage: string;
document_id: number;
title: string;
attempts: number;
max_attempts: number;
error_message: string | null;
failed_at: string | null;
}
export interface FailedListResponse {
items: FailedItem[];
total: number;
}
export interface RetryResponse {
requested: number;
retried: number;
not_retried: number;
}
export interface SkipResponse {
requested: number;
skipped: number;
not_skipped: number;
}
+121
View File
@@ -0,0 +1,121 @@
// 처리 머신 보드 / 상태 스트립 / 드로어 공용 표시 헬퍼.
// 상태 표현은 dot + 칩 (이모지 금지 원칙) — 토큰 클래스만 사용.
import type { MachineState } from '$lib/types/queue';
/** 머신 상태 한글 라벨 */
export const MACHINE_STATE_LABEL: Record<MachineState, string> = {
active: '가동',
deferred: '보류',
idle: '대기',
};
/** 상태 dot 색 — 가동=success / 보류=warning / 대기=faint */
export function machineDotClass(state: MachineState): string {
if (state === 'active') return 'bg-success';
if (state === 'deferred') return 'bg-warning';
return 'bg-faint';
}
/** 상태 칩 톤 — 가동=accent / 보류=warn / 대기=dim */
export function machineChipClass(state: MachineState): string {
if (state === 'active') return 'bg-accent/10 text-accent';
if (state === 'deferred') return 'bg-warning/10 text-warning';
return 'bg-surface-hover text-faint';
}
/** 처리율 표기 — 정수는 그대로, 소수는 한 자리 */
export function formatRate(n: number): string {
return Number.isInteger(n) ? n.toLocaleString() : n.toFixed(1);
}
/** ETA 분 → "약 N분/N시간 후 소진 예상" (추정치 — title 로 명시는 호출부 책임) */
export function etaPhrase(minutes: number): string {
if (minutes < 60) return `${Math.max(1, Math.round(minutes))}분 후 소진 예상`;
const hours = minutes / 60;
const text = hours >= 10 ? String(Math.round(hours)) : String(Math.round(hours * 10) / 10);
return `${text}시간 후 소진 예상`;
}
/** ETA 분 → 칩용 짧은 표기 ("약 12분" / "약 4.6시간" / 48h+ = "약 5.5일") */
export function etaShort(minutes: number): string {
if (minutes < 60) return `${Math.max(1, Math.round(minutes))}`;
const hours = minutes / 60;
if (hours >= 48) {
const days = hours / 24;
return `${days >= 10 ? Math.round(days) : Math.round(days * 10) / 10}`;
}
const text = hours >= 10 ? String(Math.round(hours)) : String(Math.round(hours * 10) / 10);
return `${text}시간`;
}
/** 경과 초 → "N분 전 / N시간 전 / N일 전" */
export function formatAgeSec(sec: number): string {
if (sec < 3600) return `${Math.max(1, Math.round(sec / 60))}분 전`;
if (sec < 86400) return `${Math.round(sec / 3600)}시간 전`;
return `${Math.round(sec / 86400)}일 전`;
}
/* (plan ds-board-engines-1)
* stage / () / . API label
* (raw ), · .
* / 1 (: 맥미니 ).
*/
export type FlowMachine = 'gpu' | 'macmini' | 'macbook';
export interface FlowNodeDef {
key: string;
/** 노드 표시명 */
label: string;
/** 합산할 stage 키 (다중 = 같은 엔진 공유) */
stages: string[];
machine: FlowMachine;
/** 엔진/모델 표시명 (FE 정적 — 모델 교체 시 여기 수정) */
engine: string;
/** 보조 표기 (서비스/워커명) */
sub: string;
}
/** 메인 흐름 (문서 진행 순서). 뉴스 등 소스별 스킵 경로는 그림에 안 그림 — 단순화 한계. */
export const FLOW_NODES: FlowNodeDef[] = [
{ key: 'extract', label: '추출', stages: ['extract'], machine: 'gpu', engine: 'Surya OCR', sub: 'ocr-service' },
{ key: 'markdown', label: '마크다운', stages: ['markdown'], machine: 'gpu', engine: 'Marker', sub: 'marker-service' },
{ key: 'classify', label: '분류', stages: ['classify'], machine: 'macmini', engine: 'Qwen3.6-27B', sub: 'classify + triage' },
{ key: 'summarize', label: '요약', stages: ['summarize'], machine: 'macmini', engine: 'Qwen3.6-27B', sub: 'summarize' },
{ key: 'chunkembed', label: '청크 · 임베딩', stages: ['chunk', 'embed'], machine: 'gpu', engine: 'TEI bge-m3', sub: 'text-embeddings-inference' },
{ key: 'deep', label: '심층분석', stages: ['deep_summary'], machine: 'macbook', engine: 'Qwen3.6-27B', sub: 'deep_summary' },
];
/** 보조 노드 — 메인 흐름 밖 (활동 있을 때만 보조 라인에 표시) */
export const AUX_NODES: FlowNodeDef[] = [
{ key: 'fulltext', label: '전문 수집', stages: ['fulltext'], machine: 'gpu', engine: 'Playwright', sub: 'playwright-fetcher' },
{ key: 'stt', label: '전사', stages: ['stt'], machine: 'gpu', engine: 'Whisper', sub: 'stt-service' },
{ key: 'util', label: '미리보기 · 썸네일', stages: ['preview', 'thumbnail'], machine: 'gpu', engine: '유틸', sub: 'ffmpeg' },
];
/** 머신 스트립 메타 — 모델 표기 단일 지점 */
export const MACHINE_META: Record<FlowMachine, { label: string; model: string }> = {
gpu: { label: 'GPU 서버', model: '특화 엔진' },
macmini: { label: '맥미니', model: 'Qwen3.6-27B-6bit · 24/7' },
macbook: { label: '맥북 M5 Max', model: 'Qwen3.6-27B · 야간 drain' },
};
/** 흐름 보드 단계 라벨 (드로어/상세 행 표기) */
export const FLOW_STAGE_LABEL: Record<string, string> = {
extract: '추출',
classify: '분류',
summarize: '요약',
embed: '임베딩',
chunk: '청크',
preview: '미리보기',
stt: '전사',
thumbnail: '썸네일',
deep_summary: '심층분석',
markdown: '마크다운',
fulltext: '전문',
};
export function flowStageLabel(stage: string): string {
return FLOW_STAGE_LABEL[stage] ?? stage;
}
+37
View File
@@ -8,8 +8,11 @@
import { toasts, removeToast } from '$lib/stores/toast';
import { refresh as refreshPublicConfig } from '$lib/stores/config';
import { ui } from '$lib/stores/uiState.svelte';
import { queueOverview } from '$lib/stores/queueOverview';
import { MACHINE_STATE_LABEL, machineChipClass } from '$lib/utils/queueDisplay';
import Sidebar from '$lib/components/Sidebar.svelte';
import SystemStatusDot from '$lib/components/SystemStatusDot.svelte';
import QueueDrawer from '$lib/components/QueueDrawer.svelte';
import QuickMemoButton from '$lib/components/QuickMemoButton.svelte';
import IconButton from '$lib/components/ui/IconButton.svelte';
import Drawer from '$lib/components/ui/Drawer.svelte';
@@ -65,6 +68,15 @@
let showChrome = $derived($isAuthenticated && !NO_CHROME_PATHS.some(p => $page.url.pathname.startsWith(p)));
let showSidebar = $derived(showChrome && !NO_SIDEBAR_PATHS.some(p => $page.url.pathname.startsWith(p)));
// 처리 현황 스트립 (안6 라이트) — 60s 폴링 store 공유. fetch 실패/401 시
// store 가 null → 스트립 자체를 숨김 (silent 비차단, 로그인 페이지 동일).
let queue = $derived($queueOverview);
let queueMacbook = $derived(queue?.machines?.find((m) => m.key === 'macbook') ?? null);
function toggleQueueDrawer() {
if (ui.isDrawerOpen('queue')) ui.closeDrawer();
else ui.openDrawer('queue');
}
function handleKeydown(e) {
if (e.key === '/' && !['INPUT', 'TEXTAREA'].includes(document.activeElement?.tagName)) {
e.preventDefault();
@@ -162,6 +174,28 @@
</div>
</nav>
<!-- 전 페이지 상태 스트립 (안6 라이트) — 클릭 시 우측 처리 현황 드로어 토글 -->
{#if queue}
<button
type="button"
onclick={toggleQueueDrawer}
aria-expanded={ui.isDrawerOpen('queue')}
aria-label="처리 현황 자세히 보기"
class="flex items-center gap-3 px-4 py-1.5 border-b border-default bg-surface text-[11px] text-dim shrink-0 text-left hover:bg-surface-hover transition-colors overflow-x-auto"
>
<span class="flex items-center gap-1.5 shrink-0">
<span class="w-2 h-2 rounded-full {queue.totals.processing > 0 ? 'bg-success' : 'bg-faint'}"></span>
<strong class="text-text font-semibold tabular-nums">처리 중 {queue.totals.processing.toLocaleString()}</strong>
</span>
<span class="tabular-nums shrink-0">대기 <strong class="text-text">{queue.totals.pending.toLocaleString()}</strong></span>
<span class="tabular-nums shrink-0 {queue.totals.failed > 0 ? 'text-error font-semibold' : ''}">실패 <strong class={queue.totals.failed > 0 ? '' : 'text-text'}>{queue.totals.failed.toLocaleString()}</strong></span>
{#if queueMacbook}
<span class="text-[10px] font-bold rounded-full px-2 py-0.5 shrink-0 {machineChipClass(queueMacbook.state)}">맥북 {MACHINE_STATE_LABEL[queueMacbook.state]}</span>
{/if}
<span class="ml-auto flex items-center gap-0.5 text-faint shrink-0">자세히 <ChevronDown size={11} /></span>
</button>
{/if}
<!-- 메인: 데스크탑 상시 사이드바 + 콘텐츠 -->
<div class="flex-1 min-h-0 flex">
{#if showSidebar}
@@ -191,6 +225,9 @@
</Drawer>
</div>
<!-- 처리 현황 드로어 (안6 라이트, 스트립 클릭 시 우측) -->
<QueueDrawer />
<!-- 빠른 메모 FAB -->
<QuickMemoButton />
</div>
+19 -67
View File
@@ -13,10 +13,13 @@
import { domainBgClass, domainLabel } from '$lib/utils/domainSlug';
import { user } from '$lib/stores/auth';
import { api } from '$lib/api';
import { queueOverview, refreshQueueOverview } from '$lib/stores/queueOverview';
import ProcessingFlowBoard from '$lib/components/ProcessingFlowBoard.svelte';
import type { QueueOverview } from '$lib/types/queue';
import EmptyState from '$lib/components/ui/EmptyState.svelte';
import Skeleton from '$lib/components/ui/Skeleton.svelte';
import {
Scale, FileText, Pin, ChevronRight, GraduationCap, Upload, Newspaper,
Scale, FileText, Pin, GraduationCap, Upload, Newspaper,
} from 'lucide-svelte';
import { addToast } from '$lib/stores/toast';
@@ -125,6 +128,17 @@
preview: '미리보기', thumbnail: '썸네일',
};
// ─── 처리 머신 보드 (안2) + ETA (안5 라이트) — GET /api/queue/overview ───
// 홈은 30s 폴링 (store 기본 60s 위에 추가 — inFlight 합치기로 중복 호출 0).
// 백엔드 미배포/실패 시 store=null → 보드 자체가 조용히 생략 (silent 비차단).
let queue = $derived<QueueOverview | null>($queueOverview);
onMount(() => {
void refreshQueueOverview();
const handle = setInterval(() => void refreshQueueOverview(), 30_000);
return () => clearInterval(handle);
});
interface PipelineRow {
stage: string; label: string;
pending: number; processing: number; failed: number; total: number;
@@ -166,22 +180,10 @@
let pipelineRows = $derived(
summary ? buildPipelineRows(summary.pipeline_status, summary.queue_lag ?? []) : []
);
let pipelineMax = $derived(Math.max(1, ...pipelineRows.map((r) => r.total)));
let totalFailed = $derived(summary?.failed_count ?? 0);
let totalPending = $derived(pipelineRows.reduce((s, r) => s + r.pending, 0));
let totalProcessing = $derived(pipelineRows.reduce((s, r) => s + r.processing, 0));
let pipelineManualClosed = $state(false);
let pipelineOpen = $derived(pipelineManualClosed ? false : totalFailed > 0);
function formatAge(sec: number | null): string {
if (sec == null || sec <= 0) return '';
if (sec < 60) return `${sec} `;
if (sec < 3600) return `${Math.floor(sec / 60)} `;
if (sec < 86400) return `${Math.floor(sec / 3600)}시간 `;
return `${Math.floor(sec / 86400)}일 전`;
}
function formatTime(dateStr: string) {
const d = new Date(dateStr);
if (isNaN(d.getTime())) return ''; // 빈 문자열/유효하지 않은 created_at → 'Invalid Date' 회피
@@ -420,56 +422,10 @@
</div>
</div>
<!-- ═══ 파이프라인 상세 (실패 있을 때 자동 펼침) ═══ -->
<details
class="mt-5"
open={pipelineOpen}
ontoggle={(e) => { if (!e.currentTarget.open) pipelineManualClosed = true; }}
>
<summary class="flex items-center justify-between px-5 py-3.5 bg-surface border border-default rounded-card cursor-pointer hover:bg-surface-hover transition-colors select-none list-none">
<span class="text-sm font-semibold text-text flex items-center gap-2">
<ChevronRight size={14} class="transition-transform details-chevron" />
파이프라인 상세
</span>
<span class="text-xs text-dim flex items-center gap-2.5">
{#if totalFailed > 0}<span class="text-error font-medium">실패 {totalFailed}</span>{/if}
{#if totalPending > 0}<span>대기 {totalPending}</span>{/if}
{#if totalFailed === 0 && totalPending === 0}<span>처리 완료</span>{/if}
</span>
</summary>
<div class="mt-2 px-5 py-4 bg-surface border border-default rounded-card">
<p class="text-xs text-dim mb-3">최근 24시간</p>
{#if pipelineRows.length > 0}
<div class="space-y-3">
{#each pipelineRows as row (row.stage)}
<div>
<div class="flex items-center justify-between text-xs mb-1.5">
<span class="text-dim">
{row.label}
{#if row.oldestPendingAgeSec && row.oldestPendingAgeSec > 600}
<span class="ml-1 text-warning" title="가장 오래된 pending 의 경과 시간">({formatAge(row.oldestPendingAgeSec)})</span>
{/if}
</span>
<span class="text-dim tabular-nums">
대기 <span class="text-text">{row.pending}</span> ·
처리 <span class="text-text">{row.processing}</span> ·
실패 <span class={row.failed > 0 ? 'text-error font-medium' : ''}>{row.failed}</span>
</span>
</div>
<div class="flex h-1.5 w-full overflow-hidden rounded-sm bg-bg">
{#if row.pending > 0}<div class="bg-warning h-full" style="width: {(row.pending / pipelineMax) * 100}%"></div>{/if}
{#if row.processing > 0}<div class="bg-accent h-full" style="width: {(row.processing / pipelineMax) * 100}%"></div>{/if}
{#if row.failed > 0}<div class="bg-error h-full" style="width: {(row.failed / pipelineMax) * 100}%"></div>{/if}
</div>
</div>
{/each}
</div>
{:else}
<p class="text-xs text-dim text-center py-3">처리 작업 없음</p>
{/if}
</div>
</details>
<!-- ═══ 처리 머신 보드 v2 — 파이프라인 흐름 + 상세 패널 + 실패 드로어 (ds-board-engines-1) ═══ -->
{#if queue}
<ProcessingFlowBoard overview={queue} />
{/if}
{/if}
</div>
@@ -482,7 +438,3 @@
</div>
{/snippet}
<style>
details[open] :global(.details-chevron) { transform: rotate(90deg); }
details summary::-webkit-details-marker { display: none; }
</style>
+194 -10
View File
@@ -17,6 +17,12 @@
macbook_unavailable / substrate_degraded / 기타 detail). 자동 fallback
금지 — 다른 모드로 자동 전환하지 않는다. 스트림 도중 중단 = 받은 부분
유지 + 표시.
- 대기 표시(첫 바이트 전): 경과 타이머 1초 갱신 + 3초 후 GET /api/eid/status
1회·이후 8초 간격 재조회(실패는 조용히 무시 — 기능 비차단)로 "대기"와
"고장"을 정직하게 구분. daily.busy=true 면 줄 서는 중 안내. 15초 경과 +
daily 모드면 [심층으로 전환]/[취소] 버튼 노출 — 전환은 명시 클릭만
(자동 fallback 금지 정책 위반 아님). 첫 바이트 도착/스트림 종료 시
타이머·폴링 즉시 정리.
- 이력: localStorage `eid_chat:v1` (키 상수는 $lib/eidChat — logout 시 제거와 공유).
전송 payload 는 마지막 20턴(40 messages) cap.
- 입력 한도: 메시지당 8,000자 클라 선차단(서버 422 검증과 동일 한도).
@@ -25,15 +31,25 @@
-->
<script lang="ts">
import { onMount, onDestroy } from 'svelte';
import { apiFetchRaw } from '$lib/api';
import { api, apiFetchRaw } from '$lib/api';
import { EID_CHAT_STORAGE_KEY } from '$lib/eidChat';
import Button from '$lib/components/ui/Button.svelte';
import EmptyState from '$lib/components/ui/EmptyState.svelte';
import EidEvidenceCard from '$lib/components/eid/EidEvidenceCard.svelte';
import { MessageCircle, SendHorizontal, RotateCcw, AlertCircle } from 'lucide-svelte';
type ChatMode = 'daily' | 'deep';
type ChatMessage = { role: 'user' | 'assistant'; content: string };
// deep(검색) 답변은 sources(근거)·partial 동반. daily 답변은 없음.
type EidSource = { id?: number; doc_id?: number; title?: string; score?: number };
type ChatMessage = {
role: 'user' | 'assistant';
content: string;
sources?: EidSource[];
partial?: boolean;
};
type Notice = { kind: 'warn' | 'error'; message: string; retryable: boolean };
// GET /api/eid/status 응답 — 대기 중 바쁨 신호 조회에 필요한 필드만 좁게 정의
type EidStatus = { daily?: { busy?: boolean } };
// 이력 키 — logout(stores/auth.ts) 의 이력 제거와 단일 상수 공유
const STORAGE_KEY = EID_CHAT_STORAGE_KEY;
@@ -45,6 +61,10 @@
const MAX_MESSAGE_CHARS = 8000;
// 한도 근접 카운터 노출 시작점
const COUNTER_THRESHOLD = 7500;
// 대기 표시(첫 바이트 전): 상태 폴링 시작 시점(초) / 재조회 간격(초) / 행동 버튼 노출 시점(초)
const STATUS_POLL_START_SEC = 3;
const STATUS_POLL_INTERVAL_SEC = 8;
const WAIT_ACTIONS_SEC = 15;
const DEEP_CAPTION =
'장문·무거운 질문에 적합 — 잠들어 있으면 자동 기동 (처음 응답까지 최대 ~1분)';
@@ -64,11 +84,72 @@
let streaming = $state(false);
let streamingText = $state('');
let notice = $state<Notice | null>(null);
// deep(검색) 모드 첫 바이트 전 단계 — 'searching' 이면 대기 표시를 "근거 검색 중"으로
let deepPhase = $state<'searching' | null>(null);
let scrollEl: HTMLDivElement | undefined = $state();
let textareaEl: HTMLTextAreaElement | undefined = $state();
let abortCtrl: AbortController | null = null;
// ── 대기 추적 (첫 바이트 전) ────────────────────────
// 경과 초 + daily 엔진 바쁨 여부(null = 미확인). 토큰(세대 카운터)으로
// 스트림별 소유를 구분 — abort 직후 즉시 재전송(심층 전환) 경로에서
// 이전 스트림의 늦은 정리가 새 스트림의 타이머를 죽이지 않게 한다.
let waitSeconds = $state(0);
let dailyBusy = $state<boolean | null>(null);
let waitIntervalId: ReturnType<typeof setInterval> | null = null;
let waitTokenSeq = 0;
let waitToken = 0; // 현재 활성 추적 토큰 (0 = 추적 없음)
function startWaitTracking(streamMode: ChatMode): number {
// 이전 추적 잔여 정리 (전환 재전송처럼 stop 전에 start 가 오는 경로 방어)
if (waitIntervalId !== null) {
clearInterval(waitIntervalId);
waitIntervalId = null;
}
const token = ++waitTokenSeq;
waitToken = token;
waitSeconds = 0;
dailyBusy = null;
waitIntervalId = setInterval(() => {
if (waitToken !== token) return; // 정리 누락 방어 — 무해 no-op
waitSeconds += 1;
// 바쁨 신호 폴링: 3초 경과 시 1회 + 이후 8초 간격 (3, 11, 19, ...).
// daily 모드 전용 — deep 대기는 기존 wake 안내 + 경과 타이머만.
if (
streamMode === 'daily' &&
waitSeconds >= STATUS_POLL_START_SEC &&
(waitSeconds - STATUS_POLL_START_SEC) % STATUS_POLL_INTERVAL_SEC === 0
) {
void pollEidStatus(token);
}
}, 1000);
return token;
}
// token 가드: 본인 소유 추적만 정리 — 다른 스트림이 이어받았으면 no-op
function stopWaitTracking(token: number) {
if (token !== waitToken) return;
waitToken = 0;
if (waitIntervalId !== null) {
clearInterval(waitIntervalId);
waitIntervalId = null;
}
waitSeconds = 0;
dailyBusy = null;
}
// 상태 조회 — 실패는 조용히 무시 (대기 표시는 타이머만으로 유지, 기능 비차단)
async function pollEidStatus(token: number) {
try {
const status = await api<EidStatus>('/eid/status');
if (token !== waitToken) return; // 스트림 종료/교체 후 도착한 늦은 응답 폐기
dailyBusy = status?.daily?.busy === true;
} catch {
// 무시 — 바쁨 신호는 부가 정보일 뿐 채팅 기능을 차단하지 않는다
}
}
// ── localStorage 이력 ───────────────────────────────
function persist() {
if (typeof window === 'undefined') return;
@@ -97,9 +178,15 @@
typeof (m as ChatMessage).content === 'string'
)
// 배열 크기 가드 + content 8,000자 clamp — 외부에서 손상/비대해진
// 이력이 전송 payload 를 오염시키지 않도록 복원 시점에 정규화
// 이력이 전송 payload 를 오염시키지 않도록 복원 시점에 정규화.
// sources/partial(deep 답변 근거)은 보존 — 전송 payload 엔 안 실림(runStream map 이 role/content 만).
.slice(-MAX_STORED_MESSAGES)
.map((m) => ({ role: m.role, content: m.content.slice(0, MAX_MESSAGE_CHARS) }));
.map((m) => ({
role: m.role,
content: m.content.slice(0, MAX_MESSAGE_CHARS),
sources: Array.isArray((m as ChatMessage).sources) ? (m as ChatMessage).sources : undefined,
partial: (m as ChatMessage).partial === true || undefined,
}));
}
} catch {
// 손상된 이력은 무시 (새 대화로 시작)
@@ -107,7 +194,11 @@
}
onMount(() => restore());
onDestroy(() => abortCtrl?.abort());
onDestroy(() => {
abortCtrl?.abort();
// 페이지 이탈 시 대기 타이머/폴링 정리 (abort 의 finally 와 이중이어도 무해)
if (waitIntervalId !== null) clearInterval(waitIntervalId);
});
// ── 자동 스크롤 (새 메시지 / 스트림 청크마다 하단 고정) ──
$effect(() => {
@@ -235,12 +326,39 @@
void runStream();
}
// ── 대기 중 행동 버튼 (daily + 15초 경과) ────────────
// [심층으로 전환] — 명시 클릭에 의한 모드 전환 (자동 fallback 금지 정책
// 위반 아님). 현재 fetch abort → 같은 user 턴을 mode=deep 으로 즉시 재전송.
// abort 된 이전 스트림의 finally 는 abortCtrl 비교 + 대기 token 가드로
// 새 스트림 상태를 건드리지 않는다 (새 대화 abort race 가드와 동일 구조).
function switchToDeep() {
if (!streaming || mode !== 'daily') return;
mode = 'deep'; // 모드 토글 상태도 deep 으로 갱신
abortCtrl?.abort();
void runStream();
}
// [취소] — abort 후 방금 push 한 user 턴 pop + 입력창 본문 복원
// (422 처리와 동일 패턴: 이력 오염 차단 + localStorage 재저장).
// placeholder 제거는 abort 된 스트림의 finally(streaming=false)가 처리.
function cancelWait() {
if (!streaming) return;
abortCtrl?.abort();
if (messages.length > 0 && messages[messages.length - 1].role === 'user') {
const popped = messages.pop();
if (popped && !input) input = popped.content;
persist();
}
}
async function runStream() {
notice = null;
streaming = true;
streamingText = '';
const ctrl = new AbortController();
abortCtrl = ctrl;
// 첫 바이트 전 대기 추적 시작 — 본 스트림 소유 토큰으로 정리 시점 제어
const waitTok = startWaitTracking(mode);
const payload = {
mode,
@@ -251,6 +369,9 @@
let acc = '';
let sawDone = false;
// deep(검색) 답변 동반 데이터 — daily 는 안 옴
let accSources: EidSource[] = [];
let accPartial = false;
try {
const res = await apiFetchRaw('/eid/chat', {
@@ -301,9 +422,35 @@
try {
const obj = JSON.parse(data) as {
choices?: Array<{ delta?: { content?: unknown } }>;
phase?: string;
error_reason?: string;
eid_sources?: EidSource[];
partial?: boolean;
};
// deep(검색) envelope 분기 — daily 응답엔 없음
if (obj?.phase === 'ping') return false; // heartbeat — 무시
if (obj?.phase === 'searching') {
deepPhase = 'searching'; // 대기 표시를 "근거 검색 중"으로
return false;
}
if (obj?.phase === 'error') {
// in-stream 미가용/실패 — 받은 부분 유지 + 명시 표시 (자동 fallback 0).
// 뒤따르는 [DONE] 이 sawDone 처리하므로 '중단' 오경보 없음.
notice = mapErrorReason(obj.error_reason, '');
return false;
}
if (Array.isArray(obj?.eid_sources)) {
accSources = obj.eid_sources;
accPartial = obj.partial === true;
return false;
}
const piece = obj?.choices?.[0]?.delta?.content;
if (typeof piece === 'string' && piece) {
// 첫 바이트 도착 — 대기 타이머/폴링 제거, 기존 스트리밍 표시로 전환
if (!acc) {
stopWaitTracking(waitTok);
deepPhase = null;
}
acc += piece;
streamingText = acc;
}
@@ -356,7 +503,7 @@
}
} catch (err) {
if ((err as Error)?.name === 'AbortError') {
// 새 대화 등 사용자 의도 중단 — 안내 불필요
// 새 대화 / 대기 취소 / 심층 전환 등 사용자 의도 중단 — 안내 불필요
return;
}
// 스트림 도중 네트워크 에러 — 받은 부분 유지 + 표시
@@ -368,14 +515,23 @@
}
: { kind: 'error', message: '요청에 실패했습니다 — 네트워크를 확인하세요.', retryable: true };
} finally {
// 스트림 종료 — 대기 타이머/폴링 정리. 첫 바이트에서 이미 정리됐거나
// 전환 재전송으로 새 스트림이 추적을 이어받았으면 token 가드로 no-op.
stopWaitTracking(waitTok);
// abort(새 대화/페이지 이탈) 시에는 push 하지 않음 — 새 대화로 비운
// messages 에 이전 스트림 잔여분이 흘러들어가는 race 방지.
if (acc && !ctrl.signal.aborted) {
messages.push({ role: 'assistant', content: acc });
messages.push({
role: 'assistant',
content: acc,
sources: accSources.length ? accSources : undefined,
partial: accPartial || undefined,
});
}
if (abortCtrl === ctrl) {
streaming = false;
streamingText = '';
deepPhase = null;
abortCtrl = null;
}
persist();
@@ -398,6 +554,24 @@
// 입력 길이(전송 기준 = trim 후) — 7,500자부터 카운터 노출, 8,000자 초과 차단
let inputLength = $derived(input.trim().length);
let overLimit = $derived(inputLength > MAX_MESSAGE_CHARS);
// 첫 바이트 전 placeholder 문구 — "대기"와 "고장"의 정직한 구분:
// 바쁨 확인 = 줄 서는 중 / 비-바쁨 확인 = 생성 준비 중 / 미확인 = 응답 대기 중.
// deep 모드는 폴링하지 않으므로 항상 미확인(타이머만) — wake 안내는 헤더 caption.
let waitPlaceholder = $derived(
deepPhase === 'searching'
? `이드가 문서·뉴스에서 근거를 찾는 중 · ${waitSeconds}초`
: dailyBusy === true
? `엔진이 다른 작업을 처리하고 있어요 — 차례가 오면 바로 시작됩니다 (대기 ${waitSeconds}초)`
: dailyBusy === false
? `응답 생성 준비 중 · ${waitSeconds}초`
: `응답 대기 중 · ${waitSeconds}초`
);
// 행동 버튼 노출: daily 모드 + 첫 바이트 전 + 15초 경과
let showWaitActions = $derived(
streaming && !streamingText && mode === 'daily' && waitSeconds >= WAIT_ACTIONS_SEC
);
</script>
<svelte:head>
@@ -473,25 +647,35 @@
</div>
</div>
{:else}
<div class="flex justify-start">
<div class="flex flex-col items-start">
<div class="max-w-[85%] sm:max-w-[75%] px-3.5 py-2.5 rounded-lg rounded-bl-sm bg-surface border border-default text-text text-sm whitespace-pre-wrap break-words">
{msg.content}
</div>
{#if msg.sources?.length}
<EidEvidenceCard sources={msg.sources} partial={msg.partial ?? false} />
{/if}
</div>
{/if}
{/each}
<!-- 스트리밍 중 assistant 부분 응답 -->
<!-- 스트리밍 중 assistant 부분 응답 / 첫 바이트 전 대기 표시 -->
{#if streaming}
<div class="flex justify-start">
<div class="max-w-[85%] sm:max-w-[75%] px-3.5 py-2.5 rounded-lg rounded-bl-sm bg-surface border border-default text-text text-sm whitespace-pre-wrap break-words">
{#if streamingText}
{streamingText}<span class="inline-block w-1.5 h-3.5 ml-0.5 align-middle bg-accent animate-pulse rounded-sm"></span>
{:else}
<span class="text-dim animate-pulse">응답 준비 중...</span>
<span class="text-dim animate-pulse">{waitPlaceholder}</span>
{/if}
</div>
</div>
<!-- 대기 행동 버튼: daily + 15초 경과 — 전환은 명시 클릭만 (자동 fallback 금지) -->
{#if showWaitActions}
<div class="flex justify-start gap-2">
<Button variant="secondary" size="sm" onclick={switchToDeep}>심층으로 전환</Button>
<Button variant="ghost" size="sm" onclick={cancelWait}>취소</Button>
</div>
{/if}
{/if}
<!-- 에러/안내 카드: 자동 fallback 없이 명시 표시만 -->
+62 -48
View File
@@ -30,7 +30,6 @@
import AnalysisPanel from '$lib/components/AnalysisPanel.svelte';
import ReadCounter from '$lib/components/ReadCounter.svelte';
import SectionOutline from '$lib/components/SectionOutline.svelte';
import Tabs from '$lib/components/ui/Tabs.svelte';
marked.use({ mangle: false, headerIds: false });
function renderMd(text) {
@@ -460,53 +459,68 @@
{/if}
</div>
<!-- 오른쪽 — 메타 Tabs [정보 | AI | 관리] (카드 11개 수직 스프롤 해소) -->
<aside class="min-w-0">
<Card>
<Tabs
tabs={[
{ id: 'info', label: '정보' },
{ id: 'ai', label: 'AI' },
{ id: 'manage', label: '관리' },
]}
>
{#snippet children(tab)}
<div class="pt-3 space-y-4">
{#if tab === 'info'}
{#if doc.category === 'library'}
<ReadCounter
documentId={doc.id}
initialCount={doc.read_count ?? 0}
initialLastReadAt={doc.last_read_at ?? null}
/>
{/if}
<FileInfoView {doc} />
<ProcessingStatusView {doc} />
{:else if tab === 'ai'}
<AnalysisPanel docId={doc.id} doc={doc} />
<AIClassificationEditor {doc} />
<div>
<h4 class="text-xs font-semibold text-dim uppercase mb-1.5">관련 문서</h4>
<!-- TODO(backend): GET /documents/{id}/related?limit=10 (벡터 유사도) -->
<EmptyState
icon={FileText}
title="추후 지원"
description="관련 문서 추천은 backend 연동 후 제공됩니다."
/>
</div>
{:else}
<LibraryPathEditor {doc} />
<NoteEditor {doc} />
<EditUrlEditor {doc} />
<TagsEditor {doc} />
<div class="pt-2 border-t border-default">
<DocumentDangerZone {doc} ondelete={handleDocDelete} />
</div>
{/if}
</div>
{/snippet}
</Tabs>
</Card>
<!-- 오른쪽 — 슬림 전역 인사이트 레일 (D3: 탭 게이트 제거, 요약·심층·불일치 상시 노출).
정보/관리는 접이(<details>) — 데스크탑은 인사이트 상시, 모바일은 본문 메인 + 열어서 확인. -->
<aside class="min-w-0 space-y-3">
{#if doc.category === 'library'}
<Card>
<ReadCounter
documentId={doc.id}
initialCount={doc.read_count ?? 0}
initialLastReadAt={doc.last_read_at ?? null}
/>
</Card>
{/if}
<!-- 요약·분석 — 기본 펼침(데스크탑 상시감, 모바일 접기 가능) -->
<details open class="bg-surface border border-default rounded-card overflow-hidden group">
<summary class="cursor-pointer list-none flex items-center justify-between px-3.5 py-2.5 text-xs font-semibold text-dim uppercase tracking-wide select-none">
<span>요약 · 분석</span>
<ChevronRight size={14} class="transition-transform group-open:rotate-90 text-faint" />
</summary>
<div class="px-3.5 pb-3.5 space-y-4">
<AnalysisPanel docId={doc.id} doc={doc} />
<AIClassificationEditor {doc} />
<div>
<h4 class="text-xs font-semibold text-dim uppercase mb-1.5">관련 문서</h4>
<!-- TODO(backend): GET /documents/{id}/related?limit=10 (벡터 유사도) — v1 제외(자리만) -->
<EmptyState
icon={FileText}
title="추후 지원"
description="관련 문서 추천은 backend 연동 후 제공됩니다."
/>
</div>
</div>
</details>
<!-- 문서 정보 — 접이(기본 닫힘) -->
<details class="bg-surface border border-default rounded-card overflow-hidden group">
<summary class="cursor-pointer list-none flex items-center justify-between px-3.5 py-2.5 text-xs font-semibold text-dim uppercase tracking-wide select-none">
<span>문서 정보</span>
<ChevronRight size={14} class="transition-transform group-open:rotate-90 text-faint" />
</summary>
<div class="px-3.5 pb-3.5 space-y-3">
<FileInfoView {doc} />
<ProcessingStatusView {doc} />
</div>
</details>
<!-- 관리 — 접이(기본 닫힘) -->
<details class="bg-surface border border-default rounded-card overflow-hidden group">
<summary class="cursor-pointer list-none flex items-center justify-between px-3.5 py-2.5 text-xs font-semibold text-dim uppercase tracking-wide select-none">
<span>관리</span>
<ChevronRight size={14} class="transition-transform group-open:rotate-90 text-faint" />
</summary>
<div class="px-3.5 pb-3.5 space-y-3">
<LibraryPathEditor {doc} />
<NoteEditor {doc} />
<EditUrlEditor {doc} />
<TagsEditor {doc} />
<div class="pt-2 border-t border-default">
<DocumentDangerZone {doc} ondelete={handleDocDelete} />
</div>
</div>
</details>
</aside>
</div>
+8
View File
@@ -0,0 +1,8 @@
-- Phase 2A (embedding-phase2a-1 E-1): 후보 임베딩 docs 섀도 테이블 (eval 전용, 단일 statement).
-- 평가 = exact scan 이라 벡터 인덱스 없음 (인덱스 전략 = C-1 컷오버 소관).
CREATE TABLE IF NOT EXISTS documents_cand_qwen06 (
doc_id BIGINT PRIMARY KEY,
embed_input_hash TEXT,
embedding vector(1024) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
@@ -0,0 +1,10 @@
-- Phase 2A (embedding-phase2a-1 E-1): 후보 임베딩 chunks 섀도 테이블 (eval 전용, 단일 statement).
CREATE TABLE IF NOT EXISTS document_chunks_cand_qwen06 (
id BIGINT PRIMARY KEY,
doc_id BIGINT NOT NULL,
chunk_index INTEGER,
section_title TEXT,
text TEXT,
embedding vector(1024) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
+8
View File
@@ -0,0 +1,8 @@
-- Phase 2A (embedding-phase2a-1 E-1): 후보 임베딩 docs 섀도 테이블 (eval 전용, 단일 statement).
-- 평가 = exact scan 이라 벡터 인덱스 없음 (인덱스 전략 = C-1 컷오버 소관).
CREATE TABLE IF NOT EXISTS documents_cand_qwen4 (
doc_id BIGINT PRIMARY KEY,
embed_input_hash TEXT,
embedding vector(2560) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
@@ -0,0 +1,10 @@
-- Phase 2A (embedding-phase2a-1 E-1): 후보 임베딩 chunks 섀도 테이블 (eval 전용, 단일 statement).
CREATE TABLE IF NOT EXISTS document_chunks_cand_qwen4 (
id BIGINT PRIMARY KEY,
doc_id BIGINT NOT NULL,
chunk_index INTEGER,
section_title TEXT,
text TEXT,
embedding vector(2560) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
+8
View File
@@ -0,0 +1,8 @@
-- Phase 2A (embedding-phase2a-1 E-1): 후보 임베딩 docs 섀도 테이블 (eval 전용, 단일 statement).
-- 평가 = exact scan 이라 벡터 인덱스 없음 (인덱스 전략 = C-1 컷오버 소관).
CREATE TABLE IF NOT EXISTS documents_cand_qwen4m (
doc_id BIGINT PRIMARY KEY,
embed_input_hash TEXT,
embedding vector(1024) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
@@ -0,0 +1,10 @@
-- Phase 2A (embedding-phase2a-1 E-1): 후보 임베딩 chunks 섀도 테이블 (eval 전용, 단일 statement).
CREATE TABLE IF NOT EXISTS document_chunks_cand_qwen4m (
id BIGINT PRIMARY KEY,
doc_id BIGINT NOT NULL,
chunk_index INTEGER,
section_title TEXT,
text TEXT,
embedding vector(1024) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
@@ -0,0 +1,6 @@
-- 340_documents_material_type.sql
-- 안전 자료실 분류 축 A-1 (1/12) — 자료유형 컬럼.
-- plan: safety-library-1 (PKM plans/2026-06-12-safety-library-plan.html)
-- TEXT+CHECK 방식 (PG enum 아님 — 152 의 enum ADD VALUE 동일-런 사용 불가 함정 회피).
-- 값 부여 = 수집기 ingest 시점 deterministic (classify_worker 아님 — classify-skip 경로 다수).
ALTER TABLE documents ADD COLUMN IF NOT EXISTS material_type TEXT;
@@ -0,0 +1,6 @@
-- 341_documents_material_type_check.sql
-- 안전 자료실 분류 축 A-1 (2/12) — material_type 값 공간 named CHECK.
-- plan: safety-library-1 0-1 확정 7값. 값 추가 시 = 본 제약 DROP + 재ADD 2파일 (named 라 가능).
-- NULL 은 CHECK 통과 (비안전/일반 문서는 NULL 유지 — 전수 분류 시도 금지).
ALTER TABLE documents ADD CONSTRAINT chk_documents_material_type
CHECK (material_type IN ('law', 'paper', 'book', 'incident', 'manual', 'standard', 'guide'));
@@ -0,0 +1,5 @@
-- 342_documents_jurisdiction.sql
-- 안전 자료실 분류 축 A-1 (3/12) — 관할(나라) 컬럼. 법령 1급 시민 축.
-- plan: safety-library-1 0-1. 'GB' 표기 (news_sources.country 실측 어휘와 통일, UI 라벨만 UK).
-- paper 는 NULL 허용 (국제 학술지 — 관할 개념 부적합). INT = ISO 류 국제기구 자료 유보.
ALTER TABLE documents ADD COLUMN IF NOT EXISTS jurisdiction TEXT;
@@ -0,0 +1,4 @@
-- 343_documents_jurisdiction_check.sql
-- 안전 자료실 분류 축 A-1 (4/12) — jurisdiction 값 공간 named CHECK.
ALTER TABLE documents ADD CONSTRAINT chk_documents_jurisdiction
CHECK (jurisdiction IN ('KR', 'US', 'EU', 'JP', 'GB', 'INT'));
@@ -0,0 +1,7 @@
-- 344_documents_law_jurisdiction_check.sql
-- 안전 자료실 분류 축 A-1 (5/12) — 나라 혼선 금지를 구조로 강제.
-- 법령(material_type='law')인데 jurisdiction NULL 인 행은 적재 자체가 거부된다.
-- 업로드 승인 경로는 proposed_jurisdiction 필수 입력 (KR 기본값 오염 금지 — plan A-2).
-- material_type 이 NULL 이면 식 전체가 NULL = CHECK 통과 (비법령 무영향).
ALTER TABLE documents ADD CONSTRAINT chk_documents_law_jurisdiction
CHECK (material_type <> 'law' OR jurisdiction IS NOT NULL);
@@ -0,0 +1,5 @@
-- 345_documents_published_date.sql
-- 안전 자료실 분류 축 A-1 (6/12) — 유형별 대표 날짜 (패싯 연도·freshness 단일 날짜 축).
-- 법령 = COALESCE(effective_date, promulgation_date) — plan 0-1 R2-M2 확정.
-- 논문 = 발행일 / 재해 = 발생일 / 뉴스·크롤 = extract_meta.published_at backfill (A-3).
ALTER TABLE documents ADD COLUMN IF NOT EXISTS published_date DATE;
+22
View File
@@ -0,0 +1,22 @@
-- 346_legal_acts_table.sql
-- 안전 자료실 A-1 (7/12) — 법령 레지스트리 = 워치리스트 (news_sources 패턴의 법령판).
-- plan: safety-library-1 0-2. statute_watchlist 별도 테이블 안 만듦 (R2 blocker — 이중 정의 해소, watermark 흡수).
-- KOSHA GUIDE / KGS Code 는 비대상 (guide=비법령, KGS=watch-폴더 단독 트랙 R3-M5).
-- 시드 = B-1 PR① (레거시 law_monitor 26개 superset, watch=true 전부 — R3-B1).
-- repeal_detected_at: 어댑터(코어)는 폐지 감지 마킹만, 전이는 일일 잡 단일 지점 (R3-M3).
CREATE TABLE IF NOT EXISTS legal_acts (
family_id TEXT PRIMARY KEY,
jurisdiction TEXT NOT NULL CHECK (jurisdiction IN ('KR', 'US', 'EU', 'JP', 'GB', 'INT')),
law_level TEXT NOT NULL CHECK (law_level IN ('statute', 'decree', 'rule', 'admin_rule', 'code')),
title TEXT NOT NULL,
title_ko TEXT,
parent_family_id TEXT REFERENCES legal_acts(family_id),
native_id TEXT NOT NULL,
source_api TEXT NOT NULL,
watch BOOLEAN NOT NULL DEFAULT TRUE,
poll_cycle TEXT NOT NULL DEFAULT 'daily' CHECK (poll_cycle IN ('daily', 'weekly', 'monthly', 'quarterly')),
watermark TEXT,
repeal_detected_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
+20
View File
@@ -0,0 +1,20 @@
-- 347_legal_meta_table.sql
-- 안전 자료실 A-1 (8/12) — 법령 문서 1건(=1버전 또는 1부속문서)당 1행. documents 1:0..1 위성, 최소형.
-- plan: safety-library-1 0-2. supersedes 체인 컬럼은 미포함 (개정 이벤트 10건 관찰 후 승격).
-- version_key: KR primary = MST / annex = 'MST|별표N' 합성 (같은 MST 별표 다건 UNIQUE 충돌 회피)
-- / interpretation = 소스 native id. dedup 키도 이 합성형 그대로 (R3-M4 silent skip 차단).
-- version_status 운영 계약 (B-1 PR② 일일 잡이 유일한 전이 지점, R2-B2·R3-M3):
-- 전 버전 pending 적재 → 잡이 KST 기준 시행일 도래분 current 승격 + 직전 current 를 superseded
-- + 구버전 청크 in_corpus=false 를 한 트랜잭션 처리. repeal 도 잡 경유.
-- 입법예고 등 신호류 문서는 legal_meta 없음 (legal_meta 존재 = 법령 본문).
CREATE TABLE IF NOT EXISTS legal_meta (
document_id BIGINT PRIMARY KEY REFERENCES documents(id) ON DELETE CASCADE,
family_id TEXT NOT NULL REFERENCES legal_acts(family_id),
law_doc_kind TEXT NOT NULL DEFAULT 'primary' CHECK (law_doc_kind IN ('primary', 'annex', 'interpretation')),
version_key TEXT NOT NULL,
promulgation_date DATE,
effective_date DATE,
version_status TEXT NOT NULL DEFAULT 'pending' CHECK (version_status IN ('pending', 'current', 'superseded', 'repealed')),
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
CONSTRAINT uq_legal_meta_version UNIQUE (family_id, law_doc_kind, version_key)
);
@@ -0,0 +1,4 @@
-- 348_documents_material_type_idx.sql
-- 안전 자료실 A-1 (9/12) — material_type partial index (128~131 facet 인덱스 선례).
CREATE INDEX IF NOT EXISTS idx_documents_material_type
ON documents (material_type) WHERE material_type IS NOT NULL;
@@ -0,0 +1,4 @@
-- 349_documents_jurisdiction_idx.sql
-- 안전 자료실 A-1 (10/12) — jurisdiction partial index.
CREATE INDEX IF NOT EXISTS idx_documents_jurisdiction
ON documents (jurisdiction) WHERE jurisdiction IS NOT NULL;
+6
View File
@@ -0,0 +1,6 @@
-- 350_legal_meta_family_idx.sql
-- 안전 자료실 A-1 (11/12) — point-in-time 조회 축.
-- 술어 = COALESCE(effective_date, promulgation_date) (KGS 류 시행일 미상 row 침묵 탈락 방지)
-- 이나 인덱스는 effective_date 단순형으로 시작 — COALESCE expression index 는 실측 후.
CREATE INDEX IF NOT EXISTS idx_legal_meta_family
ON legal_meta (family_id, effective_date DESC);
@@ -0,0 +1,9 @@
-- 351_documents_paper_doi_uq.sql
-- 안전 자료실 A-1 (12/12) — 논문 DOI dedup 구조 강제 (partial UNIQUE).
-- doi 보유 계약 (R3 — R2-B1): paper.doi 는 서지 Document 단일 보유.
-- OA 전문 PDF / 구매분 file Document 는 paper.doi 를 갖지 않고 paper.parent_doi 링크로 연결
-- → 인덱스 식이 NULL 이라 다중 행 허용, 2-Document 구조와 무충돌.
-- DOI 정규화(소문자·prefix 제거)는 단일 함수 경유 — 저장=조회 동일 함수 원칙 (B-3).
CREATE UNIQUE INDEX IF NOT EXISTS uq_documents_paper_doi
ON documents (lower(extract_meta #>> '{paper,doi}'))
WHERE material_type = 'paper' AND extract_meta #>> '{paper,doi}' IS NOT NULL;
@@ -0,0 +1,8 @@
-- 352_news_sources_material_type.sql
-- 안전 자료실 A-2 (1/4) — 소스 레지스트리에 자료유형 기본값.
-- plan: safety-library-1 A-2. 수집기 ingest 시점 deterministic 부여의 단일 진실 =
-- 레지스트리 행 (country 와 동일 패턴 — 코드 하드코딩/이름 매칭 회피).
-- NULL = 자료유형 비대상 (뉴스/철학 등). paper 소스는 country 가 있어도
-- documents.jurisdiction 은 NULL (국제 학술지 — 코드 레벨 규칙).
ALTER TABLE news_sources ADD COLUMN IF NOT EXISTS material_type TEXT
CHECK (material_type IN ('law', 'paper', 'book', 'incident', 'manual', 'standard', 'guide'));
@@ -0,0 +1,6 @@
-- 353_news_sources_license_scheme.sql
-- 안전 자료실 A-2 (2/4) — 소스별 라이선스 scheme (0-3 license 메타 deterministic 주입).
-- kogl(공공누리류) / ogl(UK) / public_domain(미 연방) / proprietary / unknown.
-- 미확정 소스는 보수적으로 unknown/proprietary + redistribute=false 에서 시작
-- (갱신은 근거 확보 시 완화 방향 — 보수적=빡빡 원칙).
ALTER TABLE news_sources ADD COLUMN IF NOT EXISTS license_scheme TEXT;
@@ -0,0 +1,4 @@
-- 354_news_sources_license_redistribute.sql
-- 안전 자료실 A-2 (3/4) — 재배포 가능 여부. P3 다이제스트/발행류의 구조 게이트 입력
-- (redistribute=false 소스 제외 — 사람 기억 의존 차단, 0-3).
ALTER TABLE news_sources ADD COLUMN IF NOT EXISTS license_redistribute BOOLEAN;
@@ -0,0 +1,45 @@
-- 355_news_sources_material_seed.sql
-- 안전 자료실 A-2 (4/4) — 기존 안전/공학 소스 12행 material_type + license 시드.
-- 매핑 근거 = plan safety-library-1 0-1 경계 확정 (2026-06-12 prod 레지스트리 실측 대조):
-- law=입법예고(신호) / incident=HSE·KOSHA사례·CSB·CCPS / guide=KOSHA GUIDE·TWI
-- / standard=NB·API 공지 / paper=JPVT·arXiv (jurisdiction 은 코드에서 NULL 강제).
-- 뉴스/철학 소스는 NULL 유지 (자료유형 비대상). 이름 키 = 시드 마이그레이션이 부여한 고정값.
UPDATE news_sources SET
material_type = CASE name
WHEN '고용노동부 입법행정예고' THEN 'law'
WHEN 'UK HSE Press' THEN 'incident'
WHEN 'KOSHA 재해사례' THEN 'incident'
WHEN 'US CSB 사고조사보고서' THEN 'incident'
WHEN 'CCPS Process Safety Beacon' THEN 'incident'
WHEN 'KOSHA GUIDE' THEN 'guide'
WHEN 'TWI Job Knowledge' THEN 'guide'
WHEN 'National Board 기술 아티클' THEN 'standard'
WHEN 'API 표준 공지' THEN 'standard'
WHEN 'ASME J. Pressure Vessel Technology' THEN 'paper'
WHEN 'arXiv physics.app-ph' THEN 'paper'
WHEN 'arXiv cond-mat.mtrl-sci' THEN 'paper'
END,
license_scheme = CASE name
WHEN '고용노동부 입법행정예고' THEN 'kogl'
WHEN 'KOSHA 재해사례' THEN 'kogl'
WHEN 'KOSHA GUIDE' THEN 'kogl'
WHEN 'UK HSE Press' THEN 'ogl'
WHEN 'US CSB 사고조사보고서' THEN 'public_domain'
WHEN 'TWI Job Knowledge' THEN 'proprietary'
WHEN 'National Board 기술 아티클' THEN 'proprietary'
WHEN 'API 표준 공지' THEN 'proprietary'
WHEN 'CCPS Process Safety Beacon' THEN 'proprietary'
WHEN 'ASME J. Pressure Vessel Technology' THEN 'proprietary'
WHEN 'arXiv physics.app-ph' THEN 'unknown'
WHEN 'arXiv cond-mat.mtrl-sci' THEN 'unknown'
END,
license_redistribute = CASE name
WHEN 'UK HSE Press' THEN TRUE
WHEN 'US CSB 사고조사보고서' THEN TRUE
ELSE FALSE
END
WHERE name IN ('고용노동부 입법행정예고', 'UK HSE Press', 'KOSHA 재해사례',
'US CSB 사고조사보고서', 'CCPS Process Safety Beacon', 'KOSHA GUIDE',
'TWI Job Knowledge', 'National Board 기술 아티클', 'API 표준 공지',
'ASME J. Pressure Vessel Technology', 'arXiv physics.app-ph',
'arXiv cond-mat.mtrl-sci');
+41
View File
@@ -0,0 +1,41 @@
-- 356_seed_legal_acts_kr.sql
-- 안전 자료실 B-1 PR① — legal_acts KR 시드 26행 (레거시 law_monitor 26개 superset).
-- plan: safety-library-1 B-1. watch=true 26개 전부 (R3-B1 ① — '우선순위'는 정렬일 뿐 제외 아님).
-- 법령ID/공포/시행 = 2026-06-13 lawSearch 라이브 실측 (tests/fixtures/statute_kr/seed_26laws.tsv).
-- ★ '유해ㆍ위험작업...' = 정식명에 가운뎃점(U+318D) — law_monitor 하드코딩(점 없음)은 exact match
-- 불일치로 이 법령을 영구 미매칭하던 잠복 누락이었음 (R8-m1 의 watchlist 판 실증).
-- parent 계열: 법률 → 시행령/시행규칙/위임 부령. VALUES 순서 = 부모 선행 (FK).
INSERT INTO legal_acts (family_id, jurisdiction, law_level, title, parent_family_id, native_id, source_api, watch, poll_cycle)
SELECT v.family_id, v.jurisdiction, v.law_level, v.title, v.parent_family_id, v.native_id, v.source_api, v.watch, v.poll_cycle
FROM (VALUES
-- 법률 (statute, 14)
('kr-law:001766', 'KR', 'statute', '산업안전보건법', NULL, '001766', 'law.go.kr', TRUE, 'daily'),
('kr-law:013993', 'KR', 'statute', '중대재해 처벌 등에 관한 법률', NULL, '013993', 'law.go.kr', TRUE, 'daily'),
('kr-law:001807', 'KR', 'statute', '건설기술 진흥법', NULL, '001807', 'law.go.kr', TRUE, 'daily'),
('kr-law:000237', 'KR', 'statute', '시설물의 안전 및 유지관리에 관한 특별법', NULL, '000237', 'law.go.kr', TRUE, 'daily'),
('kr-law:009502', 'KR', 'statute', '위험물안전관리법', NULL, '009502', 'law.go.kr', TRUE, 'daily'),
('kr-law:000162', 'KR', 'statute', '화학물질관리법', NULL, '000162', 'law.go.kr', TRUE, 'daily'),
('kr-law:011857', 'KR', 'statute', '화학물질의 등록 및 평가 등에 관한 법률', NULL, '011857', 'law.go.kr', TRUE, 'daily'),
('kr-law:009503', 'KR', 'statute', '소방시설 설치 및 관리에 관한 법률', NULL, '009503', 'law.go.kr', TRUE, 'daily'),
('kr-law:001854', 'KR', 'statute', '전기사업법', NULL, '001854', 'law.go.kr', TRUE, 'daily'),
('kr-law:013718', 'KR', 'statute', '전기안전관리법', NULL, '013718', 'law.go.kr', TRUE, 'daily'),
('kr-law:001850', 'KR', 'statute', '고압가스 안전관리법', NULL, '001850', 'law.go.kr', TRUE, 'daily'),
('kr-law:001849', 'KR', 'statute', '액화석유가스의 안전관리 및 사업법', NULL, '001849', 'law.go.kr', TRUE, 'daily'),
('kr-law:001872', 'KR', 'statute', '근로기준법', NULL, '001872', 'law.go.kr', TRUE, 'daily'),
('kr-law:002016', 'KR', 'statute', '환경영향평가법', NULL, '002016', 'law.go.kr', TRUE, 'daily'),
-- 대통령령 (decree, 7)
('kr-law:003786', 'KR', 'decree', '산업안전보건법 시행령', 'kr-law:001766', '003786', 'law.go.kr', TRUE, 'daily'),
('kr-law:014159', 'KR', 'decree', '중대재해 처벌 등에 관한 법률 시행령', 'kr-law:013993', '014159', 'law.go.kr', TRUE, 'daily'),
('kr-law:002111', 'KR', 'decree', '건설기술 진흥법 시행령', 'kr-law:001807', '002111', 'law.go.kr', TRUE, 'daily'),
('kr-law:009707', 'KR', 'decree', '위험물안전관리법 시행령', 'kr-law:009502', '009707', 'law.go.kr', TRUE, 'daily'),
('kr-law:004390', 'KR', 'decree', '화학물질관리법 시행령', 'kr-law:000162', '004390', 'law.go.kr', TRUE, 'daily'),
('kr-law:009694', 'KR', 'decree', '소방시설 설치 및 관리에 관한 법률 시행령', 'kr-law:009503', '009694', 'law.go.kr', TRUE, 'daily'),
('kr-law:002246', 'KR', 'decree', '고압가스 안전관리법 시행령', 'kr-law:001850', '002246', 'law.go.kr', TRUE, 'daily'),
-- 부령 (rule, 5)
('kr-law:007364', 'KR', 'rule', '산업안전보건법 시행규칙', 'kr-law:001766', '007364', 'law.go.kr', TRUE, 'daily'),
('kr-law:007363', 'KR', 'rule', '산업안전보건기준에 관한 규칙', 'kr-law:001766', '007363', 'law.go.kr', TRUE, 'daily'),
('kr-law:007844', 'KR', 'rule', '유해ㆍ위험작업의 취업 제한에 관한 규칙', 'kr-law:001766', '007844', 'law.go.kr', TRUE, 'daily'),
('kr-law:006175', 'KR', 'rule', '건설기술 진흥법 시행규칙', 'kr-law:001807', '006175', 'law.go.kr', TRUE, 'daily'),
('kr-law:009732', 'KR', 'rule', '위험물안전관리법 시행규칙', 'kr-law:009502', '009732', 'law.go.kr', TRUE, 'daily')
) AS v(family_id, jurisdiction, law_level, title, parent_family_id, native_id, source_api, watch, poll_cycle)
WHERE NOT EXISTS (SELECT 1 FROM legal_acts la WHERE la.family_id = v.family_id);
+221
View File
@@ -0,0 +1,221 @@
"""안전 자료실 A-3 백필 — 기존 코퍼스에 material_type/jurisdiction/published_date/license 소급.
plan: safety-library-1 A-3 (PKM plans/2026-06-12-safety-library-plan.html)
선례: backfill_category.py (one-off 멱등 스크립트 migration 아님, 152 단일 트랜잭션 제약 회피)
술어 (2026-06-13 prod 실측 교정 R2 blocker 반영):
1. extract_meta.source_id JOIN news_sources 레지스트리 material_type/country 전파
(KOSHA 사례 본문·CSB 페이지·HSE·MOEL·JPVT·arXiv·NB·TWI·API 공지 전부 커버.
paper jurisdiction NULL 강제 plan 0-1. KOSHA 본문의 kosha.kind='case' 가정은
실측 부정됨: kind 첨부/GUIDE 에만 존재 source_id JOIN 정본 술어)
2. kosha.kind='case_attachment' incident/KR
3. kosha.kind='guide' guide/KR (+ ofancYmd 'YYYY-MM-DD' 실측)
4. csb.kind='report_pdf' incident/US (source_id 없음 JOIN 비대상)
5. source_channel='law_monitor' law/KR (243. legal_meta 생략 MST 미보존,
버전 체인은 B-1 가동 시점부터. published_date = title '(YYYYMMDD)' 공포일 추출
extract_meta 빈값 실측, R3-m1 'NULL 허용' 보다 1 정규식이 저렴해 채움)
6. file_path LIKE '%KGS_Code%' law/KR (frontmatter = 'code' 실측 117/118,
'kgs_code' 0. 경로 술어가 단순·전수. license B-4 소관 미주입)
불변식:
- UPDATE material_type IS NULL 가드 (멱등 재실행 안전, A-2 신규 유입분 무접촉)
- material_type + jurisdiction 동일 statement (law CHECK chk_documents_law_jurisdiction 충족)
- published_date / license 각자 필드 부재 가드 (이미 있으면 무접촉)
- 업로드 Industrial_Safety 문서 = 대상 아님 (LLM 제안+승인 경로만 자동 전이 금지)
- 코퍼스(청크/임베딩) 무접촉 검색 지표 무변동이 정상
실행:
docker compose exec -T fastapi python /app/scripts/backfill_material_axis.py --dry-run
docker compose exec -T fastapi python /app/scripts/backfill_material_axis.py --apply
"""
import argparse
import asyncio
import os
import sys
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "app"))
# text() 미사용 — exec_driver_sql (정규식 콜론 함정)
from sqlalchemy.ext.asyncio import create_async_engine
# ─── 술어별 (라벨, 카운트 SQL, 적용 SQL) ───────────────────────────────────────
_KOSHA_LICENSE = ("kogl", "false", "한국산업안전보건공단(KOSHA)")
_CSB_LICENSE = ("public_domain", "true", "U.S. Chemical Safety Board")
_LAW_LICENSE = ("public_domain", "true", "국가법령정보센터")
def _license_obj(scheme: str, redistribute: str, attribution: str) -> str:
return (
f"jsonb_build_object('license', jsonb_build_object("
f"'scheme', '{scheme}', 'redistribute', {redistribute}::boolean, "
f"'attribution', '{attribution}'))"
)
STEPS: list[tuple[str, str]] = [
# 1) 레지스트리 전파 (source_id JOIN)
("1. src_join material/jurisdiction", """
UPDATE documents d SET
material_type = ns.material_type,
jurisdiction = CASE WHEN ns.material_type = 'paper' THEN NULL ELSE ns.country END
FROM news_sources ns
WHERE d.material_type IS NULL AND d.deleted_at IS NULL
AND d.extract_meta->>'source_id' ~ '^[0-9]+$'
AND ns.id = (d.extract_meta->>'source_id')::int
AND ns.material_type IS NOT NULL
"""),
# 2) KOSHA 첨부
("2. kosha 첨부 incident/KR", """
UPDATE documents SET material_type = 'incident', jurisdiction = 'KR'
WHERE material_type IS NULL AND deleted_at IS NULL
AND extract_meta#>>'{kosha,kind}' = 'case_attachment'
"""),
# 3) KOSHA GUIDE
("3. kosha GUIDE guide/KR", """
UPDATE documents SET material_type = 'guide', jurisdiction = 'KR'
WHERE material_type IS NULL AND deleted_at IS NULL
AND extract_meta#>>'{kosha,kind}' = 'guide'
"""),
# 4) CSB 보고서 PDF
("4. csb PDF incident/US", """
UPDATE documents SET material_type = 'incident', jurisdiction = 'US'
WHERE material_type IS NULL AND deleted_at IS NULL
AND extract_meta#>>'{csb,kind}' = 'report_pdf'
"""),
# 5) 레거시 law_monitor
("5. law_monitor law/KR", """
UPDATE documents SET material_type = 'law', jurisdiction = 'KR'
WHERE material_type IS NULL AND deleted_at IS NULL
AND source_channel = 'law_monitor'
"""),
# 6) KGS Code watch 폴더
("6. KGS law/KR", """
UPDATE documents SET material_type = 'law', jurisdiction = 'KR'
WHERE material_type IS NULL AND deleted_at IS NULL
AND file_path LIKE '%KGS_Code%'
"""),
# 7) published_date — crawl/news 공통 (extract_meta.published_at ISO)
("7. published_date (published_at)", """
UPDATE documents SET published_date = (extract_meta->>'published_at')::date
WHERE published_date IS NULL AND deleted_at IS NULL
AND extract_meta->>'published_at' ~ '^\\d{4}-\\d{2}-\\d{2}'
"""),
# 8) published_date — KOSHA GUIDE 공표일자 ('YYYY-MM-DD' 실측)
("8. published_date (GUIDE ofancYmd)", """
UPDATE documents SET published_date = (extract_meta#>>'{kosha,ofancYmd}')::date
WHERE published_date IS NULL AND deleted_at IS NULL
AND extract_meta#>>'{kosha,ofancYmd}' ~ '^\\d{4}-\\d{2}-\\d{2}$'
"""),
# 9) published_date — 레거시 law title 공포일 '(YYYYMMDD)'
("9. published_date (law title 공포일)", """
UPDATE documents
SET published_date = to_date(substring(title from '\\((20\\d{2}(?:0[1-9]|1[0-2])(?:0[1-9]|[12]\\d|3[01]))\\)'), 'YYYYMMDD')
WHERE published_date IS NULL AND deleted_at IS NULL
AND source_channel = 'law_monitor'
AND title ~ '\\((20\\d{2}(?:0[1-9]|1[0-2])(?:0[1-9]|[12]\\d|3[01]))\\)'
"""),
# 10) license — 레지스트리 전파 (scheme 있는 소스만)
("10. license (src_join)", """
UPDATE documents d SET
extract_meta = COALESCE(d.extract_meta, '{}'::jsonb)
|| jsonb_build_object('license', jsonb_build_object(
'scheme', ns.license_scheme,
'redistribute', COALESCE(ns.license_redistribute, false),
'attribution', ns.name))
FROM news_sources ns
WHERE d.deleted_at IS NULL AND NOT (COALESCE(d.extract_meta, '{}'::jsonb) ? 'license')
AND d.extract_meta->>'source_id' ~ '^[0-9]+$'
AND ns.id = (d.extract_meta->>'source_id')::int
AND ns.license_scheme IS NOT NULL
"""),
# 11) license — KOSHA 첨부/GUIDE (source_id 없음)
("11. license (kosha kinds)", f"""
UPDATE documents SET
extract_meta = COALESCE(extract_meta, '{{}}'::jsonb) || {_license_obj(*_KOSHA_LICENSE)}
WHERE deleted_at IS NULL AND NOT (COALESCE(extract_meta, '{{}}'::jsonb) ? 'license')
AND extract_meta#>>'{{kosha,kind}}' IN ('case_attachment', 'guide')
"""),
# 12) license — CSB PDF
("12. license (csb PDF)", f"""
UPDATE documents SET
extract_meta = COALESCE(extract_meta, '{{}}'::jsonb) || {_license_obj(*_CSB_LICENSE)}
WHERE deleted_at IS NULL AND NOT (COALESCE(extract_meta, '{{}}'::jsonb) ? 'license')
AND extract_meta#>>'{{csb,kind}}' = 'report_pdf'
"""),
# 13) license — 레거시 법령 (저작권법 제7조 비보호)
("13. license (law_monitor)", f"""
UPDATE documents SET
extract_meta = COALESCE(extract_meta, '{{}}'::jsonb) || {_license_obj(*_LAW_LICENSE)}
WHERE deleted_at IS NULL AND NOT (COALESCE(extract_meta, '{{}}'::jsonb) ? 'license')
AND source_channel = 'law_monitor'
"""),
]
VERIFY_SQL = [
("축 전수표 (material_type x jurisdiction)", """
SELECT material_type, jurisdiction, count(*) AS docs,
count(published_date) AS with_date,
count(*) FILTER (WHERE extract_meta ? 'license') AS with_license
FROM documents WHERE material_type IS NOT NULL AND deleted_at IS NULL
GROUP BY 1, 2 ORDER BY 1, 2
"""),
("law & jurisdiction NULL (0 이어야 함 — hard)", """
SELECT count(*) FROM documents
WHERE material_type = 'law' AND jurisdiction IS NULL AND deleted_at IS NULL
"""),
("잔여 미분류 안전 후보 (kosha/csb 메타 보유인데 NULL — 0 이어야 함)", """
SELECT count(*) FROM documents
WHERE material_type IS NULL AND deleted_at IS NULL
AND (extract_meta ? 'kosha' OR extract_meta ? 'csb')
"""),
]
async def main() -> None:
parser = argparse.ArgumentParser()
mode = parser.add_mutually_exclusive_group(required=True)
mode.add_argument("--dry-run", action="store_true",
help="전 UPDATE 를 트랜잭션 안에서 실행해 정확한 rowcount + 검증표를 보여주고 ROLLBACK (변경 0)")
mode.add_argument("--apply", action="store_true", help="백필 실행 (단일 트랜잭션 커밋)")
args = parser.parse_args()
db_url = os.getenv(
"DATABASE_URL", "postgresql+asyncpg://pkm:pkm@localhost:5432/pkm"
)
engine = create_async_engine(db_url)
tag = "apply" if args.apply else "dry-run"
async with engine.connect() as conn:
trans = await conn.begin()
try:
for label, sql in STEPS:
# text() 는 정규식의 '(?:' 콜론을 bind param 으로 오인 (migration 러너와
# 동일 함정) → driver 직결 실행
result = await conn.exec_driver_sql(sql)
print(f"[{tag}] {label}: {result.rowcount}")
print("\n─── 검증 (트랜잭션 내 미리보기) ───")
for label, sql in VERIFY_SQL:
result = await conn.exec_driver_sql(sql)
rows = result.fetchall()
print(f"\n{label}:")
for row in rows:
print(" ", tuple(row))
if args.apply:
await trans.commit()
print("\n[apply] 커밋 완료")
else:
await trans.rollback()
print("\n[dry-run] 전체 롤백 — 변경 0")
except Exception:
await trans.rollback()
raise
await engine.dispose()
if __name__ == "__main__":
asyncio.run(main())
+138
View File
@@ -0,0 +1,138 @@
"""B-1 PR③ — 법령 버전 체인 검증 3술어 (plan safety-library-1).
read-only 진단 E-1 관찰의 법령 게이트 도구로도 재사용 (반복 실행 안전).
검증 3술어 (R7-M1, B-1 단일 정본):
존재성 watch family 각각 primary 시리즈 current 정확 1(0건도 위반)
+ annex 시리즈당 current 1
노출 유일성 primary current 보유 family당 primary 노출(체인+레거시 매핑 합산) 정확 1
(모집단 = primary current 보유 family 한정 R8-M2)
고아 그물 law_monitor in_corpus=true 레거시 :
(a) current 보유 family 매핑되는데 flip (flip 누락) = 0
(b) 어느 watch family 에도 매핑 되는 (제명 개정 매핑 구멍) = 0
repealed family·primary current 미보유 family 레거시 보존은 위반 아님
repealed family 기대값 0 으로 면제.
실행:
docker compose exec -T fastapi python /app/scripts/verify_statute_chain.py
종료코드: 0 = 전건 PASS, 1 = 위반 (CI/관찰 게이트 )
"""
import asyncio
import os
import sys
# 컨테이너: /app/scripts → /app (workers/core/models 패키지 루트). 로컬: repo/scripts → repo/app
_here = os.path.dirname(os.path.abspath(__file__))
for _cand in (os.path.join(_here, ".."), os.path.join(_here, "..", "app")):
if os.path.isdir(os.path.join(_cand, "workers")):
sys.path.insert(0, os.path.abspath(_cand))
break
from collections import defaultdict
from sqlalchemy import select, text
from sqlalchemy.ext.asyncio import create_async_engine
from workers.statute_collector import legacy_law_name, normalize_law_name, series_suffix
async def main() -> int:
db_url = os.getenv("DATABASE_URL", "postgresql+asyncpg://pkm:pkm@localhost:5432/pkm")
engine = create_async_engine(db_url)
violations: list[str] = []
async with engine.connect() as conn:
# ── 로드 ──
acts = (await conn.execute(text(
"SELECT family_id, title, repeal_detected_at IS NOT NULL AS repealed "
"FROM legal_acts WHERE watch"))).all()
metas = (await conn.execute(text(
"SELECT family_id, law_doc_kind, version_key, version_status FROM legal_meta"))).all()
act_title = {a.family_id: a.title for a in acts}
repealed = {a.family_id for a in acts if a.repealed}
active = [a for a in acts if not a.repealed]
# family → primary current 수 / annex 시리즈별 current 수
prim_current = defaultdict(int)
annex_series_current = defaultdict(int)
for m in metas:
if m.version_status != "current":
continue
if m.law_doc_kind == "primary":
prim_current[m.family_id] += 1
else:
annex_series_current[(m.family_id, series_suffix(m.version_key))] += 1
# ── ① 존재성 ──
for a in active:
n = prim_current[a.family_id]
if n != 1:
violations.append(f"{a.family_id} ({a.title}): primary current {n}건 (정확 1 기대)")
for (fid, suf), n in annex_series_current.items():
if fid not in repealed and n > 1:
violations.append(f"{fid} annex 시리즈 {suf}: current {n}건 (≤1 기대)")
# ── ③ 고아 그물 (정규화 동등 매핑) ──
# watch family 정규화명 → family_id (current 보유 여부 동반)
norm_to_fid = {}
for a in active:
norm_to_fid[normalize_law_name(a.title)] = a.family_id
legacy = (await conn.execute(text(
"SELECT d.id, d.title, "
" EXISTS(SELECT 1 FROM document_chunks c WHERE c.doc_id=d.id AND c.in_corpus) AS exposed "
"FROM documents d WHERE d.source_channel='law_monitor' AND d.deleted_at IS NULL"))).all()
orphan_flip_miss = 0
orphan_unmapped = 0
unmapped_names = set()
for row in legacy:
if not row.exposed:
continue # in_corpus=false = 정상 (스윕됨 or 청크 없음)
name = legacy_law_name(row.title or "")
norm = normalize_law_name(name) if name else None
fid = norm_to_fid.get(norm) if norm else None
if fid is None:
orphan_unmapped += 1
if name:
unmapped_names.add(name)
elif prim_current.get(fid, 0) >= 1:
# current 보유 family 인데 레거시가 노출 중 = flip 누락
orphan_flip_miss += 1
if orphan_flip_miss:
violations.append(f"③(a) flip 누락: current 보유 family 의 노출 레거시 {orphan_flip_miss}")
if orphan_unmapped:
violations.append(
f"③(b) 무매핑 노출 레거시 {orphan_unmapped}건 — 매핑 구멍(매핑 보강 신호): "
+ ", ".join(sorted(unmapped_names))[:200])
# ── ② 노출 유일성 (primary current 보유 family 한정) ──
# 노출 primary = 체인 primary current(=1) + 레거시 매핑 노출분.
# ③(a)=0 이면 레거시 노출분 0 → 체인 1건만 = 정확 1. 별도 위반 추출은 ③(a)에 포함됨.
# (annex 노출 비동기 일반화는 may — Phase 1 미적용)
# ── 상태 요약 출력 ──
print("=== 법령 체인 검증 (B-1 PR③ 3술어) ===")
print(f"watch family: {len(acts)} (active {len(active)}, repealed {len(repealed)})")
print(f"primary current 보유 family: {sum(1 for a in active if prim_current[a.family_id]==1)}/{len(active)}")
print(f"annex current 시리즈: {len(annex_series_current)}")
exposed_legacy = sum(1 for r in legacy if r.exposed)
print(f"레거시 law_monitor: {len(legacy)}건 (in_corpus 노출 {exposed_legacy}건)")
print()
await engine.dispose()
if violations:
print(f"[FAIL] 위반 {len(violations)}건:")
for v in violations:
print(" -", v)
return 1
print("[PASS] 3술어 전건 통과 (존재성·노출 유일성·고아 그물)")
return 0
if __name__ == "__main__":
sys.exit(asyncio.run(main()))
+152
View File
@@ -0,0 +1,152 @@
"""POST /api/eid/chat mode=deep — ReAct 자동검색 SSE 변환 (ds-eid-ask-absorb P1).
DB·LLM 0: get_session/get_current_user dependency override, probe·agentic_ask_loop·
get_backend monkeypatch. 실제 검색·27B 호출 없음.
검증: 검색성phase:searching+content+eid_sources+DONE / probe 실패503 /
mid-stream BackendUnavailablein-stream error envelope / 대화성sources .
"""
from __future__ import annotations
import json
import sys
import types
from pathlib import Path
import pytest
import pytest_asyncio
from fastapi import FastAPI
from httpx import ASGITransport, AsyncClient
sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "app"))
import api.eid_chat as eid_chat # noqa: E402
from api.eid_chat import router as eid_chat_router # noqa: E402
from core.auth import get_current_user # noqa: E402
from core.database import get_session # noqa: E402
from services.llm.backends import BackendUnavailable # noqa: E402
from services.search.react_loop import ReactResult # noqa: E402
_DEEP = {"mode": "deep", "messages": [{"role": "user", "content": "콜드박스 위험성평가 찾아줘"}]}
async def _async_true() -> bool:
return True
async def _async_false() -> bool:
return False
def _build_app() -> FastAPI:
app = FastAPI()
app.include_router(eid_chat_router, prefix="/api/eid")
app.dependency_overrides[get_current_user] = lambda: types.SimpleNamespace(
id=1, username="test-user"
)
async def _fake_session():
yield None # deep 경로는 session 을 agentic_ask_loop 에 넘기기만(여기선 monkeypatch)
app.dependency_overrides[get_session] = _fake_session
return app
def _data_objs(raw: bytes) -> list[dict]:
out: list[dict] = []
for line in raw.split(b"\n"):
if line.startswith(b"data: ") and line[len(b"data: "):].strip() != b"[DONE]":
try:
out.append(json.loads(line[len(b"data: "):]))
except Exception:
pass
return out
@pytest_asyncio.fixture
async def client():
async with AsyncClient(
transport=ASGITransport(app=_build_app()), base_url="http://test"
) as ac:
yield ac
@pytest.fixture(autouse=True)
def _rules_present(monkeypatch):
# D-6 fail-closed 가드 통과 (substrate degraded 아님)
monkeypatch.setattr(eid_chat.eid_compose, "rules_present", lambda: True)
@pytest.mark.asyncio
async def test_deep_search_sse_shape(client, monkeypatch):
"""검색성 질문 → phase:searching + final content + eid_sources + DONE 순서."""
monkeypatch.setattr(eid_chat, "_probe_router_reachable", _async_true)
monkeypatch.setattr(eid_chat, "get_backend", lambda name: object())
async def _fake_loop(session, query, *, backend, **kw):
return ReactResult(
final_answer="콜드박스 위험성평가는 TK-RA-2026-OT1-01 입니다.",
iterations=1,
partial=False,
sources=[{"id": 1, "doc_id": 10, "title": "OT1 콜드박스 위험성평가", "score": 0.91}],
)
monkeypatch.setattr(eid_chat, "agentic_ask_loop", _fake_loop)
r = await client.post("/api/eid/chat", json=_DEEP)
assert r.status_code == 200
objs = _data_objs(r.content)
assert "searching" in [o.get("phase") for o in objs if "phase" in o]
content = "".join(
o["choices"][0]["delta"]["content"] for o in objs if "choices" in o
)
assert "OT1-01" in content
srcs = [o["eid_sources"] for o in objs if "eid_sources" in o]
assert srcs and srcs[0][0]["title"] == "OT1 콜드박스 위험성평가"
assert b"data: [DONE]" in r.content
@pytest.mark.asyncio
async def test_deep_conversational_no_sources(client, monkeypatch):
"""대화성(검색 불요) → ReAct early-exit, sources 빈 배열."""
monkeypatch.setattr(eid_chat, "_probe_router_reachable", _async_true)
monkeypatch.setattr(eid_chat, "get_backend", lambda name: object())
async def _chat_loop(session, query, *, backend, **kw):
return ReactResult(final_answer="안녕하세요, 이드입니다.", iterations=1, partial=False, sources=[])
monkeypatch.setattr(eid_chat, "agentic_ask_loop", _chat_loop)
r = await client.post("/api/eid/chat", json=_DEEP)
assert r.status_code == 200
objs = _data_objs(r.content)
srcs = [o["eid_sources"] for o in objs if "eid_sources" in o]
assert srcs and srcs[0] == [] # 검색 안 함 = 근거 카드 안 뜸
@pytest.mark.asyncio
async def test_deep_probe_fail_503(client, monkeypatch):
"""probe 실패(router 미도달) → 첫 바이트 전 503 router_unreachable."""
monkeypatch.setattr(eid_chat, "_probe_router_reachable", _async_false)
r = await client.post("/api/eid/chat", json=_DEEP)
assert r.status_code == 503
assert r.json()["error_reason"] == "router_unreachable"
@pytest.mark.asyncio
async def test_deep_midstream_error_envelope(client, monkeypatch):
"""검색 중 BackendUnavailable(AC 분리 등) → in-stream error envelope + DONE."""
monkeypatch.setattr(eid_chat, "_probe_router_reachable", _async_true)
monkeypatch.setattr(eid_chat, "get_backend", lambda name: object())
async def _fail_loop(session, query, *, backend, **kw):
raise BackendUnavailable("qwen-macbook", "macbook_unavailable")
monkeypatch.setattr(eid_chat, "agentic_ask_loop", _fail_loop)
r = await client.post("/api/eid/chat", json=_DEEP)
assert r.status_code == 200 # 스트림 이미 시작(probe 통과) → 200 + in-stream error
objs = _data_objs(r.content)
errs = [o for o in objs if o.get("phase") == "error"]
assert errs and errs[0]["error_reason"] == "macbook_unavailable"
assert b"data: [DONE]" in r.content
+6 -2
View File
@@ -131,6 +131,8 @@ async def test_503_substrate_degraded(client, monkeypatch):
@pytest.mark.asyncio
async def test_503_backend_unavailable_prestream(client, monkeypatch):
# call_stream 회귀(prestream 503)는 daily 로 검증 — deep 은 이제 ReAct 별 경로
# (probe·agentic_ask_loop), deep 의 503/midstream 은 test_eid_chat_deep.py 가 커버.
async def fake_call_stream(self, mode, messages, system):
raise BackendUnavailable("qwen-macbook", "macbook_unavailable")
yield b"" # pragma: no cover — async generator 형태 유지용
@@ -138,7 +140,7 @@ async def test_503_backend_unavailable_prestream(client, monkeypatch):
monkeypatch.setattr(EidAIClient, "call_stream", fake_call_stream)
r = await client.post(
"/api/eid/chat",
json={"mode": "deep", "messages": [{"role": "user", "content": "x"}]},
json={"mode": "daily", "messages": [{"role": "user", "content": "x"}]},
)
assert r.status_code == 503
js = r.json()
@@ -192,9 +194,11 @@ async def test_200_midstream_abort_quiet(client, monkeypatch):
raise BackendUnavailable("qwen-macbook", "stream_deadline_exceeded")
monkeypatch.setattr(EidAIClient, "call_stream", fake_call_stream)
# call_stream midstream 회귀는 daily 로 — deep midstream 은 in-stream error envelope
# 경로(test_eid_chat_deep.test_deep_midstream_error_envelope)로 분리됨.
r = await client.post(
"/api/eid/chat",
json={"mode": "deep", "messages": [{"role": "user", "content": "x"}]},
json={"mode": "daily", "messages": [{"role": "user", "content": "x"}]},
)
assert r.status_code == 200
assert r.content == b'data: {"x": 1}\n\n'
+5 -5
View File
@@ -104,7 +104,7 @@ async def test_anthropic_router_url_blocked(monkeypatch):
@pytest.mark.asyncio
async def test_deep_mode_alias_and_sse_line_rewrite(monkeypatch):
"""deep → qwen-macbook alias, system 은 messages[0] 단일 주입, 라인 단위 정화 중계."""
"""deep → mac-mini-default alias (맥북 백지화 2026-06-11), system 은 messages[0] 단일 주입, 라인 단위 정화 중계."""
seen: dict = {}
def handler(request: httpx.Request) -> httpx.Response:
@@ -139,7 +139,7 @@ async def test_deep_mode_alias_and_sse_line_rewrite(monkeypatch):
]
assert seen["url"].endswith("/v1/chat/completions")
body = seen["json"]
assert body["model"] == "qwen-macbook"
assert body["model"] == "mac-mini-default"
assert body["stream"] is True
assert body["max_tokens"] == 2048
assert body["temperature"] == 0.4
@@ -202,7 +202,7 @@ async def test_prestream_503_maps_reason(monkeypatch):
with pytest.raises(BackendUnavailable) as ei:
await anext(stream)
assert ei.value.reason == "macbook_unavailable"
assert ei.value.backend_name == "qwen-macbook"
assert ei.value.backend_name == "mac-mini-default"
finally:
await c.close()
@@ -253,7 +253,7 @@ async def test_prestream_400_raises_valueerror_failloud(monkeypatch):
c = EidAIClient()
try:
stream = c.call_stream("deep", _MSG, "sys")
with pytest.raises(ValueError, match="router rejected alias='qwen-macbook'"):
with pytest.raises(ValueError, match="router rejected alias='mac-mini-default'"):
await anext(stream)
finally:
await c.close()
@@ -290,7 +290,7 @@ async def test_stream_deadline_exceeded(monkeypatch):
async for _ in stream:
pass
assert ei.value.reason == "stream_deadline_exceeded"
assert ei.value.backend_name == "qwen-macbook"
assert ei.value.backend_name == "mac-mini-default"
finally:
await c.close()
+112
View File
@@ -0,0 +1,112 @@
"""GET /api/eid/status endpoint 테스트 — inline ASGI app (DB 의존 0).
실행 환경: fastapi + httpx 필요 test_eid_chat_endpoint.py 동일 idiom.
DB 0 / LLM 0: get_current_user dependency_overrides 대체, gate 점유는
llm_gate.gate_status monkeypatch (eid_chat 모듈 attribute 호출하므로 유효).
무인증 케이스는 실제 auth 경로지만 decode 단계에서 거부돼 DB 접근 반환.
"""
from __future__ import annotations
import sys
import types
from pathlib import Path
import pytest
import pytest_asyncio
from fastapi import FastAPI
from httpx import ASGITransport, AsyncClient
sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "app"))
from api.eid_chat import router as eid_chat_router # noqa: E402
from core.auth import get_current_user # noqa: E402
from services.search import llm_gate # noqa: E402
def _build_app(*, override_auth: bool = True) -> FastAPI:
"""main.py 등록 방식과 동일 prefix(/api/eid)로 라우터만 올린 inline app."""
app = FastAPI()
app.include_router(eid_chat_router, prefix="/api/eid")
if override_auth:
app.dependency_overrides[get_current_user] = lambda: types.SimpleNamespace(
id=1, username="test-user"
)
return app
@pytest_asyncio.fixture
async def client():
async with AsyncClient(
transport=ASGITransport(app=_build_app()), base_url="http://test"
) as ac:
yield ac
# ── 401 무인증 ────────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_unauthenticated_rejected():
async with AsyncClient(
transport=ASGITransport(app=_build_app(override_auth=False)),
base_url="http://test",
) as ac:
# 헤더 자체 부재 — HTTPBearer 단계 거부 (fastapi 기본 403, 버전별 401 허용)
r = await ac.get("/api/eid/status")
assert r.status_code in (401, 403)
# 위조 토큰 — decode_token 실패 → 401 (DB 접근 전 거부)
r2 = await ac.get(
"/api/eid/status", headers={"Authorization": "Bearer bogus-token"}
)
assert r2.status_code == 401
# ── 200 shape ────────────────────────────────────────────────────────────────
@pytest.mark.asyncio
async def test_200_shape(client, monkeypatch):
"""응답 shape — daily 키 아래 busy/inflight/waiters 3필드, 타입 고정."""
monkeypatch.setattr(
llm_gate, "gate_status", lambda: {"inflight": False, "waiters": 0}
)
r = await client.get("/api/eid/status")
assert r.status_code == 200, r.text
js = r.json()
assert set(js.keys()) == {"daily"}
assert set(js["daily"].keys()) == {"busy", "inflight", "waiters"}
assert isinstance(js["daily"]["busy"], bool)
assert isinstance(js["daily"]["inflight"], bool)
assert isinstance(js["daily"]["waiters"], int)
# ── busy 판정 — gate_status monkeypatch ──────────────────────────────────────
@pytest.mark.asyncio
@pytest.mark.parametrize(
"snap, expected",
[
# 유휴 — busy=false (근사: 외부 소비자 점유는 미포착)
(
{"inflight": False, "waiters": 0},
{"busy": False, "inflight": False, "waiters": 0},
),
# inflight 만 — busy=true (확실)
(
{"inflight": True, "waiters": 0},
{"busy": True, "inflight": True, "waiters": 0},
),
# waiters 만 — busy=true (inflight or waiters>0 의 or 분기)
(
{"inflight": False, "waiters": 3},
{"busy": True, "inflight": False, "waiters": 3},
),
],
)
async def test_busy_from_gate_status(client, monkeypatch, snap, expected):
monkeypatch.setattr(llm_gate, "gate_status", lambda: dict(snap))
r = await client.get("/api/eid/status")
assert r.status_code == 200, r.text
assert r.json() == {"daily": expected}
+1
View File
@@ -0,0 +1 @@
{"header": {"resultCode": "00", "resultMsg": "NORMAL_CODE"}, "body": {"pageNo": 1, "totalCount": 2845, "numOfRows": 3, "items": {"item": [{"contents": "<p><img src='https://portal.kosha.or.kr/api/compn24/auth/stdtboard/getImage.do?bbsId=B2025021314108&pstNo=20260611111536KIZXJ8&bbsAtcflNo=E0802000030001' style='width: 931px;' data-filename='6-9 부산 사상구.jpg' data-tboard-img-cvrt='Y'></p><p><br></p><p>2026. 6. 9. (화), 12:22경부산 사상구 소재 아파트에서</p><p><br></p><p>재해자가 2명이 실외기 설치 작업 중</p><p><br></p><p>베란다 난간이 파손되며 바닥으로 떨어짐</p><p><br></p><p>(사망 2명)</p><p>※ 위 내용은 신고 및 현재 파악된 내용으로 조사결과에 따라 변경될 수 있습니다.</p><div><br></div>", "keyword": "[6/9, 부산 사상구] 실외기 설치 작업 중 베란다 난간이 파손되어 떨어짐", "arno": "20260611111536KIZXJ8"}, {"contents": "<p><br><img src='https://portal.kosha.or.kr/api/compn24/auth/stdtboard/getImage.do?bbsId=B2025021314108&pstNo=20260611111355OZSS9T&bbsAtcflNo=E0802000030001' style='width: 931px;' data-filename='서 울관악구.jpg' data-tboard-img-cvrt='Y'></p><p><br></p><p>2026. 6. 9. (화), 17:26경서울 관악구 철도 공사 현장에서</p><p><br></p><p>재해자가 수직형 케이블 거치대 설치 준비 작업 중</p><p><br></p><p>개구부로 떨어짐(사망 1명)</p><p><br></p><p>※ 위 내용은 신고 및 현재 파악된 내용으로 조사결과에 따라 변경될 수 있습니다.</p><div><br></div><p></p>", "keyword": "[6/9, 서울 관악구] 수직형 케이블 거치대 설치 준비 중 개구부로 떨어짐", "arno": "20260611111355OZSS9T"}, {"contents": "<p><img src='https://portal.kosha.or.kr/api/compn24/auth/stdtboard/getImage.do?bbsId=B2025021314108&pstNo=202606111110595AR9QY&bbsAtcflNo=E0802000030001' style='width: 931px;' data-filename='5-14 전남 광양시.jpg' data-tboard-img-cvrt='Y'><br></p><p><br></p><p>2026. 5. 14. (목), 16:51경전남 광양시 소재 화학물질 제조사업장에서</p><p><br></p><p>재해자가 정제설비 내부에서 플랜지 해체 작업 중</p><p><br></p><p>고온 응축수가 쏟아져 화상을 입음(사망 1명)※ 위 내용은 신고 및 현재 파악된 내용으로 조사결과에 따라 변경될 수 있습니다.<br></p>", "keyword": "[5/14, 전남 광양시] 플랜지 해체 작업 중 고온 응축수가 쏟아져 화상", "arno": "202606111110595AR9QY"}]}}}
+26
View File
@@ -0,0 +1,26 @@
{
"_meta": {
"plan": "embedding-phase2a-1 G-1",
"measured_at": "2026-06-12",
"serving": "Ollama 0.20.0 (GPU container `ollama`), endpoint = POST /api/embed (단일 고정 — legacy /api/embeddings 사용 금지)",
"invariant": "저장=조회 동일 모델+버전, 프롬프트는 역할별 고정 (문서=plain / 쿼리=instruct prefix)"
},
"instruct_prefix_pinned": "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: ",
"models": {
"qwen3-embedding:0.6b": {
"digest": "ac6da0dfba84", "size": "639MB", "dim": 1024, "l2_normalized": true
},
"qwen3-embedding:4b": {
"digest": "df5bd2e3c74c", "size": "2.5GB(Q4)", "dim": 2560, "l2_normalized": true,
"mrl_dimensions_option": {"supported": true, "dimensions=1024": {"dim": 1024, "l2_norm": 1.0, "note": "Ollama 가 truncate+재정규화까지 수행 — 쿼리측 MRL 은 dimensions 옵션으로 처리"}}
}
},
"asymmetric_prefix_effect_0.6b": {
"doc": "압력용기의 수압시험은 설계압력의 1.3배로 실시하며, 시험 중 용접부 누설 여부를 육안으로 확인한다.",
"query": "압력용기 수압시험 기준 압력은?",
"cos_doc_vs_query_plain": 0.7446,
"cos_doc_vs_query_instruct": 0.7606,
"cos_plain_vs_instruct_query": 0.882,
"verdict": "prefix 가 쿼리 임베딩을 실질 변화시키고(0.882) 관련쌍 유사도를 올림(+0.016) — 비대칭 사용 필수"
}
}
+46
View File
@@ -0,0 +1,46 @@
<?xml version="1.0" encoding="utf-8"?>
<법령 법령키="0017662026021921374">
<기본정보>
<법령ID>001766</법령ID>
<공포일자>20260219</공포일자>
<공포번호>21374</공포번호>
<언어>한글</언어>
<법종구분 법종구분코드="A0002">법률</법종구분>
<법령명_한글><![CDATA[산업안전보건법]]></법령명_한글>
<법령명_한자><![CDATA[산업안전보건법]]></법령명_한자>
<제명변경여부>N</제명변경여부>
<한글법령여부>Y</한글법령여부>
<편장절관>40040000</편장절관>
<소관부처 소관부처코드="1492000">고용노동부</소관부처>
<전화번호>044-202-8810, 8813, 8815, 8997</전화번호>
<시행일자>20260601</시행일자>
<제개정구분>일부개정</제개정구분>
<조문별시행일자>20260601</조문별시행일자>
<조문시행일자문자열>20260801:제10조의2, 제23조, 제175조제4항제1호의2</조문시행일자문자열>
<별표편집여부>N</별표편집여부>
<공포법령여부>Y</공포법령여부>
<시행일기준편집여부>Y</시행일기준편집여부>
</기본정보>
<조문>
<조문단위 조문키="0001000">
<조문번호>1</조문번호>
<조문여부>전문</조문여부>
<조문시행일자>20260601</조문시행일자>
<조문이동이전></조문이동이전>
<조문이동이후></조문이동이후>
<조문변경여부>N</조문변경여부>
</조문단위>
<조문단위 조문키="0001001">
<조문번호>1</조문번호>
<조문여부>조문</조문여부>
<조문제목><![CDATA[목적]]></조문제목>
<조문시행일자>20260601</조문시행일자>
<조문이동이전></조문이동이전>
<조문이동이후></조문이동이후>
<조문변경여부>N</조문변경여부>
<조문내용>
<![CDATA[제1조(목적) 이 법은 산업 안전 및 보건에 관한 기준을 확립하고 그 책임의 소재를 명확하게 하여 산업재해를 예방하고 쾌적한 작업환경을 조성함으로써 노무를 제공하는 사람의 안전 및 보건을 유지ㆍ증진함을 목적으로 한다. <개정 2020.5.26>]]>
</조문내용>
</조문단위>
</조문>
</법령>
+2
View File
@@ -0,0 +1,2 @@
<?xml version="1.0" encoding="UTF-8"?><LawSearch><target>law</target><키워드>산업안전보건기준에 관한 규칙</키워드><section>lawNm</section><totalCnt>1</totalCnt><page>1</page><numOfRows>1</numOfRows><resultCode>00</resultCode><resultMsg>success</resultMsg><law id="1"><법령일련번호>273603</법령일련번호><현행연혁코드>현행</현행연혁코드><법령명한글><![CDATA[산업안전보건기준에 관한 규칙]]></법령명한글><법령약칭명><![CDATA[안전보건규칙]]></법령약칭명><법령ID>007363</법령ID><공포일자>20250901</공포일자><공포번호>00450</공포번호><제개정구분명>일부개정</제개정구분명><소관부처코드>1492000</소관부처코드><소관부처명>고용노동부</소관부처명><법령구분명>고용노동부령</법령구분명><공동부령정보></공동부령정보><시행일자>20260302</시행일자><자법타법여부></자법타법여부><법령상세링크>/DRF/lawService.do?OC=__OC_REDACTED__&amp;target=law&amp;MST=273603&amp;type=HTML&amp;mobileYn=&amp;efYd=20260302</법령상세링크></law></LawSearch>
+2
View File
@@ -0,0 +1,2 @@
<?xml version="1.0" encoding="UTF-8"?><LawSearch><target>law</target><키워드>산업안전보건법</키워드><section>lawNm</section><totalCnt>3</totalCnt><page>1</page><numOfRows>3</numOfRows><resultCode>00</resultCode><resultMsg>success</resultMsg><law id="1"><법령일련번호>283449</법령일련번호><현행연혁코드>현행</현행연혁코드><법령명한글><![CDATA[산업안전보건법]]></법령명한글><법령약칭명><![CDATA[]]></법령약칭명><법령ID>001766</법령ID><공포일자>20260219</공포일자><공포번호>21374</공포번호><제개정구분명>일부개정</제개정구분명><소관부처코드>1492000</소관부처코드><소관부처명>고용노동부</소관부처명><법령구분명>법률</법령구분명><공동부령정보></공동부령정보><시행일자>20260601</시행일자><자법타법여부></자법타법여부><법령상세링크>/DRF/lawService.do?OC=__OC_REDACTED__&amp;target=law&amp;MST=283449&amp;type=HTML&amp;mobileYn=&amp;efYd=20260601</법령상세링크></law><law id="2"><법령일련번호>284771</법령일련번호><현행연혁코드>현행</현행연혁코드><법령명한글><![CDATA[산업안전보건법 시행령]]></법령명한글><법령약칭명><![CDATA[]]></법령약칭명><법령ID>003786</법령ID><공포일자>20260324</공포일자><공포번호>36220</공포번호><제개정구분명>타법개정</제개정구분명><소관부처코드>1492000</소관부처코드><소관부처명>고용노동부</소관부처명><법령구분명>대통령령</법령구분명><공동부령정보></공동부령정보><시행일자>20260324</시행일자><자법타법여부></자법타법여부><법령상세링크>/DRF/lawService.do?OC=__OC_REDACTED__&amp;target=law&amp;MST=284771&amp;type=HTML&amp;mobileYn=&amp;efYd=20260324</법령상세링크></law><law id="3"><법령일련번호>286657</법령일련번호><현행연혁코드>현행</현행연혁코드><법령명한글><![CDATA[산업안전보건법 시행규칙]]></법령명한글><법령약칭명><![CDATA[]]></법령약칭명><법령ID>007364</법령ID><공포일자>20260529</공포일자><공포번호>00470</공포번호><제개정구분명>일부개정</제개정구분명><소관부처코드>1492000</소관부처코드><소관부처명>고용노동부</소관부처명><법령구분명>고용노동부령</법령구분명><공동부령정보></공동부령정보><시행일자>20260601</시행일자><자법타법여부></자법타법여부><법령상세링크>/DRF/lawService.do?OC=__OC_REDACTED__&amp;target=law&amp;MST=286657&amp;type=HTML&amp;mobileYn=&amp;efYd=20260601</법령상세링크></law></LawSearch>
Binary file not shown.
Binary file not shown.
+26
View File
@@ -0,0 +1,26 @@
산업안전보건법 001766 283449 20260219 20260601 법률
산업안전보건법 시행령 003786 284771 20260324 20260324 대통령령
산업안전보건법 시행규칙 007364 286657 20260529 20260601 고용노동부령
산업안전보건기준에 관한 규칙 007363 273603 20250901 20260302 고용노동부령
유해위험작업의 취업 제한에 관한 규칙 MISS
중대재해 처벌 등에 관한 법률 013993 228817 20210126 20220127 법률
중대재해 처벌 등에 관한 법률 시행령 014159 277417 20251001 20251001 대통령령
건설기술 진흥법 001807 276921 20251001 20251001 법률
건설기술 진흥법 시행령 002111 286847 20260609 20260609 대통령령
건설기술 진흥법 시행규칙 006175 286885 20260611 20260611 국토교통부령
시설물의 안전 및 유지관리에 관한 특별법 000237 266683 20241203 20251204 법률
위험물안전관리법 009502 259933 20240206 20250807 법률
위험물안전관리법 시행령 009707 273077 20250805 20250807 대통령령
위험물안전관리법 시행규칙 009732 262765 20240520 20250521 행정안전부령
화학물질관리법 000162 276815 20251001 20251001 법률
화학물질관리법 시행령 004390 280507 20251223 20251223 대통령령
화학물질의 등록 및 평가 등에 관한 법률 011857 279805 20251111 20260512 법률
소방시설 설치 및 관리에 관한 법률 009503 236977 20211130 20241201 법률
소방시설 설치 및 관리에 관한 법률 시행령 009694 284781 20260324 20260324 대통령령
전기사업법 001854 283981 20260310 20260310 법률
전기안전관리법 013718 268805 20250131 20260201 법률
고압가스 안전관리법 001850 283919 20260310 20260310 법률
고압가스 안전관리법 시행령 002246 286839 20260609 20260609 대통령령
액화석유가스의 안전관리 및 사업법 001849 276549 20251001 20251128 법률
근로기준법 001872 265959 20241022 20251023 법률
환경영향평가법 002016 276833 20251001 20251023 법률
1 산업안전보건법 001766 283449 20260219 20260601 법률
2 산업안전보건법 시행령 003786 284771 20260324 20260324 대통령령
3 산업안전보건법 시행규칙 007364 286657 20260529 20260601 고용노동부령
4 산업안전보건기준에 관한 규칙 007363 273603 20250901 20260302 고용노동부령
5 유해위험작업의 취업 제한에 관한 규칙 MISS
6 중대재해 처벌 등에 관한 법률 013993 228817 20210126 20220127 법률
7 중대재해 처벌 등에 관한 법률 시행령 014159 277417 20251001 20251001 대통령령
8 건설기술 진흥법 001807 276921 20251001 20251001 법률
9 건설기술 진흥법 시행령 002111 286847 20260609 20260609 대통령령
10 건설기술 진흥법 시행규칙 006175 286885 20260611 20260611 국토교통부령
11 시설물의 안전 및 유지관리에 관한 특별법 000237 266683 20241203 20251204 법률
12 위험물안전관리법 009502 259933 20240206 20250807 법률
13 위험물안전관리법 시행령 009707 273077 20250805 20250807 대통령령
14 위험물안전관리법 시행규칙 009732 262765 20240520 20250521 행정안전부령
15 화학물질관리법 000162 276815 20251001 20251001 법률
16 화학물질관리법 시행령 004390 280507 20251223 20251223 대통령령
17 화학물질의 등록 및 평가 등에 관한 법률 011857 279805 20251111 20260512 법률
18 소방시설 설치 및 관리에 관한 법률 009503 236977 20211130 20241201 법률
19 소방시설 설치 및 관리에 관한 법률 시행령 009694 284781 20260324 20260324 대통령령
20 전기사업법 001854 283981 20260310 20260310 법률
21 전기안전관리법 013718 268805 20250131 20260201 법률
22 고압가스 안전관리법 001850 283919 20260310 20260310 법률
23 고압가스 안전관리법 시행령 002246 286839 20260609 20260609 대통령령
24 액화석유가스의 안전관리 및 사업법 001849 276549 20251001 20251128 법률
25 근로기준법 001872 265959 20241022 20251023 법률
26 환경영향평가법 002016 276833 20251001 20251023 법률
+97
View File
@@ -0,0 +1,97 @@
"""Phase 2A E-4 비교기 — baseline vs 후보 run CSV 들의 per-query 판정.
python tests/search_eval/compare_runs.py \
--baseline baselines/<exact 재측정>.csv \
--cand qwen06=<...>.csv --cand qwen4=<...>.csv --cand qwen4m=<...>.csv \
[--epsilon 0.01] [--bootstrap 2000]
판정 출력(plan r3 E-4): 전체 graded NDCG 평균 delta · per-query win/loss/tie(|d|<ε=tie)
· 부트스트랩 95% CI · 카테고리별 평균 · 상위 개선/퇴행 쿼리. failure_expected/error 제외.
"""
from __future__ import annotations
import argparse
import csv
import random
import statistics
from pathlib import Path
def load(path: str) -> dict[str, dict]:
out = {}
with Path(path).open(encoding="utf-8") as f:
for row in csv.DictReader(f):
if row.get("failure_expected", "").lower() in ("true", "1"):
continue
if row.get("error"):
continue
try:
row["_g"] = float(row["graded_ndcg_at_10"])
except (TypeError, ValueError):
continue
out[row["id"]] = row
return out
def bootstrap_ci(deltas: list[float], n: int, seed: int = 42) -> tuple[float, float]:
rng = random.Random(seed)
means = sorted(
statistics.mean(rng.choices(deltas, k=len(deltas))) for _ in range(n)
)
return means[int(0.025 * n)], means[int(0.975 * n)]
def main() -> None:
p = argparse.ArgumentParser()
p.add_argument("--baseline", required=True)
p.add_argument("--cand", action="append", required=True, metavar="name=csv")
p.add_argument("--epsilon", type=float, default=0.01)
p.add_argument("--bootstrap", type=int, default=2000)
a = p.parse_args()
base = load(a.baseline)
print(f"baseline: {a.baseline} — scored {len(base)}, "
f"graded NDCG mean {statistics.mean(r['_g'] for r in base.values()):.4f}")
for spec in a.cand:
name, path = spec.split("=", 1)
cand = load(path)
ids = sorted(set(base) & set(cand))
if len(ids) != len(base):
print(f"{name}: 교집합 {len(ids)} != baseline {len(base)} — 누락 쿼리 확인")
deltas = [cand[i]["_g"] - base[i]["_g"] for i in ids]
mean_b = statistics.mean(base[i]["_g"] for i in ids)
mean_c = statistics.mean(cand[i]["_g"] for i in ids)
win = sum(1 for d in deltas if d > a.epsilon)
loss = sum(1 for d in deltas if d < -a.epsilon)
tie = len(deltas) - win - loss
lo, hi = bootstrap_ci(deltas, a.bootstrap)
decided = win + loss
win_rate = (win / decided * 100) if decided else 0.0
print(f"\n== {name} ==")
print(f" graded NDCG: {mean_b:.4f}{mean_c:.4f} (delta {mean_c-mean_b:+.4f}, "
f"bootstrap95% [{lo:+.4f}, {hi:+.4f}])")
print(f" per-query: win {win} / loss {loss} / tie {tie} (ε={a.epsilon}) — "
f"win-rate(결정전) {win_rate:.0f}%")
cats: dict[str, list[float]] = {}
for i in ids:
cats.setdefault(base[i].get("category", "?"), []).append(
cand[i]["_g"] - base[i]["_g"]
)
for c in sorted(cats):
ds = cats[c]
cb = statistics.mean(base[i]["_g"] for i in ids if base[i].get("category") == c)
cc = statistics.mean(cand[i]["_g"] for i in ids if base[i].get("category") == c)
print(f" {c:<18} {cb:.3f}{cc:.3f} ({statistics.mean(ds):+.3f}, n={len(ds)})")
ranked = sorted(ids, key=lambda i: cand[i]["_g"] - base[i]["_g"])
worst = [(i, round(cand[i]['_g']-base[i]['_g'],3)) for i in ranked[:3]]
best = [(i, round(cand[i]['_g']-base[i]['_g'],3)) for i in ranked[-3:][::-1]]
print(f" 개선 top3 {best} / 퇴행 top3 {worst}")
if __name__ == "__main__":
main()
@@ -0,0 +1,52 @@
label,id,category,legacy_category,intent,domain_hint,language,ocr_derived,failure_expected,query,relevant_ids,graded_relevance,returned_ids_top10,latency_ms,recall_at_10,mrr_at_10,ndcg_at_10,top3_hit,graded_ndcg_at_10,graded_recall_at_10_t2,graded_recall_at_10_t3,dedup_count,error
single,kw_001,standards,exact_keyword,fact_lookup,document,ko,0,0,산업안전보건법 제6장,3856;3868;3879,3856:3;3868:2;3879:2,30411;30423;30406;30418;3879;3868;3890;3863;3856;3908,925.1,1.000,0.200,0.490,0,0.417,1.000,1.000,0,
single,kw_002,standards,exact_keyword,fact_lookup,document,ko,0,0,중대재해 처벌 등에 관한 법률 제2장 중대산업재해,3917;3921,3917:3;3921:2,3921;3917;3919;3923;30418;10573;10571;3916;3874;3918,549.1,1.000,1.000,1.000,1,0.834,1.000,1.000,0,
single,kw_003,standards,exact_keyword,fact_lookup,document,ko,0,0,화학물질관리법 유해화학물질 영업자,3981,3981:3,3981;3985;3980;3984;3993;30412;3857;3978;3986;3983,350.4,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_004,standards,exact_keyword,fact_lookup,document,ko,0,0,근로기준법 안전과 보건,4041,4041:3,4041;3852;30407;3851;30406;30421;3877;30410;30422;3905,657.8,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_005,standards,exact_keyword,fact_lookup,document,ko,0,0,산업안전보건기준에 관한 규칙 보호구,3888,3888:3,10570;3888;3912;3913;3911;3905;3909;3889;3910;3902,665.5,1.000,0.500,0.631,1,0.631,1.000,1.000,0,
single,nl_001,korean_only,natural_language_ko,semantic_search,document,ko,0,0,기계로 인한 산업재해 관련 법령,3856;3868;3879;3854,3854:1;3856:3;3868:2;3879:2,5244;30422;30410;3878;5249;3897;3863;30421;3868;5253,697.8,0.250,0.111,0.118,0,0.083,0.333,0.000,0,
single,nl_002,korean_only,natural_language_ko,semantic_search,document,ko,0,0,사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일,3855;3867;3878,3855:3;3867:2;3878:2,30410;3855;5227;30422;30409;3917;3854;5244;30406;3867,566.2,0.667,0.500,0.432,1,0.508,0.667,1.000,0,
single,nl_003,korean_only,natural_language_ko,semantic_search,document,ko,0,0,유해화학물질을 다루는 회사가 지켜야 할 안전 의무,3980;3981;3982,3980:2;3981:2;3982:2,3980;3903;3904;3981;3760;5253;3985;3896;3917;3909,554.8,0.667,1.000,0.671,1,0.671,0.667,0.000,0,
single,nl_004,korean_only,natural_language_ko,semantic_search,document,ko,0,0,중대재해가 발생했을 때 경영책임자가 처벌받는 기준,3916;3917;3920;3921,3916:2;3917:3;3920:2;3921:2,10572;10573;3917;3918;3916;3923;5244;3919;10571;30409,418.8,0.500,0.333,0.346,1,0.399,0.500,1.000,0,
single,nl_005,korean_only,natural_language_ko,semantic_search,document,ko,0,0,안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가,3853;3865,3853:3;3865:2,3853;3876;5249;5234;4025;30420;30408;30414;6675;11677,918.5,0.500,1.000,0.613,1,0.787,0.500,1.000,0,
single,news_001,korean_only,news_ko,semantic_search,news,ko,0,0,이란과 미국의 군사 충돌,4303;4304;4307;4316;4322;4323;4327;4335,4303:2;4304:2;4307:2;4316:2;4322:2;4323:2;4327:2;4335:2,37599;27583;35225;27350;37601;32410;16081;37592;28407;37608,674.1,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,news_002,korean_only,news_ko,semantic_search,news,ko,0,0,호르무즈 해협 봉쇄,4316;4320;4322;4327,4316:3;4320:2;4322:2;4327:2,41652;41650;41634;31894;35899;22049;16392;9022;17123;11945,411.6,0.000,0.000,0.000,0,0.000,0.000,0.000,0,
single,misc_001,korean_only,other_domain,fact_lookup,document,ko,0,0,강체의 평면 운동학,4063;4065,4063:3;4065:2,4063;4065;4064;4071;4066;4068;4058;4067;5063;5105,653.8,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,misc_002,korean_only,other_domain,semantic_search,document,ko,0,0,질점의 운동역학,4060;4061;4062,4060:2;4061:2;4062:2,4060;4062;4070;4059;4061;4064;4065;4063;4058;5095,552.8,1.000,1.000,0.947,1,0.947,1.000,0.000,0,
single,news_003,english_only,news_en,semantic_search,news,en,0,0,Trump Iran ultimatum,4258;4260;4262,4258:2;4260:2;4262:2,21155;4775;4202;4776;4679;4199;4519;4668;37733;25592,462.7,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,cl_001,mixed,crosslingual_ko_en,semantic_search,document,mixed,0,0,기계 안전 가드 설계 원리,3770;3856,3770:3;3856:2,35778;5239;5244;3770;3817;3774;4540;3762;3789;5249,819.4,0.500,0.250,0.264,0,0.339,0.500,1.000,0,
single,cl_002,mixed,crosslingual_ko_en,semantic_search,document,mixed,0,0,산업 안전 입문서,3755;3775;3776;3777,3755:2;3775:2;3776:2;3777:2,5249;3760;5229;3755;3774;5230;10573;3775;3787;3818,644.5,0.500,0.250,0.291,1,0.291,0.500,0.000,0,
single,cl_003,mixed,crosslingual_ko_en,semantic_search,document,mixed,0,0,전기 안전 위험,3772;3790,3772:2;3790:2,3772;35779;3790;5260;3897;37326;3755;37216;37011;5235,1193.3,1.000,1.000,0.920,1,0.920,1.000,0.000,0,
single,news_004,mixed,news_fr,semantic_search,news,mixed,0,0,guerre en Iran,4199;4202;4210;4361;4363;4507;4519;4521,4199:2;4202:2;4210:2;4361:2;4363:2;4507:2;4519:2;4521:2,33101;33368;7740;27078;5925;5840;16010;16457;6945;4507,469.1,0.125,0.100,0.073,1,0.073,0.125,0.000,0,
single,news_005,mixed,news_crosslingual,semantic_search,news,mixed,0,0,이란 미국 전쟁 글로벌 반응,4202;4258;4262;4536;4303;4304;4316,4202:2;4258:2;4262:2;4303:2;4304:2;4316:2;4536:2,21275;16927;4457;19111;16761;35246;41629;35894;6789;35908,844.5,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_001,failure_expected,failure_expected,semantic_search,document,mixed,0,1,Rust async runtime tokio scheduler 내부 구조,,,5262;23732;36544;35645;37582;3774;30259;28766;31709;35635,533.4,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_002,failure_expected,failure_expected,semantic_search,document,ko,0,1,양자컴퓨터 큐비트 디코히어런스,,,31984;29686;16289;16384;5066;33238;20507;30566;5092;32721,290.9,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_003,failure_expected,failure_expected,semantic_search,news,ko,0,1,재즈 보컬리스트 빌리 홀리데이,,,31643;20470;35249;4634;26593;16059;9102;23082;23336;31187,265.7,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,kw_006,standards,standards,fact_lookup,document,ko,0,0,산업안전보건기준에 관한 규칙 작업장 통로,3886;3887,3886:3;3887:2,3898;3886;3908;3887;3881;3911;3912;3892;3900;3909,506.2,1.000,0.500,0.651,1,0.642,1.000,1.000,0,
single,kw_007,standards,standards,fact_lookup,document,ko,0,0,산업안전보건기준 폭발 화재 위험물 누출 방지,3896;3766,3766:1;3896:3,3896;3895;3903;13930;13931;13929;3894;3886;13944;3892,477.6,0.500,1.000,0.613,1,0.917,1.000,1.000,0,
single,kw_008,standards,standards,fact_lookup,document,ko,0,0,고압가스 안전관리법 전문,4025;4026,4025:3;4026:2,11644;11579;35775;4026;4025;11693;11645;4034;13299;13749,538.7,1.000,0.250,0.501,0,0.450,1.000,1.000,0,
single,kw_009,standards,standards,fact_lookup,document,ko,0,0,KGS FP111 가스설비 배관설비 기준,13305,13305:3,13305;13311;13306;13312;13302;13304;13309;13299;13313;13918,446.0,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_010,standards,standards,fact_lookup,document,ko,0,0,KGS FU551 가스설비 압력조정기 가스계량기,13652,13652:3,13652;11689;13657;13655;13656;13649;13651;13752;13659;13650,312.5,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_011,standards,standards,fact_lookup,document,ko,0,0,산업안전보건기준 전기로 인한 위험 방지,3897;3772,3772:1;3897:3,3897;3895;3887;13935;3902;3896;3877;3900;3899;3854,587.9,0.500,1.000,0.613,1,0.917,1.000,1.000,0,
single,en_001,english_only,english_only,semantic_search,document,en,0,0,pressure vessel flange design,5144;5136,5136:2;5144:3,5144;5140;5136;5180;36899;5137;5149;5178;5207;5148,4162.7,1.000,1.000,0.920,1,0.956,1.000,1.000,0,
single,en_002,english_only,english_only,semantic_search,document,en,0,0,ASME Section VIII introduction,5204;5206,5204:3;5206:2,5204;5212;5180;5208;5210;5143;5141;5206;5137;11634,3038.6,1.000,1.000,0.807,1,0.894,1.000,1.000,0,
single,en_003,english_only,english_only,semantic_search,document,en,0,0,impact test requirements ASME,5205;5148,5148:1;5205:3,36721;5205;5204;5178;5214;5224;5148;5140;5209;5145,5221.9,1.000,0.500,0.591,1,0.622,1.000,1.000,0,
single,en_004,english_only,english_only,semantic_search,document,en,0,0,design of vessel supports,5149,5149:3,5149;5186;5141;5137;36901;36869;5140;5139;5136;5178,5732.5,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,en_005,english_only,english_only,semantic_search,document,en,0,0,hydrogen piping ASME code,5178,5178:3,5178;5180;5210;5212;5182;5133;5207;5148;5204;5209,4941.9,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,en_006,english_only,english_only,semantic_search,document,en,0,0,ASME welding qualification requirements,5209;3771,3771:1;5209:3,37097;5224;36877;36721;5208;5209;5205;5178;5204;5180,2432.9,0.500,0.167,0.218,0,0.327,1.000,1.000,0,
single,en_007,english_only,english_only,semantic_search,document,en,0,0,pressure vessel fabrication and inspection,5208;5145,5145:2;5208:3,5208;38747;38791;36935;5189;36874;5180;36908;5187;36901,3828.4,0.500,1.000,0.613,1,0.787,0.500,1.000,0,
single,en_008,english_only,english_only,semantic_search,document,en,0,0,Industrial Safety and Health Management ergonomics,3763;3755,3755:1;3763:3,3763;3759;3774;3755;3818;3812;3778;3756;3761;3769,1866.0,1.000,1.000,0.877,1,0.974,1.000,1.000,0,
single,cl_004,mixed,mixed,semantic_search,document,mixed,0,0,ASME 압력용기 설계 실무,5207;5210;5139;5135,5135:2;5139:2;5207:3;5210:3,5139;5207;5204;5225;5206;5208;5210;5137;5182;5145,805.5,0.750,1.000,0.767,1,0.686,0.750,1.000,0,
single,cl_005,mixed,mixed,semantic_search,document,mixed,0,0,ASME 용접 코드 해설,5224;5209,5209:2;5224:3,5224;5222;5225;5209;5204;5210;5205;5178;5143;5141,737.6,1.000,1.000,0.877,1,0.932,1.000,1.000,0,
single,cl_006,mixed,mixed,semantic_search,document,mixed,0,0,pressure vessel Korean industrial safety regulation,4025;3856;5136,3856:2;4025:2;5136:1,38941;4026;36874;5145;5186;38791;39084;5143;3895;5210,1231.3,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,cl_007,mixed,mixed,semantic_search,document,mixed,0,0,KGS 코드 LPG safety standard,11647;11689;11645;4025,4025:1;11645:2;11647:3;11689:2,11647;13760;13674;13669;13774;13773;13675;13755;13924;11688,528.5,0.250,1.000,0.390,1,0.647,0.333,1.000,0,
single,cl_008,mixed,mixed,semantic_search,document,mixed,0,0,수소 가스 안전 기준 hydrogen safety,5178;5169,5169:2;5178:3,10575;5177;11671;11649;11648;13915;5241;5173;11653;5170,872.1,0.000,0.000,0.000,0,0.000,0.000,0.000,0,
single,exam_001,exam,exam,fact_lookup,document,ko,0,0,레이놀즈수 정의 공식,11504;11505,11504:3;11505:2,11504;11533;5090;4544;11509;11505;11476;11486;5132;5140,628.8,1.000,1.000,0.832,1,0.907,1.000,1.000,0,
single,exam_002,exam,exam,fact_lookup,document,ko,0,0,탱크 바닥 구멍 유체 유속 토리첼리,11500;11495;11496,11495:2;11496:2;11500:3,11500;11495;11501;5139;5090;5140;5178;11515;11517;5210,338.1,0.667,1.000,0.765,1,0.856,0.667,1.000,0,
single,exam_003,exam,exam,fact_lookup,document,ko,0,0,이상기체 음속 마하수,11514;11515,11514:3;11515:2,11514;5083;11475;5090;5084;11531;11476;11473;11479;5124,675.8,0.500,1.000,0.613,1,0.787,0.500,1.000,0,
single,exam_004,exam,exam,fact_lookup,document,ko,0,0,고압가스 용기 내압시험 영구증가량,11591;11644;11691,11591:3;11644:2;11691:2,11591;11664;13660;5177;11691;11665;13652;13942;13917;11660,357.3,0.667,1.000,0.651,1,0.785,0.667,1.000,0,
single,exam_005,exam,exam,fact_lookup,document,ko,0,0,도시가스 배관 매설 이격거리,11627;11625;11646,11625:2;11627:3;11646:1,11627;11658;11600;11625;13918;11692;13751;5177;11655;13653,400.9,0.667,1.000,0.671,1,0.883,1.000,1.000,0,
single,exam_006,exam,exam,fact_lookup,document,ko,0,0,LPG 저장탱크 안전거리 분말소화기,11617;11669;11620,11617:3;11620:1;11669:2,11595;11616;13669;11617;11649;11605;11655;11690;11658;13915,349.0,0.333,0.250,0.202,0,0.321,0.500,1.000,0,
single,exam_007,exam,exam,fact_lookup,document,ko,0,0,오리피스 차압식 유량계,11712;11711;11503,11503:2;11711:2;11712:3,11711;11712;11503;11500;11714;11713;13930;11717;11701;11715,416.5,1.000,1.000,1.000,1,0.858,1.000,1.000,0,
single,fail_004,failure_expected,failure_expected,fact_lookup,document,ko,0,1,KGS AC999 임의 가스 코드,,,11691;11647;11693;11692;13665;13661;13664;13666;13670;13773,404.6,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_005,failure_expected,failure_expected,fact_lookup,document,ko,0,1,초전도 안전 관리법 시행규칙,,,4026;35775;5236;3966;3977;5260;3971;4018;3972;3973,569.1,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
1 label id category legacy_category intent domain_hint language ocr_derived failure_expected query relevant_ids graded_relevance returned_ids_top10 latency_ms recall_at_10 mrr_at_10 ndcg_at_10 top3_hit graded_ndcg_at_10 graded_recall_at_10_t2 graded_recall_at_10_t3 dedup_count error
2 single kw_001 standards exact_keyword fact_lookup document ko 0 0 산업안전보건법 제6장 3856;3868;3879 3856:3;3868:2;3879:2 30411;30423;30406;30418;3879;3868;3890;3863;3856;3908 925.1 1.000 0.200 0.490 0 0.417 1.000 1.000 0
3 single kw_002 standards exact_keyword fact_lookup document ko 0 0 중대재해 처벌 등에 관한 법률 제2장 중대산업재해 3917;3921 3917:3;3921:2 3921;3917;3919;3923;30418;10573;10571;3916;3874;3918 549.1 1.000 1.000 1.000 1 0.834 1.000 1.000 0
4 single kw_003 standards exact_keyword fact_lookup document ko 0 0 화학물질관리법 유해화학물질 영업자 3981 3981:3 3981;3985;3980;3984;3993;30412;3857;3978;3986;3983 350.4 1.000 1.000 1.000 1 1.000 1.000 1.000 0
5 single kw_004 standards exact_keyword fact_lookup document ko 0 0 근로기준법 안전과 보건 4041 4041:3 4041;3852;30407;3851;30406;30421;3877;30410;30422;3905 657.8 1.000 1.000 1.000 1 1.000 1.000 1.000 0
6 single kw_005 standards exact_keyword fact_lookup document ko 0 0 산업안전보건기준에 관한 규칙 보호구 3888 3888:3 10570;3888;3912;3913;3911;3905;3909;3889;3910;3902 665.5 1.000 0.500 0.631 1 0.631 1.000 1.000 0
7 single nl_001 korean_only natural_language_ko semantic_search document ko 0 0 기계로 인한 산업재해 관련 법령 3856;3868;3879;3854 3854:1;3856:3;3868:2;3879:2 5244;30422;30410;3878;5249;3897;3863;30421;3868;5253 697.8 0.250 0.111 0.118 0 0.083 0.333 0.000 0
8 single nl_002 korean_only natural_language_ko semantic_search document ko 0 0 사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일 3855;3867;3878 3855:3;3867:2;3878:2 30410;3855;5227;30422;30409;3917;3854;5244;30406;3867 566.2 0.667 0.500 0.432 1 0.508 0.667 1.000 0
9 single nl_003 korean_only natural_language_ko semantic_search document ko 0 0 유해화학물질을 다루는 회사가 지켜야 할 안전 의무 3980;3981;3982 3980:2;3981:2;3982:2 3980;3903;3904;3981;3760;5253;3985;3896;3917;3909 554.8 0.667 1.000 0.671 1 0.671 0.667 0.000 0
10 single nl_004 korean_only natural_language_ko semantic_search document ko 0 0 중대재해가 발생했을 때 경영책임자가 처벌받는 기준 3916;3917;3920;3921 3916:2;3917:3;3920:2;3921:2 10572;10573;3917;3918;3916;3923;5244;3919;10571;30409 418.8 0.500 0.333 0.346 1 0.399 0.500 1.000 0
11 single nl_005 korean_only natural_language_ko semantic_search document ko 0 0 안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가 3853;3865 3853:3;3865:2 3853;3876;5249;5234;4025;30420;30408;30414;6675;11677 918.5 0.500 1.000 0.613 1 0.787 0.500 1.000 0
12 single news_001 korean_only news_ko semantic_search news ko 0 0 이란과 미국의 군사 충돌 4303;4304;4307;4316;4322;4323;4327;4335 4303:2;4304:2;4307:2;4316:2;4322:2;4323:2;4327:2;4335:2 37599;27583;35225;27350;37601;32410;16081;37592;28407;37608 674.1 0.000 0.000 0.000 1 0.000 0.000 0.000 0
13 single news_002 korean_only news_ko semantic_search news ko 0 0 호르무즈 해협 봉쇄 4316;4320;4322;4327 4316:3;4320:2;4322:2;4327:2 41652;41650;41634;31894;35899;22049;16392;9022;17123;11945 411.6 0.000 0.000 0.000 0 0.000 0.000 0.000 0
14 single misc_001 korean_only other_domain fact_lookup document ko 0 0 강체의 평면 운동학 4063;4065 4063:3;4065:2 4063;4065;4064;4071;4066;4068;4058;4067;5063;5105 653.8 1.000 1.000 1.000 1 1.000 1.000 1.000 0
15 single misc_002 korean_only other_domain semantic_search document ko 0 0 질점의 운동역학 4060;4061;4062 4060:2;4061:2;4062:2 4060;4062;4070;4059;4061;4064;4065;4063;4058;5095 552.8 1.000 1.000 0.947 1 0.947 1.000 0.000 0
16 single news_003 english_only news_en semantic_search news en 0 0 Trump Iran ultimatum 4258;4260;4262 4258:2;4260:2;4262:2 21155;4775;4202;4776;4679;4199;4519;4668;37733;25592 462.7 0.000 0.000 0.000 1 0.000 0.000 0.000 0
17 single cl_001 mixed crosslingual_ko_en semantic_search document mixed 0 0 기계 안전 가드 설계 원리 3770;3856 3770:3;3856:2 35778;5239;5244;3770;3817;3774;4540;3762;3789;5249 819.4 0.500 0.250 0.264 0 0.339 0.500 1.000 0
18 single cl_002 mixed crosslingual_ko_en semantic_search document mixed 0 0 산업 안전 입문서 3755;3775;3776;3777 3755:2;3775:2;3776:2;3777:2 5249;3760;5229;3755;3774;5230;10573;3775;3787;3818 644.5 0.500 0.250 0.291 1 0.291 0.500 0.000 0
19 single cl_003 mixed crosslingual_ko_en semantic_search document mixed 0 0 전기 안전 위험 3772;3790 3772:2;3790:2 3772;35779;3790;5260;3897;37326;3755;37216;37011;5235 1193.3 1.000 1.000 0.920 1 0.920 1.000 0.000 0
20 single news_004 mixed news_fr semantic_search news mixed 0 0 guerre en Iran 4199;4202;4210;4361;4363;4507;4519;4521 4199:2;4202:2;4210:2;4361:2;4363:2;4507:2;4519:2;4521:2 33101;33368;7740;27078;5925;5840;16010;16457;6945;4507 469.1 0.125 0.100 0.073 1 0.073 0.125 0.000 0
21 single news_005 mixed news_crosslingual semantic_search news mixed 0 0 이란 미국 전쟁 글로벌 반응 4202;4258;4262;4536;4303;4304;4316 4202:2;4258:2;4262:2;4303:2;4304:2;4316:2;4536:2 21275;16927;4457;19111;16761;35246;41629;35894;6789;35908 844.5 0.000 0.000 0.000 1 0.000 0.000 0.000 0
22 single fail_001 failure_expected failure_expected semantic_search document mixed 0 1 Rust async runtime tokio scheduler 내부 구조 5262;23732;36544;35645;37582;3774;30259;28766;31709;35635 533.4 0.000 0.000 0.000 1 0.000 0.000 0.000 0
23 single fail_002 failure_expected failure_expected semantic_search document ko 0 1 양자컴퓨터 큐비트 디코히어런스 31984;29686;16289;16384;5066;33238;20507;30566;5092;32721 290.9 0.000 0.000 0.000 1 0.000 0.000 0.000 0
24 single fail_003 failure_expected failure_expected semantic_search news ko 0 1 재즈 보컬리스트 빌리 홀리데이 31643;20470;35249;4634;26593;16059;9102;23082;23336;31187 265.7 0.000 0.000 0.000 1 0.000 0.000 0.000 0
25 single kw_006 standards standards fact_lookup document ko 0 0 산업안전보건기준에 관한 규칙 작업장 통로 3886;3887 3886:3;3887:2 3898;3886;3908;3887;3881;3911;3912;3892;3900;3909 506.2 1.000 0.500 0.651 1 0.642 1.000 1.000 0
26 single kw_007 standards standards fact_lookup document ko 0 0 산업안전보건기준 폭발 화재 위험물 누출 방지 3896;3766 3766:1;3896:3 3896;3895;3903;13930;13931;13929;3894;3886;13944;3892 477.6 0.500 1.000 0.613 1 0.917 1.000 1.000 0
27 single kw_008 standards standards fact_lookup document ko 0 0 고압가스 안전관리법 전문 4025;4026 4025:3;4026:2 11644;11579;35775;4026;4025;11693;11645;4034;13299;13749 538.7 1.000 0.250 0.501 0 0.450 1.000 1.000 0
28 single kw_009 standards standards fact_lookup document ko 0 0 KGS FP111 가스설비 배관설비 기준 13305 13305:3 13305;13311;13306;13312;13302;13304;13309;13299;13313;13918 446.0 1.000 1.000 1.000 1 1.000 1.000 1.000 0
29 single kw_010 standards standards fact_lookup document ko 0 0 KGS FU551 가스설비 압력조정기 가스계량기 13652 13652:3 13652;11689;13657;13655;13656;13649;13651;13752;13659;13650 312.5 1.000 1.000 1.000 1 1.000 1.000 1.000 0
30 single kw_011 standards standards fact_lookup document ko 0 0 산업안전보건기준 전기로 인한 위험 방지 3897;3772 3772:1;3897:3 3897;3895;3887;13935;3902;3896;3877;3900;3899;3854 587.9 0.500 1.000 0.613 1 0.917 1.000 1.000 0
31 single en_001 english_only english_only semantic_search document en 0 0 pressure vessel flange design 5144;5136 5136:2;5144:3 5144;5140;5136;5180;36899;5137;5149;5178;5207;5148 4162.7 1.000 1.000 0.920 1 0.956 1.000 1.000 0
32 single en_002 english_only english_only semantic_search document en 0 0 ASME Section VIII introduction 5204;5206 5204:3;5206:2 5204;5212;5180;5208;5210;5143;5141;5206;5137;11634 3038.6 1.000 1.000 0.807 1 0.894 1.000 1.000 0
33 single en_003 english_only english_only semantic_search document en 0 0 impact test requirements ASME 5205;5148 5148:1;5205:3 36721;5205;5204;5178;5214;5224;5148;5140;5209;5145 5221.9 1.000 0.500 0.591 1 0.622 1.000 1.000 0
34 single en_004 english_only english_only semantic_search document en 0 0 design of vessel supports 5149 5149:3 5149;5186;5141;5137;36901;36869;5140;5139;5136;5178 5732.5 1.000 1.000 1.000 1 1.000 1.000 1.000 0
35 single en_005 english_only english_only semantic_search document en 0 0 hydrogen piping ASME code 5178 5178:3 5178;5180;5210;5212;5182;5133;5207;5148;5204;5209 4941.9 1.000 1.000 1.000 1 1.000 1.000 1.000 0
36 single en_006 english_only english_only semantic_search document en 0 0 ASME welding qualification requirements 5209;3771 3771:1;5209:3 37097;5224;36877;36721;5208;5209;5205;5178;5204;5180 2432.9 0.500 0.167 0.218 0 0.327 1.000 1.000 0
37 single en_007 english_only english_only semantic_search document en 0 0 pressure vessel fabrication and inspection 5208;5145 5145:2;5208:3 5208;38747;38791;36935;5189;36874;5180;36908;5187;36901 3828.4 0.500 1.000 0.613 1 0.787 0.500 1.000 0
38 single en_008 english_only english_only semantic_search document en 0 0 Industrial Safety and Health Management ergonomics 3763;3755 3755:1;3763:3 3763;3759;3774;3755;3818;3812;3778;3756;3761;3769 1866.0 1.000 1.000 0.877 1 0.974 1.000 1.000 0
39 single cl_004 mixed mixed semantic_search document mixed 0 0 ASME 압력용기 설계 실무 5207;5210;5139;5135 5135:2;5139:2;5207:3;5210:3 5139;5207;5204;5225;5206;5208;5210;5137;5182;5145 805.5 0.750 1.000 0.767 1 0.686 0.750 1.000 0
40 single cl_005 mixed mixed semantic_search document mixed 0 0 ASME 용접 코드 해설 5224;5209 5209:2;5224:3 5224;5222;5225;5209;5204;5210;5205;5178;5143;5141 737.6 1.000 1.000 0.877 1 0.932 1.000 1.000 0
41 single cl_006 mixed mixed semantic_search document mixed 0 0 pressure vessel Korean industrial safety regulation 4025;3856;5136 3856:2;4025:2;5136:1 38941;4026;36874;5145;5186;38791;39084;5143;3895;5210 1231.3 0.000 0.000 0.000 1 0.000 0.000 0.000 0
42 single cl_007 mixed mixed semantic_search document mixed 0 0 KGS 코드 LPG safety standard 11647;11689;11645;4025 4025:1;11645:2;11647:3;11689:2 11647;13760;13674;13669;13774;13773;13675;13755;13924;11688 528.5 0.250 1.000 0.390 1 0.647 0.333 1.000 0
43 single cl_008 mixed mixed semantic_search document mixed 0 0 수소 가스 안전 기준 hydrogen safety 5178;5169 5169:2;5178:3 10575;5177;11671;11649;11648;13915;5241;5173;11653;5170 872.1 0.000 0.000 0.000 0 0.000 0.000 0.000 0
44 single exam_001 exam exam fact_lookup document ko 0 0 레이놀즈수 정의 공식 11504;11505 11504:3;11505:2 11504;11533;5090;4544;11509;11505;11476;11486;5132;5140 628.8 1.000 1.000 0.832 1 0.907 1.000 1.000 0
45 single exam_002 exam exam fact_lookup document ko 0 0 탱크 바닥 구멍 유체 유속 토리첼리 11500;11495;11496 11495:2;11496:2;11500:3 11500;11495;11501;5139;5090;5140;5178;11515;11517;5210 338.1 0.667 1.000 0.765 1 0.856 0.667 1.000 0
46 single exam_003 exam exam fact_lookup document ko 0 0 이상기체 음속 마하수 11514;11515 11514:3;11515:2 11514;5083;11475;5090;5084;11531;11476;11473;11479;5124 675.8 0.500 1.000 0.613 1 0.787 0.500 1.000 0
47 single exam_004 exam exam fact_lookup document ko 0 0 고압가스 용기 내압시험 영구증가량 11591;11644;11691 11591:3;11644:2;11691:2 11591;11664;13660;5177;11691;11665;13652;13942;13917;11660 357.3 0.667 1.000 0.651 1 0.785 0.667 1.000 0
48 single exam_005 exam exam fact_lookup document ko 0 0 도시가스 배관 매설 이격거리 11627;11625;11646 11625:2;11627:3;11646:1 11627;11658;11600;11625;13918;11692;13751;5177;11655;13653 400.9 0.667 1.000 0.671 1 0.883 1.000 1.000 0
49 single exam_006 exam exam fact_lookup document ko 0 0 LPG 저장탱크 안전거리 분말소화기 11617;11669;11620 11617:3;11620:1;11669:2 11595;11616;13669;11617;11649;11605;11655;11690;11658;13915 349.0 0.333 0.250 0.202 0 0.321 0.500 1.000 0
50 single exam_007 exam exam fact_lookup document ko 0 0 오리피스 차압식 유량계 11712;11711;11503 11503:2;11711:2;11712:3 11711;11712;11503;11500;11714;11713;13930;11717;11701;11715 416.5 1.000 1.000 1.000 1 0.858 1.000 1.000 0
51 single fail_004 failure_expected failure_expected fact_lookup document ko 0 1 KGS AC999 임의 가스 코드 11691;11647;11693;11692;13665;13661;13664;13666;13670;13773 404.6 0.000 0.000 0.000 1 0.000 0.000 0.000 0
52 single fail_005 failure_expected failure_expected fact_lookup document ko 0 1 초전도 안전 관리법 시행규칙 4026;35775;5236;3966;3977;5260;3971;4018;3972;3973 569.1 0.000 0.000 0.000 1 0.000 0.000 0.000 0
@@ -0,0 +1,52 @@
label,id,category,legacy_category,intent,domain_hint,language,ocr_derived,failure_expected,query,relevant_ids,graded_relevance,returned_ids_top10,latency_ms,recall_at_10,mrr_at_10,ndcg_at_10,top3_hit,graded_ndcg_at_10,graded_recall_at_10_t2,graded_recall_at_10_t3,dedup_count,error
single,kw_001,standards,exact_keyword,fact_lookup,document,ko,0,0,산업안전보건법 제6장,3856;3868;3879,3856:3;3868:2;3879:2,30411;30423;3879;3868;3856;3851;4041;10573;5163;30406,659.8,1.000,0.333,0.618,0,0.529,1.000,1.000,0,
single,kw_002,standards,exact_keyword,fact_lookup,document,ko,0,0,중대재해 처벌 등에 관한 법률 제2장 중대산업재해,3917;3921,3917:3;3921:2,3921;3917;10573;5244;3919;3923;30418;5227;3916;3874,517.4,1.000,1.000,1.000,1,0.834,1.000,1.000,0,
single,kw_003,standards,exact_keyword,fact_lookup,document,ko,0,0,화학물질관리법 유해화학물질 영업자,3981,3981:3,3981;3985;3980;3984;3993;3869;30412;3857;3978;3986,336.1,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_004,standards,exact_keyword,fact_lookup,document,ko,0,0,근로기준법 안전과 보건,4041,4041:3,4041;3852;30407;5227;30421;30409;30406;10573;3851;3858,599.6,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_005,standards,exact_keyword,fact_lookup,document,ko,0,0,산업안전보건기준에 관한 규칙 보호구,3888,3888:3,10570;3888;5249;3893;3887;3904;3899;3890;3895;3902,629.5,1.000,0.500,0.631,1,0.631,1.000,1.000,0,
single,nl_001,korean_only,natural_language_ko,semantic_search,document,ko,0,0,기계로 인한 산업재해 관련 법령,3856;3868;3879;3854,3854:1;3856:3;3868:2;3879:2,5244;30422;30410;3878;5249;3855;3897;30421;3868;3863,664.7,0.250,0.111,0.118,0,0.083,0.333,0.000,0,
single,nl_002,korean_only,natural_language_ko,semantic_search,document,ko,0,0,사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일,3855;3867;3878,3855:3;3867:2;3878:2,30410;3855;5227;30409;3917;3854;5244;3867;30421;3878,545.5,1.000,0.500,0.580,1,0.599,1.000,1.000,0,
single,nl_003,korean_only,natural_language_ko,semantic_search,document,ko,0,0,유해화학물질을 다루는 회사가 지켜야 할 안전 의무,3980;3981;3982,3980:2;3981:2;3982:2,3980;3904;3903;3981;3905;35780;3985;10569;11648;3857,477.8,0.667,1.000,0.671,1,0.671,0.667,0.000,0,
single,nl_004,korean_only,natural_language_ko,semantic_search,document,ko,0,0,중대재해가 발생했을 때 경영책임자가 처벌받는 기준,3916;3917;3920;3921,3916:2;3917:3;3920:2;3921:2,10573;3917;3916;3918;5227;5244;3919;10571;30409;3854,391.5,0.500,0.500,0.441,1,0.506,0.500,1.000,0,
single,nl_005,korean_only,natural_language_ko,semantic_search,document,ko,0,0,안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가,3853;3865,3853:3;3865:2,3853;3876;5249;5234;30420;30408;30414;10573;11677;3867,708.8,0.500,1.000,0.613,1,0.787,0.500,1.000,0,
single,news_001,korean_only,news_ko,semantic_search,news,ko,0,0,이란과 미국의 군사 충돌,4303;4304;4307;4316;4322;4323;4327;4335,4303:2;4304:2;4307:2;4316:2;4322:2;4323:2;4327:2;4335:2,37599;27583;35225;27350;34850;32177;15924;15976;28316;16378,601.4,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,news_002,korean_only,news_ko,semantic_search,news,ko,0,0,호르무즈 해협 봉쇄,4316;4320;4322;4327,4316:3;4320:2;4322:2;4327:2,41652;41650;41634;31894;35899;22055;9022;17123;11945;9105,402.3,0.000,0.000,0.000,0,0.000,0.000,0.000,0,
single,misc_001,korean_only,other_domain,fact_lookup,document,ko,0,0,강체의 평면 운동학,4063;4065,4063:3;4065:2,4063;4065;4066;4071;4067;4058;5105;4068;4064;5083,578.4,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,misc_002,korean_only,other_domain,semantic_search,document,ko,0,0,질점의 운동역학,4060;4061;4062,4060:2;4061:2;4062:2,4070;4060;4062;4059;4064;4066;4058;4068;5083;4065,509.9,0.667,0.500,0.531,1,0.531,0.667,0.000,0,
single,news_003,english_only,news_en,semantic_search,news,en,0,0,Trump Iran ultimatum,4258;4260;4262,4258:2;4260:2;4262:2,21155;21441;4775;4202;4776;4679;4519;37733;25592;28001,339.8,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,cl_001,mixed,crosslingual_ko_en,semantic_search,document,mixed,0,0,기계 안전 가드 설계 원리,3770;3856,3770:3;3856:2,35778;5239;5244;3758;3770;3763;4540;3787;3810;3791,699.5,0.500,0.200,0.237,0,0.305,0.500,1.000,0,
single,cl_002,mixed,crosslingual_ko_en,semantic_search,document,mixed,0,0,산업 안전 입문서,3755;3775;3776;3777,3755:2;3775:2;3776:2;3777:2,5230;5249;3755;3774;5229;10573;3851;3863;30406;3802,543.0,0.250,0.333,0.195,1,0.195,0.250,0.000,0,
single,cl_003,mixed,crosslingual_ko_en,semantic_search,document,mixed,0,0,전기 안전 위험,3772;3790,3772:2;3790:2,35779;5260;3772;3897;3790;37326;3755;37216;37011;37330,1114.4,1.000,0.333,0.544,1,0.544,1.000,0.000,0,
single,news_004,mixed,news_fr,semantic_search,news,mixed,0,0,guerre en Iran,4199;4202;4210;4361;4363;4507;4519;4521,4199:2;4202:2;4210:2;4361:2;4363:2;4507:2;4519:2;4521:2,33101;33368;7740;33374;27078;5925;26631;5840;16010;16457,426.3,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,news_005,mixed,news_crosslingual,semantic_search,news,mixed,0,0,이란 미국 전쟁 글로벌 반응,4202;4258;4262;4536;4303;4304;4316,4202:2;4258:2;4262:2;4303:2;4304:2;4316:2;4536:2,23242;4457;19111;41629;26741;6789;17242;9469;4765;4345,839.7,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_001,failure_expected,failure_expected,semantic_search,document,mixed,0,1,Rust async runtime tokio scheduler 내부 구조,,,23732;36544;25771;35645;37582;17810;28766;40739;40432;40679,465.6,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_002,failure_expected,failure_expected,semantic_search,document,ko,0,1,양자컴퓨터 큐비트 디코히어런스,,,32096;31984;16289;16384;32043;32035;28233;33551;32721;32028,270.7,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_003,failure_expected,failure_expected,semantic_search,news,ko,0,1,재즈 보컬리스트 빌리 홀리데이,,,18567;20022;20470;35249;26594;16232;23336;35498;18286;16694,258.9,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,kw_006,standards,standards,fact_lookup,document,ko,0,0,산업안전보건기준에 관한 규칙 작업장 통로,3886;3887,3886:3;3887:2,3898;3886;3887;3900;3881;3889;3912;3894;3897;3888,480.6,1.000,0.500,0.693,1,0.665,1.000,1.000,0,
single,kw_007,standards,standards,fact_lookup,document,ko,0,0,산업안전보건기준 폭발 화재 위험물 누출 방지,3896;3766,3766:1;3896:3,3896;3895;13930;3903;3897;13769;11679;11562;11567;10571,513.9,0.500,1.000,0.613,1,0.917,1.000,1.000,0,
single,kw_008,standards,standards,fact_lookup,document,ko,0,0,고압가스 안전관리법 전문,4025;4026,4025:3;4026:2,11644;11579;4025;4026;35775;11693;4034;13750;13299;13941,493.9,1.000,0.333,0.571,1,0.539,1.000,1.000,0,
single,kw_009,standards,standards,fact_lookup,document,ko,0,0,KGS FP111 가스설비 배관설비 기준,13305,13305:3,13305;13306;13311;13312;13302;13304;11688;13309;13313;13918,434.6,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_010,standards,standards,fact_lookup,document,ko,0,0,KGS FU551 가스설비 압력조정기 가스계량기,13652,13652:3,13652;11689;13657;13655;13649;13752;13648;13659;13650;13656,307.8,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_011,standards,standards,fact_lookup,document,ko,0,0,산업안전보건기준 전기로 인한 위험 방지,3897;3772,3772:1;3897:3,3897;3886;3887;13935;3895;3902;3896;3899;3877;3900,583.1,0.500,1.000,0.613,1,0.917,1.000,1.000,0,
single,en_001,english_only,english_only,semantic_search,document,en,0,0,pressure vessel flange design,5144;5136,5136:2;5144:3,5140;5136;5186;5144;5180;5212;5137;5193;36899;5182,4040.7,1.000,0.500,0.651,0,0.552,1.000,1.000,0,
single,en_002,english_only,english_only,semantic_search,document,en,0,0,ASME Section VIII introduction,5204;5206,5204:3;5206:2,5204;5212;5208;5210;5206;11634;5207;5183;5205;5182,2996.1,1.000,1.000,0.850,1,0.918,1.000,1.000,0,
single,en_003,english_only,english_only,semantic_search,document,en,0,0,impact test requirements ASME,5205;5148,5148:1;5205:3,36721;5205;5204;5178;5214;5224;5148;5186;5192;5185,5084.5,1.000,0.500,0.591,1,0.622,1.000,1.000,0,
single,en_004,english_only,english_only,semantic_search,document,en,0,0,design of vessel supports,5149,5149:3,5149;5186;5141;5136;5137;36901;36869;5139;5178;5145,5563.0,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,en_005,english_only,english_only,semantic_search,document,en,0,0,hydrogen piping ASME code,5178,5178:3,5178;5180;5212;5182;5207;5210;5148;5204;5209;5224,4870.2,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,en_006,english_only,english_only,semantic_search,document,en,0,0,ASME welding qualification requirements,5209;3771,3771:1;5209:3,5180;5178;5224;36721;36877;37097;5205;5204;36878;37051,2374.8,0.000,0.000,0.000,0,0.000,0.000,0.000,0,
single,en_007,english_only,english_only,semantic_search,document,en,0,0,pressure vessel fabrication and inspection,5208;5145,5145:2;5208:3,5186;5208;38747;38791;36935;36874;5187;5191;5182;5175,3752.4,0.500,0.500,0.387,1,0.497,0.500,1.000,0,
single,en_008,english_only,english_only,semantic_search,document,en,0,0,Industrial Safety and Health Management ergonomics,3763;3755,3755:1;3763:3,3759;3763;3774;3755;3818;5230;3812;3770;3756;3760,1817.5,1.000,0.500,0.651,1,0.635,1.000,1.000,0,
single,cl_004,mixed,mixed,semantic_search,document,mixed,0,0,ASME 압력용기 설계 실무,5207;5210;5139;5135,5135:2;5139:2;5207:3;5210:3,5139;5207;5204;5206;5225;5210;5133;5143;5141;5140,795.3,0.750,1.000,0.776,1,0.697,0.750,1.000,0,
single,cl_005,mixed,mixed,semantic_search,document,mixed,0,0,ASME 용접 코드 해설,5224;5209,5209:2;5224:3,5224;5222;5209;5225;5210;5208;5180;5204;5195;5205,724.0,1.000,1.000,0.920,1,0.956,1.000,1.000,0,
single,cl_006,mixed,mixed,semantic_search,document,mixed,0,0,pressure vessel Korean industrial safety regulation,4025;3856;5136,3856:2;4025:2;5136:1,38941;3762;4026;36889;36874;38791;5182;39084;5143;5186,1217.1,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,cl_007,mixed,mixed,semantic_search,document,mixed,0,0,KGS 코드 LPG safety standard,11647;11689;11645;4025,4025:1;11645:2;11647:3;11689:2,11647;13674;13675;11688;11645;13310;13773;13759;13668;11652,512.6,0.500,1.000,0.541,1,0.754,0.667,1.000,0,
single,cl_008,mixed,mixed,semantic_search,document,mixed,0,0,수소 가스 안전 기준 hydrogen safety,5178;5169,5169:2;5178:3,10575;5177;13924;13925;5241;11562;5178;5173;11568;11563,838.4,0.500,0.143,0.204,0,0.262,0.500,1.000,0,
single,exam_001,exam,exam,fact_lookup,document,ko,0,0,레이놀즈수 정의 공식,11504;11505,11504:3;11505:2,11504;11533;5081;11482;5088;5090;5082;5077;11509;11505,590.3,1.000,1.000,0.790,1,0.885,1.000,1.000,0,
single,exam_002,exam,exam,fact_lookup,document,ko,0,0,탱크 바닥 구멍 유체 유속 토리첼리,11500;11495;11496,11495:2;11496:2;11500:3,11500;11495;11501;5085;5071;11515;5090;5139;5086;5088,387.2,0.667,1.000,0.765,1,0.856,0.667,1.000,0,
single,exam_003,exam,exam,fact_lookup,document,ko,0,0,이상기체 음속 마하수,11514;11515,11514:3;11515:2,11479;5083;5084;11476;11514;11516;11515;11518;11517;11472,645.5,1.000,0.200,0.442,0,0.417,1.000,1.000,0,
single,exam_004,exam,exam,fact_lookup,document,ko,0,0,고압가스 용기 내압시험 영구증가량,11591;11644;11691,11591:3;11644:2;11691:2,11591;11664;5177;13652;13660;13942;11660;13653;13752;11579,338.3,0.333,1.000,0.469,1,0.674,0.333,1.000,0,
single,exam_005,exam,exam,fact_lookup,document,ko,0,0,도시가스 배관 매설 이격거리,11627;11625;11646,11625:2;11627:3;11646:1,11627;11658;11625;11692;13753;13751;11655;13653;13752;11624,399.8,0.667,1.000,0.704,1,0.905,1.000,1.000,0,
single,exam_006,exam,exam,fact_lookup,document,ko,0,0,LPG 저장탱크 안전거리 분말소화기,11617;11669;11620,11617:3;11620:1;11669:2,11595;11616;13669;11617;11649;11605;11655;11690;11658;11653,353.2,0.333,0.250,0.202,0,0.321,0.500,1.000,0,
single,exam_007,exam,exam,fact_lookup,document,ko,0,0,오리피스 차압식 유량계,11712;11711;11503,11503:2;11711:2;11712:3,11712;11711;11500;11503;11713;13930;11701;11502;11715;11717,408.6,1.000,1.000,0.967,1,0.980,1.000,1.000,0,
single,fail_004,failure_expected,failure_expected,fact_lookup,document,ko,0,1,KGS AC999 임의 가스 코드,,,11647;11668;5177;11583;11692;13665;13661;11680;11644;13666,417.6,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_005,failure_expected,failure_expected,fact_lookup,document,ko,0,1,초전도 안전 관리법 시행규칙,,,35775;3966;3875;5227;4026;4025;30419;3971;4018;4021,897.2,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
1 label id category legacy_category intent domain_hint language ocr_derived failure_expected query relevant_ids graded_relevance returned_ids_top10 latency_ms recall_at_10 mrr_at_10 ndcg_at_10 top3_hit graded_ndcg_at_10 graded_recall_at_10_t2 graded_recall_at_10_t3 dedup_count error
2 single kw_001 standards exact_keyword fact_lookup document ko 0 0 산업안전보건법 제6장 3856;3868;3879 3856:3;3868:2;3879:2 30411;30423;3879;3868;3856;3851;4041;10573;5163;30406 659.8 1.000 0.333 0.618 0 0.529 1.000 1.000 0
3 single kw_002 standards exact_keyword fact_lookup document ko 0 0 중대재해 처벌 등에 관한 법률 제2장 중대산업재해 3917;3921 3917:3;3921:2 3921;3917;10573;5244;3919;3923;30418;5227;3916;3874 517.4 1.000 1.000 1.000 1 0.834 1.000 1.000 0
4 single kw_003 standards exact_keyword fact_lookup document ko 0 0 화학물질관리법 유해화학물질 영업자 3981 3981:3 3981;3985;3980;3984;3993;3869;30412;3857;3978;3986 336.1 1.000 1.000 1.000 1 1.000 1.000 1.000 0
5 single kw_004 standards exact_keyword fact_lookup document ko 0 0 근로기준법 안전과 보건 4041 4041:3 4041;3852;30407;5227;30421;30409;30406;10573;3851;3858 599.6 1.000 1.000 1.000 1 1.000 1.000 1.000 0
6 single kw_005 standards exact_keyword fact_lookup document ko 0 0 산업안전보건기준에 관한 규칙 보호구 3888 3888:3 10570;3888;5249;3893;3887;3904;3899;3890;3895;3902 629.5 1.000 0.500 0.631 1 0.631 1.000 1.000 0
7 single nl_001 korean_only natural_language_ko semantic_search document ko 0 0 기계로 인한 산업재해 관련 법령 3856;3868;3879;3854 3854:1;3856:3;3868:2;3879:2 5244;30422;30410;3878;5249;3855;3897;30421;3868;3863 664.7 0.250 0.111 0.118 0 0.083 0.333 0.000 0
8 single nl_002 korean_only natural_language_ko semantic_search document ko 0 0 사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일 3855;3867;3878 3855:3;3867:2;3878:2 30410;3855;5227;30409;3917;3854;5244;3867;30421;3878 545.5 1.000 0.500 0.580 1 0.599 1.000 1.000 0
9 single nl_003 korean_only natural_language_ko semantic_search document ko 0 0 유해화학물질을 다루는 회사가 지켜야 할 안전 의무 3980;3981;3982 3980:2;3981:2;3982:2 3980;3904;3903;3981;3905;35780;3985;10569;11648;3857 477.8 0.667 1.000 0.671 1 0.671 0.667 0.000 0
10 single nl_004 korean_only natural_language_ko semantic_search document ko 0 0 중대재해가 발생했을 때 경영책임자가 처벌받는 기준 3916;3917;3920;3921 3916:2;3917:3;3920:2;3921:2 10573;3917;3916;3918;5227;5244;3919;10571;30409;3854 391.5 0.500 0.500 0.441 1 0.506 0.500 1.000 0
11 single nl_005 korean_only natural_language_ko semantic_search document ko 0 0 안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가 3853;3865 3853:3;3865:2 3853;3876;5249;5234;30420;30408;30414;10573;11677;3867 708.8 0.500 1.000 0.613 1 0.787 0.500 1.000 0
12 single news_001 korean_only news_ko semantic_search news ko 0 0 이란과 미국의 군사 충돌 4303;4304;4307;4316;4322;4323;4327;4335 4303:2;4304:2;4307:2;4316:2;4322:2;4323:2;4327:2;4335:2 37599;27583;35225;27350;34850;32177;15924;15976;28316;16378 601.4 0.000 0.000 0.000 1 0.000 0.000 0.000 0
13 single news_002 korean_only news_ko semantic_search news ko 0 0 호르무즈 해협 봉쇄 4316;4320;4322;4327 4316:3;4320:2;4322:2;4327:2 41652;41650;41634;31894;35899;22055;9022;17123;11945;9105 402.3 0.000 0.000 0.000 0 0.000 0.000 0.000 0
14 single misc_001 korean_only other_domain fact_lookup document ko 0 0 강체의 평면 운동학 4063;4065 4063:3;4065:2 4063;4065;4066;4071;4067;4058;5105;4068;4064;5083 578.4 1.000 1.000 1.000 1 1.000 1.000 1.000 0
15 single misc_002 korean_only other_domain semantic_search document ko 0 0 질점의 운동역학 4060;4061;4062 4060:2;4061:2;4062:2 4070;4060;4062;4059;4064;4066;4058;4068;5083;4065 509.9 0.667 0.500 0.531 1 0.531 0.667 0.000 0
16 single news_003 english_only news_en semantic_search news en 0 0 Trump Iran ultimatum 4258;4260;4262 4258:2;4260:2;4262:2 21155;21441;4775;4202;4776;4679;4519;37733;25592;28001 339.8 0.000 0.000 0.000 1 0.000 0.000 0.000 0
17 single cl_001 mixed crosslingual_ko_en semantic_search document mixed 0 0 기계 안전 가드 설계 원리 3770;3856 3770:3;3856:2 35778;5239;5244;3758;3770;3763;4540;3787;3810;3791 699.5 0.500 0.200 0.237 0 0.305 0.500 1.000 0
18 single cl_002 mixed crosslingual_ko_en semantic_search document mixed 0 0 산업 안전 입문서 3755;3775;3776;3777 3755:2;3775:2;3776:2;3777:2 5230;5249;3755;3774;5229;10573;3851;3863;30406;3802 543.0 0.250 0.333 0.195 1 0.195 0.250 0.000 0
19 single cl_003 mixed crosslingual_ko_en semantic_search document mixed 0 0 전기 안전 위험 3772;3790 3772:2;3790:2 35779;5260;3772;3897;3790;37326;3755;37216;37011;37330 1114.4 1.000 0.333 0.544 1 0.544 1.000 0.000 0
20 single news_004 mixed news_fr semantic_search news mixed 0 0 guerre en Iran 4199;4202;4210;4361;4363;4507;4519;4521 4199:2;4202:2;4210:2;4361:2;4363:2;4507:2;4519:2;4521:2 33101;33368;7740;33374;27078;5925;26631;5840;16010;16457 426.3 0.000 0.000 0.000 1 0.000 0.000 0.000 0
21 single news_005 mixed news_crosslingual semantic_search news mixed 0 0 이란 미국 전쟁 글로벌 반응 4202;4258;4262;4536;4303;4304;4316 4202:2;4258:2;4262:2;4303:2;4304:2;4316:2;4536:2 23242;4457;19111;41629;26741;6789;17242;9469;4765;4345 839.7 0.000 0.000 0.000 1 0.000 0.000 0.000 0
22 single fail_001 failure_expected failure_expected semantic_search document mixed 0 1 Rust async runtime tokio scheduler 내부 구조 23732;36544;25771;35645;37582;17810;28766;40739;40432;40679 465.6 0.000 0.000 0.000 1 0.000 0.000 0.000 0
23 single fail_002 failure_expected failure_expected semantic_search document ko 0 1 양자컴퓨터 큐비트 디코히어런스 32096;31984;16289;16384;32043;32035;28233;33551;32721;32028 270.7 0.000 0.000 0.000 1 0.000 0.000 0.000 0
24 single fail_003 failure_expected failure_expected semantic_search news ko 0 1 재즈 보컬리스트 빌리 홀리데이 18567;20022;20470;35249;26594;16232;23336;35498;18286;16694 258.9 0.000 0.000 0.000 1 0.000 0.000 0.000 0
25 single kw_006 standards standards fact_lookup document ko 0 0 산업안전보건기준에 관한 규칙 작업장 통로 3886;3887 3886:3;3887:2 3898;3886;3887;3900;3881;3889;3912;3894;3897;3888 480.6 1.000 0.500 0.693 1 0.665 1.000 1.000 0
26 single kw_007 standards standards fact_lookup document ko 0 0 산업안전보건기준 폭발 화재 위험물 누출 방지 3896;3766 3766:1;3896:3 3896;3895;13930;3903;3897;13769;11679;11562;11567;10571 513.9 0.500 1.000 0.613 1 0.917 1.000 1.000 0
27 single kw_008 standards standards fact_lookup document ko 0 0 고압가스 안전관리법 전문 4025;4026 4025:3;4026:2 11644;11579;4025;4026;35775;11693;4034;13750;13299;13941 493.9 1.000 0.333 0.571 1 0.539 1.000 1.000 0
28 single kw_009 standards standards fact_lookup document ko 0 0 KGS FP111 가스설비 배관설비 기준 13305 13305:3 13305;13306;13311;13312;13302;13304;11688;13309;13313;13918 434.6 1.000 1.000 1.000 1 1.000 1.000 1.000 0
29 single kw_010 standards standards fact_lookup document ko 0 0 KGS FU551 가스설비 압력조정기 가스계량기 13652 13652:3 13652;11689;13657;13655;13649;13752;13648;13659;13650;13656 307.8 1.000 1.000 1.000 1 1.000 1.000 1.000 0
30 single kw_011 standards standards fact_lookup document ko 0 0 산업안전보건기준 전기로 인한 위험 방지 3897;3772 3772:1;3897:3 3897;3886;3887;13935;3895;3902;3896;3899;3877;3900 583.1 0.500 1.000 0.613 1 0.917 1.000 1.000 0
31 single en_001 english_only english_only semantic_search document en 0 0 pressure vessel flange design 5144;5136 5136:2;5144:3 5140;5136;5186;5144;5180;5212;5137;5193;36899;5182 4040.7 1.000 0.500 0.651 0 0.552 1.000 1.000 0
32 single en_002 english_only english_only semantic_search document en 0 0 ASME Section VIII introduction 5204;5206 5204:3;5206:2 5204;5212;5208;5210;5206;11634;5207;5183;5205;5182 2996.1 1.000 1.000 0.850 1 0.918 1.000 1.000 0
33 single en_003 english_only english_only semantic_search document en 0 0 impact test requirements ASME 5205;5148 5148:1;5205:3 36721;5205;5204;5178;5214;5224;5148;5186;5192;5185 5084.5 1.000 0.500 0.591 1 0.622 1.000 1.000 0
34 single en_004 english_only english_only semantic_search document en 0 0 design of vessel supports 5149 5149:3 5149;5186;5141;5136;5137;36901;36869;5139;5178;5145 5563.0 1.000 1.000 1.000 1 1.000 1.000 1.000 0
35 single en_005 english_only english_only semantic_search document en 0 0 hydrogen piping ASME code 5178 5178:3 5178;5180;5212;5182;5207;5210;5148;5204;5209;5224 4870.2 1.000 1.000 1.000 1 1.000 1.000 1.000 0
36 single en_006 english_only english_only semantic_search document en 0 0 ASME welding qualification requirements 5209;3771 3771:1;5209:3 5180;5178;5224;36721;36877;37097;5205;5204;36878;37051 2374.8 0.000 0.000 0.000 0 0.000 0.000 0.000 0
37 single en_007 english_only english_only semantic_search document en 0 0 pressure vessel fabrication and inspection 5208;5145 5145:2;5208:3 5186;5208;38747;38791;36935;36874;5187;5191;5182;5175 3752.4 0.500 0.500 0.387 1 0.497 0.500 1.000 0
38 single en_008 english_only english_only semantic_search document en 0 0 Industrial Safety and Health Management ergonomics 3763;3755 3755:1;3763:3 3759;3763;3774;3755;3818;5230;3812;3770;3756;3760 1817.5 1.000 0.500 0.651 1 0.635 1.000 1.000 0
39 single cl_004 mixed mixed semantic_search document mixed 0 0 ASME 압력용기 설계 실무 5207;5210;5139;5135 5135:2;5139:2;5207:3;5210:3 5139;5207;5204;5206;5225;5210;5133;5143;5141;5140 795.3 0.750 1.000 0.776 1 0.697 0.750 1.000 0
40 single cl_005 mixed mixed semantic_search document mixed 0 0 ASME 용접 코드 해설 5224;5209 5209:2;5224:3 5224;5222;5209;5225;5210;5208;5180;5204;5195;5205 724.0 1.000 1.000 0.920 1 0.956 1.000 1.000 0
41 single cl_006 mixed mixed semantic_search document mixed 0 0 pressure vessel Korean industrial safety regulation 4025;3856;5136 3856:2;4025:2;5136:1 38941;3762;4026;36889;36874;38791;5182;39084;5143;5186 1217.1 0.000 0.000 0.000 1 0.000 0.000 0.000 0
42 single cl_007 mixed mixed semantic_search document mixed 0 0 KGS 코드 LPG safety standard 11647;11689;11645;4025 4025:1;11645:2;11647:3;11689:2 11647;13674;13675;11688;11645;13310;13773;13759;13668;11652 512.6 0.500 1.000 0.541 1 0.754 0.667 1.000 0
43 single cl_008 mixed mixed semantic_search document mixed 0 0 수소 가스 안전 기준 hydrogen safety 5178;5169 5169:2;5178:3 10575;5177;13924;13925;5241;11562;5178;5173;11568;11563 838.4 0.500 0.143 0.204 0 0.262 0.500 1.000 0
44 single exam_001 exam exam fact_lookup document ko 0 0 레이놀즈수 정의 공식 11504;11505 11504:3;11505:2 11504;11533;5081;11482;5088;5090;5082;5077;11509;11505 590.3 1.000 1.000 0.790 1 0.885 1.000 1.000 0
45 single exam_002 exam exam fact_lookup document ko 0 0 탱크 바닥 구멍 유체 유속 토리첼리 11500;11495;11496 11495:2;11496:2;11500:3 11500;11495;11501;5085;5071;11515;5090;5139;5086;5088 387.2 0.667 1.000 0.765 1 0.856 0.667 1.000 0
46 single exam_003 exam exam fact_lookup document ko 0 0 이상기체 음속 마하수 11514;11515 11514:3;11515:2 11479;5083;5084;11476;11514;11516;11515;11518;11517;11472 645.5 1.000 0.200 0.442 0 0.417 1.000 1.000 0
47 single exam_004 exam exam fact_lookup document ko 0 0 고압가스 용기 내압시험 영구증가량 11591;11644;11691 11591:3;11644:2;11691:2 11591;11664;5177;13652;13660;13942;11660;13653;13752;11579 338.3 0.333 1.000 0.469 1 0.674 0.333 1.000 0
48 single exam_005 exam exam fact_lookup document ko 0 0 도시가스 배관 매설 이격거리 11627;11625;11646 11625:2;11627:3;11646:1 11627;11658;11625;11692;13753;13751;11655;13653;13752;11624 399.8 0.667 1.000 0.704 1 0.905 1.000 1.000 0
49 single exam_006 exam exam fact_lookup document ko 0 0 LPG 저장탱크 안전거리 분말소화기 11617;11669;11620 11617:3;11620:1;11669:2 11595;11616;13669;11617;11649;11605;11655;11690;11658;11653 353.2 0.333 0.250 0.202 0 0.321 0.500 1.000 0
50 single exam_007 exam exam fact_lookup document ko 0 0 오리피스 차압식 유량계 11712;11711;11503 11503:2;11711:2;11712:3 11712;11711;11500;11503;11713;13930;11701;11502;11715;11717 408.6 1.000 1.000 0.967 1 0.980 1.000 1.000 0
51 single fail_004 failure_expected failure_expected fact_lookup document ko 0 1 KGS AC999 임의 가스 코드 11647;11668;5177;11583;11692;13665;13661;11680;11644;13666 417.6 0.000 0.000 0.000 1 0.000 0.000 0.000 0
52 single fail_005 failure_expected failure_expected fact_lookup document ko 0 1 초전도 안전 관리법 시행규칙 35775;3966;3875;5227;4026;4025;30419;3971;4018;4021 897.2 0.000 0.000 0.000 1 0.000 0.000 0.000 0
@@ -0,0 +1,52 @@
label,id,category,legacy_category,intent,domain_hint,language,ocr_derived,failure_expected,query,relevant_ids,graded_relevance,returned_ids_top10,latency_ms,recall_at_10,mrr_at_10,ndcg_at_10,top3_hit,graded_ndcg_at_10,graded_recall_at_10_t2,graded_recall_at_10_t3,dedup_count,error
single,kw_001,standards,exact_keyword,fact_lookup,document,ko,0,0,산업안전보건법 제6장,3856;3868;3879,3856:3;3868:2;3879:2,30417;30418;30406;10573;3863;3871;3862;3874;3851;3859,437.7,0.000,0.000,0.000,0,0.000,0.000,0.000,0,
single,kw_002,standards,exact_keyword,fact_lookup,document,ko,0,0,중대재해 처벌 등에 관한 법률 제2장 중대산업재해,3917;3921,3917:3;3921:2,10573;3923;30418;3916;3921;3874;3917;3918;3920;3854,449.2,1.000,0.200,0.442,0,0.393,1.000,1.000,0,
single,kw_003,standards,exact_keyword,fact_lookup,document,ko,0,0,화학물질관리법 유해화학물질 영업자,3981,3981:3,3981;3985;3980;3978;3984;3869;3983;30424;3982;3903,320.2,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_004,standards,exact_keyword,fact_lookup,document,ko,0,0,근로기준법 안전과 보건,4041,4041:3,4041;30406;3851;10573;30419;30407;4036;3915;3852;3875,530.2,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_005,standards,exact_keyword,fact_lookup,document,ko,0,0,산업안전보건기준에 관한 규칙 보호구,3888,3888:3,3905;3888;3885;3910;3895;3911;3890;3906;3913;3889,605.2,1.000,0.500,0.631,1,0.631,1.000,1.000,0,
single,nl_001,korean_only,natural_language_ko,semantic_search,document,ko,0,0,기계로 인한 산업재해 관련 법령,3856;3868;3879;3854,3854:1;3856:3;3868:2;3879:2,3856;30411;30423;3879;5249;5244;3868;30421;30406;3851,644.8,0.750,1.000,0.689,1,0.859,1.000,1.000,0,
single,nl_002,korean_only,natural_language_ko,semantic_search,document,ko,0,0,사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일,3855;3867;3878,3855:3;3867:2;3878:2,3855;3867;30410;5227;3917;3878;30418;30422;3874;30421,538.4,1.000,1.000,0.933,1,0.958,1.000,1.000,0,
single,nl_003,korean_only,natural_language_ko,semantic_search,document,ko,0,0,유해화학물질을 다루는 회사가 지켜야 할 안전 의무,3980;3981;3982,3980:2;3981:2;3982:2,3981;3985;3903;3980;30424;3867;3904;3905;3880;3760,474.4,0.667,1.000,0.671,1,0.671,0.667,0.000,0,
single,nl_004,korean_only,natural_language_ko,semantic_search,document,ko,0,0,중대재해가 발생했을 때 경영책임자가 처벌받는 기준,3916;3917;3920;3921,3916:2;3917:3;3920:2;3921:2,3917;10573;3918;3916;10572;3872;10571;3923;30418;30417,403.9,0.500,1.000,0.559,1,0.710,0.500,1.000,0,
single,nl_005,korean_only,natural_language_ko,semantic_search,document,ko,0,0,안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가,3853;3865,3853:3;3865:2,5249;10573;30420;30408;3876;3853;3867;5234;30424;4842,711.7,0.500,0.167,0.218,0,0.280,0.500,1.000,0,
single,news_001,korean_only,news_ko,semantic_search,news,ko,0,0,이란과 미국의 군사 충돌,4303;4304;4307;4316;4322;4323;4327;4335,4303:2;4304:2;4307:2;4316:2;4322:2;4323:2;4327:2;4335:2,27583;35225;34298;30806;30679;27350;27974;32155;26534;26734,566.4,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,news_002,korean_only,news_ko,semantic_search,news,ko,0,0,호르무즈 해협 봉쇄,4316;4320;4322;4327,4316:3;4320:2;4322:2;4327:2,35904;35247;22055;16532;35893;26204;35899;16392;17123;4327,333.3,0.250,0.100,0.113,0,0.074,0.250,0.000,0,
single,misc_001,korean_only,other_domain,fact_lookup,document,ko,0,0,강체의 평면 운동학,4063;4065,4063:3;4065:2,4063;4071;4065;4064;4066;4058;5083;4067;5105;4068,468.5,1.000,1.000,0.920,1,0.956,1.000,1.000,0,
single,misc_002,korean_only,other_domain,semantic_search,document,ko,0,0,질점의 운동역학,4060;4061;4062,4060:2;4061:2;4062:2,4059;4060;4070;4062;4058;4061;5113;5083;4064;4066,382.4,1.000,0.500,0.665,1,0.665,1.000,0.000,0,
single,news_003,english_only,news_en,semantic_search,news,en,0,0,Trump Iran ultimatum,4258;4260;4262,4258:2;4260:2;4262:2,4775;37733;4776;4658;4679;4519;4202;4668;4333;25974,321.8,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,cl_001,mixed,crosslingual_ko_en,semantic_search,document,mixed,0,0,기계 안전 가드 설계 원리,3770;3856,3770:3;3856:2,35778;5239;5244;3758;3770;3763;3774;4540;5253;3787,776.2,0.500,0.200,0.237,0,0.305,0.500,1.000,0,
single,cl_002,mixed,crosslingual_ko_en,semantic_search,document,mixed,0,0,산업 안전 입문서,3755;3775;3776;3777,3755:2;3775:2;3776:2;3777:2,5230;5249;3756;3882;3774;37259;3863;3755;10573;3802,599.2,0.250,0.125,0.123,1,0.123,0.250,0.000,0,
single,cl_003,mixed,crosslingual_ko_en,semantic_search,document,mixed,0,0,전기 안전 위험,3772;3790,3772:2;3790:2,35779;3790;3772;5260;3897;37326;3755;37216;37011;3774,1205.5,1.000,0.500,0.693,1,0.693,1.000,0.000,0,
single,news_004,mixed,news_fr,semantic_search,news,mixed,0,0,guerre en Iran,4199;4202;4210;4361;4363;4507;4519;4521,4199:2;4202:2;4210:2;4361:2;4363:2;4507:2;4519:2;4521:2,33101;33368;7740;33374;27078;5925;26631;5840;16010;16457,494.7,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,news_005,mixed,news_crosslingual,semantic_search,news,mixed,0,0,이란 미국 전쟁 글로벌 반응,4202;4258;4262;4536;4303;4304;4316,4202:2;4258:2;4262:2;4303:2;4304:2;4316:2;4536:2,23242;15924;4457;36648;36490;19111;35556;41629;26631;26741,875.2,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_001,failure_expected,failure_expected,semantic_search,document,mixed,0,1,Rust async runtime tokio scheduler 내부 구조,,,23732;36544;35645;37582;17810;28766;40739;40432;40679;40700,502.6,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_002,failure_expected,failure_expected,semantic_search,document,ko,0,1,양자컴퓨터 큐비트 디코히어런스,,,32096;31984;33521;16289;16384;32043;32035;22202;28233;33551,342.0,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_003,failure_expected,failure_expected,semantic_search,news,ko,0,1,재즈 보컬리스트 빌리 홀리데이,,,18567;31643;20022;20470;16059;27146;26559;26637;18286;31187,289.1,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,kw_006,standards,standards,fact_lookup,document,ko,0,0,산업안전보건기준에 관한 규칙 작업장 통로,3886;3887,3886:3;3887:2,5236;3898;3886;3902;3895;3887;5244;3889;3912;3894,537.9,1.000,0.333,0.525,1,0.514,1.000,1.000,0,
single,kw_007,standards,standards,fact_lookup,document,ko,0,0,산업안전보건기준 폭발 화재 위험물 누출 방지,3896;3766,3766:1;3896:3,3896;3895;3903;13930;3897;13769;3766;13935;10571;3867,583.1,1.000,1.000,0.818,1,0.961,1.000,1.000,0,
single,kw_008,standards,standards,fact_lookup,document,ko,0,0,고압가스 안전관리법 전문,4025;4026,4025:3;4026:2,11644;11579;4026;4025;11693;35775;11645;4034;13299;13941,539.9,1.000,0.333,0.571,0,0.508,1.000,1.000,0,
single,kw_009,standards,standards,fact_lookup,document,ko,0,0,KGS FP111 가스설비 배관설비 기준,13305,13305:3,13305;13311;13306;13312;13302;13304;11688;13309;13313;13918,488.1,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_010,standards,standards,fact_lookup,document,ko,0,0,KGS FU551 가스설비 압력조정기 가스계량기,13652,13652:3,13652;11689;13657;13655;13649;13656;13648;13659;13752;13650,362.0,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_011,standards,standards,fact_lookup,document,ko,0,0,산업안전보건기준 전기로 인한 위험 방지,3897;3772,3772:1;3897:3,3897;3758;3886;3755;5260;3772;3887;13935;3895;3774,716.8,1.000,1.000,0.832,1,0.964,1.000,1.000,0,
single,en_001,english_only,english_only,semantic_search,document,en,0,0,pressure vessel flange design,5144;5136,5136:2;5144:3,5136;5178;5140;5180;5144;5193;36899;5182;5207;5206,4109.2,1.000,1.000,0.850,0,0.642,1.000,1.000,0,
single,en_002,english_only,english_only,semantic_search,document,en,0,0,ASME Section VIII introduction,5204;5206,5204:3;5206:2,5204;5212;5208;5206;5207;5183;5210;5205;5182;5148,3037.9,1.000,1.000,0.877,1,0.932,1.000,1.000,0,
single,en_003,english_only,english_only,semantic_search,document,en,0,0,impact test requirements ASME,5205;5148,5148:1;5205:3,5205;5204;36721;5225;5178;5224;37029;5214;5210;5190,5183.0,0.500,1.000,0.613,1,0.917,1.000,1.000,0,
single,en_004,english_only,english_only,semantic_search,document,en,0,0,design of vessel supports,5149,5149:3,5149;5186;5141;5136;5140;5137;36901;36869;5139;5178,5657.1,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,en_005,english_only,english_only,semantic_search,document,en,0,0,hydrogen piping ASME code,5178,5178:3,5143;5178;5180;5212;5182;5133;5207;5210;5148;5204,4902.9,1.000,0.500,0.631,1,0.631,1.000,1.000,0,
single,en_006,english_only,english_only,semantic_search,document,en,0,0,ASME welding qualification requirements,5209;3771,3771:1;5209:3,5204;5178;5224;36721;37097;5205;37051;36963;36951;5208,2455.9,0.000,0.000,0.000,0,0.000,0.000,0.000,0,
single,en_007,english_only,english_only,semantic_search,document,en,0,0,pressure vessel fabrication and inspection,5208;5145,5145:2;5208:3,5208;5186;38747;38791;36935;5180;5182;36874;5187;36889,3808.4,0.500,1.000,0.613,1,0.787,0.500,1.000,0,
single,en_008,english_only,english_only,semantic_search,document,en,0,0,Industrial Safety and Health Management ergonomics,3763;3755,3755:1;3763:3,3759;3763;3818;3774;3812;3760;3756;3778;3757;3775,1870.9,0.500,0.500,0.387,1,0.579,1.000,1.000,0,
single,cl_004,mixed,mixed,semantic_search,document,mixed,0,0,ASME 압력용기 설계 실무,5207;5210;5139;5135,5135:2;5139:2;5207:3;5210:3,5207;5204;5139;5225;5210;5182;5140;5148;5212;5144,832.1,0.750,1.000,0.737,1,0.789,0.750,1.000,0,
single,cl_005,mixed,mixed,semantic_search,document,mixed,0,0,ASME 용접 코드 해설,5224;5209,5209:2;5224:3,5224;5222;5204;5225;5210;5205;36877;5209;37051;36721,737.6,1.000,1.000,0.807,1,0.894,1.000,1.000,0,
single,cl_006,mixed,mixed,semantic_search,document,mixed,0,0,pressure vessel Korean industrial safety regulation,4025;3856;5136,3856:2;4025:2;5136:1,38941;3762;11644;4026;5186;38791;3895;39084;13299;5185,1267.9,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,cl_007,mixed,mixed,semantic_search,document,mixed,0,0,KGS 코드 LPG safety standard,11647;11689;11645;4025,4025:1;11645:2;11647:3;11689:2,11647;13674;13669;13774;13675;11688;13667;13670;11645;13310,558.8,0.500,1.000,0.508,1,0.730,0.667,1.000,0,
single,cl_008,mixed,mixed,semantic_search,document,mixed,0,0,수소 가스 안전 기준 hydrogen safety,5178;5169,5169:2;5178:3,5177;10572;11671;5241;11688;11653;3774;5178;5173;38701,879.9,0.500,0.125,0.193,0,0.248,0.500,1.000,0,
single,exam_001,exam,exam,fact_lookup,document,ko,0,0,레이놀즈수 정의 공식,11504;11505,11504:3;11505:2,11504;11533;5081;5087;11482;5088;5090;4544;11509;11505,629.4,1.000,1.000,0.790,1,0.885,1.000,1.000,0,
single,exam_002,exam,exam,fact_lookup,document,ko,0,0,탱크 바닥 구멍 유체 유속 토리첼리,11500;11495;11496,11495:2;11496:2;11500:3,11500;11495;11501;5085;5090;3788;5086;5071;11515;11503,405.7,0.667,1.000,0.765,1,0.856,0.667,1.000,0,
single,exam_003,exam,exam,fact_lookup,document,ko,0,0,이상기체 음속 마하수,11514;11515,11514:3;11515:2,11479;5083;11514;5084;11476;11527;11516;11515;5124;11472,691.4,1.000,0.333,0.500,1,0.500,1.000,1.000,0,
single,exam_004,exam,exam,fact_lookup,document,ko,0,0,고압가스 용기 내압시험 영구증가량,11591;11644;11691,11591:3;11644:2;11691:2,11591;13660;11668;5177;13652;13943;13942;13917;11592;13752,400.9,0.333,1.000,0.469,1,0.674,0.333,1.000,0,
single,exam_005,exam,exam,fact_lookup,document,ko,0,0,도시가스 배관 매설 이격거리,11627;11625;11646,11625:2;11627:3;11646:1,11627;11658;11600;11625;11692;13918;13751;5177;11655;13653,435.4,0.667,1.000,0.671,1,0.883,1.000,1.000,0,
single,exam_006,exam,exam,fact_lookup,document,ko,0,0,LPG 저장탱크 안전거리 분말소화기,11617;11669;11620,11617:3;11620:1;11669:2,11595;11616;11617;11649;13304;11655;11690;11658;13669;13915,417.2,0.333,0.333,0.235,1,0.373,0.500,1.000,0,
single,exam_007,exam,exam,fact_lookup,document,ko,0,0,오리피스 차압식 유량계,11712;11711;11503,11503:2;11711:2;11712:3,11712;11711;11503;11500;11713;13930;11502;11716;11612;11715,436.7,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,fail_004,failure_expected,failure_expected,fact_lookup,document,ko,0,1,KGS AC999 임의 가스 코드,,,11647;11668;5177;11688;11583;11692;13665;13661;11680;13666,447.7,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_005,failure_expected,failure_expected,fact_lookup,document,ko,0,1,초전도 안전 관리법 시행규칙,,,35775;11644;3966;4026;3977;4022;3972;13299;3974;3971,586.4,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
1 label id category legacy_category intent domain_hint language ocr_derived failure_expected query relevant_ids graded_relevance returned_ids_top10 latency_ms recall_at_10 mrr_at_10 ndcg_at_10 top3_hit graded_ndcg_at_10 graded_recall_at_10_t2 graded_recall_at_10_t3 dedup_count error
2 single kw_001 standards exact_keyword fact_lookup document ko 0 0 산업안전보건법 제6장 3856;3868;3879 3856:3;3868:2;3879:2 30417;30418;30406;10573;3863;3871;3862;3874;3851;3859 437.7 0.000 0.000 0.000 0 0.000 0.000 0.000 0
3 single kw_002 standards exact_keyword fact_lookup document ko 0 0 중대재해 처벌 등에 관한 법률 제2장 중대산업재해 3917;3921 3917:3;3921:2 10573;3923;30418;3916;3921;3874;3917;3918;3920;3854 449.2 1.000 0.200 0.442 0 0.393 1.000 1.000 0
4 single kw_003 standards exact_keyword fact_lookup document ko 0 0 화학물질관리법 유해화학물질 영업자 3981 3981:3 3981;3985;3980;3978;3984;3869;3983;30424;3982;3903 320.2 1.000 1.000 1.000 1 1.000 1.000 1.000 0
5 single kw_004 standards exact_keyword fact_lookup document ko 0 0 근로기준법 안전과 보건 4041 4041:3 4041;30406;3851;10573;30419;30407;4036;3915;3852;3875 530.2 1.000 1.000 1.000 1 1.000 1.000 1.000 0
6 single kw_005 standards exact_keyword fact_lookup document ko 0 0 산업안전보건기준에 관한 규칙 보호구 3888 3888:3 3905;3888;3885;3910;3895;3911;3890;3906;3913;3889 605.2 1.000 0.500 0.631 1 0.631 1.000 1.000 0
7 single nl_001 korean_only natural_language_ko semantic_search document ko 0 0 기계로 인한 산업재해 관련 법령 3856;3868;3879;3854 3854:1;3856:3;3868:2;3879:2 3856;30411;30423;3879;5249;5244;3868;30421;30406;3851 644.8 0.750 1.000 0.689 1 0.859 1.000 1.000 0
8 single nl_002 korean_only natural_language_ko semantic_search document ko 0 0 사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일 3855;3867;3878 3855:3;3867:2;3878:2 3855;3867;30410;5227;3917;3878;30418;30422;3874;30421 538.4 1.000 1.000 0.933 1 0.958 1.000 1.000 0
9 single nl_003 korean_only natural_language_ko semantic_search document ko 0 0 유해화학물질을 다루는 회사가 지켜야 할 안전 의무 3980;3981;3982 3980:2;3981:2;3982:2 3981;3985;3903;3980;30424;3867;3904;3905;3880;3760 474.4 0.667 1.000 0.671 1 0.671 0.667 0.000 0
10 single nl_004 korean_only natural_language_ko semantic_search document ko 0 0 중대재해가 발생했을 때 경영책임자가 처벌받는 기준 3916;3917;3920;3921 3916:2;3917:3;3920:2;3921:2 3917;10573;3918;3916;10572;3872;10571;3923;30418;30417 403.9 0.500 1.000 0.559 1 0.710 0.500 1.000 0
11 single nl_005 korean_only natural_language_ko semantic_search document ko 0 0 안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가 3853;3865 3853:3;3865:2 5249;10573;30420;30408;3876;3853;3867;5234;30424;4842 711.7 0.500 0.167 0.218 0 0.280 0.500 1.000 0
12 single news_001 korean_only news_ko semantic_search news ko 0 0 이란과 미국의 군사 충돌 4303;4304;4307;4316;4322;4323;4327;4335 4303:2;4304:2;4307:2;4316:2;4322:2;4323:2;4327:2;4335:2 27583;35225;34298;30806;30679;27350;27974;32155;26534;26734 566.4 0.000 0.000 0.000 1 0.000 0.000 0.000 0
13 single news_002 korean_only news_ko semantic_search news ko 0 0 호르무즈 해협 봉쇄 4316;4320;4322;4327 4316:3;4320:2;4322:2;4327:2 35904;35247;22055;16532;35893;26204;35899;16392;17123;4327 333.3 0.250 0.100 0.113 0 0.074 0.250 0.000 0
14 single misc_001 korean_only other_domain fact_lookup document ko 0 0 강체의 평면 운동학 4063;4065 4063:3;4065:2 4063;4071;4065;4064;4066;4058;5083;4067;5105;4068 468.5 1.000 1.000 0.920 1 0.956 1.000 1.000 0
15 single misc_002 korean_only other_domain semantic_search document ko 0 0 질점의 운동역학 4060;4061;4062 4060:2;4061:2;4062:2 4059;4060;4070;4062;4058;4061;5113;5083;4064;4066 382.4 1.000 0.500 0.665 1 0.665 1.000 0.000 0
16 single news_003 english_only news_en semantic_search news en 0 0 Trump Iran ultimatum 4258;4260;4262 4258:2;4260:2;4262:2 4775;37733;4776;4658;4679;4519;4202;4668;4333;25974 321.8 0.000 0.000 0.000 1 0.000 0.000 0.000 0
17 single cl_001 mixed crosslingual_ko_en semantic_search document mixed 0 0 기계 안전 가드 설계 원리 3770;3856 3770:3;3856:2 35778;5239;5244;3758;3770;3763;3774;4540;5253;3787 776.2 0.500 0.200 0.237 0 0.305 0.500 1.000 0
18 single cl_002 mixed crosslingual_ko_en semantic_search document mixed 0 0 산업 안전 입문서 3755;3775;3776;3777 3755:2;3775:2;3776:2;3777:2 5230;5249;3756;3882;3774;37259;3863;3755;10573;3802 599.2 0.250 0.125 0.123 1 0.123 0.250 0.000 0
19 single cl_003 mixed crosslingual_ko_en semantic_search document mixed 0 0 전기 안전 위험 3772;3790 3772:2;3790:2 35779;3790;3772;5260;3897;37326;3755;37216;37011;3774 1205.5 1.000 0.500 0.693 1 0.693 1.000 0.000 0
20 single news_004 mixed news_fr semantic_search news mixed 0 0 guerre en Iran 4199;4202;4210;4361;4363;4507;4519;4521 4199:2;4202:2;4210:2;4361:2;4363:2;4507:2;4519:2;4521:2 33101;33368;7740;33374;27078;5925;26631;5840;16010;16457 494.7 0.000 0.000 0.000 1 0.000 0.000 0.000 0
21 single news_005 mixed news_crosslingual semantic_search news mixed 0 0 이란 미국 전쟁 글로벌 반응 4202;4258;4262;4536;4303;4304;4316 4202:2;4258:2;4262:2;4303:2;4304:2;4316:2;4536:2 23242;15924;4457;36648;36490;19111;35556;41629;26631;26741 875.2 0.000 0.000 0.000 1 0.000 0.000 0.000 0
22 single fail_001 failure_expected failure_expected semantic_search document mixed 0 1 Rust async runtime tokio scheduler 내부 구조 23732;36544;35645;37582;17810;28766;40739;40432;40679;40700 502.6 0.000 0.000 0.000 1 0.000 0.000 0.000 0
23 single fail_002 failure_expected failure_expected semantic_search document ko 0 1 양자컴퓨터 큐비트 디코히어런스 32096;31984;33521;16289;16384;32043;32035;22202;28233;33551 342.0 0.000 0.000 0.000 1 0.000 0.000 0.000 0
24 single fail_003 failure_expected failure_expected semantic_search news ko 0 1 재즈 보컬리스트 빌리 홀리데이 18567;31643;20022;20470;16059;27146;26559;26637;18286;31187 289.1 0.000 0.000 0.000 1 0.000 0.000 0.000 0
25 single kw_006 standards standards fact_lookup document ko 0 0 산업안전보건기준에 관한 규칙 작업장 통로 3886;3887 3886:3;3887:2 5236;3898;3886;3902;3895;3887;5244;3889;3912;3894 537.9 1.000 0.333 0.525 1 0.514 1.000 1.000 0
26 single kw_007 standards standards fact_lookup document ko 0 0 산업안전보건기준 폭발 화재 위험물 누출 방지 3896;3766 3766:1;3896:3 3896;3895;3903;13930;3897;13769;3766;13935;10571;3867 583.1 1.000 1.000 0.818 1 0.961 1.000 1.000 0
27 single kw_008 standards standards fact_lookup document ko 0 0 고압가스 안전관리법 전문 4025;4026 4025:3;4026:2 11644;11579;4026;4025;11693;35775;11645;4034;13299;13941 539.9 1.000 0.333 0.571 0 0.508 1.000 1.000 0
28 single kw_009 standards standards fact_lookup document ko 0 0 KGS FP111 가스설비 배관설비 기준 13305 13305:3 13305;13311;13306;13312;13302;13304;11688;13309;13313;13918 488.1 1.000 1.000 1.000 1 1.000 1.000 1.000 0
29 single kw_010 standards standards fact_lookup document ko 0 0 KGS FU551 가스설비 압력조정기 가스계량기 13652 13652:3 13652;11689;13657;13655;13649;13656;13648;13659;13752;13650 362.0 1.000 1.000 1.000 1 1.000 1.000 1.000 0
30 single kw_011 standards standards fact_lookup document ko 0 0 산업안전보건기준 전기로 인한 위험 방지 3897;3772 3772:1;3897:3 3897;3758;3886;3755;5260;3772;3887;13935;3895;3774 716.8 1.000 1.000 0.832 1 0.964 1.000 1.000 0
31 single en_001 english_only english_only semantic_search document en 0 0 pressure vessel flange design 5144;5136 5136:2;5144:3 5136;5178;5140;5180;5144;5193;36899;5182;5207;5206 4109.2 1.000 1.000 0.850 0 0.642 1.000 1.000 0
32 single en_002 english_only english_only semantic_search document en 0 0 ASME Section VIII introduction 5204;5206 5204:3;5206:2 5204;5212;5208;5206;5207;5183;5210;5205;5182;5148 3037.9 1.000 1.000 0.877 1 0.932 1.000 1.000 0
33 single en_003 english_only english_only semantic_search document en 0 0 impact test requirements ASME 5205;5148 5148:1;5205:3 5205;5204;36721;5225;5178;5224;37029;5214;5210;5190 5183.0 0.500 1.000 0.613 1 0.917 1.000 1.000 0
34 single en_004 english_only english_only semantic_search document en 0 0 design of vessel supports 5149 5149:3 5149;5186;5141;5136;5140;5137;36901;36869;5139;5178 5657.1 1.000 1.000 1.000 1 1.000 1.000 1.000 0
35 single en_005 english_only english_only semantic_search document en 0 0 hydrogen piping ASME code 5178 5178:3 5143;5178;5180;5212;5182;5133;5207;5210;5148;5204 4902.9 1.000 0.500 0.631 1 0.631 1.000 1.000 0
36 single en_006 english_only english_only semantic_search document en 0 0 ASME welding qualification requirements 5209;3771 3771:1;5209:3 5204;5178;5224;36721;37097;5205;37051;36963;36951;5208 2455.9 0.000 0.000 0.000 0 0.000 0.000 0.000 0
37 single en_007 english_only english_only semantic_search document en 0 0 pressure vessel fabrication and inspection 5208;5145 5145:2;5208:3 5208;5186;38747;38791;36935;5180;5182;36874;5187;36889 3808.4 0.500 1.000 0.613 1 0.787 0.500 1.000 0
38 single en_008 english_only english_only semantic_search document en 0 0 Industrial Safety and Health Management ergonomics 3763;3755 3755:1;3763:3 3759;3763;3818;3774;3812;3760;3756;3778;3757;3775 1870.9 0.500 0.500 0.387 1 0.579 1.000 1.000 0
39 single cl_004 mixed mixed semantic_search document mixed 0 0 ASME 압력용기 설계 실무 5207;5210;5139;5135 5135:2;5139:2;5207:3;5210:3 5207;5204;5139;5225;5210;5182;5140;5148;5212;5144 832.1 0.750 1.000 0.737 1 0.789 0.750 1.000 0
40 single cl_005 mixed mixed semantic_search document mixed 0 0 ASME 용접 코드 해설 5224;5209 5209:2;5224:3 5224;5222;5204;5225;5210;5205;36877;5209;37051;36721 737.6 1.000 1.000 0.807 1 0.894 1.000 1.000 0
41 single cl_006 mixed mixed semantic_search document mixed 0 0 pressure vessel Korean industrial safety regulation 4025;3856;5136 3856:2;4025:2;5136:1 38941;3762;11644;4026;5186;38791;3895;39084;13299;5185 1267.9 0.000 0.000 0.000 1 0.000 0.000 0.000 0
42 single cl_007 mixed mixed semantic_search document mixed 0 0 KGS 코드 LPG safety standard 11647;11689;11645;4025 4025:1;11645:2;11647:3;11689:2 11647;13674;13669;13774;13675;11688;13667;13670;11645;13310 558.8 0.500 1.000 0.508 1 0.730 0.667 1.000 0
43 single cl_008 mixed mixed semantic_search document mixed 0 0 수소 가스 안전 기준 hydrogen safety 5178;5169 5169:2;5178:3 5177;10572;11671;5241;11688;11653;3774;5178;5173;38701 879.9 0.500 0.125 0.193 0 0.248 0.500 1.000 0
44 single exam_001 exam exam fact_lookup document ko 0 0 레이놀즈수 정의 공식 11504;11505 11504:3;11505:2 11504;11533;5081;5087;11482;5088;5090;4544;11509;11505 629.4 1.000 1.000 0.790 1 0.885 1.000 1.000 0
45 single exam_002 exam exam fact_lookup document ko 0 0 탱크 바닥 구멍 유체 유속 토리첼리 11500;11495;11496 11495:2;11496:2;11500:3 11500;11495;11501;5085;5090;3788;5086;5071;11515;11503 405.7 0.667 1.000 0.765 1 0.856 0.667 1.000 0
46 single exam_003 exam exam fact_lookup document ko 0 0 이상기체 음속 마하수 11514;11515 11514:3;11515:2 11479;5083;11514;5084;11476;11527;11516;11515;5124;11472 691.4 1.000 0.333 0.500 1 0.500 1.000 1.000 0
47 single exam_004 exam exam fact_lookup document ko 0 0 고압가스 용기 내압시험 영구증가량 11591;11644;11691 11591:3;11644:2;11691:2 11591;13660;11668;5177;13652;13943;13942;13917;11592;13752 400.9 0.333 1.000 0.469 1 0.674 0.333 1.000 0
48 single exam_005 exam exam fact_lookup document ko 0 0 도시가스 배관 매설 이격거리 11627;11625;11646 11625:2;11627:3;11646:1 11627;11658;11600;11625;11692;13918;13751;5177;11655;13653 435.4 0.667 1.000 0.671 1 0.883 1.000 1.000 0
49 single exam_006 exam exam fact_lookup document ko 0 0 LPG 저장탱크 안전거리 분말소화기 11617;11669;11620 11617:3;11620:1;11669:2 11595;11616;11617;11649;13304;11655;11690;11658;13669;13915 417.2 0.333 0.333 0.235 1 0.373 0.500 1.000 0
50 single exam_007 exam exam fact_lookup document ko 0 0 오리피스 차압식 유량계 11712;11711;11503 11503:2;11711:2;11712:3 11712;11711;11503;11500;11713;13930;11502;11716;11612;11715 436.7 1.000 1.000 1.000 1 1.000 1.000 1.000 0
51 single fail_004 failure_expected failure_expected fact_lookup document ko 0 1 KGS AC999 임의 가스 코드 11647;11668;5177;11688;11583;11692;13665;13661;11680;13666 447.7 0.000 0.000 0.000 1 0.000 0.000 0.000 0
52 single fail_005 failure_expected failure_expected fact_lookup document ko 0 1 초전도 안전 관리법 시행규칙 35775;11644;3966;4026;3977;4022;3972;13299;3974;3971 586.4 0.000 0.000 0.000 1 0.000 0.000 0.000 0
@@ -0,0 +1,52 @@
label,id,category,legacy_category,intent,domain_hint,language,ocr_derived,failure_expected,query,relevant_ids,graded_relevance,returned_ids_top10,latency_ms,recall_at_10,mrr_at_10,ndcg_at_10,top3_hit,graded_ndcg_at_10,graded_recall_at_10_t2,graded_recall_at_10_t3,dedup_count,error
single,kw_001,standards,exact_keyword,fact_lookup,document,ko,0,0,산업안전보건법 제6장,3856;3868;3879,3856:3;3868:2;3879:2,30411;3879;3868;3856;3851;10573;5249;3867;3873;3853,697.4,1.000,0.500,0.733,0,0.617,1.000,1.000,0,
single,kw_002,standards,exact_keyword,fact_lookup,document,ko,0,0,중대재해 처벌 등에 관한 법률 제2장 중대산업재해,3917;3921,3917:3;3921:2,3921;3917;10573;3919;3923;30418;5227;10571;3916;3874,513.6,1.000,1.000,1.000,1,0.834,1.000,1.000,0,
single,kw_003,standards,exact_keyword,fact_lookup,document,ko,0,0,화학물질관리법 유해화학물질 영업자,3981,3981:3,3981;3985;3984;3983;3904;3980;3903;3905;3978;3869,319.1,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_004,standards,exact_keyword,fact_lookup,document,ko,0,0,근로기준법 안전과 보건,4041,4041:3,4041;3852;30407;3851;30406;3915;10573;3858;5249;3863,603.5,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_005,standards,exact_keyword,fact_lookup,document,ko,0,0,산업안전보건기준에 관한 규칙 보호구,3888,3888:3,10570;3888;3912;3913;3908;3911;3905;5249;3910;3893,642.7,1.000,0.500,0.631,1,0.631,1.000,1.000,0,
single,nl_001,korean_only,natural_language_ko,semantic_search,document,ko,0,0,기계로 인한 산업재해 관련 법령,3856;3868;3879;3854,3854:1;3856:3;3868:2;3879:2,5244;5227;5249;3867;3878;3863;30411;3868;3856;3879,664.3,0.750,0.125,0.354,0,0.362,1.000,1.000,0,
single,nl_002,korean_only,natural_language_ko,semantic_search,document,ko,0,0,사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일,3855;3867;3878,3855:3;3867:2;3878:2,30410;3855;30409;3917;5244;30406;3851;3867;5249;30421,547.9,0.667,0.500,0.444,1,0.516,0.667,1.000,0,
single,nl_003,korean_only,natural_language_ko,semantic_search,document,ko,0,0,유해화학물질을 다루는 회사가 지켜야 할 안전 의무,3980;3981;3982,3980:2;3981:2;3982:2,3903;3982;3760;3905;3981;3904;3980;5249;3985;10569,466.6,1.000,0.500,0.634,1,0.634,1.000,0.000,0,
single,nl_004,korean_only,natural_language_ko,semantic_search,document,ko,0,0,중대재해가 발생했을 때 경영책임자가 처벌받는 기준,3916;3917;3920;3921,3916:2;3917:3;3920:2;3921:2,10572;10573;3917;3918;3916;5227;3919;10571;30409;3854,392.0,0.500,0.333,0.346,1,0.399,0.500,1.000,0,
single,nl_005,korean_only,natural_language_ko,semantic_search,document,ko,0,0,안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가,3853;3865,3853:3;3865:2,3853;5249;5234;4025;30420;30408;10573;11677;3876;3867,726.2,0.500,1.000,0.613,1,0.787,0.500,1.000,0,
single,news_001,korean_only,news_ko,semantic_search,news,ko,0,0,이란과 미국의 군사 충돌,4303;4304;4307;4316;4322;4323;4327;4335,4303:2;4304:2;4307:2;4316:2;4322:2;4323:2;4327:2;4335:2,37599;27583;35225;27350;34850;30518;15924;32410;16081;28407,593.2,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,news_002,korean_only,news_ko,semantic_search,news,ko,0,0,호르무즈 해협 봉쇄,4316;4320;4322;4327,4316:3;4320:2;4322:2;4327:2,41652;41650;41634;31894;35899;22055;16392;9022;17123;11945,399.4,0.000,0.000,0.000,0,0.000,0.000,0.000,0,
single,misc_001,korean_only,other_domain,fact_lookup,document,ko,0,0,강체의 평면 운동학,4063;4065,4063:3;4065:2,4063;4065;4066;4071;4064;4068;4067;4058;5083;4059,592.2,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,misc_002,korean_only,other_domain,semantic_search,document,ko,0,0,질점의 운동역학,4060;4061;4062,4060:2;4061:2;4062:2,4070;4061;4060;4062;4059;4065;4066;4058;4068;4071,512.9,1.000,0.500,0.733,1,0.733,1.000,0.000,0,
single,news_003,english_only,news_en,semantic_search,news,en,0,0,Trump Iran ultimatum,4258;4260;4262,4258:2;4260:2;4262:2,21155;4775;4202;4776;4679;4199;4519;4668;37733;4658,342.4,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,cl_001,mixed,crosslingual_ko_en,semantic_search,document,mixed,0,0,기계 안전 가드 설계 원리,3770;3856,3770:3;3856:2,35778;5239;5244;3758;3791;3770;3763;3774;4540;5253,683.2,0.500,0.167,0.218,0,0.280,0.500,1.000,0,
single,cl_002,mixed,crosslingual_ko_en,semantic_search,document,mixed,0,0,산업 안전 입문서,3755;3775;3776;3777,3755:2;3775:2;3776:2;3777:2,5230;5249;3756;3761;3757;3774;3863;3755;10573;3802,565.0,0.250,0.125,0.123,1,0.123,0.250,0.000,0,
single,cl_003,mixed,crosslingual_ko_en,semantic_search,document,mixed,0,0,전기 안전 위험,3772;3790,3772:2;3790:2,35779;3790;5260;3772;3897;37326;3755;37216;37011;3774,1136.0,1.000,0.500,0.651,1,0.651,1.000,0.000,0,
single,news_004,mixed,news_fr,semantic_search,news,mixed,0,0,guerre en Iran,4199;4202;4210;4361;4363;4507;4519;4521,4199:2;4202:2;4210:2;4361:2;4363:2;4507:2;4519:2;4521:2,33101;33368;7740;33374;27078;5925;5840;16010;16457;6945,423.2,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,news_005,mixed,news_crosslingual,semantic_search,news,mixed,0,0,이란 미국 전쟁 글로벌 반응,4202;4258;4262;4536;4303;4304;4316,4202:2;4258:2;4262:2;4303:2;4304:2;4316:2;4536:2,23242;4688;15924;4457;36648;36490;19111;35556;41629;26631,823.5,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_001,failure_expected,failure_expected,semantic_search,document,mixed,0,1,Rust async runtime tokio scheduler 내부 구조,,,36544;35645;37582;17810;28766;40739;40432;40679;40700;40734,461.6,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_002,failure_expected,failure_expected,semantic_search,document,ko,0,1,양자컴퓨터 큐비트 디코히어런스,,,32096;31984;33521;16289;16384;32043;32035;22202;28233;33551,253.5,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_003,failure_expected,failure_expected,semantic_search,news,ko,0,1,재즈 보컬리스트 빌리 홀리데이,,,18567;20022;20470;27146;26559;26637;18286;31187;26759;38352,245.5,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,kw_006,standards,standards,fact_lookup,document,ko,0,0,산업안전보건기준에 관한 규칙 작업장 통로,3886;3887,3886:3;3887:2,5236;3898;3886;3902;3895;3887;5244;3889;3912;3894,498.7,1.000,0.333,0.525,1,0.514,1.000,1.000,0,
single,kw_007,standards,standards,fact_lookup,document,ko,0,0,산업안전보건기준 폭발 화재 위험물 누출 방지,3896;3766,3766:1;3896:3,3896;3895;3903;13930;3897;13769;3766;13935;10571;3867,532.8,1.000,1.000,0.818,1,0.961,1.000,1.000,0,
single,kw_008,standards,standards,fact_lookup,document,ko,0,0,고압가스 안전관리법 전문,4025;4026,4025:3;4026:2,11644;11579;4026;4025;11693;35775;11645;4034;13299;13941,505.4,1.000,0.333,0.571,0,0.508,1.000,1.000,0,
single,kw_009,standards,standards,fact_lookup,document,ko,0,0,KGS FP111 가스설비 배관설비 기준,13305,13305:3,13305;13306;13311;13312;13302;13304;11688;13309;13313;13918,439.9,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_010,standards,standards,fact_lookup,document,ko,0,0,KGS FU551 가스설비 압력조정기 가스계량기,13652,13652:3,13652;11689;13657;13655;13649;13656;13651;13648;13752;13659,324.4,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,kw_011,standards,standards,fact_lookup,document,ko,0,0,산업안전보건기준 전기로 인한 위험 방지,3897;3772,3772:1;3897:3,3897;3758;3886;3755;5260;3772;3887;13935;3895;3774,873.6,1.000,1.000,0.832,1,0.964,1.000,1.000,0,
single,en_001,english_only,english_only,semantic_search,document,en,0,0,pressure vessel flange design,5144;5136,5136:2;5144:3,5136;5178;5140;5180;5144;36899;5182;5207;5206;5149,4311.0,1.000,1.000,0.850,0,0.642,1.000,1.000,0,
single,en_002,english_only,english_only,semantic_search,document,en,0,0,ASME Section VIII introduction,5204;5206,5204:3;5206:2,5204;5212;5208;5210;5206;5141;5207;5205;5182;5148,11067.7,1.000,1.000,0.850,1,0.918,1.000,1.000,0,
single,en_003,english_only,english_only,semantic_search,document,en,0,0,impact test requirements ASME,5205;5148,5148:1;5205:3,5205;5204;36721;5225;5178;5224;37029;5214;5210;5190,5144.1,0.500,1.000,0.613,1,0.917,1.000,1.000,0,
single,en_004,english_only,english_only,semantic_search,document,en,0,0,design of vessel supports,5149,5149:3,5149;5186;5141;5140;5136;5137;36901;36869;5139;5210,5560.7,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,en_005,english_only,english_only,semantic_search,document,en,0,0,hydrogen piping ASME code,5178,5178:3,5143;5178;5180;5212;5182;5207;5210;5148;5204;5209,4909.7,1.000,0.500,0.631,1,0.631,1.000,1.000,0,
single,en_006,english_only,english_only,semantic_search,document,en,0,0,ASME welding qualification requirements,5209;3771,3771:1;5209:3,5180;5204;5224;36721;37097;5205;37051;36963;5178;36951,2380.2,0.000,0.000,0.000,0,0.000,0.000,0.000,0,
single,en_007,english_only,english_only,semantic_search,document,en,0,0,pressure vessel fabrication and inspection,5208;5145,5145:2;5208:3,5186;5208;38747;38791;36935;5180;5182;36874;5187;36897,3795.7,0.500,0.500,0.387,1,0.497,0.500,1.000,0,
single,en_008,english_only,english_only,semantic_search,document,en,0,0,Industrial Safety and Health Management ergonomics,3763;3755,3755:1;3763:3,3759;3763;3818;3774;3812;3757;3756;3775;3762;37280,1851.3,0.500,0.500,0.387,1,0.579,1.000,1.000,0,
single,cl_004,mixed,mixed,semantic_search,document,mixed,0,0,ASME 압력용기 설계 실무,5207;5210;5139;5135,5135:2;5139:2;5207:3;5210:3,5207;5204;5139;5225;5210;5182;5140;5212;5144;5224,783.3,0.750,1.000,0.737,1,0.789,0.750,1.000,0,
single,cl_005,mixed,mixed,semantic_search,document,mixed,0,0,ASME 용접 코드 해설,5224;5209,5209:2;5224:3,5224;5222;5204;5225;5210;5182;5195;5205;36877;5209,704.0,1.000,1.000,0.790,1,0.885,1.000,1.000,0,
single,cl_006,mixed,mixed,semantic_search,document,mixed,0,0,pressure vessel Korean industrial safety regulation,4025;3856;5136,3856:2;4025:2;5136:1,38941;3762;11644;4026;5186;38791;3895;39084;5204;13299,1227.3,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,cl_007,mixed,mixed,semantic_search,document,mixed,0,0,KGS 코드 LPG safety standard,11647;11689;11645;4025,4025:1;11645:2;11647:3;11689:2,11647;13674;13669;13774;13675;11688;13769;13667;13670;11645,525.2,0.500,1.000,0.503,1,0.727,0.667,1.000,0,
single,cl_008,mixed,mixed,semantic_search,document,mixed,0,0,수소 가스 안전 기준 hydrogen safety,5178;5169,5169:2;5178:3,10575;5177;10572;5241;11688;11653;5178;11693;5173;11563,838.0,0.500,0.143,0.204,0,0.262,0.500,1.000,0,
single,exam_001,exam,exam,fact_lookup,document,ko,0,0,레이놀즈수 정의 공식,11504;11505,11504:3;11505:2,11504;11533;5081;5087;11482;5088;5090;5082;11509;11505,576.2,1.000,1.000,0.790,1,0.885,1.000,1.000,0,
single,exam_002,exam,exam,fact_lookup,document,ko,0,0,탱크 바닥 구멍 유체 유속 토리첼리,11500;11495;11496,11495:2;11496:2;11500:3,11500;11495;11501;5090;5085;3788;5086;5071;5075;11515,377.2,0.667,1.000,0.765,1,0.856,0.667,1.000,0,
single,exam_003,exam,exam,fact_lookup,document,ko,0,0,이상기체 음속 마하수,11514;11515,11514:3;11515:2,11479;5083;11514;5084;11527;11516;11515;5124;11518;11471,650.0,1.000,0.333,0.511,1,0.506,1.000,1.000,0,
single,exam_004,exam,exam,fact_lookup,document,ko,0,0,고압가스 용기 내압시험 영구증가량,11591;11644;11691,11591:3;11644:2;11691:2,11591;11664;11668;5177;13943;13942;13917;11592;13752;4026,354.7,0.333,1.000,0.469,1,0.674,0.333,1.000,0,
single,exam_005,exam,exam,fact_lookup,document,ko,0,0,도시가스 배관 매설 이격거리,11627;11625;11646,11625:2;11627:3;11646:1,11627;11658;11600;11625;11692;13918;13751;5177;11655;13653,393.6,0.667,1.000,0.671,1,0.883,1.000,1.000,0,
single,exam_006,exam,exam,fact_lookup,document,ko,0,0,LPG 저장탱크 안전거리 분말소화기,11617;11669;11620,11617:3;11620:1;11669:2,11595;11616;11617;11649;13304;11655;11690;11658;13669;13915,393.7,0.333,0.333,0.235,1,0.373,0.500,1.000,0,
single,exam_007,exam,exam,fact_lookup,document,ko,0,0,오리피스 차압식 유량계,11712;11711;11503,11503:2;11711:2;11712:3,11712;11711;11503;11500;11713;11502;11716;11612;11715;11717,397.8,1.000,1.000,1.000,1,1.000,1.000,1.000,0,
single,fail_004,failure_expected,failure_expected,fact_lookup,document,ko,0,1,KGS AC999 임의 가스 코드,,,11647;11668;5177;11688;11583;11692;13665;13661;13664;11680,405.8,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
single,fail_005,failure_expected,failure_expected,fact_lookup,document,ko,0,1,초전도 안전 관리법 시행규칙,,,35775;11644;3966;4026;3977;3972;13299;3974;3971;4025,548.8,0.000,0.000,0.000,1,0.000,0.000,0.000,0,
1 label id category legacy_category intent domain_hint language ocr_derived failure_expected query relevant_ids graded_relevance returned_ids_top10 latency_ms recall_at_10 mrr_at_10 ndcg_at_10 top3_hit graded_ndcg_at_10 graded_recall_at_10_t2 graded_recall_at_10_t3 dedup_count error
2 single kw_001 standards exact_keyword fact_lookup document ko 0 0 산업안전보건법 제6장 3856;3868;3879 3856:3;3868:2;3879:2 30411;3879;3868;3856;3851;10573;5249;3867;3873;3853 697.4 1.000 0.500 0.733 0 0.617 1.000 1.000 0
3 single kw_002 standards exact_keyword fact_lookup document ko 0 0 중대재해 처벌 등에 관한 법률 제2장 중대산업재해 3917;3921 3917:3;3921:2 3921;3917;10573;3919;3923;30418;5227;10571;3916;3874 513.6 1.000 1.000 1.000 1 0.834 1.000 1.000 0
4 single kw_003 standards exact_keyword fact_lookup document ko 0 0 화학물질관리법 유해화학물질 영업자 3981 3981:3 3981;3985;3984;3983;3904;3980;3903;3905;3978;3869 319.1 1.000 1.000 1.000 1 1.000 1.000 1.000 0
5 single kw_004 standards exact_keyword fact_lookup document ko 0 0 근로기준법 안전과 보건 4041 4041:3 4041;3852;30407;3851;30406;3915;10573;3858;5249;3863 603.5 1.000 1.000 1.000 1 1.000 1.000 1.000 0
6 single kw_005 standards exact_keyword fact_lookup document ko 0 0 산업안전보건기준에 관한 규칙 보호구 3888 3888:3 10570;3888;3912;3913;3908;3911;3905;5249;3910;3893 642.7 1.000 0.500 0.631 1 0.631 1.000 1.000 0
7 single nl_001 korean_only natural_language_ko semantic_search document ko 0 0 기계로 인한 산업재해 관련 법령 3856;3868;3879;3854 3854:1;3856:3;3868:2;3879:2 5244;5227;5249;3867;3878;3863;30411;3868;3856;3879 664.3 0.750 0.125 0.354 0 0.362 1.000 1.000 0
8 single nl_002 korean_only natural_language_ko semantic_search document ko 0 0 사업주가 도급을 줄 때 산업재해를 예방하기 위해 해야 할 일 3855;3867;3878 3855:3;3867:2;3878:2 30410;3855;30409;3917;5244;30406;3851;3867;5249;30421 547.9 0.667 0.500 0.444 1 0.516 0.667 1.000 0
9 single nl_003 korean_only natural_language_ko semantic_search document ko 0 0 유해화학물질을 다루는 회사가 지켜야 할 안전 의무 3980;3981;3982 3980:2;3981:2;3982:2 3903;3982;3760;3905;3981;3904;3980;5249;3985;10569 466.6 1.000 0.500 0.634 1 0.634 1.000 0.000 0
10 single nl_004 korean_only natural_language_ko semantic_search document ko 0 0 중대재해가 발생했을 때 경영책임자가 처벌받는 기준 3916;3917;3920;3921 3916:2;3917:3;3920:2;3921:2 10572;10573;3917;3918;3916;5227;3919;10571;30409;3854 392.0 0.500 0.333 0.346 1 0.399 0.500 1.000 0
11 single nl_005 korean_only natural_language_ko semantic_search document ko 0 0 안전보건교육은 누가 받아야 하고 어떤 내용을 다루는가 3853;3865 3853:3;3865:2 3853;5249;5234;4025;30420;30408;10573;11677;3876;3867 726.2 0.500 1.000 0.613 1 0.787 0.500 1.000 0
12 single news_001 korean_only news_ko semantic_search news ko 0 0 이란과 미국의 군사 충돌 4303;4304;4307;4316;4322;4323;4327;4335 4303:2;4304:2;4307:2;4316:2;4322:2;4323:2;4327:2;4335:2 37599;27583;35225;27350;34850;30518;15924;32410;16081;28407 593.2 0.000 0.000 0.000 1 0.000 0.000 0.000 0
13 single news_002 korean_only news_ko semantic_search news ko 0 0 호르무즈 해협 봉쇄 4316;4320;4322;4327 4316:3;4320:2;4322:2;4327:2 41652;41650;41634;31894;35899;22055;16392;9022;17123;11945 399.4 0.000 0.000 0.000 0 0.000 0.000 0.000 0
14 single misc_001 korean_only other_domain fact_lookup document ko 0 0 강체의 평면 운동학 4063;4065 4063:3;4065:2 4063;4065;4066;4071;4064;4068;4067;4058;5083;4059 592.2 1.000 1.000 1.000 1 1.000 1.000 1.000 0
15 single misc_002 korean_only other_domain semantic_search document ko 0 0 질점의 운동역학 4060;4061;4062 4060:2;4061:2;4062:2 4070;4061;4060;4062;4059;4065;4066;4058;4068;4071 512.9 1.000 0.500 0.733 1 0.733 1.000 0.000 0
16 single news_003 english_only news_en semantic_search news en 0 0 Trump Iran ultimatum 4258;4260;4262 4258:2;4260:2;4262:2 21155;4775;4202;4776;4679;4199;4519;4668;37733;4658 342.4 0.000 0.000 0.000 1 0.000 0.000 0.000 0
17 single cl_001 mixed crosslingual_ko_en semantic_search document mixed 0 0 기계 안전 가드 설계 원리 3770;3856 3770:3;3856:2 35778;5239;5244;3758;3791;3770;3763;3774;4540;5253 683.2 0.500 0.167 0.218 0 0.280 0.500 1.000 0
18 single cl_002 mixed crosslingual_ko_en semantic_search document mixed 0 0 산업 안전 입문서 3755;3775;3776;3777 3755:2;3775:2;3776:2;3777:2 5230;5249;3756;3761;3757;3774;3863;3755;10573;3802 565.0 0.250 0.125 0.123 1 0.123 0.250 0.000 0
19 single cl_003 mixed crosslingual_ko_en semantic_search document mixed 0 0 전기 안전 위험 3772;3790 3772:2;3790:2 35779;3790;5260;3772;3897;37326;3755;37216;37011;3774 1136.0 1.000 0.500 0.651 1 0.651 1.000 0.000 0
20 single news_004 mixed news_fr semantic_search news mixed 0 0 guerre en Iran 4199;4202;4210;4361;4363;4507;4519;4521 4199:2;4202:2;4210:2;4361:2;4363:2;4507:2;4519:2;4521:2 33101;33368;7740;33374;27078;5925;5840;16010;16457;6945 423.2 0.000 0.000 0.000 1 0.000 0.000 0.000 0
21 single news_005 mixed news_crosslingual semantic_search news mixed 0 0 이란 미국 전쟁 글로벌 반응 4202;4258;4262;4536;4303;4304;4316 4202:2;4258:2;4262:2;4303:2;4304:2;4316:2;4536:2 23242;4688;15924;4457;36648;36490;19111;35556;41629;26631 823.5 0.000 0.000 0.000 1 0.000 0.000 0.000 0
22 single fail_001 failure_expected failure_expected semantic_search document mixed 0 1 Rust async runtime tokio scheduler 내부 구조 36544;35645;37582;17810;28766;40739;40432;40679;40700;40734 461.6 0.000 0.000 0.000 1 0.000 0.000 0.000 0
23 single fail_002 failure_expected failure_expected semantic_search document ko 0 1 양자컴퓨터 큐비트 디코히어런스 32096;31984;33521;16289;16384;32043;32035;22202;28233;33551 253.5 0.000 0.000 0.000 1 0.000 0.000 0.000 0
24 single fail_003 failure_expected failure_expected semantic_search news ko 0 1 재즈 보컬리스트 빌리 홀리데이 18567;20022;20470;27146;26559;26637;18286;31187;26759;38352 245.5 0.000 0.000 0.000 1 0.000 0.000 0.000 0
25 single kw_006 standards standards fact_lookup document ko 0 0 산업안전보건기준에 관한 규칙 작업장 통로 3886;3887 3886:3;3887:2 5236;3898;3886;3902;3895;3887;5244;3889;3912;3894 498.7 1.000 0.333 0.525 1 0.514 1.000 1.000 0
26 single kw_007 standards standards fact_lookup document ko 0 0 산업안전보건기준 폭발 화재 위험물 누출 방지 3896;3766 3766:1;3896:3 3896;3895;3903;13930;3897;13769;3766;13935;10571;3867 532.8 1.000 1.000 0.818 1 0.961 1.000 1.000 0
27 single kw_008 standards standards fact_lookup document ko 0 0 고압가스 안전관리법 전문 4025;4026 4025:3;4026:2 11644;11579;4026;4025;11693;35775;11645;4034;13299;13941 505.4 1.000 0.333 0.571 0 0.508 1.000 1.000 0
28 single kw_009 standards standards fact_lookup document ko 0 0 KGS FP111 가스설비 배관설비 기준 13305 13305:3 13305;13306;13311;13312;13302;13304;11688;13309;13313;13918 439.9 1.000 1.000 1.000 1 1.000 1.000 1.000 0
29 single kw_010 standards standards fact_lookup document ko 0 0 KGS FU551 가스설비 압력조정기 가스계량기 13652 13652:3 13652;11689;13657;13655;13649;13656;13651;13648;13752;13659 324.4 1.000 1.000 1.000 1 1.000 1.000 1.000 0
30 single kw_011 standards standards fact_lookup document ko 0 0 산업안전보건기준 전기로 인한 위험 방지 3897;3772 3772:1;3897:3 3897;3758;3886;3755;5260;3772;3887;13935;3895;3774 873.6 1.000 1.000 0.832 1 0.964 1.000 1.000 0
31 single en_001 english_only english_only semantic_search document en 0 0 pressure vessel flange design 5144;5136 5136:2;5144:3 5136;5178;5140;5180;5144;36899;5182;5207;5206;5149 4311.0 1.000 1.000 0.850 0 0.642 1.000 1.000 0
32 single en_002 english_only english_only semantic_search document en 0 0 ASME Section VIII introduction 5204;5206 5204:3;5206:2 5204;5212;5208;5210;5206;5141;5207;5205;5182;5148 11067.7 1.000 1.000 0.850 1 0.918 1.000 1.000 0
33 single en_003 english_only english_only semantic_search document en 0 0 impact test requirements ASME 5205;5148 5148:1;5205:3 5205;5204;36721;5225;5178;5224;37029;5214;5210;5190 5144.1 0.500 1.000 0.613 1 0.917 1.000 1.000 0
34 single en_004 english_only english_only semantic_search document en 0 0 design of vessel supports 5149 5149:3 5149;5186;5141;5140;5136;5137;36901;36869;5139;5210 5560.7 1.000 1.000 1.000 1 1.000 1.000 1.000 0
35 single en_005 english_only english_only semantic_search document en 0 0 hydrogen piping ASME code 5178 5178:3 5143;5178;5180;5212;5182;5207;5210;5148;5204;5209 4909.7 1.000 0.500 0.631 1 0.631 1.000 1.000 0
36 single en_006 english_only english_only semantic_search document en 0 0 ASME welding qualification requirements 5209;3771 3771:1;5209:3 5180;5204;5224;36721;37097;5205;37051;36963;5178;36951 2380.2 0.000 0.000 0.000 0 0.000 0.000 0.000 0
37 single en_007 english_only english_only semantic_search document en 0 0 pressure vessel fabrication and inspection 5208;5145 5145:2;5208:3 5186;5208;38747;38791;36935;5180;5182;36874;5187;36897 3795.7 0.500 0.500 0.387 1 0.497 0.500 1.000 0
38 single en_008 english_only english_only semantic_search document en 0 0 Industrial Safety and Health Management ergonomics 3763;3755 3755:1;3763:3 3759;3763;3818;3774;3812;3757;3756;3775;3762;37280 1851.3 0.500 0.500 0.387 1 0.579 1.000 1.000 0
39 single cl_004 mixed mixed semantic_search document mixed 0 0 ASME 압력용기 설계 실무 5207;5210;5139;5135 5135:2;5139:2;5207:3;5210:3 5207;5204;5139;5225;5210;5182;5140;5212;5144;5224 783.3 0.750 1.000 0.737 1 0.789 0.750 1.000 0
40 single cl_005 mixed mixed semantic_search document mixed 0 0 ASME 용접 코드 해설 5224;5209 5209:2;5224:3 5224;5222;5204;5225;5210;5182;5195;5205;36877;5209 704.0 1.000 1.000 0.790 1 0.885 1.000 1.000 0
41 single cl_006 mixed mixed semantic_search document mixed 0 0 pressure vessel Korean industrial safety regulation 4025;3856;5136 3856:2;4025:2;5136:1 38941;3762;11644;4026;5186;38791;3895;39084;5204;13299 1227.3 0.000 0.000 0.000 1 0.000 0.000 0.000 0
42 single cl_007 mixed mixed semantic_search document mixed 0 0 KGS 코드 LPG safety standard 11647;11689;11645;4025 4025:1;11645:2;11647:3;11689:2 11647;13674;13669;13774;13675;11688;13769;13667;13670;11645 525.2 0.500 1.000 0.503 1 0.727 0.667 1.000 0
43 single cl_008 mixed mixed semantic_search document mixed 0 0 수소 가스 안전 기준 hydrogen safety 5178;5169 5169:2;5178:3 10575;5177;10572;5241;11688;11653;5178;11693;5173;11563 838.0 0.500 0.143 0.204 0 0.262 0.500 1.000 0
44 single exam_001 exam exam fact_lookup document ko 0 0 레이놀즈수 정의 공식 11504;11505 11504:3;11505:2 11504;11533;5081;5087;11482;5088;5090;5082;11509;11505 576.2 1.000 1.000 0.790 1 0.885 1.000 1.000 0
45 single exam_002 exam exam fact_lookup document ko 0 0 탱크 바닥 구멍 유체 유속 토리첼리 11500;11495;11496 11495:2;11496:2;11500:3 11500;11495;11501;5090;5085;3788;5086;5071;5075;11515 377.2 0.667 1.000 0.765 1 0.856 0.667 1.000 0
46 single exam_003 exam exam fact_lookup document ko 0 0 이상기체 음속 마하수 11514;11515 11514:3;11515:2 11479;5083;11514;5084;11527;11516;11515;5124;11518;11471 650.0 1.000 0.333 0.511 1 0.506 1.000 1.000 0
47 single exam_004 exam exam fact_lookup document ko 0 0 고압가스 용기 내압시험 영구증가량 11591;11644;11691 11591:3;11644:2;11691:2 11591;11664;11668;5177;13943;13942;13917;11592;13752;4026 354.7 0.333 1.000 0.469 1 0.674 0.333 1.000 0
48 single exam_005 exam exam fact_lookup document ko 0 0 도시가스 배관 매설 이격거리 11627;11625;11646 11625:2;11627:3;11646:1 11627;11658;11600;11625;11692;13918;13751;5177;11655;13653 393.6 0.667 1.000 0.671 1 0.883 1.000 1.000 0
49 single exam_006 exam exam fact_lookup document ko 0 0 LPG 저장탱크 안전거리 분말소화기 11617;11669;11620 11617:3;11620:1;11669:2 11595;11616;11617;11649;13304;11655;11690;11658;13669;13915 393.7 0.333 0.333 0.235 1 0.373 0.500 1.000 0
50 single exam_007 exam exam fact_lookup document ko 0 0 오리피스 차압식 유량계 11712;11711;11503 11503:2;11711:2;11712:3 11712;11711;11503;11500;11713;11502;11716;11612;11715;11717 397.8 1.000 1.000 1.000 1 1.000 1.000 1.000 0
51 single fail_004 failure_expected failure_expected fact_lookup document ko 0 1 KGS AC999 임의 가스 코드 11647;11668;5177;11688;11583;11692;13665;13661;13664;11680 405.8 0.000 0.000 0.000 1 0.000 0.000 0.000 0
52 single fail_005 failure_expected failure_expected fact_lookup document ko 0 1 초전도 안전 관리법 시행규칙 35775;11644;3966;4026;3977;3972;13299;3974;3971;4025 548.8 0.000 0.000 0.000 1 0.000 0.000 0.000 0
+80
View File
@@ -0,0 +1,80 @@
"""B-4 — licensed_restricted 차단 술어 + watch 타깃 (material/jurisdiction/license) 매핑 순수 테스트.
차단 술어(_license_sql) retrieval 3-leg + digest 공유하는 단일 술어. 실제 제외 동작은
GPU 라이브(합성 restricted doc 검색 제외) 검증 여기선 술어 형태 + 매핑 계약만.
[[feedback_external_api_fixture_first]] / [[feedback_structural_integrity_over_path_discipline]]
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent / "app"))
from services.search.license_filter import ( # noqa: E402
restricted_exclude_orm,
restricted_exclude_sql,
)
from services.search.retrieval_service import _license_sql # noqa: E402
from workers.file_watcher import _TARGET_AXIS # noqa: E402
def test_shared_predicate_single_source():
# retrieval/digest/briefing 가 같은 술어 정의를 공유 — drift 방지(단일 source 계약)
assert _license_sql("d") == " AND " + restricted_exclude_sql("d")
assert _license_sql("") == " AND " + restricted_exclude_sql("")
assert restricted_exclude_sql("d").startswith("COALESCE(d.extract_meta")
def test_restricted_exclude_orm_constructs():
# study 풀이(explanation_rag)용 ORM 표현 — 컴파일 SQL 이 raw 술어와 동일 구조인지
from sqlalchemy.dialects import postgresql
clause = restricted_exclude_orm()
sql = str(clause.compile(dialect=postgresql.dialect(),
compile_kwargs={"literal_binds": True}))
assert "extract_meta" in sql
assert "'license'" in sql and "'restricted'" in sql # JSONB 경로 키
assert "'false'" in sql and "'true'" in sql # COALESCE 기본 + 비교값
def test_license_sql_shape_with_alias():
sql = _license_sql("d")
assert sql.startswith(" AND ") # 항상 ' AND ...' (WHERE 합성용)
assert "COALESCE(d.extract_meta -> 'license' ->> 'restricted', 'false')" in sql
assert "<> 'true'" in sql # restricted=true 만 제외
def test_license_sql_shape_no_alias():
# alias='' = 단일 FROM documents (컬럼 직접 참조)
sql = _license_sql("")
assert "COALESCE(extract_meta -> 'license' ->> 'restricted', 'false')" in sql
assert ".extract_meta" not in sql # 점 없는 컬럼 직접
def test_axis_books_papers_are_restricted():
for folder, mt in (("Books", "book"), ("Papers_Purchased", "paper")):
material, jur, lic = _TARGET_AXIS[folder]
assert material == mt
assert jur is None # 책/논문 = 관할 없음(A-2 paper NULL 강제와 정합)
assert lic["scheme"] == "proprietary"
assert lic["restricted"] is True # RAG/digest 차단 대상
assert lic["redistribute"] is False
def test_axis_manuals_proprietary_but_not_restricted():
material, jur, lic = _TARGET_AXIS["Manuals"]
assert material == "manual"
assert lic["scheme"] == "proprietary"
assert lic["restricted"] is False # 사용자 결정 2026-06-13 (검색·RAG 활용)
def test_axis_kgs_law_kr_public_not_restricted():
material, jur, lic = _TARGET_AXIS["KGS_Code"]
assert (material, jur) == ("law", "KR")
assert lic["scheme"] == "kogl"
assert lic["restricted"] is False # 법정 위임 공공 → 차단 아님
def test_axis_non_target_folder_yields_none():
# Inbox/Recordings 등 비대상 = (None, None, None) → material/license 미주입
assert _TARGET_AXIS.get("Inbox", (None, None, None)) == (None, None, None)

Some files were not shown because too many files have changed in this diff Show More