Compare commits

...

28 Commits

Author SHA1 Message Date
hyungi ebbcaf86d8 feat(observability): 큐 밖 백그라운드 작업(backfill)을 처리 머신 보드에 노출
processing_queue 는 파이프라인 stage 전용이라 hier_overnight_backfill 같은 off-queue
관리 스크립트 작업이 대시보드 보드에 안 잡혀, 다른 세션이 모르고 fastapi 를 재생성해
in-flight 재분해를 끊는 사고가 발생(2026-06-14). 사각지대 해소.

- migrations/357_background_jobs.sql: background_jobs 테이블(kind/label/state/processed/
  total/heartbeat). worker_jobs(user_id 필수, worker-pool 전용)와 별개.
- services/background_jobs.py: start/heartbeat/finish 헬퍼 — 자율 트랜잭션(즉시 commit →
  실시간 가시화) + best-effort(관측 실패가 본작업 안 깸).
- hier_overnight_backfill: 작업 시작/절 ~10개마다 heartbeat/종료 계측.
- queue_overview: /api/queue/overview 응답에 background_jobs 추가(running + 최근 6h 완료,
  stale=heartbeat 끊김 추정). SAVEPOINT 로 테이블 부재/오류 시 보드 본체 무영향.
- ProcessingFlowBoard: "백그라운드 작업" 패널(진행/경과/state, stale 끊김 경고).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-14 12:27:18 +09:00
Claude Code 6d978289b8 feat(papers): B-3 P2-PR1 oa_url 승격 분기 (arXiv 외 doi.org/KISTI/PMC OA)
arxiv_id 없는 OA 논문(oa_status gold/hybrid/green/diamond + oa_url)도 전문 승격 대상에 포함.
url = arxiv.org/pdf 또는 oa_url(friendly OA host). paywall/비-PDF 는 헤더검증서 skip(실패 격리).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 03:16:47 +00:00
Claude Code 73c6f123b8 feat(papers): B-3 P2-PR1 — arXiv 논문 전문 in-place 승격 + classify paper 요약-스킵
plan safety-library-b3-1 Phase-2. 논문을 초록(signal-only)에서 전문 md/검색으로 승격.
- paper_fulltext_promote.py: 미승격 arXiv 논문(file_format='article') → arxiv.org/pdf/{id} 다운로드
  (kosha 패턴·50MB cap·PDF 헤더검증) → NAS crawl_raw/papers/arxiv/ → in-place 갱신
  (file_format=pdf·file_type=immutable·file_path·md_status=pending, file_hash·extract_meta.paper 보존)
  → 'extract' enqueue. 1-Document(2행 분리 회피, 기존 display 스택 재사용). per-run cap 10(GPU 보호).
  arXiv=공개 프리프린트라 전문 검색/RAG 무난(restricted 불요; 유료 구매분만 Papers_Purchased restricted).
- classify_worker: material_type='paper' 가드 추가 — 요약/분류 LLM 스킵(맥미니 큐 무접촉),
  queue_consumer 가 embed/chunk/markdown 은 chain. law_monitor 스킵 패턴 동형.

CLI 전용(Phase-2 deliberate 승격·GPU 부하 사용자 통제). 파이프라인=extract→classify[skip]→embed/chunk/markdown,
marker 표시 md + hier 절구조 + 전문 검색 청크. 배포 후 라이브 검증.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-14 03:04:02 +00:00
hyungi 57c1805a8d Merge pull request 'Feat/safety library b3' (#39) from feat/safety-library-b3 into main
Reviewed-on: #39
2026-06-14 08:05:09 +09:00
hyungi cbdd4a3df7 Merge pull request 'Feat/docpage open as default' (#38) from feat/docpage-open-as-default into main
Reviewed-on: #38
2026-06-14 08:04:59 +09:00
Claude Code bf0348a3e0 feat(papers): B-3 PR5 — 구매 PDF parent_doi 스탬프 (paper_doi_reconcile 통합)
plan safety-library-b3-1 PR5. Papers_Purchased 수동 드롭 PDF(license.restricted=true)를 서지 holder 에
연결: 본문 DOI 파싱 → paper.parent_doi 링크(child, doi 미보유=인덱스 밖, unique 무충돌).
- doi.py: parse_doi_from_text(본문 전체 DOI 정규식 — PDF 구조 무관).
- paper_doi_reconcile: restricted 분기 — restricted 행은 본문 DOI→parent_doi(child),
  그 외(레거시 arXiv)는 holder 스탬프(PR4). 쿼리에 parent_doi IS NULL 추가(링크분 재처리 회피).
- file_watcher merge-only license 주입 clobber-safe 존중. enqueue 0(콘텐츠 무변경).

단위 29 passed(+parse_doi_from_text). ephemeral PASS: 합성 restricted 행 → parent_doi 링크·
paper.doi 부재·restricted 보존·스키마 수용(insert+rollback). reconcile 멱등(재실행 0 변경).
실 구매 PDF 라이브 검증 = 사용자 첫 논문 구매·드롭 시(로직 검증 완료).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 22:58:19 +00:00
Claude Code 244d526ae2 feat(papers): B-3 PR4 — 레거시 arXiv DOI reconcile + arXiv DataCite DOI 통일 (keyless)
plan safety-library-b3-1 PR4. paper.doi 없는 paper 행을 arXiv DataCite DOI 로 스탬프해
partial-unique 인덱스 편입 → 재유입 차단('동일-DOI 재유입 차단만').
- doi.py: parse_arxiv_id(본문→arXiv id) + arxiv_doi(10.48550/arxiv.{id}, OpenAlex canonical 실측 일치).
- ★arXiv DOI 통일: arxiv_collector 도 프리프린트(저널 DOI 없음)에 arxiv_doi 부여 → PR2/PR3/PR4 가 같은
  함수로 같은 paper.doi → 교차소스 dedup 성립(이전엔 프리프린트 paper.doi 부재로 PR2↔PR3 dup 갭).
- paper_doi_reconcile.py: 전용 worker(dedup_reconcile=file_hash 캐시와 별개 — 적대리뷰 B·C major).
  keyless·결정적(OpenAlex 호출 0)·in-DB·enqueue 0(콘텐츠 무변경). 선재 DOI holder 시 parent_doi
  마킹(unique 위반 회피). add_job daily 03:50 KST. __main__ CLI.

단위 28 passed(+parse_arxiv_id·arxiv_doi). 라이브 PASS (prod, running fastapi 무접촉):
레거시 197행 arXiv DataCite 스탬프·ASME 2행 skip·선재중복 0 / dedup 불변식 206 distinct 206(인덱스 무위반) /
paper summarize active 0(signal-only). 멱등.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 22:54:24 +00:00
hyungi c5bc1f773d fix(docpage): 비인접 window 를 parent_id 로 split-parent 에 흡수 (빈 본문 절 수정)
split-parent(절 헤딩)와 그 window 조각이 chunk_index 상 비인접인 경우(예: 5180 FOREWORD
헤딩 idx 1143, window idx 1233~)가 있어, 인접 흡수만 하던 collapseWindows 가 split-parent 를
빈 본문 행으로 남기고 window 들은 따로 대표 행을 만들어 "같은 제목 2행(빈 것 + 본문 있는 것)" 이
됐다. 사용자가 "본문 없는 절" 로 본 것.

- /sections API 에 parent_id 반환 (window.parent_id = 그 split-parent chunk_id, 100% _split 링크)
- collapseWindows 가 window 를 parent_id 로 split-parent 대표에 흡수(비인접 허용), 인접 heading
  fallback 유지(legacy window). 흡수 멤버에서 본문/분석 집계.
- 회귀 테스트: 비인접 parent_id 흡수 (12/12 pass)

실데이터 검증(빈 본문→0): 5180 outline 85→58·5210 318→277·5178 73→49·5151 45→40, 전부 EMPTY_BODY=0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-14 07:46:18 +09:00
Claude Code fdabca2a2f feat(papers): B-3 PR6 — OpenAlex ISSN 소스 시드 (KR/JP 안전 저널 직접 커버)
plan safety-library-b3-1 PR6 (revised). 라이브 정찰: KoreaScience=깨끗한 API 없음(OAI 404)·
J-STAGE=ToS bulk 금지, 그리고 Phase-1 메타는 OpenAlex 가 이미 전수 색인(한국안전학회지 1766건 실측)
→ 전용 스크래퍼 대신 검증된 OpenAlex 수집기에 도메인 저널 ISSN 시드 추가(전용 무료 전문 PDF=Phase-2 park).
- _JOURNAL_ISSNS(OpenAlex sources 실측): 한국안전학회지 1738-3803·한국가스학회지 1226-8402·
  KSME A/B 1226-4873·1226-4881·KSME Intl 1226-4865·JP 고압 0917-639X.
- _seeds() = ISSN 시드(cap 우선) + 키워드. build_issn_filter(primary_location.source.issn:).
  run() 루프 통합(종류별 필터, 워터마크 시드별). 적재/parser/cap/signal-only = PR3 재사용.

단위 8 passed(+ISSN 시드). 라이브 PASS: 키주입 run → 한국안전학회지 5건 적재(ISSN 우선 확인),
running fastapi 무접촉. KoreaScience/J-STAGE 전용 fulltext 수집기 = Phase-2 강등(park).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 22:42:30 +00:00
Claude Code 1fbb341e28 feat(papers): B-3 PR3 — OpenAlex 백본 수집기 (scaffold-first, signal-only, per-run cap)
plan safety-library-b3-1 PR3. 발견+dedup 글로벌 백본(JP/EU/US 색인+정본 DOI, 전문 안 줌).
- scaffold-first: OPENALEX_API_KEY 부재 시 FeedError explicit-skip(silent fallback 0). 키=무료.
- signal-only: inverted-index 초록 복원→색인(embed+chunk), summarize 0. PDF 절대 미fetch(oa_url=신호).
- 관련성 사전필터=title_and_abstract.search 키워드 + per-run cap 60(임베드 firehose 차단, 적대리뷰 A major)
  + cursor 페이징 + from_publication_date 워터마크 증분. 초록 없는 thin 레코드 skip(재료 품질).
- license: 명시 CC→redistribute true / OA·closed→false(restricted 부재=초록 RAG 사용가능, 비-CC 전문은 L-1 Phase-2).
- DOI→paper.doi(holder, 교차소스 dedup) / 없으면 openalex_id. enabled=False 행+add_job(daily 07:45 KST)+CLI.

순수 파서/초록복원/license_meta fixture 단위 7 passed(OpenAlex 실응답: cc-by/cc-by-nc-nd/None·초록 유무).
라이브 검증 PASS (prod, running fastapi 무접촉): 키없음→explicit-skip / 키주입→3건 적재
(paper/NULL/ai_summary NULL/region INT, cc-by→redist true·unspecified→false, green/gold,
큐 embed3+chunk3·summarize 0, distinct openalex_id=total, 교차소스 DOI 4 distinct 4 중복 0).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 22:30:36 +00:00
hyungi d007ad5492 fix(docpage): windowed 절에 조각별 분석(유형/신뢰도/요약) 집계 노출
절-레벨 분석(chunk_section_analysis)은 코퍼스 전역에 이미 있으나(절 보유 344문서 중 336)
window 조각의 chunk_id 에 붙어 있고, D3 는 window 를 split-parent 대표로 collapse 하며 버려서
windowed 절은 요약/유형/신뢰도가 안 떴다(분석은 대표가 아닌 조각에 있음).

- collapseWindows 가 멤버(대표+흡수 window)에서 절-레벨 분석 집계:
  sectionType=다수결(동률 첫등장) · confidence=평균 · summaries=조각 요약 배열(빈 것 제외)
- D3 트리/focus/모바일카드/이전다음이 it.sectionType/it.confidence/it.summaries 사용
- 요약은 단일 절=문단, windowed 절="절 요약 · N개 부분" 번호목록
- headingPath.test.ts: 집계 회귀 테스트 추가 (11/11 pass)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-14 07:27:41 +09:00
Claude Code 6167e03625 fix(papers): B-3 PR2 arxiv _record_success 4-arg 시그니처 (라이브 검증서 발견)
run() 종료 시 _record_success(health, now) → 누락 인자(items·not_modified) 추가
= _record_success(health, inserted, False, now) (news_collector 시그니처 일치).
일회성 compose run 라이브 검증서 TypeError 로 발견 — 배포 전 차단.

라이브 검증 PASS (prod 6건 적재, running fastapi 무접촉): material_type=paper·jurisdiction NULL·
ai_summary NULL·crawl·region=INT·license=arxiv / DOI 보유 1건 paper.doi 인덱스 진입·나머지 arxiv_id /
큐 embed6+chunk6·summarize 0(signal-only) / distinct arxiv_id=총건(dedup 불변식) / health circuit closed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 22:22:51 +00:00
hyungi b6a4821cac fix(docpage): 절 본문을 청크 text로 렌더 + window 조각 collapse
대형 split 문서는 marker 가 md_content 를 앞 5만 자만 보존하고 char_start 도 NULL 이라
char_start 슬라이스로는 절 본문이 비었다. 전체 본문은 document_chunks.text 에 절별로 보존됨.

- /sections API 가 청크 text 반환 (SectionItem.text; 소비자=D3 단독, additive)
- collapseWindows 가 window 조각 본문을 대표 절 bodyText 로 합본 (split-parent heading 제외)
- D3 페이지가 outline(collapseWindows) 단위로 렌더 → window 파편화 제거
  (5180 = 27 논리 절이 562 동일제목 조각으로 쪼개지던 문제)
- useSectionView=hasSections 로 단순화(partial/대형 문서도 절뷰), 모바일 본문 lazy 파싱
- headingPath.test.ts: bodyText 누적 회귀 테스트 추가 (10/10 pass)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-14 07:10:59 +09:00
Claude Code ba943d703a feat(papers): B-3 PR2 — arXiv 키워드 필터 수집기 (signal-only, per-run cap)
plan safety-library-b3-1 PR2 (keyless). DOI 코어(PR1) 위 첫 실수집기.
- bespoke arXiv API(Atom) 수집기: cat:{category} AND (abs:키워드) — RSS 통째(firehose) 아님.
  신규 7 카테고리(eess.SY·physics.flu-dyn/comp-ph·math.OC/NA·stat.AP·cs.CE) x 압력용기/공정안전 키워드.
- signal-only: 초록만 색인(embed+chunk), summarize 절대 미enqueue(맥미니 큐 무접촉).
- DOI 보유 -> extract_meta.paper.doi(holder, partial-unique 인덱스). 없으면 arXiv id dedup.
  교차소스 dedup = find_paper_holder(PR1) + arxiv id file_hash. paper.source_region=INT(jurisdiction NULL 유지).
- per-run insert cap(_RUN_CAP=80) — 광역 수집이 GPU embed 큐 범람 방지(적대리뷰 A major), 잔여 로깅.
- etiquette: >=3s + 429 백오프 + 카테고리별 submittedDate 워터마크 증분. https 필수(http=301).
- enabled=False news_sources 행 + main.py CronTrigger(daily 07:30 KST). __main__ CLI(--bulk/--limit).

순수 파서·쿼리빌더 fixture 단위 18 passed(arxiv 실응답 박제: DOI/journal_ref/둘다없음 3경로).
적재(run/_ingest_entry)는 news_collector signal-only 패턴 미러 — 배포 후 라이브 검증.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 22:10:25 +00:00
Claude Code 345e2cedf0 feat(papers): B-3 PR1 — DOI 정규화·dedup 코어 (normalize_doi 단일 함수 + 서지 holder 조회)
plan safety-library-b3-1 PR1 (keyless·마이그 0). 모든 논문 수집기·reconcile·구매 스탬프 공유 토대.
- normalize_doi(): 소문자·URL/doi: prefix 제거·인용 구두점(.,;) 정리. 저장=조회 단일 함수.
  괄호 '()' 보존 — 과삭제는 다른 논문 병합(데이터 손상)이라 near-dup 보다 위험.
- paper_doi_hash(): 서지 holder file_hash 키 = sha256('paper|{doi}')[:32] (statute 다중부 키 선례).
- with_paper_doi/with_parent_doi/read_paper_doi: 2-Document 계약(holder doi / child parent_doi 상호배타) extract_meta 헬퍼 (merge-safe).
- find_paper_holder(): 공유 dedup 조회 — lower(extract_meta #>> '{paper,doi}'), .scalars().first()(BBC 다중행 선례),
  EXPLAIN 으로 uq_documents_paper_doi(마이그 351 라이브) 인덱스 사용 확인.

단위 12 passed. holder DB 조회 = PR2 arXiv 실수집서 라이브 검증. 소비자 없는 순수 코드(배포·런타임 변화 0).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-13 21:50:09 +00:00
hyungi b461559d2f fix(docpage): 절 없는 문서도 인사이트 항상 표시 (fallback 개선)
사용자 "절이 없더라도 인사이트는 보여야지" — fallback(절 데이터 없는 ~92% 문서)이
모바일에서 인사이트 레일을 긴 본문 아래에 묻던 문제 수정. bodyViewer 스니펫 분리 후:
- 모바일: 인사이트 레일을 본문 위에 상시 표시
- 데스크탑: 본문 | 인사이트 레일(sticky)
(별개: 절 트리/집중 뷰는 절 분석 있는 문서에서만 활성 — 현재 4358중 333. 커버리지 확대는 후속.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 16:39:44 +09:00
hyungi 9b9790f05d fix(docpage): D3 시안 스타일 그대로 포팅 + 모바일 길이/접근성 수정
사용자 "시안대로 안했다" → 앱 토큰 재해석을 폐기하고 d3-deepened 시안의 inline
스타일을 그대로 포팅(데이터만 바인딩): 트리 좌측 색바(3×16)+연결선(ㄴ자)+활성+
저신뢰 맥동배지, 절차색 #7a8b3f, 헤더 PDF아이콘+pill칩+분류/원본/링크/관리, 절 집중
뷰(요건 requirement 배지·신뢰도 바·절요약 인용박스), 슬림 레일 카드(시안 동일).
모바일: 절구조/인사이트 안보임+무한길이("쭉 아래까지") → pill sticky + 절 본문
카드마다 접기('본문 보기', 기본 요약만)로 컴팩트화. svelte-check 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 16:29:04 +09:00
hyungi b49596135e fix(docpage): 모바일을 확정 시안 그대로 — 나란한 토글 pill + 패널 + 본문 연속
직전 모바일이 세로 details 2개라 시안(나란한 pill 토글)과 불일치
(사용자 "시안에 모바일용도 있잖아 그걸 안 만들었다") → d3-deepened 모바일 프래그먼트
충실 복제: 절 구조|인사이트 나란한 pill(기본 둘 다 접힘) + 절 구조 패널(유형 범례·
점프 링크·저신뢰·들여쓰기) + 인사이트 패널(TL;DR·핵심점·심층DEEP·불일치·분류·태그) +
본문 절 카드 연속(#m-sec 앵커, pill 탭→본문 이동). svelte-check 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 16:12:59 +09:00
hyungi 0a82a5b1bc feat(docpage): 모바일을 시안대로 — 본문 연속 절 카드 + 접이 + 탭 이동
기존 모바일(데스크탑 focus 단일절)이 시안 모바일과 불일치(사용자 "모바일은 변한게
없잖아") → 시안 모바일 충실 구현:
- 모바일(<xl) = 절 구조/인사이트 접이(기본 절구조 닫힘·인사이트 열림) + 본문이 절
  카드로 연속(각 절 제목·유형배지·절요약·본문) + 절 구조 탭하면 #m-sec 본문 앵커 이동
- 데스크탑(xl+) = 트리 | 절 집중 | 레일 (focusView 스니펫으로 분리)
- treeNav(jumpMode): 데스크탑=절 선택 / 모바일=앵커 점프
svelte-check 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 16:03:59 +09:00
hyungi 74e29e510e feat(docpage): D3 상세 페이지를 확정 시안 그대로 재구현
기존 컴포넌트 재사용/배치변경(불충실)을 폐기하고 deepened 시안을 충실히 구현:
- 좌 절 트리: 유형 색칩(정의/절차/요건)·신뢰도 dot·저신뢰 경고·레벨 들여쓰기·클릭=절 선택
- 중 절 집중 뷰: breadcrumb + 제목 + 유형 배지 + 신뢰도 막대 + 절 요약 인용 + 절 본문
  (md_content 를 char_start 로 슬라이스) + 이전/다음 절
- 우 슬림 레일: TL;DR · 핵심점 · 심층(DEEP) · 불일치 · 분류 · 태그 (읽기) + 정보/관리 접이(편집 보존)
- 절 없음 fallback: 전체 본문/뷰어 + 레일 (D3 빈 절 graceful)
- 모바일: 본문(절 집중) 메인 + 절구조/인사이트 접이
svelte-check 0. 시안=comparisons/2026-06-13-ds-docpage-d3-deepened.html.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 15:53:34 +09:00
hyungi c1555fd6ab feat(docpage): 전체 문서 목록 클릭 시 인라인 미리보기 대신 D3 상세로 이동
사용자 결정 "개선된 페이지가 앞으로 표시되야지" — /documents 브라우저에서
문서를 열면 인라인 DocumentViewer(구) 대신 개선된 /documents/[id](D3 절 구조
탐색기)로 이동. /documents = 브라우즈/검색/필터/일괄 목록(풀폭 중앙) 역할로 정리:
- selectDoc → goto(/documents/[id]) (행 클릭·키보드 enter 공통)
- 인라인 리더(DocumentViewer)·인스펙터 패널 제거, 목록 max-w-5xl 중앙
- AI 답변 카드(질문형 검색)는 목록 상단 고정으로 이동(보존)
- 검색·필터칩·일괄작업·업로드·페이지네이션 전부 유지

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 15:41:59 +09:00
hyungi 1d5755b279 Merge pull request 'feat(docpage): D3 절 구조 탐색기 — 슬림 인사이트 레일 + 절 트리 (frontend only)' (#37) from feat/ds-docpage-d3 into main
Reviewed-on: #37
2026-06-13 15:23:05 +09:00
hyungi a3e0d30569 Merge pull request 'Feat/ds board merged' (#36) from feat/ds-board-merged into main
Reviewed-on: #36
2026-06-13 15:22:53 +09:00
hyungi 540bc00dba feat(docpage): D3 절 구조 탐색기 — 슬림 인사이트 레일 + 절 트리 (frontend only)
문서 상세 /documents/[id] 재구성 (BE 무변경):
- 우측 탭(정보/AI/관리) → 슬림 전역 인사이트 레일: 요약·심층·불일치를 탭
  게이트 없이 상시 노출(details open, 모바일은 접기 가능), 정보/관리는 접이.
  → 가공 자료가 탭/온디맨드에 묻히던 IA 문제(G1) 해소.
- SectionOutline 절 목차 레벨 기반 들여쓰기(평탄→트리 모양).
- 모바일: 본문 메인 + 절목차/인사이트/정보/관리 접이 + 절 탭 본문 이동(기존 구조 활용).
관련 문서(See Also)는 v1 제외(자리만 유지). 심화 목업 = comparisons/2026-06-13-ds-docpage-d3-deepened.html.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 15:18:15 +09:00
hyungi 30c235e4c1 Merge feat/safety-library-a1 (C-1 후속 version_status+facets) into ds-board-merged
검색 결과 wrapper decoration: 법령 version_status + facets 집계(ranking 무관·additive).
2026-06-13 15:08:24 +09:00
hyungi cd439b0ff4 Merge feat/safety-library-a1 (B-4 licensed_restricted + watch 폴더 license) into ds-board-merged
B-4 PR①②: licensed_restricted 단일 술어(retrieval 3-leg/digest/briefing/study 풀이 공유)
+ file_watcher Books/Manuals/Papers_Purchased license 주입. prod 통합 브랜치 배포용.
2026-06-13 14:53:34 +09:00
hyungi 595f4b7d5e feat(board): 통합 보드 v3 — 머신 레인 + 정직 번다운/ETA (B-1·2·3·5)
ProcessingFlowBoard 를 통합안으로 재작성:
- 머신 3레인(GPU/맥미니/맥북) = "누가 일하나" + 요약 오프로드 가시화
  (요약 칩 분담 막대 맥미니 vs 맥북 + 맥북 레인 '요약 합류' 칩, summarize_by_machine 소비)
- 지배 백로그 스트립 + 정직 ETA(summarize_eta, 유입 차감 / null=소진 불가)
- 24h 번다운 SVG(유입 vs 소화) + 맥북 합류 변곡점 + 단계별 정직 ETA 미니리스트
- 신선도 '갱신 N초 전' + stale 경고(queueUpdatedAt, B-4)
- 실패 드로어 + 노드 상세 패널은 v2 자산 그대로 재사용
- 레인 stack + 칩 wrap 으로 모바일 반응형

svelte-check: 변경 파일 에러 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 14:01:50 +09:00
hyungi b630c31077 feat(board): expose summarize_by_machine for offload visibility (A-1)
요약 풀의 머신별 완료 실적(맥미니 vs 맥북)을 /api/queue/overview 응답에
summarize_by_machine 로 노출. rows_to_summarize_split 이 이미 계산하던 값의
additive 투영 — 신규 수집 SQL/마이그 0. 통합 보드 레인의 오프로드 가시화
(맥북이 요약 86% 처리) 재료. + FE 타입 동기 + store 신선도 timestamp(B-4).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 13:54:39 +09:00
29 changed files with 2962 additions and 606 deletions
+8 -3
View File
@@ -680,7 +680,12 @@ class SectionItem(BaseModel):
level: int | None = None
node_type: str | None = None # window | chapter_split | clause_split | section_split | null
is_leaf: bool
parent_id: int | None = None # 트리 부모 chunk_id. window child 의 parent_id = 그 split-parent.
# 프런트 collapseWindows 가 비인접 window 를 split-parent 에 흡수할 때 사용.
char_start: int | None = None # md_content 내 heading offset(UTF-16). jump-target 만 값, 그 외 None (Path B)
text: str | None = None # 절 본문 = 청크 원문. 대형 split 문서는 md_content 가 앞 5만 자만 보존
# (marker LARGE_DOC_MD_CONTENT_HEAD_CHARS)이고 char_start 도 NULL 이라
# md_content 슬라이스로는 본문이 비므로, 청크 text 를 직접 렌더한다.
section_type: str | None = None
summary: str | None = None # status='summarized' 인 분석행에만, 그 외 None
confidence: float | None = None
@@ -719,12 +724,12 @@ async def get_document_sections(
await session.execute(
sql_text(
"""
SELECT chunk_id, section_title, heading_path, level, node_type, is_leaf, char_start,
section_type, summary, confidence
SELECT chunk_id, section_title, heading_path, level, node_type, is_leaf, parent_id, char_start,
text, section_type, summary, confidence
FROM (
SELECT DISTINCT ON (c.id)
c.id AS chunk_id, c.chunk_index, c.section_title, c.heading_path,
c.level, c.node_type, c.is_leaf, c.char_start,
c.level, c.node_type, c.is_leaf, c.parent_id, c.char_start, c.text,
a.section_type,
CASE WHEN a.status = 'summarized' THEN a.summary ELSE NULL END AS summary,
a.confidence
+30
View File
@@ -59,6 +59,20 @@ class SummarizeEta(BaseModel):
eta_minutes: int | None
class MachineDone(BaseModel):
"""머신 1대의 summarize 완료 실적 (분담 표시용)."""
done_1h: int
done_today: int
class SummarizeByMachine(BaseModel):
"""summarize 풀의 머신별 완료 실적 분담 — 보드 레인의 '맥미니 vs 맥북'
오프로드 가시화용. rows_to_summarize_split 이 이미 계산하던 값의 노출
(ds-board-merged A-1, 신규 수집 SQL 0)."""
macmini: MachineDone
macbook: MachineDone
class TrendBucket(BaseModel):
"""summarize 24h 추이 버킷 — hour 는 KST "HH:00" 라벨."""
hour: str
@@ -89,12 +103,28 @@ class StageRow(BaseModel):
oldest_pending_age_sec: int | None
class BackgroundJobItem(BaseModel):
"""큐 밖 관리 스크립트(백필 등) 작업 — processing_queue 가 못 보는 사각지대 노출.
stale = running 인데 heartbeat 가 오래 끊김(프로세스 사망 추정)."""
id: int
kind: str
label: str | None
state: Literal["running", "done", "failed"]
processed: int
total: int | None
elapsed_sec: int
stale: bool
error: str | None
class QueueOverviewResponse(BaseModel):
machines: list[MachineCard]
stages: list[StageRow]
summarize_eta: SummarizeEta
summarize_by_machine: SummarizeByMachine
trend_24h: list[TrendBucket]
totals: Totals
background_jobs: list[BackgroundJobItem] = []
class FailedItem(BaseModel):
+12
View File
@@ -56,6 +56,9 @@ async def lifespan(app: FastAPI):
from workers.mailplus_archive import run as mailplus_run
from workers.statute_collector import run as statute_run
from workers.news_collector import run as news_collector_run
from workers.arxiv_collector import run as arxiv_collector_run
from workers.openalex_collector import run as openalex_collector_run
from workers.paper_doi_reconcile import run as paper_doi_reconcile_run
from workers.fulltext_worker import reconcile_unresolved as fulltext_reconcile_run
from workers.kosha_collector import run as kosha_collector_run
from workers.csb_collector import run as csb_collector_run
@@ -139,6 +142,9 @@ async def lifespan(app: FastAPI):
# plan ds-s1-backend-1 B-4: dedup 컬럼(duplicate_of/duplicate_count) 야간 절대 재계산.
# soft-delete 잔여 드리프트 정리(멱등, 드리프트 없으면 no-op). cron 03:30 (다른 잡과 비충돌).
scheduler.add_job(dedup_reconcile_run, CronTrigger(hour=3, minute=30, timezone=KST), id="dedup_reconcile")
# B-3 PR4: 레거시 paper 행 arXiv DataCite DOI 스탬프(재유입 차단). keyless·in-DB·enqueue 0.
# dedup_reconcile(03:30)·fulltext_reconcile(03:40) 와 별 worker·비충돌 슬롯.
scheduler.add_job(paper_doi_reconcile_run, CronTrigger(hour=3, minute=50, timezone=KST), id="paper_doi_reconcile")
# crawl-24x7 C-2: KOSHA 재해사례 diff + GUIDE 점진 백필 (daily, 새벽 잡들과 비충돌 슬롯).
scheduler.add_job(kosha_collector_run, CronTrigger(hour=6, minute=40, timezone=KST), id="kosha_collector")
# 사이클 3 C-2 잔여: CSB sitemap lastmod diff (weekly 월, cap 40 + 워터마크 점진 백필).
@@ -147,6 +153,12 @@ async def lifespan(app: FastAPI):
scheduler.add_job(api_standards_run, CronTrigger(day=5, hour=7, minute=5, timezone=KST), id="api_standards_collector")
# 사이클 3 C-2 잔여: CCPS Beacon 월간 PDF (playwright 익명 경유 — WAF 차단 시 health 로 가시화).
scheduler.add_job(ccps_collector_run, CronTrigger(day=5, hour=7, minute=20, timezone=KST), id="ccps_collector")
# B-3 PR2: arXiv 키워드 필터 수집기 (daily 07:30 KST — statute 07:00 직후 빈 슬롯).
# signal-only 초록 색인, per-run cap 으로 임베드 큐 보호. keyless.
scheduler.add_job(arxiv_collector_run, CronTrigger(hour=7, minute=30, timezone=KST), id="arxiv_collector")
# B-3 PR3: OpenAlex 백본 수집기 (daily 07:45 KST). scaffold-first(키 부재 explicit-skip),
# signal-only 초록 색인, per-run cap + cursor watermark. 키=OPENALEX_API_KEY(credentials.env).
scheduler.add_job(openalex_collector_run, CronTrigger(hour=7, minute=45, timezone=KST), id="openalex_collector")
scheduler.start()
# Phase 2.1 (async 구조): QueryAnalyzer prewarm.
+93
View File
@@ -0,0 +1,93 @@
"""off-queue 관리 스크립트(백필 등) 진행 가시화 — background_jobs (migration 357).
processing_queue 는 파이프라인 stage 전용이라 hier_overnight_backfill /
section_summary_pilot 같은 스크립트 작업은 대시보드 보드에 안 잡힌다. 이 모듈로
스크립트가 진행상황을 남기면 queue_overview 가 "백그라운드 작업" 패널로 노출한다.
설계 불변식:
- **자율 트랜잭션**: 각 기록은 engine.begin() 짧은 트랜잭션으로 즉시 commit한다.
스크립트 본 작업은 별도 세션(긴 트랜잭션)이라, 같이 묶으면 commit 전까지 안 보여
실시간 가시화가 깨진다. 그래서 전용 connection 으로 독립 commit.
- **best-effort**: 관측 기록 실패가 본 작업을 깨면 안 된다 — 모든 함수 try/except,
실패 시 warning 로그만. job_id=None 이면 조용히 no-op (start 실패해도 이어서 동작).
"""
import json
import logging
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncEngine
logger = logging.getLogger(__name__)
async def start_job(
engine: AsyncEngine, kind: str, label: str | None = None, total: int | None = None
) -> int | None:
"""작업 시작 기록 → background_jobs.id (실패 시 None — 호출측은 그대로 진행)."""
try:
async with engine.begin() as conn:
row = (
await conn.execute(
text(
"INSERT INTO background_jobs (kind, label, total) "
"VALUES (:k, :l, :t) RETURNING id"
),
{"k": kind, "l": label, "t": total},
)
).first()
return int(row[0]) if row else None
except Exception as exc: # noqa: BLE001 — 관측은 부가, 본작업 보호
logger.warning(f"[background_jobs] start 실패(무시): {type(exc).__name__}: {exc}")
return None
async def heartbeat(
engine: AsyncEngine,
job_id: int | None,
*,
processed: int | None = None,
total: int | None = None,
detail: dict | None = None,
) -> None:
"""진행 갱신(processed/total/detail). job_id=None 또는 실패 시 no-op."""
if job_id is None:
return
try:
async with engine.begin() as conn:
await conn.execute(
text(
"UPDATE background_jobs SET "
"processed = COALESCE(:p, processed), "
"total = COALESCE(:t, total), "
"detail = COALESCE(CAST(:d AS jsonb), detail), "
"updated_at = now() WHERE id = :id"
),
{
"id": job_id,
"p": processed,
"t": total,
"d": json.dumps(detail, ensure_ascii=False) if detail is not None else None,
},
)
except Exception as exc: # noqa: BLE001
logger.warning(f"[background_jobs] heartbeat 실패(무시): {type(exc).__name__}: {exc}")
async def finish_job(
engine: AsyncEngine, job_id: int | None, *, state: str = "done", error: str | None = None
) -> None:
"""종료 기록(done/failed). job_id=None 또는 실패 시 no-op."""
if job_id is None:
return
try:
async with engine.begin() as conn:
await conn.execute(
text(
"UPDATE background_jobs SET state = :s, error = :e, "
"finished_at = now(), updated_at = now() WHERE id = :id"
),
{"id": job_id, "s": state, "e": (error or None)},
)
except Exception as exc: # noqa: BLE001
logger.warning(f"[background_jobs] finish 실패(무시): {type(exc).__name__}: {exc}")
+5
View File
@@ -0,0 +1,5 @@
"""B-3 논문 수집 트랙 공유 모듈 (plan safety-library-b3-1).
doi — DOI 정규화·dedup 키·2-Document(holder/parent_doi child) extract_meta 계약 (순수).
holder — 서지 holder 공유 dedup 조회 (DB).
"""
+141
View File
@@ -0,0 +1,141 @@
"""B-3 논문 DOI 코어 — 정규화·dedup 키·2-Document(서지 holder / parent_doi child) 계약.
plan safety-library-b3-1 PR1 (keyless·마이그 0).
핵심 계약(모든 논문 수집기·reconcile·구매 PDF 스탬프가 공유):
- DOI 정규화는 이 단일 함수(normalize_doi) 경유 — **저장=조회 동일 함수**
(migration 351 주석 명시, news_collector._normalize_url 의 store=lookup 불변식 선례).
같은 논문이 다른 표기(https://doi.org/ vs doi: vs 대문자)로 들어와도 한 holder 로 붕괴.
- dedup 키 = lower(extract_meta #>> '{paper,doi}') — 라이브 partial-unique 인덱스
uq_documents_paper_doi(WHERE material_type='paper' AND ... IS NOT NULL)가 강제.
- 2-Document(R2-B1): paper.doi 는 **서지 Document 단일 보유**. OA/구매 전문 PDF 는
doi 없이 paper.parent_doi 로 holder 링크(NULL doi 라 인덱스 밖 → 다중행 무충돌).
holder 와 child 는 doi/parent_doi 를 **상호 배타**로 가진다.
"""
import hashlib
import re
# 소문자화 후 비교하므로 전부 소문자 prefix. 긴 것부터(dx.doi.org 가 doi.org 보다 먼저).
_DOI_PREFIXES = (
"https://dx.doi.org/",
"http://dx.doi.org/",
"https://doi.org/",
"http://doi.org/",
"dx.doi.org/",
"doi.org/",
"doi:",
)
def normalize_doi(raw: str | None) -> str | None:
"""DOI 정규화 — 소문자 + URL/doi: prefix 제거 + 양끝 공백·잡음 제거. 단일 함수(저장=조회).
유효 DOI(10. 으로 시작)가 아니면 None. 저장측·조회측·dedup 키 생성이 모두 이 함수를
공유해야 dedup 이 성립한다(raw 를 그대로 저장하고 정규화로 조회하면 영구 미스).
"""
if not raw:
return None
s = raw.strip().lower()
for p in _DOI_PREFIXES:
if s.startswith(p):
s = s[len(p):]
break
s = s.strip()
# 인용문 끝 잡음(마침표/쉼표/세미콜론)만 제거. 괄호 '()' 는 DOI 일부일 수 있어 보존한다
# (예: 10.1016/s0010-8650(00)80003-2) — 과삭제는 서로 다른 논문을 한 holder 로 병합하는
# 데이터 손상이라 near-dup(과소삭제)보다 위험. API 소스(OpenAlex/arXiv)의 doi 는 이미 깨끗.
s = s.rstrip(".,;")
if not s.startswith("10."):
return None
return s
# arXiv id: 신형 'YYMM.NNNNN'(+vN) 또는 구형 'archive(.SUBJ)/NNNNNNN'. 'arXiv:' 접두 흡수.
_ARXIV_ID_RE = re.compile(
r"arxiv:\s*([a-z\-]+(?:\.[a-z]{2})?/\d{7}|\d{4}\.\d{4,5})(v\d+)?", re.IGNORECASE
)
def parse_arxiv_id(text: str | None) -> str | None:
"""본문/제목에서 arXiv id(versionless) 추출. 없으면 None. 레거시 reconcile 의 입력."""
if not text:
return None
m = _ARXIV_ID_RE.search(text)
return m.group(1) if m else None
def arxiv_doi(arxiv_id: str | None) -> str | None:
"""arXiv DataCite DOI = 10.48550/arxiv.{id} (정규화). 저널 DOI 없는 프리프린트의 canonical
paper.doi 통일 키 — OpenAlex 가 프리프린트에 동일 DOI 부여(실측 확인). 모든 수집기·reconcile 가
같은 함수로 같은 DOI 를 써야 교차소스 dedup 이 성립."""
if not arxiv_id:
return None
return normalize_doi(f"10.48550/arXiv.{arxiv_id}")
_DOI_IN_TEXT_RE = re.compile(r"10\.\d{4,9}/[^\s\"'<>]+", re.IGNORECASE)
def parse_doi_from_text(text: str | None) -> str | None:
"""본문에서 첫 DOI 추출(정규화). 구매 PDF 의 paper.parent_doi 링크용(PDF 구조 무관 — 전체 스캔).
DOI 끝 구두점은 normalize_doi 가 정리. 없으면 None."""
if not text:
return None
m = _DOI_IN_TEXT_RE.search(text)
return normalize_doi(m.group(0)) if m else None
def paper_doi_hash(normalized_doi: str) -> str:
"""서지 holder 의 Document.file_hash — sha256('paper|{doi}')[:32].
statute 의 'statute|{jur}|{native_id}|{version_key}' 다중부 키 선례를 따른다.
인자는 normalize_doi() 출력(정규화 완료값)이어야 한다 — raw 를 넣으면 dedup 이 깨진다.
"""
if not normalized_doi:
raise ValueError("paper_doi_hash 는 정규화된 DOI 필요 (normalize_doi 먼저)")
return hashlib.sha256(f"paper|{normalized_doi}".encode()).hexdigest()[:32]
def read_paper_doi(extract_meta: dict | None) -> str | None:
"""holder 의 정규화 DOI 읽기 — 인덱스 식 lower(extract_meta #>> '{paper,doi}') 의 조회측 거울.
방어적 재정규화(이미 정규화돼 저장되지만 레거시·외부 주입 대비).
"""
if not extract_meta:
return None
paper = extract_meta.get("paper")
if not isinstance(paper, dict):
return None
return normalize_doi(paper.get("doi"))
def with_paper_doi(extract_meta: dict | None, normalized_doi: str) -> dict:
"""서지 holder 의 extract_meta 에 paper.doi 주입 (merge-safe, 타 키 보존).
holder 전용 — parent_doi 는 제거(상호 배타). 반환값은 새 dict(입력 비변경).
"""
if not normalized_doi:
raise ValueError("with_paper_doi 는 정규화된 DOI 필요")
meta = dict(extract_meta or {})
paper = dict(meta.get("paper") or {})
paper["doi"] = normalized_doi
paper.pop("parent_doi", None)
meta["paper"] = paper
return meta
def with_parent_doi(extract_meta: dict | None, parent_normalized_doi: str) -> dict:
"""child(OA/구매 전문 PDF)의 extract_meta 에 paper.parent_doi 주입 (merge-safe, 타 키 보존).
child 는 paper.doi 를 갖지 않는다(NULL → partial-unique 인덱스 밖, 2-Document 무충돌).
반환값은 새 dict(입력 비변경).
"""
if not parent_normalized_doi:
raise ValueError("with_parent_doi 는 정규화된 DOI 필요")
meta = dict(extract_meta or {})
paper = dict(meta.get("paper") or {})
paper["parent_doi"] = parent_normalized_doi
paper.pop("doi", None)
meta["paper"] = paper
return meta
+38
View File
@@ -0,0 +1,38 @@
"""B-3 논문 서지 holder 공유 dedup 조회.
모든 논문 수집기(OpenAlex/arXiv/KoreaScience/J-STAGE)·reconcile·구매 PDF 스탬프가
ingest 함수로 holder 존재를 확인한다(있으면 skip 또는 child 링크).
- 조회 = lower(extract_meta #>> '{paper,doi}') == normalize_doi(...) — 라이브 partial-unique
인덱스 uq_documents_paper_doi 동일 (인덱스 사용).
- .scalars().first() 교차게시·다중 landing-page 2 이상 매칭 MultipleResultsFound
raise 방지(scalar_one_or_none 금지, 2026-06 BBC 수집 중단 선례 / news_collector 동일 규율).
- 서지 holder Document **생성** 수집기/스탬프 경로가 소유한다(초록 signal 문서 vs 구매
최소 holder shape 다름). 모듈은 dedup 조회만 공유한다.
DB 조회라 모듈은 PR2(arXiv 실수집)에서 라이브 검증한다 PR1 단위 테스트 대상은 doi.py(순수).
"""
from sqlalchemy import func, select
from models.document import Document
from services.papers.doi import normalize_doi
# 인덱스 식과 동일: lower(extract_meta #>> '{paper,doi}')
_DOI_EXPR = func.lower(Document.extract_meta[("paper", "doi")].astext)
async def find_paper_holder(session, raw_or_normalized_doi):
"""정규화 DOI 로 서지 holder Document 조회. 없으면 None.
인자는 raw 정규화든 받아 normalize_doi 통일(저장=조회 동일 함수 보장).
"""
doi = normalize_doi(raw_or_normalized_doi)
if not doi:
return None
result = await session.execute(
select(Document)
.where(Document.material_type == "paper", _DOI_EXPR == doi)
.limit(1)
)
return result.scalars().first()
+50 -1
View File
@@ -213,6 +213,16 @@ def build_summarize_eta(stage_stats: dict[str, dict]) -> dict:
}
def build_summarize_by_machine(summarize_split: dict[str, dict]) -> dict:
"""summarize 머신별 완료 실적 분담 (macmini vs macbook) — 보드 레인의
오프로드 가시화용. rows_to_summarize_split 이미 만든 값을 응답 형태로
투영(done_1h/done_today , done_15m 내부 state 판정 전용이라 제외)."""
def m(key: str) -> dict:
s = summarize_split.get(key, {})
return {"done_1h": int(s.get("done_1h", 0)), "done_today": int(s.get("done_today", 0))}
return {"macmini": m("macmini"), "macbook": m("macbook")}
def build_trend(
inflow_buckets: dict[str, int],
done_buckets: dict[str, int],
@@ -292,6 +302,7 @@ def compose_overview(
),
"stages": build_stages(stage_stats),
"summarize_eta": build_summarize_eta(stage_stats),
"summarize_by_machine": build_summarize_by_machine(summarize_split),
"trend_24h": build_trend(inflow_buckets, done_buckets, now_kst),
"totals": build_totals(stage_stats),
}
@@ -401,7 +412,7 @@ async def build_overview(session: AsyncSession) -> dict:
for row in current_result
]
return compose_overview(
result = compose_overview(
rows_to_stage_stats(stage_rows),
rows_to_summarize_split(split_rows),
{row[0]: int(row[1]) for row in inflow_rows},
@@ -410,6 +421,44 @@ async def build_overview(session: AsyncSession) -> dict:
deep_enabled=deep_enabled,
now_kst=now_kst,
)
# 큐 밖 관리 스크립트(백필 등) = background_jobs (migration 357). 테이블 부재 시 graceful([]).
result["background_jobs"] = await _fetch_background_jobs(session)
return result
_BACKGROUND_JOBS_SQL = """
SELECT id, kind, label, state, processed, total,
EXTRACT(EPOCH FROM (now() - started_at))::int AS elapsed_sec,
(state = 'running' AND updated_at < now() - interval '5 minutes') AS stale,
error
FROM background_jobs
WHERE state = 'running' OR finished_at > now() - interval '6 hours'
ORDER BY (state = 'running') DESC, started_at DESC
LIMIT 20
"""
async def _fetch_background_jobs(session: AsyncSession) -> list[dict]:
"""running + 최근 6h 완료 background_jobs. 테이블 없거나 오류면 [] (보드 무영향).
요청 세션과 **별도 connection**으로 조회한다 테이블 부재(마이그 357 미적용 )
SELECT 실패가 요청 세션의 트랜잭션을 오염시키지 않도록 물리적으로 분리(실패
임시 connection만 폐기). 관측은 부가 기능이라 보드 본체를 절대 깨면 된다.
"""
try:
async with session.bind.connect() as conn: # 풀에서 독립 connection
rows = (await conn.execute(text(_BACKGROUND_JOBS_SQL))).mappings().all()
except Exception: # noqa: BLE001 — 관측 부가, 보드 본체 보호
return []
return [
{
"id": r["id"], "kind": r["kind"], "label": r["label"], "state": r["state"],
"processed": int(r["processed"] or 0), "total": r["total"],
"elapsed_sec": int(r["elapsed_sec"] or 0), "stale": bool(r["stale"]),
"error": r["error"],
}
for r in rows
]
# ─── 실패 처리 (plan ds-board-engines-1) ─────────────────────────────────────
+370
View File
@@ -0,0 +1,370 @@
"""arXiv 키워드 필터 수집기 — B-3 PR2 (plan safety-library-b3-1).
bespoke arXiv API(Atom) 수집기. 카테고리 RSS 통째(firehose) 아니라
cat:{category} AND (abs:키워드 ...) 안전/신뢰성/압력용기 관련분만 좁혀 수집한다.
- signal-only: 초록만 색인(embed+chunk), summarize 절대 미enqueue 맥미니 Qwen 무접촉.
- DOI 보유 paper.doi(서지 holder, partial-unique 인덱스 진입). 없으면 versionless arXiv id
dedup(향후 PR4 reconcile DOI 백필).
- etiquette: 요청 3s + HTTP 429 지수 백오프. 카테고리별 submittedDate 워터마크로 증분.
- per-run insert cap(_RUN_CAP) 광역 수집이 GPU bge-m3 embed 큐를 범람시키지 않게(적대리뷰 A major).
잔여는 silent-cap 금지(csb idiom): 누락 건수 로깅.
- keyless. enabled=False news_sources (6h 뉴스 사이클 비대상) + main.py CronTrigger(자체 폴링).
- arXiv API https 필수(http=301). UA = CRAWL_UA.
"""
import asyncio
import hashlib
import re
import xml.etree.ElementTree as ET
from dataclasses import dataclass, field
from datetime import datetime, timezone
import httpx
from sqlalchemy import select
from core.crawl_politeness import CRAWL_UA
from core.database import async_session
from core.utils import setup_logger
from models.document import Document
from models.news_source import NewsSource
from models.queue import enqueue_stage
from services.papers.doi import arxiv_doi, normalize_doi
from services.papers.holder import find_paper_holder
from workers.news_collector import (
FeedError,
_get_or_create_health,
_record_failure,
_record_success,
)
logger = setup_logger("arxiv_collector")
_ARXIV_API = "https://export.arxiv.org/api/query"
_SOURCE_NAME = "arXiv 안전·공학 (keyword)"
# 신규 카테고리만 — 기존 RSS 행(id 62 physics.app-ph, id 64 cond-mat.mtrl-sci)과 비중복.
_CATEGORIES = (
"eess.SY", # systems & control
"physics.flu-dyn", # 유체 — 압력/유동
"physics.comp-ph", # 전산물리
"math.OC", # 최적화·제어
"math.NA", # 수치해석 (FEM 등)
"stat.AP", # 응용통계 — 신뢰성
"cs.CE", # computational engineering
)
# 압력용기·공정안전·구조건전성 도메인 키워드(abs: OR 게이트). 좁게 유지 = 관련성↑·볼륨↓ (튜너블).
_KEYWORDS = (
"pressure vessel",
"process safety",
"structural integrity",
"fracture mechanics",
"fatigue life",
"corrosion",
)
_RUN_CAP = 80 # 1회 run 신규 적재 상한(임베드 큐 보호). bulk 시 해제.
_PAGE_SIZE = 50 # max_results per request
_MAX_PAGES_PER_CAT = 4 # 카테고리당 최대 페이지(증분이라 보통 1페이지에 워터마크 도달)
_REQ_SLEEP = 3.0 # arXiv etiquette ≥3s
_MAX_RETRY = 4
_BACKOFF_BASE = 5.0
_NS = {
"a": "http://www.w3.org/2005/Atom",
"arxiv": "http://arxiv.org/schemas/atom",
"opensearch": "http://a9.com/-/spec/opensearch/1.1/",
}
_ABS_ID_RE = re.compile(r"arxiv\.org/abs/(.+?)(v\d+)?$")
_WS_RE = re.compile(r"\s+")
# ───────────────────────── 순수 파서 (fixture 단위 테스트 대상) ─────────────────────────
@dataclass
class ArxivEntry:
arxiv_id: str # versionless, 예: "1209.2405"
version: str | None # "v1" 또는 None
title: str
summary: str # 초록
published: datetime | None
doi: str | None # normalize_doi 적용
journal_ref: str | None
primary_category: str | None
categories: list = field(default_factory=list)
abs_url: str | None = None
pdf_url: str | None = None
def _clean(text: str | None) -> str:
return _WS_RE.sub(" ", text).strip() if text else ""
def _parse_id(raw_id: str | None) -> tuple[str | None, str | None]:
"""'http://arxiv.org/abs/1209.2405v1' → ('1209.2405', 'v1'). versionless id 가 dedup 키."""
m = _ABS_ID_RE.search((raw_id or "").strip())
if not m:
return None, None
return m.group(1), m.group(2)
def _parse_dt(s: str | None) -> datetime | None:
if not s:
return None
try:
return datetime.fromisoformat(s.replace("Z", "+00:00"))
except ValueError:
return None
def build_search_query(category: str, keywords=_KEYWORDS) -> str:
"""cat:{category} AND (abs:kw1 OR abs:"kw with space" ...). 공백 키워드는 따옴표 구절."""
kw = " OR ".join(f'abs:"{k}"' if " " in k else f"abs:{k}" for k in keywords)
return f"cat:{category} AND ({kw})"
def parse_arxiv_feed(xml_text: str) -> tuple[int, list[ArxivEntry]]:
"""arXiv Atom 응답 → (total_results, [ArxivEntry]). 순수 함수."""
root = ET.fromstring(xml_text)
raw_total = root.findtext("opensearch:totalResults", default="0", namespaces=_NS)
try:
total = int(raw_total)
except (TypeError, ValueError):
total = 0
entries: list[ArxivEntry] = []
for e in root.findall("a:entry", _NS):
aid, ver = _parse_id(e.findtext("a:id", namespaces=_NS))
if not aid:
continue
prim = e.find("arxiv:primary_category", _NS)
abs_url = pdf_url = None
for ln in e.findall("a:link", _NS):
if ln.get("rel") == "alternate" and (ln.get("type") or "").startswith("text/html"):
abs_url = ln.get("href")
elif ln.get("title") == "pdf":
pdf_url = ln.get("href")
entries.append(ArxivEntry(
arxiv_id=aid,
version=ver,
title=_clean(e.findtext("a:title", namespaces=_NS)),
summary=_clean(e.findtext("a:summary", namespaces=_NS)),
published=_parse_dt(e.findtext("a:published", namespaces=_NS)),
doi=normalize_doi(e.findtext("arxiv:doi", namespaces=_NS)),
journal_ref=_clean(e.findtext("arxiv:journal_ref", namespaces=_NS)) or None,
primary_category=prim.get("term") if prim is not None else None,
categories=[c.get("term") for c in e.findall("a:category", _NS)],
abs_url=abs_url,
pdf_url=pdf_url,
))
return total, entries
# ───────────────────────── 적재 (DB — PR2 라이브 검증) ─────────────────────────
def _build_paper_meta(source: NewsSource, entry: ArxivEntry, doi: str | None) -> dict:
"""extract_meta — license + source + paper 식별. 서지 holder 는 paper.doi(있으면) 보유."""
paper: dict = {"arxiv_id": entry.arxiv_id}
if doi:
paper["doi"] = doi # partial-unique 인덱스 진입 (교차소스 dedup)
if entry.journal_ref:
paper["journal_ref"] = entry.journal_ref
if entry.primary_category:
paper["primary_category"] = entry.primary_category
meta: dict = {
"source_id": source.id,
"source_name": source.name,
"source_region": "INT", # arXiv = 국제 preprint. paper.jurisdiction 은 NULL 유지(A-2).
"paper": paper,
# arXiv 기본 라이선스 = 비배포(보수적). restricted 부재 → 초록은 RAG 사용 가능.
# (명시 CC 검출은 OAI 인터페이스 필요 — Atom API 미포함, PR 후속/관찰.)
"license": {"scheme": "arxiv", "redistribute": False, "attribution": "arXiv"},
}
if entry.published:
meta["published_at"] = entry.published.isoformat()
return meta
async def _ingest_entry(session, source: NewsSource, entry: ArxivEntry) -> bool:
"""1건 적재. 반환 = 신규 여부. signal-only(embed+chunk, summarize 없음)."""
arxiv_hash = hashlib.sha256(f"arxiv|{entry.arxiv_id}".encode()).hexdigest()[:32]
# 재수집 dedup(arXiv id) — .first()(다중행 방어)
dup = await session.execute(
select(Document.id).where(Document.file_hash == arxiv_hash).limit(1)
)
if dup.scalars().first():
return False
# arXiv canonical DOI = 저널 DOI 또는 arXiv DataCite DOI(프리프린트도 paper.doi 보유 → PR3 와 dedup)
doi = entry.doi or arxiv_doi(entry.arxiv_id)
# 교차소스 dedup(DOI holder 이미 존재 — partial-unique 인덱스 백스톱 선제 회피)
if doi and await find_paper_holder(session, doi):
return False
body = entry.summary or entry.title
doc = Document(
file_path=f"crawl/arxiv/{entry.arxiv_id}",
file_hash=arxiv_hash,
file_format="article",
file_size=len(body.encode()),
file_type="note",
title=entry.title,
extracted_text=f"{entry.title}\n\n{body}",
extracted_at=datetime.now(timezone.utc),
extractor_version="arxiv-api-signal",
md_status="skipped",
md_extraction_error="arXiv abstract: signal-only, markdown 비대상",
source_channel="crawl",
data_origin="external",
edit_url=entry.abs_url,
review_status="approved",
material_type="paper",
jurisdiction=None, # paper = NULL 불변(A-2). 지역은 extract_meta.paper.source_region.
published_date=entry.published.date() if entry.published else None,
extract_meta=_build_paper_meta(source, entry, doi),
)
session.add(doc)
await session.flush()
# signal-only: 검색 색인만. summarize/fulltext 절대 enqueue 안 함(맥미니 큐 무접촉).
await enqueue_stage(session, doc.id, "embed")
await enqueue_stage(session, doc.id, "chunk")
return True
async def _get_or_create_source(session) -> NewsSource:
result = await session.execute(
select(NewsSource).where(NewsSource.name == _SOURCE_NAME)
)
source = result.scalars().first()
if source is None:
source = NewsSource(
name=_SOURCE_NAME, feed_url=_ARXIV_API, feed_type="atom",
fetch_method="signal-only", fulltext_policy="none",
source_channel="crawl", category="Engineering", language="en",
country=None, # paper → jurisdiction NULL (country 미전파)
material_type="paper",
license_scheme="arxiv", license_redistribute=False,
enabled=False, # 6h 뉴스 사이클 비대상 — 본 워커가 자체 폴링
)
session.add(source)
await session.flush()
return source
def _watermark(source: NewsSource, category: str) -> datetime | None:
raw = (source.selector_override or {}).get("arxiv_watermark", {}).get(category)
if not raw:
return None
return _parse_dt(raw)
def _set_watermark(source: NewsSource, category: str, value: datetime) -> None:
cfg = dict(source.selector_override or {})
wm = dict(cfg.get("arxiv_watermark") or {})
wm[category] = value.isoformat()
cfg["arxiv_watermark"] = wm
source.selector_override = cfg # JSONB 변경 감지 위해 재할당
async def _fetch(client: httpx.AsyncClient, query: str, start: int) -> str:
params = {
"search_query": query, "start": start, "max_results": _PAGE_SIZE,
"sortBy": "submittedDate", "sortOrder": "descending",
}
for attempt in range(_MAX_RETRY):
resp = await client.get(_ARXIV_API, params=params)
if resp.status_code == 429:
await asyncio.sleep(_BACKOFF_BASE * (2 ** attempt))
continue
resp.raise_for_status()
return resp.text
raise FeedError(f"arXiv 429 재시도 초과: {query[:48]}")
async def run(bulk: bool = False, limit: int = 0) -> None:
"""daily 진입점(스케줄러). bulk/limit 은 CLI 전용(bulk=cap 해제·깊은 페이징)."""
now = datetime.now(timezone.utc)
async with async_session() as session:
source = await _get_or_create_source(session)
await session.commit()
source_id = source.id
run_cap = (limit or 10**9) if bulk else (min(limit, _RUN_CAP) if limit else _RUN_CAP)
inserted = 0
seen = 0
failures: list[str] = []
async with httpx.AsyncClient(
timeout=30.0, headers={"User-Agent": CRAWL_UA}, follow_redirects=True
) as client:
for category in _CATEGORIES:
if inserted >= run_cap:
break
query = build_search_query(category)
async with async_session() as session:
src = await session.get(NewsSource, source_id)
watermark = _watermark(src, category)
newest_seen: datetime | None = None
max_pages = (10**6 if bulk else _MAX_PAGES_PER_CAT)
try:
for page in range(max_pages):
if inserted >= run_cap:
break
xml_text = await _fetch(client, query, page * _PAGE_SIZE)
total, entries = parse_arxiv_feed(xml_text)
if not entries:
break
stop = False
for entry in entries:
seen += 1
if entry.published:
newest_seen = max(newest_seen or entry.published, entry.published)
# 증분: 워터마크 이하 도달 시 이 카테고리 종료(이미 본 구간)
if watermark and not bulk and entry.published <= watermark:
stop = True
break
async with async_session() as session:
src = await session.get(NewsSource, source_id)
if await _ingest_entry(session, src, entry):
inserted += 1
await session.commit()
else:
await session.rollback()
if inserted >= run_cap:
break
await asyncio.sleep(_REQ_SLEEP)
if stop or (page + 1) * _PAGE_SIZE >= total:
break
# 카테고리 워터마크 전진(이번 run 최신 발행일)
if newest_seen:
async with async_session() as session:
src = await session.get(NewsSource, source_id)
_set_watermark(src, category, newest_seen)
await session.commit()
except (httpx.HTTPError, FeedError, ET.ParseError) as e:
msg = f"[{category}] {e or repr(e)}"
logger.error(f"[arxiv] {msg}")
failures.append(msg)
async with async_session() as session:
health = await _get_or_create_health(session, source_id)
if failures and inserted == 0:
_record_failure(health, "; ".join(failures)[:500], now)
else:
_record_success(health, inserted, False, now)
await session.commit()
deferred = "" if inserted < run_cap else f" (cap {run_cap} 도달 — 잔여는 다음 run 이월)"
logger.info(
f"[arxiv] {len(_CATEGORIES)}개 카테고리 스캔 {seen}건 → 신규 {inserted}{deferred}"
+ (f" / 실패 {len(failures)}" if failures else "")
)
if __name__ == "__main__":
# CLI = 수동/백필 전용. --bulk = cap 해제·깊은 페이징, --limit N = 상한 N(라이브 검증용).
import argparse
parser = argparse.ArgumentParser(description="arXiv 안전·공학 키워드 수집기")
parser.add_argument("--bulk", action="store_true", help="cap 해제 + 깊은 페이징 백필")
parser.add_argument("--limit", type=int, default=0, help="신규 적재 상한(0=기본 cap)")
args = parser.parse_args()
asyncio.run(run(bulk=args.bulk, limit=args.limit))
+9
View File
@@ -411,6 +411,15 @@ async def process(
logger.info(f"doc {document_id}: devonagent → classify skip")
return
# 논문(material_type='paper') — 요약/분류 LLM 스킵(맥미니 큐 무접촉, B-3 signal-only 유지).
# embed/chunk/markdown 은 queue_consumer 가 chain (early-return 후에도 다음 stage enqueue).
if doc.material_type == "paper":
if not doc.ai_domain:
doc.ai_domain = "논문"
await session.commit()
logger.info(f"doc {document_id}: paper → classify skip (no summarize)")
return
if not doc.extracted_text:
raise ValueError(f"문서 ID {document_id}: extracted_text가 비어있음")
+393
View File
@@ -0,0 +1,393 @@
"""OpenAlex 백본 수집기 — B-3 PR3 (plan safety-library-b3-1).
OpenAlex = 발견+dedup 글로벌 백본(JP/EU/US 논문 색인 + 정본 DOI). 전문은 (oa_url 포인터만).
- scaffold-first: OPENALEX_API_KEY 부재 FeedError(explicit-skip, silent fallback 금지). =무료.
- signal-only: 초록(inverted-index 복원) 색인(embed+chunk), summarize 절대 미enqueue(맥미니 무접촉).
PDF 절대 OpenAlex 경유로 받음(oa_url 링크/신호일 ).
- 관련성 사전필터 = title_and_abstract.search 키워드(서버측) + per-run insert cap(임베드 firehose 차단,
적대리뷰 A major). cursor 페이징 + from_publication_date 워터마크로 증분.
- 초록 없는 thin 레코드(주로 -OA 메타) skip Phase-1 재료 품질 유지.
- DOI paper.doi(holder, partial-unique 인덱스, 교차소스 dedup). 없으면 openalex id fallback.
- license: 명시 CC redistribute=true / OA·closed false(restricted 부재 = 초록 RAG 사용 가능).
- enabled=False news_sources + main.py CronTrigger(자체 폴링). list+filter 비용 미미($1/ 크레딧).
"""
import asyncio
import hashlib
import json
import os
from dataclasses import dataclass
from datetime import date, datetime, timezone
import httpx
from sqlalchemy import select
from core.crawl_politeness import CRAWL_UA
from core.database import async_session
from core.utils import setup_logger
from models.document import Document
from models.news_source import NewsSource
from models.queue import enqueue_stage
from services.papers.doi import normalize_doi
from services.papers.holder import find_paper_holder
from workers.news_collector import (
FeedError,
_get_or_create_health,
_record_failure,
_record_success,
)
logger = setup_logger("openalex_collector")
_API = "https://api.openalex.org/works"
_SOURCE_NAME = "OpenAlex 안전·공학 (keyword)"
_ENV_KEY = "OPENALEX_API_KEY"
# 압력용기·공정안전·구조건전성 도메인 키워드(키워드별 1쿼리 = 관련성 사전필터).
_KEYWORDS = (
"pressure vessel safety",
"process safety",
"structural integrity",
"fracture mechanics",
"fatigue life assessment",
)
# 도메인 직결 저널 ISSN 시드(OpenAlex sources 실측 확인) — 키워드 매칭 누락분까지 전수 커버.
# KR 안전/가스/기계 + JP 고압. KR/JP 관심 = OpenAlex 깨끗한 API 로 직접(KoreaScience/J-STAGE 전용
# 스크래퍼 불요 — Phase-1 메타는 OpenAlex 와 중복, 전용 수집기의 유니크 가치=무료 전문 PDF=Phase-2).
_JOURNAL_ISSNS = (
("한국안전학회지", "1738-3803"),
("한국가스학회지", "1226-8402"),
("대한기계학회논문집 A", "1226-4873"),
("대한기계학회논문집 B", "1226-4881"),
("KSME International J.", "1226-4865"),
("Review of High Pressure Sci&Tech (JP)", "0917-639X"),
)
_RUN_CAP = 60 # 1회 run 신규 적재 상한(임베드 큐 보호). bulk 시 해제.
_PER_PAGE = 50
_MAX_PAGES_PER_KW = 4 # 키워드당 최대 페이지(증분이라 보통 1페이지에 워터마크 도달)
_REQ_SLEEP = 1.0 # 페이지 간 polite 간격
_MAX_RETRY = 4
_BACKOFF_BASE = 5.0
# ───────────────────────── 순수 파서 (fixture 단위 테스트 대상) ─────────────────────────
@dataclass
class OpenAlexWork:
openalex_id: str # "W2910511816"
doi: str | None # normalize_doi 적용
title: str
abstract: str # inverted-index 복원 (없으면 "")
publication_date: str | None
oa_status: str | None # closed/green/bronze/hybrid/gold/diamond
oa_url: str | None
is_oa: bool
license: str | None # cc-by / cc-by-nc-nd / None
source_name: str | None
primary_topic: str | None
work_type: str | None
def _clean(text):
return " ".join(text.split()).strip() if text else ""
def _reconstruct_abstract(inv: dict | None) -> str:
"""abstract_inverted_index({word:[positions]}) → 평문 초록. 없으면 ''."""
if not inv:
return ""
positions = [(pos, word) for word, idxs in inv.items() for pos in idxs]
positions.sort()
return " ".join(w for _, w in positions)
def license_meta(license_str: str | None, is_oa: bool, source_name: str | None) -> dict:
"""extract_meta.license — 명시 CC/public-domain 만 redistribute=true. restricted 부재(초록 색인 자유).
redistribute=false 라도 restricted 없으면 RAG 사용 가능(초록). -CC 전문의 RAG verbatim 차단은
Phase-2 전문 승격 단계가 restricted=true 처리(L-1) Phase-1(초록) 무해.
"""
attribution = source_name or "OpenAlex"
if license_str and (license_str.startswith("cc") or license_str == "public-domain"):
return {"scheme": license_str, "redistribute": True, "attribution": attribution}
return {
"scheme": "open-unspecified" if is_oa else "proprietary",
"redistribute": False,
"attribution": attribution,
}
def parse_openalex_works(json_text: str) -> tuple[int, str | None, list[OpenAlexWork]]:
"""OpenAlex /works 응답 → (count, next_cursor, [OpenAlexWork]). 순수 함수."""
d = json.loads(json_text)
meta = d.get("meta") or {}
count = meta.get("count") or 0
next_cursor = meta.get("next_cursor")
works: list[OpenAlexWork] = []
for w in d.get("results") or []:
oid = (w.get("id") or "").rstrip("/").rsplit("/", 1)[-1]
if not oid:
continue
oa = w.get("open_access") or {}
pl = w.get("primary_location") or {}
pt = w.get("primary_topic") or {}
works.append(OpenAlexWork(
openalex_id=oid,
doi=normalize_doi(w.get("doi")),
title=_clean(w.get("title")),
abstract=_reconstruct_abstract(w.get("abstract_inverted_index")),
publication_date=w.get("publication_date"),
oa_status=oa.get("oa_status"),
oa_url=oa.get("oa_url") or None,
is_oa=bool(oa.get("is_oa")),
license=pl.get("license"),
source_name=(pl.get("source") or {}).get("display_name"),
primary_topic=pt.get("display_name"),
work_type=w.get("type"),
))
return count, next_cursor, works
def build_filter(keyword: str, from_date: str | None = None) -> str:
f = f"title_and_abstract.search:{keyword}"
if from_date:
f += f",from_publication_date:{from_date}"
return f
def build_issn_filter(issn: str, from_date: str | None = None) -> str:
f = f"primary_location.source.issn:{issn}"
if from_date:
f += f",from_publication_date:{from_date}"
return f
def _seeds() -> list[tuple[str, str, str]]:
"""수집 시드 = (라벨, 워터마크키, 종류). 도메인 저널 ISSN 우선(cap 우선권) → 키워드."""
s: list[tuple[str, str, str]] = [(label, issn, "issn") for label, issn in _JOURNAL_ISSNS]
s += [(kw, kw, "kw") for kw in _KEYWORDS]
return s
# ───────────────────────── 적재 (DB — PR3 라이브 검증) ─────────────────────────
def _build_paper_meta(source: NewsSource, w: OpenAlexWork) -> dict:
paper: dict = {"openalex_id": w.openalex_id}
if w.doi:
paper["doi"] = w.doi # partial-unique 인덱스 진입(교차소스 dedup)
if w.oa_status:
paper["oa_status"] = w.oa_status
if w.oa_url:
paper["oa_url"] = w.oa_url # 링크/신호 — 자동 fetch 안 함
if w.primary_topic:
paper["topic"] = w.primary_topic
meta: dict = {
"source_id": source.id,
"source_name": source.name,
"source_region": "INT", # OpenAlex = 글로벌. paper.jurisdiction 은 NULL 유지(A-2).
"paper": paper,
"license": license_meta(w.license, w.is_oa, w.source_name),
}
if w.publication_date:
meta["published_at"] = w.publication_date
return meta
async def _ingest_work(session, source: NewsSource, w: OpenAlexWork) -> bool:
"""1건 적재. 반환 = 신규 여부. signal-only. 초록 없으면 skip(thin 레코드 배제)."""
if not w.abstract:
return False # 초록 없는 thin 레코드(주로 비-OA 메타) — Phase-1 재료 품질 유지
oid_hash = hashlib.sha256(f"openalex|{w.openalex_id}".encode()).hexdigest()[:32]
dup = await session.execute(
select(Document.id).where(Document.file_hash == oid_hash).limit(1)
)
if dup.scalars().first():
return False
if w.doi and await find_paper_holder(session, w.doi):
return False # 교차소스 dedup(arXiv 등이 이미 holder 보유)
pub_date = None
if w.publication_date:
try:
pub_date = date.fromisoformat(w.publication_date)
except ValueError:
pub_date = None
body = w.abstract
doc = Document(
file_path=f"crawl/openalex/{w.openalex_id}",
file_hash=oid_hash,
file_format="article",
file_size=len(body.encode()),
file_type="note",
title=w.title,
extracted_text=f"{w.title}\n\n{body}",
extracted_at=datetime.now(timezone.utc),
extractor_version="openalex-signal",
md_status="skipped",
md_extraction_error="OpenAlex abstract: signal-only, markdown 비대상",
source_channel="crawl",
data_origin="external",
edit_url=w.oa_url or f"https://openalex.org/{w.openalex_id}",
review_status="approved",
material_type="paper",
jurisdiction=None,
published_date=pub_date,
extract_meta=_build_paper_meta(source, w),
)
session.add(doc)
await session.flush()
await enqueue_stage(session, doc.id, "embed")
await enqueue_stage(session, doc.id, "chunk")
return True
async def _get_or_create_source(session) -> NewsSource:
result = await session.execute(
select(NewsSource).where(NewsSource.name == _SOURCE_NAME)
)
source = result.scalars().first()
if source is None:
source = NewsSource(
name=_SOURCE_NAME, feed_url=_API, feed_type="json",
fetch_method="signal-only", fulltext_policy="none",
source_channel="crawl", category="Engineering", language="en",
country=None, material_type="paper",
license_scheme="openalex", license_redistribute=False,
enabled=False,
)
session.add(source)
await session.flush()
return source
def _api_key() -> str:
key = os.getenv(_ENV_KEY, "").strip()
if not key:
raise FeedError(f"{_ENV_KEY} 미설정 — OpenAlex 수집 불가 (scaffold-first explicit-skip)")
return key
def _watermark(source: NewsSource, keyword: str) -> str | None:
return (source.selector_override or {}).get("openalex_watermark", {}).get(keyword)
def _set_watermark(source: NewsSource, keyword: str, value: str) -> None:
cfg = dict(source.selector_override or {})
wm = dict(cfg.get("openalex_watermark") or {})
wm[keyword] = value
cfg["openalex_watermark"] = wm
source.selector_override = cfg
async def _fetch(client: httpx.AsyncClient, key: str, filter_str: str, cursor: str) -> str:
params = {
"filter": filter_str, "per-page": _PER_PAGE, "cursor": cursor,
"sort": "publication_date:desc", "api_key": key,
}
for attempt in range(_MAX_RETRY):
resp = await client.get(_API, params=params)
if resp.status_code == 429:
await asyncio.sleep(_BACKOFF_BASE * (2 ** attempt))
continue
resp.raise_for_status()
return resp.text
raise FeedError(f"OpenAlex 429 재시도 초과: {filter_str[:48]}")
async def run(bulk: bool = False, limit: int = 0) -> None:
"""daily 진입점(스케줄러). 키 부재 = explicit-skip(health 실패 기록)."""
now = datetime.now(timezone.utc)
async with async_session() as session:
source = await _get_or_create_source(session)
await session.commit()
source_id = source.id
try:
key = _api_key()
except FeedError as e:
logger.warning(f"[openalex] {e}")
async with async_session() as session:
health = await _get_or_create_health(session, source_id)
_record_failure(health, str(e), now)
await session.commit()
return
run_cap = (limit or 10**9) if bulk else (min(limit, _RUN_CAP) if limit else _RUN_CAP)
inserted = 0
seen = 0
failures: list[str] = []
async with httpx.AsyncClient(
timeout=30.0, headers={"User-Agent": CRAWL_UA}, follow_redirects=True
) as client:
for label, wm_key, kind in _seeds():
if inserted >= run_cap:
break
async with async_session() as session:
src = await session.get(NewsSource, source_id)
watermark = None if bulk else _watermark(src, wm_key)
filter_str = (build_issn_filter(wm_key, watermark) if kind == "issn"
else build_filter(wm_key, watermark))
newest: str | None = None
cursor = "*"
max_pages = (10**6 if bulk else _MAX_PAGES_PER_KW)
try:
for _page in range(max_pages):
if inserted >= run_cap:
break
text = await _fetch(client, key, filter_str, cursor)
_count, next_cursor, works = parse_openalex_works(text)
if not works:
break
for w in works:
seen += 1
if w.publication_date and (newest is None or w.publication_date > newest):
newest = w.publication_date
async with async_session() as session:
src = await session.get(NewsSource, source_id)
if await _ingest_work(session, src, w):
inserted += 1
await session.commit()
else:
await session.rollback()
if inserted >= run_cap:
break
await asyncio.sleep(_REQ_SLEEP)
if not next_cursor:
break
cursor = next_cursor
if newest:
async with async_session() as session:
src = await session.get(NewsSource, source_id)
_set_watermark(src, wm_key, newest)
await session.commit()
except (httpx.HTTPError, FeedError, ValueError) as e:
msg = f"[{label}] {e or repr(e)}"
logger.error(f"[openalex] {msg}")
failures.append(msg)
async with async_session() as session:
health = await _get_or_create_health(session, source_id)
if failures and inserted == 0:
_record_failure(health, "; ".join(failures)[:500], now)
else:
_record_success(health, inserted, False, now)
await session.commit()
deferred = "" if inserted < run_cap else f" (cap {run_cap} 도달 — 잔여 다음 run 이월)"
logger.info(
f"[openalex] {len(_seeds())}개 시드(ISSN+키워드) 스캔 {seen}건 → 신규 {inserted}{deferred}"
+ (f" / 실패 {len(failures)}" if failures else "")
)
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(description="OpenAlex 안전·공학 키워드 백본 수집기")
parser.add_argument("--bulk", action="store_true", help="cap 해제 + 깊은 cursor 페이징 백필")
parser.add_argument("--limit", type=int, default=0, help="신규 적재 상한(0=기본 cap)")
args = parser.parse_args()
asyncio.run(run(bulk=args.bulk, limit=args.limit))
+102
View File
@@ -0,0 +1,102 @@
"""paper DOI reconcile — B-3 PR4(레거시 arXiv) + PR5(구매 PDF) (plan safety-library-b3-1).
paper.doi/parent_doi 없는 paper 행을 갈래로 정리:
- 레거시 arXiv 초록(holder): arXiv id arxiv_doi(10.48550/arxiv.{id}) 스탬프 partial-unique
인덱스 편입 재유입 차단('동일-DOI 재유입 차단만').
- 구매 PDF(child, license.restricted=true Papers_Purchased 드롭): 본문 DOI 파싱 paper.parent_doi
링크(서지 holder DOI 공유로 연결). child doi 미보유(인덱스 ) unique 무충돌.
- KEYLESS·결정적(OpenAlex 호출 0)·in-DB·enqueue 0(콘텐츠 무변경). dedup_reconcile(file_hash 캐시)
worker(적대리뷰 B·C major). 선재 DOI holder 존재 arXiv 행도 parent_doi 마킹(unique 위반 회피).
"""
import asyncio
from sqlalchemy import select
from core.database import async_session
from core.utils import setup_logger
from models.document import Document
from services.papers.doi import (
arxiv_doi,
parse_arxiv_id,
parse_doi_from_text,
with_paper_doi,
with_parent_doi,
)
from services.papers.holder import find_paper_holder
logger = setup_logger("paper_doi_reconcile")
_DOI_TEXT = Document.extract_meta[("paper", "doi")].astext
_PARENT_DOI_TEXT = Document.extract_meta[("paper", "parent_doi")].astext
def _is_restricted(meta: dict) -> bool:
return (meta.get("license") or {}).get("restricted") in (True, "true")
async def run(limit: int = 0) -> None:
"""paper.doi/parent_doi 없는 paper 행 reconcile(멱등). limit=0 = 전건."""
stamped = marked_dup = skipped_no_arxiv = 0
linked_purchased = skipped_purchased_no_doi = 0
async with async_session() as session:
q = (
select(Document)
.where(
Document.material_type == "paper",
_DOI_TEXT.is_(None),
_PARENT_DOI_TEXT.is_(None),
)
.order_by(Document.id)
)
if limit:
q = q.limit(limit)
rows = (await session.execute(q)).scalars().all()
for row in rows:
meta = dict(row.extract_meta or {})
paper = dict(meta.get("paper") or {})
# PR5: 구매 PDF(restricted) = child → 본문 DOI 파싱 → parent_doi 링크
if _is_restricted(meta):
doi = parse_doi_from_text(row.extracted_text)
if not doi:
skipped_purchased_no_doi += 1
continue
row.extract_meta = with_parent_doi(meta, doi)
linked_purchased += 1
continue
# PR4: 레거시 arXiv 초록(holder) = arXiv DataCite DOI 스탬프
arxiv_id = paper.get("arxiv_id") or parse_arxiv_id(row.extracted_text)
doi = arxiv_doi(arxiv_id)
if not doi:
skipped_no_arxiv += 1
continue
paper["arxiv_id"] = arxiv_id
meta["paper"] = paper
holder = await find_paper_holder(session, doi)
if holder is not None and holder.id != row.id:
row.extract_meta = with_parent_doi(meta, doi) # 선재 중복 → child 마킹
marked_dup += 1
else:
row.extract_meta = with_paper_doi(meta, doi) # holder 스탬프, 인덱스 진입
stamped += 1
# 콘텐츠 무변경 → enqueue 없음(summarize/embed/chunk 0)
await session.commit()
logger.info(
f"[paper_doi_reconcile] {len(rows)}행 → arXiv 스탬프 {stamped} · 선재중복 {marked_dup} · "
f"arXiv id 없음 skip {skipped_no_arxiv} / 구매PDF parent_doi 링크 {linked_purchased} · "
f"구매PDF DOI 없음 skip {skipped_purchased_no_doi}"
)
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(description="paper DOI reconcile (arXiv 레거시 + 구매 PDF, keyless)")
parser.add_argument("--limit", type=int, default=0, help="처리 상한(0=전건)")
args = parser.parse_args()
asyncio.run(run(limit=args.limit))
+123
View File
@@ -0,0 +1,123 @@
"""논문 arXiv 전문 승격 (in-place) — B-3 Phase-2 P2-PR1 (plan safety-library-b3-1).
arXiv 프리프린트 초록 (file_format='article', signal-only) 전문 PDF로 **in-place 승격**:
PDF 다운로드 file_format/file_type/file_path/md_status 갱신 'extract' enqueue 기존 파이프라인
(extract classify[paper skip summarize] embed/chunk/markdown) 전문 검색 청크 + md_content(marker 표시)
+ hier 절구조를 생성. 1-Document(2 분리 회피, 기존 display 스택 재사용).
- arXiv = 공개 프리프린트(arxiv.org/pdf/{id}, friendly host) 전문 검색/RAG 무난, restricted 불요.
(유료 구매 논문은 Papers_Purchased 경로가 restricted=true 별개 처리.)
- per-run cap (marker GPU ~10GB + embed 부하 보호, 4070 16GB 빡빡 idle-unload·증분). keyless.
- 요약 0 (classify paper-skip 가드). file_hash·extract_meta.paper 보존(수집기 dedup 무영향).
- CLI 전용(Phase-2 deliberate 승격, GPU 부하 사용자 통제). 스케줄 미등록.
"""
import argparse
import asyncio
import random
from pathlib import Path
import httpx
from sqlalchemy import or_, select
from core.config import settings
from core.crawl_politeness import CRAWL_UA
from core.database import async_session
from core.utils import setup_logger
from models.document import Document
from models.queue import enqueue_stage
logger = setup_logger("paper_fulltext_promote")
_ARXIV_PDF = "https://arxiv.org/pdf/{id}"
_MAX_FILE_BYTES = 50 * 1024 * 1024
_DOWNLOAD_DELAY = (2.0, 5.0)
_RUN_CAP = 10 # 1회 승격 상한(marker/embed GPU 보호). bulk 시 해제.
_ARXIV_ID_EXPR = Document.extract_meta[("paper", "arxiv_id")].astext
_OA_URL_EXPR = Document.extract_meta[("paper", "oa_url")].astext
_OA_STATUS_EXPR = Document.extract_meta[("paper", "oa_status")].astext
_REAL_OA = ("gold", "hybrid", "green", "diamond")
async def _download(url: str, dest: Path) -> int:
"""arXiv PDF 다운로드 — 크기 cap + PDF 헤더 검증 + 연속 간격(kosha 패턴)."""
await asyncio.sleep(random.uniform(*_DOWNLOAD_DELAY))
async with httpx.AsyncClient(timeout=60, follow_redirects=True) as client:
resp = await client.get(url, headers={"User-Agent": CRAWL_UA})
if resp.status_code != 200:
raise RuntimeError(f"arXiv PDF {resp.status_code}: {url}")
if len(resp.content) > _MAX_FILE_BYTES:
raise RuntimeError(f"크기 초과 {len(resp.content)}b: {url}")
if resp.content[:5] != b"%PDF-":
raise RuntimeError(f"PDF 아님(헤더 {resp.content[:8]!r}): {url}")
dest.parent.mkdir(parents=True, exist_ok=True)
dest.write_bytes(resp.content)
return len(resp.content)
async def run(bulk: bool = False, limit: int = 0) -> None:
"""미승격 arXiv 논문(file_format='article')을 전문 PDF로 in-place 승격."""
cap = (limit or 10**9) if bulk else (min(limit, _RUN_CAP) if limit else _RUN_CAP)
async with async_session() as session:
q = (
select(Document.id)
.where(
Document.material_type == "paper",
Document.file_format == "article",
or_(
_ARXIV_ID_EXPR.isnot(None),
Document.extract_meta[("paper", "oa_url")].astext.isnot(None),
),
)
.order_by(Document.id.desc())
.limit(cap)
)
ids = [r[0] for r in (await session.execute(q)).all()]
promoted = failed = 0
for doc_id in ids:
async with async_session() as session:
doc = await session.get(Document, doc_id)
if doc is None or doc.file_format != "article":
continue
paper = (doc.extract_meta or {}).get("paper") or {}
arxiv_id = paper.get("arxiv_id")
oa_status = (paper.get("oa_status") or "").lower()
if arxiv_id:
url = _ARXIV_PDF.format(id=arxiv_id)
key = arxiv_id.replace("/", "_")
elif paper.get("oa_url") and oa_status in _REAL_OA:
url = paper["oa_url"] # doi.org/KISTI/PMC (friendly OA). 비-OA·paywall 은 헤더검증서 skip
key = (paper.get("openalex_id") or paper.get("doi") or "oa").replace("/", "_")
else:
continue
rel_path = f"crawl_raw/papers/{key}.pdf"
dest = Path(settings.nas_mount_path) / rel_path
try:
size = await _download(url, dest)
except Exception as e: # noqa: BLE001 — 다운로드 실패 격리
logger.error(f"[promote] {key} 다운로드 실패: {e}")
failed += 1
continue
# in-place 승격: 초록 행 → 전문 PDF 행 (file_hash·extract_meta.paper 보존)
doc.file_path = rel_path
doc.file_format = "pdf"
doc.file_type = "immutable"
doc.file_size = size
doc.md_status = "pending" # marker 재실행(기존 'skipped' 해제)
doc.md_extraction_error = None
await enqueue_stage(session, doc.id, "extract")
await session.commit()
promoted += 1
logger.info(f"[promote] {key} → 전문 PDF in-place (doc {doc.id}, {size}b)")
logger.info(f"[paper_fulltext_promote] 승격 {promoted} · 실패 {failed} (cap {cap})")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="논문 arXiv 전문 승격 (in-place, keyless)")
parser.add_argument("--bulk", action="store_true", help="cap 해제(전건 백필 — GPU 부하 주의)")
parser.add_argument("--limit", type=int, default=0, help="승격 상한(0=기본 cap 10)")
args = parser.parse_args()
asyncio.run(run(bulk=args.bulk, limit=args.limit))
@@ -1,16 +1,18 @@
<script lang="ts">
// 처리 머신 보드 v2파이프라인 흐름 뷰 (plan ds-board-engines-1, R2 통합안).
// 메인 = 좌→우 흐름 노드(병목 amber·실패 뱃지), 노드 클릭 = 상세 패널(안1 변형),
// 실패 뱃지 클릭 = 실패 처리 드로어 (재시도/건너뛰기 — 영구 실패의 유일한 조치 경로).
// 데이터 = GET /api/queue/overview (60s 폴링 store) + GET /api/queue/failed (드로어 열 때).
// 처리 머신 보드 v3통합안 (plan ds-board-merged: C2 머신레인 + C3 번다운/정직ETA).
// · 머신 3레인(GPU/맥미니/맥북) = "누가 일하나" + 요약 오프로드(맥북 합류) 가시화
// · 지배 백로그 번다운 패널 = "언제 끝나나" + 유입 차감한 정직 ETA(summarize_eta)
// · 신선도 '갱신 N초 전' + stale 경고 / 실패 드로어·상세 패널은 v2 자산 재사용.
// 데이터 = GET /api/queue/overview (60s 폴링 store) + GET /api/queue/failed (드로어).
import { api } from '$lib/api';
import { refreshQueueOverview } from '$lib/stores/queueOverview';
import { refreshQueueOverview, queueUpdatedAt } from '$lib/stores/queueOverview';
import { addToast } from '$lib/stores/toast';
import {
AUX_NODES,
FLOW_NODES,
MACHINE_META,
type FlowNodeDef,
type FlowMachine,
etaShort,
flowStageLabel,
formatAgeSec,
@@ -20,6 +22,7 @@
FailedItem,
FailedListResponse,
MachineCurrentItem,
MachineOverview,
QueueOverview,
QueueStageRow,
RetryResponse,
@@ -82,14 +85,6 @@
);
const totalFailed = $derived(overview.totals.failed);
// 머신 스트립 — overview.machines 의 state/처리율 + 정적 모델 메타
const machineStrip = $derived(
overview.machines.map((m) => ({
...m,
meta: MACHINE_META[m.key],
})),
);
// ─── 선택 상태 (노드 상세 / 실패 드로어 — 동시에 하나만) ───
let selected = $state<string | null>(null);
let failOpen = $state(false);
@@ -194,22 +189,118 @@
await Promise.all([loadFailures(), refreshQueueOverview()]);
}
// ─── trend_24h 스파크라인 (summarize 유입 vs 소화 — API 가 주는데 미렌더이던 슬롯) ───
const spark = $derived.by(() => {
// ─── 머신 레인 (C2) — mainNodes 를 머신별로 그룹 + 머신 카드(state/처리율) 결합 ───
const machineByKey = $derived(
new Map<FlowMachine, MachineOverview>(overview.machines.map((m) => [m.key as FlowMachine, m])),
);
const LANE_ORDER: FlowMachine[] = ['gpu', 'macmini', 'macbook'];
const lanes = $derived(
LANE_ORDER.map((key) => ({
key,
meta: MACHINE_META[key],
card: machineByKey.get(key) ?? null,
nodes: mainNodes.filter((n) => n.def.machine === key),
})),
);
// 요약 오프로드 분담 — 맥미니 vs 맥북 (A-1 summarize_by_machine)
const split = $derived(overview.summarize_by_machine);
const splitTotal1h = $derived(Math.max(1, split.macmini.done_1h + split.macbook.done_1h));
const macbookSharePct = $derived(Math.round((split.macbook.done_1h / splitTotal1h) * 100));
// 맥북이 요약을 실제로 가져가는 중인가 (합류 표식 게이트)
const offloadActive = $derived(split.macbook.done_1h > 0);
// ─── 백그라운드 작업 (큐 밖 스크립트 backfill) — processing_queue 사각지대 노출 ───
const bgJobs = $derived(overview.background_jobs ?? []);
function fmtElapsed(s: number): string {
if (s < 60) return `${s}s`;
if (s < 3600) return `${Math.floor(s / 60)}m`;
return `${Math.floor(s / 3600)}h${Math.floor((s % 3600) / 60)}m`;
}
function bgDot(j: { state: string; stale: boolean }): string {
if (j.state === 'running') return j.stale ? 'bg-warning' : 'bg-success';
if (j.state === 'failed') return 'bg-error';
return 'bg-faint';
}
// ─── 지배 백로그 = 요약. 정직 ETA(유입 차감) — summarize_eta ───
const eta = $derived(overview.summarize_eta);
// 정직 ETA 라벨: eta_minutes null = 유입이 소화를 앞섬(소진 불가)
const honestEtaLabel = $derived(
eta.pending === 0
? '비어 있음'
: eta.eta_minutes != null
? etaShort(eta.eta_minutes)
: '소진 불가',
);
const honestEtaWarn = $derived(eta.pending > 0 && eta.eta_minutes == null);
/** 단계별 정직 ETA(순소화율) — 노드용. 유입>소화면 null(소진 불가) */
function netEtaLabel(n: NodeStats): string | null {
if (n.pending === 0) return '한가';
const net = n.done1h - n.created1h;
if (net > 0) return etaShort(Math.round((n.pending / net) * 60));
if (n.created1h > n.done1h) return '유입 우세';
return null;
}
// ─── 신선도 (B-4) — '갱신 N초 전' + stale 경고 (폴링 60s) ───
let now = $state(Date.now());
$effect(() => {
const id = setInterval(() => (now = Date.now()), 1000);
return () => clearInterval(id);
});
const ageSec = $derived(
$queueUpdatedAt != null ? Math.max(0, Math.round((now - $queueUpdatedAt) / 1000)) : null,
);
const stale = $derived(ageSec != null && ageSec > 90);
const freshLabel = $derived(
ageSec == null
? '갱신 대기'
: ageSec < 60
? `갱신 ${ageSec}초 전`
: `갱신 ${Math.round(ageSec / 60)}분 전`,
);
// ─── 24h 번다운 (C3) — 요약 유입 vs 소화 + 맥북 합류 변곡점 마커 ───
const burn = $derived.by(() => {
const t = overview.trend_24h;
if (!t || t.length === 0) return null;
const max = Math.max(1, ...t.map((b) => Math.max(b.inflow, b.done)));
const w = 120;
const h = 24;
const w = 300;
const h = 64;
const step = w / Math.max(1, t.length - 1);
const pts = (sel: (b: (typeof t)[number]) => number) =>
t.map((b, i) => `${(i * step).toFixed(1)},${(h - (sel(b) / max) * (h - 3) + 1).toFixed(1)}`).join(' ');
return { inflow: pts((b) => b.inflow), done: pts((b) => b.done) };
const y = (v: number) => (h - (v / max) * (h - 8) + 4).toFixed(1);
const line = (sel: (b: (typeof t)[number]) => number) =>
t.map((b, i) => `${(i * step).toFixed(1)},${y(sel(b))}`).join(' ');
const doneLine = line((b) => b.done);
const area = `0,${h} ${doneLine} ${w.toFixed(1)},${h}`;
// 합류 변곡점 = done 최대 버킷 (맥북 야간 drain 합류 추정)
let mi = 0;
t.forEach((b, i) => {
if (b.done > t[mi].done) mi = i;
});
return {
w,
h,
area,
doneLine,
inflowLine: line((b) => b.inflow),
markX: (mi * step).toFixed(1),
markHour: t[mi].hour,
markDone: t[mi].done,
peak: max,
};
});
// 머신 상태 dot 색 클래스
function dotClass(state: string): string {
return state === 'active' ? 'bg-success' : state === 'deferred' ? 'bg-warning' : 'bg-faint';
}
</script>
<div class="mt-5">
<!-- 헤더: 타이틀 + 요약 24h 스파크라인 + 실패 합계 -->
<!-- 헤더: 타이틀 + 신선도 + 실패 합계 -->
<div class="flex items-center justify-between gap-3 mb-3">
<div class="text-[11px] font-bold text-dim uppercase tracking-wider">처리 머신</div>
<div class="flex items-center gap-3">
@@ -219,80 +310,107 @@
onclick={openFailures}
>실패 {totalFailed}건 처리</button>
{/if}
{#if spark}
<div class="flex items-center gap-2 text-[10px] text-faint tabular-nums" title="요약(summarize) 단계 24시간 — 유입(회색) vs 소화(녹색)">
<svg width="120" height="24" viewBox="0 0 120 24" class="block">
<polyline points={spark.inflow} fill="none" stroke="currentColor" stroke-width="1.5" class="text-faint" />
<polyline points={spark.done} fill="none" stroke="currentColor" stroke-width="1.5" class="text-success" />
</svg>
<span>요약 24h 유입/소화</span>
</div>
{/if}
<span class="flex items-center gap-1.5 text-[10px] tabular-nums {stale ? 'text-warning' : 'text-faint'}" title="60초 폴링">
<span class="w-1.5 h-1.5 rounded-full {stale ? 'bg-warning' : 'bg-success'}"></span>
{freshLabel}{#if stale} · 갱신 지연{/if}
</span>
</div>
</div>
<!-- 머신 스트립 -->
<div class="flex flex-wrap gap-2 mb-3">
{#each machineStrip as m (m.key)}
<div class="flex items-center gap-2 bg-surface border border-default rounded-full px-3.5 py-1.5 text-xs">
<span class="w-2 h-2 rounded-full shrink-0 {m.state === 'active' ? 'bg-success' : m.state === 'deferred' ? 'bg-warning' : 'bg-faint'}"></span>
<span class="font-bold text-text">{m.meta?.label ?? m.label}</span>
<span class="text-[10px] text-faint font-mono">{m.meta?.model}</span>
<span class="text-[11px] text-dim tabular-nums">{formatRate(m.done_1h)}/h</span>
{#if m.key === 'macbook' && m.deferred_pending > 0}
<span class="text-[10px] font-semibold text-warning tabular-nums">보류 {m.deferred_pending}</span>
{/if}
<!-- 지배 백로그 스트립 (요약) + 정직 ETA -->
<div class="flex items-center flex-wrap gap-x-3 gap-y-1 bg-surface border border-warning/50 rounded-card px-3.5 py-2 mb-3">
<span class="text-[9px] font-bold text-warning border border-warning/60 rounded-full px-2 py-px">지배 백로그</span>
<span class="text-xs font-bold text-text">요약</span>
<span class="text-[11px] text-dim tabular-nums">대기 <b class="text-text">{eta.pending.toLocaleString()}</b> · 순소화 <b class="text-text">{formatRate(eta.done_rate_1h)}</b>/h · 유입 {formatRate(eta.inflow_rate_1h)}/h</span>
<span class="ml-auto flex items-center gap-1.5 border rounded-full px-2.5 py-0.5 {honestEtaWarn ? 'border-warning text-warning' : 'border-accent text-accent'}">
<span class="text-[10px] font-semibold">정직 ETA</span>
<span class="text-xs font-bold tabular-nums">{honestEtaLabel}</span>
</span>
</div>
<!-- 머신 레인 (누가 일하나 + 요약 오프로드) -->
<div class="grid gap-2 mb-3">
{#each lanes as lane (lane.key)}
<div class="bg-surface border border-default rounded-card px-3.5 py-2.5">
<div class="flex items-center gap-2 flex-wrap mb-2">
<span class="w-2 h-2 rounded-full shrink-0 {dotClass(lane.card?.state ?? 'idle')}"></span>
<span class="text-[9px] font-bold rounded px-1.5 py-px mtag-{lane.key}">{lane.meta.label}</span>
<span class="text-[10px] text-faint font-mono">{lane.meta.model}</span>
<span class="text-[11px] text-dim tabular-nums ml-1">{formatRate(lane.card?.done_1h ?? 0)}/h</span>
{#if lane.key === 'macbook' && (lane.card?.deferred_pending ?? 0) > 0}
<span class="text-[10px] font-semibold text-warning tabular-nums">보류 {lane.card?.deferred_pending}</span>
{/if}
{#if lane.card?.state === 'deferred'}
<span class="text-[9px] text-warning">잠듦 — 요약은 맥미니로 복귀</span>
{/if}
</div>
<div class="flex items-stretch gap-1.5 flex-wrap">
{#each lane.nodes as n (n.def.key)}
{@const idle = n.pending + n.processing + n.doneToday + n.failed === 0}
<button
class="relative text-left rounded-lg border px-2.5 py-1.5 transition-colors cursor-pointer hover:bg-surface-hover min-w-[96px]
{idle ? 'border-dashed border-default opacity-55' : n.inflowDominant ? 'border-warning' : 'border-default'}
{selected === n.def.key ? 'node-sel' : ''}"
onclick={() => toggleNode(n.def.key)}
title="{n.def.label} — 클릭하면 상세"
>
{#if n.failed > 0}
<span class="absolute -top-1.5 -right-1 text-[9px] font-extrabold bg-error text-white rounded-full px-1.5">{n.failed}</span>
{/if}
<div class="flex items-center gap-1 text-[11px] font-semibold text-text whitespace-nowrap">
{n.def.label}
{#if n.processing > 0}<span class="inline-block w-1.5 h-1.5 rounded-full bg-accent animate-pulse"></span>{/if}
</div>
<div class="text-sm font-extrabold tabular-nums leading-tight text-text">{n.pending.toLocaleString()}<span class="text-[9px] text-faint font-normal ml-0.5">대기</span></div>
<div class="text-[9px] text-dim tabular-nums whitespace-nowrap">{formatRate(n.done1h)}/h · 오늘 {n.doneToday.toLocaleString()}</div>
{#if n.def.key === 'summarize'}
<div class="mt-1 h-1 w-full rounded-full overflow-hidden flex" title="맥미니 {split.macmini.done_1h}/h · 맥북 {split.macbook.done_1h}/h">
<span class="block h-full mtag-macmini-bar" style="width:{100 - macbookSharePct}%"></span>
<span class="block h-full mtag-macbook-bar" style="width:{macbookSharePct}%"></span>
</div>
<div class="text-[9px] text-faint tabular-nums whitespace-nowrap mt-0.5">맥미니 {split.macmini.done_1h} · 맥북 {split.macbook.done_1h}/h</div>
{/if}
</button>
{/each}
{#if lane.key === 'macbook' && offloadActive}
<button
class="text-left rounded-lg border border-dashed border-warning/50 px-2.5 py-1.5 cursor-pointer hover:bg-surface-hover min-w-[96px]"
onclick={() => toggleNode('summarize')}
title="맥북이 요약을 맥미니에서 가져와 처리 중"
>
<div class="flex items-center gap-1 text-[11px] font-semibold text-text whitespace-nowrap">요약 합류 <span class="text-[8px] font-bold text-warning">OFFLOAD</span></div>
<div class="text-sm font-extrabold tabular-nums leading-tight text-text">{split.macbook.done_1h}<span class="text-[9px] text-faint font-normal ml-0.5">/h</span></div>
<div class="text-[9px] text-dim tabular-nums whitespace-nowrap">요약의 {macbookSharePct}% 담당</div>
</button>
{/if}
</div>
</div>
{/each}
</div>
<!-- 흐름 노드 — pt/px 헤드룸 = 실패 뱃지(-top/-right 돌출)가 스크롤 컨테이너에 잘리지 않게 -->
<div class="flex items-stretch overflow-x-auto pt-2.5 pb-1 px-2 -mx-2">
{#each mainNodes as n, i (n.def.key)}
{#if i > 0}
<div class="flex items-center text-faint text-sm px-1.5 shrink-0" aria-hidden="true"></div>
{/if}
<div
class="relative bg-surface border-[1.5px] rounded-card px-3 py-2.5 min-w-[124px] shrink-0 text-left transition-colors cursor-pointer hover:bg-surface-hover
{n.inflowDominant ? 'border-warning' : n.etaMinutes != null && n.def.stages.includes('chunk') ? 'border-success' : 'border-default'}
{selected === n.def.key ? 'node-sel' : ''}"
role="button"
tabindex="0"
onclick={() => toggleNode(n.def.key)}
onkeydown={(e) => { if (e.key === 'Enter' || e.key === ' ') { e.preventDefault(); toggleNode(n.def.key); } }}
title="{n.def.label} — 클릭하면 상세"
>
{#if n.failed > 0}
<button
class="absolute -top-2 -right-1.5 text-[9px] font-extrabold bg-error text-white rounded-full px-1.5 py-px shadow cursor-pointer"
onclick={(e) => { e.stopPropagation(); openFailures(); }}
title="실패 {n.failed}건 — 클릭하면 실패 처리"
>{n.failed}</button>
{/if}
<span class="inline-block text-[9px] font-bold rounded px-1.5 py-px mb-1.5 mtag-{n.def.machine}">
{MACHINE_META[n.def.machine].label} · {n.def.engine}
</span>
<div class="text-xs font-bold text-text flex items-center gap-1.5">
{n.def.label}
{#if n.processing > 0}
<span class="inline-block w-1.5 h-1.5 rounded-full bg-accent animate-pulse" title="처리 중 {n.processing}"></span>
{/if}
{#if n.inflowDominant}
<span class="text-[9px] font-bold text-warning">유입 우세</span>
{/if}
</div>
<div class="text-base font-extrabold tabular-nums tracking-tight leading-tight mt-0.5 text-text">
{n.pending.toLocaleString()}
</div>
<div class="text-[10px] text-dim tabular-nums">
{formatRate(n.done1h)}/h · 오늘 {n.doneToday.toLocaleString()}
{#if n.etaMinutes != null && !n.inflowDominant && n.pending > 0}
· <span class="text-accent font-semibold">{etaShort(n.etaMinutes)}</span>
{/if}
</div>
<!-- 번다운 / ETA 패널 -->
{#if burn}
<div class="bg-surface border border-default rounded-card px-3.5 py-3 mb-1">
<div class="flex items-center gap-2 mb-2">
<span class="text-[11px] font-bold text-text">요약 백로그 24시간</span>
<span class="text-[9px] text-faint">유입(회색) vs 소화(녹색)</span>
{#if offloadActive}<span class="text-[9px] text-warning ml-auto">맥북 합류 {burn.markHour} — 소화 급증</span>{/if}
</div>
{/each}
</div>
<svg viewBox="0 0 {burn.w} {burn.h}" class="block w-full" style="height:64px" preserveAspectRatio="none" role="img" aria-label="요약 백로그 24시간 번다운">
<polygon points={burn.area} fill="currentColor" class="text-success" opacity="0.12" />
<polyline points={burn.inflowLine} fill="none" stroke="currentColor" stroke-width="1.2" class="text-faint" />
<polyline points={burn.doneLine} fill="none" stroke="currentColor" stroke-width="1.6" class="text-success" />
{#if offloadActive}
<line x1={burn.markX} y1="0" x2={burn.markX} y2={burn.h} stroke="currentColor" stroke-width="1" stroke-dasharray="2 2" class="text-warning" opacity="0.7" />
{/if}
</svg>
<div class="flex flex-wrap gap-x-4 gap-y-1 mt-2 pt-2 border-t border-default text-[10px] text-dim tabular-nums">
{#each mainNodes.filter((n) => n.pending > 0 && n.def.key !== 'summarize') as n (n.def.key)}
<span class="whitespace-nowrap">{n.def.label} 대기 <b class="text-text">{n.pending.toLocaleString()}</b>{#if netEtaLabel(n)} · <span class="text-accent font-semibold">{netEtaLabel(n)}</span>{/if}</span>
{/each}
</div>
</div>
{/if}
<!-- 보조 라인 -->
<p class="text-[10px] text-faint mt-1.5 tabular-nums">
@@ -361,6 +479,32 @@
</div>
{/if}
<!-- 백그라운드 작업 (큐 밖 스크립트 backfill 등 — processing_queue 가 못 보는 사각지대) -->
{#if bgJobs.length > 0}
<div class="mt-3">
<div class="text-[11px] font-bold text-dim uppercase tracking-wider mb-2">백그라운드 작업</div>
<div class="grid gap-2">
{#each bgJobs as j (j.id)}
<div class="bg-surface border rounded-card px-3.5 py-2.5 {j.stale ? 'border-warning' : j.state === 'failed' ? 'border-error' : 'border-default'}">
<div class="flex items-center gap-2 flex-wrap">
<span class="w-2 h-2 rounded-full shrink-0 {bgDot(j)}"></span>
<span class="text-[9px] font-bold rounded px-1.5 py-px bg-default text-dim font-mono">{j.kind}</span>
<span class="text-xs font-semibold text-text truncate">{j.label ?? '작업'}</span>
<span class="text-[11px] text-dim tabular-nums ml-auto">
{#if j.total}{j.processed.toLocaleString()}/{j.total.toLocaleString()}{:else}{j.processed.toLocaleString()}{/if} · {fmtElapsed(j.elapsed_sec)}
</span>
</div>
{#if j.stale}
<div class="text-[10px] text-warning mt-1.5">heartbeat 끊김 — 프로세스 중단 추정 (재개 필요할 수 있음)</div>
{:else if j.state === 'failed'}
<div class="text-[10px] text-error mt-1.5 truncate">실패{#if j.error} · {j.error}{/if}</div>
{/if}
</div>
{/each}
</div>
</div>
{/if}
<!-- 실패 처리 드로어 -->
{#if failOpen}
<div class="border border-error/40 rounded-card mt-3 overflow-hidden bg-surface">
@@ -413,6 +557,9 @@
.mtag-gpu { background: #e7eef6; color: #3b6ea5; }
.mtag-macmini { background: #efe9f7; color: #8a5fbf; }
.mtag-macbook { background: #f7eedd; color: #b07a10; }
/* 요약 오프로드 분담 막대 채움 (맥미니 보라 / 맥북 황) */
.mtag-macmini-bar { background: #8a5fbf; }
.mtag-macbook-bar { background: #b07a10; }
.node-sel { outline: 2px solid #3b6ea5; outline-offset: 1px; }
.detail-frame { border-color: #3b6ea5; }
.detail-head { background: #e7eef6; }
@@ -43,14 +43,17 @@
{@const open = selectedId === s.chunk_id}
{@const active = activeKey != null && activeKey === s.chunk_id}
{@const typeLabel = sectionTypeLabel(s.section_type)}
{@const depth = Math.max(0, (s.level ?? 1) - 1)}
<li>
<button
type="button"
onclick={() => { toggle(item); onJump?.(s.chunk_id); }}
aria-expanded={open}
aria-current={active ? 'true' : undefined}
style="padding-left:{8 + depth * 13}px"
class={[
'w-full text-left px-2 py-1.5 rounded-md text-xs flex items-start gap-1.5 transition-colors border-l-2',
'w-full text-left pr-2 py-1.5 rounded-md text-xs flex items-start gap-1.5 transition-colors border-l-2',
depth > 0 ? 'text-[11px]' : '',
open ? 'bg-surface-active text-text border-accent' : active ? 'bg-surface text-accent-hover border-accent' : 'text-dim hover:bg-surface hover:text-text border-transparent',
].join(' ')}
>
+8 -1
View File
@@ -17,6 +17,11 @@ let pollHandle: ReturnType<typeof setInterval> | null = null;
let subscriberCount = 0;
let inFlight: Promise<void> | null = null;
// 마지막 성공 갱신 시각(epoch ms) — 보드 신선도 '갱신 N초 전' + stale 경고용
// (ds-board-merged B-4). 실패(null 수렴) 시엔 갱신 안 함 → age 가 늘어 stale 로 드러남.
const updatedAt = writable<number | null>(null);
export const queueUpdatedAt = { subscribe: updatedAt.subscribe };
const internal = writable<QueueOverview | null>(null, (_set) => {
subscriberCount += 1;
if (subscriberCount === 1 && browser) {
@@ -54,7 +59,9 @@ export async function refreshQueueOverview(): Promise<void> {
if (inFlight) return inFlight;
inFlight = (async () => {
try {
internal.set(await fetchOverview());
const ov = await fetchOverview();
internal.set(ov);
if (ov) updatedAt.set(Date.now()); // 성공 시에만 신선도 갱신 (실패=stale 유지)
} finally {
inFlight = null;
}
+23 -1
View File
@@ -43,13 +43,19 @@ export interface SummarizeEta {
eta_minutes: number | null;
}
/** 시간당 유입 vs 소화 (이번 트랙 미렌더 — 후속 추세 위젯 슬롯) */
/** 시간당 유입 vs 소화 (요약 24h 추이) */
export interface TrendPoint {
hour: string;
inflow: number;
done: number;
}
/** summarize 머신별 완료 실적 분담 (오프로드 가시화 — ds-board-merged A-1) */
export interface SummarizeByMachine {
macmini: { done_1h: number; done_today: number };
macbook: { done_1h: number; done_today: number };
}
export interface QueueTotals {
pending: number;
processing: number;
@@ -69,12 +75,28 @@ export interface QueueStageRow {
oldest_pending_age_sec: number | null;
}
/** ( ) processing_queue .
* stale = running heartbeat ( ). */
export interface BackgroundJob {
id: number;
kind: string;
label: string | null;
state: 'running' | 'done' | 'failed';
processed: number;
total: number | null;
elapsed_sec: number;
stale: boolean;
error: string | null;
}
export interface QueueOverview {
machines: MachineOverview[];
summarize_eta: SummarizeEta;
summarize_by_machine: SummarizeByMachine;
trend_24h: TrendPoint[];
stages: QueueStageRow[];
totals: QueueTotals;
background_jobs?: BackgroundJob[];
}
/** ─── 실패 처리 (ds-board-engines-1) — GET /api/queue/failed · POST /retry|/skip ─── */
@@ -83,6 +83,74 @@ test('[C2] collapseWindows: split-parent + window 들 → rail 1행, 대표=spli
assert.equal(out[0].fragmentCount, 2, 'window 조각 수 = 2 (split-parent 자신 제외)');
});
test('collapseWindows: bodyText — 정상 leaf 는 자기 본문, split-parent 는 window 본문만 이어붙임', () => {
// 정상 leaf → 자기 text 가 본문
const leaf = collapseWindows([sec({ heading_path: 'Intro', node_type: null, text: '서론 본문' })]);
assert.equal(leaf[0].bodyText, '서론 본문');
// split-parent(heading 줄뿐) + window 2개 → window 본문만 순서대로 합침(헤딩 제외)
const split = collapseWindows([
sec({ heading_path: 'Article 5', node_type: 'chapter_split', is_leaf: false, char_start: 120, text: '# Article 5' }),
sec({ heading_path: 'Article 5', node_type: 'window', is_leaf: true, text: '본문 조각1' }),
sec({ heading_path: 'Article 5', node_type: 'window', is_leaf: true, text: '본문 조각2' }),
]);
assert.equal(split.length, 1);
assert.equal(split[0].bodyText, '본문 조각1\n\n본문 조각2', 'split-parent heading 제외, window 본문만 합침');
// legacy window 런(선행 split-parent 없음) → 첫 window 자기 본문 + 흡수 조각
const legacy = collapseWindows([
sec({ heading_path: 'Pearson', node_type: 'window', text: 'p1' }),
sec({ heading_path: 'Pearson', node_type: 'window', text: 'p2' }),
]);
assert.equal(legacy.length, 1);
assert.equal(legacy[0].bodyText, 'p1\n\np2');
});
test('collapseWindows: 절-레벨 분석 집계 — windowed 절은 window 멤버에서 type 다수결/conf 평균/summaries 합본', () => {
// split-parent(분석 없음) + window 3개(요약·유형·신뢰도 보유) → 대표에 집계
const out = collapseWindows([
sec({ heading_path: 'Sec A', node_type: 'section_split', is_leaf: false, char_start: 10, text: '# Sec A', section_type: null, summary: null, confidence: null }),
sec({ heading_path: 'Sec A', node_type: 'window', text: 'b1', section_type: 'requirement', summary: '요약1', confidence: 0.9 }),
sec({ heading_path: 'Sec A', node_type: 'window', text: 'b2', section_type: 'requirement', summary: '요약2', confidence: 0.8 }),
sec({ heading_path: 'Sec A', node_type: 'window', text: 'b3', section_type: 'overview', summary: '', confidence: 1.0 }),
]);
assert.equal(out.length, 1);
assert.equal(out[0].sectionType, 'requirement', '다수결 = requirement(2) > overview(1)');
assert.ok(Math.abs(out[0].confidence! - 0.9) < 1e-9, '평균 (0.9+0.8+1.0)/3 = 0.9');
assert.deepEqual(out[0].summaries, ['요약1', '요약2'], '빈 요약 제외, 순서 유지');
// 단일 leaf 는 대표 자신의 분석
const single = collapseWindows([sec({ heading_path: 'X', node_type: null, text: 'body', section_type: 'definition', summary: '정의 요약', confidence: 0.7 })]);
assert.equal(single[0].sectionType, 'definition');
assert.equal(single[0].confidence, 0.7);
assert.deepEqual(single[0].summaries, ['정의 요약']);
// 분석 전혀 없는 절 → null/빈
const none = collapseWindows([sec({ heading_path: 'Y', node_type: null, text: 'body' })]);
assert.equal(none[0].sectionType, null);
assert.equal(none[0].confidence, null);
assert.deepEqual(none[0].summaries, []);
});
test('collapseWindows: 비인접 window 도 parent_id 로 split-parent 에 흡수 (빈 split 행 방지)', () => {
// 실데이터 버그: split-parent(chunk_index 1143)와 그 window(1233~)가 비인접 → 인접 흡수 실패로
// 빈 split 행 + 별도 window-그룹 행 2개로 쪼개짐. parent_id 링크로 정확히 합친다.
const out = collapseWindows([
sec({ chunk_id: 10, heading_path: 'FOREWORD', node_type: 'section_split', is_leaf: false, char_start: 5, text: '# FOREWORD' }),
sec({ chunk_id: 11, heading_path: 'POLICY', node_type: null, text: '정책 본문' }), // 사이에 낀 다른 절
sec({ chunk_id: 12, heading_path: 'FOREWORD', node_type: 'window', parent_id: 10, text: '서문 조각1', section_type: 'overview', summary: '요약A', confidence: 0.9 }),
sec({ chunk_id: 13, heading_path: 'FOREWORD', node_type: 'window', parent_id: 10, text: '서문 조각2', section_type: 'overview', summary: '요약B', confidence: 0.8 }),
]);
assert.equal(out.length, 2, 'FOREWORD(split, window 흡수) + POLICY = 2행 (빈 split 행 없음)');
assert.equal(out[0].section.chunk_id, 10, '대표 = split-parent(char_start 보유)');
assert.equal(out[0].bodyText, '서문 조각1\n\n서문 조각2', '비인접 window 본문을 split-parent 에 흡수');
assert.equal(out[0].fragmentCount, 2);
assert.equal(out[0].sectionType, 'overview');
assert.deepEqual(out[0].summaries, ['요약A', '요약B']);
assert.equal(out[1].section.chunk_id, 11, '사이 낀 절은 별도 행 유지');
assert.equal(out[1].bodyText, '정책 본문');
});
test('groupOrFlat: 적은 그룹 + 낮은 기타% → group (5140-류)', () => {
// 3 top segment × 4 = 12절, window 없음 → group_count 3, 기타 0%
const sections: DocumentSection[] = [];
+81 -10
View File
@@ -14,8 +14,12 @@ export interface DocumentSection {
level: number | null;
node_type: string | null; // 'window' | 'chapter_split' | 'clause_split' | 'section_split' | null
is_leaf: boolean;
/** 트리 부모 chunk_id. window child 의 parent_id = 그 split-parent (비인접 흡수에 사용). */
parent_id?: number | null;
/** md_content 내 heading offset(UTF-16). jump-target 만 값, window-child/preamble/Path A = null (Path B). */
char_start?: number | null;
/** 절 본문 = 청크 원문. split-parent 는 heading 줄뿐, window child 가 실 본문 보유. */
text?: string | null;
section_type: string | null;
summary: string | null;
confidence: number | null;
@@ -25,6 +29,17 @@ export interface DocumentSection {
export interface OutlineItem {
section: DocumentSection;
fragmentCount: number; // >1 이면 "(n조각)" 배지
/** + window child .
* split-parent heading (text) ( ) window . */
bodyText: string;
/** - . windowed window child(chunk_section_analysis)
* =split-parent . .
* - sectionType: 멤버 section_type (= )
* - confidence: 멤버 confidence
* - summaries: 멤버 ( , chunk_index ) =1, windowed=N개( ) */
sectionType: string | null;
confidence: number | null;
summaries: string[];
}
export interface OutlineGroup {
@@ -107,22 +122,78 @@ function topSegment(s: DocumentSection): string {
* fragmentCount: split-parent 0 ( ) + child = ;
* legacy window 1 ( ).
*/
/** 멤버 section_type 다수결(동률은 첫 등장 우선). 비어있으면 null. */
function majorityType(types: (string | null)[]): string | null {
const vals = types.filter((t): t is string => !!t);
if (!vals.length) return null;
const count = new Map<string, number>();
for (const t of vals) count.set(t, (count.get(t) ?? 0) + 1);
let best: string | null = null;
let bestN = -1;
for (const t of vals) {
const n = count.get(t)!;
if (n > bestN) { bestN = n; best = t; } // 첫 등장 우선 tie-break
}
return best;
}
export function collapseWindows(sections: DocumentSection[]): OutlineItem[] {
const out: OutlineItem[] = [];
const members: DocumentSection[][] = []; // out[i] 의 멤버(대표 + 흡수된 window child)
const repByChunkId = new Map<number, number>(); // split-parent chunk_id → out index (window 가 parent_id 로 흡수)
// window child 본문/멤버를 out[idx] 대표에 흡수.
const absorb = (idx: number, s: DocumentSection) => {
out[idx].fragmentCount += 1;
const t = (s.text ?? '').trim();
if (t) out[idx].bodyText = out[idx].bodyText ? `${out[idx].bodyText}\n\n${t}` : t;
members[idx].push(s);
};
for (const s of sections) {
const prev = out[out.length - 1];
const h = cleanHeading(s.heading_path);
const prevAbsorbs =
prev &&
(prev.section.node_type === 'window' || !!prev.section.node_type?.endsWith('_split')) &&
h !== '' &&
cleanHeading(prev.section.heading_path) === h;
if (s.node_type === 'window' && prevAbsorbs) {
prev!.fragmentCount += 1; // window child 흡수 — 대표(split-parent 우선)는 그대로 유지
if (s.node_type === 'window') {
// 1) parent_id 로 split-parent 대표에 흡수 — split-parent 와 window 가 chunk_index 상 비인접일 수
// 있으므로(예: 헤딩 1143, window 1233) 인접 가정 대신 트리 부모 링크로 정확히 연결한다.
let idx = s.parent_id != null ? repByChunkId.get(s.parent_id) ?? -1 : -1;
// 2) fallback: 인접 대표(legacy window run / 같은 heading split)면 흡수
if (idx < 0) {
const prev = out[out.length - 1];
const h = cleanHeading(s.heading_path);
if (
prev &&
(prev.section.node_type === 'window' || !!prev.section.node_type?.endsWith('_split')) &&
h !== '' &&
cleanHeading(prev.section.heading_path) === h
) {
idx = out.length - 1;
}
}
if (idx >= 0) {
absorb(idx, s);
continue;
}
// 3) legacy: 부모 없는 window → 자기 대표(자기 본문으로 시작)
out.push({ section: s, fragmentCount: 1, bodyText: s.text ?? '', sectionType: null, confidence: null, summaries: [] });
members.push([s]);
} else {
out.push({ section: s, fragmentCount: s.node_type?.endsWith('_split') ? 0 : 1 });
const isSplit = !!s.node_type?.endsWith('_split');
// split-parent 의 text 는 heading 줄뿐 → 본문에서 제외(window 가 본문 보유). 그 외엔 자기 본문으로 시작.
out.push({
section: s, fragmentCount: isSplit ? 0 : 1, bodyText: isSplit ? '' : (s.text ?? ''),
sectionType: null, confidence: null, summaries: [],
});
members.push([s]);
if (isSplit) repByChunkId.set(s.chunk_id, out.length - 1); // window 가 parent_id 로 찾아 흡수
}
}
// 멤버에서 절-레벨 분석 집계 (windowed 절: 대표 split-parent 엔 분석 없고 window 들이 보유).
for (let i = 0; i < out.length; i++) {
const mem = members[i];
out[i].sectionType = majorityType(mem.map((m) => m.section_type));
const confs = mem.map((m) => m.confidence).filter((c): c is number => c != null);
out[i].confidence = confs.length ? confs.reduce((a, b) => a + b, 0) / confs.length : null;
out[i].summaries = mem.map((m) => (m.summary ?? '').trim()).filter((x) => x !== '');
}
return out;
}
+22 -54
View File
@@ -8,8 +8,7 @@
import { goto } from '$app/navigation';
import { api } from '$lib/api';
import { addToast } from '$lib/stores/toast';
import { Info, X, Plus, Trash2, Tag, FolderTree, Sparkles, ChevronLeft, ArrowUpDown } from 'lucide-svelte';
import DocumentViewer from '$lib/components/DocumentViewer.svelte';
import { X, Plus, Trash2, Tag, FolderTree, Sparkles, ArrowUpDown } from 'lucide-svelte';
import MarkdownStatusBadge from '$lib/components/MarkdownStatusBadge.svelte';
import { isMdStatusVisible } from '$lib/utils/mdStatus';
import UploadDropzone from '$lib/components/UploadDropzone.svelte';
@@ -233,15 +232,12 @@
goto(`/documents${qs ? '?' + qs : ''}`, { noScroll: true });
}
async function selectDoc(doc) {
if (selectedDoc?.id === doc.id) { selectedDoc = null; return; }
selectedDoc = doc; // 즉시 표시(리더 + 기본 인스펙터)
// 인스펙터 풀 메타 하이드레이션 — 검색 결과(SearchResult)는 메타가 빈약(태그/크기/하위/md상태/읽음 없음).
// 풀 문서를 조회해 채운다(기존 GET /documents/{id}, 백엔드 무변). 리스트 모드도 md상태 등 보강.
try {
const full = await api(`/documents/${doc.id}`);
if (selectedDoc?.id === doc.id) selectedDoc = { ...doc, ...full };
} catch { /* 실패 시 기본 정보 유지 */ }
// 문서 열기 = 개선된 상세 페이지(D3 절 구조 탐색기)로 이동.
// 사용자 결정: "개선된 페이지가 앞으로 표시되야지" — 인라인 미리보기 폐기.
// /documents = 브라우즈/검색/필터/일괄 목록, 문서 열기 = /documents/[id] D3 리더.
function selectDoc(doc) {
if (!doc) return;
goto(`/documents/${doc.id}`);
}
// bulk 선택
@@ -386,8 +382,8 @@
<div class="flex h-full min-h-0">
<!-- ═══ 좌: 리스트 컬럼 ═══ -->
<div class="{selectedDoc ? 'hidden lg:flex' : 'flex'} flex-col w-full lg:w-[340px] lg:shrink-0 lg:border-r border-default min-h-0">
<!-- ═══ 문서 목록 (풀폭 중앙) — 클릭 시 D3 상세로 이동 ═══ -->
<div class="flex flex-col w-full max-w-5xl mx-auto min-h-0">
<UploadDropzone onupload={loadDocuments} />
<!-- 검색바 -->
@@ -487,6 +483,19 @@
{/if}
</div>
<!-- AI 답변 (질문형 검색) — 목록 상단 고정, 아래로 목록 스크롤 -->
{#if showAskCard}
<div class="px-3 py-2 shrink-0 border-b border-default max-h-[55vh] overflow-y-auto">
<AskAnswerCard
data={askData}
loading={askLoading}
error={askError}
onCitationClick={(docId) => goto(`/documents/${docId}`)}
onDismiss={() => { askDismissed = true; }}
/>
</div>
{/if}
<!-- 선택 toolbar -->
{#if selectionCount > 0}
<div class="flex flex-wrap items-center gap-2 px-3 py-2 shrink-0 bg-accent/10 border-y border-accent/30">
@@ -587,47 +596,6 @@
</div>
</div>
<!-- ═══ 중앙: 리더 ═══ -->
<div class="{selectedDoc ? 'flex' : 'hidden lg:flex'} flex-1 min-w-0 flex-col min-h-0">
{#if selectedDoc}
<!-- 리더 상단 바: (모바일) 뒤로 / (lg) 인스펙터 토글 -->
<div class="flex items-center gap-2 px-3 py-1.5 shrink-0 border-b border-default bg-sidebar">
<button type="button" onclick={() => { selectedDoc = null; if (ui.isDrawerOpen('meta')) ui.closeDrawer(); }}
class="lg:hidden flex items-center gap-1 text-xs text-accent-hover font-medium" aria-label="목록으로">
<ChevronLeft size={15} /> 문서
</button>
<div class="flex-1"></div>
<button type="button" onclick={toggleInfoPanel} aria-pressed={isPanelActive} title="문서 정보"
class="p-1.5 rounded-lg border transition-colors {isPanelActive ? 'border-accent text-accent bg-accent/10' : 'border-default text-dim hover:text-accent hover:border-accent'}">
<Info size={16} />
</button>
</div>
<div class="flex-1 min-h-0">
<DocumentViewer doc={selectedDoc} />
</div>
{:else if showAskCard}
<div class="p-4 lg:p-6 overflow-y-auto">
<AskAnswerCard
data={askData}
loading={askLoading}
error={askError}
onCitationClick={(docId) => goto(`/documents/${docId}`)}
onDismiss={() => { askDismissed = true; }}
/>
</div>
{:else}
<div class="hidden lg:flex flex-1 items-center justify-center text-dim text-sm">
왼쪽에서 문서를 선택하세요
</div>
{/if}
</div>
<!-- ═══ 우: 인스펙터 (xl+ inline) ═══ -->
{#if selectedDoc && inspectorOpen}
<aside class="hidden xl:flex flex-col w-[300px] shrink-0 border-l border-default bg-sidebar overflow-y-auto" aria-label="문서 정보">
{@render inspector(selectedDoc)}
</aside>
{/if}
</div>
<!-- < xl 폴백: Drawer (정보 하단/측면 시트) -->
+377 -445
View File
@@ -1,19 +1,17 @@
<script>
// Phase E.2 — detail 페이지 inline 편집.
// 기존 read-only 메타 패널(L138201)을 editors/* 스택으로 교체.
// + E.3 관련 문서 stub, + 헤더 affordance row.
// 문서 상세 /documents/[id] — 확정 시안(d3-deepened) 스타일을 그대로 포팅, 데이터만 바인딩.
// 데스크탑: 상단 헤더 띠 + [좌 절 트리(색바+연결선)][중 절 집중 뷰][우 슬림 레일]. 절 없으면 fallback.
// 모바일: 헤더 + 나란한 토글 pill(절구조|인사이트) + 본문 절 카드 연속(+탭 이동). 편집/필기/네비 보존.
import { onMount } from 'svelte';
import { page } from '$app/stores';
import { goto } from '$app/navigation';
import { api, getAccessToken } from '$lib/api';
import { isMdSuccess } from '$lib/utils/mdStatus';
import { resolveAnchorMap } from '$lib/utils/resolveAnchorMap';
import { addToast } from '$lib/stores/toast';
import { marked } from 'marked';
import DOMPurify from 'dompurify';
import { ExternalLink, Download, Link2, FileText, PenLine, X, ChevronLeft, ChevronRight, Check } from 'lucide-svelte';
import { ChevronRight, FileText } from 'lucide-svelte';
import Button from '$lib/components/ui/Button.svelte';
import Card from '$lib/components/ui/Card.svelte';
import EmptyState from '$lib/components/ui/EmptyState.svelte';
import Skeleton from '$lib/components/ui/Skeleton.svelte';
import HandwriteCanvas from '$lib/components/HandwriteCanvas.svelte';
@@ -23,97 +21,57 @@
import EditUrlEditor from '$lib/components/editors/EditUrlEditor.svelte';
import TagsEditor from '$lib/components/editors/TagsEditor.svelte';
import AIClassificationEditor from '$lib/components/editors/AIClassificationEditor.svelte';
import FileInfoView from '$lib/components/editors/FileInfoView.svelte';
import ProcessingStatusView from '$lib/components/editors/ProcessingStatusView.svelte';
import LibraryPathEditor from '$lib/components/editors/LibraryPathEditor.svelte';
import DocumentDangerZone from '$lib/components/editors/DocumentDangerZone.svelte';
import AnalysisPanel from '$lib/components/AnalysisPanel.svelte';
import ReadCounter from '$lib/components/ReadCounter.svelte';
import SectionOutline from '$lib/components/SectionOutline.svelte';
import Tabs from '$lib/components/ui/Tabs.svelte';
import { cleanHeading, pathSegments, sectionTypeLabel, collapseWindows } from '$lib/utils/headingPath';
import { domainLabel } from '$lib/utils/domainSlug';
marked.use({ mangle: false, headerIds: false });
function renderMd(text) {
return DOMPurify.sanitize(marked(text), {
USE_PROFILES: { html: true },
FORBID_TAGS: ['style', 'script'],
FORBID_ATTR: ['onerror', 'onclick'],
ALLOW_UNKNOWN_PROTOCOLS: false,
return DOMPurify.sanitize(marked(text || ''), {
USE_PROFILES: { html: true }, FORBID_TAGS: ['style', 'script'], FORBID_ATTR: ['onerror', 'onclick'], ALLOW_UNKNOWN_PROTOCOLS: false,
});
}
let doc = $state(null);
let loading = $state(true);
let error = $state(null); // 'not_found' | 'network' | null
let rawMarkdown = $state(''); // fallback: extracted_text 없을 때 원본 .md
let error = $state(null);
let rawMarkdown = $state('');
let docId = $derived($page.params.id);
// 손글씨 노트 (자료별 1:1) — "필기" 토글 시 사이드 캔버스 띄움.
// 필기
let noteOpen = $state(false);
let noteStrokes = $state(null); // { version, strokes }
let noteStrokes = $state(null);
let noteLoaded = $state(false);
async function ensureNoteLoaded() {
if (noteLoaded) return;
try {
const r = await api(`/documents/${docId}/note`);
noteStrokes = r.strokes_json && r.strokes_json.strokes ? r.strokes_json : { version: 1, strokes: [] };
} catch {
noteStrokes = { version: 1, strokes: [] };
}
try { const r = await api(`/documents/${docId}/note`); noteStrokes = r.strokes_json && r.strokes_json.strokes ? r.strokes_json : { version: 1, strokes: [] }; }
catch { noteStrokes = { version: 1, strokes: [] }; }
noteLoaded = true;
}
async function saveNote(strokesJson) {
try {
await api(`/documents/${docId}/note`, {
method: 'PUT',
body: JSON.stringify({ strokes_json: strokesJson }),
});
} catch (err) {
console.warn('필기 저장 실패', err);
}
}
async function toggleNote() {
if (!noteOpen) await ensureNoteLoaded();
noteOpen = !noteOpen;
}
async function saveNote(s) { try { await api(`/documents/${docId}/note`, { method: 'PUT', body: JSON.stringify({ strokes_json: s }) }); } catch (e) { console.warn(e); } }
async function toggleNote() { if (!noteOpen) await ensureNoteLoaded(); noteOpen = !noteOpen; }
// 인접 자료 (같은 library_path 내 이전/다음) — 학습 흐름 네비게이션
// 인접 자료
let neighbors = $state({ prev: null, next: null });
async function loadNeighbors() {
try {
neighbors = await api(`/documents/${docId}/library-neighbors`);
} catch {
neighbors = { prev: null, next: null };
}
async function loadNeighbors() { try { neighbors = await api(`/documents/${docId}/library-neighbors`); } catch { neighbors = { prev: null, next: null }; } }
async function readAndGoNext() {
try { await api(`/documents/${docId}/read`, { method: 'POST' }); addToast('success', '1회독 완료'); }
catch (err) { addToast('error', err?.detail || '회독 기록 실패'); return; }
if (neighbors.next) goto(`/documents/${neighbors.next.id}`);
}
// 절(hier section) 목차 — 본문 로드와 독립, 실패(404 포함) 무해.
// reqId guard: 문서 전환 race 시 stale 결과가 새 문서에 붙지 않게.
// 절 목차
let sections = $state([]);
let hasSections = $derived(sections.length > 0);
// 과대 절은 builder 가 window 조각(같은 제목·is_leaf)으로 분해하고 부모를 heading 만 남긴 split-parent 로
// 강등한다(예: 5180 = 27개 논리 절 → 562 window). raw sections 를 그대로 그리면 동일 제목 수백 행으로
// 파편화되므로, collapseWindows 로 논리 절 1개(대표=split-parent, bodyText=window 본문 합본)로 합친다.
let outline = $derived(collapseWindows(sections));
async function loadSections() {
const reqId = docId;
try {
const r = await api(`/documents/${reqId}/sections`);
if (reqId === docId) sections = r?.sections ?? [];
} catch {
if (reqId === docId) sections = []; // Phase 1 미배포 시 404 → 목차 숨김(graceful)
}
}
// "1회독 완료 + 다음 자료로" 한 번에
async function readAndGoNext() {
try {
await api(`/documents/${docId}/read`, { method: 'POST' });
addToast('success', '1회독 완료');
} catch (err) {
addToast('error', err?.detail || '회독 기록 실패');
return;
}
if (neighbors.next) {
goto(`/documents/${neighbors.next.id}`);
}
try { const r = await api(`/documents/${reqId}/sections`); if (reqId === docId) sections = r?.sections ?? []; }
catch { if (reqId === docId) sections = []; }
}
onMount(async () => {
@@ -121,87 +79,26 @@
doc = await api(`/documents/${docId}`);
const vt = doc.source_channel === 'news' ? 'article' : getViewerType(doc.file_format);
if ((vt === 'markdown' || vt === 'hwp-markdown') && !doc.extracted_text) {
try {
const resp = await fetch(`/api/documents/${docId}/file?token=${getAccessToken()}`);
if (resp.ok) rawMarkdown = await resp.text();
} catch (e) {
rawMarkdown = '';
}
try { const resp = await fetch(`/api/documents/${docId}/file?token=${getAccessToken()}`); if (resp.ok) rawMarkdown = await resp.text(); } catch { rawMarkdown = ''; }
}
} catch (err) {
error = err?.status === 404 ? 'not_found' : 'network';
} finally {
loading = false;
}
// 자료실 자료면 인접 자료 미리 fetch (학습 흐름 네비)
} catch (err) { error = err?.status === 404 ? 'not_found' : 'network'; }
finally { loading = false; }
if (doc && doc.category === 'library') loadNeighbors();
if (doc) loadSections();
});
let viewerType = $derived(
doc ? (doc.source_channel === 'news' ? 'article' : getViewerType(doc.file_format)) : 'none'
);
let viewerType = $derived(doc ? (doc.source_channel === 'news' ? 'article' : getViewerType(doc.file_format)) : 'none');
let canShowMarkdown = $derived(!!(isMdSuccess(doc?.md_status) && doc?.md_content?.trim()));
// 절 본문은 청크 text(절별 원문)에서 오므로 md_content 성공/존재와 무관.
// hasSections 만으로 절뷰 사용 → partial / 대형 split(md_content 5만 자 절단) 문서도 절뷰 표시.
let useSectionView = $derived(hasSections);
// PDF 분기 전용: marker_worker 가 만든 canonical markdown 이 있으면 기본으로 그것을 보여줌.
// Phase 1B 산출물의 95% 가 PDF 라 1D pilot 평가가 실사용 화면 기반이 되도록 markdown-first.
// 사용자가 "PDF 원본" 토글하면 iframe. lastDocId 로 문서 전환만 감지해서 사용자 토글이
// reactive cycle 에 덮이지 않도록 보호.
let pdfViewMode = $state('markdown'); // 'markdown' | 'pdf'
let pdfViewMode = $state('markdown');
let lastDocId = $state(null);
let canShowMarkdown = $derived(
!!(isMdSuccess(doc?.md_status) && doc?.md_content?.trim())
);
$effect(() => {
if (!doc) return;
if (doc.id !== lastDocId) {
lastDocId = doc.id;
pdfViewMode = canShowMarkdown ? 'markdown' : 'pdf';
}
// 같은 문서 안에서 markdown 이 사라지면 (success → failed 재처리 등) PDF 로 보호.
if (!canShowMarkdown && pdfViewMode === 'markdown') {
pdfViewMode = 'pdf';
}
});
// ── 개요 점프 (경로 B: BE char_start primary + string-match 폴백) ──
// 이 사이트는 항상 md_content basis(canShowMarkdown && doc.md_content) → trustBE=true.
// BE char_start 가 있으면 채택, 비면(non-PASS/미백필) resolveAnchorMap 내부에서 buildAnchorMap 로 폴백.
let anchorMap = $derived(
hasSections && canShowMarkdown && doc?.md_content
? resolveAnchorMap(doc.md_content, sections, { trustBE: true }).anchors
: {}
);
let activeKey = $state(null);
function jumpToSection(chunkId) {
const el = document.getElementById(`sec-${chunkId}`);
if (el) el.scrollIntoView({ behavior: 'smooth', block: 'start' });
}
// scroll-spy: 화면 상단(120px)을 지난 마지막 .md-anchor = 현재 절. [id] 는 window 스크롤.
$effect(() => {
void anchorMap; // 문서/섹션 변화 시 재바인딩
if (typeof window === 'undefined') return;
let raf = 0;
const onScroll = () => {
if (raf) return;
raf = requestAnimationFrame(() => {
raf = 0;
let cur = null;
document.querySelectorAll('.md-anchor').forEach((a) => {
if (a.getBoundingClientRect().top <= 120) cur = a;
});
if (cur) {
const m = cur.id.match(/^sec-(\d+)$/);
if (m) activeKey = Number(m[1]);
}
});
};
window.addEventListener('scroll', onScroll, { passive: true });
onScroll();
return () => {
window.removeEventListener('scroll', onScroll);
if (raf) cancelAnimationFrame(raf);
};
if (doc.id !== lastDocId) { lastDocId = doc.id; pdfViewMode = canShowMarkdown ? 'markdown' : 'pdf'; }
if (!canShowMarkdown && pdfViewMode === 'markdown') pdfViewMode = 'pdf';
});
function getViewerType(format) {
@@ -213,338 +110,373 @@
return 'unsupported';
}
// E.2 affordance row 핸들러
// 절 집중/모바일 상태
let selectedSectionId = $state(null);
let mTree = $state(false);
let mIns = $state(false);
let manageOpen = $state(false);
$effect(() => { if (outline.length && !outline.some((it) => it.section.chunk_id === selectedSectionId)) selectedSectionId = outline[0].section.chunk_id; });
let selectedItem = $derived(outline.find((it) => it.section.chunk_id === selectedSectionId) ?? outline[0] ?? null);
let selectedSection = $derived(selectedItem?.section ?? null);
let selIdx = $derived(outline.findIndex((it) => it.section.chunk_id === selectedItem?.section?.chunk_id));
// 절 본문 = 청크 원문(it.bodyText, window 조각 합본) 직접 렌더. 과거 char_start 로 md_content 를
// 슬라이스했으나, 대형 split 문서는 md_content 가 앞 5만 자만 보존되고 char_start 도 NULL 이라 본문이
// 비었다. 청크 text 는 절 전체를 담으므로(절 보유 문서 344개, 본문 합 평균 68KB·max 1.6MB) 그대로 렌더.
function bodyHtml(it) { return it?.bodyText ? renderMd(it.bodyText) : ''; }
let selectedBodyHtml = $derived(bodyHtml(selectedItem));
// 모바일 연속 카드: 본문은 '본문 보기' 펼칠 때만 파싱(논리 절 수백 개 × marked 즉시 파싱 회피).
let mBodyOpen = $state({});
// 절 유형 색 (시안: 정의 청 / 절차 올리브 / 요건 황)
const TYPE_META = {
definition: { label: '정의', en: 'definition', color: '#2f7d8f' },
procedure: { label: '절차', en: 'procedure', color: '#7a8b3f' },
requirement: { label: '요건', en: 'requirement', color: '#b5840a' },
};
function typeMeta(t) { return TYPE_META[t] ?? { label: sectionTypeLabel(t) || '', en: t || '', color: '#9aa090' }; }
function isLowConf(c) { return c != null && c < 0.5; }
function isMidLow(c) { return c != null && c < 0.6; }
function confColor(c) { return c == null ? '#9aa090' : c < 0.6 ? '#b5840a' : '#1f9d6b'; }
function secTitle(s) { return cleanHeading(s.section_title) || pathSegments(s.heading_path).at(-1) || '(제목 없음)'; }
function secDepth(s) { return Math.max(0, (s.level ?? 1) - 1); }
function confPct(c) { return c == null ? 0 : Math.round(c * 100); }
// 도메인 색 (시안 도메인 팔레트)
const DOMAIN_COLOR = { Industrial_Safety: '#b5840a', Engineering: '#2f7d8f', Programming: '#3d7256', General: '#7a8b3f', Reference: '#8a6a3f', Philosophy: '#7a6a9b' };
function domainColor(d) { return DOMAIN_COLOR[(d || '').split('/')[0]] ?? '#697061'; }
function fmtColor(f) { return f === 'pdf' ? '#c0564a' : f === 'md' ? '#5a8f7a' : ['m4a', 'mp3', 'wav'].includes(f) ? '#8a6aa5' : f === 'html' ? '#c2911f' : '#697061'; }
let quality = $derived(doc?.md_extraction_quality?.metrics ?? doc?.md_extraction_quality ?? null);
function copyLink() {
const url = `${window.location.origin}/documents/${docId}`;
navigator.clipboard
.writeText(url)
.then(() => addToast('success', '링크 복사됨'))
.catch(() => addToast('error', '복사 실패'));
}
function downloadOriginal() {
window.open(`/api/documents/${docId}/file?token=${getAccessToken()}&download=true`);
}
function downloadPdf() {
window.open(`/api/documents/${docId}/preview?token=${getAccessToken()}&download=true`);
}
function handleDocDelete() {
addToast('success', '문서가 삭제되어 목록으로 이동합니다.');
goto('/documents');
navigator.clipboard.writeText(`${window.location.origin}/documents/${docId}`).then(() => addToast('success', '링크 복사됨')).catch(() => addToast('error', '복사 실패'));
}
function downloadOriginal() { window.open(`/api/documents/${docId}/file?token=${getAccessToken()}&download=true`); }
function handleDocDelete() { addToast('success', '문서가 삭제되어 목록으로 이동합니다.'); goto('/documents'); }
</script>
<div class="p-4 lg:p-6">
<!-- ════ 좌 트리 (시안: 색바 + 연결선 + 활성 + 저신뢰 경고) ════ -->
{#snippet treeNav(jumpMode)}
<div class="d3tree" style="font-size:14px;">
<div style="display:flex;align-items:center;justify-content:space-between;margin-bottom:9px;">
<div style="font-size:12px;font-weight:700;color:#697061;letter-spacing:.4px;">절 구조</div>
<span style="font-size:10.5px;color:#9aa090;font-variant-numeric:tabular-nums;">{outline.length}</span>
</div>
<div style="display:flex;flex-wrap:wrap;gap:6px 8px;margin-bottom:11px;padding-bottom:10px;border-bottom:1px solid #dde3d6;">
<span style="display:inline-flex;align-items:center;gap:4px;font-size:10px;color:#697061;"><span style="width:8px;height:8px;border-radius:2px;background:#2f7d8f;"></span>정의</span>
<span style="display:inline-flex;align-items:center;gap:4px;font-size:10px;color:#697061;"><span style="width:8px;height:8px;border-radius:2px;background:#7a8b3f;"></span>절차</span>
<span style="display:inline-flex;align-items:center;gap:4px;font-size:10px;color:#697061;"><span style="width:8px;height:8px;border-radius:2px;background:#b5840a;"></span>요건</span>
</div>
{#each outline as it (it.section.chunk_id)}
{@const s = it.section}
{@const tm = typeMeta(it.sectionType)}
{@const active = !jumpMode && s.chunk_id === selectedSection?.chunk_id}
{@const child = secDepth(s) > 0}
{@const low = isMidLow(it.confidence)}
<svelte:element this={jumpMode ? 'a' : 'div'} href={jumpMode ? `#m-sec-${s.chunk_id}` : undefined} role="button" tabindex="0"
onclick={() => !jumpMode && (selectedSectionId = s.chunk_id)}
onkeydown={(e) => { if (!jumpMode && (e.key === 'Enter' || e.key === ' ')) { e.preventDefault(); selectedSectionId = s.chunk_id; } }}
class="d3node {child ? 'd3child' : ''} {active ? 'd3active' : ''}"
style="display:block;border:1px solid {active ? '#4f8a6b' : low ? '#e7d49a' : 'transparent'};border-radius:9px;padding:{child ? '6px 8px' : '7px 8px'};margin-bottom:2px;{low ? 'background:#fbf6e6;' : ''}text-decoration:none;cursor:pointer;">
<div style="display:flex;align-items:center;gap:7px;">
<span style="width:3px;height:{child ? '13px' : '16px'};border-radius:2px;background:{tm.color};flex-shrink:0;"></span>
<span class="d3title" style="font-size:{child ? '11.5px' : '12.5px'};flex:1;min-width:0;{child ? 'color:#697061;' : ''}{active ? 'color:#3d7256;font-weight:600;' : ''}overflow:hidden;text-overflow:ellipsis;white-space:nowrap;">{secTitle(s)}</span>
{#if low}
<span class="d3warn" title="저신뢰 절" style="display:inline-flex;width:14px;height:14px;border-radius:50%;background:#b5840a;color:#fff;align-items:center;justify-content:center;font-size:9px;font-weight:700;flex-shrink:0;">!</span>
{:else if !child}
<span title="신뢰도 {it.confidence != null ? it.confidence.toFixed(2) : '—'}" style="width:7px;height:7px;border-radius:50%;background:{confColor(it.confidence)};flex-shrink:0;"></span>
{/if}
</div>
</svelte:element>
{/each}
{#if quality}
<div style="margin-top:12px;padding-top:10px;border-top:1px solid #dde3d6;">
<div style="font-size:10.5px;font-weight:700;color:#697061;margin-bottom:7px;letter-spacing:.3px;">추출 품질</div>
<div style="display:grid;grid-template-columns:1fr 1fr;gap:5px;font-size:10.5px;color:#697061;font-variant-numeric:tabular-nums;">
{#if quality.headings != null}<span>headings <b style="color:#23291f;">{quality.headings}</b></span>{/if}
{#if quality.tables != null}<span>tables <b style="color:#23291f;">{quality.tables}</b></span>{/if}
{#if quality.images != null}<span>images <b style="color:#23291f;">{quality.images}</b></span>{/if}
</div>
</div>
{/if}
</div>
{/snippet}
<!-- ════ 절 집중 뷰 (데스크탑 중앙) ════ -->
{#snippet focusView()}
{#if selectedSection}
{@const tm = typeMeta(selectedItem?.sectionType)}
{@const conf = selectedItem?.confidence ?? null}
{@const summaries = selectedItem?.summaries ?? []}
<div style="display:flex;align-items:center;gap:6px;font-size:12px;color:#9aa090;margin-bottom:12px;flex-wrap:wrap;">
<span class="truncate" style="max-width:200px;">{doc.title}</span>
{#each pathSegments(selectedSection.heading_path) as seg}<span style="color:#c8d6c0;">/</span><span style="color:#697061;font-weight:600;">{seg}</span>{/each}
</div>
<div style="display:flex;align-items:center;gap:9px;flex-wrap:wrap;margin-bottom:13px;">
<h2 style="margin:0;font-size:22px;font-weight:700;color:#23291f;line-height:1.3;flex:1;min-width:180px;">{secTitle(selectedSection)}</h2>
{#if tm.label}<span style="display:inline-flex;align-items:center;gap:5px;padding:4px 11px;border-radius:999px;background:{tm.color}1a;border:1px solid {tm.color}55;font-size:12px;color:{tm.color};font-weight:600;"><span style="width:8px;height:8px;border-radius:2px;background:{tm.color};"></span>{tm.label} {tm.en}</span>{/if}
</div>
{#if conf != null}
<div style="display:flex;align-items:center;gap:9px;margin-bottom:18px;">
<span style="font-size:11px;color:#697061;font-weight:600;flex-shrink:0;">신뢰도</span>
<div style="flex:1;max-width:300px;height:7px;border-radius:999px;background:#e3ebdf;overflow:hidden;"><div style="width:{confPct(conf)}%;height:100%;background:{confColor(conf)};border-radius:999px;"></div></div>
<span style="font-size:13px;font-weight:700;color:{confColor(conf)};font-variant-numeric:tabular-nums;flex-shrink:0;">{conf.toFixed(2)}</span>
</div>
{/if}
{#if isLowConf(conf)}
<div style="display:flex;align-items:flex-start;gap:8px;background:#faf3e2;border:1px solid #ecdca3;border-radius:10px;padding:10px 12px;margin-bottom:16px;font-size:12.5px;color:#8a6306;"><span style="flex-shrink:0;width:16px;height:16px;border-radius:50%;border:1.5px solid #b5840a;color:#b5840a;font-size:10px;font-weight:800;display:inline-flex;align-items:center;justify-content:center;margin-top:1px;">!</span><span>저신뢰 절 — 표·수식 추출이 불완전할 수 있습니다. 정확한 내용은 원본을 확인하세요.</span></div>
{/if}
{#if summaries.length}
<div style="background:#ecf0e8;border-left:3px solid #4f8a6b;border-radius:0 10px 10px 0;padding:14px 16px;margin-bottom:20px;">
<div style="font-size:10.5px;font-weight:700;color:#3d7256;letter-spacing:.6px;margin-bottom:6px;">절 요약{#if summaries.length > 1} · {summaries.length}개 부분{/if}</div>
{#if summaries.length === 1}
<div style="font-size:15.5px;line-height:1.6;color:#23291f;white-space:pre-line;">{summaries[0]}</div>
{:else}
<ul style="margin:0;padding:0;list-style:none;display:flex;flex-direction:column;gap:8px;">
{#each summaries as sm, i}<li style="font-size:13.5px;line-height:1.55;color:#23291f;display:flex;gap:8px;"><span style="flex-shrink:0;color:#7a8b3f;font-weight:700;font-variant-numeric:tabular-nums;">{i + 1}</span><span style="white-space:pre-line;">{sm}</span></li>{/each}
</ul>
{/if}
</div>
{/if}
{#if selectedBodyHtml}
<div class="prose prose-base max-w-none text-text">{@html selectedBodyHtml}</div>
{:else}
<p style="color:#9aa090;font-size:14px;font-style:italic;">이 절의 본문은 추출되지 않았습니다. 헤더의 '원본'에서 확인하세요.</p>
{/if}
<div style="display:flex;justify-content:space-between;gap:10px;margin-top:20px;padding-top:14px;border-top:1px solid #dde3d6;">
{#if selIdx > 0}
{@const pv = outline[selIdx - 1].section}
<button type="button" onclick={() => (selectedSectionId = pv.chunk_id)} style="font-size:12px;color:#697061;border:1px solid #dde3d6;border-radius:9px;padding:8px 12px;background:#fff;cursor:pointer;">{secTitle(pv)}</button>
{:else}<span></span>{/if}
{#if selIdx >= 0 && selIdx < outline.length - 1}
{@const nxIt = outline[selIdx + 1]}
{@const nx = nxIt.section}
<button type="button" onclick={() => (selectedSectionId = nx.chunk_id)} style="font-size:12px;color:{isMidLow(nxIt.confidence) ? '#8a6306' : '#697061'};border:1px solid {isMidLow(nxIt.confidence) ? '#e7d49a' : '#dde3d6'};border-radius:9px;padding:8px 12px;background:#fff;cursor:pointer;display:inline-flex;align-items:center;gap:6px;">{#if isMidLow(nxIt.confidence)}<span style="display:inline-flex;width:13px;height:13px;border-radius:50%;background:#b5840a;color:#fff;align-items:center;justify-content:center;font-size:8px;font-weight:700;">!</span>{/if}{secTitle(nx)}</button>
{:else}<span></span>{/if}
</div>
{/if}
{/snippet}
<!-- ════ 우 슬림 레일 (시안 카드 스타일) ════ -->
{#snippet rail()}
<div style="display:flex;flex-direction:column;gap:11px;font-size:14px;">
{#if doc.ai_tldr || doc.ai_summary}
<div style="background:#f4f7f1;border:1px solid #dde3d6;border-radius:14px;padding:13px;">
<div style="font-size:10.5px;font-weight:700;color:#697061;letter-spacing:.4px;margin-bottom:7px;">TL;DR</div>
<div style="font-size:12px;line-height:1.5;color:#23291f;">{doc.ai_tldr || doc.ai_summary}</div>
</div>
{/if}
{#if doc.ai_bullets && doc.ai_bullets.length}
<div style="background:#f4f7f1;border:1px solid #dde3d6;border-radius:14px;padding:13px;">
<div style="font-size:10.5px;font-weight:700;color:#697061;letter-spacing:.4px;margin-bottom:8px;">핵심점</div>
<ul style="margin:0;padding:0;list-style:none;display:flex;flex-direction:column;gap:7px;">
{#each doc.ai_bullets as b}<li style="font-size:12px;line-height:1.4;display:flex;gap:6px;"><span style="color:#b5840a;font-weight:700;flex-shrink:0;">·</span><span style="flex:1;min-width:0;color:#23291f;">{b}</span></li>{/each}
</ul>
</div>
{/if}
{#if doc.ai_detail_summary}
<div style="background:#f4f7f1;border:1px solid #c8d6c0;border-radius:14px;padding:13px;">
<div style="display:flex;align-items:center;gap:6px;margin-bottom:7px;">
<span style="font-size:10.5px;font-weight:700;color:#3d7256;letter-spacing:.4px;">심층</span>
{#if doc.ai_analysis_tier === 'deep'}<span style="font-size:9px;color:#fff;background:#4f8a6b;border-radius:999px;padding:1px 7px;font-weight:600;">DEEP</span>{/if}
</div>
<div style="font-size:11.5px;line-height:1.5;color:#23291f;white-space:pre-line;">{doc.ai_detail_summary}</div>
</div>
{/if}
{#if doc.ai_inconsistencies && doc.ai_inconsistencies.length}
<div style="background:#fbf6e6;border:1px solid #e7d49a;border-radius:14px;padding:13px;">
<div style="font-size:10.5px;font-weight:700;color:#8a6306;letter-spacing:.4px;margin-bottom:7px;">불일치 {doc.ai_inconsistencies.length}</div>
<ul style="margin:0;padding:0;list-style:none;display:flex;flex-direction:column;gap:5px;">{#each doc.ai_inconsistencies as inc}<li style="font-size:11.5px;line-height:1.45;color:#23291f;">· {typeof inc === 'string' ? inc : inc.desc || inc.kind}</li>{/each}</ul>
</div>
{/if}
{#if doc.ai_domain}
<div style="background:#f4f7f1;border:1px solid #dde3d6;border-radius:14px;padding:13px;">
<div style="font-size:10.5px;font-weight:700;color:#697061;letter-spacing:.4px;margin-bottom:8px;">분류</div>
<div style="display:flex;flex-direction:column;gap:6px;font-size:11.5px;">
<div style="display:flex;justify-content:space-between;gap:8px;"><span style="color:#697061;">도메인</span><span style="display:inline-flex;align-items:center;gap:5px;color:#23291f;font-weight:600;text-align:right;"><span style="width:7px;height:7px;border-radius:50%;background:{domainColor(doc.ai_domain)};"></span>{domainLabel(doc.ai_domain)}</span></div>
{#if doc.ai_sub_group}<div style="display:flex;justify-content:space-between;gap:8px;"><span style="color:#697061;">하위</span><span style="color:#23291f;font-weight:600;">{doc.ai_sub_group}</span></div>{/if}
{#if doc.ai_analysis_tier}<div style="display:flex;justify-content:space-between;gap:8px;"><span style="color:#697061;">tier</span><span style="color:#3d7256;font-weight:600;">{doc.ai_analysis_tier}</span></div>{/if}
{#if doc.ai_confidence != null}<div style="display:flex;justify-content:space-between;gap:8px;"><span style="color:#697061;">신뢰도</span><span style="color:#1f9d6b;font-weight:700;font-variant-numeric:tabular-nums;">{doc.ai_confidence.toFixed(2)}</span></div>{/if}
</div>
</div>
{/if}
{#if doc.ai_tags && doc.ai_tags.length}
<div style="background:#f4f7f1;border:1px solid #dde3d6;border-radius:14px;padding:13px;">
<div style="font-size:10.5px;font-weight:700;color:#697061;letter-spacing:.4px;margin-bottom:8px;">태그</div>
<div style="display:flex;flex-wrap:wrap;gap:5px;">{#each doc.ai_tags as t}<span style="font-size:11px;padding:3px 8px;border-radius:999px;background:#fff;border:1px solid #dde3d6;color:#697061;">{t}</span>{/each}</div>
</div>
{/if}
<div style="background:#f4f7f1;border:1px solid #dde3d6;border-radius:14px;padding:13px;">
<div style="font-size:10.5px;font-weight:700;color:#697061;letter-spacing:.4px;margin-bottom:6px;">관련 문서</div>
<div style="font-size:11px;color:#9aa090;line-height:1.5;">벡터 유사도 기반 — 준비 중</div>
</div>
</div>
{/snippet}
<!-- ════ 절 카드 (모바일 연속 본문) ════ -->
{#snippet sectionCard(it)}
{@const s = it.section}
{@const tm = typeMeta(it.sectionType)}
<div id="m-sec-{s.chunk_id}" style="scroll-margin-top:12px;background:#f4f7f1;border:1px solid {isLowConf(it.confidence) ? '#e7d49a' : '#dde3d6'};border-radius:14px;padding:14px 15px;">
<div style="display:flex;align-items:center;gap:7px;margin-bottom:7px;">
<h2 style="margin:0;font-size:16px;font-weight:700;color:#23291f;flex:1;min-width:0;line-height:1.3;">{secTitle(s)}</h2>
{#if tm.label}<span style="flex-shrink:0;font-size:10.5px;font-weight:650;padding:2px 8px;border-radius:999px;background:{tm.color}1a;color:{tm.color};white-space:nowrap;">{tm.label}</span>{/if}
</div>
{#if isLowConf(it.confidence)}
<div style="display:flex;align-items:flex-start;gap:7px;background:#faf3e2;border:1px solid #ecdca3;border-radius:9px;padding:8px 10px;margin-bottom:10px;font-size:12px;color:#8a6306;"><span style="flex-shrink:0;width:15px;height:15px;border-radius:50%;border:1.5px solid #b5840a;color:#b5840a;font-size:10px;font-weight:800;display:inline-flex;align-items:center;justify-content:center;margin-top:1px;">!</span><span>저신뢰 — 표·수식 추출 불완전, 원본 확인 권장</span></div>
{/if}
{#if it.summaries.length}
<div style="border-left:3px solid #4f8a6b;background:#ecf0e8;border-radius:0 8px 8px 0;padding:9px 12px;margin-bottom:12px;">
<div style="font-size:9.5px;font-weight:700;color:#3d7256;letter-spacing:.5px;margin-bottom:3px;">절 요약{#if it.summaries.length > 1} · {it.summaries.length}개 부분{/if}</div>
{#if it.summaries.length === 1}
<div style="font-size:13.5px;line-height:1.55;color:#23291f;white-space:pre-line;">{it.summaries[0]}</div>
{:else}
<ul style="margin:0;padding:0;list-style:none;display:flex;flex-direction:column;gap:6px;">{#each it.summaries as sm, i}<li style="font-size:12.5px;line-height:1.5;color:#23291f;display:flex;gap:6px;"><span style="flex-shrink:0;color:#7a8b3f;font-weight:700;font-variant-numeric:tabular-nums;">{i + 1}</span><span style="white-space:pre-line;">{sm}</span></li>{/each}</ul>
{/if}
</div>
{/if}
{#if it.bodyText}
<details class="m-secbody" ontoggle={(e) => { if (e.currentTarget.open) mBodyOpen[s.chunk_id] = true; }}>
<summary style="cursor:pointer;list-style:none;font-size:12px;color:#697061;padding:5px 0;user-select:none;display:flex;align-items:center;gap:5px;">본문 보기 <span class="m-chev" style="transition:transform .16s;color:#9aa090;"></span></summary>
{#if mBodyOpen[s.chunk_id]}<div class="prose prose-sm max-w-none text-text" style="margin-top:6px;">{@html bodyHtml(it)}</div>{/if}
</details>
{/if}
</div>
{/snippet}
<div style="background:#e7ebe4;min-height:100%;" class="p-4 lg:p-6">
<div style="max-width:1360px;margin:0 auto;">
<!-- breadcrumb -->
<div class="flex items-center gap-2 text-sm mb-4 text-dim">
<a href="/documents" class="hover:text-text">문서</a>
<span class="text-faint">/</span>
<div class="flex items-center gap-2 text-sm mb-3 text-dim">
<a href="/documents" class="hover:text-text">문서</a><span class="text-faint">/</span>
<span class="truncate max-w-md text-text">{doc?.title || '로딩...'}</span>
</div>
{#if loading}
<div class="max-w-6xl mx-auto">
<Skeleton h="h-96" rounded="card" />
</div>
<Skeleton h="h-96" rounded="card" />
{:else if error === 'not_found'}
<EmptyState
icon={FileText}
title="문서를 찾을 수 없습니다"
description="삭제되었거나 접근 권한이 없을 수 있습니다."
>
<Button variant="ghost" size="sm" href="/documents">목록으로 돌아가기</Button>
</EmptyState>
<EmptyState icon={FileText} title="문서를 찾을 없습니다" description="삭제되었거나 접근 권한이 없을 수 있습니다."><Button variant="ghost" size="sm" href="/documents">목록으로</Button></EmptyState>
{:else if error === 'network'}
<EmptyState
icon={FileText}
title="문서를 불러올 수 없습니다"
description="네트워크 오류가 발생했습니다."
>
<Button variant="secondary" size="sm" onclick={() => location.reload()}>다시 시도</Button>
</EmptyState>
<EmptyState icon={FileText} title="문서를 불러올 없습니다" description="네트워크 오류"><Button variant="secondary" size="sm" onclick={() => location.reload()}>다시 시도</Button></EmptyState>
{:else if doc}
<div class="mx-auto grid grid-cols-1 gap-6 {hasSections ? 'max-w-7xl xl:grid-cols-[18rem_minmax(0,1fr)_20rem]' : 'max-w-6xl lg:grid-cols-3'}">
{#if hasSections}
<!-- 좌측 절 목차 — xl+ sticky rail (그 아래 viewport 는 본문 상단 collapsible) -->
<aside class="hidden xl:block xl:sticky xl:top-6 xl:self-start xl:max-h-[calc(100vh-3rem)] xl:overflow-y-auto">
<Card>
<SectionOutline {sections} onJump={jumpToSection} {activeKey} />
</Card>
</aside>
{/if}
<!-- 본문 (좌측 목차 없을 때 lg 2/3) -->
<div class="{hasSections ? '' : 'lg:col-span-2'} space-y-4">
{#if hasSections}
<!-- xl 미만: 절 목차 접이식 -->
<details class="xl:hidden">
<summary class="cursor-pointer text-sm text-dim px-1 py-2 select-none">절 목차 ({sections.length})</summary>
<Card class="mt-2"><SectionOutline {sections} onJump={jumpToSection} {activeKey} /></Card>
</details>
{/if}
<!-- Affordance row -->
<div class="flex flex-wrap items-center gap-2">
{#if doc.edit_url}
<Button
variant="secondary"
size="sm"
icon={ExternalLink}
href={doc.edit_url}
target="_blank"
>
Synology 편집
</Button>
{/if}
<Button variant="secondary" size="sm" icon={Download} onclick={downloadOriginal}>
원본 다운로드
</Button>
{#if doc.preview_status === 'ready'}
<Button variant="secondary" size="sm" icon={FileText} onclick={downloadPdf}>
PDF 다운로드
</Button>
{/if}
<Button variant="secondary" size="sm" icon={Link2} onclick={copyLink}>
링크 복사
</Button>
{#if doc.category === 'library'}
<Button
variant={noteOpen ? 'primary' : 'secondary'}
size="sm"
icon={noteOpen ? X : PenLine}
onclick={toggleNote}
>
{noteOpen ? '필기 닫기' : '필기'}
</Button>
{/if}
<!-- ════ 상단 띠: 문서 헤더 (시안) ════ -->
<div style="background:#f4f7f1;border:1px solid #dde3d6;border-radius:14px;padding:16px 18px;margin-bottom:14px;">
<div style="display:flex;align-items:flex-start;gap:13px;flex-wrap:wrap;">
<div style="width:40px;height:40px;border-radius:10px;background:{fmtColor(doc.file_format)};color:#fff;display:flex;align-items:center;justify-content:center;font-weight:700;font-size:10.5px;letter-spacing:.5px;flex-shrink:0;text-transform:uppercase;">{doc.file_format}</div>
<div style="flex:1;min-width:0;">
<div style="font-size:17px;font-weight:700;line-height:1.35;color:#23291f;">{doc.title}</div>
<div style="display:flex;flex-wrap:wrap;gap:6px;margin-top:8px;align-items:center;">
{#if doc.ai_domain}<span style="display:inline-flex;align-items:center;gap:5px;padding:3px 9px;border-radius:999px;background:#fff;border:1px solid #dde3d6;font-size:11.5px;color:#23291f;"><span style="width:7px;height:7px;border-radius:50%;background:{domainColor(doc.ai_domain)};"></span>{domainLabel(doc.ai_domain)}</span>{/if}
{#if doc.ai_sub_group}<span style="padding:3px 9px;border-radius:999px;background:#fff;border:1px solid #dde3d6;font-size:11.5px;color:#697061;">{doc.ai_sub_group}</span>{/if}
{#if doc.ai_analysis_tier === 'deep'}<span style="padding:3px 9px;border-radius:999px;background:#4f8a6b;color:#fff;font-size:11.5px;font-weight:600;letter-spacing:.3px;">tier DEEP</span>{/if}
{#if doc.ai_confidence != null}<span style="padding:3px 9px;border-radius:999px;background:#e3ebdf;border:1px solid #c8d6c0;font-size:11.5px;color:#3d7256;font-variant-numeric:tabular-nums;">신뢰도 {doc.ai_confidence.toFixed(2)}</span>{/if}
{#if canShowMarkdown}<span style="padding:3px 9px;border-radius:999px;background:#eafaf2;border:1px solid #b8e3cc;font-size:11.5px;color:#1f9d6b;">PDF→MD success</span>{/if}
</div>
</div>
<div style="display:flex;gap:6px;flex-shrink:0;flex-wrap:wrap;">
{#if doc.edit_url}<button type="button" onclick={() => window.open(doc.edit_url, '_blank')} style="font-size:11.5px;color:#697061;border:1px solid #dde3d6;border-radius:8px;padding:5px 9px;background:#fff;cursor:pointer;">Synology</button>{/if}
<button type="button" onclick={downloadOriginal} style="font-size:11.5px;color:#697061;border:1px solid #dde3d6;border-radius:8px;padding:5px 9px;background:#fff;cursor:pointer;">원본</button>
<button type="button" onclick={copyLink} style="font-size:11.5px;color:#697061;border:1px solid #dde3d6;border-radius:8px;padding:5px 9px;background:#fff;cursor:pointer;">링크</button>
{#if doc.category === 'library'}<button type="button" onclick={toggleNote} style="font-size:11.5px;color:{noteOpen ? '#fff' : '#697061'};border:1px solid {noteOpen ? '#4f8a6b' : '#dde3d6'};border-radius:8px;padding:5px 9px;background:{noteOpen ? '#4f8a6b' : '#fff'};cursor:pointer;">{noteOpen ? '필기 닫기' : '필기'}</button>{/if}
<button type="button" onclick={() => (manageOpen = !manageOpen)} style="font-size:11.5px;color:#697061;border:1px solid #dde3d6;border-radius:8px;padding:5px 9px;background:#fff;cursor:pointer;">관리</button>
</div>
</div>
</div>
<!-- 뷰어 — 모바일 가독성: 본문 폰트 키우고 line-height 늘림 -->
<Card class="min-h-[500px]">
{#if useSectionView}
<!-- 데스크탑(xl+): 3영역 -->
<div class="hidden xl:grid" style="grid-template-columns:252px minmax(0,1fr) 336px;gap:13px;align-items:start;">
<div style="background:#f4f7f1;border:1px solid #dde3d6;border-radius:14px;padding:13px 11px;position:sticky;top:14px;max-height:calc(100vh-2rem);overflow-y:auto;">{@render treeNav(false)}</div>
<div style="min-width:0;"><div style="background:#f4f7f1;border:1px solid #dde3d6;border-radius:14px;padding:20px 22px;">{@render focusView()}</div></div>
<div style="position:sticky;top:14px;">{@render rail()}</div>
</div>
<!-- 모바일(<xl): 나란한 토글 pill + 패널 + 본문 연속 -->
<div class="xl:hidden">
<div style="display:flex;gap:8px;margin-bottom:10px;position:sticky;top:0;z-index:5;background:#e7ebe4;padding:6px 0;">
<button type="button" onclick={() => (mTree = !mTree)} style="flex:1;display:flex;align-items:center;justify-content:space-between;gap:6px;border-radius:10px;padding:9px 12px;font-size:12.5px;font-weight:600;cursor:pointer;background:{mTree ? '#e3ebdf' : '#f4f7f1'};border:1px solid {mTree ? '#4f8a6b' : '#dde3d6'};color:{mTree ? '#23291f' : '#697061'};">절 구조 <span style="font-size:10px;color:#9aa090;font-weight:500;">{outline.length}</span><span style="transition:transform .16s;transform:rotate({mTree ? 90 : 0}deg);color:#9aa090;font-weight:700;"></span></button>
<button type="button" onclick={() => (mIns = !mIns)} style="flex:1;display:flex;align-items:center;justify-content:space-between;gap:6px;border-radius:10px;padding:9px 12px;font-size:12.5px;font-weight:600;cursor:pointer;background:{mIns ? '#e3ebdf' : '#f4f7f1'};border:1px solid {mIns ? '#4f8a6b' : '#dde3d6'};color:{mIns ? '#23291f' : '#697061'};">인사이트<span style="transition:transform .16s;transform:rotate({mIns ? 90 : 0}deg);color:#9aa090;font-weight:700;"></span></button>
</div>
{#if mTree}<div style="background:#f4f7f1;border:1px solid #dde3d6;border-radius:12px;padding:6px;margin-bottom:10px;">{@render treeNav(true)}</div>{/if}
{#if mIns}<div style="background:#f4f7f1;border:1px solid #dde3d6;border-radius:12px;padding:13px 14px;margin-bottom:10px;">{@render rail()}</div>{/if}
<div style="display:flex;flex-direction:column;gap:10px;">{#each outline as it (it.section.chunk_id)}{@render sectionCard(it)}{/each}</div>
</div>
{:else}
<!-- 절 없음 fallback: 절이 없어도 인사이트는 항상 보이게 (모바일=인사이트 상단 / 데스크탑=우측 레일) -->
{#snippet fbViewer()}
<div style="min-width:0;background:#f4f7f1;border:1px solid #dde3d6;border-radius:14px;padding:18px 20px;min-height:360px;">
{#if !hasSections && canShowMarkdown}<p style="font-size:11px;color:#9aa090;margin-bottom:12px;">이 문서는 절 분석이 없어 전체 본문으로 표시합니다. 위/옆 인사이트는 그대로 제공됩니다.</p>{/if}
{#if viewerType === 'markdown' || viewerType === 'hwp-markdown'}
<MarkdownDoc
documentId={doc.id}
mdContent={doc.md_content}
mdFrontmatter={doc.md_frontmatter}
mdStatus={doc.md_status}
mdExtractionError={doc.md_extraction_error}
mdExtractionQuality={doc.md_extraction_quality}
anchorMap={anchorMap}
extractedText={doc.extracted_text || rawMarkdown}
class="prose prose-invert prose-base lg:prose-sm max-w-none"
/>
<MarkdownDoc documentId={doc.id} mdContent={doc.md_content} mdFrontmatter={doc.md_frontmatter} mdStatus={doc.md_status} mdExtractionError={doc.md_extraction_error} mdExtractionQuality={doc.md_extraction_quality} extractedText={doc.extracted_text || rawMarkdown} class="prose prose-base max-w-none" />
{:else if viewerType === 'pdf'}
<div class="mb-2 flex items-center gap-2">
<MarkdownStatusBadge
mdStatus={doc.md_status}
mdExtractionError={doc.md_extraction_error}
mdExtractionQuality={doc.md_extraction_quality}
/>
{#if canShowMarkdown}
<Button
size="sm"
variant={pdfViewMode === 'markdown' ? 'primary' : 'secondary'}
onclick={() => (pdfViewMode = 'markdown')}
>
Markdown
</Button>
<Button
size="sm"
variant={pdfViewMode === 'pdf' ? 'primary' : 'secondary'}
onclick={() => (pdfViewMode = 'pdf')}
>
PDF 원본
</Button>
{/if}
<MarkdownStatusBadge mdStatus={doc.md_status} mdExtractionError={doc.md_extraction_error} mdExtractionQuality={doc.md_extraction_quality} />
{#if canShowMarkdown}<Button size="sm" variant={pdfViewMode === 'markdown' ? 'primary' : 'secondary'} onclick={() => (pdfViewMode = 'markdown')}>Markdown</Button><Button size="sm" variant={pdfViewMode === 'pdf' ? 'primary' : 'secondary'} onclick={() => (pdfViewMode = 'pdf')}>PDF 원본</Button>{/if}
</div>
{#if pdfViewMode === 'markdown' && canShowMarkdown}
<MarkdownDoc
documentId={doc.id}
mdContent={doc.md_content}
mdFrontmatter={doc.md_frontmatter}
mdStatus={doc.md_status}
mdExtractionError={doc.md_extraction_error}
mdExtractionQuality={doc.md_extraction_quality}
extractedText={doc.extracted_text}
class="prose prose-invert prose-base lg:prose-sm max-w-none"
/>
{:else}
<iframe
src="/api/documents/{doc.id}/file?token={getAccessToken()}"
class="w-full h-[80vh] rounded"
title={doc.title}
></iframe>
{/if}
<MarkdownDoc documentId={doc.id} mdContent={doc.md_content} mdFrontmatter={doc.md_frontmatter} mdStatus={doc.md_status} mdExtractionError={doc.md_extraction_error} mdExtractionQuality={doc.md_extraction_quality} extractedText={doc.extracted_text} class="prose prose-base max-w-none" />
{:else}<iframe src="/api/documents/{doc.id}/file?token={getAccessToken()}" class="w-full h-[80vh] rounded" title={doc.title}></iframe>{/if}
{:else if viewerType === 'image'}
<img
src="/api/documents/{doc.id}/file?token={getAccessToken()}"
alt={doc.title}
class="max-w-full rounded"
/>
<img src="/api/documents/{doc.id}/file?token={getAccessToken()}" alt={doc.title} class="max-w-full rounded" />
{:else if viewerType === 'synology'}
<EmptyState
icon={ExternalLink}
title="Synology Office 문서"
description="외부 편집기에서 열어야 합니다."
>
<Button
variant="primary"
size="sm"
href={doc.edit_url || 'https://link.hyungi.net'}
target="_blank"
>
새 창에서 열기
</Button>
</EmptyState>
<EmptyState icon={FileText} title="Synology Office 문서" description="외부 편집기에서 열어야 합니다."><Button variant="primary" size="sm" href={doc.edit_url || 'https://link.hyungi.net'} target="_blank">새 창에서 열기</Button></EmptyState>
{:else if viewerType === 'article'}
<div>
<h1 class="text-xl font-bold text-text mb-3">{doc.title}</h1>
<div class="flex items-center gap-2 mb-4 text-xs text-dim">
<span>출처: {doc.source_channel}</span>
<span class="text-faint">·</span>
<span>
{new Date(doc.created_at).toLocaleDateString('ko-KR', {
year: 'numeric',
month: 'short',
day: 'numeric',
})}
</span>
</div>
{#if doc.md_content || doc.extracted_text}
<!-- article = 텍스트 네이티브(markdown 변환 비대상). md_status='skipped' 라도
"Markdown 제외" badge 를 띄우지 않도록 mdStatus 미전달(badge 는 mdStatus 로만 구동). -->
<MarkdownDoc
documentId={doc.id}
mdContent={doc.md_content}
mdFrontmatter={doc.md_frontmatter}
mdStatus={null}
mdExtractionError={doc.md_extraction_error}
mdExtractionQuality={doc.md_extraction_quality}
extractedText={doc.extracted_text}
class="mb-6"
/>
{/if}
{#if doc.edit_url}
<Button
variant="primary"
size="sm"
icon={ExternalLink}
href={doc.edit_url}
target="_blank"
>
원문 보기
</Button>
{/if}
</div>
{:else}
<EmptyState
icon={FileText}
title="인앱 미리보기 미지원"
description="포맷: {doc.file_format}"
/>
{/if}
</Card>
{#if doc.md_content || doc.extracted_text}<MarkdownDoc documentId={doc.id} mdContent={doc.md_content} mdFrontmatter={doc.md_frontmatter} mdStatus={null} mdExtractionError={doc.md_extraction_error} mdExtractionQuality={doc.md_extraction_quality} extractedText={doc.extracted_text} class="prose prose-base max-w-none" />{/if}
{#if doc.edit_url}<div class="mt-4"><Button variant="primary" size="sm" href={doc.edit_url} target="_blank">원문 보기</Button></div>{/if}
{:else}<EmptyState icon={FileText} title="인앱 미리보기 미지원" description="포맷: {doc.file_format}" />{/if}
</div>
{/snippet}
<!-- 손글씨 노트 패드 (자료실 자료, "필기" 토글 시) -->
{#if noteOpen && doc.category === 'library' && noteLoaded}
<Card class="overflow-hidden p-0">
<div class="h-[60vh] min-h-[400px] flex flex-col">
<HandwriteCanvas
sessionId={doc.id}
initialStrokes={noteStrokes}
onChange={(strokes) => saveNote(strokes)}
/>
</div>
</Card>
{/if}
<!-- 데스크탑: 본문 | 인사이트 레일 -->
<div class="hidden xl:grid xl:grid-cols-[minmax(0,1fr)_336px] gap-3.5 items-start">
{@render fbViewer()}
<div style="position:sticky;top:14px;">{@render rail()}</div>
</div>
<!-- 모바일: 인사이트(상단 상시) + 본문 -->
<div class="xl:hidden">
<div style="margin-bottom:12px;">{@render rail()}</div>
{@render fbViewer()}
</div>
{/if}
<!-- 오른쪽 — 메타 Tabs [정보 | AI | 관리] (카드 11개 수직 스프롤 해소) -->
<aside class="min-w-0">
<Card>
<Tabs
tabs={[
{ id: 'info', label: '정보' },
{ id: 'ai', label: 'AI' },
{ id: 'manage', label: '관리' },
]}
>
{#snippet children(tab)}
<div class="pt-3 space-y-4">
{#if tab === 'info'}
{#if doc.category === 'library'}
<ReadCounter
documentId={doc.id}
initialCount={doc.read_count ?? 0}
initialLastReadAt={doc.last_read_at ?? null}
/>
{/if}
<FileInfoView {doc} />
<ProcessingStatusView {doc} />
{:else if tab === 'ai'}
<AnalysisPanel docId={doc.id} doc={doc} />
<AIClassificationEditor {doc} />
<div>
<h4 class="text-xs font-semibold text-dim uppercase mb-1.5">관련 문서</h4>
<!-- TODO(backend): GET /documents/{id}/related?limit=10 (벡터 유사도) -->
<EmptyState
icon={FileText}
title="추후 지원"
description="관련 문서 추천은 backend 연동 후 제공됩니다."
/>
</div>
{:else}
<LibraryPathEditor {doc} />
<NoteEditor {doc} />
<EditUrlEditor {doc} />
<TagsEditor {doc} />
<div class="pt-2 border-t border-default">
<DocumentDangerZone {doc} ondelete={handleDocDelete} />
</div>
{/if}
</div>
{/snippet}
</Tabs>
</Card>
</aside>
</div>
<!-- 관리 (편집/삭제) — 헤더 '관리'로 토글 -->
{#if manageOpen}
<div style="background:#f4f7f1;border:1px solid #dde3d6;border-radius:14px;padding:16px 18px;margin-top:14px;">
<div style="font-size:12px;font-weight:700;color:#697061;margin-bottom:12px;letter-spacing:.3px;">관리 · 분류 편집</div>
<div class="grid grid-cols-1 md:grid-cols-2 gap-4">
<AIClassificationEditor {doc} />
<LibraryPathEditor {doc} />
<NoteEditor {doc} />
<EditUrlEditor {doc} />
<TagsEditor {doc} />
</div>
<div class="pt-3 mt-3 border-t border-default"><DocumentDangerZone {doc} ondelete={handleDocDelete} /></div>
</div>
{/if}
{#if noteOpen && doc.category === 'library' && noteLoaded}
<div style="background:#f4f7f1;border:1px solid #dde3d6;border-radius:14px;overflow:hidden;margin-top:14px;"><div class="h-[60vh] min-h-[400px] flex flex-col"><HandwriteCanvas sessionId={doc.id} initialStrokes={noteStrokes} onChange={(s) => saveNote(s)} /></div></div>
{/if}
<!-- 모바일 sticky 하단 바 — 자료실 자료의 학습 흐름 네비게이션 -->
{#if doc.category === 'library'}
<div class="lg:hidden fixed bottom-0 inset-x-0 z-30 bg-surface border-t border-default px-3 py-2 flex items-center gap-2 shadow-lg">
<button
type="button"
onclick={() => neighbors.prev && goto(`/documents/${neighbors.prev.id}`)}
disabled={!neighbors.prev}
class="px-2 py-2 rounded text-dim disabled:opacity-30 disabled:cursor-not-allowed"
aria-label="이전 자료"
><ChevronLeft size={20} /></button>
<button
type="button"
onclick={readAndGoNext}
disabled={!neighbors.next}
class="flex-1 flex items-center justify-center gap-1.5 px-3 py-2.5 rounded-lg bg-accent text-white text-sm font-medium disabled:opacity-50"
>
<Check size={16} />
{#if neighbors.next}
1회독 완료 + 다음
{:else}
1회독 완료 (마지막 자료)
{/if}
</button>
<button
type="button"
onclick={() => neighbors.next && goto(`/documents/${neighbors.next.id}`)}
disabled={!neighbors.next}
class="px-2 py-2 rounded text-dim disabled:opacity-30 disabled:cursor-not-allowed"
aria-label="다음 자료 (회독 카운트 안 함)"
><ChevronRight size={20} /></button>
<button type="button" onclick={() => neighbors.prev && goto(`/documents/${neighbors.prev.id}`)} disabled={!neighbors.prev} class="px-3 py-2 rounded text-dim disabled:opacity-30" aria-label="이전"></button>
<button type="button" onclick={readAndGoNext} disabled={!neighbors.next} class="flex-1 px-3 py-2.5 rounded-lg bg-accent text-white text-sm font-medium disabled:opacity-50">{#if neighbors.next}1회독 완료 + 다음{:else}1회독 완료 (마지막){/if}</button>
<button type="button" onclick={() => neighbors.next && goto(`/documents/${neighbors.next.id}`)} disabled={!neighbors.next} class="px-3 py-2 rounded text-dim disabled:opacity-30" aria-label="다음"></button>
</div>
<!-- 본문이 sticky 바 뒤에 가리지 않도록 패딩 -->
<div class="lg:hidden h-20"></div>
{/if}
{/if}
</div>
</div>
<style>
.d3node:hover { background: #ecf0e8; }
.d3active:hover { background: #e3ebdf; }
.d3child { position: relative; }
.d3child::before { content: ""; position: absolute; left: 2px; top: -3px; bottom: 50%; width: 1px; background: #cdd6c4; }
.d3child::after { content: ""; position: absolute; left: 2px; top: 50%; width: 7px; height: 1px; background: #cdd6c4; }
.m-secbody[open] .m-chev { transform: rotate(90deg); }
.d3warn { animation: d3pulse 2.4s ease-in-out infinite; }
@keyframes d3pulse { 0%, 100% { box-shadow: 0 0 0 0 rgba(181, 132, 10, .35); } 50% { box-shadow: 0 0 0 3px rgba(181, 132, 10, 0); } }
</style>
+19
View File
@@ -0,0 +1,19 @@
-- 2026-06-14 PR-Background-Jobs-Observability: 큐 밖 관리 스크립트(백필 등) 진행 가시화.
-- processing_queue 는 파이프라인 stage 전용 — hier_overnight_backfill / section_summary_pilot
-- 같은 off-queue 관리 스크립트는 여기에 진행상황을 남겨 대시보드 보드가 노출한다.
-- worker_jobs(user_id NOT NULL, worker-pool 전용)와 별개 — 이건 owner 없는 관리 작업 heartbeat.
-- 단일 statement (asyncpg multi-statement 불허 컨벤션). 인덱스는 소량 테이블이라 생략.
CREATE TABLE IF NOT EXISTS background_jobs (
id BIGSERIAL PRIMARY KEY,
kind TEXT NOT NULL, -- 'hier_redecompose' | 'section_summary' | ...
label TEXT, -- 사람이 읽는 대상 표기 (예: 'doc 5210 (Sec VIII)')
state TEXT NOT NULL DEFAULT 'running'
CHECK (state IN ('running', 'done', 'failed')),
processed INTEGER NOT NULL DEFAULT 0, -- 처리한 단위 수 (절/leaf 등)
total INTEGER, -- 전체 단위 수 (미상이면 NULL)
detail JSONB NOT NULL DEFAULT '{}'::jsonb,
error TEXT,
started_at TIMESTAMPTZ NOT NULL DEFAULT now(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
finished_at TIMESTAMPTZ
);
+20 -3
View File
@@ -32,6 +32,7 @@ from core.config import settings
from services.hier_decomp.builder import build_hier_tree
from services.hier_decomp.persist import persist_hier_tree
from services.search.llm_gate import Priority, acquire_mlx_gate
from services.background_jobs import finish_job, heartbeat, start_job
# 단일 진실: 절 분석 상수/헬퍼 (PROMPT_VERSION 일치 = 멱등 보존)
from section_summary_pilot import (
@@ -140,8 +141,10 @@ def _make_engine():
return create_async_engine(os.environ["DATABASE_URL"], pool_pre_ping=True)
async def _analyze_doc_leaves(session, client, doc_id, doc_domain, model_name, stop_at):
"""doc 의 미분석 hier leaf 분석 → upsert. stop_at(epoch) 넘으면 leaf 경계 중단."""
async def _analyze_doc_leaves(session, client, doc_id, doc_domain, model_name, stop_at,
engine=None, job_id=None, base_processed=0):
"""doc 의 미분석 hier leaf 분석 → upsert. stop_at(epoch) 넘으면 leaf 경계 중단.
engine/job_id 주어지면 background_jobs ~10절마다 진행 heartbeat(보드 가시화)."""
rows = (await session.execute(LEAF_SQL, {"doc": doc_id, "pv": PROMPT_VERSION})).mappings().all()
ok = fail = skip = 0
timings, types = [], []
@@ -187,6 +190,8 @@ async def _analyze_doc_leaves(session, client, doc_id, doc_domain, model_name, s
"content_hash": r["content_hash"], "error": err,
})
await session.commit()
if job_id and (ok + fail + skip) % 10 == 0:
await heartbeat(engine, job_id, processed=base_processed + ok + fail + skip)
await session.commit()
return {"ok": ok, "fail": fail, "skip": skip, "leaves": len(rows),
"timings": timings, "types": types, "aborted": aborted}
@@ -256,6 +261,12 @@ async def cmd_run(args):
_candidate_params(allowlist, doc_ids))).mappings().all()
_log(f"후보 doc {len(cands)} 선별. 시작.")
# 관측: 큐 밖 작업이라 대시보드 보드가 못 보므로 background_jobs 에 진행 노출(best-effort)
_job_kind = "hier_redecompose" if reprocess else "hier_backfill"
_job_label = (f"doc {args.doc} {'재분해' if reprocess else '분해'}" if doc_ids
else f"{len(cands)}개 문서 {'재분해' if reprocess else '분해'}")
job_id = await start_job(engine, _job_kind, _job_label, total=None)
for c in cands:
if time.time() >= stop_at:
_log(f"⏰ deadline 버퍼 도달 — doc 경계에서 중단 (처리 {tot_docs} doc)")
@@ -272,7 +283,10 @@ async def cmd_run(args):
"timings": [], "types": [], "aborted": False}
else:
async with sm() as session:
astat = await _analyze_doc_leaves(session, client, doc_id, doc_domain, model_name, stop_at)
astat = await _analyze_doc_leaves(
session, client, doc_id, doc_domain, model_name, stop_at,
engine=engine, job_id=job_id,
base_processed=(tot_ok + tot_fail + tot_skip))
except Exception as exc:
_log(f" ✗ doc={doc_id} 처리 실패(건너뜀): {type(exc).__name__}: {repr(exc)[:160]}")
continue
@@ -280,6 +294,8 @@ async def cmd_run(args):
tot_docs += 1
tot_ok += astat["ok"]; tot_fail += astat["fail"]; tot_skip += astat["skip"]
all_timings += astat["timings"]; all_types += astat["types"]
await heartbeat(engine, job_id, processed=(tot_ok + tot_fail + tot_skip),
total=tot_leaves_created)
avg = statistics.mean(astat["timings"]) if astat["timings"] else 0
_log(f" ✓ doc={doc_id} ({len(body):,}{doc_domain.split('/')[0]}) "
f"leaf생성={leaves_created} 분석ok={astat['ok']} fail={astat['fail']} skip={astat['skip']} "
@@ -287,6 +303,7 @@ async def cmd_run(args):
if astat["aborted"]:
_log("⏰ leaf 분석 중 deadline 도달 — 중단")
break
await finish_job(engine, job_id, state="done")
finally:
await client.close()
await engine.dispose()
+383
View File
@@ -0,0 +1,383 @@
<?xml version='1.0' encoding='UTF-8'?>
<feed xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns:arxiv="http://arxiv.org/schemas/atom" xmlns="http://www.w3.org/2005/Atom">
<id>https://arxiv.org/api/m9A/71G4hH6NGyarIQjqA3n6Zzk</id>
<title>arXiv Query: search_query=abs:"pressure vessel"&amp;id_list=&amp;start=0&amp;max_results=10</title>
<updated>2026-06-13T21:57:59Z</updated>
<link href="https://arxiv.org/api/query?search_query=abs:%22pressure+vessel%22&amp;start=0&amp;max_results=10&amp;id_list=" type="application/atom+xml"/>
<opensearch:itemsPerPage>10</opensearch:itemsPerPage>
<opensearch:totalResults>89</opensearch:totalResults>
<opensearch:startIndex>0</opensearch:startIndex>
<entry>
<id>http://arxiv.org/abs/1209.2405v1</id>
<title>A Survey of Pressure Vessel Code Compliance for Superconducting RF Cryomodules</title>
<updated>2012-09-11T19:34:46Z</updated>
<link href="https://arxiv.org/abs/1209.2405v1" rel="alternate" type="text/html"/>
<link href="https://arxiv.org/pdf/1209.2405v1" rel="related" type="application/pdf" title="pdf"/>
<summary>Superconducting radio frequency (SRF) cavities made from niobium and cooled with liquid helium are becoming key components of many particle accelerators. The helium vessels surrounding the RF cavities, portions of the niobium cavities themselves, and also possibly the vacuum vessels containing these assemblies, generally fall under the scope of local and national pressure vessel codes. In the U.S., Department of Energy rules require national laboratories to follow national consensus pressure vessel standards or to show "a level of safety greater than or equal to" that of the applicable standard. Thus, while used for its superconducting properties, niobium ends up being treated as a low-temperature pressure vessel material. Niobium material is not a code listed material and therefore requires the designer to understand the mechanical properties for material used in each pressure vessel fabrication; compliance with pressure vessel codes therefore becomes a problem. This report summarizes the approaches that various institutions have taken in order to bring superconducting RF cryomodules into compliance with pressure vessel codes.</summary>
<category term="physics.acc-ph" scheme="http://arxiv.org/schemas/atom"/>
<published>2012-09-11T19:34:46Z</published>
<arxiv:comment>7 pp</arxiv:comment>
<arxiv:primary_category term="physics.acc-ph"/>
<author>
<name>Thomas Peterson</name>
<arxiv:affiliation>Fermilab</arxiv:affiliation>
</author>
<author>
<name>Arkadiy Klebaner</name>
<arxiv:affiliation>Fermilab</arxiv:affiliation>
</author>
<author>
<name>Tom Nicol</name>
<arxiv:affiliation>Fermilab</arxiv:affiliation>
</author>
<author>
<name>Jay Theilacker</name>
<arxiv:affiliation>Fermilab</arxiv:affiliation>
</author>
<author>
<name>Hitoshi Hayano</name>
<arxiv:affiliation>KEK, Tsukuba</arxiv:affiliation>
</author>
<author>
<name>Eiji Kako</name>
<arxiv:affiliation>KEK, Tsukuba</arxiv:affiliation>
</author>
<author>
<name>Hirotaka Nakai</name>
<arxiv:affiliation>KEK, Tsukuba</arxiv:affiliation>
</author>
<author>
<name>Akira Yamamoto</name>
<arxiv:affiliation>KEK, Tsukuba</arxiv:affiliation>
</author>
<author>
<name>Kay Jensch</name>
<arxiv:affiliation>DESY</arxiv:affiliation>
</author>
<author>
<name>Axel Matheisen</name>
<arxiv:affiliation>DESY</arxiv:affiliation>
</author>
<author>
<name>John Mammosser</name>
<arxiv:affiliation>Jefferson Lab</arxiv:affiliation>
</author>
<arxiv:doi>10.1063/1.4707088</arxiv:doi>
<link rel="related" href="https://doi.org/10.1063/1.4707088" title="doi"/>
</entry>
<entry>
<id>http://arxiv.org/abs/2003.02057v1</id>
<title>Investigation of Unit-1 Nuclear Reactor of the Fukushima Daiichi by Cosmic Muon Radiography</title>
<updated>2020-03-03T03:21:53Z</updated>
<link href="https://arxiv.org/abs/2003.02057v1" rel="alternate" type="text/html"/>
<link href="https://arxiv.org/pdf/2003.02057v1" rel="related" type="application/pdf" title="pdf"/>
<summary>We have investigated the status of the nuclear fuel assemblies in Unit-1 reactor of the Fukushima Daiichi Nuclear Power plant by the method called Cosmic Muon Radiography. In this study, muon tracking detectors were placed outside of the reactor building. We succeeded in identifying the inner structure of the reactor complex such as the reactor containment vessel, pressure vessel, and other structures of the reactor building, through the concrete wall of the reactor building. We found that a large amount of fuel assemblies was missing in the original fuel loading zone inside the pressure vessel. It can be naturally interpreted that most of the nuclear fuel was melt and dropped down to the bottom of the pressure vessel or even below.</summary>
<category term="physics.ins-det" scheme="http://arxiv.org/schemas/atom"/>
<category term="hep-ex" scheme="http://arxiv.org/schemas/atom"/>
<published>2020-03-03T03:21:53Z</published>
<arxiv:comment>14 pages, 17 figures</arxiv:comment>
<arxiv:primary_category term="physics.ins-det"/>
<author>
<name>Hirofumi Fujii</name>
<arxiv:affiliation>High Energy Accelerator Research Organization</arxiv:affiliation>
</author>
<author>
<name>Kazuhiko Hara</name>
<arxiv:affiliation>University of Tsukuba</arxiv:affiliation>
</author>
<author>
<name>Kohei Hayashi</name>
<arxiv:affiliation>High Energy Accelerator Research Organization</arxiv:affiliation>
</author>
<author>
<name>Hidekazu Kakuno</name>
<arxiv:affiliation>Tokyo Metropolitan University</arxiv:affiliation>
</author>
<author>
<name>Hideyo Kodama</name>
<arxiv:affiliation>High Energy Accelerator Research Organization</arxiv:affiliation>
</author>
<author>
<name>Kanetada Nagamine</name>
<arxiv:affiliation>High Energy Accelerator Research Organization</arxiv:affiliation>
</author>
<author>
<name>Kotaro Sato</name>
<arxiv:affiliation>High Energy Accelerator Research Organization</arxiv:affiliation>
</author>
<author>
<name>Shin-Hong Kim</name>
<arxiv:affiliation>University of Tsukuba</arxiv:affiliation>
</author>
<author>
<name>Atsuto Suzuki</name>
<arxiv:affiliation>High Energy Accelerator Research Organization</arxiv:affiliation>
</author>
<author>
<name>Takayuki Sumiyoshi</name>
<arxiv:affiliation>Tokyo Metropolitan University</arxiv:affiliation>
</author>
<author>
<name>Kazuki Takahashi</name>
<arxiv:affiliation>University of Tsukuba</arxiv:affiliation>
</author>
<author>
<name>Fumihiko Takasaki</name>
<arxiv:affiliation>High Energy Accelerator Research Organization</arxiv:affiliation>
</author>
<author>
<name>Shuji Tanaka</name>
<arxiv:affiliation>High Energy Accelerator Research Organization</arxiv:affiliation>
</author>
<author>
<name>Satoru Yamashita</name>
<arxiv:affiliation>University of Tokyo</arxiv:affiliation>
</author>
</entry>
<entry>
<id>http://arxiv.org/abs/1609.07515v1</id>
<title>Low Background Stainless Steel for the Pressure Vessel in the PandaX-II Dark Matter Experiment</title>
<updated>2016-09-21T10:33:04Z</updated>
<link href="https://arxiv.org/abs/1609.07515v1" rel="alternate" type="text/html"/>
<link href="https://arxiv.org/pdf/1609.07515v1" rel="related" type="application/pdf" title="pdf"/>
<summary>We report on the custom produced low radiation background stainless steel and the welding rod for the PandaX experiment, one of the deep underground experiments to search for dark matter and neutrinoless double beta decay using xenon. The anthropogenic 60 Co concentration in these samples is at the range of 1 mBq/kg or lower. We also discuss the radioactivity of nuclear-grade stainless steel from TISCO which has a similar background rate. The PandaX-II pressure vessel was thus fabricated using the stainless steel from CISRI and TISCO. Based on the analysis of the radioactivity data, we also made discussions on potential candidate for low background metal materials for future pressure vessel development.</summary>
<category term="physics.ins-det" scheme="http://arxiv.org/schemas/atom"/>
<category term="hep-ex" scheme="http://arxiv.org/schemas/atom"/>
<published>2016-09-21T10:33:04Z</published>
<arxiv:primary_category term="physics.ins-det"/>
<author>
<name>Tao Zhang</name>
</author>
<author>
<name>Changbo Fu</name>
</author>
<author>
<name>Xiangdong Ji</name>
</author>
<author>
<name>Jianglai Liu</name>
</author>
<author>
<name>Xiang Liu</name>
</author>
<author>
<name>Xuming Wang</name>
</author>
<author>
<name>Chunfa Yao</name>
</author>
<author>
<name>Xunhua Yuan</name>
</author>
<arxiv:doi>10.1088/1748-0221/11/09/T09004</arxiv:doi>
<link rel="related" href="https://doi.org/10.1088/1748-0221/11/09/T09004" title="doi"/>
</entry>
<entry>
<id>http://arxiv.org/abs/2308.09786v1</id>
<title>Mechanical design of the optical modules intended for IceCube-Gen2</title>
<updated>2023-08-18T19:20:09Z</updated>
<link href="https://arxiv.org/abs/2308.09786v1" rel="alternate" type="text/html"/>
<link href="https://arxiv.org/pdf/2308.09786v1" rel="related" type="application/pdf" title="pdf"/>
<summary>IceCube-Gen2 is an expansion of the IceCube neutrino observatory at the South Pole that aims to increase the sensitivity to high-energy neutrinos by an order of magnitude. To this end, about 10,000 new optical modules will be installed, instrumenting a fiducial volume of about 8 km^3. Two newly developed optical module types increase current sensitivity per module by a factor of three by integrating 16 and 18 newly developed four-inch PMTs in specially designed 12.5-inch diameter pressure vessels. Both designs use conical silicone gel pads to optically couple the PMTs to the pressure vessel to increase photon collection efficiency. The outside portion of gel pads are pre-cast onto each PMT prior to integration, while the interiors are filled and cast after the PMT assemblies are installed in the pressure vessel via a pushing mechanism. This paper presents both the mechanical design, as well as the performance of prototype modules at high pressure (70 MPa) and low temperature (-40 degree Celsius), characteristic of the environment inside the South Pole ice.</summary>
<category term="astro-ph.IM" scheme="http://arxiv.org/schemas/atom"/>
<category term="astro-ph.HE" scheme="http://arxiv.org/schemas/atom"/>
<published>2023-08-18T19:20:09Z</published>
<arxiv:comment>Presented at the 38th International Cosmic Ray Conference (ICRC2023). See arXiv:2307.13048 for all IceCube-Gen2 contributions</arxiv:comment>
<arxiv:primary_category term="astro-ph.IM"/>
<author>
<name>Yuya Makino</name>
<arxiv:affiliation>for the IceCube-Gen2 Collaboration</arxiv:affiliation>
</author>
</entry>
<entry>
<id>http://arxiv.org/abs/0804.0261v1</id>
<title>Circulation in Blowdown Flows</title>
<updated>2008-04-01T22:22:32Z</updated>
<link href="https://arxiv.org/abs/0804.0261v1" rel="alternate" type="text/html"/>
<link href="https://arxiv.org/pdf/0804.0261v1" rel="related" type="application/pdf" title="pdf"/>
<summary> The blowdown of high pressure gas in a pressure vessel produces rapid adiabatic cooling of the gas remaining in the vessel. The gas near the wall is warmed by conduction from the wall, producing radial temperature and density gradients that affect the flow, the mass efflux rate and the thermodynamic states of both the outflowing and the contained gas. The resulting buoyancy-driven flow circulates gas through the vessel and reduces, but does not eliminate, these gradients. The purpose of this note is to estimate when blowdown cooling is rapid enough that the gas in the pressure vessel is neither isothermal nor isopycnic, though it remains isobaric. I define a dimensionless number, the buoyancy circulation number BC, that parametrizes these effects.</summary>
<category term="physics.flu-dyn" scheme="http://arxiv.org/schemas/atom"/>
<published>2008-04-01T22:22:32Z</published>
<arxiv:comment>5 pp., no figures</arxiv:comment>
<arxiv:primary_category term="physics.flu-dyn"/>
<arxiv:journal_ref>J. Pressure Vessel Tech. 131, 034501 (2009)</arxiv:journal_ref>
<author>
<name>J. I. Katz</name>
</author>
</entry>
<entry>
<id>http://arxiv.org/abs/1204.0234v1</id>
<title>Substantiation of Thermodynamic Criteria of Explosion Safety in Process of Severe Accidents in Pressure Vessel Reactors</title>
<updated>2012-03-27T11:21:14Z</updated>
<link href="https://arxiv.org/abs/1204.0234v1" rel="alternate" type="text/html"/>
<link href="https://arxiv.org/pdf/1204.0234v1" rel="related" type="application/pdf" title="pdf"/>
<summary>The paper represents original development of thermodynamic criteria of occurrence conditions of steam-gas explosions in the process of severe accidents. The received results can be used for modelling of processes of severe accidents in pressure vessel reactors.</summary>
<category term="physics.gen-ph" scheme="http://arxiv.org/schemas/atom"/>
<published>2012-03-27T11:21:14Z</published>
<arxiv:comment>5 pages, 1 figure</arxiv:comment>
<arxiv:primary_category term="physics.gen-ph"/>
<author>
<name>V. I. Skalozubov</name>
</author>
<author>
<name>V. N. Vashchenko</name>
</author>
<author>
<name>S. S. Jarovoj</name>
</author>
<author>
<name>V. Yu. Kochnyeva</name>
</author>
</entry>
<entry>
<id>http://arxiv.org/abs/2511.11485v1</id>
<title>Data-efficient U-Net for Segmentation of Carbide Microstructures in SEM Images of Steel Alloys</title>
<updated>2025-11-14T17:01:02Z</updated>
<link href="https://arxiv.org/abs/2511.11485v1" rel="alternate" type="text/html"/>
<link href="https://arxiv.org/pdf/2511.11485v1" rel="related" type="application/pdf" title="pdf"/>
<summary>Understanding reactor-pressure-vessel steel microstructure is crucial for predicting mechanical properties, as carbide precipitates both strengthen the alloy and can initiate cracks. In scanning electron microscopy images, gray-value overlap between carbides and matrix makes simple thresholding ineffective. We present a data-efficient segmentation pipeline using a lightweight U-Net (30.7~M parameters) trained on just \textbf{10 annotated scanning electron microscopy images}. Despite limited data, our model achieves a \textbf{Dice-Sørensen coefficient of 0.98}, significantly outperforming the state-of-the-art in the field of metallurgy (classical image analysis: 0.85), while reducing annotation effort by one order of magnitude compared to the state-of-the-art data efficient segmentation model. This approach enables rapid, automated carbide quantification for alloy design and generalizes to other steel types, demonstrating the potential of data-efficient deep learning in reactor-pressure-vessel steel analysis.</summary>
<category term="cs.LG" scheme="http://arxiv.org/schemas/atom"/>
<category term="cond-mat.mtrl-sci" scheme="http://arxiv.org/schemas/atom"/>
<published>2025-11-14T17:01:02Z</published>
<arxiv:primary_category term="cs.LG"/>
<arxiv:journal_ref>Machine Learning and the Physical Sciences Workshop @ NeurIPS 2025 https://openreview.net/forum?id=xYY5pn4f8N</arxiv:journal_ref>
<author>
<name>Alinda Ezgi Gerçek</name>
</author>
<author>
<name>Till Korten</name>
</author>
<author>
<name>Paul Chekhonin</name>
</author>
<author>
<name>Maleeha Hassan</name>
</author>
<author>
<name>Peter Steinbach</name>
</author>
</entry>
<entry>
<id>http://arxiv.org/abs/2511.09689v1</id>
<title>An ASME-Compliant Helium-4 Evaporation Refrigerator for the SpinQuest Experiment</title>
<updated>2025-11-12T19:45:47Z</updated>
<link href="https://arxiv.org/abs/2511.09689v1" rel="alternate" type="text/html"/>
<link href="https://arxiv.org/pdf/2511.09689v1" rel="related" type="application/pdf" title="pdf"/>
<summary>This paper presents the design, safety basis, and commissioning results of a 1 K liquid helium-4 (4He) evaporation refrigerator developed for the Fermilab SpinQuest Experiment (E1039). The system represents the first high power helium evaporation refrigerator operated in a fixed target scattering experiment at Fermilab and was engineered to comply with the Fermilab ES\&amp;H Manual (FESHM) requirements governing pressure vessels, piping, cryogenic systems, and vacuum vessels. The design is mapped to ASME B31.3 (Process Piping) and the ASME Boiler and Pressure Vessel Code (BPVC) for pressure boundary integrity and overpressure protection, with documented compliance to FESHM Chapters 5031 (Pressure Vessels), 5031.1 (Piping Systems), and 5033 (Vacuum Vessels). This work documents the methodology used to reach compliance and approval for the 4He evaporation refrigerator at Fermilab which the field lacks. Design considerations specific to the high radiation target-cave environment including remotely located instrumentation approximately 20 m from the cryostat are summarized, together with the relief-system sizing methodology used to accommodate transient heat loads from dynamic nuclear polarization microwaves and the high-intensity proton beam. Commissioning data from July 2024 confirms that the system satisfies all thermal performance and safety objectives.</summary>
<category term="physics.ins-det" scheme="http://arxiv.org/schemas/atom"/>
<published>2025-11-12T19:45:47Z</published>
<arxiv:comment>For IEEE Transactions in Nuclear Physics, 11 pages, 14 figures</arxiv:comment>
<arxiv:primary_category term="physics.ins-det"/>
<author>
<name>Jordan D. Roberts</name>
</author>
<author>
<name>Vibodha Bandara</name>
</author>
<author>
<name>Kenichi Nakano</name>
</author>
<author>
<name>Dustin Keller</name>
</author>
</entry>
<entry>
<id>http://arxiv.org/abs/1507.04072v1</id>
<title>High-Voltage Terminal Test of Test Stand for 1-MV Electrostatic Accelerator</title>
<updated>2015-07-15T02:41:11Z</updated>
<link href="https://arxiv.org/abs/1507.04072v1" rel="alternate" type="text/html"/>
<link href="https://arxiv.org/pdf/1507.04072v1" rel="related" type="application/pdf" title="pdf"/>
<summary>The Korea Multipurpose Accelerator Complex (KOMAC) has been developing a 300-kV test stand for a 1-MV electrostatic accelerator ion source. The ion source and accelerating tube will be installed in a high-pressure vessel. The ion source in the high-pressure vessel is required to have a high reliability. The test stand has been proposed and developed to confirm the stable operating conditions of the ion source. The ion source will be tested at the test stand to verify the long-time operating conditions. The test stand comprises a 300-kV high-voltage terminal, a battery for the ion-source power, a 60-Hz inverter, 200-MHz RF power, a 5-kV extraction power supply, a 300-kV accelerating tube, and a vacuum system. The results of the 300-kV high-voltage terminal tests are presented in this paper.</summary>
<category term="physics.acc-ph" scheme="http://arxiv.org/schemas/atom"/>
<published>2015-07-15T02:41:11Z</published>
<arxiv:comment>International Conference on Accelerators and Beam Utilization (ICABU2014)</arxiv:comment>
<arxiv:primary_category term="physics.acc-ph"/>
<arxiv:journal_ref>Yong-Sub Cho KNS (2014); W. Sima IEEE (2004) 480-483; LA-UR-87-126 (1987); Jeong-tae Kim KNS (2014)</arxiv:journal_ref>
<author>
<name>Sae-Hoon Park</name>
</author>
<author>
<name>Yu-Seok Kim</name>
</author>
<arxiv:doi>10.3938/jkps</arxiv:doi>
<link rel="related" href="https://doi.org/10.3938/jkps" title="doi"/>
</entry>
<entry>
<id>http://arxiv.org/abs/2005.05585v1</id>
<title>Investigation of the Status of Unit 2 Nuclear Reactor of the Fukushima Daiichi by the Cosmic Muon Radiography</title>
<updated>2020-05-12T07:26:37Z</updated>
<link href="https://arxiv.org/abs/2005.05585v1" rel="alternate" type="text/html"/>
<link href="https://arxiv.org/pdf/2005.05585v1" rel="related" type="application/pdf" title="pdf"/>
<summary>We have investigated the status of the nuclear debris in the Unit-2 Nuclear Reactor of the Fukushima Daiichi Nuclear Power plant by the method called Cosmic Muon Radiography. In this measurement, the muon detector was placed outside of the reactor building as was the case of the measurement for the Unit-1 Reactor. Compared to the previous measurements, the detector was down-sized, which made us possible to locate it closer to the reactor and to investigate especially the lower part of the fuel loading zone. We identified the inner structures of the reactor such as the containment vessel, pressure vessel and other objects through the thick concrete wall of the reactor building. Furthermore, the observation showed existence of heavy material at the bottom of the pressure vessel, which can be interpreted as the debris of melted nuclear fuel dropped from the loading zone.</summary>
<category term="physics.ins-det" scheme="http://arxiv.org/schemas/atom"/>
<published>2020-05-12T07:26:37Z</published>
<arxiv:comment>11 figures and 2 tables</arxiv:comment>
<arxiv:primary_category term="physics.ins-det"/>
<author>
<name>Hirofumi Fujii</name>
</author>
<author>
<name>Kazuhiko Hara</name>
</author>
<author>
<name>Shugo Hashimoto</name>
</author>
<author>
<name>Kohei Hayashi</name>
</author>
<author>
<name>Hidekazu Kakuno</name>
</author>
<author>
<name>Hideyo Kodama</name>
</author>
<author>
<name>Gi Meiki</name>
</author>
<author>
<name>Masato Mizokami</name>
</author>
<author>
<name>Shinya Mizokami</name>
</author>
<author>
<name>Kanetada Nagamine</name>
</author>
<author>
<name>Kotaro Sato</name>
</author>
<author>
<name>Shunsuke Sekita</name>
</author>
<author>
<name>Hiroshi Shirai</name>
</author>
<author>
<name>Shin-Hong Kim</name>
</author>
<author>
<name>Takayuki Sumiyoshi</name>
</author>
<author>
<name>Atsuto Suzuki</name>
</author>
<author>
<name>Yoshihisa Takada</name>
</author>
<author>
<name>Kazuki Takahashi</name>
</author>
<author>
<name>Yu Takahashi</name>
</author>
<author>
<name>Fumihiko Takasaki</name>
</author>
<author>
<name>Daichi Yamada</name>
</author>
<author>
<name>Satoru Yamashita</name>
</author>
</entry>
</feed>
File diff suppressed because one or more lines are too long
+75
View File
@@ -0,0 +1,75 @@
"""B-3 PR2 — arXiv 파서·쿼리빌더 순수 단위 테스트 (plan safety-library-b3-1).
fixture = arXiv API 실응답 박제(abs:"pressure vessel" relevance 10
DOI 보유 / journal_ref 보유 / 없음 3경로 포함). run()/적재(DB) PR2 라이브 검증.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent / "app"))
from workers.arxiv_collector import ( # noqa: E402
build_search_query,
parse_arxiv_feed,
)
FIX = Path(__file__).parent / "fixtures" / "arxiv_search_pressure_vessel.xml"
def _entries():
total, entries = parse_arxiv_feed(FIX.read_text(encoding="utf-8"))
return total, {e.arxiv_id: e for e in entries}, entries
# ─── 피드 레벨 ───
def test_feed_total_and_count():
total, by_id, entries = _entries()
assert total == 89 # fixture totalResults (페이징 재료)
assert len(entries) == 10
def test_versionless_ids():
_, by_id, entries = _entries()
# arxiv_id 는 versionless (버전 접미는 .version 으로 분리)
assert all("/" not in e.arxiv_id for e in entries)
assert "1209.2405" in by_id and by_id["1209.2405"].version == "v1"
# ─── DOI 보유 entry ───
def test_entry_with_doi():
_, by_id, _ = _entries()
e = by_id["1209.2405"]
assert e.doi == "10.1063/1.4707088" # normalize_doi 적용(소문자·정규화)
assert e.journal_ref is None
assert e.primary_category == "physics.acc-ph"
assert e.title.startswith("A Survey of Pressure Vessel")
assert len(e.summary) > 200 # 초록 본문
assert e.published is not None
assert e.abs_url and "/abs/" in e.abs_url
assert e.pdf_url and "pdf" in e.pdf_url
# ─── journal_ref 만 (DOI 없음) — 압력용기 저널 출판분 ───
def test_entry_journal_ref_without_doi():
_, by_id, _ = _entries()
e = by_id["0804.0261"]
assert e.doi is None
assert e.journal_ref and "Pressure Vessel" in e.journal_ref
# ─── 둘 다 없음(최근 preprint) 경로도 존재 ───
def test_entry_neither_doi_nor_journal_ref_exists():
_, _, entries = _entries()
assert any(e.doi is None and e.journal_ref is None for e in entries)
# ─── 쿼리 빌더 ───
def test_build_search_query():
q = build_search_query("eess.SY", ["pressure vessel", "safety"])
assert q == 'cat:eess.SY AND (abs:"pressure vessel" OR abs:safety)'
+106
View File
@@ -0,0 +1,106 @@
"""B-3 PR3 — OpenAlex 파서·초록복원·license 순수 단위 테스트 (plan safety-library-b3-1).
fixture = OpenAlex /works 실응답 박제(process safety/pressure vessel OA 5
cc-by/cc-by-nc-nd/license None, 초록 있음/없음). run()/적재(DB) PR3 라이브 검증.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent / "app"))
from workers.openalex_collector import ( # noqa: E402
_reconstruct_abstract,
_seeds,
build_filter,
build_issn_filter,
license_meta,
parse_openalex_works,
)
FIX = Path(__file__).parent / "fixtures" / "openalex_works_response.json"
def _works():
count, cursor, works = parse_openalex_works(FIX.read_text(encoding="utf-8"))
return count, {w.openalex_id: w for w in works}, works
# ─── 피드 레벨 ───
def test_count_and_results():
count, by_id, works = _works()
assert count == 1111
assert len(works) == 5
assert all(w.openalex_id.startswith("W") and "/" not in w.openalex_id for w in works)
# ─── 초록 보유 + CC 라이선스 ───
def test_work_with_abstract_and_cc():
_, by_id, _ = _works()
w = by_id["W2910511816"]
assert w.doi and w.doi.startswith("10.") and w.doi == w.doi.lower() # normalize_doi
assert len(w.abstract) > 50 # inverted-index 복원
assert w.oa_status == "diamond" and w.is_oa is True
assert w.license == "cc-by"
assert license_meta(w.license, w.is_oa, w.source_name)["redistribute"] is True
# ─── 초록 없는 thin 레코드(skip 대상) ───
def test_work_without_abstract():
_, by_id, _ = _works()
w = by_id["W3107397139"]
assert w.abstract == "" # inverted-index 부재 → 빈 초록
lm = license_meta(w.license, w.is_oa, w.source_name)
assert lm["redistribute"] is False # license None → 비배포
# ─── cc-by-nc-nd 도 CC 계열 → redistribute True ───
def test_cc_variant_redistribute():
_, by_id, _ = _works()
w = by_id["W4391130399"]
assert w.license == "cc-by-nc-nd"
assert license_meta(w.license, w.is_oa, w.source_name)["redistribute"] is True
# ─── 초록 inverted-index 복원 순서 ───
def test_reconstruct_abstract_order():
inv = {"Safety": [0], "of": [1, 4], "pressure": [2], "vessels": [3], "design": [5]}
assert _reconstruct_abstract(inv) == "Safety of pressure vessels of design"
assert _reconstruct_abstract(None) == ""
assert _reconstruct_abstract({}) == ""
# ─── license_meta 분기 ───
def test_license_meta_branches():
assert license_meta("cc-by", True, "X")["redistribute"] is True
assert license_meta("cc0", True, "X")["redistribute"] is True
none_oa = license_meta(None, True, "X")
assert none_oa["redistribute"] is False and none_oa["scheme"] == "open-unspecified"
closed = license_meta(None, False, "X")
assert closed["redistribute"] is False and closed["scheme"] == "proprietary"
# ─── 쿼리 빌더 ───
def test_build_filter():
assert build_filter("process safety") == "title_and_abstract.search:process safety"
assert build_filter("process safety", "2026-06-01") == \
"title_and_abstract.search:process safety,from_publication_date:2026-06-01"
# ─── PR6: ISSN 소스 시드 (KR/JP 안전 저널 직접 커버) ───
def test_build_issn_filter_and_seeds():
assert build_issn_filter("1738-3803") == "primary_location.source.issn:1738-3803"
assert build_issn_filter("1738-3803", "2026-01-01") == \
"primary_location.source.issn:1738-3803,from_publication_date:2026-01-01"
seeds = _seeds()
kinds = [k for _, _, k in seeds]
assert kinds[0] == "issn" # ISSN 시드가 키워드보다 먼저(cap 우선권)
assert any(v == "1738-3803" and k == "issn" for _, v, k in seeds) # 한국안전학회지 포함
+141
View File
@@ -0,0 +1,141 @@
"""B-3 PR1 — 논문 DOI 코어 순수 단위 테스트 (plan safety-library-b3-1).
holder.find_paper_holder(DB 조회) PR2 arXiv 실수집 라이브 검증 여기선 순수 함수만.
"""
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).parent.parent / "app"))
from services.papers.doi import ( # noqa: E402
arxiv_doi,
normalize_doi,
paper_doi_hash,
parse_arxiv_id,
parse_doi_from_text,
read_paper_doi,
with_paper_doi,
with_parent_doi,
)
# ─── normalize_doi: 단일 함수(저장=조회) ───
def test_normalize_strips_url_and_lowercases():
assert normalize_doi("https://doi.org/10.1585/PFR.15.2402039") == "10.1585/pfr.15.2402039"
assert normalize_doi("http://dx.doi.org/10.1115/1.4045678") == "10.1115/1.4045678"
assert normalize_doi("doi:10.1016/j.jlp.2020.104321") == "10.1016/j.jlp.2020.104321"
assert normalize_doi("DOI: 10.1234/ABC") == "10.1234/abc"
def test_normalize_trims_whitespace_and_citation_noise():
assert normalize_doi(" https://doi.org/10.1234/abc ") == "10.1234/abc"
assert normalize_doi("10.1234/abc.") == "10.1234/abc"
assert normalize_doi("10.1234/abc;") == "10.1234/abc"
def test_normalize_preserves_parens_in_doi():
# 괄호는 DOI 일부일 수 있어 보존 (과삭제 = 다른 논문 병합 = 데이터 손상, near-dup 보다 위험)
assert normalize_doi("10.1016/s0010-8650(00)80003-2") == "10.1016/s0010-8650(00)80003-2"
assert normalize_doi("https://doi.org/10.1016/S0010-8650(00)80003-2") == "10.1016/s0010-8650(00)80003-2"
def test_normalize_rejects_non_doi():
assert normalize_doi(None) is None
assert normalize_doi("") is None
assert normalize_doi(" ") is None
assert normalize_doi("not-a-doi") is None
assert normalize_doi("arXiv:2606.08108") is None # arXiv id 는 DOI 아님
def test_normalize_is_idempotent_store_equals_lookup():
# 저장측·조회측이 같은 함수를 거치면 표기 차이가 한 값으로 붕괴 (dedup 성립 조건)
forms = [
"https://doi.org/10.1/X",
"doi:10.1/x",
"10.1/X",
" HTTPS://DOI.ORG/10.1/x ",
]
assert {normalize_doi(f) for f in forms} == {"10.1/x"}
assert normalize_doi(normalize_doi("https://doi.org/10.1/X")) == "10.1/x" # 멱등
# ─── paper_doi_hash: holder file_hash 키 ───
def test_paper_doi_hash_deterministic_len32():
h = paper_doi_hash("10.1234/abc")
assert len(h) == 32
assert h == paper_doi_hash("10.1234/abc")
def test_paper_doi_hash_distinct_per_doi():
assert paper_doi_hash("10.1/a") != paper_doi_hash("10.1/b")
# ─── 2-Document extract_meta 계약 (holder doi / child parent_doi 상호 배타) ───
def test_with_paper_doi_holder_shape_and_merge_safe():
meta = with_paper_doi({"license": {"scheme": "cc_by"}, "source_id": 7}, "10.1/x")
assert meta["paper"]["doi"] == "10.1/x"
assert "parent_doi" not in meta["paper"]
assert meta["license"]["scheme"] == "cc_by" # 타 키 보존
assert meta["source_id"] == 7
def test_with_parent_doi_child_shape_no_doi():
meta = with_parent_doi({"license": {"scheme": "proprietary"}}, "10.1/holder")
assert meta["paper"]["parent_doi"] == "10.1/holder"
assert "doi" not in meta["paper"] # child 는 doi 미보유 (partial-unique 인덱스 밖)
assert meta["license"]["scheme"] == "proprietary"
def test_holder_child_mutually_exclusive():
child = with_parent_doi({}, "10.1/p")
promoted = with_paper_doi(child, "10.1/self")
assert promoted["paper"]["doi"] == "10.1/self"
assert "parent_doi" not in promoted["paper"]
def test_input_not_mutated():
src = {"paper": {"doi": "10.1/old"}}
with_parent_doi(src, "10.1/new")
assert src["paper"]["doi"] == "10.1/old" # 원본 dict 불변
# ─── read_paper_doi: 인덱스 식의 조회측 거울 ───
def test_read_paper_doi():
assert read_paper_doi({"paper": {"doi": "10.1/x"}}) == "10.1/x"
assert read_paper_doi({"paper": {"doi": "https://doi.org/10.1/X"}}) == "10.1/x" # 방어적 재정규화
assert read_paper_doi({}) is None
assert read_paper_doi(None) is None
assert read_paper_doi({"paper": {"parent_doi": "10.1/p"}}) is None # child 는 doi 없음
assert read_paper_doi({"paper": {}}) is None
# ─── PR4: arXiv id 파싱 + arXiv DataCite DOI (교차소스 dedup 통일 키) ───
def test_parse_arxiv_id():
assert parse_arxiv_id("Title arXiv:2606.10236v1 Announce Type: new Abstract") == "2606.10236"
assert parse_arxiv_id("see arXiv:2601.02852 for details") == "2601.02852"
assert parse_arxiv_id("arXiv:cond-mat/0703470v2") == "cond-mat/0703470"
assert parse_arxiv_id("no arxiv here") is None
assert parse_arxiv_id(None) is None
def test_arxiv_doi_canonical():
# OpenAlex canonical 실측 일치: 10.48550/arxiv.{id} (소문자)
assert arxiv_doi("2606.10236") == "10.48550/arxiv.2606.10236"
assert arxiv_doi(None) is None
# 수집기·reconcile 가 같은 함수 → 같은 paper.doi (교차소스 dedup 성립)
assert arxiv_doi(parse_arxiv_id("x arXiv:2606.10236v1 y")) == "10.48550/arxiv.2606.10236"
# ─── PR5: 구매 PDF 본문 DOI 파싱 (parent_doi 링크용, PDF 구조 무관) ───
def test_parse_doi_from_text():
assert parse_doi_from_text("ref https://doi.org/10.1016/j.jlp.2024.105474 end") == "10.1016/j.jlp.2024.105474"
assert parse_doi_from_text("DOI 10.1115/1.4045678. Next.") == "10.1115/1.4045678"
assert parse_doi_from_text("no doi here") is None
assert parse_doi_from_text(None) is None
+26
View File
@@ -103,6 +103,32 @@ def test_summarize_pool_split_attribution():
assert macbook["pending"] == 0 # 풀 pending 은 macmini 만
def test_summarize_by_machine_projection():
"""build_summarize_by_machine = split 의 done_1h/done_today 를 머신별로 투영
(done_15m 제외 내부 state 판정 전용)."""
from services.queue_overview import build_summarize_by_machine
split = _split(
macbook={"done_1h": 226, "done_today": 312, "done_15m": 60},
macmini={"done_1h": 37, "done_today": 94, "done_15m": 9},
)
sbm = build_summarize_by_machine(split)
assert sbm == {
"macmini": {"done_1h": 37, "done_today": 94},
"macbook": {"done_1h": 226, "done_today": 312},
}
assert "done_15m" not in sbm["macbook"]
def test_compose_overview_includes_summarize_by_machine():
"""compose_overview 응답 계약에 summarize_by_machine 포함 (FE 레인 분담 재료)."""
now_kst = datetime(2026, 6, 13, 13, 0, tzinfo=KST)
stats = {"summarize": _stage(pending=1317, done_1h=264)}
split = _split(macbook={"done_1h": 226, "done_today": 312}, macmini={"done_1h": 37, "done_today": 94})
ov = compose_overview(stats, split, {}, {}, [], deep_enabled=True, now_kst=now_kst)
assert ov["summarize_by_machine"]["macbook"]["done_1h"] == 226
assert ov["summarize_by_machine"]["macmini"]["done_today"] == 94
def test_deep_disabled_deep_summary_counts_to_macmini():
stats = {"deep_summary": _stage(pending=2, processing=1, done_1h=3, done_today=4)}
machines = build_machines(stats, _split(), [], deep_enabled=False)