44 Commits

Author SHA1 Message Date
Hyungi Ahn
06da098eab fix: 법령 분할 — 조문키 000 기반 장(章) 단위 분할로 변경
국가법령 XML은 <편>/<장> 태그가 아닌 <조문단위 조문키="xxxx000">에
"제X장 ..." 형태로 장 구분자가 포함됨. 이를 파싱하여 분할.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 15:05:48 +09:00
Hyungi Ahn
1668be0a75 fix: 법령 저장 후 즉시 commit — 알림 실패가 DB 롤백하지 않도록
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 14:50:39 +09:00
Hyungi Ahn
93c5805060 feat: 법령 API 전면 개편 — 26개 법령, 분할 저장, 변경 이력 추적
- 모니터링 법령 12개 → 26개 (산업안전/건설/위험물/소방/전기/가스/근로/환경)
- lawSearch.do로 검색, lawService.do로 본문 조회
- 대형 법령 편/장 단위 분할 저장 (fallback: 편→장→전체)
- 저장 경로: PKM/Inbox/ (AI 자동 분류 연계)
- 변경 감지 시 user_note에 이력 자동 기록
- CalDAV + SMTP 알림

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 14:47:08 +09:00
Hyungi Ahn
b4ca918125 fix: 벡터 검색 asyncpg 캐스트 — ::vector → cast(:embedding AS vector)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 14:35:14 +09:00
Hyungi Ahn
e23c4feaa0 feat: 검색 전면 개편 — 필드별 가중치 + 벡터 합산 + match reason
검색 대상: title > ai_tags > user_note > ai_summary > extracted_text
- 필드별 가중치: title(3.0), tags(2.5), note(2.0), summary(1.5), text(1.0)
- 벡터 검색: 별도 쿼리로 분리, 결과 합산 (asyncpg 충돌 방지)
- match_reason: 어떤 필드에서 매칭됐는지 반환
- 중복 제거 + 점수 합산

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 14:33:34 +09:00
Hyungi Ahn
e7cd710e69 fix: hybrid 검색 단순화 — FTS + ILIKE (vector/trgm 복잡 쿼리 제거)
asyncpg 파라미터 바인딩 충돌 문제 근본 해결.
한국어 검색: ILIKE fallback으로 안정 동작.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 14:16:36 +09:00
Hyungi Ahn
3236b8d812 fix: 검색 500 에러 (ILIKE % 이스케이프) + 한글 조합 중 Enter 방지
- ILIKE '%' → '%%' (SQLAlchemy text() 파라미터 충돌 해결)
- e.isComposing 체크로 한글 조합 완료 전 Enter 무시

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 14:14:07 +09:00
Hyungi Ahn
4d205b67c2 fix: 검색 UX 개선 — Enter 키 기반 + 한국어 검색 ILIKE fallback
- 프론트: debounce 자동검색 제거 → Enter 키로만 검색 (한글 조합 문제 해결)
- 백엔드: trgm threshold 0.1로 낮춤 + ILIKE '%검색어%' fallback 추가
- hybrid 검색 score threshold 0.01 → 0.001로 낮춤

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 14:10:47 +09:00
Hyungi Ahn
b54cc25650 fix: 미분류 판단 기준 변경 — file_path 기반 → ai_domain 없음 기준
파일을 물리적으로 이동하지 않으므로 file_path로 미분류 판단 불가.
ai_domain이 NULL 또는 빈 문자열인 문서를 미분류로 취급.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 14:05:41 +09:00
Hyungi Ahn
d63a6b85e1 feat: 사이드바 3단계 재귀 트리 + 너비 확장 (320px)
- tree API: domain 경로를 파싱하여 계층 구조로 반환
  (Industrial_Safety → Practice → Patrol_Inspection)
- Sidebar: 재귀 snippet으로 N단계 트리 렌더링
- domain 필터: prefix 매칭 (상위 클릭 시 하위 전부 포함)
- 사이드바 너비: 260px → 320px

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 14:03:36 +09:00
Hyungi Ahn
7f5e09096a fix: 문서 삭제 시 processing_queue FK 제약 해결 + 변환본/preview 함께 삭제
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 13:51:35 +09:00
Hyungi Ahn
63f75de89d fix: Qwen3.5 thinking 모드 비활성화 (enable_thinking: false)
JSON 응답에 Thinking Process 텍스트가 섞이는 문제 해결.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 13:38:10 +09:00
Hyungi Ahn
6d73e7ee12 feat: 분류 체계 전면 개편 — taxonomy + document_type + confidence
- config.yaml: 6개 domain × 3단계 taxonomy + 13개 document_types 정의
- classify.txt: 영문 프롬프트, taxonomy 경로 기반 분류 + 분류 규칙 주입
- classify_worker: taxonomy 검증, confidence 기반 분류, document_type 저장
- migration 008: document_type, importance, ai_confidence 컬럼
- API: DocumentResponse에 document_type, importance, ai_confidence 추가

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 13:32:20 +09:00
Hyungi Ahn
1b5fa95a9f feat: 오피스 → ODF 변환 + 원본/편집본 분리 아키텍처
- original_path/format/hash + conversion_status 필드 추가 (migration 007)
- extract_worker: 텍스트 추출 후 xlsx→ods, docx→odt 등 ODF 변환
  - 변환본은 .derived/{doc_id}.ods 에 저장
  - 원본 메타 보존 (original_path/format/hash)
- file_watcher: .derived/ .preview/ 디렉토리 제외
- DocumentViewer: ODF 포맷이면 편집 버튼 자동 표시
  - edit_url 있으면 "편집", 없으면 "Synology Drive에서 열기"

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 13:11:43 +09:00
Hyungi Ahn
b937eb948b feat: Noto 다국어 폰트 추가 (fonts-noto-core/extra — 라틴/아랍/태국 등)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 12:45:02 +09:00
Hyungi Ahn
1030bffc82 fix: LibreOffice 한글/CJK 폰트 추가 (fonts-noto-cjk, fonts-nanum)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 12:43:13 +09:00
Hyungi Ahn
733f730e16 fix: preview enum 누락 + AI summary thinking 제거 + CLAUDE.md 전면 갱신
- queue.py: process_stage enum에 'preview' 추가
- classify_worker: ai_summary에 strip_thinking() 적용
- CLAUDE.md: 현재 아키텍처 전면 반영 (파이프라인, UI, 인프라, 코딩규칙)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 12:38:59 +09:00
Hyungi Ahn
6893ea132d refactor: preview 병렬 트리거 + 파일 이동 제거 + domain 색상 바
- queue_consumer: extract 완료 시 classify + preview 동시 등록
- classify_worker: _move_to_knowledge() 제거, 파일 원본 위치 유지
- DocumentCard: 좌측 domain별 색상 바 (4px) 추가

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 12:31:57 +09:00
Hyungi Ahn
47e9981660 fix: Qwen3.5 Thinking Process 텍스트 제거 — JSON 파싱 개선
첫 번째 { 이전의 모든 비-JSON 텍스트를 제거하여
thinking/reasoning preamble이 있어도 JSON 추출 가능.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 11:44:21 +09:00
Hyungi Ahn
03b0612aa2 fix: extract_worker OFFICE_FORMATS 블록에 return 누락 수정
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 11:28:09 +09:00
Hyungi Ahn
a5186bf4aa fix: 스프레드시트 텍스트 추출 — csv 필터 사용 (txt:Text는 Calc 미지원)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 11:21:29 +09:00
Hyungi Ahn
b37043d651 fix: LibreOffice 한글 파일명 호환 — 영문 임시파일로 복사 후 변환
extract_worker, preview_worker 모두 적용.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 11:18:06 +09:00
Hyungi Ahn
45448b4036 feat: extract_worker에 LibreOffice 텍스트 추출 추가 (오피스 포맷)
- xlsx, docx, pptx, odt, ods, odp, odoc, osheet 지원
- LibreOffice --convert-to txt로 텍스트 추출 (60s timeout)
- 추가 의존성 없음 (Docker에 이미 설치된 LibreOffice 사용)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 11:12:19 +09:00
Hyungi Ahn
41072a2e6d feat: 수동 편집 URL — 정보 패널에서 Synology Drive 링크 입력/관리
- edit_url 컬럼 추가 (migration 006)
- PreviewPanel: 편집 링크 입력/수정/표시 UI
- DocumentViewer: edit_url 있으면 편집 버튼에서 해당 URL로 새 탭
- API: DocumentResponse/DocumentUpdate에 edit_url 필드

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 10:37:44 +09:00
Hyungi Ahn
4bea408bbd feat: Markdown 편집기 + PDF 변환 파이프라인 + 뷰어 포맷 분기
- Markdown split editor: textarea + marked preview, Ctrl+S 저장
- PUT /api/documents/{id}/content: 원본 파일 저장 + extracted_text 갱신
- GET /api/documents/{id}/preview: PDF 미리보기 캐시 서빙
- preview_worker: LibreOffice headless → PDF 변환 (timeout 60s, retry 1회)
- queue_consumer: preview stage 추가 (embed 후 자동 트리거)
- DocumentViewer: 포맷별 분기 (markdown/pdf/preview-pdf/image/text/cad)
- 오피스/CAD 문서: 새 탭 편집 버튼
- Dockerfile: LibreOffice headless 설치
- migration 005: preview_status, preview_hash, preview_at 컬럼

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 10:10:03 +09:00
Hyungi Ahn
17d41a8526 feat: Phase 1D+2 — 모바일 대응, 스마트 그룹, 메모, 태그 편집
- 모바일: 카드 클릭 시 detail 페이지로 이동 (뷰어 패널 미표시)
- 스마트 그룹: 사이드바에 최근 7일/법령 알림/이메일 프리셋 필터
- 메모: user_note 컬럼 추가 (migration 004), PATCH API, PreviewPanel 인라인 편집
- 태그 편집: PreviewPanel에서 태그 추가(+)/삭제(×) 기능
- DB 모델 + API 스키마 user_note 필드 추가

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 09:27:18 +09:00
Hyungi Ahn
87747866b6 feat: Phase 1A — 사이드바 트리 네비게이션 + domain/sub_group 필터
- Sidebar.svelte: /api/documents/tree 기반 domain→sub_group 트리,
  접기/펼치기, active highlight, 모바일 drawer
- documents/+page.svelte: 2-pane 레이아웃, URL params 기반 필터,
  빈 상태 개선, 카드 정보 밀도 향상 (domain 경로, 태그, origin 배지)
- documents.py: sub_group 필터 파라미터 추가
- app.css: domain 7색 + sidebar CSS 변수

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 08:54:09 +09:00
Hyungi Ahn
faf9bda77a fix: set correct Content-Type and inline disposition for file serving
PDF was downloading instead of displaying because media_type was None
(defaulting to octet-stream). Now maps file extensions to proper MIME
types and sets Content-Disposition: inline for in-browser viewing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 08:38:10 +09:00
Hyungi Ahn
1affcb1afd fix: add query param token auth for file serving (iframe compat)
iframe/img tags can't send Bearer headers. File endpoint now accepts
?token= query parameter for authentication. Frontend passes access
token in URL for PDF/image viewers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 08:34:45 +09:00
Hyungi Ahn
e14084d5cd feat: add file serving endpoint GET /api/documents/{id}/file
Returns original document file from NAS. Fixes 404 on PDF/image
viewer in frontend. Updated frontend iframe/img src to match.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 08:32:51 +09:00
Hyungi Ahn
62f5eccb96 fix: isolate each worker call in independent async session
Shared session between queue consumer and workers caused
MissingGreenlet errors in APScheduler context. Each worker
call now gets its own session with explicit commit/rollback.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 08:29:14 +09:00
Hyungi Ahn
87683ca000 security: NAS 마운트 검증 + AI 서비스 포트 제한 + deploy 문서 갱신
- NAS fail-fast: 시작 시 /documents/PKM 존재 확인, NFS 미마운트 방지
- ollama/ai-gateway 포트를 127.0.0.1로 제한 (외부 무인증 접근 차단)
- deploy.md: Caddy HTTPS 자동발급 → 앞단 프록시 HTTPS 종료 구조 반영

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 08:25:07 +09:00
Hyungi Ahn
17c1b7cf30 fix: set refresh cookie secure=False, samesite=lax for reverse proxy chain
Nginx terminates TLS and forwards HTTP internally. Secure=True cookies
don't get sent when the backend sees HTTP connections.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 06:53:59 +09:00
Hyungi Ahn
cfa95ff031 feat: implement Phase 4 SvelteKit frontend + backend enhancements
Backend:
- Add dashboard API (today stats, inbox count, law alerts, pipeline status)
- Add /api/documents/tree endpoint for sidebar domain/sub_group tree
- Migrate auth to HttpOnly cookie for refresh token (XSS defense)
- Add /api/auth/logout endpoint (cookie cleanup)
- Register dashboard router in main.py

Frontend (SvelteKit + Tailwind CSS v4):
- api.ts: fetch wrapper with refresh queue pattern, 401 single retry,
  forced logout on refresh failure
- Auth store: login/logout/refresh with memory-based access token
- UI store: toast system, sidebar state
- Login page with TOTP support
- Dashboard with 4 stat widgets + recent documents
- Document list with hybrid search (debounce, URL query state, mode select)
- Document detail with format-aware viewer (markdown/PDF/HWP/Synology/fallback)
- Metadata panel (AI summary, tags, processing history)
- Inbox triage UI (batch select, confirm dialog, domain override)
- Settings page (password change, TOTP status)

Infrastructure:
- Enable frontend service in docker-compose
- Caddy path routing (/api/* → fastapi, / → frontend) + gzip

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 06:46:19 +09:00
Hyungi Ahn
46537ee11a fix: Codex 리뷰 P1/P2 버그 4건 수정
- [P1] migration runner 도입: schema_migrations 추적, advisory lock,
  단일 트랜잭션 실행, SQL 검증 (기존 DB 업그레이드 대응)
- [P1] eml extract 큐 조건 분기: extract_worker 미지원 포맷 큐 스킵
- [P2] iCalendar escape_ical_text() 추가: RFC 5545 준수
- [P2] 이메일 charset 감지: get_content_charset() 사용 + payload None 방어

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 15:55:38 +09:00
Hyungi Ahn
d93e50b55c security: fix 5 review findings (2 high, 3 medium)
HIGH:
- Lock setup TOTP/NAS endpoints behind _require_setup() guard
  (prevented unauthenticated admin 2FA takeover after setup)
- Sanitize upload filename with Path().name + resolve() validation
  (prevented path traversal writing outside Inbox)

MEDIUM:
- Add score > 0.01 filter to hybrid search via subquery
  (prevented returning irrelevant documents with zero score)
- Implement Inbox → Knowledge file move after classification
  (classify_worker now moves files based on ai_domain)
- Add Anthropic Messages API support in _request()
  (premium/Claude path now sends correct format and parses
  content[0].text instead of choices[0].message.content)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 15:33:31 +09:00
Hyungi Ahn
31d5498f8d feat: implement Phase 3 automation workers
- Add automation_state table for incremental sync (last UID, last check)
- Add law_monitor worker: 국가법령정보센터 API → NAS/DB/CalDAV VTODO
  (LAW_OC 승인 대기 중, 코드 완성)
- Add mailplus_archive worker: IMAP(993) → .eml NAS save + DB + SMTP
  notification (imaplib via asyncio.to_thread, timeout=30)
- Add daily_digest worker: PostgreSQL/pipeline stats → Markdown + SMTP
  (documents, law changes, email, queue errors, inbox backlog)
- Add CalDAV VTODO helper and SMTP email helper to core/utils.py
- Wire 3 cron jobs in APScheduler (law@07:00, mail@07:00+18:00,
  digest@20:00) with timezone=Asia/Seoul

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 15:24:50 +09:00
Hyungi Ahn
a5312c044b fix: replace deprecated regex with pattern in search Query param
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 15:02:44 +09:00
Hyungi Ahn
4b695332b9 feat: implement Phase 2 core features
- Add document CRUD API (list/get/upload/update/delete with auth)
  - Upload saves to Inbox + auto-enqueues processing pipeline
  - Delete defaults to DB-only, explicit flag for file deletion
- Add hybrid search API (FTS 0.4 + trigram 0.2 + vector 0.4 weighted)
  - Modes: fts, trgm, vector, hybrid (default)
  - Vector search gracefully degrades if GPU unavailable
- Add Inbox file watcher (5min interval, new file + hash change detection)
- Register documents/search routers and file_watcher scheduler in main.py
- Add IVFFLAT vector index migration (lists=50, with tuning guide)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 14:49:12 +09:00
Hyungi Ahn
299fac3904 feat: implement Phase 1 data pipeline and migration
- Implement kordoc /parse endpoint (HWP/HWPX/PDF via kordoc lib,
  text files direct read, images flagged for OCR)
- Add queue consumer with APScheduler (1min interval, stage chaining
  extract→classify→embed, stale item recovery, retry logic)
- Add extract worker (kordoc HTTP call + direct text read)
- Add classify worker (Qwen3.5 AI classification with think-tag
  stripping and robust JSON extraction from AI responses)
- Add embed worker (GPU server nomic-embed-text, graceful failure)
- Add DEVONthink migration script with folder mapping for 16 DBs,
  dry-run mode, batch commits, and idempotent file_path UNIQUE
- Enhance ai/client.py with strip_thinking() and parse_json_response()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 14:35:36 +09:00
Hyungi Ahn
23ee055357 fix: replace passlib with bcrypt directly (passlib+bcrypt 5.0 incompatible)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 14:00:43 +09:00
Hyungi Ahn
e63d2971a9 fix: update TemplateResponse call for Starlette 1.0 API
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 13:58:16 +09:00
Hyungi Ahn
a601991f48 feat: implement Phase 0 auth system, setup wizard, and Docker config
- Add users table to migration, User ORM model
- Implement JWT+TOTP auth API (login, refresh, me, change-password)
- Add first-run setup wizard with rate-limited admin creation,
  TOTP QR enrollment (secret saved only after verification), and
  NAS path verification — served as Jinja2 single-page HTML
- Add setup redirect middleware (bypasses /health, /docs, /openapi.json)
- Mount config.yaml, scripts, logs volumes in docker-compose
- Route API vs frontend traffic in Caddyfile
- Include admin seed script as CLI fallback

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 13:21:45 +09:00
Hyungi Ahn
131dbd7b7c feat: scaffold v2 project structure with Docker, FastAPI, and config
동작하는 최소 코드 수준의 v2 스캐폴딩:

- docker-compose.yml: postgres, fastapi, kordoc, frontend, caddy
- app/: FastAPI 백엔드 (main, core, models, ai, prompts)
- services/kordoc/: Node.js 문서 파싱 마이크로서비스
- gpu-server/: AI Gateway + GPU docker-compose
- frontend/: SvelteKit 기본 구조
- migrations/: PostgreSQL 초기 스키마 (documents, tasks, processing_queue)
- tests/: pytest conftest 기본 설정
- config.yaml, Caddyfile, credentials.env.example 갱신

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 10:20:15 +09:00