hyungi/hyungi_document_server
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
182 Commits 3 Branches 1 Tag
f5c3dea8331acc857d52e0a116af0434c2f7d21b
Commit Graph

9 Commits

Author SHA1 Message Date
Hyungi Ahn
c81b728ddf refactor(search): Phase 2.1 QueryAnalyzer를 async-only 구조로 전환 
## 철학 수정 (실측 기반)

gemma-4-26b-a4b-it-8bit MLX 실측:
  - full query_analyze.txt (prompt_tok=2406) → 10.5초
  - max_tokens 축소 무효 (모델 자연 EOS 조기 종료)
  - 쿼리 길이 영향 거의 없음 (프롬프트 자체가 지배)
  → 800ms timeout 가정은 13배 초과. 동기 호출 완전히 불가능.

따라서 QueryAnalyzer는 "즉시 실행하는 기능" → "미리 준비해두는 기능"으로
포지셔닝 변경. retrieval 경로에서 analyzer 동기 호출 **금지**.

## 구조

```
query → retrieval (항상 즉시)
         ↘ trigger_background_analysis (fire-and-forget)
            → analyze() [5초+] → cache 저장

다음 호출 (동일 쿼리) → get_cached() 히트 → Phase 2 파이프라인 활성화
```

## 변경 사항

### app/prompts/query_analyze.txt
 - 5971 chars → 2403 chars (40%)
 - 예시 4개 → 1개, 규칙 설명 축약
 - 목표 prompt_tok 2406 → ~600 (1/4)

### app/services/search/query_analyzer.py
 - LLM_TIMEOUT_MS 800 → 5000 (background이므로 여유 OK)
 - PROMPT_VERSION v1 → v2 (cache auto-invalidate)
 - get_cached / set_cached 유지 — retrieval 경로 O(1) 조회
 - trigger_background_analysis(query) 신규 — 동기 함수, 즉시 반환, task 생성
 - _PENDING set으로 task 참조 유지 (premature GC 방지)
 - _INFLIGHT set으로 동일 쿼리 중복 실행 방지
 - prewarm_analyzer() 신규 — startup에서 15~20 쿼리 미리 분석
 - DEFAULT_PREWARM_QUERIES: 평가셋 fixed 7 + 법령 3 + 뉴스 2 + 실무 3

### app/api/search.py
 - 기존 sync analyzer 호출 완전 제거
 - analyze=True → get_cached(q) 조회만 O(1)
   - hit: query_analysis 활용 (Phase 2.2/2.3 파이프라인 조건부 활성화)
   - miss: trigger_background_analysis(q) + 기존 경로 그대로
 - timing["analyze_ms"] 제거 (경로에 LLM 호출 없음)
 - notes에 analyzer cache_hit/cache_miss 상태 기록
 - debug.query_analysis는 cache hit 시에만 채워짐

### app/main.py
 - lifespan startup에 prewarm_analyzer() background task 추가
 - 논블로킹 — 앱 시작 막지 않음
 - delay_between=0.5로 MLX 부하 완화

## 기대 효과

 - cold 요청 latency: 기존 Phase 1.3 그대로 (회귀 0)
 - warm 요청 + prewarmed: cache hit → query_analysis 활용
 - 예상 cache hit rate: 초기 70~80% (prewarm) + 사용 누적
 - Phase 2.2/2.3 multilingual/filter 기능은 cache hit 시에만 동작

## 참조

 - memory: feedback_analyzer_async_only.md (영구 룰 저장)
 - plan: ~/.claude/plans/zesty-painting-kahan.md ("철학 수정" 섹션)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-08 14:47:09 +09:00
Hyungi Ahn
a6c19ef76c feat: 뉴스 자동 수집 시스템 — 6개국 신문 RSS/API 
- news_sources 테이블 (소스 관리, UI 동적 제어)
- news_collector 워커: RSS(feedparser) + NYT API
  - 중복 체크: hash(title+date+source) + URL normalize
  - category 표준화, summary HTML 정제, timezone UTC
  - 30일 이내만 embed, source별 try/catch
- News API: 소스 CRUD + 수동 수집 트리거
- APScheduler: 6시간 간격 자동 수집
- 대상: 경향/아사히/NYT/르몽드/신화/슈피겔

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-06 13:38:07 +09:00
Hyungi Ahn
87683ca000 security: NAS 마운트 검증 + AI 서비스 포트 제한 + deploy 문서 갱신 
- NAS fail-fast: 시작 시 /documents/PKM 존재 확인, NFS 미마운트 방지
- ollama/ai-gateway 포트를 127.0.0.1로 제한 (외부 무인증 접근 차단)
- deploy.md: Caddy HTTPS 자동발급 → 앞단 프록시 HTTPS 종료 구조 반영

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-03 08:25:07 +09:00
Hyungi Ahn
cfa95ff031 feat: implement Phase 4 SvelteKit frontend + backend enhancements 
Backend:
- Add dashboard API (today stats, inbox count, law alerts, pipeline status)
- Add /api/documents/tree endpoint for sidebar domain/sub_group tree
- Migrate auth to HttpOnly cookie for refresh token (XSS defense)
- Add /api/auth/logout endpoint (cookie cleanup)
- Register dashboard router in main.py

Frontend (SvelteKit + Tailwind CSS v4):
- api.ts: fetch wrapper with refresh queue pattern, 401 single retry,
  forced logout on refresh failure
- Auth store: login/logout/refresh with memory-based access token
- UI store: toast system, sidebar state
- Login page with TOTP support
- Dashboard with 4 stat widgets + recent documents
- Document list with hybrid search (debounce, URL query state, mode select)
- Document detail with format-aware viewer (markdown/PDF/HWP/Synology/fallback)
- Metadata panel (AI summary, tags, processing history)
- Inbox triage UI (batch select, confirm dialog, domain override)
- Settings page (password change, TOTP status)

Infrastructure:
- Enable frontend service in docker-compose
- Caddy path routing (/api/* → fastapi, / → frontend) + gzip

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-03 06:46:19 +09:00
Hyungi Ahn
31d5498f8d feat: implement Phase 3 automation workers 
- Add automation_state table for incremental sync (last UID, last check)
- Add law_monitor worker: 국가법령정보센터 API → NAS/DB/CalDAV VTODO
  (LAW_OC 승인 대기 중, 코드 완성)
- Add mailplus_archive worker: IMAP(993) → .eml NAS save + DB + SMTP
  notification (imaplib via asyncio.to_thread, timeout=30)
- Add daily_digest worker: PostgreSQL/pipeline stats → Markdown + SMTP
  (documents, law changes, email, queue errors, inbox backlog)
- Add CalDAV VTODO helper and SMTP email helper to core/utils.py
- Wire 3 cron jobs in APScheduler (law@07:00, mail@07:00+18:00,
  digest@20:00) with timezone=Asia/Seoul

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 15:24:50 +09:00
Hyungi Ahn
4b695332b9 feat: implement Phase 2 core features 
- Add document CRUD API (list/get/upload/update/delete with auth)
  - Upload saves to Inbox + auto-enqueues processing pipeline
  - Delete defaults to DB-only, explicit flag for file deletion
- Add hybrid search API (FTS 0.4 + trigram 0.2 + vector 0.4 weighted)
  - Modes: fts, trgm, vector, hybrid (default)
  - Vector search gracefully degrades if GPU unavailable
- Add Inbox file watcher (5min interval, new file + hash change detection)
- Register documents/search routers and file_watcher scheduler in main.py
- Add IVFFLAT vector index migration (lists=50, with tuning guide)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 14:49:12 +09:00
Hyungi Ahn
299fac3904 feat: implement Phase 1 data pipeline and migration 
- Implement kordoc /parse endpoint (HWP/HWPX/PDF via kordoc lib,
  text files direct read, images flagged for OCR)
- Add queue consumer with APScheduler (1min interval, stage chaining
  extract→classify→embed, stale item recovery, retry logic)
- Add extract worker (kordoc HTTP call + direct text read)
- Add classify worker (Qwen3.5 AI classification with think-tag
  stripping and robust JSON extraction from AI responses)
- Add embed worker (GPU server nomic-embed-text, graceful failure)
- Add DEVONthink migration script with folder mapping for 16 DBs,
  dry-run mode, batch commits, and idempotent file_path UNIQUE
- Enhance ai/client.py with strip_thinking() and parse_json_response()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 14:35:36 +09:00
Hyungi Ahn
a601991f48 feat: implement Phase 0 auth system, setup wizard, and Docker config 
- Add users table to migration, User ORM model
- Implement JWT+TOTP auth API (login, refresh, me, change-password)
- Add first-run setup wizard with rate-limited admin creation,
  TOTP QR enrollment (secret saved only after verification), and
  NAS path verification — served as Jinja2 single-page HTML
- Add setup redirect middleware (bypasses /health, /docs, /openapi.json)
- Mount config.yaml, scripts, logs volumes in docker-compose
- Route API vs frontend traffic in Caddyfile
- Include admin seed script as CLI fallback

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 13:21:45 +09:00
Hyungi Ahn
131dbd7b7c feat: scaffold v2 project structure with Docker, FastAPI, and config 
동작하는 최소 코드 수준의 v2 스캐폴딩:

- docker-compose.yml: postgres, fastapi, kordoc, frontend, caddy
- app/: FastAPI 백엔드 (main, core, models, ai, prompts)
- services/kordoc/: Node.js 문서 파싱 마이크로서비스
- gpu-server/: AI Gateway + GPU docker-compose
- frontend/: SvelteKit 기본 구조
- migrations/: PostgreSQL 초기 스키마 (documents, tasks, processing_queue)
- tests/: pytest conftest 기본 설정
- config.yaml, Caddyfile, credentials.env.example 갱신

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 10:20:15 +09:00