feat(board): 처리 보드 v2 — 파이프라인 흐름 뷰·엔진 구분·실패 재시도/건너뛰기 (ds-board-engines-1)

- 흐름 뷰 메인: 좌→우 노드(머신·엔진 태그, 유입 우세 amber, 실패 뱃지) + 머신 스트립(모델 표기) + trend_24h 스파크라인 첫 렌더
- 노드 클릭 상세 패널: KV 4칸 + 다중 stage 행 + 지금 처리 중
- 실패 처리 드로어: 에러 패턴 그룹 + 재시도/건너뛰기 (영구 실패의 첫 사용자 조치 경로)
- API: stages[].done_1h/created_1h 노출 + GET /api/queue/failed + POST /api/queue/retry|/skip (uq_queue_active 충돌 skip, 건너뛰기는 enqueue_next_stage 미호출)
- 엔진/모델 표기 = queueDisplay.ts 정적 맵 단일 지점 (모델 교체 시 1곳)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
hyungi
2026-06-12 01:05:04 +00:00
parent 8ac1dbf4a8
commit 5581d3f1ce
7 changed files with 744 additions and 71 deletions
+94 -7
View File
@@ -1,20 +1,30 @@
"""처리 머신 보드 API — GET /api/queue/overview (plan ds-processing-ui-6an).
"""처리 머신 보드 API — /api/queue/* (plan ds-processing-ui-6an → ds-board-engines-1).
홈 stage 평면 테이블을 "머신 관점 보드(누가 일하나)"로 — 집계 로직은
services/queue_overview.py (순수 판정부 분리). 응답 스키마는 FE 와 계약 고정.
응답에 raw 모델명 노출 금지 — 머신 label 만.
- GET /overview: 홈 stage 평면 테이블을 "머신 관점 보드(누가 일하나)"로 — 집계
로직은 services/queue_overview.py (순수 판정부 분리). 응답 스키마는 FE 와
계약 고정. 응답에 raw 모델명 노출 금지 — 머신 label 만 (엔진/모델 표기는
FE 정적 맵 책임).
- GET /failed + POST /retry|/skip: 실패 처리 (ds-board-engines-1) — 영구 실패
(자동 재시도 3회 소진)의 유일한 사용자 조치 경로. 일괄 조치는 FE 가 그룹의
id 목록을 모아 보낸다 (서버측 패턴 매칭 없음 — raw 식별자/패턴 미수신).
"""
from datetime import datetime
from typing import Annotated, Literal
from fastapi import APIRouter, Depends
from pydantic import BaseModel
from pydantic import BaseModel, Field
from sqlalchemy.ext.asyncio import AsyncSession
from core.auth import get_current_user
from core.database import get_session
from models.user import User
from services.queue_overview import build_overview
from services.queue_overview import (
build_overview,
fetch_failed_items,
retry_failed,
skip_failed,
)
router = APIRouter()
@@ -64,11 +74,17 @@ class Totals(BaseModel):
class StageRow(BaseModel):
"""단계별 현황 행 — '단계 상세' 패널용 (완료 가시화)."""
"""단계별 현황 행 — 흐름 노드/상세 패널용.
done_1h/created_1h = 처리율·유입률 (유입 우세 판정 + ETA 의 FE 재료,
ds-board-engines-1 추가 — 수집 SQL 에 이미 있던 값의 노출).
"""
stage: str
pending: int
processing: int
failed: int
done_1h: int
created_1h: int
done_today: int
oldest_pending_age_sec: int | None
@@ -81,6 +97,40 @@ class QueueOverviewResponse(BaseModel):
totals: Totals
class FailedItem(BaseModel):
"""영구 실패 행 — 실패 드로어 표시 단위."""
id: int
stage: str
document_id: int
title: str
attempts: int
max_attempts: int
error_message: str | None
failed_at: datetime | None
class FailedListResponse(BaseModel):
items: list[FailedItem]
total: int
class QueueActionRequest(BaseModel):
"""재시도/건너뛰기 대상 — 실패 행 id 목록 (FE 가 그룹핑 후 전달)."""
ids: list[int] = Field(min_length=1, max_length=300)
class RetryResponse(BaseModel):
requested: int
retried: int
not_retried: int
class SkipResponse(BaseModel):
requested: int
skipped: int
not_skipped: int
@router.get("/overview", response_model=QueueOverviewResponse)
async def get_queue_overview(
user: Annotated[User, Depends(get_current_user)],
@@ -88,3 +138,40 @@ async def get_queue_overview(
):
"""머신 관점 처리 보드 + summarize ETA 집계 (라이브 계산, 신규 테이블 0)"""
return QueueOverviewResponse.model_validate(await build_overview(session))
@router.get("/failed", response_model=FailedListResponse)
async def get_failed_items(
user: Annotated[User, Depends(get_current_user)],
session: Annotated[AsyncSession, Depends(get_session)],
):
"""영구 실패 행 목록 (문서 제목 포함, 최대 300건)"""
items = await fetch_failed_items(session)
return FailedListResponse(
items=[FailedItem.model_validate(i) for i in items],
total=len(items),
)
@router.post("/retry", response_model=RetryResponse)
async def retry_failed_items(
body: QueueActionRequest,
user: Annotated[User, Depends(get_current_user)],
session: Annotated[AsyncSession, Depends(get_session)],
):
"""실패 행 재시도 — attempts 리셋 + pending 복귀.
not_retried = 같은 (문서, 단계) 의 active 행 충돌(uq_queue_active) 또는
이미 failed 가 아닌 행 (중복 클릭 등) — 건드리지 않고 건수만 보고.
"""
return RetryResponse.model_validate(await retry_failed(session, body.ids))
@router.post("/skip", response_model=SkipResponse)
async def skip_failed_items(
body: QueueActionRequest,
user: Annotated[User, Depends(get_current_user)],
session: Annotated[AsyncSession, Depends(get_session)],
):
"""실패 행 건너뛰기 — completed 마킹(payload.skipped_by_user) + 연쇄 없음"""
return SkipResponse.model_validate(await skip_failed(session, body.ids))
+104 -1
View File
@@ -22,7 +22,7 @@ from datetime import datetime, timedelta
from posixpath import basename
from zoneinfo import ZoneInfo
from sqlalchemy import text
from sqlalchemy import bindparam, text
from sqlalchemy.ext.asyncio import AsyncSession
from core.config import settings
@@ -258,6 +258,8 @@ def build_stages(stage_stats: dict[str, dict], now=None) -> list[dict]:
"pending": st["pending"],
"processing": st["processing"],
"failed": st["failed"],
"done_1h": st["done_1h"],
"created_1h": st["created_1h"],
"done_today": st["done_today"],
"oldest_pending_age_sec": age,
})
@@ -408,3 +410,104 @@ async def build_overview(session: AsyncSession) -> dict:
deep_enabled=deep_enabled,
now_kst=now_kst,
)
# ─── 실패 처리 (plan ds-board-engines-1) ─────────────────────────────────────
# 실패 = 자동 재시도(max_attempts=3) 소진 후 영구 정지 상태. 여기 함수들은
# 사용자 명시 조치 전용 — 자동 호출 경로 없음 (보드 실패 드로어가 유일 호출자).
# 실패 행은 completed_at 이 비어 있을 수 있어(소비자 실패 경로가 미기록)
# started_at 을 시각 fallback 으로 쓴다.
_FAILED_LIST_SQL = """
SELECT q.id, q.stage, q.document_id, q.attempts, q.max_attempts,
q.error_message,
COALESCE(q.completed_at, q.started_at) AS failed_at,
d.title, d.original_filename, d.file_path
FROM processing_queue q
JOIN documents d ON d.id = q.document_id
WHERE q.status = 'failed'
ORDER BY q.stage, COALESCE(q.completed_at, q.started_at) DESC NULLS LAST
LIMIT 300
"""
# 재시도: failed → pending (attempts 리셋 = 자동 재시도 3회 새로 부여).
# error_message 는 감사용으로 보존 — 성공 시 완료 행에 남아도 무해.
# uq_queue_active((doc,stage) pending/processing 부분 유니크)와 충돌하는 행 —
# 같은 문서·단계가 이미 재enqueue 된 경우 — 는 건드리지 않고 건수만 보고.
_RETRY_SQL = """
UPDATE processing_queue q
SET status = 'pending', attempts = 0,
started_at = NULL, completed_at = NULL
WHERE q.id IN :ids
AND q.status = 'failed'
AND NOT EXISTS (
SELECT 1 FROM processing_queue p
WHERE p.document_id = q.document_id
AND p.stage = q.stage
AND p.status IN ('pending', 'processing')
AND p.id <> q.id
)
RETURNING q.id
"""
# 건너뛰기: failed → completed + payload 마킹 (감사 추적).
# enqueue_next_stage 는 의도적으로 호출하지 않는다 — 실패 문서(빈 텍스트 등)가
# 하류 단계로 흘러가는 것 방지. 후속 단계가 필요하면 재시도가 정상 경로.
_SKIP_SQL = """
UPDATE processing_queue
SET status = 'completed', completed_at = NOW(),
payload = COALESCE(payload, '{}'::jsonb)
|| jsonb_build_object('skipped_by_user', true,
'skipped_at', NOW()::text)
WHERE id IN :ids AND status = 'failed'
RETURNING id
"""
async def fetch_failed_items(session: AsyncSession) -> list[dict]:
"""영구 실패 행 목록 (문서 제목 포함, 최대 300건)."""
rows = (await session.execute(text(_FAILED_LIST_SQL))).all()
return [
{
"id": r[0],
"stage": r[1],
"document_id": r[2],
"attempts": int(r[3] or 0),
"max_attempts": int(r[4] or 0),
"error_message": r[5],
"failed_at": r[6],
"title": display_title({
"document_id": r[2],
"title": r[7],
"original_filename": r[8],
"file_path": r[9],
}),
}
for r in rows
]
async def retry_failed(session: AsyncSession, ids: list[int]) -> dict:
"""failed → pending 복귀. not_retried = active 충돌 + 이미 failed 아님."""
unique_ids = list(set(ids))
stmt = text(_RETRY_SQL).bindparams(bindparam("ids", expanding=True))
retried = (await session.execute(stmt, {"ids": unique_ids})).all()
await session.commit()
return {
"requested": len(unique_ids),
"retried": len(retried),
"not_retried": len(unique_ids) - len(retried),
}
async def skip_failed(session: AsyncSession, ids: list[int]) -> dict:
"""failed → completed(건너뛰기 마킹). 후속 단계 연쇄 없음."""
unique_ids = list(set(ids))
stmt = text(_SKIP_SQL).bindparams(bindparam("ids", expanding=True))
skipped = (await session.execute(stmt, {"ids": unique_ids})).all()
await session.commit()
return {
"requested": len(unique_ids),
"skipped": len(skipped),
"not_skipped": len(unique_ids) - len(skipped),
}
@@ -0,0 +1,419 @@
<script lang="ts">
// 처리 머신 보드 v2 — 파이프라인 흐름 뷰 (plan ds-board-engines-1, R2 통합안).
// 메인 = 좌→우 흐름 노드(병목 amber·실패 뱃지), 노드 클릭 = 상세 패널(안1 변형),
// 실패 뱃지 클릭 = 실패 처리 드로어 (재시도/건너뛰기 — 영구 실패의 유일한 조치 경로).
// 데이터 = GET /api/queue/overview (60s 폴링 store) + GET /api/queue/failed (드로어 열 때).
import { api } from '$lib/api';
import { refreshQueueOverview } from '$lib/stores/queueOverview';
import { addToast } from '$lib/stores/toast';
import {
AUX_NODES,
FLOW_NODES,
MACHINE_META,
type FlowNodeDef,
etaShort,
flowStageLabel,
formatAgeSec,
formatRate,
} from '$lib/utils/queueDisplay';
import type {
FailedItem,
FailedListResponse,
MachineCurrentItem,
QueueOverview,
QueueStageRow,
RetryResponse,
SkipResponse,
} from '$lib/types/queue';
let { overview }: { overview: QueueOverview } = $props();
// ─── 노드 통계 합성 ───
interface NodeStats {
def: FlowNodeDef;
/** 다중 stage 노드(청크·임베딩)는 같은 문서가 양쪽 큐에 있어 max — 합산 = 이중계산 */
pending: number;
processing: number;
failed: number; // 실패는 행 단위 사실이라 합산
done1h: number;
created1h: number;
doneToday: number;
oldestAgeSec: number | null;
etaMinutes: number | null;
inflowDominant: boolean;
perStage: QueueStageRow[];
}
const stageBy = $derived(new Map(overview.stages.map((s) => [s.stage, s])));
function nodeStats(def: FlowNodeDef): NodeStats {
const rows = def.stages
.map((s) => stageBy.get(s))
.filter((r): r is QueueStageRow => r != null);
const pending = rows.reduce((m, r) => Math.max(m, r.pending), 0);
const done1h = rows.reduce((m, r) => Math.max(m, r.done_1h), 0);
const created1h = rows.reduce((m, r) => Math.max(m, r.created_1h), 0);
const oldest = rows.reduce<number | null>(
(m, r) => (r.oldest_pending_age_sec == null ? m : Math.max(m ?? 0, r.oldest_pending_age_sec)),
null,
);
return {
def,
pending,
processing: rows.reduce((s, r) => s + r.processing, 0),
failed: rows.reduce((s, r) => s + r.failed, 0),
done1h,
created1h,
doneToday: rows.reduce((m, r) => Math.max(m, r.done_today), 0),
oldestAgeSec: oldest,
etaMinutes: pending > 0 && done1h > 0 ? Math.round((pending / done1h) * 60) : null,
inflowDominant: pending > 0 && created1h > done1h,
perStage: rows,
};
}
const mainNodes = $derived(FLOW_NODES.map(nodeStats));
const auxAll = $derived(AUX_NODES.map(nodeStats));
const auxActive = $derived(
auxAll.filter((n) => n.pending + n.processing + n.failed + n.doneToday > 0),
);
const auxIdle = $derived(
auxAll.filter((n) => n.pending + n.processing + n.failed + n.doneToday === 0),
);
const totalFailed = $derived(overview.totals.failed);
// 머신 스트립 — overview.machines 의 state/처리율 + 정적 모델 메타
const machineStrip = $derived(
overview.machines.map((m) => ({
...m,
meta: MACHINE_META[m.key],
})),
);
// ─── 선택 상태 (노드 상세 / 실패 드로어 — 동시에 하나만) ───
let selected = $state<string | null>(null);
let failOpen = $state(false);
function toggleNode(key: string) {
selected = selected === key ? null : key;
if (selected) failOpen = false;
}
const selectedNode = $derived(
[...mainNodes, ...auxAll].find((n) => n.def.key === selected) ?? null,
);
function nodeCurrent(def: FlowNodeDef): MachineCurrentItem[] {
return overview.machines.flatMap((m) => m.current.filter((c) => def.stages.includes(c.stage)));
}
// ─── 실패 드로어 ───
let failItems = $state<FailedItem[]>([]);
let failLoading = $state(false);
let busy = $state(false);
let expanded = $state<Record<string, boolean>>({});
async function openFailures() {
failOpen = true;
selected = null;
await loadFailures();
}
async function loadFailures() {
failLoading = true;
try {
const r = await api<FailedListResponse>('/queue/failed');
failItems = r.items;
} catch {
addToast('error', '실패 목록을 불러오지 못했습니다');
} finally {
failLoading = false;
}
}
interface FailGroup {
key: string;
stage: string;
pattern: string;
items: FailedItem[];
}
// 그룹핑 = stage + 에러 메시지 prefix(36자) — 같은 원인(ReadTimeout 등) 묶음
const failGroups = $derived.by(() => {
const map = new Map<string, FailGroup>();
for (const it of failItems) {
const pattern = (it.error_message ?? '(메시지 없음)').slice(0, 36);
const key = `${it.stage}::${pattern}`;
const g = map.get(key);
if (g) g.items.push(it);
else map.set(key, { key, stage: it.stage, pattern, items: [it] });
}
return [...map.values()].sort(
(a, b) => a.stage.localeCompare(b.stage) || b.items.length - a.items.length,
);
});
async function retryIds(ids: number[]) {
if (busy || ids.length === 0) return;
busy = true;
try {
const r = await api<RetryResponse>('/queue/retry', {
method: 'POST',
body: JSON.stringify({ ids }),
});
addToast(
'success',
`재시도 ${r.retried}건 큐 재진입${r.not_retried > 0 ? ` (${r.not_retried}건 제외 — 이미 활성/처리됨)` : ''}`,
);
await afterAction();
} catch {
addToast('error', '재시도 요청 실패');
} finally {
busy = false;
}
}
async function skipIds(ids: number[]) {
if (busy || ids.length === 0) return;
busy = true;
try {
const r = await api<SkipResponse>('/queue/skip', {
method: 'POST',
body: JSON.stringify({ ids }),
});
addToast('success', `건너뛰기 ${r.skipped}건 처리 (해당 단계 제외)`);
await afterAction();
} catch {
addToast('error', '건너뛰기 요청 실패');
} finally {
busy = false;
}
}
async function afterAction() {
await Promise.all([loadFailures(), refreshQueueOverview()]);
}
// ─── trend_24h 스파크라인 (summarize 유입 vs 소화 — API 가 주는데 미렌더이던 슬롯) ───
const spark = $derived.by(() => {
const t = overview.trend_24h;
if (!t || t.length === 0) return null;
const max = Math.max(1, ...t.map((b) => Math.max(b.inflow, b.done)));
const w = 120;
const h = 24;
const step = w / Math.max(1, t.length - 1);
const pts = (sel: (b: (typeof t)[number]) => number) =>
t.map((b, i) => `${(i * step).toFixed(1)},${(h - (sel(b) / max) * (h - 3) + 1).toFixed(1)}`).join(' ');
return { inflow: pts((b) => b.inflow), done: pts((b) => b.done) };
});
</script>
<div class="mt-5">
<!-- 헤더: 타이틀 + 요약 24h 스파크라인 + 실패 합계 -->
<div class="flex items-center justify-between gap-3 mb-3">
<div class="text-[11px] font-bold text-dim uppercase tracking-wider">처리 머신</div>
<div class="flex items-center gap-3">
{#if totalFailed > 0}
<button
class="text-[11px] font-semibold text-error hover:underline cursor-pointer"
onclick={openFailures}
>실패 {totalFailed} 처리</button>
{/if}
{#if spark}
<div class="flex items-center gap-2 text-[10px] text-faint tabular-nums" title="요약(summarize) 단계 24시간 — 유입(회색) vs 소화(녹색)">
<svg width="120" height="24" viewBox="0 0 120 24" class="block">
<polyline points={spark.inflow} fill="none" stroke="currentColor" stroke-width="1.5" class="text-faint" />
<polyline points={spark.done} fill="none" stroke="currentColor" stroke-width="1.5" class="text-success" />
</svg>
<span>요약 24h 유입/소화</span>
</div>
{/if}
</div>
</div>
<!-- 머신 스트립 -->
<div class="flex flex-wrap gap-2 mb-3">
{#each machineStrip as m (m.key)}
<div class="flex items-center gap-2 bg-surface border border-default rounded-full px-3.5 py-1.5 text-xs">
<span class="w-2 h-2 rounded-full shrink-0 {m.state === 'active' ? 'bg-success' : m.state === 'deferred' ? 'bg-warning' : 'bg-faint'}"></span>
<span class="font-bold text-text">{m.meta?.label ?? m.label}</span>
<span class="text-[10px] text-faint font-mono">{m.meta?.model}</span>
<span class="text-[11px] text-dim tabular-nums">{formatRate(m.done_1h)}/h</span>
{#if m.key === 'macbook' && m.deferred_pending > 0}
<span class="text-[10px] font-semibold text-warning tabular-nums">보류 {m.deferred_pending}</span>
{/if}
</div>
{/each}
</div>
<!-- 흐름 노드 -->
<div class="flex items-stretch overflow-x-auto pb-1">
{#each mainNodes as n, i (n.def.key)}
{#if i > 0}
<div class="flex items-center text-faint text-sm px-1.5 shrink-0" aria-hidden="true"></div>
{/if}
<div
class="relative bg-surface border-[1.5px] rounded-card px-3 py-2.5 min-w-[124px] shrink-0 text-left transition-colors cursor-pointer hover:bg-surface-hover
{n.inflowDominant ? 'border-warning' : n.etaMinutes != null && n.def.stages.includes('chunk') ? 'border-success' : 'border-default'}
{selected === n.def.key ? 'node-sel' : ''}"
role="button"
tabindex="0"
onclick={() => toggleNode(n.def.key)}
onkeydown={(e) => { if (e.key === 'Enter' || e.key === ' ') { e.preventDefault(); toggleNode(n.def.key); } }}
title="{n.def.label} — 클릭하면 상세"
>
{#if n.failed > 0}
<button
class="absolute -top-2 -right-1.5 text-[9px] font-extrabold bg-error text-white rounded-full px-1.5 py-px shadow cursor-pointer"
onclick={(e) => { e.stopPropagation(); openFailures(); }}
title="실패 {n.failed} 클릭하면 실패 처리"
>{n.failed}</button>
{/if}
<span class="inline-block text-[9px] font-bold rounded px-1.5 py-px mb-1.5 mtag-{n.def.machine}">
{MACHINE_META[n.def.machine].label} · {n.def.engine}
</span>
<div class="text-xs font-bold text-text flex items-center gap-1.5">
{n.def.label}
{#if n.processing > 0}
<span class="inline-block w-1.5 h-1.5 rounded-full bg-accent animate-pulse" title="처리 중 {n.processing}"></span>
{/if}
{#if n.inflowDominant}
<span class="text-[9px] font-bold text-warning">유입 우세</span>
{/if}
</div>
<div class="text-base font-extrabold tabular-nums tracking-tight leading-tight mt-0.5 text-text">
{n.pending.toLocaleString()}
</div>
<div class="text-[10px] text-dim tabular-nums">
{formatRate(n.done1h)}/h · 오늘 {n.doneToday.toLocaleString()}
{#if n.etaMinutes != null && !n.inflowDominant && n.pending > 0}
· <span class="text-accent font-semibold">{etaShort(n.etaMinutes)}</span>
{/if}
</div>
</div>
{/each}
</div>
<!-- 보조 라인 -->
<p class="text-[10px] text-faint mt-1.5 tabular-nums">
{#each auxActive as n, i (n.def.key)}
{i > 0 ? ' · ' : '보조: '}{n.def.label}({n.def.engine}) 대기 {n.pending.toLocaleString()} · {formatRate(n.done1h)}/h{n.failed > 0 ? ` · 실패 ${n.failed}` : ''}
{/each}
{#if auxIdle.length > 0}
{auxActive.length > 0 ? ' — ' : ''}한가: {auxIdle.map((n) => n.def.label).join(' · ')}
{/if}
— 뉴스 등 일부 소스는 분류/추출을 건너뜀 (흐름 그림은 대표 경로)
</p>
<!-- 상세 패널 (노드 클릭) -->
{#if selectedNode}
<div class="border rounded-card mt-3 overflow-hidden bg-surface detail-frame">
<div class="flex items-center gap-2.5 px-4 py-2.5 text-xs font-bold detail-head">
{selectedNode.def.label}{selectedNode.def.engine}
<span class="text-[10px] font-mono font-medium text-dim bg-surface border border-default rounded px-1.5">{selectedNode.def.sub} · {MACHINE_META[selectedNode.def.machine].label}</span>
<button class="ml-auto text-[11px] text-dim font-normal cursor-pointer hover:text-text" onclick={() => (selected = null)}>닫기</button>
</div>
<div class="px-4 pb-3.5">
<div class="grid grid-cols-2 md:grid-cols-4 gap-2.5 my-2.5">
<div class="bg-bg border border-default rounded-card px-3 py-2">
<div class="text-[9px] text-faint uppercase tracking-wide">대기</div>
<div class="text-lg font-extrabold tabular-nums text-text">{selectedNode.pending.toLocaleString()}</div>
</div>
<div class="bg-bg border border-default rounded-card px-3 py-2">
<div class="text-[9px] text-faint uppercase tracking-wide">처리율 (1h)</div>
<div class="text-lg font-extrabold tabular-nums text-text">{formatRate(selectedNode.done1h)}<span class="text-[11px] text-dim font-semibold">/h</span></div>
</div>
<div class="bg-bg border border-default rounded-card px-3 py-2">
<div class="text-[9px] text-faint uppercase tracking-wide">오늘 완료</div>
<div class="text-lg font-extrabold tabular-nums text-text">{selectedNode.doneToday.toLocaleString()}</div>
</div>
<div class="bg-bg border border-default rounded-card px-3 py-2">
<div class="text-[9px] text-faint uppercase tracking-wide">소진 예상</div>
<div class="text-lg font-extrabold tabular-nums {selectedNode.inflowDominant ? 'text-warning' : 'text-accent'}">
{#if selectedNode.inflowDominant}유입 우세{:else if selectedNode.etaMinutes != null}{etaShort(selectedNode.etaMinutes)}{:else if selectedNode.pending === 0}한가{:else}{/if}
</div>
</div>
</div>
{#if selectedNode.perStage.length > 1}
{#each selectedNode.perStage as row (row.stage)}
<div class="flex items-center gap-2.5 py-1.5 border-t border-default text-xs">
<span class="font-semibold text-text min-w-[72px]">{flowStageLabel(row.stage)}</span>
<span class="ml-auto text-dim tabular-nums">
대기 <strong class="text-text">{row.pending.toLocaleString()}</strong>
· {formatRate(row.done_1h)}/h · 오늘 {row.done_today.toLocaleString()}
{#if row.failed > 0}· <span class="text-error font-semibold">실패 {row.failed}</span>{/if}
</span>
</div>
{/each}
{/if}
<div class="text-[11px] text-dim border-t border-dashed border-default mt-2 pt-2 tabular-nums">
{#if selectedNode.oldestAgeSec != null && selectedNode.oldestAgeSec > 600}
가장 오래 기다린 항목 {formatAgeSec(selectedNode.oldestAgeSec)}
{/if}
{#each nodeCurrent(selectedNode.def) as c, i (c.document_id + c.stage)}
{i === 0 && !(selectedNode.oldestAgeSec != null && selectedNode.oldestAgeSec > 600) ? '' : ' · '}지금: {c.title} ({flowStageLabel(c.stage)})
{/each}
{#if selectedNode.failed > 0}
· <button class="text-error font-semibold cursor-pointer hover:underline" onclick={openFailures}>실패 {selectedNode.failed} 처리</button>
{/if}
</div>
</div>
</div>
{/if}
<!-- 실패 처리 드로어 -->
{#if failOpen}
<div class="border border-error/40 rounded-card mt-3 overflow-hidden bg-surface">
<div class="flex items-center gap-2.5 px-4 py-2.5 bg-error/5 text-xs font-bold text-text">
실패 처리
<span class="text-[10px] font-semibold text-error">영구 실패 {failItems.length}건 — 자동 재시도 3회 소진, 수동 조치 대기</span>
<button class="ml-auto text-[11px] text-dim font-normal cursor-pointer hover:text-text" onclick={() => (failOpen = false)}>닫기</button>
</div>
{#if failLoading}
<p class="text-xs text-dim text-center py-4">불러오는 중…</p>
{:else if failItems.length === 0}
<p class="text-xs text-dim text-center py-4">영구 실패 항목 없음</p>
{:else}
{#each failGroups as g (g.key)}
<div class="px-4 py-2.5 border-t border-default">
<div class="flex items-center gap-2 flex-wrap text-xs font-bold text-text mb-1">
{flowStageLabel(g.stage)} {g.items.length}
<span class="text-[10px] font-mono font-medium text-error bg-error/10 rounded px-1.5 py-px">{g.pattern}{g.items[0]?.error_message && g.items[0].error_message.length > 36 ? '…' : ''}</span>
</div>
{#each expanded[g.key] ? g.items : g.items.slice(0, 4) as it (it.id)}
<div class="flex items-center gap-2.5 py-1 border-t border-dashed border-default/60 text-xs">
<span class="flex-1 min-w-0 truncate text-text" title={it.title}>{it.title}</span>
<span class="text-[10px] font-mono text-faint shrink-0 tabular-nums">시도 {it.attempts}/{it.max_attempts}</span>
<span class="text-[10px] font-mono text-error shrink-0 max-w-[260px] truncate" title={it.error_message ?? ''}>{it.error_message ?? ''}</span>
<button class="text-[10px] font-bold border border-accent text-accent rounded px-2 py-0.5 shrink-0 cursor-pointer hover:bg-accent/10 disabled:opacity-40" disabled={busy} onclick={() => retryIds([it.id])}>재시도</button>
<button class="text-[10px] font-bold border border-default text-faint rounded px-2 py-0.5 shrink-0 cursor-pointer hover:bg-surface-hover disabled:opacity-40" disabled={busy} onclick={() => skipIds([it.id])}>건너뛰기</button>
</div>
{/each}
{#if g.items.length > 4 && !expanded[g.key]}
<button class="text-[10px] text-dim cursor-pointer hover:text-text mt-1" onclick={() => (expanded = { ...expanded, [g.key]: true })}> {g.items.length - 4} 펼치기</button>
{/if}
{#if g.items.length > 1}
<div class="flex gap-2 mt-1.5">
<button class="text-[10px] font-bold border border-accent text-accent rounded px-2.5 py-0.5 cursor-pointer hover:bg-accent/10 disabled:opacity-40" disabled={busy} onclick={() => retryIds(g.items.map((x) => x.id))}>그룹 전체 재시도 ({g.items.length})</button>
<button class="text-[10px] font-bold border border-default text-faint rounded px-2.5 py-0.5 cursor-pointer hover:bg-surface-hover disabled:opacity-40" disabled={busy} onclick={() => skipIds(g.items.map((x) => x.id))}>그룹 전체 건너뛰기</button>
</div>
{/if}
</div>
{/each}
<p class="text-[10px] text-faint px-4 py-2 border-t border-default">
재시도 = 시도 횟수 리셋 후 큐 재진입 (자동 재시도 3회 새로 부여) · 건너뛰기 = 이 단계 완료 처리(후속 단계 연쇄 없음, 감사 마킹) · 같은 오류가 반복되는 항목(빈 텍스트 등)은 건너뛰기 권장
</p>
{/if}
</div>
{/if}
</div>
<style>
/* 머신 색 — 디자인 토큰 외 3색 (gpu 청/macmini 보라/macbook 황) — 이 컴포넌트 한정 */
.mtag-gpu { background: #e7eef6; color: #3b6ea5; }
.mtag-macmini { background: #efe9f7; color: #8a5fbf; }
.mtag-macbook { background: #f7eedd; color: #b07a10; }
.node-sel { outline: 2px solid #3b6ea5; outline-offset: 1px; }
.detail-frame { border-color: #3b6ea5; }
.detail-head { background: #e7eef6; }
</style>
+34
View File
@@ -61,6 +61,10 @@ export interface QueueStageRow {
pending: number;
processing: number;
failed: number;
/** 최근 1시간 완료 — 노드 처리율·ETA 재료 (ds-board-engines-1) */
done_1h: number;
/** 최근 1시간 유입 — 유입 우세 판정 재료 (ds-board-engines-1) */
created_1h: number;
done_today: number;
oldest_pending_age_sec: number | null;
}
@@ -72,3 +76,33 @@ export interface QueueOverview {
stages: QueueStageRow[];
totals: QueueTotals;
}
/** ─── 실패 처리 (ds-board-engines-1) — GET /api/queue/failed · POST /retry|/skip ─── */
export interface FailedItem {
id: number;
stage: string;
document_id: number;
title: string;
attempts: number;
max_attempts: number;
error_message: string | null;
failed_at: string | null;
}
export interface FailedListResponse {
items: FailedItem[];
total: number;
}
export interface RetryResponse {
requested: number;
retried: number;
not_retried: number;
}
export interface SkipResponse {
requested: number;
skipped: number;
not_skipped: number;
}
+79
View File
@@ -36,3 +36,82 @@ export function etaPhrase(minutes: number): string {
const text = hours >= 10 ? String(Math.round(hours)) : String(Math.round(hours * 10) / 10);
return `${text}시간 후 소진 예상`;
}
/** ETA 분 → 칩용 짧은 표기 ("약 4.6시간" / "약 12분") */
export function etaShort(minutes: number): string {
if (minutes < 60) return `${Math.max(1, Math.round(minutes))}`;
const hours = minutes / 60;
const text = hours >= 10 ? String(Math.round(hours)) : String(Math.round(hours * 10) / 10);
return `${text}시간`;
}
/** 경과 초 → "N분 전 / N시간 전 / N일 전" */
export function formatAgeSec(sec: number): string {
if (sec < 3600) return `${Math.max(1, Math.round(sec / 60))}분 전`;
if (sec < 86400) return `${Math.round(sec / 3600)}시간 전`;
return `${Math.round(sec / 86400)}일 전`;
}
/* (plan ds-board-engines-1)
* stage / () / . API label
* (raw ), · .
* / 1 (: 맥미니 ).
*/
export type FlowMachine = 'gpu' | 'macmini' | 'macbook';
export interface FlowNodeDef {
key: string;
/** 노드 표시명 */
label: string;
/** 합산할 stage 키 (다중 = 같은 엔진 공유) */
stages: string[];
machine: FlowMachine;
/** 엔진/모델 표시명 (FE 정적 — 모델 교체 시 여기 수정) */
engine: string;
/** 보조 표기 (서비스/워커명) */
sub: string;
}
/** 메인 흐름 (문서 진행 순서). 뉴스 등 소스별 스킵 경로는 그림에 안 그림 — 단순화 한계. */
export const FLOW_NODES: FlowNodeDef[] = [
{ key: 'extract', label: '추출', stages: ['extract'], machine: 'gpu', engine: 'Surya OCR', sub: 'ocr-service' },
{ key: 'markdown', label: '마크다운', stages: ['markdown'], machine: 'gpu', engine: 'Marker', sub: 'marker-service' },
{ key: 'classify', label: '분류', stages: ['classify'], machine: 'macmini', engine: 'Qwen3.6-27B', sub: 'classify + triage' },
{ key: 'summarize', label: '요약', stages: ['summarize'], machine: 'macmini', engine: 'Qwen3.6-27B', sub: 'summarize' },
{ key: 'chunkembed', label: '청크 · 임베딩', stages: ['chunk', 'embed'], machine: 'gpu', engine: 'TEI bge-m3', sub: 'text-embeddings-inference' },
{ key: 'deep', label: '심층분석', stages: ['deep_summary'], machine: 'macbook', engine: 'Qwen3.6-27B', sub: 'deep_summary' },
];
/** 보조 노드 — 메인 흐름 밖 (활동 있을 때만 보조 라인에 표시) */
export const AUX_NODES: FlowNodeDef[] = [
{ key: 'fulltext', label: '전문 수집', stages: ['fulltext'], machine: 'gpu', engine: 'Playwright', sub: 'playwright-fetcher' },
{ key: 'stt', label: '전사', stages: ['stt'], machine: 'gpu', engine: 'Whisper', sub: 'stt-service' },
{ key: 'util', label: '미리보기 · 썸네일', stages: ['preview', 'thumbnail'], machine: 'gpu', engine: '유틸', sub: 'ffmpeg' },
];
/** 머신 스트립 메타 — 모델 표기 단일 지점 */
export const MACHINE_META: Record<FlowMachine, { label: string; model: string }> = {
gpu: { label: 'GPU 서버', model: '특화 엔진' },
macmini: { label: '맥미니', model: 'Qwen3.6-27B-6bit · 24/7' },
macbook: { label: '맥북 M5 Max', model: 'Qwen3.6-27B · 야간 drain' },
};
/** 흐름 보드 단계 라벨 (드로어/상세 행 표기) */
export const FLOW_STAGE_LABEL: Record<string, string> = {
extract: '추출',
classify: '분류',
summarize: '요약',
embed: '임베딩',
chunk: '청크',
preview: '미리보기',
stt: '전사',
thumbnail: '썸네일',
deep_summary: '심층분석',
markdown: '마크다운',
fulltext: '전문',
};
export function flowStageLabel(stage: string): string {
return FLOW_STAGE_LABEL[stage] ?? stage;
}
+3 -61
View File
@@ -14,9 +14,7 @@
import { user } from '$lib/stores/auth';
import { api } from '$lib/api';
import { queueOverview, refreshQueueOverview } from '$lib/stores/queueOverview';
import {
MACHINE_STATE_LABEL, machineChipClass, machineDotClass, formatRate, etaPhrase,
} from '$lib/utils/queueDisplay';
import ProcessingFlowBoard from '$lib/components/ProcessingFlowBoard.svelte';
import type { QueueOverview } from '$lib/types/queue';
import EmptyState from '$lib/components/ui/EmptyState.svelte';
import Skeleton from '$lib/components/ui/Skeleton.svelte';
@@ -460,65 +458,9 @@
</div>
</div>
<!-- ═══ 처리 머신 보드 (안2) + ETA 라인 (안5 라이트) ═══ -->
<!-- ═══ 처리 머신 보드 v2 — 파이프라인 흐름 + 상세 패널 + 실패 드로어 (ds-board-engines-1) ═══ -->
{#if queue}
<div class="mt-5">
<div class="text-[11px] font-bold text-dim uppercase tracking-wider mb-3">처리 머신</div>
<div class="grid grid-cols-1 md:grid-cols-3 gap-3">
{#each queue.machines as m (m.key)}
<div class="bg-surface border border-default rounded-card p-4">
<!-- 헤더: 상태 dot + 라벨 + state 칩 -->
<div class="flex items-center justify-between gap-2 mb-2">
<span class="flex items-center gap-2 text-[13px] font-bold text-text min-w-0">
<span class="w-2 h-2 rounded-full shrink-0 {machineDotClass(m.state)}"></span>
<span class="truncate">{m.label}</span>
</span>
<span class="text-[10px] font-bold rounded-full px-2 py-0.5 shrink-0 {machineChipClass(m.state)}">
{MACHINE_STATE_LABEL[m.state]}
</span>
</div>
<!-- 담당 단계 칩 -->
{#if m.stages.length > 0}
<div class="flex flex-wrap gap-1 mb-2.5">
{#each m.stages as s (s)}
<span class="text-[10px] font-semibold rounded-full px-2 py-0.5 bg-surface-hover text-dim">{queueStageLabel(s)}</span>
{/each}
</div>
{/if}
<!-- 대기 · 처리율 · 오늘 -->
<div class="text-xs text-dim tabular-nums">
대기 <strong class="text-text">{m.pending.toLocaleString()}</strong>
· 처리율 <strong class="text-text">{formatRate(m.done_1h)}/h</strong>
· 오늘 <strong class="text-text">{m.done_today.toLocaleString()}</strong>
</div>
<!-- 맥북 보류 (sleep 등 자동 재개 대기) -->
{#if m.key === 'macbook' && m.deferred_pending > 0}
<div class="text-[11px] font-semibold text-warning mt-1.5 tabular-nums">보류 {m.deferred_pending.toLocaleString()}건 — 자동 재개 대기</div>
{/if}
<!-- 지금 처리 중인 문서 -->
{#if m.current.length > 0}
<div class="text-[11px] text-dim border-t border-dashed border-default mt-2.5 pt-2 truncate"
title={m.current.map((c) => `${c.title} (${queueStageLabel(c.stage)})`).join(' · ')}>
지금: {m.current[0].title} ({queueStageLabel(m.current[0].stage)}){m.current.length > 1 ? ` 외 ${m.current.length - 1}` : ''}
</div>
{/if}
</div>
{/each}
</div>
<!-- ETA 한 줄 (안5 라이트 — 추정치) -->
<div class="text-xs text-dim mt-2.5 px-1 tabular-nums"
title="현재 페이스 기반 추정치 — 유입 변동 시 달라질 수 있습니다">
요약 대기 <strong class="text-text">{queue.summarize_eta.pending.toLocaleString()}</strong>
— 소화 {formatRate(queue.summarize_eta.done_rate_1h)}/h
· 유입 {formatRate(queue.summarize_eta.inflow_rate_1h)}/h
{#if queue.summarize_eta.eta_minutes != null}
· <span class="text-accent font-semibold">{etaPhrase(queue.summarize_eta.eta_minutes)}</span>
{:else}
· 유입 우세(백필 중)
{/if}
</div>
</div>
<ProcessingFlowBoard overview={queue} />
{/if}
<!-- ═══ 단계 상세 (기존 stage 테이블 — 접힘 강등, 실패 있을 때 자동 펼침) ═══ -->
+11 -2
View File
@@ -379,5 +379,14 @@ def test_build_stages_order_fields_and_age():
assert by["extract"]["failed"] == 2
assert by["extract"]["oldest_pending_age_sec"] is None
# 전 stage 행 존재 (빈 단계 숨김은 FE 몫)
assert {"stage", "pending", "processing", "failed", "done_today",
"oldest_pending_age_sec"} == set(rows[0].keys())
assert {"stage", "pending", "processing", "failed", "done_1h", "created_1h",
"done_today", "oldest_pending_age_sec"} == set(rows[0].keys())
def test_build_stages_exposes_rates():
"""ds-board-engines-1: done_1h/created_1h 노출 — 흐름 노드 처리율·ETA·유입 우세 재료."""
from services.queue_overview import build_stages
stats = {"embed": _stage(pending=4, done_1h=600, created_1h=120, done_today=900)}
rows = build_stages(stats)
embed = next(r for r in rows if r["stage"] == "embed")
assert (embed["done_1h"], embed["created_1h"], embed["done_today"]) == (600, 120, 900)