feat(ask): Phase 3.5b guardrails — verifier + telemetry + grounding 강화
Phase 3.5a(classifier+refusal gate+grounding) 위에 4개 Item 추가: Item 0: ask_events telemetry 배선 - AskEvent ORM 모델 + record_ask_event() — ask_events INSERT 완성 - defense_layers에 input_snapshot(query, chunks, answer) 저장 - refused/normal 두 경로 모두 telemetry 호출 Item 3: evidence 간 numeric conflict detection - 동일 단위 다른 숫자 → weak flag - "이상/이하/초과/미만" threshold 표현 → skip (FP 방지) Item 4: fabricated_number normalization 개선 - 단위 접미사 건/원 추가, 범위 표현(10~20%) 양쪽 추출 - bare number 2자리 이상만 (1자리 FP 제거) Item 1: exaone semantic verifier (판단권 잠금 배선) - verifier_service.py — 3s timeout, circuit breaker, severity 3단계 - direct_negation만 strong, numeric/intent→medium, 나머지→weak - verifier strong 단독 refuse 금지 — grounding과 교차 필수 - 6-tier re-gate (4라운드 리뷰 확정) - grounding strong 2+ OR max_score<0.2 → verifier skip Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -25,6 +25,7 @@ from typing import Any
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
|
||||
from core.database import async_session
|
||||
from models.ask_event import AskEvent
|
||||
from models.search_failure import SearchFailureLog
|
||||
|
||||
logger = logging.getLogger("search_telemetry")
|
||||
@@ -306,3 +307,47 @@ async def record_search_event(
|
||||
failure_reason="low_confidence",
|
||||
context=base_ctx,
|
||||
)
|
||||
|
||||
|
||||
# ─── /ask 전용 telemetry (Phase 3.5b) ─────────────────────
|
||||
|
||||
|
||||
async def record_ask_event(
|
||||
query: str,
|
||||
user_id: int | None,
|
||||
completeness: str | None,
|
||||
synthesis_status: str | None,
|
||||
confidence: str | None,
|
||||
refused: bool,
|
||||
classifier_verdict: str | None,
|
||||
max_rerank_score: float,
|
||||
aggregate_score: float,
|
||||
hallucination_flags: list[str],
|
||||
evidence_count: int,
|
||||
citation_count: int,
|
||||
defense_layers: dict[str, Any],
|
||||
total_ms: int,
|
||||
) -> None:
|
||||
"""ask_events INSERT. background task에서 호출 — 에러 삼킴."""
|
||||
try:
|
||||
async with async_session() as session:
|
||||
row = AskEvent(
|
||||
query=query,
|
||||
user_id=user_id,
|
||||
completeness=completeness,
|
||||
synthesis_status=synthesis_status,
|
||||
confidence=confidence,
|
||||
refused=refused,
|
||||
classifier_verdict=classifier_verdict,
|
||||
max_rerank_score=max_rerank_score,
|
||||
aggregate_score=aggregate_score,
|
||||
hallucination_flags=hallucination_flags,
|
||||
evidence_count=evidence_count,
|
||||
citation_count=citation_count,
|
||||
defense_layers=defense_layers,
|
||||
total_ms=total_ms,
|
||||
)
|
||||
session.add(row)
|
||||
await session.commit()
|
||||
except SQLAlchemyError as exc:
|
||||
logger.warning(f"ask_event insert failed: {exc}")
|
||||
|
||||
Reference in New Issue
Block a user