Hyungi Ahn
5bfbb79641
VERIFIER_NUMERIC_PROMOTE 환경변수로 numeric_conflict severity 승격 실험.
verifier_service.py:
- _NUMERIC_PROMOTE = os.getenv('VERIFIER_NUMERIC_PROMOTE', '0') == '1'
(import time 평가 — env 변경 시 process restart 필수)
- _SEVERITY_MAP['numeric_conflict']: env=1 → critical=strong / minor=medium,
env=0 (기본) → 둘 다 medium (기존 동작 유지)
- direct_negation 은 env 무관 항상 strong (안전장치)
verifier.txt:
- numeric_conflict 정의에 critical/minor 분리 명시 (core quantity vs peripheral)
- "Range values satisfy any answer within range" rule 추가
- severity mapping 갱신: numeric_conflict 분기 명시
search.py re-gate (Tier 1~7 재번호, B2 신규 Tier 4):
- v_strong_numeric = sum(1 for f in v_strong
if f.startswith('verifier_numeric_conflict'))
- Tier 4 (신규): g_strong + v_strong_numeric >= 1 + low_conf → refuse
re_gate value: 'refuse(grounding+verifier_numeric)'
- 원칙 유지: verifier strong 단독 refuse 금지 — g_strong 교차 필수
- 호환성: 기존 re_gate string literals 그대로 유지, 신규 1개만 추가
credentials.env.example: VERIFIER_NUMERIC_PROMOTE=0 (off, B3 통과 후 production 전환)
tests/test_verifier_numeric_promote.py: 4 케이스 (env off / on / explicit 0 /
direct_negation invariant). monkeypatch.setenv + importlib.reload 패턴.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
43 lines
2.5 KiB
Plaintext
43 lines
2.5 KiB
Plaintext
You are a grounding verifier. Given an answer and its evidence sources, check if the answer contradicts or fabricates information. Respond ONLY in JSON.
|
|
|
|
## Contradiction Types (IMPORTANT — severity depends on type)
|
|
- **direct_negation** (CRITICAL): Answer directly contradicts evidence. Examples: evidence "의무" but answer "권고"; evidence "금지" but answer "허용"; negation reversal ("~해야 한다" vs "~할 필요 없다").
|
|
- **numeric_conflict**: Answer states a number different from evidence. "50명" in evidence but "100명" in answer. Only flag if the same concept is referenced. severity=critical when the number is the CORE answered quantity (amount/count/rate/date/duration that the query asked for); severity=minor when the number is peripheral (e.g., example/footnote).
|
|
- **intent_core_mismatch**: Answer addresses a fundamentally different topic than the query asked about.
|
|
- **nuance**: Answer overgeneralizes or adds qualifiers not in evidence (e.g., "모든" when evidence says "일부").
|
|
- **unsupported_claim**: Answer makes a factual claim with no basis in any evidence.
|
|
|
|
## Rules
|
|
1. Compare each claim in the answer against the cited evidence. A claim with [n] citation should be checked against evidence [n].
|
|
2. NOT a contradiction: Paraphrasing, summarizing, or restating the same fact in different words. Korean formal/informal style (합니다/한다) differences.
|
|
3. Numbers must match exactly after normalization (1,000 = 1000). Range values (e.g., "100~200명") satisfy any answer within range.
|
|
4. Legal/regulatory terms must preserve original meaning (의무 ≠ 권고, 금지 ≠ 제한, 허용 ≠ 금지).
|
|
5. Maximum 5 contradictions (most severe first: direct_negation > numeric_conflict > intent_core_mismatch > nuance > unsupported_claim).
|
|
|
|
## Output Schema
|
|
{
|
|
"contradictions": [
|
|
{
|
|
"type": "direct_negation" | "numeric_conflict" | "intent_core_mismatch" | "nuance" | "unsupported_claim",
|
|
"severity": "critical" | "minor",
|
|
"claim": "answer 내 해당 구절 (50자 이내)",
|
|
"evidence_ref": "대응 근거 내용 (50자 이내, [n] 포함)",
|
|
"explanation": "모순 이유 (한국어, 30자 이내)"
|
|
}
|
|
],
|
|
"verdict": "clean" | "minor_issues" | "major_issues"
|
|
}
|
|
|
|
severity mapping:
|
|
- direct_negation → "critical"
|
|
- numeric_conflict → "critical" if the number is the CORE answered quantity, else "minor"
|
|
- All other types → "minor"
|
|
|
|
If no contradictions: {"contradictions": [], "verdict": "clean"}
|
|
|
|
## Answer
|
|
{answer}
|
|
|
|
## Evidence
|
|
{numbered_evidence}
|