feat: Hybrid 부하 판단 — health latency baseline + 조건부 inference

- model_adapter: measure_inference_latency() (max_tokens=1, 최소 부하)
- backend_registry:
  - health latency baseline 학습 (초기 5회 max, 이후 EMA)
  - get_load_status(): inference 우선, health/queue 보조
  - cache 30s + cooldown 10s + asyncio.Lock으로 자기증폭 루프 방지
  - 조건: health > baseline*3 또는 사용자 명시 요청 시에만 ping
- worker:
  - "system_status" 액션 — 사용자 상태 조회 시 force_measure
  - _build_system_status() 응답 빌더 (health/baseline/ping/queue)
  - route busy 안내를 get_load_status 기반으로 변경

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Hyungi Ahn
2026-04-07 07:46:57 +09:00
parent fcd29a82c3
commit 875a4f80d2
3 changed files with 158 additions and 25 deletions

View File

@@ -115,3 +115,19 @@ class ModelAdapter:
return resp.status_code < 500
except Exception:
return False
async def measure_inference_latency(self) -> float:
"""ping 메시지로 실제 inference latency 측정. max_tokens=1로 최소 부하.
반환: 밀리초. 실패 시 -1.0"""
import time as _time
original_max = self.max_tokens
self.max_tokens = 1
try:
start = _time.monotonic()
await self.complete_chat("ping")
return (_time.monotonic() - start) * 1000
except Exception:
logger.warning("inference latency measurement failed for %s", self.name)
return -1.0
finally:
self.max_tokens = original_max