# 🔌 Circuit Breakers — Resilencia avanzada
## 🤔 ¿Qué hago? ¿Cómo lo hago? ¿Y para qué lo hago?
**¿Qué hago?** Implemento el patrón Circuit Breaker para proteger klaude-proxy contra fallos de dependencias (Anthropic, Qdrant, embeddings).
**¿Cómo lo hago?** Cada llamada a un servicio externo pasa por un circuit breaker con 3 estados: CLOSED (normal) → OPEN (rechaza) → HALF_OPEN (prueba reset).
**¿Para qué lo hago?** Evitar cascadas de fallos. Cuando Anthropic está caído, el circuit breaker abre inmediatamente, evitando timeout chains que degradan toda la plataforma.
---
## 🏗️ Arquitectura
### Estados del Circuit Breaker
```
CLOSED (normal)
↓ (5+ fallos)
OPEN (rechaza inmediatamente)
↓ (30s timeout)
HALF_OPEN (prueba 1 llamada)
├─ éxito → CLOSED (recuperado)
└─ fallo → OPEN (vuelve a esperar 30s)
```
### Parámetros de configuración
| Parámetro | Valor | Razón |
| --- | --- | --- |
| **failure_threshold** | 5 | Abrir después de 5 fallos consecutivos |
| **recovery_timeout** | 30s | Esperar 30s en OPEN antes de intentar reset |
| **backoff_multiplier** | 2.0 | Retry exponencial: 1s → 2s → 4s → 8s → 30s |
---
## 📍 Integración por servicio
### 1. Anthropic API (`proxy/anthropic_client.py`)
```python
# Circuit breaker para forward_blocking y forward_stream
_cb_forward = CircuitBreaker("anthropic_forward", failure_threshold=5)
# Uso:
response = await _cb_forward.call_async(
_forward_blocking_impl,
request_data,
incoming_headers
)
```
**Comportamiento:**
- ✅ CLOSED: Llamadas pasan normalmente
- 🔴 OPEN: `CircuitBreakerError` — cliente recibe 503
- 🟠 HALF_OPEN: 1 llamada de prueba permitida
**Casos de uso:**
- Anthropic rate limited (429)
- Anthropic servidor down (5xx)
- Timeout de conexión a Anthropic
- Latencia extrema (>30s)
---
### 2. Qdrant Cache (`proxy/cache.py`) — *opcional*
Implementar análogo:
```python
_cb_qdrant = CircuitBreaker("qdrant_cache", failure_threshold=5)
# En search():
results = await _cb_qdrant.call_async(
client.search,
collection_name="klaude_cache",
query_vector=vector,
limit=5
)
```
**Casos de uso:**
- Qdrant conexión rechazada
- Qdrant colección corrupta
- Network timeout a Qdrant
---
### 3. Embeddings Model (`proxy/embeddings.py`) — *opcional*
```python
_cb_embeddings = CircuitBreaker("embeddings_model", failure_threshold=3)
# En get_embedding():
embedding = await _cb_embeddings.call_async(
model.encode,
text
)
```
**Casos de uso:**
- Modelo ONNX load failure
- OOM (out of memory)
- CUDA device error
---
## 📊 Monitoreo y alerting
### Métricas expuestas
```bash
GET /metrics
```
Métricas relevantes:
- `requests_failed_total{error_type="CircuitBreakerError"}` — contador de rechazos
- `requests_total{status="503"}` — status 503 por circuit breaker abierto
- Endpoint `/health/ready` — retorna false si circuit breaker OPEN
### PrometheusRule (helm/prometheus-rules.yaml)
```yaml
- alert: CircuitBreakerOpen
expr: requests_failed_total > 0.05 * requests_total
for: 5m
severity: warning
annotations:
summary: "Circuit breaker OPEN en {{ $labels.endpoint }}"
```
---
## 🔄 Flujo de recuperación
### Escenario: Anthropic se cae a las 14:00
```
14:00:00 — Anthropic comienza a devolver 503
14:00:15 — Circuit breaker acumula 5 fallos → pasa a OPEN
Requests posteriores: CircuitBreakerError (503 al cliente)
14:00:30 — Anthropic se recupera, pero circuit sigue OPEN
14:00:35 — timeout alcanzado (30s)
Circuit breaker pasa a HALF_OPEN
Próxima request intentará de nuevo
14:00:40 — Request de prueba exitosa → CLOSED
Tráfico normal restaurado ✅
```
### Escenario: Antropic sigue caído
```
14:00:35 — HALF_OPEN, request de prueba
14:00:36 — Falla (aún 503) → vuelve a OPEN
14:01:06 — timeout alcanzado, retry de reset
...
```
---
## ⚙️ API de uso
### Decorador `@circuit_breaker_async`
```python
from circuit_breaker import circuit_breaker_async
@circuit_breaker_async("my_service", failure_threshold=3, recovery_timeout=30)
async def call_external_service(data):
response = await httpx_client.post(url, json=data)
return response.json()
```
El decorador expone la instancia: `call_external_service._circuit_breaker`
### Clase `CircuitBreaker` directo
```python
from circuit_breaker import CircuitBreaker
cb = CircuitBreaker("my_cb", failure_threshold=5)
# Async
result = await cb.call_async(some_async_func, arg1, arg2)
# Status
print(cb.status)
# {
# "name": "my_cb",
# "state": "closed",
# "failure_count": 2,
# "time_since_last_state_change": 45.3
# }
```
### `RetryWithBackoff`
```python
from circuit_breaker import RetryWithBackoff
retry = RetryWithBackoff(
max_attempts=3,
initial_delay=1.0,
max_delay=30.0,
multiplier=2.0
)
async def is_retryable(exc):
# Retryable solo si es transient (429, 5xx, timeout)
return isinstance(exc, (httpx.TimeoutException, httpx.HTTPStatusError))
result = await retry.execute(
async_function,
arg1,
arg2,
is_retryable=is_retryable
)
```
---
## 🧪 Testing
### Test de estado CLOSED
```python
def test_circuit_breaker_closed():
cb = CircuitBreaker("test")
async def success_func():
return "ok"
result = await cb.call_async(success_func)
assert result == "ok"
assert cb.state == CircuitState.CLOSED
```
### Test de estado OPEN
```python
@pytest.mark.asyncio
async def test_circuit_breaker_opens_after_threshold():
cb = CircuitBreaker("test", failure_threshold=2)
async def fail_func():
raise Exception("failed")
for _ in range(2):
with pytest.raises(Exception):
await cb.call_async(fail_func)
# 3ra llamada sin excepción, pero CB abierto
with pytest.raises(CircuitBreakerError):
await cb.call_async(success_func)
```
---
## 🚀 Mejoras futuras
- [ ] Circuit breaker per-endpoint (fine-grained control)
- [ ] Histograma de fallos por error type (429 vs 5xx vs timeout)
- [ ] Adaptive timeout (backoff basado en latencia)
- [ ] Bulkhead pattern (limitar concurrent requests per service)
- [ ] Fallback strategies (cached responses, default values)
---
## 📚 Referencias
- [Release It! — Michael Nygard (Circuit Breaker pattern)](https://pragprog.com/titles/mnee2/release-it-second-edition/)
- [AWS Well-Architected — Resilience](https://docs.aws.amazon.com/wellarchitected/latest/userguide/resilience.html)
- [Google SRE Book — Cascading Failures](https://sre.google/sre-book/handling-overload/)