From cohere-pack
Executes Cohere API incident runbook: triages outages with curl endpoint tests and status checks, mitigates via kubectl, handles postmortems for integration failures.
npx claudepluginhub jeremylongshore/claude-code-plugins-plus-skills --plugin cohere-packThis skill is limited to using the following tools:
Rapid incident response procedures for Cohere API v2 outages. Covers triage, mitigation, communication, and postmortem for Chat, Embed, Rerank, and Classify endpoints.
Runs Cohere API v2 production deployment checklist: auth, code quality, models, performance, health checks, rollback for go-live.
Executes incident response for Anthropic Claude API outages, errors, rate limits, and latency with triage scripts, status checks, and mitigation steps.
Executes Mistral AI incident triage, mitigation, and postmortem using bash scripts to check API status, rate limits, auth errors, and service health via curl and kubectl.
Share bugs, ideas, or general feedback.
Rapid incident response procedures for Cohere API v2 outages. Covers triage, mitigation, communication, and postmortem for Chat, Embed, Rerank, and Classify endpoints.
| Level | Definition | Response Time | Example |
|---|---|---|---|
| P1 | All Cohere endpoints down | < 15 min | API returning 5xx globally |
| P2 | Degraded (rate limits, high latency) | < 1 hour | 429 errors, P95 > 10s |
| P3 | Single endpoint affected | < 4 hours | Embed works, Chat fails |
| P4 | Non-blocking issue | Next business day | Slow response, minor errors |
# 1. Check Cohere service status
curl -s https://status.cohere.com/api/v2/status.json | jq '.status.description'
# 2. Test each endpoint directly
echo "--- Chat ---"
curl -s -o /dev/null -w "%{http_code}" \
-X POST https://api.cohere.com/v2/chat \
-H "Authorization: Bearer $CO_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"command-r7b-12-2024","messages":[{"role":"user","content":"ping"}]}'
echo -e "\n--- Embed ---"
curl -s -o /dev/null -w "%{http_code}" \
-X POST https://api.cohere.com/v2/embed \
-H "Authorization: Bearer $CO_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"embed-v4.0","texts":["test"],"input_type":"search_document","embedding_types":["float"]}'
echo -e "\n--- Rerank ---"
curl -s -o /dev/null -w "%{http_code}" \
-X POST https://api.cohere.com/v2/rerank \
-H "Authorization: Bearer $CO_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"rerank-v3.5","query":"test","documents":["a","b"]}'
# 3. Check our app health
curl -sf https://api.yourapp.com/api/health | jq '.cohere'
# 4. Check error rate (last 5 min)
curl -s "localhost:9090/api/v1/query?query=rate(cohere_errors_total[5m])" | jq '.data.result'
Cohere API returning errors?
├─ YES: Is status.cohere.com showing incident?
│ ├─ YES → Cohere-side outage. Enable fallback. Monitor status page.
│ └─ NO → Check our API key and configuration.
│ ├─ 401 → API key revoked or wrong. Check CO_API_KEY.
│ ├─ 429 → Rate limited. Check if trial key in prod.
│ ├─ 400 → Bad request. Check request format (v1 vs v2?).
│ └─ 5xx → Cohere server issue. Retry with backoff.
└─ NO: Is our app healthy?
├─ YES → Intermittent. Monitor.
└─ NO → Our infrastructure. Check pods, memory, network.
# Verify API key is set
kubectl get secret cohere-secrets -o jsonpath='{.data.CO_API_KEY}' | base64 -d | head -c4
echo "..."
# Test key directly
curl -s -o /dev/null -w "%{http_code}" \
-H "Authorization: Bearer $(kubectl get secret cohere-secrets -o jsonpath='{.data.CO_API_KEY}' | base64 -d)" \
-H "Content-Type: application/json" \
https://api.cohere.com/v2/chat \
-d '{"model":"command-r7b-12-2024","messages":[{"role":"user","content":"test"}]}'
# If key is invalid: rotate
# 1. Generate new key at dashboard.cohere.com
# 2. Update secret
kubectl create secret generic cohere-secrets \
--from-literal=CO_API_KEY=NEW_KEY \
--dry-run=client -o yaml | kubectl apply -f -
# 3. Restart pods
kubectl rollout restart deployment/app
# Check if using trial key in production (trial = 20 calls/min)
KEY_LEN=$(kubectl get secret cohere-secrets -o jsonpath='{.data.CO_API_KEY}' | base64 -d | wc -c)
echo "Key length: $KEY_LEN chars"
# If trial key: upgrade to production key at dashboard.cohere.com
# If production key: reduce concurrency
kubectl set env deployment/app COHERE_MAX_CONCURRENT=5
# Enable request queuing
kubectl set env deployment/app COHERE_QUEUE_ENABLED=true
# Enable graceful degradation
kubectl set env deployment/app COHERE_FALLBACK_ENABLED=true
# If using RAG: fall back to rerank-only (skip chat)
# If using agents: fall back to cached responses
# Monitor Cohere status page for resolution
watch -n 30 'curl -s https://status.cohere.com/api/v2/status.json | jq .status.description'
import { CohereError, CohereTimeoutError } from 'cohere-ai';
async function resilientChat(message: string): Promise<string> {
try {
const response = await cohere.chat({
model: 'command-a-03-2025',
messages: [{ role: 'user', content: message }],
});
return response.message?.content?.[0]?.text ?? '';
} catch (err) {
if (err instanceof CohereError && err.statusCode === 429) {
// Fallback to cheaper model (may have separate rate limit)
const fallback = await cohere.chat({
model: 'command-r7b-12-2024',
messages: [{ role: 'user', content: message }],
maxTokens: 200,
});
return fallback.message?.content?.[0]?.text ?? '';
}
if (err instanceof CohereError && (err.statusCode ?? 0) >= 500) {
return 'Cohere is temporarily unavailable. Please try again shortly.';
}
throw err;
}
}
P[1-4] INCIDENT: Cohere Integration
Status: INVESTIGATING / MITIGATED / RESOLVED
Impact: [e.g., "RAG answers unavailable, chat degraded to cached responses"]
Root cause: [e.g., "Cohere API returning 503 — confirmed on status.cohere.com"]
Current action: [e.g., "Enabled fallback mode, monitoring for recovery"]
Next update: [time]
Cohere Integration — Degraded Performance
Some AI-powered features may be slower than usual or temporarily unavailable.
We are monitoring the situation and will provide updates as available.
Last updated: [timestamp]
# Export error logs from incident window
kubectl logs -l app=my-cohere-app --since=1h | grep -i "cohere\|CohereError" > incident-logs.txt
# Export metrics
curl "localhost:9090/api/v1/query_range?query=cohere_errors_total&start=$(date -d '2 hours ago' +%s)&end=$(date +%s)&step=60" > metrics.json
# Cohere status history
curl -s https://status.cohere.com/api/v2/incidents.json | jq '.incidents[:3]'
## Incident: Cohere [endpoint] [error type]
**Date:** YYYY-MM-DD HH:MM - HH:MM UTC
**Duration:** X hours Y minutes
**Severity:** P[1-4]
**Detection:** [Alert name / user report / health check]
### Summary
[1-2 sentences]
### Timeline
- HH:MM — Alert fired: cohere_errors_total spike
- HH:MM — On-call acknowledged, began triage
- HH:MM — Root cause identified: [cause]
- HH:MM — Mitigation applied: [action]
- HH:MM — Service restored
### Root Cause
[Was it Cohere-side (status page incident) or our configuration?]
### Action Items
- [ ] Add circuit breaker for [endpoint] — @owner — due date
- [ ] Improve fallback for [scenario] — @owner — due date
- [ ] Add alert for [missed signal] — @owner — due date
For data handling, see cohere-data-handling.