Implement FireCrawl load testing, auto-scaling, and capacity planning strategies. Use when running performance tests, configuring horizontal scaling, or planning capacity for FireCrawl integrations. Trigger with phrases like "firecrawl load test", "firecrawl scale", "firecrawl performance test", "firecrawl capacity", "firecrawl k6", "firecrawl benchmark".
From firecrawl-packnpx claudepluginhub nickloveinvesting/nick-love-plugins --plugin firecrawl-packThis skill is limited to using the following tools:
Guides Next.js Cache Components and Partial Prerendering (PPR) with cacheComponents enabled. Implements 'use cache', cacheLife(), cacheTag(), revalidateTag(), static/dynamic optimization, and cache debugging.
Migrates code, prompts, and API calls from Claude Sonnet 4.0/4.5 or Opus 4.1 to Opus 4.5, updating model strings on Anthropic, AWS, GCP, Azure platforms.
Details PluginEval's skill quality evaluation: 3 layers (static, LLM judge), 10 dimensions, rubrics, formulas, anti-patterns, badges. Use to interpret scores, improve triggering, calibrate thresholds.
Load testing, scaling strategies, and capacity planning for FireCrawl integrations.
// firecrawl-load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
stages: [
{ duration: '2m', target: 10 }, // Ramp up
{ duration: '5m', target: 10 }, // Steady state
{ duration: '2m', target: 50 }, // Ramp to peak
{ duration: '5m', target: 50 }, // Stress test
{ duration: '2m', target: 0 }, // Ramp down
],
thresholds: {
http_req_duration: ['p(95)<500'], # HTTP 500 Internal Server Error
http_req_failed: ['rate<0.01'],
},
};
export default function () {
const response = http.post(
'https://api.firecrawl.com/v1/resource',
JSON.stringify({ test: true }),
{
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${__ENV.FIRECRAWL_API_KEY}`,
},
}
);
check(response, {
'status is 200': (r) => r.status === 200, # HTTP 200 OK
'latency < 500ms': (r) => r.timings.duration < 500, # HTTP 500 Internal Server Error
});
sleep(1);
}
# Install k6
brew install k6 # macOS
# or: sudo apt install k6 # Linux
# Run test
k6 run --env FIRECRAWL_API_KEY=${FIRECRAWL_API_KEY} firecrawl-load-test.js
# Run with output to InfluxDB
k6 run --out influxdb=http://localhost:8086/k6 firecrawl-load-test.js # 8086 = configured value
# kubernetes HPA
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: firecrawl-integration-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: firecrawl-integration
minReplicas: 2
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Pods
pods:
metric:
name: firecrawl_queue_depth
target:
type: AverageValue
averageValue: 100
import { Pool } from 'generic-pool';
const firecrawlPool = Pool.create({
create: async () => {
return new FireCrawlClient({
apiKey: process.env.FIRECRAWL_API_KEY!,
});
},
destroy: async (client) => {
await client.close();
},
max: 20,
min: 5,
idleTimeoutMillis: 30000, # 30000: 30 seconds in ms
});
async function withFireCrawlClient<T>(
fn: (client: FireCrawlClient) => Promise<T>
): Promise<T> {
const client = await firecrawlPool.acquire();
try {
return await fn(client);
} finally {
firecrawlPool.release(client);
}
}
| Metric | Warning | Critical |
|---|---|---|
| CPU Utilization | > 70% | > 85% |
| Memory Usage | > 75% | > 90% |
| Request Queue Depth | > 100 | > 500 |
| Error Rate | > 1% | > 5% |
| P95 Latency | > 1000ms | > 3000ms |
interface CapacityEstimate {
currentRPS: number;
maxRPS: number;
headroom: number;
scaleRecommendation: string;
}
function estimateFireCrawlCapacity(
metrics: SystemMetrics
): CapacityEstimate {
const currentRPS = metrics.requestsPerSecond;
const avgLatency = metrics.p50Latency;
const cpuUtilization = metrics.cpuPercent;
// Estimate max RPS based on current performance
const maxRPS = currentRPS / (cpuUtilization / 100) * 0.7; // 70% target
const headroom = ((maxRPS - currentRPS) / currentRPS) * 100;
return {
currentRPS,
maxRPS: Math.floor(maxRPS),
headroom: Math.round(headroom),
scaleRecommendation: headroom < 30
? 'Scale up soon'
: headroom < 50
? 'Monitor closely'
: 'Adequate capacity',
};
}
## FireCrawl Performance Benchmark
**Date:** YYYY-MM-DD
**Environment:** [staging/production]
**SDK Version:** X.Y.Z
### Test Configuration
- Duration: 10 minutes
- Ramp: 10 → 100 → 10 VUs
- Target endpoint: /v1/resource
### Results
| Metric | Value |
|--------|-------|
| Total Requests | 50,000 |
| Success Rate | 99.9% |
| P50 Latency | 120ms |
| P95 Latency | 350ms |
| P99 Latency | 800ms |
| Max RPS Achieved | 150 |
### Observations
- [Key finding 1]
- [Key finding 2]
### Recommendations
- [Scaling recommendation]
Write k6 test script with appropriate thresholds.
Set up HPA with CPU and custom metrics.
Execute test and collect metrics.
Record results in benchmark template.
| Issue | Cause | Solution |
|---|---|---|
| k6 timeout | Rate limited | Reduce RPS |
| HPA not scaling | Wrong metrics | Verify metric name |
| Connection refused | Pool exhausted | Increase pool size |
| Inconsistent results | Warm-up needed | Add ramp-up phase |
k6 run --vus 10 --duration 30s firecrawl-load-test.js
const metrics = await getSystemMetrics();
const capacity = estimateFireCrawlCapacity(metrics);
console.log('Headroom:', capacity.headroom + '%');
console.log('Recommendation:', capacity.scaleRecommendation);
set -euo pipefail
kubectl scale deployment firecrawl-integration --replicas=5
kubectl get hpa firecrawl-integration-hpa
For reliability patterns, see firecrawl-reliability-patterns.