Optimize FireCrawl API performance with caching, batching, and connection pooling. Use when experiencing slow API responses, implementing caching strategies, or optimizing request throughput for FireCrawl integrations. Trigger with phrases like "firecrawl performance", "optimize firecrawl", "firecrawl latency", "firecrawl caching", "firecrawl slow", "firecrawl batch".
From firecrawl-packnpx claudepluginhub nickloveinvesting/nick-love-plugins --plugin firecrawl-packThis skill is limited to using the following tools:
Guides Next.js Cache Components and Partial Prerendering (PPR) with cacheComponents enabled. Implements 'use cache', cacheLife(), cacheTag(), revalidateTag(), static/dynamic optimization, and cache debugging.
Migrates code, prompts, and API calls from Claude Sonnet 4.0/4.5 or Opus 4.1 to Opus 4.5, updating model strings on Anthropic, AWS, GCP, Azure platforms.
Details PluginEval's skill quality evaluation: 3 layers (static, LLM judge), 10 dimensions, rubrics, formulas, anti-patterns, badges. Use to interpret scores, improve triggering, calibrate thresholds.
Optimize FireCrawl API performance with caching, batching, and connection pooling.
| Operation | P50 | P95 | P99 |
|---|---|---|---|
| Read | 50ms | 150ms | 300ms |
| Write | 100ms | 250ms | 500ms |
| List | 75ms | 200ms | 400ms |
import { LRUCache } from 'lru-cache';
const cache = new LRUCache<string, any>({
max: 1000, # 1000: 1 second in ms
ttl: 60000, // 1 minute # 60000: 1 minute in ms
updateAgeOnGet: true,
});
async function cachedFireCrawlRequest<T>(
key: string,
fetcher: () => Promise<T>,
ttl?: number
): Promise<T> {
const cached = cache.get(key);
if (cached) return cached as T;
const result = await fetcher();
cache.set(key, result, { ttl });
return result;
}
import Redis from 'ioredis';
const redis = new Redis(process.env.REDIS_URL);
async function cachedWithRedis<T>(
key: string,
fetcher: () => Promise<T>,
ttlSeconds = 60
): Promise<T> {
const cached = await redis.get(key);
if (cached) return JSON.parse(cached);
const result = await fetcher();
await redis.setex(key, ttlSeconds, JSON.stringify(result));
return result;
}
import DataLoader from 'dataloader';
const firecrawlLoader = new DataLoader<string, any>(
async (ids) => {
// Batch fetch from FireCrawl
const results = await firecrawlClient.batchGet(ids);
return ids.map(id => results.find(r => r.id === id) || null);
},
{
maxBatchSize: 100,
batchScheduleFn: callback => setTimeout(callback, 10),
}
);
// Usage - automatically batched
const [item1, item2, item3] = await Promise.all([
firecrawlLoader.load('id-1'),
firecrawlLoader.load('id-2'),
firecrawlLoader.load('id-3'),
]);
import { Agent } from 'https';
// Keep-alive connection pooling
const agent = new Agent({
keepAlive: true,
maxSockets: 10,
maxFreeSockets: 5,
timeout: 30000, # 30000: 30 seconds in ms
});
const client = new FireCrawlClient({
apiKey: process.env.FIRECRAWL_API_KEY!,
httpAgent: agent,
});
async function* paginatedFireCrawlList<T>(
fetcher: (cursor?: string) => Promise<{ data: T[]; nextCursor?: string }>
): AsyncGenerator<T> {
let cursor: string | undefined;
do {
const { data, nextCursor } = await fetcher(cursor);
for (const item of data) {
yield item;
}
cursor = nextCursor;
} while (cursor);
}
// Usage
for await (const item of paginatedFireCrawlList(cursor =>
firecrawlClient.list({ cursor, limit: 100 })
)) {
await process(item);
}
async function measuredFireCrawlCall<T>(
operation: string,
fn: () => Promise<T>
): Promise<T> {
const start = performance.now();
try {
const result = await fn();
const duration = performance.now() - start;
console.log({ operation, duration, status: 'success' });
return result;
} catch (error) {
const duration = performance.now() - start;
console.error({ operation, duration, status: 'error', error });
throw error;
}
}
Measure current latency for critical FireCrawl operations.
Add response caching for frequently accessed data.
Use DataLoader or similar for automatic request batching.
Configure connection pooling with keep-alive.
| Issue | Cause | Solution |
|---|---|---|
| Cache miss storm | TTL expired | Use stale-while-revalidate |
| Batch timeout | Too many items | Reduce batch size |
| Connection exhausted | No pooling | Configure max sockets |
| Memory pressure | Cache too large | Set max cache entries |
const withPerformance = <T>(name: string, fn: () => Promise<T>) =>
measuredFireCrawlCall(name, () =>
cachedFireCrawlRequest(`cache:${name}`, fn)
);
For cost optimization, see firecrawl-cost-tuning.