Firecrawl Deploy Integration

Overview

Deploy applications using Firecrawl's web scraping API (api.firecrawl.dev) to production. Covers API key management, webhook endpoint deployment for async crawl results, and self-hosted Firecrawl deployment options using Docker.

Prerequisites

• Firecrawl API key stored in FIRECRAWL_API_KEY environment variable
• Application using @mendable/firecrawl-js SDK
• Platform CLI installed (vercel, docker, or gcloud)
• Webhook endpoint for async crawl results

Instructions

Step 1: Configure Secrets

# Vercel
vercel env add FIRECRAWL_API_KEY production

# Cloud Run
echo -n "your-key" | gcloud secrets create firecrawl-api-key --data-file=-

Step 2: Deploy Scraping API

// api/scrape.ts
import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

export async function POST(req: Request) {
  const { url, formats } = await req.json();

  const result = await firecrawl.scrapeUrl(url, {
    formats: formats || ["markdown"],
  });

  return Response.json({
    markdown: result.markdown,
    metadata: result.metadata,
  });
}

Step 3: Self-Hosted Firecrawl (Docker)

# docker-compose.yml
version: "3.8"
services:
  firecrawl:
    image: mendableai/firecrawl:latest
    ports:
      - "3002:3002"  # 3002 = configured value
    environment:
      - REDIS_URL=redis://redis:6379  # 6379: Redis port
      - PLAYWRIGHT_BROWSERS_PATH=/browsers
    depends_on:
      - redis

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"  # Redis port

  app:
    build: .
    ports:
      - "3000:3000"  # 3000: 3 seconds in ms
    environment:
      - FIRECRAWL_API_URL=http://firecrawl:3002
    depends_on:
      - firecrawl

Step 4: Webhook Endpoint for Async Crawls

// api/webhooks/firecrawl.ts
export async function POST(req: Request) {
  const { type, id, data } = await req.json();

  if (type === "crawl.completed") {
    await processScrapedPages(id, data.pages);
  }

  return Response.json({ received: true });
}

Step 5: Health Check

export async function GET() {
  try {
    const result = await firecrawl.scrapeUrl("https://example.com", {
      formats: ["markdown"],
    });
    return Response.json({ status: result ? "healthy" : "degraded" });
  } catch {
    return Response.json({ status: "unhealthy" }, { status: 503 });  # HTTP 503 Service Unavailable
  }
}

Error Handling

Issue	Cause	Solution
Rate limited	Too many scrape requests	Queue requests with delays
Scrape blocked	Target site protection	Use waitFor and browser options
API key invalid	Key expired	Regenerate at firecrawl.dev dashboard
Self-hosted memory	Playwright overhead	Increase container memory to 2GB+

Examples

Basic usage: Apply firecrawl deploy integration to a standard project setup with default configuration options.

Advanced scenario: Customize firecrawl deploy integration for production environments with multiple constraints and team-specific requirements.

Resources

• Firecrawl Documentation
• Firecrawl Self-Hosting

Next Steps

For webhook handling, see firecrawl-webhooks-events.

Output

• Configuration files or code changes applied to the project
• Validation report confirming correct implementation
• Summary of changes made and their rationale