Firecrawl Multi-Environment Setup

Overview

Firecrawl's credit-based pricing model makes environment separation critical. Development should use conservative crawl limits to avoid burning production credits during testing.

Prerequisites

• Firecrawl account with API key from firecrawl.dev
• Optional: self-hosted Firecrawl instance for development
• Understanding of credit consumption model (1 credit = 1 page scraped)

Environment Strategy

Environment	API Key	Crawl Limit	Concurrency	Credit Budget
Development	Dev key or self-hosted	10 pages max	1	Minimal
Staging	Staging key	100 pages max	2	Limited
Production	Prod key	Configured per task	Full	Monitored

Instructions

Step 1: Configuration Structure

// config/firecrawl.ts
import FirecrawlApp from "@mendable/firecrawl-js";

type Env = "development" | "staging" | "production";

interface FirecrawlConfig {
  apiKey: string;
  apiUrl?: string;              // override for self-hosted
  maxPagesPerCrawl: number;     // hard limit per crawl job
  maxDepth: number;
  concurrency: number;
  waitFor: number;              // ms to wait for JS rendering
}

const configs: Record<Env, FirecrawlConfig> = {
  development: {
    apiKey: process.env.FIRECRAWL_API_KEY_DEV || "fc-localdev",
    apiUrl: process.env.FIRECRAWL_API_URL_DEV, // point to self-hosted if available
    maxPagesPerCrawl: 10,         // strict limit in dev to protect credits
    maxDepth: 2,
    concurrency: 1,
    waitFor: 2000,  # 2000: 2 seconds in ms
  },
  staging: {
    apiKey: process.env.FIRECRAWL_API_KEY_STAGING!,
    maxPagesPerCrawl: 100,
    maxDepth: 3,
    concurrency: 2,
    waitFor: 3000,  # 3000: 3 seconds in ms
  },
  production: {
    apiKey: process.env.FIRECRAWL_API_KEY_PROD!,
    maxPagesPerCrawl: 500,        // per-task limit, set lower for specific jobs  # HTTP 500 Internal Server Error
    maxDepth: 5,
    concurrency: 5,
    waitFor: 3000,  # 3 seconds in ms
  },
};

export function getFirecrawlConfig(): FirecrawlConfig {
  const env = (process.env.NODE_ENV || "development") as Env;
  return configs[env] || configs.development;
}

export function getFirecrawlClient(): FirecrawlApp {
  const cfg = getFirecrawlConfig();
  return new FirecrawlApp({
    apiKey: cfg.apiKey,
    ...(cfg.apiUrl ? { apiUrl: cfg.apiUrl } : {}),
  });
}

Step 2: Self-Hosted Firecrawl for Development

# docker-compose.dev.yml - Run Firecrawl locally in dev
version: "3.8"
services:
  firecrawl:
    image: mendableai/firecrawl:latest
    ports:
      - "3002:3002"  # 3002 = configured value
    environment:
      - USE_DB_AUTHENTICATION=false
      - PORT=3002

Set FIRECRAWL_API_URL_DEV=http://localhost:3002 and any API key in .env.local to use local instance with no credit consumption.

Step 3: Credit-Aware Scraping Wrapper

// lib/firecrawl-service.ts
import { getFirecrawlClient, getFirecrawlConfig } from "../config/firecrawl";

export async function safeScrape(url: string, options?: any) {
  const firecrawl = getFirecrawlClient();
  const cfg = getFirecrawlConfig();

  return firecrawl.scrapeUrl(url, {
    formats: ["markdown"],
    onlyMainContent: true,
    waitFor: cfg.waitFor,
    ...options,
  });
}

export async function safeCrawl(url: string, customLimit?: number) {
  const firecrawl = getFirecrawlClient();
  const cfg = getFirecrawlConfig();

  const limit = Math.min(customLimit ?? cfg.maxPagesPerCrawl, cfg.maxPagesPerCrawl);

  return firecrawl.asyncCrawlUrl(url, {
    limit,
    maxDepth: cfg.maxDepth,
    scrapeOptions: { formats: ["markdown"], onlyMainContent: true },
  });
}

Step 4: Environment Variable Setup

# .env.local (development)
FIRECRAWL_API_KEY_DEV=fc-dev-abc123
FIRECRAWL_API_URL_DEV=http://localhost:3002  # 3002: optional self-hosted

# GitHub Actions - Staging environment
FIRECRAWL_API_KEY_STAGING=fc-staging-def456

# GitHub Actions - Production environment
FIRECRAWL_API_KEY_PROD=fc-prod-xyz789

Step 5: CI/CD Pipeline Integration

# .github/workflows/deploy.yml
jobs:
  test-scraping:
    environment: staging
    env:
      FIRECRAWL_API_KEY_STAGING: ${{ secrets.FIRECRAWL_API_KEY_STAGING }}
      NODE_ENV: staging
    steps:
      - run: node scripts/test-firecrawl.js
        # Uses staging config: 100-page limit, staging API key

  deploy:
    environment: production
    env:
      FIRECRAWL_API_KEY_PROD: ${{ secrets.FIRECRAWL_API_KEY_PROD }}
      NODE_ENV: production

Error Handling

Issue	Cause	Solution
Credits depleted in dev	No page limit in dev config	Always set maxPagesPerCrawl: 10 for development
Self-hosted not responding	Docker container not running	Check docker-compose up firecrawl
402 Payment Required	Production credits exhausted	Monitor credit balance before large jobs
Different results per env	waitFor mismatch	Standardize JS wait time or test with prod settings

Examples

Check Active Configuration

import { getFirecrawlConfig } from "./config/firecrawl";

const cfg = getFirecrawlConfig();
console.log(`Max pages: ${cfg.maxPagesPerCrawl}, depth: ${cfg.maxDepth}`);
console.log(`API URL: ${cfg.apiUrl || "https://api.firecrawl.dev"}`);

Budget Check Before Large Crawl

set -euo pipefail
# Check current credit balance
curl -s "https://api.firecrawl.dev/v1/team/credits" \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY_PROD" | jq .credits_remaining

Resources

• Firecrawl API Documentation
• Firecrawl Self-Hosting Guide
• Firecrawl Pricing

Next Steps

For deployment configuration, see firecrawl-deploy-integration.

Output

• Configuration files or code changes applied to the project
• Validation report confirming correct implementation
• Summary of changes made and their rationale