FireCrawl Migration Deep Dive

Overview

Comprehensive guide for migrating to or from FireCrawl, or major version upgrades.

Prerequisites

• Current system documentation
• FireCrawl SDK installed
• Feature flag infrastructure
• Rollback strategy tested

Migration Types

Type	Complexity	Duration	Risk
Fresh install	Low	Days	Low
From competitor	Medium	Weeks	Medium
Major version	Medium	Weeks	Medium
Full replatform	High	Months	High

Pre-Migration Assessment

Step 1: Current State Analysis

set -euo pipefail
# Document current implementation
find . -name "*.ts" -o -name "*.py" | xargs grep -l "firecrawl" > firecrawl-files.txt

# Count integration points
wc -l firecrawl-files.txt

# Identify dependencies
npm list | grep firecrawl
pip freeze | grep firecrawl

Step 2: Data Inventory

interface MigrationInventory {
  dataTypes: string[];
  recordCounts: Record<string, number>;
  dependencies: string[];
  integrationPoints: string[];
  customizations: string[];
}

async function assessFireCrawlMigration(): Promise<MigrationInventory> {
  return {
    dataTypes: await getDataTypes(),
    recordCounts: await getRecordCounts(),
    dependencies: await analyzeDependencies(),
    integrationPoints: await findIntegrationPoints(),
    customizations: await documentCustomizations(),
  };
}

Migration Strategy: Strangler Fig Pattern

Phase 1: Parallel Run
┌─────────────┐     ┌─────────────┐
│   Old       │     │   New       │
│   System    │ ──▶ │  FireCrawl   │
│   (100%)    │     │   (0%)      │
└─────────────┘     └─────────────┘

Phase 2: Gradual Shift
┌─────────────┐     ┌─────────────┐
│   Old       │     │   New       │
│   (50%)     │ ──▶ │   (50%)     │
└─────────────┘     └─────────────┘

Phase 3: Complete
┌─────────────┐     ┌─────────────┐
│   Old       │     │   New       │
│   (0%)      │ ──▶ │   (100%)    │
└─────────────┘     └─────────────┘

Implementation Plan

Phase 1: Setup (Week 1-2)

set -euo pipefail
# Install FireCrawl SDK
npm install @firecrawl/sdk

# Configure credentials
cp .env.example .env.firecrawl
# Edit with new credentials

# Verify connectivity
node -e "require('@firecrawl/sdk').ping()"

Phase 2: Adapter Layer (Week 3-4)

// src/adapters/firecrawl.ts
interface ServiceAdapter {
  create(data: CreateInput): Promise<Resource>;
  read(id: string): Promise<Resource>;
  update(id: string, data: UpdateInput): Promise<Resource>;
  delete(id: string): Promise<void>;
}

class FireCrawlAdapter implements ServiceAdapter {
  async create(data: CreateInput): Promise<Resource> {
    const firecrawlData = this.transform(data);
    return firecrawlClient.create(firecrawlData);
  }

  private transform(data: CreateInput): FireCrawlInput {
    // Map from old format to FireCrawl format
  }
}

Phase 3: Data Migration (Week 5-6)

async function migrateFireCrawlData(): Promise<MigrationResult> {
  const batchSize = 100;
  let processed = 0;
  let errors: MigrationError[] = [];

  for await (const batch of oldSystem.iterateBatches(batchSize)) {
    try {
      const transformed = batch.map(transform);
      await firecrawlClient.batchCreate(transformed);
      processed += batch.length;
    } catch (error) {
      errors.push({ batch, error });
    }

    // Progress update
    console.log(`Migrated ${processed} records`);
  }

  return { processed, errors };
}

Phase 4: Traffic Shift (Week 7-8)

// Feature flag controlled traffic split
function getServiceAdapter(): ServiceAdapter {
  const firecrawlPercentage = getFeatureFlag('firecrawl_migration_percentage');

  if (Math.random() * 100 < firecrawlPercentage) {
    return new FireCrawlAdapter();
  }

  return new LegacyAdapter();
}

Rollback Plan

set -euo pipefail
# Immediate rollback
kubectl set env deployment/app FIRECRAWL_ENABLED=false
kubectl rollout restart deployment/app

# Data rollback (if needed)
./scripts/restore-from-backup.sh --date YYYY-MM-DD

# Verify rollback
curl https://app.yourcompany.com/health | jq '.services.firecrawl'

Post-Migration Validation

async function validateFireCrawlMigration(): Promise<ValidationReport> {
  const checks = [
    { name: 'Data count match', fn: checkDataCounts },
    { name: 'API functionality', fn: checkApiFunctionality },
    { name: 'Performance baseline', fn: checkPerformance },
    { name: 'Error rates', fn: checkErrorRates },
  ];

  const results = await Promise.all(
    checks.map(async c => ({ name: c.name, result: await c.fn() }))
  );

  return { checks: results, passed: results.every(r => r.result.success) };
}

Instructions

Assess current configuration

Document existing implementation and data inventory.

Step 2: Build Adapter Layer

Create abstraction layer for gradual migration.

Step 3: Migrate Data

Run batch data migration with error handling.

Step 4: Shift Traffic

Gradually route traffic to new FireCrawl integration.

Output

• Migration assessment complete
• Adapter layer implemented
• Data migrated successfully
• Traffic fully shifted to FireCrawl

Error Handling

Issue	Cause	Solution
Data mismatch	Transform errors	Validate transform logic
Performance drop	No caching	Add caching layer
Rollback triggered	Errors spiked	Reduce traffic percentage
Validation failed	Missing data	Check batch processing

Examples

Quick Migration Status

const status = await validateFireCrawlMigration();
console.log(`Migration ${status.passed ? 'PASSED' : 'FAILED'}`);
status.checks.forEach(c => console.log(`  ${c.name}: ${c.result.success}`));

Resources

• Strangler Fig Pattern
• FireCrawl Migration Guide

Flagship+ Skills

For advanced troubleshooting, see firecrawl-advanced-troubleshooting.