Skill

brightdata-local-dev-loop

Configure Bright Data local development with hot reload and testing. Use when setting up a development environment, configuring test workflows, or establishing a fast iteration cycle with Bright Data. Trigger with phrases like "brightdata dev setup", "brightdata local development", "brightdata dev environment", "develop with brightdata".

npx claudepluginhub flight505/skill-forge --plugin brightdata-pack

Tool Access

This skill is limited to using the following tools:

ReadWriteEditBash(npm:*)Bash(pnpm:*)Grep

Preview

Set up a fast, reproducible local development workflow for Bright Data scraping projects with mocked proxy responses, cached results, and vitest integration.

SKILL.md

Similar Skills

github-deep-research

63.9k

Conducts multi-round deep research on GitHub repos via API and web searches, generating markdown reports with executive summaries, timelines, metrics, and Mermaid diagrams.

2 files

bytedance-deer-flow-1

Stats

Parent Repo Stars0

Parent Repo Forks0

Last CommitApr 30, 2026

Actions

View Source View Plugin View on GitHub View README

Help us improve

Share bugs, ideas, or general feedback.

Bright Data Local Dev Loop

Overview

Set up a fast, reproducible local development workflow for Bright Data scraping projects with mocked proxy responses, cached results, and vitest integration.

Prerequisites

Completed brightdata-install-auth setup
Node.js 18+ with npm/pnpm
brd-ca.crt SSL certificate downloaded

Instructions

Step 1: Create Project Structure

my-scraper/
├── src/
│   ├── brightdata/
│   │   ├── proxy.ts         # Proxy configuration helper
│   │   ├── scraper.ts       # Scraping functions
│   │   └── cache.ts         # Response caching for dev
│   └── index.ts
├── tests/
│   ├── fixtures/            # Cached HTML responses
│   │   └── example.html
│   └── scraper.test.ts
├── .env.local               # Local credentials (git-ignored)
├── .env.example             # Template for team
├── brd-ca.crt               # Bright Data SSL cert (git-ignored)
└── package.json

Step 2: Build Proxy Configuration Module

// src/brightdata/proxy.ts
import 'dotenv/config';

export interface BrightDataProxy {
  host: string;
  port: number;
  auth: { username: string; password: string };
}

export function getProxy(options?: {
  country?: string;
  city?: string;
  session?: string;
}): BrightDataProxy {
  const { BRIGHTDATA_CUSTOMER_ID, BRIGHTDATA_ZONE, BRIGHTDATA_ZONE_PASSWORD } = process.env;
  if (!BRIGHTDATA_CUSTOMER_ID || !BRIGHTDATA_ZONE || !BRIGHTDATA_ZONE_PASSWORD) {
    throw new Error('Missing BRIGHTDATA_* environment variables');
  }

  let username = `brd-customer-${BRIGHTDATA_CUSTOMER_ID}-zone-${BRIGHTDATA_ZONE}`;
  if (options?.country) username += `-country-${options.country}`;
  if (options?.city) username += `-city-${options.city}`;
  if (options?.session) username += `-session-${options.session}`;

  return {
    host: 'brd.superproxy.io',
    port: 33335,
    auth: { username, password: BRIGHTDATA_ZONE_PASSWORD },
  };
}

Step 3: Add Response Cache for Development

// src/brightdata/cache.ts — cache scraped pages to avoid burning proxy credits
import { createHash } from 'crypto';
import { existsSync, readFileSync, writeFileSync, mkdirSync } from 'fs';
import { join } from 'path';

const CACHE_DIR = join(process.cwd(), '.scrape-cache');

export function getCachedResponse(url: string): string | null {
  const key = createHash('md5').update(url).digest('hex');
  const path = join(CACHE_DIR, `${key}.html`);
  return existsSync(path) ? readFileSync(path, 'utf-8') : null;
}

export function setCachedResponse(url: string, html: string): void {
  mkdirSync(CACHE_DIR, { recursive: true });
  const key = createHash('md5').update(url).digest('hex');
  writeFileSync(join(CACHE_DIR, `${key}.html`), html);
}

Step 4: Configure Testing with Mocked Responses

// tests/scraper.test.ts
import { describe, it, expect, vi, beforeEach } from 'vitest';
import axios from 'axios';

vi.mock('axios');
const mockedAxios = vi.mocked(axios);

describe('Bright Data Scraper', () => {
  beforeEach(() => vi.clearAllMocks());

  it('should scrape through proxy and return HTML', async () => {
    mockedAxios.get.mockResolvedValueOnce({
      status: 200,
      data: '<html><head><title>Test</title></head></html>',
    });

    const { scrape } = await import('../src/brightdata/scraper');
    const html = await scrape('https://example.com');
    expect(html).toContain('<title>Test</title>');

    // Verify proxy was configured
    expect(mockedAxios.get).toHaveBeenCalledWith(
      'https://example.com',
      expect.objectContaining({
        proxy: expect.objectContaining({ host: 'brd.superproxy.io' }),
      }),
    );
  });

  it('should retry on 502 proxy errors', async () => {
    mockedAxios.get
      .mockRejectedValueOnce({ response: { status: 502 } })
      .mockResolvedValueOnce({ status: 200, data: '<html>OK</html>' });

    const { scrapeWithRetry } = await import('../src/brightdata/scraper');
    const html = await scrapeWithRetry('https://example.com');
    expect(html).toContain('OK');
    expect(mockedAxios.get).toHaveBeenCalledTimes(2);
  });
});

Step 5: Package Scripts

{
  "scripts": {
    "dev": "tsx watch src/index.ts",
    "scrape": "tsx src/index.ts",
    "test": "vitest",
    "test:watch": "vitest --watch",
    "test:live": "BRIGHTDATA_LIVE=1 vitest --testPathPattern=integration"
  }
}

Output

Proxy config module with geo-targeting support
Response cache to avoid burning credits during development
Mocked test suite that doesn't require live proxy access
Live integration test flag (BRIGHTDATA_LIVE=1)

Error Handling

Error	Cause	Solution
`Missing BRIGHTDATA_* vars`	No `.env.local`	Copy from `.env.example`
Cache stale	Old cached HTML	Delete `.scrape-cache/` directory
Mock not working	Import order	Use `vi.mock()` before dynamic imports
SSL errors in tests	CA cert path	Tests use mocks, not live proxy

Resources

Next Steps

See brightdata-sdk-patterns for production-ready code patterns.