scrapingbee

ScrapingBee CLI

Command-line client for the ScrapingBee API: scrape URLs (single or batch), crawl sites, check usage and credits, and use Google, Fast Search, Amazon, Walmart, YouTube, and ChatGPT from the terminal.

Requirements

• Python 3.10+

Setup: Install (below), then authenticate (Configuration). You need a ScrapingBee API key before any command will work.

Installation

Recommended — install with uv (no virtual environment needed):

curl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install scrapingbee-cli

Alternative — install with pip in a virtual environment:

pip install scrapingbee-cli

From source: clone the repo and run pip install -e . in the project root.

Configuration

You need a ScrapingBee API key:

1. scrapingbee auth – Validate and save the key to config (use --api-key KEY for non-interactive; --show to print config path).
2. Environment – export SCRAPINGBEE_API_KEY=your_key
3. .env file – In the current directory or ~/.config/scrapingbee-cli/.env

Remove the stored key with scrapingbee logout. Get your API key from the ScrapingBee dashboard.

Usage

scrapingbee [command] [arguments] [options]

• scrapingbee --help – List all commands.
• scrapingbee [command] --help – Options and parameters for that command.

Options are per-command. Each command has its own set of options — run scrapingbee [command] --help to see them. Common options across batch-capable commands include --output-file, --output-dir, --input-file, --input-column, --concurrency, --output-format, --overwrite, --retries, --backoff, --resume, --update-csv, --no-progress, --extract-field, --fields, --smart-extract, --deduplicate, --sample, --post-process, --on-complete, --scraping-config, and --verbose. For details, see the documentation.

Parameter values: Choice parameters accept both hyphens and underscores interchangeably (e.g. --sort-by price-low and --sort-by price_low both work).

Commands

Command	Description
usage	Check credits and max concurrency
auth / logout	Save or remove API key
docs	Print docs URL; --open to open in browser
scrape [url]	Scrape a URL (HTML, JS, screenshot, extract)
crawl	Crawl sites following links, with AI extraction and save-pattern filtering
google / fast-search	Search SERP APIs
amazon-product / amazon-search	Amazon product and search
walmart-search / walmart-product	Walmart search and product
youtube-search / youtube-metadata	YouTube search and video metadata
chatgpt	ChatGPT API (--search true for web-enhanced responses)
export	Merge batch/crawl output to ndjson, txt, or csv (with --flatten, --columns)
schedule	Schedule commands via cron (--name, --list, --stop)
tutorial	Interactive step-by-step guide to CLI features (--chapter N, --reset, --list, --output-dir)

Batch mode: Commands that take a single input support --input-file (one line per input, or .csv with --input-column) and --output-dir. Use --output-format csv or --output-format ndjson to stream all results to a single file (or stdout) instead of individual files. Add --deduplicate to remove duplicate URLs, --sample N to test on a subset, or --post-process 'jq .title' to transform each result. Use --resume to skip already-completed items after interruption. Run bare scrapingbee --resume to discover incomplete batches in the current directory.

Parameters and options: Use space-separated values (e.g. --render-js false), not --option=value. For full parameter lists, response formats, and credit costs, see scrapingbee [command] --help and the ScrapingBee API documentation.

Key features