Crawl any website and save pages as local markdown files. Use when you need to download documentation, knowledge bases, or web content for offline access or analysis. No code required - just provide a URL.
Crawls websites and saves pages as local markdown files for offline access and analysis.
/plugin marketplace add tavily-ai/tavily-plugins/plugin install tavily@tavily-pluginsThis skill inherits all available tools. When active, it can use any tool Claude has access to.
scripts/crawl_url.pyCrawls websites using Tavily Crawl API and saves each page as a separate markdown file in a flat directory structure.
Tavily API Key Required - Get your key at https://tavily.com
Add to ~/.claude/settings.json:
{
"env": {
"TAVILY_API_KEY": "tvly-your-api-key-here"
}
}
Restart Claude Code after adding your API key.
Use this skill when the user wants to:
Execute the crawl script with a URL and optional instruction:
python scripts/crawl_url.py <URL> [--instruction "guidance text"]
https://docs.stripe.com/api)--instruction, -i: Natural language guidance for the crawler (e.g., "Focus on API endpoints only")--output, -o: Output directory (default: <repo_root>/crawled_context/<domain>)--depth, -d: Max crawl depth (default: 2, range: 1-5)--breadth, -b: Max links per level (default: 50)--limit, -l: Max total pages to crawl (default: 50)The script creates a flat directory structure at <repo_root>/crawled_context/<domain>/ with one markdown file per crawled page. Filenames are derived from URLs (e.g., docs_stripe_com_api_authentication.md).
Each markdown file includes:
python scripts/crawl_url.py https://docs.anthropic.com
Crawls the Anthropic docs with default settings, saves to <repo_root>/crawled_context/docs_anthropic_com/.
python scripts/crawl_url.py https://react.dev --instruction "Focus on API reference pages and hooks documentation"
Uses natural language instruction to guide the crawler toward specific content.
python scripts/crawl_url.py https://docs.stripe.com/api -o ./stripe-api-docs
Saves results to a custom directory.
python scripts/crawl_url.py https://nextjs.org/docs --depth 3 --breadth 100 --limit 200
Increases crawl depth, breadth, and page limit for more comprehensive coverage.
TAVILY_API_KEY environment variable (loads from .env if available)<repo_root>/crawled_context/<domain>/ directory regardless of original URL hierarchyUse when working with Payload CMS projects (payload.config.ts, collections, fields, hooks, access control, Payload API). Use when debugging validation errors, security issues, relationship queries, transactions, or hook behavior.