From tavily
Discovers and lists URLs on websites via Tavily CLI without extracting content. Use to find specific pages on large sites, map site structure, list pages, or prepare for targeted extraction.
npx claudepluginhub tavily-ai/skills --plugin tavilyThis skill is limited to using the following tools:
Discover URLs on a website without extracting content. Faster than crawling.
Discovers and lists URLs on a website using firecrawl map CLI via Bash, with optional search filtering to find specific pages or map site structure before scraping or crawling.
Crawls websites to extract content from multiple pages via Tavily CLI. Saves pages as local markdown files with depth/breadth limits, path filtering, and semantic instructions. Use for bulk doc downloads or site content collection.
Conducts multi-round deep research on GitHub repos via API and web searches, generating markdown reports with executive summaries, timelines, metrics, and Mermaid diagrams.
Share bugs, ideas, or general feedback.
Discover URLs on a website without extracting content. Faster than crawling.
If tvly is not found on PATH, install it first:
curl -fsSL https://cli.tavily.com/install.sh | bash && tvly login
Do not skip this step or fall back to other tools.
See tavily-cli for alternative install methods and auth options.
# Discover all URLs
tvly map "https://docs.example.com" --json
# With natural language filtering
tvly map "https://docs.example.com" --instructions "Find API docs and guides" --json
# Filter by path
tvly map "https://example.com" --select-paths "/blog/.*" --limit 500 --json
# Deep map
tvly map "https://example.com" --max-depth 3 --limit 200 --json
| Option | Description |
|---|---|
--max-depth | Levels deep (1-5, default: 1) |
--max-breadth | Links per page (default: 20) |
--limit | Max URLs to discover (default: 50) |
--instructions | Natural language guidance for URL filtering |
--select-paths | Comma-separated regex patterns to include |
--exclude-paths | Comma-separated regex patterns to exclude |
--select-domains | Comma-separated regex for domains to include |
--exclude-domains | Comma-separated regex for domains to exclude |
--allow-external / --no-external | Include external links |
--timeout | Max wait (10-150 seconds) |
-o, --output | Save output to file |
--json | Structured JSON output |
Use map to find the right page, then extract it. This is often more efficient than crawling an entire site:
# Step 1: Find the authentication docs
tvly map "https://docs.example.com" --instructions "authentication" --json
# Step 2: Extract the specific page you found
tvly extract "https://docs.example.com/api/authentication" --json
extract or crawl for content.--instructions for semantic filtering when path patterns aren't enough.