Web crawling with Tantivy full-text search indexing. Supports crawl, search, and auto-crawl. WHEN: User wants to "scrape a website", "crawl documentation", "search crawled content", "index a site". WHEN NOT: Single page fetch (use browser_navigate), web search (use web_search).
/plugin marketplace add cyrup-ai/kodegen-claude-plugin/plugin install kg@kodegenThis skill inherits all available tools. When active, it can use any tool Claude has access to.
mcp__plugin_kg_kodegen__scrape_url crawls websites, saves content as Markdown, and builds a Tantivy full-text search index. Uses an action-based interface with connection isolation and background execution support.
| Action | Description | Required Parameters |
|---|---|---|
SEARCH | Search with auto-crawl (RECOMMENDED) | url, query |
CRAWL | Explicit crawl | url |
READ | Check crawl progress | None |
LIST | Show all active crawls | None |
KILL | Cancel crawl | None |
| Parameter | Type | Default | Description |
|---|---|---|---|
action | string | "CRAWL" | Action to perform |
url | string | null | Target URL (required for CRAWL/SEARCH) |
crawl_id | number | 0 | Crawl instance (0, 1, 2...) |
query | string | null | Search query (SEARCH action) |
max_depth | number | 3 | Maximum crawl depth |
limit | number | null | Max pages to crawl |
await_completion_ms | number | 600000 | Timeout (10 min default) |
crawl_rate_rps | number | 2 | Requests per second |
search_limit | number | 10 | Max search results |
search_offset | number | 0 | Search pagination offset |
search_highlight | boolean | true | Highlight matches |
Auto-crawls if index doesn't exist:
{
"action": "SEARCH",
"url": "https://ratatui.rs",
"crawl_id": 0,
"query": "layout widgets"
}
{
"action": "CRAWL",
"crawl_id": 0,
"url": "https://docs.rs/tokio"
}
{
"action": "CRAWL",
"url": "https://example.com/docs",
"max_depth": 2,
"limit": 50,
"crawl_rate_rps": 1
}
{
"action": "READ",
"crawl_id": 0
}
{ "action": "LIST" }
{
"action": "KILL",
"crawl_id": 0
}
Tantivy supports advanced queries:
| Query Type | Example | Description |
|---|---|---|
| Text | layout components | Search all fields |
| Phrase | "exact phrase" | Exact match |
| Boolean | layout AND widgets | Logical operators |
| Field | title:layout | Search specific field |
| Fuzzy | layot~2 | Allow 2 character differences |
Content saved to .kodegen/citescrape/{domain}/:
.kodegen/citescrape/ratatui.rs/
├── manifest.json # Crawl metadata
├── .search_index/ # Tantivy search index
├── index.md # Homepage
├── tutorials/
│ └── hello-world.md
└── api/
└── widgets.md
SEARCH with url and query (auto-crawls if needed)CRAWL with url, max_depth, limitREADSEARCH actioncrawl_id - use different numbers for parallel crawlsREAD to check on background crawlsCreating algorithmic art using p5.js with seeded randomness and interactive parameter exploration. Use this when users request creating art using code, generative art, algorithmic art, flow fields, or particle systems. Create original algorithmic art rather than copying existing artists' work to avoid copyright violations.
Applies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.
Create beautiful visual art in .png and .pdf documents using design philosophy. You should use this skill when the user asks to create a poster, piece of art, design, or other static piece. Create original visual designs, never copying existing artists' work to avoid copyright violations.