Zyte Web Data for Claude Code
From a plain-English prompt to a working Scrapy spider.
Install
claude plugin marketplace add zyte-ai/claude-skills
claude plugin install zyte-web-data@zyte-ai
If Claude Code is already running, reload plugins in the active session:
/reload-plugins
See also: Discovering and installing plugins
What it does
This is Zyte's official Claude Code plugin that generates production-ready Scrapy spiders with web-poet page objects from a plain-English prompt. Give it a URL and describe what you want to extract. It handles site exploration, schema discovery, code generation, and smoke testing: no boilerplate, no manual selector hunting.
The plugin explores the target site, discovers available fields, and presents a schema for your approval before generating a single line of code. After you confirm the schema, it creates a Scrapy project with all dependencies configured, generates web-poet page objects and test fixtures, wires up the spider, and runs a smoke test to verify that extraction is working before handing the project back to you.
Optionally, use /scrape-scrapy-cloud to deploy directly to Scrapy Cloud for scheduled runs, job history, and monitoring. A free tier is available.
Use cases
The /scrape skill works on any website with repeating structured content: detail pages linked from a listing or category page. Examples from the skill:
- Product catalogs
- Job listings
- Recipes
How does it work?
The /scrape skill orchestrates five stages automatically:
1. Decide which fields to extract → /scrape-define
2. Analyze the website → /scrape-spec
3. Create the Scrapy project → /scrape-ensure-project
4. Generate the extraction code → /scrape-codegen
5. Generate the spider → /scrape-create-spider
Each stage feeds directly into the next. When the pipeline completes, you have a runnable spider and a passing test suite:
uv run scrapy crawl <spider_name>
uv run pytest fixtures/
Skills
Orchestration
| Skill | Description |
|---|
scrape | End-to-end web scraping workflow — from URL to working spider with web-poet page objects |
Pipeline stages (called automatically by /scrape)
| Skill | Description |
|---|
scrape-define | Quick schema definition: explore one detail page, discover fields, fast approval loop |
scrape-spec | Explore diverse pages and validate the extraction spec: downloads pages, compares variants, optional browser review |
scrape-explore-site | Explore a website to find and save diverse pages (start, list, detail) with classified links |
scrape-analyze-page | Extract all available fields with values from a detail page |
scrape-ensure-project | Ensure a Scrapy project exists with scrapy-poet and Zyte API support |
scrape-codegen | Generate web-poet page object code from an extraction spec |
scrape-codegen-analyze | Analyze an HTML page to produce field extraction instructions for code generation |
scrape-codegen-generate | Generate web-poet page object code from per-page extraction analyses |
scrape-create-spider | Generate a Scrapy spider that wires page objects together |
Utilities
| Skill | Description |
|---|
scrape-add-page-object | Add an empty web-poet page object to a Scrapy project |
scrape-review-schema | Generate an HTML review page for schema and extracted data verification |
Deployment
| Skill | Description |
|---|
scrape-scrapy-cloud | Deploy projects, schedule spiders, list/stop jobs, and view items or logs on Scrapy Cloud |
scrape-zyte-login | Set up your Zyte account and credentials |
Prerequisites
- Claude Code (CLI or desktop app)
uv — used to create and manage the Scrapy project
Project dependencies (scrapy, scrapy-poet, scrapy-zyte-api, web-poet, extruct, price-parser, pytest) are installed automatically by the skills.
Quickstart
Any scraping prompt triggers the skill automatically. For example:
/scrape https://books.toscrape.com/ products
