docpull

docpull

Public web to agent-ready Markdown. Fast, local, browser-free.

docpull is a Python CLI, SDK, and MCP server for pulling public static and server-rendered web pages into clean Markdown, NDJSON, OKF bundles, SQLite, or local archives. Documentation crawling is its sharpest default workflow, but the product is broader than docs: blogs, API references, vendor pages, OpenAPI specs, and other server-rendered web content all fit. It uses async HTTP instead of Playwright, preserves source metadata, and is built for agent-selected URLs with SSRF, XXE, DNS-rebinding, and CRLF-injection protections enabled by default.

Use docpull when you need to:

• Load public web pages, docs, API references, or blogs into an LLM context.
• Build inspectable source corpora for RAG/search pipelines.
• Give a coding agent a local MCP tool for fetching, caching, grepping, and reading docs.
• Mirror public web content for offline work while keeping source attribution.

docpull intentionally does not render JavaScript. For JS-only apps, browser automation, paid extraction APIs, or general web crawling, see docs/scraping-boundary.md and docs/alternatives.md.

Try it in 60 seconds

pip install docpull
docpull https://docs.python.org/3/library/asyncio.html --single
docpull https://docs.python.org/3/library/asyncio.html --profile llm --stream | jq .

Install

pip install docpull

# Optional extras
pip install 'docpull[llm]'           # tiktoken for token-accurate chunking
pip install 'docpull[trafilatura]'   # alternative extractor for noisy pages
pip install 'docpull[mcp]'           # run as an MCP server for AI agents
pip install 'docpull[parallel]'      # Parallel API context packs
pip install 'docpull[observability]' # Raindrop benchmark tracing
pip install 'docpull[all]'           # everything above

Quick start

# Scrape a page graph and save Markdown
docpull https://docs.example.com

# Scrape one page, no crawl — the fast path for agents
docpull https://docs.example.com/guide --single

# LLM-ready NDJSON with 4k-token chunks streamed to stdout
docpull https://docs.example.com --profile llm --stream | jq .

# Open Knowledge Format bundle for agent/wiki interoperability
docpull https://example.com --format okf -o ./site-okf

# Mirror scraped content for offline use
docpull https://docs.example.com --profile mirror --cache

Project traction

Framework-aware extraction

docpull inspects each page before running the generic extractor and can pull content directly from framework data feeds: