Fetches any URL and converts to markdown using baoyu-fetch CLI (Chrome CDP with adapters for X/Twitter, YouTube transcripts, Hacker News, Defuddle). Handles login/CAPTCHA waits for saving webpages.
npx claudepluginhub jimliu/baoyu-skills --plugin baoyu-skillsThis skill uses the workspace's default tool permissions.
Fetches any URL via `baoyu-fetch` CLI (Chrome CDP + site-specific adapters) and converts it to clean markdown.
references/adapters.mdreferences/config/first-time-setup.mdreferences/quality-gate.mdscripts/baoyu-fetchscripts/bun.lockscripts/lib/adapters/generic/index.tsscripts/lib/adapters/hn/index.tsscripts/lib/adapters/index.tsscripts/lib/adapters/types.tsscripts/lib/adapters/x/article.tsscripts/lib/adapters/x/index.tsscripts/lib/adapters/x/login.tsscripts/lib/adapters/x/match.tsscripts/lib/adapters/x/payloads.tsscripts/lib/adapters/x/session.tsscripts/lib/adapters/x/shared.tsscripts/lib/adapters/x/single.tsscripts/lib/adapters/x/thread-loader.tsscripts/lib/adapters/x/thread.tsscripts/lib/adapters/x/types.tsConverts webpage URLs to Markdown files using standard, AI-optimized (with YAML frontmatter, summaries, key points), or dual modes. Saves locally for archiving web documents or AI context processing.
Extracts clean Markdown from any URL using ezycopy CLI. Handles JS-rendered pages with headless Chrome, retries on failure, and auto-installs tool if needed.
Converts web URLs to cleaned Markdown with site-specific routing: r.jina.ai for general/X/Twitter pages, defuddle.md for YouTube, browser-impersonated extraction for WeChat/Zhihu/Feishu using Mozilla Readability and Turndown.
Share bugs, ideas, or general feedback.
Fetches any URL via baoyu-fetch CLI (Chrome CDP + site-specific adapters) and converts it to clean markdown.
When this skill prompts the user, follow this tool-selection rule (priority order):
AskUserQuestion, request_user_input, clarify, ask_user, or any equivalent.Concrete AskUserQuestion references below are examples — substitute the local equivalent in other runtimes.
Important: The CLI source is vendored in {baseDir}/scripts/lib. scripts/package.json installs only third-party runtime dependencies.
Agent Execution Instructions:
{baseDir}${BUN} runtime: if bun installed → bun; else suggest installing Bun{baseDir}/scripts/node_modules does not exist, run ${BUN} install --cwd {baseDir}/scripts${READER} = {baseDir}/scripts/baoyu-fetch${READER} in this document with the resolved valueCheck EXTEND.md in priority order — the first one found wins:
| Priority | Path | Scope |
|---|---|---|
| 1 | .baoyu-skills/baoyu-url-to-markdown/EXTEND.md | Project |
| 2 | ${XDG_CONFIG_HOME:-$HOME/.config}/baoyu-skills/baoyu-url-to-markdown/EXTEND.md | XDG |
| 3 | $HOME/.baoyu-skills/baoyu-url-to-markdown/EXTEND.md | User home |
| Result | Action |
|---|---|
| Found | Read, parse, apply settings |
| Not found | MUST run first-time setup (see below) — do NOT silently create defaults |
EXTEND.md supports: download media by default, default output directory.
When EXTEND.md is not found, you MUST use AskUserQuestion to gather preferences before creating EXTEND.md. NEVER create EXTEND.md with silent defaults. Generation is BLOCKED until setup completes. Batch all three questions into a single call:
imgs/ and videos/./url-to-markdown/{domain}/{slug}.md~/.baoyu-skills/ (all projects).baoyu-skills/ (this project only)After answers, write EXTEND.md, confirm "Preferences saved to [path]", then continue.
Full template: references/config/first-time-setup.md.
| Key | Default | Values | Description |
|---|---|---|---|
download_media | ask | ask / 1 / 0 | ask = prompt each time, 1 = always, 0 = never |
default_output_dir | empty | path or empty | Default output directory (empty = ./url-to-markdown/) |
EXTEND.md → CLI mapping:
| EXTEND.md key | CLI argument | Notes |
|---|---|---|
download_media: 1 | --download-media | Requires --output to be set |
default_output_dir: ./posts/ | Agent constructs --output ./posts/{domain}/{slug}.md | Agent generates path, not a direct flag |
Value priority: CLI arguments → EXTEND.md → skill defaults.
# Default: headless capture, markdown to stdout
${READER} <url>
# Save to file
${READER} <url> --output article.md
# Save with media download
${READER} <url> --output article.md --download-media
# Wait for interaction (login/CAPTCHA) — auto-detect and continue
${READER} <url> --wait-for interaction --output article.md
# Wait for interaction — manual control (Enter to continue)
${READER} <url> --wait-for force --output article.md
# JSON output
${READER} <url> --format json --output article.json
# Force specific adapter
${READER} <url> --adapter youtube --output transcript.md
| Option | Description |
|---|---|
<url> | URL to fetch |
--output <path> | Output file path (default: stdout) |
--format <type> | Output format: markdown (default) or json |
--json | Shorthand for --format json |
--adapter <name> | Force adapter: x, youtube, hn, or generic (default: auto-detect) |
--headless | Force headless Chrome (no visible window) |
--wait-for <mode> | Interaction wait mode: none (default), interaction, or force |
--wait-for-interaction | Alias for --wait-for interaction |
--wait-for-login | Alias for --wait-for interaction |
--timeout <ms> | Page load timeout (default: 30000) |
--interaction-timeout <ms> | Login/CAPTCHA wait timeout (default: 600000 = 10 min) |
--interaction-poll-interval <ms> | Poll interval for interaction checks (default: 1500) |
--download-media | Download images/videos to local imgs/ and videos/, rewrite markdown links. Requires --output |
--media-dir <dir> | Base directory for downloaded media (default: same as --output directory) |
--cdp-url <url> | Reuse existing Chrome DevTools Protocol endpoint |
--browser-path <path> | Custom Chrome/Chromium binary path |
--chrome-profile-dir <path> | Chrome user data directory (default: BAOYU_CHROME_PROFILE_DIR env or ./baoyu-skills/chrome-profile) |
--debug-dir <dir> | Write debug artifacts (document.json, markdown.md, page.html, network.json) |
CRITICAL: treat default headless capture as provisional. Some sites render differently in headless mode and can silently return low-quality content without failing the CLI.
After every headless run, inspect the saved markdown. See references/quality-gate.md for the full checklist, recovery workflow, and capture-mode table. Read it whenever a run looks suspicious or the user asks about login/CAPTCHA handling.
The agent must construct the output file path — baoyu-fetch does not auto-generate paths.
Algorithm:
default_output_dir or default ./url-to-markdown/example.com){base_dir}/{domain}/{slug}/{slug}.md — each URL gets its own directory so media files stay isolated{slug}-YYYYMMDD-HHMMSS/{slug}-YYYYMMDD-HHMMSS.mdPass the constructed path to --output. Media files (--download-media) are saved into subdirectories next to the markdown file, keeping each URL's assets self-contained.
See references/adapters.md for the adapter catalog (X, YouTube, Hacker News, generic), per-adapter notes, the media download flow (ask / always / never), and the JSON output schema. Read it before answering adapter-specific questions or handling media prompts.
| Variable | Description |
|---|---|
BAOYU_CHROME_PROFILE_DIR | Chrome user data directory (can also use --chrome-profile-dir) |
Troubleshooting: Chrome not found → use --browser-path. Timeout → increase --timeout. Login/CAPTCHA → --wait-for interaction. Debug → --debug-dir to inspect captured HTML and network logs.
Custom configurations via EXTEND.md. See Preferences section above for paths and supported keys.