web-scraper

Web Scraper Skill

When you need to interact with a webpage, extract data, or read its content, use the tools provided in this directory. The scraper handles rendering JavaScript via Playwright and writes the output into Markdown files in chunks of 128k characters to avoid context limit issues.

Usage Instructions

1. Determine Environment & check dependencies: Before running the Python scraping script, you MUST decide which dependency scanning script to use based on your operating system:

   • Windows (Command Prompt): Run skills\web-scraper\setup\lib-scan.bat
   • Windows (PowerShell): Run .\skills\web-scraper\setup\lib-scan.ps1
   • Linux / macOS: Run skills/web-scraper/setup/lib-scan.sh

2. Install Dependencies if needed: If the lib-scan script indicates that dependencies are missing (non-zero exit code or "Missing dependencies" message), you MUST run the corresponding install script:

   • Windows (Command Prompt): Run skills\web-scraper\setup\install.bat
   • Windows (PowerShell): Run .\skills\web-scraper\setup\install.ps1
   • Linux / macOS: Run skills/web-scraper/setup/install.sh

3. Run the Scraper: Execute the python script, passing the target URL as an argument:

   python skills/web-scraper/scripts/scrape.py <url>
   

   Optional Parameters:

   • -o, --output: Specify the output Markdown file path (default: scraped_output.md).
   • -p, --port: Port for Chrome debugging via CDP (default: 9222). Set to 0 to disable CDP and run a fresh headless browser.

   Debug Mode Chrome (Auto-Start): By default, the script attempts to connect to Chrome via CDP on port 9222. If the port is not open, it automatically runs start_debug_chrome.bat to launch a dedicated Chrome instance with an isolated profile (chrome-debug-profile). This allows the agent to scrape sites requiring user login without interfering with the user's normal browser. The user can manually log in on this debug browser window if prompted.

4. Read the Output: The script will write the scraped content to the file specified by the -o parameter. If the content exceeds the 128k limit, it splits the output into multiple parts (e.g. scraped_output_part_1.md, scraped_output_part_2.md). Read these generated files to process the data.