web-scraper

Scraper App

Intelligent document and web content extraction SDK. Parse web pages, Excel spreadsheets, PowerPoint presentations, Python source files, and more into clean, structured, LLM-ready output.

Features

• Web scraping - Extract articles from any blog or news site (RSS, sitemap, HTML)
• Document parsing - Excel (.xlsx/.xls/.csv), PowerPoint (.pptx), Python (.py), PDF
• LLM-ready output - Markdown, plain text, token counts, and RAG-ready chunks
• Smart routing - Auto-detects input type (URL, file, directory) and dispatches to the right parser
• Batch processing - Parallel multi-document parsing with concurrency control
• 92.2% F1 score - Mozilla Readability extraction validated against the Dragnet benchmark

Installation

npm install @tyroneross/scraper-app

Quick Start

Extract a web article

import { extractArticle } from '@tyroneross/scraper-app';

const article = await extractArticle('https://example.com/blog/post');
console.log(article.title);
console.log(article.markdown);
console.log(`${article.wordCount} words, ${article.readingTime} min read`);

Parse any document (auto-detect)

import { parse } from '@tyroneross/scraper-app';

// URL → web scraper
const web = await parse('https://example.com/article');

// Excel file → structured tables
const excel = await parse('./data/report.xlsx');

// PowerPoint → slides with notes
const pptx = await parse('./deck.pptx');

// Python source → functions, classes, docstrings
const py = await parse('./scripts/main.py');

// Directory → batch parse all supported files
const dir = await parse('./documents/');

LLM-ready output

import { scrapeForLLM } from '@tyroneross/scraper-app/llm';

const { markdown, tokens, chunks, frontmatter } = await scrapeForLLM(url);
// tokens: estimated count for context window management
// chunks: pre-split for RAG applications

Batch processing

import { parseMultiple } from '@tyroneross/scraper-app';

const results = await parseMultiple(
  ['./report.xlsx', './deck.pptx', 'https://blog.example.com/post'],
  { concurrency: 4 }
);

API Reference

Core Functions

Function	Description
parse(input)	Auto-detect input type and parse (URL, file, directory, raw HTML)
parseMultiple(inputs, opts)	Parse multiple inputs in parallel
extractArticle(url)	Extract a single article from a URL
scrapeWebsite(url, opts)	Discover and extract multiple articles from a site
smartScrape(url)	Auto-detect single article vs. listing page

Module Exports

Import	Use For
@tyroneross/scraper-app	Core: parse, parseMultiple, extractArticle, scrapeWebsite, smartScrape
@tyroneross/scraper-app/llm	LLM output: scrapeForLLM, toLLMFormat, estimateTokens
@tyroneross/scraper-app/batch	Batch: scrapeUrls, extractArticles
@tyroneross/scraper-app/parsers	Direct access: parseExcelFile, parsePptxFile, parsePythonFile
@tyroneross/scraper-app/cache	Caching: createCache, MemoryCache, FileCache
@tyroneross/scraper-app/validation	Validation: validateUrl, canScrape, isValidUrl
@tyroneross/scraper-app/testing	Testing: createMockScraper, enableMockMode
@tyroneross/scraper-app/debug	Debug: enableDebugMode, DebugSession
@tyroneross/scraper-app/react	React hook: useScraper
@tyroneross/scraper-app/express	Express router: createScraperRouter
@tyroneross/scraper-app/optimizations	Performance: createConnectionPool, parallelFetch, fastExtract

Supported Formats

Format	Extensions	Parser
Web pages	URLs (http/https)	Mozilla Readability + Cheerio
Excel	.xlsx, .xls, .csv, .tsv, .xlsb, .ods	SheetJS (single-pass optimized)
PowerPoint	.pptx	Custom ZIP + regex parser (up to 14x faster than node-pptx-parser)
Python	.py	Static analysis (regex-based, no runtime needed)
PDF	.pdf	pdf-parse
HTML	Raw HTML strings	Cheerio + Readability

Performance

The document parsers have been rebuilt from scratch for speed:

Parser	Speedup vs. v1	Method
PowerPoint	5-14x faster	Single-pass ZIP, regex-first XML extraction, chart/diagram support
Excel	1-1.6x faster	Single-pass processing (builds rows, markdown, CSV simultaneously)

Benchmarked across 24 test files with 100% accuracy.

Examples

Discover articles from a blog