lc:generate-scraper

Generate Larascraper Scraper

Create a Larascraper scraper that fetches a target URL with Puppeteer and parses it. The action chain is a little query builder for the page: chain browser actions (click, type, wait, waitForSelector, scroll, ...), branch and loop with conditional flow (when() / repeatUntil() + Condition), solve simple captchas (solveCaptcha()), and download files/PDFs (FileScraper + submitAndCapture()).

Subcommands

Subcommand	Description
(no argument)	Prompt for the scraper name and the target URL / what to scrape.
[ScraperName]	Generate the scraper class, then ask for the target URL.
[ScraperName] [url]	Generate the scraper for the given URL and infer the selectors/actions.

This is a generator skill -- it always creates files. There is no analyze-only mode.

Process

Step 0: Verify Larascraper Is Installed

1. Use Grep to check if edulazaro/larascraper exists in composer.json.
2. If NOT found, stop and inform the user:

   Larascraper is not installed. Install it with:
   composer require edulazaro/larascraper
   php artisan larascraper:install   # installs the Node packages Puppeteer needs
   

3. If found, also confirm the Node side is ready (the scraper shells out to Node + Puppeteer). If node_modules/puppeteer-extra is missing, tell the user to run php artisan larascraper:install. The runner will otherwise fail with a clear message, but it is better to catch it up front.

Step 1: Read the Base Scraper Class

Read the installed package from vendor to match the actual installed version, not assumptions:

1. Use Read to load vendor/edulazaro/larascraper/src/Scraper.php and the Concerns/BuildsActions.php trait (the action methods live there). Note:
   • The fluent config methods: scrape(), proxy(), timeout(), headers(), retry(), run().
   • The browser action methods: click(), clickAndWait(), type(), select(), hover(), press(), waitForSelector(), waitForNavigation(), wait(), scroll() / scrollToBottom().
   • Conditional flow (2.1+): when($condition, $then, $else) and repeatUntil($condition, $body, max:, delay:), plus the Support\Condition helper (Condition::selectorExists/selectorMissing/textContains/urlContains/captured).
   • Captcha (2.1+): solveCaptcha($imageSelector, $inputSelector, $options).
   • File downloads (2.2+): the concrete EduLazaro\Larascraper\FileScraper class, the submitAndCapture($formSelector, $options) action, and $result->file / $result->contentType on the response.
   • That handle() parses $this->crawler (a Symfony\Component\DomCrawler\Crawler).
2. Check what the installed version actually exposes (methods/classes may be absent in older versions). If Concerns/BuildsActions.php or FileScraper.php are missing, the install predates those features: use only what is present and tell the user to composer update edulazaro/larascraper for the rest.

Step 2: Parse Arguments

• Scraper name: e.g., BikeScraper. Append Scraper suffix if not already present. Supports subfolders: News/MegaScraper.
• Target URL: the page to scrape. If not provided, ask for it.
• What to extract: ask the user what fields they want (title, price, list of items, ...), unless obvious from the request.

Step 3: Study Existing Scrapers

1. Use Glob to find existing scrapers in app/Scrapers/.
2. If any exist, Read one or two to match conventions: imports, how handle() is structured, return shape, how they are invoked (proxy, retry).
3. Note the project's proxy/retry conventions so the generated usage snippet matches (several projects always chain ->retry(3, 20)->proxy(...)).

Step 4: Generate the Scraper Class

Prefer the package's own generator (it uses the up-to-date stub):

php artisan make:scraper {ScraperName}

Be Docker-aware: if a docker-compose.yml / Sail setup exists, run artisan inside the container (e.g., docker exec -u sail -w /var/www/html <container> php artisan make:scraper {ScraperName}).

This creates app/Scrapers/{ScraperName}.php extending Scraper with a handle() skeleton. If make:scraper is unavailable, Write the file manually:

<?php

namespace App\Scrapers;

use EduLazaro\Larascraper\Scraper;

/**
 * Scrapes a page and returns the parsed data.
 */
class {ScraperName} extends Scraper
{
    /**
     * Parse the fetched page and return the scraped data.
     *
     * @return array
     */
    protected function handle(): array
    {
        return [
            //
        ];
    }
}

Step 5: Inspect the Target Page

Determine the right selectors from the real page, never guess:

1. Quick structural peek with WebFetch on the target URL to understand the markup, OR
2. Better, run the scraper once to get the exact HTML Puppeteer sees (JS already executed), then inspect it. A throwaway run via tinker:

   php artisan tinker --execute="dd(\App\Scrapers\{ScraperName}::scrape('{url}')->run()->html);"
   

   (Docker-aware. run() returns a ScraperResponse, so read ->html for the markup and ->data for the parsed result.) Read the returned HTML to pick stable selectors.
3. Decide whether the content is available on load or needs interaction:
   • Cookie/consent wall blocking content -> click(), or when(Condition::selectorExists('#banner'), fn($b) => $b->click('#accept')) if it only sometimes appears.
   • Content behind a search/filter form -> type() + press('Enter', waitForNavigation: true) or click($submit, waitForNavigation: true).
   • Pagination / "load more" -> click() + waitForSelector(), or repeatUntil(Condition::selectorExists('#end'), fn($b) => $b->clickAndWait('a.next'), max: N).
   • Lazy / infinite-scroll content -> repeatUntil(Condition::selectorExists('#footer'), fn($b) => $b->scrollToBottom()->wait(500), max: N).
   • Content rendered late by JS -> waitForSelector() on the element you need.
   • Simple image captcha guarding the page -> solveCaptcha('#captcha-img', '#captcha-input'), usually inside a repeatUntil because OCR is imperfect (see Step 7). Requires php artisan larascraper:install --captcha.
   • The result is a file/PDF (download, export, document behind a form) -> use FileScraper + submitAndCapture() instead of parsing HTML (see Step 7).

Step 6: Fill handle() (parse the crawler)

handle() only parses $this->crawler. Fill it with the selectors found in Step 5:

protected function handle(): array
{
    return [
        'title' => $this->crawler->filter('h1')->text(''),
        'price' => $this->crawler->filter('.price')->text(''),
    ];
}

For a list of items, map over a node list:

protected function handle(): array
{
    return $this->crawler->filter('.product-card')->each(function ($node) {
        return [
            'name'  => $node->filter('.name')->text(''),
            'price' => $node->filter('.price')->text(''),
            'link'  => $node->filter('a')->attr('href'),
        ];
    });
}

Use ->text('') / ->attr('...') with safe defaults so a missing node does not throw.

Step 7: Build the Usage Snippet With Actions

Actions are chained at the call site (Scraper::scrape($url)-> ... ->run()), not inside handle(). run() returns a ScraperResponse (->success, ->status, ->error, ->html, ->data); the parsed handle() output is in ->data. (In larascraper 1.x run() returned the data directly -- check the installed version from Step 1.) Produce a usage snippet tailored to the site. Examples:

Plain page (no interaction):

$result = {ScraperName}::scrape('{url}')
    ->retry(3, 20)
    ->run();

$items = $result->data; // parsed handle() output (check $result->success first)

Behind a cookie wall + lazy load:

$result = {ScraperName}::scrape('{url}')
    ->click('#accept-cookies')
    ->scrollToBottom()
    ->waitForSelector('.product-card')
    ->run();

Search form:

$result = {ScraperName}::scrape('{url}')
    ->type('#search', 'zelda')
    ->press('Enter', waitForNavigation: true)
    ->waitForSelector('.results')
    ->run();

Pagination ("next" that reloads):

$result = {ScraperName}::scrape('{url}')
    ->clickAndWait('a.next')
    ->waitForSelector('.results')
    ->run();

Conditional flow (only-if + retry loop):

use EduLazaro\Larascraper\Support\Condition;

$result = {ScraperName}::scrape('{url}')
    ->when(                                          // only if the banner is there
        Condition::selectorExists('#cookie-banner'),
        fn ($b) => $b->click('#accept'),
    )
    ->repeatUntil(                                   // load more until the footer shows (bounded)
        Condition::selectorExists('#end-of-list'),
        fn ($b) => $b->scrollToBottom()->wait(500),
        max: 10,
    )
    ->run();

The when()/repeatUntil() closures receive a sub-builder ($b) with the same action methods. repeatUntil() is always bounded by max (and an optional delay between iterations to avoid hammering the server).

Simple captcha (read image, retry until accepted):

use EduLazaro\Larascraper\Support\Condition;

$result = {ScraperName}::scrape('{url}')
    ->repeatUntil(
        Condition::selectorMissing('#captcha-img'),  // stop once the captcha is gone
        fn ($b) => $b
            ->solveCaptcha('#captcha-img', 'input[name=captcha]')
            ->clickAndWait('#submit'),
        max: 6,
        delay: 1000,
    )
    ->run();

solveCaptcha() needs the optional OCR packages: php artisan larascraper:install --captcha.

Downloading a file / PDF (use FileScraper, read $result->file):

use EduLazaro\Larascraper\FileScraper;

$result = FileScraper::scrape('{url}')
    ->submitAndCapture('form', ['expect' => 'application/pdf'])
    ->run();

if ($result->success && $result->file) {
    file_put_contents('document.pdf', $result->file);   // raw bytes; type in $result->contentType
}

For a file behind a captcha, combine both with Condition::captured() (stop once the file is grabbed):

FileScraper::scrape('{url}')
    ->repeatUntil(
        Condition::captured(),
        fn ($b) => $b->solveCaptcha('#captcha-img', 'input[name=captcha]')
                     ->submitAndCapture('form', ['expect' => 'application/pdf']),
        max: 8, delay: 500,
    )
    ->run();

FileScraper is used directly (no class to write); the bytes land in $result->file, not $result->data.

Match the project's proxy/retry conventions from Step 3 when present.

Step 8: Verify

1. Read the generated class to confirm it is correct.
2. Run php -l on the file (Docker-aware).
3. If it is safe to do so, run the scraper once (Step 5 tinker command) and confirm $result->success === true and that $result->data is populated. Refine selectors/actions if any come back empty.
4. If an action fails (e.g., a selector never appears), $result->success is false and $result->error holds the message -- adjust the selector or add a waitForSelector() before it.

Step 9: Show the Final Usage

Show the user the tailored call (Step 7) plus how to read the result, and remind them that handle() receives the HTML after all actions have run.

Actions Reference

These build an ordered list Puppeteer runs in a single browser session, after navigating and before handle() parses the HTML. The waits happen inside Node (where the page is alive), not in PHP.

Method	Use it for
->click($selector)	Click an element (waits for it first).
->click($selector, waitForNavigation: true) / ->clickAndWait($selector)	A click that loads a new page.
->type($selector, $text)	Fill an input.
->select($selector, $value)	Pick a <select> option by value.
->hover($selector)	Reveal hover menus.
->press($key)	Press a key; pass waitForNavigation: true when it submits.
->waitForSelector($selector)	Wait for lazy/JS content to appear.
->waitForNavigation()	Wait for a navigation to finish.
->wait($ms)	Fixed pause in milliseconds.
->scroll('bottom'|'top') / ->scrollToBottom()	Trigger lazy / infinite scroll.
->when($cond, $then, $else?)	Run a branch only if a condition holds (closures receive a sub-builder).
->repeatUntil($cond, $body, max:, delay:)	Repeat a branch until a condition holds. Always bounded by max.
->solveCaptcha($img, $input, $opts?)	OCR a simple image captcha and type it. Needs --captcha install.
->submitAndCapture($form, ['expect'=>...])	Submit a form and capture the response file (use with FileScraper).

$selector is any CSS selector, including attribute selectors like [name=email] or input[name=captcha].

Conditions (EduLazaro\Larascraper\Support\Condition, for when()/repeatUntil()):

Condition	True when...
Condition::selectorExists($selector)	an element matching the selector exists
Condition::selectorMissing($selector)	no element matching the selector exists
Condition::textContains($text, $selector?)	the text is found (in $selector, or the whole page)
Condition::urlContains($text)	the current URL contains the substring
Condition::captured()	a file has been captured by submitAndCapture()

Navigation tip: for a click or key press that loads a new page, use waitForNavigation: true on that action (or clickAndWait()) instead of a separate ->waitForNavigation() -- it arms the wait before the action, avoiding a race.

Important Notes

• handle() is parse-only; it must not perform interactions. All interaction lives in the action chain at the call site.
• For HTML scrapers, generate a class extending Scraper with a handle(). For pure file/PDF downloads, use FileScraper directly (no class, no handle()); the result is in $result->file, not $result->data.
• repeatUntil() must always be bounded: pass a sensible max, and a delay when each iteration hits a remote server, so the loop never hammers it.
• Always derive selectors from the real page (Step 5), never guess from the URL.
• Use ->text('') / ->attr('...') defaults so missing nodes do not throw.
• Respect the target site: keep timeouts and retry counts reasonable, and honor the project's existing proxy usage.
• solveCaptcha() only handles simple image (text) captchas, not reCAPTCHA/hCaptcha. It needs the optional OCR packages (php artisan larascraper:install --captcha).
• If the installed package lacks a method/class, it is an older version -- use what is present and suggest composer update edulazaro/larascraper.