webpatser / dusk Goto Github PK

Laravel Dusk is an incredibly simple to use browser automation and testing tool. However, it was made with testing in mind, and as such, was never really meant for the production environment limelight.

Not anymore. I've gone ahead and ripped out all the parts that expose routes, or interact with your users.

So we can scrape the web with an amazingly expressive API in a framework we all know and love! All the methods you know and love -- minus the ones on interacting with or acting as users -- are there. See the official docs below for more details.

When using this tool as a scraper, you're typically limited to making dusk tests, and then triggering them via php artisan dusk.

You can automate this process and apply them to a command bus you can schedule!

Here's how:

Make your dusk test as you have before. Then, make a new command, making sure to give it a unique signature.

In the handle method, simply new up the class of the test you just created, initialize it and run it:

/**
 * Execute the console command.
 *
 * @return mixed
 */
public function handle()
{
    $event = new \Tests\Browser\Test;
    $event->prepare();
    $event->testExample();
}

Do whatever you had planned with the data you're scraping inside the test itself and you're golden.

When you make a test for dusk, make sure you include $browser->quit(); to ensure that the browser cleans up after itself. Do this after you've processed the data you've scraped.

I'll make a 2.0 branch in the coming days to keep pace with the official package. It brings headless operation which will really help with some serious scraping.

Just kidding! Version 1.1.0 upgrades the binary and allows for headless operation.

To run your scraping headless, check out DuskTestCase.php, make sure to add ChromeOptions to your use statements, and set the $options variable and pass it through to the RemoteWebDriver like pictured below.

use Facebook\WebDriver\Chrome\ChromeOptions;

/**
 * Create the RemoteWebDriver instance.
 *
 * @return \Facebook\WebDriver\Remote\RemoteWebDriver
 */
protected function driver()
{
    $options = (new ChromeOptions)->addArguments(['--headless']);

    return RemoteWebDriver::create(
        'http://localhost:9515', DesiredCapabilities::chrome()->setCapability(
            ChromeOptions::CAPABILITY, $options
        )
    );
}

If anyone wants to help kick in a PR.

Documentation for Dusk can be found on the Laravel website.

Dusk-Secure is open-sourced software licensed under the MIT license