luccavallari / opencrawler Goto Github PK

Welcome to the OpenCrawler wiki! OpenCrawler is an open source web scraper written in PHP and in a standalone file.

Open Crawler is a web spider (which can be freey interpretated as the spider that just we know, because of the form of the WWW just like the web of the spider) which does go all around the web and catch detailed informations for every web site is just visiting, just like Google do when it does index your website (or also like Bing or Yahoo).

Open Crawler was born as an Open Source project (so any programmer can freely contribute or modify the source code), to let people to have an indexer by their own, freely (and for free, of course) licensed with the GNU GPL license. It can be used in many area that can be from a personal Search Engine (just like Google) to a small website search (so you don't have to use any third part search engines into your website).

Any person who like the World Wide Web, and of course all the professional programmers, or not ;)

require_once 'OpenCrawler.php';
$OpenCrawler = new OpenCrawler();
$OpenCrawler -> loadUrl('http://www.example.com/');

// magic things...

// Debug informations
echo '<pre>';
print_r($OpenCrawler);