Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page.
PHP 100.00%
crawler's Introduction
TO USE:
1. Edit config.PHP with appropriate database and domain information
2. (for now) in phpMyAdmin insert the seed URL into the urls table.
* URL should be something like: www.fcc.gov
* URL should have a trailing slash
* (for now) May also want to set clicks to '0' to avoid problems
3. Open crawler.php
4. (optional) open stats.php to watch progress
TIPS:
Changes to php.ini
1. Increase memory limit (1GB)
2. Remove execution time limit
Changes to mysql.ini
* Increased max query size (to avoid "mysql went away" error)
Additional documentation (source code) in (/source)
Thanks for the script. I am facing some problems to setup it. I am running the script on a linux server, and would like to crawl another server that is a windows one. But it always returns 0 urls to crawl. MySQL setup is correct, and I have also inserted on phpMyAdmin the entry on URLs table. Would there be anything else to be done?
Hi, I'm playing with the tool and have followed instructions for configuration, but when I visit the crawl.php page I get the message "No URL's to Crawl". Am I missing something?
Hi Sir,
I am a computer engineering student and wish to use the crawler posted by you on GitHub link as below: https://github.com/FCC/Crawler
I am facing the issue that even after correctly setting up the database information, I am getting "No URLs to crawl". Please guide me in using the crawler.