habitats / crawlie Goto Github PK

#Crawlie – A general purpose web crawler

##Features

• Custom priority/heuristic functionality to guide the crawler through the web
• Never go the same place twice!
• Massive parallelization, decide for yourself how far you want to go!
• Tree traversal back to seed from any discovered page
• Automatically store your results to a neat database (SQLite as of now)
• Download discovered files based on suffix, URL or regex matching
• Simple graphical user interface
• Automatically save your current session and all working data and resume later!
• Loosely coupled and easily maintainable architecture

##How to use ###Jar file Download the jar, configure the properties file and run. Source Download the source, the required libraries and run the Crawlie.java class. Required libraries

• guava-16.0.1
• jsoup-1.7.3
• sqlite-jdbc-3.7.2

##How it works The crawler starts from a seed, then checks all the links on that page, and continues to the url that yielded the highest priority given the configuration. This can be searching for pdf-files, url that include “hello” etc. It will periodically store all the analyzed pages to a database. Due to the database becoming insanely heavy I’ve chosen not to include the source files. The crawler can also mark files/urls for download. For instance, one scenario could be: Go to hello.com and download all the images on that domain.

##Potential weaknesses

• Requires massive amounts of computer power if you intend to use it extensively
• High, monotonically increasing, memory usage due to having to cache discovered URLs in order to not enter a crawler loop

##Future improvements The improvements and potential is highly dependent on the line of work the crawler is used for. However, some specifics might be:

• Implement a GUI for the configuration
• Use big data analytics to make sense of data collected
• Implement a relevance filter to prioritize pages based on content rather than URL

##How to expand Expansion is easy. It’s already heavily parallelized with SWING (the GUI) being the biggest bottleneck. If instead of writing to a local database, decentralized instances could write to one common one. Depending on the implementation this could be very suitable for clusters and other multi- and/or super-computers.