This is a mini size crawler made by Python3. It's easy to use and extend. Good for study. Simply crawler every item on a page and save it into local csv file. Then go to the next page.
- Save every item on the page into cvs and then go to the next page
- Pause and rerun
- Suppoer almost all kernels (PhantomJS, Firefox etc.)
- Also can use Requests for those static pages, speed up
- requests
- selenium
- time
- csv
- os
- pickle
- datetime
- shutil
- numpy
- pandas
- BeautifulSoup4
- re
- math
- Easy to maintain
- Simple enough to use and extend
- Less Memory leak
minicrawler/spider/basespider.py
- crawl
- getTotalKeys (leave it to sub class)
- getCurrentPage (leave it to sub class)
- gotoNextPage (leave it to sub class, optional)
- saveCurrentIndex
- saveContentToWorkbook
minicrawler/provider/requestor.py
minicrawler/provider/webbrowser.py
- load
- getContent (leave it to different kernel)
- navigate (leave it to different kernel)
- quit
A Wikipedia (static page)
- https://en.wikipedia.org/wiki/List_of_colleges_and_universities_in_the_United_States_by_endowment
- Static page
- Only one page
minicrawler/spider/richspider.py
B US news (dynamic with url pagination)
- http://colleges.usnews.rankingsandreviews.com/best-colleges/rankings/national-universities
- Dynamic page
- Change url to next page
minicrawler/spider/nuspider.py
C Startclass (dynamic with button pagination)
- http://faculty-salaries.startclass.com/
- Dynamic page
- Click the pagenition button for the next page
minicrawler/spider/salaryspider.py