jizhang27 / quickflix-crawler Goto Github PK

These urls are all aggregated by google, however our crawler could not found

• http://www.metacritic.com/movie/the-magnificent-seven
• http://www.newyorker.com/goings-on-about-town/movies/the-magnificent-seven
• http://www.nytimes.com/2016/09/23/movies/magnificent-seven-review-denzel-washington.html?referrer=google_kp&_r=0
• http://www.avclub.com/review/magnificent-seven-gets-uninspired-remake-242722

Please help to add more entires following format:
'url1',
'url2',

Prioritises certain domains over others depending on whether it contains reviews / whether it is useful (based on some metric)

fetch_html(): request timeout and bad URLs etc
parse_review(): error loading string into json

Pic
Description
Release date:
Director:
Actors:
Producers:

• Crawler pops URLs from the queue, visits it (or not) and parses the review (if the page has one) and adds new URLs to the queue
• Request rate controlled by some time-delay
• Needs #3 to be done

The crawler needs access to:

• list of URLs that have already been visited
• queue of URLs that have yet to be visited

And of course:

• data tables for movie reviews

To check if the requests library supports it. Otherwise need to implement ourselves.

need to do check by url before saving review to db