This is a Python crawler to check the newly available houses on Redfin website.
-
Update the target URL in
redfinIrvine.py
as you like.Redfin has a special
region id
for each city, e.g. Irvine is 9361.URL = "https://www.redfin.com/city/9361/CA/Irvine/filter/property-type=house+condo+townhouse,max-price=750k,max-beds=3,min-baths=2"
Or you can go with zipcode:https://www.redfin.com/zipcode/94086
-
Update
base_path
andwork_path
inredfinIrvine.py
on your machine. -
Use
crontab
(on Mac or Linux) to schedule the Python scripts. e.g.:0 10,16 * * * /anaconda3/bin/python3 /Users/zixxu/Documents/Learning/crawler/redfinIrvine.py >| /Users/zixxu/Documents/Learning/crawler/log/redfinIrvine.log 2>&1 5 13 * * * /anaconda3/bin/python3 /Users/zixxu/Documents/Learning/crawler/storeData.py >| /Users/zixxu/Documents/Learning/crawler/log/storeData.log 2>&1
Note: ">|" means overwrite, need
noclobber
flag on (set -o | grep noclobber
) -
There will be a
.csv
file downloaded which will be processed by Python scripts. Then host the webpage through Nginx:sudo nginx
And go to
http://localhost:[port]
to see the crawler results. You can edit the Nginx config file at/usr/local/etc/nginx/
.
- Server side: Use an actual server instead of local computer.
We can use AWS to hold the Python scripts and Webpages, also move
crontab
scheduler there. - Client side: Use React.js instead of web component.