#####Scraper for aqicn.org
####Install
#####OS X:
- Install from Python 3.4
- Install the required libraries using pip (see below for libraries)
- Download aqicn.py
- Setup the output directory inside the script
- run the script aqicn.py (you may need to chmod +x)
#####Cent OS:
- You must download and compile Python 3.4
Follow this guide http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/
wget https://www.python.org/ftp/python/3.4.1/Python-3.4.1.tgz --no-check-certificate
tar -xzvf Python-3.4.1.tgz
cd Python-3.4.1
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
make && sudo make altinstall
#PyPi will be installed at /usr/local/bin/pip3.4
sudo /usr/local/bin/pip3.4 install requests
sudo /usr/local/bin/pip3.4 install beautifulsoup4
sudo /usr/local/bin/pip3.4 install python-dateutil
- Download aqicn.py
- Setup the output directory inside the script
- run the script aqicn.py (you may need to chmod +x)
- change the script to look use the python /usr/local/bin/python3.4 instaed of /usr/bin/python
####Output
#####Console It should look like this
Getting the main page
Finding the json
There are 3181 cities
Scraping Weifang () 1 of 3181 ['36.706774', '119.161756'] 1510
Saved cur_pm25,cur_pm10,cur_o3,cur_no2,cur_so2,cur_co,aqi for city Weifang ()
Errors will be printed as they occur. Sometimes unicode handling in Python has some issues. #####Files Files will be saved to
[directory]/Data/AQICN/Variable/YYYY/MM/yyyymmdd-variable.csv
You can change the base [directory] by editing the script. It is defaulted to "./"
####Troubleshooting
If you UnicodeException you made need to set the enviroment variable PYTHONIOENCODING to utf-8
###Written in Python 3.4
####Libraries Uses PyPi Versions of
- Requests2.2.1: pip install requests
- BeautifulSoup4.3.2: pip install beautifulsoup4
- python-dateutil: pip install python-dateutil
Just run the command after the colon to install them with pip. pip is installed with Python 3.4
###Developed in Eclipse ####Eclipse Plugins