-Country of data source: Peru
-Specific coverage within country: Lima

URL: (A) http://peruclima.pe/?p=calidad-de-aire OR (B) http://calidaddelaire.minam.gob.pe/estaciones.php [These appear to be the same sources, separate confirmation would be good though]
-Comments on URL: [Ex. You have to click on x, then y to see values]
For (B), you will need to click on the red dots to go to table like this: http://www.senamhi.gob.pe/?p=0412&txt=112192

-Pollutants measured: PM2.5, PM10, CO, SO2, NO2

-Are units of measurement clearly indicated (e.g. do you see 'AQI', 'ppm' or 'ug/m^3'?) Note: We cannot directly input AQI, API, AQHI, etc. values directly into our system.
Yes, physical measurements (e.g. ppm, ug/m^3) are clearly labelled in tables. Though note for (A), you will see the map shows a presumed AQI, but as soon as you hover over the location, you see a nice table with great labels including timestamp, coords, and units of measurements.

-Comments about units: [ex. 'They are viewable on {this page}, see the table]
None.

-Are there date+timestamps for pollutant measurements?
Yes.

-Are time intervals otherwise indicated on the site (e.g. 'Measurements are 15 minute averages'):
Hourly. It's pretty clear from the tables.

*-Are coordinates for the stations clearly indicated or discernible? Comment. (e.g. do you see a map or a list of coordinates): * [<--I can't get the *bold * function to work here! Weird.}
For (A), they are clearly listed when you hover over a spot. For (B), when you click on a station (little red dot), they are also listed.

-Are you with the host agency and may we contact you?: No.

-Any other comments about this source:
This source looks awesome.
Another source in Lima: #99

[OPTIONAL]Your name/contact info/interest in open air quality data: n/a

@olafveerman 's issue transported to this repository:

Measurements for a lot of Belgian measuring stations: http://www.ircel.be/nl/luchtkwaliteit/metingen
This page shows rolling averages for most parameters.

When you drill down, it's possible to get the actual hourly measurements and not the rolling averages. For example:

go to this page: http://www.ircel.be/en/air-quality/measurements/particulate-matter/pm10?set_language=en
click on table with detailed info per monitoring site
Have not found a programmatic way to access this data, it might need to be scraped.

We had an outage of a few days due to a silly error in the Dockerfile, however there weren't any alerts triggered or anything. So we should figure out how to monitor that we're consistently getting new data added to the system correctly.

We should probably have something that runs periodically and checks to make sure new data has been added but it should probably also be aware of how often a data source is to add data? We don't want to fire off any alarms if we haven't gotten data for 1.5 hours if something only reports every hour, or 24 hours. This could be tied to the better email handling #24.

Currently does not appear to be working ('server busy')

http://oab.ambientebogota.gov.co/es/el-observatorio-y-las-localidades

Taiwan PM2.5:

http://taqm.epa.gov.tw/pm25/en/PM25A.aspx?area=10

Station descriptions (including coordinates) available by clicking on station name on link directly above:
http://taqm.epa.gov.tw/taqm/en/Site/Keelung.aspx

Ozone, as well as PM2.5, is available on the main page, but not sure how it is accessible programmatically:
http://taqm.epa.gov.tw/taqm/en/

For Iran (#51) and possibly other sources, we may have to back-calculate AQI values to pollutant concentration. It seems that Iran uses the EPA's methodology. Should we write a generic tool in lib that does this calculation?

Data

This tool would accept an object, or array of objects with a measurement. Each measurement needs at least the parameter and the value. We can store a reference to the original value and an annotation about the conversion.

{ 
  city: 'Tehran',
  location: 'emam khomeyni',
  parameter: 'so',
  value: 150
}

would become something like:

{ 
  city: 'Tehran',
  location: 'emam khomeyni',
  parameter: 'so',
  unit: 'ppm',
  value: 0.22,
  original: {
    value: 150,
    methodology: 'EPA AQI to concentration'
  }
}

Methodology

On page 16 of this EPA publication:

@RocketD0g Can we use this?

cc @jflasher

This looks like it could be a great addition.

• create schema
• replace the validateData method in utils
• provide better info about why a given measurement fails to pass

http://spacetelescope.github.io/understanding-json-schema/
http://jsonschemalint.com/draft4/#

UPDATE: 20 Mar 2016: Please see my comment below (also timestamped 20 Mar 2016) for updated information and formatting of this issue.

Note: While several sources are listed, none are straightforward. If you'd like a straightforward new data issue, visit this issue for adding data sources from Tehran, Iran: #51

Below are listed potential other sources in India, contributed by the OpenAQ Community:
These are listed as high priority because members of the community suggested them.

(1) http://www.cpcb.gov.in/CAAQM/frmUserAvgReportCriteria.aspx
Note: Several Sites around India. I have not been able to access any data yet from this link with any combination of places I have put in. I'm also uncertain if it yields AQI or physical concentrations yet.

(2) http://164.100.160.234:9000/
Note: Several sites around India. These values are in AQI.

(3) http://www.cpcb.gov.in/CAAQM/mapPage/frmindiamap.aspx (same presumed sources as (1), just different view)
Note: Several sites around India. To date, we've had issues programmatically accessing these data - something to do with a timeout issue perhaps? Do you recall,@jflasher?

(4) http://mpcb.gov.in/envtdata/demoPage1.php
Note: Several sites around India. Looks like daily-averaged data. Unclear if data are available in real-time.

http://www.stateair.mn

They also have hourly historical data going back to October.

Think I might be able to do this with Kimono and will give it a shot. If not, will add 'help wanted' shortly.

This is listed as high priority b/c Tehran, Iran is experiencing very bad AQ. Media outlets report schools shut/will be shutting down to AQ. Plus it is in a region where we have minimal to no coverage currently in our system.

Most useful info:
List of stations and coordinates: http://31.24.238.89/home/station.aspx
Hourly physical concentration data: http://31.24.238.89/home/DataArchive.aspx
(also downloadable via csv)

Other info:
General map (in AQI format): http://air.tehran.ir/Default.aspx?tabid=193
General map with hourly data (in AQI format): http://31.24.238.89/home/OnlineAQI.aspx
It appears they are using the US EPA scale. I assume this means they are doing a similar calculation for US EPA

Data is now reporting on the US AirNow System for US Emb sites in Hanoi, India, and UB. Not clear if we will be able to access it via the AirNow ftp database or if we'll need to scrape:

http://airnow.gov/index.cfm?action=airnow.global_summary.

142 sources for Romania, updated hourly:
http://www.calitateaer.ro/valori.php

https://www.kimonolabs.com/

This is actually a good thing since it's been breaking on a regular basis anyways.

@jflasher originally created this issue:

Too many emails are getting sent, figure out a more sane way to handle this. They are currently disabled via Heroku scheduler task until this is fixed.

These are labelled as a priority because we have gotten a request for Vancouver, though coverage over all of BC would be great.

• General British Columbia AQ page: http://www.bcairquality.ca/
• Go here: http://www.bcairquality.ca/readings/index.html
• Click on 'inset' of Vancouver
• Then click on the actual map locations (e.g. "Vancouver International Airport")
• This will then show the actual data.
  ...I'm going to keep clicking around to hopefully find a better/easier way to access these data.

Some of them are just objects at the moment.

Hourly/ real-time data reported on airnow.gov is aggregated into geographical areas and reported as an AQI using the NowCast formula (a weighted 12-hour average) so we are unable to use it for direct actual pollution measurements.

In search of sites that display Washington DC AQ data, found this:
http://www.nature.nps.gov/air/data/current/index.cfm
This lists hourly ozone data in national parks around the US.

http://newdelhi.usembassy.gov/airqualitydataemb.html

See table with AQ (only PM2.5) data for 5 cities in India.

Originally created by @olafveerman :
Coordinates for the Dutch stations are hard-coded in the adapter: https://github.com/openaq/openaq-api/blob/master/adapters/netherlands.js#L98

Instead, they should/could be parsed from: http://www.lml.rivm.nl/meetnet/index.php

Listed as a priority given recent AQ issues in region, but NOTE: There is an issue with these data as they are currently presented: They are shown as an index value of combined pollutants into a unitless value. The FAQ's indicate to contact an individual for historical concentrations in non-index units, though currently unsure how to do that.

After discussions with WRI/GFW, they are particularly interested in AQ data from countries affected by the fires in Indonesia.

AQ data for Malaysia:
http://apims.doe.gov.my/v2/
(click 'apims' when given choice at portal)

FAQ's (and 5th one down is the one on hourly concentration data): http://apims.doe.gov.my/v2/faq.html

Seeking help in identifying who to contact about access to hourly AQ data.

There are a ton of China air quality data sources. Here are some:
aepb.gov.cn
bjmemc.com.cn
cdemc.cn
cfhb.gov.cn
cepb.gov.cn
dl.gov.cn
dyhb.gov.cn
nbemc.gov.cn
sdein.gov.cn
fjepb.gov.cn
gsep.gansu.gov.cn
qhepb.gov.cn
gdep.gov.cn
gxepb.gov.cn
ghb.gov.cn
dloer.gov.cn
hebei.gov.cn
hljdep.gov.cn
hnep.gov.cn
hbepb.gov.cn
hbt.hunan.gov.cn
nmgepb.gov.cn
jshb.gov.cn
jxepb.gov.cn
shbj.klmy.gov.cn
lnemc.cn
lzhb.gov.cn
nnems.gov.cn
nxep.gov.cn
ordoshb.gov.cn
semc.gov.cn
sxhjjcz.com.cn
szhec.gov.cn
tjemc.org.cn
xzep.gov.cn
wlmqhb.gov.cn
whepb.gov.cn
xianemc.gov.cn
xnepb.gov.cn
xjepb.gov.cn
ynepb.gov.cn
zjepb.gov.cn
Sources from: aqicn.org/sources

There is also a site that provides an API to Chinese AQ data:
http://pm25.in/

However the issue is that from what I can tell, these data are not shared in their raw format only AQI (I have not clicked on each and everyone however).

Anyone know differently or find ones above that are shared our in raw format?

Of course listed as a priority due to recent AQ issues in the region, but there are issues accessing data at an appropriate time resolution (@olafveerman, correct/edit if I'm mistaken).

After discussions with WRI/GFW, they are particularly interested in AQ data from countries affected by the fires in Indonesia.

Here is the general AQ data page for Indonesia:
http://www.bmkg.go.id/BMKG_Pusat/Kualitas_Udara/

Here is an example of one of the page (PM10). I am unsure if we can programmatically get data from this page either from the graph (hover over point) or hit 'view data':
http://www.bmkg.go.id/BMKG_Pusat/Kualitas_Udara/Informasi_Partikulat.bmkg

@jflasher 's original issue:
If we don't set timeouts on the requests, Heroku may time us out which feels worse. Makes me wonder if we should have a system-wide request object that gets passed around so we can set defaults in one place?

Listed as medium priority b/c it was requested, but frankly there's not much there, so didn't list it as high priority:

This is the only site in the DC area I'm finding hourly real-time air quality measurements, not AQI. And it is only for one location (McMillan Reservoir).

http://www.nature.nps.gov/air/webcams/parks/nacccam/washcam.cfm

This site gives a nice synopsis of how AQ data in the region is handled via EPA, DC govt, and NPS:
http://doee.dc.gov/service/ambient-air-quality-monitoring

Please comment if other sites with actual measurements in the DC area are available.

48 hour Moscow data
http://mosecom.ru/air/air-today/station/spirid/table.html

We've gotten a request for NYC data.

There are several on this site for NYC and also NY State. Here is one for NY City (CCNY):
http://www.dec.ny.gov/airmon/stationStatus.php?stationNo=73

I am going to be making this invalid to avoid DynamoDB insert errors. There shouldn't be an empty item in the attribution array.

{ date: 
   { utc: '2016-01-07T00:00:00.000Z',
     local: '2016-01-07T00:00:00+00:00' },
  parameter: 'co',
  location: 'Amsterdam-Einsteinweg',
  value: 861.5,
  unit: 'µg/m³',
  city: 'Amsterdam',
  attribution: 
   [ { name: 'RIVM', url: 'http://www.lml.rivm.nl/' },
     { name: '' } ],
  averagingPeriod: { value: 1, unit: 'hours' },
  coordinates: { latitude: 52.3813, longitude: 4.84523 },
  country: 'NL',
  sourceName: 'Netherlands',
  Timestamp: '2016-01-07T00:00:00.000Z',
  LocationParameter: 'co-Amsterdam-Einsteinweg' }

We've gotten two requests to pull in SF data as we can for use for people working on low-cost sensors.

http://www.baaqmd.gov/about-air-quality/current-air-quality/air-monitoring-data?DataViewFormat=daily&DataView=tech&StartDate=10/15/2015&ParameterId=316

Haven't been avoiding this for any particular reason, but should definitely add this in the near future.

Map of stations with coordinates and current readings and stations' current data are here (though not on unique urls):

http://www.havaizleme.gov.tr/Default.ltr.aspx

'TÜM İSTASYONLAR' = all stations

Clicking on the stations reveals site coordinates (click 'station description) and pollutant types measured.

(Sidenote: They use an AQI system with breakpoints same as the US EPA)

Related to openaq/openaq-api#141, trying to first get writing to DynamoDB working. I'm writing to a table (yay!), but there are some weird behaviors (boo). Specifically, using Texas and Sao Paulo data sources as examples, if I write all the Texas data, it saves fine, but then when I go to write Sao Paulo data into the same table, I seem to be losing some of the Texas data. I have no idea how writing more data into the database can make other data disappear. I thought I could be accidentally overwrite Texas data with Sao Paulo data, but I have done a few things to convince myself that this isn't the case, so not sure what it is at the moment.

Example output before and after a Sao Paulo data source looks like below

Brizo: ~
$ aws --profile openaq dynamodb scan --table-name measurements --filter-expression 'country = :country' --expression-attribute-values {\":country\":{\"S\":\"US\"}} --select COUNT
{
    "Count": 1366, 
    "Items": [], 
    "ScannedCount": 1366, 
    "ConsumedCapacity": null
}

Brizo: ~
$ aws --profile openaq dynamodb scan --table-name measurements --filter-expression 'country = :country' --expression-attribute-values {\":country\":{\"S\":\"US\"}} --select COUNT
{
    "Count": 899, 
    "Items": [], 
    "ScannedCount": 3045, 
    "ConsumedCapacity": null
}

The total number of records does increase, but doesn't seem to get to the right amount. And now even weirder, when I try to insert records in batches after each other, the total number of records in the database decreases? Craziness!

Listed as a priority because we currently do not have any data from the entire continent of Africa.

Data: http://live.rbcaa.org.za/Default.htm
Coordinates: Click on station/station information for coordinates
('trs' in pollutant table is 'total reduced sulfur' - we won't want it

Was seeing some errors in Texas return data that seemed to be stopping whole data process.

Listed a priority given recent AQ issues in area but NOTE: There is an issue with these data as they are currently presented: They are not listed as individual stations, but rather as aggregated stations that represent different regions of the city. WRI/GFW is contacting NEA on our behalf to see if the station-level data are accessible. We will not proceed with these data until we received the station-level information.

After discussions with WRI/GFW, they are particularly interested in AQ data from countries affected by the fires in Indonesia.

AQ data for Singapore:
http://www.nea.gov.sg/anti-pollution-radiation-protection/air-pollution-control/psi/pollutant-concentrations/type/PM25-1Hr#pollutant

If anyone knows differently, please comment.

http://www.thenational.ae/uae/sharjah-to-make-air-quality-data-from-seven-new-stations-open-to-public

These sources are available via RSS feed at:
http://stateair.net/dos/
RSS Feeds:
Chennai: http://stateair.net/dos/RSS/Chennai/Chennai-PM2.5.xml
Hanoi: http://stateair.net/dos/RSS/Hanoi/Hanoi-PM2.5.xml
Hyderbad:http://stateair.net/dos/RSS/Hyderabad/Hyderabad-PM2.5.xml
Kolkata: http://stateair.net/dos/RSS/Kolkata/Kolkata-PM2.5.xml
Mumbai: http://stateair.net/dos/RSS/Mumbai/Mumbai-PM2.5.xml
New Delhi: http://stateair.net/dos/RSS/NewDelhi/NewDelhi-PM2.5.xml
Ulaanbaatar: http://stateair.net/dos/RSS/Ulaanbaatar/Ulaanbaatar-PM2.5.xml

Geographic Coordinates:
Chennai: 13.052371, 80.251932
Hanoi: 21.021770, 105.819002 (source: Google Maps, US Embassy in Hanoi, need to verify this is the location of the monitor)
Hyderbad: 17.443464, 78.474890 (source: Google Maps)
Kolkata: 22.547142, 88.351048 (source: Google Maps)
Mumbai: 19.066023, 72.868702 (source: Google Maps)
New Delhi: 28.598096, 77.189066 (source: Google Maps)
Ulaanbaatar: 47.928444, 106.930189 (source: Google Maps)

We could also add the Mission China data, which is also similar on RSS. Currently we use Kimono.

It has different units so wasn't included initially.

Data submitted by someone from the Source Google form for Slovenia:

Hourly: http://www.arso.gov.si/xml/zrak/ones_zrak_urni_podatki_zadnji.xml
Daily: http://www.arso.gov.si/xml/zrak/ones_zrak_dnevni_podatki_zadnji.xml"

Hourly AQ data for several places in CA that have high levels of PM relative to the rest of the US (specifically: Kern + Merced Counties, CA, the city of Fresno, CA, etc.):

http://www.valleyair.org/Programs/RAAN/raan_monitoring_system.htm

This could help catch errors in the adapters and fix them more quickly.

Hourly data for each pollutant, using the following configuration shown in this attached image:

Listed as a priority b/c we got a request for it. Not listed as the highest priority b/c can't figure out if these sources are coming from specific stations vs. aggregated or where those stations are specifically located. Can't really use until we can determine this.

Data here: http://air.utah.gov/currentconditions.php?id=slc
Though this might be better to get the data from:
http://air.utah.gov/trendcharts.php?id=slc

These data are labelled as a priority b/c they have been requested. Also these data (in my imperfect estimation) look fairly straightforward to get.

http://www.airqualityontario.com/history/pollutant.php?stationid=31103&pol_code=124&start_day=5&start_month=11&start_year=2015&station_id=31103&chart=0&submitter=Update

fix it!

According to http://www.timeanddate.com/worldclock/netherlands/amsterdam, should be +1 UTC, but recent data looks like

{
"_id": "56744fa651dc60c200e1969d",
"date": {
"utc": "2015-12-18T19:00:00.000Z",
"local": "2015-12-18T19:00:00+00:00"
},
"parameter": "so2",
"location": "Amsterdam-Hoogtij",
"value": 2.3,
"unit": "µg/m³",
"city": "Amsterdam",
"coordinates": {
"latitude": 52.428,
"longitude": 4.77348
},
"country": "NL"
}

Originaly @jflasher 's issue.

Japanese data was listed as a priority as we got a request to have this in our database longterm:

Sources for Yokohama:

http://cgi.city.yokohama.lg.jp/kankyou/saigai/data/taiki/all/all_0000_00_001.html
http://www.ihe.pref.miyagi.jp/telem/dayreportitem/?itemSelect=10&day=2015%E5%B9%B410%E6%9C%8804%E6%97%A5

Appears to be hourly but unsure. Joe is contacting Miyagi Prefecture regarding details, potentially existing API, station coordinates.

Tokyo:
Just oxides? Need help with translation:
http://www.ox.kankyo.metro.tokyo.jp/index.php?chiku=1
http://www.ox.kankyo.metro.tokyo.jp/

Main page: http://www.kankyo.metro.tokyo.jp/nature/index.html