Deploy script for fetching online data from SIRI about open-bus HOT 7 CLOSED

Attached is a list of bus stops.
There are stops served by routes that also call at train stations.
So if route 11 calls at Tel-Aviv Center, all the stops for line 11 are included in this file.
If we can't poll data for the all the 27K bus stops in Israel, we still might be able to poll data for this list (~16K).

stops_connected_to_stations.txt

from open-bus.

Implementation:

1. The relevant scheme can be found here: core/schema.sql
2. fetch_and_store_arrivals.py is indeed the currently preferred way for fetching SIRI data. It was tested 2 months ago, but I'm not sure what's its current status.
3. I think we should create a script which gets a DB name, querying frequency, and experiment duration in hours (e.g. 168 hours - a week) and fetch_and_store_arrivals.py params; and will create the requested DB and the querying cron.

Scaling:

Querying about ~16K stationsat once is quite a lot, @pumbaoz says it may be considered as a DDoS attack on SIRI servers. Can we narrow the list to ~300? (Maybe refer to a partial set of train stations, and switch the set after we have enough data).
For similar reasons, it may be better to lower the querying frequency. @kobigro and @pumbaoz agreed 5 minutes will be a legitimate frequency. We may increase the frequency once we make sure all goes well.

In addition, we should make sure our server has enough resources for this mission.

from open-bus.

OK, so here's another idea. Let's poll all (or many) stops along one or two important lines.

This will allow us to see how well we can infer the bus location based on the SIRI data.

I suggest taking 189 Tel-Aviv - Holon, and maybe another line in Jerusalem \ Haifa. I can ask Gil from 15minutes for suggestions.

Does that seem to make sense?

@nitzangur , are you planning to work on this or should we find another volunteer for that?

from open-bus.

I added the siri cron script, it's in siri/pull_data.cron. It's currently set to run every 5 minutes.
To deploy it , we need to decide on a list of stops to poll(stops.txt) and create the schema on the server.

from open-bus.

I suggest we start with the stops for lines 189 and TLV and 71 in Jerusalem.
I am attaching files with the lists of stops for each of these lines. The lists were created using https://github.com/hasadna/open-bus/blob/master/gtfs/parser/line_stops_finder.py, on the GTFS data from 24/8.

I think the field we need for SIRI is the stop code, and not the stop id, which I think is GTFS specific. But unless anyone knows for sure please test it to see which one is correct.

suggested stops.zip

from open-bus.

I tried running fetch_and_store on our new server. I made a few small changes, like removing the username and password from the code (for increased security) and turning off the Open Train proxy by default.

The current status is:

1. There's some permission problems when writing to the DB:
   psycopg2.ProgrammingError: permission denied for sequence responses_id_seq
   I am hoping someone with more PostGRES experience can help me fix this on Monday.
2. The cronjob will have to include command line parameters for the user, password and stops file name. Anyway this will have to be done during deployment because user and password shouldn't be in the repository.
3. Unfortunately we again don't have permissions to access SIRI directly from our server (as its IP address changed). I am checking with @uda and @maryLoi what we can do about it.
4. Polling 219 stops (as I asked for in my previous post) seems to be all right with SIRI, but the reply is very large (~0.5M) and proxying through Open Train breaks (raises http.client.IncompleteRead). This isn't a huge problem since we meant to use the proxy for testing only.

from open-bus.

Good news everyone!

Our SIRI access has finally been resolved last week.

I have installed fetch_and_store_arrivals on our server. It is configured to run every minute from 5am to 1am, Sunday-Friday.

It polls data for lines 19 in Jerusalem, line 480 (Jer-TLV) and line 947 (Jer-Haifa).

I expect that within a week we will have enough data to start analysing it and seeing if it's helpful at all.

from open-bus.