Hello to all,

I'm very interested in this project. So thanks to the authors.
Can u upload again the json files, for example incofer.json for costa rica?
I have checked all links but are dead

Thanks

The current route creation is very much custom for Florianópolis. After the abstraction work happening in (#9) I'd suggest working on a default implementation for route generation based on OSM's Public Transport Schema 2.

Managua can be a good start as the data quality is very consistent and complying the standard data structure without exceptions.

The stops parameter of Route Class is a list the is being assigned with two different object types during the default execution of the script.

In osm_connector it is a list of strings, where each string corresponds to the node ID of the stop in OSM. In the default creator the stop parameter is changed to a list of osm2gtfs Stop Objects.

There are some function calls and variable names in Portuguese language. I have spotted them primarily in the osm_routes.py file. These should be cleaned up.

Hello to all!

I´m trying to run osm2gtfs using ubuntu 14 64 bits.

Steps i have made:

1. Installed Python Version 2.7.6
2. Installed PIP (apt-get install python-pip)
3. Clone osm2gtfs git repository to /opt
4. cd into osm2gtfs folder and run pip install -e . the following message appears

**Obtaining file:///opt/osm2gtfs
Running setup.py (path:/opt/osm2gtfs/setup.py) egg_info for package from file:///opt/osm2gtfs

Downloading/unpacking overpy>=0.4 (from osm2gtfs==0.0.1)
Downloading overpy-0.4.tar.gz (41kB): 41kB downloaded
Running setup.py (path:/tmp/pip_build_root/overpy/setup.py) egg_info for package overpy
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
your setuptools is too old (<12)
setuptools_scm functionality is degraded
zip_safe flag not set; analyzing archive contents...

Installed /tmp/pip_build_root/overpy/pytest_runner-3.0-py2.7.egg

warning: no files found matching 'docs.Makefile'

Downloading/unpacking transitfeed (from osm2gtfs==0.0.1)
Downloading transitfeed-1.2.14.tar.gz (347kB): 347kB downloaded
Running setup.py (path:/tmp/pip_build_root/transitfeed/setup.py) egg_info for package transitfeed

Installing collected packages: overpy, transitfeed, osm2gtfs
Running setup.py install for overpy

warning: no files found matching 'docs.Makefile'

Running setup.py install for transitfeed
changing mode of build/scripts-2.7/feedvalidator.py from 644 to 755
changing mode of build/scripts-2.7/schedule_viewer.py from 644 to 755
changing mode of build/scripts-2.7/kmlparser.py from 644 to 755
changing mode of build/scripts-2.7/kmlwriter.py from 644 to 755
changing mode of build/scripts-2.7/merge.py from 644 to 755
changing mode of build/scripts-2.7/unusual_trip_filter.py from 644 to 755
changing mode of build/scripts-2.7/location_editor.py from 644 to 755
changing mode of build/scripts-2.7/feedvalidator_googletransit.py from 644 to 755
changing mode of build/scripts-2.7/shape_importer.py from 644 to 755

changing mode of /usr/local/bin/feedvalidator.py to 755
changing mode of /usr/local/bin/kmlparser.py to 755
changing mode of /usr/local/bin/schedule_viewer.py to 755
changing mode of /usr/local/bin/location_editor.py to 755
changing mode of /usr/local/bin/shape_importer.py to 755
changing mode of /usr/local/bin/feedvalidator_googletransit.py to 755
changing mode of /usr/local/bin/unusual_trip_filter.py to 755
changing mode of /usr/local/bin/kmlwriter.py to 755
changing mode of /usr/local/bin/merge.py to 755

Running setup.py develop for osm2gtfs

Creating /usr/local/lib/python2.7/dist-packages/osm2gtfs.egg-link (link to .)
Adding osm2gtfs 0.0.1 to easy-install.pth file
Installing osm2gtfs script to /usr/local/bin

Installed /opt/osm2gtfs

Successfully installed overpy transitfeed osm2gtfs
Cleaning up...**

5. Check if transitfeed is installed
   pip install transitfeed
   Requirement already satisfied (use --upgrade to upgrade): transitfeed in /usr/local/lib/python2.7/dist-packages

6. Check if overpy is installed
   pip install overpy
   Requirement already satisfied (use --upgrade to upgrade): overpy in /usr/local/lib/python2.7/dist-packages

7. Finally i run osm2gtfs command
   osm2gtfs --config incofer.json --output testinconfer

But i get an error

Traceback (most recent call last):
File "/usr/local/bin/osm2gtfs", line 9, in
load_entry_point('osm2gtfs==0.0.1', 'console_scripts', 'osm2gtfs')()
File "/opt/osm2gtfs/osm2gtfs/osm2gtfs.py", line 35, in main
config = Configuration(args)
File "/opt/osm2gtfs/osm2gtfs/core/configuration.py", line 32, in init
self._prepare_dates()
File "/opt/osm2gtfs/osm2gtfs/core/configuration.py", line 98, in _prepare_dates
if 'start_date' in config['feed_info']:
KeyError: 'feed_info'

What i doing wrong? Some one knows how to solve it?

Thanks

Currently, the data about the schedule information is getting pulled in a very custom way into the script.

Others would like to use the script. For now I see different ways for feeding the script:

• Currently @grote uses a json file with route numbers and then pulls data from the Fenix network website and parses the time information.
• @jamescr is putting schedule information directly in a json file
• We (MapaNica) would like to use a CSV file with frequencies and run some logic on it.

I suggest moving different ways on pulling schedule data into different files, which could be handled similar to small plugins. Basically all of them then have to output the same information to the main script: Stop times of the first and last stops (mostly terminals, I guess).

Probably we could specify in the config file (#3) the name of the schedule information plugin. Do you have any suggestions on how to implement this properly?

This is an important step in order to make this tool generically usable.

In code currently stops without names get assigned "[" + self.stop_no_name + "]" and after trying to find a good name (OsmConnector._find_best_name_for_unnamed_stop()) it just uses stop_no_name without brackets to not query again and again. We probably should move to None.

Hello,

currently a lot of custom configuration data is stored in code. I'd like to push it to be more generic.

I'd suggest to rely on a config.json file similar to geom2gtfs does: https://github.com/conveyal/geom2gtfs/blob/master/kingco.json.example

We could start with a simple structure, based on the general information used for querying from OSM and from GTFS' agency.txt and start and end date from calendar.txt:

{
    "query":{
            "network": "", 
            "bbox": {"e": "-48.2711", "n": "-27.2155", "s": "-27.9410", "w": "-49.0155"},
    },
    "agency":{
            "agency_id": "",
            "agency_name": "", 
            "agency_url": "", 
            "agency_timezone": "",
            "agency_lang": "",
            "agency_phone": "", 
            "agency_fare_url": "",
    },
    "feed_info":{
            "publisher_name": "", 
            "publisher_url":  "",
            "licence_name":  "",
            "licence_url":  "",
            "version":  "",
    },


    "gtfs_mode":3,
    "start_date": "",
    "end_date":"",
    "schedule_information":  "",
    "output": "", 
}

What do you think? Is this a way to go?

In order to generate proper GTFS files good data quality in OSM is recommended, if not required.A little writeup/manual would help users how to assure data quality.

I think the wiki here would be the best place to live for this information. Maybe once it's getting to be more complete we could integrate it into learnOSM.org

Just a rough brainstorm list of the contents I suggest to include:

• Use JOSM, the PT_Assistant plugin, and the OSM standard public_transport:version=2
• Clean up all verification errors related to PT.
• Make a list of recommended tags to be on the relations and which are used in osm2gtfs

Originally the script was created with bus routes in mind but this could also work for train routes (as the case of incofer train routes)

The idea is to add an optional argumnet: "--route-type". the default value will be "bus".

I'm testing Transportr (a locally compiled version) with incofer data and I notice that some stops are avoided as line change points. Example: instead of changing from line 2 to line 4 in stop A is being done in stop B affecting times of the total trip. I suspect maybe that non grouped stops could be the problem, as stop A of line 2 and stop A of line 4 aren't exactly in the same point but are in the same station:

-84.0693107,Estación Atlántico,9.9349735,4310018677,0
-84.0686079,Estación Atlántico,9.9346699,4628999925,0

Adding support for grouping stops (stations) in the GTFS file seems simple. The GTFS part is solved using the parent_station and location_type of the stops.txt (https://developers.google.com/transit/gtfs/reference/stops-file). And from OSM we could check for the stop_area relation the contains both stop_position. At the end we should have something like this:

stop_lon,stop_name,stop_lat,stop_id,location_type, parent_station
-84.0693107,Estación Atlántico,9.9349735,4310018677,0, ID
-84.0686079,Estación Atlántico,9.9346699,4628999925,0, ID
lon, Estación Atlántico, lat, ID, 1,

We need documentation on how the script is technically structured, how to extend the script and how to make it work for other cities and other type of schedule information. I suggest to use the wiki here in this github repo.

Hi there,

the script obtains some attributes from the route from OpenStreetMap, such as reference number (ref) and the name. And then there are additional attributes hard-coded in the script, these are:

• route_desc
• route_url
• route_color
• route_text_color

Those should be pulled from OpenStreetMap and not be hard coded.

The tagging schema provides tags on bus routes for description, colour and we can also add website. For the text color we could either come up with a new tag in OSM or define it in the script's config file (#3)

Hi,

in code there is a blacklist of some routes hard-coded into the script. Are these from a different network? Or is there any other reason they are excluded. I'd suggest to mark them in the data of OSM to be able to sort them out, and removing the blacklist in code.

osm2gtfs downloads data from OpenStreetMap (Public Transport Schema version 2) and puts it into transitfeed's structure to generate GTFS.

Currently there is a parallel data structure (mainly living in the classes RouteMaster, Route, Stop) with some regional touch. We already started some discussion about this in #26, but I think it's worth to discuss this separately.

Resuming the past comments on this:

We were considering to use directly transitfeed's (routes, stops and trips objects) instead of a parallel data structure. The downside of this would be that we would make ourselves totally dependent on transitfeed's classes which might change without further notice.

I further think it's not really possible because stops associated to routes (and variants) are connected in transitfeed's objects through the trips. Something osm2gtfs generates at the very end based on stops over routes. So this would just not work, as far as I understand.

Ergo, we need a parallel structure(!?). So, let's think about this and how this would look like.

We need documentation on how the script works, which functionalities it has and how the config file is structured. I suggest to use the wiki here in this github repo.

Debugging, such as screen output and special functions like Route.print_shape_for_leaflet() are currently within regular code. I'd like to optimize this. Right now I'm thinking about:

• a Debug class for special functions
• making debug print output always depending on a flag (probably set through a bash argument)

Currently, all the requirements are just listed in the README.md, but they should probably be moved into a requirements.txt file for easy installation.

Currently, this repository includes different implementations for cities. Particularly the trip_creators are very much exclusively targeted and programmed for a special data source. However for any new person coming to this script it is almost impossible to find out where to get those data sources, in order to run osm2gtfs with the existing implementations, for example for the sake of testing. As they are so heavily connected, the data (source) on the schedule should be provided, too.

Instead of adding the actual data, I'd suggest to include a source field to the configuration file where a valid url to the file containing the time information should be specified. The script would then download the data, save it into the data directory and use it. Probably also an argument (similar to the --refresh- ones should be implemented, e.g. --refresh-time-table).

The data = OsmHelper() object initiation queries and creates the whole data structure of all routes and stops already in the constructor.

This is not very efficient, because even when updating only the routes (or even one route) or only the stops, everything get's queried and built anyway.

I guess, we should move those queries and data tree builders into the respective functions, instead of building the whole data tree in the constructor function.

My local company Expresso Lorenzutti have made time tables available in PDF format. I understand it might be difficult to accommodate all forms of publishing time tables, and have therefor started to write a scraper for this. I would like to save it in a simple, common format, that osm2gtfs can read directly. One easy solution would be to save it as JSON. If you have any suggestions to formats I am open.

Hi,

I think the current bounding box approach is not ideal. Transit networks can overlap and do not end necessarily at squared boxes.

I'd suggest to run a query to obtain relations with

• type=route
• public_transport:version=2
• network=XY

(and probably maintain the bounding box optional to just make the overpass api query more performant).

Does this make sense? I'd happy to provide a pull request for this. But I also think it's best to get feedback before. Thanks!

I've been working on the script and building the creators for my city and i noticed that if i run the
script with the :

--refresh-all flags it work fine
but if i try to run it with out the flag the script doesn't work and i raise the " Should have implemented this"
like in the route_creator and in trips_creator.

so that tells me the script is not reading my creators and in my config.json i did specify the selector

"selector" : "irtramma"

any ideas why is this happening? i did run incofer with the data that james has in one of his repos and works fine.

PEP 8 recommends:

Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability.

And

Class names should normally use the CapWords convention.

Our creator modules (filenames) however are currently using CapWords convention. This should be changed to fit PEP 8.

Some ideas:

• Improve find_best_name_for_unnamed_stop(), if possible
• Move out of Fenix into standard procedure
• Make it configurable to use it or not through config file

The current stops creation is very much custom for Florianópolis. After the abstraction work happening in (#9) I'd suggest working on a default implementation for stops generation based on OSM's Public Transport Schema 2.

Managua can be a good start as the data quality is very consistent and complying the standard data structure without exceptions.

We need a way to make sure our changes and refactors do not break the current creators.
Maybe a shell script ...

The process could be, for each creator :

• launch osm2gtfs with mocked or persisted OSM input data
• unzip the GTFS output file
• display a diff of all the csv files

Currently the filename and path of the output file is hard-coded into the script. We should handle this more generic. Either as another argument on the command line or inside the config file (#3). I prefer the first solution, but before implementing, I wanted to check with you guys.

Cheers!

If the json config file includes in the feed info only one of the dates: "start_date" or "end_date" then the other is generated automatically. But when this happen the object type for the generated date is datetime.date and the type of the date from config file is datetime.datetime. When the dates validation is done an TypeError occurs: «can't compare datetime.datetime to datetime.date»

This could be fixed using datetime.datetime in both cases. I'll create a PR is you agree.

Currently we have the following folder (package) structure

• creators (extendible part of the application)
• factory (handles logic to for the creators)
• osmhelper (everything else: OsmConnector, Cache, data structure)
• data (data caching, input and output)
• src (external python packages)

I'm proposing to merge factory and osmhelper into one single package called core. What do you think?

Right now the code includes Portuguese language. This is probably not very good and we should think about how to localize it. Potfiles are probably the way to go!?

A load_agency(config) method returning a transitfeed.Agency object created using data from the json config file. In the json file under the "agency" object, the valid keys will be any member of the transitfeed.Agency._FIELD_NAMES array.

Hello, I hope you don't mind me asking some questions. if this is not the correct place for this, feel free to close this issue.

I'm working on a personal project (mostly for learning) building an open source progressive web app that'll show you bus routes between two points, since I've found no free alternatives that work well in Argentina.

What I have so far is a basic frontend interface and a python backend that runs querys on a local postgis db to which I have loaded osm data for my region, here is the code. These seems to kinda work but I'm having some trouble with osm2pgsql splitting bus routes into chunks when the route makes a circle and determining the direction so that I only get the A -> B routes.
what do you think of this approach?

yesterday I saw your youtube video about transportr and I wonder if using gtfs data is strictly necessary in order to search routes (if I don't care about arrival and departure times yet) ?

another question would be if you could provide me some basic info about how I would adapt this library so I can try to get data from Argentina and contribute that to navitia. There are some closed source apps like Moovit and mapa.buenosaires.gob.ar that have departure and arrival times, so I'm trying to figure out how to get them as well. (the latter being a government website, so I send some emails but they have not answered yet)

Thanks in advance

When generating an end_date with a start_date in January or running the script in January without a specified start_date the script throws the following error: calendar.IllegalMonthError: bad month number 0; must be 1-12. A special check needs to be introduced for this particular case.

Providing transfers information could enhance the use of the generated GTFS for journeys computation.
There could be several ways of doing it :

• a new feature in osm2gtfs
• using an external tool

After creating a new route or stop from the API, the new object should be written to the file right away.

If there's an error (such as two many requests), you can resume right where you left off instead of starting from scratch again.

Making the config file argument required is not ideal. I suggest to have config.json as a default argument for the config parameter, but don't include this file in the repository and even add it to .gitignore.

This enables people to have their local config file and play with it without it being tracked by version control and without having to specify the file every time they want to run the script. If people are happy with the official file in the repository, they can symlink config-fenix.json to config.json.

What is the difference between travel_time and duration attributes of Route object ?

(Sorry for the delayed question related with PR #66.)

Currently the stop_id for the GTFS is extracted based on the id coming from OpenStreetMap. This is nice, because ideally it allows mapping between a stop and its object in OpenStreetMap.

Some context: OSM has three types of elements: node, way, relation. They have separate spaces for IDs:

Element types have their own ID space, so there could be a node with id=100 and a way with id=100, which are unlikely to be related or geographically near to each other.

Currently it seems to work like this (in the standard stop creator): if the stop is a node or a way it gets it's ID (even though they are not the same). If it is a relation of type stop_area, then a SA gets prepended to the ID.

This is not ideal, because IDs should be solid (and not change over time), and they should not conflictive (like IDs between nodes and ways).

google/transitfeed#394 has been merged and released as 1.2.16.The workaround in the code to add the feed_info.txt manually should be removed now.

Also, Accra regression tests need to be fixed. There are different stop times now.

I'm creating this issue just to keep issue#55 with the original theme.

In my opinion, all other creators should ideally be merged here. This makes sense for example when >we refactor the code. If the creators live in other repositories, they need to be manually adjusted to
whatever changed in the upstream repository.

Make sense to keep the creators in the same repo, also if we try to make generic default creators will be very useful to have them together.

And what about config files? A config folder to keep them could be a solution. What you think?

@grote worte:

For the dates, we need a more flexible solution, because when used with Navitia, the difference between start and end date can never be more than a year and ideally, for this use-case, I like these dates to automatically move forward when I run the script again without needing to update them somewhere manually.

Great idea. Maybe we can make the script to understand syntax like now and +1year, etc...

There is a validation error when start_date is automatically generated:

transitfeed.problems.InvalidValue: Invalid value 2017301 in field start_date

This is because datetime.datetime() returns the numer of the month as one digit (3) instead of two (03).

When building a stops list i\m getting the following error:

Traceback (most recent call last):
File "osm2gtfs.py", line 101, in
main()
File "osm2gtfs.py", line 67, in main
stops_creator.add_stops_to_schedule(schedule, data)
File "C:\Projecten\osm2gtfs\creators\stops_creator.py", line 20, in add_stops_to_schedule
stops = data.get_stops()
File "C:\Projecten\osm2gtfs\core\osm_connector.py", line 239, in get_stops
self._get_names_for_unnamed_stops()
File "C:\Projecten\osm2gtfs\core\osm_connector.py", line 488, in _get_names_for_unnamed_stops
self._find_best_name_for_unnamed_stop(stop)
File "C:\Projecten\osm2gtfs\core\osm_connector.py", line 518, in find_best_name_for_unnamed_stop
""" % (stop.lat, stop.lon, stop.lat, stop.lon))
File "C:\Python27\lib\site-packages\overpy_init.py", line 123, in query
raise exception.OverpassTooManyRequests
overpy.exception.OverpassTooManyRequests: Too many requests

or

Traceback (most recent call last):
File "osm2gtfs.py", line 101, in
main()
File "osm2gtfs.py", line 67, in main
stops_creator.add_stops_to_schedule(schedule, data)
File "C:\Projecten\osm2gtfs\creators\stops_creator.py", line 20, in add_stops_to_schedule
stops = data.get_stops()
File "C:\Projecten\osm2gtfs\core\osm_connector.py", line 211, in get_stops
self._get_names_for_unnamed_stops()
File "C:\Projecten\osm2gtfs\core\osm_connector.py", line 488, in _get_names_for_unnamed_stops
self._find_best_name_for_unnamed_stop(stop)
File "C:\Projecten\osm2gtfs\core\osm_connector.py", line 545, in find_best_name_for_unnamed_stop
candidate.get_nodes(resolve_missing=True))
File "C:\Python27\lib\site-packages\overpy_init.py", line 860, in get_nodes
tmp_result = self.result.api.query(query)
File "C:\Python27\lib\site-packages\overpy_init.py", line 126, in query
raise exception.OverpassGatewayTimeout
overpy.exception.OverpassGatewayTimeout: Server load too high

Is there a way to avoid it?

Instead of printing out random messages, it would be nice to get a kind of "report" of the data it ran on, things to improve on OpenStreetMap, etc. These information should be grouped.

Hi,

I'm writing here not sure if it is the right place, so sorry in advance for that. I've installed osm2gtfs because I need to generate GTFS data from Birmingham and Newcastle (uk), but there is no GTFS information anywhere.

My question is if it is possible to create GTFS data for these cities only having OSM data. If it is possible, there is information/guide to follow?

Thank you,

Proma

The OsmHelper is the part of the script that retrieves data from OpenStreetMap and puts it into the own class/object data structure (to be) discussed in #30. It has included a simple caching functionality that saves data obtained from OpenStreetMap into files and grabs them, if present, to avoid unnecessary hits to the OSM's Overpass API.

I think the current structure of the OsmHelper can be improved and I'm doing a very first draft here, so we have a base for discussion:

OsmHelper

- config
- routes {route_id, RouteMaster}
- stops {osm_id, Stop}
-------
+ get_routes(): Dic of RouteMaster
+ get_route(String route_id): RouteMaster
+ get_stops(): Dic of Stop
+ get_stop(String osm_id): Stop

- query_routes(): Dic of RouteMaster
- query_route(ref): RouteMaster
- query_stops(): List of Stop
- query_stop(osm_id): Stop

+ cache_refresh(name): void
- cache_write(name, content): void
- cache_read(name): content

Some thoughts:

• Using generic read and write caching functions (calling for routemasters, routevariants and stops)
• Use one single cache_refresh function, that takes as an argument if all or a certain part of the cache shall be refreshed.
• All RouteVariants are going to be added to a RouteMaster object, even if there is no master relation in OSM. A simple check creates a RouteMaster either based on the master relation or just on the fly, if non-existant.
• The get-functions are wrappers that check for cached data, return this or query from OSM if data is not cached.

Please add a (open) licence to the repository, to enable collaboration. Thanks!

The current trips creation is very much custom for Florianópolis. This issue here is the place to discuss how a default implementation for trips creation could and should look like. Probably it's going to be in the direction of @jamescr Incofer trips creation, as he is using a standard json containing schedule information for all stops. Fenix and Managua is doing more fancy and custom stuff to create a schedule, which the can live (partially) in the respective individual creators.