Save objects to file after every API query about osm2gtfs HOT 7 CLOSED

This might be easy to do within refactoring OsmHelper #31

from osm2gtfs.

Would it make sense to save to files after queries and cache again after building the structure into the RouteMaster, Route and Stop objects? Or should we only cache right after the query? Or should we only cache when everything is built into the objects?

from osm2gtfs.

Since disk access is a lot faster than network access, we should cache as often as possible.

Also, we might be able to use the latest change of the relation/node to decide whether we need to refresh the cached data.

I will push a commit soon that adds the first piece to that puzzle. Hopefully, that will allow us to build up the cache faster in several steps when tweaking the queries.

from osm2gtfs.

Just a note/though: Caching should probably be handled through a separate class and not inside the OsmHelper itself. It has not necessarily something to do with OSM data retrieval, so why should it live there? A separate class would make more sense and we'd be prepared for the future in case caching becomes more advanced.

from osm2gtfs.

With the latest commit caching now lives in a separate Cache class. We also cache after every find_best_name_for_unnamed_stop call. The only thing that is missing for this issue, is to actually cache queried raw result sets right after querying. But this doesn't seem to be possible, because Python can't pickle lambda functions:

If doing:

# Obtain raw data about routes from OpenStreetMap
result = self._query_routes()
# Cache and return whole result set
Cache.write_data('routes_raw', result)

I get an ugly error:

pickle.PicklingError: Can't pickle <function <lambda> at 0x7ff255d0e320>: it's not found as overpy <lambda>

Any solution to this? Or shall we close this issue as resolved.

from osm2gtfs.

Is it a big problem to not cache query data? Isn't it sufficient to just cache the resulting objects? Creating them shouldn't take too long, right?

from osm2gtfs.

It's fine. Just for development purposes it would have been nice to cache before. But it's not that important. I was just mentioning it to explain that the initial request was not completely realizable. Closing here then.

from osm2gtfs.