Simplify the early stages of creating a `StreetNetwork` from OSM data about osm2streets HOT 6 OPEN

Thanks for writing this up! I'm totally in favor.

The trimming operation is better defined in the domain of the OSM data itself, because the OSM data is points and lines.

I really like this idea! I think I've tried to get external tools to do that clipping before, but never gotten exactly the right thing working. Currently we tend to assume something like complete_ways strategy (https://osmcode.org/osmium-tool/manual.html) is passed in as input.

One consequence of rethinking the logic at the OSM level is treating very long roads or railways equivalently. Currently we special case railways that start and end outside the boundary but pass somewhere inside. No reason we shouldn't treat roads the same. (Old traffic sim assumptions leaking through were the original reason.)

The takeaway here is that StreetNetwork should store a full copy of the original OSM data

In https://github.com/a-b-street/osm2streets/tree/osm_tags I'm getting rid of osm_tags on Road. Separately a caller could hold onto a Document, which maps way IDs to Tags. A Road tracks multiple way IDs. From a code structure perspective, does that still seem like what we want to do?

I agree this would very nicely clean up current hacks around crossing_nodes. Right now a very downstream caller outside of this library makes use of that data, but they could do the association with their own road representation themselves. Eventually osm2streets will natively handle crossing and barrier points in some other way.

The idea is that an interactive editor...

That's sort of the intention behind the raw map editor, but in its current form, it's overly complex because of the old way that IDs used to be managed. The refactor you describe will make building this kind of app much, much easier.

from osm2streets.

I think I've tried to get external tools to do that clipping before, but never gotten exactly the right thing working. Currently we tend to assume something like complete_ways strategy (osmcode.org/osmium-tool/manual.html) is passed in as input.

Yeah, the complete_ways strategy is how I imaging raw OSM data coming in because that's how the data download in JOSM works. All ways are complete and some stick out of the bounding box. The UI then draws the downloaded area on the map (showing "out of bounds" as yellow hatches), so the user can understand both 1) the extent of the objects they are editing and 2) where there may be unseen features nearby or connected to the loaded objects.

Because the OSM data model allows splitting ways wherever the mapper wants to, I think any "out of bounds" data should be ignored consistently, so it would be really cool for OsmExtract to clip strictly to the boundary polygon, truncating ways that cross over the boundary by inserting dummy nodes right at the boundary. It would want to keep areas as closed loops by doing a boolean intersection I guess... and maybe the clipped result is less useful for areas, but I feel like clipping our linear highway features in this way is still the best compromise.

OsmExtract might need to allow multiple clipped objects per id for ways that leave the boundary then come back in, or otherwise represent which sections are outside the boundary.

One consequence of rethinking the logic at the OSM level is treating very long roads or railways equivalently. Currently we special case railways that start and end outside the boundary but pass somewhere inside. No reason we shouldn't treat roads the same. (Old traffic sim assumptions leaking through were the original reason.)

Yes, I'd like to see the clipping be exhaustive at the OSM level, doing proper intersection checks for every line segment, though moving the clipping logic out of StreetNetwork land and into OSM land will be an improvement, even if the implementation is a rudimentary to begin with.

The takeaway here is that StreetNetwork should store a full copy of the original OSM data

In osm_tags I'm getting rid of osm_tags on Road. Separately a caller could hold onto a Document, which maps way IDs to Tags. A Road tracks multiple way IDs. From a code structure perspective, does that still seem like what we want to do?

I like the idea of OsmExtract being the ergonomic collection of raw data that the caller would refer to. Maybe the StreetNetwork (optionally) stores the extract itself and maintains a two-way mapping as it makes edits. That is, Roads and Intersections etc. point back to the OSM IDs and OsmExtract points to StreetNetwork IDs. StreetNetwork would be responsible for keeping this mapping up to date as it modifies its representation. It depends if node_id_to_road_id and node_id_to_intersection_id are useful long term.

The idea is that an interactive editor...

That's sort of the intention behind the raw map editor, but in its current form, it's overly complex because of the old way that IDs used to be managed. The refactor you describe will make building this kind of app much, much easier.

Even before we get there, thinking about how StreetNetwork can encapsulate the complexities behind those sorts of operations is how I make progress on understanding those complexities :D

from osm2streets.

It would want to keep areas as closed loops by doing a boolean intersection I guess

We have some support for that today. Areas get more complex, because they're often relations with some of the members missing in the complete-ways extract.

allow multiple clipped objects per id for ways that leave the boundary then come back in, or otherwise represent which sections are outside the boundary.

This is the case that

StreetNetwork would be responsible for keeping this mapping up to date as it modifies its representation

Hmm, maintaining a two-way mapping sounds tougher. Right now each Road and Intersection points back to a list of OSM IDs. We can always go build the reverse mapping (many-to-many I think...) by looking at the whole StreetNetwork. If we wanted to cheaply maintain a two-way mapping, we'd have to decide where to store it and plumb it around. If it lives in StreetNetwork, the borrow checker might fight us if we have an &mut method on Road or Intersection changing IDs. But maybe we never tend to do that in practice.

from osm2streets.

I had a go at starting this idea, because our current haphazard handling of map edge geometry is complicating the geometry refactor. A few problems...

Tests: We don't have an explicit boundary for any of the tests. I think we need to go back and record one, because we otherwise assume a bounding box covering everything in the OSM input. It'd be good to do this anyway, so we can make obnoxious cases like quad_intersection stop stretching super far because of the old behavior about not clipping light rail. StreetExplorer could remember + generate a GeoJSON with the bbox passed to Overpass, and add a button to download it.

Implement OsmExtract::new(doc: osm::Document, boundary: Option<Vec>) -> Self which doesn't depend on anything StreetNetwork. It can clip the raw OSM features as it reads them from doc.

Currently OsmExtract has callback methods like handle_way. This is so a consumer like A/B Street can handle non-road stuff still. This is making me debate where the OSM-level clipping should happen. Maybe handle_way should take the Way, mutate it by clipping if needed, and return Option<Way> to the caller. If None, it was a road and has been handled. If not, the caller can use it, and additionally it's guaranteed to be clipped.

This gets a bit tricky with areas. We already have glue_multipolygon and a few other things in this repo anyway, only used by A/B Street. Maybe we move some of the area clipping logic from A/B Street back here. Eventually osm_reader + Document could be a separate Rust library, totally orthogonal to osm2streets concerns. There are a few OSM-reading crates out there, but I don't think any attempt clipping. Which is maybe kind of reasonable, because possibly the right thing to do depends on semantics deeper than the OSM data model. (A way can represent both an area and a linestring-like thing, and the clipping behavior for each is a bit different.)

I'm leaning towards a different API... after reading a Document, we optionally clip it. Nothing to do with OsmExtract -- it would already receive a clipped document. That avoids the funny handle_way API.

I think the complication of clipping at the document level is introducing geometry that doesn't come from OSM. Way already has both nodes and pts, so I think this'll be OK.

from osm2streets.

Something started in a branch; it's much simpler. Two more questions raised:

1. Is it time to get rid of OriginalRoad? Can we just remember a Road's osm_way_ids? Is it ever useful to track which segments they came from, or could we just stitch this together from the two endpoints osm_node_ids?
2. If we don't provide a clip and infer one from the OSM data, we'll never detect map edges. That's... kind of expected. Should we always require a boundary as input maybe, to simplify the API? I'll browse through all my use cases and see if there's always one readily at hand.

I'll open a few PRs to start pushing this through.

from osm2streets.

I had to back away a bit more from the idea of clipping an OSM document being blind to the types of things being clipped. When loop roads like https://www.openstreetmap.org/way/128464171 are split by a boundary, it was breaking previously. The previous way was also throwing out some geometry for large multipolygon relations that crossed the boundary.

We can revisit generic clipping again, but we need osm2areas or something with tests first. :) A/B Street importer is my proxy for that in the meantime.

from osm2streets.