There are areas that are marked as landuse=industrial or landuse=retail and it would be classify POIs inside one of them really inside that area. For example there's a retail area that is not inside any city and if I want to search for a McDonald's inside of it in openstreetmap.org I can search for McDonald's Gan HaZafon but I can't do it in mimir.

Israel have a complicated way to define the administration borders so I want to ask how to handle it before opening a PR.

Israel as a country have the following hierarchy:

• Country (Israel)
• Districts (Northern District, Southern District, etc), they are shown as states in cosmogony.
• Council (City council, local council and regional council), they are mapped as cities.
• Villages which are only under a regional council

So basically admin_level=8 can mean either a city council/local council which are actually a city but they can also mean a regional council which is a collection of villages or kibbutzim.

Sometimes the council will be marked as place=city/town and sometimes inside the council area there would be another area that is tagged as place=city/town/village.

Maybe using the place tag would be better for cities and the council would be for state district or something like that?

Some ideas to test the quality of our dataset:

Non closed boundaries
We need to log the list of the boundaries that could not be imported because they are not valid polygon / multipolygon

Hierarchy coherence

• an object should never be bigger than its parent
• child zones of a region should not overlap

Coverage stat and tests
By country statistics:
Compute the geographical coverage in states, cities, etc. (example: 88% city coverage, which means that 88% of the country territory is inside a city)

Persist expected values and test them in CI:
for example:

• Each country must have a 100% state coverage
• France country must have at least 99% city coverage
• etc

Volumetric stat and tests
Stat:
same as below, but only raw numbers, without geographical concerns (example: Australia country has 17 states)

Test:

• Australia should have 17 states
• Australia should have between 1600 and 1700 cities
• etc

Expected values for each country must be in a config file (CSV, YAML ?) and not inside the code source, so that anybody can update it if needed.

Ways can also be a boundary of an administration, for instance https://www.openstreetmap.org/way/14375181

Hi, I'm a contributor to coronadatascraper, an open source project aiming to scrape official websites all around the world for COVID numbers.

My problem is that I'm trying to find a system which we can use to define different hierarchies within countries. I created the country-levels project from Natural Earth dataset, but I'm not happy with it as for example it contains bad Admin 1 divisions in Spain.

Can you help us with your experience? My aim is to make a short code based system, like hasc:ES.CL or something similar which we can use to refer to a region. We'd need to have a GeoJSON for each region + a Wikidata link for population fetching. Is this possible somehow with your project?

Here is a relevant issue which I've opened, if you could contribute to the discussion it'd be great!
https://github.com/lazd/coronadatascraper/issues/286

Hi, I hope you're all doing well!

I have tried building and launching my own Cosmogony, because I have my own genre of this project(https://mappumappu.000webhostapp.com/index.html), albeit on a smaller scale so it really picked my interest!
But when I try to do a full flow of the cosmogony process
(1 - Fetch an ...osm.pbf from Geofabrik;
2 - Build a .json with the Cosmogony project;
3 - try to run Cosmogony Explorer with the .json.)
I get an error in the tests assertion part. I have tried with the Portugal-latest and Poland-latest osm.pbf files. It does, however, build the Explorer with the files already in the Cosmogony project(for example the luxembourg_filtered.osm.pbf in the cosmogony\tests\data folder), although it shows nothing besides the UI of the explorer(don't know if that's the expected behaviour). Do I need to add the .osm.pbf file to the ../tests/data so it knows what to expect?
I don't know what I can do to fix it so all help would be appreciated.

P.S.: There are two images attached, the first with the result of cosmogony json build and the second with more info of the error.
Also, the commands I'm running are:

cargo run --release -- generate -i portugal.osm.pbf

PATH_TO_COSMOGONY_DIR=../cosmogony_data pipenv run inv -e run-local --cosmogony-file-name=cosmogony_Portugal.json --build-dockers

(cosmogony build)

(error in Docker run)

This part of admin is an enclave : https://www.openstreetmap.org/way/137109677
It is part of the city Pieve del Cairo, whereas it is fully inside the city of Isola Sant'Antonio.

In Cosmogony, the enclave is is both cities 👎
This may be because the enclave shares one point (one node) with one of the city boundary.

Hi,

First of all, thanks for this great piece of software.

I may have encountered a bug while working on some partial OSM data set.
I am using geofabrik.de which provides the OSM data per country
http://download.geofabrik.de/

Cosmogony is working well for France, Belgium and Germany.
But for Canada it is not working properly.
http://download.geofabrik.de/north-america/canada-latest.osm.pbf

The shape (the geometry if you prefer) of Canada is not computed correctly.
Displaying this shape shows only the Haïda Gwaïi archipelago and not the main land !
As a direct consequence the hierarchy is not computed correctly

For example, It computes
Mount Royal, Montreal (06), Quebec
But it should compute
Mount Royal, Montreal (06), Quebec, Canada

Any idea how to fix it ?

Best regards,

To generate a cosmogony of the world, you currently need a lot of ram since the whole planet.osm.pbf needs to be loaded.

We could read split non overlapping osm files, without a shared parent (eg. split by continent or country) and merge the generated cosmogony.

Since there is no contract on the indexes (id and parent) being consecutive, it seems a dumb offset by files, so each files will have ids in a separate range, will do the job.

I have a first shot at this, and need to test it on real cases but do you have any thoughts on this (I haven't worked on cosmogony for a while, maybe I'm completely off the mark)?

ping @prhod @amatissart

I'm getting a segfault when executing on the current weekly OSM planet.pbf (i.e. exported 2021-06-26) using the github master.

No problems when executing on a limited dataset (sweden.pbf from GeoFabrik).

Any suggestions?

I'll try again as soon as the next weekly planet.pbf is released.

[2021-06-30T20:26:00Z INFO  cosmogony_builder::additional_zones] Ignoring place with id node:4295699280 and country relation:2202162 as parent
[2021-06-30T20:26:00Z INFO  cosmogony_builder::additional_zones] Ignoring place with id node:4295699281 and country relation:2202162 as parent
[2021-06-30T20:26:00Z INFO  cosmogony_builder::additional_zones] Ignoring place with id node:4318991429 and country relation:2103120 as parent
[2021-06-30T20:26:03Z INFO  cosmogony_builder::additional_zones] We'll compute voronois partitions for 14005 parent zones
intersection failure: impossible to build a geometry from a nullptr in "Geometry::intersection
Last error: TopologyException: Input geom 0 is invalid: Self-intersection at or near point 11.341666499999999 8.7999999999999989 at 11.341666499999999 8.799999999999998" (Unknown GEOS error)
intersection failure: impossible to build a geometry from a nullptr in "Geometry::intersection
Last error: TopologyException: Input geom 0 is invalid: Self-intersection at or near point 11.341666499999999 8.7999999999999989 at 11.341666499999999 8.799999999999998" (Unknown GEOS error)
intersection failure: impossible to build a geometry from a nullptr in "Geometry::intersection
Last error: TopologyException: Input geom 0 is invalid: Self-intersection at or near point 11.550000000000001 8.3166669999999989 at 11.550000000000001 8.316666999999998" (Unknown GEOS error)

...

extrude_existing_town: difference failed for node:150968654: NoConstructionFromNullPtr("Geometry::difference\nLast error: TopologyException: Input geom 1 is invalid: Nested shells at or near point -121.29731 38.243600000000001 at -121.29731 38.24360000000000")
extrude_existing_town: difference failed for node:29941752: NoConstructionFromNullPtr("Geometry::difference\nLast error: TopologyException: Input geom 1 is invalid: Nested shells at or near point -89.753559199999998 43.144259999999996 at -89.753559199999998 43.14425999999999")
extrude_existing_town: difference failed for node:1178614100: NoConstructionFromNullPtr("Geometry::difference\nLast error: TopologyException: Input geom 1 is invalid: Self-intersection at or near point 133.2864505703993 44.099531000885996 at 133.2864505703993 44.09953100088599")
extrude_existing_town: difference failed for node:1346990699: NoConstructionFromNullPtr("Geometry::difference\nLast error: TopologyException: Input geom 1 is invalid: Self-intersection at or near point 133.2864505703993 44.099531000885996 at 133.2864505703993 44.09953100088599")
extrude_existing_town: difference failed for node:1080033707: NoConstructionFromNullPtr("Geometry::difference\nLast error: TopologyException: Input geom 1 is invalid: Self-intersection at or near point 133.2864505703993 44.099531000885996 at 133.2864505703993 44.09953100088599")

...

Last error: TopologyException: Input geom 0 is invalid: Self-intersection at or near point 4.0499999999999998 7.8749999999999991 at 4.0499999999999998 7.874999999999999" (Unknown GEOS error)
intersection failure: impossible to build a geometry from a nullptr in "Geometry::intersection
Last error: TopologyException: Input geom 0 is invalid: Self-intersection at or near point 4.0499999999999998 7.8749999999999991 at 4.0499999999999998 7.874999999999999" (Unknown GEOS error)
intersection failure: impossible to build a geometry from a nullptr in "Geometry::intersection
Last error: TopologyException: Input geom 0 is invalid: Self-intersection at or near point 3.7916669999999995 7.9249999999999998 at 3.7916669999999995 7.924999999999999" (Unknown GEOS error)
Failed to compute voronoi for parent relation:3720587: impossible to build a geometry from a nullptr in "Geometry::voronoi
Last error: TopologyException: Input geom 1 is invalid: Self-intersection at or near point 4.3749999999999991 7.9499999999999993 at 4.3749999999999991 7.949999999999999"
extrude_existing_town: difference failed for node:150963743: NoConstructionFromNullPtr("Geometry::difference\nLast error: TopologyException: Input geom 1 is invalid: Nested shells at or near point -122.78478179999999 45.340228699999997 at -122.78478179999999 45.34022869999999")
extrude_existing_town: difference failed for node:4571365408: NoConstructionFromNullPtr("Geometry::difference\nLast error: TopologyException: Input geom 1 is invalid: Self-intersection at or near point 107.95787713017623 26.570585016953814 at 107.95787713017623 26.57058501695381")
extrude_existing_town: difference failed for node:4571483306: NoConstructionFromNullPtr("Geometry::difference\nLast error: TopologyException: Input geom 1 is invalid: Self-intersection at or near point 107.95787713017623 26.570585016953814 at 107.95787713017623 26.57058501695381")
Segmentation fault (core dumped)

We will need a visualization tool (or maybe several tools ?) for our day to day usage of cosmogony.
The purpose of this issue is to gather our needs. Then we may summarize it in the readme of this repo.

Visual coverage by zone type
we want to explore the world on a map, select an zone type and see the existing zones of this type on the map to get an idea of the coverage
a POC has been done in this repo, PR : osm-without-borders/cosmogony_explorer#1

View zone metadata
we want to select a zone and get all its metadata (names in different languages, wikidata id, etc)

All zones containing a point
we want to click the map and see all the zones including the point

Explore the hierarchy
we want to select a zone, and get an idea of its hierarchy :

• see all its parent zones
• see all its direct child zones
• see all its child zones, cascading the hierarchy
• see its other linked zones

Download some zones
we want to select some zones (selecting from the map and/or using the hierarchy) and download them in a GIS friendly format (at least geojson, with metadata as properties).

Quality assurance
We want some dashboard with the coverage tests results described in issue #4

I think this may be an issue in https://github.com/Qwant/osm_boundaries_utils_rs but I'm reporting here to bring it to your attention, I'm happy to close it if you think it should only exist there.

I realized that ZoneType=Country zones were missing for India and Bangladesh. In fact, it seems much more is missing, possibly because the larger country boundary is missing?

I'm reporting this not as a matter of pedantic precision but with the intention to alert you that it seems this entire area (India/Bangladesh) is missing.

Note however that the flanking islands are present:

For Bangladesh, the only part that is present is this little chunk (osm id: relation:3921211, specifically this chunk, name: রংপুর বিভাগ). Interestingly, I think it is precisely one of those enclaves (not sure)

I think it may be related to many enclaves being present between India and Bangladesh. See this wikipedia article, in fact there are even counter-counter enclaves, see this diagram on wikipedia:

The interesting thing is that southern Netherlands at the border with Belgium there are some enclaves as well but those seem to be handled fine (Yellow is the Netherlands and green is Belgium). Maybe it has to do with there being counter and counter-counter enclaves in India/Bangladesh? Wikipedia seems to claim they both create the only counter-counter enclave in the world.

I think this is just not supported right now?

https://github.com/Qwant/osm_boundaries_utils_rs/blob/master/src/boundaries.rs#L143-L144

I did notice that for example has a role of subarea within Bangladesh, and subarea is not in the list of roles to extract here, might it be as simple as adding it?

The Problem

There are a lot of areas around the world where it is close to impossible to obtain a proper partition of a country with available polygon data for all levels (especially city and below).

On the other hand, OSM has an abundance of city nodes for example.

How we can solve this

A first order approximation of the polygons for levels where OSM has a good coverage of nodes could be done via a Voronoi Diagram (https://en.wikipedia.org/wiki/Voronoi_diagram).

In practice

For preliminary tests, Postgis offers the ST_VoronoiPolygons (https://postgis.net/docs/ST_VoronoiPolygons.html) function.

Example output applied to a set of city nodes:

This method would need to be refined and robustified to adapt to existing admin levels of higher level and situations where data is partially available (and other scenarios that might emerge).

This repository has been created in the QwantResearch organization by convenience, should we move it to another organization ?

And if we do it, should we create a Cosmogony organization or should we use an existing one ?

When looking at a map, it would be nice to know what I am looking at.

I see two possible approaches for now:

By viewport

The area that covers the largest % of the screen (and less that 90%, otherwise it would always display « Europe »).

This requires that one displays a map, which is not always the case.

A geocoder that allows to search a restaurant needs to know where the restaurant is.

The nice thing is that it would show Central Park instead of Manhattan.

By a tag on the administrative level

It could be computed from the size instead of the administrative level. This would allow to handle differently Tokyo (2100km²) and Paris (100km²).

This tag could be attached to profile of who is asking.

Hi,

Years ago, I have been using cosmogony with the parameters filter-langs and country-code They are not recognized anymore by the latest version. I have not found any relevant information on that matter.

cosmogony --filter-langs fr en --country-code FR -i FR/FR.osm.pbf -o FR/FR.2.osm.json

Could someone be nice enough to help on that matter ?

Cheers

Philippe

Unable to build cosmogony as described into README.md + Pull Request

hznor@local:~/mimirsbrunn/cosmogony$ cargo build --release

[...]

  Compiling termcolor v1.1.3
   Compiling heck v0.4.0
   Compiling geo-types v0.7.7
   Compiling flat_map v0.0.10
error: expected `,`, found `.`
   --> /home/hznor/.cargo/registry/src/github.com-1ecc6299db9ec823/geo-types-0.7.7/src/geometry/geometry_collection.rs:111:25
    |
111 | #[deprecated(since = 0.7.5, note = "Use `GeometryCollection::from(vec![geom])` instead.")]
    |                         ^ expected `,`

   Compiling smartstring v0.2.10
   Compiling clap_derive v4.0.13
   Compiling clap_lex v0.3.0
error: could not compile `geo-types` due to previous error
warning: build failed, waiting for other jobs to finish...

This typo has since been fixed in the repo: georust/geo.
Incriminated line 111:

• https://github.com/georust/geo/blob/geo-types-0.7.7/geo-types/src/geometry/geometry_collection.rs

Now corrected by:

• https://github.com/georust/geo/blob/geo-types-0.7.8/geo-types/src/geometry/geometry_collection.rs

The solution seems to be to upgrade the geo-types dependency to version 0.7.8.

I have sent a Pull Request to address this issue.

Duplicates finder

Given a .pbf, an administrative level, we want to find any duplicate.

For instance any way having the same name many kilometers away within the same city. This would help to know a focus should be given on a specific area to avoid any ambiguity for a geocoder

When I browse to http://cosmogony.world/#/7.59/4.26/50.80/168112 and I switch to "Country region", I get an empty region for Bruxelles (normally, it's admin_level=4) but I see the two other regions Flanders and Wallonie (both admin_level=4)
See 1st screenshot

Then, I choose "State". I see Bruxelles area not empty and when I click on "Bruxelles", I get the polygon with admin_level=4 I was expecting one level above (at "Country region"). See 2nd screenshot

It doesn't seem normal. Can you confirm it's an issue?

Here's the issue to start the discussion about schema of our zones "hierarchy".

The aim of this issue is to fill the concerned section in the README

here are my non structured thoughts:

categories

I like libpostal categories, libpostal is quite a reference in the address parsing world and we can hope their categories can handle all the countries specificities all around the world, but I don't think it handles all the corner cases (and it's not the only category out there, for example Wof uses another).

libpostal does not handle non administrative regions apart from the suburb (and maybe the country_region). So it would be difficult to represent Marne-la-Vallée or parc du mercantour

There is also the question of postal codes. I don't know whereas we could/should have postal codes zones in the hierarchy (should we create a separate issue for this ?)

Pyramidal hierarchy or graph-based ?

Can a zone have at most one parent or can it have several.

I fill that it might be a failing of Wof to have a pyramidal hierarchy. I don't think it will complicate cosmogony that much to be able to have several parents.
I don't think it's useful for purely administrative regions (but maybe there are countries where it's relevant), but for non-administrative regions I think a pyramidal hierarchy will be too restrictive.

Eg. what would we link Marne-la-Vallée to ? ile de france ? but then it would be difficult to link it back to the cities that are part of it.
The same apply for non official suburbs that can span across several district

links coherence

Wof hierarchy is nice, but being linked to all parents brings incoherence (like france empire that contains france country but the empire has less descendant than the country.
I fill like outputting only the first level of relationship force the dataset to be coherent (even if so it will make the dataset harder to use without tools)

When I try to generate a .json file based on either unfiltered or filtered by boundaries/admin_level planet.osm.pbf the process gets 'killed' after a while. I'll leave more info below and attached in the image.

I'm using WSL2 and the steps I'm doing are just:
1 - Get planet.osm.pbf file(in my case I got it from the torrent version);
2 - Filter pbf file by either boundary or admin_level, which would result in a considerably smaller file. Additionally I'm also filtering some other info out. The commands I'm using are these:
osmium tags-filter planet-latest.osm.pbf r/admin_level --overwrite -o planet-admin.osm.pbf
osmconvert planet-admin.osm.pbf -o=planet-admin.osm
osmfilter planet-admin.osm --drop-tags="barrier= building= highway= landuse= office= place= waterway=" -o=planet-admin-noplace.osm
osmconvert planet-admin-noplace.osm -o=planet-admin-noplace.osm.pbf;

3 - Build cosmogony with command:
cargo run --release -- generate -i ../../planet-admin-noplace.osm.pbf
It's in this command that the process is getting "killed".

(Note: This happens with unfiltered planet.osm.pbf as well. I also don't know if this could be related to #118 ...)

I'm not able to access the website - is this service still running or are there any DNS issues?

To make it easier to reuse, we need to add the ability to persist our cosmogony to a postgis database.