mapbox / hecate Goto Github PK

Add all feature properties to tiles endpoint to allow for full styling

Add optional query parameters to the delta list endpoint to allow querying by something like: start/end, last x minutes and by a specific user.

Allow deletions of bounds via the web UI

On successful creation of a style, the server should respond with the numeric ID of the style instead of a true

error[E0432]: unresolved import `std::boxed::FnBox`
 --> /build/.cargo/registry/src/github.com-1ecc6299db9ec823/rocket-0.4.1/src/fairing/ad_hoc.rs:2:5
  |
2 | use std::boxed::FnBox;
  |     ^^^^^^^^^^^^^^^^^ no `FnBox` in `boxed`

error: aborting due to previous error

Seems similar to rwf2/Rocket#1042
May you please update to Rocket 0.4.2?

cc/ @ingalls

A GET request with an invalid bbox to the /api/data/features/endpoint currently returns a 500 error.
For example: http://localhost:8888/api/data/features?bbox=badbbox

It might be nice to have some validation of the bbox parameter and get a 400 response if the bbox isn't in the expected format.

Not sure where the server is erroring out, but maybe here? /src/lib.rs#L460

cc: @mattficke

Every function should be fully Rust Doc'd

https://rustbyexample.com/meta/doc.html

Once all functions are rustdoc'd we should enforce rust doc compliance

Now that the styles endpoint is in place, add adjustable limit to the number of styles per user.

Once feature props are in the vector tiles add the style table and ui components for multiple styles

Once #16 has landed let's get some example schema files that we can include with the project.

Let's start by creating a JSON schema for validating the properties of an address type feature.

An example:

{
    "source": "openaddresses,us-ca-alameda_county",
    "number": "123a",
    "street": "Main St"
}

An example of a JSON schema for OpenAddreses is here: https://github.com/openaddresses/openaddresses/blob/master/schema/source_schema.json

An in depth book about JSON Schema is here: https://spacetelescope.github.io/understanding-json-schema/

A quick walkthrough is here: http://json-schema.org/example1.html

Context

@samely and I noticed that certain features weren't being rendered in the Hecate UI, though they were queryable. See the following errors in the console:

The address we were looking for is 900 Windemere Cir, Longview, Texas 75601, United States, z/x/y 19.39/32.5103247/-94.6816671

cc/ @ingalls

Ensure the web UI converts from the default timezone to give accurate timestamps

Add option to delete your own styles

Hi Nick,

should Hecate be used from JOSM as OSM API replacement? Or with proprietary Mapbox tools nobody (yet) knows?

cc/ @ingalls

Since there was no INSTALL file I've just run cargo build --release

This fails due to unsupported proc_macro. My rust is 1.33, so this shouldn't be too old??

How may I correctly compile hecate?

cc/ @ingalls

Per convo with @ingalls 💻

🕳️ There is no way to identify where/why a corrupted .geojson file fails while importing whit the current message we get.

ok - beginning upload of 1595 features
ok - upload complete
ok - beginning upload of 1593 features

Upload Error: Uploaded to Line 1595
Body must be valid GeoJSON Feature

Suggested resource

👉 geojsonhint

Next steps

• Add geoJSON validator

Context

Best practice is to require email confirmation

Ensure we follow best practice, allowing hecate to be configured to send outgoing email from a provided address

• Confirm Email on Account Registration
• Add ability to Reset Password

cc/ @ingalls

While recognizing that there's some contention in how best to represent versioning of APIs over time, it still seems like common practice to put version numbers in URLs. The OSM part of the API is versioned to match the current OSM version but other parts of the API are not. Were we to write scripts that interacted with the API, it might be nice to be able to easily enforce their compatibility as the underlying API might change out from under them, and a version number seems like a good way to do that. (Also, it might be nice to more clearly separate the URL namespaces of the OSM compat part of the API from the "native" part?)

Add endpoint for setting/removing admin privleges

A new key value was added in #70, which helps us avoid creating duplicate features in the database. The current behavior is to return an error when attempting to insert a feature with a pre-existing key. It would be helpful if there was also an "upsert" option, so that the conflicting feature is updated with the new value instead.

Postgres includes this feature: https://www.postgresql.org/docs/9.6/static/sql-insert.html

cc @ingalls @lukasmartinelli @yhahn

Add button on feature panel to display feature history.

The feature history API can be used to retrieve the data

http://localhost:8000/api/data/feature/1/history

Context

If the server contains no global styles but the user has styles, the No Custom Styles should be hidden.

cc/ @ingalls

@ingalls @samely in https://github.com/mapbox/Hecate/blob/master/README.md#additional-members I see

id: The unique integer id of a given feature. Note that all features get a unique id accross GeoJSON Geometry Type

key: [Optional] A String containing a value that hecate will ensure remains unique accross all features. Can be a natural id (wikidata id, PID, etc), computed property hash, geometry hash etc. The specifics are left up to the client. Should an attempt at importing a Feature with a differing id but identical key be made, the feature with will be rejected, ensuring the uniqueness of the key values. By default this value will be NULL. Duplicate NULL values are allowed.

I think that the roles of these properties should be switched to make the spec more aligned with GeoJSON on the web.

"key" as defined in the Hecate spec is exactly what "id" is meant to be in the GeoJSON spec. GeoJSON ids should be strings (URIs or UUIDs, in my opinion). The only reason it says "or int" in the GeoJSON spec is because the IETF WG failed to nail down the specification well and because the GIS architects in the working group had a hard time seeing past the shapefile paradigm.

I realize that Hecate needs an integer key to integrate with other parts of our platform, and I think "key" would be the perfect fit for a semi-proprietary GeoJSON extension member.

Object stores like CouchDB use strings for identifiers (UUIDs and GUIDs are recommended at http://docs.couchdb.org/en/2.1.1/intro/api.html#documents). Hecate should, too, for the same reasons.

Add a server endpoint for setting server wide settings.

The first server wide setting would be server style.

Context

The admin panel for styles overflows after 3 styles. This should scroll

cc/ @ingalls

Implement GET: /api request which would return a JSON object containing public metadata about the server.

{
     "version": "0.15.0"
}

URL: /api/data/bounds/us_ca

<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>

Test database connection in main.rs before the rocket server is instantiated and throw human readable error if connection problems are encountered.

Current error is:

ERROR 2018-06-14T11:50:38Z: r2d2: IO error: Cannot assign requested address (os error 99)

cc/ @samely

Context

Does hecate support the export of osm.pbf files? This is what the official server does; I would like to have this to create Mapnik tiles from data inside a hecate instance.

cc/ @ingalls

Add GET endpoint for /api/tiles/<z>/<x>/<y>/meta

This would return a JSON object containing at minimum the time at which the vector tile was created.

{
    "created": "ISO DATESTAMP"
}

Allow changing of a users password

Following the installation guide, the database connection cannot be established:

ERROR 2018-11-19T23:24:23Z: r2d2: database error: FATAL: password authentication failed for user "hecate"

It does work however with user postgres for the db writer role. For example, this will work:

cargo run -- --database_read "hecate_read:123456@localhost:5432/hecate" --database "postgres:123456@localhost:5432/hecate"

This doesn't work:

cargo run -- --database_read "hecate_read:123456@localhost:5432/hecate" --database "hecate:123456@localhost:5432/hecate"

(I changed passwords prior to running the example above in order to rule out empty password config issues).

Add a UI component for the styles work introduced in: https://github.com/ingalls/Hecate/releases/tag/v0.14.0

Current Hecate relies on the Rocket web framework, which requires the use of the unstable/nightly Rust compiler. Nightly contains language features that are subject to change, and also breaks occasionally, and at least per the current instructions (which recommend installing the latest nightly), each developer contributing to the project might end up with a different compiler with different features, behavior, and bugs depending on when they happen to set up their dev environment.

Rocket itself doesn't seem particularly close to running on stable (it seems like a project mostly interesting in pushing the language envelope rather than running on stable), and independent of stable vs unstable, the lead dev anticipates a major API overhaul to move towards async as the Rust async story matures.

The ergonomics do seem nice, but in the interests of building on a solid foundation and making it easy for everyone to work on it, I wonder if we shouldn't consider moving to a framework with a stable API that runs on the stable compiler? Nickel seems like a possibility, or if we wanted to get a head-start on async, maybe Gotham ?

Context

While exploring some of the addresses we have loaded into Hecate, I noticed that I was unable to retrieve metadata information for some point features. I'm not seeing any errors in the console and there doesn't appear to be any pattern to the point features that do/do not return.

cc/ @ingalls

Show an option to load the history of a given feature via the Feature Panel

Then show via full history or JSON Diff view

Hard-revert a delta if you need to erase all traces of a near-catastrophe.

Context

Ensure the UI doesn't attempt to display the EOF character from the streaming UI

cc/ @ingalls

Add GET endpoint for /api/tiles/<z>/<x>/<y>/regen

When requested this endpoint would skip the cache and regenerate/recache the requested tile

We've had discussions in the past about running multiple internal instances of the OSM stack for different data projects, and I imagine we might want to do the same with this tool. It would be nice if you could then later combine the contents of two such servers and not have their IDs collide. Could we consider UUIDs instead of autoincrementing integers? (Not sure this would play nice with JOSM 🤔 )

Add toggle for setting style as public/private

Hecate currently generates Vector tiles on request and caches them for 24hrs before they are regenerated at the next request.

This behavior could be greatly improved once https://github.com/ingalls/tilecover is feature complete (missing Polygons and MultiPolygons). Generating a tile-cover report on each delta, and invalidating referenced tiles.

Next Actions:

• Finish ingalls/tilecover Polygon & MultiPolygon Support

Context

cc/ @ingalls

Rel: ingalls/hecate-example#2

Currently Hecate allows invalid/improbable geo coordinates like [ 938.832710598893414, -199.38005089759827 ]. I'm not sure if this is necessarily a bug, since it's within the scope of an ambitious imagination that someone might want Hecate to be able to eat strange GeoJSON points.

But it seems pretty extreme, and I'd reckon validating coords would save more headaches around "Oops, I mixed up my lat and long" than it would "Why won't Hecate allow me to plot a point off the edge of the earth?"

A next question would be if this kind of validation should happen "hard", like somewhere around https://github.com/mapbox/Hecate/blob/master/src/feature/mod.rs#L179, or if this job might be abstracted to another kind of configurable schema.

Context

Look into a diff algorithm for storing the geometries & properties on a given delta. Since our delta storage tables currently holds the complete geometry/properties, it grows very quickly over time - especially for large, automated edits.

cc/ @ingalls

Allow an API to block/ban a user without deleting their account.

Allow admins to access/search through a list of users.

It looks like Hecate preserves the OSM distinction between publicly accessible read-only content and auth-required read-write access. I wonder if this makes sense going forward. If a usecase for this service is potentially store internal-use/private data, reads would also need to be protected. It seems like, freed from the constraints of OSM's code, you'd either want to secure everything at the network level, or everything with username/password, but I'm not sure why you'd additional levels of auth/protection for some parts of the service than for others. Maybe it's just a constraint imposed by a desire to still interoperate with JOSM?

A key: null JSON property on an uploaded feature should not return a Duplicate Key Value error when attempting an upload.

Duplicate NULL values are allowed.

cc/ @mattficke @lukasmartinelli

 {15,2,3}
 {4,5,6,7,8,9,10,11}
 {5,4}
 {5,4}
 {5,4}
 {4}

Output from https://github.com/ingalls/hecate-example where the first feature uploaded was given id: 15.

Hecate should either error out on action: create features with an id or version property or should dispose of them before entering them into the database.

Will sketch out erroring, as per the docs they should not be included.

Affects

When reverting a small mistake I made, the references in the affected property didn't actually refer to the objects modified in the delta, resulting in the reversion creating a larger mess.

cc/ @mapbox/geocoding-gang

Currently all of the get style list endpoints return the uid of the user. This makes subsequent requests straighforward as they use the uid over the plaintext username. This makes display more difficult however as it would require two request.