Giter Club home page Giter Club logo

labs-postgis-preview's Introduction

CircleCI

PostGIS Preview

A lightweight node api and frontend for quickly previewing PostGIS queries. Pull Requests Welcomed! Take a look at the open issues

preview

Gitter Chat

Join the conversation on Gitter

Contributing

Take a look a the open issues. Come chat on Gitter if you have questions. Create a Pull Request on the develop branch.

Why

Our team at the NYC Department of City Planning needed to be able to test out PostGIS queries in a local environment and iterate quickly. CartoDB provides this functionality, giving users a SQL pane and a map view to quickly see the geometries returned from the database (This UI and SQL preview workflow are inspired by the CartoDB editor)

When asking on Twitter if anyone had solutions to this problem, responses included:

  • Run queries in pgadmin and use ST_asGeoJson(), copy and paste the geojson into geojson.io
  • Use QGIS dbmanager. This works, but requires a few clicks once the data are returned to add them to the map.
  • Use various command line tools that show previews in the terminal or send the results to geojson.io programmatically.

How it works

The express.js app has a single endpoint: /sql that is passed a SQL query q as a url parameter. That query is passed to PostGIS using the pg-promise module. The resulting data are transformed into topojson using a modified dbgeo module (modified to include parsing WKB using the WKX module), and the response is sent to the frontend.

The frontend is a simple Bootstrap layout with a Leaflet map, CartoDB basemaps, a table, and a SQL pane. The TopoJSON from the API is parsed using omnivore by Mapbox, and added to the map as an L.geoJson layer with generic styling.

How to Use

  • Clone this repo
  • Have a PostGIS instance running somewhere that the node app can talk to
  • Create a .env file in the root of the repo based on .env-sample, setting DATABASE_URL with a connection string for your database (DATABASE_URL=postgres://user:password@host:port/database)
  • Install dependencies yarn install
  • Run the express app yarn start, or yarn devstart if you are developing and want the server to restart on file changes.
  • Load the frontend http://localhost:4000
  • Query like a boss, see your geoms on the map!

Contributing

Pull Requests Welcomed! Take a look at the open issues Join the conversation on Gitter

Notes

  • PostGIS preview expects your geometry column to be called geom, and that it contains WGS84 geometries.

Tests

yarn test

Optional Feature Requirements

At some time in the near future, this app will allow the option to serve mvt locally, instead of geojson. However, for that to work, you will need the following:

  • PostGIS 2.4 (and PostGres 10); and,
  • lib-protobuf-c.

labs-postgis-preview's People

Contributors

allthesignals avatar andycochran avatar brianbancroft avatar chriswhong avatar jczaplew avatar jimmyrocks avatar jwalgran avatar mapmeld avatar maptastik avatar matt-baker avatar mgd722 avatar vitaly-t avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

labs-postgis-preview's Issues

Cannot scroll table

When the number of items in the table exceeds the height of the window it isn't possible to scroll down

License missing

Am I wrong, or this repository doesn't have yet a license? Really interesting -- and useful!!! -- work! I'm interesting in contributing. :-)

Add database connection modal

Let's make this thing even more user-friendly and just have a modal for the database connection info.

I am not sure how to implement this.

  • The credentials need to make their way back to the server and stored somewhere (new express endpoint?)
  • They should persist when the server stops and starts again (manually edit .env file?)
  • Let the user test the connection before closing the modal

Any ideas?

Here's a connection modal from Franchise we can use for inspiration. We should totally throw in an elephant or two.

franchise

Restore query history

Let's get the query history working again.

Store 10 previous queries in localstorage, read the latest one when the app loads and pre-load it into the SQL pane.

Records with no geometry map to 0,0

Records with no geometry (NULL or blank) map to 0,0 in PostGIS Preview, which is is confusing to the user. It would be preferable if records with no geometry just did not render.

Data table persists

The data table persists from a previous query. I queried the data and only included the field geom; however, the fields and data from a previous query, which included more fields other than geom, remain in the data table in Data View.

Add copy-able lat/lon div to map view

Add a div to the map view that shows the current lat/lon of the center of the map, so that it can be copied and pasted easily into your postGIS queries.

Maybe also wrap it in it as ST_Geomfromtext() for those who may need it in that format?

Start / Setup environment issues?

Maybe there's a version requirement that isn't documented or something? Cannot even start:

yarn install v1.7.0
[1/4] ๐Ÿ”  Resolving packages...
success Already up-to-date.
โœจ  Done in 0.43s.
Sparrow:~/Code/labs-postgis-preview pramsey(develop)$ yarn start
yarn run v1.7.0
$ node server.js
/Users/pramsey/Code/labs-postgis-preview/routes/sql.js:38
    );
    ^

SyntaxError: Unexpected token )
    at createScript (vm.js:56:10)
    at Object.runInThisContext (vm.js:97:10)
    at Module._compile (module.js:549:28)
    at Object.Module._extensions..js (module.js:586:10)
    at Module.load (module.js:494:32)
    at tryModuleLoad (module.js:453:12)
    at Function.Module._load (module.js:445:3)
    at Module.require (module.js:504:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/pramsey/Code/labs-postgis-preview/server.js:39:17)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Sparrow:~/Code/labs-postgis-preview pramsey(develop)$ git status
On branch develop
Your branch is up to date with 'origin/develop'.

nothing to commit, working tree clean

Improvements to NULL geom handling

Current issues with NULL geom handling:

  1. To provide more context to issue #44 , currently the app only checks whether the table's first row has a null geom and then assumes 0,0 for the geom of all records rather than checking through the rest of the records which may have valid geometries.

  2. If there is a null geom further down in the table, the query throws an error "must start with number, buffer, array or string"

App could default to only showing records where geom is not null -- or find a better way to still include the null geom records in the table but not render them on the map. Maybe the rendering component could always default to include "where geom is not null."

Add max-height to infowindow

When there are lots of columns, the infowindow becomes very tall and the map pans to fit the top of the infowindow in the view. Suggest max-height + scrolling once it gets past a certain height.

Relation to Table does not exist

I am typing my query in query tab as
Select * from Fuel (My Table/layer present in the postgis Database)
But em getting message realtion "Fuel" Does not exist. Same result for other tables.
When i am using this query in pgadmin there is no issue.
Can anyone guide me what to do.
Any suggestion
image

Allow user to switch between GeoJSON Output and MVT

  • User should be allowed to switch between using GeoJSON and MVT
  • User should be made aware on what is required to make it work
  • The app should fail gracefully (give notification and switch back to geojson) if requirements are not met.

Export to CSV -{"error":"options is not defined"}

I love the application by the way. The easiest and most comprehensive solution of its kind I have found,

However when I attempt to export to csv I receive this error:

{"error":"options is not defined"}

Additionally the http request in the url is as follows.

http://localhost:4000/sql?q=%2F_%20___Type%20your%20SQL%20here___%20_%2F%0ASELECT%20*%20FROM%20apex_join_pts_wgs84%20WHERE%20%22COMPANY%22%20%3D%20%27DIAMOND%20SHAMROCK%27%3B&format=csv

Was there some setup or configuration that I have missed?

Add Table View

The geojson properties object returned by the API can be thrown into a table view. Bootstrap has some with "fake pagination" that will work, but I would love to hear recommendations.

Refine Datatables Layout

Datatables has some advanced configuration for layouts, scrolling, etc. I would like to have the header and footer of the datatable remain static (fixed to the top and bottom of the container) but also allow for left-right AND top-bottom scrolling.

Allow for queries with no geom

Queries with no geom throw an error, presumably when dbgeo is doing its thing. If the query has no geom, we should flash a message on the map that says there was no geom, but you can see the rest of the results on the data view.

DATABASE_URL in environment over-rides configuration

This may be "by design" actually, since it makes it faster to switch databases, but it's a surprise if (as with me) you have a legacy DATABASE_URL lying around for other reasons. Feel free to close if this is expected behaviour.

Handle mix of valid and null geometries

When a query result contains some rows with a valid geometry and others with a NULL geometry the following error is thrown:

App.js:63 Uncaught (in promise) TypeError: Cannot read property 'type' of null
    at <anonymous>:85:62

This occurs because line 63 of App.js checks the geometry type by looking at the first item in the GeoJSON FeatureCollection, but if that item is NULL the error is thrown.

Handle Points and LineStrings

Right now the app only visualizes polygons. Add some handling for displaying Points and Linestrings.

This is easy to do with geojson by checking the type of a feature's geometry.
For MVT mode, we may need to return the geometrytype in the initialization response.

SSR?

This wouldn't be part of a milestone (or a pressing issue for that matter), but if we're serving a react app, we might want to do server side rendering of the index route. Webpack, babel... all the fun stuff we hate setting up.

The benefits is that the code will be cleaner, and it will be a lot easier to add dependencies.

The downside is that we'd have to config ssr, babel and Webpack.

Simple Infowindow

Add a simple infowindow to the leaflet map that shows all attributes when you click on a geometry

Data table persists

Data table persists if no attributes are included in the last query. My result set only included a column called "geom" but the table in Data View included columns/data from the last query, which had other columns other than geom.

Disable table view in MVT mode OR get table view working in MVT mode

The table is simply created from properties in geojson mode. When using MVTs, we don't have data to populate the table view, so we either need a proper pagination endpoint on the API, or we should just disable table view when in MVT mode.

Since MVT mode is experimental at this point anyway, I vote for the latter, just disable table view if the user is using MVTs.

Test/Handle Big Requests

What should the app do if the request is too big (so much returned that rendering it in leaflet will be problematic).

Should it just be a row limit with a message saying the response is truncated, or should the API not send anything if it's going to be too big?

Refactor frontend using a framework

The frontend has Query UI, error notifications, map, table view, etc. I think it's time to consider a framework.

Our team at NYC Planning Labs has had a lot of success combining Ember.js with MapboxGL.

Also open to using React, just think we should commit the built frontend code so it's easy for people to get started quickly without the build step.

Any suggestions?

Data table persists

The data table in the Data View persists from a past query. I queried the data and only returned the field geom; however, the data from a previous query, which included other fields than geom, remained in the data table.

Add Download feature

This could all be done client-side with data urls. Download the geojson currently being shown, or transform it into CSV.

Make this an Electron App

Not all PostGIS users are web developers.

As late as 2015, I had no idea what build tooling was. NPM, HTTP Servers, not to mention front-end architectures. For those people who want the means to use this as a straightforward tool to preview and test PostGIS queries on a local database, we should think about making this an Electron App.

I have no idea how we should tackle this, and I don't believe this should be on our roadmap for 0.2

https://electronjs.org/

Multi-line queries sometimes throw errors

postgis_preview_and_script_js_ _postgis-preview

If you put your SQL on several lines like in this screenshot it will return an error. However I have seen the multi-line queries work after deleting the new line and adding again, sometimes... its very odd.

Can anyone repro?

Any ideas on this one @mgd722?

Add recent queries

My thought is that you can add a backward and forward arrow to the queries and 20 (or whatever) recent queries can be stored in localstorage.

Wish List: Query running

Add query running loader gif so that i know its in process. Additionally clear previous errors on run.

Data table persists

The data table in the Data View persists from a past query. I queried the data and only returned the field geom; however, the data from a previous query, which included other fields than geom, remained in the data table.

Restore data table

Let's get the data table working again.

  • User should be able to use the buttons in the header to toggle between map and table view. (Header needs to be brought into App component)
  • Now may be the time to consider a different table that may play better with react and be more performant.

Display several spatial columns

Hi postgis-preview is really great! Thanks!

Only I couldnt manage to display two different geometry columns from one query as in


SELECT st_astext(sq.point), sq.point as geom, sq.poly  as geom2
from 
(
select ST_GeomFromText('POINT(-71.064544 42.28787)',4326) as point,
ST_GeomFromText('POLYGON((-71.1776585052917 42.3902909739571,-71.1776820268866 42.3903701743239,-71.1776063012595 42.3903825660754,-71.1775826583081 42.3903033653531,-71.1776585052917 42.3902909739571))') as poly
) sq;

Restore map popup

When we moved to mapboxGL, we lost the simple popup on clicking a feature on the map.

The popup should include a simple table of the keys & values for the feature's properties.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.