Giter Club home page Giter Club logo

mesh-api's Introduction

Mesh API

Screenshot of Google Earth showing data from Mesh API. There are blue dots showing nodes on buildings and green and yellow lines showing links between nodes.

๐Ÿšง Work in progress!

Contributing

Before committing code, please run yarn precommit to format your code and run the tests. Only commit your code when it's formatted and the tests pass. You can add it as a git precommit hook if you like.

Endpoints

https://api.nycmesh.net/v1/nodes
https://api.nycmesh.net/v1/links
https://api.nycmesh.net/v1/buildings
https://api.nycmesh.net/v1/members
https://api.nycmesh.net/v1/requests
https://api.nycmesh.net/v1/search
https://api.nycmesh.net/v1/los
https://api.nycmesh.net/v1/kml

Architecture

  • Netlify Functions for hosting
  • Express for handling requests
  • PostgreSQL for main db
  • PostGIS for line of sight db
  • DigitalOcean Spaces (S3) for storing panorama images
  • Auth0 for access control

Running locally

Clone the repo: git clone [email protected]:olivernyc/nycmesh-api.git
Install dependencies: yarn install
Run the local server: yarn start

You'll need a .env file with the following values:

DATABASE_URL=postgres://$user:$pass@$host:$port/$db
LOS_DATABASE_URL=postgres://$user:$pass@$host:$port/$db

S3_BUCKET=
S3_ENDPOINT=
S3_ID=
S3_KEY=

JWKS_URI=
JWT_AUDIENCE=
JWT_ISSUER=

SLACK_TOKEN=
SLACK_INSTALL_CHANNEL=
SLACK_PANO_CHANNEL=
SLACK_REQUEST_CHANNEL=

OSTICKET_API_KEY=

ACUITY_USER_ID=
ACUITY_API_KEY=

Schema

Currently, we use node numbers to represent join requests, members, and nodes. This schema is an attempt to detangle our data and create a common definition of the various components of the mesh.

Building

A physical location.

id address lat lng alt bin notes

Member

A person in the mesh community. For example, a node-owner, donor or installer.

id name email phone

Node

A specific location on the network. Typically one per building.

id lat lng alt status name location
  • id
  • lat
  • lng
  • alt
  • status (active, dead)
  • name (optional) // e.g. "Saratoga", "SN1"
  • location (optional) // Human readable location, e.g. "Roof", "Basement"
  • notes (optional)
  • create_date
  • abandon_date (optional)
  • building_id
  • member_id

Join Request

  • id
  • date
  • roof_access
  • member_id
  • building_id

Panorama

  • id
  • url
  • date
  • request_id

Device Type

  • id
  • name
  • manufacturer
  • range
  • width

Device

A unit of hardware. Routers, radios, servers, etc.

  • id
  • status (in stock, active, dead)
  • name (optional)
  • ssid (optional)
  • notes (optional)
  • lat
  • lng
  • alt
  • azimuth (direction in degrees, default 0)
  • create_date
  • abandon_date (optional)
  • device_type_id
  • node_id

Link

A connection between two devices. For example, an ethernet cable or wireless connection.

  • id
  • status (active, dead)
  • create_date
  • device_a_id
  • device_b_id

Example Queries

Most join requests by member

SELECT
	COUNT(members.id) AS count,
	members.name AS member_name
FROM
	requests
	RIGHT JOIN members ON requests.member_id = members.id
GROUP BY
	members.id
ORDER BY
	count DESC;

Join requests in active node buildings

SELECT
	SUBSTRING(buildings.address, 1, 64) AS building_address,
	COUNT(DISTINCT requests.member_id) AS request_count,
	COUNT(DISTINCT nodes.member_id) AS node_count,
	JSON_AGG(DISTINCT nodes.id) AS node_ids,
	JSON_AGG(DISTINCT members.email) AS request_emails
FROM
	buildings
	JOIN requests ON buildings.id = requests.building_id
	JOIN members ON members.id = requests.member_id
	JOIN nodes ON buildings.id = nodes.building_id
WHERE
	nodes.status = 'active'
GROUP BY
	buildings.id
HAVING
	COUNT(DISTINCT requests.member_id) > COUNT(DISTINCT nodes.member_id)
ORDER BY
	request_count DESC

Tallest buildings with panos

SELECT
buildings.alt,
COUNT(DISTINCT requests.id) as request_count,
SUBSTRING(buildings.address, 1, 64) as building_address,
ARRAY_AGG(DISTINCT nodes.id) as node_ids,
ARRAY_AGG(DISTINCT panoramas.url) as pano_ids
FROM buildings
JOIN requests
ON buildings.id = requests.building_id
FULL JOIN nodes
ON buildings.id = nodes.building_id
JOIN panoramas
ON requests.id = panoramas.request_id
WHERE requests.roof_access IS TRUE
GROUP BY buildings.id
ORDER BY buildings.alt DESC;

Most join requests by building

SELECT
SUBSTRING(buildings.address, 1, 64) as building_address,
COUNT(buildings.id) as count
FROM requests
RIGHT JOIN buildings
ON requests.building_id = buildings.id
GROUP BY buildings.id
ORDER BY count DESC;

And node count

SELECT
buildings.alt as building_height,
-- COUNT(requests.id) as request_count,
COUNT(buildings.id) as node_count,
SUBSTRING (buildings.address, 1, 64) as building_address
FROM nodes
RIGHT JOIN buildings
ON nodes.building_id = buildings.id
RIGHT JOIN requests
ON nodes.building_id = requests.building_id
GROUP BY buildings.id
ORDER BY node_count DESC;

Node ids in building

SELECT array_agg(id) FROM nodes WHERE nodes.building_id = \$1;

Most nodes by building

SELECT
buildings.alt as building_height,
COUNT(buildings.id) as node_count,
SUBSTRING (buildings.address, 1, 64) as building_address
FROM nodes
RIGHT JOIN buildings
ON nodes.building_id = buildings.id
GROUP BY buildings.id
ORDER BY node_count DESC;

Nodes and join requests by building

SELECT
buildings.id,
COUNT(DISTINCT requests.id) as request_count,
COUNT(DISTINCT nodes.id) as node_count,
ARRAY_AGG(DISTINCT nodes.id) as node_ids,
SUBSTRING(buildings.address, 1, 64) as building_address
FROM buildings
JOIN requests
ON buildings.id = requests.building_id
JOIN nodes
ON buildings.id = nodes.building_id
GROUP BY buildings.id
ORDER BY request_count DESC;

Tallest buildings

SELECT
buildings.alt as building_height,
COUNT(nodes.id) as node_count,
SUBSTRING(buildings.address, 1, 64) as building_address
FROM nodes
RIGHT JOIN buildings
ON nodes.building_id = buildings.id
GROUP BY buildings.id
ORDER BY building_height DESC;

Tallest buildings with nodes

SELECT
buildings.id as building_id,
buildings.alt as building_height,
COUNT(nodes.id) as node_count,
array_agg(nodes.id) as node_ids,
SUBSTRING(buildings.address, 1, 64) as building_address
FROM buildings
LEFT JOIN nodes
ON buildings.id = nodes.building_id
GROUP BY buildings.id
-- HAVING COUNT(nodes.id) > 0 -- Toggle this line to hide/show nodeless buildings
ORDER BY building_height DESC;

Tallest buildings with join requests

SELECT
buildings.id as building_id,
buildings.alt as building_height,
COUNT(requests.id) as request_count,
array_agg(requests.id) as request_ids,
SUBSTRING(buildings.address, 1, 64) as building_address
FROM buildings
LEFT JOIN requests
ON buildings.id = requests.building_id
GROUP BY buildings.id
-- HAVING COUNT(nodes.id) > 0 -- Toggle this line to hide/show nodeless buildings
ORDER BY building_height DESC;

Line of Sight

DB Setup

Install lxml:

pip3 install lxml

Set up the db:

node scripts/reset-los-db.js

Download the building data:

curl -o building_data.zip http://maps.nyc.gov/download/3dmodel/DA_WISE_GML.zip
unzip building_data.zip -d building_data
rm building_data.zip

Insert the data

{
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA1_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA2_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA3_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA4_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA5_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA6_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA7_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA8_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA9_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA10_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA11_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA12_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA13_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA14_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA15_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA16_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA17_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA18_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA19_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA20_3D_Buildings_Merged.gml buildings
	python3 ./scripts/gml_to_pgsql.py ./building_data/DA_WISE_GMLs/DA21_3D_Buildings_Merged.gml buildings
} | psql $LOS_DATABASE_URL

Now we are ready to make queries!

Making Queries

Let's check for line of sight between Supernode 1 and Node 3.

Step 1: Look up BINs:

Use NYC GeoSearch or NYC Building Information Search.

Supernode 1 BIN: 1001389
Node 3 BIN: 1006184

Step 2: Get building midpoints:

SELECT ST_AsText(ST_Centroid((SELECT geom FROM ny WHERE bldg_bin = '1001389'))) as a,
ST_AsText(ST_Centroid((SELECT geom FROM ny WHERE bldg_bin = '1006184'))) as b;
#                     a                     |                    b
# ------------------------------------------+------------------------------------------
#  POINT(987642.232749068 203357.276907034) | POINT(983915.956115596 198271.837494287)
# (1 row)

Step 3: Get building heights:

SELECT ST_ZMax((SELECT geom FROM ny WHERE bldg_bin = '1001389')) as a,
ST_ZMax((SELECT geom FROM ny WHERE bldg_bin = '1006184')) as b;
#         a         |        b
# ------------------+------------------
#  582.247499999998 | 120.199699999997
# (1 row)

Step 4: Check for intersections:

SELECT a.bldg_bin
FROM ny AS a
WHERE ST_3DIntersects(a.geom, ST_SetSRID('LINESTRINGZ (983915 198271 582, 987642 203357 120)'::geometry, 2263));
#  bldg_bin
# ----------
# (0 rows)

There are no intersections. We have line of sight!

mesh-api's People

Contributors

bhny avatar davidbalbert avatar greenberga avatar olivermesh avatar olivernyc avatar sammbeller avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mesh-api's Issues

Line of sight accuracy

There are a few things we can do to improve line of sight accuracy.

  • Check los from each corner of the roof to actual antenna location, rather than between roof midpoints.
  • Similarly, make sure los is within beam width of sectors.
  • Check for intersections with entire fresnel zone, rather than a straight line. Bonus: calculate fraction of zone obstructed.

Verify acuity webhook

We need to verify acuity message signatures to make sure they are not forged. https://developers.acuityscheduling.com/docs/webhooks

// Get hash of message using shared secret:
var hasher = crypto.createHmac('sha256', secret);
hasher.update(buf.toString());
var hash = hasher.digest('base64');

// Compare hash to Acuity signature:
if (hash !== req.header('X-Acuity-Signature')) {
    throw new Error('This message was forged!');
}

Weird BINs

SELECT COUNT(*) FROM ny WHERE bldg_bin = '1000000' returns 335
SELECT COUNT(*) FROM ny WHERE bldg_bin = '2000000' returns 2490
SELECT COUNT(*) FROM ny WHERE bldg_bin = '3000000' returns 4620
SELECT COUNT(*) FROM ny WHERE bldg_bin = '4000000' returns 19874
SELECT COUNT(*) FROM ny WHERE bldg_bin = '5000000' returns 3591

Any idea what's going on here? Is the source data invalid?

TODO

  • Finish endpoints (for updating members, nodes, requests, devices) Moved to #62
  • Figure out dashboard editing flow, especially for common tasks like new node after install, adding members to nodes, setting device locations Moved to #63
  • Set up db migrations Moved to #64
  • Add some tests Moved to #65
  • Pagination Moved to #66
  • Update map to show data from api Moved to #67
  • Panorama upload form (api work is done) Moved to #68

Add PATCH method to /requests/:id

Currently there's a GET and POST to /requests for getting and creating a join request. We need to have a PATCH method available for /requests/:id for modifying the state of the join request once the request has been allocated a node_id.

For example, updating the Request's status from open to installed

Add tests

Would be great to test individual endpoints and the full join flow!

Write tests

Unit and integration tests, where makes sense.

CI actions & test reporting

Add CI GH Actions to build, test, and run the app as well as codecov reporting and build status icons for readme.md

Connect API to new docker Postgres

In completing #64 we setup a new socket database, now we need to connect the system to this database for use in local development and potentially production.

We should also delete the old create database script once this is done

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.