Giter Club home page Giter Club logo

geoarrow-js's Introduction

geoarrow-js

A minimal TypeScript implementation of GeoArrow building on top of Arrow JS.

It complements the work-in-progress geoarrow-wasm, which will provide Rust-based operations on GeoArrow memory.

Features

  • Performant spatial operations.
  • Rich static typing of geometry arrays.
  • Tree shakeable.
  • Utilities for sharing Arrow data across Web Workers (not specific to GeoArrow)

Spatial operations

Only spatial operations that are implemented on binary representations of geometry will be added to this repo.

This means that geoarrow-js will not, say, use algorithms from Turf, because that would require conversions to and from GeoJSON for the operation.

Implemented algorithms:

Refer to the algorithm namespace in the docs.

Web Worker utilities

Refer to the worker namespace. Use preparePostMessage to obtain references to all underlying ArrayBuffer objects, so they can be transfered instead of copied.

import * as arrow from "apache-arrow";
import {
  preparePostMessage,
  rehydrateVector,
} from "@geoarrow/geoarrow-js/worker";

const originalVector = arrow.makeVector(new Int32Array([1, 2, 3]));
const [preparedVector, arrayBuffers] = preparePostMessage(originalVector);

// Here we use structuredClone to simulate a postMessage but on the main thread
const receivedVector = structuredClone(preparedVector, {
  transfer: arrayBuffers,
});
const rehydratedVector = rehydrateVector(receivedVector);

Ecosystem

geoarrow-js is designed to be used seamlessly with WebAssembly-based GeoArrow operations, such as those in the JavaScript bindings of the Rust GeoArrow implementation, and with rendering libraries, such as deck.gl, with the help of @geoarrow/deck.gl-layers.

For more background on my plans for GeoArrow ecosystem in JS and WebAssembly, refer to this thread.

geoarrow-js's People

Contributors

kylebarron avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

geoarrow-js's Issues

Failed to execute 'structuredClone' on prepared vector

Trying to use the preparePostMessage and rehydrateVector helpers on the README example I get the error message:

Uncaught DOMException: Failed to execute 'structuredClone' on 'Window': (index) => isChunkedValid(unchunkedData, index) could not be cloned.

See stackblitz example: https://stackblitz.com/edit/geoarrow-readme-worker-example?file=src%2Fmain.ts&terminal=dev
And open devtools to see the error.

npm list:

+-- @esbuild/[email protected] extraneous
+-- @geoarrow/[email protected]
+-- @rollup/[email protected] extraneous
+-- @rollup/[email protected] extraneous
+-- [email protected]
+-- [email protected]
`-- [email protected]

postMessage helper

Ensure arrow arrays are being transferred correctly in postMessage. Might also need a clone function or similar to disconnect an array from its backing buffer

hard cloning should check `offset === 0`

I.e. in this example if you look at

table.getChildAt(0)

you'll see

Vector [IntVector<Int>] {
  _offsets: Uint32Array(4) [ 0, 1, 2, 3 ],
  data: [
    Data {
      type: [Int32 [Int]],
      children: [],
      dictionary: undefined,
      offset: 0,
      length: 1,
      _nullCount: 0,
      stride: 1,
      values: [Int32Array],
      nullBitmap: Uint8Array(0) []
    },
    Data {
      type: [Int32 [Int]],
      children: [],
      dictionary: undefined,
      offset: 1,
      length: 1,
      _nullCount: 0,
      stride: 1,
      values: [Int32Array],
      nullBitmap: Uint8Array(0) []
    },
    Data {
      type: [Int32 [Int]],
      children: [],
      dictionary: undefined,
      offset: 0,
      length: 1,
      _nullCount: 0,
      stride: 1,
      values: [Int32Array],
      nullBitmap: Uint8Array(0) []
    }
  ],
  type: Int32 [Int] { isSigned: true, bitWidth: 32 },
  stride: 1,
  numChildren: 0,
  length: 3
}

So the second Data is a zero copy slice on the input. This means that if you checked if it's shared, it would say it's not shared because the view occupies the whole buffer? Need to double check this.

In any case, implementing hard cloning with slicing shouldn't be too hard, copying from the arrowjs method that has subarrays

Include field string keys on typedefs?

export type Point2 = arrow.FixedSizeList<arrow.Float> & {
  children: [arrow.Field<arrow.Float> & { name: "xy" }];
};

function tmp(x: arrow.DataType) {
  let z = x.children[0].name;
  // string
}
function tmp2(x: Point2) {
  let z = x.children[0].name;
  z
  // "xy"
}

Would provide a distinction between otherwise structurally-equal types (linestring--multipoint and polygon--multilinestring). Right now, if you pass in a multipoint type into a function like mapCoords you'll get linestring back

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.