kylebarron / arrow-js-ffi Goto Github PK

Materialize views into full array on the JS side

Failed in https://github.com/kylebarron/arrow-js-ffi/actions/runs/7721619256/job/21048446815

npm ci doesn't work with yarn.lock. and in any case I think the install likely would fail without the rust crate for the dev dep

Hi,
I've been trying to setup small example of getting data from parquet to json.
So far everything works except when I try to get data from parseRecordBatch, after I generate recordBatches and try to pass them into apache-arrow Table it fails with Maximum call stack size exceeded (screenshot attached)

I am not sure if I am doing something wrong getting wasmArrowTable from parquet.toFFI or that data is large (using your geo location data from parquet-wasm repo).
This is combination of multiple examples that I could find that made some sense so I am sorry if I am butchering code.
And I am pretty new to parquet and arrow.

Here is all the demo code that I have needed to reproduce it. Bun is used but function generateData should be easily converted to what you need to reproduce it

import { dir } from '@stricjs/utils';
import { Router } from '@stricjs/router';
import { Table, tableFromIPC } from "apache-arrow";
import { readParquet, _memory } from "parquet-wasm/node/arrow2";
import { parseRecordBatch } from "arrow-js-ffi";

const generateData = async () => {
    const path = "./public/geo.parquet"
    const file = await Bun.file(path);
    const buff = await file.arrayBuffer()
    const bytes = new Uint8Array(buff);
    console.log(bytes.length);
    const prq = readParquet(bytes)
    const wasmArrowTable = prq.toFFI();
    console.log("numBatches", wasmArrowTable.numBatches())

    const recordBatches = [];
for (let i = 0; i < wasmArrowTable.numBatches(); i++) {
  const recordBatch = parseRecordBatch(
    _memory().buffer,
    wasmArrowTable.arrayAddr(i),
    wasmArrowTable.schemaAddr(),
    true
  );
  recordBatches.push(recordBatch);
}

console.log(recordBatches[0])
const table = new Table(recordBatches[0]);
wasmArrowTable.drop();

}

export default new Router()
  .get('/', () => new Response('Hi'))
  .get('/test', async () => {
    await generateData()
    new Response("DONE")
  })
  .get('/make', async () => {
    await makeTable()
    new Response("DONE")
  })
  .get('/*', dir('./public'));

Arrow JS is a big library! It's not really a tenable dependency for a very bundle size conscious library or application.

This is actually the same story as in C/C++/Python. The C++ Arrow library got so big that many projects didn't want to depend on it. That's why nanoarrow was created. As a super minimal library that works with the C Data Interface representation of Arrow arrays.

I think there's definitely potential for a low level Arrow library in JS, that hews very closely to the C Data Interface.

Data structures would be essentially the JS counterpart of C Data Interface structs. All array data (no matter the logical type) would be a Uint8Array, that could later be viewed as another type or as strings.

Because array data are all Uint8Arrays, it means an array could either be "owned" in JS memory or "viewed" from wasm memory. So the memory safety wouldn't be great, but this is JS after all!

It would make sense to have toArrowJS and fromArrowJS functions that convert to and from Arrow JS arrays/Data instances.

An emphasis should be placed on a functional api instead of a class API to keep bundle size low.

Ideally, this would allow high-performance programs to rely on Arrow memory without fear of a huge bundle size impact! But this would be complementary not competitive with Arrow JS.

May need to coerce offsets to int32 to be used in arrow js

Should parallel parseRecordBatch.. should take in an ArrowSchema struct, which it asserts to be of a struct type, then unpack the internal fields to return an Arrow JS Schema

I previously found that importing from * as arrow meant that esbuild couldn't treeshake at all. geoarrow/geoarrow-js#20

So maybe this lib should use path imports? It does have to import every type necessarily because it doesn't know what data type the C struct will be.

I originally tried to parse an array of strings, and the last item in the list was always null. Does the offsets array always need to be length + 1 in arrow?

Ref #61

E.g. LargeList[LargeList[LargeString]] and make sure that the Large -> Small conversion is working correctly for the nested types

Hey Kyle

I am attempting to wasm build rust-wasm portion as indicated:

➜  rust-wasm git:(main) wasm-pack build --target web
[INFO]: 🎯  Checking for the Wasm target...
[INFO]: 🌀  Compiling to Wasm...
   Compiling rust-wasm v0.1.0 (/Users/anirudh.vyas/rust/Polars-wasm-mwe/rust-wasm)
error[E0609]: no field `0` on type `Result<Row<'_>, PolarsError>`
  --> src/lib.rs:42:22
   |
42 |         for j in row.0.iter() {
   |                      ^

For more information about this error, try `rustc --explain E0609`.
error: could not compile `rust-wasm` due to previous error
Error: Compiling your crate to WebAssembly failed
Caused by: failed to execute `cargo build`: exited with exit status: 101
  full command: "cargo" "build" "--lib" "--release" "--target" "wasm32-unknown-unknown"

Would you know what am I doing wrong?

Hey! Thank you for this library.

I noticed that there is a mismatch in major versions for the apache-arrow dependency.

The peerDependencies specify ^10:

while the devDependencies specify ^12:

• https://www.totaltypescript.com/typescript-5-2-new-keyword-using
• https://devblogs.microsoft.com/typescript/announcing-typescript-5-2/#using-declarations-and-explicit-resource-management

https://stackoverflow.com/a/840504

ref

I think for a sliced array that offset should indeed not be zero.

I think it makes sense to explore writing to Wasm memory.

• writeField(field: arrow.Field, malloc: (length: Number) -> Number): Number would take a field and a malloc function. It would write the field manually into wasm memory using malloc, and at the end returning the pointer to the written number struct.
• Requires a memory copy, but still that's better than going through IPC, which requires a single memory buffer.

A good reference here, esp in relation to Rust is Matu Radei's great blog post: https://radu-matei.com/blog/practical-guide-to-wasm-memory/#passing-arrays-to-rust-webassembly-modules