zandaqo / structurae Goto Github PK

The single first example is not working due to an undefined TextEncoder in the module.
The original code, as follows.

workspace/structurae-demo/node_modules/structurae/lib/string-view.js:364
StringView.encoder = new TextEncoder();
^

ReferenceError: TextEncoder is not defined
at Object. (/mnt/c/users/u0165298/workspace/structurae-demo/node_modules/structurae/lib/string-view.js:364:26)
at Module._compile (internal/modules/cjs/loader.js:776:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:690:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/mnt/c/users/u0165298/workspace/structurae-demo/node_modules/structurae/lib/record-array.js:17:20)
at Module._compile (internal/modules/cjs/loader.js:776:30)

const { ObjectView, ArrayViewMixin } = require('structurae');

class Person extends ObjectView {}
Person.schema = {
id: { type: 'uint32' },
name: { type: 'string', length: 10 },
};

// an array class for Person objects
const PeopleArray = ArrayViewMixin(Person);

// create an empty array view of 10 Person objects
const people = PeopleArray.of(10);

// create an array view from a given array
const hitchhikers = PeopleArray.from([
{ id: 1, name: 'Arthur' },
{ id: 2, name: 'Ford' },
]);
const arthur = hitchhikers.get(0);
//=> Person [14]
arthur.toObject();
//=> { id: 1, name: 'Arthur' }

// set the first object data
hitchhikers.set(0, { id: 3, name: 'Trillian' });
hitchhikers.get(0).toObject();
//=> { id: 3, name: 'Trillian' }

hitchhikers.toObject();
//=> [{ id: 1, name: 'Arthur' }, { id: 2, name: 'Ford' }]

The documentation gives an example alternative compare function for as:

const { BinaryHeap } = require('structurae');

class MaxHeap extends BinaryHeap {}
MaxHeap.compare = (a, b) => b - a; 

This example will not compile in a typescript project, as BinaryHeap.compare has the signature:

static compare(a: any, b: any): boolean;

It appears that the implementation does indeed expect a boolean result from compare, and as such the documentation should probably be something like:

const { BinaryHeap } = require('structurae');

class MaxHeap extends BinaryHeap {}
MaxHeap.compare = (a, b) => a > b; 

Was curious if it is possible to convert Java objects into a View and sent to browser for JavaScript consumption? Any examples?

Hello! First of all, thank you for the marvelous (and quite unique) way of handling binary data in JavaScript! This library is paving the way for an intra-worker communication I've been developing. Copying/Serializing objects have been an absurd overhead due to the application communication throughput so I started looking into SharedArrayBuffers to transfer data directly.

I already got the gist of BitArrays by themselves, but I'm really struggling to understand the view.create<T> expectations when the schema uses some kind of these special data handlers.

I wrote a basic pseudo-code to explain what I am trying to achieve.
The ??? comments are for sections I am specifically clueless.

interface User {
  username: string
  attributes: number[] // ???
}

const UserView = view.create<User>({
  $id: 'User',
  type: 'object',
  
  properties: {
    username: {
      type: 'string',
      maxLength: 10,
    },

    attributes: {
      type: 'array',
      btype: 'uint32', // ???
      maxLength: 4,
    },
  },
})

const user = UserView.from({
   username: 'vinerz',
   attributes: [0, 1, 0, 1], // ???
 })

// work with the user data
user.getView('attributes').setBit(2, 1) 

// transfer control to worker
worker.postMessage({ user: user.buffer }, [user.buffer])

Could you please shed some light here?
Is it possible or am I thinking in a wrong way?

Could the binary structures serve as the basis for persistent data structures, implemented as HAMT?

Thanks for making your work on structurae available.

Issue

I'm running into the following when using in a node / typescript project

• An import error occurs when importing symbols from structurae 4.0.x
  import { View } from 'structurae'

Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: No "exports" main defined in /tmp/structurae-test/node_modules/structurae/package.json
    at new NodeError (node:internal/errors:371:5)
    at throwExportsNotFound (node:internal/modules/esm/resolve:453:9)
    at packageExportsResolve (node:internal/modules/esm/resolve:669:7)
    at resolveExports (node:internal/modules/cjs/loader:482:36)
    ...

Steps to Reproduce

• Minimally, create new typescript project node (package.json)
  • pnpm add structurae
  • `import { View } from 'structurae'
  • Using a basic tsconfig.json

    {
      "compilerOptions": {
        "outDir": "./lib",
        "rootDir": "./src"
      },
      "include": ["src/**/*"],
      "exclude": ["node_modules"]
    }

• Here's a bare bones project to illustrate: kloshih/structurae-test.

Hi zandaqo!

Thanks for the excellent work you're putting into this library. I'm working on some generative art projects and am looking forward to speeding things up considerably using structurae.

Also, thanks for the typings! I'm using TypeScript, and having that already included was awesome.

I noticed a small bug: in the typings file, the options argument for the Graph constructors is listed as having the type GridOptions. GridOptions only has the properties rows, columns, and pad. But according to the API reference, Graph options have the size, pad, and directed properties.

I've already modified my local copy of index.d.ts. Let me know if you'd like me to submit a PR with the corrections.

Thanks again for you hard work!

Cheers,
Thomas

Hi @zandaqo . After cloning and running the tests, they fail due to the undefined TextEncoder.
Copy that :

FAIL test/string-array-view.spec.js
● Test suite failed to run

ReferenceError: TextEncoder is not defined

  362 |  * @type TextEncoder
  363 |  */
> 364 | StringView.encoder = new TextEncoder();
      |                          ^
  365 |
  366 | /**
  367 |  * @type TextDecoder

  at Object.TextEncoder (lib/string-view.js:364:26)
  at Object.require (lib/record-array.js:17:20)

Test Suites: 6 failed, 16 passed, 22 total
Tests: 280 passed, 280 total
Snapshots: 0 total
Time: 23.737s
Ran all test suites.
npm ERR! Test failed. See above for more details.

Is it possible to provide some kind of TypedArray to store value referencing to js object?

When working with web workers, all methods are stripped away during transfer. Is there a way to initialize the sortedArray at the other end, without the end to sort it again? https://github.com/zandaqo/structurae/blob/master/sorted-array.ts#L30

Hello again! Thanks for the last fix. So I'm hoping to be able to use structurae for my binary protocol, but I'm having trouble reconstructing the view from binary on the client.

Simple example:

// Server
const transfer = TransferMessage.from({
  property: 'a string'
})

ws.send(transfer.buffer)

// Client
const message = TransferMessage.from(event.data)
console.log(message.toObject())

// Prints an empty structure
{ property: '' }

Maybe I'm not doing this right, or maybe its not possible? Its not clear in the documentation. It would be super useful if this were the API to create an object from a buffer.

One additional thing that I would want (but could implement outside structurae) is the ability to tag the buffer with the type of Protocol.

For instance, The first 8 bytes tell you which object to decode the remaining buffer into. Maybe this can be accomplished by wrapping the message? Some tips and insight would be appreciated. Thanks!

I currently have a npm project. And trying to use structurae appears to fail when generating typings in vite project..

$ tsc -p tsconfig.build.json && vite build
       ../../node_modules/structurae/types/grid.d.ts(42,65): error TS2526: A 'this' type is available only in a non-static member of a class or interface.
       ../../node_modules/structurae/types/grid.d.ts(57,86): error TS2344: Type 'T' does not satisfy the constraint 'abstract new (...args: any) => any'.
       ../../node_modules/structurae/types/grid.d.ts(64,94): error TS2344: Type 'T_1' does not satisfy the constraint 'abstract new (...args: any) => any'.
       ../../node_modules/structurae/types/symmetric-grid.d.ts(42,65): error TS2526: A 'this' type is available only in a non-static member of a class or interface.
       ../../node_modules/structurae/types/symmetric-grid.d.ts(57,71): error TS2344: Type 'T' does not satisfy the constraint 'abstract new (...args: any) => any'.
       ../../node_modules/structurae/types/symmetric-grid.d.ts(64,94): error TS2344: Type 'T_1' does not satisfy the constraint 'abstract new (...args: any) => any'.
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Most objects in this library inherit from DataView or TypedArray. That means they have a number of methods which don't respect the type abstraction.

e.g. since Uint16View extends DataView, it has a setFloat32 method. Note this also writes outside of the 2 bytes one expects the view to refer to.
e.g. since GridMixin extends a TypedArray, it has map, at, findIndex, which all use linear indexing.

Furthermore, Int32View extends Uint8View. This renders the type annotation x: Uint8View practically meaningless.

It would be nice if this library used less inheritance. It does make sense for binary-backed objects to implement the more minimal ArrayBufferView interface ({buffer, byteLength, byteOffset}).

It would be nice to have a markdown file showing the benchmark results.

Hello. I have some Typescript written to do various Matrix operations (some may need verification). Repo just created here: https://github.com/zachary-kaelan/performant-typescript

I can Regex some or all of them to go into a Grid class with numeric data, if you feel that would fit with or add to your project.

Hello!

First of all, thanks for your hard work on this library! @SupremeTechnopriest and I are really loving it so far :).

I am curious if you have plans to implement, or have any ideas on a custom implementation, of iterators/cursors over your ArrayViews? I want to run small, mostly pure functions over the binary data and mutate it as I go.

interface TimestampedData {
  ts: UTCTimestamp;
}

// @ts-ignore
export class MarketDataArray<T extends TimestampedData> extends SortedArray<T> {
  override unique = true;

  static override compare<T extends TimestampedData>(a: T, b: T): 0 | -1 | 1 {
    if (b.ts == a.ts) return 0;
    else if (b.ts - a.ts > 0) return -1;
    else return 1;
  }
}

const seedData = [{ ts: 0 as UTCTimestamp, value: 1}];
const data = MarketDataArray.from(seedData);
data.push({ ts: 0 as UTCTimestamp, value: 999 }); // value is still 1

If the data is new but is unique by for example timestamp, it appears that there isn't much way around to replacing it using push or unshift?

class DescSortedArray extends SortedArray {
  static compare(a, b) {
    return b - a
  }
}

const sorted = new DescSortedArray();

sorted.push(3)
sorted.push(1)
sorted.push(5)
sorted.push(4)
sorted.push(2)

console.log(sorted)

Is it normal to see an unsorted array?

DescSortedArray(5) [ 3, 1, 5, 4, 2, unique: false ]

Is there any chance to enable auto sort when I call push() or I have to call sort() after all push() calls?

Thanx!

I have a situation where I need basically a sliding window buffer used between web workers. I was hoping the SortedMixin would work on an ArrayView of ObjectViews.

Is it possible to support SharedArrayBuffers?

Grid has a size property which represents the log of the number of columns. This is confusingly named, since it seems like it should be some size of the grid itself. Maybe stride would be more apropos.

I am building a pretty big graph built up of sub smaller graphs with the nodes as ObjectViews. Since the graph relates the objects in myriads ways I assume that the storage solution I choose for the objects, i.e. how they are serialized to a structure out of memory, like JSON, can be purely pragmatic. The objects don't need to be stored in some sort of hierarchy. Would there be some benefit in storing them in CollectionViews or something like that? It would be nice if they were stored in a way that they could be deserialized trivially and safely back into a graph. Of course the graph itself is easy to serialize

Tried with both yarn && yarn benchmark and npm i && npm run benchmark getting the same output on node v12.3.0 on a Ubuntu 19.10 box.

> [email protected] benchmark /home/delaney/work/black/structurae
> node bench/index.js
/home/delaney/work/structurae/bench/bits.js:20
class Person extends BitField {}
                     ^

TypeError: Class extends value #<Object> is not a constructor or null
    at Object.<anonymous> (/home/delaney/work/structurae/bench/bits.js:20:22)
    at Module._compile (internal/modules/cjs/loader.js:956:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:973:10)
    at Module.load (internal/modules/cjs/loader.js:812:32)
    at Function.Module._load (internal/modules/cjs/loader.js:724:14)
    at Module.require (internal/modules/cjs/loader.js:849:19)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/home/delaney/work/structurae/bench/index.js:2:6)
    at Module._compile (internal/modules/cjs/loader.js:956:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:973:10)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

@zandaqo Thank you for your great work! These structures are awesome!

Could you please make a new data structure SortedMap? Or it is possible to do it from existing structures and I just didn't read it?

It looks like "default" isn't a property in the ObjectViewField type. Looks like a bug to me :).