zandaqo / structurae Goto Github PK
View Code? Open in Web Editor NEWData structures for high-performance JavaScript applications.
License: MIT License
Data structures for high-performance JavaScript applications.
License: MIT License
Was curious if it is possible to convert Java objects into a View
and sent to browser for JavaScript consumption? Any examples?
Hello. I have some Typescript written to do various Matrix operations (some may need verification). Repo just created here: https://github.com/zachary-kaelan/performant-typescript
I can Regex some or all of them to go into a Grid class with numeric data, if you feel that would fit with or add to your project.
Grid
has a size
property which represents the log of the number of columns. This is confusingly named, since it seems like it should be some size of the grid itself. Maybe stride
would be more apropos.
class DescSortedArray extends SortedArray {
static compare(a, b) {
return b - a
}
}
const sorted = new DescSortedArray();
sorted.push(3)
sorted.push(1)
sorted.push(5)
sorted.push(4)
sorted.push(2)
console.log(sorted)
Is it normal to see an unsorted array?
DescSortedArray(5) [ 3, 1, 5, 4, 2, unique: false ]
Is there any chance to enable auto sort when I call push() or I have to call sort() after all push() calls?
Thanx!
Tried with both yarn && yarn benchmark
and npm i && npm run benchmark
getting the same output on node v12.3.0 on a Ubuntu 19.10 box.
> [email protected] benchmark /home/delaney/work/black/structurae
> node bench/index.js
/home/delaney/work/structurae/bench/bits.js:20
class Person extends BitField {}
^
TypeError: Class extends value #<Object> is not a constructor or null
at Object.<anonymous> (/home/delaney/work/structurae/bench/bits.js:20:22)
at Module._compile (internal/modules/cjs/loader.js:956:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:973:10)
at Module.load (internal/modules/cjs/loader.js:812:32)
at Function.Module._load (internal/modules/cjs/loader.js:724:14)
at Module.require (internal/modules/cjs/loader.js:849:19)
at require (internal/modules/cjs/helpers.js:74:18)
at Object.<anonymous> (/home/delaney/work/structurae/bench/index.js:2:6)
at Module._compile (internal/modules/cjs/loader.js:956:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:973:10)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Is it possible to provide some kind of TypedArray to store value referencing to js object?
workspace/structurae-demo/node_modules/structurae/lib/string-view.js:364
StringView.encoder = new TextEncoder();
^
ReferenceError: TextEncoder is not defined
at Object. (/mnt/c/users/u0165298/workspace/structurae-demo/node_modules/structurae/lib/string-view.js:364:26)
at Module._compile (internal/modules/cjs/loader.js:776:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:787:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:690:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/mnt/c/users/u0165298/workspace/structurae-demo/node_modules/structurae/lib/record-array.js:17:20)
at Module._compile (internal/modules/cjs/loader.js:776:30)
const { ObjectView, ArrayViewMixin } = require('structurae');
class Person extends ObjectView {}
Person.schema = {
id: { type: 'uint32' },
name: { type: 'string', length: 10 },
};
// an array class for Person objects
const PeopleArray = ArrayViewMixin(Person);
// create an empty array view of 10 Person objects
const people = PeopleArray.of(10);
// create an array view from a given array
const hitchhikers = PeopleArray.from([
{ id: 1, name: 'Arthur' },
{ id: 2, name: 'Ford' },
]);
const arthur = hitchhikers.get(0);
//=> Person [14]
arthur.toObject();
//=> { id: 1, name: 'Arthur' }
// set the first object data
hitchhikers.set(0, { id: 3, name: 'Trillian' });
hitchhikers.get(0).toObject();
//=> { id: 3, name: 'Trillian' }
hitchhikers.toObject();
//=> [{ id: 1, name: 'Arthur' }, { id: 2, name: 'Ford' }]
Hi zandaqo!
Thanks for the excellent work you're putting into this library. I'm working on some generative art projects and am looking forward to speeding things up considerably using structurae.
Also, thanks for the typings! I'm using TypeScript, and having that already included was awesome.
I noticed a small bug: in the typings file, the options argument for the Graph constructors is listed as having the type GridOptions. GridOptions only has the properties rows
, columns
, and pad
. But according to the API reference, Graph options have the size
, pad
, and directed
properties.
I've already modified my local copy of index.d.ts. Let me know if you'd like me to submit a PR with the corrections.
Thanks again for you hard work!
Cheers,
Thomas
I currently have a npm project. And trying to use structurae appears to fail when generating typings in vite project..
$ tsc -p tsconfig.build.json && vite build
../../node_modules/structurae/types/grid.d.ts(42,65): error TS2526: A 'this' type is available only in a non-static member of a class or interface.
../../node_modules/structurae/types/grid.d.ts(57,86): error TS2344: Type 'T' does not satisfy the constraint 'abstract new (...args: any) => any'.
../../node_modules/structurae/types/grid.d.ts(64,94): error TS2344: Type 'T_1' does not satisfy the constraint 'abstract new (...args: any) => any'.
../../node_modules/structurae/types/symmetric-grid.d.ts(42,65): error TS2526: A 'this' type is available only in a non-static member of a class or interface.
../../node_modules/structurae/types/symmetric-grid.d.ts(57,71): error TS2344: Type 'T' does not satisfy the constraint 'abstract new (...args: any) => any'.
../../node_modules/structurae/types/symmetric-grid.d.ts(64,94): error TS2344: Type 'T_1' does not satisfy the constraint 'abstract new (...args: any) => any'.
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Hello again! Thanks for the last fix. So I'm hoping to be able to use structurae for my binary protocol, but I'm having trouble reconstructing the view from binary on the client.
Simple example:
// Server
const transfer = TransferMessage.from({
property: 'a string'
})
ws.send(transfer.buffer)
// Client
const message = TransferMessage.from(event.data)
console.log(message.toObject())
// Prints an empty structure
{ property: '' }
Maybe I'm not doing this right, or maybe its not possible? Its not clear in the documentation. It would be super useful if this were the API to create an object from a buffer.
One additional thing that I would want (but could implement outside structurae) is the ability to tag the buffer with the type of Protocol.
For instance, The first 8 bytes tell you which object to decode the remaining buffer into. Maybe this can be accomplished by wrapping the message? Some tips and insight would be appreciated. Thanks!
Hello! First of all, thank you for the marvelous (and quite unique) way of handling binary data in JavaScript! This library is paving the way for an intra-worker communication I've been developing. Copying/Serializing objects have been an absurd overhead due to the application communication throughput so I started looking into SharedArrayBuffers
to transfer data directly.
I already got the gist of BitArrays
by themselves, but I'm really struggling to understand the view.create<T>
expectations when the schema uses some kind of these special data handlers.
I wrote a basic pseudo-code to explain what I am trying to achieve.
The ???
comments are for sections I am specifically clueless.
interface User {
username: string
attributes: number[] // ???
}
const UserView = view.create<User>({
$id: 'User',
type: 'object',
properties: {
username: {
type: 'string',
maxLength: 10,
},
attributes: {
type: 'array',
btype: 'uint32', // ???
maxLength: 4,
},
},
})
const user = UserView.from({
username: 'vinerz',
attributes: [0, 1, 0, 1], // ???
})
// work with the user data
user.getView('attributes').setBit(2, 1)
// transfer control to worker
worker.postMessage({ user: user.buffer }, [user.buffer])
Could you please shed some light here?
Is it possible or am I thinking in a wrong way?
Is it possible to support SharedArrayBuffers?
When working with web workers, all methods are stripped away during transfer. Is there a way to initialize the sortedArray at the other end, without the end to sort it again? https://github.com/zandaqo/structurae/blob/master/sorted-array.ts#L30
The documentation gives an example alternative compare function for as:
const { BinaryHeap } = require('structurae');
class MaxHeap extends BinaryHeap {}
MaxHeap.compare = (a, b) => b - a;
This example will not compile in a typescript project, as BinaryHeap.compare has the signature:
static compare(a: any, b: any): boolean;
It appears that the implementation does indeed expect a boolean result from compare, and as such the documentation should probably be something like:
const { BinaryHeap } = require('structurae');
class MaxHeap extends BinaryHeap {}
MaxHeap.compare = (a, b) => a > b;
Thanks for making your work on structurae available.
I'm running into the following when using in a node / typescript project
import { View } from 'structurae'
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: No "exports" main defined in /tmp/structurae-test/node_modules/structurae/package.json
at new NodeError (node:internal/errors:371:5)
at throwExportsNotFound (node:internal/modules/esm/resolve:453:9)
at packageExportsResolve (node:internal/modules/esm/resolve:669:7)
at resolveExports (node:internal/modules/cjs/loader:482:36)
...
pnpm add structurae
{
"compilerOptions": {
"outDir": "./lib",
"rootDir": "./src"
},
"include": ["src/**/*"],
"exclude": ["node_modules"]
}
I have a situation where I need basically a sliding window buffer used between web workers. I was hoping the SortedMixin would work on an ArrayView of ObjectViews.
Could the binary structures serve as the basis for persistent data structures, implemented as HAMT?
Hello!
First of all, thanks for your hard work on this library! @SupremeTechnopriest and I are really loving it so far :).
I am curious if you have plans to implement, or have any ideas on a custom implementation, of iterators/cursors over your ArrayViews? I want to run small, mostly pure functions over the binary data and mutate it as I go.
Hi @zandaqo . After cloning and running the tests, they fail due to the undefined TextEncoder.
Copy that :
FAIL test/string-array-view.spec.js
โ Test suite failed to run
ReferenceError: TextEncoder is not defined
362 | * @type TextEncoder
363 | */
> 364 | StringView.encoder = new TextEncoder();
| ^
365 |
366 | /**
367 | * @type TextDecoder
at Object.TextEncoder (lib/string-view.js:364:26)
at Object.require (lib/record-array.js:17:20)
Test Suites: 6 failed, 16 passed, 22 total
Tests: 280 passed, 280 total
Snapshots: 0 total
Time: 23.737s
Ran all test suites.
npm ERR! Test failed. See above for more details.
It would be nice to have a markdown file showing the benchmark results.
Most objects in this library inherit from DataView
or TypedArray
. That means they have a number of methods which don't respect the type abstraction.
e.g. since Uint16View
extends DataView
, it has a setFloat32
method. Note this also writes outside of the 2 bytes one expects the view to refer to.
e.g. since GridMixin
extends a TypedArray
, it has map
, at
, findIndex
, which all use linear indexing.
Furthermore, Int32View
extends Uint8View
. This renders the type annotation x: Uint8View
practically meaningless.
It would be nice if this library used less inheritance. It does make sense for binary-backed objects to implement the more minimal ArrayBufferView
interface ({buffer, byteLength, byteOffset}
).
@zandaqo Thank you for your great work! These structures are awesome!
Could you please make a new data structure SortedMap? Or it is possible to do it from existing structures and I just didn't read it?
I am building a pretty big graph built up of sub smaller graphs with the nodes as ObjectView
s. Since the graph relates the objects in myriads ways I assume that the storage solution I choose for the objects, i.e. how they are serialized to a structure out of memory, like JSON, can be purely pragmatic. The objects don't need to be stored in some sort of hierarchy. Would there be some benefit in storing them in CollectionViews or something like that? It would be nice if they were stored in a way that they could be deserialized trivially and safely back into a graph. Of course the graph itself is easy to serialize
interface TimestampedData {
ts: UTCTimestamp;
}
// @ts-ignore
export class MarketDataArray<T extends TimestampedData> extends SortedArray<T> {
override unique = true;
static override compare<T extends TimestampedData>(a: T, b: T): 0 | -1 | 1 {
if (b.ts == a.ts) return 0;
else if (b.ts - a.ts > 0) return -1;
else return 1;
}
}
const seedData = [{ ts: 0 as UTCTimestamp, value: 1}];
const data = MarketDataArray.from(seedData);
data.push({ ts: 0 as UTCTimestamp, value: 999 }); // value is still 1
If the data is new but is unique by for example timestamp, it appears that there isn't much way around to replacing it using push
or unshift
?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.