Giter Club home page Giter Club logo

borsh-js's Introduction

Borsh JS

Project license Project license Discord Travis status NPM version Size on NPM

Borsh JS is an implementation of the Borsh binary serialization format for JavaScript and TypeScript projects.

Borsh stands for Binary Object Representation Serializer for Hashing. It is meant to be used in security-critical projects as it prioritizes consistency, safety, speed, and comes with a strict specification.

Tip

We strongly recommend to use borsh-js alongside the amazing project Borsher, which we plan to merge in borsh.

Examples

(De)serializing a Value

import * as borsh from 'borsh';

const encodedU16 = borsh.serialize('u16', 2);
const decodedU16 = borsh.deserialize('u16', encodedU16);

const encodedStr = borsh.serialize('string', 'testing');
const decodedStr = borsh.deserialize('string', encodedStr);

(De)serializing an Object

import * as borsh from 'borsh';

const value = {x: 255, y: BigInt(20), z: '123', arr: [1, 2, 3]};
const schema = { struct: { x: 'u8', y: 'u64', 'z': 'string', 'arr': { array: { type: 'u8' }}}};

const encoded = borsh.serialize(schema, value);
const decoded = borsh.deserialize(schema, encoded);

API

The package exposes the following functions:

  • serialize(schema: Schema, obj: any, validate: boolean = true): Uint8Array - serializes an object obj according to the schema schema. Setting validate to false will skip the validation of the schema.
  • deserialize(schema: Schema, buffer: Uint8Array, validate: boolean = true): any - deserializes an object according to the schema schema from the buffer buffer. Setting validate to false will skip the validation of the schema.

Schemas

Schemas are used to describe the structure of the data being serialized or deserialized. They are used to validate the data and to determine the order of the fields in the serialized data.

You can find examples of valid in the test folder.

Tip

We strongly recommend to use borsh-js alongside the amazing project Borsher, which we plan to merge in borsh.

Basic Types

Basic types are described by a string. The following types are supported:

  • u8, u16, u32, u64, u128 - unsigned integers of 8, 16, 32, 64, and 128 bits respectively.
  • i8, i16, i32, i64, i128 - signed integers of 8, 16, 32, 64, and 128 bits respectively.
  • f32, f64 - IEEE 754 floating point numbers of 32 and 64 bits respectively.
  • bool - boolean value.
  • string - UTF-8 string.

Arrays, Options, Maps, Sets, Enums, and Structs

More complex objects are described by a JSON object. The following types are supported:

  • { array: { type: Schema, len?: number } } - an array of objects of the same type. The type of the array elements is described by the type field. If the field len is present, the array is fixed-size and the length of the array is len. Otherwise, the array is dynamic-sized and the length of the array is serialized before the elements.
  • { option: Schema } - an optional object. The type of the object is described by the type field.
  • { map: { key: Schema, value: Schema }} - a map. The type of the keys and values are described by the key and value fields respectively.
  • { set: Schema } - a set. The type of the elements is described by the type field.
  • { enum: [ { struct: { className1: structSchema1 } }, { struct: { className2: structSchema2 } }, ... ] } - an enum. The variants of the enum are described by the className1, className2, etc. fields. The variants are structs.
  • { struct: { field1: Schema1, field2: Schema2, ... } } - a struct. The fields of the struct are described by the field1, field2, etc. fields.

Type Mappings

Javascript Borsh
number u8 u16 u32 i8 i16 i32
bigint u64 u128 i64 i128
number f32 f64
number f32 f64
boolean bool
string UTF-8 string
type[] fixed-size byte array
type[] dynamic sized array
object enum
Map HashMap
Set HashSet
null or type Option

Contributing

Install dependencies:

yarn install

Continuously build with:

yarn dev

Run tests:

yarn test

Run linter

yarn lint

Publish

Prepare dist version by running:

yarn build

When publishing to npm use np.

License

This repository is distributed under the terms of both the MIT license and the Apache License (Version 2.0). See LICENSE-MIT and LICENSE-APACHE for details.

borsh-js's People

Contributors

ailisp avatar artob avatar bartosz-lipinski avatar cbot918 avatar chadoh avatar chefsale avatar dimfeld avatar exromany avatar frol avatar gagdiez avatar heroes-bounty[bot] avatar ilblackdragon avatar joncinque avatar k06a avatar maksymzavershynskyi avatar mehtaphysical avatar mikedotexe avatar mikhailok avatar popcornpaws avatar sekhmet avatar tbezman avatar vgrichina avatar volovyks avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

borsh-js's Issues

yarn build and yarn test fails

borsh-js (master) yarn build
yarn run v1.22.18
$ tsc -p ./tsconfig.json
node_modules/@types/prettier/index.d.ts:41:54 - error TS2315: Type 'IsTuple' is not generic.

41 type IndexProperties<T extends

{ length: number }

= IsTuple extends true


node_modules/@types/prettier/index.d.ts:53:6 - error TS2456: Type alias 'IsTuple' circularly references itself.

53 type IsTuple<T> = T extends [] ? true : T extends <span class="error">[infer First, ...infer Remain]</span> ? IsTuple<Remain> : false;
~~~~~~~

node_modules/@types/prettier/index.d.ts:53:65 - error TS2574: A rest element type must be an array type.

53 type IsTuple<T> = T extends [] ? true : T extends <span class="error">[infer First, ...infer Remain]</span> ? IsTuple<Remain> : false;

node_modules/@types/prettier/index.d.ts:53:84 - error TS2315: Type 'IsTuple' is not generic.

53 type IsTuple = T extends [] ? true : T extends [infer First, ...infer Remain] ? IsTuple : false;


node_modules/@types/prettier/index.d.ts:96:5 - error TS2589: Type instantiation is excessively deep and possibly infinite.

96 call<
~~~~

node_modules/@types/prettier/index.d.ts:131:5 - error TS2589: Type instantiation is excessively deep and possibly infinite.

131 each<
~~~~

node_modules/@types/prettier/index.d.ts:165:5 - error TS2589: Type instantiation is excessively deep and possibly infinite.

165 map<
~~~

Found 7 errors.

error Command failed with exit code 2.
info Visit [https://yarnpkg.com/en/docs/cli/run](https://yarnpkg.com/en/docs/cli/run) for documentation about this command.

Security Policy violation SECURITY.md

This issue was automatically created by Allstar.

Security Policy Violation
Security policy not enabled.
A SECURITY.md file can give users information about what constitutes a vulnerability and how to report one securely so that information about a bug is not publicly visible. Examples of secure reporting methods include using an issue tracker with private issue support, or encrypted email with a published key.

To fix this, add a SECURITY.md file that explains how to handle vulnerabilities found in your repository. Go to https://github.com/near/borsh-js/security/policy to enable.

For more information, see https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository.


This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

How to serialize a floating point number

I see there is no supported 'f32' or 'f64' types. I am new to typescript so maybe I'm missing something here, but is there a work-around to serialize a number type that has a floating point value? Maybe send as a byte array of u8 that reflects IEEE standard?

e.g. 0.0001

How to convert uint8array data to BTreeMap or vector/array using typescript.

Hello, I'm serializing a BTreeMap using BorshSerialize in rust. How to deserialize this uint8array serialized data to vector/array or BTreeMap in Typescript using borsh-js or any other library? Please help

Below is my rust code

#[derive( Clone, BorshDeserialize, BorshSerialize, Copy)]
pub struct BuyOrderData {
    pub owner_public_key: [u8;32],
    pub receiving_bc_public_key: [u8;32],
    pub amount_qc: u64,
}

#[derive(Default)]
pub struct BuyOrderDataBTreeMap {
    is_initialized: bool,
    bids:  BTreeMap<u128, BuyOrderData>,
}



const INITIALIZED_BYTES: usize = 1;
const BTREEMAP_LENGTH: usize = 4;
const BTREEMAP_STORAGE: usize = 8600;
const ORDER_DATA_HEAP_ACCOUNT_STATE_SPACE: usize = INITIALIZED_BYTES + BTREEMAP_LENGTH + BTREEMAP_STORAGE;


impl BuyOrderDataBTreeMap {
    pub fn set_initialized(&mut self) {
        self.is_initialized = true;
    }

    pub fn add(&mut self, new_key:u128, new_owner_public_key:  [u8;32], new_receiving_bc_public_key:  [u8;32],  new_amount_qc: u64) {
        self.bids.insert(new_key, BuyOrderData {owner_public_key: new_owner_public_key, receiving_bc_public_key: new_receiving_bc_public_key, amount_qc: new_amount_qc});
    }

    pub fn remove(&mut self, key: u128) {
        self.bids.remove(key);
    }

}

impl Sealed for BuyOrderDataBTreeMap {}

impl IsInitialized for BuyOrderDataBTreeMap {
    fn is_initialized(&self) -> bool {
        self.is_initialized
    }
}

#[allow(clippy::ptr_offset_with_cast)]
impl Pack for BuyOrderDataBTreeMap {
    const LEN: usize = ORDER_DATA_HEAP_ACCOUNT_STATE_SPACE;
    fn unpack_from_slice(src: &[u8]) -> Result<Self, ProgramError> {
        let src = array_ref![src, 0, BuyOrderDataBTreeMap::LEN];
        let (is_initialized_src, data_len_src, data_src) = array_refs![src, INITIALIZED_BYTES, BTREEMAP_LENGTH, BTREEMAP_STORAGE];
 
        let is_initialized = match is_initialized_src {
            [0] => false,
            [1] => true,
            _ => return Err(ProgramError::InvalidAccountData),
        };
        let data_len = u32::from_le_bytes(*data_len_src) as usize;
        if data_len == 0 {
            Ok(BuyOrderDataBTreeMap {
                is_initialized,
                bids: BTreeMap::<u128, BuyOrderData>::new(),
            })
        }  else {
            let data_dser =
                BTreeMap::<u128, BuyOrderData>::try_from_slice(&data_src[0..data_len]).unwrap();
            Ok(BuyOrderDataBTreeMap {
                is_initialized,
                bids: data_dser,
            })
        }
    }

    fn pack_into_slice(&self, dst: &mut [u8]) {
        let dst = array_mut_ref![dst, 0, BuyOrderDataBTreeMap::LEN];
        let (is_initialized_dst, data_len_dst, data_dst) = mut_array_refs![dst, INITIALIZED_BYTES, BTREEMAP_LENGTH, BTREEMAP_STORAGE];
        is_initialized_dst[0] = self.is_initialized as u8;
        let orders_store_heap_data = self.bids.try_to_vec().unwrap();
        let data_len = orders_store_heap_data.len();
        data_len_dst[..].copy_from_slice(&(data_len as u32).to_le_bytes());
        data_dst[0..data_len].copy_from_slice(&orders_store_heap_data);

    }
}

`global` is not defined in the browser

The check for if the TextDecoder polyfill is required currently references global, which only exists on Node.js and not in the browser.

const TextDecoder =
typeof (global as any).TextDecoder !== "function"
? encoding.TextDecoder
: (global as any).TextDecoder;

With bundlers that don't automatically polyfill global this causes problems and requires hacks to make it work. What you would think of one of the following solutions?

  • typeof TextDecoder !== "function" ? encoding.TextDecoder : TextDecoder
  • typeof globalThis.TextDecoder !== "function" ? encoding.TextDecoder : globalThis.TextDecoder

globalThis has been available for a while now, but I think the first one is best since it doesn't make any assumptions about the name of the global variable. Any thoughts? I'm happy to submit a PR for this.

Support the `bool` type

While the Borsh spec doesn't officially support bool, the Rust implementation does map bool to u8 as 1/0.

Can we add bool and/or boolean to the JS implementation for convenience?

Support fixed size array

Right now only fixed sized byte arrays are supported. According to the spec fixed sized arrays should be supported for any type.

How to deserialize enum?

In the readme, the documentation mentions about enum.

{ enum: [{ className1: { struct: {...} } }, { className2: { struct: {...} } }, ... ] } - an enum. The variants of the enum are described by the className1, className2, etc. fields. The variants are structs.

But the current implementation seems not correct.

const struct = schema.enum[valueIndex].struct;

Buffer is not defined

import 

 {Buffer} 

 from "buffer";
 import 

 {deserialize, serialize} 

 from "borsh";

class Test {
 constructor(data) 

 { Object.assign(this, data); } 

}

const value = new Test(

 { x: 255, y: 20, z: '123', q: [1, 2, 3] } 

);
 const schema = new Map([[Test, 

 { kind: 'struct', fields: [['x', 'u8'], ['y', 'u64'], ['z', 'string'], ['q', [3]]] } 

]]);
 const buffer = serialize(schema, value);

Does not work

How to serialize deserialize an array of some type

I am using borsh-js and getting the error Error: Class Array is missing in schema. I would like to deserialize an array of ChatMessage that comes into my js client code as Buffer of u8 array. Note when I deserialize just one class instance it works.

`
class ChatMessage {
archive_id: string = DUMMY_TX_ID;
created_on: string = DUMMY_CREATED_ON; // max milliseconds in date
constructor(
fields:

{ archive_id: string; created_on: string }

| undefined = undefined
) {
if (fields)

{ this.archive_id = fields.archive_id; this.created_on = fields.created_on; }

}
}
const ChatMessageSchema = new Map([
[
ChatMessage,

{ kind: "struct", fields: [ ["archive_id", "String"], ["created_on", "String"], ], }

,
],
]);
const result = serialize(
ChatMessageSchema,
sampleChatMessages
);
`

How to serialize UnorderedMap (Map)?

How I can deserialize in borsh UnorderedMap?

Now my schema is

 const schema = new Map([
 [
 Record,
 {
 kind: "struct",
 fields: [
 ["x2", 

 { kind: 'map', key: 'string', value: 'u128'} 

],
 ],
 },
 ],
 ]);

Rust struct:

 pub struct Counter 

 { x2: UnorderedMap<AccountId, Balance>, } 

But I have error: "Expected buffer length 2036811841 isn't within bounds"

When I try serialize same struct in js, it dont work too

 const value = new Test({ x2: 

 {"1":"1"} 

 });
 const schema = new Map([
 [
 Test,
 {
 kind: "struct",
 fields: [
 ["x2", 

 { kind: 'map', key: 'string', value: 'string'} 

],
 ]
 }
 ]
 ]);
 const buffer = borsh.serialize(schema, value);
 console.log(buffer.toString('base64'))

Error: The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received type number (0)

Schema for nested struct (classes)

I have nested classes but am flummoxed on how to write the schema for it. No matter what I try I get errors...

A simple example anyone?

How to serialize Enum?

I haven't found any example to to serialize an enum. Can you please give an example to to do it. For example, here is a Rust enum:

#<span class="error">[derive(BorshSerialize, BorshDeserialize, PartialEq, Debug)]</span>
enum CalcInstruction {
Init,
Plus 

{ value: i32 } 

,
Minus 

{ value: i32, comment: String } 

,
}

deserialize method not mapping properties properly

Hi I have a class type defined with schema mentioned below

`class CampaignDetails {
name: string
description: string
image_link: string
admin: Buffer
amount_donated: number

constructor()
constructor(
name?: string,
description?: string,
image_link?: string,
admin?: Buffer,
amount_donated?: number
)

{ ;(this.name = name!), (this.description = description!), (this.image_link = image_link!), (this.admin = admin!), (this.amount_donated = amount_donated!) }

static schema: any = new Map([
[
CampaignDetails,

{ kind: 'struct', fields: [ ['admin', [32]], ['name', 'string'], ['description', 'string'], ['image_link', 'string'], ['amount_donated', 'u64'], ], }

,
],
])
}`

but when it deserialize the data the structure of it is changing,

`CampaignDetails
admin: undefined
amount_donated: undefined
description: undefined
image_link: undefined
name:
admin: Uint8Array(32) [245, 127, 181, 215, 162, 110, 87, 142, 194, 140, 167, 145, 235, 30, 49, 124, 184, 88, 251, 208, 221, 249, 65, 163, 2, 122, 182, 245, 58, 188, 96, 142, buffer: ArrayBuffer(32), byteLength: 32, byteOffset: 0, length: 32, Symbol(Symbol.toStringTag): 'Uint8Array']
amount_donated: BN

{negative: 0, words: Array(3), length: 1, red: null}

description: "dogs are the best"
image_link: "https://hips.hearstapps.com/hmg-prod.s3.amazonaws.com/images/dog-puppy-on-garden-royalty-free-image-1586966191.jpg?crop=0.752xw:1.00xh;0.175xw,0&resize=640:*"
name: "Save dogs"
[[Prototype]]: Object
[[Prototype]]: Object`

below is the code for deserializing

` const getAllCampaigns = async () => {
let accounts = await connection.getProgramAccounts(programId)

accounts.forEach( =>

{ let campaignData = deserialize( CampaignDetails.schema, //@ts-ignore CampaignDetails, x.account.data ) console.log(campaignData) }

)
}`

Custom Serialize/Deserialize

What do you think about allowing classes to provide custom serialize/deserialize methods?

By adding something like this to serializeStruct

function serializeStruct(schema: Schema, obj: any, writer: any) {
  if (typeof obj.borshSerialize === 'function') {
    obj.borshSerialize(writer);
    return;
  }
.....
}
.....

function deserializeStruct(
  schema: Schema,
  classType: any,
  reader: BinaryReader
) {
  if(typeof classType.borshDeserialize) {
    classType.borshDeserialize(reader);
    return;
  }
....
}

we could then do this:

export class PublicKey {
    keyType: KeyType;
    data: Uint8Array;

    static borsheDeserialize(reader: BinaryReader) {
        reader.readU8
    }

    static borshDeserialize(reader: BinaryReader): PublicKey {
        const keyType = reader.readU8();

        let data;
        if(keyType === KeyType.ED25519) {
            data = reader.readFixedArray(32);
        } else if(keyType === KeyType.SECP256k1) {
            data = reader.readFixedArray(64);
        }

        return new PublicKey({ keyType, data });
    }

    borshSerialize(writer: BinaryWriter) {
        writer.writeU8(this.keyType);
        writer.writeFixedArray(this.data);
    }
}

If this looks good I'm happy to implement it and submit a PR.

Type is not strict when serialize `u64` or `u128`

When schema is u64 or u128, the input value type can be number, bigint, string, boolean. But when schema is u8, u16, u32, the input value type can only be number, they are not consistent, that is strange.

I think we should make input value type be strict when serialize u64 or u128, I mean ONLY allow bigint.
Alternatively, we should make input type NOT be strict when serialize u8, u16, u32, I mean allow number, bigint, string and boolean

Schemaless serialization?

Is it possible to have a feature that auto-generates a schema from a plain-old-javascript-object? Or does Borsch not support that?

Deserialize requires type assertion

The deserialize method is not implemented as documented with a Class type parameter:

deserialize(schema: Schema, buffer: Uint8Array, class?: Class): any

Instead, it is:

deserialize(schema: Schema, buffer: Uint8Array, validate: boolean): DecodeTypes

Since there is no link to the type implementing Schema, a type assertion is required to assign deserialized values to custom types:

let data = ...;

type MyType = {
    value: string;
};
const MyTypeSchema: Schema = {
  struct: {
    value: "string"
  }
};

// No explicit type
const myTypeInstance_1 = borsh.deserialize(MyTypeSchema, data);
myTypeInstance_1.value; // Unresolved variable value 

// Use DecodeTypes
const myTypeInstance_2: DecodeTypes = borsh.deserialize(MyTypeSchema, data);
myTypeInstance_2.value; // Unresolved variable value

// Use MyType without type assertion
const myTypeInstance_3: MyType = borsh.deserialize(MyTypeSchema, data); // Initializer type DecodeTypes is not assignable to variable type MyType 

// Use type assertion
const myTypeInstance_4: MyType = <MyType>borsh.deserialize(MyTypeSchema, data);
myTypeInstance_4.value; // OK

Is this the way the library is supposed to be used? In my opinion the previous interface was cleaner. In any case, the README file is not in line with the recent changes.

Enum schema failed with typescript check when (de)serialize

Depending on the documentation, the enum schema should be

{ enum: [ { struct: { className1: structSchema1 } }, { struct: { className2: structSchema2 } }, ... ] }

Suppose we have a schema like

const borsh_schema = {
  enum: [
    { 
      struct: { 
        transfer: { 
          struct: {
            recipient: 'string',
            amount: "u64",
          },
        }
      }
    },
    { 
      struct: { 
        deploy: {
          struct: {
            contract: { array: { type: "u8" } },
            cbi_version: "u32",
          },
        }
      } 
    },
  ],
}

The type of schema will become (this is intended behavior)

const borsh_schema: {
    enum: ({
        struct: {
            transfer: {
                struct: {
                    recipient: string;
                    amount: string;
                };
            };
            deploy?: undefined;
        };
    } | {
        struct: {
            deploy: {
                struct: {
                    contract: {
                        array: {
                            type: string;
                        };
                    };
                    cbi_version: string;
                };
            };
            transfer?: undefined;
        };
    })[];
}

But this will fail the type check when we want to pass schema to (de)serialize

deserialize(borsh_schema, bytes)
// this will failed the type check

Typescript Playground: https://tsplay.dev/WPM05w

deserialize returns any because deserializeStruct isn't generic

My editor didn't pick up an annotation on deserialize so I wondered whether deserialize could be made generic over classType so that it actually returns the thing it is trying to deserializing the buffer into but apparently [it is already generic over classType! ](

borsh-js/borsh-ts/index.ts

Lines 440 to 445 in 261d9cd

export function deserialize<T>(
schema: Schema,
classType: { new (args: any): T },
buffer: Buffer,
Reader = BinaryReader
): T {
)

I'm not sure if i'm not seeign some obvious blocker to this, but from a cursory look at it โ€“ deserialize still always returns any because the deserializeStruct(...which deserialize calls?) [doesn't have a return type](

borsh-js/borsh-ts/index.ts

Lines 397 to 422 in 261d9cd

function deserializeStruct(
schema: Schema,
classType: any,
reader: BinaryReader
) {
if (typeof classType.borshDeserialize === "function") {
return classType.borshDeserialize(reader);
}
const structSchema = schema.get(classType);
if (!structSchema) {
throw new BorshError(`Class ${classType.name} is missing in schema`);
}
if (structSchema.kind === "struct") {
const result = {};
for (const [fieldName, fieldType] of schema.get(classType).fields) {
result[fieldName] = deserializeField(
schema,
fieldName,
fieldType,
reader
);
}
return new classType(result);
}
). Is that true?

I'm doing a tutorial on figment and at least that's how their borsh/lib type declarations read, maybe their version is lagging behind?

/// <reference types="node" />
import BN from 'bn.js';
export declare function baseEncode(value: Uint8Array | string): string;
export declare function baseDecode(value: string): Buffer;
export declare type Schema = Map<Function, any>;
export declare class BorshError extends Error 

{ originalMessage: string; fieldPath: string[]; constructor(message: string); addToFieldPath(fieldName: string): void; } 

export declare class BinaryWriter 

{ buf: Buffer; length: number; constructor(); maybeResize(): void; writeU8(value: number): void; writeU16(value: number): void; writeU32(value: number): void; writeU64(value: number | BN): void; writeU128(value: number | BN): void; writeU256(value: number | BN): void; writeU512(value: number | BN): void; private writeBuffer; writeString(str: string): void; writeFixedArray(array: Uint8Array): void; writeArray(array: any[], fn: any): void; toArray(): Uint8Array; } 

export declare class BinaryReader 

{ buf: Buffer; offset: number; constructor(buf: Buffer); readU8(): number; readU16(): number; readU32(): number; readU64(): BN; readU128(): BN; readU256(): BN; readU512(): BN; private readBuffer; readString(): string; readFixedArray(len: number): Uint8Array; readArray(fn: any): any[]; } 

export declare function serialize(schema: Schema, obj: any): Uint8Array;
export declare function deserialize(schema: Schema, classType: any, buffer: Buffer): any;
export declare function deserializeUnchecked(schema: Schema, classType: any, buffer: Buffer): any;

Remove dependency on text-encoding-utf-8

    1. Bakground

Should it rather not be up to the user to actually polyfill this dependency when needed?

I tried to setup near-api-js with [snowpack](https://www.snowpack.dev/) , and it fails since the text-encoding-utf-8 library does not use default export for the mjs build but rather a normal [export](https://github.com/arv/text-encoding-utf-8/blob/17484662e1e6881accbd69d5e51f3587c901311e/lib/encoding.lib.mjs#L665), which gives errors like these:

![Screenshot 2021-12-02 at 23 08 19](https://user-images.githubusercontent.com/155505/144511056-97c3567a-423b-4a56-9ddd-683b669c4918.png)

I hijacked the borsh module and removed the imports on text-encoding-utf-8, and stuff still works ๐Ÿ‘Œ๐Ÿฝ

    1. TextEncoder and TextDecoder Support

The text-encoding-utf-8 library has not been updated in four years (not that it is a bad sign), but the TextEncoder and TextDecoder are largely supported by most browsers now, mostly old Internet Explorer which does not support this and which has now been deprecated and no longer supported.

![Screenshot 2021-12-02 at 23 01 23](https://user-images.githubusercontent.com/155505/144510188-b737ffef-ceb7-4e84-a06e-604c07a0f668.png)

    1. Proposal
      Remove the import and dependency on text-encoding-utf-8 and rather specify in the README that if one needs support for this one should rather polyfill this manually instead.

Serialize Uint8Array?

noob alert:

I'm trying to serialize a Uint8Array to be sent to Rust WASI expecting a Vec to de-serialize and am not sure of how to declare the model. What I have is:

class AccumulatorSer {
constructor(value: Uint8Array) 

{ this.accumulator = value; } 

public accumulator: Uint8Array;
};

const ACCUMULATOR_SCHEMA = new Map([[AccumulatorSer, 

{ kind: 'struct', fields: [['accumulator', 'Uint8Array']] } 

]]);

// in function
let in_acc = new AccumulatorSer(Uint8Array.from(tracking_account_state.inputAccumulator.slice(8)));

serialize(ACCUMULATOR_SCHEMA, in_acc);

I get this error:

TypeError: writer<span class="error">[capitalizeFirstLetter(...)]</span> is not a function

Also, can I avoid having to wrap the Uint8Array in a class to build the schema?

Non-ASCII characters not being properly serialized

When we try to serialize a string containing non-ASCII characters, instead of using the multi-byte Unicode representation only the last 8 bits of the char value are written into the buffer.

Precise code where this happens is here.

    encode_string(value: unknown): void {
        this.checkTypes && utils.expect_type(value, 'string', this.fieldPath);
        const _value = value as string;

        // 4 bytes for length
        this.encoded.store_value(_value.length, 'u32');

        // string bytes
        for (let i = 0; i < _value.length; i++) {
            this.encoded.store_value(_value.charCodeAt(i), 'u8');
        }
    }

As a result the output buffer is not compatible with the standard deserialization for UTF-8 Strings.

Solution

Actually encode the character as multi-byte.

deserialize method not allowing Class implementation for classType parameter

I have been trying to deserialize solana account info with defined class type and schema but unfortunately I couldnt deserialize it.

let accounts = await connection.getProgramAccounts(programId)

accounts.forEach( =>

{ let campaignData = deserialize( CampaignDetails.schema, new CampaignDetails(), x.account.data ) }

)

error: Argument of type 'CampaignDetails' is not assignable to parameter of type 'new (args: any) => unknown'.
Type 'CampaignDetails' provides no match for the signature 'new (args: any): unknown'.ts(2345)

class CampaignDetails {
name: string
description: string
image_link: string
admin: Buffer
amount_donated: number

constructor()
constructor(
name?: string,
description?: string,
image_link?: string,
admin?: Buffer,
amount_donated?: number
)

{ ;(this.name = name!), (this.description = description!), (this.image_link = image_link!), (this.admin = admin!), (this.amount_donated = amount_donated!) }

static schema: any = new Map([
[
CampaignDetails,

{ kind: 'struct', fields: [ ['admin', [32]], ['name', 'string'], ['description', 'string'], ['image_link', 'string'], ['amount_donated', 'u64'], ], }

,
],
])
}

Any suggestionss ?

Deserializing Option type should return null in the None case

Hello,

Based on what's documented in the README:
A buffer containing an option type should deserialize to null if the value is of type None, or to the specific type if the value is Some

| option | `null` or type |

However, the implementation in deserializeField seems to be returning undefined instead of null.

return undefined;

Is this an oversight? Be happy to put in a PR for a quick change.

Thanks!

Add `u256`

Would you consider a PR for u256? This would be useful to serialize 256 bit cryptographic keys.

Is it possible to have a property with 2 possible values for the length in a borsh-js(v.1.0.0) schema?

Is it possible to have a property with 2 possible values for the length in a borsh-js(v.1.0.0) schema?

For example, I want to have a class with a publicKey property of type Uint8Array that has a length of 32 or 65 bytes.

Basically what I'm trying to do is this: https://github.com/near/near-api-js/pull/985/files#diff-dfff95256a781858ca8cc28ea405fc0e61dee26055dd6544465040df364a6ea4R183 but I don't see how this can be done with borsh-js v1.0.0.

This is how we are using borsh-js v1.0.0 https://github.com/near/near-api-js/blob/master/packages/transactions/src/schema.ts#L90-L230

Tried removing len: 32 but that didn't help.

How to serialize or create schema for vec of structs

in rust

#[derive(Debug, BorshDeserialize, BorshSerialize)]
pub struct ContentStorage {
    pub posts: Vec<Post>
}

#[derive(Debug, BorshDeserialize, BorshSerialize)]
pub struct Post {
    pub post_id: String,            // unique string
    pub image_url: Option<String>,  // ipfs link
    pub text: Option<String>,       // text for curr post
    pub owner: [u8; 32],            // post's owner
    pub likes_count: u32,           // likes of current post
    pub repost_count: u32,          // reposts
    pub comments: Vec<String>      // list of comments
}

in ts

export class ContentStorage {
    posts: Post[]
    static schema: Schema = new Map([
        [ContentStorage,
            {
                kind: 'struct',
                fields: [
                    ['posts', ['structs']],
                ]
            }],
    ]);

    constructor(posts: Post[]) {
        this.posts = posts
    }

    serialize(): Uint8Array {
        return serialize(ContentStorage.schema, this);
    }

    deserialize(data: Buffer): ContentStorage {
        return deserialize(ContentStorage.schema, ContentStorage, data);
    }
}

when I try to run it my result:
TypeError: writer[capitalizeFirstLetter(...)] is not a function

TypeError: reader[capitalizeFirstLetter(...)] is not a function

I am getting this error in an Angular14 project while trying to deserialize/decode a Uint8Array array.

ERROR Error: Uncaught (in promise): TypeError: reader[capitalizeFirstLetter(...)] is not a function
TypeError: reader[capitalizeFirstLetter(...)] is not a function
    at deserializeField (index.js:356:20)
    at deserializeStruct (index.js:410:33)
    at deserializeUnchecked (index.js:438:12)

Add option to ignore trailing bytes

Currently the library throws when the buffer supplied for deserialization is longer than expected:

throw new BorshError(`Unexpected ${buffer.length - reader.offset} bytes after deserialized data`);

This is nice when the input schema is of a fixed size. However, there's a use case where you have:

  • a schema with option fields; and
  • a fixed size storage backend.

under this scenario the trailing bytes are actually useless, and it's impossible to know the length of meaningful bytes without performing the deserialization that throws.

A possible solution would be to add an optional parameter on deserialize() which ignores the excess bytes.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.