Giter Club home page Giter Club logo

jstp's Introduction

Metarhia Logo

Travis CI AppVeyor CI Coverage Status NPM Version NPM Downloads/Month NPM Downloads

JSTP / JavaScript Transfer Protocol

JSTP is an RPC protocol and framework which provides two-way asynchronous data transfer with support of multiple parallel non-blocking interactions that is so transparent that an app may not even distinguish between local async functions and remote procedures.

And, as a nice bonus, there's a blazing fast JSON5 implementation bundled in!

This project is bound by a Code of Conduct.

Installation

JSTP works in Node.js and web browsers:

$ npm install --save @metarhia/jstp

Or, alternatively, there is jstp.umd.js UMD bundle.

We also have official client-side implementations for Swift and Java that work effortlessly on iOS and Android 🎉

There is also an interactive CLI provided by this package:

$ npm install -g @metarhia/jstp
$ jstp-cli

Getting Started

Server:

'use strict';

const jstp = require('@metarhia/jstp');

// Application is the core high-level abstraction of the framework. An app
// consists of a number of interfaces, and each interface has its methods.
const app = new jstp.Application('testApp', {
  someService: {
    sayHi(connection, name, callback) {
      callback(null, `Hi, ${name}!`);
    },
  },
});

// Let's create a TCP server for this app. Other available transports are
// WebSocket and Unix domain sockets. One might notice that an array of
// applications is passed the `createServer()`. That's because it can serve
// any number of applications.
const server = jstp.net.createServer([app]);
server.listen(3000, () => {
  console.log('TCP server listening on port 3000 🚀');
});

Client:

'use strict';

const jstp = require('@metarhia/jstp');

// Create a TCP connection to server and connect to the `testApp` application.
// Clients can have applications too for full-duplex RPC,
// but we don't need that in this example. Client is `null` in this example,
// this implies that username and password are both `null`
// here — that is, the protocol-level authentication is not leveraged in this
// example. The next argument is an array of interfaces to inspect and build
// remote proxy objects for. Remaining arguments are for
// net.connect (host and port) and last argument is a callback
// to be called on successful connection or error.
jstp.net.connectAndInspect(
  'testApp',
  null,
  ['someService'],
  3000,
  'localhost',
  handleConnect
);

function handleConnect(error, connection, app) {
  if (error) {
    console.error(`Could not connect to the server: ${error}`);
    return;
  }

  // The `app` object contains remote proxy objects for each interface that has
  // been requested which allow to use remote APIs as regular async functions.
  // Remote proxies are also `EventEmitter`s: they can be used to `.emit()`
  // events to another side of a connection and listen to them using `.on()`.
  app.someService.sayHi('JSTP', (error, message) => {
    if (error) {
      console.error(`Oops, something went wrong: ${error}`);
      return;
    }
    console.log(`Server said "${message}" 😲`);
  });
}

Project Maintainers

Kudos to @tshemsedinov for the initial idea and proof-of-concept implementation. Current project team is:

jstp's People

Contributors

aqrln avatar belochub avatar dimon-durak avatar dzyubspirit avatar gyfk avatar igorgo avatar lundibundi avatar mille-nium avatar nechaido avatar tshemsedinov avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jstp's Issues

Make parser single-pass

Avoid extra memory copying by eliminating jstp::parser::internal::PrepareString and extracting lexems on demand instead.

Специфікація пакета 'State'

Для опису операції state, del: використовується окреме позначення '[]'.

{state:[-12,'object.path.value'],inc:5}
{state:[-13,'object.path.value'],dec:1}
{state:[-14,'object.path.value'],set:12}
{state:[-15,'object.path.value'],del:[]}

Для приведення до єдиного вигляду можна використати:

{state:[-12,'object.path.value'],inc:5}
{state:[-13,'object.path.value'],dec:1}
{state:[-14,'object.path.value'],set:12}
{state:[-15,'object.path.value'],del:0}

або

{state:[-12,'object.path.value'],inc:[5]}
{state:[-13,'object.path.value'],dec:[1]}
{state:[-14,'object.path.value'],set:[12]}
{state:[-15,'object.path.value'],del:[]}

Get rid of redundant parameters in functions

Some public callbacks and, maybe, core functions take redundant parameters that can be extracted from the other ones. For example, there's no need to pass an application to a function if a connection is already supplied.

Build should fail when native extension can't be built on CI

I checked CI logs today and found out that our native extension was not built for a while now because of compilation errors. It caused no errors in CI, because it just used JavaScript implementation of parser which is working fine.
I think that we maybe need to add some environment variable check for CI so that tests won't pass when native extension can't be built.

Switch to a new branching model

Rationale

As @tshemsedinov suggested, we should change the branching model (originating from Impress, which, in turn, is derived from the Node.js) to be consistent with the rest of our projects.

Model description

master is the main development branch. It hosts the latest working but possibly unstable version of the project. Each major (since 1.0.0) or minor (before the release of 1.0.0) version has its own stable branch (i.e., v0.4, v0.5, v1.x, v2.x etc). Any new features are developed in separate branches and merged into master when they are ready and all tests pass. Commits from master are merged into the current version branch and may be cherry-picked onto previous versions branches if there is an urgency to do so. Releases are tagged (using Git's lightweight tags, not annotated ones) and published to npm from version branches.

Implications

Since our master and develop branches are in sync now, this migration will not break the workflow and does not require any changes in the codebase. These rules apply to new commits only. Versions released before the introduction of this branching model (except 0.4.4) will not be tagged and branches for pre-0.4 releases will not be created.

FYI @belochub

Refactor module.exports proposal

Here we use object for module.exports to export submodule interface and require to import it in lib/.

I propose to export function(api) with api argument to pass submodule interfaces and node.js api interfaces to each submodule for crosslinking. As we did in metarhia/Common and assemble all submodules in the following way: metarhia/Common/common.js

Fix string serialization

Current string serializer is only correct for the simplest cases and must be rewritten totally.

Semantic versioning and version compatibility

Note: this issue doesn't enforce any new policies, it just documents the approach we already use implicitly, for the sake of clarity.

Though we have not reached v1.0.0 yet, we already have to deal with correct versioning of the package. For example, Impress and our proprietary apps currently support JSTP 0.6, which is the latest version, but JSTP 0.7 will have some backward incompatibilities. And if the migration is not a problem for Impress, it actually is in the latter case. But there are some commits that should be landed on v0.6 too (e.g., 6069f18 and 851a2c6). It's just one of the constantly recurring examples which demonstrates the importance of proper version compatibility.

When we reach 1.0, we will just adhere to semantic versioning strictly, but semver says nothing about versions <1.0.0. Thus we extend semver's rules applying them to these versions this way: in 0.minor.patch scheme minor acts as a major semver version and patch as both minor and patch, regardless of whether a change is a feature or a bugfix.

Accordingly, if labels named semver-major or semver-minor are added to any issue or pull-request before we have released v1.0.0, they actually assume minor and patch subversions.

/cc @tshemsedinov @belochub

Broken import in `tools/cli.js`

It went unnoticed during the review, but I think I broke it in 087d0f5. Please self-assign if someone is willing to fix it right now, otherwise I'll do that when I get to my editor :)

API versioning

It must be possible to provide several versions of an API and to select one during a handshake.

Support different serialization formats

Now JSTP works only with one JSON5 like serialization format. While it would be much more extendable use abstract interface that provides such functions as:

  • parse()
  • serialize()
  • parseNetworkPacket()
  • serializeNetworkPacket()

This way will be able to add new formats, for example JSON or some binary format.

Use local variable instead of this

I think we have multiple problems:
https://github.com/metarhia/JSTP/blob/master/lib/server.js#L71
https://github.com/metarhia/JSTP/blob/master/lib/server.js#L90
https://github.com/metarhia/JSTP/blob/master/lib/server.js#L103
https://github.com/metarhia/JSTP/blob/master/lib/server.js#L64

Proposed fix:

Server.prototype.getClients = function() {
  if (this._cachedClientsArray.length === 0) {
    var server = this; // FIX
    this._cachedClientsArray = Object.keys(this.clients).map(function(cid) {
      return server.clients[cid]; // FIX
    });
  }
  return this._cachedClientsArray;
};

Incorrect term usage (s/packet(s)/message(s))?

It is not technically correct to say "JSTP packets" since packets are the PDUs of layer 3 protocols. Layer 5 protocols don't have a standardized concept of PDU (and what they transfer is simply referred to as data) but in practice their data units are commonly called messages.

Binary data transfer

  • Support streaming but with chunks, because connection should not be blocked by single large data transmission stream
  • Client and server will arrange chunk size in handshake packets
  • Stream initialization packet will contain metadata about data type (file, streaming video or audio, etc.), file name (optional), total size (optional)
  • Both sides will be able to suspend, resume and terminate stream

Data-driven approach

  1. Модель данных транслируется в схему GUI, пример impress.core.schema.js
  2. Схему GUI можно написать и руками
  3. Схема GUI рендерится на разных платформах
  4. GUI взаимодействует с пользователем и собирает реакцию
  5. Реакция приводит к вызову API или работе с GlobalStorage
  6. Обращение к GlobalStorage преобразуется в вызовы API
  7. Вызовы API попадают на сервер и меняют модель данных
  8. Интерфейсы API частично создаются из модели данных (CRUD и другие типовые)
  9. Модель данных мы меняем и она формирует до 70% системы
  10. Примерно 20% дописываем руками на любом этапе в декларативном стиле
  11. Еще 10% дописываем руками в императивном стиле
  12. Круг замкнулся, данными начали, данными кончили

Предложения по реализации binary stream

Нам нужно сделать JSTP совместимым с передачей бинарных данных в режиме стриминга, по частям, блоками, как торренты. Это не срочно, но жду ваших предложений, @aqrln и @RayGron Это может быть отдельный проект, по трансляции и ретрансляции видеопотока в двоичном виде через пиринговую сеть.

Delayed packet delivery

  • Default delivery behavior is immediate send
  • Optionally we may send delayed packet if no active connection we still may send:
    • with TTL (N milliseconds)
    • with expiration time (certain time)
      (this two cases may have only different interface but same implementation)

JSTP may store packets in GlobalStorage compatible interface, that may use memory or memory + persistent storage provider (if available for certain platform).

Support the ES2015 syntax for Unicode literals

ECMAScript 6.0 introduces a new Unicode literal form for long code points. For example, instead of '\uD83D\uDE3A' you can naturally write '\u{1F63A}'. We should support it in the JSRS parser as it is a part of JavaScript syntax and should be valid in JSTP.

Неоднозначність декодування записів

Проблема

За основу взято опис RecordData:

var metadata = {firstName: 'string',  lastName: 'string'};
var data = ['John', 'Doe'];
var obj = api.jstp.decode(data, metadata);

Проблема полягає у тому, що у загальному випадку, ми не можемо гарантувати, що правильним ключам поставляться у відповідність правильні значення. Тобто, для описаного прикладу можливі два варіанти:

// Варіант 1
console.dir(obj);  // { firstName: 'John', lastName: 'Doe' }

// Варіант 2
console.dir(obj);  // { firstName: 'Doe', lastName: 'John' }

Причина такої неоднозначності у тому, що специфікація JS не визначає порядок ключів у об'єкті, залишаючи на розсуд тих, хто реалізує її.

Пропозиції

Необхідно змінити формат метаданих таким чином, щоб він включав послідовність полів.

Варіант 1. Додаткове поле в метаданих

{
    firstName: 'string',  
    lastName: 'string',
    __order__: ['firstName', 'lastName'],
};

Варіант 2. Метадані як список

[
    ['firstName', 'string'],
    ['lastName', 'string'],
]

// або

[
    { field: 'firstName', type: 'string'},
    { field: 'lastName', type: 'string'},
]

Із урахуванням #2, я схиляюся до видозміненого першого варіанту (за основу взято коментар від 29 березня 2016 р):

{
  firstName: [ /* ... */ ],
  lastName: [ /* ... */ ],
  __meta__: {
    cache: { /* ... */ },
    order: ['firstName', 'lastName'],
  }
}

Implement heartbeat packets

In order to be able to know whether a connection is still established, heartbeat packets must be supported.

Metadata, DSL, IDL or schema

Вместо

{
  country: 'string',
  city: 'string',
  zip: 'number(5)',
  street: 'string',
  building: 'string',
  room: '[number]'
}

Делаем что-то типа:

{
  country: ['string'],
  city: ['string', { default: 'Kiev' }],
  zip: ['number', [0, 99999]],
  street: ['string', [5, 30]],
  building: ['string', [1, 5], 'nullable'],
  room: ['number', 'nullable']
}

Добавить методы синхронизации данных

Были: inc, dec, set, del
Заменили: set на let, del на delete
Добавляем: push, pop, shift, unshift, delete для работы с множествами
delete работает с хешами {}, массивами [] и множествами new Set()

Rewrite JSTP Record Serialization in C++

Given the same object and 100000 iterations:

  • JSON.stringify — 134.377 ms
  • JSON.parse — 201.747 ms
  • jstp.stringify — 492.840 ms (3.67x slower than JSON)
  • jstp.parse (naive three-line implementation) — 45634.546 ms (226.2x slower than JSON)
  • jstp.parse (secure implementation with validation and timeouts) — 61266.695 ms (303.68x slower than JSON)
  • jstp.parse (no validation, shared sandbox) — 5449 ms (27x slower than JSON)
  • jstp.parse (validation, shared sandbox) — 9011.31 ms (44.67x slower than JSON)

We have an efficient implementation of the parser and serializer in C++, it must be rewritten using V8 data types and integrated into this library.

Update: the prototype of a parser as a native Node.js addon gives the following results:

  • json.stringify — N/A (not finished yet)
  • json.parse — 652.148 ms (3.23x slower than JSON, 13.82x faster than our most recent version of the JavaScript parser)

Can we parse query syntax?

{
  name: 'Marcus',
  age > 20,
  city != 'Kiev'
}

or even

{
  name = 'Marcus',
  country <> 'China',
  score: 200..500,
}

or even

{
  name = 'Marcus',
  country  'China',
  score  [200,500),
  city  {'Kiev', 'Roma'}
}

It would be great to get something like following after parsing:

{
  name: ['=', 'Marcus'],
  age: ['>', 20],
  score: ['>=', 200, '<', 500],
  city: ['Kiev', 'Roma']
}

Reconnecting to existing sessions

Now we have an abstraction of session but it isn't used anywhere. There must be a possibility to connect to an existing session specifying its ID in handshake packet. When a client loses its connection, it must reconnect to the session automatically and transparently.

Optimize Connection events

  • Review existing events, remove unnecessary ones and rename long ones.
  • Prevent possible deopts in Connection#_emitPacketEvent or even get rid of this method completely.

Transition to ES6 and Node.js >=6.x

Unfortunately we cannot do it right now because of existing code, but as soon as the oldest supported version of Impress is 0.3, the following changes must be applied to this package:

  • Rewrite the code to ECMAScript 2015+
  • Use Jest, which is ES6 and Node.js >=4.x only, (or another testing framework if there will be anything better than Jest; right now there isn't) instead of Mocha and expect.js

UPD: migrating tests from Mocha and Chai is still important but it isn't a matter of transition to ES6.

Impossibility to catch `error` event arising in connections during handshake

At the moment, when a server is being created and someone sends incorrect first packet (which must be handshake, but can be anything), node will crash with message "Unhandled 'error' event", even if there is an error event listener for this server.
That happens because we forward error event listener from transport to server (as connectionError) only after handshake was processed.
I guess possible fix will be to forward error event to server on connection creation.

'\0' as package delimiter

Wouldn't it be better to use null characters as package delimiters? It is only one byte instead of three and it will under no circumstances appear in text data.

Реализовать api.vm для браузера

  • api.vm.createContext
  • api.vm.createScript
  • api.vm.isContext
  • api.vm.runInDebugContext
  • api.vm.Script
  • script.runInContext и api.vm.runInContext
  • script.runInNewContext и api.vm.runInNewContext
  • script.runInThisContext и api.vm.runInThisContext

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.