infinyon / fluvio-client-node Goto Github PK

View Code? Open in Web Editor NEW

9.0 9.0 9.0 9.64 MB

Node.js client for Fluvio

Home Page: https://infinyon.github.io/fluvio-client-node/

License: Apache License 2.0

Makefile 1.93% Rust 55.33% JavaScript 1.61% TypeScript 41.13%

fluvio-client nodejs streaming

fluvio-client-node's People

Contributors

Stargazers

Watchers

Forkers

vijaylaxmid ryanmtate simlay realpha warycat sehz bencleary estebanborai morenol

fluvio-client-node's Issues

CI: audit update watch semver ecosystem updates and fix npm audit issues

reenable halt on failed npm audit check

JSDoc and TypeScript Definitions

nodejs fluvio/client package throwing improper architecture error on Apple Silicon

What happened
When following the Hello Fluvio Node Example on a Mac M1 (arm64) an error is thrown trying to run producer.ts.

npx ts-node producer.ts

Error: dlopen(/Users/lance/code/learn-fluvio/fluvio-demo/node_modules/@fluvio/client/dist/darwin/index.node, 0x0001): tried: '/Users/lance/code/learn-fluvio/fluvio-demo/node_modules/@fluvio/client/dist/darwin/index.node' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/index.node' (no such file), '/usr/lib/index.node' (no such file)
    at Object.Module._extensions..node (node:internal/modules/cjs/loader:1179:18)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:999:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/Users/lance/code/learn-fluvio/fluvio-demo/node_modules/@fluvio/client/src/index.ts:43:16)
    at Module._compile (node:internal/modules/cjs/loader:1097:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1149:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
  code: 'ERR_DLOPEN_FAILED'

# confirm node process is running on arm64 architecture
node -p process.arch
arm64

Expected behavior
I would expect the fluvio/client package to support Apple Silicon.

Describe the setup

local fluvio install with Kinds cluster

Release Channel      : stable
Fluvio CLI           : 0.9.18
Fluvio CLI SHA256    : f97262eec5fcb1f0b05c3cd6d1a7cdf40cfe83d1c5b702be49b82aa4575f409e
Fluvio Platform      : 0.9.18 (kind-fluvio)
Git Commit           : 2b6397f56e3935807ebb9e84c60517f815426d51
OS Details           : Darwin 12.1 (kernel 21.2.0)
=== Plugin Versions ===
Fluvio Runner (fluvio-run)     : 0.0.0
Infinyon Cloud CLI (fluvio-cloud) : 0.1.6

NPM Fluvio/Client Package:

 "dependencies": {
    "@fluvio/client": "^0.10.0"
  }

How to reproduce it (as minimally and precisely as possible)
Steps to reproduce the behavior:

Follow the steps outlined in the NodeJs HelloWorld example
Alternatively, if you just const fluvio = new Fluvio(); into a typescript file and run it will trigger the issue.

Add Stream (an EventEmitter) abstraction to client

Related to #43, let's add the Stream abstraction.

Ship with linked binary; pre-build

Make release process smoother

If you modify the rust a full release requires 3 different releases:

These also need to be done in a specific order. Let's make this a bit smoother.

Update client to latest interfaces changes in fluvio rust client

add API to consume with smart stream filter

Infinite loop when disconnected from server

Start a fluvio connection and disconnect the server.

The client continuously prints this message in the log

Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.274 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.276 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.276 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.276 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.276 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.276 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.276 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.276 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.276 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.276 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.277 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.277 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.277 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.277 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.277 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.277 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.277 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.277 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.277 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.278 ERROR fluvio::sync::controller: No more items to receive from stream!
^Cerver has terminated connection
Dec 21 13:38:19.278 ERROR fluvio::sync::controller: No more items to receive from stream!
Dec 21 13:38:19.278 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection
Dec 21 13:38:19.278 ERROR fluvio_socket::multiplexing: No more responses, server has terminated connection

Reference docs on Consume Iterator are out of date

This doc is out of date
https://infinyon.github.io/fluvio-client-node/classes/PartitionConsumer.html doesn't match

w.r.t.
https://github.com/infinyon/fluvio-client-node/blob/master/examples/consumeIterator.ts

Add M1 support to macOS releases.

It's unclear how to do this. We may need to figure out how to add support for install via git/https via the github url on this.

[Bug]: Create topic from node, gets 1 replica & 3 partitions, creating from CLI gets 1 replica & 1 partition

Create a topic from CLI

$ fluvio topic create topic-from-cli
topic "topic-from-cli" created
$ fluvio topic list
 NAME            TYPE      PARTITIONS  REPLICAS  IGNORE-RACK  STATUS                   REASON 
 topic-from-cli  computed      1          1                   resolution::provisioned

Setup Node Environment

$ mkdir custom-chat && cd custom-chat && npm init -y && \
npm install typescript ts-node @types/node -D && \
npm install @fluvio/client -S

create producer.ts with the following content

import Fluvio, { TopicReplicaParam } from "@fluvio/client";

// Create Fluvio Client Instance
const fluvio = new Fluvio();

// Handle Fluvio errors
process.on('unhandledRejection', error => {
    console.log(error);
});

(async () => {
    // Connect the fluvio cluster;
    await fluvio.connect();

    // Connect to admin interface
    const admin = await fluvio.admin();
    const topic = await admin.findTopic("topic-from-node");

    if (topic == null) {
       await admin.createTopic("topic-from-node");
    }

})();

Check the topics at the CLI:

$ fluvio topic list
 NAME             TYPE      PARTITIONS  REPLICAS  IGNORE-RACK  STATUS                              REASON 
 topic-from-cli   computed      1          1                        resolution::provisioned         
 topic-from-node  computed      1          3                   resolution::insufficient-resources  need 2 more SPU

Topic from node has 3 replicas, whereas topic from cli has only 1.

There are 2 issues here

Node client should default to 1 partition.
Node client should work with options to change the number of partitions.

When GC collects Fluvio Instance, PartitionConsumer stream closes

This might be fixed when infinyon/fluvio#645 is fixed but it's unclear. We may need to add a reference to the fluvio instance someplace to make sure it's not GC'd.

Add api to consume using smartmodule by name

Add generic API for admin client

Use `--release` for nj-cli in publishing the native fluvio packages to npm

Currently the package sizes on npmjs are 93MB and 24MB for linux and macOS packages on npmjs.com. It looks like those are debug builds that were released.

Add javascript support and examples

Need the ability to add topics, produce and consume form javascript. The project as it stands requires typescript which is great for large projects, but not ideal for small scripts.

Requirements:

javascript support
examples

Update Alpha Release Developer Docs

Update the developer docs to include information about building the native modules and publishing the @fluvio/client, @fluvio/native, and @fluvio/native-<platform> modules.

$ ts-node ./examples/createTopic.ts
requiring platform specific module
/Users/aj/projects/github/fluvio-client-node/node_modules/uuid/dist/esm-browser/index.js:1
export { default as v1 } from './v1.js';
^^^^^^

SyntaxError: Unexpected token 'export'
at wrapSafe (internal/modules/cjs/loader.js:1055:16)
at Module._compile (internal/modules/cjs/loader.js:1103:27)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1159:10)
at Module.load (internal/modules/cjs/loader.js:988:32)
at Function.Module._load (internal/modules/cjs/loader.js:896:14)
at Module.require (internal/modules/cjs/loader.js:1028:19)
at require (internal/modules/cjs/helpers.js:72:18)
at Object. (/Users/aj/projects/github/fluvio-client-node/examples/createTopic.ts:3:1)
at Module._compile (internal/modules/cjs/loader.js:1139:30)
at Module.m._compile (/Users/aj/.nvm/versions/node/v13.5.0/lib/node_modules/ts-node/src/index.ts:1043:23)

There's no nice way for the client to disconnect a PartitionConsumer stream

The stream method in PartitionConsumer will never finish. This causes the consume example to never finish.

One method for ending the stream on the client side is to wrap the stream call in a promise like the integration tests do. This is a bit gross and should be abstracted.

fix clippy warnings

error: `to_string` applied to a type that implements `Display` in `format!` args
   --> src/admin.rs:101:70
    |
101 |             FluvioError::Other(format!("serialization error: {}", err.to_string()))
    |                                                                      ^^^^^^^^^^^^ help: remove this
    |
    = note: `-D clippy::to-string-in-format-args` implied by `-D warnings`
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args

error: `to_string` applied to a type that implements `Display` in `format!` args
   --> src/admin.rs:120:74
    |
120 |                 FluvioError::Other(format!("serialization error: {}", err.to_string()))
    |                                                                          ^^^^^^^^^^^^ help: remove this
    |
    = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#to_string_in_format_args

PartitionConsume.fetch should have an optional parameter of Offset

Right now fetch requires an Offset instance, this should be an optional parameter which defaults to new Offset({index: 0, from: OffsetFrom.Beginning}) when passed into the rust.

Add api to list smart modules

Published package should be one package rather than 4.

Currently, there's 4 sub-package in this project:

@fluvio/native
@fluvio/native-linux
@fluvio/native-darwin
@fluvio/native-win but this sin't published yet.

Maximum call stack size exceed for `listTopics` if there are lots of topics

If there are around 800 topics (each with a UUID name) already in the cluster, the node client will error on listTopcis with this stacktrace:

npx ts-node ./examples/listTopic.ts
requiring platform specific module
connecting client to sc
problem getting topic RangeError: Maximum call stack size exceeded
    at arrayBufferToString (/home/simlay/projects/infinyon/fluvio-client-node/src/index.ts:867:32)
    at FluvioAdmin.<anonymous> (/home/simlay/projects/infinyon/fluvio-client-node/src/index.ts:386:16)
    at Generator.next (<anonymous>)
    at fulfilled (/home/simlay/projects/infinyon/fluvio-client-node/src/index.ts:5:58)

https://stackoverflow.com/questions/22123769/rangeerror-maximum-call-stack-size-exceeded-why looks to be the same issue in our arrayBufferToString.

format README.md

CI: Fluvio CI with latest fluvio action is broken due to cargo regression

Action: Fluvio CI with latest fluvio

cargo add fluvio --git https://github.com/infinyon/fluvio --branch master is expected to update the Cargo.toml line from:

fluvio = { features = ["admin"], git = "https://github.com/infinyon/fluvio.git", tag = "v0.10.14" }

fluvio = { features = ["admin"], git = "https://github.com/infinyon/fluvio.git" }

but fails to recognize admin feature defined in the fluvio crate in that repo crates/fluvio/Cargo.toml

Run cargo add fluvio --git https://github.com/infinyon/fluvio --branch master
    Updating git repository `[https://github.com/infinyon/fluvio`](https://github.com/infinyon/fluvio%60)
      Adding fluvio (git) to dependencies.
error: unrecognized feature for crate fluvio: admin
no features available for crate fluvio

Add Key/Value support for Node client

Once Key/Value support is added for the Rust client (#828), we need to update the node client interface to also allow sending Key/Value records.

This is a subtask of infinyon/fluvio#101.

AsyncIterator support for PartitionConsumer

Right now the stream api is:

consumer.stream(offset, (data) => {
})

Right now it's impossible to stop this stream because this spawns a new task on the rust side of things. This means that stream can't be unit tested either.

Long term we should support the AsyncIterator pattern so we can do things like...

for await (const x of PartitionConsumer.iterable()) {
}

where the iterable call returns something that is an AsyncIterator.

This sort of interface would be more composable and we'd be able to more easily unit test (breaks would could the interator to get GC'd). And then we can wrap that in an event emitter if we'd like.

This could be done by implementing #44 and wrapping that or it can be done rust side with node-bindgen depending on the amount of work.

@realpha has expressed some interest in this. Thoughts?

CI using GitHub action

unit test
integration test using minikube(linux)

[Bug]: Creating a topic with parameters from Node returns timeout.

Start with a fresh system

$ fluvio install cluster --local

Setup Node Enviroment

$ mkdir custom-chat && cd custom-chat && npm init -y && \
npm install typescript ts-node @types/node -D && \
npm install @fluvio/client -S

create producer.ts with the following content

import Fluvio, { TopicReplicaParam } from "@fluvio/client";

// Create Fluvio Client Instance
const fluvio = new Fluvio();

// Handle Fluvio errors
process.on('unhandledRejection', error => {
    console.log(error);
});

(async () => {
    // Connect the fluvio cluster;
    await fluvio.connect();

    // Connect to admin interface
    const admin = await fluvio.admin();
    const topic = await admin.findTopic("hello-fluvio");

    if (topic == null) {
        let topicParams: TopicReplicaParam = {
            partitions: 1,
            replicationFactor: 1,
            ignoreRackAssignment: true
        };

       await admin.createTopic("hello-fluvio", topicParams);
    }

})();

Run script to create topic

$ npx ts-node ./producer.ts
requiring platform specific module
[Error: Fluvio socket error]

Once the system is in that state, a topic with this name can no longer be added to Fluvio

$ fluvio topic create hello-fluvio
Error: 
   0: Fluvio client error
   1: Fluvio socket error
   2: time out in send and request: 3

Location:
   src/cli/src/root_cli.rs:138

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Detect disconnects and perhaps auto-reconnect

Right now the node client doesn't know if it's been disconnected. We (the node client library) should detect if the connection has been dropped and either emit the disconnection or auto-reconnect. We could make the auto-reconnect an option.

Better Error Handling

Add better error handling for NodeJS client.

Currently, there are a number of places where the client might fail, and no error will be provided to the user.

Update Examples to be more idiomatic and show best practices for using fluvio.

Right now in CI we compile the examples and run them but they're all in try catch blocks and never fail. Let's update them to show more use cases of the fluvio node client.

Publish CI should create a github tag for the current version being published.

update to latest release of rust client for OSS release

Update rust client to be 0.4

Update Node.js Client

Update client to use alpha api for client

Add ToRecords for FetchablePartitionResponse

Getting the actual messages out of a fetch in PartitionConsumer is a bit of a pain and pretty much requires doing:

const batchesToRecords = (response: FetchablePartitionResponse) => {
    let records = new Array();
    response.records.batches.forEach(batch => {
        batch.records.forEach(record => {
            records.push(record.value);
        });
    })
    return records;
};

Let's add something like toRecords to the FetchablePartitionResponse body which returns an Array<string> or something.

Add docs to release artifacts

We want to build an archive of reference docs from older versions on the fluvio-website. The docs are already generated and hosted. Only need to keep track of each version. This will be easier to accomplish if we don't have to scrape with git.

Related infinyon/fluvio-website#460