Giter Club home page Giter Club logo

nodejs-questdb-client's Introduction

QuestDB Logo

 

QuestDB community Slack channel QuestDB open source contributors QuestDB on Apache Maven

English | 简体中文 | 繁體中文 | العربية | Italiano | Українська | Español | Português | 日本語 | Türkçe | हिंदी | Tiếng Việt

QuestDB

QuestDB is an open-source time-series database for high throughput ingestion and fast SQL queries with operational simplicity.

QuestDB is well-suited for financial market data, IoT sensor data, ad-tech and real-time dashboards. It shines for datasets with high cardinality and is a drop-in replacement for InfluxDB via support for the InfluxDB Line Protocol.

QuestDB implements ANSI SQL with native time-series SQL extensions. These SQL extensions make it simple to filter and downsample data, or correlate data from multiple sources using relational and time-series joins.

We achieve high performance by adopting a column-oriented storage model, parallelized vector execution, SIMD instructions, and low-latency techniques. The entire codebase is built from the ground up in Java, C++ and Rust with no dependencies and zero garbage collection.

QuestDB supports schema-agnostic streaming ingestion using the InfluxDB line protocol and a REST API for bulk imports and exports. The QuestDB SQL Web Console is an interactive SQL editor facilitating CSV import. Finally, QuestDB also includes the Postgres Wire Protocol for programmatic queries.

Popular tools that integrate with QuestDB include Apache Kafka, Grafana, Superset, Telegraf and Apache Flink.

Try QuestDB

We provide a live demo provisioned with the latest QuestDB release and sample datasets:

  • Trips: 10 years of NYC taxi trips with 1.6 billion rows
  • Trades: live crypto market data with 30M+ rows per month
  • Pos: geolocations of 250k unique ships over time

Checkout our interactive real-time market data dashboards and NYC Taxi Data Analytics Dashboards powered by QuestDB and Grafana.

Query Execution time
SELECT sum(double) FROM trips 0.15 secs
SELECT sum(double), avg(double) FROM trips 0.5 secs
SELECT avg(double) FROM trips WHERE time in '2019' 0.02 secs
SELECT time, avg(double) FROM trips WHERE time in '2019-01-01' SAMPLE BY 1h 0.01 secs
SELECT * FROM trades LATEST ON timestamp PARTITION BY symbol 0.00025 secs

Our demo is running on c5.metal instance and using 24 cores out of 96.

Get started

Install QuestDB

To run QuestDB, Docker can be used to get started quickly:

docker run -p 9000:9000 -p 9009:9009 -p 8812:8812 questdb/questdb

macOS users can use Homebrew:

brew install questdb
brew services start questdb

questdb start // To start questdb
questdb stop  // To stop questdb

The QuestDB downloads page provides direct downloads for binaries and has details for other installation and deployment methods.

QuestDB Cloud

QuestDB Cloud is the fully managed version of QuestDB, with additional features such as Role-based access control, Cloud-native Replication, Compression, monitoring and cloud-native snapshots. Get started with $200 credits.

Connect to QuestDB

You can interact with QuestDB using the following interfaces:

Insert data

Below are the official QuestDB clients for ingesting data via the InfluxDB Line Protocol:

End-to-end quickstart

Want to walk through everything, from streaming ingestion to visualization with Grafana? Check out our multi-path quickstart repository.

How QuestDB compares to other open source TSDBs

Checkout our benchmark blog post which compares QuestDB and InfluxDB across functionality, maturity and performance.

A chart comparing the ingestion rate of QuestDB, InfluxDB and TimescaleDB.

Resources

📚 Read the docs

❓ Get support

🚢 Deploy QuestDB

Contribute

We welcome contributions to the project, whether source code, documentation, bug reports, feature requests or feedback. To get started with contributing:

✨ As a sign of our gratitude, we also send QuestDB swag to our contributors. Claim your swag.

A big thanks goes to the following wonderful people who have contributed to QuestDB: (emoji key):


clickingbuttons
💻 🤔 📓
ideoma
💻 📓 ⚠️
tonytamwk
💻 📓
sirinath
🤔
igor-suhorukov
💻 🤔
mick2004
💻 📦
rawkode
💻 🚇

solidnerd
💻 🚇
solanav
💻 📖
shantanoo-desai
📝 💡
alexprut
💻 🚧
lbowman
💻 ⚠️
chankeypathak
📝
upsidedownsmile
💻

Nagriar
💻
piotrrzysko
💻 ⚠️
mpsq
💻
siddheshlatkar
💻
Yitaek
✅ 💡
gabor-boros
✅ 💡
kovid-r
✅ 💡

TimBo93
🐛 📓
zikani03
💻
jaugsburger
💻 🚧
TheTanc
📆 🖋 🤔
davidgs
🐛 🖋
kaishin
💻 💡
bluestreak01
💻 🚧 ⚠️

patrickSpaceSurfer
💻 🚧 ⚠️
chenrui333
🚇
bsmth
📖 🖋
Ugbot
💬 📓 📢
lepolac
💻 🔧
tiagostutz
📓 🐛 📆
Lyncee59
🤔 💻

rrjanbiah
🐛
sarunas-stasaitis
🐛
RiccardoGiro
🐛
duggar
🐛
postol
🐛
petrjahoda
🐛
t00
🐛

snenkov
📓 🐛 🤔
marregui
💻 🤔 🎨
bratseth
💻 🤔 📓
welly87
🤔
fuzzthink
🤔 📓
nexthack
💻
g-metan
🐛

tim2skew
🐛 📓
ospqsp
🐛
SuperFluffy
🐛
nu11ptr
🐛
comunidadio
🐛
mugendi
🤔 🐛 📖
paulwoods222
🐛

mingodad
🤔 🐛 📖
houarizegai
📖
jjsaunier
🐛
zanek
🤔 📆
Geekaylee
📓 🤔
lg31415
🐛 📆
null-dev
🐛 📆

ultd
🤔 📆
ericsun2
🤔 🐛 📆
giovannibonetti
📓 🐛 📆
wavded
📓 🐛
puzpuzpuz
📖 💻 📓
rstreics
💻 🚇 📖
mariusgheorghies
💻 🚇 📖

pswu11
🖋 🤔 🎨
insmac
💻 🤔 🎨
eugenels
💻 🤔 🚧
bziobrowski
💻 📆
Zapfmeister
💻 📓
mkaruza
💻
DylanDKnight
📓 🐛

enolal826
💻
glasstiger
💻
argshook
💻 🤔 🎨 🐛
amunra
💻 📖 🐛
GothamsJoker
💻
kocko
💻
jerrinot
💻 🤔 🐛

rberrelleza
💻
Cobalt-27
💻
eschultz
💻
XinyiQiao
💻
terasum
📖
PlamenHristov
💻
tris0laris
📝 🤔

HeZean
💻 🐛
iridess
💻 📖
selmanfarukyilmaz
🐛
donet5
🤔 🐛
Zahlii
🐛
salsasepp
🐛
EmmettM
🐛 ⚠️

robd003
🤔
AllenEdison
🐛
CSharpDummy
🐛
shimondoodkin
🐛 🤔
huuhait
🐛 🤔
alexey-milovidov
🐛
suconghou
🐛

allegraharris
💻
oliver-daniel
💻
kerimsenturk5734
📖

This project adheres to the all-contributors specification. Contributions of any kind are welcome!

nodejs-questdb-client's People

Contributors

amunra avatar argshook avatar eugenels avatar glasstiger avatar jerrinot avatar juanarbol avatar puzpuzpuz avatar sklarsa avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar

Forkers

timmikeladze apapacy juanarbol suryatmodulus arleneio breavyn woss

nodejs-questdb-client's Issues

Specify minimal supported Node.js version or support older ones like 14

Scenario:

  • Create a QuestDB Cloud instance
  • Run generated ILP ingestion script for Node.js with Node.js 14, e.g. v14.17.6
  • Observe the following error:
$ node insert-client.js 
Successfully connected to <host>:<port>
Authenticating with <host>:<port>
(node:392696) UnhandledPromiseRejectionWarning: TypeError [ERR_INVALID_ARG_TYPE]: The "key.key" property must be of type string or an instance of Buffer, TypedArray, or DataView. Received an instance of Object
    at prepareAsymmetricKey (internal/crypto/keys.js:288:13)
    at Object.createPrivateKey (internal/crypto/keys.js:349:5)
    at authenticate (/home/puzpuzpuz/projects/node-experiments/node_modules/@questdb/nodejs-client/src/sender.js:387:40)
    at TLSSocket.<anonymous> (/home/puzpuzpuz/projects/node-experiments/node_modules/@questdb/nodejs-client/src/sender.js:133:43)
    at TLSSocket.emit (events.js:400:28)
    at addChunk (internal/streams/readable.js:290:12)
    at readableAddChunk (internal/streams/readable.js:265:9)
    at TLSSocket.Readable.push (internal/streams/readable.js:204:10)
    at TLSWrap.onStreamRead (internal/stream_base_commons.js:188:23)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:392696) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:392696) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

The reason is that crypto.createPrivateKey(key) supports JWK object starting from v15.12.0: https://nodejs.org/api/crypto.html#cryptocreateprivatekeykey

Messages in console always visible

I want to use this module, but since it uses console.info to show some messages, it is a little bit slow due to the sync code with console.
The debug/info messages should be optional and defined as part of the options when the user create the sender

How to query?

I see there is sender implementation but where is reading?

Reduce the number of necessary parameters when using ILP auth

For connecting via ILP auth, only the key_id and the private_key/d parameter should be needed. However, this client also requests the public key pair, which is annoying as those are not really needed, and the developer need to treat them as secrets.

Some official clients like Go, JAVA, or C# don't need the extra parameters.

Once changed, the documentation should be updated to reflect the simplified connection params.

Cant insert data no matter what I do

Hi,

Just installed questdb and trying to insert some data. I have created the table as such:

CREATE TABLE 'chartfeed' (
timestamp TIMESTAMP,
name SYMBOL capacity 256 CACHE,
quoteid INT,
marketid INT,
open INT,
high INT,
low INT,
close INT,
) timestamp (timestamp);

And below is the code Im using to insert:

` const { Sender } = require('@questdb/nodejs-client');

async function run() {
// create a sender using HTTP protocol
const sender = new Sender({ protocol: 'tcp', host: 'localhost', port: 9009, bufferSize: 4096 });
await sender.connect();

// add rows to the buffer of the sender
await sender.table('chartfeed')
    .symbol('name', 'TEST')
    .intColumn('quoteid', 1234)
    .intColumn('marketid', 5678)
    .intColumn('open', 33450)
    .intColumn('high', 33480)
    .intColumn('low', 33410)
    .intColumn('close', 33460)
    .timestampColumn('timestamp',1715809721)


// flush the buffer of the sender, sending the data to QuestDB
// the buffer is cleared after the data is sent, and the sender is ready to accept new data
await sender.flush();

// close the connection after all rows ingested
await sender.close();

}

run().then(function(){
console.log('good')
}).catch(function(err){
console.error('error',err)
}); `

Console log prints:

Successfully connected to localhost:9009
Connection to 127.0.0.1:9009 is closed
good

But however no data is inserted in the table when I inspect with the web console.

Screenshot 2024-05-15 at 23 20 44

Cant see what Im doing wrong?

Client should skip columns if value is null

When adding columns to the sender, if we pass a null value, as in

sender.table('prices').symbol('instrument', 'EURUSD')
            .floatColumn('bid', null).at(Date.now(), 'ms');

It would be convenient that the column was ignored, rather than raising an error. It would make things easier for programmatic access

Missing method to insert binary data

Hi,
in the QuestDB spec there's a column type "binary".
We want to use this column type to store base64 encoded data.
Unfortunately, there's no "binaryColumn" method in sender.js.
Is this planned or is there any other way to insert binary data using this library?
Thanks

geohash support

would be great to have the geohash support, I am willing to contribute but I would need some guidence

Inserting data from more than one script/pm2 process causes Web console to refuse connection

Hi,

I have just managed to code a script that pulls data from an API and inserts into my questdb. I am using PM2 to manage the processes where each process runs at every second.

I have setup 4 processes to pull data from 4 different markets and the moment I have all 4 running, when I try to visit web console it gives me a refused connection. Dialling back PM2 to 1 process only and I can login just fine.

Is there a limitation on clients connected somewhere that I can change?

It seems everything grinds to a halt the moment I have all 4 script inserting data

Here is a gist:

https://gist.github.com/hmpmarketing/b18ba2b6db4fb0af370fbfea1f49c366

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.