mcollina / mqtt-level-store Goto Github PK

View Code? Open in Web Editor NEW

24.0 7.0 11.0 23 KB

Store your in-flight MQTT message on Level, for Node

License: MIT License

JavaScript 100.00%

mqtt-level-store's Introduction

MQTT.js Level Store

Persistent Store for in-flight MQTT.js packets. Works in Node and the Browser thanks to level.

Installation

npm install mqtt-level-store --save

Usage

'use strict';

var mqtt = require('mqtt'),
  levelStore = require('mqtt-level-store');
  manager = levelStore('path/to/db');

var client = mqtt.connect({
  port: 8883,
  incomingStore: manager.incoming,
  outgoingStore: manager.outgoing
});

//// or
// var client = mqtt.connect('mqtt://test.mosca.io', {
//  port: 8883,
//  incomingStore: manager.incoming,
//  outgoingStore: manager.outgoing
//});

client.on('connect', function() {
  console.log('connected');
  client.publish('hello', 'world', {qos: 1}, function() {
    console.log('published');
    client.end();
  });
});

Note, path/to/db is a folder, not a file. Files will be created at the given path.

License

MIT

mqtt-level-store's People

Contributors

Stargazers

Watchers

Forkers

oroce charliexp behrad abcd-ca mitsos1os redboltz ogis-fujiwara amaurioliveira ogis-yamazaki climax777 nunesd

mqtt-level-store's Issues

how to use in browser

var manager = '/a.json';
    client = mqtt.connect('ws://localhost:3000', {
            username: logid
            , password: logpass
            , incomingStore: manager.incoming
            , outgoingStore: manager.outgoing
             ,keepalive : 0
          });

mqtt.Store();

I tried this code in a html file but no action in a.json file

Is there any example please?

end event fires prematurely

I inititally thought the problem is in MQTT itself

Though after thorough testing, I found that level-sublevel's readStream sends 'end' event prematurely. It sends it after the last item is read, rather than the last read(1) === null. This causes a bunch of problems (see below)
mqttjs/MQTT.js#1319

mqtt support, modify mqtt client

Good Day
@mcollina great work, I was going to implement this with nedb but this seems very interesting.

One small issue, there is a problem with when using this package with current mqtt.js version.

When a in-flight message is loaded in setupStream the message is indeed loaded, but after the message is sent, when the check for callback is done in handleAck, }because no callback was assigned in the this.outgoing array, the message is not deleted from the LevelDB storage.

How do you propose to fix this issue cleanly? I managed to do this assigning a dummy callback before the that._sendPacket(packet) function, but maybe you can propose a bigger change in the client.js library.

Kind regards

ldb files are not cleaned event after successful publishes

I kept my device offline for 2 days and let it gather data in the store (140 KB every minute)

After I put it online, it uploaded a lot of data and then stopped. Now it was only uploading the new incoming data, so I guess it uploaded everything.

However, before putting the device online, I saw there were files of about 230 MB.
After all of the uploading completed, the files were still there. After few more fresh data uploads, some of the files were removed, however, there are still files of about 190 MB.

Is there a setting I am missing? How does this cleanup happen?

Question: Are there any data size limitations?

I am talking about the total size of data that I can have offline in base network connectivity is lost.

I was using mqtt-nedb-store but nedb is does not support more than 256 MB as it needs to parse the file to json, and 256MB is the limit of js string.

Here it looks like the data is being written to multiple small files, which I think avoids that issue.

So are there any limitations that I should know of?

update dependencies and add Node 10

Bump level-sublevel

Currently we are still using level-sublevel 6.x, which depends on levelup 0.19, which does not run on node v4.

cc @oroce

See dominictarr/level-sublevel#101.

[proposal]replacing level-browserify with level

https://github.com/Level/level-browserify is deprecated now.
The readme said that they recommend to replace to https://github.com/Level/level

I've replaced on my local environment and works well but level requires Node.js 8+.
I personally think that Node 8+ is good enough. AWS lambda supports Node 8.10 https://aws.amazon.com/about-aws/whats-new/2018/04/aws-lambda-supports-nodejs/?nc1=h_ls

Current .travis.yml supports older version. https://github.com/mcollina/mqtt-level-store/blob/master/.travis.yml

If we can remove old version support, I will send a pull request to replace level-browserify with level.

What do you think?

How to know when all messages have been sent?

Hi, I have had good success with this library. I have a device that sometimes goes offline for long periods and upon reconnecting sends its mqtt messages that were queued in the mqtt-level-store. This works.

What I am missing is a way to know when it's safe to power-off my device. I want to be able to know if all mqtt messages have been sent from the store or if it's still busy sending. Is there a hook for this already that you can point me to? Otherwise, is there a particular place in the source code you can point me to to develop it myself?

Thanks!

Keys limitation (Offline storage issue)

Hi there!

context:
Mqtt protocol has from 1 to 65.535 ids for in flight messages - messages in the middle of transmission between the client and the server.

This storage uses the messageId as key to store a timestamp, and then that timestamp to store the pacakge. But there is a little trouble.

problem:
Mqtt's messageId is generated with a random value between 1 and 65.535, which leave us with only 65.535 possible keys, and with an increasing risk of losing data.

risks:
When the storage is empty, we have a low risk of repeating keys, but when it starts growing, the chance of generating a random value between 1 and 65.535 that we already have stored gets higher, and after 32k packages stored (after a few days without internet connection), the chance is to lose half of the packages.

proposal:
If the local storage could use another, longer and not randomly assigned message id for local storage this would solve the problem.

Then would have to coordinate so that in flight message id's do not clash, and keep track of which message mqtt is sending (for qos > 0), because it can deliver it (and have to delete it from storage), or can fail (and have to insert it in the storage again, if it was pulled out).

Hope I explained myself...

Best regards!

levelstore started too late

in some cases, the levelstore started too late after the client connected to broker. this makes some of messages are lost.
I'd like to know if there is any way to detect if the level store's successfully launched before establish the connection to broker.

Error: Could not locate the bindings file. Tried

I create a file directory, why the error？

→ e:\WS_WorkSpace\Svn-RoadNetTraffic\mqtt-collect-server\node_modules\leveldown\build\leveldown.node
→ e:\WS_WorkSpace\Svn-RoadNetTraffic\mqtt-collect-server\node_modules\leveldown\build\Debug\leveldown.node
→ e:\WS_WorkSpace\Svn-RoadNetTraffic\mqtt-collect-server\node_modules\leveldown\build\Release\leveldown.node
→ e:\WS_WorkSpace\Svn-RoadNetTraffic\mqtt-collect-server\node_modules\leveldown\out\Debug\leveldown.node
→ e:\WS_WorkSpace\Svn-RoadNetTraffic\mqtt-collect-server\node_modules\leveldown\Debug\leveldown.node
→ e:\WS_WorkSpace\Svn-RoadNetTraffic\mqtt-collect-server\node_modules\leveldown\out\Release\leveldown.node
→ e:\WS_WorkSpace\Svn-RoadNetTraffic\mqtt-collect-server\node_modules\leveldown\Release\leveldown.node
→ e:\WS_WorkSpace\Svn-RoadNetTraffic\mqtt-collect-server\node_modules\leveldown\build\default\leveldown.node
→ e:\WS_WorkSpace\Svn-RoadNetTraffic\mqtt-collect-server\node_modules\leveldown\compiled\8.8.0\win32\x64\leveldown.node