gamaops / hfx-bus Goto Github PK
View Code? Open in Web Editor NEWRedis backed high frequency exchange bus
License: MIT License
Redis backed high frequency exchange bus
License: MIT License
To scale Redis horizontally we have to support distributed routing:
Describe the bug
message.load method is not handling unexistent payoad key properly.
To Reproduce
Try to load an unexistent payload key.
Expected behavior
Thrown a custom error class telling that this key doesn't exist, allow an option to just ignore the key.
Hello,
just want to let you now that your implementation for stream support was used to add stream support to Redis-Commander, a redis web ui (joeferner/redis-commander#351). The PR of the other user is not merged right now but will be soon.
(current WIP for integration: https://github.com/sseide/redis-commander/tree/streams)
I'll add some more license/copyright informations to the file at our project and will add a link back to your project from our readme. Hope its ok to use it (your license implies it). But just wanna let you know in case there are some objections or helpful hints...
Best regards,
Stefan Seide
When the same connection is used for both, the blocking call xreadgroup will block any other commands.
Steps to reproduce the behavior:
Bus S4KaCspAW1-3f-KU started!
{ event: 'healthz:ping:pending',
asyncListener: [AsyncFunction],
error:
{ Error: Undefined payload key "ping"
at Object.message.load (/home/adi/leaflet/cc-live-map/node_modules/hfxbus/lib/hfx-bus.js:318:12)
at process._tickCallback (internal/process/next_tick.js:68:7) data: { key: 'ping', message: [Object] } },
args:
[ { messageId: '_kpGUaDD0oiG_OKk',
trackerId: 'yVBM5NZLjPPb27ZQ',
streamId: '1557251758314-0',
replyTo: [Array],
groupName: 'healthz',
streamName: 'ping',
isAcknowledged: false,
isResolved: null,
_release: [Function: _release],
ack: [AsyncFunction: ack],
load: [AsyncFunction],
save: [AsyncFunction],
drop: [AsyncFunction],
purge: [AsyncFunction],
resolve: [AsyncFunction],
reject: [AsyncFunction],
ping: null } ] }
{ event: 'healthz:ping:pending',
asyncListener: [AsyncFunction],
error:
{ Error: Undefined payload key "ping"
at Object.message.load (/home/adi/leaflet/cc-live-map/node_modules/hfxbus/lib/hfx-bus.js:318:12)
at process._tickCallback (internal/process/next_tick.js:68:7) data: { key: 'ping', message: [Object] } },
args:
[ { messageId: '8AQzDFL3atkk4R7U',
trackerId: 'ZuGDFH7V3RxN_vcb',
streamId: '1557251763305-0',
replyTo: [Array],
groupName: 'healthz',
streamName: 'ping',
isAcknowledged: false,
isResolved: null,
_release: [Function: _release],
ack: [AsyncFunction: ack],
load: [AsyncFunction],
save: [AsyncFunction],
drop: [AsyncFunction],
purge: [AsyncFunction],
resolve: [AsyncFunction],
reject: [AsyncFunction],
ping: null } ] }
[...]
Please complete the following information:
I have the following use case:
As we can see here Redis tells us that clients can be used to shard data across nodes, so I think the ideal feature achieve this setup is:
crc16(routingValue) mod numberOfHAUnits
should give us the index of client that the command should useI'll start the work on this feature and it'll be a minor change to keep our current major v2.
HFXBus need stress tests, we need to plan which scenarios will be tested. The test must differentiate times by HFBus processing time and Redis/Network response time.
Hi,
thank you so much for your hard work!
There aren't a lot of projects utilizing redis streams at all and I love how very complete your implementation is!
I just read through the documentation and didn't find a way to fetch multiple items at once.
I know it's very event driven, but is this still possible somehow?
It would be awesome if up to X items could be read at once (non blocking) in intervals, for preprocessing them before processing and acknowledging them as a batch.
(Use case is some minor serialization during processing inside consumers running in parallel)
Thank you very much!
Hi,
Could you please explain the general flow in terms of Redis structures and key names?
I see a Redis Stream ("ping"), okay..., looking through the code I also see some pieces related to PubSub but PUBSUB CHANNELS *
doesn't list anything.
A short architecture paragraph would be great.
...something like https://www.slideshare.net/RedisLabs/redisconf18-building-lightweight-microservices-using-redis starting from slide 57
Thank you
PS: also 'hfxbus' is said to be the default namespace, shouldn't the stream name be hfxbus:ping
? or is this related to NodeJs/EventEmitter/RedisConsumerGroups/PubSub... what's the purpose of heartz
?
I'm just learning Redis & NodeJS & Javascript & Co... so apologies for the some of the, maybe, dumb questions
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.