ce-rust / cerk Goto Github PK
View Code? Open in Web Editor NEWCloudEvents Router with a Microkernel
License: Apache License 2.0
CloudEvents Router with a Microkernel
License: Apache License 2.0
In some special scenarios, the Kernel can be blocked forever in case the channel from the Kernel to the port is full.
Port A: Pushed 31 incoming CloudEvents to the Kernel -> The port is blocked now.
Kernel: Pushes 31 outgoing CloudEvents to Port A -> Kernel is blocked
-> Deadlock
Use the health check. With the help of the health check port, this scenario can be identified, and the router can be restarted. This is okay if the delivery guarantee at-least-once is used.
the example needs to be fixed and reactivated in the pipeline
https://buildkite.com/ce-rust/cerk/builds/643#0181e289-4c86-43b2-9dfa-144d15a9610f
The MQTT Port currently only supports QoS 0 (At Most Once Delivery Guarantee). The reason is that the used MQTT library doesn't allow the delay of the PUBACK
message until the message is processed (see eclipse/paho.mqtt.c#522).
We tried to patch the C
and rust
library (see eclipse/paho.mqtt.c@master...ce-rust:feat/ack-after-processing and eclipse/paho.mqtt.rust@master...ce-rust:feat/ack-after-processing-0.8.0).
This patch allowed us to block in the message callback till the message was successfully delivered over another port.
But then we run into another problem: The C
library uses the same threads and mutex for all the connections. In the case there are multiple MQTT Ports, the blocking of the message handler results in the blocking of the other connections => Dead Lock.
We then tried to fork the process before opening the connections. This way the different ports would not share the same threads and mutex. We tried to implement that in the port crate (see https://github.com/ce-rust/cerk/tree/feat/mqtt-port-subprocess) but unfortunately, forking the process in that stage of the router lifecycle results in unexpected behavior (see https://thorstenball.com/blog/2014/10/13/why-threads-cant-fork/).
Possible solutions:
PUBACK
PUBACK
Scheduler
component for the router that is not based on threads but on processes. (requires major changes in cerk
, e.g. BrokerEvent
must be serializable.Currently,
mosquitto_client
is compiled with theWITH_TLS
flag set tono
. This means users ofmosquitto-client-wrapper
are not able to open secure connections.
ce-rust/mosquitto-client-wrapper#9
This means that the MQTT Mosquitto Port is currently not able to support TLS.
Implement port which connects to an open UNIX socket and listens on CloudEvents.
Currently, the router routes messages to ports.
However, most messaging protocols know the concept of channels.
It would be nice if the routing does not only allow a port destination but also a port+channel destination.
Example:
We have an AMQP Port that is connected to a RabbitMQ Service called port A.
Port A has a configuration of two exchanges: Exchange 1 and exchange 2.
Messages can be routed to port A.
Port A currently forwards the messages to all configured exchanges (1&2) on RabbitMQ.
It would be nice to be able to select the destination port AND channel, e.g., port A & exchange 1.
Workaround:
One port per channel group that should be addressable has to be configured.
This means that there is an overhead of multiple TCP connections to the same service.
For the example this means: Port A with exchange 1 and port B with exchange 2 have to be configured.
For the cerk_port_mqtt_mostquitto
we are using the mosquitto_client library. Unfortunately, the published version does not support the latest version of libmossquitto
even though there is an open pull request to add support for a while.
Because of that, we are using not the version of the package published on crates.io but the patched one (see #77 (comment)). This prevents us from publishing the cerk_port_mqtt_mostquitto
on crates.io
There are different ways to approach this issue:
The new release management introduced in #80 could easily be automated by just pushing a tag to master.
The version bump command would look something like cargo workspaces version custom "0.2.8" --no-git-push --allow-branch="*release*" --yes
cerk/src/kernel/config.rs:12:3
warning: derive helper attribute is used before it is introduced
--
| --> cerk/src/kernel/config.rs:12:3
| \|
| 12 \| #[serde(untagged)]
| \| ^^^^^
| 13 \| #[derive(Deserialize, Serialize)]
| \| ----------- the attribute is introduced here
| \|
| = note: `#[warn(legacy_derive_helpers)]` on by default
| = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
| = note: for more information, see issue #79202 <https://github.com/rust-lang/rust/issues/79202>
Harmonize the error handling by using anyhow for error handling in all components.
cerk/cerk/src/kernel/broker_event.rs
Line 10 in 6bd6210
the example needs to be fixed and reactivated in the pipeline
https://buildkite.com/ce-rust/cerk/builds/643#0181e289-4c94-4925-9eb2-2babc5b45ed4
with #![deny(missing_docs)]
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.