Giter Club home page Giter Club logo

rabbitmq-message-timestamp's Introduction

RabbitMQ Message Timestamp Plugin

This plugin fills the timestamp property and timestamp_in_ms header of a message as it enters RabbitMQ with the current (server node) timestamp value.

Supported RabbitMQ Versions

This plugin targets RabbitMQ 3.6.0 until 3.11.

⚠️ Starting in RabbitMQ 3.12, this plugin is deprecated ⚠️

In RabbitMQ 3.12 and later versions, instead of using this plugin, use the following rabbitmq.conf snippet:

message_interceptors.incoming.set_header_timestamp.overwrite = false

To allow timestamps being overwritten use:

message_interceptors.incoming.set_header_timestamp.overwrite = true

Limitations

This plugin cannot be used together with rabbitmq-routing-node-stamp as they override the same extension point.

Installation

Binary builds of this plugin can be obtained from the Community Plugins page.

See Plugin Installation for details about how to install plugins that do not ship with RabbitMQ.

Building from Source

You can build and install it like any other plugin (see the plugin development guide).

Usage

Just enable the plugin with the following command:

rabbitmq-plugins enable rabbitmq_message_timestamp

The plugin will then hook into the basic.publish process in order to add the current timestamp as seen by the broker.

Always overwrite timestamps

This plugin will not overwrite an existing timestamp on a message. To always overwrite, create an advanced.config file for RabbitMQ with the following content, or add the rabbitmq_message_timestamp term to your existing file:

[
    {rabbitmq_message_timestamp, [
        {overwrite_timestamps, true}
    ]}
].

Limitations

The plugin hooks into the basic.publish path, so expect a small throughput reduction when using this plugin, since it has to modify every message that crosses RabbitMQ.

This plugin should not be enabled at the same time as any other interceptors that hook into the basic.publish process, such as the rabbitmq-routing-node-stamp plugin. Enabling more than one interceptor that is registered to the basic.publish process will cause all AMQP 0-9-1 connections to fail when creating a new channel.

If there's enough demand, we could add in the future a way for only time-stamping messages that crosses certain exchanges, say by applying policies.

LICENSE

See the LICENSE file

rabbitmq-message-timestamp's People

Contributors

acogoluegnes avatar ansd avatar biiiipy avatar binarin avatar dcorbacho avatar dumbbell avatar gerhard avatar gmr avatar hradilf avatar kjnilsson avatar lukebakken avatar michaelklishin avatar mkuratczyk avatar spring-operator avatar videlalvaro avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rabbitmq-message-timestamp's Issues

Closing AMQP connections after activating the plugin

I obtained the binary for RabbitMQ 3.6.14 from the following sources:

https://dl.bintray.com/rabbitmq/community-plugins/rabbitmq_message_timestamp-1.0.0.ez
https://dl.bintray.com/rabbitmq/community-plugins/rabbitmq_message_timestamp-3.6.x-0b333722.ez
https://dl.bintray.com/rabbitmq/community-plugins/rabbitmq_message_timestamp-3.6.x-6a09867d.ez

The reason I tried several versions is that the link on the community plugin list is broken:

http://www.rabbitmq.com/community-plugins.html
rabbitmq_message_timestamp
Download for 3.6.x
is referencing https://dl.bintray.com/rabbitmq/community-plugins/rabbitmq_delayed_message_exchange-0.0.1.ez

However, if I enable the plugin using one of the provided binaries my Clients can't connect!

The error log in /var/log/rabbitmq/... is as follows:

=ERROR REPORT==== 11-Jan-2018::11:51:14 ===
Error on AMQP connection <0.440.0> (IP:55842 -> IP:5672, vhost: '/', user: 'admin', state: running), channel 1:
{{case_clause,
{'EXIT',
{undef,
[{os,system_time,[seconds],[]},
{rabbit_timestamp_interceptor,intercept,3,
[{file,"src/rabbit_timestamp_interceptor.erl"},{line,46}]},
{rabbit_channel_interceptor,call_module,4,
[{file,"src/rabbit_channel_interceptor.erl"},{line,74}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},
{rabbit_channel,handle_cast,2,
[{file,"src/rabbit_channel.erl"},{line,470}]},
{gen_server2,handle_msg,2,
[{file,"src/gen_server2.erl"},{line,1047}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}}},
[{rabbit_channel_interceptor,call_module,4,
[{file,"src/rabbit_channel_interceptor.erl"},{line,74}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},
{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,470}]},
{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1047}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}

=WARNING REPORT==== 11-Jan-2018::11:51:14 ===
Non-AMQP exit reason '{{case_clause,
{'EXIT',
{undef,
[{os,system_time,[seconds],[]},
{rabbit_timestamp_interceptor,intercept,3,
[{file,"src/rabbit_timestamp_interceptor.erl"},
{line,46}]},
{rabbit_channel_interceptor,call_module,4,
[{file,"src/rabbit_channel_interceptor.erl"},
{line,74}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},
{rabbit_channel,handle_cast,2,
[{file,"src/rabbit_channel.erl"},{line,470}]},
{gen_server2,handle_msg,2,
[{file,"src/gen_server2.erl"},{line,1047}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}}},
[{rabbit_channel_interceptor,call_module,4,
[{file,"src/rabbit_channel_interceptor.erl"},
{line,74}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},
{rabbit_channel,handle_cast,2,
[{file,"src/rabbit_channel.erl"},{line,470}]},
{gen_server2,handle_msg,2,
[{file,"src/gen_server2.erl"},{line,1047}]},
{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,239}]}]}'

=INFO REPORT==== 11-Jan-2018::11:51:14 ===
closing AMQP connection <0.440.0> (IP:55842 -> IP:5672, vhost: '/', user: 'admin')

My activated plugins are as follows ($ rabbitmq-plugins list):

Configured: E = explicitly enabled; e = implicitly enabled
| Status: * = running on rabbit@st0cvm3000839
|/
[e*] amqp_client 3.6.14
[e*] cowboy 1.0.4
[e*] cowlib 1.0.2
[ ] rabbitmq_amqp1_0 3.6.14
[ ] rabbitmq_auth_backend_ldap 3.6.14
[ ] rabbitmq_auth_mechanism_ssl 3.6.14
[ ] rabbitmq_consistent_hash_exchange 3.6.14
[ ] rabbitmq_event_exchange 3.6.14
[ ] rabbitmq_federation 3.6.14
[ ] rabbitmq_federation_management 3.6.14
[ ] rabbitmq_jms_topic_exchange 3.6.14
[E*] rabbitmq_management 3.6.14
[e*] rabbitmq_management_agent 3.6.14
[ ] rabbitmq_management_visualiser 3.6.14
[E*] rabbitmq_message_timestamp [Note: no Version Number]
[ ] rabbitmq_mqtt 3.6.14
[ ] rabbitmq_random_exchange 3.6.14
[ ] rabbitmq_recent_history_exchange 3.6.14
[ ] rabbitmq_sharding 3.6.14
[ ] rabbitmq_shovel 3.6.14
[ ] rabbitmq_shovel_management 3.6.14
[ ] rabbitmq_stomp 3.6.14
[ ] rabbitmq_top 3.6.14
[ ] rabbitmq_tracing 3.6.14
[ ] rabbitmq_trust_store 3.6.14
[e*] rabbitmq_web_dispatch 3.6.14
[ ] rabbitmq_web_mqtt 3.6.14
[ ] rabbitmq_web_mqtt_examples 3.6.14
[ ] rabbitmq_web_stomp 3.6.14
[ ] rabbitmq_web_stomp_examples 3.6.14
[ ] sockjs 0.3.4

I don't understand: what is the issue?

TImestamp plugin doesn't seem to apply timestamps to messages arriving via Federated exchange before hitting x-consistent-hash exchange

I have a scenario where I have 2 clusters running in two different DC's. I have a pub-sub exchange federated between both clusters. On one cluster, I have an x-consistent-hash exchange bound to the pub-sub exchange. The x-consistent-hash exchange is load balancing across 3 queues, routing on a hash of the "timestamp" property. If that property isn't set, the x-consistent-hash exchange never delivers the message.

When I publish the message directly onto the pub-sub exchange on local cluster, the timestamp property gets set on the message before it hits the x-consistent-hash exchange and it gets delivered to one of the 3 queues. When I publish the message on the remote cluster, I see that it federates over to my local pub-sub exchange, but then never gets delivered on the x-consistent-hash exchange. When I add a 2nd queue directly to the local pub-sub exchange, it does get delivered.

So the symptoms I am seeing is that for federated messages, it doesn't appear to be adding the timestamp header before it hits the x-consistent hash exchange.

This works:

Publisher --> cluster1 --> pub-sub exchange --> x-consistent-hash (hashing "timestamp" property) --> 1 of 3 target queues

This doesn't work:
Publisher --> cluster 2 --> pub-sub exchange --> federate to Cluster 1 pub-sub-exchange --> x-consistent-hash (hashing "timestamp" property) --> message is lost here, never makes it to target queue

This works (to prove federation to regular queue works without the x-consistent-hash being dependent on the timestamp header):
Publisher --> cluster 2 --> pub-sub exchange --> federate to Cluster 1 pub-sub-exchange --> queue bound to pub sub exchange (see timestamp header on the message).

This also works (to prove federation to x-consistent-hash exchange works that isn't dependent on the timestamp header):
Publisher --> cluster 2 --> pub-sub exchange --> federate to Cluster 1 pub-sub-exchange --> x-consistent-hash (hashing on default "routing key" property) --> 1 of 3 target queues

Support 3.9.0

This plugin doesn't support the current version of rabbitmq-server.

I tried to build a local release but got some sort of permission error, so I would greatly appreciate if you could make a formal release =)

Lost `head_message_timestamp` after restart

Once set head_message_timestamp (by plugin) is lost after rabbitmq restart.

Is this the expected behavior?

Erlang 20.1, Rabbitmq 3.7.0, rabbitmq-message-timestamp plugin 3.7.0

Timestamp as message exits broker

Not sure if this is a piece of functionality that belongs in this plugin or a separate one (or if this is even a bad idea). At a high level I was wondering if it would be possible to also attach the timestamp when the message was consumed from the queue.

For some background, the reason that this would be useful is that as part of a metrics/monitoring solution, it would be good to keep an eye on how long messages are waiting in queue to be processed. This can help help drive consumer scale up/down solutions. Without keeping track of timings at the message queue layer, you risk having inaccurate measurements between producers/consumers due to clock drift if the consumers/producers are on different servers.

Can someone build a release?

I tried to build a release that includes the latest master (which support v3.9), but I keep running into this make error:

ERLC rabbit_auth_backend_ldap.erl rabbit_auth_backend_ldap_app.erl rabbit_auth_backend_ldap_util.erl rabbit_log_ldap.erl
/home/eblouin/rabbitmq/rabbitmq-message-timestamp/.erlang.mk/rabbitmq-server/deps/rabbitmq_auth_backend_ldap/include/logging.hrl:1: can't find include lib "rabbit_common/include/logging.hrl"

elixir 1.12.0-rc.1-otp-23
erlang 23.3.4.5
GNU Make 4.2.1

support 3.12.x

rabbit server 3.12.0 has beed released - could you please update this plugin so it will be supported by 3.12.x server releases?
@lukebakken ?

timestamp_in_ms missing via stomp protocol

Hi.
I've installed v3.7.0 (latest on the community builds) plugin into rabbitmq 3.7.2. I don't see timestamp_in_ms header returned via STOMP, but I am able to see it via the management console, if I inspect a message. I'm also able to see it via AMQP (using php-amqplib).
To rule out stomp client implementation issues, here it is via telnet:

[root@24596575663a app]# telnet
telnet> open mq 61613
Trying 172.24.0.2...
Connected to mq.
Escape character is '^]'.
CONNECT
accept-version:1.2
host:/
login:guest
passcode:guest

^@
CONNECTED
server:RabbitMQ/3.7.2
session:session-NosUdMkAzMWUNqJ6V4ZAoQ
heart-beat:0,0
version:1.2


SUBSCRIBE
id:0
destination:/queue/foo
ack:client

^@
MESSAGE
subscription:0
destination:/queue/foo
message-id:T_0@@session-NosUdMkAzMWUNqJ6V4ZAoQ@@1
redelivered:false
ack:T_0@@session-NosUdMkAzMWUNqJ6V4ZAoQ@@1
timestamp:1515482597
content-type:text/plain
content-length:11

hello world

And here is the same message when viewed via the management console:
message_with_timestamp_in_ms

No curses lib functions found

when running make on my Debian 8 amd64, I get this following error:

configure: error: No curses library functions found
configure: error: /bin/bash '/home/david/otp_src_19.0/erts/configure' failed for erts

any suggestions?

Set timestamp if not already present

Very handy! And of course being a plugin, users can decide whether or not they want Rabbit to set the timestamps.
However, this applies globally - maybe there are cases where some publishers are supplying timestamps that they don't want replaced? E.g.

  • a queue is part of a bridge or routing setup and it's the time the message was enqueued upstream that's significant, not the time it was bridged/routed
  • (similar) a queue is part of a pipeline of activities and the timestamp reflects the start of the overall pipeline

Generally, I'm thinking of contexts where Rabbit is connecting a variety of different apps with somewhat varying practices. This plugin provides a huge benefit in ensuring that timestamps (and timestamp monitoring 😃) will always be available, but if a timestamp is already there there's no reason for it to interfere.

FWIW I think the reason JMS doesn't work like this is that the timestamp is considered internal and typically used for TTL expiry etc. whereas in Rabbit it has no internal role.

Timestamp in nano_seconds

I use timestamp info to ensure message order. But sometimes I have some issues with collision. What about have an option to use os:system_time(nano_seconds) for better precision?

Plugin doesn't support current server version. Actual broker version: "3.8.0"

Hello,
After upgrading Rabbit server to ver 3.8.0 - plugin "rabbitmq_message_timestamp" stop working

sudo rabbitmq-plugins enable rabbitmq_message_timestamp
Enabling plugins on node rabbit@lb-serv:
rabbitmq_message_timestamp
Error:
Failed to enable some plugins:
    rabbitmq_message_timestamp:
        Plugin doesn't support current server version. Actual broker version: "3.8.0", supported by the plugin: ["3.7.0-3.7.x"]

Support 3.10.8

I'm getting error when enabling for 3.10.8

`$ rabbitmq-plugins enable rabbitmq_routing_node_stamp

Enabling plugins on node rabbit@my-server-01:
rabbitmq_routing_node_stamp
Error:
Failed to enable some plugins:
rabbitmq_routing_node_stamp:
Plugin doesn't support current server version. Actual broker version: "3.10.8", supported by the plugin: ["3.10.x-3.10.x"]`

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.