jik0 / rabbitmq-federation Goto Github PK

Purpose
=======

The high level goal is to transmit messages between brokers without
requiring clustering. This is useful for the following reasons:

1) Federated brokers may be in different administrative
   domains. Clustered brokers form a single administrative domain.

   a) Federated brokers may have different users and virtual hosts.
      Federated brokers only need to partially trust each other.

   b) Federated brokers may run different versions of RabbitMQ and
      Erlang.

2) Federated brokers only speak AMQP to each other, and the federation
   mechanism is designed to deal with intermittent connectivity.
   Federation is therefore much more WAN-friendly.

3) Brokers can contain federated and local-only components - you don't
   need to federate everything if you don't want to.

4) Ultimately, greater scalability should be possible, since more complex
   routing topologies avoid the need for n^2 connections between n
   brokers.

For the time being, federation is primarily useful in pub/sub scenarios.


Limitations
===========

You can't federate headers exchanges. You can't make a broker federate
with a new upstream without restarting it. There's no status
reporting. These will likely get fixed.


HOWTO
=====

The plugin provides a federated exchange type. A federated exchange
has type 'x-federation'. However, this type does not provide any
routing logic. The routing logic is implemented by a backing type,
provided to the exchange as an argument.

Messages can be published to a federated exchange like any
other. However, a federated exchange also receives messages from
one or more "upstream" exchanges, located on remote brokers.

(Well, you don't need to have upstream exchanges, and they don't need
to be remote. But then you don't get anything very useful.)

An upstream exchange can be a regular exchange or a federation
exchange. It is expected that upstream and downstream exchanges have
the same type (or backing type). Mixing types will lead to strange
routing behaviour.

Inter-broker communication is implemented using AMQP (optionally
secured with SSL). Bindings are grouped together and bind / unbind
commands are sent to the upstream exchange. Therefore the federation
plugin only receives messages over the wire for which the downstream
exchange has a subscription. To promote scalability the bindings are
sent upstream asynchronously - so the effect of adding or removing a
binding is only guaranteed to be seen eventually.

A typical use would be to have the same "logical" exchange distributed
over many brokers. This would be achieved by having the exchange
declared as a federated exchange in each broker, with upstreams
corresponding to all the other brokers.

Another would be massive fanout - a single "root" exchange in one
broker can be treated as an upstream by many other brokers. In turn
each of these can be the upstream for many more downstreams, and so
on.

Other topologies are of course possible.

Federated exchanges can be set up statically via broker configuration,
or declared dynamically over AMQP.


Static Configuration
====================

A verbose static configuration might look like:

  {rabbitmq_federation,
   [ {exchanges, [[{exchange,     "my-exchange"},
                   {virtual_host, "/"},
                   {type,         "topic"},
                   {durable,      true},
                   {auto_delete,  false},
                   {internal,     false},
                   {upstream_set, "my-upstreams"}]
                 ]},
     {upstream_sets, [{"my-upstreams", [[{connection, "upstream-server"},
                                         {exchange,   "my-upstream-x"},
                                         {max_hops,   2}],
                                        [{connection, "another-server"}]
                                       ]}
                     ]},
     {connections, [{"upstream-server", [{host,            "upstream-server"},
                                         {protocol,        "amqps"},
                                         {port,            5671},
                                         {virtual_host,    "/"},
                                         {username,        "myusername"},
                                         {password,        "secret"},
                                         {mechanism,       default},
                                         {prefetch_count,  1000},
                                         {reconnect_delay, 5},
                                         {heartbeat,       1},
                                         {queue_expires,   30000},
                                         {message_ttl,     10000},
                                         {ssl_options,
                                          [{cacertfile, "/path/to/cacert.pem"},
                                           {certfile,   "/path/to/cert.pem"},
                                           {keyfile,    "/path/to/key.pem"},
                                           {verify,     verify_peer},
                                           {fail_if_no_peer_cert, true}
                                          ]}
                                        ]},
                    {"another-server", [...elided...]}
                   ]},
     {local_username, "myusername"},
     {local_nodename, "my-server"}
   ]
  }

The list of exchanges looks like a set of exchange.declares for the
most part, but with each declaration including the name of an
"upstream_set", representing a list of remote exchanges whose messages
should be federated to the local exchange. Note that the type
parameter must match the type of the upstream exchanges for routing to
work at all sensibly.

Each element in the upstream_set contains a mapping from a name to a
list of upstreams. Each element in this list can contain the following
properties:

connection
  - name of a connection from the connection list. Mandatory.

exchange
  - name for exchange to connect to. Default is to use the same name
    as the downstream exchange. (Therefore one upstream_set can be
    refered to by many local exchanges as long as exchange names are
    the same across all the nodes in your federation.)

max_hops
  - the maximum number of times a message received over this link may
    have been forwarded (including traversing this link). The default
    for max_hops is 1. This prevents messages from looping forever
    when there are circular topologies, and can reduce or prevent
    message duplication.

Note that the static configuration will declare the downstream
exchanges (on the local broker). It does not ensure the upstream
exchanges exist.

The connections list provides information on how to connect to brokers
mentioned in the upstream sets. It can contain the following properties:

host
  - hostname to connect to

protocol
  - "amqp" or "amqps". Default is "amqp".

port
  - port to connect to. Default is 5672, or 5671 when using SSL.

virtual_host
  - virtual host to connect to. Default is to use the virtual host for
    the downstream exchange.

username
  - user to connect as. Default is "guest". The user will need the
    ability to create exchanges and queues with names beginning with
    "federation:". It will also need to be able to bind to the upstream
    exchange.

password
  - password to use. Default is "guest".

mechanism
  - SASL mechanism to use. One of:
      'default'  - to use PLAIN or AMQPLAIN by negotiation (this is the default)
      'EXTERNAL' - to use SASL EXTERNAL authentication - e.g.
                   rabbitmq-auth-mechanism-ssl

prefetch_count
  - limit on the maximum number of unacknowledged messages in flight
    per link. Default is 'none'.

reconnect_delay
  - time in seconds to wait to reconnect to the broker after being
    disconnected. Default is 1.

heartbeat
  - AMQP heartbeat interval (in seconds) on the connection, or none. Default
    is 'none'.

expires
  - Messages are buffered in the upstream broker in a queue before being
    sent downstream. This setting controls how long the upstream queue
    lasts before being deleted if the downstream is disconnected (i.e.
    it controls the "x-expires" argument for the upstream queue). The
    value is in milliseconds. Default is 'none', meaning the queue should
    never expire.

message_ttl
  - Controls how long to keep upstream messages buffered, in milliseconds
    (i.e. the "x-message-ttl" argument for the upstream queue). Default is
    'none', meaning messages should never expire.

ssl_options
  - Specifies how to make client SSL connections. See the Erlang
    client documentation for more details.

The local_username parameter specifies how to connect to the local
broker. The default is "guest". This user will need the ability to
publish messages to the downstream exchange.

The local_nodename parameter specifies the name this node should use
to identify itself in the federation. If not specified it will attempt
to build a "long form" version of the usual Erlang node name (but
using the machine's FQDN). You should only have to specify this if
your DNS will not give each machine in the federation unique names.

Declaring Federation Exchanges Over AMQP
========================================

This is a less common case, but in case you want to do this:

 * Declare the downstream exchange with type "x-federation".

 * Give it arguments "type" and "upstream-set", both of type "long
   string". "type" should refer to its backing type; the type of the
   upstream exchanges. "upstream-set" should be the name of an
   upstream_set from the static configuration (note that it's a hyphen
   over AMQP but an underscore in the configuration).

Example using the Java API:

Map<String, Object> args = new HashMap<String, Object>();
args.put("type", "topic");
args.put("upstream-set", "my-upstream-set");

Channel ch = ...;
ch.exchangeDeclare("my-federated-exchange", "x-federation", true, false, args);

Note On Clustering
==================

When using federation with clustering, all cluster nodes must have the
federation plugin installed. Any node may establish connections to
upstream exchanges. If a node fails, any upstream links will fail over
to a surviving node.

Possible Future Enhancements
============================

* Simple status reporting, similar to the shovel

* Modify upstream sets dynamically

* Mode to federate all exchanges

* Status information and control in the management plugin

* Smarter routing logic

* Support for RPC-like patterns