Giter Club home page Giter Club logo

zbroker's Introduction

zbroker - the ZeroMQ broker project

The zbroker project is a container for arbitrary ZeroMQ-based messaging services. The current list of messaging services is:

  • ZPIPES - reliable, distributed named pipes

General Operation

To build zbroker:

git clone git://github.com/jedisct1/libsodium.git
for project in libzmq czmq zyre zbroker; do
    git clone git://github.com/zeromq/$project.git
done
for project in libsodium libzmq czmq zyre zbroker; do
    cd $project
    ./autogen.sh
    ./configure && make check
    sudo make install
    sudo ldconfig
    cd ..
done

To run zbroker:

zbroker [broker-name]

Where 'broker-name' is a string that is unique on any given host. The default broker name is 'local'. To end the broker, send a TERM or INT signal (Ctrl-C).

ZPIPES Overview

The ZPIPES Protocol

The following ABNF grammar defines the ZPIPES protocol:

ZPIPES = reader | writer

reader = input-command *( read-command | ping-command )
         close-command
input-command = c:input ( s:input-ok | s:input-failed )
read-command = c:read ( s:read-ok | s:read-end
                      | s:read-timeout | s:read-failed )
close-command = c:close ( s:close-ok | s:close-failed )

writer = output-command *write-command close-command
output-command = c:output ( s:output-ok | s:output-failed )
write-command = c:write ( s:write-ok
                        | s:write-timeout | s:write-failed )
ping-command = c:ping s:ping-ok

;         Create a new pipe for reading
input           = signature %d1 pipename
signature       = %xAA %xA0             ; two octets
pipename        = string                ; Name of pipe

;         Input request was successful
input_ok        = signature %d2

;         Input request failed
input_failed    = signature %d3 reason
reason          = string                ; Reason for failure

;         Create a new pipe for writing
output          = signature %d4 pipename
pipename        = string                ; Name of pipe

;         Output request was successful
output_ok       = signature %d5

;         Output request failed
output_failed   = signature %d6 reason
reason          = string                ; Reason for failure

;         Read a chunk of data from pipe
read            = signature %d7 size timeout
size            = number-4              ; Number of bytes to read
timeout         = number-4              ; Timeout, msecs, or zero

;         Read was successful
read_ok         = signature %d8 chunk
chunk           = chunk                 ; Chunk of data

;         Pipe is closed, no more data
read_end        = signature %d9

;         Read ended with timeout
read_timeout    = signature %d10

;         Read failed due to error
read_failed     = signature %d11 reason
reason          = string                ; Reason for failure

;         Write chunk of data to pipe
write           = signature %d12 chunk timeout
chunk           = chunk                 ; Chunk of data
timeout         = number-4              ; Timeout, msecs, or zero

;         Write was successful
write_ok        = signature %d13

;         Write ended with timeout
write_timeout   = signature %d14

;         Read failed due to error
write_failed    = signature %d15 reason
reason          = string                ; Reason for failure

;         Close pipe
close           = signature %d16

;         Close was successful
close_ok        = signature %d17

;         Close failed due to error
close_failed    = signature %d18 reason
reason          = string                ; Reason for failure

;         Signal liveness
ping            = signature %d19

;         Respond to ping
ping_ok         = signature %d20

;         Command was invalid at this time
invalid         = signature %d21

; A chunk has 4-octet length + binary contents
chunk           = number-4 *OCTET

; Strings are always length + text contents
string          = number-1 *VCHAR

; Numbers are unsigned integers in network byte order
number-1        = 1OCTET
number-4        = 4OCTET

The ZPIPES API

The zpipes_client class provides the public API.

Ownership and Contributing

The contributors are listed in AUTHORS. This project uses the MPL v2 license, see LICENSE.

The contribution policy is the standard ZeroMQ C4.1 process. Please read this RFC if you have never contributed to a ZeroMQ project.

zbroker's People

Contributors

hintjens avatar rpedde avatar taotetek avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

zbroker's Issues

Problem: zbroker does not use config file

zbroker should use a configuration file to specify the bind endpoint, log level, and log file.

Option for syslog with configurable facility would be a nice-to-have.

Problem: zpipes retains data for disconnected pipes

Named pipe semantics are to flush the pipe contents when both reader and writer disconnect. I think that's probably the expected behavior here. It's very surprising to connect to an "old" pipe and find data in it.

Problem: zbroker library structure too complex

When adding a library for the thin client, we'll get 3 different libraries, which seems too complex.

Solution: classic ZeroMQ-based client & server can go into same libzbroker library. Thin libzmtp based client can go into a separate libzbrokercli client library.

Problem: syslog system call is dangerous

Michael Haberler reports on zeromq-dev,

I notice zlog.c was added to czmq recently
in the machinekit project we used syslog from a cyclic process which at times generates log messages in short succession, but should not block on syslog()
we had all sorts of problems with blocking, funny hangs in futex calls and whatnot with syslog
we got rid of them for good by replacing syslog by syslog_async :
http://thekelleys.org.uk/syslog-async/READ-ME
license is GPL2/GPL3 only
see also: http://lists.sip-router.org/pipermail/users/2008-October/020101.html

We can't use syslog_async, however the protocol is straight-forward so:

Solution: implement our own syslog client using ZMQ_STREAM socket.

Need specification of ipc:// protocol

Problem: there's no formal spec for the ZeroMQ ipc:// protocol

Solution: (1) collect the basic info of the protocol frames used, and then (2) write up as a formal ZeroMQ RFC.

Feature request: configurable logging

It would be nice to be able to change animate logging via configuration rather than compilation.

Honoring SIGHUP for config re-read would be useful as well.

In some kind of far-future super utopian world, a command-line client that talked the management protocol to push a single setting into a running server would be great. :)

Problem: rapid writer opens asserts

This might be related to the other assert, as it's in the same pattern:

open writer on hosta
open corresponding reader on hostb
open new writer on hostb

write to new writer handle, reader asserts:

zpipes_server.c:417: pipe_send_data: Assertion `self->reader' failed.
Aborted

Writer animation:

14-05-02 17:14:23 I: starting zpipes server using config in '/etc/zbroker.cfg'
14-05-02 17:14:23 I: joining cluster as DEC3D1B4C98A5003B10285003C1CD50F
14-05-02 17:14:23 N: starting zpipes_server service
14-05-02 17:14:23 N: binding zpipes service to 'ipc://@/zpipes/local'
14-05-02 17:14:33 I: ZPIPES server appeared at 64ECAC5F2DDC5D91320A6CD4AA3079A4
14-05-02 17:14:42      0: start:
14-05-02 17:14:42      0:     OUTPUT
14-05-02 17:14:42      0:         $ lookup or create pipe
14-05-02 17:14:42      0:         $ open pipe writer
14-05-02 17:14:42      0:         > before writing
14-05-02 17:14:42      0: before writing:
14-05-02 17:14:42      0:     ok
14-05-02 17:14:42      0:         $ send OUTPUT_OK
14-05-02 17:14:42      0:         > writing
14-05-02 17:14:42      1: start:
14-05-02 17:14:42      1:     OUTPUT
14-05-02 17:14:42      1:         $ lookup or create pipe
14-05-02 17:14:42      1:         $ open pipe writer
14-05-02 17:14:42      1:         > before writing
14-05-02 17:14:42      1: before writing:
14-05-02 17:14:42      1:     ok
14-05-02 17:14:42      1:         $ send OUTPUT_OK
14-05-02 17:14:42      1:         > writing
14-05-02 17:14:42      1: writing:
14-05-02 17:14:42      1:     WRITE
14-05-02 17:14:42      1:         $ process write request
14-05-02 17:14:42      1:         > processing write
14-05-02 17:14:42      1: processing write:
14-05-02 17:14:42      1:     have reader
14-05-02 17:14:42      1:         $ pass data to reader
14-05-02 17:14:42      1:         $ send WRITE_OK
14-05-02 17:14:42      1:         > writing
14-05-02 17:14:42      1: writing:
14-05-02 17:14:42      1:     CLOSE
14-05-02 17:14:42      1:         $ close pipe writer
14-05-02 17:14:42      1:         $ send CLOSE_OK
14-05-02 17:14:42      1:         > start
14-05-02 17:14:52      0: writing:
14-05-02 17:14:52      0:     expired
14-05-02 17:14:52      0:         $ close pipe writer
14-05-02 17:14:52      0:         $ terminate
14-05-02 17:14:52      1: start:
14-05-02 17:14:52      1:     expired
14-05-02 17:14:52      1:         $ terminate
14-05-02 17:14:52 I: ZPIPES server vanished from 64ECAC5F2DDC5D91320A6CD4AA3079A4

Reader animation:

14-05-02 17:14:33 I: starting zpipes server using config in 'zbroker.cfg'
14-05-02 17:14:33 I: joining cluster as 64ECAC5F2DDC5D91320A6CD4AA3079A4
14-05-02 17:14:33 N: starting zpipes_server service
14-05-02 17:14:33 N: binding zpipes service to 'ipc://@/zpipes/local'
14-05-02 17:14:33 I: ZPIPES server appeared at DEC3D1B4C98A5003B10285003C1CD50F
14-05-02 17:14:42    643: start:
14-05-02 17:14:42    643:     INPUT
14-05-02 17:14:42    643:         $ lookup or create pipe
14-05-02 17:14:42    643:         $ open pipe reader
14-05-02 17:14:42    643:         > before reading
14-05-02 17:14:42    643: before reading:
14-05-02 17:14:42    643:     ok
14-05-02 17:14:42    643:         $ send INPUT_OK
14-05-02 17:14:42    643:         > reading
lt-zbroker: zpipes_server.c:417: pipe_send_data: Assertion `self->reader' failed.
Aborted

Strangely, the pipe that was written to didn't have a reader. I'm thinking some wires got crossed here. The writer should have blocked on no reader, but it sent data.

Let me know if there is more repro data I can provide.

Problem: N-to-N pipe semantics undefined

Solution: define these semantics.

One possibility is a loosely-connected pub-sub model that allows N publishers talk to N subscribers, using the pipe as sync point. Thus:

  • A writer would block until there is at least one reader.
  • Each reader to a specific pipe would get all messages written to it.
  • A reader would get all data from a specific writer, or none of it. That is, a reader that missed message #1 from a writer would discard or skip all later messages.

Problem: no codecs for thin client (libzmtp)

Solution: port the zproto codec to generate code for libzmtp, and port the client API to this codec.

For now, this can be in a directory 'clients' in zbroker; it'll build a separate library.

Problem: zpipes_client_new asserts on rapid open

Trying to repo another problem I ran into this one:

in quick succession:
open writer on hosta
open reader on hostb
open writer on hosta

second writer open asserts:

zpipes_client.c:76: zpipes_client_new: Assertion `0' failed.
Aborted

Animate data:

14-05-02 17:07:01    420: start:
14-05-02 17:07:01    420:     OUTPUT
14-05-02 17:07:01    420:         $ lookup or create pipe
14-05-02 17:07:01    420:         $ open pipe writer
14-05-02 17:07:01    420:         > before writing
14-05-02 17:07:01    420: before writing:
14-05-02 17:07:01    420:     ok
14-05-02 17:07:01    420:         $ send OUTPUT_OK
14-05-02 17:07:01    420:         > writing
14-05-02 17:07:01    421: start:
14-05-02 17:07:01    421:     INPUT
14-05-02 17:07:01    421:         $ lookup or create pipe
14-05-02 17:07:01    421:         $ open pipe reader
14-05-02 17:07:01    421:         > before reading
14-05-02 17:07:01    421: before reading:
14-05-02 17:07:01    421:     error
14-05-02 17:07:01    421:         $ send INPUT_FAILED
14-05-02 17:07:01    421:         $ terminate
14-05-02 17:07:11    420: writing:
14-05-02 17:07:11    420:     expired
14-05-02 17:07:11    420:         $ close pipe writer
14-05-02 17:07:11    420:         $ terminate

I suspect this comment is wrong:

    {
        zpipes_msg_send_input (self->dealer, pipe_name);
        if (s_expect_reply (self, ZPIPES_MSG_INPUT_OK))
            assert (false);     //  Cannot happen in current use case                                                                                                                                                                                          
    }

:)

Problem: sometimes reader and writer pipes don't connect

Repro:

  1. open writer pipe
  2. sleep(1)
  3. open reader pipe
  4. write to writer pipe

In some cases, the reader and writer don't connect, and the writer blocks indefinitely.

writer animation:

zbroker service/0.0.1
Copyright (c) 2014 the Contributors
This Software is provided under the MPLv2 License on an "as is" basis,
without warranty of any kind, either expressed, implied, or statutory.

14-05-05 12:46:37 I: starting zpipes server using config in 'zbroker-2.cfg'
14-05-05 12:46:37 I: joining cluster as 00D325
14-05-05 12:46:37 N: starting zpipes_server service
14-05-05 12:46:37 N: binding zpipes service to 'ipc://@/zpipes/local2
14-05-05 12:46:37 I: ZPIPES server appeared at 5AB1DE
14-05-05 12:46:37 I: ZPIPES server vanished from 5AB1DE
14-05-05 12:46:37 I: ZPIPES server appeared at 5AB1DE
14-05-05 12:46:38 write_test: attach local writer
14-05-05 12:46:38 write_test: broadcast we are now writer
14-05-05 12:46:38 D:    289:write_test          : open local writer
14-05-05 12:46:39 D:    289:write_test          : close local writer
interrupted
14-05-05 12:46:39 N: terminating zpipes_server service

reader animation:

zbroker service/0.0.1
Copyright (c) 2014 the Contributors
This Software is provided under the MPLv2 License on an "as is" basis,
without warranty of any kind, either expressed, implied, or statutory.

14-05-05 12:46:37 I: starting zpipes server using config in 'zbroker-1.cfg'
14-05-05 12:46:37 I: joining cluster as 5AB1DE
14-05-05 12:46:37 N: starting zpipes_server service
14-05-05 12:46:37 N: binding zpipes service to 'ipc://@/zpipes/local'
14-05-05 12:46:37 I: ZPIPES server appeared at 002BAA
14-05-05 12:46:37 I: ZPIPES server appeared at 002BAA
14-05-05 12:46:37 write_test: attach local reader
14-05-05 12:46:37 write_test: broadcast we are now reader
14-05-05 12:46:37 D:    289:write_test          : open local reader
14-05-05 12:46:38 D: remote=002BAA command=HAVE WRITER pipe=write_test unicast=0
14-05-05 12:46:38 write_test: attach remote writer
14-05-05 12:46:38 write_test: tell peer we are now reader
14-05-05 12:46:39 write_test: tell peer we stopped being reader
14-05-05 12:46:39 D:    289:write_test          : close local reader
interrupted
14-05-05 12:46:39 N: terminating zpipes_server service

The writer was stuck in zpipes_client_write

I find this difficult to reproduce on a single node with two brokers, but it evidences itself frequently on true multi-node config.

Problem: lack full cluster test case

Solution: simulate N cluster nodes doing some work

Perhaps we could use simulation to also measure different kinds of activity, throughput, latency, etc.

Problem: zyre doesn't have a stable release

Currently, zbroker is version tied to zeromq 4.0.4 and czmq 2.2.0. It would be nice to tie to a tagged version of zyre as well.

Solution: tag Zyre, double-check that ZRE is stable.

Problem: no detection for unexpected disconnections

Symptom: server does not get a valid Close command and so does not clean up the pipe.

Solution: configurable ping-pong heartbeating. Eventually, this should be done at the ZMTP (libzmq / libzmtp) level.

Problem: pipelining broken

Symptom: as the "expecting chunk" state deals only with internal events, "fetch" commands are held in a queue until the FSM hits the "reading" state. Adding "close" to the state switches that off (in the current implementation), so pipelined "fetch" commands are held until they can be safely processed.

Solution: don't mix protocol command events and internal events. We could perhaps filter commands, e.g. queue fetch and then accept close. Not sure of the semantics of this, though.

Better solution: document the state machine generator properly to explain this. Second, deal with EINTR differently.

reader/writer issue

Not sure how this goes wrong, but...

open reader and writer
write small amount of data
attempt to read a large amount of data (blocks, too small)
close write handle

at this point, expected is that reader returns the data in buffer, and subsequent read returns 0. instead, the reader gets dropped, the read request blocks, and the pipe doesn't terminate.

I'll post traces in a bit

Problem: client API uses chunks, not a stream

One symptom is that data will be discarded if a client doesn't provide the right size of buffer.

Solution: expose a classic stream API, and map internally to chunks both on sending and receiving.

Problem: no log details of read/write requests

This makes it hard to know what the applications are doing.

Solution:

  • Log remote server in reader/writer connect messages
  • Log remote server in reads/writes as well
  • Log read/write chunk size, timeouts

Feature request: detailed logging

Animate logging would be sufficient for debugging if it added some extra details:

  • all logs should include the pipe name they are working on (better transaction greppability)
  • at a minimum, reader/writer connect messages should contain the remote server that connected, but reads and writes could (should?) show the remote id as well.
  • block size and read/write request size might be helpful as well

zpipe reads block on closed writer

Right now, zpipes implements an infinite pipe. But in order to implement true named pipe semantics over zpipes, it is necessary that the client can detect remote writer close and return a zero byte read.

Problem: zpipes_client_read returns ambiguous results

in zpipes_client.c, zpipes_client_read returns 0 ambiguously.

zpipes_msg_recv returns NULL on both malloc error and on EINTR (blowing the entire message frame, it looks like). Returning a 0 here looks like a graceful pipe shutdown, and should return an error (-1 with errno set to EIO? It's probably not retryable, so EINTR is probably out.)

Same with a timeout condition -- this should return an error as well, so as to distinguish itself from a graceful pipe close.

Problem: client potentially loses data

If the buffer provided by a client is smaller than the size of the pending data chunk, the excess data in the chunk is thrown away.

The client read function really needs to keep an offset of the position of the current chunk, and continue to serve from it until the chunk is exhausted.

In a perfect world, we'd never return short reads, either. Reads shorter then requested size happen, but are surprising unless at end of file, I think. So a single read in excess of the pending chunk size should probably continue to pull chunks until the client supplied buffer is full.

This might be something better deferred to after the raw client re-write, though. Just noting this for posterity.

Problem: segv on multi-broker

Reliable segv:

  1. open write pipe on one broker
  2. open corresponding read pipe on another broker
  3. segv
14-05-01 18:02:48 I: joining cluster as DD561851F4DAA8B14004F9CF9F3C0CB5
14-05-01 18:02:48 N: starting zpipes_server service
14-05-01 18:02:48 N: binding zpipes service to 'ipc://@/zpipes/local'
14-05-01 18:02:48 I: ZPIPES server appeared at 616A81862091A61E1A61C70EC4C77DC9
14-05-01 18:02:53    825: start:
14-05-01 18:02:53    825:     INPUT
14-05-01 18:02:53    825:         $ lookup or create pipe
14-05-01 18:02:53    825:         $ open pipe reader

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff55b3700 (LWP 21517)]
0x00007ffff7bc5dcc in s_client_execute (self=0xffffffffffffffff, event=10)
    at zpipes_server_engine.h:521
521         self->next_event = event;
(gdb) bt
#0  0x00007ffff7bc5dcc in s_client_execute (self=0xffffffffffffffff, event=10)
    at zpipes_server_engine.h:521
#1  0x00007ffff7bc532c in engine_send_event (client=0xffffffffffffffff, 
    event=have_reader_event) at zpipes_server_engine.h:340
#2  0x00007ffff7bcb353 in pipe_attach_local_reader (self=0x7fffec001510, 
    reader=0x7fffec001890) at zpipes_server.c:220
#3  0x00007ffff7bcc44a in open_pipe_reader (self=0x7fffec001890) at zpipes_server.c:481
#4  0x00007ffff7bc60ff in s_client_execute (self=0x7fffec001890, event=2)
    at zpipes_server_engine.h:562
#5  0x00007ffff7bca95e in s_server_client_message (loop=0x60fcd0, item=0x61f3b0, 
    argument=0x60ecd0) at zpipes_server_engine.h:1447
#6  0x00007ffff776dc6c in zloop_start (self=0x60fcd0) at zloop.c:463
#7  0x00007ffff7bcaac7 in s_server_task (args=0x0, ctx=0x608ec0, pipe=0x6091f0)
    at zpipes_server_engine.h:1465
#8  0x00007ffff777915f in s_thread_shim (args=0x608e90) at zthread.c:81
#9  0x00007ffff6ce1b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#10 0x00007ffff72550ed in clone () from /lib/x86_64-linux-gnu/libc.so.6
#11 0x0000000000000000 in ?? ()
(gdb) 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.