originally: https://bitbucket.org/gdamore/mangos/issue/17

While doing some work on another project, I discovered that rep saves the message Header, instead of making a copy of it. If the message is recycled, this can result in data corruption. We must be careful to make a copy of any data we need to keep; the framework will free the inbound message once the SendHook is done, and that can lead to use-after-free bugs.

Fixed in cset 622c0f9

Apparently the changes to use time.Duration instead of time.Time broke the compat module. This needs to be fixed.

Originally here: https://bitbucket.org/gdamore/mangos/issue/8

Fixed in cset 8a1a0c3

Websocket transport following the nanomsg RFC.

Ryan found a deadlock in his testing -- turns out that we can hang hard waiting for Close() to complete.

Tests are failing inconsistently, I've run go test github.com/gdamore/mangos/test using go version go1.4 linux/amd64 10 times in a row, 8 of them failed. During the last run process stuck apparently, so I killed it with ABRT to get goroutine trace. Please see result of all runs here: https://gist.github.com/artyom/e613dbaad04bfcf9034f

Package version tested:

root@sandbox:mangos # git describe --always
d591c81

At present there is no way to detect or be notified if a device broke, and there is no way to terminate a device.

nanomsg does it through nn_device/nn_term.

The README.md has all kinds of cautionary notes. At this point people are using this in real production scenarios, and our comments kind of scare people away. Its time to remove them.

Originally here: https://bitbucket.org/gdamore/mangos/issue/6

Fixed in cset a82e7d1

So, we've seen another failure on the bus test -- only on Linux and with GOMAXPROCS = 18 and with -race. I've convinced myself that this is an edge case that can occur if we burst too many messages to the broadcast channel before the reader (another go routine) can pick them up. Normally we shouldn't queue these up so fast, but 5 is clearly an arbitrarily chosen value and is too small -- the test itself bursts messages 7 at a time.

A better solution is to make this the same depth as Tyler's new WriteQLen value. This sadly does mean that we wind up possibly queueing up 2x the requested depth for bus protocol, but it avoids dispatcher-latency induced drops. (They can still happen, but they are probably extraodinarily unlikely -- you'd have to have partial drain on one side, while continuously bursting on the send side.)

Notably, surveyor is "safe" from this since there should never be more than a single survey outstanding. (It uses a fixed queue of 1 instead of 5.)

Originally: https://bitbucket.org/gdamore/mangos/issue/14

The pipe selection for some protocols may spend too much time cycling through all possible pipes. We should examine this further, and see if we can reduce or eliminate unnecessary list churn. Perhaps we should prefer to continue to a single pipe until back pressure is exerted. (Right now we round robin every message.)

However:

The new code works quite differently now. Protocols can decide how to select pipes. Furthermore, we're seeing very good perf. already, and the architecture is to let Go's channel scheduler handle this. But, we can revisit later if we find cause to once we have experience with sockets with thousands of connected peers. For now, closing WONTFIX.

There appears to be an inconsistency between the docstring for OptionSurveyTime and the actual behavior of surveyor sockets. The docstring says, if not specified, the default surveyor deadline is 1 second. However, it looks like the default is actually 0, meaning the surveyor waits infinitely.

Originally filed here:

https://bitbucket.org/gdamore/mangos/issue/1/add-support-for-pub-sub

Originally: https://bitbucket.org/gdamore/mangos/issue/15

Fixed in cset 66590d4

I noticed today that in one case BUS errors can wind up with a use after free. This only happens if we arrange to hit a flow control condition in the bus protocol. Effectively, we wind up freeing the wrong message.

Originally: https://bitbucket.org/gdamore/mangos/issue/13

We need the ability to use priorities ala SNDPRIO, RCVPRIO.

However:

Its actually unclear whether this is useful. Furthermore, the architecture involving the use of multiple goroutines means that this is impractical without adding a new set of channels and additional latency.
Closing WONTFIX for now. Revisit when/if we ever get a business case.

Originally here: https://bitbucket.org/gdamore/mangos/issue/9

Fixed in cset 8a1a0c3

The whole rendezvous logic in the inproc implementation is far too byzantine, making use of go channels and go routines to distraction. Ultimately, there was a deadlock there, I'm pretty sure. This can be greatly simplified.

Originally: https://bitbucket.org/gdamore/mangos/issue/10

Performance numbers show that buffer allocation is hurting us. Allow the application to preallocate the message buffer for receive. (For send this isn't necessary.)

Fixed a while ago (before I was tracking these more closely).

Hi Guys, Another noobs here. I want to ask something.

1. Suppose i got 2 worker behind a device, worker 'a' and 'b'. The requester want to send msg to worker 'a'. If i do this on micro controller, i send the msg to all worker, but only worker that being specified from the sender respond it, the other just throw the msg. If i do this using zmq, i got some socket id option so that i send it to that ID. When i use nanomsg, i'm thinking about the bus socket but how i can do it?
2. Can i see what the message that going through or even stuck in the socket? Or using some proxy device/monitoring device/forwarder with pub socket?

I've looked through the docs and the Surveyor source. I've noticed that we're dropping any messages that exceed the timeout. Which looks a little passive for my tastes.

• We receive a message from a responder
• If the time taken is greater than the timeout
• We drop the message.

I'm assuming the implementation would have to change to have a more active timeout.
IE: Have the surveyor to pass a message when a responder has timed out.

Am I wrong in my assessment? Was there something I missed?

Originally: https://bitbucket.org/gdamore/mangos/issue/2

The IPC transport only works with Unix domain sockets. While Go has no native support for Named Sockets, the libnanomsg API does. We could probably come up with something for Windows, if there was a demand.
Note that Windows users can always use the TCP transport.

Is it possible for a respondent to connect to more than one surveyor socket? E.g.

respondent, _ := respondent.NewSocket()
respondent.AddTransport(tcp.NewTransport())
respondent.Dial("tcp://127.0.0.1:5000")
respondent.Dial("tcp://127.0.0.1:5001")

The current behavior seems to be undefined. With the above code, surveys are always received from either the first or second but not both. If surveys can't be received from multiple sockets, I'd expect the behavior to be consistent when trying to connect to multiple sockets. Either:

1. Return error when trying to connect when already connected to surveyor
2. Receive surveys from the first-made connection
3. Receive surveys from the last-made connection

Thoughts?

We see, when running under -race, test failures from TestBusTLS. A lot of the history is in #41 but also you can see this:

=== RUN TestBusTLS
--- FAIL: TestBusTLS (0.87s)
    common_test.go:503: Address tls+tcp://127.0.0.1:43934, 5 Cases
    common_test.go:149: Jan 16 17:39:01.787: Id 0: Got dup START from 1
    common_test.go:149: Jan 16 17:39:01.875: Id 0: Got dup START from 2
    common_test.go:149: Jan 16 17:39:01.875: Id 0: Got dup START from 1
    common_test.go:149: Jan 16 17:39:01.881: Id 0: Sent all 7 messages
    common_test.go:149: Jan 16 17:39:01.883: Id 1: Got all 7 messages
    common_test.go:149: Jan 16 17:39:01.883: Id 4: Got all 7 messages
    common_test.go:149: Jan 16 17:39:01.883: Id 3: Got all 7 messages
    common_test.go:149: Jan 16 17:39:01.883: Id 2: Got all 7 messages
    common_test.go:149: Jan 16 17:39:01.883: Id 3: Sent all 7 messages
    common_test.go:154: Jan 16 17:39:01.884: Id 0: Got dup message #6 from 3
    common_test.go:154: Jan 16 17:39:01.884: Id 0: RecvHook failed
    common_test.go:149: Jan 16 17:39:01.884: Id 4: Sent all 7 messages
    common_test.go:149: Jan 16 17:39:01.884: Id 0: Testing complete
    common_test.go:149: Jan 16 17:39:01.885: Id 1: Sent all 7 messages
    common_test.go:149: Jan 16 17:39:01.885: Id 1: Testing complete
    common_test.go:149: Jan 16 17:39:01.886: Id 2: Sent all 7 messages
    common_test.go:149: Jan 16 17:39:01.887: Id 2: Testing complete
    common_test.go:149: Jan 16 17:39:01.887: Id 3: Testing complete
    common_test.go:149: Jan 16 17:39:01.887: Id 4: Testing complete
FAIL
exit status 1
FAIL    github.com/gdamore/mangos/test  2.625s

The errors seem to be limited to TestBusTLS; none of the other protocols or transport combinations seem to matter. (Note that StarTest does not use TLS.)

Originally: https://bitbucket.org/gdamore/mangos/issue/12

We'd like to have the ability to pass additional metadata. Specifically, a transport could provide details about the pipe that the message was received on. Imagine that with TLS we could provide the information from the certificate. Or the peer address. Or with IPC knowing the credentials correspond to a root socket -- which would give a good validation that a system service wasn't being spoofed -- would be incredibly useful.

Originally: https://bitbucket.org/gdamore/mangos/issue/20

SSL3.0 is insecure. See https://www.openssl.org/~bodo/ssl-poodle.pdf We need to disable it.

Fixed in cset c4b8bf0

While running tests I was still getting intermittent duplicate frees. I modified
message.Free() to save the stack trace of the goroutine that executed it. That
way when a duplicate free is detected I can print the stack of the previous free
call. Sure enough there is another dup free during error what was not fixed in
271babf. I have a PR incoming.

diff --git a/message.go b/message.go
index 2ae59a5..49352de 100644
--- a/message.go
+++ b/message.go
@@ -15,6 +15,8 @@
 package mangos

 import (
+   "fmt"
+   "runtime"
    "sync/atomic"
 )

@@ -30,6 +32,7 @@ type Message struct {
    hbuf   []byte
    bsize  int
    refcnt int32
+   LastStack []byte
 }

 type msgCacheInfo struct {
@@ -56,6 +59,10 @@ func (m *Message) Free() {
    if v := atomic.AddInt32(&m.refcnt, -1); v > 0 {
        return
    }
+   if li := atomic.LoadInt32(&m.refcnt); li < 0 {
+       panic(fmt.Sprintf("msg refcnt (%d) should never be < 0, double free occurred\n===BEGIN STACK:\n%s\n===END STACK\n", li, m.LastStack));
+   }
+   runtime.Stack(m.LastStack, false)
    for i := range messageCache {
        if m.bsize == messageCache[i].maxbody {
            ch = messageCache[i].cache
@@ -99,6 +106,7 @@ func NewMessage(sz int) *Message {
        m.bbuf = make([]byte, 0, sz)
        m.hbuf = make([]byte, 0, 32)
        m.bsize = sz
+       m.LastStack = make([]byte, 4096)
    }

m.refcnt = 1

And the output:

rpz@ubuntu:~/gowrk/src/github.com/gdamore/mangos/test$ ~/repeat.sh 10 device_test.go common_test.go tls_test.go
error SendMsg: use of closed network connection
panic: msg refcnt (-1) should never be < 0, double free occurred
===BEGIN STACK:
goroutine 74 [running]:
github.com/gdamore/mangos.(*Message).Free(0xc2080fe090)
    /home/rpz/gowrk/src/github.com/gdamore/mangos/message.go:65 +0x2bb
github.com/gdamore/mangos.(*connipc).Send(0xc20810a000, 0xc2080fe090, 0x7f04f5a0e8c0, 0xc20800a450)
    /home/rpz/gowrk/src/github.com/gdamore/mangos/conn.go:167 +0x236
github.com/gdamore/mangos.(*pipe).SendMsg(0xc20813e100, 0xc2080fe090, 0x0, 0x0)
    /home/rpz/gowrk/src/github.com/gdamore/mangos/pipe.go:91 +0x7c
github.com/gdamore/mangos/protocol/pair.(*pair).sender(0xc2081420c0, 0x7f04f5a1add0, 0xc20813e100)
    /home/rpz/gowrk/src/github.com/gdamore/mangos/protocol/pair/pair.go:47 +0x15c
created by github.com/gdamore/mangos/protocol/pair.(*pair).AddEndpoint
    /home/rpz/gowrk/src/github.com/gdamore/mangos/protocol/pair/pair.go:79 +0x13b

===END STACK


goroutine 74 [running]:
github.com/gdamore/mangos.(*Message).Free(0xc2080fe090)
    /home/rpz/gowrk/src/github.com/gdamore/mangos/message.go:63 +0x263
github.com/gdamore/mangos/protocol/pair.(*pair).sender(0xc2081420c0, 0x7f04f5a1add0, 0xc20813e100)
    /home/rpz/gowrk/src/github.com/gdamore/mangos/protocol/pair/pair.go:49 +0x27c
created by github.com/gdamore/mangos/protocol/pair.(*pair).AddEndpoint
    /home/rpz/gowrk/src/github.com/gdamore/mangos/protocol/pair/pair.go:79 +0x13b

Originally: https://bitbucket.org/gdamore/mangos/issue/19

John Goodall created an issue 2014-08-30
I want to use this library for a project, but having trouble getting an overview without having any examples to look at. On the 'server' side, is the general flow?
Create a new transport
Use the transport to create a new listener (NewAcceptor)
Use the listener to create a server (Accept)
Then how do I hook the SP to the Listener?
A working example would be a great help. Once there is one simple example, I can help add additional ones (maybe take the C examples from Getting Started with nanomsg and implementing in go.

Fixed in cset f2844c3

Originally here: https://bitbucket.org/gdamore/mangos/issue/7

I'd like to add support for TLS, using an experimental syntax. ("tis://") to begin with?
While there are some configuration directives specific to TLS and our implementation, I'd like to assume that the semantics are otherwise identical to TCP.

Fixed in: cset 0dcfb19

I wanted to start the discussion on how we can handle cycles with the STAR protocol. I find this protocol extremely useful, but I think it's unreasonable to require systems to avoid cycles.

It seems the simplest solution is to just allow cycles and add a TTL and, optionally, an ID on messages. This way, messages won't propagate infinitely. Applications which need to ensure idempotency can make use of an ID, whether its part of the protocol or the message body itself. Obviously the downside is there's additional overhead with adding more onto the message.

It looks like GC is minimized by using a message cache so messages are reused (https://github.com/gdamore/mangos/blob/master/message.go#L41). This is currently implemented using channels to provide a buffer of disposed messages. My experience has been that channels are generally slowish. I'm curious if a decision was consciously made to not use a sync.Pool? I'd be interested to see benchmarks for both.

Originally here: https://bitbucket.org/gdamore/mangos/issue/5

Add support for the BUS protocol. This is just a broadcast blaster.

Fixed in: cset 7991f4b

Originally here: https://bitbucket.org/gdamore/mangos/issue/3

The timeouts and deadlines are hard coded. Also retransmit times. We should make those available via socket options.

Fixed in cset 7c35c9d

It may be faster/better to use sync.Pool for Go message pooling. In the past we found it was slightly less efficient than our hand coded version using channels, but Go 1.4 may have improved here. We should investigate again.

The package and doc URLs currently refer to Bitbucket, they should probably point to Github now.

Hi - noob here. I guess I fail to understand the paradigm and dont see it in the docs - are we locked into the pattern we choose or can we mix and match features? Ie pair with broadcast?

Does pair only work with pair? And pub only with sub? or can I mix Sub to pair? Its unclear to muh noobness what the paradigm is here.

I'm trying to do a pair dialer client and pair server. Do I have to free_msg like in nano msg to get the next message? I dont see that in the docs... Im not getting the Second msg to my "server" socket - and writes are not going out either. Do I have to respond before I can get the 2nd msg?

Also +1 for an example of a server / client that writes arbitrary msgs instead of one write and exit; the extra go boiler plate for wrapping the read/writes to chans/funcs is a bit confusing.

Thanks

In sending messages through the publisher, bus, and star protocols at least, we have a queue depth for each of the fanout targets. It would be nice to have this be tunable. Right now its set to 5 as a hard coded limit.

Send/Recv deadline is an absolute time.Time:

    // OptionRecvDeadline is the absolute time when the next Recv should
    // timeout.  The value is a time.Time.  Zero value may be passed to
    // indicate that no timeout should be applied.  By default there is
    // no timeout is used.
    OptionRecvDeadline = "RECV-DEADLINE"

    // OptionSendDeadline is the absolute time when the next Send should
    // timeout.  The value is a time.Time.  Zero value may be passed to
    // indicate that no timeout should be applied.  By default there is
    // no timeout.
    OptionSendDeadline = "SEND-DEADLINE"

This is kind of annoying. To me, it would make more sense to take a time.Duration, e.g. socket.SetOption(mangos.OptionRecvDeadline, 5 * time.Second). This way, I don't have to calculate the deadline and set the option for every send/recv.

Even better, it could probably take either a time.Time or time.Duration.

The docstrings on OptionWriteQLen and OptionReadQLen says the default size is 10, but it's now 128.

At least one user has indicated a strong desire for the tcpmux facility. Martin has done the work to make this work in nanomsg, we should support in mangos.

Originally: https://bitbucket.org/gdamore/mangos/issue/11

The protocols that handle the header (req/rep in particular) should learn to grok a message that is passed up with the header "pre-processed". This would prevent a requirement to copy / split the headers in the inproc transport.

However:

Its unclear that this is worthwhile. We're already getting rather good performance as it stands. Tweaking this further would allow us to actually outperform nanomsg, but to what end? Inproc isn't really a real-world deployment -- especially in a Go environment where its more natural to use channels.

Originally: https://bitbucket.org/gdamore/mangos/issue/16

We need a standard handler for the Device pattern.

Fixed in cset cset 22773ff

There are a few things that are utterly meaningless:

Send() on either a REQ or REP raw socket -- no header == no way to deliver it properly.
Recv() on a raw REP socket (raw REQ is probably OK, although you won't be able to match replies to requests sent unless you have support for it at the upper layer.)
Send() or Recv() on a respondant socket. (No way to route the reply!)

Other non-routing protocols may be ok.

But -- it may be simpler (and support future protocol extensions that add routing information to headers) if we simply insist that Send() and Recv() are in appropriate / unsupported on raw sockets. All the time. (Frankly, except for those that it matters for, there is no reason to use a RAW socket anyway ... e.g. RAW push/pull sockets don't differ meaningfully from cooked ones.)

Originally here: https://bitbucket.org/gdamore/mangos/issue/4

Fixed in: 8a1a0c3

These protocols normally never do a read. As a consequence, they will only notice a disconnect of a socket on the far end when the server attempts to send data to them. This can, in some applications, lead to extensive waste of files.

These protocols should do a blocking read, discarding any results obtained, so that we can reap disconnected TCP connections. (Presumably the same problem may exist with UNIX domain sockets, with the same corrective action.)

This was discovered by Jason Aten when testing his goq project with mangos, using the compatibility layer.

There are currently no examples that display how to setup Req/Rep to handle requests in parallel. Something I think many people will have use for.

Well, it would be nice to have this automated.

We should put a copy of the LICENSE, and a NOTICE file as well, in the repo.

Originally: https://bitbucket.org/gdamore/mangos/issue/21

So we control both ends (no compatibility issues, no direct browser connectivity).
SSL3 is broken badly, but even TLS 1.0 and 1.1 have security flaws. Just force 1.2 for now.

Fixed in cset 7e8399e

It turns out that the message sizes can have a nasty impact on overall memory consumption, particularly as we support buffering. Imagine for example 1000 subscribers attached to a publisher, and a message size of 1M. That requires 1G of RAM for buffering, and it gets worse if the message is copied for buffering. (I don't think it is... I think copying only occurs in the bottom transport.)

Anyway, this should be tunable, e.g. OptionMaxMessageSize or somesuch.