shutdownHeartbeats is not shutting down heartbeats about stompngo HOT 16 CLOSED

Are you reading the correct channel to get ERROR frames? If the reader takes an EOF, that should get sent back to the client. That logic has been there a long time.

ERROR frames will not be sent on a channel that is subscription specific.

ERROR frames (and RECEIPT frames) are sent on the connection wide MessageData channel:

error_frame := <- connection.MessageData // Normally in a select

Those frames can not be sent on a subscription specific channel because their headers do not contain either DESTINATION or SUBSCRIPTION, so a specific subscription cannot be identified.

With a logger set, you should see a log message containing:

RDR_RECEIVE_FRAME

Show me a few (5-10) messages right before and right after that.

Repeat: what are your Heart Beat values ??????

A typical way to handle reads in an app:

                var md stompngo.MessageData
		select {
		case md = <-subscribeChannel:
                        // MESSAGE frames here
		case md = <-conn.MessageData:
			// Frames RECEIPT or ERROR here
                        // check md.Error for not nil
		}

from stompngo.

First, it looks like that channel is not being used. I'll look into integrating it. Thanks.

Second, heartbeat is set to 500,500.

Third, here are some log lines:

SEND start destination:...
WTR_WIREWRITE start
invoking write deadline callback 1
ERROR: write tcp <source>-><dest>: i/o timeout
WTR_WIREWRITE COMPLETE SEND destination:...
SEND end destination:...
RDR_RELOOP
RDR_RECEIVE_FRAME
RDR_RECEIVE_FRAME MESSAGE subscription:<uuid>
destination:<dest>
message-id:<msg_id>
redelivered:false
ack:<ack_id>
content-length:<num>
...
RDR_RECEIVE_ERR <nil>]
SEND start destination:...
WTR_WIREWRITE start
invoking write deadline callback 1
ERROR: write tcp <source>-><dest>: i/o timeout
WTR_WIREWRITE COMPLETE SEND destination:...
SEND end destination:...
HDERR ends
RDR_CONN_ERR EOF
RDR_SHUTDOWN <ts>
WTR_WIREWRITE shutdown W received
WTR_SHUTDOWN <ts>
Heartbeat Send ends <ts>

I'll let you know if the MessageData channel works for us.

from stompngo.

Several points:

Update to the HEAD of DEV please.

The system can not still be trying to send heartbeats. The Heartbeat send ticker does end. That is plain reviewing that log snippet.

It you have not been using the MessageData channel before this - well, you are doing it wrong. Perhaps you should spend some time reviewing stompngo examples:

https://github.com/gmallard/stompngo_examples

Your heartbeat values are both way too low in my opinion. This is not unusual to see. My very best rule of thumb regarding heartbeat values is: make them as large as possible while still meeting your application's other requirements. Although 500ms is allowed by the spec's, it is an abuse of the concept. Please review my heartbeat recommendations:

https://github.com/gmallard/stompngo/wiki/Heart-Beats

But just think about 500 ms. How long does it take to ping one server from another on your net? 500 ms is too low.

from stompngo.

I didn't get a chance to update yet, but I think I'm getting closer. I'm at the point where I can see that I get an EOF because the connection is dropped and I can see the readers and writers shutting down, but disconnecting doesn't seem to work. I get a "DISCONNECT start" when I try it, but that's it. I thought it was because I was trying to disconnect while it was still connected, so I put some code in to check to see if I was connected and only disconnect if I was, but I got the same issue. Looking at the code, I can see that shutdown method handles ensuring that the connection flag is set to false, but it doesn't appear that it is run except when called via the Disconnect function.

Is there something that I'm missing or is this a legit bug?

from stompngo.

From lots of experience - with stompngo as well as the Ruby stomp gem - I can tell you that it is entirely possible for a network connection to be so messed up that calls to DISCONNECT will always fail one way or another.

If you have a situation where you know that the net.Conn is "not right" I would avoid calling DISCONNECT at all.

It sounds like you are in a situation where:

a) The connection writer is down (has shutdown)
b) You call DISCONNECT

I think this is likely to hang - because the writer is not "receiving" data on it's input channel.

But it also might panic.

In any case, if the network connection is truly down - all bets are off. There is nothing that stompngo could possibly do about it.

from stompngo.

Yes, that is the situation I'm in except that I don't know why I'm getting connection errors with this app.

My real issue that I was trying to get across was that the "connected" flag was not being set to false when the readers/writers shut down. To me, if the readers and writers have all shut down, this flag should be set to false so I can have my code work around it. I know I can work around it another way. From an API perspective, however, I feel it would be cleaner to be able to check to see if the client is still connected and act accordingly.

from stompngo.

from stompngo.

When I was calling Disconnect, I would see "SHUTDOWN start" and that's it. I do not see it at any other time.

from stompngo.

Regarding connection errors in general: what is in your broker's logs ??

That will likely yield some clues .....

from stompngo.

I'm starting to think that I'm either overwhelming the connection or there's something up with this network. I ran a couple tests this morning. The first, with a 10-second heartbeat, resulted in the broker closing the connection due to a skipped heartbeat. The second, I increased it to a 30-second heartbeat and I can see that the connection was closed, but no error.

Since that appears to be more on my end than yours, I'd like to go back to the original purpose of this ticket which is if the readers and writers shut down due to a connection error, should the stompngo client still report as being connected to the broker. I think your code can at least set the required flag to indicate that the connection has crapped out in that case. Then, you can make the Disconnect code safer by checking for that flag and making sure it doesn't get stuck. Granted, this may not be a very common case, but I'm personally a proponent of making sure it's hard to do something you shouldn't be doing.

from stompngo.

from stompngo.

from stompngo.

So, at HEAD of DEV is:

1. More aggressive set of connected flag to false in both reader and writer shutdown logic paths

2. Optional use of retry when short network writes are detected. Note that this logic is not the default. The default is to report/return short writes as an error. Activate this new logic by:

    conn.ShortWriteRecovery(true)

from stompngo.

Thanks. I got my issue worked out, so I'll try them out when I get a chance.

As for my issue, in case you're interested, my best theory is that the work being performed in batches took too long between acks. I shrunk the batch size down a good bit and it started performing much better and more reliably.

from stompngo.

Please close this if you are satisfied with current results.

from stompngo.

The changes for setting the connected flag look good. I'm not sure when I'll get back to this, so since things are working for me right now, I'll close this. Thanks!

from stompngo.