Comments (23)
Furthermore, the line above does not seem to take the ReadDeadline
into account, if I am not mistaken?
from stompngo.
I do have test(s) for disconnect receipts, and do not see hangs.
What stompngo version are you running ?
Do you have a short example that shows this behavior ?
Can you get me a stack trace of the hang (on Linux press Ctrl-\ when the hang occurs)?
Can you enable logging and show me that output ?
ReadDeadLine is reset between all calls to the TCP reader. They have to be. Otherwise nothing would work right.
You could always request no receipt as a work around.
from stompngo.
Thank you very much for your prompt reply, much appreciated!
stompngo version: v1.0.12
I'm currently working on a minimal example, I'll come back to you as soon as I know more or have it available.
Please find the recorded stack trace with enabled connection logging attached:
stackdumpWithLogMasked.txt
I enabled the logging with conn.SetLogger
. Please let me know if you'd like me to log something else as well.
ReadDeadLine is reset between all calls to the TCP reader. They have to be. Otherwise nothing would work right.
Alright, I see. Thanks a lot for the information.
You could always request no receipt as a work around.
Unfortunately I have to request a receipt to make sure that the disconnection has worked properly, for reasons that are probably beyond the scope of this.
Thank you very much for your help.
If you have any questions/comments, please do not hesitate to say so.
from stompngo.
Are you fully draining your input queues/topics ?
It looks like the system is deadlocked trying to send you a MESSAGE frame which you have not received.
from stompngo.
thanks a lot for your help. now I'm draining the topic in a parallel goroutine while disconnecting, and it doesn't hang anymore -- however, now, it never receives a receipt. conn.DisconnectReceipt
is empty, as well as all subsequent messages on conn.
from stompngo.
short update: I see now that it works for some servers, for some not. I'll further investigate whether the error is in on the side of server's STOMP implementation, and come back to you if that error source is excluded. In the meantime I'll close this issue.
Thanks a lot for your prompt and helpful input, much appreciated!
from stompngo.
By now I can say that the error is not on the side of the servers.
My most recent insights are: Your library sees and records the receipt (visible in the logs):tada:, but when I try to access it with conn.DisconnectReceipt
, it's sometimes no longer there. 🤔 Because of that, I can't verify that the receipt ID is the right one. Does the connection shutdown logic delete the recorded receipt, s.t. this issue would be due to a race condition?
from stompngo.
The receipt is never deleted by stompngo code.
Show me the logs you are looking at please. The receipt should actually be in the logs (see code in disconnect.go).
Also ..... I need some code that recreates this. Because I cannot recreate it here .......
from stompngo.
I've now taken another look at the logs and found something potentially interesting.
LOG: 2019/10/31 13:42:09.991834 connection.go:182: ID:< SOME ID, MASKED > < $HOME - MASKED >/go/pkg/mod/github.com/gmallard/[email protected]/disconnect.go 84 [DISCONNECT dr receipt:6
{{ []} <nil>}]
Contrary to my original suspicion, the logged receipt in this line is not the one in the received message (which is in fact empty, as logged) -- it is a second mentioning of the earlier sent receipt.
This is the code creating the log entry:
c.log(DISCONNECT, "dr", ch, c.DisconnectReceipt)
where ch
is a clone of the headers the user passed as the method's argument + a receipt, if the user didn't disallow it explicitly.
I suggest to check in the library whether the received receipt is the valid & expected one, e.g. with
conn.DisconnectReceipt.Message.Headers.ContainsKV(stomp.HK_RECEIPT_ID, receipt)
and with listening to the receipt with some safety margin. For example, one can start listening already right before sending the DISCONNECT
frame, and keep listening some time until a receipt arrives. If the receipt doesn't arrive within a given timeout, one can mark the disconnection as unsuccessful -- I'm interested in hearing your input on how to handle this best, what follow-up actions to take if the disconnection didn't work. So far I've closed the underlying connection with a defer
statement and stopped program execution to avoid the message buffer exploding on the server side.
I've implemented this mentioned procedure, let me know if you'd like me to contribute the code.
The only part that's missing is to start listening to the receipt already before sending the DISCONNECT
frame, I'll do that now and keep you posted.
Also, I'm working on a small example, I'll come back to you as soon as I know more.
from stompngo.
I'd like to give a short progress update.
As mentioned earlier, this issue doesn't always appear. I wrote a small example to connect and then disconnect with a receipt, but so far it works. There seems to be some sort of race condition or the like. Should I be able to reliably reproduce the error at some point, I'll tell you.
Since I have no idea where the issue comes from, and since the hypothetical race condition seems nontrivial to debug, I've now chosen the following procedure (and will change it back as soon as this issue is solved): If I don't get a receipt, it's not too bad, since I close the underlying connection afterwards anyways. So, I attempt to do it the nice way -- with a receipt -- but if it doesn't work, the program doesn't throw an error, it just warns, closes the underlying connection, and exits, since this is anyway the last action (at least in the current implementation).
If you'd like to suggest a different procedure or would recommend a different approach, please feel free to let me know, I'd be happy hear your feedback.
from stompngo.
Put a check in your code that looks at what is actually in conn.DisconnectReceipt. I want to know if it is actually an ERROR frame.
That is entirely possible, and my code does not handle that properly. I am working on a fix for that.
from stompngo.
Ah, I think there might be a small misunderstanding :) I did not receive ERROR frames, but entirely empty ones; I have checks for this in the code:
if r.Message.Command == "" && (len(r.Message.Headers) == 0) && (len(r.Message.Body) == 0) {
emptyMsgCount += 1
}
In general , thanks a lot for looking into this, much appreciated!
from stompngo.
Two items:
-
There is an enhancement/correction to Disconnect on HEAD of the dev branch. You might try that to see if it provides any more data about this.
-
This section in the specification:
https://stomp.github.io/stomp-specification-1.2.html#Connection_Lingering
Read the last sentence of that section.
from stompngo.
Sorry for the delayed message.
Thanks a lot for pointing me to that sentence in the specification, that is very helpful!
Thanks also for the changes in the dev branch, they look good to me. If possible I'll try them out sometime soon and tell you how it went.
from stompngo.
Note that there is another reference to this behavior in:
https://stomp.github.io/stomp-specification-1.2.html#DISCONNECT
Question: have you tried this with heartbeats on?
from stompngo.
Indeed, this is very helpful! Thanks a lot, your help is much appreciated.
Yes, I had heartbeats on (my heartbeats "offer" before negotiation: [0, 10000]).
from stompngo.
There is another enhancement to DISCONNECT on the dev branch.
It allows the user to optionally specify a "disconnect timeout". After the timeout expires, if no RECEIPT frame has been received, DISCONNECT will return an error indicating a timeout.
from stompngo.
Also, I pushed v1.0.13 to github a bit ago. Includes all the DISCONNECT enhancements. Please give that a try.
from stompngo.
Please excuse the delayed reply.
Thanks a lot for the new features, this looks good to me.
At the moment I've switched from ActiveMQ to something else. I'm quite busy atm, but generally I'd be happy to try the changes out -- I'll let you know if I get a chance :)
Two small suggestions looking at the code: How about implementing timeouts with
case <- time.After(<timeout>)
instead of creating a ticker
and assigning the channel output to _
here?
This is just a minor suggestion, nothing crucial.
Source: https://gobyexample.com/timeouts , last visited: 19. November 2019
from stompngo.
I am always curious about brokers. I would like to ask: from AMQ to what ?
Here I test with:
- AMQ
- Rabbit
- Artemis
- Apollo (the last release, since development has ceased)
Using time.After is a good idea, thanks, I will probably tweak the code to do that.
from stompngo.
I switched from AMQ to a totally different approach that doesn't involve messaging at all -- not because of the messaging itself, but because my program needs to fetch information from somewhere else after all. :)
You're welcome, as already said just a small suggestion, nothing important.
Thank you very much for helping with this, very much appreciated! And in general thank you very much for maintaining this library.
from stompngo.
Apologies for the very tardy reply.
Removal of TIcker form that code has been on DEV since 12/23/19. Commit:
I hope you have been able to try that.
I am going to close this issue based on the above. If you need to re-open it, please d so.
Regards, Guy
from stompngo.
Thanks much for your message :) no worries about the delay.
I opened the issue during an internship, by now I don't have access to the code anymore for which I experienced the issue. Closing the issue sounds good to me.
from stompngo.
Related Issues (20)
- Date race analysis HOT 4
- Potential blocking on unsubscribe with message in flight HOT 27
- Tests Fail various ways, golang at 8b22109 HOT 2
- Race in disconnect HOT 3
- Go get fetching the go1 branch HOT 9
- Support timeouts for network reads/writes HOT 16
- shutdownHeartbeats is not shutting down heartbeats HOT 16
- Abstract Connection to an interface HOT 2
- Panic: non-positive interval for NewTicker HOT 3
- readBody::utils.go panics when message body length is unexpectedly zero
- Is this a potential bug of SetWriteDeadline? HOT 3
- time.Ticker overflow HOT 1
- Potential blocking on Connection.Disconnect() HOT 2
- Use case in production environment HOT 1
- SSL Port error handling enhancement HOT 1
- golang docking ActiveMQ HOT 1
- Header Decode Shortcoming HOT 1
- Question about heartbeats HOT 2
- Disconnect has incomplete error checking HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stompngo.