launchdarkly / ruby-eventsource Goto Github PK
View Code? Open in Web Editor NEWServer-Sent Events client for Ruby
License: Other
Server-Sent Events client for Ruby
License: Other
I can't find any params to set this . ty.
This was mentioned in a comment on a Ruby SDK issue filed by @camillof. Currently, the Ruby SSE implementation uses Socketry to implement read timeouts on streaming connections. Socketry uses an old version of hitimes
, which apparently in some environments can cause this error:
/usr/local/DFS/gems/hitimes-1.3.1/lib/hitimes.rb:56:in `<top (required)>': Unable to find binary extension, was hitimes installed correctly? The following paths were tried. (LoadError)
/Users/camillo/hitimes/2.6/hitimes : incompatible library version - /usr/local/DFS/gems/hitimes-1.3.1/lib/hitimes/2.6/hitimes.bundle
/Users/camillo/hitimes/hitimes : cannot load such file -- hitimes/hitimes
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from /usr/local/DFS/gems/socketry-0.5.1/lib/socketry.rb:10:in `<top (required)>'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
from /usr/local/DFS/gems/ld-eventsource-1.0.3/lib/ld-eventsource/impl/streaming_http.rb:5:in `<top (required)>'
The 2.0 release of hitimes
seems to fix this. However, Socketry is unmaintained so a patch to use this update may not become available. Options would include forking Socketry or pursuing a different strategy to implement timeouts.
Ruby Server SDK calls Eventsource with large payloads of JSON data, causing Rails Puma workers to timeout on the default 90s timeout. We have identified likely causes which will be highlighted below.
The code for handling streaming responses chunks on record breaks (/\r\n/
approx), but uses a mechanism which can lead to exponential growth on large data
rows:
def read_line
loop do
@lock.synchronize do
i = @buffer.index(/[\r\n]/)
if !i.nil? && !(i == @buffer.length - 1 && @buffer[i] == "\r")
i += 1 if (@buffer[i] == "\r" && @buffer[i + 1] == "\n")
return @buffer.slice!(0, i + 1).force_encoding(Encoding::UTF_8)
end
end
return nil if !read_chunk_into_buffer
end
end
Specifically this line is worrying:
i = @buffer.index(/[\r\n]/)
When LD is initialized I believe that it tries to send the entirety of the payload in one line of data, but server responses are chunked streams. If the full response is 150Mb (not theoretical) and the chunk size is say 1Mb (guessing for example) that means it won't hit a record-break until 150Mb.
In other words, it does this (-
is one chunk):
buffer = ''
buffer.index(/[\r\n]/)
buffer = '-'
buffer.index(/[\r\n]/)
buffer = '--'
buffer.index(/[\r\n]/)
buffer = '---'
buffer.index(/[\r\n]/)
buffer = '----'
buffer.index(/[\r\n]/)
buffer = '-----'
buffer.index(/[\r\n]/)
# ...
buffer = '-' * 150
buffer.index(/[\r\n]/)
Repeat until you get to 150Mb from what I think might be happening and you've done a substantial number of reads over the same data.
The problem is that eventsource is reading the entirety of the buffer and storing the entirety of the response, rather than checking for record breaks in each chunk.
Why does that become an issue? I believe this chunk of code is relevant:
message = JSON.parse(message.data, symbolize_names: true)
all_data = Impl::Model.make_all_store_data(message[:data])
@feature_store.init(all_data)
@initialized.make_true
...and that won't be triggered by conn.on_event { |event| process_message(event) }
(src) until that entire JSON payload is processed from chunks.
We believe that the Ruby Server SDK is trying to retrieve the entirety of the flag data as a singular response from Ruby Eventsource, and for large clients (150Mb+) this will cause crashes.
Current experimentation I've been working on has been to prevent full reads of a partial buffer, but I believe this reflects more on the entirety of flag data being sent in a single message. If it truly is all being sent in a single response this will be a severe issue for larger clients.
when i depoly sse client to docker ,it costs 30 / 40 more secends to receive events.
do someone know how to reslove this?
Hi. I am hoping to get some guidance on what to do with the following error:
WARN -- ld-eventsource: Unexpected error from event source: #<Faraday::ConnectionFailed wrapped=#<Errno::EINVAL: Invalid argument - setsockopt(2)>>
How would I proceed to debug what is going on with this? Thanks.
Currently the CI doesn't test Ruby 3.1 or 3.2, or JRuby 9.4. It's also locking bundler to a very old version that may be an issue for more recent Rubies.
Ideally CI should cover all not EOLed Rubies. I'd be happy to make this change, but because the library is using Circle CI I can't test on a fork. Just let me know if there's willingness to work on a PR for this and I'll submit an update that we can use as a base.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.