Curb does not seem to perform persistent connections about curb HOT 13 CLOSED

I believe there's an individual socket pool for each Easy handle. If you use Multi, it will keep a shared connection pool. Or reutilize the Easy handle by setting the url again and calling #perform.

from curb.

Switching this example to use Curb::Multi I see the same behavior (multiple sockets created).

The man page for curl_easy_perform() says:

   You can do any amount of calls to curl_easy_perform(3) while using  the
   same handle. If you intend to transfer more than one file, you are even
   encouraged to do so. libcurl will then attempt to re-use the same  con-
   nection for the following transfers, thus making the operations faster,
   less CPU intense and using less network resources. Just note  that  you
   will have to use curl_easy_setopt(3) between the invokes to set options
   for the following curl_easy_perform.

Which leads me to believe that Curl::Easy#perform should behave as I expect.

from curb.

this should fix the issue. http://github.com/taf2/curb/commit/73e11030c9debfc6c51f32e6aef4f597281cf6db

It keeps the original multi handle around between invocations.

I added two benchmarks in bench/curb_easy.rb and bench/nethttp_test.rb

here's the results for me:

time ruby bench/curb_easy.rb
Duration 0.059045 seconds

real 0m0.083s
user 0m0.019s
sys 0m0.014s

time ruby bench/nethttp_test.rb
Duration 0.063115 seconds

real 0m0.731s
user 0m0.406s
sys 0m0.070s

from curb.

Just to point it out, libcurl already uses a multi handle to perform a easy handle even if you don't manually add the handle.

from curb.

I think the issue with curb here is different... Unless libcurl provides away to get at that multi handle? For each easy request curb was creating a new multi handle, effectively throwing away any open connections. Now instead, when calling the curb easy perform it will reuse it's internal multi handle... Hope that makes sense?

from curb.

I'm wondering why create a multi for each easy handle if libcurl does that by itself inside easy_perform?

from curb.

curl_easy_perform blocks the whole ruby interpreter. To avoid the blocking and have libcurl play nice with ruby, all IO is run through rb_thread_select. In order to expose that kind of IO a multi interface was necessary... Unless there is an interface to expose what select method libcurl uses or access the easy handle's internal multi interface?

from curb.

Did some testing in 1.8.7 and 1.9.1 and it does some evil blocking indeed. Since we don't have access to libcurl's select or internal multi, a suggestion would be to fit rb_thread_schedule() in a callback and get rid of multi and the fds hassle.

from curb.

you mean for example have a libcurl callback on the easy handle such as on_progress or another that calls maybe rb_thread_schedule or even yields ?

from curb.

Yup, but according to libcurl docs, on_progress callback does not seem to be a good choice since it keeps being called every second even if there is no transfer running. I've did a quick test by placing rb_thread_schedule inside CURLOPT_WRITEFUNCTION callback and works . I didn't run any benchmark tho.

from curb.

if there's not transfer running than it's blocked... write function would only be called when data is received?

from curb.

I really think select is the right thing here, you want the OS to tell you when to idle and when to read/write otherwise let ruby run.

from curb.

Alright, it's using persistent connections now

from curb.