The sequential dispatcher has better performance because it does not need to start a fiber for every submitted effect. However, this means it can only run one effect at a time.

Typically it's possible to replace one parallel dispatcher with multiple sequential ones e.g. a dispatcher for reading and a dispatcher for writing, since there can only be one read and write in progress at a time.

Actual behaviour

Invalid transfers result in never terminating application.

    client.expect[String]("")

curl logs

* No URL set

when verbose is enabled, and transfer does not start or crashes.

another example:

    client.expect[String]("unsupported://server")

curl output:

* Protocol "unsupported" not supported or disabled in libcurl
* Closing connection -1

It maybe related to #30

Expected behavior

It must fail with corresponding error from curl

Original title: Test fails building from source on Ubuntu 22.04, X86_64

This issue to capture my experience whilst debugging another problem. I report it in case others try
to build from source.

I cloned this repository and build it, having make sure I am using the libcurl4-openssl-dev used in this CI.
When I do sbt test, the executable runs, but I get two instances of curl_easy_pause: 43: one for 'get' and
one for 'put'.

Additional notes from @armanbilge

• this seems to be due to a change since libcurl 7.68.0, which is currently used in CI and what I developed against.
  #8 (comment)
• it is also reproducible in macOS
  #5 (comment)

Follow up to #59 (comment). A user in #scala-native discord recently encountered this problem when trying to use http4s-curl. I was confused but lucky I remembered that comment and was able to suggest that they add that flag to fix their build.

Suffice to say, it's a bit unfortunate that this is necessary, since the @link("curl") is supposed to handle it.

If I understand correctly this is the code that introduced the problem.

I wonder if we could refactor that code, so instead of directly calling curl_version_info method, it takes the result of that method call and extracts the field from the struct. Then, we can call curl_version_info() directly from SN, and pass its result to that glue code helper. It's quite awkward on our end, but better for UX.

Libcurl 8 is out! There is no API nor ABI break in this version as stated in the release doc, however tests include a check on specific version which causes builds to fail on macos and windows.

This a major version number bump but without any ground-breaking changes or fireworks. We decided it was about time to reset the minor number down to more a manageable level and doing it exactly on curl’s 25th birthday made it extra fun. There is no API nor ABI break in this version.

https://daniel.haxx.se/blog/2023/03/20/curl-8-0-0-is-here/

The general pattern for http4s clients is to have a builder, and use that to create the client.

That pattern will not work here, because the client is constructed for us by the runtime. Therefore a config data class CurlClientConfig would be the way to do this. It would be similar in spirit to IORuntimeConfig in that it should be possible to override in CurlApp.

There's some fun to scope out the various curl configuration options. There's also the question of client-level config vs request-level config, which we can expose with an attribute.

And I'm not really sure why. For example:

// the server responds 400 parse error: invalid character in chunk size
Request[IO](POST).withEntity(Stream.emit[IO, String]("blah"))

// this just hangs
Request[IO](POST).withEntity[Stream[IO, String]](Stream.empty)

Currently to use http4s-curl we use CurlApp which replaces the IORuntime with a CurlRuntime. This means that it is not possible to integrate http4s-curl with FS2-based applications (Ember, Skunk, Lepus) and other native libraries (e.g. SNUnit).

In Cats Effect 3.6 the default runtime offers file descriptor I/O polling that works with all of these libraries. The idea would be to use IOApp as usual and then to construct a Resource[IO, Client[IO]] with a CurlClientBuilder (similar to Ember).

Here's a rough sketch for how to implement a Client[IO] using file descriptor polling:

1. Create a MapRef[IO, FileDescriptor, (Fiber, Fiber)] to keep track of a dedicated fiber for polling each relevant file descriptor for each type of interest (read interest vs write interest).

2. Configure the CURLMOPT_SOCKETFUNCTION to get notified about when cURL needs to start/stop monitoring sockets for read/writes. Use these callbacks to start/cancel fibers in the MapRef.
   https://curl.se/libcurl/c/CURLMOPT_SOCKETFUNCTION.html

3. To add a new socket you should register the file descriptor.

4. Then, start a fiber looping on read-readiness and another looping on write-readiness. Whenever the socket is ready, the curl_multi_socket_action should be invoked.
   https://curl.se/libcurl/c/curl_multi_socket_action.html

5. A callback for CURLMOPT_TIMERFUNCTION should also be registered. It should start a fiber that sleeps the requested time and then calls curl_multi_socket_action.
   https://curl.se/libcurl/c/CURLMOPT_TIMERFUNCTION.html

When setting up a transfer, we use curl slist, which needs to be freed manually.
https://curl.se/libcurl/c/curl_slist_free_all.html

Somehow this error manifested into a 6 hour hang on CI.

==> X org.http4s.curl.CurlClientSuite.3 post echos 12.67s org.http4s.client.UnexpectedStatus: unexpected HTTP status: 502 Bad Gateway for request POST https://postman-echo.com/post

https://github.com/http4s/http4s-curl/actions/runs/3182527644/jobs/5188586017

//> using lib "org.http4s::http4s-curl::0.2.0"

import cats.effect.IO
import org.http4s.curl._

object App extends CurlApp.Simple {
  def run = curlClient.expect[String]("https://example.com").flatMap(IO.println)
}

Yields:

org.http4s.client.UnexpectedStatus: unexpected HTTP status: 100 Continue for request GET https://example.com

I guess we need special handling for 100 responses? 🤔

Or maybe we can make curl not send this.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/100

To have a server check the request's headers, a client must send Expect: 100-continue as a header in its initial request and receive a 100 Continue status code in response before sending the body.

So far I've been lazy and just print the error code. But it's actually not that hard to do something smarter and more user-friendly.

https://curl.se/libcurl/c/curl_easy_strerror.html
https://curl.se/libcurl/c/CURLOPT_ERRORBUFFER.html

There's probably a few places we can take advantage of this.

https://scala-native.org/en/latest/changelog/0.4.8.html#easier-access-to-arrays-underlying-memory

For historical interest:

When I clone from source, build, and run "sbt test" I get two
instances of the error curl_easy_setop: 1, one for 'get' and one for 'put'.

This was several recursion levels deep from the work I was trying to
get done, so I report it only in case the information may help others.

• https://curl.se/docs/websocket.html
• https://curl.se/libcurl/c/curl_ws_recv.html
• https://curl.se/libcurl/c/curl_ws_send.html
• https://curl.se/libcurl/c/curl_ws_meta.html

https://github.com/typelevel/keypool

Easy handles are meant and designed to be reused. When you have done a single transfer with the easy handle, you can immediately use it again for your next transfer.

https://everything.curl.dev/libcurl/easyhandle#reuse

After the transfer has been made, you can set new options and make another transfer, or if you are done, cleanup the session by calling curl_easy_cleanup. If you want persistent connections, you do not cleanup immediately, but instead run ahead and perform other transfers using the same easy handle.

https://curl.se/libcurl/c/libcurl-easy.html

Sounds like a mistake that we're not reusing these 😅