elixir-plug / plug_cowboy Goto Github PK

https://phoenixframework.org/blog/phoenix-1-4-0-released references plug_cowboy version 2.5, is that coming out anytime soon?

Description
I'm not sure it's bug or not, but I observe a strange behavior of server. I face with the task of starting two or more supervised http server with different ports, and I did the following:

1. I've created simple mix application with supervised DynamicSupervisor

defmodule Master do
  @moduledoc false
  use Application
  require Logger

  @doc false
  def start(_type, _args) do
    children = [
      {
        DynamicSupervisor,
        strategy: :one_for_one,
        name: Master.DynamicSupervisor
      }
    ]
    Supervisor.start_link(children, strategy: :one_for_one, name: __MODULE__)
  end

  def start_server(port) do
    DynamicSupervisor.start_child(Master.DynamicSupervisor,{
      Plug.Cowboy,
      scheme: :http,
      plug: Router,
      options: [
        port: port
      ]
    })
  end
end

This module contain just one custom function, it's Master.start_server/1, it receive the port as argument, and trying to start PlugCowboy with this port as child of DynamicSupervisors.

2. I've added a simple router with "/echo"

defmodule Router do
  use Plug.Router
  require Logger

  plug Plug.Parsers, parsers: [{:multipart, length: 20_000_000}]
  plug(:match)
  plug(:dispatch)

  # Creates basic params for Plug.Conn
  defp get_base_connection(conn), do: conn |> put_resp_content_type("application/json")

  get "/echo", do: conn |> get_base_connection |> send_resp(200, Jason.encode!(%{result: "Success!"}))

  # 404 page error
  match _, do: send_resp(conn, 404, Jason.encode!(%{code: 404, message: "Page not found"}))
end

3. And have tried to start it via iex -S mix

What I've got
I've got

Interactive Elixir (1.9.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)>

So far so good. And now, I'm trying to start my first server:

iex(1)> Master.start_server 7000
{:ok, #PID<0.545.0>}

Still okay, but, if I try to run a second server on another port:

iex(2)> Master.start_server 7001

I got this:

03:49:12.712 [error] Failed to start Ranch listener Router.HTTP in :ranch_tcp:listen([cacerts: :..., key: :..., cert: :..., port: 7000]) for reason :eaddrinuse (address already in use)

{:error,
 {:shutdown,
  {:failed_to_start_child, :ranch_acceptors_sup,
   {:listen_error, Router.HTTP, :eaddrinuse}}}}

What's strange?

1. I've started my second server with port=7001, but error message contain port=7000 (my first server's port).
2. If we try to call this function again (with same args) - we will got:

iex(3)> Master.start_server 7001
{:ok, #PID<0.668.0>}

All log of my console*

Interactive Elixir (1.9.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Master.start_server 7000
{:ok, #PID<0.558.0>}
iex(2)> Master.start_server 7001

03:49:12.712 [error] Failed to start Ranch listener Router.HTTP in :ranch_tcp:listen([cacerts: :..., key: :..., cert: :..., port: 7000]) for reason :eaddrinuse (address already in use)

{:error,
 {:shutdown,
  {:failed_to_start_child, :ranch_acceptors_sup,
   {:listen_error, Router.HTTP, :eaddrinuse}}}}
iex(3)> Master.start_server 7001
{:ok, #PID<0.668.0>}
iex(4)> 

Hello! just a friendly request to release version v2.2.2 with the new early_error events...

@josevalim

Howdy,

I just ran a new mix phx.create today and it is broken out of the box. On initial load you see the following error:

{:response_error, :invalid_header, :"Response cookies must be set using cowboy_req:set_resp_cookie/3,4."}

I went down a bit of a rabbit hole trying to get it to work, but luckily had created a new project a couple of days ago which worked and still works fine.

Seems Cowboy has released an update today (2.11) and that doesn't play nice with plug_cowboy, which asks for the latest cowboy (from what I can see). So todays build running with 2.11 doesn't work, but the build with 2.10 works fine.

Problem is that Cowboy isn't an explicit dependency for a new Phoenix project, plug_cowboy is. So we can't downgrade the Cowboy version to a working one.

I'm currently working on a custom Logger backend to report crashes to a 3rd party service. For HTTP requests, I'd like to pull some metadata from the Plug.Conn and attach that to the report.

To get access to the conn in the Logger backend, I think it could be added to the metadata here:

Thoughts?

If this change makes sense, I can open a PR.

I found some trace of it in 040c53f but not on master?

I expected config :phoenix, :filter_parameters to filter sensitive parameters from logs. But exceptions logged by PlugCowboy exposes all parameters.

[error] #PID<0.823.0> running MyAppWeb.Endpoint (connection #PID<0.821.0>, stream id 1) terminated
Server: localhost:80 (http)
Request: GET /signin/callback?sensitive-parameters-here
** (exit) an exception was raised:
    ** (FunctionClauseError) ...

I thought of two solutions, both less than ideal.

1. :filtered_parameters are passed to PlugCowboy during start to configure translator. They show up as [FILTERED] as expected. Possible duplicate code with Phoenix.
2. Don’t log query strings in PlugCowboy. Possibly confusing.

There are a category of requests that cowboy itself will respond to before it invokes the Plug.Cowboy.Handler, and thus they aren't detectable by any telemetry instrumentation currently in plug or plug_cowboy...

The early_error cowboy callback is implemented, and a subset of requests are logged:

It would be nice to emit a telemetry event for these requests so that they aren't lost. They consume real resources, but are currently invisible to any monitoring solution based on the exiting Plug events.

Thoughts @josevalim?

We are creating a new Cowboy stream handler that will emit telemetry events:

• https://github.com/beam-telemetry/cowboy_telemetry

Once it's released, we should adopt it in plug_cowboy and include it by default, and also adopt it for the early_error logging.

I don't know if this library is the place to implement this, but shutdowns due to idle_timeout need to be made more obvious. I just lost over an hour of my time understanding that my application was silently failing because of this.

ninenines/cowboy#1385

Hi everybody,
I hope this is not to offtopic, but is it possible to create websocket connections with plug_cowboy?

If yes, could you point my to any resources that show how to do it?

Thanks in advance

Hello,
I am trying to send a list of values in query parameters using the key-value notation, but I am not getting the expected results.

Key-Value
We can pass query parameters as key-value pairs like this:
?obj[eq]=val

Fetched query parameters for this request would be:
%{ "obj" => %{"eq" => "val"} }

List of value
Similarly we can pass a list using the array notation:
?arr[]=val1&arr[]=val2

This would yield:
%{ "arr" => ["val1", "val2"] }

Key-Value where value is a list
But if we try passing a list of key-value pairs we only get the last value
?obj_arr[in]=val1&obj_arr[in]=val2

This yields:
%{ "obj_arr" => %{"in" => "val2"} }

Expected result
It would be nice, if the request above would result in:
%{ "obj_arr" => %{"in" =>[ "val1", "val2"]} }

I am using:
Erlang/OTP 26
Elixir 1.16.0
plug_cowboy 2.7.0

Hi,

It looks like the exception logger from the plug here adds the entire conn to the logger metadata. The conn can contain sensitive information such as headers with authorization keys, cookies with sensitive values, or maybe even request/response bodies with sensitive data in them that should not be logged. In cases where metadata: :all is used this leads to those sensitive values being logged out which may be great for debugging, is not great for production environments where those values need to be scrubbed and or other serious implications.

Should this be added to the metadata by default given the high sensitivity of those values and the somewhat common practice to allow all metadata?

Additionally, I do not see any mention of this in the docs for the plug. Maybe it's somewhere in the phoenix or plug docs though and I missed it.

Last but not least, thank you for the awesome project, we really appreciate all the hard work put into this community <3

Hi 👋,

children = [
      Plug.Cowboy.child_spec(
        scheme: :http,
        plug: WSServer.Router,
        options: [
          dispatch: dispatch(),
          port: port(),
          transport_options: [
            num_acceptors: 2,
            max_connections: 16_384 # => default
          ],
          protocol_options: [
            idle_timeout: 1000 # => not getting applied
          ]
        ]
      )
    ]

But is getting applied inside SocketHandler init callback

def init(req, state) do
    Logger.info("[SocketHandler] init")
    dbg({req, state})
    state = %{registry_key: req.path}

    {:cowboy_websocket, req, state, %{idle_timeout: :infinity}}
end

Is there anything I'm missing ?

Elixir 1.10.2 (compiled with Erlang/OTP 21)

Mix 1.10.2 (compiled with Erlang/OTP 21)

Plug 2.1

Hello all,

Should I open a quick pull-request to update the documentation?

When I tried to add the Plug to my application's Supervision tree as specified by the documentation on Hex:

children = [
  {Plug.Cowboy, scheme: :http, plug: MyApp, options: [port: 4040]}
]

Supervisor.start_link(children, strategy: :one_for_one)

I got the following error message after running the command mix run --no-halt:

20:00:38.644 [info]  Starting Concise application.

20:00:38.689 [info]  Application concise exited: Concise.Application.start(:normal, []) returned an error: shutdown: failed to start child: {:ranch_listener_sup, ConcisePlug.HTTP}
    ** (EXIT) shutdown: failed to start child: :ranch_acceptors_sup
        ** (EXIT) :badarg
** (Mix) Could not start application concise: Concise.Application.start(:normal, []) returned an error: shutdown: failed to start child: {:ranch_listener_sup, ConcisePlug.HTTP}
    ** (EXIT) shutdown: failed to start child: :ranch_acceptors_sup
        ** (EXIT) :badarg

However, after following the instructions provided by the project's README, it worked out fine at the end.

children = [
      Plug.Cowboy.child_spec(scheme: :http, plug: MyRouter, port: 4001)
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)

After updating to Cowboy version 2.11.0, compatibility issues have arisen with plug_cowboy version 2.7.0, leading to failures in environments where plug_cowboy requests the latest version of Cowboy. This issue manifests specifically when running mix phx.create, resulting in an application that crashes on initial load with a header handling error related to set-cookie directives.

As identified by a Cowboy maintainer, the root cause of this issue stems from Cowboy's recent update to be more explicit in what is not allowed concerning the set-cookie header. The way Plug sets the set-cookie header does not comply with the HTTP specification, as Cowboy cannot handle it like other headers due to its special nature. Previously, this incompatibility did not surface due to coincidental factors but has now become evident with Cowboy's stricter enforcement.

Can we get a proper fix for it?

We have a Phoenix application that generates very large API responses (approaching 1Gb in size) in both JSON and CSV formats. To keep memory usage down, we are streaming these large responses back to the client using Plug.Conn.send_chunked and Plug.Conn.chunk, and we "consume" the stream using Enum.reduce_while, as suggested in the documentation for Plug.Conn.chunk.

While this approach has worked fine for API responses that are not extraordinarily large, it has been causing problems for our very large responses. Specifically, when downloading these large API responses (using curl for example), the download will start out reasonably fast, but then after a period of time it will slow down to a crawl. By using the observer tool I was able to pinpoint the problem: the cowboy_clear:connection_process process gets overloaded with millions of messages in its mailbox and also consumes a large amount of memory. When the system gets into this state is when the chunked HTTP response slows down so much that it has virtually stopped altogether.

(Incidentally, aborting the download on the client-side (such as Ctrl-C on curl) will eventually cause the server-side to clean itself up and the memory usage quickly returns to normal levels.)

The root problem appears to be that Plug.Cowboy.Conn.chunk/2 calls :cowboy_req.stream_body/3, which immediately sends a message to its stream handler process. Since there is no backpressure mechanism, this means that the Stream that is being chunked back to the client is fully "realized" almost immediately, so memory usage spikes to the hold the entire stream in memory. It also means that the Cowboy stream handler process gets overloaded with messages in its process mailbox, and this in turn causes the extreme slow down mentioned above.

I cannot share our real codebase that demonstrates this issue, but I have put together a demo project that simulates the scenario and reproduces the behavior.

In this demo project, there is a controller with a flood action that generates a large stream of CSV data and streams it to the client: https://github.com/pro-football-focus/plug_chunk_backpressure_demo/blob/25f93204024e479a965cda61ddb401ff792de8e6/lib/chunk_backpressure_demo_web/controllers/demo_controller.ex#L6

This flood action demonstrates the behavior described above, where the cowboy_clear:connection_process mailbox gets overloaded and the streaming response slows to a crawl.

In that same controller, there is another action called backpressure, which streams the same large CSV response to the client, but in which I've implemented a rudimentary backpressure mechanism: https://github.com/pro-football-focus/plug_chunk_backpressure_demo/blob/25f93204024e479a965cda61ddb401ff792de8e6/lib/chunk_backpressure_demo_web/controllers/demo_controller.ex#L12

The backpressure action checks the size of the process mailbox for the cowboy stream handler, and if it goes over a certain threshold (arbitrarily set at 500 in the demo), then it waits until the mailbox size goes back down below that threshold before sending the chunk to cowboy. (This is done using the wait_for_it library, but that is an incidental detail and not important to demonstrate the issue.)

This rudimentary backpressure mechanism not only keeps the memory usage low and prevents the overload of the stream handler process mailbox, but it also speeds up the response dramatically! The CSV data produced in the demo is 300Mb in size, and downloading it with curl took nearly 6 minutes using the flood action (with no backpressure) but took only 3.5 seconds with the backpressure action:

$ time curl -o flood.csv http://localhost:4000/flood              
curl -o flood.csv http://localhost:4000/flood  2.88s user 27.72s system 8% cpu 5:50.33 total

$ time curl -o backpressure.csv http://localhost:4000/backpressure 
curl -o backpressure.csv http://localhost:4000/backpressure  0.34s user 2.95s system 93% cpu 3.518 total

I think this convincingly demonstrates the need for a good backpressure mechanism in Plug.Cowboy.Conn.chunk/2.

I don't intend to suggest that inspecting the process mailbox size like I've done in the demo should be the basis for such a backpressure mechanism, though. In fact, it is quite possible that there is something like a quantum "observer effect" happening here: the very act of checking the size of the mailbox slows things down enough to "accidentally" provide backpressure. That said, I'm hoping that someone on the plug_cowboy team can come up with a production-ready backpressure mechanism that allows for streaming large responses efficiently.

See failures on https://travis-ci.org/elixir-plug/plug_cowboy/jobs/544024765

Problem

I'm trying to use custom options on Plug.Conn.read_body/2 to prevent the reading to stop before the size I need.

# my method

  defp get_body(conn) do
    IO.inspect(">>>>>> Starting read conn.body")

    opts = [
      # :length - sets the maximum number of bytes to read from the body on every call, defaults to 8_000_000 bytes
      length: 30_000_000,
      # :read_length - sets the amount of bytes to read at one time from the underlying socket to fill the chunk, defaults to 1_000_000 bytes
      read_length: 30_000_0000,
      # :read_timeout - sets the timeout for each socket read, defaults to 15_000 milliseconds
      read_timeout: 300_000
    ]

    case read_body(conn, opts) do
      {status, binary, conn} ->
        IO.inspect(">>>>>> Ending read conn.body: #{status}")
        {:ok, binary, conn}

      _ ->
        :no_body
    end
  end

result:

13:40:54.025 request_id=2lnq07rbse5509m5ts0005p1 [info] PATCH /files/bc235362-fe22-11e8-9328-3c15c2d33dce
13:40:54.122 request_id=2lnq07rbse5509m5ts0005p1 [debug] Processing with MyAppWeb.UploadController.patch/2
  Parameters: %{"uid" => "bc235362-fe22-11e8-9328-3c15c2d33dce"}
  Pipelines: [:api]
13:40:54.122 request_id=2lnq07rbse5509m5ts0005p1 [debug] [upload] path chunk of upload bc235362-fe22-11e8-9328-3c15c2d33dce
">>>>>> Starting read conn.body"
">>>>>> Ending read conn.body: more"
[
  min_size: 5242880,
  part_size: 1300176,
  "file.offset": 0,
  "file.size": 1794211,
  part_size_min_size: true,
  "file.offset_min_size_file.size": true
]
last_part?: false
part_too_small?: true
13:41:09.341 request_id=2lnq07rbse5509m5ts0005p1 [info] Sent 409 in 15315ms

Versions

{:plug, "1.7.1"},
{:plug_cowboy, "2.0.0"}
{:cowboy, "2.6.1"}

Extra

Some configurations that i've tried:

config :my_app, MyApp.Endpoint,
  http: [
    port: port,
    # 3 min in ms
    timeout: 600_0000,
    protocol_options: [
      chunked: true,
      http10_keepalive: true,
      # 2 min in ms
      idle_timeout: 600_0000,
      inactivity_timeout: 600_0000,
      linger_timeout: 30_0000,
      # 2 min in ms
      request_timeout: 600_0000,
      # 3 min in ms
      # shutdown_timeout: 180_000,
      # 10 min in ms
      shutdown_timeout: 600_0000,
      # 5 MB in bytes

      # max_empty_lines (5)
      # Maximum number of empty lines before a request.
      max_empty_lines: 5000,

      # max_header_name_length (64)
      # Maximum length of header names.
      max_header_name_length: 64000,

      # max_header_value_length (4096)
      # Maximum length of header values.
      max_header_value_length: 4_096_000,

      # max_headers (100)
      # Maximum number of headers allowed per request.
      max_headers: 100_000,

      # max_keepalive (100)
      # Maximum number of requests allowed per connection.
      max_keepalive: 100_000,

      # max_method_length (32)
      # Maximum length of the method.
      max_method_length: 32000,
      max_skip_body_length: 20_000_000,
      max_request_line_length: 2_000_0000
    ]
  ]

Context

This library depends on cowboy ~> 2.7 which is using the deprecated :erlang.get_stacktrace/0. It happens that when an unexpected error raises, we could end up with this situation:

 Ranch protocol #PID<0.10445.0> of listener Wharever.Endpoint.HTTP (connection #PID<0.10444.0>, stream id 1) terminated
an exception was raised:
    ** (UndefinedFunctionError) function :erlang.get_stacktrace/0 is undefined or private
        :erlang.get_stacktrace()
        (cowboy 2.7.0) /builds/whatever/deps/cowboy/src/cowboy_stream_h.erl:314: :cowboy_stream_h.request_process/3
        (stdlib 3.17.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

We are in front of an outdated dependency that prevents us from using Phoenix with OTP 24, for example.

Proposed solution

I think it's time to upgrade this library, so:

• Upgrade Elixir to ~ 1.13
• Make it compatible with OTP to 24
• Upgrade Cowboy to >= 2.8, as it stops using the old way.

Let me know if it makes sense and I'll submit a PR with the changes.

Hi.
I put my plug on post request in router:

post("/upload", Project.Plugs.UploadHooks, :call)

There is a small part of plug:

  ## UploadHooks module
  def call(conn, _) do
    conn
    |> get_req_header("hook-name")
    |> trigger_hook(conn, conn.params)
  end

  ## PRE-CREATE hook clauses
  ## This event will be triggered before an upload is created.
  ## At this stage, verification and authorization of upload metadata is performed.
  defp trigger_hook(["pre-create"], conn, %{
         "Upload" => %{"MetaData" => %{"user_id" => user_id, "filename_token" => token}}
       }) do
    with {:ok, _} <- validate_user_id(user_id),
         {:ok, _} <- Druid.find_id_by_token(token),
         %User{} <- Accounts.find_user(user_id),
         {:ok, _} <- Resources.get_asset_by_token(token) do
      
      Logger.info("""
      #{inspect __MODULE__}[pre-create]:
      Upload allowed for asset #{inspect token}\
      """)
      send_resp(conn, 204, "OK")
    else
      e ->
        Logger.info("""
        #{inspect __MODULE__}[pre-create]:
        Upload not allowed - #{inspect e}
        """)
        send_resp(conn, 403, "Forbidden")
    end
  end

And this is how it works: the hook worked successfully and it looks like it will send 204 response, but after 2ms this exception was raised. The client receives 500 status response

Even if i do this then get the same:

conn
|> send_resp(204, "OK")
|> halt()

Current implementation of connection draining works in a way that make it stop creating new connections, but that do not prevent existing connections from accepting and processing new requests. That mean that single client that is connected to the node will prevent it from going down (potentially disrupting any other client that tries to connect to the given node).

I do not know the solution (maybe this issue should be forwarded to Ranch or Cowboy) but maybe there already exists some solution in these deps that would allow handling such problem.

I attempted to use master branch's plug and tried to install plug_cowboy.
However, I can not install it, and I get an error like the one below.

Failed to fetch record for 'hexpm/plug_cowboy' from registry (using cache)
This could be because the package does not exist, it was spelled incorrectly or you don't have permissions to it
** (Mix) No package with name plug_cowboy (from: mix.exs) in registry

Have not you registered to hex yet? Or should I not use it yet?

It took me a while to figure out that in order to use Plug.Cowboy.Drainer with phoenix endpoint one must use it like this:

# lib/myapp/application.ex
  def children do
    [
    # ...
      MyappWeb.Endpoint,
      {Plug.Cowboy.Drainer, refs: [MyappWeb.Endpoint.HTTP]},
    ]
  end

Would it make sense to mention this in README or Drainer docs?

I noticed that when compress is set to true, the order of the stream handlers is: [:cowboy_compress_h | @default_stream_handlers], where @default_stream_handlers is [:cowboy_telemetry_h, :cowboy_stream_h].

Does this work as I understand it in that the telemetry does not take into account the compression handler? Is this working as intended or should the compression be taken into account? I can imagine this is a breaking change but we could put it behind a flag or support compress: [metrics: true] or something like that.

The v2.5.0 git tag is missing from this repository so the view source button on hexdocs (https://hexdocs.pm/plug_cowboy/Plug.Cowboy.html) gives a 404 (https://github.com/elixir-plug/plug_cowboy/blob/v2.5.0/lib/plug/cowboy.ex#L1)

Has anyone tried using OpenTelemetry with Cowboy and a Plug? The idea is to have a root span start when cowboy starts a request and a child span for the plug duration. I know I can hook into telemetry events but I need to link the created spans together. What are the ways in which you could pass an OpenTelemetry context up to the plug, or is this simply not possible without forking / using ETS?

Asking for a friend ;) opentelemetry-beam/opentelemetry_phoenix#28

See: https://hex.pm/packages/plug_cowboy/2.3.0

According to the Cowboy documentation, a metrics_callback function can be passed to the options when starting a new Endpoint as follows:
https://ninenines.eu/docs/en/cowboy/2.4/manual/cowboy_http2/

config :my_app, MyApp.Endpoint,
  http: [port: 4000, 
         metrics_callback:  &metrics_callback/1,
         stream_handlers: [:cowboy_metrics_h, :cowboy_stream_h]]

but I get this error when I try to start it:

** (Mix) Could not start application free_license_server: MyApp.Application.start(:normal, []) returned an error: shutdown: failed to start child: MyApp.Endpoint
    ** (EXIT) shutdown: failed to start child: {:ranch_listener_sup, MyApp.Endpoint.HTTP}
        ** (EXIT) shutdown: failed to start child: :ranch_acceptors_sup
            ** (EXIT) :badarg

One fix was to to add :metrics_callback to the list of protocol_options

@protocol_options [:compress, :stream_handlers, :metrics_callback]
in the Plug.Cowboy module.

I hope that this issue makes sense.

I'm on ranch 1.5, so this might not be applicable to you.

I'm using Plug.Cowboy with a Phoenix project and can't confirm if this is caused by Phoenix, Plug.Cowboy, or my own local configuration. Phoenix on master and Plug.Cowboy on master.

1. Ranch expects the transport opts to be a proplist. They are currently being passed as a map.

2. After reworking plug_cowboy locally to convert the transport opts %{max_connections: 16384, num_acceptors: 100, socket_opts: [port: 4000]} to a Keyword list/proplist, I was getting a :badarg exception. I'm not sure where socket_opts is coming from (maybe Phoenix?), but I pulled the port config out of socket opts and was able to start without crashing.

Here is the specific exception in question:

[info] Application administration exited: Administration.Application.start(:normal, []) returned an error: shutdown: failed to start child: AdministrationWeb.Endpoint
    ** (EXIT) an exception was raised:
        ** (FunctionClauseError) no function clause matching in :proplists.get_value/3
            (stdlib) proplists.erl:215: :proplists.get_value(:num_acceptors, %{max_connections: 16384, num_acceptors: 100, socket_opts: [port: 4000]}, 10)
            (ranch) /data/repos/turret/deps/ranch/src/ranch.erl:116: :ranch.child_spec/5
            (plug_cowboy) lib/plug/cowboy.ex:185: Plug.Cowboy.child_spec/1
            (phoenix) lib/phoenix/endpoint/cowboy2_adapter.ex:44: Phoenix.Endpoint.Cowboy2Adapter.child_spec/3
            (phoenix) lib/phoenix/endpoint/supervisor.ex:106: anonymous fn/6 in Phoenix.Endpoint.Supervisor.server_children/4
            (elixir) lib/enum.ex:1925: Enum."-reduce/3-lists^foldl/2-0-"/3
            (phoenix) lib/phoenix/endpoint/supervisor.ex:97: Phoenix.Endpoint.Supervisor.server_children/4
            (phoenix) lib/phoenix/endpoint/supervisor.ex:57: Phoenix.Endpoint.Supervisor.init/1
            (stdlib) supervisor.erl:295: :supervisor.init/1
            (stdlib) gen_server.erl:374: :gen_server.init_it/2
            (stdlib) gen_server.erl:342: :gen_server.init_it/6
            (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

Hi there, we are running into a problem where the call to chunk/2 is hanging. I've traced it down to the following. Recently, cowboy2 added a backpressure mechanism for streaming content: ninenines/cowboy#1376

This new mechanism sends ack messages back to the caller process in order to provide the backpressure. As noted in the comments here (added as part of ninenines/cowboy@eaa0526) , it is no longer possible to send streaming data to cowboy2's handler from multiple processes:

https://github.com/ninenines/cowboy/blob/master/src/cowboy_stream_h.erl#L223

However, when we have a simple HTTP controller in phoenix which streams data it appears that it (unsurprisingly) sends to the same HTTP/2 cowboy handler from multiple request handle PIDs when requests come in concurrently through the same HTTP/2 transport. This breaks the contract, and is resulting in the backpressure mechanism hanging when enough data is pushed at once.

The temporary workaround (which may still fail I think) is to increase the new buffer size to a sufficient amount so all data is ack'ed immediately. I'm not sure how to resolve this properly, it seems like the cowboy plug would need to start multiplexing streamed data somehow through a new process 😬

Hi!

My app has been shutting down seemingly randomly. This only happens on my local machine (not on other devs, not in production environments) so I suspect it has something to do with my set up.

There are no logs when my app shuts down. It doesn't seem to correlate to any particular behavior. This has been happening for at least a couple weeks with regular daily development happening.

I finally got a clue by enabling SASL logging:

[error] Child Plug.Cowboy.Drainer of Supervisor MyApp.Endpoint shut down abnormally
** (exit) killed
Pid: #PID<0.1011.0>
Start Call: Plug.Cowboy.Drainer.start_link([refs: [MyAppWeb.Endpoint.HTTP]])
Restart: :permanent
Shutdown: 5000
Type: :worker

[notice] Application my_app exited: shutdown

Based on looking at what Plug.Cowboy.Drainer does, this seems like the error could be a red herring? Any ideas how I could get more diagnostic information? I highly suspect this is a memory limit or running out of file descriptors or something else specific to my environment but I can't seem to get any more clues.

I'm using plug_cowboy 2.6.0 with Phoenix 1.7.0-rc.2 on Elixir 1.14.3, OTP 25

The docs seem to state that you can pass any option from Plug.SSL.configure/1 (see this)

But in practice, that doesn't seem to work for me. I don't know enough about plug to know which rabbit hole to go down for this. Any help is appreciated.

iex)> opts = [
  certfile: "/path/to/my.crt",
  keyfile: "/path/to/my.key",
  host: "coolhost.dev",
  rewrite_on: [:x_forwarded_proto],
  port: 443
]

iex)> Plug.Cowboy.https MyApp.Router, [], opts
{:error,
 {{:shutdown, {:failed_to_start_child, :ranch_acceptors_sup, :badarg}},
  {:child, :undefined, {:ranch_listener_sup, MyApp.Router.HTTPS},
   {:ranch_listener_sup, :start_link,
    [
      MyApp.Router.HTTPS,
      :ranch_ssl,
      %{
        connection_type: :supervisor,
        max_connections: 16384,
        num_acceptors: 100,
        socket_opts: [
          next_protocols_advertised: ["h2", "http/1.1"],
          alpn_preferred_protocols: ["h2", "http/1.1"],
          reuse_sessions: true,
          secure_renegotiate: true,
          certfile: '/path/to/my.crt',
          keyfile: '/path/to/my.key',
          port: 443,
          host: "coolhost.dev",
          rewrite_on: [:x_forwarded_proto]
        ]
      },
      :cowboy_tls,
      %{
        connection_type: :supervisor,
        env: %{
          dispatch: [
            {:_, [],
             [{:_, [], Plug.Cowboy.Handler, {MyApp.Router, []}}]}
          ]
        },
        stream_handlers: [Plug.Cowboy.Stream]
      }
    ]}, :permanent, :infinity, :supervisor, [:ranch_listener_sup]}}}

Also fails when trying to use child_spec/1

iex)> spec = Plug.Cowboy.child_spec(plug: MyApp.Router, scheme: :https, options: opts)
iex)> DynamicSupervisor.start_child(MyApp.Supervisor, spec)
{:error, {:shutdown, {:failed_to_start_child, :ranch_acceptors_sup, :badarg}}}

The only options I can get to work are :certfile and :keyfile. This may be user-error and I might just need help understanding how to specify the SSL opts correctly?

> mix deps.compile
# ...
===> Compiling cowboy
src/cowboy_clear.erl:16:2: Warning: behaviour ranch_protocol undefined

src/cowboy_tls.erl:16:2: Warning: behaviour ranch_protocol undefined

===> Compiling src/cowboy_http.erl failed
src/cowboy_http.erl:{155,14}: can't find include lib "cowlib/include/cow_inline.hrl"; Make sure cowlib is in your app file's 'applications' list
src/cowboy_http.erl:{156,14}: can't find include lib "cowlib/include/cow_parse.hrl"; Make sure cowlib is in your app file's 'applications' list
src/cowboy_http.erl:512:11: undefined macro 'IS_TOKEN/1'
src/cowboy_http.erl:658:74: undefined macro 'IS_WS/1'

src/cowboy_http.erl:492:6: function parse_method/4 undefined
src/cowboy_http.erl:646:4: function parse_hd_name/4 undefined

src/cowboy_http.erl:517:1: Warning: function parse_uri/3 is unused
src/cowboy_http.erl:534:1: Warning: function parse_uri_authority/3 is unused
src/cowboy_http.erl:538:1: Warning: function parse_uri_authority/5 is unused
src/cowboy_http.erl:563:1: Warning: function parse_uri_path/5 is unused
src/cowboy_http.erl:573:1: Warning: function parse_uri_query/6 is unused
src/cowboy_http.erl:582:1: Warning: function skip_uri_fragment/6 is unused

Seems like calling up an "app file's 'applications' list" means I should be using:
https://hexdocs.pm/plug_cowboy/Plug.Cowboy.html#child_spec/1
inside lib/<app>/application.ex -

defmodule Pain do
  def start(_type, _args) do
    children = [
      # Start the Telemetry supervisor
      PainWeb.Telemetry,
      # Start the Ecto repository
      # Pain.Repo,
      # Start the PubSub system
      {Phoenix.PubSub, name: Pain.PubSub},
      # Start Finch
      {Finch, name: Pain.Finch},
      # Start the Endpoint (http/https)
      { Plug.Cowboy, scheme: :http, plug: PainWeb.Endpoint },
      PainWeb.Endpoint,
      # Start a worker by calling: Pain.Worker.start_link(arg)
      # {Pain.Worker, arg}
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: Pain.Supervisor]
    Supervisor.start_link(children, opts)
  end
  # ...
end

And here's config:

# config/config.exs
config :pain, PainWeb.Endpoint,
  url: [host: System.get_env("PHX_HOST") || "0.0.0.0"],
  adapter: Phoenix.Endpoint.Cowboy2Adapter,
  render_errors: [
    formats: [html: PainWeb.ErrorHTML, json: PainWeb.ErrorJSON],
    layout: false,
  ],
  pubsub_server: Pain.PubSub,
  live_view: [signing_salt: "aaaabbbb"]
# ...

# config/prod.exs
import Config
config :pain, PainWeb.Endpoint,
  server: true,
  http: [ ip: {0, 0, 0, 0}, port: System.get_env("PORT") ],
  cache_static_manifest: "priv/static/cache_manifest.json"
# ...

👋🏼 Howdy!

In NervesHub we use websockets for connections with devices and utilize mTLS with them. We wanted to the ability to prevent socket connections as low as possible with some rate limiting and were enabled to do this with Thousand Island in a past PR.

However, due to some recently discovered problems with socket disconnects, we've had to switch back to cowboy until that can be further investigated. While switching, I found that :ranch does a :ranch_transport behavior that can be implemented, but after doing so we realized there is no easy way to use that with :cowboy or Plug.Cowboy via Phoenix.

The end result is a somewhat hack replicating the Phoenix.Endpoint.Cowboy2Adapter and injecting our ranch_transport module into the specific child_spec generated.

Is it desirable to have that functionality either added here or in Phoenix.Endpoint.Cowboy2Adapter? I'd be happy to add, but don't want to do the work if not wanted. We can simply leave this small change in until figuring out how Bandit/Thousand Island can be adjusted for our needs and switch back

I'm currently trying to use the latest opentelemetry_phoenix, which depends on telemetry ~> 1.0 via opentelemetry_telemetry ~> 1.0.

plug_cowboy is currently my blocker, since it depends on cowboy_telemetry ~> 0.3. cowboy_telemetry version 0.4.0 added support for telemetry ~> 1.0

I have a problem in my web application: some clients don't wait for HTTP response and disconnect early. As a result, the erlang process executing this request on my server terminates instantly, while I need to do some resource cleanup when it happens.

So I need some callback to be called on my Plug to notify me of client closing connection before termination.

Please advise if it's already possible, thanks!

Currently Plug.Cowboy.Translator only logs exceptions that were raised in the Plug pipeline only if the Plug.Exception.status/1 for the error returns a status code greater than or equal to 500:

Would you be open for making this behaviour configurable? My thinking is that while implementing Plug.Exception is a great way to customize status codes (and in case of Phoenix, rendered responses), it's not always desirable to swallow the exception completely.

cowboy version lower than 2.8.0 is using a deprecated_function erlang, get_stacktrace, making all error log to undef error:
Ranch protocol #PID<0.409.0> of listener Plug.Cowboy.TranslatorTest.HTTP (connection #PID<0.408.0>, stream id 1) terminated\n** (exit) :undef

error reproducing:
plug_cowboy: Release v2.5.2
erlang otp: 24
mix test test/plug/cowboy/translator_test.exs

There was a recent change to remove the timeout config option:

• c267211

If this option is still specified, it's now causing the option to get passed to the underlying transport, where it causes a badarg error when starting ranch...

I'm wondering if we should do more to protect against this... drop the option or raise an error?

@josevalim

10:20:40 18:20:40.815 [error] Child :ranch_acceptors_sup of Supervisor #PID<0.637.0> (:ranch_listener_sup) failed to start
10:20:40 ** (exit) :badarg
10:20:40 Start Call: :ranch_acceptors_sup.start_link(Main.HttpServer.HTTP, :ranch_tcp)
10:20:40 
10:20:40 18:20:40.815 [error] Process #PID<0.639.0> terminating
10:20:40 ** (exit) :badarg
10:20:40     (kernel) inet_tcp.erl:142: :inet_tcp.listen/2
10:20:40     (ranch) /work/src/source.datanerd.us/after/nerd-graph/deps/ranch/src/ranch_acceptors_sup.erl:39: :ranch_acceptors_sup.init/1
10:20:40     (stdlib) supervisor.erl:295: :supervisor.init/1
10:20:40     (stdlib) gen_server.erl:374: :gen_server.init_it/2
10:20:40     (stdlib) gen_server.erl:342: :gen_server.init_it/6
10:20:40     (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
10:20:40 Initial Call: :ranch_acceptors_sup.init/1
10:20:40 Ancestors: [#PID<0.637.0>, Main.Supervisor, #PID<0.634.0>]
10:20:40 
10:20:40 18:20:40.823 [error] Child {:ranch_listener_sup, Main.HttpServer.HTTP} of Supervisor Main.Supervisor failed to start
10:20:40 ** (exit) shutdown: failed to start child: :ranch_acceptors_sup
10:20:40     ** (EXIT) :badarg
10:20:40 Start Call: :ranch_listener_sup.start_link(Main.HttpServer.HTTP, :ranch_tcp, %{max_connections: 16384, num_acceptors: 100, socket_opts: [port: 3001, timeout: :infinity]}, :cowboy_clear, %{compress: true, env: %{dispatch: [{:_, [], [{:_, [], Plug.Cowboy.Handler, {Main.HttpServer, []}}]}]}, idle_timeout: 300000, max_header_value_length: 8192, stream_handlers: [:cowboy_compress_h, Plug.Cowboy.Stream]})
10:20:40 
10:20:40 18:20:40.823 [error] Process #PID<0.633.0> terminating
10:20:40 ** (exit) exited in: Main.Application.start(:normal, [])
10:20:40     ** (EXIT) shutdown: failed to start child: {:ranch_listener_sup, Main.HttpServer.HTTP}
10:20:40         ** (EXIT) shutdown: failed to start child: :ranch_acceptors_sup
10:20:40             ** (EXIT) :badarg
10:20:40     (kernel) application_master.erl:138: :application_master.init/4
10:20:40     (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
10:20:40 Initial Call: :application_master.init/4
10:20:40 Ancestors: [#PID<0.632.0>]
10:20:40 ** (Mix) Could not start application main: Main.Application.start(:normal, []) returned an error: shutdown: failed to start child: {:ranch_listener_sup, Main.HttpServer.HTTP}
10:20:40     ** (EXIT) shutdown: failed to start child: :ranch_acceptors_sup
10:20:40         ** (EXIT) :badarg

Looks like the release v2.6.1 is cut on non main branch. The changelog at the master is still for v2.6.0. Some code changes are missing as well.

Latest hex package is v2.6.1: https://diff.hex.pm/diff/plug_cowboy/2.6.0..2.6.1.

Continuing discussion from phoenixframework/phoenix#3869...

Current plug telemetry handlers for :stop and :exception are susceptible to being terminated during their execution because the cowboy process can exit at any time after the request is complete.

This may be avoided by leveraging the built-in handler system of cowboy, especially cowboy_metrics_h which provides a way to execute a callback at the end of each request with stats about its execution.

Wiring up the additional handler is trivial, the thing to figure out is what data from it to expose, and what to name the telemetry event...

https://ninenines.eu/docs/en/cowboy/2.8/manual/cowboy_metrics_h/

• Should we all the data directly or transform it so we aren't exposing raw cowboy data structures?

• Is [:plug_cowboy, :metrics] a reasonable name since it's providing summarized metrics (ie: not start and stop for spans)? Or [:plug_cowboy, :terminate] since it's called in the terminate phase of the cowboy process

In order to handle WebSocket connections (or have any other custom cowboy2 handler) alongside using a Plug.Router to route other http requests, I have to manually configure Cowboy's dispatch using the :dispatch option.

It's great that you have provided this option! 👍

However, it is only briefly referred to here and there is no mention of the Plug.Adapters.Cowboy.Handler module which is necessary to configure Plug.Router manually. This makes me wonder if doing this is supported functionality?

It would be nice to have the example:

dispatch:     [
      {^hostname, [
        {"/to_custom_handler", WebServerApp.CustomHandler, []},
        {:_, Plug.Adapters.Cowboy.Handler, {WebServerApp.Router, []}}
      ]}
    ]

Thanks 😺 - PS The docs may very well already contain this, if so - my mistake 👍

I'm in the process of upgrading an app relying on plug_cowboy version 1.0.0.

The changelog (https://github.com/elixir-plug/plug_cowboy/blob/master/CHANGELOG.md) does not mention anything related to 1.0.0.

Are there breaking changes between 1.0.0 and the rest of the upgrades?

I have investigated and cannot see much changes (v1.0.0...v2.0.0) but would prefer to be sure!

Thank you (I can create a doc PR with my findings to help others).