holsee / chroxy Goto Github PK

View Code? Open in Web Editor NEW

253.0 7.0 27.0 340 KB

Headless Chrome as a Service

Home Page: https://hex.pm/packages/chroxy

License: MIT License

Elixir 96.42% Dockerfile 3.58%

elixir chrome headless-chrome browser-automation

chroxy's People

Contributors

Stargazers

Watchers

chroxy's Issues

Update Elixir & Erlang Versions used in CI

Update: https://github.com/holsee/chroxy/blob/master/.travis.yml

Bad address

I'm trying to get chroxy running in a docker container but am running into the following error on startup. I've confirmed that chrome is on the machine and runs without an issue.

headless@3f7c2ef65f01:~/chroxy-0.3.2$ mix run --no-halt

02:53:39.180 pid=<0.212.0> module=Chroxy.Application [info]  Started application

02:53:39.202 pid=<0.214.0> module=Chroxy.ProxyListener [debug] Listening on port: 1331

02:53:39.225 pid=<0.216.0> module=Chroxy.ChromeManager [warn]  ARGS: [{Chroxy.ChromeServer, [page_wait_ms: "200", crash_dumps_dir: "/tmp", verbose_logging: 0]}, {Chroxy.ProxyServer, [packet_trace: false]}, {Chroxy.Endpoint, [scheme: :http, port: "1330"]}, {:included_applications, []}, {Chroxy.ProxyListener, [host: "127.0.0.1", port: "1331"]}, {:chrome_remote_debug_port_from, "9222"}, {:chrome_remote_debug_port_to, "9223"}]

02:53:39.482 pid=<0.217.0> module=Chroxy.ChromeServer [info]  [CHROME: 477] stdout: "Pid 477: cannot execute '(null)': Bad address"

02:53:39.709 pid=<0.322.0> module=Chroxy.ChromeServer [info]  [CHROME: 478] stdout: "Pid 478: cannot execute '(null)': Bad address"

02:53:39.945 pid=<0.324.0> module=Chroxy.ChromeServer [info]  [CHROME: 479] stdout: "Pid 479: cannot execute '(null)': Bad address"

02:53:40.173 pid=<0.326.0> module=Chroxy.ChromeServer [info]  [CHROME: 480] stdout: "Pid 480: cannot execute '(null)': Bad address"

02:53:40.247 pid=<0.216.0> module=Chroxy.ChromeServer [error] ChromeServer #PID<0.217.0> did not reply to ready request

02:53:40.247 pid=<0.216.0> module=Chroxy.ChromeManager [error] Failed to start chrome on port 9222

Add OTP and Elixir version to logs

On app start, it would be useful to log the OTP and Elixir version to help with debugging issues relating to versions of the VM and language.

Is this similar to Puppeteer?

Is this library similar to Puppeteer - https://pptr.dev/ - in terms of the protocol it uses?

I'm working on a project with a forked version of Chroxy. There are all sorts of reasons why we are using a fork but ideally we want to use the main line code base going forward. One issue is that we have a bunch of flags set for Chrome here's what our config looks like (there are even more flags in our config) 😆

chrome_flags: [
    "--user-agent=\"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36\"",
    "--disable-dev-shm-usage",
    "--disable-features=VizDisplayCompositor",
    "--enable-features=NetworkService,NetworkServiceInProcess",
    "--headless",
    "--disable-gpu",
    "--no-sandbox",
    "--ignore-certificate-errors",
    "--incognito",
    "--allow-running-insecure-content",
    "--v8-cache-options=off",
    "--aggressive-cache-discard",
    "--disable-cache",
    "--disable-application-cache",
    "--disable-offline-load-stale-cache",
    "--no-first-run"
  ]

Also we were planning on using the Docker version of Chroxy. As such, I'm trying to figure out how to make these flags configurable with the app and docker.

Deprecated time units used

12:46:03.637 pid=<0.342.0> module=Chroxy.ChromeServer [info]  [CHROME: 93] "DevTools listening on ws://127.0.0.1:9223/devtools/browser/15596190-f1d1-4888-911b-38ca580d1225"

12:46:23.060 pid=<0.347.0> module=Plug.Logger [info]  GET /api/v1/connection
warning: deprecated time unit: :nanoseconds. A time unit should be :second, :millisecond, :microsecond, :nanosecond, or a positive integer
  (plug) lib/plug/request_id.ex:60: Plug.RequestId.generate_request_id/0
  (plug) lib/plug/request_id.ex:48: Plug.RequestId.get_request_id/2
  (plug) lib/plug/request_id.ex:42: Plug.RequestId.call/2
  (chroxy) lib/chroxy/endpoint.ex:1: Chroxy.Endpoint.plug_builder_call/2


12:46:23.303 pid=<0.348.0> module=Chroxy.ChromeProxy [debug] Obtained new page from ChromeServer [#PID<0.236.0>]

12:46:23.303 pid=<0.341.0> module=Chroxy.ProxyRouter [debug] put object with key: B6E2D84F83211AC9667C2057ED47D952 - value: #PID<0.348.0>
warning: deprecated time unit: :micro_seconds. A time unit should be :second, :millisecond, :microsecond, :nanosecond, or a positive integer
  (plug) lib/plug/logger.ex:36: anonymous fn/2 in Plug.Logger.call/2
  (logger) lib/logger.ex:867: Logger.normalize_message/2
  (logger) lib/logger.ex:690: Logger.__do_log__/3
  (plug) lib/plug/logger.ex:34: anonymous fn/3 in Plug.Logger.call/2

Crashes after some time

I deployed 0.4.2 to production but it crashes after sometime it crashed. The relevant logs - https://gist.github.com/j-manu/fa27bd11d1f336144f50557b7855538a

Can't run with Elixir 1.8.1 & OTP 21

15:26:32.873 [info]  Application chroxy exited: exited in: Chroxy.Application.start(:normal, [])
    ** (EXIT) an exception was raised:
        ** (ArgumentError) argument error
            :erlang.binary_to_integer(nil)
            (chroxy) lib/chroxy/endpoint.ex:11: Chroxy.Endpoint.child_spec/0
            (chroxy) lib/chroxy/application.ex:20: Chroxy.Application.start/2
            (kernel) application_master.erl:277: :application_master.start_it_old/4
** (Mix) Could not start application chroxy: exited in: Chroxy.Application.start(:normal, [])
    ** (EXIT) an exception was raised:
        ** (ArgumentError) argument error
            :erlang.binary_to_integer(nil)
            (chroxy) lib/chroxy/endpoint.ex:11: Chroxy.Endpoint.child_spec/0
            (chroxy) lib/chroxy/application.ex:20: Chroxy.Application.start/2
            (kernel) application_master.erl:277: :application_master.start_it_old/4

Number and weight of chrome instances can grow large

We have had chroxy running continuously on a machine for a long time - weeks or months.
We noticed that the machine was getting low on memory.

Running ps -o pid,user,%mem,rss,lim,command --sort %mem ax | less (on Linux - the equivalent on MacOS is ps -o pid,user,%mem,rss,lim,command -me) showed that the processes consuming most of the memory were chrome instances.

Restarting chroxy dramatically reduced both the number and weight of the chrome processes.
Before restart, there were 52 chrome processes using 4.4GB of memory.
After restart, there were 11 chrome processes using 666MB of memory.

I don't know a way to reproduce this situation and, in our case, it's reasonable to restart chroxy periodically during known periods of low activity. But I thought it was worth reporting.

Supported Versions List

I am working on listing the support elixir / OTP versions and updating the CI config to reflect this.

The targets will be:

Latest Elixir (1.11) to 1.8
OTP 21+

This unit of work has been triggered by cowboy update which broke elixir 1.7 / older OTP support.

Support Chromium and configurable Chrome Path

Short term goal would be to support configuration of the path (as per other configuration through environment variables).

This function would need to be modified:

chroxy/lib/chroxy/chrome_server.ex

Lines 347 to 354 in a1997bd

 defp chrome_path do 

 case :os.type() do 

 {:unix, :darwin} -> 

 "/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome" 

 {:unix, _} -> 

 "/usr/bin/google-chrome" 

 end

From early testing, Chromium was fully compatible.

Log upstream and downstream connection information to help debug client issues.

Also put info level logging that is noise into debug level.

Docker container for Chroxy

We want to be able to run Chroxy within a docker container out of the box, so adding Dockerfile to this project would be a good start.

Address is use

When performing basic stress testing the following error occurs:

10:27:31.339 request_id=Ff2tuFY7vDKkyNYAAEki pid=<0.2931.0> module=Plug.Logger [info]  Sent 200 in 202ms

10:27:31.340 pid=<0.2627.0> module=gen_server [error] GenServer Chroxy.ProxyListener terminating
** (CaseClauseError) no case clause matching: {:error, :emfile}
    (chroxy 0.6.3) lib/chroxy/proxy_listener.ex:85: Chroxy.ProxyListener.handle_cast/2
    (stdlib 3.9) gen_server.erl:637: :gen_server.try_dispatch/4
    (stdlib 3.9) gen_server.erl:711: :gen_server.handle_msg/6
    (stdlib 3.9) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_cast", {:accept, [dyn_hook: #Function<0.68892742/1 in Chroxy.ChromeProxy.handle_call/3>]}}
State: %{listen_socket: #Port<0.1125>}

10:27:31.340 pid=<0.2933.0> module=Plug.Logger [info]  GET /api/v1/connection

10:27:31.341 pid=<0.2935.0> module=Chroxy.ProxyListener [error] TCP Listen failed due to: :eaddrinuse

When the Chroxy.ProxyListener fails to obtain address to bind to, within a race condition between the restart of the tcp listener and the clean-up of the native resource after the crash, then the system will enter a bad state where Chrome page sessions will be created as part of the http request for a page session to chroxy, but the websocket proxy will not be able to be created.

Proxy configuration per-connection

Hi! Are there plans to have a new mode which allows consumers to specify their own proxy configuration? I don't think it's possible to specify proxy configuration on a per-tab basis.

I believe you eluded to it here #5 (comment); I wanted to get the conversation rolling around implementation ideas.

Load balancing / Chroxy

I was wondering if Chroxy was designed to coordinate requests across multiple physical instances of machines running Chroxy?

Intermittent 500's

I'm seeing random and intermittent failures when trying to interface with Chroxy. I'm testing in a isolated environment with plenty of resources and currently have 8 available port ranges for chrome.

When running one off requests I will get a handful of 500 errors and then a handful of successful responses.

The log of the failed response is:


21:29:36.171 pid=<0.791.0> module=Plug.Logger [info]  GET /api/v1/connection

21:29:36.171 pid=<0.216.0> module=Chroxy.ChromeManager [info]  Selected chrome server: {:undefined, #PID<0.326.0>, :worker, [Chroxy.ChromeServer]}

21:29:37.175 pid=<0.792.0> module=Chroxy.ChromeProxy [debug] Obtained new page from ChromeServer [#PID<0.326.0>]

21:29:37.175 request_id=2ksqq0mmjsj7v5fgks000181 pid=<0.791.0> module=Plug.Logger [info]  Sent 200 in 1003ms

21:29:37.178 pid=<0.214.0> module=Chroxy.ProxyListener [info]  Connection accepted, spawning proxy server to manage connection

21:29:37.180 pid=<0.793.0> module=Chroxy.ProxyServer [debug] Downstream connection established

21:29:37.181 pid=<0.793.0> module=Chroxy.ProxyServer [debug] Upstream socket closed, terminating proxy

21:29:37.181 pid=<0.654.0> module=Chroxy.ChromeProxy [info]  Proxy connection down - closing page

21:29:37.181 pid=<0.216.0> module=Chroxy.ChromeManager [info]  ChromeManager linked process #PID<0.654.0> exited with reason: :normal

Remove entry from `ProxyRouter` when page is closed

The down/2 callback hook in ChromeProxy should remove the entry for the page from the ProxyRouter (ets table) when page is closed.

Chroxy to Cowboy 2.x

Add support for Cowboy 2.x, originally raised in #17 by @bglusman

Installation in a Phoenix project

Chroxy can't be installed in a recent Phoenix project as Cowboy version has been bumped to 2.7 and chroxy has a dependency on Cowboy 2.6.3.

Refresh README and link to video

The README needs some love and the following tutorial covers using Chroxy
https://www.youtube.com/watch?v=FGUEDu8A7GY

Bump Deps

mix hex.outdated

Dependency               Current  Latest  Update possible
chrome_remote_interface  0.1.0    0.2.0   No
cowboy                   1.1.2    2.6.3   No
erlexec                  1.9.5    1.10.0  No
ex_doc                   0.19.3   0.20.2  Yes
exexec                   0.1.0    0.2.0   Yes
jason                    1.1.2    1.1.2
plug                     1.7.2    1.8.0   Yes
plug_cowboy              1.0.0    2.0.2   No

This should be split into 3.

Low risk (jason, ex_doc, chrome_remote_interface)
Native Port Drivers (erl_exec, exexec)
Plug & Cowboy

Cant run with elixir 1.6.4 and erlang 20.3

    ** (EXIT) an exception was raised:
        ** (ArgumentError) argument error
            :erlang.binary_to_integer(nil)
            (chroxy) lib/chroxy/endpoint.ex:11: Chroxy.Endpoint.child_spec/0
            (chroxy) lib/chroxy/application.ex:29: Chroxy.Application.start/2
            (kernel) application_master.erl:273: :application_master.start_it_old/4```

Support Firefox remote connections

https://wiki.mozilla.org/Remote
https://developer.mozilla.org/en-US/docs/Tools/Remote_Debugging/Debugging_Firefox_Desktop
https://firefox-source-docs.mozilla.org/remote/index.html

https://wiki.mozilla.org/WebDriver/Meetings/2018-11-26

Spawn Connections Without Using The API

This can be optional. i.e those who want to do it with API can continue doing that. Have the proxy listen on another port (say 1332) and when the the client connects to ws://localhost:1332/ and the proxy server spawns a new connection (instead of the API server) and it continues from there.

Here is how I envisioned it.

Proxy listens for a connection
For a NEW connection, it spawns a new ChromeProxy instance.
The connection is handed over to the ChromeProxy instance.
From this point on it is just like current architecture

This makes it easier to load balance multiple chroxy instancers because it will be a single connection. Currently it takes two connections (one to api and one ws) and those can go to different instances. The only way to solve it from what I see is to ensure that a client machine always gets routes to the same chroxy instance but that means that a single client machine can only use one chroxy instance

ChroxyClient 0.2.0 issue via ChromeRemoteInterface Issue

Due to ChromeRemoteInterface having an incomplete protocol file in version 0.2.0, Chroxy Client 0.2.0 has been impacted. See: holsee/chroxy_client#2

This will be resolved and new versions pushed to hex when ChromeRemoteInterface merges the PR.

StackTrace on requesting new PageSession

We're having an issue with Chroxy whereby it eventually gets into a state where it fails to estbalish the connection properly anymore.

We are running Chroxy locally on the box as a separate service.
We will make a call for a new Page sessions like so:
ChroxyClient.page_session!(%{host: "localhost", port: 1330})

When everything is working fine the Chroxy logs look like this:

15:26:17.067 pid=<0.416.0> module=Plug.Logger [info]  GET /api/v1/connection
15:26:17.299 pid=<0.417.0> module=Chroxy.ChromeProxy [debug] Obtained new page from ChromeServer [#PID<0.301.0>]
15:26:17.299 pid=<0.406.0> module=Chroxy.ProxyRouter [debug] put object with key: 09641A45CD7EF339E90AC9536C1C6352 - value: #PID<0.417.0>
15:26:17.301 request_id=FfTq1rmOxGgNfRUAAABC pid=<0.416.0> module=Plug.Logger [info]  Sent 200 in 233ms
15:26:17.304 pid=<0.297.0> module=Chroxy.ProxyListener [info]  Connection accepted, spawning proxy server to manage connection
15:26:17.309 pid=<0.418.0> module=Chroxy.ProxyServer [debug] Resolved dynamic hook `%{mod: Chroxy.ChromeProxy, ref: #PID<0.417.0>}` for ProxyServer
15:26:17.311 pid=<0.418.0> module=Chroxy.ProxyServer [debug] Downstream connection established

But eventually we get in a bad state, and the Logs we get back look like this:

21:25:41.828 pid=<0.9502.0> module=Plug.Logger [info]  GET /api/v1/connection
21:25:42.032 pid=<0.9503.0> module=Chroxy.ChromeProxy [debug] Obtained new page from ChromeServer [#PID<0.9043.0>]
21:25:42.032 pid=<0.1328.0> module=Chroxy.ProxyRouter [debug] put object with key: 4212795C766AB0B83DFC89E687F4A0CB - value: #PID<0.9503.0>
21:25:42.033 pid=<0.8763.0> module=Chroxy.ProxyListener [info]  Connection accepted, spawning proxy server to manage connection
21:25:42.033 request_id=FfTqzoTtW2ZgnYUAAAri pid=<0.9502.0> module=Plug.Logger [info]  Sent 200 in 205ms
21:25:42.033 pid=<0.9504.0> module=Chroxy.ProxyServer [debug] Resolved dynamic hook `%{mod: Chroxy.ChromeProxy, ref: #PID<0.9498.0>}` for ProxyServer
21:25:42.034 pid=<0.9504.0> module=Chroxy.ProxyServer [debug] Downstream connection established
21:25:42.034 pid=<0.9504.0> module=Chroxy.ProxyServer [debug] Upstream socket closed, terminating proxy
21:25:42.034 pid=<0.9498.0> module=Chroxy.ChromeProxy [info]  Proxy connection down - closing page
21:25:42.034 pid=<0.1328.0> module=Chroxy.ProxyRouter [debug] deleting object with key: C2230CCF04096F06B0AEF1A50F9AE6FA
21:25:42.034 pid=<0.1328.0> module=Chroxy.ProxyRouter [debug] received DOWN message for: #Reference<0.3639764170.3576692738.192501>

and the Client code eventually gets back a stack trace:

21:25:42.034 [debug] WebSocket: ws://127.0.0.1:1331/devtools/page/4212795C766AB0B83DFC89E687F4A0CB
** (EXIT from #PID<0.2498.0>) shell process exited with reason: an exception was raised:
    ** (MatchError) no match of right hand side value: {:error, %WebSockex.ConnError{original: :timeout}}
        (chrome_remote_interface) lib/page_session.ex:123: ChromeRemoteInterface.PageSession.init/1
        (stdlib) gen_server.erl:374: :gen_server.init_it/2
        (stdlib) gen_server.erl:342: :gen_server.init_it/6
        (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

As soon as the Downstream connection is established, we immediately get the message saying that the Upstream socket closed, and then it shuts everything down. I've poured through the code, but I just don't have any idea what's causing this.

If we shut down the entire Chroxy Application and restart it, then it goes back to working as normal. If we shut down and restart the client Application, it makes no difference.

I'm really up against a wall here so any help you could offer would be appreciated.

how can we use it puppeteer?

Any suggestions? Puppetter Api

Protocol.UndefinedError on ChromeServer termination

When running Chroxy with root permissions (which is not supported see the Dockerfile for example how to avoid doing so), you can see the issue with ChromeServer process termination reason being logged as follows:

web_1       | 12:33:59.391 pid=<0.2084.0> module=gen_server line=888 [error] GenServer #PID<0.2084.0> terminating
web_1       | ** (Protocol.UndefinedError) protocol String.Chars not implemented for {{:exit_status, 4}, {:gen_server, :call, [:exec, {:port, {{:run, '/usr/bin/google-chrome --headless --disable-gpu --disable-translate --disable-extensions --disable-background-networking --safebrowsing-disable-auto-update --enable-logging --disable-sync --metrics-recording-only --disable-default-apps --mute-audio --no-first-run --no-sandbox --incognito --remote-debugging-port=9222 --crash-dumps-dir= --v=0', [:pty, :stderr, :stdin, :stdout]}, :link, false}}, 30000]}}. This protocol is implemented for: Postgrex.Query, Postgrex.Copy, Decimal, Integer, Float, Version.Requirement, Time, NaiveDateTime, DateTime, Date, BitString, List, Atom, Version, URI
web_1       |     (elixir) /usr/local/src/elixir/lib/elixir/lib/string/chars.ex:3: String.Chars.impl_for!/1
web_1       |     (elixir) /usr/local/src/elixir/lib/elixir/lib/string/chars.ex:22: String.Chars.to_string/1
web_1       |     (chroxy) lib/chroxy/chrome_server.ex:126: Chroxy.ChromeServer.terminate/2
web_1       |     (stdlib) gen_server.erl:673: :gen_server.try_terminate/3
web_1       |     (stdlib) gen_server.erl:858: :gen_server.terminate/10
web_1       |     (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
web_1       | Last message: :launch

Support for Cowboy 2?

Right now Chroxy is locked on Cowboy 1.x, which blocks updating to Cowboy 2 with Phoenix 1.4... guessing that could be relaxed to support both?

Unable to run any session programmatically.

So the issue I'm having is, if I run the commands inside an IEX session everything works as expected, however if I call this GenServer inside another module / plug request it timesout.

edit: NVM I was causing a deadlock in my code.

	defp chrome_path do
	case :os.type() do
	{:unix, :darwin} ->
	"/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome"

	{:unix, _} ->
	"/usr/bin/google-chrome"
	end

holsee / chroxy Goto Github PK

chroxy's People

Contributors

Stargazers

Watchers

Forkers

chroxy's Issues

Recommend Projects

Recommend Topics

Recommend Org