holsee / chroxy Goto Github PK
View Code? Open in Web Editor NEWHeadless Chrome as a Service
Home Page: https://hex.pm/packages/chroxy
License: MIT License
Headless Chrome as a Service
Home Page: https://hex.pm/packages/chroxy
License: MIT License
I'm trying to get chroxy running in a docker container but am running into the following error on startup. I've confirmed that chrome is on the machine and runs without an issue.
headless@3f7c2ef65f01:~/chroxy-0.3.2$ mix run --no-halt
02:53:39.180 pid=<0.212.0> module=Chroxy.Application [info] Started application
02:53:39.202 pid=<0.214.0> module=Chroxy.ProxyListener [debug] Listening on port: 1331
02:53:39.225 pid=<0.216.0> module=Chroxy.ChromeManager [warn] ARGS: [{Chroxy.ChromeServer, [page_wait_ms: "200", crash_dumps_dir: "/tmp", verbose_logging: 0]}, {Chroxy.ProxyServer, [packet_trace: false]}, {Chroxy.Endpoint, [scheme: :http, port: "1330"]}, {:included_applications, []}, {Chroxy.ProxyListener, [host: "127.0.0.1", port: "1331"]}, {:chrome_remote_debug_port_from, "9222"}, {:chrome_remote_debug_port_to, "9223"}]
02:53:39.482 pid=<0.217.0> module=Chroxy.ChromeServer [info] [CHROME: 477] stdout: "Pid 477: cannot execute '(null)': Bad address"
02:53:39.709 pid=<0.322.0> module=Chroxy.ChromeServer [info] [CHROME: 478] stdout: "Pid 478: cannot execute '(null)': Bad address"
02:53:39.945 pid=<0.324.0> module=Chroxy.ChromeServer [info] [CHROME: 479] stdout: "Pid 479: cannot execute '(null)': Bad address"
02:53:40.173 pid=<0.326.0> module=Chroxy.ChromeServer [info] [CHROME: 480] stdout: "Pid 480: cannot execute '(null)': Bad address"
02:53:40.247 pid=<0.216.0> module=Chroxy.ChromeServer [error] ChromeServer #PID<0.217.0> did not reply to ready request
02:53:40.247 pid=<0.216.0> module=Chroxy.ChromeManager [error] Failed to start chrome on port 9222
On app start, it would be useful to log the OTP and Elixir version to help with debugging issues relating to versions of the VM and language.
Is this library similar to Puppeteer - https://pptr.dev/ - in terms of the protocol it uses?
I'm working on a project with a forked version of Chroxy. There are all sorts of reasons why we are using a fork but ideally we want to use the main line code base going forward. One issue is that we have a bunch of flags set for Chrome here's what our config looks like (there are even more flags in our config) ๐
chrome_flags: [
"--user-agent=\"Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36\"",
"--disable-dev-shm-usage",
"--disable-features=VizDisplayCompositor",
"--enable-features=NetworkService,NetworkServiceInProcess",
"--headless",
"--disable-gpu",
"--no-sandbox",
"--ignore-certificate-errors",
"--incognito",
"--allow-running-insecure-content",
"--v8-cache-options=off",
"--aggressive-cache-discard",
"--disable-cache",
"--disable-application-cache",
"--disable-offline-load-stale-cache",
"--no-first-run"
]
Also we were planning on using the Docker version of Chroxy. As such, I'm trying to figure out how to make these flags configurable with the app and docker.
12:46:03.637 pid=<0.342.0> module=Chroxy.ChromeServer [info] [CHROME: 93] "DevTools listening on ws://127.0.0.1:9223/devtools/browser/15596190-f1d1-4888-911b-38ca580d1225"
12:46:23.060 pid=<0.347.0> module=Plug.Logger [info] GET /api/v1/connection
warning: deprecated time unit: :nanoseconds. A time unit should be :second, :millisecond, :microsecond, :nanosecond, or a positive integer
(plug) lib/plug/request_id.ex:60: Plug.RequestId.generate_request_id/0
(plug) lib/plug/request_id.ex:48: Plug.RequestId.get_request_id/2
(plug) lib/plug/request_id.ex:42: Plug.RequestId.call/2
(chroxy) lib/chroxy/endpoint.ex:1: Chroxy.Endpoint.plug_builder_call/2
12:46:23.303 pid=<0.348.0> module=Chroxy.ChromeProxy [debug] Obtained new page from ChromeServer [#PID<0.236.0>]
12:46:23.303 pid=<0.341.0> module=Chroxy.ProxyRouter [debug] put object with key: B6E2D84F83211AC9667C2057ED47D952 - value: #PID<0.348.0>
warning: deprecated time unit: :micro_seconds. A time unit should be :second, :millisecond, :microsecond, :nanosecond, or a positive integer
(plug) lib/plug/logger.ex:36: anonymous fn/2 in Plug.Logger.call/2
(logger) lib/logger.ex:867: Logger.normalize_message/2
(logger) lib/logger.ex:690: Logger.__do_log__/3
(plug) lib/plug/logger.ex:34: anonymous fn/3 in Plug.Logger.call/2
I deployed 0.4.2 to production but it crashes after sometime it crashed. The relevant logs - https://gist.github.com/j-manu/fa27bd11d1f336144f50557b7855538a
15:26:32.873 [info] Application chroxy exited: exited in: Chroxy.Application.start(:normal, [])
** (EXIT) an exception was raised:
** (ArgumentError) argument error
:erlang.binary_to_integer(nil)
(chroxy) lib/chroxy/endpoint.ex:11: Chroxy.Endpoint.child_spec/0
(chroxy) lib/chroxy/application.ex:20: Chroxy.Application.start/2
(kernel) application_master.erl:277: :application_master.start_it_old/4
** (Mix) Could not start application chroxy: exited in: Chroxy.Application.start(:normal, [])
** (EXIT) an exception was raised:
** (ArgumentError) argument error
:erlang.binary_to_integer(nil)
(chroxy) lib/chroxy/endpoint.ex:11: Chroxy.Endpoint.child_spec/0
(chroxy) lib/chroxy/application.ex:20: Chroxy.Application.start/2
(kernel) application_master.erl:277: :application_master.start_it_old/4
We have had chroxy running continuously on a machine for a long time - weeks or months.
We noticed that the machine was getting low on memory.
Running ps -o pid,user,%mem,rss,lim,command --sort %mem ax | less
(on Linux - the equivalent on MacOS is ps -o pid,user,%mem,rss,lim,command -me
) showed that the processes consuming most of the memory were chrome instances.
Restarting chroxy dramatically reduced both the number and weight of the chrome processes.
Before restart, there were 52 chrome processes using 4.4GB of memory.
After restart, there were 11 chrome processes using 666MB of memory.
I don't know a way to reproduce this situation and, in our case, it's reasonable to restart chroxy periodically during known periods of low activity. But I thought it was worth reporting.
I am working on listing the support elixir / OTP versions and updating the CI config to reflect this.
The targets will be:
Latest Elixir (1.11) to 1.8
OTP 21+
This unit of work has been triggered by cowboy update which broke elixir 1.7 / older OTP support.
Short term goal would be to support configuration of the path (as per other configuration through environment variables).
This function would need to be modified:
chroxy/lib/chroxy/chrome_server.ex
Lines 347 to 354 in a1997bd
From early testing, Chromium was fully compatible.
info
level logging that is noise into debug
level.We want to be able to run Chroxy within a docker container out of the box, so adding Dockerfile to this project would be a good start.
When performing basic stress testing the following error occurs:
10:27:31.339 request_id=Ff2tuFY7vDKkyNYAAEki pid=<0.2931.0> module=Plug.Logger [info] Sent 200 in 202ms
10:27:31.340 pid=<0.2627.0> module=gen_server [error] GenServer Chroxy.ProxyListener terminating
** (CaseClauseError) no case clause matching: {:error, :emfile}
(chroxy 0.6.3) lib/chroxy/proxy_listener.ex:85: Chroxy.ProxyListener.handle_cast/2
(stdlib 3.9) gen_server.erl:637: :gen_server.try_dispatch/4
(stdlib 3.9) gen_server.erl:711: :gen_server.handle_msg/6
(stdlib 3.9) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_cast", {:accept, [dyn_hook: #Function<0.68892742/1 in Chroxy.ChromeProxy.handle_call/3>]}}
State: %{listen_socket: #Port<0.1125>}
10:27:31.340 pid=<0.2933.0> module=Plug.Logger [info] GET /api/v1/connection
10:27:31.341 pid=<0.2935.0> module=Chroxy.ProxyListener [error] TCP Listen failed due to: :eaddrinuse
When the Chroxy.ProxyListener
fails to obtain address to bind to, within a race condition between the restart of the tcp listener and the clean-up of the native resource after the crash, then the system will enter a bad state where Chrome page sessions will be created as part of the http request for a page session to chroxy, but the websocket proxy will not be able to be created.
Hi! Are there plans to have a new mode which allows consumers to specify their own proxy configuration? I don't think it's possible to specify proxy configuration on a per-tab basis.
I believe you eluded to it here #5 (comment); I wanted to get the conversation rolling around implementation ideas.
I was wondering if Chroxy was designed to coordinate requests across multiple physical instances of machines running Chroxy?
I'm seeing random and intermittent failures when trying to interface with Chroxy. I'm testing in a isolated environment with plenty of resources and currently have 8 available port ranges for chrome.
When running one off requests I will get a handful of 500 errors and then a handful of successful responses.
The log of the failed response is:
21:29:36.171 pid=<0.791.0> module=Plug.Logger [info] GET /api/v1/connection
21:29:36.171 pid=<0.216.0> module=Chroxy.ChromeManager [info] Selected chrome server: {:undefined, #PID<0.326.0>, :worker, [Chroxy.ChromeServer]}
21:29:37.175 pid=<0.792.0> module=Chroxy.ChromeProxy [debug] Obtained new page from ChromeServer [#PID<0.326.0>]
21:29:37.175 request_id=2ksqq0mmjsj7v5fgks000181 pid=<0.791.0> module=Plug.Logger [info] Sent 200 in 1003ms
21:29:37.178 pid=<0.214.0> module=Chroxy.ProxyListener [info] Connection accepted, spawning proxy server to manage connection
21:29:37.180 pid=<0.793.0> module=Chroxy.ProxyServer [debug] Downstream connection established
21:29:37.181 pid=<0.793.0> module=Chroxy.ProxyServer [debug] Upstream socket closed, terminating proxy
21:29:37.181 pid=<0.654.0> module=Chroxy.ChromeProxy [info] Proxy connection down - closing page
21:29:37.181 pid=<0.216.0> module=Chroxy.ChromeManager [info] ChromeManager linked process #PID<0.654.0> exited with reason: :normal
The down/2
callback hook in ChromeProxy
should remove the entry for the page from the ProxyRouter
(ets table) when page is closed.
Chroxy can't be installed in a recent Phoenix project as Cowboy version has been bumped to 2.7 and chroxy has a dependency on Cowboy 2.6.3.
The README needs some love and the following tutorial covers using Chroxy
https://www.youtube.com/watch?v=FGUEDu8A7GY
mix hex.outdated
Dependency Current Latest Update possible
chrome_remote_interface 0.1.0 0.2.0 No
cowboy 1.1.2 2.6.3 No
erlexec 1.9.5 1.10.0 No
ex_doc 0.19.3 0.20.2 Yes
exexec 0.1.0 0.2.0 Yes
jason 1.1.2 1.1.2
plug 1.7.2 1.8.0 Yes
plug_cowboy 1.0.0 2.0.2 No
This should be split into 3.
** (EXIT) an exception was raised:
** (ArgumentError) argument error
:erlang.binary_to_integer(nil)
(chroxy) lib/chroxy/endpoint.ex:11: Chroxy.Endpoint.child_spec/0
(chroxy) lib/chroxy/application.ex:29: Chroxy.Application.start/2
(kernel) application_master.erl:273: :application_master.start_it_old/4```
This can be optional. i.e those who want to do it with API can continue doing that. Have the proxy listen on another port (say 1332) and when the the client connects to ws://localhost:1332/ and the proxy server spawns a new connection (instead of the API server) and it continues from there.
Here is how I envisioned it.
This makes it easier to load balance multiple chroxy instancers because it will be a single connection. Currently it takes two connections (one to api and one ws) and those can go to different instances. The only way to solve it from what I see is to ensure that a client machine always gets routes to the same chroxy instance but that means that a single client machine can only use one chroxy instance
Due to ChromeRemoteInterface having an incomplete protocol file in version 0.2.0, Chroxy Client 0.2.0 has been impacted. See: holsee/chroxy_client#2
This will be resolved and new versions pushed to hex when ChromeRemoteInterface merges the PR.
We're having an issue with Chroxy whereby it eventually gets into a state where it fails to estbalish the connection properly anymore.
We are running Chroxy locally on the box as a separate service.
We will make a call for a new Page sessions like so:
ChroxyClient.page_session!(%{host: "localhost", port: 1330})
When everything is working fine the Chroxy logs look like this:
15:26:17.067 pid=<0.416.0> module=Plug.Logger [info] GET /api/v1/connection
15:26:17.299 pid=<0.417.0> module=Chroxy.ChromeProxy [debug] Obtained new page from ChromeServer [#PID<0.301.0>]
15:26:17.299 pid=<0.406.0> module=Chroxy.ProxyRouter [debug] put object with key: 09641A45CD7EF339E90AC9536C1C6352 - value: #PID<0.417.0>
15:26:17.301 request_id=FfTq1rmOxGgNfRUAAABC pid=<0.416.0> module=Plug.Logger [info] Sent 200 in 233ms
15:26:17.304 pid=<0.297.0> module=Chroxy.ProxyListener [info] Connection accepted, spawning proxy server to manage connection
15:26:17.309 pid=<0.418.0> module=Chroxy.ProxyServer [debug] Resolved dynamic hook `%{mod: Chroxy.ChromeProxy, ref: #PID<0.417.0>}` for ProxyServer
15:26:17.311 pid=<0.418.0> module=Chroxy.ProxyServer [debug] Downstream connection established
But eventually we get in a bad state, and the Logs we get back look like this:
21:25:41.828 pid=<0.9502.0> module=Plug.Logger [info] GET /api/v1/connection
21:25:42.032 pid=<0.9503.0> module=Chroxy.ChromeProxy [debug] Obtained new page from ChromeServer [#PID<0.9043.0>]
21:25:42.032 pid=<0.1328.0> module=Chroxy.ProxyRouter [debug] put object with key: 4212795C766AB0B83DFC89E687F4A0CB - value: #PID<0.9503.0>
21:25:42.033 pid=<0.8763.0> module=Chroxy.ProxyListener [info] Connection accepted, spawning proxy server to manage connection
21:25:42.033 request_id=FfTqzoTtW2ZgnYUAAAri pid=<0.9502.0> module=Plug.Logger [info] Sent 200 in 205ms
21:25:42.033 pid=<0.9504.0> module=Chroxy.ProxyServer [debug] Resolved dynamic hook `%{mod: Chroxy.ChromeProxy, ref: #PID<0.9498.0>}` for ProxyServer
21:25:42.034 pid=<0.9504.0> module=Chroxy.ProxyServer [debug] Downstream connection established
21:25:42.034 pid=<0.9504.0> module=Chroxy.ProxyServer [debug] Upstream socket closed, terminating proxy
21:25:42.034 pid=<0.9498.0> module=Chroxy.ChromeProxy [info] Proxy connection down - closing page
21:25:42.034 pid=<0.1328.0> module=Chroxy.ProxyRouter [debug] deleting object with key: C2230CCF04096F06B0AEF1A50F9AE6FA
21:25:42.034 pid=<0.1328.0> module=Chroxy.ProxyRouter [debug] received DOWN message for: #Reference<0.3639764170.3576692738.192501>
and the Client code eventually gets back a stack trace:
21:25:42.034 [debug] WebSocket: ws://127.0.0.1:1331/devtools/page/4212795C766AB0B83DFC89E687F4A0CB
** (EXIT from #PID<0.2498.0>) shell process exited with reason: an exception was raised:
** (MatchError) no match of right hand side value: {:error, %WebSockex.ConnError{original: :timeout}}
(chrome_remote_interface) lib/page_session.ex:123: ChromeRemoteInterface.PageSession.init/1
(stdlib) gen_server.erl:374: :gen_server.init_it/2
(stdlib) gen_server.erl:342: :gen_server.init_it/6
(stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
As soon as the Downstream connection is established, we immediately get the message saying that the Upstream socket closed, and then it shuts everything down. I've poured through the code, but I just don't have any idea what's causing this.
If we shut down the entire Chroxy Application and restart it, then it goes back to working as normal. If we shut down and restart the client Application, it makes no difference.
I'm really up against a wall here so any help you could offer would be appreciated.
Any suggestions? Puppetter Api
When running Chroxy with root permissions (which is not supported see the Dockerfile for example how to avoid doing so), you can see the issue with ChromeServer
process termination reason
being logged as follows:
web_1 | 12:33:59.391 pid=<0.2084.0> module=gen_server line=888 [error] GenServer #PID<0.2084.0> terminating
web_1 | ** (Protocol.UndefinedError) protocol String.Chars not implemented for {{:exit_status, 4}, {:gen_server, :call, [:exec, {:port, {{:run, '/usr/bin/google-chrome --headless --disable-gpu --disable-translate --disable-extensions --disable-background-networking --safebrowsing-disable-auto-update --enable-logging --disable-sync --metrics-recording-only --disable-default-apps --mute-audio --no-first-run --no-sandbox --incognito --remote-debugging-port=9222 --crash-dumps-dir= --v=0', [:pty, :stderr, :stdin, :stdout]}, :link, false}}, 30000]}}. This protocol is implemented for: Postgrex.Query, Postgrex.Copy, Decimal, Integer, Float, Version.Requirement, Time, NaiveDateTime, DateTime, Date, BitString, List, Atom, Version, URI
web_1 | (elixir) /usr/local/src/elixir/lib/elixir/lib/string/chars.ex:3: String.Chars.impl_for!/1
web_1 | (elixir) /usr/local/src/elixir/lib/elixir/lib/string/chars.ex:22: String.Chars.to_string/1
web_1 | (chroxy) lib/chroxy/chrome_server.ex:126: Chroxy.ChromeServer.terminate/2
web_1 | (stdlib) gen_server.erl:673: :gen_server.try_terminate/3
web_1 | (stdlib) gen_server.erl:858: :gen_server.terminate/10
web_1 | (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
web_1 | Last message: :launch
Right now Chroxy is locked on Cowboy 1.x, which blocks updating to Cowboy 2 with Phoenix 1.4... guessing that could be relaxed to support both?
So the issue I'm having is, if I run the commands inside an IEX session everything works as expected, however if I call this GenServer inside another module / plug request it timesout.
edit: NVM I was causing a deadlock in my code.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.