akira / exq_ui Goto Github PK

As an end user, I want to enqueue scheduled jobs from the UI

Acceptance Criteria

• Add "Enqueue" button in scheduled jobs UI view
• Clicking on "Enqueue" button, queues scheduled job

If you are interested in becoming a maintainer, please reach out to @akira! Thanks!

Following https://gist.github.com/chrismccord/bb1f8b136f5a9e4abc0bfc07b832257e

While updating phoenix to 1.4 I'm switching cowboy to plug_cowboy but that requires cowboy ~>2.5

Failed to use "cowboy" (versions 2.5.0 to 2.6.1) because
  exq_ui (version 0.9.0) requires ~> 1.0
  plug_cowboy (versions 2.0.0 and 2.0.1) requires ~> 2.5

Interactive Elixir (1.3.0-dev) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> time = Timex.Date.from({{2017, 9, 5}, {15, 54, 0}}) |> Timex.Date.to_timestamp
{1504, 569600, 0}
iex(2)> {:ok, ack} = Exq.enqueue_at(Exq, "bots", time, Sentinel.Execution, ["arg1", "arg2"])
{:ok, "c2e61709-04b5-4983-bea1-7c64a5cd2589"}
iex(3)>
15:57:48.763 [error] Ranch listener ExqUi.RouterPlug.HTTP had connection process started with :cowboy_protocol:start_link/4 at #PID<0.395.0> exit with reason: {{%ArgumentError{message: "argument error"}, [{:erlang, :binary_to_float, ["1504569600"], []}, {ExqUi.RouterPlug.Router, :score_to_time, 1, [file: 'web/router.ex', line: 199]}, {ExqUi.RouterPlug.Router, :"-map_score_to_jobs/1-fun-0-", 1, [file: 'web/router.ex', line: 209]}, {Enum, :"-map/2-lists^map/1-0-", 2, [file: 'lib/enum.ex', line:
1092]}, {ExqUi.RouterPlug.Router, :"-do_match/4-fun-5-", 1, [file: 'web/router.ex', line: 80]}, {ExqUi.RouterPlug.Router, :plug_builder_call, 2, [file: 'web/router.ex', line: 34]}, {Plug.Adapters.Cowboy.Handler, :upgrade, 4, [file: 'lib/plug/adapters/cowboy/handler.ex', line: 15]}, {:cowboy_protocol, :execute, 4, [file: 'src/cowboy_protocol.erl', line: 442]}]}, {ExqUi.RouterPlug, :call, [%Plug.Conn{adapter: {Plug.Adapters.Cowboy.Conn, :...}, assigns: %{}, before_send: [], body_params:
%Plug.Conn.Unfetched{aspect: :body_params}, cookies: %Plug.Conn.Unfetched{aspect: :cookies}, halted: false, host: "localhost", method: "GET", owner: #PID<0.395.0>, params: %Plug.Conn.Unfetched{aspect: :params}, path_info: ["api", "scheduled"], peer: {{127, 0, 0, 1}, 48621}, port: 4040, private: %{}, query_params: %Plug.Conn.Unfetched{aspect: :query_params}, query_string: "", remote_ip: {127, 0, 0, 1}, req_cookies: %Plug.Conn.Unfetched{aspect: :cookies}, req_headers: [{"host", "localhost:4040"},
{"connection", "keep-alive"}, {"accept", "application/json, text/javascript, /; q=0.01"}, {"x-requested-with", "XMLHttpRequest"}, {"user-agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"}, {"referer", "http://localhost:4040/"}, {"accept-encoding", "gzip, deflate, sdch"}, {"accept-language", "en-US,en;q=0.8,pt;q=0.6,he;q=0.4"}, {"cookie", "_ga=GA1.1.975442743.1458598151"}], request_path: "/api/scheduled", resp_body: nil,
resp_cookies: %{}, resp_headers: [{"cache-control", "max-age=0, private, must-revalidate"}], scheme: :http, script_name: [], secret_key_base: nil, state: :unset, status: nil}, [exq_opts: [name: Exq.Api], namespace: ""]]}}

We should provide a first class way to auth. For now i think basic auth is fine, but rather than rely on nginx or an upstream plug we should bake it in.

Moved over from akira/exq#46

Clicking on "clear worker list" does not seem to reload after clearing.

A module called JsonApi in the file web/json_api.ex is clashing with JSONAPI in case insensitive FS.

14:45:27.182 [error] Loading of /otp-app/_build/test/lib/exq_ui/ebin/Elixir.JSONAPI.beam failed: :badfile

14:45:27.182 [error] beam/beam_load.c(1428): Error loading module 'Elixir.JSONAPI':
  module name in object code is Elixir.JsonApi

It would be nice to be able to clear jobs out of the retry list manually after they have passed there fail limit. The problem that I'm am running into is if a worker failed and won't actually be able to be retried successfully, it will always be in the retry list and on restart of my elixir application when exq boots up it is runs again only to fail. There is most likely a better way to solve this, but currently I have been flushing redis to clear out the jobs.

At the moment the only option available in Failures view is to clear failed jobs queue.

It would have been useful to be able to restart them (individually or all of them).

https://monosnap.com/file/KJAiY6SQGorG2TpV0VWJEPFKlWNGkT - says all I think

Will try to fix it later 😄

This is not a big, but more of a doubt. There are several jobs scheduled under multiple queues, but the size has a value other than zero only for one of the queues. Also, the size is bigger than the total number of scheduled jobs. Can someone tell me why is that happening and what exactly is this size?

Exq_ui doesn't contain the assets anymore since 7a02ab5, so the web admin has no longer the needed static assets (js and css) to render properly.

Also building the JS distribution manually myself fails, since ember-validations is no longer found on the given url "http://builds.dockyard.com.s3.amazonaws.com/ember-validations/ember-validations-latest.js"

It seems that exq_ui is using a fairly outdated version of poison. It would be really good if it could support more recent versions, because recent versions of dependencies like poison are used in many other packages, making it a hard choice to make to pick exq_ui.

Seems like chart has broken at some point, figure out what's not working

Hi, I'm having difficulty in accessing exq_ui once it's deployed, although it's working properly using my local computer. I checked the logs to see if there are any error but didn't find any. Hope you could help me. I've been stuck here since Friday, May 4, 2018.

Server Log

Erlang/OTP 20 [erts-9.3] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:10] [hipe] [kernel-poll:false]^M
^M
Starting ExqUI on Port 4009^M
06:21:22.007 [info] Running WorkerWeb.Endpoint with Cowboy using http://:::4008^M

netstat -ntl doesn't display 4009 port.

Currently, units on the left Y axis are being displayed on a scale from 0 to 1. This should actually indicate the count of jobs instead.

Right now the chart updates but we should also be updating the stats at the top of the page with the latest stats.

Moved from akira/exq#49

When I click on Workers (/exq/processes) I often see this broken table:

And sometimes it's not broken but it's missing data:

Other tables like the one for Scheduled looks fine, with all its data.

ExqUI is still using Poison. We should convert it to use custom serializer used by Exq.

See akira/exq#362

Hi there,

I had ExqUI working before, but now I'm getting 404's when it tries to load the assets (/assets/vendor.css, /assets/exqui.css, /assets/vendor.js, /assets/exqui.js). The HTML page is being served correctly (I can go to /exq in my app), it's just failing to load the assets.

Any idea what might be causing this?

I have HTTP basic auth set up on accessing /exq, but this was working previously.

This is part of a Phoenix app. I have the following in config.exs:

config :exq_ui, server: false

And this in my router:

  pipeline :exq do
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_flash
    # plug :protect_from_forgery ## Docs show pipeline w/o CSRF, Exq UI fails if include this
    plug :put_secure_browser_headers
    plug ExqUi.RouterPlug, namespace: "exq"
  end

  scope "/exq", ExqUi do
    pipe_through [:http_basic_auth, :exq]
    forward "/", RouterPlug.Router, :index
  end

Is there a way to prevent the exq_ui server from being started when running mix test? Phoenix for example provides the server: false option in your Endpoint config.

I've followed the guides on https://github.com/akira/exq_ui#using-with-plug and I have exq_ui working with this configuration:

# *** config.ex ***

config :exq_ui,
  server: true

# *** router.ex ***

  pipeline :admin do
    plug(BasicAuth, use_config: {:myapp, :admin_auth})
  end

  pipeline :exq do
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_flash
    plug :put_secure_browser_headers
    plug ExqUi.RouterPlug, namespace: "admin/exq"
  end

  scope "/admin/exq", ExqUi do
    pipe_through [:admin, :exq]
    forward "/", RouterPlug.Router, :index
  end

This works fine and I can access http://localhost:4000/admin/exq and I'm prompted for my credentials, all fine.

But... I've found that I can also access http://localhost:4040 and I won't be prompted for credentials.

Is there a way to disable access and only enable it via the configured route?

The following two lines:

plug Exq.RouterPlug, namespace: "exq"
...
forward "/exq", Exq.RouterPlug.Router, :index

Should read (missing "Ui"):

plug ExqUi.RouterPlug, namespace: "exq"
...
forward "/exq", ExqUi.RouterPlug.Router, :index

I have run into a strange race condition that is hard to duplicate, but I believe has to do with exq_ui starting exq.

I am running exq with start_on_application: false and starting it in my application just like the docs say. supervisor(Exq, []). I am doing this so it is starting after my ecto repo.

What I have seen happen is: a job is enqueued immediately after my server comes online. The job shows up in the retry column of the exq web UI. There are no console logs, and no logs in the web ui as to why it failed. If I click the 'Retry' button, I see the ecto sandbox error in my console, similar to:

10:55:45.507 [error] Process #PID<0.388.0> raised an exception
** (DBConnection.OwnershipError) cannot find ownership process for #PID<0.388.0>.

When using ownership, you must manage connections in one
of the three ways:

  * By explicitly checking out a connection
  * By explicitly allowing a spawned process
  * By running the pool in shared mode

If I kick off another job, or even retry this one again it will usually work.

I have put some log statements in my exq and exq_ui startup sequence and I see the following

• my application boots
• exq_ui application starts
• exq_ui starts exq due to: https://github.com/akira/exq_ui/blob/master/lib/exq_ui.ex#L17
• exq starts AGAIN due to my config: supervisor(Exq, [])

At this point there are 2 separate Exq's running. I suspect 1 of is started before Ecto was started, and so it picks up the job and it continues to fail. Though I am not sure why it would fail silently like it does.

This is very difficult to replicate, and I cannot do it reliably. But fundamentally, should exq_ui be calling start_link on exq?

I'm using the docker image elixir:1.8 and i'm getting the following warming.

warning: Plug.Adapters.Cowboy2 is deprecated, please use Plug.Cowboy instead
(plug) lib/plug/adapters/cowboy2.ex:12: Plug.Adapters.Cowboy2.http/3
(exq_ui) lib/exq_ui.ex:24: ExqUi.launch_app/0
(kernel) application_master.erl:277: :application_master.start_it_old/4

First of all, thanks for your work on exq and exq_ui, it's really great so far! I do have one hangup, though. I'm using exrm for deployment, since that was recommended, and I have exq and exq_ui in my applications function, also as recommended. However, when I try to start the server, I get:

Application exq exited: Exq.start(:normal, []) returned an error: already started: #PID<0.1350.0>

I've tried all permutations of including/disincluding exq and exq_ui as well as different configs to no effect. Can you provide any help?

When trying to start exq_ui with the following configs:

  config :exq,
    start_on_application: false

  config :exq_ui,
    web_port: 4040,
    web_namespace: "",
    server: false

The app crashes when starting:

21:40:02.619 [error] Failed to connect to Redis (127.0.0.1:6379): connection refused
21:40:02.632 [error] GenServer ExqUi.Api terminating
** (MatchError) no match of right hand side value: {:error, %Redix.ConnectionError{reason: :closed}}
    (exq) lib/exq/redis/connection.ex:63: Exq.Redis.Connection.smembers!/2
    (exq) lib/exq/redis/job_queue.ex:296: Exq.Redis.JobQueue.queue_size/2
    (exq) lib/exq/api/server.ex:84: Exq.Api.Server.handle_call/3
    (stdlib) gen_server.erl:661: :gen_server.try_handle_call/4
    (stdlib) gen_server.erl:690: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
Last message (from #PID<0.1811.0>): :queue_size
State: %Exq.Api.Server.State{namespace: "exq", redis: ExqUi.Redis.Client}
Client #PID<0.1811.0> is alive

    (stdlib) gen.erl:167: :gen.do_call/4
    (elixir) lib/gen_server.ex:1006: GenServer.call/3
    (exq_ui) lib/exq_ui.ex:20: ExqUi.launch_app/0
    (kernel) application_master.erl:277: :application_master.start_it_old/4

I am using exq_ui along phoenix. The workers are doing their thing and if I go to exq_ui i see the stats getting updated (16 processed, 13 failed, etc).

The names of the queues are also listed.

But I can't see any, for example, the failures. The failures table is empty even though the stats say that '13 failed'.

Any ideas on what to check?

Thanks in advance.

As an end user, I want to re-enqueue dead jobs in the UI

Acceptance Criteria

• Add "Requeue" button in dead job UI view for each job
• Clicking on "Requeue" button, requeues dead jobs

Hi! I have an issue with exq_ui package. When I try to access the dashboard, I am getting the erlang gen_server error: :noproc. I am running the page through Plug. When I run it on a separate port with mix exq.ui, everything works ok.

What is strange is that everything worked ok until a few days ago when it stopped. I have no idea how to solve this.

Here is the whole stack trace:

Elixir.ErlangError erlang error: {:noproc, {GenServer, :call, [ExqUi.Api, {:stats, "processed"}, 5000]}} 
    lib/gen_server.ex:596 GenServer.call/3
    web/router.ex:45 anonymous fn/1 in ExqUi.RouterPlug.Router.do_match/4
    web/router.ex:34 ExqUi.RouterPlug.Router.plug_builder_call/2
    lib/phoenix/router/route.ex:154 Phoenix.Router.Route.forward/4
    lib/phoenix/router.ex:261 MyApp.Router.dispatch/2
    web/router.ex:1 MyApp.Router.do_call/2
    lib/plug/error_handler.ex:64 MyApp.Router.call/2
    lib/my_app/endpoint.ex:1 MyApp.Endpoint.phoenix_pipeline/1

We use (and love) exq for our Elixir/Ember application. I've noticed that you're having trouble maintaining the Ember version upgrades, etc. Would you like a PR to upgrade it further to the latest ember LTS version? (3.4)? Just want to make sure someone isn't already working on this.

As an end user, I want to re-enqueue jobs in the "Retry" queue in the UI

Acceptance Criteria

• Add "Requeue" button in retry queue UI view for each job
• Clicking on "Requeue" button, requeues retry jobs right away

Hi,

Is there any way to avoid [Object object] in the args column? Actually, this object is just a simple map:

%{id: "something"}

But I cannot see the id because the UI give this:

Any hack to fix it?

Thank you for your time

At the moment UI truncates backtrace in failures and they are useless.

Seems that whole backtrace gets transmitted over the API, so it is purely UI thing.

I suggest adding a link will allow to unwrap the whole backtrace if they are truncated.

Currently this is written in Ember, but it does need continual updating and maintenance given that the dependent libraries are being deprecated / updated.

As a maintainer, I don't have much Ember experience and am personally more familiar with libraries such as React.js.

I am considering rewrite of the UI to React.js / Redux / React Router.

Alternatives considered:

• Vue.js / Vuex / Vue Router
• Clojurescript (with Reagent or Om.next).

However, given that React.js seems the most widely used, I am inclined toward React.js and Redux.

Hi there!

I had previously been lazy and was running my Phoenix app in production using mix to launch directly on the console:

MIX_ENV=prod mix phoenix.server

Exq and ExqUi both work great with this method. I am trying to package an Exrm release, though, and it compiles fine but then dies when I try to load a page:

** (exit) an exception was raised:
    ** (UndefinedFunctionError) undefined function: ExqUi.RouterPlug.call/2 (module ExqUi.RouterPlug is not available)

I have no idea why the behavior would be any different. Does anything jump out at you as to why it wouldn't see the ExqUi.RouterPlug module? I would think it's because I might need to add :exq_ui to my application list, but when I do that the app crashes because it tries to start ExqUi twice.

I got an error when starting the Web UI using this command:
$ mix exq.ui

  def application do
    [
      applications: [:exq_ui, :phoenix, :phoenix_ecto, :exq],
      extra_applications: [:logger]
    ]
  end

  defp deps do
    [
      {:phoenix, "~> 1.5.4"},
      {:phoenix_ecto, "~> 4.1"},
      {:exq, "~> 0.14.0"},
      {:jason, "~> 1.0"},
      {:exq_ui, "~> 0.11.0"},
      {:plug_cowboy, "~> 2.0"}
    ]
  end

This is the error:

15:31:56.880 [warn]  Failed to lookup telemetry handlers. Ensure the telemetry application has been started.
Starting ExqUI on Port 4040

15:31:56.880 [warn]  Failed to lookup telemetry handlers. Ensure the telemetry application has been started.
** (exit) exited in: :gen_server.call(:telemetry_handler_table, {:insert, :plug_cowboy, [[:cowboy, :request, :early_error]], #Function<1.55428787/4 in Plug.Cowboy."-fun.handle_event/4-">, nil})
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (stdlib 3.13) gen_server.erl:238: :gen_server.call/2
    lib/plug/cowboy.ex:233: Plug.Cowboy.run/4
    lib/exq_ui.ex:24: ExqUi.launch_app/0
    lib/mix/tasks/exq.ui.ex:7: Mix.Tasks.Exq.Ui.run/1
    (mix 1.10.4) lib/mix/task.ex:330: Mix.Task.run_task/3
    (mix 1.10.4) lib/mix/cli.ex:82: Mix.CLI.run_task/2

I don't want to change global logger level, possible to silence just exq_ui logger?

[info] GET /exq/api/realtimes [info] Sent 200 in 587µs [info] GET /exq/api/realtimes [info] Sent 200 in 599µs [info] GET /exq/api/realtimes [info] Sent 200 in 540µs
This is flooding console every second.

Expand tests for router.ex (https://github.com/akira/exq/blob/master/web/router.ex).

Add cases for each type of endpoint along with data expected.

I upgraded to Phoenix 1.4 with webpack. And it's not fetching css and js exq assets. So the ember app is not loading.

Doing curl http://localhost:4000/exq/assets/vendor.js from terminal returns this

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <title>Exqui</title>
    <meta name="description" content="">
    <meta name="viewport" content="width=device-width, initial-scale=1">


<meta name="exqui/config/environment" content="......." />

    <base href="/exq/" />

    <script>
      window.exqNamespace = "exq/";
    </script>

    <link rel="stylesheet" href="assets/vendor.css">
    <link rel="stylesheet" href="assets/exqui.css">


  </head>
  <body>


    <script src="assets/vendor.js"></script>
    <script src="assets/exqui.js"></script>


  <iframe src="/phoenix/live_reload/frame" style="display: none;"></iframe>
</body>
</html>

Seems like something changed in routing static assets. Any ideas?

Edit: I'm running off master d3f2a51

Is there a way to view this dashboard in a deployed Heroku instance? Thanks! =)

As an end user, I want to delete scheduled job in the UI

Acceptance Criteria

• Add "Delete" button in scheduled jobs UI view for each job
• Clicking on "Delete" button, removes job from scheduled queue

Hi,
It's possible to quite queue like sidekiq?

Moved from
akira/exq#51

Is it normal that the graph on Dashboard looks like this, using only one colour?

I'm using the default view (I didn't follow the instructions to set up Ember).

As an end user, I want to delete jobs from the "Retry" queue via the UI

Acceptance Criteria

• Add "Delete" button in retry jobs UI view
• Clicking on "Delete" button, deletes jobs

When I click "Enqueued" or "Queues" I got this error:

ember.debug.js:29019 Error while processing route: queues.index Assertion Failed: You must include an 'id' for queue in an object passed to 'push' Error: Assertion Failed: You must include an 'id' for queue in an object passed to 'push'

ember.debug.js:18008 Error: Assertion Failed: You must include an 'id' for queue in an object passed to 'push'

There is no error, if there is no enqueued tasks.

It seems like even with these silent configs:

config :exq, start_on_application: false
config :exq_ui, server: false

exq_ui raises %Redix.Error{message: "NOAUTH Authentication required."} exception when I have authentication on a local redis setup. It happens due to this check https://github.com/akira/exq_ui/blob/master/lib/exq_ui.ex#L22 appears too late. It should be at least before this line https://github.com/akira/exq_ui/blob/master/lib/exq_ui.ex#L20 as it goes to redis and passes it some commands, which for sure need provide a password.
The main problem it brings is the inability to run any CI scripts on CI providers, GH Actions, and so on.

I am using the new version of Phoenix:

     {:phoenix, "~> 1.2.0-rc"},
     {:phoenix_ecto, "~> 3.0.0-rc.0"},
     {:phoenix_html, "~> 2.5.1"},
     {:phoenix_pubsub, "~> 1.0.0-rc"},

With exq_ui, I get this error:

Router.Helpers is not loaded and could not be found