edgurgel / verk_web Goto Github PK

#84 here is the PR

Since Endpoint is all macros it checks :authorization config during verk_web compile time. Since compiling dependency happens only once, changing config after dependencies have been compiled has no effect. I think the reasonable solution would be to have a plug that checks the config on init. BasicAuth does exactly that, but because of https://github.com/edgurgel/verk_web/blob/master/lib/verk_web/endpoint.ex#L39 it is not even loaded.

Is there a way to customize mounting options? Trying to integrate it into phoenix app.
Would be nice to protect Verk UI with simple http auth or something

In exq_ui i can simply add necessary stuff via pipelines:
https://github.com/akira/exq_ui#using-with-plug

Initially reported here: #62

This is more to start a discussion if this is a wanted feature.

Here is my current implementation, that so far looks promising...

• https://github.com/ruby2elixir/verk_web/tree/mount_plug

The inspiration came from:

• http://neuvians.io/2016/03/15/using-a-phoenix-app-as-a-dependency/
• https://github.com/maxneuvians/phoenix_dependency_test

this problem is not properly solved yet:

• https://www.bountysource.com/issues/23491390-correctly-scope-static-file-paths-generated-in-forwarded-endpoints

What was done:

• check-in generated static files into Git (simpler deployment)
• provide the plug for mounting
• adjust readme
• remove VerkWeb.Endpoint from application to start

It depends on my fork of Verk atm, that's why it makes no sense to open a pull request...
@edgurge what are your thoughts about this feature?

It's hard to know how long ago a job was started/enqueued by looking only the UTC time.

Similar to edgurgel/verk#54

Hi! Love Verk. Question. Is there a way to mount the verk route under a protected scope as opposed to using basic auth? We use pow to protect routes and would like to put the /verk route under an authenticated scope instead if basic auth.

something like:

  scope "/",  do
    pipe_through [:browser, :protected]
    
    forward("/verk", VerkWeb.MountRoute)
  end

Hi,

The newest release of phoenix (including cowboy2) no longer has the method Phoenix.Router.Route.forward/4. I receive the following error trying to redirect to /verk.

function Phoenix.Router.Route.forward/4 is undefined or private

It is happening inside the plug/mount.ex file in the defp forward method.

EDIT: For now, I got around it by adding this to my router directly.

  scope "/", VerkWeb do
    forward("/verk", Endpoint)
  end

Hello,

Thanks for writing this very useful library!

I have a use case where I want to manually run the scheduled job earlier than the scheduled time through the interface.

I imagine a button next to the details button on the scheduled jobs tab. This would have a javascript "Are you sure" alert and then just enqueue the scheduled job immediately.

I'd be interested in writing a PR for this if you think it would be something you'd want to add.

Hey Ed,

When you enqueue or schedule a job you get back a job id that's a big long number, however, there doesn't look like there is a way to delete a job by that ID. My ideal case is to store that number in my db table so I can delete the future scheduled job if I need to.

I had a look in verk_web, found this:

There was also a delete_job function I found, but that required the original json, not just the ID.

Appreciate advice on where to go next.

Thanks!

When running the phoenix server. I see plenty of those messages:
[warn] Ignoring unmatched topic "rooms:jobs" in xxx.UserSocket

As a beginner to elixir phoenix, some help would be awesome.

Thanks

I have the following code in endpoint.ex:

defmodule App.Endpoint do
  use Phoenix.Endpoint, otp_app: :app
  if Application.get_env(:verk_web, :authorization) do
    IO.puts("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
    IO.inspect(Application.get_env(:verk_web, :authorization))
    use VerkWeb.Mount, path: "/admin/verk"
    IO.puts("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
  else
    raise "Verk authorization"
  end

If I execute this in dev it ask for authorization. When I compile to prod I can see this message:

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[realm: "Admin", username: "admin", password: "password"]
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

If I execute a remote console in the prod server I can get Application.get_env(:verk_web, :authorization) correctly. However I does not ask for authorization. I don't have more ideas to solve this.

I use AWS LoadBalancer and Docker with AWS ECS.

Any help it is very welcome :)

Thanks in advance.

Internal links in the VerkWeb are behaving weirdly, see screenshot:

• page_path(@conn, :index) -> / (incorrect)
• mount_static_path(@conn, "/js/app.js") -> /verk/js/app.js (correct)

Details:

• phoenix 1.3.0
• verk 1.1.0
• verk_web 1.0.2 (same behavior for master as well)
• plug 1.4.3

mounted into endpoint via

defmodule App.Web.Endpoint do
  use Phoenix.Endpoint, otp_app: :app
  use VerkWeb.Mount, path: "/verk"
  ...

I have an older project with phoenix 1.2 and verk 0.13.5 and everything works as expected.
git diff between these versions don't show any related changes so it's probably behavior change on phoenix/plug side.

Did anyone encounter something similar? Thoughts on debugging?

P.S. related - https://github.com/edgurgel/verk_web/blob/master/web/views/layout_view.ex#L5 - that's why mount_static_path works well.

Add a retry button so we can retry instantly some jobs that failed

We should be able to clear a list of enqueued jobs for a queue.

Sometimes you want to select all jobs and then unselect one. This can be quite tedious 😢

Maybe a checkbox placed inside the table header?

Not sure if this is an issue with my setup but trying to run the master branch of VerkWeb locally doesn't work for me:

$ mix deps.get
$ npm install && bower install
$ mix phoenix.server

[info] GET /
[debug] Processing by VerkWeb.PageController.index/2
  Parameters: %{}
  Pipelines: [:browser]
[info] Sent 500 in 12ms
[error] #PID<0.2415.0> running VerkWeb.Endpoint terminated
Server: localhost:4000 (http)
Request: GET /
** (exit) an exception was raised:
    ** (ArgumentError) argument error
        (stdlib) :ets.select(:queue_stats, [{{:"$1", :"$2", :"$3", :"$4", :_, :_}, [], [{{:"$1", :"$2", :"$3", :"$4"}}]}])
        (verk) lib/verk/queue_stats_counters.ex:23: Verk.QueueStatsCounters.all/0
        (verk) lib/verk/queue_stats.ex:22: Verk.QueueStats.all/0
        (verk_web) web/controllers/page_controller.ex:6: VerkWeb.PageController.index/2
        (verk_web) web/controllers/page_controller.ex:1: VerkWeb.PageController.action/2
        (verk_web) web/controllers/page_controller.ex:1: VerkWeb.PageController.phoenix_controller_pipeline/2
        (verk_web) lib/phoenix/router.ex:261: VerkWeb.Router.dispatch/2
        (verk_web) web/router.ex:1: VerkWeb.Router.do_call/2
        (verk_web) lib/verk_web/endpoint.ex:1: VerkWeb.Endpoint.phoenix_pipeline/1
        (verk_web) lib/plug/debugger.ex:123: VerkWeb.Endpoint."call (overridable 3)"/2
        (verk_web) lib/phoenix/endpoint/render_errors.ex:34: VerkWeb.Endpoint.call/2
        (plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
        (cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4

Is there something else I need to do to run this in development?

Using x from now format.

It would be interesting to know how many workers this instance has and how many are idle.

Error occurs on application startup

Error log:

02:15:10.347 [info] Application verk_web exited: VerkWeb.start(:normal, []) returned an error: shutdown: failed to start child: VerkWeb.Endpoint
--
  |  | ** (EXIT) an exception was raised:
  |  | ** (UndefinedFunctionError) function Plug.Cowboy.child_spec/1 is undefined (module Plug.Cowboy is not available)
  |  | Plug.Cowboy.child_spec([scheme: :http, plug: {VerkWeb.Endpoint, []}, options: [dispatch: [_: [{:_, Phoenix.Endpoint.Cowboy2Handler, {VerkWeb.Endpoint, []}}]], port: 4000, otp_app: :verk_web]])
  |  | (phoenix 1.4.17) lib/phoenix/endpoint/cowboy2_adapter.ex:44: Phoenix.Endpoint.Cowboy2Adapter.child_spec/3
  |  | (phoenix 1.4.17) lib/phoenix/endpoint/supervisor.ex:108: anonymous fn/6 in Phoenix.Endpoint.Supervisor.server_children/4
  |  | (elixir 1.10.4) lib/enum.ex:2111: Enum."-reduce/3-lists^foldl/2-0-"/3
  |  | (phoenix 1.4.17) lib/phoenix/endpoint/supervisor.ex:99: Phoenix.Endpoint.Supervisor.server_children/4
  |  | (phoenix 1.4.17) lib/phoenix/endpoint/supervisor.ex:59: Phoenix.Endpoint.Supervisor.init/1

Elixir 1.10.4

relevant mix.exs:

def application do
    [
      applications: [
        :verk,
        :verk_web,
      ]
    ]
  end

def deps do
 {:verk, "~> 1.7.1"},
 {:verk_web, "~> 1.5.0"}
end

relevant releases.exs:

config :verk_web, VerkWeb.Endpoint,
  http: [port: System.get_env("PORT", "4000") |> String.to_integer()],
  url: [host: System.get_env("URL_HOST", "localhost")],
  server: true,
  pubsub: [name: VerkWeb.PubSub, adapter: Phoenix.PubSub.PG2]

I've tried several things, but nothing really works. It does seem like some sort of a dependency problem, but I really can't figure it out. Seems like this error is related to phoenix 1.4, but since I don't have it as an explicit dependency I'm not sure what to do. Any help will be greatly appreciated. Thanks in advance :)

Related to edgurgel/verk#49

Hello,

It seems it does not work for the newest versions of Phoenix:

Failed to use "phoenix" (version 1.3.0) because            
  phoenix_live_reload (version 1.1.1) requires ~> 1.0      
  verk_web (version 1.1.0) requires ~> 1.2.1               
  mix.exs specifies ~> 1.3.0    

Any plan or solutions about this?

Thanks for your time

I have several instances running a Verk-backed application and I’d like to see the stats for the machines in aggregate. I’m having a couple issues.

1. Is there a way to run a single VerkWeb node that gets stats for all the processors? When I tried running VerkWeb on it’s own I get an argument error:

   18:36:07.483 [error] #PID<0.337.0> running VerkWeb.Endpoint terminated
   Server: localhost:4000 (http)
   Request: GET /
   ** (exit) an exception was raised:
   ** (ArgumentError) argument error
       (stdlib) :ets.select(:queue_stats, [{{:"$1", :"$2", :"$3", :"$4", :_, :_}, [], [{{:"$1", :"$2", :"$3", :"$4"}}]}])
       (verk) lib/verk/queue_stats_counters.ex:23: Verk.QueueStatsCounters.all/0
       (verk) lib/verk/queue_stats.ex:22: Verk.QueueStats.all/0
       (verk_web) web/controllers/page_controller.ex:6: VerkWeb.PageController.index/2
       (verk_web) web/controllers/page_controller.ex:1: VerkWeb.PageController.action/2
       (verk_web) web/controllers/page_controller.ex:1: VerkWeb.PageController.phoenix_controller_pipeline/2
       (verk_web) lib/phoenix/router.ex:261: VerkWeb.Router.dispatch/2
       (verk_web) web/router.ex:1: VerkWeb.Router.do_call/2
   

2. Unfortunately it looks like VerkWeb relies on the :queue_stats ets table to get queue specific stats, rather than using the data in Redis. The “All-time stats” numbers are coming from Redis and are reporting correctly, but queue specific stats are only showing jobs processed by the node on which that VerkWeb instance is running. Is this expected behavior?

It would be neat to add a Dockerfile and publish to docker hub a standalone verk web server.

It should have something like: "This job is likely gone (aka processed)"

Let's say I have 2 nodes, node_id: "1" and node_id: "2" running on the same redis server.

Currently, the "Failed", "Enqueued", "Scheduled" jobs properly list all jobs in the cluster (that is, jobs saved in Redis, no matter on what node is the VerkWeb being run).

Unfortunately, the "Running jobs" count and list lists only jobs running in the given node.

It would be cool if it would list all the jobs that are inprogress in all currently living nodes.

Possibly related to edgurgel/verk#157 ?

P.S. Until we implement it, we could mention it in the README and/or in VerkWeb interface.

Similar implementation to Retries

GenStage architecture was implemented in the Verk 0.13.6

Still trying to triage this, but upgrading to 0.13.3 breaks my application by preventing startup.

Startup crash log: https://gist.github.com/keyan/29f1c503cc3de1865bbc9531e33692e8
My mix.lock: https://gist.github.com/keyan/3d372b125579e684116a5f0f9713eb09

I might have to make some application changes on my end in order to work with the new version. At the very least I think the changes were large enough and added additional features and so warrant a minor version bump.

I have verk_web configured as a separate app in our umbrella project (not mounted in any phoenix apps).

We have one proxy app that kindof plugs requests to the other phoenix endpoints.

Shortened:

defmodule ProxyWeb.Endpoint do
  @domains %{
    "admin.app1.com" => AdminWeb.Endpoint,
    "verk.app1.com" => VerkWeb.Endpoint
  }
  def init(opts), do: opts

  def call(conn, _) do
    with domain <- get_domain(conn),
         endpoint <- Map.get(@domains, domain, @default_host) do
      endpoint.call(conn, endpoint.init(:supervisor, endpoint))
    end
  end
end

VerkWeb is totally fine here, except for websockets:

[info] Converted error Phoenix.Router.NoRouteError to 404 response
[info] GET /socket/websocket

I'm not quite sure what I should do from here, but happy to provide more detail or even PR if we find it's a real issue.

Move queue/:queue to queue/:queue/busy and put enqueued jobs on queue/:queue. Use recently added Verk.Queue to list (edgurgel/verk#18)

** (ArgumentError) lists in Phoenix.HTML and templates may only contain integers representing bytes, binaries or other lists, got invalid entry: {:a, "b"}

https://github.com/edgurgel/verk_web/blob/master/web/templates/dead/index.html.eex#L66

I think it would be really useful to see some stats around "dead" (as Verk calls them) jobs on VerkWeb. As of now, the only job failure stats we get are related to failed attempts, even when subsequent retries may succeed. It is difficult to tell how many job failures ended with the job reaching the max retry count and dying.

It is not entirely clear to me how difficult it would be but it very helpful to have another tab, "Dead", that has similar data as the "Retries" tab.

The inspect feature makes no sense for jobs that are not running (scheduled, retries, enqueued)

Hi!

So we have an issue, which is, we currently run our Elixir app on Heroku (not for much longer but we are currently stuck) and we want to use Heroku workers to handle the job load. However, it seems I can't configure our "web" app to not accept worker requests but still host verk_web on the same app.

We have the dashboard working but it shows no queues in the web interface because we have used the VERK_DISABLED example in the docs to disable it on our API instance. The problem is that we need to host the verk_web panel on the API instance and host the workers on separated machines but since we return [] for no queues, the dashboard thinks there is nothing there. We definitely don't want the API app to act as a worker.

Do you have any advice???

Similar to Retries