Giter Club home page Giter Club logo

broadway_dashboard's Introduction

Broadway Dashboard

Online documentation

BroadwayDashboard is a tool to analyze Broadway pipelines. It provides some insights about performance and errors for your running pipelines.

It works as an additional page for the Phoenix LiveDashboard.

You can inspect pipelines on remote nodes that are not running BroadwayDashboard too. See Distribution for details.

Broadway Dashboard

Integration with Phoenix LiveDashboard

You can add this page to your Phoenix LiveDashboard by adding as a page in the live_dashboard macro at your router file.

live_dashboard "/dashboard",
  additional_pages: [
    broadway: {BroadwayDashboard, pipelines: [MyBroadway]}
  ]

The :pipelines option accept pipeline names (the :name option of your Broadway). By omitting the :pipelines option, BroadwayDashboard will try to autodiscover your pipelines.

live_dashboard "/dashboard",
  additional_pages: [
    broadway: BroadwayDashboard
  ]

Once configured, you will be able to access the BroadwayDashboard at /dashboard/broadway.

Using from the command line with PLDS

It's possible to use Broadway Dashboard without having to install it on your application. PLDS stands for Phoenix LiveDashboard Standalone and it's a CLI version of Phoenix LiveDashboard with some tools pre-installed. One of those tools is Broadway Dashboard.

To install PLDS on your machine, you can run:

$ mix escript.install hex plds

Then connect to your running node with:

$ plds server --connect mynode --open

For more information about the usage, please check the PLDS documentation.

Distribution

Phoenix LiveDashboard works with distribution out of the box, and it's not different with Broadway Dashboard! You can inspect your pipelines that are running on connected nodes.

You can also inspect pipelines from nodes that are not running the same system of your dashboard. This is possible because we "copy" the essential parts of this tool to the remote node when it's not running BroadwayDashboard. We stop the tracking once the node that started it is disconnected.

Installation

Add the following to your mix.exs and run mix deps.get:

def deps do
  [
    {:broadway_dashboard, "~> 0.4.0"}
  ]
end

After that, proceed with instructions described in Integration with Phoenix LiveDashboard above.

Acknowledgment

This project is based on Marlus Saraiva's work from his presentation at ElixirConf 2019.

In that talk he presented a graph showing the work of a Broadway pipeline, which is essentially the same we display in this project. Thank you, Marlus! <3

Development

After cloning this project, you can use the following command to run a development server with a sample pipeline:

$ mix dev

This is going to start a server running Phoenix LiveDashboard.

Use mix test to run the test suite.

License

Copyright 2021 Dashbit

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

broadway_dashboard's People

Contributors

feng19 avatar harisudhanravi avatar josevalim avatar kianmeng avatar leductam avatar louisvisser avatar moxley avatar philss avatar vittoriabitton avatar walter avatar wojtekmach avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

broadway_dashboard's Issues

Broadway pipelines tab fails to load

Current versions i am using:

broadway 1.0.3
broadway_dashboard 0.2.2
broadway_kafka 0.3.4
phoenix_live_dashboard 0.6.4
phoenix_live_view 0.17.7

router.ex

live_dashboard "/dashboard",
        metrics: TelemetrySupervisor,
        additional_pages: [
          broadway: BroadwayDashboard
        ]
    end

Issue:
Whenever i load up my /dashboard i am unable to access the /dashboard/broadway tab. It will load for a while and says the Dashboard is not connected.

Only meaningful log i can find

iex([email protected])1> 17:24:34.953 request_id=0558abf4-4a16-42a1-8232-e47e5a950a18 [info] GET /ingest/dashboard/broadway
iex([email protected])1> 17:24:34.970 request_id=0558abf4-4a16-42a1-8232-e47e5a950a18 [info] Sent 200 in 17ms
iex([email protected])1> 17:24:35.867 [info] CONNECTED TO Phoenix.LiveView.Socket in 95µs
  Transport: :websocket
  Serializer: Phoenix.Socket.V2.JSONSerializer
  Parameters: %{"_csrf_token" => "JXMuGARSIggZMRUFBjNjL3QrcCosUzYQHEK5wgQyKAahTj6BGJ6Oofcx", "vsn" => "2.0.0"}
iex([email protected])1> 17:24:35.961 [info] Function passed as a handler with ID {BroadwayDashboard.Telemetry, #PID<0.1495.0>} is local function.
This mean that it is either anonymous function or capture of function without module specified. That may cause performance penalty when calling such handler. For more details see note in `telemetry:attach/4` documentation.

Screen Shot 2022-05-10 at 7 14 20 PM

[error] Handler {BroadwayDashboard.Telemetry, #PID<0.xxx.0>} has failed and has been detached.

I tried integrate broadway_dashboard to live LiveDashboard but got the error:

[error] Handler {BroadwayDashboard.Telemetry, #PID<0.xxx.0>} has failed and has been detached.
Reason={:badkey, :time, %{system_time: 16274584813510685}}
Stacktrace=[
  {BroadwayDashboard.Telemetry, :handle_event, 4,
   [file: 'lib/broadway_dashboard/telemetry.ex', line: 47]},
  {:telemetry, :"-execute/3-fun-0-", 4, ...

My deps

defp deps do
    [
      {:phoenix, github: "phoenixframework/phoenix", override: true},
      {:phoenix_html, github: "phoenixframework/phoenix_html", override: true},
      {:phoenix_live_reload, "~> 1.2", only: :dev},
      {:phoenix_live_view, github: "phoenixframework/phoenix_live_view", override: true},
      {:floki, ">= 0.30.0", only: :test},
      {:phoenix_live_dashboard, github: "phoenixframework/phoenix_live_dashboard", override: true},
      {:esbuild, "~> 0.1", runtime: Mix.env() == :dev},
      {:swoosh, "~> 1.3"},
      {:telemetry_metrics, "~> 0.4"},
      {:telemetry_poller, "~> 0.4"},
      {:jason, "~> 1.0"},
      {:plug_cowboy, "~> 2.0"},
      {:broadway, github: "dashbitco/broadway", override: true},
      {:broadway_rabbitmq, github: "dashbitco/broadway_rabbitmq", override: true},
      {:broadway_dashboard, github: "dashbitco/broadway_dashboard", override: true}
    ]
  end

Please help to clarify this issue.

Add the ability to control zoom in and out to the view of pipeline

When we have a large number of concurrency in batches or processors the view of the whole pipeline becomes very difficult.

image

I was wondering if we can add the ability to add some zoom control and navigation to this graph?
Something like this: https://codepen.io/osublake/pen/oGoyYb
Or this: https://www.petercollingridge.co.uk/tutorials/svg/interactive/pan-and-zoom/

I've looked to the source code of this project, but I'vent found where is the SVG generation or the place where I can implement this feature with JS.

Yes, I know that I can see the whole pipeline inspecting the element and setting the svg to use max-width as 100% and they will be more responsive. But I think this would be nice if we don't have to do this every time.

I believe this will improve the DX.

"Throughput" and "All time" counters history

Hi

I have implemented a history module for the live dashboard, and it works for Phoenix and my custom metrics but does not work for BroadwayDashboard.

live_dashboard "/dashboard",
      metrics: MyMegaApp.Telemetry,
      additional_pages: [
        broadway: BroadwayDashboard
      ],
      metrics_history: {MyMegaApp.Telemetry.History, :metrics_history, []}

The MyMegaApp.Telemetry.History.metrics_history/1 contains a logger, so I can see that BroadwayDashboard does not call for history.

How can counters history be implemented in BroadwayDashboard?

Package versions:

%{
  "broadway": "1.0.0",
  "broadway_dashboard": "0.2.1",
  "broadway_rabbitmq": "0.7.0",
  "phoenix_live_dashboard": "0.5.3",
  "telemetry": "0.4.3",
  "telemetry_metrics": "0.6.0",
  "telemetry_poller": "0.5.0"
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.