fede1024 / kafka-view Goto Github PK

View Code? Open in Web Editor NEW

215.0 215.0 31.0 2.56 MB

Kafka web UI

License: MIT License

CSS 0.54% JavaScript 10.42% Rust 88.10% Dockerfile 0.40% Makefile 0.54%

kafka user-interface web-interface

kafka-view's People

Contributors

Stargazers

Watchers

kafka-view's Issues

topic_groups API is incorrect

kafka-view/src/web_server/api.rs

Lines 117 to 119 in b3729eb

 let mut groups: HashMap<(ClusterId, String), GroupInfo> = cache.groups 

 .lock_iter(|iter| { 

 iter.filter(|&(&(ref c, ref t), g)| filter_fn(c, t, &g.name))

Here Fn(&ClusterId, &TopicName, &String) -> bool the second argument isn't topic name but group name cause the APIs always returns Offsets only state

some column can't get the data

The support for TLS has landed in Rocket, and kafka-view should have an opt-in support for it. This would also the browser to accept the brotli compression, which makes the API responses considerably faster.

Docker image

The Dockerfile cannot be built.
I am not sure why (not a Rust dev :D)
Even the CI on master fails with something like the following:

[...]
 Compiling tokio-core v0.1.17
   Compiling scheduled-executor v0.4.0
   Compiling maud v0.20.0
   Compiling rocket_contrib v0.4.0
error: linking with `x86_64-unknown-linux-musl-gcc` failed: exit code: 1
  |
  = note: "x86_64-unknown-linux-musl-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-m64" "-nostdlib" "/root/.rustup/toolchains/nightly-2019-03-22-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crt1.o" "/root/.rustup/toolchains/nightly-2019-03-22-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/crti.o" "-L" "/root/.rustup/toolchains/nightly-2019-03-22-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib"
[...]

SASL/Kerberos and SASL/PLAIN Based support

Team,
Quick Question, Does the Kafka-view support on SASLKeberos or SASL/PLAIN support?

Display partition assignment in consumer group members list

Add a new column Partitions to indicate which partitions each consumer member holds.

Docker image contains 1G of unneeded stuff

$ du -sh .cargo
504M	.cargo
$ du -sh .rustup
871M	.rustup```

Prob just needs a dockerignore file

Consume topic content directly from the web UI.

Thanks for this tool.

Readme says: Consume topic content directly from the web UI. I see the message tailer working. Is it possile to retrieve messages from a specific offset? Any reason why this would or wouldn't be possible in kafka-view?

Initial cache state loading is too slow

For a long running kafka-view, restart may never succeed because the initial cache state loading is way too slow (> 10 minutes).

2017-09-17 15:45:51+0800: (main) INFO - kafka_view - Kafka-view is starting up!
2017-09-17 15:45:51+0800: (main) INFO - kafka_view::cache - Started creating state
2017-09-17 15:55:51+0800: (poll) ERROR - rdkafka::client - librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): example.com:9092/bootstrap: Receive failed: Disconnected
2017-09-17 15:55:51+0800: (polling thread) ERROR - rdkafka::client - librdkafka: Global error: BrokerTransportFailure (Local: Broker transport failure): example.com:9092/bootstrap: Receive failed: Disconnected
...

Upgrade to serde 1.0

Is offsets.storage=zookeeper supported?

As far as I can see, old-style zookeeper offsets are not displayed with default settings (I'm using docker image).
Are they supported?

Make API easier to use directly

The API currently used for generating tables returns JSON blobs containing lists of lists. Since the semantics of every field depends on the position within the inner list, changing the API will break any API user. Dictionaries should be used instead, as it's currently already done for the /api/clusters/<cluster>/topics endpoint.

Add integration tests

It would be useful to have integration tests.x

Some possible test scenarios are:

Produce some data into Kafka, wait for metrics to refresh, then check rate, topic size.
Create a topic with a specific layout (partitions/replicas), and verify that the topology reported by the API matches the expected one.
Verify that the topic tailer returns the messages as they are being produced.
Create a consumer group, commit some offsets, and verify that the API reports the correct values.
Delete a topic, check that the topic disappears from the API and from the cache.

This could be easily achieved by running kafka-view in a dockerized environment using docker-compose, similarly to what is done for the integration tests of rdkafka.

Add timestamp and key to the topic tailer

If timestamps and keys are available, they should be printed in the topic tailer.

More clear installation steps

Please, make a more clear instructions to installation. Not everyone knows where to get rustup.
It also would be nice to have binary releases/deb/rpm packages.

Add a license

It's unclear how this project is licensed.

Dockerfile?

Would love to test it, and I think it'd be best to have a Dockerfile.
Looking forward to it :)

failed to build librdkafka on OSX

➜  kafka-view git:(master) ✗ cargo build
   Compiling semver-parser v0.7.0
   Compiling autocfg v0.1.2
   Compiling libc v0.2.51
   Compiling rand_core v0.4.0
   Compiling proc-macro2 v0.4.27
   Compiling unicode-xid v0.1.0
   Compiling byteorder v1.3.1
   Compiling arrayvec v0.4.10
   Compiling stable_deref_trait v1.1.1
   Compiling matches v0.1.8
   Compiling yansi v0.4.0
   Compiling nodrop v0.1.13
   Compiling smallvec v0.6.9
   Compiling syn v0.15.30
   Compiling lazy_static v1.3.0
   Compiling cfg-if v0.1.7
   Compiling cc v1.0.34
   Compiling version_check v0.1.5
   Compiling scopeguard v0.3.3
   Compiling memoffset v0.2.1
   Compiling httparse v1.3.3
   Compiling slab v0.4.2
   Compiling lazycell v1.2.1
   Compiling bitflags v1.0.4
   Compiling futures v0.1.26
   Compiling untrusted v0.6.2
   Compiling safemem v0.3.0
   Compiling percent-encoding v1.0.1
   Compiling fnv v1.0.6
   Compiling language-tags v0.2.2
   Compiling typeable v0.1.2
   Compiling memchr v2.2.0
   Compiling traitobject v0.1.0
   Compiling yansi v0.5.0
   Compiling serde v1.0.90
   Compiling pkg-config v0.3.14
   Compiling num-traits v0.2.6
   Compiling indexmap v1.0.2
   Compiling state v0.4.1
   Compiling ryu v0.2.7
   Compiling regex v1.1.5
   Compiling bitflags v0.7.0
   Compiling ucd-util v0.1.3
   Compiling num-integer v0.1.39
   Compiling same-file v1.0.4
   Compiling alloc-no-stdlib v2.0.1
   Compiling utf8-ranges v1.0.2
   Compiling unicode-width v0.1.5
   Compiling scoped-tls v0.1.2
   Compiling rustc-demangle v0.1.13
   Compiling itoa v0.4.3
   Compiling literalext v0.1.1
   Compiling maud_htmlescape v0.17.0
   Compiling linked-hash-map v0.5.2
   Compiling quick-error v1.2.2
   Compiling ansi_term v0.11.0
   Compiling vec_map v0.8.1
   Compiling dtoa v0.4.3
   Compiling strsim v0.7.0
   Compiling snowflake v1.3.0
   Compiling kafka-view v0.3.3 (/Users/me/source/rust/kafka-view)
   Compiling termcolor v1.0.4
   Compiling unicode-bidi v0.3.4
   Compiling owning_ref v0.4.0
   Compiling tokio-trace-core v0.1.0
   Compiling thread_local v0.3.6
   Compiling rand_core v0.3.1
   Compiling log v0.4.6
   Compiling crossbeam-utils v0.6.5
   Compiling unicode-normalization v0.1.8
   Compiling rand_pcg v0.1.2
   Compiling rand_chacha v0.1.1
   Compiling rand v0.6.5
   Compiling backtrace v0.3.15
   Compiling semver v0.9.0
   Compiling unicase v1.4.2
   Compiling pear_codegen v0.1.2
   Compiling rocket_codegen v0.4.0
   Compiling rocket v0.4.0
   Compiling walkdir v2.2.7
   Compiling regex-syntax v0.6.6
   Compiling alloc-stdlib v0.2.1
   Compiling textwrap v0.10.0
   Compiling humantime v1.2.0
   Compiling ring v0.13.5
   Compiling libz-sys v1.0.25
   Compiling lz4-sys v1.8.3
   Compiling miniz-sys v0.1.11
   Compiling yaml-rust v0.4.3
   Compiling maud_macros v0.20.0
   Compiling rand_hc v0.1.0
   Compiling rand_xorshift v0.1.1
   Compiling rand_isaac v0.1.1
   Compiling lock_api v0.1.5
   Compiling log v0.3.9
   Compiling crossbeam-queue v0.1.2
   Compiling rustc_version v0.2.3
   Compiling brotli-decompressor v2.3.0
   Compiling tokio-sync v0.1.4
   Compiling tokio-executor v0.1.7
   Compiling mime v0.2.6
   Compiling parking_lot_core v0.4.0
   Compiling rand_os v0.1.3
   Compiling rand_jitter v0.1.3
   Compiling iovec v0.1.2
   Compiling net2 v0.2.33
   Compiling num_cpus v1.10.0
   Compiling time v0.1.42
   Compiling num_cpus v0.2.13
   Compiling fsevent-sys v0.1.6
   Compiling isatty v0.1.9
   Compiling filetime v0.2.4
   Compiling atty v0.2.11
   Compiling base64 v0.9.3
   Compiling base64 v0.10.1
   Compiling crossbeam-epoch v0.7.1
   Compiling tokio-timer v0.2.10
   Compiling tokio-current-thread v0.1.6
   Compiling idna v0.1.5
   Compiling quote v0.6.11
   Compiling aho-corasick v0.7.3
   Compiling bytes v0.4.12
   Compiling futures-cpupool v0.1.8
   Compiling rdkafka-sys v0.11.1-0
   Compiling clap v2.32.0
   Compiling fsevent v0.2.17
   Compiling mio v0.6.16
   Compiling error-chain v0.12.0
   Compiling crossbeam-deque v0.7.1
   Compiling syn v0.14.9
   Compiling flate2 v0.2.20
   Compiling brotli v3.3.0
   Compiling notify v4.0.10
   Compiling tokio-io v0.1.12
   Compiling url v1.7.2
   Compiling mio-uds v0.6.7
   Compiling mio-extras v2.0.5
   Compiling tokio-threadpool v0.1.13
   Compiling tokio-codec v0.1.1
   Compiling parking_lot v0.7.1
   Compiling tokio-reactor v0.1.9
   Compiling cookie v0.11.0
   Compiling hyper v0.10.15
   Compiling tokio-fs v0.1.6
   Compiling zookeeper_derive v0.4.0
   Compiling env_logger v0.6.1
   Compiling tokio-uds v0.2.5
   Compiling tokio-udp v0.1.3
   Compiling tokio-tcp v0.1.3
   Compiling tokio v0.1.18
error: failed to run custom build command for `rdkafka-sys v0.11.1-0`
process didn't exit successfully: `/Users/me/source/rust/kafka-view/target/debug/build/rdkafka-sys-72cb70cef33dc91f/build-script-build` (exit code: 101)
--- stdout
Configuring librdkafka
using cache file config.cache
checking for OS or distribution... ok (osx)
checking for C compiler from CC env... failed
checking for gcc (by command)... ok (cached)
checking for C++ compiler from CXX env... failed
checking for C++ compiler (g++)... ok (cached)
checking executable ld... ok (cached)
checking executable nm... ok (cached)
checking executable objdump... ok (cached)
checking executable strip... ok (cached)
checking for debug symbols compiler flag (-g...)... ok (cached)
checking for pkgconfig (by command)... ok (cached)
checking for install (by command)... failed
checking for PIC (by compile)... ok (cached)
checking for GNU-compatible linker options... failed
checking for OSX linker options... ok (cached)
checking for GNU linker-script ld flag... failed
checking for Solaris linker-script ld flag... failed (ignore)
checking for __atomic_32 (by compile)... ok (cached)
checking for __atomic_64 (by compile)... ok (cached)
checking for socket (by compile)... ok (cached)
parsing version '0x000b01ff'... ok (0.11.1)
checking for libpthread (by pkg-config)... failed
checking for libpthread (by compile)... ok (cached)
checking for zlib (by pkg-config)... ok
checking for zlib (by compile)... ok (cached)
checking for libcrypto (by pkg-config)... failed
checking for libcrypto (by compile)... failed (disable)
checking for liblz4 (by pkg-config)... ok
checking for liblz4 (by compile)... ok (cached)
checking for crc32chw (by compile)... ok (cached)
checking for regex (by compile)... ok (cached)
checking for librt (by pkg-config)... failed
checking for librt (by compile)... failed
checking for strndup (by compile)... ok (cached)
checking for strerror_r (by compile)... ok (cached)
checking for libdl (by pkg-config)... failed
checking for libdl (by compile)... ok (cached)
checking for nm (by env NM)... ok (cached)
Generated Makefile.config
Generated config.h

Configuration summary:
  prefix                   /usr/local
  ARCH                     x86_64
  CPU                      generic
  GEN_PKG_CONFIG           y
  ENABLE_DEVEL             n
  ENABLE_VALGRIND          n
  ENABLE_REFCNT_DEBUG      n
  ENABLE_SHAREDPTR_DEBUG   n
  ENABLE_LZ4_EXT           y
  ENABLE_SSL               n
  ENABLE_SASL              n
  MKL_APP_NAME             librdkafka
  MKL_APP_DESC_ONELINE     The Apache Kafka C/C++ library
  MKL_DISTRO               osx
  SOLIB_EXT                .dylib
  CC                       gcc
  CXX                      g++
  LD                       ld
  NM                       nm
  OBJDUMP                  objdump
  STRIP                    strip
  CPPFLAGS                 -gstrict-dwarf -O2 -fPIC -Wall -Wsign-compare -Wfloat-equal -Wpointer-arith -Wcast-align
  PKG_CONFIG               pkg-config
  INSTALL                  install
  HAS_LDFLAGS_STATIC       n
  LIB_LDFLAGS              -shared -dynamiclib -Wl,-install_name,$(DESTDIR)$(libdir)/$(LIBFILENAME)
  RDKAFKA_VERSION_STR      0.11.1
  MKL_APP_VERSION          0.11.1
  LIBS                     -L/usr/local/Cellar/lz4/1.9.2/lib -llz4 -lpthread -lz -ldl
  CFLAGS                   -I/usr/local/Cellar/lz4/1.9.2/include
  CXXFLAGS                 -Wno-non-virtual-dtor
  SYMDUMPER                $(NM) -g
  exec_prefix              /usr/local
  bindir                   /usr/local/bin
  sbindir                  /usr/local/sbin
  libexecdir               /usr/local/libexec
  datadir                  /usr/local/share
  sysconfdir               /usr/local/etc
  sharedstatedir           /usr/local/com
  localstatedir            /usr/local/var
  libdir                   /usr/local/lib
  includedir               /usr/local/include
  infodir                  /usr/local/info
  mandir                   /usr/local/man
Generated config.cache

Now type 'make' to build

--- stderr
Building and linking librdkafka statically
Running command: "./configure --disable-sasl --disable-ssl --enable-static" in dir: librdkafka
thread 'main' panicked at 'Command failed with error: No such file or directory (os error 2)', /Users/me/.cargo/registry/src/github.com-1ecc6299db9ec823/rdkafka-sys-0.11.1-0/build.rs:23:21
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

Add metrics charts

Kafka-view periodically collects metrics from all clusters, such as message rate, byte rate and topic sizes. It would be useful if, instead of just keeping the last value for each metric, kafka-view could store past values as well and show them in form of a chart.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

	let mut groups: HashMap<(ClusterId, String), GroupInfo> = cache.groups
	.lock_iter(\|iter\| {
	iter.filter(\|&(&(ref c, ref t), g)\| filter_fn(c, t, &g.name))