zix99 / rare Goto Github PK
View Code? Open in Web Editor NEWCreate console histograms, bar graphs, tables, heatmaps and more in realtime using regex and expressions.
Home Page: https://rare.zdyn.net/
License: GNU General Public License v3.0
Create console histograms, bar graphs, tables, heatmaps and more in realtime using regex and expressions.
Home Page: https://rare.zdyn.net/
License: GNU General Public License v3.0
Though DeferLogs
is only called in a few spots, logger.print is called in many and can cause an edge-case race condition with swapping the logger
I've taken screenshots to preserve image highlighting; here's the cut-and-paste friendly test case:
echo "a: 1 b: 2 c: 3" | ./rare f -m 'a: (\d+) b: (\d+) c: (\d+)' -e '{1} {2} {3}'
Everything seems to work fine using the short '-e' with histogram, analyze, etc; it's just filter that is impacted.
Here's for example a snippet that shows a histogram, tracking monthly changes in a git repo:
git log --first-parent --pretty=format:"%ad%x09%s" --date="format:%Y-%m" | \
awk '{ print $1 }' | rare histogram -a --sort date
It spits out something like:
2021-10 10 [ 4.6%] ████████████
2021-11 42 [19.3%] ██████████████████████████████████████████████████
2021-12 8 [ 3.7%] █████████▌
2022-01 4 [ 1.8%] ████▊
2022-02 7 [ 3.2%] ████████▍
2022-03 2 [ 0.9%] ██▍
2022-04 19 [ 8.7%] ██████████████████████▋
2022-05 3 [ 1.4%] ███▋
2022-06 9 [ 4.1%] ██████████▊
2022-07 7 [ 3.2%] ████████▍
2022-08 4 [ 1.8%] ████▊
2022-09 10 [ 4.6%] ████████████
2022-10 15 [ 6.9%] █████████████████▉
2022-11 27 [12.4%] ████████████████████████████████▏
2022-12 3 [ 1.4%] ███▋
2023-01 5 [ 2.3%] ██████
2023-02 10 [ 4.6%] ████████████
2023-03 8 [ 3.7%] █████████▌
2023-04 8 [ 3.7%] █████████▌
2023-05 11 [ 5.0%] █████████████
2023-06 6 [ 2.8%] ███████▏
Now, how can I get it sorted in descending order? I can't use cmd-line sort
, it breaks the histogram.
I'm curious the Readme could included the memory and CPU usage between standard unix tools ga and rare?
Another take to benchmark on embedded device is useful.
Hey, I just discovered this and really liked it, however I found a bug when trying to follow large standard output.
if I have the following python file which just generates a bunch of numbers:
import random
BOXES = 100
BALLS = 100
def sim():
boxes = [0 for a in range(BOXES)]
for ball in range(BALLS):
boxes[random.randint(0, BOXES-1)] += 1
return len([b for b in boxes if b == 0])
for i in range(10000000):
print(sim())
And I want to use rare to show the distribution of the output live while the simulation runs, I would do
python3 ./random_100_box_ball.py | rare histo -f -x --sortkey -n 30 --batch 1
However, the updates through the pipe come once a second, even after setting the batch size to something small like 1.
If I instead piped the output to a file, and then read from the file using rare, it produces the live following that I was after
python3 random_100_box_ball.py > test.txt
rare histo -f --sortkey -n 30 --batch 1 ./test.txt
batch settings are respected and the updates can be tuned when not using pipe. I was wondering if you knew why this was the case? And if you could get pipes to emulate the behaviour seen when you instead follow a file?
N.B. I'm running ubuntu 20.04 WSL2 on Windows 10, using the prebuilt binary (rare, but rare-pcre suffers from the same issue)
04:55:37 ~ -> rare --version
rare version 0.2.1, 29f1bd5; regex: re2
04:55:39 ~ -> rare-pcre --version
rare-pcre version 0.2.1, 29f1bd5; regex: libpcre2
It would be useful to support using a log scale for generating heatmaps with wide distributions.
Edit: by the way, thanks a ton for this tool. I've avoided learning gnuplot for years and rare looks like it will help me do that indefinitely 😄.
I see you have no LICENSE file for this project. I see the README.md
has the GPL-3.0-or-later header. I see .goreleaser.yml
says GPLv2
.
I would suggest adding the GPL-3.0-or-later license as a LICENSE
or COPYING
file, changing GPLv2
to GPL-3.0-or-later
, and adding license headers to all files. I could send a patch for the first two if you would like.
Hey,
Just tried to install from Homebrew and I get an error:
❯ brew tap zix99/rare
==> Tapping zix99/rare
Cloning into '/usr/local/Homebrew/Library/Taps/zix99/homebrew-rare'...
remote: Enumerating objects: 45, done.
remote: Counting objects: 100% (45/45), done.
remote: Compressing objects: 100% (30/30), done.
remote: Total 45 (delta 14), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (45/45), 6.65 KiB | 3.33 MiB/s, done.
Resolving deltas: 100% (14/14), done.
Error: Invalid formula: /usr/local/Homebrew/Library/Taps/zix99/homebrew-rare/rare.rb
rare: Calling bottle :unneeded is disabled! There is no replacement.
Please report this issue to the zix99/rare tap (not Homebrew/brew or Homebrew/core):
/usr/local/Homebrew/Library/Taps/zix99/homebrew-rare/rare.rb:9
Error: Cannot tap zix99/rare: invalid syntax in tap!
Is there anything I need to do other than
brew tap zix99/rare && brew install rare
I'm doing this on a 2019 Mac Book Pro (Intel)
Thank you
First off, great work with the heatmaps feature!
I've been using heatmaps for numerical data, in particular nginx response times. I usually convert them it integers in rare by matching eg 0.053
(seconds) with (\d+)\.(\d{3})
and then using the expression {sumi {multi {1} 1000} {3}}
to convert to milliseconds.
I've found that the table and heatmap sort the column names as strings, and not numerically if possible. The results in meaningless heatmaps:
I made a small change to a local checkout of rare to basically test if the column headers could be converted to integers and then sort them numerically if they can:
index 1be0be8..040b889 100644
--- a/pkg/aggregation/table.go
+++ b/pkg/aggregation/table.go
@@ -103,8 +103,23 @@ func (s *TableAggregator) OrderedColumns() []string {
func (s *TableAggregator) OrderedColumnsByName() []string {
keys := s.Columns()
+ // check if keys can be sorted numerically:
+ numeric := true
+ for _,k := range keys {
+ if _, err := strconv.Atoi(k); err != nil {
+ numeric = false
+ break
+ }
+ }
+
sort.Slice(keys, func(i, j int) bool {
- return keys[i] < keys[j]
+ if numeric {
+ k0, _ := strconv.Atoi(keys[i])
+ k1, _ := strconv.Atoi(keys[j])
+ return k0 < k1
+ } else {
+ return keys[i] < keys[j]
+ }
})
return keys
The same heatmap is now much more meaningful:
Would you be interested in incorporating the above diff into rare?
I wanted to draw a histogram for float values but I got BAD-TYPE
error. Are decimal values not supported at all or do we need to process them with a trick?
Integer values work:
$ echo -e "1\n10\n15\n20" | rare h -e "{bucket {0} 10}"
10 2
20 1
0 1
Matched: 4 / 4 (Groups: 3)
11 B (0 B/s) | <stdin>
But when I use decimal values, I get error
echo -e "1.5\n10.2\n15.5\n20.0" | rare h -e "{bucket {0} 10}"
<BAD-TYPE> 4
Matched: 4 / 4 (Groups: 1)
19 B (0 B/s) | <stdin>
README currently states that to build go 1.11 or higher is required:
Requires GO 1.11 or higher (Uses go modules)
Looks like rare
is now making us of embed
(#32) which requires 1.16
First of all, let me say, this is an awesome project. Nice work! I was in the process of writing something similar but much much worse; I think I might use this as a library instead!
I've manage to cause a panic in the following scenario:
echo '1,2,3,4,5,6,7,8,9,0' | rare filter -m "(^[^,]*)(,([^,]*)){5}"
panic: runtime error: slice bounds out of range [11:10]
goroutine 1 [running]:
rare/pkg/color.WrapIndices(0xc000500000, 0x13, 0xc0000b4590, 0x6, 0x6, 0x206860, 0x1207860)
/Users/ondrejb/Documents/git/rare/pkg/color/coloring.go:95 +0x832
rare/cmd.filterFunction(0xc00024c840, 0x0, 0xc0004752d0)
/Users/ondrejb/Documents/git/rare/cmd/filter.go:33 +0x276
github.com/urfave/cli.HandleAction(0x13b20c0, 0x144c828, 0xc00024c840, 0xc00024c840, 0x0)
/Users/ondrejb/go/pkg/mod/github.com/urfave/[email protected]/app.go:523 +0xfd
github.com/urfave/cli.Command.Run(0x142c0a8, 0x6, 0x1429f04, 0x1, 0x0, 0x0, 0x0, 0x1441dfa, 0x44, 0x0, ...)
/Users/ondrejb/go/pkg/mod/github.com/urfave/[email protected]/command.go:174 +0x58e
github.com/urfave/cli.(*App).Run(0xc0004d6000, 0xc000090040, 0x4, 0x4, 0x0, 0x0)
/Users/ondrejb/go/pkg/mod/github.com/urfave/[email protected]/app.go:276 +0x7d4
main.cliMain(0xc000090040, 0x4, 0x4, 0x0, 0x0)
/Users/ondrejb/Documents/git/rare/main.go:101 +0x666
main.main()
/Users/ondrejb/Documents/git/rare/main.go:105 +0x49
To save you parsing manually, the three groups here are the 1st column of a csv, the 6th column but including the leading comma, and the 6th field without the leading comma.
When I make the second (outer) group non capturing, ie. (^[^,]*)(?:,([^,]*)){5}
everything works fine and I get the 1st and 6th field (groups {1} and {3}). Obviously, if I use the --nocolor option, or if I use an expression eg. -e '{1} {2} {3}
, everything is fine.
I haven't looked deep into the code yet, but obviously since the match groups overlap, the starting index of the inner group lies inside the outer group, and the colouring logic doesn't account for this scenario.
I'd suggest that the inner match should take precedence when colouring the matching text (ie. inner match colours "overwrite" the outer group)
I'll have a crack at making a pull request to fix this myself soon.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.