iangrunert / git-ratchet Goto Github PK

Adding or removing measures separately works, but it's the combination of the two that causes a problem.

To reproduce the error:

# Setup
mkdir ratchet-test && cd ratchet-test
git init

# Add two measures, works as expected
echo "First" > LOG.txt
git add LOG.txt && git commit -m "First"
echo -e "measure-A,5\nmeasure-B,4" | git ratchet -v check -w

# Remove a measure, works as expected
echo "Second" >> LOG.txt && git commit -am "Second"
echo -e "measure-A,5" | git ratchet -v check -w

# Add a new measure, works as expected
echo "Third" >> LOG.txt && git commit -am "Third"
echo -e "measure-A,5\nmeasure-C,3" | git ratchet -v check -w

# Add a new measure AND remove an old measure, THROWS AN EXCEPTION
echo "Fourth" >> LOG.txt && git commit -am "Fourth"
echo -e "measure-C,3\nmeasure-D,2" | git ratchet -v check -w

The error thrown is:

panic: runtime error: index out of range

goroutine 1 [running]:
github.com/iangrunert/git-ratchet/store.CompareMeasures(0x228250, 0x6, 0xc20803ca21, 0x28, 0xc20804a700, 0x2, 0x2, 0xc20804a340, 0x2, 0x2, ...)
	/home/travis/gopath/src/github.com/iangrunert/git-ratchet/store/reader.go:250 +0x100c
github.com/iangrunert/git-ratchet/cmd.Check(0x228250, 0x6, 0x0, 0x3b0100, 0x225750, 0x3, 0xc208032601, 0x3b3bb0, 0xc208048000, 0x28efc0)
	/home/travis/gopath/src/github.com/iangrunert/git-ratchet/cmd/check.go:51 +0xfc4
main.func·002(0xc2080426c0, 0xc20803c720, 0x0, 0x3)
	/home/travis/gopath/src/github.com/iangrunert/git-ratchet/git-ratchet.go:42 +0x10d
github.com/spf13/cobra.(*Command).execute(0xc2080426c0, 0xc20803c660, 0x3, 0x3, 0x0, 0x0)
	/home/travis/gopath/src/github.com/spf13/cobra/command.go:603 +0x86b
github.com/spf13/cobra.(*Command).ExecuteC(0xc208042b40, 0xc2080426c0, 0x0, 0x0)
	/home/travis/gopath/src/github.com/spf13/cobra/command.go:689 +0x501
github.com/spf13/cobra.(*Command).Execute(0xc208042b40, 0x0, 0x0)
	/home/travis/gopath/src/github.com/spf13/cobra/command.go:648 +0x3a
main.main()
	/home/travis/gopath/src/github.com/iangrunert/git-ratchet/git-ratchet.go:98 +0xc06

After some further testing, I've found that it only happens if the number of removed measures is greater than or equal to the number of added measures:

• Add 1, remove 1 – fails
• Add 1, remove 2 – fails
• Add 2, remove 1 – works

My case:
Using ESLint, if there are no errors for a particular rule (say "no-var") then the ESLint output file won't include this rule (it won't tell us "no-var: 0 errors"). So git-ratchet doesn't know about this rule when we start keeping track of our ESLint results.
If later on that rule comes up with 1 error, git-ratchet counts it as a new measure, which aren't treated as errors, and so it doesn't complain even though it's technically increased from 0 errors to 1.

It would be good if there was a flag we could pass to git-ratchet to handle these cases where we want to count new measures as errors.

I would expect that when I excuse a measure exceeding the slack that a new baseline is set for future checks. Currently this doesn't occur, and the next commit succeeds (because git-ratchet checks hash^1 for the excuses), but the commit after fails.

Reproduction script:

# initialise a new test repo
mkdir /tmp/test-ratchet
cd /tmp/test-ratchet
git init .
echo "Hello" > README
git add README && git commit -m"Initial"

# create an "origin" repo
mkdir /tmp/test-origin
git init --bare /tmp/test-origin
git remote add origin /tmp/test-origin
git push origin master

# initial measure
echo "measure,1"|git ratchet -v check -s 0 -w

# add a commit
echo "Hello" >> README
git commit -am"More"

# measure rising - error exit code as expected
echo "measure,2"|git ratchet -v check -s 0 -w

# excuse the measure
git-ratchet -v excuse -n "measure" -e "Testing"

# measure rising - OK, excused, 0 exit code
echo "measure,2"|git ratchet -v check -s 0 -w

# another commit
echo "Hello" >> README
git commit -am"More"

# measure rising - OK, previous excuse read (hash^1)
echo "measure,2"|git ratchet -v check -s 0 -w

# another commit
echo "Hello" >> README
git commit -am"More"

# measure fails - baseline was not re-written to 1 after excuse
echo "measure,2"|git ratchet -v check -s 0 -w

git-ratchet calls out to git log to get the data for each note in the repository:

git --no-pager log --no-notes --notes=git-ratchet-1-{prefix} --pretty=format:'%H,%an <%ae>,%at,"%N",' HEAD

If a committer's name (the %an part) includes quotation marks, it will break the CSV format that git-ratchet expects. Thus, when git-ratchet passes the output of that git log command to Go's "encoding/csv", it'll throw an error:

> git --no-pager log --show-notes=git-ratchet-1-aprefix --pretty=format:'%H,%an <%ae>,%at,"%N",' HEAD
9a0e08bd28efb18a279bef619436115695c1acad,Christopher Darroch <[email protected]>,1437014747,"",
f14ba5ec39c9b8c3536dd8ac149c5d7d9cdc7700,Chris "Daz" Darroch <[email protected]>,1437355939,"",
8f88af5ab2b93fb5bd2af7f2d1eb7fcca3175234,Christopher Darroch <[email protected]>,1437014595,"first-measure,1,1
second-measure,2,2
",
934fdedf2d97ebb037393af086f3b8b0ff10bc34,Christopher Darroch <[email protected]>,1437014341,"",

> git-ratchet check -p "asge-v1-eslint" -v < input.csv
INFO: 2015/07/28 Reading measures stored in git
INFO: 2015/07/28 git --no-pager log --show-notes=git-ratchet-1-aprefix --pretty=format:'%H,%an <%ae>,%at,"%N",' HEAD
FATAL: 2015/07/28 line 2, column 48: bare " in non-quoted-field

Just exploring this project, and I'm having trouble getting started.

I've downloaded the binary, made it executable, and when I run git ratchet check -w and enter values, nothing seems to happen.

> git ratchet check -w
  test,100

Am I misunderstanding instructions?

It would be useful to verify we've installed the latest version, by having a version command.

Ie:

> git-ratchet version 
git-ratchet version: v0.1.3

Sometimes values in different tests could differ 10-100 times. It's hard to choose a slack value that'd be meaningful in those cases. Would be super cool to be able to use percents!

git-ratchet currently works by assuming metrics are only ever added, and never disappear.

The behaviour of git-ratchet when a metric disappears is to bail hard.

Tools such as ESLint -- which reports on the number of defects or errors -- will not report a metric when the count is zero.

This puts git-ratchet and ESLint (and tools like it) at odds: if you improve your code (i.e., you reduce your code's defects), the numbers go to zero, which causes them to stop being reported, which makes git-ratchet sad.

One possible solution would be to add an --on-missing flag to git-ratchet, whose options could be:

• error [default] : fail (return a non-zero exit code) when a metric is missing from the ratchet input
• set-zero: adjust the value of the metric to zero (0) and continue

Use of this flag would support ESLint's behaviour of only reporting stats with a non-zero error count. This is also beneficial for other tools who will not (or cannot) report rules that did not fail.

For example, something like git-ratchet baseline -p pageweightv1 would show the most recent commit id where the metric was equal to the baseline for each known metric in the most recent ratchet note.

The use case for this is to effectively find the "last known good" commit. I guess normally that commit shouldn't be that far away, but we've neglected our page weight build for a long time and now want to easily identify where the baseline was set. I've done it manually for a few metrics, but it might be useful for others.

Steps to reproduce:

git init .
echo "Hello">README.txt
git add README.txt && git commit -m"Test"
echo "test,1"|git ratchet check -vw
git-ratchet dump # works fine
git notes add -m"Hello"
git-ratchet dump # fails
FATAL: 2015/06/30 Badly formatted measures

It looks like the command git --no-pager log --show-notes=git-ratchet-1-master --pretty=format:'%H,%an <%ae>,%at,"%N",' HEAD picks up notes from the git-ratchet-1-master ref AND the default ref.

According to http://git-scm.com/docs/git-show --show-notes is deprecated and --notes should be used - doing git --no-pager log --notes=git-ratchet-1-master --pretty=format:'%H,%an <%ae>,%at,"%N",' HEAD shows the correct notes.

Howdy,

The git-ratchet installation using go get on our build agents is failing in CI. I am able to reproduce the failure locally, too:

▶ go get gopkg.in/iangrunert/git-ratchet.v0
# gopkg.in/iangrunert/git-ratchet.v0
../../go/src/gopkg.in/iangrunert/git-ratchet.v0/git-ratchet.go:27: not enough arguments in call to cmd.Check
▶ go version
go version go1.4.2 darwin/amd64

The latest v0 release was 20 days ago but the failure only started happening recently, so it be a problem with the way we're installing the package.

Any help or insights appreciated, thanks.