release-engineering / retrodep Goto Github PK

Run the Go vendoring process in reverse

License: GNU General Public License v3.0

Go 96.16% Makefile 1.59% Shell 2.25%

retrodep's Introduction

Retrodep

This command inspects a Go source tree with vendored packages and attempts to work out the versions of the packages which are vendored, as well as the version of top-level package itself.

It does this by comparing file hashes of the packages with those from the upstream repositories.

If no semantic version tag matches but a commit is found that matches, a pseudo-version is generated.

Installation

go get github.com/release-engineering/retrodep

Running

retrodep: help requested
usage: retrodep [OPTION]... PATH
  -debug
    	show debugging output
  -deps
    	show vendored dependencies (default true)
  -diff string
    	compare with upstream ref (implies -deps=false)
  -exclude-from exclusions
    	ignore directory entries matching globs in exclusions
  -help
    	print help
  -importpath string
    	top-level import path
  -o string
    	output format, one of: go-template=...
  -only-importpath
    	only show the top-level import path
  -template string
    	go template to use for output with Reference fields (deprecated)
  -x	exit on the first failure

In many cases retrodep can work out the import path for the top-level project. In those cases, simply supply the directory name to examine:

$ retrodep src

If it cannot determine the import path, provide it with -importpath:

$ retrodep -importpath github.com/example/name src

By default both the top-level project and its vendored dependencies are examined. To ignore vendored dependencies supply -deps=false:

$ retrodep -deps=false -importpath github.com/example/name src

If there are additional local files not expected to be part of the upstream version they can be excluded:

$ cat exclusions
.git
Dockerfile
$ ls -d src/Dockerfile src/.git
src/Dockerfile
src/.git
$ retrodep -exclude-from=exclusions src

Exit code

Exit code	Reason
0	all versions were found (or -diff: no changes)
1	any error was encountered other than those below
2	a version was missing
3	import path needed but not supplied
4	no Go source code was found at the provided path
5	in -diff mode, changes were found

Example output

$ retrodep $GOPATH/src/github.com/docker/distribution
github.com/docker/distribution:v2.7.1
github.com/docker/distribution:v2.7.1/github.com/Azure/azure-sdk-for-go:v16.2.1
github.com/docker/distribution:v2.7.1/github.com/Azure/go-autorest:v10.8.1
github.com/docker/distribution:v2.7.1/github.com/Shopify/logrus-bugsnag:v0.0.0-0.20171204154709-577dee27f20d
github.com/docker/distribution:v2.7.1/github.com/aws/aws-sdk-go:v1.15.11
github.com/docker/distribution:v2.7.1/github.com/beorn7/perks:v0.0.0-0.20160804124726-4c0e84591b9a
github.com/docker/distribution:v2.7.1/github.com/bshuster-repo/logrus-logstash-hook:0.4
github.com/docker/distribution:v2.7.1/github.com/bugsnag/bugsnag-go:v1.0.3-0.20150204195350-f36a9b3a9e01
...

In this example,

github.com/docker/distribution is the top-level package, and the upstream semantic version tag v2.7.1 matches
github.com/Azure/azure-sdk-for-go etc are vendored dependencies of distribution
github.com/Azure/azure-sdk-for-go, github.com/Azure/go-autorest, github.com/aws/awk-sdk-go, and github.com/bshuster-repo/logrus-logstash-hook all had matches with upstream semantic version tags
github.com/bugsnag/bugsnag-go matched a commit from which tag v1.0.2 was reachable (note: v1.0.2, not v1.0.3 -- see below)
github.com/beorn7/perks matched a commit from which there were no reachable semantic version tags

Pseudo-versions

The pseudo-versions generated by this tool are:

v0.0.0-0.yyyyddmmhhmmss-abcdefabcdef (commit with no relative tag)
vX.Y.Z-pre.0.yyyyddmmhhmmss-abcdefabcdef (commit after semver vX.Y.Z-pre)
vX.Y.(Z+1)-0.yyyyddmmhhmmss-abcdefabcdef (commit after semver vX.Y.Z)
tag-1.yyyyddmmhhmmss-abcdefabcdef (commit after tag)

Diff mode

When supplying the -diff option, retrodep compares with a specific version only, and outputs the differences (in unified diff format) between the local files and the upstream files.

To compare source code in src with a known upstream version of a package, use it like this:

$ retrodep -diff v1.2.0 github.com/example/name src

No output (and a zero exit code) means the source code in src matches the upstream version v1.2.0 of github.com/example/name. Otherwise, the differences in src compared to the upstream version are shown in unified diff format, and the exit code is 5.

Files in src that are not in the upstream version are presented as diffs compared with "/dev/null". Files in the upstream version but not in src are ignored.

Limitations

The vendor directory is assumed to be complete.

Original source code is assumed to be available.

Only git and repositories are currently supported, and working 'git' and 'hg' executables are assumed to be available.

Non-Go code is not considered, e.g. binary-only packages, or CGo.

Commits with additional files (e.g. *_linux.go) are identified as matching when they should not.

Packages vendored from forks will not have matching commits.

Files marked as "export-subst" in .gitattributes files in the vendored copy are ignored.

retrodep's People

Contributors

Stargazers

Watchers

Forkers

ralphcorderoy rmohr jasinner vkrizan litleleprikon mmirecki ejegrova nirzari cachito-testing tkdchen tichavskym psturc

retrodep's Issues

Failed to run go get

go get fails with the following message:

 > go get github.com/release-engineering/backvendor
# github.com/release-engineering/backvendor/backvendor
go/src/github.com/release-engineering/backvendor/backvendor/gosource.go:112:9: undefined: yaml.NewDecoder

Write errors to stderr

https://github.com/release-engineering/backvendor/blob/3a4f5a07095ffdb746b2f4855ee6d8515855516c/main.go#L65

print the version once if there's an exact match to semantic version

In the example output in the README there is this line:

github.com/opencontainers/image-spec@87998cd070d9e7a2c79f8b153a26bea0425582e5 =v1.0.0 ~v1.0.0

Why do we bother printing ~v1.0.0 does it have any meaning?

Delete temporary directory on failure in NewWorkingTree

Several failure paths leave the temporary directory there.

.gitattributes processed regardless of whether repository uses git

The .gitattributes files should only be processed if the upstream repository came from git.

Remove FileHash root

This is never used.

Rev and Ver swapped sometimes?

I'm seeing a strange issue when running the backvendor tool on one repo:

backvendor -exclude-from=exclusions -importpath github.com/brancz/kube-rbac-proxy .
..
github.com/PuerkitoBio/[email protected] ~8a290539e2e8629dbc4e6bad948158f790ec31f4

Notice the Rev, and Ver seems to be swapped, it happens for all vendored dependencies, but only on this project.

Use filepath.Rel() to get relative path

https://github.com/release-engineering/backvendor/blob/3a4f5a07095ffdb746b2f4855ee6d8515855516c/backvendor/filehash.go#L121
https://github.com/release-engineering/backvendor/blob/3a4f5a07095ffdb746b2f4855ee6d8515855516c/backvendor/vendored.go#L51

Support detecting release maintenance branches

Some projects (e.g. Kubernetes) use maintenance branches (e.g. release-1.9) to track backports to stable versions in between stable releases. It might be useful to recognize these in some capacity,

I get "line not understood" error

When running backvendor from master against kubevirt.io/kubevirt I get the following error:

2018/09/21 17:39:32 /home/rmohr/go/src/kubevirt.io/kubevirt: line not understood: 100644 blob 483f8ffa1669545951f89ccdc5372013edc05771	vendor/github.com/russross/blackfriday/testdata/Amps and angle encoding.html

That did not happen two days ago.

Godep changes source code it vendors

When godep vendors source code it removes "import comments" from package statements. We will need to accommodate this somehow.

Perhaps we need to

identify whether the top-level package using godep (look for Godeps/Godeps.json)
if so, for each upstream tag/commit in a vendored project, try first with 'git ls-tree ...' and then if that fails try building a FileHashes map from files in the working tree (that match the subset we are looking for) that we modify in the same way as godep

Show dependency layers in output

It would be great if we could nest the output so that we could see easily which package was responsible for pulling in another package.

Using grafana as an example. In the vendor directory it has a dependency:

github.com/mattn/go-colorable

That currently shows up in the output like this:

github.com/mattn/go-colorable@167de6bfdfba052fa6b2d3664c8f5272e23c9072 =v0.0.9 ~v0.0.9

However extra analysis of he vendor directory could tell us that it's not used directly by grafana but is used as part of both:

github.com/inconshreveable/log15
github.com/fatih/color

So potentially we could nest the output to show that:

github.com/inconshreveable/log15@0decfc6c20d9ca0ad143b0e89dcaa20f810b4fb3 =v2.13 ~v2.13 / github.com/mattn/go-colorable@167de6bfdfba052fa6b2d3664c8f5272e23c9072 =v0.0.9 ~v0.0.9

github.com/fatih/color@5b77d2a35fb0ede96d138fc9a99f5c9b6aef11b4 =v1.7.0 ~v1.7.0 / github.com/mattn/go-colorable@167de6bfdfba052fa6b2d3664c8f5272e23c9072 =v0.0.9 ~v0.0.9

This would be a 'nice-to-have' feature. I wouldn't consider it a high priority as the analysis team can work out the dependency hierarchy by inspecting the source code. It might even be better suited to a unique tool, not backvendor itself.

Use go.mod to discover import path

Each go.mod file in the tree defines a separately-versioned project. We should:

search recursively for go.mod files not in vendor directories or directories which themselves contain a go.mod file
if any are found, use this as the list of projects (instead of searching each subdirectory of the provided dir)
for each found go.mod, extract the module name from it to use as the import path

Use the real parser to find ImportComment

Should use the authoritative go/build parser to find Package.ImportComment instead.

Allow JSON output

As suggested in #96, it might be useful to have JSON output as an option.

Add a -diff option

If we know which version we expect to find but retrodep fails to agree, it would be useful to be able to ask it for the full list of differences between upstream and local.

It could work like this: retrodep -diff VER DIR

The -diff option implies -deps=false, and only compares against the given upstream version VER. For each difference it finds it should invoke diff -u to show it. It should not stop at the first difference it finds but keep going until all have been shown.

Use go.mod to find expected versions of dependencies

Following on from #106, we should read the requirements from go.mod in order to determine the initial version to attempt matching against. (This is similar to the way we read glide.yaml for this information.)

Retrodep fails if retrieving any of the repo's fails.

When resolving any of the repo's fails, retrodep will exit without printing any results.
I would rather expect that it will continue, printing the version of the problematic package as "?".
Some of the packages that cause an issue:

kubevirt.io/qe-tools/pkg/ginkgo-reporters
kubevirt.io/qe-tools/pkg/polarion-xml

#122 is an interim patch that does not print the problematic packages, but at least allows retrodep to continue with the correct ones.

Check file errors for IsNotExist

https://github.com/release-engineering/backvendor/blob/3a4f5a07095ffdb746b2f4855ee6d8515855516c/backvendor/filehash.go#L95
https://github.com/release-engineering/backvendor/blob/3a4f5a07095ffdb746b2f4855ee6d8515855516c/backvendor/gosource.go#L69
https://github.com/release-engineering/backvendor/blob/3a4f5a07095ffdb746b2f4855ee6d8515855516c/backvendor/gosource.go#L95
https://github.com/release-engineering/backvendor/blob/3a4f5a07095ffdb746b2f4855ee6d8515855516c/backvendor/vendored.go#L94

Template the top level output

As a follow up to issue #33, include the top level package in the template.

Detection of current import path seems to fail for common case

The use case of "I want to run backvendor on this project" seems like a common use case, but seems to fail for me unless I manually pass the -importpath. For example

$ go version
go version go1.11 linux/amd64
$ echo $GOPATH
/home/directxman12/dev/atomic/go-path
$ pwd
/home/directxman12/dev/atomic/go-path/src/github.com/openshift/service-idler
$ /path/to/backvendor
backvendor: missing path
usage: backvendor [-help] [-importpath=toplevel] [-deps=false] path
$ /path/to/backvendor .
.: unable to determine import path
Provide import path with -importpath
$ cd $GOPATH/src && /path/to/backvendor github.com/openshift/service-idler
./github.com/openshift/service-idler: unable to determine import path
Provide import path with -importpath

These all seem like pretty standard invocations with an easily discoverable import path that backvendor should probably support without a manual -importpath argument.

Package backvendor should not write directly to stderr

When executed commands fail the output should not be written to stderr but returned to the caller in an error.