banyanops / collector Goto Github PK

Hi,

Is there any documentation on data privacy? specifically, i would like to know if the data(vulnerability) in our private images goes out of our local environment for analyzing?

Add ability to limit CPU and memory usage, and execution time of the containers under inspection. Maybe disable their networking also.

You mentioned that "For further details about how one might use this in an enterprise, please check out Banyan. This SAAS service offers deeper analysis of your data and provides a dashboard showing which of your images are compliant to your policies (e.g., which of your images have security vulnerabilities, etc.) along with real-time updates and email notifications. " in README.md.But I browsed Banyan, and I can't find the way for deeper analysis.Or where is the SAAS you mentioned in README.md?
(Sorry,my English is not good)

Users have asked for the ability to gather and check information about the Dockerfile and default container settings associated with each image. A registry query in metadata.go gets those container settings but we're discarding that info right now. In addition, Dockerfile info can be obtained from the layer history.

Collector processes every image in a registry for the specified repositories (and all images of all repositories if no repository is specified). Such thoroughness is often overkill. It would help to implement an option to specify N, the maximum number of images to process for each repository. It probably makes sense to process only the newest N images in each repo, and simply report the existence of older ones without processing them.

I'm running the collector as a docker container, and it communicates via a unix socket with the docker daemon. When trying to check an image, it seems like the pullImage action succeeds, but then the collector tries to create a container from the image based on the ID, which fails. I substituted the actual registry and repo with and . I might be doing something wrong, but perhaps you want to take a look. Thanks in advance :)

[13:19:13 2016/01/12 +0000] [INFO] (main.checkRepoList:261) Repolist: /banyandir/hostcollector/repolist not specified
[13:19:13 2016/01/12 +0000] [INFO] (main.checkRepoList:291) Limiting collection to the following repos:
[13:19:13 2016/01/12 +0000] [INFO] (main.checkRepoList:293) <repo>
[13:19:13 2016/01/12 +0000] [INFO] (main.DoIteration:74) Obtained 1 new metadata items in this iteration
[13:19:13 2016/01/12 +0000] [INFO] (collector.(*FileWriter).AppendImageMetadata:72) Appending image metadata to file...
[13:19:13 2016/01/12 +0000] [INFO] (main.DoIteration:105) Starting to apply maxImages limit to repo <repo>
[13:19:13 2016/01/12 +0000] [INFO] (collector.PullImage:31) PullImage downloading /images/create?fromImage=<registry>/<repo>:latest, Image ID: 69f597566a32764bdd546bc2b517e125fa1321e21ae54007e92269a47adae399
[13:19:17 2016/01/12 +0000] [INFO] (collector.createCmd:256) Executing command: docker [PATH=/banyancollector/bin:$PATH bash-static /banyancollector/defaultscripts/pkgextractscript.sh]
[13:19:17 2016/01/12 +0000] [EROR] (except.Error:27) DockerAPI URL: http://example.com/containers/create status code: 404error: No such image: 69f597566a32764bdd546bc2b517e125fa1321e21ae54007e92269a47adae399:latest
 : Error in Remote Docker API call:  /containers/create {"User":"0","AttachStdin":false,"AttachStdout":true,"AttachStderr":true,"Tty":false,"Env":null,"Cmd":["PATH=/banyancollector/bin:$PATH bash-static /banyancollector/defaultscripts/pkgextractscript.sh"],"Entrypoint":["/banyancollector/bin/bash-static","-c"],"Image":"69f597566a32764bdd546bc2b517e125fa1321e21ae54007e92269a47adae399","Labels":null,"WorkingDir":"","HostConfig":{"Binds":["/root/.banyan/hosttarget:/banyancollector:ro"],"Links":null,"Privileged":false,"VolumesFrom":null}}
[13:19:17 2016/01/12 +0000] [EROR] (except.Error:27) DockerAPI URL: http://example.com/containers/create status code: 404error: No such image: 69f597566a32764bdd546bc2b517e125fa1321e21ae54007e92269a47adae399:latest
 : Error in creating container
[13:19:17 2016/01/12 +0000] [EROR] (except.Error:27) DockerAPI URL: http://example.com/containers/create status code: 404error: No such image: 69f597566a32764bdd546bc2b517e125fa1321e21ae54007e92269a47adae399:latest
 : Error in running script:  pkgextractscript.sh
[13:19:17 2016/01/12 +0000] [INFO] (collector.(*FileWriter).WriteImageAllData:37) Writing image (pkg and other) data into file...
[13:19:17 2016/01/12 +0000] [INFO] (main.DoIteration:158) No pulled images left to process in this iteration
[13:19:17 2016/01/12 +0000] [INFO] (main.InfLoop:336) Looping in 60 seconds

Currently you can create a file in $BANYAN_HOST_DIR/hostcollector/repolist with one namespace/repo on each line indicating repos for Collector to inspect. You can change the contents of the list on-the-fly and Collector will eventually read the updated list and process the new repos. A few improvements seem desirable:

• Read the updated file sooner, instead of first processing all images from the previous list of repos.
• Allow users to wildcard the repo field, e.g. banyanops/*

Since collector is long-running, the authentication info in .dockercfg or .docker/config.json can change during a run. Collector should detect such changes and use the updated auth info.

Hi,

    I am trying to use this tool and issued the below command on Ubuntu Linux 14.04 setup. It keeps on iterating showing , image downloaded etc etc. but does not terminate. Where am I going wrong? 

"
sudo docker run --rm -v ~/.docker:/root/.docker -v ~/.dockercfg:/root/.dockercfg -v /var/run/docker.sock:/var/run/docker.sock -v $HOME/.banyan:/banyandir -v /banyancollector/data/userscripts -e BANYAN_HOST_DIR=$HOME/.banyan banyanops/collector registry-1.docker.io banyanops/nginx
"

Attached is the screenshot how it looks on infinite loop. I need this to get it running ASAP. Immediate help would highly be appreciated.

docker run --rm -v ~/.docker:/root/.docker -v ~/.docfg:/root/.dockercfg -v /var/run/docker.sock:/var/run/docker.sock -v $HOME/.banyan:/banyandir -v /home/nehajain/collector/data/userscripts:/banyancollector/data/userscripts -e BANYAN_HOST_DIR=$HOME/.banyan banyanops/collector localhost:50000 --registryproto=v2 

Running this command in the collector directory gives me :

[07:26:44 2019/05/27 +0000] [INFO] (main.checkRepoList:357) Repolist: /banyandir/hostcollector/repolist not specified
[07:26:44 2019/05/27 +0000] [INFO] (collector.DockerAPI:186) DockerAPI http://example.com/version
[07:26:44 2019/05/27 +0000] [INFO] (main.main:466) registry API URL: https://localhost:50000
[07:26:44 2019/05/27 +0000] [INFO] (collector.DockerAPI:186) DockerAPI http://example.com/version
[07:26:44 2019/05/27 +0000] [INFO] (main.main:474) Docker version 18.9.6
[07:26:44 2019/05/27 +0000] [WARN] (except.Warn:52) open /banyandir/hostcollector/imagelist: no such file or directory : Error in opening /banyandir/hostcollector/imagelist : perhaps a fresh start?
[07:26:44 2019/05/27 +0000] [INFO] (main.main:482) Fresh start: No previously collected images were found in /banyandir/hostcollector/imagelist
[07:26:44 2019/05/27 +0000] [WARN] (except.Warn:52) open /banyandir/hostcollector/imagelist_ManifestHash: no such file or directory : Error in opening /banyandir/hostcollector/imagelist_ManifestHash : perhaps a fresh start?
[07:26:44 2019/05/27 +0000] [INFO] (collector.GetImageMetadata:483) Get Repos
[07:26:44 2019/05/27 +0000] [EROR] (except.Error:16) v2 registry search/catalog interface not yet supported in collector

What else should I specify in the command?

Follow-on to issue #5

Collector currently supports the old auth flow for Docker Hub.

Looking forward, we'll need to support the new Registry v2 auth flow.

For this, what would be a good Authorization Server setup to develop and test against?

[13:24:10 2016/01/05 -0800] [EROR] (except.Error:27) Get http://example.com/version: dial unix /var/run/docker.sock: connect: no such file or directory :DockerAPI URL http://example.com/version client request failed

I'm not sure what example.com is...not specifing that via the CLI at all:

sudo COLLECTOR_DIR=$PWD $GOPATH/bin/collector http://REDACTED hello-appname

Any ideas?

I tried running a vanilla setup, and got the following error:
[17:14:40 PDT 2015/05/14] [EROR] (main.runAllScripts:107) Post http://example.com/containers/create: dial unix /var/run/docker.sock: no such file or directory : Error in running script: listUsers.py
[17:14:40 PDT 2015/05/14] [EROR] (main.runAllScripts:107) Post http://example.com/containers/create: dial unix /var/run/docker.sock: no such file or directory : Error in running script: dotfilescript.sh

Ran with:
./collector -u data/userscripts/ index.docker.io library/busybox

Seems that all scripts were failing:
[17:14:40 PDT 2015/05/14] EROR Post http://example.com/containers/create: dial unix /var/run/docker.sock: no such file or directory :doDockerAPI URL http://example.com/containers/create client request failed
[17:14:40 PDT 2015/05/14] EROR Post http://example.com/containers/create: dial unix /var/run/docker.sock: no such file or directory : Error in Remote Docker API call: /containers/create
[17:14:40 PDT 2015/05/14] EROR Post http://example.com/containers/create: dial unix /var/run/docker.sock: no such file or directory : Error in creating container
[17:14:40 PDT 2015/05/14] EROR Post http://example.com/containers/create: dial unix /var/run/docker.sock: no such file or directory : Error in running script: pkgextractscript.sh

Got the same behaviour when run inside a container...

When I tried to run the command with go installed, there is an errror:

$go get -u github.com/banyanops/collector/...

# github.com/banyanops/collector/cmd/collector
../go/src/github.com/banyanops/collector/cmd/collector/main.go:406:31: not enough arguments in call to log4go.NewFileLogWriter
	have (string)
	want (string, bool)

Anyone can help me?
Besides, the version of go is go1.13.4.

$go version
go version go1.13.4 linux/amd64

Quay.IO doesn't support all the same APIs as Private Registry v1.

We probably need to treat it a totally new kind of Registry.

Hello,

I followed the steps you provided for go:

After running go get -v -u github.com/banyanops/collector/ I went ahead and cd to the source directory and ran the command you specified to run the collector. I get the following error:

sudo: /Users/sbehrens/mygo/bin/collector: command not found

I looked and I do not have a bin directory in the GOPATH. Was there another step that I was supposed to do that wasn't in the documentation?

go get bitbucket.org/banyanops/mysqllib

package bitbucket.org/banyanops/mysqllib: https://api.bitbucket.org/1.0/repositories/banyanops/mysqllib: 403 FORBIDDEN

Collector currently connects to the Docker daemon via the Docker UNIX socket. It should be extended to connect to a TCP endpoint as well with correct use of certs for TLS.

After updating Docker on my Macbook to version 17.03.0-ce, I am now getting the following error.

[19:05:34 2017/03/19 -0400] [EROR] (except.Error:16) strconv.ParseInt: parsing "0-ce": invalid syntax
[19:05:34 2017/03/19 -0400] [EROR] (except.Fail:18) Could not determine Docker version

A (new?) feature in Google Cloud lets you add a service account JSON key to your $HOME/.docker/config.json by issuing "docker login" with special arguments as described here (https://cloud.google.com/container-registry/docs/auth).

So the setup of Docker for GCR now looks conventional. We will simplify Collector, removing special flags for GCR and packages imported from Kubernetes. Collector special support for GCE metadata authentication will also go away as part of this change.

docker run --rm -v ~/.docker:/root/.docker -v ~/.docfg:/root/.dockercfg -v /var/run/docker.sock:/var/run/docker.sock -v $HOME/.banyan:/banyandir -v /home/nehajain/collector/data/userscripts:/banyancollector/data/userscripts -e BANYAN_HOST_DIR=$HOME/.banyan banyanops/collector localhost:50000 --registryproto=v2 

Running this command in the collector directory gives me :

[07:26:44 2019/05/27 +0000] [INFO] (main.checkRepoList:357) Repolist: /banyandir/hostcollector/repolist not specified
[07:26:44 2019/05/27 +0000] [INFO] (collector.DockerAPI:186) DockerAPI http://example.com/version
[07:26:44 2019/05/27 +0000] [INFO] (main.main:466) registry API URL: https://localhost:50000
[07:26:44 2019/05/27 +0000] [INFO] (collector.DockerAPI:186) DockerAPI http://example.com/version
[07:26:44 2019/05/27 +0000] [INFO] (main.main:474) Docker version 18.9.6
[07:26:44 2019/05/27 +0000] [WARN] (except.Warn:52) open /banyandir/hostcollector/imagelist: no such file or directory : Error in opening /banyandir/hostcollector/imagelist : perhaps a fresh start?
[07:26:44 2019/05/27 +0000] [INFO] (main.main:482) Fresh start: No previously collected images were found in /banyandir/hostcollector/imagelist
[07:26:44 2019/05/27 +0000] [WARN] (except.Warn:52) open /banyandir/hostcollector/imagelist_ManifestHash: no such file or directory : Error in opening /banyandir/hostcollector/imagelist_ManifestHash : perhaps a fresh start?
[07:26:44 2019/05/27 +0000] [INFO] (collector.GetImageMetadata:483) Get Repos
[07:26:44 2019/05/27 +0000] [EROR] (except.Error:16) v2 registry search/catalog interface not yet supported in collector

What else should I specify in the command?

It would be useful to add a mode that would allow Collector to detect and gather information from local images on a Docker host instead of having to pull from a registry.

Hi,
I've been trying to run collector in the standalone mode, for a local image, that I had pulled from dockerhub previously. However, I'm getting the following errors, about it being unable to read the docker config file. I get the same error if I try with the example given in collector --help.

[DEBG] (main.doFlags:63) Creating directory: /home/chaitanya/.banyan [DEBG] (main.doFlags:63) Creating directory: /home/chaitanya/.banyan/hostcollector [DEBG] (main.doFlags:63) Creating directory: /home/chaitanya/.banyan/hostcollector [DEBG] (main.doFlags:63) Creating directory: /home/chaitanya/.banyan/hostcollector/banyanout [DEBG] (main.doFlags:63) Creating directory: /home/chaitanya/.banyan/hosttarget/defaultscripts [DEBG] (main.doFlags:63) Creating directory: /home/chaitanya/.banyan/hosttarget/userscripts [DEBG] (main.doFlags:63) Creating directory: /home/chaitanya/.banyan/hosttarget/bin [INFO] (main.checkRepoList:357) Repolist: /home/chaitanya/.banyan/hostcollector/repolist not specified [INFO] (main.checkRepoList:387) Limiting collection to the following repos: [INFO] (main.checkRepoList:389) sassmann/debian-firefox [INFO] (collector.DockerAPI:185) DockerAPI http://example.com/version [EROR] (except.Error:23) Could not read /home/chaitanya/.docker/config.json, trying $HOME/.dockercfg [EROR] (except.Fail:22) Could not read%!(EXTRA string=/home/chaitanya/.dockercfg)
Any ideas as to what am I missing? Also let me know, if there exists a mailing-list or something where asking this would be more appropriate.
Thanks!

Docker 1.7 saves login credentials to $HOME/.docker/config.json instead of $HOME/.dockercfg. Collector needs to be updated to handle this correctly depending on the Docker version.

The latest commit that moved things into a fsutil package (f2f5aeb) broke builds for me, while the previous commit (33d25bc) works fine. Output from go get -u github.com/banyanops/collector/..., the command recommended in the "Getting started" section, is

package github.com/banyanops/collector
    imports github.com/banyanops/collector/config
    imports github.com/docker/docker/pkg/homedir
    imports github.com/docker/libcontainer/user
    imports github.com/docker/docker/pkg/mflag
    imports github.com/banyanops/collector/fsutil
    imports github.com/banyanops/collector/fsutil
    imports github.com/banyanops/collector/fsutil: cannot find package "github.com/banyanops/collector/fsutil" in any of:
    /usr/lib/go/src/pkg/github.com/banyanops/collector/fsutil (from $GOROOT)
    /home/jfktrey/Docker/collector/src/github.com/banyanops/collector/fsutil (from $GOPATH)

I followed the instructions to clone from git,
and ran the following command:

I have docker running on the host and can start containers with the same image as banyanops/nginx

But when I ran:
sudo COLLECTOR_DIR=$PWD $GOPATH/bin/collector index.docker.io banyanops/nginx

It goes into this infinite loop of attempting to get some metadata from docker hub (as shown
below)
This has been going on for 8 hours or more. What am I doing wrong?

Thanks for you help!

[23:44:15 2015/06/17 -0500] INFO Looping in 60 seconds
[23:45:15 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:45:16 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:45:18 2015/06/17 -0500] INFO No new metadata in this iteration
[23:45:18 2015/06/17 -0500] INFO Looping in 60 seconds
[23:46:18 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:46:19 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:46:20 2015/06/17 -0500] INFO No new metadata in this iteration
[23:46:20 2015/06/17 -0500] INFO Looping in 60 seconds
[23:47:20 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:47:21 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:47:23 2015/06/17 -0500] INFO No new metadata in this iteration
[23:47:23 2015/06/17 -0500] INFO Looping in 60 seconds
[23:48:23 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:48:24 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:48:25 2015/06/17 -0500] INFO No new metadata in this iteration
[23:48:25 2015/06/17 -0500] INFO Looping in 60 seconds
[23:49:25 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:49:26 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:49:28 2015/06/17 -0500] INFO No new metadata in this iteration
[23:49:28 2015/06/17 -0500] INFO Looping in 60 seconds
[23:50:28 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:50:29 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:50:30 2015/06/17 -0500] INFO No new metadata in this iteration
[23:50:30 2015/06/17 -0500] INFO Looping in 60 seconds
[23:51:30 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:51:31 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:51:33 2015/06/17 -0500] INFO No new metadata in this iteration
[23:51:33 2015/06/17 -0500] INFO Looping in 60 seconds
[23:52:33 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:52:34 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:52:37 2015/06/17 -0500] INFO No new metadata in this iteration
[23:52:37 2015/06/17 -0500] INFO Looping in 60 seconds
[23:53:37 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:53:39 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:53:41 2015/06/17 -0500] INFO No new metadata in this iteration
[23:53:41 2015/06/17 -0500] INFO Looping in 60 seconds
[23:54:41 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:54:42 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:54:43 2015/06/17 -0500] INFO No new metadata in this iteration
[23:54:43 2015/06/17 -0500] INFO Looping in 60 seconds
[23:55:43 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:55:44 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:55:46 2015/06/17 -0500] INFO No new metadata in this iteration
[23:55:46 2015/06/17 -0500] INFO Looping in 60 seconds
[23:56:46 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:56:46 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:56:48 2015/06/17 -0500] INFO No new metadata in this iteration
[23:56:48 2015/06/17 -0500] INFO Looping in 60 seconds
[23:57:48 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:57:49 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:57:50 2015/06/17 -0500] INFO No new metadata in this iteration
[23:57:50 2015/06/17 -0500] INFO Looping in 60 seconds
[23:58:50 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:58:51 2015/06/17 -0500] INFO Get Tags and Metadata from Docker Hub
[23:58:53 2015/06/17 -0500] INFO No new metadata in this iteration
[23:58:53 2015/06/17 -0500] INFO Looping in 60 seconds

[23:59:53 2015/06/17 -0500] INFO Get Repos from Docker Hub
[23:59:54 2015/06/17 -0500] INFO Get Tags and Metadata from
[23:59:55 2015/06/17 -0500] INFO No new metadata in this iteration
[23:59:55 2015/06/17 -0500] INFO Looping in 60 seconds
[00:00:55 2015/06/18 -0500] INFO Get Repos from Docker Hub
[00:00:56 2015/06/18 -0500] INFO Get Tags and Metadata from Docker Hub
[00:00:57 2015/06/18 -0500] INFO No new metadata in this iteration
[00:00:57 2015/06/18 -0500] INFO Looping in 60 seconds

The version of git (1.7.1) on CentOS 6 seems to be too old to handle gopkg.in packages, and so go get -u github.com/banyanops/collector hangs after successfully downloading everything except gopkg.in/yaml.v2. A workaround is to manually download the yaml.v2 package:

git clone --branch v2 https://github.com/go-yaml/yaml $GOPATH/src/gopkg.in/yaml.v2

After that, cd $GOPATH/src/github.com/banyanops/collector; go install should work.

Currently, Collector uses token authentication to query the Docker Hub index service and registries. Other services like Google Container Registry use the same protocol, and so Collector should recognize that and use token authentication for these services, too, instead of using it only for Docker Hub.