Giter Club home page Giter Club logo

git-all-secrets's People

Contributors

anshumanbh avatar ballinballen avatar lanzay avatar matt-cox-simplisafe avatar sam-martin avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

git-all-secrets's Issues

Small code clarification

Hello, I miss to understand the reason of this code

		func(orgclone *sync.WaitGroup, urlToClone string, directory string) {
			enqueueJob(func() {
				gitclone(urlToClone, directory, orgclone)
			})
		}(&orgclone, urlToClone, directory)

Is there a particular reason to not doing this?


			enqueueJob(func() {
				gitclone(urlToClone, directory, &orgclone)
			})

After cloning I get error

panic: exit status 128
goroutine 110 [running]:
main.check(0x7c8460, 0xc4201922c0)
/data/main.go:40 +0x174
main.gitclone(0xc42024d3e0, 0x24, 0xc42016f4c0, 0x15, 0xc42022ffe0)
/data/main.go:53 +0x108
created by main.cloneorgrepos
/data/main.go:92 +0x429

Windows 10 git-all-secrets docker is being used.

No Result

Hello @anshumanbh ,

I run git-all-secrets on some repo which already having some sensitive info but generated the results file is showing blank..no result.

I ran ./git-all-secrets -token=xxxmytokenxxxx -repoURL=https://github.com/some/info -toolName=thog -thogEntropy

Cloning of: https://github.com/some/info finished

Starting to scan: https://github.com/some/info

Finished truffleHog Scanning for: some_info
Scanning of: https://github.com/some/info finished

Combining the output into one file

Can you please help me out on this issue.

Running without docker

First of all, thanks for creating and working on this tool, it’s really cool job.

But i’ve got a question- is it possible for running this tool without docker?

Have a nice day, ultras

Username for 'https://github.com': panic: signal: killed

When I run the below

docker run -it abhartiya/tools_gitallsecrets:v6 -token=REDACTED -org=REDACTED -output=results.txt

It starts to list the repos and fails with the below

Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': panic: signal: killed

goroutine 905 [running]:
main.check(0x7fa4c0, 0xc4203b64c0)
	/data/main.go:41 +0x100
main.gitclone(0xc420be2e80, 0x38, 0xc420ea5a20, 0x1e, 0xc420c00f10)
	/data/main.go:54 +0x108
created by main.cloneorgrepos
	/data/main.go:93 +0x405

container not authenticating after mapping ssh key

I am mapping my ssh key as shown in the documentation, it seems like the container isn't using the correct username when running the following:

docker run -it -v ~/.ssh/id_rsa:/root/.ssh/id_rsa abhartiya/tools_gitallsecrets -token=REDACTED -org=REDACTED 
Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Use
rname for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com':
Password for 'https://github.com': signal: interrupt: Cloning into '/tmp/repos/org/.........REDACTED

Did I miss something?

add support for gitlab

Gitlab is getting more and more popular. Your tool does exactly what we need at our environment, but we have on-premise gitlab, so as I understood from documentation it's not suitable.

Just a suggestion in case you are actively working on it.

"results.txt" has shell color characters in it

Example:

Tool: truffleHog
OrgorUser: terraform RepoName: fluentd-sumologic

^[[92mReason: Generic Password^[[0m
^[[92mDate: 2018-03-28 21:58:30^[[0m
^[[92mHash: e05234bc366d0ab0f5174f1085d84c032dd8b8a6^[[0m
^[[92mFilepath: aaa/bbb.txt^[[0m
^[[92mBranch: origin/master^[[0m
^[[92mCommit: Updated s3 key for remote state
^[[0m
^[[93m+  password = "admin"^[[0m

Crash when analysing huge organization

Hi !

I'm getting this crash when I try to run the tool over a "huge" org (>140 repositories). Just after:

Scanning all org repositories now..This may take a while so please be patient

panic: exit status 1

goroutine 2727 [running]:
main.check(0x7c8460, 0xc4201e2020)
        /data/main.go:40 +0x174
main.runTrufflehog(0xc420164000, 0x1f, 0xc4201d7710, 0xf, 0x7ffce0564ec0, 0x6, 0x0, 0x0)
        /data/main.go:213 +0x1d1
main.runGitTools(0x6b46b2, 0x3, 0xc420164000, 0x1f, 0xc4203b51f0, 0xc4201d7710, 0xf, 0x7ffce0564ec0, 0x6)
        /data/main.go:233 +0x1af
created by main.scanorgrepos
        /data/main.go:379 +0x209

Any idea on how can I help to debug this crash?

Scanning Private Repos are failing

After the newest update of my docker image, I am no longer able to use my ssh key to scan private repos. When I try, I am given the following stack trace during the cloning process:

exit status 128: Cloning into '/tmp/repos/org/[REDACTED]'...

fatal: cannot run ssh: No such file or directory
fatal: unable to fork

panic: exit status 128

goroutine 61 [running]:
main.gitclone(0xc420469950, 0x26, 0xc4200b3110, 0x22, 0xc42026b280)
	/go/src/github.com/anshumanbh/git-all-secrets/main.go:97 +0x3bf
main.executeclone.func1.1()
	/go/src/github.com/anshumanbh/git-all-secrets/main.go:137 +0x4e
main.enqueueJob.func1(0xc420553c80)
	/go/src/github.com/anshumanbh/git-all-secrets/main.go:67 +0x27
created by main.enqueueJob
	/go/src/github.com/anshumanbh/git-all-secrets/main.go:66 +0x5b

I imaged my machine before I pulled the latest image, and the same command runs fine. For context, here is the command I am trying:
docker run -it -v ~/.ssh/[REDACTED]:/root/.ssh/id_rsa abhartiya/tools_gitallsecrets -token=[REDACTED] -org=[REDACTED] -scanPrivateReposOnly -orgOnly

Improper line breaks in /root/.ssh/config

The line breaks that the Dockerfile is putting in the /root/.ssh/config file are not being interpreted properly.

# cat /root/.ssh/config
Host *github.com \n  IdentitiesOnly yes \n  StrictHostKeyChecking no \n  UserKnownHostsFile=/dev/null \n  IdentityFile /root/.ssh/id_rsa \n  \n Host github.*.com \n  IdentitiesOnly yes \n  StrictHostKeyChecking no \n  UserKnownHostsFile=/dev/null \n  IdentityFile /root/.ssh/id_rsa

Question regarding performance/efficiency

While the tool has the ability to scan all possible repo's related to an org, it takes quite a while to complete. I tried using just the trufflehog scanning portion of the tool, but it still manages to take weeks at a time to scan a specific org. I understand it may be due to repo sizes. I'm curious to find out whether there are any ways to fine tune the performance of these scans.

404 Not Found

docker run -it abhartiya/tools_gitallsecrets -token= -org cnooc -orgOnly 1
Org was specified combined with orgOnly, the tool will proceed to scan only the org repos and nothing related to its users
Cloning the repositories of the organization: cnooc
If the token provided belongs to a user in this organization, this will also clone all public AND private repositories of this org, irrespecitve of the scanPrivateReposOnly flag being set..
panic: GET https://api.github.com/orgs/cnooc/repos?per_page=10: 404 Not Found []

goroutine 1 [running]:
main.check(0x71ee20, 0xc420528050)
/go/src/github.com/anshumanbh/git-all-secrets/main.go:80 +0xec
main.cloneorgrepos(0x721cc0, 0xc4200a0010, 0xc4200b8fc0, 0x7ffe5d12cf39, 0x5, 0x0, 0x0)
/go/src/github.com/anshumanbh/git-all-secrets/main.go:158 +0x1b9
main.main()
/go/src/github.com/anshumanbh/git-all-secrets/main.go:953 +0x2d6

flag provided but not defined: -orgOnly

Defined -orgOnly=1 but getting an error: "flag provided but not defined: -orgOnly"

sudo docker run -it abhartiya/tools_gitallsecrets:v3 -token=<> -orgOnly=1 -org=<>

Tool requires the github user/pass ?

Hi,

I reopen this issue, firstly, I want to confirm my scanning on my private repo (not other person's repo for any illegitimate purpose).

I have generated token from "https://github.com/settings/tokens" in section "Personal access tokens". But the tool still require to authenticate with github account

Any hint ? Thanks

Best Regards,
VietNC

orgURL could be another option

For enterprise github (specific to a company), there isn't a way to pass both the orgName and enterprise github domain URL. Currently, this component could be used for just one repo at a time for an ORG.

Not able to clone repos that are being migrated

There are cases when certain repos cannot be cloned because they are being migrated or whatever. In those cases, git-all-secrets needs to move on.

An example error message - "Access to this repository has been disabled while it is being migrated.". Repo is - https://github.com/walmartlabs/grunt-castle

Is it possible to exclude some folders in repo from scanning?

We are using Sitecore Unicorn which serializes data from database to files on disk and we are checking this files in repository.
File structure includes GUID values which are treated as positives. And we have a lot of files like this, scattered in folders through whole solution.
Hence a question - is it possible to exclude from scanning folders, which names starts/contains/matches given string?

Tool requires the github user/pass ?

Hi, When I try to run git-all-secrets, it requires the github user/pass, but when i watch the video demo, it doesn't.

Even I provided the correct github user/pass, it still can't run :

$ docker run --rm -it abhartiya/tools_gitallsecrets:v6 -org=MyOrg -token=d2d2cf23d24d9e267d88d08b34caf48bne895776 -output=results.txt
Since org was provided, the tool will proceed to scan all the org repos, then all the user repos and user gists in a recursive manner
Cloning the repositories of the organization: MyOrg
[...skip repo info...]
Username for 'https://github.com': Username for 'https://github.com': [email protected]
Password for 'https://[email protected]@github.com': Password for 'https://[email protected]':
panic: exit status 128

goroutine 28 [running]:
main.check(0x7fa4c0, 0xc420286780)
	/data/main.go:41 +0x100
main.gitclone(0xc4202feac0, 0x32, 0xc42030a9a0, 0x19, 0xc420280510)
	/data/main.go:54 +0x108
created by main.cloneorgrepos
	/data/main.go:93 +0x405

Anyone can help ? Thanks

Best Regards,
VietNC

not able to get output

cat results.txt
Tool: truffleHog
Tool: repo-supervisor


when i see result from results.txt . everytime above output shown

scan organization repos meet "index out of range"

When "Done cloning org repos." and then start "Listing users of the organization and their repositories and gists" it shows:

panic: runtime error: index out of range

goroutine 1 [running]:
main.cloneusergists(0x721c20, 0xc420018028, 0xc42008efc0, 0xc420374ac0, 0xb, 0x0, 0x0)
	/go/src/github.com/anshumanbh/git-all-secrets/main.go:249 +0x595
main.main()
	/go/src/github.com/anshumanbh/git-all-secrets/main.go:975 +0x77a

feature to exclude some gits from cloning

Hi,

Thanks for this amazing tool and automation! Much appreciate it.

Sometimes, when cloning company's gits we get lots of "unwanted" repo forked by employees. So, would be possible to exclude it from cloning the line command ?

Cheers

Issue with scanning repositories using git-all-secrets

When I am trying to scan all the repositories in our organization, facing the below issue:

$sudo docker run -it -v ~/.ssh/id_rsa abhartiya/tools_gitallsecrets -token=mytoken -org=myorg -scanPrivateReposOnly
.........
.........

Warning: Permanently added 'github.com,192.30.253.112' (RSA) to the list of known hosts.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0755 for '/root/.ssh/id_rsa' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
key_load_private_type: bad permissions
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

panic: exit status 128

goroutine 462 [running]:
main.gitclone(0xc420435b60, 0x2c, 0xc4201fb6a0, 0x1e, 0xc420540a10)
/data/main.go:75 +0x387
main.executeclone.func1.1()
/data/main.go:106 +0x4e
main.enqueueJob.func1(0xc42083d590)
/data/main.go:45 +0x27
created by main.enqueueJob
/data/main.go:47 +0x6a

All the files in .ssh folder has 600 permission.

Can you suggest anything to solve this issue? If you can post a video in youtube for this, that will be a great help for everyone. Thank you.

Request for feature - Web Interface option for results

Hi, can there be an option for the results to be put into a Bootstrap GUI, like GitRob?

Example: Having it part of the main program as an argument that spins up Bootstrap and automatically parses the results, or have a second executable that parses the results.txt into Bootstrap.

Username / Password authentication not working

Hi, when I run docker run -it abhartiya/tools_gitallsecrets -token=myToken -org=myOrg -orgOnly=true -toolName=thog
I get all the repos listed, and then this:

Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': sbberk
Password for 'https://[email protected]': Username for 'https://github.com': Password for 'https://[email protected]':

And I can't enter my correct username/password to get past this check. Thanks!

Github action

Dear @anshumanbh ,
thank you for this very nice looking project! I was wondering: do you have a github action as well for this? Would love to integrate it into our scanner benchmark at OWASP/wrongsecrets#424 ,
with kind regards,
Jeroen

Ask username with token

Hello,
I am testing this product to scan our repositories stored in github (SaaS mode). These deposits are private and public.
To allow the tool to retrieve the projects, I give it a token with all the accesses but I have the following message:

image

I use the following command :
docker run --rm -it abhartiya/tools_gitallsecrets -org "my_org" -token "my_token" -orgOnly

index-pack-failed

Hello @anshumanbh ,
I am still getting the issue after cloning half of the repo from any org.
exit status 128: Cloning into '/tmp/repos/org/xxx/est4js'... error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function. fatal: The remote end hung up unexpectedly fatal: early EOF fatal: index-pack failed

goroutine error

Hi,

I'm getting this error when I ran using go as well as the docker.

I've no idea what's that about.

Would you please be able to help ?

Cheers


panic: exit status 1

goroutine 1320 [running]:
main.check(0x7c6460, 0xc4201f7560)
	/home/v/Desktop/tools/git-all-secrets/main.go:38 +0x174
main.runGitTools(0xc420db57a0, 0x21, 0xc4202c8880, 0xc420db4ee0, 0x11, 0x7ffeb7d763cc, 0x9)
	/home/v/Desktop/tools/git-all-secrets/main.go:178 +0x1f0
created by main.scanorgrepos
	/home/v/Desktop/tools/git-all-secrets/main.go:279 +0x1bf
exit status 2

No output generated

Hi dude,

First of all, thank you for your tool, it seems to be very useful for the use-case that I'm currently covering.
However, I'm having a trouble and I can't find what am I missing.
I'm running the tool in the following way:

docker run --rm -it abhartiya/tools_gitallsecrets -repoURL https://github.com/XXXX/XXXX -token XXXXXXX -output test.txt

And, after finish the scan, no matter what repo, org or user is scanning, it never drops any output.
I attach a pic to show you the case.
imagen

I'm using the latest image and running on Ubuntu 18.04.

Any idea ?

Thank you !

Scan hangs

I am trying to scan my repository, but it just hangs without any traces
Here what I get in output:


Starting to clone: https://github.com/nameHere/repoName.git

Username for 'https://github.com': userNameHere
Password for 'https://[email protected]':
Cloning of: https://github.com/nameHere/repoName.git finished

Starting to scan: https://github.com/nameHere/repoName.git

And it hangs there in screen for a long while do not outputting anything

ssl certificate error

I attempted to scan a public repo and got the following error. Is this common? I was able to clone the repo separately outside of this tool.

exit status 128: Cloning into '/tmp/repos/juniormint88/test-secrets'...
fatal: unable to access 'https://github.com/juniormint88/test-secrets/': SSL certificate problem: self signed certificate in certificate chain

x509 Error when attempting to access Org

Fairly new to Github, so this may be an obvious fix, but would appreciate the assistance.


panic: Get https://api.github.com/orgs/hudgov/repos?per_page=10: x509: certificate signed by unknown authority

goroutine 1 [running]:
main.check(0x71f1e0, 0xc420301440)
/go/src/github.com/anshumanbh/git-all-secrets/main.go:80 +0xec
main.cloneorgrepos(0x721cc0, 0xc420018028, 0xc42008eea0, 0x7ffc4d9ddf43, 0x6, 0x0, 0x0)
/go/src/github.com/anshumanbh/git-all-secrets/main.go:158 +0x1b9
main.main()
/go/src/github.com/anshumanbh/git-all-secrets/main.go:953 +0x2d6

Weird issue

Hey dude when i run this against a users repo i am getting the following despite providing a user token?

Capture.png

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.