anshumanbh / git-all-secrets Goto Github PK

View Code? Open in Web Editor NEW

1.1K 40.0 196.0 113 KB

A tool to capture all the git secrets by leveraging multiple open source git searching tools

License: MIT License

Go 95.04% Shell 0.25% Dockerfile 4.71%

git-all-secrets's Introduction

git-all-secrets

About

git-all-secrets is a tool that can:

Clone multiple public/private github repositories of an organization and scan them,
Clone multiple public/private github repositories of a user that belongs to an organization and scan them,
Clone a single public/private repository of an organization and scan it,
Clone a single public/private repository of a user and scan it,
Clone a single public/secret gist of a user and scan it
Clone a team's repositories in an organization and scan them,
All of the above together!! Oh yeah!! Simply provide an organization name and get all their secrets. If you also want to get secrets of a team within an organization, just mention the team name along with the org.
Clone and scan Github Enterprise repositories and gists as well.

Scanning is done by multiple open source tools such as:

truffleHog - scans commits for high entropy strings and user provided regular expressions,
repo-supervisor - scans for high entropy strings in .js and .json files

NOTE - More such tools can be added in future, if desired! NOTE - Scanning can be done by all the tools or any one of them by specifying the toolName flag.

If all the tools are used to scan, the final output from the tool combines the output from all files from all the tools into one consolidated output file.

Getting started

The easiest way to run git-all-secrets is via Docker and I highly recommend installing Docker if you don't already have it. Once you have Docker installed,

Type docker run --rm -it abhartiya/tools_gitallsecrets --help to understand the different flags it can take as inputs.
Once you know what you want to scan, type something like docker run -it abhartiya/tools_gitallsecrets -token=<> -org=<>. You can also specify a particular tool to use for scanning by typing something like docker run -it abhartiya/tools_gitallsecrets -token=<> -org=<> -toolName=<>. Options are thog and repo-supervisor.
If you want to run truffleHog with the default regex AND the high entropy settings, provide the thogEntropy flag like this - docker run -it abhartiya/tools_gitallsecrets -token=<> -org=<> -toolName=thog -thogEntropy.
After the container finishes running, retrieve the container ID by typing docker ps -a.
Once you have the container ID, get the results file from the container to the host by typing docker cp <container-id>:/root/results.txt .

Flags/Options

-token = Github personal access token. We need this because unauthenticated requests to the Github API can hit the rate limiting pretty soon!
-org = Name of the Organization to scan. This will scan all public repos in the org + all the repos & gists of all users in the org. If you are using a token of a user who is a part of this org, it will also clone and scan all the secret gists belonging to that user as well as all the private repos in that org that the user has access to. However, it will NOT clone and scan any private repositories of this user belonging to this org. To scan private repositories of users, please use the scanPrivateReposOnly flag with the user flag along with the SSH key mounted on a volume.
-user = Name of the User to scan. This will scan all the repos & gists of this user. If the token provided is the token of the user, secret gists will also be cloned and scanned. But, only public repos will be cloned and scanned. To scan private repositories of this user, please use the scanPrivateReposOnly flag with the user flag along with the SSH key mounted on a volume.
-repoURL = HTTPS URL of the Repo to scan. This will scan this repository only. For public repos, mentioning the https URL of the repo will suffice. However, if you wish to scan a private repo, then you need to provide the ssh URL along with the SSH key mounted on a volume and the scanPrivateReposOnly flag.
-gistURL = HTTPS URL of the Gist to scan. This will scan this gist only. There is no concept of public or secret gist as long as you have the URL. Even if you have a secret gist, if someone knows the HTTPS URL of your secret gist, they can access it too.
-output = This is the name of the file where all the results will get stored. By default, this is results.txt.
-cloneForks = This is the optional boolean flag to clone forks of org and user repositories. By default, this is set to 0 i.e. no cloning of forks. If forks are to be cloned, this value needs to be set to 1. Or, simply mention -cloneForks along with other flags.
-orgOnly = This is the optional boolean flag to skip cloning user repositories belonging to an org. By default, this is set to 0 i.e. regular behavior. If user repo's are not to be scanned and only the org repositories are to be scanned, this value needs to be set to 1. Or, simply mention -orgOnly along with other flags.
-toolName = This is the optional string flag to specify which tool to use for scanning. By default, this is set to all i.e. thog and repo-supervisor will all be used for scanning. Values are either thog or repo-supervisor.
-teamName = Name of the Organization Team which has access to private repositories for scanning. This flag is not fully tested so I can't guarantee the functionality.
-scanPrivateReposOnly = This is the optional boolean flag to specify if you want to scan private user repositories or not. Mentioning this will NOT scan public user repositories. And, you need to provide the SSH key by mounting the volume onto the container. Also, this only works with either the user flag, the repoURL flag or the org flag.

When the org flag is mentioned along with the scanPrivateReposOnly flag and without the orgOnly flag, it will scan the public AND the private repos belonging to this org to which the user has access to (whose token is provided). It will then continue to scan ONLY the private repositories of the user (whose token is provided). Finally, it will continue to scan all public and secret gists of this user (whose token is provided). In a nutshell, the scanPrivateReposOnly flag only really affects the user and the repoURL flag.
-enterpriseURL = Optional flag to provide the enterprise Github URL, if you wish to scan enterprise repositories. It should be something like https://github.org.com/api/v3 along with the SSH key mounted onto the container. Refer to scanning github enterprise below.
-threads = Default value is 10. This is to limit the number of threads if your system is not beefy enough. For the most part, leaving this to 10 should be okay.
-thogEntropy = This is an optional flag that basically tells if you want to get back high entropy based secrets from truffleHog or not. The high entropy secrets from truffleHog produces a LOT of noise so if you don't really want all that noise and if you are running git-all-secrets on a big organization, I'd recommend not to mention this flag. By default, this is set to False which means truffleHog will only produce result based on the Regular expressions in the rules.json file. If you are scanning a fairly small org with a limited set of repos or a user with a few repos, mentioning this flag makes more sense.
-mergeOutput = Optional flag to merge and deduplicate the ouput of the tools used (currently truffleHog and repo-supervisor). Default value is False.
-blacklist = Repo names provided as comma separated values that should NOT be scanned.

Note

The token flag is compulsory. This can't be empty.
The org, user, repoURL and gistURL can't be all empty at the same time. You need to provide just one of these values. If you provide all of them or multiple values together, the order of precendence will be org > user > repoURL > gistURL. For instance, if you provide both the flags -org=secretorg123 and -user=secretuser1 together, the tool will complain that it doesn't need anything along with the org value. To run it against a particular user only, just need to provide the user flag and not the org flag.
When specifying the scanPrivateReposOnly flag:
- One must mount a volume containing the private SSH key onto the Docker container using the -v flag.
- It should be used anytime a private repository is scanned. Please use the ssh url when using the flag and not the https URL.
- Please make sure the token being used actually belongs to the user whose private repository/gist you are trying to scan otherwise there will be errors.
- The SSH key that you will be using should NOT have a passphrase set if you want this tool to work without any manual intervention.
Refer to scanning private repositories below.
When specifying teamName it is important that the provided token belong to a user which is a member of the team. Unexpected results may occur otherwise. Refer to scanning an organization team below.
When specifying the enterpriseURL flag, it will always consider the SSH url even if you provide the https url of a repository. All the enterprise cloning/scanning happens via the ssh url and not the https url.
As mentioned above, make sure the SSH key being used (to scan the ssh URL) does not have any passphrase set.

Scanning Private Repositories

The most secure way to scan private repositories is to clone using the SSH URLs. To accomplish this, one needs to place an appropriate SSH key which has been added to a Github User. Github has helpful documentation for configuring your account. Make sure this key does not have any passphrase set on it. Once you have the SSH key, simply mount it to the Docker container via a volume. It is as simple as typing the below commands:

docker run -it -v ~/.ssh/id_rsa_personal:/root/.ssh/id_rsa abhartiya/tools_gitallsecrets -token=<> -user=<> -scanPrivateReposOnly

docker run -it -v ~/.ssh/id_rsa_personal:/root/.ssh/id_rsa abhartiya/tools_gitallsecrets -token=<> -repoURL=<> -scanPrivateReposOnly

Here, I am mapping my personal SSH key id_rsa_personal stored locally to /root/.ssh/id_rsa inside the container so that git-all-secrets will try to clone the repo via ssh and will use the SSH key stored at /root/.ssh/id_rsa inside the container. This way, you are not really storing anything sensitive inside the container. You are just using a file from your local machine. Once the container is destroyed, it no longer has access to this key.

Scanning an Organization Team

The Github API limits the circumstances where a private repository is reported. If one is trying to scan an Organization with a user which is not an admin, you may need to provide the team which provides repository access to the user. In order to do this, use the teamName flag along with the org flag. Example is below:

docker run --it -v ~/.ssh/id_rsa_personal:/root/.ssh/id_rsa abhartiya/tools_gitallsecrets -token=<> -org=<> -teamName <>

Scanning Github Enterprise

git-all-secrets now supports scanning Github Enterprise as well. If you have your own Github Enterprise hosted behind a VPN or something, make sure you are connected on the VPN or on the correct network that has access to the Github Enterprise repos. The enterpriseURL is what you'd need to scan your Github Enterprise repos. Below are some examples:

Example 1:

docker run -it -v ~/.ssh/id_rsa_gitenterprise:/root/.ssh/id_rsa -token <token> -enterpriseURL https://github.<org>.com/api/v3 -repoURL https://github.<org>.com/<user>/<repo>.git

Here, I am now mounting my github enterprise SSH key onto the container, followed by my personal access token, the enterprise URL to which the requests will be sent and the repo I want to scan.

Example 2:

docker run -it -v ~/.ssh/id_rsa_gitenterprise:/root/.ssh/id_rsa -token <token> -enterpriseURL https://github.<org>.com/api/v3 -repoURL https://github.<org>.com/<user>/<repo>.git -toolName thog -thogEntropy

Above, I am now just running truffleHog against the repository with the Entropy settings.

Example 3:

docker run -it -v ~/.ssh/id_rsa_gitenterprise:/root/.ssh/id_rsa -token <token> -enterpriseURL https://github.<org>.com/api/v3 -user <username> -scanPrivateReposOnly

Above, I am scanning only the private repositories of the user whose token is provided with all the tools (repo-supevisor and thog), but without the entropy setting of truffleHog.

TODO

Test team scanning functionality
~~Fix the Goroutine bug~~ - Hopefully DONE!
~~Support scanning Github Enterprise~~ DONE!
~~Support cloning and scanning private repositories of an org~~ - DONE!
~~Replace gitsecrets by the new Regex functionality in truffleHog~~ - DONE!
~~Add support for scanning private user repositories via SSH keys~~ - DONE!
~~Add flag to avoid scanning forks~~ - DONE!

Details

Features

You can add your own regular expressions in the rules.json file and include it when executing docker run using the argument -v $(pwd)/rules.json:/root/truffleHog/rules.json.
The tool looks for some default regular expressions. If needed, it can also be made for high entropy strings. All this happens via the truffleHog tool.
It can look for high entropy strings in .js and .json files via the repo-supervisor tool.
It scans users gists, which most of the tools dont.
If there is a new tool that is good, it can be integrated into git-all-secrets pretty effortlessly.
It is built for integration with other tools and frameworks. It takes in a few input parameters and produces an output file of the results. Pretty straightforward!
It supports scanning Github Enterprise orgs/users/repos/gists as well.
Most of the tools out there are made to scan individual repositories. If you want to loop it over multiple repositories, you'd have to write your own for loop in a shell script or something like that. git-all-secrets can help you scan multiple repositories at one go.
You can now merge outputs from both the tools into a json file which can then be used in other automation type tools/frameworks

Motivation

I looked at a large number of open source tools that could be potentially used to look for secrets in github repositories. Some of the top tools that I thought were good are: gitrob, truffleHog and git-secrets.

Gitrob is meant to be a standalone tool that is pretty difficult to integrate with other tools because it has its own database and UI to see all the secrets discovered. It also produces a ton of false positives, more than truffleHog. And, it doesn't really highlight the secrets discovered. It just looks at the files and their extensions, not the actual content. So, although Gitrob is a great tool to get started with, I would recommend running it every once in a while to understand what the attack surface looks like and see if it has changed.

Then, there is truffleHog that looks for secrets in the actual contents of the file by looking at Shannon's entropy and prints the output on the screen. It takes in a repository URL or a repository directory as an argument. This is a pretty good tool although it does have its share of false positives. Some of the other drawbacks are:

We can't use it recursively to scan directories that contain multiple repositories.
There is no way we can use truffleHog to identify secrets that follow a certain pattern but don't have a high enough entropy i.e. we can't make it look for secrets that we know of but not necessarily have high entropy to be considered as a secret.
It prints the output on the screen so not really useful for automation as such.

Finally, there is git-secrets which can flag things like AWS secrets. The best part is that you can add your own regular expressions as well for secrets that you know it should be looking for. A major drawback is that it doesn't do a good job on finding high entropy strings like truffleHog does. You can also only scan a particular directory that is a repository so no recursion scanning from a directory of repositories either.

So, as you can see, there are decent tools out there, but they had to be combined somehow. There was also a need to recursively scan multiple repositories and not just one. And, what about gists? There are organizations and users. Then, there are repositories for organizations and users. There are also gists by users. All of these should be scanned. And, scanned such that it could be automated and easily consumed by other tools/frameworks.

Changelog

2/9/19 - Added blacklist flag to skip some repo names while scanning. Thank you @sjqnn for the PR. Built and pushed a new Docker image. GLHF!
7/31/18 - Made trugglehog's installation simpler by using pip. @mhmdiaa fixed a bug wrt trufflehog's output function where it wasn't merging and sorting properly. Updated the regex file to include things like password. Built and pushed a new Docker image. GLHF!
7/15/18 - Updated repo-supervisor's fork because the upstream had some changes. Rebuilt a new Docker image using the latest Trufflehog. Provided the rules.json file that contains all the regexes that Trufflehog uses to find secrets. Added the ability to also merge outputs (in json) for both the tools using the -mergeOutput flag. Drastically reduced the Docker image size by using multi-stage builds and dep for managing dependencies. Huge shout out to @mhmdiaa for all of this!
12/12/17 - For some large repos, truffleHog fails and exits. But, we don't want to stop there. We want to notify the user that scanning failed for that repo and continue scanning the other repos. This is now implemented in the latest docker image.
12/11/17 - Removed gitsecrets because truffleHog supports regex functionality now. Simply, adding your regexes in the rules.json file and rebuilding the Docker image will basically give us the functionality that gitsecrets was giving previously so there is no need for gitsecrets anymore. I also added support for scanning Github Enterprise repos & gists. @high-stakes helped get a PR in that (hopefully) fixes the Goroutine bug by limiting the amount of threads. Finally, support for scanning private repositories for an organization was added as well.
12/08/17 - Removed my own fork of truffleHog. Using the upstream version now along with the new regex functionality of truffleHog + entropy mode. Soon, I believe we can replace both gitsecrets and repo supervisor by just truffleHog once some issues are fixed.
12/07/17 - I updated the documentation with some more details and explanation around the different flags.
12/05/17 - Integrated scanning support for private repositories via SSH key. This has been an ask for the longest time and it is now possible to do so. Also, changed the docker image tag scheme. From now on, the latest image will have the latest tag. And, all the previous versions will be tagged with a number. All this couldn't have been possible without the SimpliSafe team, specially Matthew Cox (https://github.com/matthew-cox). So, a big shoutout to you Matt!
10/14/17 - Built and pushed the new image abhartiya/tools_gitallsecrets:v6. This new image has the newer version of git-secrets as well as repo-supervisor i.e. I merged some upstream changes into my fork alongwith some additional changes I had already made in my fork. The new image uses these changes so everything is latest and greatest!
10/14/17 - Built and pushed the new image abhartiya/tools_gitallsecrets:v5. This image fixes a very stupid and irritating bug which was possibly causing repo supervisor to fail. Something changed in the way Environment values are being read in Dockerfile which resulted in repo supervisor not understanding which node path to use. Node hell!
9/29/17 - Built and pushed the new image with the orgOnly flag - abhartiya/tools_gitallsecrets:v4
8/22/17 - Added -orgOnly toggle by kciredor: analyzes specified organization repo and skips user repo's.
6/26/17 - Removed some output in repo-supevisor that printed out errors when there were no secrets found. Unnecessary output! Built and pushed the new image - abhartiya/tools_gitallsecrets:v3
6/25/17 - Added the flag toolName to specify which tool to use for scanning. Built and pushed the new image - abhartiya/tools_gitallsecrets:v2
6/14/17 - Added repo-supervisor as a scanning tool, also updated and added the version number to the docker image - abhartiya/tools_gitallsecrets:v1
6/14/17 - Added the flag cloneForks to avoid cloning forks of org and user repos. By default, this is false. If you wish to scan forks, just set the value to 1 i.e. -cloneForks=1

Donate

If you want to show some love, my BTC wallet address is 1PtMhXWCcMZCitcDfaEBe7jnV9sjKoNvq7.

git-all-secrets's People

Contributors

Stargazers

Watchers

Forkers

tobey123 rohitbemax olivierh59500 nqis adolfoeliazat kbahaxor irivera007 techlord-rce m00zh33 menzow mgcfish chubbymaggie pentestingtools drpep 99 arjuneng05 5up3rc mmcniece ngocngv mingyen high-stakes backstopit elorion armagheadon quikilr danehammer vinothzomato peval syntaxerror4 ragodev zricethezav fingerleakers mr-brody rajsek loganding crazybus minkione johnjohnsp1 makyotox jayyogesh weirdindiankid amreetvishal mrhacker46 aridoshikagithub samhaxr binaryreaper essobi permikomnaskaltara secxena alexmenguy mhmdiaa 23ranjan blacklanternsecurity 5cr1p70633k fzxcp3 imbici mosburn d-3lf slowmistio atefbb ibuystuff allisone-0 mmg1 ballinballen shiv4nsh poeblu orf53975 jayaramyalla peeper35 xiaolushuo fr34k8 driver-217 abbas512 r3dfruitrollup tejomayonline io-security-solutions michaelgrifalconi peoplenet hpxpat honsa bbhunter cyberbuck shahid1996 dennyx hanklehigh ayomawdb lzhengqc axiom215 qsdj damnsl jkalandaibm devendrasuthar sh1nu11bi lanzay toygang la3b0z marciopocebon youssefkhaoulaj vikasvarshney secbydefault

git-all-secrets's Issues

Support non-Github hosted repositories

support scanning local repositories and repositories on other sites/privately hosted servers.

Running without docker

First of all, thanks for creating and working on this tool, it’s really cool job.

But i’ve got a question- is it possible for running this tool without docker?

Have a nice day, ultras

Scan hangs

I am trying to scan my repository, but it just hangs without any traces
Here what I get in output:


Starting to clone: https://github.com/nameHere/repoName.git

Username for 'https://github.com': userNameHere
Password for 'https://[email protected]':
Cloning of: https://github.com/nameHere/repoName.git finished

Starting to scan: https://github.com/nameHere/repoName.git

And it hangs there in screen for a long while do not outputting anything

Github action

Dear @anshumanbh ,
thank you for this very nice looking project! I was wondering: do you have a github action as well for this? Would love to integrate it into our scanner benchmark at OWASP/wrongsecrets#424 ,
with kind regards,
Jeroen

Weird issue

Hey dude when i run this against a users repo i am getting the following despite providing a user token?

orgURL could be another option

For enterprise github (specific to a company), there isn't a way to pass both the orgName and enterprise github domain URL. Currently, this component could be used for just one repo at a time for an ORG.

After cloning I get error

panic: exit status 128
goroutine 110 [running]:
main.check(0x7c8460, 0xc4201922c0)
/data/main.go:40 +0x174
main.gitclone(0xc42024d3e0, 0x24, 0xc42016f4c0, 0x15, 0xc42022ffe0)
/data/main.go:53 +0x108
created by main.cloneorgrepos
/data/main.go:92 +0x429

Windows 10 git-all-secrets docker is being used.

Docker Image failed to build due to missing SSH client

ssh and ssh-keyscan missing from base image

running only truffleHog gives blank results

When I use the -toolName=thog, the tool gives out 0 findings vs the default run of the tool which gives the expected findings.

No Result

Hello @anshumanbh ,

I run git-all-secrets on some repo which already having some sensitive info but generated the results file is showing blank..no result.

I ran ./git-all-secrets -token=xxxmytokenxxxx -repoURL=https://github.com/some/info -toolName=thog -thogEntropy

Cloning of: https://github.com/some/info finished

Starting to scan: https://github.com/some/info

Finished truffleHog Scanning for: some_info
Scanning of: https://github.com/some/info finished

Combining the output into one file

Can you please help me out on this issue.

feature to exclude some gits from cloning

Hi,

Thanks for this amazing tool and automation! Much appreciate it.

Sometimes, when cloning company's gits we get lots of "unwanted" repo forked by employees. So, would be possible to exclude it from cloning the line command ?

Cheers

Not able to clone repos that are being migrated

There are cases when certain repos cannot be cloned because they are being migrated or whatever. In those cases, git-all-secrets needs to move on.

An example error message - "Access to this repository has been disabled while it is being migrated.". Repo is - https://github.com/walmartlabs/grunt-castle

index-pack-failed

Hello @anshumanbh ,
I am still getting the issue after cloning half of the repo from any org.
exit status 128: Cloning into '/tmp/repos/org/xxx/est4js'... error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function. fatal: The remote end hung up unexpectedly fatal: early EOF fatal: index-pack failed

Issue with scanning repositories using git-all-secrets

When I am trying to scan all the repositories in our organization, facing the below issue:

$sudo docker run -it -v ~/.ssh/id_rsa abhartiya/tools_gitallsecrets -token=mytoken -org=myorg -scanPrivateReposOnly
.........
.........

Warning: Permanently added 'github.com,192.30.253.112' (RSA) to the list of known hosts.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: UNPROTECTED PRIVATE KEY FILE! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0755 for '/root/.ssh/id_rsa' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
key_load_private_type: bad permissions
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

panic: exit status 128

goroutine 462 [running]:
main.gitclone(0xc420435b60, 0x2c, 0xc4201fb6a0, 0x1e, 0xc420540a10)
/data/main.go:75 +0x387
main.executeclone.func1.1()
/data/main.go:106 +0x4e
main.enqueueJob.func1(0xc42083d590)
/data/main.go:45 +0x27
created by main.enqueueJob
/data/main.go:47 +0x6a

All the files in .ssh folder has 600 permission.

Can you suggest anything to solve this issue? If you can post a video in youtube for this, that will be a great help for everyone. Thank you.

Improper line breaks in /root/.ssh/config

The line breaks that the Dockerfile is putting in the /root/.ssh/config file are not being interpreted properly.

# cat /root/.ssh/config
Host *github.com \n  IdentitiesOnly yes \n  StrictHostKeyChecking no \n  UserKnownHostsFile=/dev/null \n  IdentityFile /root/.ssh/id_rsa \n  \n Host github.*.com \n  IdentitiesOnly yes \n  StrictHostKeyChecking no \n  UserKnownHostsFile=/dev/null \n  IdentityFile /root/.ssh/id_rsa

flag provided but not defined: -orgOnly

Defined -orgOnly=1 but getting an error: "flag provided but not defined: -orgOnly"

sudo docker run -it abhartiya/tools_gitallsecrets:v3 -token=<> -orgOnly=1 -org=<>

Scanning Private Repos are failing

After the newest update of my docker image, I am no longer able to use my ssh key to scan private repos. When I try, I am given the following stack trace during the cloning process:

exit status 128: Cloning into '/tmp/repos/org/[REDACTED]'...

fatal: cannot run ssh: No such file or directory
fatal: unable to fork

panic: exit status 128

goroutine 61 [running]:
main.gitclone(0xc420469950, 0x26, 0xc4200b3110, 0x22, 0xc42026b280)
	/go/src/github.com/anshumanbh/git-all-secrets/main.go:97 +0x3bf
main.executeclone.func1.1()
	/go/src/github.com/anshumanbh/git-all-secrets/main.go:137 +0x4e
main.enqueueJob.func1(0xc420553c80)
	/go/src/github.com/anshumanbh/git-all-secrets/main.go:67 +0x27
created by main.enqueueJob
	/go/src/github.com/anshumanbh/git-all-secrets/main.go:66 +0x5b

I imaged my machine before I pulled the latest image, and the same command runs fine. For context, here is the command I am trying:
docker run -it -v ~/.ssh/[REDACTED]:/root/.ssh/id_rsa abhartiya/tools_gitallsecrets -token=[REDACTED] -org=[REDACTED] -scanPrivateReposOnly -orgOnly

Small code clarification

Hello, I miss to understand the reason of this code

		func(orgclone *sync.WaitGroup, urlToClone string, directory string) {
			enqueueJob(func() {
				gitclone(urlToClone, directory, orgclone)
			})
		}(&orgclone, urlToClone, directory)

Is there a particular reason to not doing this?


			enqueueJob(func() {
				gitclone(urlToClone, directory, &orgclone)
			})

goroutine error

Hi,

I'm getting this error when I ran using go as well as the docker.

I've no idea what's that about.

Would you please be able to help ?

Cheers


panic: exit status 1

goroutine 1320 [running]:
main.check(0x7c6460, 0xc4201f7560)
	/home/v/Desktop/tools/git-all-secrets/main.go:38 +0x174
main.runGitTools(0xc420db57a0, 0x21, 0xc4202c8880, 0xc420db4ee0, 0x11, 0x7ffeb7d763cc, 0x9)
	/home/v/Desktop/tools/git-all-secrets/main.go:178 +0x1f0
created by main.scanorgrepos
	/home/v/Desktop/tools/git-all-secrets/main.go:279 +0x1bf
exit status 2

Tool requires the github user/pass ?

Hi, When I try to run git-all-secrets, it requires the github user/pass, but when i watch the video demo, it doesn't.

Even I provided the correct github user/pass, it still can't run :

$ docker run --rm -it abhartiya/tools_gitallsecrets:v6 -org=MyOrg -token=d2d2cf23d24d9e267d88d08b34caf48bne895776 -output=results.txt
Since org was provided, the tool will proceed to scan all the org repos, then all the user repos and user gists in a recursive manner
Cloning the repositories of the organization: MyOrg
[...skip repo info...]
Username for 'https://github.com': Username for 'https://github.com': [email protected]
Password for 'https://[email protected]@github.com': Password for 'https://[email protected]':
panic: exit status 128

goroutine 28 [running]:
main.check(0x7fa4c0, 0xc420286780)
	/data/main.go:41 +0x100
main.gitclone(0xc4202feac0, 0x32, 0xc42030a9a0, 0x19, 0xc420280510)
	/data/main.go:54 +0x108
created by main.cloneorgrepos
	/data/main.go:93 +0x405

Anyone can help ? Thanks

Best Regards,
VietNC

Username for 'https://github.com': panic: signal: killed

When I run the below

docker run -it abhartiya/tools_gitallsecrets:v6 -token=REDACTED -org=REDACTED -output=results.txt

It starts to list the repos and fails with the below

Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': panic: signal: killed

goroutine 905 [running]:
main.check(0x7fa4c0, 0xc4203b64c0)
	/data/main.go:41 +0x100
main.gitclone(0xc420be2e80, 0x38, 0xc420ea5a20, 0x1e, 0xc420c00f10)
	/data/main.go:54 +0x108
created by main.cloneorgrepos
	/data/main.go:93 +0x405

"results.txt" has shell color characters in it

Example:

Tool: truffleHog
OrgorUser: terraform RepoName: fluentd-sumologic

^[[92mReason: Generic Password^[[0m
^[[92mDate: 2018-03-28 21:58:30^[[0m
^[[92mHash: e05234bc366d0ab0f5174f1085d84c032dd8b8a6^[[0m
^[[92mFilepath: aaa/bbb.txt^[[0m
^[[92mBranch: origin/master^[[0m
^[[92mCommit: Updated s3 key for remote state
^[[0m
^[[93m+  password = "admin"^[[0m

add support for gitlab

Gitlab is getting more and more popular. Your tool does exactly what we need at our environment, but we have on-premise gitlab, so as I understood from documentation it's not suitable.

Just a suggestion in case you are actively working on it.

not able to get output

cat results.txt
Tool: truffleHog
Tool: repo-supervisor

when i see result from results.txt . everytime above output shown

ssl certificate error

I attempted to scan a public repo and got the following error. Is this common? I was able to clone the repo separately outside of this tool.

exit status 128: Cloning into '/tmp/repos/juniormint88/test-secrets'...
fatal: unable to access 'https://github.com/juniormint88/test-secrets/': SSL certificate problem: self signed certificate in certificate chain

Username / Password authentication not working

Hi, when I run docker run -it abhartiya/tools_gitallsecrets -token=myToken -org=myOrg -orgOnly=true -toolName=thog
I get all the repos listed, and then this:

Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': sbberk
Password for 'https://[email protected]': Username for 'https://github.com': Password for 'https://[email protected]':

And I can't enter my correct username/password to get past this check. Thanks!

scan organization repos meet "index out of range"

When "Done cloning org repos." and then start "Listing users of the organization and their repositories and gists" it shows:

panic: runtime error: index out of range

goroutine 1 [running]:
main.cloneusergists(0x721c20, 0xc420018028, 0xc42008efc0, 0xc420374ac0, 0xb, 0x0, 0x0)
	/go/src/github.com/anshumanbh/git-all-secrets/main.go:249 +0x595
main.main()
	/go/src/github.com/anshumanbh/git-all-secrets/main.go:975 +0x77a

Is it possible to exclude some folders in repo from scanning?

We are using Sitecore Unicorn which serializes data from database to files on disk and we are checking this files in repository.
File structure includes GUID values which are treated as positives. And we have a lot of files like this, scattered in folders through whole solution.
Hence a question - is it possible to exclude from scanning folders, which names starts/contains/matches given string?

404 Not Found

docker run -it abhartiya/tools_gitallsecrets -token= -org cnooc -orgOnly 1
Org was specified combined with orgOnly, the tool will proceed to scan only the org repos and nothing related to its users
Cloning the repositories of the organization: cnooc
If the token provided belongs to a user in this organization, this will also clone all public AND private repositories of this org, irrespecitve of the scanPrivateReposOnly flag being set..
panic: GET https://api.github.com/orgs/cnooc/repos?per_page=10: 404 Not Found []

goroutine 1 [running]:
main.check(0x71ee20, 0xc420528050)
/go/src/github.com/anshumanbh/git-all-secrets/main.go:80 +0xec
main.cloneorgrepos(0x721cc0, 0xc4200a0010, 0xc4200b8fc0, 0x7ffe5d12cf39, 0x5, 0x0, 0x0)
/go/src/github.com/anshumanbh/git-all-secrets/main.go:158 +0x1b9
main.main()
/go/src/github.com/anshumanbh/git-all-secrets/main.go:953 +0x2d6

container not authenticating after mapping ssh key

I am mapping my ssh key as shown in the documentation, it seems like the container isn't using the correct username when running the following:

docker run -it -v ~/.ssh/id_rsa:/root/.ssh/id_rsa abhartiya/tools_gitallsecrets -token=REDACTED -org=REDACTED

Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Use
rname for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com': Username for 'https://github.com':
Password for 'https://github.com': signal: interrupt: Cloning into '/tmp/repos/org/.........REDACTED

Did I miss something?

Crash when analysing huge organization

Hi !

I'm getting this crash when I try to run the tool over a "huge" org (>140 repositories). Just after:

Scanning all org repositories now..This may take a while so please be patient

panic: exit status 1

goroutine 2727 [running]:
main.check(0x7c8460, 0xc4201e2020)
        /data/main.go:40 +0x174
main.runTrufflehog(0xc420164000, 0x1f, 0xc4201d7710, 0xf, 0x7ffce0564ec0, 0x6, 0x0, 0x0)
        /data/main.go:213 +0x1d1
main.runGitTools(0x6b46b2, 0x3, 0xc420164000, 0x1f, 0xc4203b51f0, 0xc4201d7710, 0xf, 0x7ffce0564ec0, 0x6)
        /data/main.go:233 +0x1af
created by main.scanorgrepos
        /data/main.go:379 +0x209

Any idea on how can I help to debug this crash?

Question regarding performance/efficiency

While the tool has the ability to scan all possible repo's related to an org, it takes quite a while to complete. I tried using just the trufflehog scanning portion of the tool, but it still manages to take weeks at a time to scan a specific org. I understand it may be due to repo sizes. I'm curious to find out whether there are any ways to fine tune the performance of these scans.

x509 Error when attempting to access Org

Fairly new to Github, so this may be an obvious fix, but would appreciate the assistance.

panic: Get https://api.github.com/orgs/hudgov/repos?per_page=10: x509: certificate signed by unknown authority

goroutine 1 [running]:
main.check(0x71f1e0, 0xc420301440)
/go/src/github.com/anshumanbh/git-all-secrets/main.go:80 +0xec
main.cloneorgrepos(0x721cc0, 0xc420018028, 0xc42008eea0, 0x7ffc4d9ddf43, 0x6, 0x0, 0x0)
/go/src/github.com/anshumanbh/git-all-secrets/main.go:158 +0x1b9
main.main()
/go/src/github.com/anshumanbh/git-all-secrets/main.go:953 +0x2d6

Ask username with token

Hello,
I am testing this product to scan our repositories stored in github (SaaS mode). These deposits are private and public.
To allow the tool to retrieve the projects, I give it a token with all the accesses but I have the following message:

I use the following command :
docker run --rm -it abhartiya/tools_gitallsecrets -org "my_org" -token "my_token" -orgOnly

Tool requires the github user/pass ?

Hi,

I reopen this issue, firstly, I want to confirm my scanning on my private repo (not other person's repo for any illegitimate purpose).

I have generated token from "https://github.com/settings/tokens" in section "Personal access tokens". But the tool still require to authenticate with github account

Any hint ? Thanks

Best Regards,
VietNC

Request for feature - Web Interface option for results

Hi, can there be an option for the results to be put into a Bootstrap GUI, like GitRob?

Example: Having it part of the main program as an argument that spins up Bootstrap and automatically parses the results, or have a second executable that parses the results.txt into Bootstrap.

No output generated

Hi dude,

First of all, thank you for your tool, it seems to be very useful for the use-case that I'm currently covering.
However, I'm having a trouble and I can't find what am I missing.
I'm running the tool in the following way:

docker run --rm -it abhartiya/tools_gitallsecrets -repoURL https://github.com/XXXX/XXXX -token XXXXXXX -output test.txt

And, after finish the scan, no matter what repo, org or user is scanning, it never drops any output.
I attach a pic to show you the case.

I'm using the latest image and running on Ubuntu 18.04.

Any idea ?

Thank you !