Building the docker image fails when trying to compile hyperscan:

$ docker build --rm=true --tag=deepfenceio/secretscanning:latest -f Dockerfile .

[snip]

#6 341.4 [ 54%] Building CXX object CMakeFiles/hs_compile.dir/src/nfagraph/ng_literal_decorated.cpp.o
#6 341.6 [ 54%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/nfa/truffle.c.o
#6 343.6 [ 54%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/som/som_runtime.c.o
#6 343.9 [ 54%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/som/som_stream.c.o
#6 344.3 [ 54%] Building CXX object CMakeFiles/hs_compile.dir/src/nfagraph/ng_mcclellan.cpp.o
#6 345.9 [ 54%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/nfa/sheng.c.o
#6 346.6 [ 54%] Building CXX object CMakeFiles/hs_compile.dir/src/nfagraph/ng_limex.cpp.o
#6 347.0 [ 54%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/rose/block.c.o
#6 347.5 [ 55%] Building CXX object CMakeFiles/hs_compile.dir/src/nfagraph/ng_limex_accel.cpp.o
#6 348.8 [ 55%] Building CXX object CMakeFiles/hs_compile.dir/src/nfagraph/ng_misc_opt.cpp.o
#6 350.8 [ 55%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/nfa/shufti.c.o
#6 350.8 [ 55%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/rose/catchup.c.o
#6 353.6 [ 55%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/nfa/tamarama.c.o
#6 389.1 c++: internal compiler error: Killed (program cc1plus)
#6 389.1 Please submit a full bug report,
#6 389.1 with preprocessed source if appropriate.
#6 389.1 See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
#6 389.1 CMakeFiles/hs_compile.dir/build.make:1536: recipe for target 'CMakeFiles/hs_compile.dir/src/nfagraph/ng_mcclellan.cpp.o' failed
#6 389.1 make[2]: *** [CMakeFiles/hs_compile.dir/src/nfagraph/ng_mcclellan.cpp.o] Error 4
#6 389.1 make[2]: *** Waiting for unfinished jobs....
#6 389.2 [ 56%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/rose/init.c.o
#6 392.1 [ 56%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/nfa/truffle.c.o
#6 392.3 [ 57%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/som/som_runtime.c.o
#6 392.3 [ 57%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/som/som_stream.c.o
#6 392.3 [ 57%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/rose/stream.c.o
#6 393.0 [ 57%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/rose/match.c.o
#6 394.3 CMakeFiles/Makefile2:512: recipe for target 'CMakeFiles/hs_compile.dir/all' failed
#6 394.3 make[1]: *** [CMakeFiles/hs_compile.dir/all] Error 2
#6 394.3 [ 57%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/rose/block.c.o
#6 394.8 [ 57%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/rose/catchup.c.o
#6 395.1 [ 57%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/rose/init.c.o
#6 398.7 [ 57%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/rose/program_runtime.c.o
#6 398.9 [ 57%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/util/multibit.c.o
#6 399.9 [ 57%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/rose/stream.c.o
#6 400.1 [ 58%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/rose/match.c.o
#6 400.2 [ 58%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/rose/program_runtime.c.o
#6 403.2 [ 58%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/util/multibit.c.o
#6 404.2 [ 58%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/util/simd_utils.c.o
#6 406.8 [ 58%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/util/state_compress.c.o
#6 408.1 [ 58%] Building C object CMakeFiles/hs_exec_shared_core2.dir/src/database.c.o
#6 408.1 [ 58%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/util/simd_utils.c.o
#6 408.4 [ 59%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/util/state_compress.c.o
#6 411.4 [ 59%] Building C object CMakeFiles/hs_exec_shared_corei7.dir/src/database.c.o
#6 421.1 [ 59%] Built target hs_exec_shared_corei7
#6 421.3 [ 59%] Built target hs_exec_shared_core2
#6 421.3 make: *** [all] Error 2
#6 421.3 Makefile:129: recipe for target 'all' failed
------
executor failed running [/bin/sh -c mkdir -p /usr/local/include/ &&     cd /usr/local/include/ &&     git clone https://github.com/intel/hyperscan.git &&     mkdir /usr/local/include/hs &&     cd /usr/local/include/hs &&     export MAKEFLAGS=-j$(nproc) &&     cmake -DBUILD_STATIC_AND_SHARED=1 /usr/local/include/hyperscan &&     echo "/usr/local/lib" | tee --append /etc/ld.so.conf.d/usrlocal.conf &&     cd /usr/local/include/hs && make && make install]: exit code: 2

I'm relatively new to docker, so if you can provide the steps to grab any log files you might be interested in, I'm happy to do so. Pulling the docker image from docker hub appears to work fine, and is a valid workaround.

Changes required in the modules dependent on fetcher service

I want to do a filesystem scan (I'm mounting a subpath) and I would like to skip scanning all files, called, say conf.yml
Is there a way to achieve this in config.yaml?

Hello

Is it possible to use my own configuration file for, only, exceptions i.e. I want to keep yours but f.i. I wish to be able to skip anything under /var/www/vendor (excluded folder).

As far I understand, I can create a config.yaml file and use it with --config-path but if I do this, your file won't be loaded anymore so I first need to copy/paste yours in mine and make some changes (https://github.com/deepfence/SecretScanner/blob/master/config.yaml)

This is bad since yours will be upgraded in the future for more rules so, on my side, I just need to be able to extend it with my own rules like my exclusions.

Also, is it possible to foresee/change the --config argument so I can provide a filename (like .config/secret-scanner.yml. The name 'config.yaml is too generic; I've already plenty of configuration files.

Thanks

Design and develop docker extension for SecretScanner with following features.

• Ability to list all images present locally
• Ability to run secret scan on selected image
• Result in human readable table format instead of JSON
• In-depth details of each scan result

Encountered an error while running the deepfenceio/deepfence_secret_scanner:2.0.0 docker image.

flag provided but not defined: -output-path

Noticed that 2 related json output options were remove from an earlier PR https://github.com/deepfence/SecretScanner/pull/97/files

• json-filename and
• output-path

However, those options are still being used / referenced in some docs. They probably should be updated so as not to cause confusions.

E.g.

Side note, slightly curious about the reason behind the sudden change removing json output support, I didn't find much explanation / change info from the PR itself.

tried to run a scan-Job:

image: docker:latest
services:
  - docker:dind 

variables:
  DOCKER_HOST: tcp://docker:2375 
  DOCKER_DRIVER: overlay2 
  DOCKER_TLS_CERTDIR: ""
  IMAGE_NAME: myownimage 
  IMAGE_TAG: v1

secret-detection-image:
  stage: image-scan
  image:
    name: deepfenceio/deepfence_secret_scanner:2.1.0   
    entrypoint: [""]
  script:
    - echo ${REGISTRY_TOKEN} | docker login --username ${REGISTRY_USER} --password-stdin $CI_REGISTRY  
    - /home/deepfence/usr/SecretScanner -config-path /home/deepfence/usr --image-name ${DOCKER_ENV_CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${IMAGE_TAG}

results in:

$ /home/deepfence/usr/SecretScanner -config-path /home/deepfence/usr --image-name ${DOCKER_ENV_CI_REGISTRY_IMAGE}/${IMAGE_NAME}:${IMAGE_TAG}
INFO[2023-11-30T13:58:15Z] main.go:131 Scanning image registry.gitlab.com/...omited.../...omited.../myownimage:v1 for secrets... 
ERRO[2023-11-30T13:58:16Z] utils.go:46 cmd: /usr/bin/podman --remote --url unix:///run/podman/podman.sock ps 
ERRO[2023-11-30T13:58:16Z] utils.go:47 exit status 125                              
WARN[2023-11-30T13:58:16Z] autodetect.go:256 podman ps:exit status 125: Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: Get "http://d/v4.5.1/libpod/_ping": dial unix /run/podman/podman.sock: connect: no such file or directory 
ERRO[2023-11-30T13:58:23Z] process_image.go:65 scanImage: Could not save container image: could not detect container runtime. Check if the image name is correct. 
FATA[2023-11-30T13:58:23Z] main.go:134 main: error while scanning image: %scould not detect container runtime 
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

Hi, I used this tool to scan an image whose size is 26.83GB, then I received a fatal error:

scanImage: Could not save container image: exit status 1. Check if the image name is correct.
main: error while scanning image: exit status 1
panic: Fatal error....

goroutine 1 [running]:
github.com/deepfence/SecretScanner/core.(*Logger).Log(0x4000800f5b, 0x5, {0x1094c61, 0x24}, {0xc00045dee8, 0x1, 0x1})
/home/deepfence/src/SecretScanner/core/log.go:68 +0x25e
github.com/deepfence/SecretScanner/core.(*Logger).Fatal(...)
/home/deepfence/src/SecretScanner/core/log.go:73
main.runOnce()
/home/deepfence/src/SecretScanner/main.go:108 +0x145
main.main()
/home/deepfence/src/SecretScanner/main.go:148 +0x114

For diversity/inclusion reasons, replace 'blacklist' with 'exclude':

Config keys:

• blacklisted_strings
• blacklisted_extensions
• blacklisted_paths
• blacklisted_entropy_extensions

Error messages:

• "Blacklisted string %s matched"
• "matchString: Skipping matches containing blacklisted strings"
• "processHsRegexMatch: Skipping matches containing blacklisted strings"

If possible, retain the blacklist_* config keys for backwards compatibility, using them as undocumented aliases for the equivalent exclude_ keys.

Hello,
I run SecretScanner as a http server.
But I do not know how to use curl command to POST data to http server.
This is my example:

curl -X POST http://0.0.0.0:8080/secret-scan -d '{"image_name_with_tag_list": ["<my_image_name:tag>"]}' -H 'Content-Type: application/json'

And output is:

{"error":"Image Name with tag list is required "}

I have trace code, and in http.go file, I found the runSecretScan func in line 49.
But I always POST fail to http server, what is POST data format?

Hi,
When the image scan is executed the SecretScanner runs docker save and then extract the content of the saved tar then perform a scan on it.

It would be great to have a functionality to provide the tar image directly for scanning so that we don't have to use Docker daemon to run docker save first.

I read the documentation and found that I can exclude certain paths by appending them to exclude_paths.

Trying to evaluate this tool, I found the following false positive:

    {
      "Image Layer ID": "xxxxxxxxxxx",
      "Matched Rule ID": 135,
      "Matched Rule Name": "Contains a private key",
      "Matched Part": "contents",
      "String to Match": "",
      "Signature to Match": "-----BEGIN (EC|RSA|DSA|OPENSSH|PGP) PRIVATE KEY",
      "Severity": "medium",
      "Severity Score": 5.08,
      "Starting Index of Match in Original Content": 0,
      "Relative Starting Index of Match in Displayed Substring": 0,
      "Relative Ending Index of Match in Displayed Substring": 26,
      "Full File Name": "usr/local/share/.cache/yarn/v6/npm-proxy-agent-5.0.0-d31405c10d6e8431fde96cba7a0c027ce01d633b-integrity/node_modules/proxy-agent/test/ssl-cert-snakeoil.key",
      "Matched Contents": "-----BEGIN RSA PRIVATE KEY-----"
    },

While there's a private key in the path, it's added to the container by the repo's dependencies. So inorder to remove any detection of secrets in the usr/local/share/.cache path, I appended it to exclude_paths list:

Snippet:

# Secret Scanner Configuration File

blacklisted_strings: ["node_modules"] # skip matches containing any of these strings (case sensitive)
blacklisted_extensions: [".exe", ".jpg", ".jpeg", ".png", ".gif", ".bmp", ".tiff", ".tif", ".psd", ".xcf", ".zip", ".tar.gz", ".ttf", ".lock", ".pem"]
blacklisted_paths: ["{sep}var{sep}lib{sep}docker", "{sep}var{sep}lib{sep}containerd", "{sep}var{sep}lib{sep}containers", "{sep}var{sep}lib{sep}crio", "{sep}var{sep}run{sep}containers", "{sep}bin", "{sep}boot", "{sep}dev", "{sep}lib", "{sep}lib64", "{sep}media", "{sep}proc", "{sep}run", "{sep}sbin", "{sep}usr{sep}lib", "{sep}sys", "{sep}home{sep}kubernetes"]
exclude_paths: ["{sep}var{sep}lib{sep}docker", "{sep}var{name_sep}lib{name_sep}docker","{sep}var{sep}lib{sep}containerd", "{sep}var{name_sep}lib{name_sep}containerd", "{sep}usr{sep}local{sep}share{sep}.cache"] # use {sep} for the OS' path seperator and {name_sep} for -  (i.e. / or \)

signatures:
  - part:  'extension'

And ran the command: docker run -it --rm --name=deepfence-secretscanner -v $(pwd):/home/deepfence/output -v /var/run/docker.sock:/var/run/docker.sock deepfenceio/deepfence_secret_scanner:latest -image-name <image>:latest --config-path secretscanner (where the config.yaml is saved in secretscanner directory).

SecretScanner still detects this path and the false positive.

PS: I tried using {name_sep} instead of {sep}, and tried adding the path to blacklisted_strings and blacklisted_paths. Nothing worked so far.

Container build tools have tar exporters, it would be helpful to support scanning of existing tarballs.

ie a flag for -tar-path that skips the saveImageData() step (removing the need for docker) and uses the supplied image tarball

idea :- https://docs.github.com/en/enterprise-cloud@latest/code-security/secret-scanning/managing-alerts-from-secret-scanning | need same feature using our secretscanner | ---> how to :- SARIF support for secret scanning using github action

Hi,

I'm running the SecretScanner in local mode using the docker image:
docker run --rm -v /tmp/vulnerable_repo:/target -v /tmp/test_sec:/artifacts deepfenceio/deepfence_secret_scanner -config-path /artifacts -debug-level DEBUG -multi-match -maximum-file-size 512 -max-multi-match 5 -json-filename report.json -output-path /artifacts -local /target

The tool works perfectly, and it detect all the repository secrets.

But in some cases the target repositories may contain files with meta attributes. For example:

{
    "mariadb": {
        "host": "MARIADB_HOST",
        "port": "MARIADB_PORT",
        "rootMariaPwd": "MYSQL_ROOT_PASSWORD_EXAMPLE",
    },
}

I'm triying to whitelist all the matches containing the string _EXAMPLE with no success:

# Secret Scanner Configuration File

blacklisted_extensions: []
blacklisted_paths: []
blacklisted_strings:
- _EXAMPLE

signatures:
- name: Generic credentials
  part: contents
  regex: (?i)(?:'|"){0,1}(?:[a-z0-9\-_.]{0,25})(?:key|api|apikey|token|secret|client|pass|pwd|passwd|password|auth|cred|authentication)(?:[0-9a-z\-_\s.]{0,20})(?:'|"){0,1}(?:=|>|:=|\|\|:|<=|=>|:)(?:'|\"|\s|=){0,5}([0-9a-z\-_\.=:@!]{8,512})['|\"|\n|\r|\s]
  regextype: 'large'
  severity: high
  severityscore: 10

Is there something wrong with my configuration ? or it is a SecretScanner bug ?

root@bb1:~# SecretScanner -image-name worker1
Initializing....
Scanning image worker1 for secrets...
scanImage: Could save container image failed to save image: invalid output path: directory "/tmp/Deepfence/SecretScanning/df_worker1" does not exist

findSecretsInImage: failed to save image: invalid output path: directory "/tmp/Deepfence/SecretScanning/df_worker1" does not exist

main: error while scanning image: failed to save image: invalid output path: directory "/tmp/Deepfence/SecretScanning/df_worker1" does not exist

panic: Fatal error....

goroutine 1 [running]:
github.com/deepfence/SecretScanner/core.(*Logger).Log(0x7ffc1c756662, 0x5, {0x56cb7e, 0x24}, {0xc00033ff50, 0x1, 0x1})
	/root/go/src/github.com/deepfence/SecretScanner/core/log.go:68 +0x25e
github.com/deepfence/SecretScanner/core.(*Logger).Fatal(...)
	/root/go/src/github.com/deepfence/SecretScanner/core/log.go:73
main.main()
	/root/go/src/github.com/deepfence/SecretScanner/main.go:139 +0x14c

Ubuntu 20
go version go1.17.6 linux/amd64

Exclude files and secrets which comes with the system like

• /etc/ssh/ssh_host_dsa_key
• shadow files

passing a flag like, -container-runtime or -cr during runtime would reduce the overall time taken to scan images when underlying runtime is known to user.

$ go get github.com/deepfence/SecretScanner go: downloading github.com/deepfence/SecretScanner v1.1.2 go: github.com/deepfence/SecretScanner upgrade => v1.1.2 go get: github.com/deepfence/[email protected] requires github.com/deepfence/[email protected]: invalid version: unknown revision 000000000000

What is the problem here? May I have the latest updated the documentation or can you fix the go.mod file?

Please check this issue for more details.
deepfence/ThreatMapper#373

Hi I saw blacklisted_paths in config.yaml
I use customization content pattern, but result is not skip blacklisted_paths path
Does blacklisted_paths work on scan image mode?

SecretScanner takes >250 mb of memory when scanning hosts with huge number of files

1. Type:
   circle ci config file
   keys such as auth, username, password is not detected

Ex: auth:
username: circleclidockeruser
password: circleclidockerpassword # context / project UI env-var reference

2. API keys in openweather maps (must be shown as a warning)

Example:
Highligted text is not detected:
url = "https://api.openweathermap.org/data/2.5/find?q=Palo+Alto&units=imperial&type=accurate&mode=json&APPID=`ba3447bf3NOTREAL18414e1f995f68aeb6d`"

3. Variable names such as username, user, password and passwd are not detected

Example:
USER = 'realusername'
PASSWORD = 'Realpassword@1234'

usr = 'anotherrealluser'
passwd = 'anotherrealpasswd'

4. There is no username and password in this line,
   https://github.com/apache/jmeter/blob/efe50ca5b150cdbdb578886f8b4d98d3f1ea264f/xdocs/usermanual/curl.xml#L173
   I get a false positive scan result:

 {
      "Image Layer ID": "",
      "Matched Rule ID": 118,
      "Matched Rule Name": "Username and password in URI",
      "Matched Part": "contents",
      "String to Match": "",
      "Signature to Match": "([\\w+]{1,24})(://)([^$\u003c]{1})([^\\s\";]{1,}):([^$\u003c]{1})([^\\s\";/]{1,})@[-a-zA-Z0-9@:%._\\+~#=]{1,256}\\.[a-zA-Z0-9()]{1,24}([^\\s]+)",
      "Severity": "high",
      "Severity Score": 7.53,
      "Starting Index of Match in Original Content": 9408,
      "Relative Starting Index of Match in Displayed Substring": 50,
      "Relative Ending Index of Match in Displayed Substring": 72,
      "Full File Name": "/deepfence/mnt/root/jmeter/xdocs/usermanual/curl.xml",
      "Matched Contents": "xy\u003c/b\u003e\u003csource\u003ecurl 'https://example.invalid/' -x '�[31mhttps://aa:[email protected]�[0mvalid:8042'\u003c/source\u003e\u003c/p\u003e"
    }

5. However this line must be detected as there is a mention of passwd
   https://github.com/apache/jmeter/blob/efe50ca5b150cdbdb578886f8b4d98d3f1ea264f/xdocs/usermanual/curl.xml#L174

"https://example.invalid" -u 'user:passwd' --basic</source></p>

Detect Kubernetes service account token in secret scan

pre-commit - A framework for managing and maintaining multi-language pre-commit hooks. https://pre-commit.com

SecretScanner is currently a standalone application that needs to be run every time we want to process some data. The output produced is a json payload.
Going forward, we want to make it easy to plug SecretScanner with other processes.
ThreatMapper has some plugin definition on how to server content over gRPC. We want to reuse that architecture to serve API via SecretScanner and later integrate it inside ThreatMapper

I suggest creating a GitHub Action for SecretScanner to allow developers to easily have their code changes scanned for secrets (such as validation of Pull Requests before they are merged). Should be pretty straightforward using your filesystem scanning option. If there was a documented way to do this, I would be interested in trying it.

As pointed out here: #22 (comment), SecretScanner relies on docker to execute commands.
Some systems use k8s only have containerd runtime available. We need to support those case.
Proposed solution is to rely on https://github.com/deepfence/vessel project to allow more cntainer runtimes.

output/output.go:9:2: github.com/deepfence/[email protected] (replaced by ./agent-plugins-grpc): reading agent-plugins-grpc/go.mod: open /home/User/Desktop/Tools/SecretScanner/agent-plugins-grpc/go.mod: no such file or directory

Add more signatures to the config
https://github.com/dxa4481/truffleHogRegexes/blob/master/truffleHogRegexes/regexes.json

I'm getting this error:

output/output.go:11:2: github.com/deepfence/[email protected] (replaced by ./agent-plugins-grpc): reading agent-plugins-grpc/go.mod: open /go/src/github.com/deepfence/SecretScanner/agent-plugins-grpc/go.mod: no such file or directory

The secret scanner process running inside agent gets Killed on VM 143.198.68.242
Process log:

      "Full File Name": "/fenced/mnt/host/var/snap/docker/common/var-lib-docker/overlay2/5043b3ac304141297e67251e1ce08efd890a2e0b738a4fdc0087daee3d95ca9c/diff/var/log/dpkg.log",
      "Matched Contents": ".log"
    }

Killed

top:

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                       
1375878 root      20   0 1404896  40476  22884 D  19.3   1.0   0:13.14 SecretScanner  

The memory % and cpu usage does not grow with time.
Issue has been produced only in this VM