project-ncl / sourceclear-invoker Goto Github PK

Wrapper project for invoking the SourceClear Scanner

License: Apache License 2.0

Java 100.00%

sourceclear-invoker's Introduction

SourceClear Invoker

This project is designed to project a simple wrapper around the existing SourceClear command line in order to provide the ability to parse the results and output a configurable JUnit pass/fail test. This can then be used as part of a pipeline in order to verify source repositories and final deliverables.

Requirements

glide: brew install glide on macOS, check here for installation instructions on other platforms
srcclr:
- Linux: SourceClear should be installed via its rpm (it should check for that). The yum repo for that is:

    [SourceClear]
    name=SourceClear
    baseurl=https://download.sourceclear.com/redhat/noarch/
    enabled=1
    gpgcheck=1
    gpgkey=https://download.sourceclear.com/redhat/SRCCLR-GPG-KEY

macOS: with homebrew - brew install srcclr

Usage

Currently this project will build a jar-with-dependencies although this is primarily aimed at local testing only. It provides a simplified interface to SourceClear e.g.

Usage: SrcClrWrapper [-dehV] [--email-address=<emailAddress>]
                     [--email-server=<emailServer>] [-c=<product>]
                     [--processor=<processor>] [-t=<threshold>] [COMMAND]
Wrap SourceClear and invoke it.
      --email-address=<emailAddress>[,<emailAddress>...]
                         Comma separated list of email addresses to notify. Domain portion of
                         first will be used as FROM address
      --email-server=<emailServer>
                         SMTP Server to use to send notification email
  -p, --product=<product>    Product Name
  -v, --product-version=<version>   Version of the product
  --memory               Memory allocation for the forked process. If not set will default to the current VM size.
  --package=<subpackage> CPE Subpackage Name
  --trace                Enables trace logging from SourceClear. Disables JSON output.
  --log                  Directory path to output log file containing results. Defaults to 'target'. Set to empty to disable.
  --json                 Directory path to output processed JSON as a file. Defaults to 'target'. Set to empty to disable.
  -d, --debug            Enable debug.
  -e, --exception        Throw exception on vulnerabilities found.
  -h, --help             Show this help message and exit.
  --processor=<processor>
                         Processor to use to analyse SourceClear results. Default is
                           'cvss'
  --profile=<profile>   Profile for VeraCode scanning. Defaults to empty
                            (default profile).
  -t, --threshold=<threshold>
                         Threshold on which exception is thrown. Only used with CVSS
                           Processor
  -V           Print version information and exit.


Usage: SrcClrWrapper scm [-dehV] [--ref=REF] --url=URL [-t=<threshold>]
Scan a SCM URL
      --ref=REF     the SCM reference (e.g. git sha, tag)
      --url=URL     the SCM url
  -d, --debug       Enable debug.
  -e, --exception   Throw exception on vulnerabilities found.
  -h, --help        Show this help message and exit.
  -t, --threshold=<threshold>
                    Threshold on which exception is thrown.
  -V    Print version information and exit.
  --maven-param     Extra Maven parameters

Note : for the SCM --url parameter both . and file:// URLs are supported to scan the local filesystem.


Usage: SrcClrWrapper binary [-dehV] --name=NAME --url=URL
                            [-t=<threshold>]
Scan a remote binary
      --url=URL     the remote file url
  -d, --debug       Enable debug.
  -e, --exception   Throw exception on vulnerabilities found.
  -h, --help        Show this help message and exit.
  -t, --threshold=<threshold>
                    Threshold on which exception is thrown.
  -V     Print version information and exit.

Its main use is to be ran inside Jenkins as a JUnit test suite e.g.

mvn -Dmaven.buildNumber.skip=true -Pjenkins clean test  '-DargLine=-Dsourceclear="--product-version=1.0.0 -p=koji-build-finder scm --url=https://github.com/release-engineering/koji-build-finder.git --ref=koji-build-finder-1.0.0"'

Features

It supports reading a configuration from the command or from $HOME/.srcclr/invoker.properties.
It can send a notification email to a specified email address with a summary of any problems found.
It can examine either the CVSS score returned from SourceClear or examine the CVE identifier and then query the results using the CPE (product name) against the Red Hat Security Data API ( https://access.redhat.com/labs/securitydataapi/ )
Sample Jenkins jobs are provided in the jenkins directory.

Notes

Currently the code requires the product name and version to be passed in. It will assemble a CPE from that information. While we did consider using the CPE parser library but due to the fact we are not currently parsing or comparing CPEs the extra library isn't required.

sourceclear-invoker's People

Contributors

Stargazers

Watchers

Forkers

rnc jeremychoi cunningt rgdoliveira tombentley pombredanne gsaslis paulgallagher75

sourceclear-invoker's Issues

`mvn package` fails with `"glide" is not installed or not found on PATH`

I'm on OSX (10.14) and after I've installed the srcclr command-line tool, and activated it with srcclr activate I now get a bunch of errors when running mvn package, including get a different error:

2019-11-20/15:36:12.742 com.sourceclear.engine.component.ComponentEngineBuilder	DEBUG	Invalid collector name is specified: "SOFILE"
2019-11-20/15:36:12.742 com.sourceclear.engine.component.ComponentEngineBuilder	DEBUG	collectorsToRun: []
2019-11-20/15:36:12.743 com.sourceclear.engine.component.ComponentEngineBuilder	DEBUG	collectorsToSkip: []
2019-11-20/15:36:12.864 com.sourceclear.agent.commands.ScanCommand	INFO	"glide" is not installed or not found on PATH
2019-11-20/15:36:12.864 com.sourceclear.agent.commands.ScanCommand	INFO	Error while configuring engine
2019-11-20/15:36:12.932 com.sourceclear.agent.EntryPointImpl	DEBUG	Exiting with 1

com.sourceclear.agent.FatalException: SourceClear is unable to continue with scanning the project because at least one\nof the build systems we found for your project is not ready for build.\nWe suggest you scan the project again after reviewing and correcting the errors.
	at com.sourceclear.agent.commands.ScanCommand.handleScanErrors(ScanCommand.java:972)
	at com.sourceclear.agent.commands.ScanCommand.executeRepoScan(ScanCommand.java:749)
	at com.sourceclear.agent.commands.ScanCommand.execute(ScanCommand.java:469)
	at com.sourceclear.agent.EntryPointImpl.runVerb(EntryPointImpl.java:371)
	at com.sourceclear.agent.EntryPointImpl.dispatchVerbOptions(EntryPointImpl.java:320)
	at com.sourceclear.agent.EntryPointImpl.apply(EntryPointImpl.java:156)
	at com.sourceclear.agent.Main.start(Main.java:96)
	at com.sourceclear.agent.Main.main(Main.java:101)
Caused by:
	com.sourceclear.engine.component.CollectionException: SourceClear is unable to continue with scanning the project because at least one\nof the build systems we found for your project is not ready for build.\nWe suggest you scan the project again after reviewing and correcting the errors.
		at com.sourceclear.agent.commands.ScanCommand.handleScanErrors(ScanCommand.java:972)
		at com.sourceclear.agent.commands.ScanCommand.executeRepoScan(ScanCommand.java:749)
		at com.sourceclear.agent.commands.ScanCommand.execute(ScanCommand.java:469)
		at com.sourceclear.agent.EntryPointImpl.runVerb(EntryPointImpl.java:371)
		at com.sourceclear.agent.EntryPointImpl.dispatchVerbOptions(EntryPointImpl.java:320)
		at com.sourceclear.agent.EntryPointImpl.apply(EntryPointImpl.java:156)
		at com.sourceclear.agent.Main.start(Main.java:96)
		at com.sourceclear.agent.Main.main(Main.java:101)
	at org.zeroturnaround.exec.InvalidExitUtil.checkExit(InvalidExitUtil.java:27)
	at org.zeroturnaround.exec.WaitForProcess.call(WaitForProcess.java:114)
	at org.zeroturnaround.exec.ProcessExecutor.waitFor(ProcessExecutor.java:1097)
	at org.zeroturnaround.exec.ProcessExecutor.execute(ProcessExecutor.java:925)
	at com.redhat.engineering.srcclr.SrcClrInvoker.execSourceClear(SrcClrInvoker.java:142)
	at com.redhat.engineering.srcclr.SCM.call(SCM.java:121)
	at com.redhat.engineering.srcclr.SCM.call(SCM.java:40)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1743)
	at picocli.CommandLine.access$900(CommandLine.java:145)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2157)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2116)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1928)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:2282)
	at com.redhat.engineering.srcclr.SrcClrWrapper.invokeWrapper(SrcClrWrapper.java:130)
	at com.redhat.engineering.srcclr.SCBase.exeSC(SCBase.java:44)
	at com.redhat.engineering.srcclr.internal.SourceClearInvokerTest.runScmGoScan(SourceClearInvokerTest.java:227)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.contrib.java.lang.system.internal.LogPrintStream$1$1.evaluate(LogPrintStream.java:30)
	at org.junit.contrib.java.lang.system.internal.PrintStreamHandler$3.evaluate(PrintStreamHandler.java:48)
	at org.junit.contrib.java.lang.system.internal.LogPrintStream$1.evaluate(LogPrintStream.java:26)
	at org.junit.contrib.java.lang.system.internal.LogPrintStream$1$1.evaluate(LogPrintStream.java:30)
	at org.junit.contrib.java.lang.system.internal.PrintStreamHandler$3.evaluate(PrintStreamHandler.java:48)
	at org.junit.contrib.java.lang.system.internal.LogPrintStream$1.evaluate(LogPrintStream.java:26)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[DEBUG] Caught an internal exception
picocli.CommandLine$ExecutionException: Error while calling command (scanning https://github.com/srcclr/example-go-glide version 96b1262): com.redhat.engineering.srcclr.utils.InternalException: Error executing SourceClear
	at picocli.CommandLine.executeUserObject(CommandLine.java:1752)
	at picocli.CommandLine.access$900(CommandLine.java:145)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2157)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2116)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1928)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:2282)
	at com.redhat.engineering.srcclr.SrcClrWrapper.invokeWrapper(SrcClrWrapper.java:130)
	at com.redhat.engineering.srcclr.SCBase.exeSC(SCBase.java:44)
	at com.redhat.engineering.srcclr.internal.SourceClearInvokerTest.runScmGoScan(SourceClearInvokerTest.java:227)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.contrib.java.lang.system.internal.LogPrintStream$1$1.evaluate(LogPrintStream.java:30)
	at org.junit.contrib.java.lang.system.internal.PrintStreamHandler$3.evaluate(PrintStreamHandler.java:48)
	at org.junit.contrib.java.lang.system.internal.LogPrintStream$1.evaluate(LogPrintStream.java:26)
	at org.junit.contrib.java.lang.system.internal.LogPrintStream$1$1.evaluate(LogPrintStream.java:30)
	at org.junit.contrib.java.lang.system.internal.PrintStreamHandler$3.evaluate(PrintStreamHandler.java:48)
	at org.junit.contrib.java.lang.system.internal.LogPrintStream$1.evaluate(LogPrintStream.java:26)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: com.redhat.engineering.srcclr.utils.InternalException: Error executing SourceClear
	at com.redhat.engineering.srcclr.SrcClrInvoker.execSourceClear(SrcClrInvoker.java:174)
	at com.redhat.engineering.srcclr.SCM.call(SCM.java:121)
	at com.redhat.engineering.srcclr.SCM.call(SCM.java:40)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1743)
	... 43 common frames omitted
Caused by: org.zeroturnaround.exec.InvalidExitValueException: Unexpected exit value: 1, allowed exit values: [0], executed command [/usr/local/Cellar/srcclr/3.4.17/libexec/jre/bin/java, -jar, /usr/local/Cellar/srcclr/3.4.17/libexec/srcclr-3.4.17.jar, --debug, scan, --json, --skip-collectors="SOFile", --url, https://github.com/srcclr/example-go-glide, --ref, 96b1262, --no-upload] in directory /var/folders/0h/28qktgvj01j9hp61hck2p7sc0000gn/T/sourceclear-invoker-5314633448743086581 with environment {GIT_BRANCH=null, GIT_URL=null, GOPATH=/var/folders/0h/28qktgvj01j9hp61hck2p7sc0000gn/T/sourceclear-gopath-8004744084892554207}, output was 7590 bytes (truncated):

sourceclear-invoker hiding GC overhead limit exceeded error

Getting the following when running sourceclear-invoker on a large project (https://github.com/apache/camel-quarkus) on a
Jenkins with 8192MB RAM. When I ssh in and run with the srcclr command line tool, I could see that it was a java.lang.OutOfMemoryError: GC overhead limit exceeded error, but running with the invoker seemed to swallow the error.

Is there an option I can pass to get more log from the client?

`INFO com.redhat.engineering.srcclr.SCM- Scanning local file system with /home/jenkins/workspace/pncgeneration/directory
INFO c.r.engineering.srcclr.SrcClrInvoker- Invoking in environment {GIT_BRANCH=null, GIT_URL=null, GOPATH=/tmp/sourceclear-gopath-1527597488020721299} with command {/opt/srcclr-3.7.18/jre/bin/java -Xmx3702521856 -jar /opt/srcclr-3.7.18/srcclr-3.7.18.jar -Dsun.jnu.encoding=UTF-8 -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2,TLSv1.1,TLSv1 scan --json /tmp/sourceclear-json-4749302074553051076/7aefdab9-eebe-4e02-880d-190b55b83d24 /home/jenkins/workspace/pncgeneration/directory --ref camel-quarkus-1.8.1.fuse-800004 --allow-dirty --loud}
ERROR c.r.engineering.srcclr.SrcClrInvoker-
ERROR c.r.engineering.srcclr.SrcClrInvoker- This project does not seem to build.
ERROR c.r.engineering.srcclr.SrcClrInvoker- Because of this, SourceClear cannot scan it. Please ensure that the project compiles prior to scanning.
ERROR c.r.engineering.srcclr.SrcClrInvoker- 2021-05-20/00:53:59.939 com.sourceclear.engine.component.collectors.JsonComponentGraphNativeCollector ERROR Encountered errors while collecting component information.

ERROR c.r.engineering.srcclr.SrcClrInvoker- Invalid exit: Unexpected exit value: 1, allowed exit values: [0], executed command [/opt/srcclr-3.7.18/jre/bin/java, -Xmx3702521856, -jar, /opt/srcclr-3.7.18/srcclr-3.7.18.jar, -Dsun.jnu.encoding=UTF-8 -Dfile.encoding=UTF-8 -Dhttps.protocols=TLSv1.2,TLSv1.1,TLSv1, scan, --json, /tmp/sourceclear-json-4749302074553051076/7aefdab9-eebe-4e02-880d-190b55b83d24, /home/jenkins/workspace/pncgeneration/directory, --ref, camel-quarkus-1.8.1.fuse-800004, --allow-dirty, --loud] in directory /tmp/sourceclear-invoker-722534425713724011 with environment {GIT_BRANCH=null, GIT_URL=null, GOPATH=/tmp/sourceclear-gopath-1527597488020721299}, output was 167 bytes:
2021-05-20/00:53:59.939 com.sourceclear.engine.component.collectors.JsonComponentGraphNativeCollector ERROR Encountered errors while collecting component information.`

NoSuchElementException runing tool

Trying to run the tool on Kafka 2.4, using java -jar target/srcclr-1.4-jar-with-dependencies.jar --product=cpe:/a:redhat:amq_streams:1.4 --product-version=1.4 scm --url=https://github.com/apache/kafka.git --ref=2.4.0, at the end of the scan (after the report URL is logged) I get

Exception in thread "main" java.util.NoSuchElementException: No value present
	at java.util.Optional.get(Optional.java:135)
	at com.redhat.engineering.srcclr.notification.DefaultStringNotifier.toString(DefaultStringNotifier.java:30)
	at com.redhat.engineering.srcclr.notification.LogFileNotifier.notify(LogFileNotifier.java:47)
	at com.redhat.engineering.srcclr.SrcClrWrapper.notifyListeners(SrcClrWrapper.java:210)
	at com.redhat.engineering.srcclr.SCM.call(SCM.java:131)
	at com.redhat.engineering.srcclr.SCM.call(SCM.java:40)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1743)
	at picocli.CommandLine.access$900(CommandLine.java:145)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2157)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2116)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1928)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:2282)
	at com.redhat.engineering.srcclr.SrcClrWrapper.invokeWrapper(SrcClrWrapper.java:130)
	at com.redhat.engineering.srcclr.SrcClrWrapper.main(SrcClrWrapper.java:118)

sourceclear-invoker cannot correctly scan 3scale apicast-operator repo

I try to scan the repository and it seems like sourceclear-invoker is hitting some issue parsing the output:

$ mvn -Pjenkins clean test -DargLine='-Xmx4G -Dsourceclear="--product=3scale/apicast-operator --product-version=2.11.0 scm --url=https://github.com/3scale/apicast-operator.git --ref=master "'
[INFO] Scanning for projects...
[INFO]
[INFO] -------------------< com.redhat.engineering:srcclr >--------------------
[INFO] Building SourceClear JUnit Wrapper 1.6-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.0.0:clean (default-clean) @ srcclr ---
[INFO] Deleting /Users/yorgos/dev/redhat/3scale/sourceclear-invoker/target
[INFO]
[INFO] --- buildnumber-maven-plugin:1.4:create (buildnumber) @ srcclr ---
[INFO] ShortRevision tag detected. The value is '8'.
[INFO] Executing: /bin/sh -c cd '/Users/yorgos/dev/redhat/3scale/sourceclear-invoker' && 'git' 'rev-parse' '--verify' '--short=8' 'HEAD'
[INFO] Working directory: /Users/yorgos/dev/redhat/3scale/sourceclear-invoker
[INFO] Storing buildNumber: e055da2f at timestamp: 01 February 2021
[INFO] Storing buildScmBranch: master
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) @ srcclr ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (avoid-trip-hazards) @ srcclr ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-best-practices) @ srcclr ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ srcclr ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ srcclr ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 43 source files to /Users/yorgos/dev/redhat/3scale/sourceclear-invoker/target/classes
[INFO] /Users/yorgos/dev/redhat/3scale/sourceclear-invoker/src/main/java/com/redhat/engineering/srcclr/SrcClrWrapper.java: /Users/yorgos/dev/redhat/3scale/sourceclear-invoker/src/main/java/com/redhat/engineering/srcclr/SrcClrWrapper.java uses or overrides a deprecated API.
[INFO] /Users/yorgos/dev/redhat/3scale/sourceclear-invoker/src/main/java/com/redhat/engineering/srcclr/SrcClrWrapper.java: Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-commonjava-standards) @ srcclr ---
[INFO] Skipping Rule Enforcement.
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ srcclr ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ srcclr ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 12 source files to /Users/yorgos/dev/redhat/3scale/sourceclear-invoker/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M3:test (default-test) @ srcclr ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.redhat.engineering.srcclr.SourceClearTest
[INFO] Retrieved argument [--product, 3scale/apicast-operator, --product-version, 2.11.0, scm, --url=https://github.com/3scale/apicast-operator.git, --ref=master]
[INFO] Invoking in environment {GIT_BRANCH=null, GIT_URL=null, GOPATH=/var/folders/0h/28qktgvj01j9hp61hck2p7sc0000gn/T/sourceclear-gopath-4886510764422553483} command [/usr/local/Cellar/srcclr/3.7.9/libexec/jre/bin/java, -jar, /usr/local/Cellar/srcclr/3.7.9/libexec/srcclr-3.7.9.jar, scan, --json, /var/folders/0h/28qktgvj01j9hp61hck2p7sc0000gn/T/sourceclear-json-8946556462077758082/6d97b96e-0e5b-4e69-9fb9-4162475961bc, --url, https://github.com/3scale/apicast-operator.git, --ref, master] ....
[INFO] Found vulnerability 'Directory Traversal' with CVE ID 2014-9358 in library github.com/docker/docker::v0.7.3
[INFO] Found vulnerability 'Directory Traversal' with CVE ID 2014-9356 in library github.com/docker/docker::v0.7.3
[INFO] Found vulnerability 'DNS Rebinding' with CVE ID 2018-1099 in library github.com/coreos/etcd::v3.3.10
[INFO] Found vulnerability 'Cross-site Request Forgery (CSRF)' with CVE ID 2018-1098 in library github.com/coreos/etcd::v3.3.10
[INFO] Found vulnerability 'Denial Of Service (DoS)' with CVE ID 2020-27813 in library github.com/gorilla/websocket::v1.4.0
[INFO] Found vulnerability 'Authorization Bypass' with CVE ID 2020-26160 in library github.com/dgrijalva/jwt-go::v3.2.0
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 61.659 s <<< FAILURE! - in com.redhat.engineering.srcclr.SourceClearTest
[ERROR] runSourceClear(com.redhat.engineering.srcclr.SourceClearTest)  Time elapsed: 61.621 s  <<< ERROR!
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
	at com.redhat.engineering.srcclr.SourceClearTest.runSourceClear(SourceClearTest.java:35)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   SourceClearTest.runSourceClear:35->SCBase.exeSC:44 » IndexOutOfBounds Index: 0...
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:06 min
[INFO] Finished at: 2021-02-01T22:45:11+02:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project srcclr: There are test failures.
[ERROR]
[ERROR] Please refer to /Users/yorgos/dev/redhat/3scale/sourceclear-invoker/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Help output throws "InternalException: Command line parse exception"

Running the tool with no options or --help prints some help info followed by a stacktrace.

$ java -jar target/srcclr-1.4-SNAPSHOT-jar-with-dependencies.jar
Missing required options [--product=<product>, --product-version=<version>]
Usage: SrcClrWrapper [-dehV] [--trace] [--email-server=<emailServer>]
                     -p=<product> [--package=<packageName>]
                     [--processor=<processor>] [-t=<threshold>] -v=<version>
                     [--email-address=<emailAddresses>[,
                     <emailAddresses>...]]... [COMMAND]
Wrap SourceClear and invoke it.
  -d, --debug               Enable debug.
  -e, --exception           Throw exception on vulnerabilities found.
      --email-address=<emailAddresses>[,<emailAddresses>...]
                            Comma separated list of email address to notify.
                              Domain portion will be used as FROM address
      --email-server=<emailServer>
                            SMTP Server to use to send notification email
  -h, --help                Show this help message and exit.
  -p, --product=<product>   Product Name (in same format as CPE Product Name)
      --package=<packageName>
                            Package name. It's optional but required for RHOAR,
                              e.g. (vertx|swarm|springboot).
      --processor=<processor>
                            Processor (cve|cvss) to use to analyse SourceClear
                              results. Default is cve
  -t, --threshold=<threshold>
                            Threshold on which exception is thrown. Only used
                              with CVSS Processor
      --trace               Enable trace. Will DISABLE JSON OUTPUT
  -v, --product-version=<version>
                            Version of the product
  -V, --version             Print version information and exit.
Commands:
  scm     Scan a SCM URL (unmatched args are passed directly to SourceClear)
  binary  Scan a remote binary (unmatched args are passed directly to
            SourceClear)
Exception in thread "main" com.redhat.engineering.srcclr.utils.InternalException: Command line parse exception
	at com.redhat.engineering.srcclr.SrcClrWrapper.invokeWrapper(SrcClrWrapper.java:127)
	at com.redhat.engineering.srcclr.SrcClrWrapper.main(SrcClrWrapper.java:111)

no junit reports created if tests failed

I am seeing junit reports created successfully when tests pass, but not when they fail:

[DEBUG]   } ]
[DEBUG] }
[WARN] Potential vulnerability without a CVE with CVSS score of 5.0 found as Leakage Of Sensitive Data Over Websocket Protocol in library actioncable::5.2.3
[WARN] Potential vulnerability without a CVE with CVSS score of 6.4 found as Open Redirect in library actionpack::5.2.3
[DEBUG] Looking up https://access.redhat.com/labs/securitydataapi/cve/CVE-2019-16892.json
[INFO] Found vulnerability 'Denial Of Service (DoS)' with CVE ID 2019-16892 in library rubyzip::1.2.3
[DEBUG] Looking up https://access.redhat.com/labs/securitydataapi/cve/CVE-2019-15587.json
[INFO] Found vulnerability 'Cross-Site Scripting (XSS)' with CVE ID 2019-15587 in library loofah::2.2.3
[INFO] Report is <redacted>

[INFO] Writing log file to target/vulnerabilityLogFile.txt
[ERROR] Found issues when scanning Found 2 vulnerabilities : <redacted>?login=saml
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 29.112 s <<< FAILURE! - in com.redhat.engineering.srcclr.SourceClearTest
[ERROR] runSourceClear(com.redhat.engineering.srcclr.SourceClearTest)  Time elapsed: 28.405 s  <<< FAILURE!
java.lang.AssertionError: Found 2 vulnerabilities : <redacted>
	at com.redhat.engineering.srcclr.SourceClearTest.runSourceClear(SourceClearTest.java:40)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   SourceClearTest.runSourceClear:40 Found 2 vulnerabilities : <redacted>
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:24 min
[INFO] Finished at: 2019-11-22T05:18:19Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M3:test (default-test) on project srcclr: There are test failures.
[ERROR] 
[ERROR] Please refer to /productization/srcclr-invoker/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[Pipeline] echo
********************************************************************************
WARNING, non-zero return code, this may not be fatal, but is worth noting.
Command: install -m 600 /etc/srcclr/agent.yml.template /etc/srcclr/agent.yml ; mvn -f /productization/srcclr-invoker/pom.xml -Pjenkins clean test -DargLine='-Dsourceclear=" --product=3scale/zync --product-version=2.7.0  scm --url=https://code.engineering.redhat.com/gerrit/3scale/zync.git --ref=master_product   --quick --debug "' ; cp -r /productization/srcclr-invoker/target/surefire-reports /workDir/workspace 
Return Code: 1
********************************************************************************
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] junit
Recording test results
No test report files were found. Configuration error?

sourceclear-invoker cannot correctly scan 3scale-operator repo

I am trying to scan the 3scale-operator repository with:

$ mvn -Pjenkins clean test -DargLine='-Dsourceclear="--product=3scale/operator --product-version=2.11.0 scm --url=https://github.com/3scale/3scale-operator.git --ref=master "'

and sourceclear-invoker reports back the following error message:

...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.redhat.engineering.srcclr.SourceClearTest
[INFO] Retrieved argument [--product, apicast-operator, --product-version, 2.9.0, scm, --url=https://code.engineering.redhat.com/gerrit/3scale/apicast-operator.git, --ref=master_product]
[INFO] Invoking in environment {GIT_BRANCH=null, GIT_URL=null, GOPATH=/var/folders/0h/28qktgvj01j9hp61hck2p7sc0000gn/T/sourceclear-gopath-6828806545921668884} command [/usr/local/Cellar/srcclr/3.4.21/libexec/jre/bin/java, -jar, /usr/local/Cellar/srcclr/3.4.21/libexec/srcclr-3.4.21.jar, scan, --json, --url, https://code.engineering.redhat.com/gerrit/3scale/apicast-operator.git, --ref, master_product] ....
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 16.949 s <<< FAILURE! - in com.redhat.engineering.srcclr.SourceClearTest
[ERROR] runSourceClear(com.redhat.engineering.srcclr.SourceClearTest)  Time elapsed: 16.798 s  <<< ERROR!
com.redhat.engineering.srcclr.utils.InternalException:
Invalid split - gave group count of 2 for 2021-02-01/17:29:04.129 com.sourceclear.agent.commands.ScanCommand	ERROR	Unable to scan and generate report: null

	at com.redhat.engineering.srcclr.SourceClearTest.runSourceClear(SourceClearTest.java:35)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   SourceClearTest.runSourceClear:35->SCBase.exeSC:44 » Internal Invalid split - ...
[INFO]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  21.251 s
[INFO] Finished at: 2021-02-01T19:29:05+02:00
[INFO] ------------------------------------------------------------------------

However, I can scan other projects, by only changing product name and url:

$ mvn -Pjenkins clean test -DargLine='-Dsourceclear="--product=3scale/zync --product-version=2.11.0 scm --url=https://github.com/3scale/zync.git --ref=master "'

This makes me think there is something in the output returned for this repository that sourceclear-invoker does not handle correctly.

do not work

mvn -Dmaven.buildNumber.skip=true -Pjenkins clean test  '-DargLine=-Dsourceclear="--product-version=1.8.0.265.b01-1.fc32.x86_64 -p=java-1.8.0-openjdk binary   --url=file:////usr/lib/jvm/java-1.8.0-openjdk-1.8.0.265.b01-1.fc32.x86_64/"'

INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.redhat.engineering.srcclr.SourceClearTest
[INFO] Retrieved argument [--product-version, 1.8.0.265.b01-1.fc32.x86_64, -p, java-1.8.0-openjdk, binary, --url=file:////usr/lib/jvm/java-1.8.0-openjdk-1.8.0.265.b01-1.fc32.x86_64/]
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.404 s <<< FAILURE! - in com.redhat.engineering.srcclr.SourceClearTest
[ERROR] runSourceClear(com.redhat.engineering.srcclr.SourceClearTest)  Time elapsed: 0.192 s  <<< ERROR!
java.nio.file.FileAlreadyExistsException: /tmp/sourceclear-remote-cache-3695434843701160635
	at com.redhat.engineering.srcclr.SourceClearTest.runSourceClear(SourceClearTest.java:35)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   SourceClearTest.runSourceClear:35->SCBase.exeSC:44 » FileAlreadyExists /tmp/so...
[INFO] 
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------


rm /tmp/sourceclear-remote-cache-3695434843701160635
rm: cannot remove '/tmp/sourceclear-remote-cache-3695434843701160635': No such file or directory

or more funny:

java -jar target/srcclr-1.5-SNAPSHOT-jar-with-dependencies.jar binary [email protected]  --product=java-1.8.0-openjdk  --product-version=1.8.0.265.b01-1.fc32.x86_64   --url=file:////usr/lib/jvm/java-1.8.0-openjdk-1.8.0.265.b01-1.fc32.x86_64/
Missing required options [--product=<product>, --product-version=<version>]

Invoker should support local checkout

SourceClear's scm tool supports running on a local checkout, which would allow quite a few features -

ability to checkout not using sourceclear (you could checkout private repos more easily, and checkout protected repos like code.engineering repos)
ability to patch the directory with settings.xml and srcclr.yml before running

Stracktrace with scm scanning

--url takes a URL (obviously), but it would be nice if the tool didn't throw an exception if given something like [email protected]:strimzi/strimzi-kafka-oauth.git which would be acceptable to git clone but is not a URL. An error message to say "use a URL" would suffice, but it would be even nicer if it accepted that.

$ java -jar target/srcclr-1.4-SNAPSHOT-jar-with-dependencies.jar --product='cpe:/a:redhat:amq_streams:1.3' --product-version='1.3.0.CR1' scm --url [email protected]:strimzi/strimzi-kafka-oauth.git --ref master
INFO com.redhat.engineering.srcclr.SCM- Scanning local file system with [email protected]:strimzi/strimzi-kafka-oauth.git
INFO c.r.engineering.srcclr.SrcClrInvoker- Invoking in environment {GIT_BRANCH=null, GIT_URL=null, GOPATH=/tmp/sourceclear-gopath-218570056154285958} command [/opt/srcclr-3.4.3/jre/bin/java, -jar, /opt/srcclr-3.4.3/srcclr-3.4.3.jar, scan, --json, --skip-collectors="SOFile", [email protected]:strimzi/strimzi-kafka-oauth.git, --ref, master] ....
ERROR c.r.engineering.srcclr.SrcClrInvoker- 
ERROR c.r.engineering.srcclr.SrcClrInvoker- Invalid exit 
org.zeroturnaround.exec.InvalidExitValueException: Unexpected exit value: 1, allowed exit values: [0], executed command [/opt/srcclr-3.4.3/jre/bin/java, -jar, /opt/srcclr-3.4.3/srcclr-3.4.3.jar, scan, --json, --skip-collectors="SOFile", [email protected]:strimzi/strimzi-kafka-oauth.git, --ref, master] in directory /tmp/sourceclear-invoker-8487097337571164151 with environment {GIT_BRANCH=null, GIT_URL=null, GOPATH=/tmp/sourceclear-gopath-218570056154285958}, output was 0 bytes:

	at org.zeroturnaround.exec.InvalidExitUtil.checkExit(InvalidExitUtil.java:27)
	at org.zeroturnaround.exec.WaitForProcess.call(WaitForProcess.java:114)
	at org.zeroturnaround.exec.ProcessExecutor.waitFor(ProcessExecutor.java:1097)
	at org.zeroturnaround.exec.ProcessExecutor.execute(ProcessExecutor.java:925)
	at com.redhat.engineering.srcclr.SrcClrInvoker.execSourceClear(SrcClrInvoker.java:138)
	at com.redhat.engineering.srcclr.SCM.call(SCM.java:119)
	at com.redhat.engineering.srcclr.SCM.call(SCM.java:40)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1743)
	at picocli.CommandLine.access$900(CommandLine.java:145)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2157)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2116)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1928)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:2282)
	at com.redhat.engineering.srcclr.SrcClrWrapper.invokeWrapper(SrcClrWrapper.java:123)
	at com.redhat.engineering.srcclr.SrcClrWrapper.main(SrcClrWrapper.java:111)
Exception in thread "main" com.redhat.engineering.srcclr.utils.InternalException: Error executing SourceClear 
	at com.redhat.engineering.srcclr.SrcClrInvoker.execSourceClear(SrcClrInvoker.java:164)
	at com.redhat.engineering.srcclr.SCM.call(SCM.java:119)
	at com.redhat.engineering.srcclr.SCM.call(SCM.java:40)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1743)
	at picocli.CommandLine.access$900(CommandLine.java:145)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2157)
	at picocli.CommandLine$RunAll.handle(CommandLine.java:2116)
	at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1928)
	at picocli.CommandLine.parseWithHandlers(CommandLine.java:2282)
	at com.redhat.engineering.srcclr.SrcClrWrapper.invokeWrapper(SrcClrWrapper.java:123)
	at com.redhat.engineering.srcclr.SrcClrWrapper.main(SrcClrWrapper.java:111)
Caused by: org.zeroturnaround.exec.InvalidExitValueException: Unexpected exit value: 1, allowed exit values: [0], executed command [/opt/srcclr-3.4.3/jre/bin/java, -jar, /opt/srcclr-3.4.3/srcclr-3.4.3.jar, scan, --json, --skip-collectors="SOFile", [email protected]:strimzi/strimzi-kafka-oauth.git, --ref, master] in directory /tmp/sourceclear-invoker-8487097337571164151 with environment {GIT_BRANCH=null, GIT_URL=null, GOPATH=/tmp/sourceclear-gopath-218570056154285958}, output was 0 bytes:

	at org.zeroturnaround.exec.InvalidExitUtil.checkExit(InvalidExitUtil.java:27)
	at org.zeroturnaround.exec.WaitForProcess.call(WaitForProcess.java:114)
	at org.zeroturnaround.exec.ProcessExecutor.waitFor(ProcessExecutor.java:1097)
	at org.zeroturnaround.exec.ProcessExecutor.execute(ProcessExecutor.java:925)
	at com.redhat.engineering.srcclr.SrcClrInvoker.execSourceClear(SrcClrInvoker.java:138)
	... 10 more