@carllp Take a look in your test-results.xml file (copy it to artifacts so you can download it), it likely has 2 <test-run> nodes which is invalid xml format, and Nunit only allows one per file

I've opened an issue for it as the script itself is written to do this specifically for each site that is scanned, in my case it was http vs https

from csedevops.

This doesn't look like a problem related to the handlebars template. The error is related to the scanner not being able to publish the result files. Normally this happens on a self-hosted agent that has a version of docker installed via the SNAP package manager. Are you by any chance running a similar self-hosted agent @carllp ?

from csedevops.

@doymturner The agent the task is being run on is the Ubuntu 18.04 Microsoft-hosted agent.

from csedevops.

Can you share your full pipeline (cleansed)? At least all of the bits related to zap scanner?

from csedevops.

tasklog_12.log

Sure. I believe it's been sufficiently cleansed of all unique identifiers.

As a side note, if you could point me to some documentation (via DM or here) to speed up the cleansing of pipeline logs it would be much appreciated.

from csedevops.

Took a second look at the original error you posted. Did you make any changes to the handlebar template when adding it to your pipeline? Could you share the bits of your pipeline YAML related to the extension and report publishing (including the template)?

FYI: We are working on additional reporting options for the extension, but plan to leave the options for custom reporting such as with the handlebars template so I'm keen to figure out the issue.

from csedevops.

Sorry for the late reply @doymturner.

Fixed quoted text below by including only the bash scripting code and not displayName and condition lines. Please ignore.

We did edit the bash script convert report.json to test-results.xml slightly as it was throwing errors (see log below). I assume that this is the culprit as to why the nunit test report doesn't work. The two removed lines work fine in the handlebars template script so we're not sure why they throw errors in this step.

We removed the following two lines so that the conversion would run:
displayName: 'owasp nunit template'
condition: always()

The error report if the lines are included:

=========================================================
Generating script.
========================== Starting Command Output ====
/home/vsts/work/_temp/: line 2: displayName:: command not found
/home/vsts/work/_temp/: line 3: syntax error near unexpected token (' /home/vsts/work/_temp/: line 3: condition: always()'

##[error]Bash exited with code '2'.
##[section]Finishing: Bash Script - Convert report.json to test-results.xml

Here's how it looks in our pipeline. If you know what's syntactically wrong, please let me know.

Appreciate your help with this.

from csedevops.

As requested, 1st snippet is the handlebars YAML and the 2nd is the report conversion. 3rd is the Publish Test Results YAML.

steps:

• bash: |
  sudo npm install -g handlebars-cmd

  cat <<EOF > owaspzap/nunit-template.hbs
  {{#each site}}
  
  <test-run
      id="2"
      name="Owasp test"
      start-time="{{../[@generated]}}"  >
      <test-suite
          id="{{@index}}"
          type="Assembly"
          name="{{[@name]}}"
          result="Failed"
          failed="{{alerts.length}}">
          <attachments>
              <attachment>
                  <filePath>owaspzap/report.html</filePath>
              </attachment>
          </attachments>
      {{#each alerts}}<test-case
          id="{{@index}}"
          name="{{alert}}"
          result="Failed"
          fullname="{{alert}}"
          time="1">
              <failure>
                  <message>
                      <![CDATA[{{{desc}}}]]>
                  </message>
                  <stack-trace>
                      <![CDATA[
  Solution:
  {{{solution}}}
  
  Reference:
  {{{reference}}}
  
  instances:{{#each instances}}
  * {{uri}}
      - {{method}}
      {{#if evidence}}- {{{evidence}}}{{/if}}
                      {{/each}}]]>
                  </stack-trace>
              </failure>
      </test-case>
      {{/each}}
      </test-suite>
  </test-run>
  {{/each}}
  EOF
  

  displayName: 'Bash Script - Install handlebars and generate report template'

steps:

• bash: |
  handlebars owaspzap/report.json < owaspzap/nunit-template.hbs > owaspzap/test-results.xml

  displayName: 'Bash Script - Convert report.json to test-results.xml'

steps:

• task: PublishTestResults@2
  displayName: 'Publish Test Results owaspzap/test-results.xml'
  inputs:
  testResultsFormat: NUnit
  testResultsFiles: 'owaspzap/test-results.xml'
  condition: succeededOrFailed()

from csedevops.

Error in the publish test results task is slightly different now.

It now shows:
##[warning]Failed to read /home/vsts/work/r1/a/owaspzap/test-results.xml. Error : Data at the root level is invalid. Line 284, position 4..

Instead of
/home/vsts/work/r1/a/owaspzap/test-results.xml. Error : Data at the root level is invalid. Line 22, position 4..

Full log:

` ##[section]Starting: Publish Test Results test-results.xml

Task : Publish Test Results
Description : Publish test results to Azure Pipelines
Version : 2.161.1
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/publish-test-results

[command]/usr/bin/dotnet --version
3.0.101
##[warning]Failed to read /home/vsts/work/r1/a/owaspzap/test-results.xml. Error : Data at the root level is invalid. Line 284, position 4..
##[warning]Failed to publish test results: Object reference not set to an instance of an object.
##[section]Async Command Start: Publish test results
##[section]Async Command End: Publish test results
##[section]Finishing: Publish Test Results test-results.xml
`

from csedevops.

I have the same issue as carllp and I have solved it by creating whole task in yaml instead of DevOps separate components.

from csedevops.

In my case, this was resolved by moving my final EOF over two spaces. In the spacing adjustments in the YAML file it became offset thus causing a problem as shown here. Once it was aligned, the warning didn't show up anymore. To be clear - the task was succeeding but a warning was being output on the PublishTestResults task.

from csedevops.

@carllp Take a look in your test-results.xml file (copy it to artifacts so you can download it), it likely has 2 <test-run> nodes which is invalid xml format, and Nunit only allows one per file

I've opened an issue for it as the script itself is written to do this specifically for each site that is scanned, in my case it was http vs https

So how did you resolve the issue? NM. Thanks!

from csedevops.

The sample shows the following

sudo npm install -g handlebars-cmd

  cat < owaspzap/nunit-template.hbs

... snip

  EOF

When running in a bash task this causes failure. Fix by moving everything from cat to EoF two spaces to the left.

from csedevops.

hi
I am getting same issue now.
I have tried all the options but nothing works (yaml,azure devops tasks) , removing EOF and other stuff..
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
##[warning]Failed to read /home/vsts/work/1/s/owaspzap/test-results.xml. Error : Data at the root level is invalid. Line 59, position 4..
Async Command Start: Publish test results
Async Command End: Publish test results
Finishing: Publish Test Results owaspzap/test-results.xml

I am not able to see test tab eventhough build is successful

from csedevops.

trigger:

• master

pool:
vmImage: 'ubuntu-latest'

steps:

• task: owaspzap@1
  inputs:
  threshold: '150'
  scantype: 'targetedScan'
  url: 'example.com'
  port: '443'

• task: CopyFiles@2
  inputs:
  SourceFolder: 'owaspzap/'
  Contents: '**'
  TargetFolder: '$(Build.ArtifactStagingDirectory)'

• task: PublishBuildArtifacts@1
  condition: always()
  inputs:
  ArtifactName: 'owasp_zap_reports'

• bash: |
  sudo npm install -g handlebars-cmd

  cat < owaspzap/nunit-template.hbs

  
  {{#each site}}
  
  
  $BUILD_SOURCESDIRECTORY/owaspzap/report.html
  
  
  {{#each alerts}}
  
  
  
  
  
  <![CDATA[
  Solution:
  {{{solution}}}

  Reference:
  {{{reference}}}

  instances:{{#each instances}}

  • {{uri}}
    • {{method}}
      {{#if evidence}}- {{{evidence}}}{{/if}}
      {{/each}}]]>
      
      
      
      {{/each}}
    {{/each}}
  EOF

displayName: 'owasp nunit template'
condition: always()

• bash: ' handlebars owaspzap/report.json < owaspzap/nunit-template.hbs > owaspzap/test-results.xml'
  displayName: 'generate nunit type file'
  condition: always()
• task: PublishTestResults@2
  inputs:
  testResultsFormat: 'Nunit'
  testResultsFiles: 'owaspzap/test-results.xml'

from csedevops.

Can anyone help me

from csedevops.

I've found solution for this, I'll just paste it from marketplace reviews.

Sithu Kyaw
9/23/2020
I tried it just now and it works fine except not able to attach "owaspzap/report.html" for Publish Test Result task.

After a couple of troubleshooting, I found Handlebars script needs minor adjustment by adding $(System.DefaultWorkingDirectory)/ in front of 'owasp/report.html' inside . Good work!

<attachment>
       <filePath>$(System.DefaultWorkingDirectory)/owaspzap/report.html</filePath>
</attachment>

from csedevops.

In Step 1 - Remove the EOF statement at the end. Keep only the below code. It will work fine

sudo npm install -g handlebars-cmd

 cat <<EOF > $(System.DefaultWorkingDirectory)/owaspzap/nunit-template.hbs
 {{#each site}}

 <test-run
     id="2"
     name="Owasp test"
     start-time="{{../[@generated]}}"  >
     <test-suite
         id="{{@index}}"
         type="Assembly"
         name="{{[@name]}}"
         result="Failed"
         failed="{{alerts.length}}">
         <attachments>
             <attachment>

$(System.DefaultWorkingDirectory)/owaspzap/report.html


{{#each alerts}}





<![CDATA[
Solution:
{{{solution}}}

 Reference:
 {{{reference}}}

 instances:{{#each instances}}
 * {{uri}}
     - {{method}}
     {{#if evidence}}- {{{evidence}}}{{/if}}
                     {{/each}}]]>
                 </stack-trace>
             </failure>
     </test-case>
     {{/each}}
     </test-suite>
 </test-run>
 {{/each}}

from csedevops.

I have had the same problem, tried all different possibilites. But always get error file not found..

##[warning]File at 
                  $(Build.SourcesDirectory)/OWASP-ZAP-Report.xml
                 was not found
Failed to upload run logs to LogStore, Trying to upload to FileService
##[warning]Skipping attachment as it is not available on disk: 
                  $(Build.SourcesDirectory)/OWASP-ZAP-Report.xml

also by injecting env variable to xslt

##[warning]File at /home/vsts/work/1/s/OWASP-ZAP-Report.xml
                 was not found
Failed to upload run logs to LogStore, Trying to upload to FileService
##[warning]Skipping attachment as it is not available on disk: /home/vsts/work/1/s/OWASP-ZAP-Report.xml

also added echo to confirm the location of file

echo `pwd`/`ls OWASP-ZAP-Report.xml`

/home/vsts/work/1/s/OWASP-ZAP-Report.xml

from csedevops.

In my case the pipeline runs successfully. However the Passed test results are not shown in the test report. I only see the failed or others. This is the template I am using.

stages:

• stage: OWASP_ZAP_Stage # !CHANGE! per env
  jobs:

  • job : OWASP_ZAP
    variables:
    websiteurl: 'https://qatest.gdc.net/status'
    pool:
    vmImage: 'ubuntu-latest'
    steps:

    • task: Bash@3
      displayName: 'Prepare OWASPZAP output directory for Docker'
      inputs:
      targetType: 'inline'
      script: |
      sudo mkdir $(System.DefaultWorkingDirectory)/owaspzap
      sudo chmod -R 777 $(System.DefaultWorkingDirectory)/owaspzap

    • task: Bash@3
      displayName: 'Run OWASPZAP Docker Image and scan $(websiteurl)'
      inputs:
      targetType: 'inline'
      script: |
      docker run -v $(System.DefaultWorkingDirectory)/owaspzap:/zap/wrk:rw -t owasp/zap2docker-stable zap-baseline.py -t $(websiteurl) -J report.json -i --autooff

    • bash: |
      sudo npm install -g handlebars-cmd

      cat < owaspzap/nunit-template.hbs

      
      {{#each site}}
      
      
      $(System.DefaultWorkingDirectory)/owaspzap/report.html
      
      
      {{#each alerts}}
      
      
      
      
      
      <![CDATA[
      Solution:
      {{{solution}}}

      Reference:
      {{{reference}}}

      instances:{{#each instances}}

      • {{uri}}
        • {{method}}
          {{#if evidence}}- {{{evidence}}}{{/if}}
          {{/each}}]]>
          
          
          
          {{/each}}
        {{/each}}
      displayName: 'OWASP NUnit template'

    • bash: ' handlebars owaspzap/report.json < owaspzap/nunit-template.hbs > owaspzap/test-results.xml'
      displayName: 'Generate NUnit type file for $(websiteurl)'

    Publish results to AZDO Test page

    • task: PublishTestResults@2
      displayName: 'Publish Test Results to AZDO'
      inputs:
      testResultsFormat: NUnit
      testResultsFiles: 'owaspzap/test-results.xml'
      testRunTitle: '$(websiteurl)'
      failTaskOnFailedTests: false
      condition: always()

from csedevops.

Hello everyone! I'm don't use this extension for ZAP testing, but got the same issue - unable to upload test results. So my pipeline yaml looks like this:

trigger: none

pr: none

pool:
  vmImage: ubuntu-latest

steps:
- script: docker run --rm --user root -v $(System.DefaultWorkingDirectory):/zap/wrk/:rw -i owasp/zap2docker-stable zap-full-scan.py -d -j -I -t https://blah-blah -J owasp-zap/zap-report.json
  displayName: 'Run dynamic testing'

- bash: |
    sudo npm install -g handlebars-cmd
    handlebars owasp-zap/zap-report.json < owasp-zap/nunit-template.hbs > owasp-zap/test-results.xml
    python3 owasp-zap/splitter.py test-results.xml
  displayName: 'Generate test result files'

- task: PublishTestResults@2
  displayName: 'Publish rest results'
  inputs:
    testResultsFormat: NUnit
    testResultsFiles: 'owasp-zap/test-results-*.xml'
    mergeTestResults: true

So, at the root of my repo i had directory owasp-zap that contains file nunit-template.hbs:

{{#each site}}

<test-run
    id="2"
    name="Owasp test"
    start-time="{{../[@generated]}}"  >
    <test-suite
        id="{{@index}}"
        type="Assembly"
        name="{{[@name]}}"
        result="Failed"
        failed="{{alerts.length}}">
    {{#each alerts}}<test-case
        id="{{@index}}"
        name="{{alert}}"
        result="Failed"
        fullname="{{alert}}"
        time="1">
            <failure>
                <message>
                    <![CDATA[{{{desc}}}]]>
                </message>
                <stack-trace>
                    <![CDATA[
Solution:
{{{solution}}}

Reference:
{{{reference}}}

instances:{{#each instances}}
* {{uri}}
    - {{method}}
    {{#if evidence}}- {{{evidence}}}{{/if}}
                    {{/each}}]]>
                </stack-trace>
            </failure>
    </test-case>
    {{/each}}
    </test-suite>
</test-run>
{{/each}}

And file splitter.py:

import xml.dom.minidom as minidom
import argparse


def parse_file():
    parser = argparse.ArgumentParser(description='File path parser')
    parser.add_argument('-f', '--file', type=str, help='A required string - path to report file.')
    args = parser.parse_args()
    return args.file


def get_xml(document: str) -> list:
    with open(document, 'r') as file:
        return [line for line in file]


def to_file(line: str, file_name: str):
    with open(file_name, 'a') as f:
        f.write(line)


def extract_separate_documents(lines: list):
    count = 0

    for line in lines:
        line: str
        if '<test-run' in line:
            count += 1
            print('report:', count)
        if count > 0:
            to_file(line, f'test-results-{count}.xml')


document = parse_file()

if not document:
    print("Exception! Report file not specified.")
    exit(1)

lines = get_xml(document=document)
extract_separate_documents(lines)

My ZAP testing run generates json file, that should be procced by handlebars tool and I got test-results.xml in owasp-zap dir. In my case test-results.xml file contains two root block that doesn't correspond to NUnit type. Than I wrote a Python script that splits this xml result file on more files named test-results-{count}.xml based on tag. And in PublishTestResults task i provided those files as an input and set mergeTestResults parameter to true. This is working fine in my case and may be suitable for yours.

from csedevops.