techsneeze / dmarcts-report-viewer Goto Github PK

I am having a problem with running the viewer. I get the following error message:

Query failed: 'dmarc.report.mindate' isn't in GROUP BY (Error #1055)

This is a system running Debian 9, MariaDB 10.1, PHP Version 7.0.30-0+deb9u1.
Do you have an idea of what I am doing wrong? I think one issue could be that my MariaDB server has strict mode enabled, which entails ONLY_FULL_GROUP_BY (https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html#sqlmode_only_full_group_by).

In case a DMARC reports sender includes more detailled information regarding DKIM signatures (mostly the name of the selectors observed), dmarcts-report-viewer does not check whether any of those selectors was successful validated, but only checks the first one.

This issue causes successfully validated messages to show up as "DKIM failed", especially if the sender's setup involves dual-signing - for example, using a RSA and ED25519 DKIM key. The latter is smaller and faster to sign and validate, but not yet fully supported everywhere.

DMARC report XML snipped of a message displayed as "DKIM fail":

  <record>
    <row>
      <source_ip>x</source_ip>
      <count>1</count>
      <policy_evaluated>
        <disposition>none</disposition>
        <dkim>pass</dkim>
        <spf>pass</spf>
      </policy_evaluated>
    </row>
    <identifiers>
      <header_from>x</header_from>
    </identifiers>
    <auth_results>
      <dkim>
        <domain>x</domain>
        <result>fail</result>
        <selector/>
      </dkim>
      <dkim>
        <domain>x</domain>
        <result>pass</result>
        <selector>202003rsa</selector>
      </dkim>
      <spf>
        <domain>x</domain>
        <result>pass</result>
      </spf>
    </auth_results>
  </record>

In contrast, this record will be displayed as "DKIM pass", although it contains exactly the same information:

  <record>
    <row>
      <source_ip>x</source_ip>
      <count>1</count>
      <policy_evaluated>
        <disposition>none</disposition>
        <dkim>pass</dkim>
        <spf>pass</spf>
      </policy_evaluated>
    </row>
    <identifiers>
      <header_from>x</header_from>
    </identifiers>
    <auth_results>
      <dkim>
        <domain>x</domain>
        <result>pass</result>
        <selector>202003rsa</selector>
      </dkim>
      <dkim>
        <domain>x</domain>
        <result>fail</result>
        <selector/>
      </dkim>
      <spf>
        <domain>x</domain>
        <result>pass</result>
      </spf>
    </auth_results>
  </record>

Thanks for having a look at this - and thanks for providing dmarcts-report-viewer, it is most useful. :-)

The DMARC result filter filters on complete reports. I I get a report with thousand of entries I need to scroll to see which ones failed. It would be better to show only the failed ones, and not the ones that 'passed'. Currently it's filtering if a report completely or partially passed (mixed) but this doesn't make much sense. We don't care who reported it but we are mainly interested to see the failed entries.

Hello,
since PHP 8.1, long2ip() doesn't accept string argument anymore.

Line 151 in dmarcts-report-viewer.php

        if ( $row['ip'] ) {
            $ip = long2ip($row['ip']);

should become something like

        if ( $row['ip'] ) {
            $ip = long2ip(intval($row['ip']));

When i navigate on the dmarc report viewer, if I access a report with a large number of records, the page throws this php error :

Fatal error: Uncaught ValueError: DOMDocument::loadXML(): Argument #1 ($source) must not be empty in /var/www/viewer/dmarcts-report-viewer-report-data.php:213 Stack trace: #0 /var/www/viewer/dmarcts-report-viewer-report-data.php(213): DOMDocument->loadXML() #1 /var/www/viewer/dmarcts-report-viewer-report-data.php(59): formatXML() #2 /var/www/viewer/dmarcts-report-viewer-report-data.php(366): tmpl_reportData() #3 {main} thrown in /var/www/viewer/dmarcts-report-viewer-report-data.php on line 213

It would be useful to be able to choose to view all data for a specified year and not just a specified month as the current setup allows.

I'm trying to run dmarcts-report-viewer on my local host set up using apache2. I have moved the dmarcts report-parser folder along with all the other files into the /var/www/html directory. My issue is that when I direct the browser to 127.0.0.1/dmarcts-report-parser/dmarcts-report-viewer/dmarcts-report-viewer.php, it shows a blank white web page.

I'm running this on a Linux Ubuntu VM.

Using php7.1 as I encountered a few errors with php5 and ubuntu.

I know the directory is correctly linked to the php file as I removed the <?php tag and the same web page displayed all the code in the file, but it wont display the SQL database.

I have also checked my SQL database on my localhost and it is populated using dmarcts-report-parser.

Error Log from apache:

[Fri Sep 22 15:55:50.307842 2017] [php7:error] [pid 12389] [client 127.0.0.1:54908] PHP Fatal error: Uncaught Error: Class 'mysqli' not found in /var/www/html/dmarcts-report-parser/dmarcts-report-viewer/dmarcts-report-viewer.php:216\nStack trace:\n#0 {main}\n thrown in /var/www/html/dmarcts-report-parser/dmarcts-report-viewer/dmarcts-report-viewer.php on line 216, referer: http://127.0.0.1/dmarcts-report-parser/dmarcts-report-viewer/

Cheers

None of the data retrieved from the database has html-entity escaping applied before outputting it. Therefore a maliciously crafted DMARC report could trigger the execution of arbitrary javascript code when viewed in the report viewer (for example if org_name in the DMARC report would be '<script>alert('XSS')</script>').

The simplest (fewest changes required) fix is to apply the htmlspecialchars() function to all data before output. This could be done with an array_map() call in the appropriate places, eg: $row = array_map('htmlspecialchars', $row); immediately after every place where the $row variable is set within the tmpl_reportList and tmpl_reportData functions.

Just installed dmarcts-report-parser, and it seems like there is some classes that is missing:

[Mon Sep 19 11:22:47.678382 2022] [php7:error] [pid 64682] [client 172.31.20.135:58211] PHP Fatal error: Uncaught Error: Class 'DOMDocument' not found in /srv/dmarcts-report-viewer/dmarcts-report-viewer-report-data.php:210\nStack trace:\n#0 /srv/dmarcts-report-viewer/dmarcts-report-viewer-report-data.php(59): formatXML()\n#1 /srv/dmarcts-report-viewer/dmarcts-report-viewer-report-data.php(366): tmpl_reportData()\n#2 {main}\n thrown in /srv/dmarcts-report-viewer/dmarcts-report-viewer-report-data.php on line 210, referer: http://dmarc-webui.xxx.se/

FYI - I cloned down the GIT-repo, and not the release

When clicking on one of the report entries, the following 500 error occurs.

Fatal error: Uncaught Error: Class "DOMDocument" not found in /var/www/storage/dmarc/dmarcts-report-viewer-report-data.php:184 Stack trace: #0 /var/www/storage/dmarc/dmarcts-report-viewer-report-data.php(58): formatXML() #1 /var/www/storage/dmarc/dmarcts-report-viewer-report-data.php(321): tmpl_reportData() #2 {main} thrown in /var/www/storage/dmarc/dmarcts-report-viewer-report-data.php on line 184

With Firefox, when XML is displayed, it pushes the list beneath the xml viewer and as such it's not possible to select anything on the list.

I can provide a screenshot if needed.

Hi,

while setting up Rspamd with SPD/DKIM/DMARC on my own mail server recently, I today stumbled over this project. I have deployed it right a away and its awesome as it is simple to setup. Nice work!

Would you mind me to upload this package to Debian? If not, then I'd like to ask you for a release of the current code base (if it is releasable).

Thanks+Greets
Mike (aka sunweaver at debian.org)

I've been working on an enhancement to dmarcts-report-parser to add TLS reporting (see Issue 65). Since any addition to the parser would be useless without a corresponding addition to the viewer, I have started modifying the viewer to incorporate TLS reports.

As a proof-of-concept, I simply copied the dmarcts*.php files to tlsts*.php files and quickly modified them to display the TLS reports. I have managed to get most of the features working in TLS reports (JSON highlighting is eluding me at the moment).

What I have found is that the main files (dmarcts- and tlsts-report-viewer.php) and the *-list.php files are similar enough that I can merge them and deal with the differences in one set of files (say, viewer.php and viewer-list.php). I haven't yet made a decision about the *-data files; they are different enough to maybe warrant separate files (say, tls-viewer-data.php and dmarc-viewer-data.php). I will have to do some more investigation.

Because this feature introduces major new functionality, I think it warrants a version bump. @techsneeze, could you please create a v2.0 branch or something similar? Also, maybe label this issue "enhancement"?

The dmarcts-report-parser encodes the raw XML with 'encode_base64', but you don't encode it.
dmarcts-report-viewer-report-data.php line 209 should be:
$dom->loadXML(base64_decode($raw_xml));

I would like to recommend option to set the following:

• Sort order by "end date" - reason, when a report from 24th Oct to 27th Nov comes in on 28th Nov, you don't see it because the list is sorted by start date, and so only appears if you view data for October or change the display period to all

I achieved this myself by swapping the "order by" mindate/maxdate of the SQL

I am creating an options screen for dmarcts to replace the option values in dmarcts-report-viewer-config.php and dmarcts-report-viewer.js. The options would be stored in the database along with the DMARC reports, in a separate table. So far, I have the following options:

• General Settings
  • Host lookup: On/Off
  • Default sort column: dropdown list
  • Default sorting order: Asc/Desc
  • Unknown SPF/DKIM Alignments (treat unknown results as alignment failures or keep as "unknown"): On/Off
• Filters
  • Default DMARC Result: dropdown list
  • Show Only Report Data records that match DMARC Result: On/Off
  • Default Report Data Status: dropdown list
  • Default period: Current Month/All
  • Default domain: dropdown list
  • Default reporter: dropdown list
• Appearance
  • Default css file: dropdown list
  • Report List - Initial Height: number
• Developer
  • Debug messages: On/Off
  • Debug Window - Initial Height: number
  • Debug Window - Maximum Height: number

Some of the above options reference features not yet implemented but planned.

Any suggestions for more options?

Hi there,

the "Messages" column that displays numbers with thousands separators (e.g., 12,345) is not being sorted numerically as expected. Instead, it appears to be sorted as a string, where "10,000" comes before "2,000" due to the leading "1" in "10,000". This issue leads to a non-intuitive display of data where numerical values are not correctly ordered from smallest to largest (or vice versa).

Thousands separator seems useful, so maybe this should be handled in the javascript.

opening the page as of b5b8055 shows the following errors in console:

Uncaught TypeError: document.getElementById(...) is null
    showReportlist https://myhost/dmarcts-report-viewer/dmarcts-report-viewer.js:95
    onload https://myhost/dmarcts-report-viewer/dmarcts-report-viewer.php:1

I can also not see selDomain in the page source in the browser, which causes the exception.

disclaimer: this is a fresh install, i don't know if it's actually related to the recent changes

Hey!

I'd like to report this PHP error that appears in the logs:

NOTICE: PHP message: PHP Deprecated: Optional parameter $option declared before required parameter $var is implicitly treated as a required parameter in /var/www/viewer/dmarcts-report-viewer-options.php on line 216

2024/01/28 10:14:26 [error] 99#99: *39 FastCGI sent in stderr: "PHP message: PHP Deprecated: Optional parameter $option declared before required parameter $var is implicitly treated as a required parameter in /var/www/viewer/dmarcts-report-viewer-options.php on line 216" while reading response header from upstream, client: 172.18.0.1, server: _, request: "GET /dmarcts-report-viewer-options.php HTTP/1.1", upstream: "fastcgi://unix:/run/php-fpm.sock:", host: "127.0.0.1:9012", referrer: "https://xxxxxx/"

There's a SQL Injection possiblity because $_GET['report'] isn't validated to be numeric or escaped.

echo tmpl_page( ""
    .tmpl_reportList($allowed_reports)
    .tmpl_reportData( (isset($_GET["report"]) ? $_GET["report"] : false ), $allowed_reports )
);
if (!$reportnumber) {
    return "";
}
$sql = "SELECT * FROM rptrecord where serial = $reportnumber";

I'd suggest adding validation:

if(isset($_GET['report']) && is_numeric($_GET['report'])){
        $reportid=$_GET['report'];
}elseif(!isset($_GET['report'])){
        $reportid=false;
}else{
        die('Invalid Report ID');
}
// Generate Page with report list and report data (if a report is selected).
echo tmpl_page( ""
        .tmpl_reportList($allowed_reports)
        .tmpl_reportData( $reportid, $allowed_reports )
);

Patch attached.
dmarcts-report-viewer.txt

When a report isn't being stored in the DB (due to $maxsize_xml)

dmarcts-report-parser.pl: google.com: 3694144787112198549: Skipping storage of large XML (75036 bytes) as defined in config file.

I am unable to access it in the viewer, the following error is displayed

Fatal error: Uncaught ValueError: DOMDocument::loadXML(): Argument #1 ($source) must not be empty 
in /var/www/viewer/dmarcts-report-viewer-report-data.php:213
Stack trace:
  #0 /var/www/viewer/dmarcts-report-viewer-report-data.php(213): DOMDocument->loadXML()
  #1 /var/www/viewer/dmarcts-report-viewer-report-data.php(59): formatXML()
  #2 /var/www/viewer/dmarcts-report-viewer-report-data.php(366): tmpl_reportData()
  #3 {main} thrown in /var/www/viewer/dmarcts-report-viewer-report-data.php on line 213

I would like to propose a feature request:
Diagrams (Pie and line/bar charts) that show the overall statistics of the current view/timerange for:

• Line or bar chart with amount of registered reports on each day
• Pie chart that shows the total amount of passed, mixed and failed reports for DMARC, SPF and DKIM (Like the colored icons in each row but combined)

Would it be conceivable for you to include this to the viewer?

dmarcts-report-viewer.zip

Hi,

I found an error in dmarcts-report-viewer.php

The $html[] = ""; code on lines 249,271 and 294 should be above the closing bracket on the lines above.

When you have only one domain or one period the menu does not show correctly

Martin

I'm getting this error when trying to open the viewer.

Fatal error: Uncaught TypeError: setcookie(): Argument #3 ($expires_or_options) must be of type array|int, string given in /var/www/storage/dmarc/dmarcts-report-viewer.php:222 Stack trace: #0 /var/www/storage/dmarc/dmarcts-report-viewer.php(222): setcookie() #1 {main} thrown in /var/www/storage/dmarc/dmarcts-report-viewer.php on line 222

I'm on PHP 8.0 with FPM.

I succeeded to install your solution on my linux station.
I configured NGINX and PHP-FPM 7.2.
I imported my existing DMARC reports (about 1500) with the perl script from the other github project "dmarcts-report-parser".
This part is good.
I can view the reports in the web interface.
But when I want to click on one report to get detail I have this error:

2021/01/15 11:08:02 [error] 11445#11445: *26 FastCGI sent in stderr: "PHP message: PHP Fatal error: Uncaught Error: Class 'DOMDocument' not found in /var/www/html/dmarcts-report-viewer/dmarcts-report-viewer-report-data.php:144 Stack trace: #0 /var/www/html/dmarcts-report-viewer/dmarcts-report-viewer-report-data.php(54): formatXML('<?xml version="...') #1 /var/www/html/dmarcts-report-viewer/dmarcts-report-viewer-report-data.php(239): tmpl_reportData(1490, Array, 1) #2 {main} thrown in /var/www/html/dmarcts-report-viewer/dmarcts-report-viewer-report-data.php on line 144" while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /dmarcts-report-viewer/dmarcts-report-viewer-report-data.php?report=1490&hostlookup=1&p=2021-01 HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "localhost", referrer: "http://localhost/dmarcts-report-viewer/dmarcts-report-viewer.php"

Hi!

After upgrading, i could no longer view dmarc reports. I traced commits and found out it's probally caused by commit 4c10b11, after which the bug occurs. Looking over the code, I can't for the life of me figure out what is causing this. Below you'll find debug info and the sql error. It would be great if you clould take a look at this!

Debug info

D=
O=

DMARC=all

sql where = 

(I prettified this to increase legibility)
Data List sql: 

SELECT
  report.*,
  rcount,
  dkim_align_min,
  spf_align_min,
  dkim_result_min,
  spf_result_min,
  dmarc_result_min,
  dmarc_result_max
FROM
  report
  LEFT JOIN (
    SELECT
      SUM(rcount) AS rcount,
      serial,
      dkim_align,
      spf_align,
      dkimresult,
      spfresult,
      MIN(
        (
          CASE
            WHEN dkim_align = 'fail' THEN 0
            WHEN dkim_align = 'pass' THEN 2
            ELSE 1
          END
        )
      ) AS dkim_align_min,
      MIN(
        (
          CASE
            WHEN spf_align = 'fail' THEN 0
            WHEN spf_align = 'pass' THEN 2
            ELSE 1
          END
        )
      ) AS spf_align_min,
      MIN(
        (
          CASE
            WHEN dkimresult = 'fail' THEN 0
            WHEN dkimresult = 'pass' THEN 2
            ELSE 1
          END
        )
      ) AS dkim_result_min,
      MIN(
        (
          CASE
            WHEN spfresult = 'fail' THEN 0
            WHEN spfresult = 'pass' THEN 2
            ELSE 1
          END
        )
      ) AS spf_result_min,
      MIN(
        (
          CASE
            WHEN dkim_align = 'fail' THEN 0
            WHEN dkim_align = 'pass' THEN 1
            ELSE 3
          END
        ) + (
          CASE
            WHEN spf_align = 'fail' THEN 0
            WHEN spf_align = 'pass' THEN 1
            ELSE 3
          END
        )
      ) AS dmarc_result_min,
      MAX(
        (
          CASE
            WHEN dkim_align = 'fail' THEN 0
            WHEN dkim_align = 'pass' THEN 1
            ELSE 3
          END
        ) + (
          CASE
            WHEN spf_align = 'fail' THEN 0
            WHEN spf_align = 'pass' THEN 1
            ELSE 3
          END
        )
      ) AS dmarc_result_max
    FROM
      rptrecord
    GROUP BY
      serial
  ) AS rptrecord ON report.serial = rptrecord.serial
GROUP BY
  serial
ORDER BY
  maxdate DESC

SQL error:

Query failed: Expression #3 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'dmarc.rptrecord.dkim_align' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by (Error #1055)

$ mysql -V
mysql Ver 14.14 Distrib 5.1.73, for redhat-linux-gnu (x86_64) using readline 5.1
$ php-fpm -v
PHP 5.3.3 (fpm-fcgi) (built: Mar 22 2017 12:28:05)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies

Nothing is showed by the app except default html.
Tried to run from cli, got this error:

php dmarcts-report-viewer.php > /dev/null
PHP Notice: Undefined offset: 1 in /usr/share/nginx/html/dmarcts-report-viewer/dmarcts-report-viewer.php on line 76
PHP Warning: Invalid argument supplied for foreach() in /usr/share/nginx/html/dmarcts-report-viewer/dmarcts-report-viewer.php on line 76

Am I missing something?

commit a5ead0b breaks the Raw Report XML.

Rather than showing the actual XML it shows it HTML escaped, so <pre><code class='xml'>&lt;?xml version=&quot;1.0&quot; and not <?xml version="1.0"...

The problem is in dmarcts-report-viewer-report-data.php

$row['raw_xml'] = formatXML($row['raw_xml'], $reportnumber);
$row = array_map('html_escape', $row);

which should probably be

$row['raw_xml'] = formatXML($row['raw_xml'], $reportnumber);
foreach ($row as $k => $v) {
	if ($k !== 'raw_xml') {
		$row[$k] = html_escape($v);
	}
}

Hello @techsneeze

perhaps it would be good to add a screenshot to the description page? :)

Have a nice day!