Giter Club home page Giter Club logo

mail-dmarc's Introduction

Status Badges

Build Status

Coverage Status

NAME

Mail::DMARC - Perl implementation of DMARC

VERSION

version 1.20240314

SYNOPSIS

DMARC: Domain-based Message Authentication, Reporting and Conformance

my $dmarc = Mail::DMARC::PurePerl->new(
  ... # see the documentation for the "new" method for required args
);

my $result = $dmarc->validate();

if ( $result->result eq 'pass' ) {
   ...continue normal processing...
   return;
};

# any result that did not pass is a fail. Now for disposition

if ( $result->evalated->disposition eq 'reject' ) {
   ...treat the sender to a 550 ...
};
if ( $result->evalated->disposition eq 'quarantine' ) {
   ...assign a bunch of spam points...
};
if ( $result->evalated->disposition eq 'none' ) {
   ...continue normal processing...
};

DESCRIPTION

This module is a suite of tools for implementing DMARC. It adheres to the 2013 DMARC draft, intending to implement every MUST and every SHOULD.

This module can be used by...

  • MTAs and filtering tools like SpamAssassin to validate that incoming messages are aligned with the purported sender's policy.
  • email senders, to receive DMARC reports from other mail servers and display them via CLI and web interfaces.
  • MTA operators to send DMARC reports to DMARC author domains.

When a message arrives via SMTP, the MTA or filtering application can pass in a small amount of metadata about the connection (envelope details, SPF and DKIM results) to Mail::DMARC. When the validate method is called, Mail::DMARC will determine if:

a. the header_from domain exists
b. the header_from domain publishes a DMARC policy
c. if a policy is published...
d. does the message conform to the published policy?
e. did the policy request reporting? If so, save details.

The validation results are returned as a Mail::DMARC::Result object. If the author domain requested a report, it was saved to the Report Store. The Store class includes a SQL implementation that is tested with SQLite, MySQL and PostgreSQL.

There is more information available in the $result object. See Mail::DMARC::Result for complete details.

Reports are viewed with the dmarc_view_reports program or with a web browser and the dmarc_httpd program.

Aggregate reports are sent to their requestors with the dmarc_send_reports program.

For aggregate reports that you have been sent, the dmarc_receive program will parse the email messages (from IMAP, Mbox, or files) and save the report results into the Report Store.

The report store can use the same database to store reports you have received as well as reports you will send. There are several ways to identify the difference, including:

  • received reports will have a null value for report_policy_published.rua
  • outgoing reports will have null values for report.uuid and report_record.count

CLASSES

Mail::DMARC - the perl interface for DMARC

Mail::DMARC::Policy - a DMARC policy

Mail::DMARC::PurePerl - Pure Perl implementation of DMARC

Mail::DMARC::Result - the results of applying policy

Mail::DMARC::Report - Reporting: the R in DMARC

Mail::DMARC::Report::Send - send reports via SMTP & HTTP

Mail::DMARC::Report::Receive - receive and store reports from email, HTTP

Mail::DMARC::Report::Store - a persistent data store for aggregate reports

Mail::DMARC::Report::View - CLI and CGI methods for viewing reports

Mail::DMARC::libopendmarc - an XS implementation using libopendmarc

METHODS

new

Create a DMARC object.

my $dmarc = Mail::DMARC::PurePerl->new;

Populate it.

$dmarc->source_ip('192.0.1.1');
$dmarc->envelope_to('recipient.example.com');
$dmarc->envelope_from('sender.example.com');
$dmarc->header_from('sender.example.com');
$dmarc->dkim( $dkim_verifier );
$dmarc->spf([
    {   domain => 'example.com',
        scope  => 'mfrom',
        result => 'pass',
    },
    {
        scope  => 'helo',
        domain => 'mta.example.com',
        result => 'fail',
    },
]);

Run the request:

my $result = $dmarc->validate();

Alternatively, pass in all the required parameters in one shot:

my $dmarc = Mail::DMARC::PurePerl->new(
        source_ip     => '192.0.1.1',
        envelope_to   => 'example.com',
        envelope_from => 'cars4you.info',
        header_from   => 'yahoo.com',
        dkim          => $dkim_results,  # same format
        spf           => $spf_results,   # as previous example
        );
my $result = $dmarc->validate();

source_ip

The remote IP that attempted sending the message. DMARC only uses this data for reporting to domains that request DMARC reports.

envelope_to

The domain portion of the RFC5321.RcptTo, (aka, the envelope recipient), and the bold portion in the following example:

RCPT TO:<user@example.com>

envelope_from

The domain portion of the RFC5321.MailFrom, (aka, the envelope sender). That is the the bold portion in the following example:

MAIL FROM:<user@example.com>

header_from

The domain portion of the RFC5322.From, aka, the From message header.

From: Ultimate Vacation <sweepstakes@example.com>

You can instead pass in the entire From: header with header_from_raw.

header_from_raw

Retrieve the header_from domain by parsing it from a raw From field/header. The domain portion is extracted by get_dom_from_header, which is fast, generally effective, but also rather crude. It has limits, so read the description.

dkim

If Mail::DKIM::Verifier was used to validate the message, just pass in the Mail::DKIM::Verifier object that processed the message:

$dmarc->dkim( $dkim_verifier );

Otherwise, pass in an array reference. Each member of the DKIM array results represents a DKIM signature in the message and consists of the 4 keys shown in this example:

$dmarc->dkim( [
        {
            domain      => 'example.com',
            selector    => 'apr2013',
            result      => 'fail',
            human_result=> 'fail (body has been altered)',
        },
        {
            # 2nd signature, if present
        },
    ] );

The dkim results can also be build iteratively by passing in key value pairs or hash references for each signature in the message:

$dmarc->dkim( domain => 'sig1.com', result => 'fail' );
$dmarc->dkim( domain => 'sig2.com', result => 'pass' );
$dmarc->dkim( { domain => 'example.com', result => 'neutral' } );

Each hash or hashref is appended to the dkim array.

Finally, you can pass a coderef which won't be called until the dkim method is used to read the dkim results. It must return an array reference as described above.

The dkim result is an array reference.

domain

The d= parameter in the DKIM signature

selector

The s= parameter in the DKIM signature

result

The validation results of this signature. One of: none, pass, fail, policy, neutral, temperror, or permerror

human result

Additional information about the DKIM result. This is comparable to Mail::DKIM::Verifier->result_detail.

spf

The spf method works exactly the same as dkim. It accepts named arguments, a hashref, an arrayref, or a coderef:

$dmarc->spf(
    domain => 'example.com',
    scope  => 'mfrom',
    result => 'pass',
);

The SPF domain and result are required for DMARC validation and the scope is used for reporting.

domain

The SPF checked domain

scope

The scope of the checked domain: mfrom, helo

result

The SPF result code: none, neutral, pass, fail, softfail, temperror, or permerror.

DESIGN & GOALS

Correct

The DMARC spec is lengthy and evolving, making correctness a moving target. In cases where correctness is ambiguous, options are generally provided.

Easy to use

Providing an implementation of DMARC that SMTP utilities can utilize will aid DMARC adoption.

The list of dependencies appears long because of reporting. If this module is used without reporting, the number of dependencies not included with perl is about 5.

Maintainable

Since DMARC is evolving, this implementation aims to be straight forward and easy to alter and extend. The programming style is primarily OO, which carries a small performance penalty but dividends in maintainability.

When multiple options are available, such as when sending reports via SMTP or HTTP, calls should be made to the parent Send class to broker the request. When storing reports, calls are made to the Store class which dispatches to the SQL class. The idea is that if someone desired a data store other than those provided by perl's DBI class, they could easily implement their own. If you do, please fork it on GitHub and share.

Fast

If you deploy this in an environment where performance is insufficient, please profile the app and submit a report and preferably, patches.

SEE ALSO

Mail::DMARC on GitHub

2015-03 RFC 7489

DMARC Best Current Practices

HISTORY

The daddy of this perl module was a DMARC module for the qpsmtpd MTA.

AUTHORS

CONTRIBUTORS

COPYRIGHT AND LICENSE

This software is copyright (c) 2024 by Matt Simerson.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

mail-dmarc's People

Contributors

bigio avatar cgf1 avatar dlucredativ avatar hopper262 avatar jeanpaulgalea avatar m50 avatar manwar avatar marcbradshaw avatar msimerson avatar priyadi avatar rjbs avatar waffle-iron avatar xpunkt avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mail-dmarc's Issues

Prevent aggregated report duplication

Hi,

If I process again a aggregated report, yet imported, it ll be duplicated.
In 'report' table report stay unique, but in 'report_record' table information is duplicated.

Is it possible to prevent duplicate integration?

Regards,

dmarc_update_public_suffix_list ignores file location specified in mail-dmarc.ini

reading

https://metacpan.org/pod/distribution/Mail-DMARC/bin/dmarc_update_public_suffix_list

To download a new Public Suffix List to the location specified my mail-dmarc.ini

In my config

cat /usr/local/etc/auth-milter/mail-dmarc.ini
	...
	[dns]
	timeout            = 5
	public_suffix_list = /usr/local/etc/auth-milter/public_suffix_list
	...

When I exec

dmarc_update_public_suffix_list --config-file=/usr/local/etc/auth-milter/mail-dmarc.ini

it's not creating/updating the .ini-file-specified public_suffix_list

instead, it returns

Public Suffix List file /usr/lib/perl5/site_perl/5.26.1/auto/share/dist/Mail-DMARC/public_suffix_list not modified

checking

ls -al \
 /usr/lib/perl5/site_perl/5.26.1/auto/share/dist/Mail-DMARC/public_suffix_list \
 /usr/local/etc/auth-milter/public_suffix_list
	ls: cannot access '/usr/local/etc/auth-milter/public_suffix_list': No such file or directory
	-r--r--r-- 1 root root 204K Oct 10 08:33 /usr/lib/perl5/site_perl/5.26.1/auto/share/dist/Mail-DMARC/public_suffix_list

Percentage stuff not working (fix inside)

I found a problem in the percentage algorithm in file lib/Mail/DMARC/PurePerl.pm

The comparison in line 68 is wrong. It should be: int( rand(100) ) < $policy->pct
(Think of a pct=100 which should give the same behavior as omitted pct)

Furthermore the action in this conditional is not sufficient - we need to set the disposision before return:
$self->result->disposition($effective_p);

Thank you for this nice library :)

Tests started to fail

I see the following test failure on my smoker systems:

#   Failed test 'verify_external_reporting, mail-dmarc.tnpi.net, theartfarm.com'
#   at t/04.PurePerl.t line 116.
#          got: '1'
#     expected: undef

#   Failed test 'verify_external_reporting, override rua'
#   at t/04.PurePerl.t line 124.
#     Structures begin differing at:
#          $got->{uri} = 'mailto:[email protected]'
#     $expected->{uri} = 'mailto:[email protected]'

#   Failed test 'save aggregate'
#   at t/04.PurePerl.t line 94.
# Looks like you failed 3 tests of 111.
t/04.PurePerl.t .............................. 
Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/111 subtests 

parameters for "one-shot" passing to new do not match method arguments

I tried to get a Mail::DMARC::PurePerl via new, but hit a couple snags.

  1. You can't pass a Mail::DKIM::Verifier as the dkim argument in new, only via the dkim setter. If you do, you get "dkim needs to be an array reference!"
  2. You can't pass a hashref as the spf argument in new, because it wants an arrayref. If you pass the documented pairs as an arrayref, you get "Can't use string ("domain") as a HASH ref while "strict refs"". Maybe it wants an arrayref of hashrefs, but I gave up at that point.

The new method and the individual setters, for these two at least, have very different behaviors. Please either make them the same, or document each.

Thanks!

dist:zilla

This question is particularly directed at @marcbradshaw . How do you feel about the DZ? This is one of the two modules I used it in and I've just entirely removed it from the other. My primary reasons for jettisoning it are:

  1. having to install half of CPAN on each machine I work on
  2. having the code in the github repo be different than the published module
  3. it raises the bar for contributors

Seriously, I've composed and filed this issue while waiting for authordeps to install on my laptop. Yeah, sure, they were installed once but then OS updates, or perl versions, etc. and then it has to happen all over again. Slowly. Because perl, and Moose.

Apologies

@msimerson just an FYI and an apology. I accidentally pushed some of my local merge commits up to master, and reverted the push shortly after. Nothing new went up, but there were some ugly merges in the commit log for a while.

Change of behaviour?

I have just merged the latest master into my branch and notice a change in behaviour.

Previously, a DKIM pass and SPF softfail resulted in an overall DMARC pass, this has changed to a DMARC fail.

Before

Authentication-Results: test.module;
    dkim-adsp=pass (ADSP policy from marcbradshaw.net);
    dkim=pass (1024-bit rsa key) header.d=marcbradshaw.net [email protected] header.b=DU/QdEi3;
    dkim=pass (2048-bit rsa key) header.d=1e100.net [email protected] header.b=aIL04w54;
    dmarc=pass header.from=marcbradshaw.net;
    iprev=fail policy.iprev=123.123.123.123 (NOT FOUND);
    spf=softfail [email protected] smtp.helo=bad.name.google.com;
    x-ptr=fail x-ptr-helo=bad.name.google.com x-ptr-lookup=
Received-SPF: softfail (marcbradshaw.net: Sender is not authorized by default to use '[email protected]' in 'mfrom' identity, however domain is not currently prep    ared for false failures (mechanism '~all' matched)) receiver=test.module; identity=mailfrom; envelope-from="[email protected]"; helo=bad.name.google.com; client-i    p=123.123.123.123

After

Authentication-Results: test.module;
    dkim-adsp=pass (ADSP policy from marcbradshaw.net);
    dkim=pass (1024-bit rsa key) header.d=marcbradshaw.net [email protected] header.b=DU/QdEi3;
    dkim=pass (2048-bit rsa key) header.d=1e100.net [email protected] header.b=aIL04w54;
    dmarc=fail (p=none) header.from=marcbradshaw.net;
    iprev=fail policy.iprev=123.123.123.123 (NOT FOUND);
    spf=softfail [email protected] smtp.helo=bad.name.google.com;
    x-ptr=fail x-ptr-helo=bad.name.google.com x-ptr-lookup=
Received-SPF: softfail (marcbradshaw.net: Sender is not authorized by default to use '[email protected]' in 'mfrom' identity, however domain is not currently prep    ared for false failures (mechanism '~all' matched)) receiver=test.module; identity=mailfrom; envelope-from="[email protected]"; helo=bad.name.google.com; client-i    p=123.123.123.123

Parse error in Policy

Odd number of elements in anonymous hash at /usr/share/perl5/Mail/DMARC/Policy.pm line 31.

No context as yet, putting this here as a reminder to investigate.

save_aggregate in DMARC.pm fails when provided with DKIM verifier object.

I have run into a problem when I provide the dmarc object with the DKIM verifier object in line 259 in DMARC.pm. $self->dkim does not exist. If I provide the dmarc object with and explicit array of dkim results however there is no problem.

...
my $dmarc = Mail::DMARC::PurePerl->new();
my $dkim = Mail::DKIM::Verifier->new();

$dmarc->source_ip('127.0.0.1');
$dmarc->envelope_to('yahoo.ca');
$dmarc->envelope_from('yahoo.ca');
$dmarc->header_from('yahoo.ca');

my $fh;
open($fh, "<:raw", $filename)
or return("Can't open file $filename");

$dkim->load($fh);
$dkim->CLOSE;
$dmarc->dkim($dkim); <--- this does not work, while $dmarc->dkim(domain => 'yahoo.ca', result => 'fail'); this does.

...
my $dmarcResult = $dmarc->validate(); <--- croaks if I use dkim verifier directly.

Invalid DKIM results cause SQL error

When DKIM returns an invalid result, for example "invalid (public key: DNS error: query timed out)" and is then passed to DMARC an error inserting into the database is generated.

This also happens if a message has multiple signatures and one or more of them is invalid.

Should we be inserting the invalid record, or simply ignoring it?

query called by Mail::DMARC::Report::Store::SQL, 527#012#011INSERT INTO report_record_dkim#012 (report_record_id, domain_id,selector,result,human_result)#012VALUES (??)#12#012#0111376426, 113974, dkim, invalid, invalid (public key: DNS error: query timed out) at /usr/share/perl5/Mail/DMARC/Report.pm line 77

Sending reports in TLS fails for certain domains

Hi,

I found the following issues when sending reports to certain domains.

When sending to the MX for rua.agari.com, specifically etl.cp.prod.agari.com, connect_smtp_tls returns a valid smtp object making one think all is good. There is a bug however as when invoking a starttls, you get a failure. While the server supports TLS, the handshake fails. One needs to examine the status code of the starttls before using TLS.

Secondly, in email STARTTLS is only really used for encryption purposes and the certificate is not necessarily verified. Self signed certificates do exist (see agari example above). So the patch includes SSL_VERIFY_NONE when constructing and SMTPS object and also checking the return value.

Third, some servers require a HELO after a STARTTLS (), e.g. mx1.sophos.com. That is not happening, so I'm issuing a hello after starttls. A second hello does not hurt.

Finally, the specified timeout kicks in a lot on a lot of live servers. Have increased to 30 seconds.

See the attached patch:

starttls.patch.txt

create Aggregate reports on hour-aligned or day-aligned slices

The aggregate reports should be generated on a stable timeframe, somewhat irrespective of exactly when the generation job itself is run.
The consequence of not doing this is that the reported window of time slides, depending on how long it takes any prior jobs to run. Here is a view of that occurring in reports I received for 2015-03-09:
(I've anonymized the real provider into 'somebody.com')

 DMARC provider |  data start time    |   data end time     |  received by us at
----------------+---------------------+---------------------+---------------------
 somebody.com   | 2015-03-09 00:00:10 | 2015-03-10 00:00:10 | 2015-03-10 00:20:15
 somebody.com   | 2015-03-09 00:01:44 | 2015-03-10 00:01:44 | 2015-03-10 00:20:16
 somebody.com   | 2015-03-09 00:02:39 | 2015-03-10 00:02:39 | 2015-03-10 00:20:16
 somebody.com   | 2015-03-09 00:04:46 | 2015-03-10 00:04:46 | 2015-03-10 00:20:18
 somebody.com   | 2015-03-09 00:08:18 | 2015-03-10 00:08:18 | 2015-03-10 00:20:19
 somebody.com   | 2015-03-09 00:10:02 | 2015-03-10 00:10:02 | 2015-03-10 00:20:20
 somebody.com   | 2015-03-09 00:12:46 | 2015-03-10 00:12:46 | 2015-03-10 00:20:21
 somebody.com   | 2015-03-09 00:14:33 | 2015-03-10 00:14:33 | 2015-03-10 00:20:21
 somebody.com   | 2015-03-09 00:14:33 | 2015-03-10 00:14:33 | 2015-03-10 00:20:22
 somebody.com   | 2015-03-09 00:21:28 | 2015-03-10 00:21:28 | 2015-03-10 01:20:05
 somebody.com   | 2015-03-09 00:22:48 | 2015-03-10 00:22:48 | 2015-03-10 01:20:06
 somebody.com   | 2015-03-09 00:25:33 | 2015-03-10 00:25:33 | 2015-03-10 01:20:07
 somebody.com   | 2015-03-09 00:32:21 | 2015-03-10 00:32:21 | 2015-03-10 01:20:10
 somebody.com   | 2015-03-09 00:32:29 | 2015-03-10 00:32:29 | 2015-03-10 01:20:10
 somebody.com   | 2015-03-09 00:35:45 | 2015-03-10 00:35:45 | 2015-03-10 01:20:10
 somebody.com   | 2015-03-09 00:36:48 | 2015-03-10 00:36:48 | 2015-03-10 01:21:43
 somebody.com   | 2015-03-09 00:37:09 | 2015-03-10 00:37:09 | 2015-03-10 01:22:54
 somebody.com   | 2015-03-09 00:37:14 | 2015-03-10 00:37:14 | 2015-03-10 01:23:27
 somebody.com   | 2015-03-09 00:40:31 | 2015-03-10 00:40:31 | 2015-03-10 01:23:49
 somebody.com   | 2015-03-09 00:41:09 | 2015-03-10 00:41:09 | 2015-03-10 01:23:50
 somebody.com   | 2015-03-09 00:45:36 | 2015-03-10 00:45:36 | 2015-03-10 01:24:22
 somebody.com   | 2015-03-09 00:45:41 | 2015-03-10 00:45:41 | 2015-03-10 01:24:26
 somebody.com   | 2015-03-09 00:45:52 | 2015-03-10 00:45:52 | 2015-03-10 01:24:48
 somebody.com   | 2015-03-09 00:45:57 | 2015-03-10 00:45:57 | 2015-03-10 01:25:21

You can see that the period of the reports generated is sliding.

I think that the run timeframe should always be for the entire prior single-hour timeframe, if the run is for a 1-hour slice. (E.G. 00:00 - 00:59:59 even if the generation is run at 00:24:00)
Or, if the run timeframe is for 1 day, it should always generate the entire prior UTC day, 00:00 - 23:59:59

added dmarc_whitelist hosts

Format: IP whitespace type whitespace comment

Reason types: forwarded sampled_out trusted_forwarder mailing_list local_policy other

any IP without a type specified will default to 'other'

Comment is a free form text entry

127.0.0.3 trusted_forwarder Test Comment

127.0.0.1 local_policy localhost
::1 local_policy localhost

140.211.11.3 mailing_list apache.org

141.42.206.35 mailing_list charite.de

146.112.225.21 mailing_list phishtank.com

168.100.1.1 mailing_list cloud9.net
168.100.1.3 mailing_list cloud9.net
168.100.1.4 mailing_list cloud9.net
168.100.1.7 mailing_list cloud9.net

2604:8d00:0:1::3 mailing_list cloud9.net
2604:8d00:0:1::4 mailing_list cloud9.net
2604:8d00:0:1::7 mailing_list cloud9.net

198.148.79.53 mailing_list clamav.net

208.69.40.157 mailing_list blackops.org

95.128.36.21 mailing_list kolabsys.com
95.128.36.22 mailing_list kolabsys.com
95.128.36.23 mailing_list kolabsys.com

Update of public_suffix_list

Would it be possible to update the default public_suffix_list in the distribution please? I realise I can update this myself, but it would be useful to know that I don't have to remember to do so next time I install the module ;-)

Thanks for a great module,

Andy

dmarc_send_reports retries sending forever

Hi,

I've noticed that there are cases where a report is not removed from the database despite failure to send the report. For example, in the case below, the server returns a 554 error. This is a permanent error. Shouldn't dmarc_dend_reports delete this entry from the DB instead of retrying? Perhaps we should examine the $smtp->code in line 395 of dmarc_send_reports.bat and delete records which have resulted in permanent failures, i.e. 5XX codes. I have attached a patch which deals with the error when dealing with DATA command (copied from existing code which dealt with 5XX for recipients).

ID: 44
Domain: isdg.net
rua: mailto:[email protected]

getting MX for isdg.net
delivering message to [email protected], via mail.isdg.net
DATA for domain isdg.net report rejected
554 Message Not Accepted by filter.

DBI error: NOT NULL constraint failed: report_error.time
Sending error query called by Mail::DMARC::Report::Store::SQL, 498
INSERT INTO report_error (report_id, error ) VALUES (??)
44, 554 Message Not Accepted by filter.
at C:/Program Files (x86)/Vircom/modusGate/perl/site/lib/Mail/DMARC/Report/Store.pm line 17.

dmarc_send_reports.patch.txt

Infinite loop on report sending

Hi Matt,

Thanks for accepting the merge requests.

I have found an issue with dmarc_send_reports. If there are records in the database which for whatever reason cannot be sent then the dmarc_send_reports script enters an infinite loop.
Reports are only deleted on a successful send, and the retrieve_todo method call will retrieve the same report on subsequent runs.

I have patched a local copy to check for a report_id repeating, and simply delete the repeated id however I am not certain this is the best way to handle the error.

thoughts?

SQLite tests failing

~/.cpanm/work/1420752724.36287/Mail-DMARC-1.20141230$ perl -Mblib t/12.Report.Store.SQL.t
ok 1 - use Mail::DMARC::Report::Store::SQL;
ok 2 - An object of class 'Mail::DMARC::Report::Store::SQL' isa 'Mail::DMARC::Report::Store::SQL'
ok 3 - db_connect
ok 4 - An object of class 'DBIx::Simple' isa 'DBIx::Simple'
ok 5 - query_insert, report, 1
not ok 6 - query_delete

...and so on, but that's the only failure.

I am using DBD::SQLite 1.31 on perl 5.8.8.

metacharacters produce FN validation

sub is_valid_p {
    my ( $self, $p ) = @_;
    croak "unspecified p" if !defined $p;
    return ( grep {/^$p$/i} qw/ none reject quarantine / ) ? 1 : 0;
}

can be spoofed with "v=DMARC1; p=.*"

not that that is of any real value, but the idea of modding the pattern /^$x$/ to /^\Q$x\E$/ is what's intended here.

loop in dmarc_send_report

I received some mails from sp-assu.fr and other domains that have the same rua; the rua seems not legit, Google replies to dmarc_send_report with error 450 and dmarc_send_report goes in a loop.

$ host -ttxt sp-assu.fr
sp-assu.fr descriptive text "v=spf1 include:spf.sendinblue.com mx ~all"
$ host -ttxt _dmarc.sp-assu.fr
_dmarc.sp-assu.fr descriptive text "v=DMARC1\; p=none\; sp=none\; rua=mailto:[email protected]!10m\; ruf=mailto:[email protected]!10m\; rf=afrf\; pct=100\; ri=86400"

ID:     12878
Domain: sp-assu.fr
rua:    mailto:[email protected]!10m
getting MX for mailinblue.com
delivering message to [email protected], via aspmx.l.google.com
RCPT TO [email protected] rejected
        450 4.2.1 The user you are trying to contact is receiving mail at a rate that
4.2.1 prevents additional messages from being delivered. Please resend your
4.2.1 message at a later time. If the user is able to receive mail at that
4.2.1 time, your message will be delivered. For more information, please
4.2.1 visit
4.2.1  https://support.google.com/mail/answer/6592 n63si4220274wme.21 - gsmtp

Incorrect handling of Internationalized Domain Names when checking public suffix list

Internationalized domain names are failing the public suffix list check.

For example, .政务 which translates to punycode .xn--zfr164b

Domains in the psl are utf-8 rather than punycode, fixes are to read in the psl in a utf safe way, and convert incoming domain names from punycode to unicode before checking.

My current fix brings in a dependency of Net::IDN::Encode, would appreciate thoughts on the best way to approach this. @msimerson

fastmailops@1ba5b44

Sub domain reporting at policy level domain.

@tomkicamp mentioned this one in relation to another issue yesterday.

I think I've seen a surfeit of subdomain reporting from (provider). In other words, more than expected. It's up to a bit of interpretation in the spec itself, but the largest providers behave this way: an aggregate report ought to be generated for the domain where the policy is held, only. Subdomains seen in From headers, for which there is no explicit DMARC record at that level, are reported up in the policy_domain level.

So, a report for sub.example.com could be rolled up into the report for example.com, removing the need for individual reports for the domain and all of its sub domains.

Some required values missing from agg report xml

Some values specified in the schema are not added to the generated aggregate report xml.
Specifically, values which are optional in the policy are required to be added to the report as their default values.

http://dmarc.org/dmarc-xml/0.1/rua.xsd

I have a branch which adds these defaults, which I will merge once I'm happy with it.
https://github.com/fastmail/mail-dmarc/tree/report_published_validity

We also seem to be adding some values to results which are not specified in the schema, I'm looking into this and will add it in another issue if required.

evals without error handling lose data

Mail::DMARC::PurePerl has the following code:

eval { $self->is_dkim_aligned; };
eval { $self->is_spf_aligned;  };

...which is good, because it prevents an error checking SPF from crashing the whole program. On the other hand, it simply drops any error on the floor, which can make diagnosing problems a bit confusing. (Especially so because both SPF and DKIM can come from user-provided callbacks.)

I have locally written those as "eval { ...; 1 } or warn $@" for now, but maybe you've got some other means by which you think this could or should be done (error callback?).

mail-dmarc.ini don't offer a port setting for the imap server

lib/Mail/DMARC/Report/Receive.pm assume port 993 if IO::Socket::SSL is installed.
that's no true for my host: I use IMAP on port 143 only.

I solved that by changing lib/Mail/DMARC/Report/Receive.pm, line 29
my $port = defined $self->config->{imap}{port} ? $self->config->{imap}{port} : $self->get_imap_port();

that allow setting "port = 143" in /etc/mail-dmarc.ini

yes, that does not express encryption should be used or not. But in my case I limit the connection to localhost so plaintext don't hurt :-)

Andreas

dmarc_view_reports --geoip v6 support

Hello,

We run your software on IPv6 enabled networks and the dmarc_view_reports --geoip only supports lookups for IPv4.

I personally want to lookup both type of addresses and only display the country code; so currently I use cut/awk/sed/whatever to drop the continent & city fields.

What do you think about the following? Is it something that you would accept upstream?

  1. Replace the current --geoip code to use the MaxMind v6 database, which includes entries for v4 addresses too. The lookup for v4 requires the address to be prefixed with ::ffff: (e.g. ::ffff:127.0.0.1), which is easy to handle. Although this might make some of your existing users unhappy when upgrading, because they will need to change the MaxMind database type. Ideally I don't want to keep support for v4 only, because it complicates the library calls.

  2. Add --geoip-country, --geoip-city, --geoip-continent flags to only show certain fields. Specifying --geoip is equivalent to having all three enabled.

Cheers.

verify_external_reporting croaks when validating rua from yahoo.ca

I've recently started playing around with mail-dmarc, it is awesome. I have run into an issue when creating reports however. When doing a test with yahoo.ca, one enters the verify_external_reporting sub in PurePerl.pm. A TXT record is corrently returned (v=DMARC1) for the $dest (yahoo.ca._report._dmarc.yahoo.com).

But when evaluating line 548, the parsing of the $dmarc_str (v=DMARC1) croaks as it is incomplete. My workaround is not to perform this: eval { $policy = $self->policy->parse($dmarc_str) }; ## no critic (Eval)

`report_record`.`count` tinyint range is too limited

Hello,

Is there a specific reason why the count field [1] is tinyint unsigned? In MySQL [2] the range for that is only 0-255.

I've ran into a bug when we received reports with count value of 400 or more, which causes an INSERT error due to the limited range. We're running MariaDB 10.1.x with a bunch of options to force traditional SQL, escalating truncation warnings to errors and so on. Not sure if default MySQL installs have this problem, it might be that the value is silently truncated to 255.

I haven't had time to actually patch and test it, but I would be happy to if you are willing to merge this upstream.

Cheers and thanks for the software!

[1] https://github.com/jeanpaulgalea/mail-dmarc/blob/master/share/mail_dmarc_schema.mysql#L236
[2] https://dev.mysql.com/doc/refman/5.7/en/integer-types.html

result->reason is an arrayref of objects, not hashrefs

The docs for Result indicate that ->reason will return hashrefs, but actually it's objects.

I was pretty confused by the whole ->reason method, actually. Its name is singular, but it returns potentially many things. It returns an arrayref, which makes me worry a bit about mutation, but that's less of a big deal. When there is no reason, it returns undef instead of [].

I didn't propose a documentation patch because I wonder whether the whole way that one can inspect a Result to determine what really happened can't be generally improved overall. If nothing else, though, this ticket represents the one definite way in which docs disagree with reality. :-)

'fo' attribute validator is invalid

the fo() routine doesn't grok the spec allowed "colon separated list" of ( 0|1|d|s )
in:

croak "invalid fo: $[1]" if 0 == grep {/^$[1]$/ix} qw/ 0 1 d s /;

repeat by:

... fo=0:s; ...

relevant rfc7489 stuff:

 dmarc-fo        = "fo" *WSP "=" *WSP
                   ( "0" / "1" / "d" / "s" )
                   *(*WSP ":" *WSP ( "0" / "1" / "d" / "s" ))

is deferred evaluation of SPF / DKIM possible?

Sometimes, we don't care about SPF or DKIM results if there's no DMARC policy to instruct on their use. It seems that we always need to do the SPF and DKIM checks before checking DMARC, which can result in quite a few unneeded DNS queries.

Is there a way to defer those checks until the DMARC policy makes it clear that one of more of them is needed?

If not, would you be receptive to a change to allow it? One possible implementation would allow the passing of a callback for spf and dkim, to be used to lazily populate the attributes as needed.

Use of uninitialized value

[001] dmarc forensic report for example.org from ip xxx.xxx,xxx.xxx
Use of uninitialized value in split at /usr/lib64/perl5/vendor_perl/5.24.1/Mail/DMARC/Report/Receive.pm line 140.
Use of uninitialized value $c_type in string eq at /usr/lib64/perl5/vendor_perl/5.24.1/Mail/DMARC/Report/Receive.pm line 141.
Use of uninitialized value $c_type in string eq at /usr/lib64/perl5/vendor_perl/5.24.1/Mail/DMARC/Report/Receive.pm line 142.
Use of uninitialized value $c_type in string eq at /usr/lib64/perl5/vendor_perl/5.24.1/Mail/DMARC/Report/Receive.pm line 147.
Use of uninitialized value $c_type in string eq at /usr/lib64/perl5/vendor_perl/5.24.1/Mail/DMARC/Report/Receive.pm line 153.
Use of uninitialized value $c_type in string eq at /usr/lib64/perl5/vendor_perl/5.24.1/Mail/DMARC/Report/Receive.pm line 153.
Use of uninitialized value $c_type in string eq at /usr/lib64/perl5/vendor_perl/5.24.1/Mail/DMARC/Report/Receive.pm line 162.
Use of uninitialized value $c_type in concatenation (.) or string at /usr/lib64/perl5/vendor_perl/5.24.1/Mail/DMARC/Report/Receive.pm line 169.
Unknown message part

v1.20150222 doesn't seem to include recent changes

From the changelog in master:

1.20150222 2015-02-22 America/Los_Angeles

 - tolerate missing SPF auth type scope
 - initialized config file first (was non-deterministic)

…but a comparison of the last two releases doesn't show this change. There is no sort present, and get_config can still return a filename instead of a Config::Tiny object.

The code in git looks correct, so I'm not sure what happened. I did a dzil build on master (38ad62d) and the changes to Mail::DMARC::new and Mail::DMARC::Base::get_config were there. I really don't know what happened, but the released code doesn't seem to match the repo or the changelog.

confusing/inexact comment

seeing this in 1.20150317, Mail::DMARC::Base.pm, line 236:

# You need Perl 5.01 or later at lib/Mail/DMARC/DNS.pm line 83.

ah, there isn't such a thing in this distro. where should this comment actually point?

thanks!

CPAN release

Hi Matt,

Could I request a new CPAN release, please and thank you. :)

`install_deps.pl` installed to PATH

install_deps.pl looks like a utility script that is only intended to be run from a checkout, not installed to the host machine.

As such, it shouldn't be in the bin dir with all the other dmarc tools.

are resources being initialized more often than needed?

While looking at something else, I noticed the PSL being loaded more frequently than I expected. It's initialized in is_public_suffix in Mail::DMARC::Base, and stored in $self->{public_suffixes}. This comment is in there:

    # load PSL into hash for fast lookups, esp. for long running daemons
    my %psl = map { $_ => 1 }
              grep { $_ !~ /^[\/\s]/ } # weed out comments & whitespace
              map { chomp($_); $_ }    # remove line endings
              <$fh>;

…but as near as I can tell, the hash isn't persistent for long running daemons. I might be wrong about that, in which case please let me know what I missed and close the ticket!

If I'm not wrong, though, I think the issue is the dual purpose of Mail::DMARC objects. They are both validators and validation state objects. What I had expected to see was a reusable validator object which accepted requests and provided results. That way, the validator could be initialized once and run forever, reusing its components (which could include its resolver, a PSL, and so on).

I looked at Mail::DMARC::HTTP, and I found that (for example) serve_validator seems to create a new Mail::DMARC::PurePerl object for each request, meaning that each request will reparse the PSL, initialize a new DNS resolver, and so on.

Have I misunderstood? If not, do we want to address this?

Tests use live configuration when run outside of distzilla

Create a build using dzil build
cd to the build directory,
run perl Makefile.pl && make test

If there is a live configuration installed then the tests run against this rather than the configuration in t/, possibly resulting in dmarc records being written to the database.

Quick and dirty workaround is to copy the test config down to the build directory, but a longer term fix is probably a good idea.

improved handling of multiple From addresses

A conversation on [email protected] regarding "handling multiple domains in RFC5322.from (section 6.6.1)" set me to thinking. Back when Mail::DMARC was first authored, the spec was fairly vague on what should be done about messages with multiple From addresses. Back then, I sampled my traffic, found that messages with multiple domains in the From header almost never exist, and punted, by choosing to only use the domain in the last from address. Today, the spec is still a little vague but my opinion is that The Right Thing to do is:

  • evaluate every domain in the From header
  • apply the most strict policy

TODO

  • get_from_dom() should be revamped to properly parse the From header and return an array of addresses.
  • validate() should expect an array from get_from_dom and check each domain for a policy, applying the strictest one.

Domains listed in public_suffix_list fail to align

@msimerson could I have your opinion on something please,

We had a ticket today relating to emails from 1password.com which were failing dmarc despite having a dkim signature from 1password.com.

These were failing because 1password.com is listed in the PRIVATE section of the public suffix list, and Mail::DKIM::PurePerl->is_dkim_alligned() has a check for is_public_suffix(), noting section 4.3.1 of the spec.

From the rfc it is clear that we shouldn't be stopping before a psl listed domain when finding an org domain, but it is ambiguous as to wether psl listed domains can be aligned with a direct match.

Clearly, in this case it is reasonable for 1password.com to be aligned as the signature domain and from domain match, the same would be true for any domain in the PRIVATE section.

For domains in the ICANN section it is less clear, there shouldn't really be a case in practice where an ICANN psl listed domain can sign, so there should never be an alignment, we still stop at the correct boundary, so I can't see any issue with dropping the check in is_dkim_aligned(), but it may be worth keeping that check for ICANN domains, and relaxing it for PRIVATE domains, however the amount of work in that is higher than dropping the check altogether.

What is your preferred solution, drop the check and allow psl listed domains to align when there is a direct match, or split the check and only allow alignment for domains listed in the PRIVATE section?

Thanks,
-Marc

Extra fields in XML aggregate report

The following fields are not defined in the XML schema for aggregate reports (http://dmarc.org/dmarc-xml/0.1/rua.xsd).

record->identifiers->envelope_from
record->auth_results->dkim->selector
record->auth_results->spf->scope

This is causing reports to fail when validated against the schema.

What will be the impact, if any, of removing these fields from the generated reports?

tagging: @msimerson for input

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.