patschi / parsedmarc-dockerized Goto Github PK

Hello, I've bee running this for a while now and the DMARC Summary has been working, but DMARC Forensic Samples dashboard just shows "Error" and the displays look like they are permanently refreshing.

Not sure where/how to look for relevant logs.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update docker.elastic.co/elasticsearch/elasticsearch docker tag to v8
• chore(deps): update docker.elastic.co/kibana/kibana docker tag to v8

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Affected Versions:
Kibana Versions 7.0.0 through 7.17.8 and 8.0.0 through 8.6.0

https://discuss.elastic.co/t/elastic-7-17-9-8-5-0-and-8-6-1-security-update/324661

in my setup I have O365 as a servers adn with O365 disabling basic Auth I am getting login failures, parcedmarc can not loging to email account using iMAP

Hi, I've a fresh installation on a ZorinOS 16 (20.04.1-Ubuntu), connexion to https://myIP:9999 show the elasticsearch dashbord, with all boards in error (SPF, DKIM, DMARC...) and complains about a warning

Configuration missing
server.publicBaseUrl is missing and should be configured when running in a production environment. Some features may not behave correctly.

The second error at the bottom, sounds logical because elasticsearch didn't receive the dmarc data
No matching indices found: No indices match pattern "dmarc_aggregate*"

As you can see in this screen capture

my config.ini

[general]
save_aggregate = True
save_forensic = True
log_file = /var/log/parsedmarc/

[imap]
# IMAP login
host = imap.gmail.com
port = 993
user = [email protected]
password = "my password"
ssl = True

# advanced
[mailbox]
watch = True
reports_folder = DMARC
archive_folder = DmarcProcessed
delete = False

# advanced advanced
test = True

### NO EDIT REQURIED DOWN BELOW
[elasticsearch]
hosts = elasticsearch:9200
ssl = False

Folder for the log file is root ownered, but nothing recorded
I tried with the debug = True option in the config.ini, no more information

docker do not complain during the initialization process

/opt/patschi_parsedmarc-dockerized# docker-compose up -d
Creating network "patschi_parsedmarc-dockerized_parsedmarc-network" with driver "bridge"
Creating network "patschi_parsedmarc-dockerized_default" with the default driver
Creating patschi_parsedmarc-dockerized_geoipupdate_1     ... done
Creating patschi_parsedmarc-dockerized_parsedmarc-init_1 ... done
Creating patschi_parsedmarc-dockerized_elasticsearch_1   ... done
Creating patschi_parsedmarc-dockerized_parsedmarc_1      ... done
Creating patschi_parsedmarc-dockerized_kibana_1          ... done
Creating patschi_parsedmarc-dockerized_nginx_1           ... done

Did I missed something ?
Thanks in advance

I did a complete fresh install (Ubuntu 22.04 + parsedmarc-dockerized) yesterday due to the field error issues (see #23 ).
I only changed the version tags for elastic/kibana in docker-compose.yml

Now kibana is showing everything ok but imap mails are only pulled once after reboot.
The log says parsedmarc is idling/trying to retrieve.
In the install before that worked fine.

Any ideas?

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• chore(deps): update actions/checkout action to v4
• chore(deps): update docker.elastic.co/elasticsearch/elasticsearch docker tag to v8
• chore(deps): update docker.elastic.co/kibana/kibana docker tag to v8
• chore(deps): update docker/build-push-action action to v5
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update docker.elastic.co/elasticsearch/elasticsearch docker tag to v7.17.18
• chore(deps): update docker.elastic.co/kibana/kibana docker tag to v7.17.18

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

I was able to run docker-compose up with some sample data I placed in a test folder with an Office365 mailbox. I'd like to now re-run this with production data.

Because I'm a little inexperienced with docker, I have a few questions:

• When running docker-compose up -d, is parsedmarc running as a service where it will occasionally pull down messages from my O365 via MSGraph?
• If it is not running as a service, how do I "re-run" the dockerized parsedmarc so that it can ingest new DMARC reports sent to my mailbox?

Traceback (most recent call last):
File "/opt/pypy/bin/parsedmarc", line 8, in
sys.exit(_main())
File "/opt/pypy/lib/pypy3.8/site-packages/parsedmarc/cli.py", line 440, in _main
opts.imap_password = imap_config["password"]
File "/opt/pypy/lib/pypy3.8/configparser.py", line 1255, in getitem
return self._parser.get(self._name, key)
File "/opt/pypy/lib/pypy3.8/configparser.py", line 800, in get
d)
File "/opt/pypy/lib/pypy3.8/configparser.py", line 395, in before_get
self._interpolate_some(parser, option, L, value, section, defaults, 1)
File "/opt/pypy/lib/pypy3.8/configparser.py", line 445, in _interpolate_some
"found: %r" % (rest,))
configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(', found: '%kGDzy'

Hello !
I am testing this utility to process the dmarc. but i have this problem.

parsedmarc -c /etc/parsedmarc/config.ini
0it [00:00, ?it/s]
ERROR:cli.py:947:Gmail API Error
Traceback (most recent call last):
File "/opt/pypy/lib/pypy3.8/site-packages/parsedmarc/cli.py", line 943, in _main
oauth2_port=opts.gmail_api_oauth2_port
File "/opt/pypy/lib/pypy3.8/site-packages/parsedmarc/mail/gmail.py", line 46, in init
creds = _get_creds(token_file, credentials_file, scopes, oauth2_port)
File "/opt/pypy/lib/pypy3.8/site-packages/parsedmarc/mail/gmail.py", line 31, in _get_creds
oauth2_port=oauth2_port)
File "/opt/pypy/lib/pypy3.8/site-packages/google_auth_oauthlib/flow.py", line 497, in run_local_server
bind_addr or host, port, wsgi_app, handler_class=_WSGIRequestHandler
File "/opt/pypy/lib/pypy3.8/wsgiref/simple_server.py", line 154, in make_server
server = server_class((host, port), handler_class)
File "/opt/pypy/lib/pypy3.8/socketserver.py", line 452, in init
self.server_bind()
File "/opt/pypy/lib/pypy3.8/wsgiref/simple_server.py", line 50, in server_bind
HTTPServer.server_bind(self)
File "/opt/pypy/lib/pypy3.8/http/server.py", line 138, in server_bind
socketserver.TCPServer.server_bind(self)
File "/opt/pypy/lib/pypy3.8/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use

make this change in : docker compose expose port 8080 in parsedmarc

Have you tried using gmail_api?

Thanks

As of today, Februari 15, the parsedmarc container doesn't start. I tried building the image myself, but it gives the same error.

Traceback (most recent call last):
  File "/opt/pypy/bin/parsedmarc", line 5, in <module>
    from parsedmarc.cli import _main
  File "/opt/pypy/lib/pypy3.10/site-packages/parsedmarc/__init__.py", line 31, in <module>
    from parsedmarc.mail import MailboxConnection
  File "/opt/pypy/lib/pypy3.10/site-packages/parsedmarc/mail/__init__.py", line 2, in <module>
    from parsedmarc.mail.graph import MSGraphConnection
  File "/opt/pypy/lib/pypy3.10/site-packages/parsedmarc/mail/graph.py", line 10, in <module>
    from msgraph.core import GraphClient
ModuleNotFoundError: No module named 'msgraph'

Fresh install, the panel to the left of "Message Source Countries" says unrecognized layerType EMS_VECTOR_TILE. The panel to the right says there are 49 messages all from US. This is new so I don't have any RUF reports yet if that is what it wants. Is this a bug or is it waiting for some data I don't have yet?

Hi,
the main dashboard works splendidly, however the Forensic samples one is empty with following errors:

same for source_country.keyword & source_ip_address.keyword

Any idea if these need to be updated to different values, or is it parsedmarc config that needs to be amended?

Thanks for any insights!

Hi ,

is there a way to use htpassword on the nginx because i run the docker containers i get me straigt to the details page an evryone can access this.

so my idea was to add htpassword but how to i transfer the htpasswd to the docker container?

Hi,
I'm a bit confused about this - since the elasticsearch version is fixed (to 7.8.1), this should not happen. Maybe I messed up something else?

parsedmarc_1 | Traceback (most recent call last):
parsedmarc_1 | File "/opt/pypy/bin/parsedmarc", line 8, in
parsedmarc_1 | sys.exit(_main())
parsedmarc_1 | File "/opt/pypy/site-packages/parsedmarc/cli.py", line 583, in _main
parsedmarc_1 | forensic_indexes=[es_forensic_index])
parsedmarc_1 | File "/opt/pypy/site-packages/parsedmarc/elastic.py", line 244, in migrate_indexes
parsedmarc_1 | if not Index(aggregate_index_name).exists():
parsedmarc_1 | File "/opt/pypy/site-packages/elasticsearch_dsl/index.py", line 414, in exists
parsedmarc_1 | return self._get_connection(using).indices.exists(index=self._name, **kwargs)
parsedmarc_1 | File "/opt/pypy/site-packages/elasticsearch/client/utils.py", line 168, in _wrapped
parsedmarc_1 | return func(*args, params=params, headers=headers, **kwargs)
parsedmarc_1 | File "/opt/pypy/site-packages/elasticsearch/client/indices.py", line 333, in exists
parsedmarc_1 | "HEAD", _make_path(index), params=params, headers=headers
parsedmarc_1 | File "/opt/pypy/site-packages/elasticsearch/transport.py", line 413, in perform_request
parsedmarc_1 | _ProductChecker.raise_error(self._verified_elasticsearch)
parsedmarc_1 | File "/opt/pypy/site-packages/elasticsearch/transport.py", line 630, in raise_error
parsedmarc_1 | raise UnsupportedProductError(message)
parsedmarc_1 | elasticsearch.exceptions.UnsupportedProductError: The client noticed that the server is not a supported distribution of Elasticsearch
parsedmarc-dockerized_parsedmarc_1 exited with code 1

I'm using the following command to start parsedmarc in docker-compose

command: ["-c", "/etc/parsedmarc.ini", "--verbose", "-o", "/output", "--aggregate-json-filename", "/output/aggregate.json", "--forensic-json-filename", "/output/forensic.json", "--debug"]

it is stored perfectly into opensearch but I assumed that the local files would be created too. But the json files just contain [] after a report is processed. Also the csv files do not look complete. Debug does not show an error related to writing into the files.

Hello all,

When I start the project, the container parsedmarc-1 don't start
When I go in the logs I see that :

2024-04-03 11:07:52    ERROR:cli.py:500:A file does not exist at /etc/parsedmarc/config.ini

2024-04-08 09:41:27   Traceback (most recent call last):
2024-04-08 09:41:27   File "/opt/pypy/lib/pypy3.10/site-packages/parsedmarc/cli.py", line 1211, in _main
2024-04-08 09:41:27     mailbox_connection = IMAPConnection(
2024-04-08 09:41:27   File "/opt/pypy/lib/pypy3.10/site-packages/parsedmarc/mail/imap.py", line 24, in __init__
2024-04-08 09:41:27     self._client = IMAPClient(host, user, password, port=port,
2024-04-08 09:41:27   File "/opt/pypy/lib/pypy3.10/site-packages/mailsuite/imap.py", line 151, in __init__
2024-04-08 09:41:27     imapclient.IMAPClient.__init__(self,
2024-04-08 09:41:27   File "/opt/pypy/lib/pypy3.10/site-packages/imapclient/imapclient.py", line 288, in __init__
2024-04-08 09:41:27     self._imap = self._create_IMAP4()
2024-04-08 09:41:27   File "/opt/pypy/lib/pypy3.10/site-packages/imapclient/imapclient.py", line 325, in _create_IMAP4
2024-04-08 09:41:27     return tls.IMAP4_TLS(
2024-04-08 09:41:27   File "/opt/pypy/lib/pypy3.10/site-packages/imapclient/tls.py", line 44, in __init__
2024-04-08 09:41:27     imaplib.IMAP4.__init__(self, host, port)
2024-04-08 09:41:27   File "/opt/pypy/lib/pypy3.10/imaplib.py", line 202, in __init__
2024-04-08 09:41:27     self.open(host, port, timeout)
2024-04-08 09:41:27   File "/opt/pypy/lib/pypy3.10/site-packages/imapclient/tls.py", line 52, in open
2024-04-08 09:41:27     sock = socket.create_connection(
2024-04-08 09:41:27   File "/opt/pypy/lib/pypy3.10/socket.py", line 824, in create_connection
2024-04-08 09:41:27     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
2024-04-08 09:41:27   File "/opt/pypy/lib/pypy3.10/socket.py", line 955, in getaddrinfo
2024-04-08 09:41:27     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
2024-04-08 09:41:27 _socket.gaierror: [Errno -2] Name or service not known
2024-04-08 09:42:28    ERROR:cli.py:1223:IMAP Error

I'm working on the version v4.28.0 on docker desktop on windows 11.
Do you have any idea of how to solve this problem ?

Thank you

Is there a way to use localfiles for the parsing?
I.e. mounting a volume and placing the report files there for analyzing

I just did a clean install and when I go to the web site, there are no graphics. Every box has an error that says: "[esaggs] > The field "message_count" associated with this object no longer exists in the index pattern. Please use another field.

Thanks for all the updates to the Repository!

I now updated to the newest Repo version.

I also adjusted my parsedmarc config based on the new config.sample.ini
eb00207

But instead of using Processed as archive_folder it used the default value of Archive.

Looking at docker images I found
patschi/parsedmarc latest 210dcbdce820 2 years ago 484MB

Some update of parsedmarc added the new section [mailbox] but this doesn't work with the previous version I still had on the system.
After doing docker compose pull and restart everything it works now.

So maybe it's helpful to add version tags for the Docker images and use them in the docker-compose.yml

Recently went to check on this great solution and my SPF Alignment Over Time and DKIM Alignment Over Time panels only show "Trying to initialize aggs without index pattern". I did a fresh install and this persists. Anything I can do to fix it?

Thank you.