rseichter / fangfrisch Goto Github PK

View Code? Open in Web Editor NEW

85.0 85.0 9.0 1.61 MB

Update and verify unofficial Clam Anti-Virus signatures

License: GNU General Public License v3.0

Makefile 1.21% Shell 4.99% Python 93.80%

fangfrisch's People

Stargazers

Watchers

Forkers

hydrapolic kprkpr captainsifff intelowlproject waycarbon andrewnimmo rfc1036 wangshifeng

fangfrisch's Issues

Can't create database structure; fangfrisch command not found?

Hello, thank you for working on an elegant update tool for ClamAV! This is awesome!

I am curious if I can get some help, stuck on this step (I'm on Fedora 32 desktop workstation) entered normally as myself (underprivileged user), not under root. i think i installed fangfrisch correctly in earlier steps.

[soffeas@localhost ~]$ pip3.8 install --user fangfrisch
Collecting fangfrisch
  Using cached https://files.pythonhosted.org/packages/e7/a8/e3044dbfff8abdf7a63bfa0a026a79cbeb94b99fa8d8bb74ffaa4d6c630b/fangfrisch-1.2.0-py3-none-any.whl
Requirement already satisfied: requests>=2.22.0 in /usr/lib/python3.8/site-packages (from fangfrisch) (2.22.0)
Requirement already satisfied: SQLAlchemy>=1.3.13 in ./.local/lib/python3.8/site-packages (from fangfrisch) (1.3.20)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib/python3.8/site-packages (from requests>=2.22.0->fangfrisch) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/lib/python3.8/site-packages (from requests>=2.22.0->fangfrisch) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3.8/site-packages (from requests>=2.22.0->fangfrisch) (1.25.7)
Installing collected packages: fangfrisch
Successfully installed fangfrisch-1.2.0

[soffeas@localhost ~]$ fangfrisch 
usage: fangfrisch [-h] [-c CONF] [-f] [-p PROVIDER]
                  {dumpconf,dumpmappings,initdb,refresh}
fangfrisch: error: the following arguments are required: action

[soffeas@localhost ~]$ sudo -u clamav -- fangfrisch --conf /etc/fangfrisch.conf initdb
sudo: fangfrisch: command not found

i can't make sense of this.

help?

Add support for syslog-based logging

In order to avoid relying on Cron email reports, logging via syslog should be supported.

Malwarepatrol changing URLs!

Just got that E-Mail from malwarepatrol and wanted to inform you, just in case you didn't know:

Hello Subscriber,

We hope this message finds you well. We're reaching out to inform you about some upcoming changes to our Risk Indicators / OSINT data feed.

We've recently made improvements to make it easier to parse, enhancing your experience with the data. The content of the OSINT data feed remains unchanged; however, there are some updates regarding the links for two of the three feeds in the set: Risk Indicators and High Risk IPs.

Please note the following key points:

Improved Formatting: We've enhanced the JSON format of the OSINT data feed to ensure better readability and ease of integration into your systems.
New Links: The Risk Indicators and High Risk IPs feeds' new links are already available in the customer portal.
Action Required: We kindly ask you to update your systems or scripts to use the new links for the Risk Indicators and High Risk IPs feeds. The old links will be disabled on March 22, 2024.
TOR Exit Nodes: The link and format for TOR Exit Nodes remain unchanged. You can continue accessing this feed as usual without any interruption.

Should you have any questions or require assistance during this transition, please don't hesitate to reach out to our dedicated support team at [email protected].

We appreciate your continued trust and support as we strive to provide you with the best possible service.

Best regards,

Malware Patrol Team

urlhaus digest mismatch

The v3.7.3 Script runs well but sometimes (for ex. today, Fri, 27 Nov 2020 10:45:03 ) I get odd messages in the log file / status mail like:
WARNING: https://urlhaus.abuse.ch/downloads/urlhaus.ndb sha256 digest mismatch (expected fc554db6a3a6d2fee4056a0b80d4724a4701dc100f7034c80c75c41ce382963f, got 8ff4b8d6c964ed579bdba6fc1931382daed2d567d70e7e26f20b4d240abd1797)
Seems to be due to urlhaus, but I don't necessarily want to switch off the checksum checks completely.
So, any hint's?

Option to disable single data source from provider

As in #8 the Problem already exist.
As mention in malware.expert.fp ab1762c69439bdd1d6381726e832544f:37:buy.subscriotion.from.malware.expert for this you need a subscription.
Now i want to disable the signatures.

url_malwareexpert_fp = /dev/null
url_malwareexpert_hdb = /dev/null
url_malwareexpert_ldb = /dev/null
url_malwareexpert_ndb = /dev/null

will probably work but brings error in syslog

Feb  8 15:15:00 x fangfrisch[8220]: Invalid URL '/dev/null.sha256': No schema supplied. Perhaps you meant http:///dev/null.sha256?
Feb  8 15:15:00 x Traceback (most recent call last):
Feb  8 15:15:00 x File "/usr/lib/python3.8/site-packages/fangfrisch/refresh.py", line 106, in refresh
Feb  8 15:15:00 x digest = get_digest(ci)
Feb  8 15:15:00 x File "/usr/lib/python3.8/site-packages/fangfrisch/download.py", line 70, in get_digest
Feb  8 15:15:00 x download = _download(f'{ci.url}.{ci.check}', max_size)
Feb  8 15:15:00 x File "/usr/lib/python3.8/site-packages/fangfrisch/download.py", line 57, in _download
Feb  8 15:15:00 x response = _session.get(url, stream=True, timeout=30)
Feb  8 15:15:00 x File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
Feb  8 15:15:00 x return self.request('GET', url, **kwargs)
Feb  8 15:15:00 x File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 528, in request
Feb  8 15:15:00 x prep = self.prepare_request(req)
Feb  8 15:15:00 x File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 456, in prepare_request
Feb  8 15:15:00 x p.prepare(
Feb  8 15:15:00 x File "/usr/lib/python3.8/site-packages/requests/models.py", line 316, in prepare
Feb  8 15:15:00 x self.prepare_url(url, params)
Feb  8 15:15:00 x File "/usr/lib/python3.8/site-packages/requests/models.py", line 390, in prepare_url
Feb  8 15:15:00 x raise MissingSchema(error)
Feb  8 15:15:00 x requests.exceptions.MissingSchema: Invalid URL '/dev/null.sha256': No schema supplied. Perhaps you meant http:///dev/null.sha256?

Feature: (Optionally) Cleanup unused signatures (db)

Let us say I am using two sources for signatures - sanesecurity and urlhaus.

Tomorrow I decide that sanesecurity signature are not required or are to be disabled temporarily because there is some issue with them.

So I remove sanesecurity from fangfrisch.conf but old signatures will still remain in /var/lib/clamav and will continue to be used by clamav.

So can you provide a feature where I run fangfrisch cleanunusedsigs and it cleans up files related to unused sources from /var/lib/clamav

It will check which data sources are enabled and delete files related to other databases which are not enabled. (ofcourse it should not delete other signature files which it has not downloaded but are there in /var/lib/clamav)

Modernise Python build (PEP 517)

The Python build is currently lacking both pyproject.toml and setup.cfg (see Python Packaging User Guide and PEP 517). This needs to be addressed.

Option to enable specific data sources

Is it possible to specify only the data sources you want from a feed? I see you can disable datasources but I want to specify only the data sources I want. For instance, for Sanesecurity I want to only enable the ones with Low FP score.

Thanks

Overriding on_update_exec in configuration sections

The option on_update_exec is currently only permitted in the configuration's DEFAULT section. It would be useful to be able to override this option within provider sections.

Use yes/no or true/false for enabled

Currently, the docs use yes/false for the enabled option. Of cource, this is valid, but using one of the usual pairs yes/no or true/false is less surprising.

compatibility Problems with Ubuntu?

Hi,

thank you for this project an your time!
Could it be that it is not supported on ubuntu?

mx01 fangfrisch # python3 -m venv venv
mx01 fangfrisch # source venv/bin/activate
(venv) mx01 fangfrisch # pip3 install fangfrisch
Collecting fangfrisch
  Could not find a version that satisfies the requirement fangfrisch (from versions: )
No matching distribution found for fangfrisch

regards

Celevra

Proxy configuration

fangfrisch-1.9.0

Please provide an option to locally configure a specific proxy server for fangfrisch passing to the requests library, if a global HTTPS_PROXY in the environment is not an option.

Improve setup on Gentoo

Hello, seeing that you're also the maintainer on Gentoo, i was wondering if it might be possible to improve the setup on there. I've not tested Arch which also seem to have a package. Maybe it's similar there.

Fangfrisch was suggested as a new alternative to clamav-unofficial-sigs but when installing it, it lacks of a number of defaults that would usually be provided from other packages. I got it working but migrating a number of machines to it, some would have to rerun the commands that would simply set documented defaults.

The package does not create a /var/lib/fangfrisch directory when it is essential to have it. Even if you're not going to add the creation of such a directory to the Ebuild, it might be a good idea to add the initial commands to the Ebuild info instead of the link to your homepage. As a new user, i was forced to go to your website just to find 2 or 3 commands that i needed to enter.

The other thing i wonder about is the lack of a default config file location, when the file is already placed there. Wouldn't it be easier if Fangfrisch use the default but keeps the -c option for those using another location? That way you could simply run fangfrisch initdb without the need of providing the default config, that was placed to /etc/fangfrisch.conf during install.

rsync support for sanesecurity updates

I have recently started trying fangfrisch as a replacement for the old clamav-unofficial-sigs package, but now I have serious doubts about its viability.
That script only uses rsync and, while rsync.sanesecurity.net has an healthy list of IPs, I cannot find a list of official HTTP mirrors for sanesecurity.
The default configuration uses an australian HTTP mirror which is seriously overloaded and rarely responds in less that 30 seconds. Also, this is not a sensible default because the bandwidth costs for Australia are very high.

Also, I see that the default configuration still downloads files like malware.expert.* and phishtank.ndb, which have been empty for years.

".yes" extension in securiteinfo ?

I'm unable to fetch the securiteinfo using a paid account. All files are ending with .yes extension and trigger an 403 forbidden error. Removing the yes extension, works.

Who is adding that ?

# sudo -u clamav -- /var/lib/fangfrisch/venv/bin/fangfrisch --conf /etc/fangfrisch/fangfrisch.conf refresh
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfoandroid.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfoascii.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfohtml.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/javascript.ndb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfopdf.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfo.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfo.ign2.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfo0hour.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfo.mdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfoold.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/spam_marketing.ndb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfo.yara.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfo.pdb.yes download failed: 403 Forbidden


# cat /etc/fangfrisch/fangfrisch.conf
[DEFAULT]
db_url = sqlite:////var/lib/fangfrisch/db.sqlite

local_directory = /var/lib/clamav
on_update_exec = clamdscan --reload
on_update_timeout = 42

[sanesecurity]
enabled = yes

[securiteinfo]
enabled = yes
integrity_check = yes
customer_id = xxxxxx
url_0hour = ${prefix}securiteinfo0hour.hdb
url_securiteinfo_mdb = ${prefix}securiteinfo.mdb
url_old = ${prefix}securiteinfoold.hdb
url_spam_marketing = ${prefix}spam_marketing.ndb
url_yara = ${prefix}securiteinfo.yara
url_pdb = ${prefix}securiteinfo.pdb

[urlhaus]
interval = 1h
enabled = yes
max_size = 2MB

[twinwave]
enabled = yes
integrity_check = disabled
interval = 1h
prefix = https://raw.githubusercontent.com/twinwave-security/twinclams/master/
url_twinclams = ${prefix}twinclams.ldb
url_twinwave_ign2 = ${prefix}twinwave.ign2

[clampunch]
enabled = yes
max_size = 2M
integrity_check = disabled
interval = 24h
prefix = https://raw.githubusercontent.com/wmetcalf/clam-punch/master/
url_miscreantpunch099low = ${prefix}MiscreantPunch099-Low.ldb
url_exexor99 = ${prefix}exexor99.ldb
url_miscreantpuchhdb = ${prefix}miscreantpunch.hdb

[rfxn]
enabled = yes
interval= 4h
integrity_check = disabled
prefix = https://www.rfxn.com/downloads/
url_rfxn_ndb = ${prefix}rfxn.ndb
url_rfxn_hdb = ${prefix}rfxn.hdb
url_rfxn_yara = ${prefix}rfxn.yara

[interserver]
enabled = yes
interval = 1d
integrity_check = disabled
prefix = https://rbldata.interserver.net/
url_interserver_sha256 = ${prefix}interserver256.hdb
url_interserver_topline = ${prefix}interservertopline.db
url_interserver_shell = ${prefix}shell.ldb
url_interserver_whitelist = ${prefix}whitelist.fp

[ditekshen]
enabled = yes
interval = 1d
integrity_check = disabled
prefix = https://raw.githubusercontent.com/ditekshen/detection/master/clamav/
url_ditekshen_ldb = ${prefix}clamav.ldb
filename_ditekshen_ldb = ditekshen.ldb

error in refreshing the rules

Hello!

first of all, thank you for this work!

While trying to integrate this tool with IntelOwl, I found a problem that always happen when I launch fangfrisch refresh

intelowl_malware_tools_analyzers | INFO: /var/lib/clamav/junk.ndb updated (6989248 bytes)
intelowl_malware_tools_analyzers | INFO: /var/lib/clamav/jurlbl.ndb updated (830565 bytes)
intelowl_malware_tools_analyzers | INFO: /var/lib/clamav/jurlbla.ndb updated (88784 bytes)
intelowl_malware_tools_analyzers | Traceback (most recent call last):
intelowl_malware_tools_analyzers |   File "/usr/local/bin/fangfrisch", line 8, in <module>
intelowl_malware_tools_analyzers |     sys.exit(main())
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/fangfrisch/__main__.py", line 64, in main
intelowl_malware_tools_analyzers |     ClamavRefresh(args).refresh_all()
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/fangfrisch/refresh.py", line 142, in refresh_all
intelowl_malware_tools_analyzers |     if self.refresh(ci):
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/fangfrisch/refresh.py", line 134, in refresh
intelowl_malware_tools_analyzers |     RefreshLog.update(ci, digest.data)
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/fangfrisch/db.py", line 190, in update
intelowl_malware_tools_analyzers |     entry: RefreshLog = _query_url(ci.url, session)
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/fangfrisch/db.py", line 239, in _query_url
intelowl_malware_tools_analyzers |     return session.query(RefreshLog).filter(RefreshLog.url == url).first()
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2752, in first
intelowl_malware_tools_analyzers |     return self.limit(1)._iter().first()  # type: ignore
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2855, in _iter
intelowl_malware_tools_analyzers |     result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2229, in execute
intelowl_malware_tools_analyzers |     return self._execute_internal(
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2114, in _execute_internal
intelowl_malware_tools_analyzers |     conn = self._connection_for_bind(bind)
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1981, in _connection_for_bind
intelowl_malware_tools_analyzers |     return trans._connection_for_bind(engine, execution_options)
intelowl_malware_tools_analyzers |   File "<string>", line 2, in _connection_for_bind
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/state_changes.py", line 137, in _go
intelowl_malware_tools_analyzers |     ret_value = fn(self, *arg, **kw)
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1108, in _connection_for_bind
intelowl_malware_tools_analyzers |     conn = bind.connect()
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3245, in connect
intelowl_malware_tools_analyzers |     return self._connection_cls(self)
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
intelowl_malware_tools_analyzers |     self._dbapi_connection = engine.raw_connection()
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3269, in raw_connection
intelowl_malware_tools_analyzers |     return self.pool.connect()
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 455, in connect
intelowl_malware_tools_analyzers |     return _ConnectionFairy._checkout(self)
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1270, in _checkout
intelowl_malware_tools_analyzers |     fairy = _ConnectionRecord.checkout(pool)
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 719, in checkout
intelowl_malware_tools_analyzers |     rec = pool._do_get()
intelowl_malware_tools_analyzers |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 157, in _do_get
intelowl_malware_tools_analyzers |     raise exc.TimeoutError(
intelowl_malware_tools_analyzers | sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)

At one point, during download, SQLAlchemy breaks.
I think that by incrementing the pool_size and max_overflow values (see doc) it would fix the problem.

I can open a really little PR for this if you like

Tests infeasible to run for downstreams

I was looking into enabling the frangfrisch test suite to run during package build and in the Debian CI infrastructure, but quickly ran into issues.

During package build, there is no internet access (to make sure that the Debian archive is self-contained). As a result, any tests that require internet access to run, will fail. This is not a huge issue as it's common that really testing things requires doing something on a network and mocking the required network connections is non-trivial.
In our CI environment, tests are run periodically, so multiple connects from a single host are normal. Low thresholds for blocking access make running the tests in CI unreliable.

In the course of trying to enable the tests, I quickly started getting errors like:

requests.exceptions.ProxyError: HTTPSConnectionPool(host='seichter.de', port=443): Max retries exceeded with url: /favicon-16x16.png (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7d1992a690>: Failed to establish a new connection: [Errno 111] Connection refused')))

I don't know what would be the best solution to this, but currently, for Debian, running these tests is not feasible.

Scott K

comparison

how is this different from clamav-unofficial-sigs ?

anyways, enjoy: http://sid.ethz.ch/debian/fangfrisch/

Guide

Any setup guide for this?

Configurable connection timeouts

Connection timeouts should be configurable, at least at the data provider level and possibly at URL level. The current, hard coded timeout is not flexible enough to handle known slow connections. The new feature must take scheduling considerations into account, e.g. avoiding problems due to multiple Fangfrisch instances running concurrently.

Possible to use environment variables for db_url?

Hi there,

Is there was a simple way to pass the db_url connection string via an environment variable? Right now, our database isn't located on the same machine as fangfrisch so we have password protection enabled for the database user. The issue is that it seems like db_url in the configuration file requires a hardcoded string that would include the database password being written to disk.

Would it be possible to just read the default db_url to also read from FANGFRISCH_DB_URL or more generally, allow us to configure which environment variable the any configuration should be read from?

I dug through the source code and all the issues but didn't anything about environment variables generally being supported.

Thanks for the help!

log_method syslog not working with metalog

Coming from hvisage/metalog#32 ...

Setting log_method to syslog doesn't log anything to metalog:

# grep log /etc/fangfrisch.conf 
log_level = DEBUG
log_method = syslog

Removing the log_method and falling back to "console", the logging to STDOUT works as expected.

Gentoo Linux 23.0
app-antivirus/fangfrisch-1.9.0
app-antivirus/clamav-1.2.2
dev-lang/python-3.12.3-r1
app-admin/metalog-20230719

Timeout Sanesecurity from AS3320

Yo,

I'm seeing alot of these when updating Sanesecurity:

ERROR: HTTPConnectionPool(host='ftp.swin.edu.au', port=80): Read timed out. (read timeout=30)

I tested with a browser and it is slow as molasses, coming from Deutsche Telekom AS3320. It works, but is really slow. I checked traceroute for IPv6 and that looks very nasty. Problem has persisted for months I just ignored it so far.

I fixed it temporarily by hot-modifying timeout=30 here: https://github.com/rseichter/fangfrisch/blob/master/fangfrisch/download.py#L57 I set it to 120 seconds and the error is gone.

Any chance to make this configurable? Or some other option?

(Edit: And bump version number so Arch picks it up...)

Thanks, Stephan

Option to disable single data source from provider

The sanesecurity provider downloads scanmnailer.ndb. However this is unmaintained and its author discourages its use:
http://www.scamnailer.info/

I would like to disable this source on my installation, without disabling the complete sanesecurity provider. I could create a custom sanesecurity provider without the scanmailer source, but an easier way would be helpful.

Struggling with setting up Fangfrisch database on MX Linux

I use MX Linux and am working on a hardened respin.

I installed Fangfrisch, following the instructions as written up to here https://rseichter.github.io/fangfrisch/#dbsetup in a root shell.

Running this command sudo -u clamav -- fangfrisch --conf /etc/fangfrisch.conf initdb
(even when I set my user profile to the clamav usergroup) only got me this error.
sudo: fangfrisch: command not found

So I was forced to run Fangfrisch, using the initdb and refresh commands as root.

When I tried to refresh the database, here's the output I got (minus the receipt number for Malware Patrol).

(venv) root@mx:/var/lib/fangfrisch# fangfrisch --conf /etc/fangfrisch.conf -f refresh
WARNING: https://lists.malwarepatrol.net/cgi/getfile?product=32&receipt=[REDACTED]&list=clamav_basic content length unknown
ERROR: HTTPSConnectionPool(host='www.securiteinfo.com', port=443): Read timed out.
Traceback (most recent call last):
  File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/urllib3/response.py", line 438, in _error_catcher
    yield
  File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/usr/lib/python3.7/http/client.py", line 461, in read
    n = self.readinto(b)
  File "/usr/lib/python3.7/http/client.py", line 505, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.7/ssl.py", line 1052, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.7/ssl.py", line 911, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/requests/models.py", line 758, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/urllib3/response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/urllib3/response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/urllib3/response.py", line 443, in _error_catcher
    raise ReadTimeoutError(self._pool, None, "Read timed out.")
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.securiteinfo.com', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/fangfrisch/refresh.py", line 122, in refresh
    payload = get_payload(ci)
  File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/fangfrisch/download.py", line 81, in get_payload
    return StatusDataPair(True, download.data.content)
  File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/requests/models.py", line 836, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/requests/models.py", line 765, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.securiteinfo.com', port=443): Read timed out.
ERROR: /bin/sh: 1: clamdscan: not found

Here is the conf file I used (Reformatted to txt since .conf isn't supported here).
Fangfrisch.txt

fangfrisch traceback on timeout with clamd

Traceback (most recent call last):
File "/usr/lib/python-exec/python3.9/fangfrisch", line 33, in
sys.exit(load_entry_point('fangfrisch==1.4.0', 'console_scripts', 'fangfrisch')())
File "/usr/lib/python3.9/site-packages/fangfrisch/main.py", line 64, in main
ClamavRefresh(args).refresh_all()
File "/usr/lib/python3.9/site-packages/fangfrisch/refresh.py", line 150, in refresh_all
run_command(command, config.on_update_timeout(),
File "/usr/lib/python3.9/site-packages/fangfrisch/util.py", line 92, in run_command
p: CompletedProcess = run(command, capture_output=True, encoding='utf-8', shell=True, timeout=timeout)
File "/usr/lib/python3.9/subprocess.py", line 507, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/lib/python3.9/subprocess.py", line 1134, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib/python3.9/subprocess.py", line 1980, in _communicate
self._check_timeout(endtime, orig_timeout, stdout, stderr)
File "/usr/lib/python3.9/subprocess.py", line 1178, in _check_timeout
raise TimeoutExpired(
subprocess.TimeoutExpired: Command 'clamdscan --reload' timed out after 30 seconds

privacy

you might want to drop the google stuff?

W: fangfrisch: privacy-breach-generic [] (https://fonts.googleapis.com/css?family=open+sans:300,300italic,400,400italic,600,600italic%7cnoto+serif:400,400italic,700,700italic%7cdroid+sans+mono:400,700) [usr/share/doc/fangfrisch/index.html]

Multiple exceptions on connection failure

When a connection fails, the following appears in the logs for each failed item.

Could this be made less spammy?

Operating System: Arch Linux
Kernel Version: 6.8.4-zen1-1-zen (64-bit)
python 3.11.8
urllib3 1.26.18

ERROR: HTTPConnectionPool(host='mirror.sentries.org', port=80): Max retries exceeded with url: /sanesecurity/badmacro.ndb.sha256 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x745cb54d2fd0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.11/http/client.py", line 1298, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1344, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1293, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.11/http/client.py", line 1052, in _send_output
    self.send(msg)
  File "/usr/lib/python3.11/http/client.py", line 990, in send
    self.connect()
  File "/usr/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x745cb54d2fd0>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='mirror.sentries.org', port=80): Max retries exceeded with url: /sanesecurity/badmacro.ndb.sha256 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x745cb54d2fd0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.11/site-packages/fangfrisch/refresh.py", line 120, in refresh
    digest = get_digest(ci)
             ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/fangfrisch/download.py", line 73, in get_digest
    download = _download(f'{ci.url}.{ci.check}', max_size, ci.connection_timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/fangfrisch/download.py", line 60, in _download
    response = _session.get(url, stream=True, timeout=timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='mirror.sentries.org', port=80): Max retries exceeded with url: /sanesecurity/badmacro.ndb.sha256 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection

Config File Location

Hey this may seem like an obvious thing to most other people but I can't seem to figure out where to create a config file and what to call it. Any advice?

configparser.NoOptionError: No option 'db_url' in section: 'DEFAULT'

Hi, trying to run fangfrisch refresh, I get the error below.
OS: Manjaro, up-to-date
Python: 3.12
installed from AUR via yay

fangfrisch refresh
Traceback (most recent call last):
  File "/usr/lib/python3.12/configparser.py", line 767, in get
    value = d[option]
            ~^^^^^^^^
  File "/usr/lib/python3.12/collections/__init__.py", line 1015, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/collections/__init__.py", line 1007, in __missing__
    raise KeyError(key)
KeyError: 'db_url'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/fangfrisch", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/python3.12/site-packages/fangfrisch/__main__.py", line 65, in main
    DbMeta.assert_version_match()
  File "/usr/lib/python3.12/site-packages/fangfrisch/db.py", line 78, in assert_version_match
    DbMeta.init(False)
  File "/usr/lib/python3.12/site-packages/fangfrisch/db.py", line 63, in init
    db_url = config.db_url()
             ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/fangfrisch/config/config.py", line 96, in db_url
    return self.parser.get(configparser.DEFAULTSECT, DB_URL)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/configparser.py", line 770, in get
    raise NoOptionError(option, section)
configparser.NoOptionError: No option 'db_url' in section: 'DEFAULT'

Check for malformed database files before reloading clamd

When a malformed database is downloaded and installed by fangfrisch, the reload of the signatures in clamd will fail and clamd will exit completely, possibly breaking your mail set up:

Apr 28 17:29:07 hostname clamd[947]: Wed Apr 28 17:29:07 2021 -> Reading databases from /var/lib/clamav
Apr 28 17:29:36 hostname clamd[947]: LibClamAV Error: cli_parseadd(): PCRE subsig mismatched '/' delimiter
Apr 28 17:29:36 hostname clamd[947]: LibClamAV Error: Problem parsing database at line 1456
Apr 28 17:29:36 hostname clamd[947]: LibClamAV Error: Can't load /var/lib/clamav/twinclams.ldb: Malformed database
Apr 28 17:29:36 hostname clamd[947]: LibClamAV Error: cli_loaddbdir(): error loading database /var/lib/clamav/twinclams.ldb
Apr 28 17:29:36 hostname clamd[947]: Wed Apr 28 17:29:36 2021 -> !reload db failed: Malformed database
Apr 28 17:29:38 hostname clamd[947]: Wed Apr 28 17:29:38 2021 -> Terminating because of a fatal error.
Apr 28 17:29:40 hostname systemd[1]: clamav-daemon.service: Main process exited, code=exited, status=1/FAILURE
Apr 28 17:29:40 hostname systemd[1]: clamav-daemon.service: Failed with result 'exit-code'.

Before reloading the database, fangfrisch should check the database with clamscan, and if it fails, it should remove the broken files, before reloading clamd.

For example this command shows that the db is broken:

$ clamscan -d /var/lib/clamav/twinclams.ldb
LibClamAV Error: cli_parseadd(): PCRE subsig mismatched '/' delimiter
LibClamAV Error: Problem parsing database at line 1456
LibClamAV Error: Can't load /var/lib/clamav/twinclams.ldb: Malformed database
ERROR: Malformed database

python 3.11 issue

running with python 3.10 is no problem:

# /usr/lib/python-exec/python3.10/fangfrisch -c /etc/fangfrisch.conf refresh
# no output

but with 3.11 I get this error:

# /usr/lib/python-exec/python3.11/fangfrisch -c /etc/fangfrisch.conf refresh

Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.11/fangfrisch", line 33, in <module>
    sys.exit(load_entry_point('fangfrisch==1.5.0', 'console_scripts', 'fangfrisch')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/fangfrisch/__main__.py", line 64, in main
    ClamavRefresh(args).refresh_all()
  File "/usr/lib/python3.11/site-packages/fangfrisch/refresh.py", line 142, in refresh_all
    if self.refresh(ci):
       ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/fangfrisch/refresh.py", line 112, in refresh
    elif not RefreshLog.is_outdated(ci.url, ci.interval):
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/fangfrisch/db.py", line 144, in is_outdated
    entry: RefreshLog = _query_url(url, RefreshLog._session())
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/fangfrisch/db.py", line 239, in _query_url
    return session.query(RefreshLog).filter(RefreshLog.url == url).first()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/query.py", line 2752, in first
    return self.limit(1)._iter().first()  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/query.py", line 2855, in _iter
    result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
                                                  ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2229, in execute
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2114, in _execute_internal
    conn = self._connection_for_bind(bind)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1981, in _connection_for_bind
    return trans._connection_for_bind(engine, execution_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in _connection_for_bind
  File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 137, in _go
    ret_value = fn(self, *arg, **kw)
                ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1108, in _connection_for_bind
    conn = bind.connect()
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3245, in connect
    return self._connection_cls(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
                             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3269, in raw_connection
    return self.pool.connect()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 455, in connect
    return _ConnectionFairy._checkout(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1270, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 719, in checkout
    rec = pool._do_get()
          ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 157, in _do_get
    raise exc.TimeoutError(
sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)

error when initdb on an existing database

hey, first of all thank you for this software!

I wanted to ask why it is that running initdb on an existing database gives an error ?

The reason it is a bit surprising is that in order to make a systemd service for this, you have to make an "init" oneshot service that will run once to init the db. However if you ever disable and re-enable the service, the init will fail.
To that reason I'm making an init service on nixos that initdb with --force (to make sure it never errors out, that way you can enable and disable fangfrisch at will).
I just wanted to know if there were any considerations I missed.

Thank you again for this!

Python 3.6 optional?

The dependencies declared in the setup state Python 3.7, but I'm wondering if you could relax it slightly to accept Python 3.6 as well, to make upgrade processes easier where Python 3.7 isn't always available yet in the system python.

fangfrisch exception occured

Hey,

just found out about Fangfrisch from the Archwiki (https://wiki.archlinux.org/title/ClamAV#Adding_more_databases/signatures_repositories).

Wanted to test it out in a clean Archlinux install inside KVM (virt-manager).

On Archlinux (up-to-date until few minutes ago, no desktop or something, install few minutes old and nothing installed except installer installs himself) following error is showing:

>> fangfrisch refresh

Traceback (most recent call last):
  File "/usr/lib/python3.10/configparser.py", line 790, in get
    value = d[option]
  File "/usr/lib/python3.10/collections/__init__.py", line 986, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/usr/lib/python3.10/collections/__init__.py", line 978, in __missing__
    raise KeyError(key)
KeyError: 'db_url'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/fangfrisch", line 33, in <module>
    sys.exit(load_entry_point('fangfrisch==1.6.0', 'console_scripts', 'fangfrisch')())
  File "/usr/lib/python3.10/site-packages/fangfrisch/__main__.py", line 65, in main
    DbMeta.assert_version_match()
  File "/usr/lib/python3.10/site-packages/fangfrisch/db.py", line 78, in assert_version_match
    DbMeta.init(False)
  File "/usr/lib/python3.10/site-packages/fangfrisch/db.py", line 63, in init
    db_url = config.db_url()
  File "/usr/lib/python3.10/site-packages/fangfrisch/config/config.py", line 84, in db_url
    return self.parser.get(configparser.DEFAULTSECT, DB_URL)
  File "/usr/lib/python3.10/configparser.py", line 793, in get
    raise NoOptionError(option, section)
configparser.NoOptionError: No option 'db_url' in section: 'DEFAULT'

I tried fangfrisch manualy, because systemctl status fangfrisch.timer stucks here:

systemd[1]: Started Download unofficial clamav virus definition files.

Edit:

Just checked again, now i have following:


INFO: /var/lib/fangfrisch/signatures/blurl.ndb updated (348100 bytes)
INFO: /var/lib/fangfrisch/signatures/bofhland_cracked_URL.ndb updated (3448 bytes)
INFO: /var/lib/fangfrisch/signatures/bofhland_malware_attach.hdb updated (106247 bytes

So the Daemon seems to work. Only to run manualy not.

rseichter / fangfrisch Goto Github PK

fangfrisch's People

Stargazers

Watchers

Forkers

fangfrisch's Issues

Recommend Projects

Recommend Topics

Recommend Org