rseichter / fangfrisch Goto Github PK
View Code? Open in Web Editor NEWUpdate and verify unofficial Clam Anti-Virus signatures
License: GNU General Public License v3.0
Update and verify unofficial Clam Anti-Virus signatures
License: GNU General Public License v3.0
Hello, thank you for working on an elegant update tool for ClamAV! This is awesome!
I am curious if I can get some help, stuck on this step (I'm on Fedora 32 desktop workstation) entered normally as myself (underprivileged user), not under root. i think i installed fangfrisch correctly in earlier steps.
[soffeas@localhost ~]$ pip3.8 install --user fangfrisch
Collecting fangfrisch
Using cached https://files.pythonhosted.org/packages/e7/a8/e3044dbfff8abdf7a63bfa0a026a79cbeb94b99fa8d8bb74ffaa4d6c630b/fangfrisch-1.2.0-py3-none-any.whl
Requirement already satisfied: requests>=2.22.0 in /usr/lib/python3.8/site-packages (from fangfrisch) (2.22.0)
Requirement already satisfied: SQLAlchemy>=1.3.13 in ./.local/lib/python3.8/site-packages (from fangfrisch) (1.3.20)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/lib/python3.8/site-packages (from requests>=2.22.0->fangfrisch) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/lib/python3.8/site-packages (from requests>=2.22.0->fangfrisch) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/lib/python3.8/site-packages (from requests>=2.22.0->fangfrisch) (1.25.7)
Installing collected packages: fangfrisch
Successfully installed fangfrisch-1.2.0
[soffeas@localhost ~]$ fangfrisch
usage: fangfrisch [-h] [-c CONF] [-f] [-p PROVIDER]
{dumpconf,dumpmappings,initdb,refresh}
fangfrisch: error: the following arguments are required: action
[soffeas@localhost ~]$ sudo -u clamav -- fangfrisch --conf /etc/fangfrisch.conf initdb
sudo: fangfrisch: command not found
i can't make sense of this.
help?
In order to avoid relying on Cron email reports, logging via syslog should be supported.
Just got that E-Mail from malwarepatrol and wanted to inform you, just in case you didn't know:
Hello Subscriber,
We hope this message finds you well. We're reaching out to inform you about some upcoming changes to our Risk Indicators / OSINT data feed.
We've recently made improvements to make it easier to parse, enhancing your experience with the data. The content of the OSINT data feed remains unchanged; however, there are some updates regarding the links for two of the three feeds in the set: Risk Indicators and High Risk IPs.
Please note the following key points:
Improved Formatting: We've enhanced the JSON format of the OSINT data feed to ensure better readability and ease of integration into your systems.
New Links: The Risk Indicators and High Risk IPs feeds' new links are already available in the customer portal.
Action Required: We kindly ask you to update your systems or scripts to use the new links for the Risk Indicators and High Risk IPs feeds. The old links will be disabled on March 22, 2024.
TOR Exit Nodes: The link and format for TOR Exit Nodes remain unchanged. You can continue accessing this feed as usual without any interruption.
Should you have any questions or require assistance during this transition, please don't hesitate to reach out to our dedicated support team at [email protected].
We appreciate your continued trust and support as we strive to provide you with the best possible service.
Best regards,
Malware Patrol Team
The v3.7.3 Script runs well but sometimes (for ex. today, Fri, 27 Nov 2020 10:45:03 ) I get odd messages in the log file / status mail like:
WARNING: https://urlhaus.abuse.ch/downloads/urlhaus.ndb sha256 digest mismatch (expected fc554db6a3a6d2fee4056a0b80d4724a4701dc100f7034c80c75c41ce382963f, got 8ff4b8d6c964ed579bdba6fc1931382daed2d567d70e7e26f20b4d240abd1797)
Seems to be due to urlhaus, but I don't necessarily want to switch off the checksum checks completely.
So, any hint's?
As in #8 the Problem already exist.
As mention in malware.expert.fp ab1762c69439bdd1d6381726e832544f:37:buy.subscriotion.from.malware.expert for this you need a subscription.
Now i want to disable the signatures.
url_malwareexpert_fp = /dev/null
url_malwareexpert_hdb = /dev/null
url_malwareexpert_ldb = /dev/null
url_malwareexpert_ndb = /dev/null
will probably work but brings error in syslog
Feb 8 15:15:00 x fangfrisch[8220]: Invalid URL '/dev/null.sha256': No schema supplied. Perhaps you meant http:///dev/null.sha256?
Feb 8 15:15:00 x Traceback (most recent call last):
Feb 8 15:15:00 x File "/usr/lib/python3.8/site-packages/fangfrisch/refresh.py", line 106, in refresh
Feb 8 15:15:00 x digest = get_digest(ci)
Feb 8 15:15:00 x File "/usr/lib/python3.8/site-packages/fangfrisch/download.py", line 70, in get_digest
Feb 8 15:15:00 x download = _download(f'{ci.url}.{ci.check}', max_size)
Feb 8 15:15:00 x File "/usr/lib/python3.8/site-packages/fangfrisch/download.py", line 57, in _download
Feb 8 15:15:00 x response = _session.get(url, stream=True, timeout=30)
Feb 8 15:15:00 x File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
Feb 8 15:15:00 x return self.request('GET', url, **kwargs)
Feb 8 15:15:00 x File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 528, in request
Feb 8 15:15:00 x prep = self.prepare_request(req)
Feb 8 15:15:00 x File "/usr/lib/python3.8/site-packages/requests/sessions.py", line 456, in prepare_request
Feb 8 15:15:00 x p.prepare(
Feb 8 15:15:00 x File "/usr/lib/python3.8/site-packages/requests/models.py", line 316, in prepare
Feb 8 15:15:00 x self.prepare_url(url, params)
Feb 8 15:15:00 x File "/usr/lib/python3.8/site-packages/requests/models.py", line 390, in prepare_url
Feb 8 15:15:00 x raise MissingSchema(error)
Feb 8 15:15:00 x requests.exceptions.MissingSchema: Invalid URL '/dev/null.sha256': No schema supplied. Perhaps you meant http:///dev/null.sha256?
Let us say I am using two sources for signatures - sanesecurity and urlhaus.
Tomorrow I decide that sanesecurity signature are not required or are to be disabled temporarily because there is some issue with them.
So I remove sanesecurity from fangfrisch.conf but old signatures will still remain in /var/lib/clamav
and will continue to be used by clamav.
So can you provide a feature where I run fangfrisch cleanunusedsigs
and it cleans up files related to unused sources from /var/lib/clamav
It will check which data sources are enabled and delete files related to other databases which are not enabled. (ofcourse it should not delete other signature files which it has not downloaded but are there in /var/lib/clamav
)
The Python build is currently lacking both pyproject.toml
and setup.cfg
(see Python Packaging User Guide and PEP 517). This needs to be addressed.
Is it possible to specify only the data sources you want from a feed? I see you can disable datasources but I want to specify only the data sources I want. For instance, for Sanesecurity I want to only enable the ones with Low FP score.
Thanks
The option on_update_exec
is currently only permitted in the configuration's DEFAULT section. It would be useful to be able to override this option within provider sections.
Currently, the docs use yes/false
for the enabled option. Of cource, this is valid, but using one of the usual pairs yes/no
or true/false
is less surprising.
Hi,
thank you for this project an your time!
Could it be that it is not supported on ubuntu?
mx01 fangfrisch # python3 -m venv venv
mx01 fangfrisch # source venv/bin/activate
(venv) mx01 fangfrisch # pip3 install fangfrisch
Collecting fangfrisch
Could not find a version that satisfies the requirement fangfrisch (from versions: )
No matching distribution found for fangfrisch
regards
Celevra
fangfrisch-1.9.0
Please provide an option to locally configure a specific proxy server for fangfrisch passing to the requests library, if a global HTTPS_PROXY in the environment is not an option.
Hello, seeing that you're also the maintainer on Gentoo, i was wondering if it might be possible to improve the setup on there. I've not tested Arch which also seem to have a package. Maybe it's similar there.
Fangfrisch was suggested as a new alternative to clamav-unofficial-sigs but when installing it, it lacks of a number of defaults that would usually be provided from other packages. I got it working but migrating a number of machines to it, some would have to rerun the commands that would simply set documented defaults.
The package does not create a /var/lib/fangfrisch directory when it is essential to have it. Even if you're not going to add the creation of such a directory to the Ebuild, it might be a good idea to add the initial commands to the Ebuild info instead of the link to your homepage. As a new user, i was forced to go to your website just to find 2 or 3 commands that i needed to enter.
The other thing i wonder about is the lack of a default config file location, when the file is already placed there. Wouldn't it be easier if Fangfrisch use the default but keeps the -c
option for those using another location? That way you could simply run fangfrisch initdb
without the need of providing the default config, that was placed to /etc/fangfrisch.conf
during install.
I have recently started trying fangfrisch as a replacement for the old clamav-unofficial-sigs package, but now I have serious doubts about its viability.
That script only uses rsync and, while rsync.sanesecurity.net
has an healthy list of IPs, I cannot find a list of official HTTP mirrors for sanesecurity.
The default configuration uses an australian HTTP mirror which is seriously overloaded and rarely responds in less that 30 seconds. Also, this is not a sensible default because the bandwidth costs for Australia are very high.
Also, I see that the default configuration still downloads files like malware.expert.*
and phishtank.ndb
, which have been empty for years.
I'm unable to fetch the securiteinfo using a paid account. All files are ending with .yes
extension and trigger an 403 forbidden
error. Removing the yes
extension, works.
Who is adding that ?
# sudo -u clamav -- /var/lib/fangfrisch/venv/bin/fangfrisch --conf /etc/fangfrisch/fangfrisch.conf refresh
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfoandroid.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfoascii.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfohtml.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/javascript.ndb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfopdf.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfo.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfo.ign2.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfo0hour.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfo.mdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfoold.hdb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/spam_marketing.ndb.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfo.yara.yes download failed: 403 Forbidden
ERROR: https://www.securiteinfo.com/get/signatures/xxxxxx/securiteinfo.pdb.yes download failed: 403 Forbidden
# cat /etc/fangfrisch/fangfrisch.conf
[DEFAULT]
db_url = sqlite:////var/lib/fangfrisch/db.sqlite
local_directory = /var/lib/clamav
on_update_exec = clamdscan --reload
on_update_timeout = 42
[sanesecurity]
enabled = yes
[securiteinfo]
enabled = yes
integrity_check = yes
customer_id = xxxxxx
url_0hour = ${prefix}securiteinfo0hour.hdb
url_securiteinfo_mdb = ${prefix}securiteinfo.mdb
url_old = ${prefix}securiteinfoold.hdb
url_spam_marketing = ${prefix}spam_marketing.ndb
url_yara = ${prefix}securiteinfo.yara
url_pdb = ${prefix}securiteinfo.pdb
[urlhaus]
interval = 1h
enabled = yes
max_size = 2MB
[twinwave]
enabled = yes
integrity_check = disabled
interval = 1h
prefix = https://raw.githubusercontent.com/twinwave-security/twinclams/master/
url_twinclams = ${prefix}twinclams.ldb
url_twinwave_ign2 = ${prefix}twinwave.ign2
[clampunch]
enabled = yes
max_size = 2M
integrity_check = disabled
interval = 24h
prefix = https://raw.githubusercontent.com/wmetcalf/clam-punch/master/
url_miscreantpunch099low = ${prefix}MiscreantPunch099-Low.ldb
url_exexor99 = ${prefix}exexor99.ldb
url_miscreantpuchhdb = ${prefix}miscreantpunch.hdb
[rfxn]
enabled = yes
interval= 4h
integrity_check = disabled
prefix = https://www.rfxn.com/downloads/
url_rfxn_ndb = ${prefix}rfxn.ndb
url_rfxn_hdb = ${prefix}rfxn.hdb
url_rfxn_yara = ${prefix}rfxn.yara
[interserver]
enabled = yes
interval = 1d
integrity_check = disabled
prefix = https://rbldata.interserver.net/
url_interserver_sha256 = ${prefix}interserver256.hdb
url_interserver_topline = ${prefix}interservertopline.db
url_interserver_shell = ${prefix}shell.ldb
url_interserver_whitelist = ${prefix}whitelist.fp
[ditekshen]
enabled = yes
interval = 1d
integrity_check = disabled
prefix = https://raw.githubusercontent.com/ditekshen/detection/master/clamav/
url_ditekshen_ldb = ${prefix}clamav.ldb
filename_ditekshen_ldb = ditekshen.ldb
Hello!
first of all, thank you for this work!
While trying to integrate this tool with IntelOwl, I found a problem that always happen when I launch fangfrisch refresh
intelowl_malware_tools_analyzers | INFO: /var/lib/clamav/junk.ndb updated (6989248 bytes)
intelowl_malware_tools_analyzers | INFO: /var/lib/clamav/jurlbl.ndb updated (830565 bytes)
intelowl_malware_tools_analyzers | INFO: /var/lib/clamav/jurlbla.ndb updated (88784 bytes)
intelowl_malware_tools_analyzers | Traceback (most recent call last):
intelowl_malware_tools_analyzers | File "/usr/local/bin/fangfrisch", line 8, in <module>
intelowl_malware_tools_analyzers | sys.exit(main())
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/fangfrisch/__main__.py", line 64, in main
intelowl_malware_tools_analyzers | ClamavRefresh(args).refresh_all()
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/fangfrisch/refresh.py", line 142, in refresh_all
intelowl_malware_tools_analyzers | if self.refresh(ci):
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/fangfrisch/refresh.py", line 134, in refresh
intelowl_malware_tools_analyzers | RefreshLog.update(ci, digest.data)
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/fangfrisch/db.py", line 190, in update
intelowl_malware_tools_analyzers | entry: RefreshLog = _query_url(ci.url, session)
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/fangfrisch/db.py", line 239, in _query_url
intelowl_malware_tools_analyzers | return session.query(RefreshLog).filter(RefreshLog.url == url).first()
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2752, in first
intelowl_malware_tools_analyzers | return self.limit(1)._iter().first() # type: ignore
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2855, in _iter
intelowl_malware_tools_analyzers | result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2229, in execute
intelowl_malware_tools_analyzers | return self._execute_internal(
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 2114, in _execute_internal
intelowl_malware_tools_analyzers | conn = self._connection_for_bind(bind)
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1981, in _connection_for_bind
intelowl_malware_tools_analyzers | return trans._connection_for_bind(engine, execution_options)
intelowl_malware_tools_analyzers | File "<string>", line 2, in _connection_for_bind
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/state_changes.py", line 137, in _go
intelowl_malware_tools_analyzers | ret_value = fn(self, *arg, **kw)
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1108, in _connection_for_bind
intelowl_malware_tools_analyzers | conn = bind.connect()
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3245, in connect
intelowl_malware_tools_analyzers | return self._connection_cls(self)
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
intelowl_malware_tools_analyzers | self._dbapi_connection = engine.raw_connection()
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3269, in raw_connection
intelowl_malware_tools_analyzers | return self.pool.connect()
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 455, in connect
intelowl_malware_tools_analyzers | return _ConnectionFairy._checkout(self)
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1270, in _checkout
intelowl_malware_tools_analyzers | fairy = _ConnectionRecord.checkout(pool)
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 719, in checkout
intelowl_malware_tools_analyzers | rec = pool._do_get()
intelowl_malware_tools_analyzers | File "/usr/local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 157, in _do_get
intelowl_malware_tools_analyzers | raise exc.TimeoutError(
intelowl_malware_tools_analyzers | sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)
At one point, during download, SQLAlchemy breaks.
I think that by incrementing the pool_size
and max_overflow
values (see doc) it would fix the problem.
I can open a really little PR for this if you like
I was looking into enabling the frangfrisch test suite to run during package build and in the Debian CI infrastructure, but quickly ran into issues.
During package build, there is no internet access (to make sure that the Debian archive is self-contained). As a result, any tests that require internet access to run, will fail. This is not a huge issue as it's common that really testing things requires doing something on a network and mocking the required network connections is non-trivial.
In our CI environment, tests are run periodically, so multiple connects from a single host are normal. Low thresholds for blocking access make running the tests in CI unreliable.
In the course of trying to enable the tests, I quickly started getting errors like:
requests.exceptions.ProxyError: HTTPSConnectionPool(host='seichter.de', port=443): Max retries exceeded with url: /favicon-16x16.png (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f7d1992a690>: Failed to establish a new connection: [Errno 111] Connection refused')))
I don't know what would be the best solution to this, but currently, for Debian, running these tests is not feasible.
Scott K
how is this different from clamav-unofficial-sigs ?
anyways, enjoy: http://sid.ethz.ch/debian/fangfrisch/
Any setup guide for this?
Connection timeouts should be configurable, at least at the data provider level and possibly at URL level. The current, hard coded timeout is not flexible enough to handle known slow connections. The new feature must take scheduling considerations into account, e.g. avoiding problems due to multiple Fangfrisch instances running concurrently.
Hi there,
Is there was a simple way to pass the db_url connection string via an environment variable? Right now, our database isn't located on the same machine as fangfrisch so we have password protection enabled for the database user. The issue is that it seems like db_url in the configuration file requires a hardcoded string that would include the database password being written to disk.
Would it be possible to just read the default db_url to also read from FANGFRISCH_DB_URL or more generally, allow us to configure which environment variable the any configuration should be read from?
I dug through the source code and all the issues but didn't anything about environment variables generally being supported.
Thanks for the help!
Coming from hvisage/metalog#32 ...
Setting log_method to syslog doesn't log anything to metalog:
# grep log /etc/fangfrisch.conf
log_level = DEBUG
log_method = syslog
Removing the log_method and falling back to "console", the logging to STDOUT works as expected.
Gentoo Linux 23.0
app-antivirus/fangfrisch-1.9.0
app-antivirus/clamav-1.2.2
dev-lang/python-3.12.3-r1
app-admin/metalog-20230719
Yo,
I'm seeing alot of these when updating Sanesecurity:
ERROR: HTTPConnectionPool(host='ftp.swin.edu.au', port=80): Read timed out. (read timeout=30)
I tested with a browser and it is slow as molasses, coming from Deutsche Telekom AS3320. It works, but is really slow. I checked traceroute for IPv6 and that looks very nasty. Problem has persisted for months I just ignored it so far.
I fixed it temporarily by hot-modifying timeout=30 here: https://github.com/rseichter/fangfrisch/blob/master/fangfrisch/download.py#L57 I set it to 120 seconds and the error is gone.
Any chance to make this configurable? Or some other option?
(Edit: And bump version number so Arch picks it up...)
Thanks, Stephan
The sanesecurity provider downloads scanmnailer.ndb. However this is unmaintained and its author discourages its use:
http://www.scamnailer.info/
I would like to disable this source on my installation, without disabling the complete sanesecurity provider. I could create a custom sanesecurity provider without the scanmailer source, but an easier way would be helpful.
I use MX Linux and am working on a hardened respin.
I installed Fangfrisch, following the instructions as written up to here https://rseichter.github.io/fangfrisch/#dbsetup in a root shell.
Running this command sudo -u clamav -- fangfrisch --conf /etc/fangfrisch.conf initdb
(even when I set my user profile to the clamav usergroup) only got me this error.
sudo: fangfrisch: command not found
So I was forced to run Fangfrisch, using the initdb and refresh commands as root.
When I tried to refresh the database, here's the output I got (minus the receipt number for Malware Patrol).
(venv) root@mx:/var/lib/fangfrisch# fangfrisch --conf /etc/fangfrisch.conf -f refresh
WARNING: https://lists.malwarepatrol.net/cgi/getfile?product=32&receipt=[REDACTED]&list=clamav_basic content length unknown
ERROR: HTTPSConnectionPool(host='www.securiteinfo.com', port=443): Read timed out.
Traceback (most recent call last):
File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/urllib3/response.py", line 438, in _error_catcher
yield
File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/urllib3/response.py", line 519, in read
data = self._fp.read(amt) if not fp_closed else b""
File "/usr/lib/python3.7/http/client.py", line 461, in read
n = self.readinto(b)
File "/usr/lib/python3.7/http/client.py", line 505, in readinto
n = self.fp.readinto(b)
File "/usr/lib/python3.7/socket.py", line 589, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.7/ssl.py", line 1052, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.7/ssl.py", line 911, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/requests/models.py", line 758, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/urllib3/response.py", line 576, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/urllib3/response.py", line 541, in read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/urllib3/response.py", line 443, in _error_catcher
raise ReadTimeoutError(self._pool, None, "Read timed out.")
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.securiteinfo.com', port=443): Read timed out.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/fangfrisch/refresh.py", line 122, in refresh
payload = get_payload(ci)
File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/fangfrisch/download.py", line 81, in get_payload
return StatusDataPair(True, download.data.content)
File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/requests/models.py", line 836, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
File "/var/lib/fangfrisch/venv/lib/python3.7/site-packages/requests/models.py", line 765, in generate
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.securiteinfo.com', port=443): Read timed out.
ERROR: /bin/sh: 1: clamdscan: not found
Here is the conf file I used (Reformatted to txt since .conf isn't supported here).
Fangfrisch.txt
Traceback (most recent call last):
File "/usr/lib/python-exec/python3.9/fangfrisch", line 33, in
sys.exit(load_entry_point('fangfrisch==1.4.0', 'console_scripts', 'fangfrisch')())
File "/usr/lib/python3.9/site-packages/fangfrisch/main.py", line 64, in main
ClamavRefresh(args).refresh_all()
File "/usr/lib/python3.9/site-packages/fangfrisch/refresh.py", line 150, in refresh_all
run_command(command, config.on_update_timeout(),
File "/usr/lib/python3.9/site-packages/fangfrisch/util.py", line 92, in run_command
p: CompletedProcess = run(command, capture_output=True, encoding='utf-8', shell=True, timeout=timeout)
File "/usr/lib/python3.9/subprocess.py", line 507, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/lib/python3.9/subprocess.py", line 1134, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib/python3.9/subprocess.py", line 1980, in _communicate
self._check_timeout(endtime, orig_timeout, stdout, stderr)
File "/usr/lib/python3.9/subprocess.py", line 1178, in _check_timeout
raise TimeoutExpired(
subprocess.TimeoutExpired: Command 'clamdscan --reload' timed out after 30 seconds
you might want to drop the google stuff?
W: fangfrisch: privacy-breach-generic [] (https://fonts.googleapis.com/css?family=open+sans:300,300italic,400,400italic,600,600italic%7cnoto+serif:400,400italic,700,700italic%7cdroid+sans+mono:400,700) [usr/share/doc/fangfrisch/index.html]
When a connection fails, the following appears in the logs for each failed item.
Could this be made less spammy?
Operating System: Arch Linux
Kernel Version: 6.8.4-zen1-1-zen (64-bit)
python 3.11.8
urllib3 1.26.18
ERROR: HTTPConnectionPool(host='mirror.sentries.org', port=80): Max retries exceeded with url: /sanesecurity/badmacro.ndb.sha256 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x745cb54d2fd0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/urllib3/util/connection.py", line 72, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/socket.py", line 962, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/urllib3/connectionpool.py", line 416, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/lib/python3.11/http/client.py", line 1298, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1344, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1293, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.11/http/client.py", line 1052, in _send_output
self.send(msg)
File "/usr/lib/python3.11/http/client.py", line 990, in send
self.connect()
File "/usr/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x745cb54d2fd0>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/urllib3/connectionpool.py", line 799, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='mirror.sentries.org', port=80): Max retries exceeded with url: /sanesecurity/badmacro.ndb.sha256 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x745cb54d2fd0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/fangfrisch/refresh.py", line 120, in refresh
digest = get_digest(ci)
^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/fangfrisch/download.py", line 73, in get_digest
download = _download(f'{ci.url}.{ci.check}', max_size, ci.connection_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/fangfrisch/download.py", line 60, in _download
response = _session.get(url, stream=True, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='mirror.sentries.org', port=80): Max retries exceeded with url: /sanesecurity/badmacro.ndb.sha256 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection
Hey this may seem like an obvious thing to most other people but I can't seem to figure out where to create a config file and what to call it. Any advice?
Hi, trying to run fangfrisch refresh
, I get the error below.
OS: Manjaro, up-to-date
Python: 3.12
installed from AUR via yay
fangfrisch refresh
Traceback (most recent call last):
File "/usr/lib/python3.12/configparser.py", line 767, in get
value = d[option]
~^^^^^^^^
File "/usr/lib/python3.12/collections/__init__.py", line 1015, in __getitem__
return self.__missing__(key) # support subclasses that define __missing__
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/collections/__init__.py", line 1007, in __missing__
raise KeyError(key)
KeyError: 'db_url'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/fangfrisch", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/lib/python3.12/site-packages/fangfrisch/__main__.py", line 65, in main
DbMeta.assert_version_match()
File "/usr/lib/python3.12/site-packages/fangfrisch/db.py", line 78, in assert_version_match
DbMeta.init(False)
File "/usr/lib/python3.12/site-packages/fangfrisch/db.py", line 63, in init
db_url = config.db_url()
^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/site-packages/fangfrisch/config/config.py", line 96, in db_url
return self.parser.get(configparser.DEFAULTSECT, DB_URL)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/configparser.py", line 770, in get
raise NoOptionError(option, section)
configparser.NoOptionError: No option 'db_url' in section: 'DEFAULT'
When a malformed database is downloaded and installed by fangfrisch, the reload of the signatures in clamd will fail and clamd will exit completely, possibly breaking your mail set up:
Apr 28 17:29:07 hostname clamd[947]: Wed Apr 28 17:29:07 2021 -> Reading databases from /var/lib/clamav
Apr 28 17:29:36 hostname clamd[947]: LibClamAV Error: cli_parseadd(): PCRE subsig mismatched '/' delimiter
Apr 28 17:29:36 hostname clamd[947]: LibClamAV Error: Problem parsing database at line 1456
Apr 28 17:29:36 hostname clamd[947]: LibClamAV Error: Can't load /var/lib/clamav/twinclams.ldb: Malformed database
Apr 28 17:29:36 hostname clamd[947]: LibClamAV Error: cli_loaddbdir(): error loading database /var/lib/clamav/twinclams.ldb
Apr 28 17:29:36 hostname clamd[947]: Wed Apr 28 17:29:36 2021 -> !reload db failed: Malformed database
Apr 28 17:29:38 hostname clamd[947]: Wed Apr 28 17:29:38 2021 -> Terminating because of a fatal error.
Apr 28 17:29:40 hostname systemd[1]: clamav-daemon.service: Main process exited, code=exited, status=1/FAILURE
Apr 28 17:29:40 hostname systemd[1]: clamav-daemon.service: Failed with result 'exit-code'.
Before reloading the database, fangfrisch should check the database with clamscan, and if it fails, it should remove the broken files, before reloading clamd.
For example this command shows that the db is broken:
$ clamscan -d /var/lib/clamav/twinclams.ldb
LibClamAV Error: cli_parseadd(): PCRE subsig mismatched '/' delimiter
LibClamAV Error: Problem parsing database at line 1456
LibClamAV Error: Can't load /var/lib/clamav/twinclams.ldb: Malformed database
ERROR: Malformed database
running with python 3.10 is no problem:
# /usr/lib/python-exec/python3.10/fangfrisch -c /etc/fangfrisch.conf refresh
# no output
but with 3.11 I get this error:
# /usr/lib/python-exec/python3.11/fangfrisch -c /etc/fangfrisch.conf refresh
Traceback (most recent call last):
File "/usr/lib/python-exec/python3.11/fangfrisch", line 33, in <module>
sys.exit(load_entry_point('fangfrisch==1.5.0', 'console_scripts', 'fangfrisch')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/fangfrisch/__main__.py", line 64, in main
ClamavRefresh(args).refresh_all()
File "/usr/lib/python3.11/site-packages/fangfrisch/refresh.py", line 142, in refresh_all
if self.refresh(ci):
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/fangfrisch/refresh.py", line 112, in refresh
elif not RefreshLog.is_outdated(ci.url, ci.interval):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/fangfrisch/db.py", line 144, in is_outdated
entry: RefreshLog = _query_url(url, RefreshLog._session())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/fangfrisch/db.py", line 239, in _query_url
return session.query(RefreshLog).filter(RefreshLog.url == url).first()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/query.py", line 2752, in first
return self.limit(1)._iter().first() # type: ignore
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/query.py", line 2855, in _iter
result: Union[ScalarResult[_T], Result[_T]] = self.session.execute(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2229, in execute
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2114, in _execute_internal
conn = self._connection_for_bind(bind)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1981, in _connection_for_bind
return trans._connection_for_bind(engine, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<string>", line 2, in _connection_for_bind
File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 137, in _go
ret_value = fn(self, *arg, **kw)
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1108, in _connection_for_bind
conn = bind.connect()
^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3245, in connect
return self._connection_cls(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
self._dbapi_connection = engine.raw_connection()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3269, in raw_connection
return self.pool.connect()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 455, in connect
return _ConnectionFairy._checkout(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1270, in _checkout
fairy = _ConnectionRecord.checkout(pool)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 719, in checkout
rec = pool._do_get()
^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 157, in _do_get
raise exc.TimeoutError(
sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30.00 (Background on this error at: https://sqlalche.me/e/20/3o7r)
hey, first of all thank you for this software!
I wanted to ask why it is that running initdb on an existing database gives an error ?
The reason it is a bit surprising is that in order to make a systemd service for this, you have to make an "init" oneshot service that will run once to init the db. However if you ever disable and re-enable the service, the init will fail.
To that reason I'm making an init service on nixos that initdb with --force (to make sure it never errors out, that way you can enable and disable fangfrisch at will).
I just wanted to know if there were any considerations I missed.
Thank you again for this!
The dependencies declared in the setup state Python 3.7, but I'm wondering if you could relax it slightly to accept Python 3.6 as well, to make upgrade processes easier where Python 3.7 isn't always available yet in the system python.
Hey,
just found out about Fangfrisch from the Archwiki (https://wiki.archlinux.org/title/ClamAV#Adding_more_databases/signatures_repositories).
Wanted to test it out in a clean Archlinux install inside KVM (virt-manager).
On Archlinux (up-to-date until few minutes ago, no desktop or something, install few minutes old and nothing installed except installer installs himself) following error is showing:
>> fangfrisch refresh
Traceback (most recent call last):
File "/usr/lib/python3.10/configparser.py", line 790, in get
value = d[option]
File "/usr/lib/python3.10/collections/__init__.py", line 986, in __getitem__
return self.__missing__(key) # support subclasses that define __missing__
File "/usr/lib/python3.10/collections/__init__.py", line 978, in __missing__
raise KeyError(key)
KeyError: 'db_url'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/fangfrisch", line 33, in <module>
sys.exit(load_entry_point('fangfrisch==1.6.0', 'console_scripts', 'fangfrisch')())
File "/usr/lib/python3.10/site-packages/fangfrisch/__main__.py", line 65, in main
DbMeta.assert_version_match()
File "/usr/lib/python3.10/site-packages/fangfrisch/db.py", line 78, in assert_version_match
DbMeta.init(False)
File "/usr/lib/python3.10/site-packages/fangfrisch/db.py", line 63, in init
db_url = config.db_url()
File "/usr/lib/python3.10/site-packages/fangfrisch/config/config.py", line 84, in db_url
return self.parser.get(configparser.DEFAULTSECT, DB_URL)
File "/usr/lib/python3.10/configparser.py", line 793, in get
raise NoOptionError(option, section)
configparser.NoOptionError: No option 'db_url' in section: 'DEFAULT'
I tried fangfrisch manualy, because systemctl status fangfrisch.timer stucks here:
systemd[1]: Started Download unofficial clamav virus definition files.
Edit:
Just checked again, now i have following:
INFO: /var/lib/fangfrisch/signatures/blurl.ndb updated (348100 bytes)
INFO: /var/lib/fangfrisch/signatures/bofhland_cracked_URL.ndb updated (3448 bytes)
INFO: /var/lib/fangfrisch/signatures/bofhland_malware_attach.hdb updated (106247 bytes
So the Daemon seems to work. Only to run manualy not.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.