Comments (10)
funilrys
commented on May 25, 2024
1
Hi @spirillen.
I will look into this later.
On my test and dev machine, everything is working with a local MariaDB instance. I never used MySQL myself. But will try to find out how it goes with MySQL.
Cheers.
P.S: You may ask yourself: Why is MySQL available then? Well, @mitchellkrogza machines were using MySQL and I found it challenging to implement both 😹
from pyfunceble.
funilrys
commented on May 25, 2024
Hi @spirillen, Thanks for the feedback!
Can you create a PR? 😄
I'm currently working in a branch which is incompatible with the current state of the dev branch so I can't handle that directly. However, I'll be glad to merge your PR to the dev branch.
Do you mind adding the missing ; at the following lines into a PR to the dev branch?
Line 189 in 971e141
Line 217 in 971e141
Once your PR will be merged, into the dev branch the fix will be available around the world in the next 3 minutes as it's a file that is downloaded and not shipped with the PyFunceble package.
Thanks again.
Cheers,
Nissar
from pyfunceble.
spirillen
commented on May 25, 2024
Hi @funilrys Of curse :=)
I'll be on it later, I'm about adding some s%&(¤rs 😨 to my blacklists :)
from pyfunceble.
funilrys
commented on May 25, 2024
Please DO NOT close this issue manually. This issue is going to be closed automatically once 2009ab9 will be merged to master.
from pyfunceble.
spirillen
commented on May 25, 2024
Hey @funilrys have you seen this https://mariadb.com/kb/en/library/authentication-from-mariadb-104/ it also goes for oracles mysql 5.7.
It seems like we need to first generate a user over the socket and then run the sql import.....
OR
Have the sql script to simply connect over the unix:socket which also is faster...
It took a loot of frustration before i found the post to see why the mysql|mariadb didn't work as it keep screaming an sql error on line 1
After finally getting Travis-ci to run properly with mysql, then both mysql and mariadb returns errors on line 1
But the server and workstation, it just runs as supposed to??
Feel free to play around on https://github.com/spirillen/Dead-Domains I add you to the dev.
from pyfunceble.
spirillen
commented on May 25, 2024
Hi hate mysql myself, but it was the "straight up" choice for the test with Travis-CI 😈 and it tolk me some time time to figure out that travis don't start mariadb after install
But I was a little fast in my previous post about the workstation.... just got this error after testing 5 - 6 times mariadb 10.4
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/PyFunceble/file_multiprocess_core.py", line 96, in run
Process.run(self)
File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.7/site-packages/PyFunceble/file_core.py", line 488, in _test_line
status = self.__process_test(subject)
File "/usr/local/lib/python3.7/site-packages/PyFunceble/file_core.py", line 374, in __process_test
return self.domain(domain2idna(subject))
File "/usr/local/lib/python3.7/site-packages/PyFunceble/file_core.py", line 222, in domain
inactive_db=self.inactive_db,
File "/usr/local/lib/python3.7/site-packages/PyFunceble/status.py", line 144, in get
whois_db=self.whois_db,
File "/usr/local/lib/python3.7/site-packages/PyFunceble/expiration_date.py", line 155, in get
self._extract()
File "/usr/local/lib/python3.7/site-packages/PyFunceble/expiration_date.py", line 476, in _extract
self.__extract_from_record()
File "/usr/local/lib/python3.7/site-packages/PyFunceble/expiration_date.py", line 449, in __extract_from_record
self.subject, self.expiration_date, self.whois_record
File "/usr/local/lib/python3.7/site-packages/PyFunceble/whois_db.py", line 423, in add
self[subject] = data
File "/usr/local/lib/python3.7/site-packages/PyFunceble/whois_db.py", line 245, in __setitem__
self.__setitem_mysql(index, value)
File "/usr/local/lib/python3.7/site-packages/PyFunceble/whois_db.py", line 232, in __setitem_mysql
"digest": digest,
File "/usr/local/lib/python3.7/site-packages/pymysql/cursors.py", line 170, in execute
result = self._query(query)
File "/usr/local/lib/python3.7/site-packages/pymysql/cursors.py", line 328, in _query
conn.query(q)
File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 517, in query
self._affected_rows = self._read_query_result(unbuffered=unbuffered)
File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 732, in _read_query_result
result.read()
File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 1075, in read
first_packet = self.connection._read_packet()
File "/usr/local/lib/python3.7/site-packages/pymysql/connections.py", line 684, in _read_packet
packet.check_error()
File "/usr/local/lib/python3.7/site-packages/pymysql/protocol.py", line 220, in check_error
err.raise_mysql_exception(self._data)
File "/usr/local/lib/python3.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.InternalError: (1366, "Incorrect string value: '\\xE9\\x99\\x88\\xE4\\xBA\\x9A...' for column `PyFunceble`.`pyfunceble_whois`.`record` at row 1")
Test command
pyfunceble -h -ex -dbr 1 --plain --idna -db --database-type mariadb -m -p 4 -f hosts
Source file:
wget -q https://gitlab.com/my-privacy-dns/matrix/matrix/raw/master/source/adware/wildcard.list -O hosts
By the way, how was the October fest in München? 🍻 Mit leaderhosen und allez
from pyfunceble.
spirillen
commented on May 25, 2024
Third note:
On my dual CPU 8 core server, I periodically see this error popping up, but scripts continues
Just to help tracking the error(s). Same command as before just with -p 8
/usr/local/lib/python3.6/dist-packages/pymysql/cursors.py:329: Warning: (1366, "Incorrect string value: '\\xEF\\xBC\\x8C en...' for column `PyFunceble`.`pyfunceble_whois`.`record` at row 1")
self._do_get_result()
from pyfunceble.
funilrys
commented on May 25, 2024
@spirillen Oktoberfest is always too short 😉
I tried with some domains which should actually raise such warning but I got nothing 🤔 Is it another MySQL issue ?
Probably a wrong charset used 🤔
from pyfunceble.
spirillen
commented on May 25, 2024
As I just dropped a note in the commit. I believe, or it feels like, the way travis is set up to work, it is not following the normal path as described in the docs for MariaDB, and that's why I had to make that work-around to get the user access from a non-socket connection.
So a thought: Why not make Pyf connect to sql over the unix:socket as it usually is faster anyway?
from pyfunceble.
funilrys
commented on May 25, 2024
Look https://github.com/spirillen/Dead-Domains/commit/58b1d40f3d32c27df9ea7a8667dd1c231098b8c8 😉
So a thought: Why not make Pyf connect to sql over the unix:socket as it usually is faster anyway?
That can also be possible 🤔 Will have to think about it!
from pyfunceble.
Related Issues (20)
- BUG: https://secure.fanboy.co.nz/fanboy-cookiemonster_ubo.txt unable to scan list HOT 8
- tests with errors HOT 11
- FEATURE: Preload/Continue like the CI workflow ... but without Git HOT 2
- BUG: Cant install latest version with pip HOT 2
- BUG: urls in domain lists.... HOT 2
- FEATURE: Special Rules for forumactif.com HOT 1
- Contribution Tracking
- DOC: Moving away from restructured text HOT 8
- BUG: URL in file header should be changed
- BUG: dead domain query HOT 9
- BUG: sqlalche braekes after finishing...
- FEATURE: Reputation filter using the proxy connection HOT 3
- FEATURE: Sharing WHOIS
- FEATURE: meilisearch or redis support vs RamDrive
- BUG: TypeError(f"<data> should be {dict}, {type(data)} given.") HOT 5
- BUG: log file not created... HOT 2
- Special Rules, are they working as expected? HOT 2
- Unstable special rules HOT 1
- BUG: Object of type datetime is not JSON serializable HOT 2
- FEATURE: Timestamp in CLI output
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyfunceble.