Comments (6)
funilrys
commented on May 25, 2024
Hi @jawz101,
Unfortunately, I can't reproduce on my personal machine or a clean raspberry pi 🤔
What I did:
-
On all machine, always clean
/etc/hostsas it may contain @Ultimate-Hosts-Blacklist, @dead-hosts members or other parasites. -
Launch the following on my test machine (fresh Arch Linux which is always reinstalled every x months)
-
Launch the following (for safety/check in our case) on a raspberry pi which is also configured and used as a test environment for PyFunceble.
-
Create a file called issue-19 with the following content
a4.tl apptrk.a4.tl els.a4.tl jrs.a4.tl ldap.a4.tl preroll.a4.tl sdk.a4.tl adform.net adformdsp.net adkmob.com bp.adkmob.com ssdk.adkmob.com adleads.com 0.0.0.0 a4.tl 0.0.0.0 adformdsp.net 0.0.0.0 adform.net 0.0.0.0 adkmob.com 0.0.0.0 adleads.com 0.0.0.0 admoda.com 0.0.0.0 adsmogo.mobi 0.0.0.0 adsmogo.net 0.0.0.0 adywind.com 0.0.0.0 adzerk.net 0.0.0.0 alexajstrack.com 0.0.0.0 applifier.info 0.0.0.0 appnexus.net 0.0.0.0 apxadtracking.net 0.0.0.0 atti.com 0.0.0.0 avazunativeads.com 0.0.0.0 cpro.baidu.cn 0.0.0.0 bayctrk.com 0.0.0.0 billymobile.com 0.0.0.0 cb-cdn.com 0.0.0.0 cedexis-radar.net 0.0.0.0 chartboosts.com 0.0.0.0 clickkydsp.com 0.0.0.0 cnbc7.com -
Get the version of PyFunceble
$ pyfunceble --version pyfunceble 1.7.0. (Blue Bontebok: Moth) -
Run a test
$ pyfunceble -f issue-19 -a --split --clean -nu- Note1: I used
-abecause I wanted the full output. - Note2: I used
--splitbecause I did not want to have the copy of the screen output split by status. - Note3: I used
--cleanin order to clean myoutputdirectory before starting to test. - Note4: I used
-nubecause I wanted to have the copy of the screen output in one file calledoutput/results.txt.
- Note1: I used
-
Read
output/results.txt# File generated by https://github.com/funilrys/PyFunceble # Date of generation: Wed 16 Jan 04:01:04 CET 2019 Domain Status Expiration Date Source HTTP Code Analyze Date a4.tl ACTIVE 20-sep-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 adformdsp.net ACTIVE 02-aug-2023 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 adform.net ACTIVE 23-sep-2023 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 adkmob.com ACTIVE 26-dec-2020 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 adleads.com ACTIVE 18-sep-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 admoda.com ACTIVE 27-mar-2021 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 adsmogo.mobi ACTIVE 11-sep-2020 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 adsmogo.net ACTIVE 11-sep-2020 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 adywind.com ACTIVE 11-apr-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 adzerk.net ACTIVE 12-may-2024 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 alexajstrack.com ACTIVE 04-jun-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 applifier.info ACTIVE 03-apr-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 appnexus.net ACTIVE 12-sep-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 apptrk.a4.tl ACTIVE Unknown NSLOOKUP 302 Wed 16 Jan 04:01:04 CET 2019 apxadtracking.net ACTIVE 06-aug-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 atti.com ACTIVE 23-feb-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 avazunativeads.com ACTIVE 16-feb-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 bayctrk.com ACTIVE 12-jan-2020 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 billymobile.com ACTIVE 20-apr-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 bp.adkmob.com ACTIVE Unknown NSLOOKUP 403 Wed 16 Jan 04:01:04 CET 2019 cb-cdn.com ACTIVE 22-apr-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 cedexis-radar.net ACTIVE 27-sep-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 chartboosts.com ACTIVE 20-may-2025 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 clickkydsp.com ACTIVE 16-nov-2019 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 cnbc7.com ACTIVE 30-nov-2018 WHOIS *** Wed 16 Jan 04:01:04 CET 2019 cpro.baidu.cn INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 04:01:04 CET 2019 els.a4.tl ACTIVE Unknown NSLOOKUP 403 Wed 16 Jan 04:01:04 CET 2019 jrs.a4.tl ACTIVE Unknown NSLOOKUP *** Wed 16 Jan 04:01:04 CET 2019 ldap.a4.tl ACTIVE Unknown NSLOOKUP *** Wed 16 Jan 04:01:04 CET 2019 preroll.a4.tl ACTIVE Unknown NSLOOKUP 403 Wed 16 Jan 04:01:04 CET 2019 sdk.a4.tl ACTIVE Unknown NSLOOKUP 403 Wed 16 Jan 04:01:04 CET 2019 ssdk.adkmob.com ACTIVE Unknown NSLOOKUP 403 Wed 16 Jan 04:01:04 CET 2019 -
Here is the
/etc/hostsI use while testing with PyFunceble. (if a domain is listed there and no Apache/nginx server is available on the machine, a NSLOOKUP do not resolve)$ cat /etc/hosts 127.0.0.1 localhost 127.0.0.1 localhost.localdomain 127.0.0.1 local 255.255.255.255 broadcasthost ::1 localhost ::1 ip6-localhost ::1 ip6-loopback fe80::1%lo0 localhost ff00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 127.0.0.1 +*+*+*+*+*+*.***** www.+*+*+*+*+*+*+*+*+*.****** # Network or projects Internal domain 0.0.0.0 0.0.0.0
I hope that we will find a solution or a protocol to reproduce your behavior/issue!
Cheers,
Nissar
P.S: Yes It is late here 😸 (I could not sleep)
from pyfunceble.
funilrys
commented on May 25, 2024
Hi @jawz101,
The first CI tests result are the same as mine: https://travis-ci.com/funilrys/PyFunceble/jobs/170715225#L1512-L1578
Cheers,
Nissar
from pyfunceble.
jawz101
commented on May 25, 2024
Thanks. Here was my initial results. I'll post back when I test things. This was done on a new little rasberry pi running raspbian so I'm still figuring out its quirks
mats.zip
from pyfunceble.
funilrys
commented on May 25, 2024
Hi @jawz101, I may have found your issue.
Can you look at your local configuration file (.PyFunceble.yaml) and tell me what is the assignment of the no_whois index?
This is the default state:
PyFunceble/.PyFunceble_production.yaml
Line 40 in 1d18d37
I'll launch the CI with it deactivated so that you can compare but this is what I get if I disable the request of the WHOIS record which actually means no_whois: True:
$ pyfunceble --version
pyfunceble 1.7.0. (Blue Bontebok: Moth)
$ pyfunceble -f issue-19 -a --split --clean -nu -nw
Note0: no_whois:True will disable the usage/request of the WHOIS record.
Note1: -nw switch the default value (from .PyFunceble.yaml) of no_whois.
Note2: no_whois is set to False on my env so using -nw set no_whois to True.
Note3: if no_whois is set to True by default (at .PyFunceble.yaml) the usage of -nw will set no_whois to False.
# File generated by https://github.com/funilrys/PyFunceble
# Date of generation: Wed 16 Jan 08:01:09 CET 2019
Domain Status Expiration Date Source HTTP Code Analyze Date
a4.tl INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
adformdsp.net INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
adform.net INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
adkmob.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
adleads.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
admoda.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
adsmogo.mobi INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
adsmogo.net INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
adywind.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
adzerk.net INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
alexajstrack.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
applifier.info INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
appnexus.net INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
apptrk.a4.tl ACTIVE Unknown NSLOOKUP 302 Wed 16 Jan 08:01:09 CET 2019
apxadtracking.net INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
atti.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
avazunativeads.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
bayctrk.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
billymobile.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
bp.adkmob.com ACTIVE Unknown NSLOOKUP 403 Wed 16 Jan 08:01:09 CET 2019
cb-cdn.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
cedexis-radar.net INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
chartboosts.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
clickkydsp.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
cnbc7.com INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
cpro.baidu.cn INACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
els.a4.tl ACTIVE Unknown NSLOOKUP 403 Wed 16 Jan 08:01:09 CET 2019
jrs.a4.tl ACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
ldap.a4.tl ACTIVE Unknown NSLOOKUP *** Wed 16 Jan 08:01:09 CET 2019
preroll.a4.tl ACTIVE Unknown NSLOOKUP 403 Wed 16 Jan 08:01:09 CET 2019
sdk.a4.tl ACTIVE Unknown NSLOOKUP 403 Wed 16 Jan 08:01:09 CET 2019
ssdk.adkmob.com ACTIVE Unknown NSLOOKUP 403 Wed 16 Jan 08:01:09 CET 2019
Now as you probably know, if you get INACTIVE with NSLOOKUP as the source with no_whois:False that actually means:
- We could not extract the expiration date from the WHOIS record.
nslookup xxtell us NXDOMAIN or similar which means that no IP is linked to the domain.
On the other hand with no_whois:True:
- We don't request the whois record.
nslookup xxtell us NXDOMAIN or similar which means that no IP is linked to the domain.
Back to my previous test most of them were marked as ACTIVE (and WHOIS as source) on my env and Travis CI. Here is how we flag explain more than simply:
no_whois:False- We could extract the expiration date from the WHOIS record.
- We flag the domain as
ACTIVE.
OR
no_whois:False- We could not extract the expiration date from the WHOIS record.
nslookup xxdoes not give use NXDOMAIN or similar.- We flag the domain as
ACTIVE
OR
no_whois:False- We could not extract the expiration date from the WHOIS record.
nslookup xxtell us NXDOMAIN or similar which means that no IP is linked to the domain.- We flag the domain as
INACTIVE
Note: We do not use the CLI tool nslookup but I used nslookup or NXDOMAIN to simply explain.
Cheers,
Nissar
from pyfunceble.
funilrys
commented on May 25, 2024
- With
no_whois:False: https://travis-ci.com/funilrys/PyFunceble/jobs/170736790#L1151-L1217 - With
no_whois:True: https://travis-ci.com/funilrys/PyFunceble/jobs/170736790#L1219-L1285
from pyfunceble.
jawz101
commented on May 25, 2024
ok... looks like my router was having some DNS issues. I reset it and things started working properly
from pyfunceble.
Related Issues (20)
- FEATURE: Preload/Continue like the CI workflow ... but without Git HOT 2
- BUG: Cant install latest version with pip HOT 2
- BUG: urls in domain lists.... HOT 2
- FEATURE: Special Rules for forumactif.com HOT 1
- Contribution Tracking
- DOC: Moving away from restructured text HOT 8
- BUG: URL in file header should be changed
- BUG: dead domain query HOT 9
- BUG: sqlalche braekes after finishing...
- FEATURE: Reputation filter using the proxy connection HOT 3
- FEATURE: Sharing WHOIS
- FEATURE: meilisearch or redis support vs RamDrive
- BUG: TypeError(f"<data> should be {dict}, {type(data)} given.") HOT 5
- BUG: log file not created... HOT 2
- Special Rules, are they working as expected? HOT 2
- Unstable special rules HOT 1
- BUG: Object of type datetime is not JSON serializable HOT 2
- FEATURE: Timestamp in CLI output
- BUG: domains can't start with a dot... HOT 2
- pyfunceble.funilrys.com
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyfunceble.