c-mueller / ads Goto Github PK
View Code? Open in Web Editor NEWCoreDNS plugin to block ads. Inspried by PiHole.
License: Apache License 2.0
CoreDNS plugin to block ads. Inspried by PiHole.
License: Apache License 2.0
I noticed my coredns pods where downloading excessively, and did that every 1 one minute. It turns out this happens after the first auto-update-interval
.
This is because for at
Line 161 in d6b2ab8
failCount < u.RetryCount
as failCount
does not increment.
IMPACT: after auto-update-interval
HTTP lists are downloaded every minute
Would be nice if the whitelist
and blacklist
options accepts URL's and file-locations and maybe obsolete the list
option?
Examples:
blacklist https://raw.githubusercontent.com/cbuijs/accomplist/master/deugniets/plain.black.domain.list
whitelist https://raw.githubusercontent.com/cbuijs/accomplist/master/deugniets/plain.white.domain.list
blacklist-regex /etc/coredns/my-sweet-regex.list
blacklist doubleclick.net
whitelist /etc/coredns/banks.list
etc...
If a Qname should get blocked the plugin currently only returns a A Record no matter what was originally requested.
In order to do things properly IPv6 requests should return a IPv6 address instead.
Setting this address also requires adding another config option.
First of all, thanks you to has spend time on this project, your plugin convince me to switch from pi-hole to CoreDns as my home dns !!
I have setup a graphana dashboard for my CoreDns instance, and i would like to be able to have the number of blocked request at a given moment. There is only two metrics exported for now coredns_ads_blocked_request_count_total
and coredns_ads_request_count_total
; unfortunately these metrics does not fit my need
I have juste read the documentation of prometheus, for this need you should need to use histogram metric
If you don't have enough time to do this it's not a problem, i could try to work on it (event if i am not really comfortable with go language )
Once again, thanks for sharing with us your work 🥇
While the first update of the Blocklist file will occur properly, i.e. last time + update interval
even if CoreDNS crashes or gets restarted. But the second one and all following ones will occur at start time + (update interval * update iteration)
This is wrong. it should occur at last time + update interval
Hello, I just noticed a probleme in the ads blocker.
When a error occure while fetching the URL lists, descibed bellow, the whole initialization process is stopped and the next lists are not fetched. The result is than all list are not loaded and the blocker not work as expected. Can you find what is the problem please ?
Thanks for your time,
dns | [INFO] plugin/ads: Initializing CoreDNS 'ads' plugin. Version 0.2.1
dns | [INFO] plugin/ads: Initializing CoreDNS 'ads' list update routines...
dns | [DEBUG] plugin/hosts: Parsed hosts file into 4 entries
dns | .:53
dns | CoreDNS-1.6.7
dns | linux/amd64, go1.13.8, da7f65b3-dirty
dns | [DEBUG] plugin/ads: Fetching list "https://hosts-file.net/psh.txt"...
dns | [ERROR] plugin/ads: Get https://hosts-file.net/psh.txt: net/http: TLS handshake timeout
dns | [INFO] plugin/ads: Initializing CoreDNS 'ads' plugin. Version 0.2.1
dns | [INFO] plugin/ads: Initializing CoreDNS 'ads' list update routines...
dns | [DEBUG] plugin/hosts: Parsed hosts file into 4 entries
dns | .:53
dns | CoreDNS-1.6.7
dns | linux/amd64, go1.13.8, da7f65b3-dirty
dns | [DEBUG] plugin/ads: Fetching list "https://hosts-file.net/psh.txt"...
dns | [ERROR] plugin/ads: Get https://hosts-file.net/psh.txt: dial tcp 3.234.198.254:443: i/o timeout
To get some statistics or block websites that are not on a blocklist some sort of HTTP API might be useful.
One huge foucs here should be simlicity to keep the functionality very simple:
Some notes on the functionality and mappings:
Authentication using basic auth
Mappings:
GET /api/stats
-> show PiHole llike stats of the last 24hGET /api/Blocklists
-> Return a list of the blocklists currently used (including manually added ones and regex)GET /api/blocklists/:name
-> Get the specific blocklists contentsPOST /api/blocklist/regex/add
-> Add A Regex to blockPOST /api/blocklist/regex/delete
-> Delete A Regex to blockLine 30 in a915674
https://someonewhocares.org/hosts/zero/hosts
https://en.wiktionary.org/wiki/%C9%A2
ɢoogle.com
Hi,
First: Thanks for this very cool plugin! Keep up the good work!
But...
When following the instructions to add the plugin and compile coredns, following errors occurr during the make:
# github.com/c-mueller/ads
coredns/go/src/github.com/c-mueller/ads/setup.go:51:24: cannot use c (type *"github.com/mholt/caddy".Controller) as type *"github.com/caddyserver/caddy".Controller in argument to metrics.MustRegister
coredns/go/src/github.com/c-mueller/ads/setup.go:52:24: cannot use c (type *"github.com/mholt/caddy".Controller) as type *"github.com/caddyserver/caddy".Controller in argument to metrics.MustRegister
coredns/go/src/github.com/c-mueller/ads/setup.go:53:24: cannot use c (type *"github.com/mholt/caddy".Controller) as type *"github.com/caddyserver/caddy".Controller in argument to metrics.MustRegister
coredns/go/src/github.com/c-mueller/ads/setup.go:54:24: cannot use c (type *"github.com/mholt/caddy".Controller) as type *"github.com/caddyserver/caddy".Controller in argument to metrics.MustRegister
coredns/go/src/github.com/c-mueller/ads/setup.go:67:21: cannot use c (type *"github.com/mholt/caddy".Controller) as type *"github.com/caddyserver/caddy".Controller in argument to dnsserver.GetConfig
I see this with many "external" plugins and I think something changed at one point with the go modules and they are always related to Caddy.
Could you provide info/documentation how to succesfull compile? E.g. how do you do it? Using the latest coredns v1.5.2 and go version go1.12.7 linux/amd64.
Cheers,
-Chris.
First of all, very nice project! I have replaced my pihole with coredns+ads. So now I have names for my network and ad blocking. And it's a bit faster than pihole, I did few benchmarks using DNSBench. So it servers names and blocks ads yet it's faster than pihole, which only blocks ads.
Currently whitelist would be very good feature as many of the blocklists are quite aggressive and I can't exclude names I was able on pihole.
Is it possible to make a release with the last changes to be compatible with version 1.8 of coredns?
The list "https://v.firebog.net/hosts/Kowabit.txt" blocks legitimate sites:
msn.com
live.com
facebook.com
twitter.com
instagram.com
facebook.de
facebook.net
apple.com
yahoo.com
yahoo.de
deviantart.com
pinterest.com
microsoft.com
...
stackoverflow.com
...
googlecode.com
...
Currently only exact matching of Domain Names is supported. We also want to support rules based on wildcards and regular expressions (regex).
In the first step, the definition of such rules should only be possible through the corefile, by a definitition like:
ads {
wildcard domain-to-wildcard-block.com
}
or
ads {
regex <REGEX TO BLOCK>
}
With the following config adblocking only works on normal dns not doh and dot.
.:53 https://.:443 tls://.:853 {
tls fullchain.pem privkey.pem
ads
forward . 1.1.1.1
errors
reload
}
Basic Dns:
kdig @127.0.0.1 -p 1053 aoredi.com +short
127.0.0.1
DNS over TLS:
kdig @127.0.0.1 +tls-host=localhost aoredi.com +short
109.206.162.83
Am I doing something wrong or is this intended?
Currently the plugin will download the blocklists everytime CoreDNS is launched.
In order to prevent this the blocklists should be persisted on disk.
Updates should only take place by a cronjob.
Apr 18 08:30:21 dns_server rc.local[1012]: [INFO] plugin/ads: Updating lists from Local files...
Apr 18 08:30:21 dns_server rc.local[1012]: [INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 1 entries into whitelist
Apr 18 08:31:21 dns_server rc.local[1012]: [INFO] plugin/ads: Updating lists from Local files...
Apr 18 08:31:21 dns_server rc.local[1012]: [INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 1 entries into whitelist
这是我的配置:
ads {
blacklist https://raw.githubusercontent.com/privacy-protection-tools/anti-AD/master/anti-ad-domains.txt
whitelist file:///home/snakwu/CoreDNS/whitelist.txt
auto-update-interval 24h
list-store /home/snakwu/CoreDNS/ads-cache
}
Currently, as of version 0.2.5, changing the auto-update-interval does not update the intervall between updates of the http lists.
This must be resolved and a config option for the file update interval must be added.
hi,c-mueller!
When I add ads to the source code of coerdns, and then make, no matter whether my Corefile is added or not, the entire network becomes slow!
plugin.cfg
metadata:metadata
cancel:cancel
tls:tls
reload:reload
nsid:nsid
bufsize:bufsize
root:root
bind:bind
debug:debug
trace:trace
ready:ready
health:health
pprof:pprof
prometheus:metrics
errors:errors
log:log
dnstap:dnstap
dns64:dns64
acl:acl
any:any
chaos:chaos
loadbalance:loadbalance
cache:cache
rewrite:rewrite
dnssec:dnssec
autopath:autopath
template:template
transfer:transfer
ads:github.com/c-mueller/ads
hosts:hosts
clouddns:clouddns
k8s_external:k8s_external
kubernetes:kubernetes
file:file
auto:auto
secondary:secondary
etcd:etcd
loop:loop
fanout:github.com/networkservicemesh/fanout
forward:forward
grpc:grpc
erratic:erratic
whoami:whoami
on:github.com/caddyserver/caddy/onevent
sign:sign
Currently whitelist and blacklist rules can only be defined within the corefile. In order to change them CoreDNS has to get restarted. In order to allow a simple method to reconfigure rules dynamically the following feature should be implemented:
Rule can be defined in a sepaerate file (or maybe multiple ones) this file gets reloaded every 5 seconds or maybe through a filesystem hook (on change). The idea of this should be simmilar to the reloading mechanism of the hosts
plugin.
For syntax i think a ignore
file like syntax is a good approach (excluding wildcards)
A rulefile could look like this:
# A comment
!google.com # Whitelist google.com
my-domain.com # Blacklist my-domain.com
!<REGEX> # Whitelist a regex
<REGEX> # Blacklist a regex
Consider the use of some sort of storage adapters for the block list like:
Some could be:
Based on a External storage solution
Based on Filesystem storage
We could also investigate the use of lookup through the storage solution, instead of loading the whole blocklist in memory
CGO_ENABLED=0 go build -v -ldflags="-s -w -X github.com/coredns/coredns/coremain.GitCommit=f6262eb2-dirty" -o coredns
github.com/c-mueller/ads
# github.com/c-mueller/ads
/go/pkg/mod/github.com/c-mueller/[email protected]/setup.go:57:4: undefined: metrics.MustRegister
/go/pkg/mod/github.com/c-mueller/[email protected]/setup.go:58:4: undefined: metrics.MustRegister
/go/pkg/mod/github.com/c-mueller/[email protected]/setup.go:59:4: undefined: metrics.MustRegister
/go/pkg/mod/github.com/c-mueller/[email protected]/setup.go:60:4: undefined: metrics.MustRegister
make: *** [Makefile:16: coredns] Error 2
Hi there,
I have noticed a problem when using https://coredns.io/plugins/reload/.
It seems that HTTP/S lists are not being updated on a reload.
I dynamically inject lists into the Corefile, and lists seem to only become active on a restart, not a reload.
Maybe [INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
is related to this?
Let me know how I can help out to debug/fix this :)
. {
reload
prometheus 0.0.0.0:9153
ads {
blacklist https://blocklistproject.github.io/List/ads.txt
blacklist https://blocklistproject.github.io/Lists/abuse.txt
blacklist https://blocklistproject.github.io/Lists/crypto.txt
blacklist https://blocklistproject.github.io/Lists/drugs.txt
blacklist https://blocklistproject.github.io/Lists/fraud.txt
blacklist https://blocklistproject.github.io/Lists/gambling.txt
blacklist https://blocklistproject.github.io/Lists/malware.txt
blacklist https://blocklistproject.github.io/Lists/phishing.txt
blacklist https://blocklistproject.github.io/Lists/porn.txt
blacklist https://blocklistproject.github.io/Lists/ransomware.txt
blacklist https://blocklistproject.github.io/Lists/redirect.txt
blacklist https://blocklistproject.github.io/Lists/scam.txt
blacklist https://blocklistproject.github.io/Lists/tiktok.txt
blacklist https://blocklistproject.github.io/Lists/tracking.txt
list-store /etc/coredns/blocklists
default-lists
target 127.0.0.1
target-ipv6 ::1
}
forward . tls://1.1.1.1 {
except fritz.box home lan local
tls_servername cloudflare-dns.com
health_check 5s
}
cache 30
}
[INFO] plugin/ads: Initializing CoreDNS 'ads' plugin. Version 0.2.5
[INFO] plugin/ads: Initializing CoreDNS 'ads' list update routines...
.:53
[INFO] plugin/reload: Running configuration MD5 = e60acf6ca0a2a149cfee6c1356874087
CoreDNS-1.8.3
linux/amd64, go1.15.8, 4293992b-dirty
[INFO] plugin/ads: Loaded Whitelist (HTTP) Length: 0
[INFO] plugin/ads: Loaded Blacklist (HTTP) Length: 2990552
[INFO] plugin/ads: Updating lists from HTTP URLs...
[INFO] plugin/ads: Scheduled next update in 23h53m16.626635257s
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
----->> Blocklist gets removed from Corefile here!
[INFO] Reloading
[INFO] plugin/ads: Initializing CoreDNS 'ads' plugin. Version 0.2.5
[INFO] plugin/reload: Running configuration MD5 = 988a0e01e30fe01dfa814abd9f642fed
[INFO] Reloading complete
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
Currently (v0.2.4) the plugin does not compile against the latest master
revisions. To fix this the Package path of caddy must be changed from github.com/caddyserver/caddy
to github.com/coredns/caddy
.
Once CoreDNS launches, the plugin will initialize itself buit it will not yet load the blocklists, either from the web or from disk. The Blocklists will get loaded approximately 5 seconds after the launch of CoreDNS. During this period the plugin does only block by the rules defined in the Corefile. Blocking the rest will occur once the blocklists have been loaded.
The idea is to try to load the blocklists on startup (within the onStartup
function) instead of 5 seconds later.
The initial decision to introduce the delay was the fact that i wanted to ensure CoreDNS is running before we try looking up any domainnames (ie. the ones from the blocklist URLs) because this could cause coredns to crash if you use it as the on the machine (container) it is running on. Since it is not able to resolve these names in this case
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.