Hi,

First: Thanks for this very cool plugin! Keep up the good work!

But...

When following the instructions to add the plugin and compile coredns, following errors occurr during the make:

# github.com/c-mueller/ads
coredns/go/src/github.com/c-mueller/ads/setup.go:51:24: cannot use c (type *"github.com/mholt/caddy".Controller) as type *"github.com/caddyserver/caddy".Controller in argument to metrics.MustRegister
coredns/go/src/github.com/c-mueller/ads/setup.go:52:24: cannot use c (type *"github.com/mholt/caddy".Controller) as type *"github.com/caddyserver/caddy".Controller in argument to metrics.MustRegister
coredns/go/src/github.com/c-mueller/ads/setup.go:53:24: cannot use c (type *"github.com/mholt/caddy".Controller) as type *"github.com/caddyserver/caddy".Controller in argument to metrics.MustRegister
coredns/go/src/github.com/c-mueller/ads/setup.go:54:24: cannot use c (type *"github.com/mholt/caddy".Controller) as type *"github.com/caddyserver/caddy".Controller in argument to metrics.MustRegister
coredns/go/src/github.com/c-mueller/ads/setup.go:67:21: cannot use c (type *"github.com/mholt/caddy".Controller) as type *"github.com/caddyserver/caddy".Controller in argument to dnsserver.GetConfig

I see this with many "external" plugins and I think something changed at one point with the go modules and they are always related to Caddy.

Could you provide info/documentation how to succesfull compile? E.g. how do you do it? Using the latest coredns v1.5.2 and go version go1.12.7 linux/amd64.

Cheers,
-Chris.

The list "https://v.firebog.net/hosts/Kowabit.txt" blocks legitimate sites:

msn.com
live.com
facebook.com
twitter.com
instagram.com
facebook.de
facebook.net
apple.com
yahoo.com
yahoo.de
deviantart.com
pinterest.com
microsoft.com
...
stackoverflow.com
...
googlecode.com
...

With the following config adblocking only works on normal dns not doh and dot.

.:53 https://.:443 tls://.:853 {
    tls fullchain.pem privkey.pem
    ads
    forward . 1.1.1.1
    errors
    reload
}

Basic Dns:

kdig @127.0.0.1 -p 1053 aoredi.com +short
127.0.0.1

DNS over TLS:

kdig @127.0.0.1 +tls-host=localhost  aoredi.com +short
109.206.162.83

Am I doing something wrong or is this intended?

Currently only exact matching of Domain Names is supported. We also want to support rules based on wildcards and regular expressions (regex).

In the first step, the definition of such rules should only be possible through the corefile, by a definitition like:

ads {
  wildcard domain-to-wildcard-block.com
}

or

ads {
  regex <REGEX TO BLOCK>
}

Hi there,

I have noticed a problem when using https://coredns.io/plugins/reload/.

It seems that HTTP/S lists are not being updated on a reload.
I dynamically inject lists into the Corefile, and lists seem to only become active on a restart, not a reload.

Maybe [INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist is related to this?

Let me know how I can help out to debug/fix this :)

Config

. {
    reload
    prometheus 0.0.0.0:9153
    ads {
      blacklist https://blocklistproject.github.io/List/ads.txt
      blacklist https://blocklistproject.github.io/Lists/abuse.txt
      blacklist https://blocklistproject.github.io/Lists/crypto.txt
      blacklist https://blocklistproject.github.io/Lists/drugs.txt
      blacklist https://blocklistproject.github.io/Lists/fraud.txt
      blacklist https://blocklistproject.github.io/Lists/gambling.txt
      blacklist https://blocklistproject.github.io/Lists/malware.txt
      blacklist https://blocklistproject.github.io/Lists/phishing.txt
      blacklist https://blocklistproject.github.io/Lists/porn.txt
      blacklist https://blocklistproject.github.io/Lists/ransomware.txt
      blacklist https://blocklistproject.github.io/Lists/redirect.txt
      blacklist https://blocklistproject.github.io/Lists/scam.txt
      blacklist https://blocklistproject.github.io/Lists/tiktok.txt
      blacklist https://blocklistproject.github.io/Lists/tracking.txt
      list-store /etc/coredns/blocklists
      default-lists

      target 127.0.0.1
      target-ipv6 ::1
    }
    forward . tls://1.1.1.1 {
       except fritz.box home lan local
       tls_servername cloudflare-dns.com
       health_check 5s
    }
    cache 30
}

Logs

[INFO] plugin/ads: Initializing CoreDNS 'ads' plugin. Version 0.2.5
[INFO] plugin/ads: Initializing CoreDNS 'ads' list update routines...
.:53
[INFO] plugin/reload: Running configuration MD5 = e60acf6ca0a2a149cfee6c1356874087
CoreDNS-1.8.3
linux/amd64, go1.15.8, 4293992b-dirty
[INFO] plugin/ads: Loaded Whitelist (HTTP) Length: 0
[INFO] plugin/ads: Loaded Blacklist (HTTP) Length: 2990552
[INFO] plugin/ads: Updating lists from HTTP URLs...
[INFO] plugin/ads: Scheduled next update in 23h53m16.626635257s
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist
[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist


----->> Blocklist gets removed from Corefile here!

[INFO] Reloading
[INFO] plugin/ads: Initializing CoreDNS 'ads' plugin. Version 0.2.5
[INFO] plugin/reload: Running configuration MD5 = 988a0e01e30fe01dfa814abd9f642fed
[INFO] Reloading complete


[INFO] plugin/ads: Updating lists from Local files...
[INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 0 entries into whitelist

https://someonewhocares.org/hosts/zero/hosts https://en.wiktionary.org/wiki/%C9%A2

ɢoogle.com

I noticed my coredns pods where downloading excessively, and did that every 1 one minute. It turns out this happens after the first auto-update-interval.

This is because for at

IMPACT: after auto-update-interval HTTP lists are downloaded every minute

To get some statistics or block websites that are not on a blocklist some sort of HTTP API might be useful.

One huge foucs here should be simlicity to keep the functionality very simple:

Some notes on the functionality and mappings:
Authentication using basic auth

Mappings:

• GET /api/stats -> show PiHole llike stats of the last 24h
• GET /api/Blocklists -> Return a list of the blocklists currently used (including manually added ones and regex)
• GET /api/blocklists/:name -> Get the specific blocklists contents
• POST /api/blocklist/regex/add -> Add A Regex to block
• POST /api/blocklist/regex/delete -> Delete A Regex to block

First of all, very nice project! I have replaced my pihole with coredns+ads. So now I have names for my network and ad blocking. And it's a bit faster than pihole, I did few benchmarks using DNSBench. So it servers names and blocks ads yet it's faster than pihole, which only blocks ads.

Currently whitelist would be very good feature as many of the blocklists are quite aggressive and I can't exclude names I was able on pihole.

Currently (v0.2.4) the plugin does not compile against the latest master revisions. To fix this the Package path of caddy must be changed from github.com/caddyserver/caddy to github.com/coredns/caddy.

Currently the plugin will download the blocklists everytime CoreDNS is launched.

In order to prevent this the blocklists should be persisted on disk.
Updates should only take place by a cronjob.

Once CoreDNS launches, the plugin will initialize itself buit it will not yet load the blocklists, either from the web or from disk. The Blocklists will get loaded approximately 5 seconds after the launch of CoreDNS. During this period the plugin does only block by the rules defined in the Corefile. Blocking the rest will occur once the blocklists have been loaded.

The idea is to try to load the blocklists on startup (within the onStartup function) instead of 5 seconds later.

The initial decision to introduce the delay was the fact that i wanted to ensure CoreDNS is running before we try looking up any domainnames (ie. the ones from the blocklist URLs) because this could cause coredns to crash if you use it as the on the machine (container) it is running on. Since it is not able to resolve these names in this case

Is it possible to make a release with the last changes to be compatible with version 1.8 of coredns?

First of all, thanks you to has spend time on this project, your plugin convince me to switch from pi-hole to CoreDns as my home dns !!

I have setup a graphana dashboard for my CoreDns instance, and i would like to be able to have the number of blocked request at a given moment. There is only two metrics exported for now coredns_ads_blocked_request_count_total and coredns_ads_request_count_total; unfortunately these metrics does not fit my need

I have juste read the documentation of prometheus, for this need you should need to use histogram metric

If you don't have enough time to do this it's not a problem, i could try to work on it (event if i am not really comfortable with go language )

Once again, thanks for sharing with us your work 🥇

Consider the use of some sort of storage adapters for the block list like:

Some could be:

Based on a External storage solution

• SQL
• BoltDB
• Redis
• LevelDB

Based on Filesystem storage

• JSON (Used currently)
• Protobufs

We could also investigate the use of lookup through the storage solution, instead of loading the whole blocklist in memory

Would be nice if the whitelist and blacklist options accepts URL's and file-locations and maybe obsolete the list option?

Examples:

blacklist https://raw.githubusercontent.com/cbuijs/accomplist/master/deugniets/plain.black.domain.list
whitelist https://raw.githubusercontent.com/cbuijs/accomplist/master/deugniets/plain.white.domain.list
blacklist-regex /etc/coredns/my-sweet-regex.list
blacklist doubleclick.net
whitelist /etc/coredns/banks.list

etc...

Currently, as of version 0.2.5, changing the auto-update-interval does not update the intervall between updates of the http lists.

This must be resolved and a config option for the file update interval must be added.

If a Qname should get blocked the plugin currently only returns a A Record no matter what was originally requested.

In order to do things properly IPv6 requests should return a IPv6 address instead.
Setting this address also requires adding another config option.

hi,c-mueller!
When I add ads to the source code of coerdns, and then make, no matter whether my Corefile is added or not, the entire network becomes slow!

plugin.cfg

metadata:metadata
cancel:cancel
tls:tls
reload:reload
nsid:nsid
bufsize:bufsize
root:root
bind:bind
debug:debug
trace:trace
ready:ready
health:health
pprof:pprof
prometheus:metrics
errors:errors
log:log
dnstap:dnstap
dns64:dns64
acl:acl
any:any
chaos:chaos
loadbalance:loadbalance
cache:cache
rewrite:rewrite
dnssec:dnssec
autopath:autopath
template:template
transfer:transfer
ads:github.com/c-mueller/ads
hosts:hosts
clouddns:clouddns
k8s_external:k8s_external
kubernetes:kubernetes
file:file
auto:auto
secondary:secondary
etcd:etcd
loop:loop
fanout:github.com/networkservicemesh/fanout
forward:forward
grpc:grpc
erratic:erratic
whoami:whoami
on:github.com/caddyserver/caddy/onevent
sign:sign

Hello, I just noticed a probleme in the ads blocker.

When a error occure while fetching the URL lists, descibed bellow, the whole initialization process is stopped and the next lists are not fetched. The result is than all list are not loaded and the blocker not work as expected. Can you find what is the problem please ?

Thanks for your time,

dns                    | [INFO] plugin/ads: Initializing CoreDNS 'ads' plugin. Version 0.2.1
dns                    | [INFO] plugin/ads: Initializing CoreDNS 'ads' list update routines...
dns                    | [DEBUG] plugin/hosts: Parsed hosts file into 4 entries
dns                    | .:53
dns                    | CoreDNS-1.6.7
dns                    | linux/amd64, go1.13.8, da7f65b3-dirty
dns                    | [DEBUG] plugin/ads: Fetching list "https://hosts-file.net/psh.txt"...
dns                    | [ERROR] plugin/ads: Get https://hosts-file.net/psh.txt: net/http: TLS handshake timeout

dns                    | [INFO] plugin/ads: Initializing CoreDNS 'ads' plugin. Version 0.2.1
dns                    | [INFO] plugin/ads: Initializing CoreDNS 'ads' list update routines...
dns                    | [DEBUG] plugin/hosts: Parsed hosts file into 4 entries
dns                    | .:53
dns                    | CoreDNS-1.6.7
dns                    | linux/amd64, go1.13.8, da7f65b3-dirty
dns                    | [DEBUG] plugin/ads: Fetching list "https://hosts-file.net/psh.txt"...
dns                    | [ERROR] plugin/ads: Get https://hosts-file.net/psh.txt: dial tcp 3.234.198.254:443: i/o timeout

Currently whitelist and blacklist rules can only be defined within the corefile. In order to change them CoreDNS has to get restarted. In order to allow a simple method to reconfigure rules dynamically the following feature should be implemented:

Rule can be defined in a sepaerate file (or maybe multiple ones) this file gets reloaded every 5 seconds or maybe through a filesystem hook (on change). The idea of this should be simmilar to the reloading mechanism of the hosts plugin.

For syntax i think a ignore file like syntax is a good approach (excluding wildcards)
A rulefile could look like this:

# A comment
!google.com # Whitelist google.com
my-domain.com # Blacklist my-domain.com
!<REGEX> # Whitelist a regex
<REGEX> # Blacklist a regex

While the first update of the Blocklist file will occur properly, i.e. last time + update interval even if CoreDNS crashes or gets restarted. But the second one and all following ones will occur at start time + (update interval * update iteration)

This is wrong. it should occur at last time + update interval

Apr 18 08:30:21 dns_server rc.local[1012]: [INFO] plugin/ads: Updating lists from Local files...
Apr 18 08:30:21 dns_server rc.local[1012]: [INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 1 entries into whitelist
Apr 18 08:31:21 dns_server rc.local[1012]: [INFO] plugin/ads: Updating lists from Local files...
Apr 18 08:31:21 dns_server rc.local[1012]: [INFO] plugin/ads: [File Update] Loaded 0 entries into Blacklist and 1 entries into whitelist

这是我的配置:

ads {
    blacklist https://raw.githubusercontent.com/privacy-protection-tools/anti-AD/master/anti-ad-domains.txt
    whitelist file:///home/snakwu/CoreDNS/whitelist.txt
    auto-update-interval 24h
    list-store /home/snakwu/CoreDNS/ads-cache
    }