anudeepnd / blacklist Goto Github PK

@anudeepND this domain delivers images and blocking it just cripples some web pages.As seen here:
https://www.autoblog.com/2018/10/18/2018-audi-r8-v10-plus-competition-package/

Hi there,

I've been looking into adding your list to my auto updated dnsmasq filter, though I found quite a lot of possible false calls.

You might want to consider removing the following items:

vmall.com
taobao.com
toplinkxchange.com
shellcat.ru
sigidwiki.com
jappy.tv
kingsoft.com
skyrock.mobi
skyrock.net
pwnews.net
rusnext.ru
nianticlabs.com
nikopik.com
nordpresse.be
nrj.fr
minimilitiahackcheats.com
magicbricks.com
launchforth.io
ksosoft.com
hideipaddr.com
fairtop.in
fedsig.com
fireeye.com
dailymaza.co
delcomeyersdale.com
columbiarecords.com
birddl.com
azadmob.in

We do not have a coin miner anymore. Even when we did, it was opt-in and didn't start unless you wanted it to.

If you are going to maintain this list, you need to review the sites better before adding them.

www.goodolddownloads.com

Thanks,
GOD

$ git blame -L '/ptpb\.pw/,+1' adservers.txt
92f62b7d (Debian 2018-12-10 13:53:25 +0000 32115) 0.0.0.0 ptpb.pw

ptpb is a lightweight pastebin and url shortener

Hello! Thank you for your effort!
Possible problems:
s2.dmcdn.net, static2.dmcdn.net - blocks thumbnail images of the videos on dailymotion.com
lx1.dyncdn.cc - blocks movie/tv cover images on 1337x.to
vid3.ec.dmcdn.net - I've found this domain also in your list. Not sure what type of videos it blocks on dailymotion. Ads-videos? Could be a false positive too.

@anudeepND i'm not sure how that specific domain ended up on your radar but *.oloadcdn.net
domains are part of openload.co and that's where the content is hosted. It's harmless and would advise to remove it as there will be not possible to watch anything from openload.co
For more info see our conversation.
Thank you and i hope you're well as you've been absent for a while now. 👍

settings.data.microsoft.com is false positive:
The following endpoint is used as a way for apps to dynamically update their configuration. Apps such as System Initiated User Feedback and the Xbox app use it. If you turn off traffic for this endpoint, an app that uses this endpoint may stop working.

https://docs.microsoft.com/en-us/windows/privacy/manage-windows-1809-endpoints

I don't know if this is intentional, but the domain googleadapis.l.google.com was added in 62b873d and is causing the problem, since fonts.googleapis.com contains a CNAME record to this domain.

Hi @anudeepND i'm follow the good guys on Ultimate.Hosts.Blacklist by @funilrys

There is a question about the domains i title of issue on the at least these two issues

Ultimate-Hosts-Blacklist/Ultimate.Hosts.Blacklist#455 and my own https://github.com/spirillen/rpz-block-list/issues/78

Could you do us that favor to enlighten us about why they are listed

bilder.bild.de imho should not be in there - I don't think ads are served from there, only images.

You block the following Windows Update servers with your adservers.txt:
fe2.update.microsoft.com.akadns.net
sls.update.microsoft.com.akadns.net
statsfe2.update.microsoft.com.akadns.net

All are legit:

Microsoft uses global load balancers that can appear in network trace-routes. For example, an endpoint for *.akadns.net might be used to load balance requests to an Azure datacenter, which can change over time.
https://docs.microsoft.com/en-us/windows/privacy/windows-endpoints-1809-non-enterprise-editions

@anudeepND i'll post a link to test it (it's safe) ,and i'm not saying that the domain in question is false positive but put it simply,it's working as the "man in the middle"(most likely collecting data in the process) but blocking it,one would not be able to reach the final destination.Here is the link(not sure how long will be active but if it's not ,let me know and i'll give you another one),click on the big green SEE DEAL button:
https://slickdeals.net/f/12903043-dell-xps-9380-i7-4k-touch-16gb-ddr4-ram-512gb-ssd-376-in-points-1505-tax

There are other redirect domains like this one and i'll try to trace them and report them due to braking of functionality(which i truly hate to be honest)

reg.ru is a major domain registrar and webhosting provider, this definitely false-positive.

What about the instagram's ads?
How can blocking them?

Found out about this domain (which is the parent company behind the popular Androidpit websites)

tracking-opi.fonpit.de

Here is a link to show the problem and as you can see the whole web page's icons,css etc. coming from that domain and therefore the page is all broken up:
https://dottech.org/167885/how-to-add-more-options-to-right-click-menu-in-chrome-tip/

A couple of sub domains for PushBullet (software to mirror notifications to computers/other devices) are listed in adservers.txt and are false positives.

• api.pushbullet.com
• widget.pushbullet.com

I'm not sure what the widget sub domain is used for, but the api sub domain is required for the software to work

ag.innovid.com 

Missing from current list of domains.

Match found in https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts:
cdn.innovid.com
rtr.innovid.com
s.innovid.com
s-static.innovid.com
dts.innovid.com
innovid.com
Match found in http://sysctl.org/cameleon/hosts:
dts.innovid.com
rtr.innovid.com
s-static.innovid.com
s.innovid.com
Match found in https://hosts-file.net/ad_servers.txt:
s.innovid.com
Match found in http://www.squidblacklist.org/downloads/dg-ads.acl:
innovid.com
Match found in https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt:
frontend5.innovid.com
frontend6.innovid.com
frontendqa.innovid.com
innovid.com
priority.innovid.com
tch.innovid.com

Hi Anudeep,

Please can you remove the domain c.tadst.com from your blocklist, as it is preventing some legitimate content on timeanddate.com from working (logos and maps not loading).

Best regards

I've had to whitelist a few domains to enable some of my Android TV (SHIELD TV) apps. Specifically:

CBS:
cbsinteractive.hb.omtrdc.net
pubads.g.doubleclick.net

NBC:
nbcume.sc.omtrdc.net

Not confident of your policy on what constitutes a "false positive", but these are necessary for the apps in question to function (though they likely also do assist in serving ads).

Match found in https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt:
vanguard.demdex.net

Hi, Vanguard isn't an ad server. This domain is for the adobe analytics used for personalize services and a few other items on their site. I recommend an unblock as it is not an ad server. Thanks.

Hello Anudeep,

Thank you for providing this great blocklist. I do have a question about a couple Mozilla domains:

blocklists.settings.services.mozilla.com

This URL is used by Firefox and Thunderbird to ban extensions that cause a security threat against its users. More information about how the blocklist works can be found in this FAQ. You can also view the contents of the blocklist. Some of these extensions look pretty nasty, I definitely would not want them installed on my computer. Perhaps you will consider removing this domain from your lists so that users can remain protected against these malware extensions.

shavar.services.mozilla.com

From Mozilla's site:

Originally designed for phishing protection the protocol was co-opted so that the tracking protection project would be able to publish larger data sets without incurring large bandwidth usage for mobile clients.

So this is Firefox's version of Google's 'Safe Browsing' feature. I can see why you would block this, but unfortunately its also the service that provides domains to block for Firefox's built-in tracking protection.

firefox.settings.services.mozilla.com

I use a Firefox account to sync my setting across devices. My understanding is that everything is fully encrypted using a key derived from the user password and cannot be recovered/decrypted on the server without the users password. I did find that they do store IP addresses for security reasons, but I do not believe it is stored unless you are actively using this service.

I've whitelisted all those domains on my end, so this isn't causing me any trouble, but I thought I might open a ticket so that I could get your viewpoint on them.

I was just wondering if you could unblock deals.thehackernews.com please

As per this tweet: vice-web-statics-cdn.vice.com is blocked by your blocklist, and causes www.vice.com to not load content such as pictures.

Cheers 😄

Blocking fast.wistia.com breaks training videos on codewithmosh.com

My understanding is that fast.wistia.com is a hosting provider for videos - kind of like vimeo.com for businesses. While the videos can be advertisements, they can also be legitimate content videos.

see title

Hello,

js.braintreegateway.com is used for payment processing. BrainTree is a PayPal owned company that provides credit-card processing forms for any commercial business. More can be read about their JavaScript SDK here: https://developers.braintreepayments.com/start/hello-client/javascript/v3.

By blocking this script, any website that uses BrainTree's SDK for credit card processing will fail, breaking online shopping and checkouts. I had to whitelist it this morning while doing some christmas shopping :)

I was looking at the https://github.com/Ultimate-Hosts-Blacklist/anudeepND_blacklist_ad_server/tree/master/output/domains, there are 13k inactive domains. Most of them seems to taken down or no longer valid. What are your suggestions @dnmTX and @lightswitch05 ? Do I need to remove them?

Hello and thank you for this great list. I've found that blocking tools.google.com in adservers.txt breaks google chrome updates. This is unfortunate since I love blocking Google and avoid Chrome whenever possible - but unfortunately it is required for some sites and having an out-of-date version is a big security problem. Anyways, I've whitelisted it on my side, so no real problem, just thought you might want to take this issue into consideration.

@anudeepND i was checking on the filtering progress on your lists in Ultimate-Hosts-Blacklist and there is a section for INVALID Domains which already has bunch of them listed.Some are misspelled,some has space in between,some are just nonexistent. Would advise to take a look and fix it on your end as they don't really serve any purpose but just being there.
If you need any help to sort them out,let me know.
Thank you 👍
For Example:
From INVALID domains: 0.0.0.0 miscellaneous
is 0.0.0.0 miscellaneous magnetic.t.domdex.com in your lists
and so on.....

please remove this line from blacklist:
||yastatic.net^

because its blocking some content:

@anudeepND found that you have some duplicates.Listing them here so you can remove them:
b.scorecardresearch.com
cdn.teads.tv
i.skimresources.com
ml.pubnative.net
p.skimresources.com -this one has 3 entries
trk.vidible.tv
t.skimresources.com
p.adsymptotic.com
nqs-lw3.youboranqs01.com
nqs-nl-c6-c2.youboranqs01.com
nqs-nl1-c2.youboranqs01.com
nqs-nl2.youboranqs01.com

Close when done.Thanks.

Hello @anudeepND ,

Because I want to be transparent, I just wanted to inform you that we added your hosts file to Ultimate.Hosts.Blacklist through Ultimate-Hosts-Blacklist/anudeepND_blacklist_ad_server.

Cheers,
Nissar

P.S: Feel free to close/comment after reading this information.

@anudeepND currently you have three domains listed that would interfere with windows update:
statsfe1.ws.microsoft.com
statsfe2.update.microsoft.com.akadns.net
statsfe2.ws.microsoft.com

This is based out of @crazy-max's lists which is very accurate and curated frequently . Please check if there are more just in case as i found them by chance and i'm Windows user and need Windows Update working.
Thank you.

@anudeepND those domains among setting up cookies,tracking/telemetry(i assume) also deliver images and blocking them entirely results in some web pages being broken up(porn sites mostly but still...).I believe in such a domains it's good to let the AdBlockers to take over and do the job as they can whitelist/block whatever it's needed.Would advise to whitelist them.Thank you.

Screenshot from my AdBlocker just as example:

@anudeepND looks like that domain is responsible for downloading Adobe Flash Player.Any input on why it's blocked here?
For reference see: Ultimate-Hosts-Blacklist/Ultimate.Hosts.Blacklist#451

Currently symcb.com is included in the wildcard file, but blocking this whole domain and its subdomains prevents HTTPS certificate revocation status checking in Chrome, resulting in HTTPS/certificate errors.

gp.symcb.com and tn.symcb.com are also included in the adservers file. I cannot yet confirm if these are used for certificate purposes or something else, but I managed to fix the issue by removing symcb.com from my wildcards.

Edit: example site to test with https://symantec.com. I am using Chrome 62 on Windows 10.

This domain on your blacklist cdn.syndication.twimg.com seems to target the widgets used on sites to embed Tweets.

For reference, this is the site I noticed it on
http://www.iflscience.com/environment/truck-driver-ignored-signs-and-drove-across-perus-2000yearold-nasca-lines/

And I narrowed it down with the help of this forum
https://support.mozilla.org/en-US/questions/1189443?page=2
Specifically, this section
" This address should pop up a download dialog (no need to download):
https://cdn.syndication.twimg.com/tweets.json?callback=__twttr.callbacks.cb0&ids=950804340675088384&lang=en&suppress_response_codes=true&tz=GMT-0800 "

CDN is a Content Delivery Network, and as far as I can tell it seems to load images. Another domain I saw used it for shopping and keeping the pictures on their server rather than the store's own website. Here is the Wiki for reference
https://en.wikipedia.org/wiki/Content_delivery_network

Is there a history of abusing the Twitter CDN for ads? If not would it be possible to remove this from your blacklist?

"server.cpmstar.com" I believe is another server that hosts ads, at least on the websites that I visit.

This is the location service from Mozilla.
It's safe

dmd.metaservices.microsoft.com is false positive in your list.
It's just used for device metadata: https://docs.microsoft.com/en-us/windows/privacy/manage-windows-1809-endpoints

Can you check your adservers list. This st.adxxx.o domain seems to be an incomplete domain.

https://community.ubnt.com/ uses this domain for its CSS layout. With it blocked the layout is jacked.

• ubnt.i.lithium.com

Also cisco support forums uses this for its CSS layout which is also on this list.

• kxiwq67737.i.lithium.com

There is probably more support sites but I use these two regular

The following end up blocking a lot of sites, bing being one of them.

0.0.0.0 a-0001.a-msedge.net
0.0.0.0 a-0002.a-msedge.net
0.0.0.0 a-0003.a-msedge.net
0.0.0.0 a-0004.a-msedge.net
0.0.0.0 a-0005.a-msedge.net
0.0.0.0 a-0006.a-msedge.net
0.0.0.0 a-0007.a-msedge.net
0.0.0.0 a-0008.a-msedge.net
0.0.0.0 a-0009.a-msedge.net

C:\Users\Test-User>nslookup www.bing.com
Server:  UnKnown
Address:  10.255.255.3

Non-authoritative answer:
Name:    a-0001.a-msedge.net
Address:  0.0.0.0
Aliases:  www.bing.com
          a-0001.a-afdentry.net.trafficmanager.net

I've noticed my cellphone performing requests to epdg.epc.mncXXX.mccXXX.pub.3gppnetwork.org which was not present in your list. So I decided to script a DNS lookup on all the possible subdomain combination on 3gppnetwork.org

#!/bin/bash
# First Loop between 000 and 999
for a in `seq -s " " -w 000 999`
do
	#Second loop between 000 and 999
	for b in `seq -s " " -w 000 999`
	do
		#Using dig, try to retrieve the IP address (-t A) on the hostnames
		CONTENT=`dig @1.1.1.1 -t A epdg.epc.mnc$a.mcc$b.pub.3gppnetwork.org`
		#If we find "SOA" in the dig response, means that hostname has no associated IP address
		if [[ $CONTENT =~ .*SOA.* ]]
		then
			echo "epdg.epc.mnc"$a".mcc"$b".pub.3gppnetwork.org" >> 3gpp_dead.txt
		else
			echo "epdg.epc.mnc"$a".mcc"$b".pub.3gppnetwork.org" >> 3gpp_alive.txt
		fi
	done
done

Hopefully this can be easily adjusted to cover many other ad-network.
Obviously the ad-network is able to enable/disable any subdomain not picked up today. In that case, you can decide to:

1. Blacklist all the possible subdomain, even if they are not alive (bigger file).
2. Run the script periodically and update the list with only the live subdomains (slow but more accurate results).
3. Switch to more sophisticated mechanisms to ban entirely subdomains (e.g.: *.pub.3gppnetwork.org)

Meanwhile the alive list is attachd
3gpp_alive.txt

Hello, I'm trying to implement your Facebook host file in the latest version of AdAway, but it seems that the entries aren't making it into my host file.

Hello, I found a new host from facebook today: instagram.fcdg1-1.fna.fbcdn.net

This rule ||g.alicdn.com^ that simply blocking makes site taobao.com broken. It's unnecessary since similar but more precise rules are exist in Easylist China etc..

Hello,

I just want to report this two:

Two domains in one line or... are those intentional?

Thanks!