About this Issue

Document important changes to this repo that resulted in changes in configuration and/or list creation script

Documentation

• License updates #223
  • Update to list creation script with new entity list structure mozilla-services/shavar-list-creation#155
  • Update to client endpoints and its distribution within Mozilla bug
  • Update to licensing on Firefox bug
  • Update to tools mozilla/trackingprotection-tools#18

About this Issue

As mentioned in #111, we want to apply the #110 changes to the rest of the versioned lists when there are actual updates so we don't trigger non-meaningful update.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1432650 for an example where Disconnect's reclassification of an entry from Advertising to Content had no effect.

We should aim to detect these problems at pull request time.

This is a Yandex domain it is already in the list of entities.

About this Issue

To make the PR tests be more robust and comprehensive we should add additional checks/tests in Travis CI to check for the following:

Acceptance Criteria

• Check the diff between shavar-prod-lists and disconnect-tracking-protection and have Travis CI fail if it ever finds domains in our list that aren't in Disconnect's
• Check the list files updated (run shavar-list-creation) from the changes in the shavar-prod-lists, see #171 for more details
• Check that the size of the files updated does not exceed the max file size (4Mb), see #170 for more details.
• Remove the "tag" counts in the json_verify.py test as it is an outdated format

The current list verification script contains some nested logic that makes it difficult to follow on first read. We should be able to simplify it by porting the checks over to something like pytest.

While processing the entity list, ensure that each property & resource domain only occurs in a single entity, to prevent down-stream bugs where a duplicate resource domain might be mis-identified as belonging to an entity.

See mozilla/blok#104 (comment)

Hello! My name is Russell Vaughan and I'm a Sales Engineer at GoSquared. A client had raised that the Firefox's enhanced tracking protection now blocks our chat widget in strict mode as it is loaded from gosquared.com also, which is listed in your blacklist.

Our chat widget is not reliant on our Analytics tracking. If you block just our tracking endpoints, opposed to all of gosquared.com, our chat widget will still load and be functional. However, all tracking will still be blocked. We've worked with most of the other tracking blocking tools to solve this.

Could we ask that you use data.gosquared.com and data2.gosquared.com for your blacklist opposed to gosquared.com?

Please do let me know if you have any questions or need anything further.

Disconnect's canonical list now contains category tags on some of the domains. Our tests currently fail when these are present. We should figure out what needs to be updated to allows us to merge versions of the list with tags present.

Because JSECOIN is 100%, explicit USER-OPT-IN, users will always be asked whether they want their CPU used for mining at JSECOIN-hosted-sites. If this were to be allowed as the Default behavior of Firefox the user would not be able to distinguish between User-Approved (Safe) Miners and unsafe Miners that steal CPU Cycles. Web Hoster and User Consents are the first principles of JSECOIN Design (and perhaps existing or future other Miners too).

Eventually, it may be useful to provide a Checkbox (or equivalent) that By-Default allows Opt-In Miners, because Opting-In should be the user’s choice. It’s the secret monetizing by stealing user CPU cycles that is to be discouraged, not legitimate means of monetizing anything with which users explicitly agree. JSECOIN needs to be removed from this code.

The "Disconnect" category is currently remapped to the Advertising, Analytics, and Social categories using this file. This adds complexity to list parsing, and has led to a lot of confusion when folks look at the plaintext version of the list. We should remove this category from the upstream list if possible.

hi, there are updates in the Disconnect list that were made a month ago that I still don't see here. Is there an anticipated date to pick up the lastest Disconnect list? I assume this happens automatically periodically but I can't seem to find any schedule anywhere in docs.

About this PR

As an extension to PR #120 additional checks are needed in Travis CI to better review changes to update the tracking protections lists, further discussion here:

Acceptance Criteria

• Add file size check that changes do not exceed 4Mb for branch versions 73 and higher
• Add file size check that changes do not exceed 1Mb for branch versions 72 and lower

hs.fi is a newspaper site owned by sanoma.fi. Subscribers log in on tili.sanoma.fi (as you can see by clicking "Kirjaudu" in the top right corner of hs.fi) and the login state is supposed to be reflected on hs.fi. With ETP in "strict" mode, hs.fi fails to discover the logged-in state if the login has been performed in an earlier session. This started happening in Firefox for iPad a couple of weeks earlier than it started happening in Fenix.

My undebugged hypothesis is that we treat sanoma.fi as a third party relative to hs.fi and strip the login cookie from requests to sanoma.fi (or block requests altogether) when hs.fi is the location bar domain.

Please add a same-entity annotation that allows hs.fi to perform cookieful requests to sanoma.fi.

While at it, it would make sense to also allow other sanoma.fi properties to make cookieful requests to the IdP. These are listed as linked logos on https://oma.sanoma.fi/v2/aihe/sanoma-tili/mika-on-sanoma-tili-ja-mihin-sita-kaytetaan

The job runs on jenkins now, but it looks like both Remote Settings AND s3 upload is disabled?

Upload to Remote Setting and S3 disabled.

Right now, tracking protection breakage is handled using the "Content" category of the Tracking Protection lists. That is, when a tracking domain is found to cause an unacceptable level of breakage, it is moved to the "Content" category of the Disconnect list, which is only blocked when the "Strict" version of the Disconnect list is enabled. This has a number of downsides. The main one is that severe breakage on a few popular sites may lead to a tracking domain being whitelisted on all sites.

We should instead consider pairwise whitelisting in these scenarios. That is, when breakage is discovered that would normally lead to a domain being moved from the Basic list to the Strict list, we instead add a pairwise whitelist for that domain. For example, if we discover that tracker.example breaks news.example and video.example, then we only whitelist tracker.example on those two sites when in the "Basic" mode of protection. The platform should already have support for this with our entity lists.

The Disconnect category of the main blocklist file contains googletagmanager.com here, however this is not included in the Google remapping file here. Does that mean we drop googletagmanager.com?

About this Issue

In order to make sure all the domains are proper, we should check the new domains being added, or the entire file, with canonicalize function from shavar-list-creation.

Level 2 category of Disconnect.me breaks videos hosted on googleusercontent.com on websites such kissanime.ac/ru, animehub.ac. etc, please remove googleusercontent.com from the Level 2
list.

@patjack Currently I'm able to reproduce this in Firefox 72

Testurl -- https://kissanime.ac/Anime/Fairy-Tail.78285/Episode-013?id=828&s=ptserver

Hi,

I'm writing on behalf of Internet Brands.

Our website editor has had an upgrade a couple of months ago. Now it uses a WebSockets connection to synchronize some information between multiple browser sessions for same site editing. In Firefox, when the user has the Enhaned Tracking Protection set to Strict, the browser blocks the connection to the WebSockets server wss://smbwebmgr-sockets.internetbrands.com .

In the disconnect-blacklist.json file, there in an entry for the internetbrands.com domain, which obviously blocks ALL (?) subdomains, too. I'm pretty certain that this entry is too broad. Effectively, on all subdomains of the domain, it disables everything for which this list is used for (obviously WS connections are one use case). https://github.com/mozilla-services/shavar-prod-lists/blob/master/disconnect-blacklist.json#L5300

Is this the right place to report this? If not, please advise where to turn to.

I'm assuming these domain have the same owner, but I have not verified that. This leads to the breakage described in Bug 1545092.

About this Issue

Files in /tests and social-tracking-protection-blocklist.json are Mozilla curated lists. We should update the licenses in these files.

Acceptance Criteria

• License updated in .json files in /tests folder no update needed as these are old lists
• License updated in social-tracking-protection-blocklist.json can stay as MPL
• Update the words on the LICENSE file to indicate which files fall under the Attribution-NonCommercial-ShareAlike 4.0 International

We currently don't use:

• adGuard-blacklist.json
• easyList-blacklist.json
• easyPrivacy-blacklist.json
• fanBoyAnnoyance-blacklist.json
• fanBoySocial-blacklist.json

These were added for a test and are no longer needed. Might be nice to remove them + the corresponding config files to clean things up a bit.

About this Issue

After #211 we have decided to move to Attribution-NonCommercial-ShareAlike 4.0 International. We should update the license key value pair in .JSON files from Disconnect

Acceptance Criteria

• Update license value in disconnect-blocklist.json
• Add license key value in disconnect-entitylist.json
• Follow-up issue created on shavar-prod-lists to document structure change and script change #249
• Follow-up issue created on trackingprotection-tools repo to update entity processing mozilla/trackingprotection-tools#18

Right now we have the ability to test content that lists are consumed and applied correctly end-to-end by writing test pages which include our test domains, e.g., trackertest.org. However, we don't include domains in all of the lists we generate from the base Tracking Protection blocklist included in this repo. This makes it hard to do end-to-end tests for certain lists (e.g., Bug 1515818).

Ideally, we'd have a unique test domain present in each of the generated lists to get us do an end-to-end test. I've suggested an approach for doing so in Bug 1514852 Comment 1.

Assuming we have a unique domain in each list -- assume it's <category>.trackertest.example. E.g., analytics.trackertest.example in the Analytics category, fingerprinting.trackertest.example with the "fingerprinting": "true" tag, and effdnt.trackertest.example with the "dnt": "eff" tag.

We should decide where we'd like to consume these domains; I see two options:

1. We can ask Disconnect to add a test domain to each category / tag on the list.
2. We can add these test domains automatically via the lists2safebrowsing.py .

The benefit of (2) is that we can add these domains at will without coordinating with Disconnect. However, if the list creation script has a bug, it could end up creating lists that only have the single test domain. There is a benefit to consuming the test domains in the same way we consume all other domains.

I'm comfortable enough with the other set of checks we have in place to ensure we don't serve empty lists, and thus prefer (2).

@groovecoder do you have a preference?

1. Summary

It would be nice, if AddToAny service will be removed from Firefox blacklist. At the time social buttons of service block in Firefox (beginning at Firefox 63) by default.

If I need to make another actions, that AddToAny buttons works for Firefox users with Always value of Trackers key, please, tell me.

2. Example

• Simple AddToAnyButtons on Codepen

<!-- AddToAny BEGIN -->
<div class="a2a_kit a2a_kit_size_32 a2a_floating_style a2a_vertical_style" style="left:0px; top:150px;">
<a class="a2a_dd" href="https://www.addtoany.com/share"></a>
<a class="a2a_button_mastodon"></a>
<a class="a2a_button_diaspora"></a>
</div>
<script async src="https://static.addtoany.com/menu/page.js"></script>
<!-- AddToAny END -->

3. Expected behavior

If Only in private windows value of Trackers key:

Buttons are shown:

4. Non-expected behavior

Else Always value of Trackers key:

No buttons:

5. Argumentation

5.1. Do Not Track

Privacy Policy from official site:

Do Not Track (DNT):

    + When a supported browser's DNT header is enabled, we prevent tracking across sites where AddToAny is used.
    + For example, we disable Like & Tweet buttons to prevent Facebook & Twitter tracking when DNT is enabled.
    + See our full Do Not Track Compliance Policy.

Full Do Not Track Compliance Policy in AddToAny site.

5.2. EFF

As user, I use Privacy Badger of Electronic Frontier Foundation (EFF) for tracking preventing → Privacy Badger allow AddToAny:

Firefox support EFF blacklists.

5.3. Disabling tracking

See on my Codepen pen:

a2a_config = a2a_config or {}
a2a_config.no_3p = true
a2a_config.track_links = false

This code prevents any tracking:

• AddToAny JavaScript API
• AddToAny: removing the “spy” from the share-ware

Thanks.

iOS requires a fingerprinting list in normalized-lists. For instance it exists here on 72 https://github.com/mozilla-services/shavar-prod-lists/tree/72.0

When "tracking protection" is enabled, I can not ask Yandex about images ( https://yandex.com/images/ ) via AJAX. I do not know if it is possible to allow it while still have the tracking disallowed. I encountered this issue when dealing with WebExtensions.

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC001)

About this PR

As an extension to PR #120 additional checks are needed in Travis CI to better review changes to update the tracking protections lists, information on what domains on what category has been changed is needed:

Acceptance Criteria

• Summary of diff between the branch to be merged and the current branch is needed. Sample output TBD.

Example: https://www.google.com/amp/s/www.nbcnews.com/news/amp/ncna1027251

The Tracking Protection list is used in a number of current and experimental features. We should update the README to remove language that is specific to using the list to block tracking resources.

Hello,

When are you planning to update your disconnect list?

It hasn't been updated for 2 months and it is out of sync with the list at disconnectme.

If a domain is not on any of the blacklists (strict or standard) then it doesn't have to be in the entity list.

About this Issue

Currently there are too much debug output message that is not useful, such as the 200 responses:

ar 10 19:31:07 ip-172-31-30-24 docker-shavar[8789]: [pid: 1|app: 0|req: 128787/290012] 172.31.28.224 () {52 vars in 877 bytes} [Tue Mar 10 19:31:07 2020] POST /downloads?client=navclient-auto-ffox&appver=73.0&pver=2.2 => generated 7 bytes in 1 msecs (HTTP/1.1 200) 2 headers in 78 bytes (1 switches on core 0)
Mar 10 19:31:07 ip-172-31-30-24 docker-shavar[8789]: [pid: 1|app: 0|req: 128788/290013] 172.31.14.23 () {52 vars in 930 bytes} [Tue Mar 10 19:31:07 2020] POST /downloads?client=navclient-auto-ffox&appver=52.9&pver=2.2 => generated 110 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 80 bytes (1 switches on core 1)
Mar 10 19:31:07 ip-172-31-30-24 docker-shavar[8789]: [pid: 1|app: 0|req: 128790/290014] 172.31.9.81 () {52 vars in 915 bytes} [Tue Mar 10 19:31:07 2020] POST /downloads?client=navclient-auto-ffox&appver=52.9&pver=2.2 => generated 320 bytes in 0 msecs (HTTP/1.1 200) 2 headers in 80 bytes (1 switches on core 1)
Mar 10 19:31:07 ip-172-31-30-24 docker-shavar[8789]: [pid: 1|app: 0|req: 128791/290015] 172.31.14.96 () {52 vars in 866 bytes} [Tue Mar 10 19:31:07 2020] POST /downloads?client=navclient-auto-ffox&appver=73.0&pver=2.2 => generated 7 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 78 bytes (1 switches on core 0)
Mar 10 19:31:07 ip-172-31-30-24 docker-shavar[8789]: [pid: 1|app: 0|req: 128791/290016] 172.31.28.99 () {54 vars in 889 bytes} [Tue Mar 10 19:31:07 2020] POST /downloads?client=navclient-auto-ffox&appver=68.5&pver=2.2 => generated 123 bytes in 1 msecs (HTTP/1.1 200) 2 headers in 80 bytes (1 switches on core 1)

We should create a flag to enable/disable the suppression of the 200 responses so that logs like:

Mar 10 19:21:17 ip-172-31-30-24 docker-shavar[8532]: AttributeError: 'LogRecord' object has no attribute 'message'
Mar 10 19:21:18 ip-172-31-30-24 docker-shavar[8532]: Top level Sentry exception caught - failed creating log record
Mar 10 19:21:18 ip-172-31-30-24 docker-shavar[8532]: Skipping 75.0 version support for google-trackwhite-digest256 since the file does not exist in S3
Mar 10 19:21:18 ip-172-31-30-24 docker-shavar[8532]: {"time": "2020-03-10T19:21:18.047933Z", "v": 1, "message": "Skipping 75.0 version support for google-trackwhite-digest256 since the file does not exist in S3", "hostname": "ip-172-31-30-24", "pid": 1, "op": "shavar", "name": "shavar"}
Mar 10 19:21:18 ip-172-31-30-24 docker-shavar[8532]: Traceback (most recent call last):
Mar 10 19:21:18 ip-172-31-30-24 docker-shavar[8532]: File "/usr/local/lib/python2.7/site-packages/raven/handlers/logging.py", line 66, in emit
Mar 10 19:21:18 ip-172-31-30-24 docker-shavar[8532]: return self._emit(record)
Mar 10 19:21:18 ip-172-31-30-24 docker-shavar[8532]: File "/usr/local/lib/python2.7/site-packages/raven/handlers/logging.py", line 145, in _emit
Mar 10 19:21:18 ip-172-31-30-24 docker-shavar[8532]: handler_kwargs['formatted'] = text_type(record.message)
Mar 10 19:21:18 ip-172-31-30-24 docker-shavar[8532]: AttributeError: 'LogRecord' object has no attribute 'message

are surfaced.

Acceptance Criteria

• Flag is created in .ini to enable/disable the suppression of the 200 responses
• Enabling/disabling the flag suppresses the 200 responses

We need to revert the changes in disconnectme/disconnect-tracking-protection@f2550bc that were added for the reasons described in disconnectme/disconnect-tracking-protection#100. These changes push the generated entity list over the url classifier's chunk size limit, which is causing all Firefox clients to fail to update any tracking protection list.

The next steps here are to:

• Revert the Google entity list changes.
• Create a new whitelist similar in configuration to mozstd-trackwhite-digest256, perhaps mozstd-trackwhite-google-digest256.
• Duplicate the Google entity over to that whitelist.
• Consume the new whitelist alongside the standard whitelist in all channels and for all classifier features.
• Exclude the Google entity from mozstd-trackwhite-digest256.
• Add the changes from this PR back into the main list

About this Issue

#322 pointed out that there may be discrepancy between Shavar lists and https://github.com/disconnectme/disconnect-tracking-protection lists. We want to check that there are no other unexpected discrepancy and that checks are happening to prevent that

Acceptance Criteria

• Check the diff between Disconnect's and Shavar's lists

We should be able to automatically summarize the list changes as part of the test script. This will make manual review of list updates easier.

I'm a developer at IBM and the product I'm working on is embeddable on third-party sites.

ibm.com was added to the list of services here: https://github.com/mozilla-services/shavar-prod-lists/blame/master/disconnect-blacklist.json

FYI I saw that it was lifted from the disconnectme/disconnect-tracking-protection repo by the original developer (@carbureted) so I created an issue in that repo as well.

I think this was an oversight since tracking and analytics services are not the sole offerings of IBM. Please remove ibm.com from the list, or change it to the specific offending subdomain(s).

Thank you!

There seems to be an issue with merging the Disconnect changes.

In the Disconnect repo, crashlytics.com was moved from Twitter to Google last year (due to acquisition by Google) in 2019:

disconnectme/disconnect-tracking-protection@b75f0d3

This change does not seem to be merged into your disconnect-blacklist.json.

I was just confused about this behaviour, since I thought these changes would be merged automatically on your side.

Recent Disconnect list entity updates, removals, or domains going from invasive to general fingerprinting. Please new the newest version. Thank so much!

Commit: Moves zadn.vn

We're opening this issue because your project has used Travis CI within the last 6 months. If you have already migrated off it, you can close and ignore this issue.

Travis CI is ending free builds on public repositories. travis-ci.com stopped providingthem in early November, and travis-ci.org will stop after December 31, 2020. To avoid disruptions to your workflows, you must migrate to another CI service.

For production use cases, we recommend switching to CircleCI. This service is already widely used within Mozilla. There is a guide to migrating from Travis CI to CircleCI available here.

For non production use cases, we recommend either CircleCI or Github Actions. There is a guide to migrating from Travis CI to Github Actions available here. Github Actions usage within Mozilla is new, and you will have to work with our github administrators to enable specific actions following this process.

If you have any questions, reach out in #github-admin:mozilla.org on matrix.

Steps to reproduce in Nightly:

Navigate to GCP console
Login and Select "Monitoring" from the left menu
You will be redirected to a Stackdriver login page
Attempts to login will result in Shield + failure to follow link

iovation/iesnare is a product in the fraud and security space. There is some confusion as to why this is on a blacklist when similar products are not on the list.

This issue will be used to track the specific changes that need to be applied to 73 while cutting a 73 versioned branch from master.

The properties section of the Google organization in the entity list has unnecessary entries. Taking them out reduces the size of the list from 314k (10052 entries) to 300k (9599 entries).

Here are the details:

• Properties which aren't (302 to google.com):
  • googleapis.com
  • googlesyndication.com
• Properties which aren't (404 when visiting them directly):
  • destinationurl.com
  • googleadservices.com
  • googletagservices.com
  • googleusercontent.com
  • gstatic.com
• Properties which aren't (no DNS for TLD or www.):
  • 2mdn.net
• Properties which moved (301 elsewhere):
  • admob.com
  • doubleclick.net
  • gmail.com
  • google-analytics.com
  • googleapps.com
  • googleartproject.com
  • googlemail.com
  • googlevideo.com
  • postini.com
  • recaptcha.net

About this Issue

After merging the changes from #110 I saw that the staging S3 showed that there were files updated when we expected to updates:

Diffs of the logs before and after merging #110 shows that the order of the domains have changed causing the list to be "changed" (look at the diff between the .log files attached).
ads-track-digest256-BEFORE.log
ads-track-digest256.log
We should wait to apply #110 to versioned branches so that the domains are added alphabetically to prevent not-real update like this one. We should also remove all the Disconnect category references in the services and configs.

Acceptance Criteria

• We should wait to apply #110 to versioned branches so that we deploy meaningful updates and not a simple re-ordering of the items in the list. Issue filed #112
• The domains added in the list should be alphanumerical to prevent not-real update like this one. Issue filed mozilla-services/shavar-list-creation#132
• We should also remove all the Disconnect category references in the services and configs. See mozilla-services/shavar-list-creation#118 for more details