DEPRECATED - Tools to import/export Mozilla Addons XML blocklist into Kinto
Home Page: https://kinto-reader.dev.mozaws.net/v1/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/46.0/
License: Apache License 2.0
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot]")
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Refs mozilla/addons-server#3054 and bug 1283601
What we need is for gfx entries to have two additional text fields: Max application version, Min application version. They should both be optional. If either has a value, the versionRange node should be added as a child of the
gfxBlacklistEntry, like in these examples:<versionRange maxVersion="49.9" /> <versionRange minVersion="40.0a1" /> <versionRange minVersion="48.0.2" maxVersion="49.*" />
The latest kinto-lambda-blockpages_generator v3.2.0 is not working for Python 3.6 run time. We're seeing this error on -prod.
ds from /buckets/blocklists/collections/addons...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
Reading records from /buckets/blocklists/collections/plugins...
[WARNING] 2018-02-28T17:44:29.743Z 068f0194-1caf-11e8-b0fb-7f265a34aae0 Reading records from /buckets/blocklists/collections/plugins...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
Generating index...
[WARNING] 2018-02-28T17:44:30.403Z 068f0194-1caf-11e8-b0fb-7f265a34aae0 Generating index...
write() argument must be str, not bytes: TypeError
Traceback (most recent call last):
File "/var/task/aws_lambda.py", line 238, in blockpages_generator
generator_main(args)
File "/var/task/amo2kinto/generator.py", line 121, in main
generate(kinto_client, collections, args.target_dir, 'collection.tpl', 'record.tpl')
File "/var/task/amo2kinto/generator.py", line 82, in generate
generate_index(records, col_tpl, target_dir)
File "/var/task/amo2kinto/generator.py", line 39, in generate_index
f.write(res.encode('utf8'))
TypeError: write() argument must be str, not bytes
END RequestId: 068f0194-1caf-11e8-b0fb-7f265a34aae0
REPORT RequestId: 068f0194-1caf-11e8-b0fb-7f265a34aae0 Duration: 3192.11 ms Billed Duration: 3200 ms Memory Size: 256 MB Max Memory Used: 58 MB
It doesn't work with python run time v2.7 either. The error being:
START RequestId: 32894d9f-1cb4-11e8-bacf-cf97e862fd59 Version: $LATEST
Syntax error in module 'aws_lambda': invalid syntax (__init__.py, line 65)
END RequestId: 32894d9f-1cb4-11e8-bacf-cf97e862fd59
REPORT RequestId: 32894d9f-1cb4-11e8-bacf-cf97e862fd59 Duration: 0.57 ms Billed Duration: 100 ms Memory Size: 256 MB Max Memory Used: 34 MB
The kinto-lambda-validate_signature sometimes fails without error or traceback information. So far, what we have noticed is that it only happens when checking /buckets/blocklists/collections/addons collection, and it logs Signature KO. with a few lines of other information. We don't know why this happens or what triggers it at this point. And this seems to be intermittent too, for example it can happen a few times in one hour, and then doesn't happen at all most time of the day (The lambda itself runs every 5 minutes).
And just fyi, if it is a successful run, the lambda logs a single Signature OK for each of the input collections.
We've included the logs of one run of the lambda below. Please let us know if there are questions. Thanks!
This script is supposed to be run only once to import the existing blocklist entries into a blank server.
But still, it would be safer that it nevers overwrites anything by default.
Especially the collection schemas.
If we change an existing schema in production, and if records are created in a form that is not supported by clients, it could have disastrous consequences 😨
Dependabot couldn't authenticate with https://pypi.python.org/simple/.
You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.
All of the tests in this library are unit tests. They work OK to verify the logic in each module, but no test ensures that the code functions correctly in situ. We should add a functional test, particularly to test_generator.py.
Refs #80 (comment).
Why do we use lxml rather that python's sdtlib package ?
As discussed here, they can be omitted for now in the amo ui
https://github.com/mozilla-services/amo-blocklist-ui/issues/35
But in the client we have to be ready to support them from day 1, in case they are added in the ui.
I suggest that we don't strip them when importing.
Which makes wonder how does the UI behaves when a record with some unknown fields are loaded in a form. Are they lost or preserved ?
No JSON object could be decoded:
ValueError Traceback (most recent call last):
File "/var/task/lambda.py", line 58, in json2kinto importer_main(flatten_args)
File "/var/task/amo2kinto/importer.py", line 192, in main permissions=constants.COLLECTION_PERMISSIONS)
File "/var/task/amo2kinto/importer.py", line 49, in sync_records permissions=permissions)
File "/var/task/amo2kinto/kinto.py", line 37, in get_kinto_records collection, bucket, permissions=permissions, if_not_exists=True)
File "/var/task/kinto_http/__init__.py", line 357, in create_collection safe=safe)
File "/var/task/kinto_http/__init__.py", line 159, in _create_if_not_exists return create_method(**kwargs) File "/var/task/kinto_http/__init__.py", line 365, in create_collection headers=headers)
File "/var/task/kinto_http/session.py", line 84, in request message = '{0} - {1}'.format(resp.status_code, resp.json())
File "/var/task/requests/models.py", line 841, in json return complexjson.loads(self.text, **kwargs) File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded
As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:
If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].
(Message COC001)
note: sometimes cron commands do not have $PATH we need to make sure that the command can find the `diff`` command.
We got paged because BadSignatureError happened to kinto-lambda-validate_signature lambda. However when looking into the lambda logs, it is not obvious what the lambda tried to validate and failed.
The error just looks like this.
: BadSignatureError
Traceback (most recent call last):
File "/var/task/aws_lambda.py", line 87, in validate_signature
raise exception
File "/var/task/aws_lambda.py", line 74, in validate_signature
signer.verify(serialized, signature)
File "/var/task/kinto_signer/signer/local_ecdsa.py", line 85, in verify
raise BadSignatureError(e)
kinto_signer.signer.exceptions.BadSignatureError
The request is that when logging the BadSignatureError, the validate lambda should also log what it tried to validate and failed. This helps troubleshoot during an incident.
We should probably document this :)
In file included from src/lxml/lxml.etree.c:320:0:
src/lxml/includes/etree_defs.h:14:31: fatal error: libxml/xmlversion.h: Aucun fichier ou dossier de ce type
compilation terminated.
The scripts will run on Lambda. Let's make sure it works, or if it needs some tweaking let's do them
as we did in kinto-amo
Schemas were moved to cloudops
In my current Gecko patch, I had addons.
The current schema of amo-blocklist-ui (mozilla-services/amo-blocklist-ui@d16c882) defines addonId as a required field for addons entries.
But the importer here does include it : https://github.com/mozilla-services/kinto2xml/blob/6d6ce1418c465b11d3539e49c937c1cafb136c36/kinto2xml/importer.py#L17
So the functional test of importing and validating on Kinto should fail. But it does not.
Note:
addonId is being renamed to guid, but still we should figure out why validation is green. https://github.com/mozilla-services/amo-blocklist-ui/pull/87
See https://github.com/mozilla-services/amo2kinto/blob/master/amo2kinto/synchronize.py#L73-L78
It should go through to-review and then to-sign with a different user.
or kinto-blocklist-tools or whatever more specific than kinto2xml
The blocklists client states: https://dxr.mozilla.org/mozilla-central/source/services/common/blocklist-clients.js#360
Also we never actually used it so far it is something that @mozmark might want to use in the future.
It needs to be added in the JSON schema as well as in the blocklist xml file generator.
We're opening this issue because your project has used Travis CI within the last 6 months. If you have already migrated off it, you can close and ignore this issue.
Travis CI is ending free builds on public repositories. travis-ci.com stopped providingthem in early November, and travis-ci.org will stop after December 31, 2020. To avoid disruptions to your workflows, you must migrate to another CI service.
For production use cases, we recommend switching to CircleCI. This service is already widely used within Mozilla. There is a guide to migrating from Travis CI to CircleCI available here.
For non production use cases, we recommend either CircleCI or Github Actions. There is a guide to migrating from Travis CI to Github Actions available here. Github Actions usage within Mozilla is new, and you will have to work with our github administrators to enable specific actions following this process.
If you have any questions, reach out in #github-admin:mozilla.org on matrix.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.