The push-okr from mozilla-services

1.3 Create report containing the top 5 forms of site abuse or misuse.

(e.g. bad encryption, send to an invalid URL, etc.)

4.3 Implement the uptime dashboard

Depends on #12.

3.4 Relax VAPID checks for Partner Endpoints

5.3 Have database of historical load-test results

For future regression testing, we should begin recording load-test results in a database.

1.2 Use AWS TTL's for db tables

Issues:

3.2 Create dashboard showing internal vs. external push usage

1.4 Transition away from Aha to another system

4.1 Create Push KPI page on Mana with embedded graphs

Full KPI plan is at https://docs.google.com/document/d/1I45VfUnyMmHYm2bVrh7CGpaUK0826fNiIWbF30qS_I0/edit

The new mana page should use embedded graphs as needed from re:dash and DataDog.

2.3 Revamp load/smoke-testing architecture to address shortcomings in loads-broker.

Using AWS, create a new architecture to address the short-comings we have experienced with loads-broker.

1.1 Develop key push metrics document

Define metrics are we using to identify funnel & engagement.
Attempt to determine a causality between DAU and Push Subscriptions.

2.1 Report generated twice a month for status email...

2.1 Report generated twice a month for status email that includes AWS resource cost per-user, and cost of service mis-use/abuse

2.2 Four code refactor issues completed and obsolete legacy code removed...

2.2 Four code refactor issues completed and obsolete legacy code removed to improve quality of code and pay down technical debt

3.1 Publish an educational developer blog post.

Produce a push system blog post highlighting best practices.

1.5 Setup and hold a meeting with DevRel for Hacks post on Topics

Setup and hold a meeting with DevRel about possible Hacks post on new Topic functionality.

2.1 Meet with existing partners to gather information and needs/objectives

4.2 Push Docs and Specs reviewed (update for latest changes & pull in new library updates).

1.4 Dashboard of Quality of Service

Simple dashboard indicating server availability & use for the service as a whole. Including error rates, profile timing for transaction latency, etc.
Data on nginx cache hits vs. not
95 and 99 percentile timings for message delivery
95 and 99 percentile round-trip latency
Error rates
Time to Deliver from storage vs. online
Message delivery timing breakdowns based on message destination
Heavy senders by IP/User-Agent
Odd URL targets (Excessive messages to small groups of URL’s)

1.2 Implement Kibana dashboard for data without external team deps from 1.1

Issue mozilla-services/push-service#56

1.1 Partner endpoint ready for production deployment

3.3 Create dashboard breaking down external VAPID usage

2.1 Automate smoke-test integration with GitHub.

(Q: bclark adding

AP-loadtester scenario that verifies smoke-test
Test ready version of Ardere (AWS based load testing system).
automated smoke testing for QA may involve tarek.)

4.3 Push usage of third-party libraries on GitHub is logged and verified.

5.1 Improve the loads-broker for basic QA requirements

Document and implement QA requirements for consistent load-test running with metric capturing.

3.1 Additional bridge API methods provided...

3.1 Additional bridge API methods provided to help the client team understand errors they are receiving on the server side.

mozilla-services/autopush@71035fa
mozilla-services/autopush#858

1.1 Rust connection node productized

Issues:

3.1 High volume authenticated endpoint

Deployed to stage environment with load-testing results.

3.1 Create and implement a Rust PoC

3.1 Test alternate endpoint/connection deployment scenarios to reduce needed EC2 instances

Pypy timings and resource requirements vs. older cpython
Running multiple app servers per instance
Record message processing times for eventual comparisons to gauge effectiveness of cost reduction strategies

3.2 Update documentation to latest specs/tooling

Review existing documentation and update to reflect new specs/tooling.
Agree via consensus that the documents are valid to current system

3.3 Retire SimplePush

Issues:

mozilla-services/autopush#799

Related PR's:

mozilla-services/autopush/pull/981

3.2 Release improvements to Assist libraries for subscription providers...

3.2 Release improvements to Assist libraries for subscription providers. (pywebpush & vapid to support latest Draft / update test pages to use latest Drafts)

4.2 Create a plan for an uptime dashboard

5.2 Regularly run automated load-testing

Our load-tests should be run regularly on a schedule.

2.2 Start saving load testing results/metrics

Ensure we have started saving load testing results/metrics.

2.3 Dashboard for memory use over time

1.3 Jenkins API integration with Ardere

1.2 New smoke-tester for autopush

1.3 Create decision criteria document for spec & non-spec Push features

A decision criteria document should be created, and included on the push-service repo that tracks how we decide to implement spec/non-spec Push features, such as: