Suggested approach:

• In SearchRecommendation, after the result property is determined, but before the classifiers are applied, do an HTTP request for result.url and store the DOM on a property of the SearchRecommendation instance.
• Pass the DOM during the construction of each classifier instance for the result.
• Have helpers on the base classifier to help with the querying of the DOM.
• Each classifier will then have the ability to query the DOM as necessary.
• A new classifier can be written to get the apple touch icon for websites, and a separate classifier can be written to get the og:image for article-type content.

Per https://github.com/mozilla-services/dockerflow

This will help us keep response sizes slim and develop/experiment with new classifiers without punishing users not currently utilizing them.

Moved over from mozilla/universal-search#100, seems like it belongs here.

There are a number of resources in AWS that are used for the deprecated summarizer server or a tiny-machine deployment. As we are no longer using either of those, we should remove those resources.

We should write a classifier applied to all results that attempts to return a favicon from Embedly.

When that i finished, we should back out the changes from #63.

Cloud Services' Dockerflow uses circleci to build docker containers (https://github.com/mozilla-services/Dockerflow). Testpilot just committed a circle.yml that works, we might be able to steal most of that:

mozilla/testpilot@87b7c6e
mozilla/testpilot@86774eb

or there is an example in the Dockerflow link and @mostlygeek can help with questions

In #63, we temporarily turned EmbedlyClassifier on for all results in order to get access to favicons for every site. In #77, we will revert those changes when adding a new classifier solely responsible for dealing with favicons. That means we will see other Embedly data—most notably key images—for results where EmbedlyClassifier is not applied.

When toying around with it, that feels a lot worse. One example of that:

Embedly data is currently included for top level domains, directories, and Wikipedia articles. We don't want to always include it—there are quite a few bytes involved in that—but we could definitely afford to expand the current definition.

Moved from mozilla/universal-search#123

Any CDN or proxy we end up with will respect the standard caching headers, so we should make sure we are sending them.

This can be a no-cache header for empty results and a cache for 24 hours header for results with stuff in them or whatever. cache-control, expires, etags, pragma, etc. whatever you kids are using these days.

🚨

We should write a command that prepopulates the cache based on some combination of a list of common/popular searches and short combinations of characters.

Copied from mozilla/universal-search#1

The operations infrastructure is expecting some endpoints for each project to reply in a standardized fashion. This will be expected for all Test Pilot experiments as well as Test Pilot itself. They are handy for debugging too!

1. Respond to /__version__ with a JSON version object (moved to #41).
2. Respond to /__heartbeat__ with a HTTP 200 or 5xx on error

this endpoint is used by external monitoring to check the health of the service
it is recommended to check the app's dependencies (database, caches, etc) here

1. Respond to /__lbheartbeat__ with an HTTP 200

   used by the load balancer to check if the server and application is OK
   do not include dependency checks as this check may trigger automatic node termination and replacement

• Get CircleCI builds running (#40)
• Remove Travis badge from README
• Get coverage/coveralls working with CircleCI
• Remove Travis webhook

Rather than attempt to calculate the recommendation on a cache miss, we want to kick off a worker process to do so and immediately return a 202.

It's currently shimmed to behave similar to this, but it calculates and caches the value in the request/response cycle before returning the 202.

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/header-caching.html#header-caching-web-selecting

This issue was originally posted as mozilla/universal-search#120 on the universal-search repo, moved to the server repo.

@wresuolc said:

We talked about this in a meeting, but I forget which one.

Now that we have stage, we should run a load tester on it to get a rough idea of what kind of performance we can expect. Even if it's just returning nulls from memcache, it would be helpful to get a rough idea of requests/sec.

Ideally testing this won't cost us a fortune in API queries. :)

@chuckharmston replied:

I don't think we should consider this until we have some sort of upstream cache; either Varnish-like or a CDN.

We could avoid costing us money in API queries by disallowing outbound HTTP requests from the worker server. All the jobs would fail, but as far as load testing the application server, it should be fine (still 1 memcached read per request).

When our server queries Bing's suggestion engine and the query is equal to the top suggestion, Bing tries to try to be helpful and returns full web search results, e.g. this for facebook. Absolutely bizarre behavior, but the fix should be pretty simple.

Reported by @mostlygeek here: https://bugzilla.mozilla.org/show_bug.cgi?id=1256417#c7

Currently it uses MD5 instead of SHA-256. The speed difference is nominal, but SHA-256 is considerably more secure.

Searches are currently case-sensitive. No need for that.

We don't verify the user is above the age of consent, so I'm not sure we should be showing preview content (images/text) from adult sites in response to user keystrokes. I dunno. Maybe legal should be involved? cc @nchapman

Technically speaking, the BOSS API docs mention filters for 'porn' and 'hate', but only provide an example for 'porn':

https://yboss.yahooapis.com/ysearch/web?q={keywords}&format=xml&filter=-porn

¯\_(ツ)_/¯

I got a pagerduty alert. Investigating found that there were thousands of connections open to the redis server:

> netstat -aln | grep 6379 | wc -l
28232

In the logs I see:

...
Apr 08 04:01:59 ip-172-31-3-146 docker[2782]: File "/usr/local/lib/python3.5/site-packages/redis/connection.py", line 482, in _connect
Apr 08 04:01:59 ip-172-31-3-146 docker[2782]: sock.connect(socket_address)
Apr 08 04:01:59 ip-172-31-3-146 docker[2782]: OSError: [Errno 99] Cannot assign requested address
Apr 08 04:01:59 ip-172-31-3-146 docker[2782]: During handling of the above exception, another exception occurred:
Apr 08 04:01:59 ip-172-31-3-146 docker[2782]: Traceback (most recent call last):
Apr 08 04:01:59 ip-172-31-3-146 docker[2782]: File "/usr/local/lib/python3.5/site-packages/redis/client.py", line 572, in execute_command
Apr 08 04:01:59 ip-172-31-3-146 docker[2782]: connection.send_command(*args)
...

The important line in the log is: OSError: [Errno 99] Cannot assign requested address

This is a OS / kernel error that comes up when no more connections to an IP can be made. Most likely this is the result of the __heartbeat__ endpoint being monitored and it opens but doesn't close connections to redis.

TL;DR: Access to the BOSS APIs will continue until March 31, 2016.

Per discussions in #66.

I'm going to keep Yahoo around for now, since the plan is apparently to keep riding those coattails as long as we are able.

Looks like the BOSS service will be removed in ~2 weeks, according to https://developer.yahoo.com/boss/search/:

"At Yahoo, we’re always looking for ways to streamline and simplify products for our customers. With this focus in mind, we will discontinue the BOSS JSON Search API on March 31, 2016.

Access to the BOSS APIs will continue until March 31, 2016. Moving forward, customers leveraging the BOSS JSON Search API can instead use YPA, a Javascript Solution that provides algorithmic web results with search ads for publishers who manage their own search engine results pages (SERPs). Click here to apply or learn more about YPA, or if you are working with a Yahoo Partner Manager, they can help you explore your options."

We should put them in separate files in a new docs directory, and link to the various documentation from the README.

The original plan was to use a multiple-container Docker configuration and use one of AWS' services that allow multiple-container deployment. That's changed, and we've now been asked to use Dockerflow, which is a substantial departure. I've tried to reuse as much work as possible, but there have been enough gotchas that I've talked myself into starting over with the production architecture in mind.

This bug will track design efforts, and bugs resulting from it will be tracked in the Deployment-ready milestone.

Desired end state:

• On successful builds on master, CI will build the application into a Docker container that is then published to Docker Hub and deployed.
• The built container is Dockerflow-compliant.
• Local development is as contributor-friendly as possible.

Proposal:

• Refactor from a multiple-container configuration to a single-container one that only manages the application.
• For local development, use a Compose configuration that manages data storage and task queues.
• Eliminate nginx entirely; use uWSGI directly as a server, and use Flask's send-file for our few static files.

Feedback wanted: @mostlygeek @6a68

The BOSS docs say that filter=-porn is the query string argument to use for adult content filtering: https://developer.yahoo.com/boss/search/boss_api_guide/web.html#filter

Dependency on #2.

Our application should respond to /version with a JSON version object

This was extracted from #34, since it's made much easier by #40 but shouldn't block the former from being merged.

We'd like (as-close-as-is-pragmatically-possible-to-) 100% unit testing coverage on all the code introduced by e2c278b before deploying for usage.

We should use a proxy for embedly to avoid giving up our client's IP addresses. @jaredkerim wrote one at https://github.com/mozilla/embedly-proxy . Let's use it!

Tim says they are using it successfully in production right now so we should be good to go.

https://www.contributejson.org/

As defined here: https://github.com/mozilla/universal-search/blob/master/docs/metrics.md#on-the-server

To help in debugging what part of a service is down __heartbeat__ should return a hint of what's down.

For example:

• memcache down
• redis down
• celery down

It would help us in determine which part of the stack to start investigating in.

That way, the addon can see if results for "fa" and "fac" are identical. If so, they can skip any DOM manipulations entirely.

python-memcached was used instead of pylibmc for AWS compatibility reasons that are no longer relevant.

@mostlygeek found a bug in the Circle CI configuration; tests weren't actually being run, which means that linting fails weren't actually breaking the build. Once #70 is merged, do the following:

• Move linting to Travis
• Remove the excludes being added in #70
• Fix any linting errors that have snuck through

e.g. __heartbeat__ and not heartbeat. When not escaped, Markdown bolds them: version

I'm seeing this in the logs:

Apr 15 00:18:00 ip-172-31-36-68 docker[2852]: [pid: 10|app: 0|req: 73/73] 127.0.0.1 () {32 vars in 395 bytes} [Fri Apr 15 00:18:00 2016] GET /__lbheartbeat__ => generated 0 bytes in 0 msecs (HTTP/1.0 200) 5 headers in 189 bytes (1 switches on core 0)
Apr 15 00:18:00 ip-172-31-36-68 docker[2852]: --------------------------------------------------------------------------------
Apr 15 00:18:00 ip-172-31-36-68 docker[2852]: INFO in middleware [/app/recommendation/mozlog/middleware.py:63]:
Apr 15 00:18:00 ip-172-31-36-68 docker[2852]: --------------------------------------------------------------------------------

Should I set RECOMMENDATION_ENV to prod or stage? Looking through the code it seems setting it to something other than development will disable the debugging output.

We currently only run Embedly for Wikipedia pages and top level directories/domains. Let's run it on everything so we can get access to all the favicons.

Right now, we blindly return a URL for a Clearbit logo by concatenating a base URL with the result's hostname. Let's check for its existence first.