audiocommons / ac-mediator Goto Github PK

The search endpoint misses important functionalities like the ability to filter results from third party content providers. Adding support for filters is necessary before making the first public release of the Audio Commons Mediator.

Should we already start including API versioning in api url endpoints? /api/v1/...?

After linking a 3rd party service the link services page should be reloaded. We tried with window.opener.location.reload() which worked fine in our local dev environment, but in production it raises an 'is not a function' error. In production we tried with window.top.location.reload() and it does not raise an error but the page does not get reloaded.

When users link with services (e.g. user A links his Audio Commons account with Freesound), some credentials (e.g. access token, refresh token, ...) are returned. We currently store these as a json object in our database. We should probably encrypt them in some way, but we also need to be able to de-encrypt them when we make requests to the services on behalf of users. What should be the best way to approach this?

Out current development setup includes a Docker container with Nginx to serve https requests. (For development) We use a custom ssl certificate which is included in our repository. This certificate will expire at some point so we should probably automatically generate a new certificate every time the Docker container is rebuild.

There are the instructions that we originally followed to generate the self-signed certificates for development.

Dockerfile for django/gunicorn container uses an ADD . instruction to copy all code to /code/ folder in the container and then performs pip install -r requirements.txt This works but it's not optimal for the Docker cache as contents of /code/ change when files are edited and therefore cache is invalidated before the pip install ..., causing Python dependencies to be re-installed every time we build the container (see here).
We should first use COPY instruction to copy requirements.txt, then run pip install and then do the ADD . (just as suggested in the basic django+docker tutorial).

As reported by @viviengenet:

Calling the OAuth endpoint without trailing slash at the end, does not return a 404 but instead says no credentials have been provided:

curl -X POST -d "client_id=CLIENT_ID&grant_type=password&username=USERNAME&password=PASSWORD" https://m.audiocommons.org/api/o/token
{"status_code":401,"detail":"Authentication credentials were not provided."}

We can expect that most 3rd party services will implement some kind of oauth like mechanism so we can link accounts. This requires refreshing access tokens when these expire and we need a way to handle that to avoid users having to link their Audio Commons accounts too often. Ideally users should only need to link their Audio Commons account with 3rd party services once.

Access tokens that expire can be renewed using a refresh token. This is not a problem and can be implemented as some kind of routine in the "request" method of BaseACService (for example). If request returns expired access token, get a new token (using the refresh token) and repeat request.
However, what happens when refresh tokens expire? Then users needs to re-link the account. To avoid that we should make sure that we renew access token before the corresponding refresh tokens expire.

Refresh tokens typically have long life cycles (some times they never expire), so we probably need some kind of asynchronous job that checks for refresh tokens that are about to expire and renews the corresponding access tokens. To do that the expiration time of the refresh token should probably be indicated somewhere in ACServiceAuthMixin.

Some things must be changed in the form that is presented to developers when they want to create an API client (credentials):

• client id and client secret should not be editable
• client type and authorization grant can be hidden and set to password mode

We'll also need to update some labels and add help text in Redirect uris field

Request distributor is in charge of processing an incoming request and propagating it to the different services that can answer it. Current simple implementation propagates the request sequentially and in a synchronous-blocking fashion, meaning that process waits until responses from ALL services are received.

Alternatively we should provide an asynchronous implementation which propagates requests in parallel and on each response aggregates the contents using the results aggregator. Current implementation of results aggregator should probably be fine for the asynchronous results distributor. If we provide this asynchronous mode, then we should probably have an API endpoint for 'collecting' request results which given a request_id/response_id returns currently aggregated results and its status (whether all results have already been collected or not).

Maybe the synchronous/asynchronous mode could be a preference option for 3rd party applications.

Our Docker setup includes an Nginx container which routes requests to the Django application via Gunicorn. Requests made over http are redirected to the https version. Our intention is to only allow https and this is why we added this redirect. However the redirect is probably not a good idea as it could happen in some situations that after the http request with sensible data (such as access tokens) the client is transparently redirected to https without even noticing. That would mean that http requests would still be made.

We need to handle http requests in a better way, most probably returning an error if the request is made inside the domain of the API, and allowing the redirect if the request is performed in the web frontend.

The title is quite self-explicative ;)

When API credentials are created, developer should also be able to check a number of services from a services list. When end users login to audiocommons.org and go to "link services" page, we could recommend the services recommended by the apps the user has used (the ones for which she has granted permissions).

When users register they are automatically activated.
We probably need to have them inactive by default and send an email with a link to activate the account.

We should also consider adding captcha recognition to the registration form.

Currently the acservice package is provided as part of the Audio Commons Mediator.
New third party services are expected to be added to the ecosystem by implementing a "service plugin" using the acservice package. To facilitate this, acservice should be provided as a package completely separated from Audio Commons Mediator (should also be in a different repo).

Dependencies form the acservice package to the rest of the mediator should therefore be eliminated. If I'm not wrong these dependencies are only exception classes.

Also, the service integration testing functionality should be separated from the mediator and probably provided as a script in acservice package which might generate an output html file with test results (or output to console). Then the service integration testing part of the mediator can be eliminated as won't be really required there.

Services configuration should also be decoupled from the mediator and config parameters should be loaded from a file per service.

API endpoints are prepared so that authentication mechanisms can be defined but this has not yer been enabled (current API works without authentication).

According to our design decisions we will ONLY allow OAuth2 authentication, but we will allow the password grant therefore it will be easier to implement for 3rd party developers.

Requiring OAuth2 authentication should be a matter of simple configuration with our current Django Rest Framework + Django OAuth Toolkit setup.

It would be desirable to have some way of easily monitoring Redis and Celery state.
For Celery we want to know number of workers and active workers. For Redis we want to know number of keys in store and possibly size?

Project is missing password change and password recovery functionalities.
We can use the standard ones provided by Django.

The current API lacks a "/me" endpoint which returns basic user information such as an Audio Commons user_id and username so that applications can identify the users and link their accounts with any local data they want to store (or simply do other things like displaying the username in the interface).

We want to provide valid linked data format (json-ld) that can load Audio Commons responses into a triplestore. See initial experiments in branch https://github.com/AudioCommons/ac-mediator/tree/jsonld_output

We can use this issue to discuss how to implement that and make experiments.

Every time a new request is made to the endpoints of the Audio Commons API (e.g. search), a response dictionary is created and stored in a shared key-value store (we use a Redis backend). This dictionary is updated as soon as new responses from 3rd party services are received and can be consumed by clients using the /collect endpoint of the Audio Commons API. We should figure out how to clean up old stored responses that have already been collected.

We implemented the setting DELETE_RESPONSES_AFTER_CONSUMED that will delete the response dictionary immediately after it has been collected. This will only allow the response to be consumed (collected) once. We probably need a softer solution like a periodic deletion of old responses. We can probably set a periodic task with Celery (all infrastructure is already set up as we use Celery to perform requests) and use the response_dictionary['meta']['sent_timestamp'] value of stored responses to decide which ones to delete.