swissdatasciencecenter / renku-gateway Goto Github PK
View Code? Open in Web Editor NEWGateway between the renku UI and the various renku backend services
Home Page: https://renkulab.io
License: Apache License 2.0
Gateway between the renku UI and the various renku backend services
Home Page: https://renkulab.io
License: Apache License 2.0
the sooner the better, especially with work being done on integration with other services.
README
should be updated with relevant new chart valuesendpoints.json
Update: After the introduction of traefik we don't really need REST docs anymore, but the gateway-auth endpoints should be documented properly.
There's a need for backend services working independently to UI to have access to users' GitLab oauth tokens. As renku-gateway
seems to be currently in charge of managing tokens, it sounds reasonable to create such an endpoint in that service.
GET <renku-gateway-url>/oauth-tokens/{email}
Headers: renku-graph-token -> <some token>
Response:
renku-graph-token
is absent or invalid{ oauth-token: "<user oauth token>" }
This endpoint must be secured and accessible only to known services.
A request to /api/v4/user
with an expired token results in the following response:
{id: 1, name: "Administrator", username: "root", state: "active",โฆ}
Depending on the order in which services come up during deployment, there are cases where the UI enters an infinite redirection loop when trying to log in.
the query for /api/graph/<namespace>/<project>/lineage
is:
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX wfdesc: <http://purl.org/wf4ever/wfdesc#>
PREFIX wf: <http://www.w3.org/2005/01/wf/flow#>
PREFIX wfprov: <http://purl.org/wf4ever/wfprov#>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
SELECT ?v ?w
WHERE {
{
SELECT ?entity
WHERE {
?entity prov:atLocation ?project .
FILTER (?project = <https://dev.renku.ch/gitlab/rokroskar/zurich-bikes-data>)
?entity (prov:qualifiedGeneration/prov:activity | ^prov:entity/^prov:qualifiedUsage) ?qactivity .
FILTER (?qactivity = <file:///85a49605d9a579221841720540ec6248c6775f49#>)
}
GROUP BY ?entity
}
{
?entity prov:qualifiedGeneration/prov:activity ?activity .
BIND (?entity AS ?v)
BIND (?activity AS ?w)
}
UNION
{
?activity prov:qualifiedUsage/prov:entity ?entity .
BIND (?activity AS ?v)
BIND (?entity AS ?w)
}
}
The project URL should be without gitlab/
and the commit sha without a #
at the end.
There are tons of error messages from traefik showing up in the gateway logs.
time="2019-04-18T12:42:01Z" level=error msg="Error while Peeking first byte: EOF"
For requests to gitlab we could rely on an existing python API wrapper: https://github.com/python-gitlab/python-gitlab
It seems to be actively maintained and brings things like paging through results for free.
It looks at in some circumstances when Gatway does the POST to webhook-services' /projects/:id/webhooks
it does send OAUTH-TOKEN
header with some invalid token. We tried a few tests on the UI-team environment and when we did curl webhook-services' endpoint from its container console it works properly. So effectively these two calls where succeeding while failing with 401 from gitlab, when the same endpoint was hit by the Gateway.
curl -XPOST --header "OAUTH-TOKEN: 07fbaa9b39f0e208ab1cf48a819888dfe8740613cbd25bc0d2cb58700872e874" http://localhost:9001/projects/72/webhooks/validation -v
curl -XPOST --header "OAUTH-TOKEN: 07fbaa9b39f0e208ab1cf48a819888dfe8740613cbd25bc0d2cb58700872e874" http://localhost:9001/projects/72/webhooks -v
import app
should work without side effectsimport app.config
should never sys.exit
but rather raise an exception if necessary, better to the checks in the create_app
The proxy/gateway should give clients the possibility to subscribe to events. The API should be something like:
The generate events, the proxy can initially use the gitlab web hooks. When notifications come in over the web hooks, these will be forwarded to the web socket.
See:
So far, we have assumed that a user corresponding to the identified keycloak user would already exist in GitLab. In the future this will not be the case. My suggestion is to just try with the keycloak username and - in case of a {"message":"404 User with ID or username \'demo\' Not Found"
response - post a new user and try again.
MVP
post - MVP
The current test coverage is minimal and arguably not very meaningful. Also,, tests should rely on monkey-patching rather than checking for pytest in the module path.
It seems like some of the browser cookies are forwarded to the backend services where they might be used to authenticate API requests instead of the oAuth access token. This might lead to unexpected results.
Setting up the tests around keycloak keys requires some refactoring.
Once it has the redis backend.
When upgrading keycloak to version 4.8.3 we get 500 Internal Server Error
, right after clicking login button on Renku.
Gateway pod logs:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/quart/app.py", line 1440, in handle_request
return await self.full_dispatch_request(request_context)
File "/usr/local/lib/python3.7/site-packages/quart/app.py", line 1462, in full_dispatch_request
result = await self.handle_user_exception(error)
File "/usr/local/lib/python3.7/site-packages/quart/flask_patch/app.py", line 23, in new_handle_user_exception
return await old_handle_user_exception(self, error)
File "/usr/local/lib/python3.7/site-packages/quart/app.py", line 893, in handle_user_exception
raise error
File "/usr/local/lib/python3.7/site-packages/quart/app.py", line 1460, in full_dispatch_request
result = await self.dispatch_request(request_context)
File "/usr/local/lib/python3.7/site-packages/quart/app.py", line 1508, in dispatch_request
return await handler(**request_.view_args)
File "/code/app/auth/web.py", line 206, in get_tokens
audience=app.config['OIDC_CLIENT_ID']
File "/usr/local/lib/python3.7/site-packages/jwt/api_jwt.py", line 105, in decode
self._validate_claims(payload, merged_options, **kwargs)
File "/usr/local/lib/python3.7/site-packages/jwt/api_jwt.py", line 141, in _validate_claims
self._validate_aud(payload, audience)
File "/usr/local/lib/python3.7/site-packages/jwt/api_jwt.py", line 205, in _validate_aud
raise InvalidAudienceError('Invalid audience')
jwt.exceptions.InvalidAudienceError: Invalid audience
Line where this audience value is set : web.py
A logout should also trigger logout from all "3rd party" backend services (like Jupyterhub and GitLab).
Implement the server-side changes to match SwissDataScienceCenter/renku-ui#307:
This service definitely needs an API spec asap. Related to #29
To find out information about a group the UI uses the endpoint /groups/[group-path]. Group-path needs to be URL-encoded (since a group can be a subgroup of another group), but the gateway is unencoding this path and as a result, Gitlab cannot process the request correctly.
Use gunicorn / uWSGI to run the gateway in production.
When building from a tag, the resulting docker image should not be tagged latest
. Applies to all other repos too.
I can view notebooks/DataPreprocess.ipynb
but not notebooks/Data Preprocess.ipynb
from the UI.
Files that are inside a folder in the file explorer inside a project can't be previewed.
docker-compose.yml
file of renga repo.register-gitlab-oauth-applications
).api.renga.local
And provide the corresponding endpoint in the gateway.
This test does not work, maybe related to master not working ?
In proxy.py, if in
def authorize(headers):
line 122 del headers['Authorization'] is commented out, the test will run properly.
If the line is not commented out, def authorize(headers) will return an empty list.
Existing header exchange functionality could be singled out as an individual service which is called by the forwardAuth
middleware (https://docs.traefik.io/v2.0/middlewares/forwardauth/). The proxying of requests would be taken care of by traefik.
Introduce two versions of the API endpoints:
/api/v0/resource
should proxy to GitLabs API v4/api/v1/resource
should parse the information according to the Renku schema and use Renku terminology (ku instead of api, etc)alternatively, the api version could be specified in the request header.
In the longer run the gateway has to handle state, i.e. store user-specific access tokens for all backend services which do not accept a keycloak token for their API. This will allow us to integrate backend services for which we do not have admin rights (for example not sudo-token for GitLab). The tokens could for example be stored in a redis store.
Once #37 is patched, add some logic to attempt to issue a new token
Currently, there is no coherent strategy about what should be logged in development and production mode respectively. It would be nice to have one parameter which consistently sets the log level throughout the entire application (also that of the imported libraries such as oauthlib).
Update: #140 partially addressed this issue for the traefik
part of the gateway. For the auth
part this remains to be done. Furthermore it should be possible override the log level assumed by the development/production
setting through a chart value.
Should add the sequence diagram to the main docs so we have a better overview of what is going on in here.
At the moment, the graph query only returns the links between files and commits. We want to be able to use more information:
usedBy
, generatedBy
)output_1
All of the logger.info()
logging is missing in the k8s logs while logger.debug()
logs are present. This can lead to situations where the gateway pod is dying without leaving any logs at all.
Original description by @rokroskar:
At the moment, there are no logs provided when things go wrong during bootstrapping of the gateway. For example, if keycloak isn't reachable, there is no message just an unresponsive service. This makes debugging of new deployments difficult.
There are obsolete values set in values.yaml
and there are necessary values which are not documented in values.yaml
. This should be fixed.
With a configuration file
The gitlab sudo token and the oidc client secret are passed as standard environment variables. Would be better to use the k8s secrets for these values.
Create a dedicated projects endpoint which assembles all the project information as described here: https://github.com/SwissDataScienceCenter/renga-ui/blob/development/doc/data_model.md
It seems that the gitlab sudo token isn't needed or used anymore. We should remove it from the charts and the related uses of sudo from the code.
The GitLab api returns several pagination-related X-... and link headers. Make sure these are forwarded by the gateway.
A global setting of gitlab tells where it has to redirect after logout. On instances of gitlab that are not dedicated to Renku, like gitlab.com
, this means the administrator has to choose which users will get a "wrong" redirection.
A solution is to copy the workaround for JupyterHub and use the logout iframe trick.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.