redhat-developer / che-starter Goto Github PK

View Code? Open in Web Editor NEW

5.0 10.0 23.0 645 KB

REST API for managing Eclipse Che workspaces

License: Eclipse Public License 1.0

Java 97.45% Shell 2.17% Dockerfile 0.38%

che-starter's Introduction

che-starter

REST endpoints for managing Eclipse Che workspaces.

Building from command line

Apache Maven is used for building the project:

    $ mvn clean verify

Running

There are several ways for running the project:

Spring Boot Maven Plugin:

    $ mvn spring-boot:run

From Eclipse IDE you can simply right-click on Application.java -> Run As.. -> Java Application
From command line:

    $ java -jar target/che-starter-1.0-SNAPSHOT.jar

To tell it which application properties file to use (located in the src/main/resources directory) specify the spring.profiles.active parameter, like so:

    $ java -Dspring.profiles.active=local -jar target/che-starter-1.0-SNAPSHOT.jar

Once the service is running, it would be available with Swagger documentation on http://localhost:10000/

Docker container:

docker build -t rhche/che-starter .
docker run -p 10000:10000 -t rhche/che-starter

Profiles

There are two available profiles, local and test. To select which profile to use, add the -Dspring.profiles.active property to the command line:

    $ java -Dspring.profiles.active=local -jar target/che-starter-1.0-SNAPSHOT.jar

Debugging

There are several ways for debugging the project:

From Eclipse IDE you can simply right-click on Application.java -> Debug As.. -> Java Application
Running the project in the debug mode from the command line:

    $ java -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=8000,suspend=n -Dspring.profiles.active=local -jar target/che-starter-1.0-SNAPSHOT.jar

Integration test

Integration tests are running as part of the build against multi-tenant Che server deployed on prod-preview osd. In order to successfully run those tests locally OSIO_USER_TOKEN env var must be set before building the project:

    $ export OSIO_USER_TOKEN=<OSIO_PROD_PREVIEW_USER_TOKEN>

Tests can be skipped via -DskipTests mvn command line argument.

CI Jobs

On success, the second job will push the che-starter image to Docker Hub. Another copy is pushed to the local CentOS CI registry. The CentOS CI Registry hosted image can then be used by other components in the CentOS CI services, either as triggers or as a point of integration.

Live preview

http://che-starter-dsaas-preview.b6ff.rh-idev.openshiftapps.com

Testing with Minishift

che-starter can be tested locally against Minishift. The instructions for running Minishift can be found in the How to deploy che-starter on Minishift ? document.

Code Conventions

Indent using spaces only
New line in the end

Apache Maven Checkstyle Plugin is used for validating the code conventions. All conventions can be found in the checkstyle.xml

License

EPL 1.0, See LICENSE file.

License Maven Plugin is used for license management. In order to update headers in source files run the following command:

    $ mvn license:update-file-header

che-starter's People

Contributors

Stargazers

Watchers

Forkers

ibuziuk jfchevrette kbsingh johnsteele l0rd snjeza hectorj2f vpavlin sbryzak sunix katka92 screwtsw rhopp samuzzal-choudhury jmelis gastaldi riuvshin tinakurian

che-starter's Issues

Type of created project is wrong

Need to compare jetty vs undertow performance

Switch from tomcat to jetty boost che-starter performance dramatically. Need to compare jetty vs. underflow [1] performance.
[1] http://undertow.io/

Need to remove @Ignore from 'createWorkspaceTest'

@Ignore("Ignoring because of https://github.com/redhat-developer/che-starter/pull/122")
Had to add this after adding support for setting GitHub user info automatically.
@mlabuda I guess vertx test server should be fixed on 'che-vertx-server' side
see #122

Kubernetes / OpenShift client should be closed after request execution

@mlabuda feel free to add more details

Need to add 'Namespace' parameter to all endpoints

For now OpenShift project is hard-coded in application.properties - https://github.com/redhat-developer/che-starter/blob/master/src/main/resources/application.properties#L16
The name should be obtained from namespace parameter passed from Platform.

Need to split POST workspace endpoint into two separate: for creating new and starting existing

POST /workspace = create new
PATCH /workspace/{name} = edit existing (start existing)

Make che-starter more robust in terms of resource look up

Resource look up (route, DC...) could be based on the information that the deployment is done for a che server image.

Need to change filtering based on description "repo#branch"

Description will be used as metadata for Platform

Need to create Delete workspace endpoint

Readme.md testing with minishift section improvements

@IlyaBuziuk some comments about the doc:

Is minishfit v1.0 required? If that is true we should also change the default login/password
To create the PV we could just reuse the code instructions in rh-che instead of installing gofabric8
Instructions contains 4 logins and that can be confusing. Maybe addressing my second point would help removing the need for the last login
If we use che.$(minishift ip).nip.io address we could remove manaul editing of /etc/hosts

GitHub token must be set during workspace start

If che-server is idled GitHub token will be lost. Need to obtain and set it during workspace startup

Need to return link for starting workspace in GET workspace response

"PATCH /workspace/{id} = edit existing, operate"

Need to pass config.name in POST worskspace endpoint

Need to add custom exception for Keycloak Token processing

setting committer details may fail due to GH settings

We have observed the below stacktrace when a github user's email was set not to be displayed

7:28 PM
org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:94)
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:531)
    at io.fabric8.che.starter.client.WorkspacePreferencesClient.setCommiterInfo(WorkspacePreferencesClient.java:43)
    at io.fabric8.che.starter.controller.WorkspaceController.createWorkspace(WorkspaceController.java:156)
    at io.fabric8.che.starter.controller.WorkspaceController.create(WorkspaceController.java:113)

Live previews are no longer updated on each commit to master

Master CI job [1] is now fixed and image is published to the registry, however changes are not reflected on

[1] https://ci.centos.org/job/devtools-che-starter-build-master/

504 Gateway Time-out when trying to start che-server when it was idled

It takes too long to start che-sever if it was idled:

<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

deployCheIfSuspended [1] is a long-running task
[1] https://github.com/redhat-developer/che-starter/blob/master/src/main/java/io/fabric8/che/starter/openshift/CheDeploymentConfig.java#L41

POST / PATCH workspace endpoints should contain workspace name reference

Workspace and Project templates are read on every request

JSON from templates [1] [2] is obtained on every single request. It should be read only once though

[1] https://github.com/redhat-developer/che-starter/blob/master/src/main/java/io/fabric8/che/starter/template/ProjectTemplate.java#L69
[2] https://github.com/redhat-developer/che-starter/blob/master/src/main/java/io/fabric8/che/starter/template/WorkspaceTemplate.java#L75

Need to add endpoint for "go to the line" functionality

Route protocol is now hard-coded to http:// (need to add https:// support)

https://github.com/redhat-developer/che-starter/blob/master/src/main/java/io/fabric8/che/starter/openshift/CheServerRoute.java#L43

We should provide workspace IDE URL on workspace creation

Che starter creates a wrong workspace

I uses swagger output for CURL to create a workspace:
curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' --header 'Authorization: EmHPEus9-BTL6DMfvy9mdmlhG2WahNW1ezKFe8Eo6pA' -d '{ "branch": "master", "description": "https://github.com/mlabuda/vertx-with-che#master", "id": "aaasdsad", "name": "dasdasdaasdas", "repo": "https://github.com/mlabuda/vertx-with-che.git", "stack": "vert.x" }' 'http://localhost:10000/workspace/oso?masterUrl=https%3A%2F%2F192.168.99.100%3A8443%2F&namespace=eclipse-che'

Try to replace masterURL and Authorization header token by your values and see what happens. It should create a stack on vert.x centos image, but it creates it on eclipse ubuntu jdk8 image.

Add rule to checkstyle plugin for using spaces only in code formatting

Create project API does not take into account workspaceId

I have found out that create project API does not take into account workspaceId. Is it expected behavior ?
Endpoint for project creation [1] does not include placeholder for workspace ID, even though on project creation step there is a line[2] which tries to set it somehow:

        // Next we create a new project within the workspace
        String url = CheRestEndpoints.CREATE_PROJECT.generateUrl(server.getUrl(), workspaceId);

[1] https://github.com/redhat-developer/che-starter/blob/master/src/main/java/io/fabric8/che/starter/client/CheRestEndpoints.java#L24
[2] https://github.com/redhat-developer/che-starter/blob/master/src/main/java/io/fabric8/che/starter/client/ProjectClient.java#L79

OpenShift endpoint details must be stored as a secret or configmap

from #96 discussion:

I would recommend moving the OPENSHIFT endpoint details to a secret or configmap and set defaults that work for local development, rather than production endpoints. We can then manage the secret in prod / stage/ dev as needed.

Add description to Create WS params, it should be just passed

We should not generate description on our own, we dont care what is there

Provide a way for "fail safe"

E.g. when we receive a call in Che starter and we create a resource/resources on OpenShift, we should be able to perform clean up if something go really wrong and the desired state is not achieved. E.g. workspace start-up fail, we should be able to perform clean up on such "broken" resources. Basically calls should/could behave like transactions = either call is performed whole and successful, or we "fail safe back" as it has never happened.

che-starter CI jobs are down

https://ci.centos.org/job/devtools-che-starter/
https://ci.centos.org/job/devtools-che-starter-build-master/

Need to support obtaining OSO token from ENV var instead of using Keycloak server

From the discussion on mattermost:

the issue is that we're currently using the Users oso token to check status of CheHost etc. The user shouldn't hae access to even see the namespace let alone fiddle with the DeploymentConfig etc..
From che-starter side it's a simple matter of optionally getting the token from a env variable instead of looking it up via KC for the OSO token

if the ENV var is set e.g. OSO_ADMIN_TOKEN this token should be used for all operations on the cluster

Reevaluate and refactor WorkspaceCreateParams

Get rid of json template files for token, project, server, workspace creation

json files are used as a request body - need to replace it with proper DTO

401 (Unauthorized) error should contain options with a body stating where is the error

Another thing I need from che starter is a fine grained "401 (Unauthorized)" response on LIST/POST options with a body stating where you got the error e.g. is is the KC token that is broken, or the token for oso or from github that way we can report back to 'someone' in Platform that can redirect user to refresh tokens etc.
che-starter knows which 'broker' you got the token from and if it worked or not. so report that 401

{broker: 'x', msg: 'invalid token' }

Commands are not available after workspace creation

Need to add Stacks commands to workspace creation

[1] https://github.com/eclipse/che/blob/master/ide/che-core-ide-stacks/src/main/resources/stacks.json#L127

Need to increase timeout for workspace creation

workspace is created just fine:

{
"href": "http://che.192.168.42.126.nip.io/che/zkren1m2",
"rel": "ide url",
"method": "GET"
}
However, I am getting issues with project creation - 503 service is unavailable:

2017-03-28 19:26:06.246 INFO 20469 --- [cTaskExecutor-2] i.f.c.s.c.ProjectClient : Creating project against workspace agent URL: http://wsagent.v2o1ryh71ln3hisd.che.192.168.42.126.nip.io/api/project/batch
2017-03-28 19:26:06.916 ERROR 20469 --- [cTaskExecutor-2] .a.i.SimpleAsyncUncaughtExceptionHandler : Unexpected error occurred invoking async method 'public void io.fabric8.che.starter.client.ProjectClient.createProject(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String) throws java.io.IOException,io.fabric8.che.starter.exception.ProjectCreationException'.

org.springframework.web.client.HttpServerErrorException: 503 Service Unavailable

The problem is coupled with timeout that is now = 1 min. This value needs to be increased

Model the workspace creation request using DTOs instead of a template

This will allow for more flexibility and future-proofing.

Do we need to update memmory limits for workspace creation ?

See https://github.com/redhat-developer/che-starter/blob/master/src/main/resources/templates/workspace_template.json#L12

cc. @amisevsk @l0rd

This might be fixed on fabric8 template side

Need to provide endpoints for starting idled che-server and getting it's state

For now scaling up che-server pod is a sync, long-running task which results in 504 error - #145

Need to provide endpoints for starting idled che-server and getting it's state

PATCH /server - start idled che-server
GET /server - return state of the che-server pod

che-starter fails to talk with OSO due to certificate issue

Che-Starter deployed on prod-preview[1] fails to talk to our OSO[2], with what seems to be similar issues as what KeyCloak had with certificates. Locally (as a jar and in docker) the certificate issue is not reproducible and che-starter can talk with OSO[2] without any issues.

Running keytool -list -keystore $JAVA_HOME/lib/security/cacerts -storepass almighty | grep tsrv in both local and oso container return the same value
tsrv.devshift.net-1, Mar 16, 2017, trustedCertEntry,
Certificate is installed but for some reason is not used when kubernetes client is called

[1] http://che-starter.prod-preview.openshift.io/
[2] https://tsrv.devshift.net:8443

Create tests for most of API endpoints.

Need to set user name & email in workspace git preferences

PUT request to wsmaster/api/preferences should do the trick

API for obtaining user info based on github token is already in place - #73

cc: @sunix

Need to reduce che-starter image size

From email thread "che-starter size"

KB:
The che-starter built image is over 0.5GB in size. It takes 20 times
longer to deploy than the entire core api (13 seconds V/s 0.8 )
What options do we have here to try and address this ?
Pavol:
do we need JDK for running it, isn't JRE enough? You can decrease image size by ~100MB by not installing JDK and cleaning Yum cache.

JDK is now required because of the certificate installation ( javac [1] is involved)

[1] https://github.com/redhat-developer/che-starter/blob/master/Dockerfile#L21

Need to create a mapping between Stack ID and project Type

For now all projects are created with 'maven' projectType [1]. Need to create a mapping between Stack Id and project Type e.g. "node-default" -> "node-js"

All project types are defined in https://github.com/eclipse/che/blob/master/ide/che-core-ide-templates/src/main/resources/samples.json

[1] https://github.com/redhat-developer/che-starter/blob/master/src/main/resources/templates/project_template.json#L3

Need to track disk space on workspace creation

If there there is not enough disk space - throw custom exception with "out of disk space" message

Can not create a new workspace when 1 is already running

Failure is on resource limits and both workspaces seems to be killed:

Error creating: pods "che-ws-dm89d8k9yx96epes-1200552331-" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=2539m,limits.memory=1300Mi, used: limits.cpu=3906m,limits.memory=2000Mi, limited: limits.cpu=4,limits.memory=2Gi

Most things are aync in openshift so things are just 'marked' for shutdown etc..
Probably need to poll the state before continuing execution [1]
However, if we really need to poll the pod state from openshift and not workspace state from che-server this becomes complicated...

[1] https://github.com/redhat-developer/che-starter/blob/master/src/main/java/io/fabric8/che/starter/client/WorkspaceClient.java#L134

POST /server endpoint is not working

API for creation che-server is not working now and it is not required for summit. However, It would be great to implement it

Keycloak endpoints should not be hard-coded

from @alexeykazakov comment to https://issues.jboss.org/browse/CHE-142

BTW, don't use harcoded FQDN URLs to our services in che-started (or in other places) - https://github.com/redhat-developer/che-starter/blob/master/src/main/java/io/fabric8/che/starter/client/keycloak/KeycloakEndpoint.java#L22
Use relative URLs instead: : : //sso./auth/realms/fabric8/broker/github/token
So if the request to the che-starter is to che.domain.org or whatever then use sso.domain.org/auth/.. to call our Keycloak. Then we won't need to change the code when switching to another domain name.

Create a status endpoint for health check

Discussed this with @ibuziuk - interim option would be to use /swagger-ui-html but as it returns ~3.5KB, it might not be optimal for health checks which are called every couple seconds.

We need an endpoint which will return status 200 if everything is ok and the app is fully started and working and 404 (or other error code) if it is not - this is used also for readinessProbe which influences whether the container started properly or not when you do new deployment so it will help us to make sure new deployments actually work.

Please ask should you need more information

Internal update of Che

Che needs to do any 'internal' update itself, e.g. be able to run Che n+1 on the same PVC as Che n.

che-starter needs to pass the keycloak authentication token that it receives from platform to che server with every request

Expected header:
Authorization: Bearer <KEYCLOAK_TOKEN>