frosner / cluster-broccoli Goto Github PK

• https://gitter.im/FRosner/cluster-broccoli
• in README.md

Description

The black-box test should bring up the docker container of Cluster Broccoli, also start Nomad and/or Consul if required and then execute the integration tests.

Inside the travis build, after the unit tests succeeded, it should build the docker image, run it together with Nomad and Consul and then execute the integration tests.

Test Cases

• no nomad no consul
• Check that creating an instance of each example template + starting it really starts the expected service. This is crucial to make sure that every example is actually working.

References

• http://stackoverflow.com/questions/25818138/why-do-integration-tests-in-a-play-scala-project-are-not-executed-at-sbt-ittes
• https://www.digitalocean.com/community/tutorials/how-to-configure-a-continuous-integration-testing-environment-with-docker-and-docker-compose-on-ubuntu-14-04

Problem

When the instances are persisted in a file it needs to be locked to avoid concurrent access. However, this prevents us from starting multiple Broccoli instances for load balancing or having different configurations.

Solution

Support CouchDB as a persistent instance storage and talk to it over the HTTP API.
This requires us to rework the configuration properties:

• broccoli.templates.storage.type how to 'persist' templates (currently only file supported, which is the directory structure)
• broccoli.templates.storage.url path to look for templates (folder path if type is fs)
• broccoli.instances.storage.type how to 'persist' templates (fs or couchdb)
• broccoli.instances.storage.url path to look for instances (directory if fs, HTTP endpoint of CouchDB if couchdb)
• remove broccoli.templatesDir and broccoli.instancesFile (search all code and documentation)
• Broccoli must only return 200 on an operation if it was successful. So the HTTP API should be blocking in order to avoid users making changes an realizing that someone else overwrote them.
• Link in README + Wiki page explaining the storage back-ends and configuration options. Especially important in order to know how to handle version upgrades with breaking schema changes.
• Nomad job prefix still needs to be respected
• Actually it doesn't make any sense to persist the service and job status information.

References

• https://hub.docker.com/_/couchdb/
• http://docs.couchdb.org/en/2.0.0/intro/api.html

Tasks

• Change the file back-end to a directory based back-end so that each instance gets its own JSON. Reason behind that is that 1) it is easier to navigate and manually delete instances, 2) if during a write something gets corrupted it's only one instance and 3) it will be consistent with the DB back-end where we are writing also instance by instance mainly to avoid version conflicts if people edit different instances. (#121)
• The DB back-end needs to regularly poll the DB for new versions because otherwise you won't notice if another Broccoli is making changes? Or shall we say that multiple changing Broccolis is not supported? => ask the back-end on every request.
• Should Broccoli store any state in-memory or just query the back-end on every request? => After discussing with @Gerrrr we decided not to keep any state but always query the back-end. The queries should be blocking so the HTTP API is only returning 200 if the request was successful. Later we can make it async.
• How to unit test CouchDB? => https://www.playframework.com/documentation/2.5.x/ScalaTestingWebServiceClients
• Add an http-api-test that adds 1000 instances and checks whether they can get retrieved properly (with couchdb)
• no FIXMEs and no TODOs left around this topic
• script in the Wiki documentation that transfers instances from one broccoli to another (for migrating between storages). it should ask the first one and submit it to the second one

Problem

Right now, when the InstanceService receives a message that Consul is not reachable, it does nothing.

Solution

It should not show services but a warning instead, that Consul is not reachable.

Problem

Sometimes when we update the templates and restart Broccoli, we might want to update also the instances in order to apply the new template. The usual way would be to stop them and recreate them. However, this is tedious and requires either some manual work or at least saving the JSON representation of all the instances.

Also, sometimes you might want to use the same instance but just transfer it to a different template.

Solution

There should be a button to convert an instance to an existing current template version.

Use POST and add another field in the JSON which indicates that the instance should update it's template before changing the parameter values.

• In the UI we could just implement it as part of the create/edit modal by adding a drop down where you can select a template. If it's an edit modal, there is a check box whether you want to refresh it (so it does not overwrite the template with every update).
• Also it will be useful to have some primitive versioning of the templates. We can start simple by using an MD5 hash of the template JSON file and show it behind the template.
• Template version needs to be computed also based on meta information (meta.json content)

Questions

• At the moment we don't allow illegal states of instances. I.e. if you try to fill parameter values that do not match the template, it fails. However, this might be annoying when transfering instances between templates. Maybe it is nice to just validate on the point of submission and mark illegal instances somehow? => we should not have illegal states

Problem

We will have Nomad jobs that don't match any template. This can happen if you change the name of a template and forget to stop old ones.

Solution

Show them in a separate category. However, it should not be allowed to start or stop them. They should also respect the prefix (if defined), i.e. you should only see unmatched instances that match the prefix but no template.

InstanceService.updateStatusesBasedOnNomad is the place where we need to change.

Problem

Blocked loading mixed active content "http://ajax.googleapis.com/ajax/libs/angularjs/1.5.2/angular.min.js"[Learn More]
Blocked loading mixed active content "http://cdnjs.cloudflare.com/ajax/libs/restangular/1.3.1/restangular.min.js"[Learn More]

Solution

Include both of them as https.

Problem

An advanced user of Cluster Broccoli needs to know how to create new templates.

Solution

Quick how-to + reference to the example template folder, covering:

• folder structure
• description.txt
• template.json
  • variable replacement {{}}
  • HCL to JSON (nomad run -output)
  • {{id}} as Nomad job ID, because it won't work otherwise (relates to #16)

https://github.com/FRosner/cluster-broccoli/wiki/Template-Definition

Problem

Right now, the example templates depend on snapshot versions of the docker images and wrappers etc.

Solution

Tag the examples and point to the tags.

Problem

When discovering a service of an instance through Consul, we only get address and port, but not protocol.

Solution

Right now, we encode the protocol as tags (e.g. protocol:http). We should decide whether this is the way to go (esp. because : is not supported through the Consul DNS interface and throws warnings) and then document how to do it.

We will go for protocol-http.

Progress on https://github.com/FRosner/cluster-broccoli/projects/2

Problem

When a nomad job is started through the UI, it would be nice to be able to get logs about this job. A Broccoli instance can create one (one-time batch or system jobs) or multiple (periodic jobs) Nomad jobs. A Nomad job has one or multiple allocations which can be pending, running, failing or finishing. The UI should reflect this in a consistent but understandable way.

Solution

The Broccoli HTTP API should have a possibility to query for out and err logs of an instance and the UI should show it somewhere.

References

• https://www.nomadproject.io/docs/http/client-fs.html

https://imagelayers.io/?images=frosner%2Fcluster-broccoli:latest

https://github.com/FRosner/cluster-broccoli/wiki/REST-API-v1

• functionality
• screenshots

Problem

Right now, a service is displayed as soon as it is discoverable. You will not see whether it is healthy. This yields to people clicking on Web-UI links that are not (yet) reachable.

Solution

Ask Consul whether the service is healthy and show this in the UI somehow (e.g. by not allowing to click or changing the color or something like this).

localhost:8500/v1/health/service/<service>?passing returns an empty array if there is no service passing all health checks.

Problem

Right now, each template needs to have the job ID to be exactly {{id}}, so that the mapping between an instance ID and a job ID works. Right now, the two are equal.

Solution

It is not clear and we should think about it, whether to set the instance ID based on the complete Nomad job ID instead of only the {{id}} variable.

Example:

job "zeppelin-{{id}}" {
  # ...
}

Here it might make sense to set the Broccoli instance ID to zeppelin-frank instead of frank, if ID = "frank".

Problem

Travis should push the image containing only the binaries and not activator. openjdk:8-jre should be used as a base image to be lean and small. The activator image can still remain in a different folder so people can use it for quick development.

References

• https://sebest.github.io/post/using-travis-ci-to-build-docker-images/

Tasks

• adjust README example
• adjust README dockerhub badge
• different tags are explained in dockerhub
• travis only pushes to dockerhub on master

Problem

It is quite boring and text-heavy to only have the name "zeppelin" and "jupyter".

Solution

Allow icon.svg inside a template and add it to the front-end.

For the non-docker version,

• It's not mentioned anywhere that you also need to install play
• it's not explained how to start it after packaging it with activator

Problem

After creating a new instance, it is immutable. This forces a user to stop his instance, recreate it and start it again. This can be really tedious if your templates contain a lot of parameters.

Solution

Allow updates on instances.

Design Draft

Updating instances will be done using the POST method on the instances/<id> endpoint. I will deprecate the old POST endpoint which takes only a JSON string for updating an instance status (but keep it for compatibility reasons) and include status updates in it as well.

The POST request contains a JSON object which can be a partial instance. If you want to update the status, you'd have to post:

curl -v -H 'Content-Type: application/json' \
  -X POST -d '{ "status": "running" }' \
  'http://localhost:9000/api/v1/instances/my-http'

Allowed statuses are: "running" and "stopped"

If you want to update the parameter values, you'd have to post:

curl -v -H 'Content-Type: application/json' \
  -X POST -d '{ "parameterValues": { "id": "my-http", "cpu": "250" } }' \
  'http://localhost:9000/api/v1/instances/my-http'

If a request is invalid (e.g. invalid key) HTTP status code 400 (invalid request) will be returned.

Acceptance Criteria

• it needs to check for validity (e.g. trying to change a parameter value that does not exist)
• the instance ID needs to be immutable
• update HTTP API

Problem

Right now, if you restart the web application, all instances are reset. The Nomad jobs will keep running but they are not visible in Broccoli.

Solution

Persist instances.

Problem

Sometimes we might want to allow customization but don't enforce it. There should be a way to define default values in the job templates.

Solution

• introduce syntax for default values in templates ({{name:cpu default:500}})
• prefill the front-end form with the default values
• update HTTP API documentation if necessary

Problem

Right now, all JS libraries are loaded from the internet. This has two problems. First, it doesn't work if there is no internet. Second, it is annoying if the internet connection is slow.

Solution

Is there any problem in just putting all JS files in assets?

Problem

Right now, services are advertised using the service address and port. This has several issues, the main one being that if nomad reschedules the job, using the IP might not allow finding it back. Also it is easier to remember.

Solution

• Configuration flag to enable service names instead of IPs + the consul domain (default is enabled)
• Configuration to customize the consul domain (default is ".service.consul")

References

• https://www.consul.io/docs/agent/options.html#_domain

• update README and Wiki (screenshots, etc.)
• remove SNAPSHOT
• git tag
• activator dist
• Github release
• Dockerhub tag
• bump version

Currently, a line with a "running" service is looking red because the "Stop" button is red, and a line with a "stopped" service looks green because the "Start" button is green. This is misleading.

I would propose to have a green "running" and a red "stopped" status labels.
For action buttons, something like glyphicon-play and glyphicon-stop could be used.

Problem

It would be nice to see some more information about resource utilization.

Solution

• Individual resources used of each instance on each node
• Overall available resources of each node (https://www.nomadproject.io/docs/http/nodes.html / https://www.nomadproject.io/docs/http/node.html)

Problem

Right now, GET /templates returns an array of template IDs. This is not extendable, if we wanted to add more information for each template.

Also the actual JSON template file should not get returned.

Solution

• Make GET /templates return all templates and not only IDs
• Don't return template JSON anywhere
• Document in Wiki

• remove SNAPSHOT
• git tag
• activator dist
• Github release
• Dockerhub tag
• bump version

Problem

Clicking on a service link will close the Broccoli UI.

Solution

Open link in a new tab.

• es gibt eine scala lib für consul
• Akka hat ein Trait für Logging "ActorLogging", das logging kann per logback configuriert werden
• mir erschließt sich der Sin für die @volatile Notation in Aktoren nicht, da sie per se single threaded sind
• Aktoren mit State sind besser per FSM umzusetzen
• InstanceService braucht kein system als Argument, ist per context.system schon als member von Actor enthalten
• Durations können mit implicit conversions aus Ints erstellt werden "1 second" (import scala.concurrent.duration._)
• Futures, desssen ergebnis an den sender geschickt werden mappen/recovern und an den sender pipen (akka.pattern.pipe)
• Json deserialisieren (http://json2caseclass.cleverapps.io/ kann dabei helfen)
• Model: Case classes ohne vars. Wenn members geändert werden müssen die case class kopieren Foo.copy(member = newValue)
• json templating: Play JsonTransformer benutzen, um json properties anhand eines pfades zu setzen. so kann man hässliche string replaces umgehen. Vllt. in das template ein objekt mit zwei members, "template" -> das tempalte JsObject, "template-parameters" -> die parameter definition (name, JsPath to replace, js datatype (JsNumber, JsString, JsValue))

Problem

1. Try to create an instance but only put some of the parameter values (using the HTTP API)
2. Receive the following error:

<!DOCTYPE html>
<html lang="en">
    <head>
        <title>Error</title>
        <link rel="shortcut icon" href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAlFJREFUeNqUU8tOFEEUPVVdNV3dPe8xYRBnjGhmBgKjKzCIiQvBoIaNbly5Z+PSv3Aj7DSiP2B0rwkLGVdGgxITSCRIJGSMEQWZR3eVt5sEFBgTb/dN1yvnnHtPNTPG4PqdHgCMXnPRSZrpSuH8vUJu4DE4rYHDGAZDX62BZttHqTiIayM3gGiXQsgYLEvATaqxU+dy1U13YXapXptpNHY8iwn8KyIAzm1KBdtRZWErpI5lEWTXp5Z/vHpZ3/wyKKwYGGOdAYwR0EZwoezTYApBEIObyELl/aE1/83cp40Pt5mxqCKrE4Ck+mVWKKcI5tA8BLEhRBKJLjez6a7MLq7XZtp+yyOawwCBtkiBVZDKzRk4NN7NQBMYPHiZDFhXY+p9ff7F961vVcnl4R5I2ykJ5XFN7Ab7Gc61VoipNBKF+PDyztu5lfrSLT/wIwCxq0CAGtXHZTzqR2jtwQiXONma6hHpj9sLT7YaPxfTXuZdBGA02Wi7FS48YiTfj+i2NhqtdhP5RC8mh2/Op7y0v6eAcWVLFT8D7kWX5S9mepp+C450MV6aWL1cGnvkxbwHtLW2B9AOkLeUd9KEDuh9fl/7CEj7YH5g+3r/lWfF9In7tPz6T4IIwBJOr1SJyIGQMZQbsh5P9uBq5VJtqHh2mo49pdw5WFoEwKWqWHacaWOjQXWGcifKo6vj5RGS6zykI587XeUIQDqJSmAp+lE4qt19W5P9o8+Lma5DcjsC8JiT607lMVkdqQ0Vyh3lHhmh52tfNy78ajXv0rgYzv8nfwswANuk+7sD/Q0aAAAAAElFTkSuQmCC">
        <style>
            html, body, pre {
                margin: 0;
                padding: 0;
                font-family: Monaco, 'Lucida Console', monospace;
                background: #ECECEC;
            }
            h1 {
                margin: 0;
                background: #A31012;
                padding: 20px 45px;
                color: #fff;
                text-shadow: 1px 1px 1px rgba(0,0,0,.3);
                border-bottom: 1px solid #690000;
                font-size: 28px;
            }
            p#detail {
                margin: 0;
                padding: 15px 45px;
                background: #F5A0A0;
                border-top: 4px solid #D36D6D;
                color: #730000;
                text-shadow: 1px 1px 1px rgba(255,255,255,.3);
                font-size: 14px;
                border-bottom: 1px solid #BA7A7A;
            }
        </style>
    </head>
    <body>
        <h1>Oops, an error occurred</h1>

        <p id="detail">
            This exception has been logged with id <strong>70ljpnhk3</strong>.
        </p>

    </body>
</html>

• remove SNAPSHOT
• git tag
• activator dist
• Github release
• Dockerhub tag
• bump version

Problem

It is not possible to see if the Broccoli REST API is reachable or not from the JS front-end.

Solution

Add an indicator (like a green / red light or something).

Problem

The API should not be mixed with the front-end.

Solution

Move the rest API from the / to /api/v1

Problem

When the template JSON is built in Instance, people might be able to "inject" JSON. We might want to allow to configure an input sanitization (e.g. escaping quotes or other JSON characters). However as a user might want to allow JSON inside we should make it configurable whether you want to sanitize and how (e.g. escape quotes).

Solution

Make sanitization configurable. Should it be on a global level (all input), on a template-parameter level or both? => I think best is to actually treat text parameters as JSON text, which sanitizes it automatically. Thus when you can put "key" : {{value}} instead of "key" : "{{value}}" if {{value}} is a text parameter.

And then we can just add another parameter type which we call "raw" in addition to "string". This relates to #71 also.

• Introduce a new config object for templates and instances (like it is done for nomad in https://github.com/FRosner/cluster-broccoli/pull/254/files#diff-44c416010933ac7e004b47d735296d38)
• Important: If we make "string" the default parameter type then it will break all existing templates. However I think that "string" is the most reasonable default because it is safe. To avoid putting "raw" as default we can either 1) encode the Broccoli version into the template somehow and when we detect that there is no version present we choose "raw" as default, otherwise "string", or 2) add a global configuration parameter to change the default so we can allow users to have some deprecation period.
• Refactor instance parsing so I can inject the config option which parameter type to use as default.

• Models
• Services
• Controllers

Tasks

• Models
• Services
• Controllers

References

• http://andyshora.com/unit-testing-best-practices-angularjs.html

Problem

When creating a new instance, the focus is not inside the textbox.

Solution

Put the focus inside the text box.

Problem

Right now, the CSS is put in a style tag directly in the index.html page.

Solution

Let's put it in a separate CSS file and include it.

Problem

{{spark-master}} as a parameter will not work now because the regex is looking only for numbers and characters.

Solution

Also support - and _ in the Template regex.

Problem

Right now, templates are parsed off of a JSON template for the nomad job + a description text file + an image. In order to make this more structured and be prepared for also a create / update REST endpoint for templates later on, we might want to standardise it.

Solution

• Instead of a description.txt, have another JSON document with meta information (like template name, description, etc.).
• Update https://github.com/FRosner/cluster-broccoli/wiki/Template-Definition

Problem

Right now it is not possible to distinguish jobs created from Broccoli from normal Nomad jobs. Also it is not possible to use ACLs as they are based on prefixes.

Solution

Allow a configurable job prefix. It will be used to set the job ID and also for asking Nomad about the job status.

• prefix is used for creation, updating and showing of instances
• prefix can be queried using the about endpoint
• property should be called broccoli.instances.prefix

Links

Relates to: #16

Available Applications

Application Instances

Problem

One should not only be able to add instances, but also to delete them.

Solution

Add a button to the UI and a REST endpoint accepting DELETE requests to delete an instance. Deletion should also send a DELETE request to Nomad.