Giter Club home page Giter Club logo

sbt-docker-compose's Introduction

SBT Docker Compose plugin

sbt-docker-compose plugin provides ultimate solution for running integration tests against docker containers with health checking support.

Plugin consists of three runnable tasks: dockerCompose, dockerComposeTest, dockerComposeTestQuick which will be discussed in the next sections.

Comparison to similar projects

Tapad's sbt-docker-compose plugin is the alternative to this plugin, it's main downside is using sbt commands instead of tasks which makes it ineffcicient in multi-module projects. Also it doesn't expose docker-compose commands as it's public API, so you cannot modify options passed to docker-compose cli.

On the contrary sbt-docker-compose plugin is using tasks, exposes docker compose comamnds as it's public API, runs health check tests before running integration tests, provides containers host/port as system property, uses SBT's built in task for running tests/it-test and so many other features.

Features SBT docker compose plugin Tapad's SBT docker compose plugin
Multiproject support Yes No
Health check support Yes No
Testframework agnostic Yes No
Tag substitution Yes No
Container host/port as SystemProperty Yes No
Docker Compose commands as configurable SBT Setting keys Partially No
Autocompletion support Yes No

How to Setup/Enable?

1 - Add the sbt-docker-compose plugin to the plugins.sbt file:

addSbtPlugin("com.github.ehsanyou" % "sbt-docker-compose" % "1.0.0")

2 - Enable the auto-plugin on your desired sbt project:

enablePlugins(DockerCompose)

Race conditions

In order to prevent race conditions caused by identical docker-compose project name in CI nodes, a new option --project-name-suffix has been introduced, value of this option will append to the final project name.

How to use?

Plugin consists of three runnable tasks dockerCompose, dockerComposeTest, dockerComposeTestQuick.

dockerCompose :

Behaves exactly like docker-compose cli, takes options like --project-name with autocomplete support, the only difference is limitiations on supported commands.

It only supports up and down commands, the aim is to facilate running integration test against docker containers, aformentioned commands cover majority of use cases.

In addition to that dockerCompose task takes an extra optional option: --tags which is part of tag subsitution feature that will be explained later.

Task structure:

// dockerCompose [docker-compose options i.e: -p] [docker-compose commands [up, down]] [docker-compose commands options]
dockerCompose -p myproject up -d --tags myservice:new-image-tag

Using SBT's autocomplete feature is the easies way to experiment.

dockerComposeTest :

It runs containers defined in the docker-compose file, provides their host/port as System properties, so they are easily accessable in tests.

Then it waits 'till containers return healthy and runs tests/it-tests depend on how its parameterized, regardless of the test result, it will shutdown the containers and returns with proper status code.

It also supports tag susbstitution same with dockerCompose task.

If you want to see container logs while running integration tests configure dockerComposeTestLogging setting key on your sbt project.

dockerComposeTestLogging := true // default value is false

Task structure:

// dockerComposeTest [docker-compose options i.e -p] [test/testOnly/testQuick/it:test,...] [docker-compose up command options: i.e --force-recreate]
dockerComposeTest --no-ansi it:testOnly "*MyClass" --force-recreate
dockerComposeTestQuick :

It's the same with dockerComposeTest except that it skips shutting down the booted containers.

It helps to save some time during development i.e in cases when it's required to run integration tests multiple times against same set of containers.

Default Behaviour

It's possible to provide default behaviour for supported commands [up, down, test], so it's not required to pass options to the task every time it's invoked.

    dockerComposeCommandOptions :=
        DockerComposeCmd()
            .withOption(("-p", "projectname))

    dockerComposeUpCommandOptions := 
        DockerComposeUpCmd()
            .withOption("-d")
            
    dockerComposeDownCommandOptions := 
        DockerComposeDownCmd()
            .withOption("--remove-orphans")
            
    /* 
      `dockerComposeTestCommand` is not part of docker-compose api,
      because dockerComposeTest runs `docker-compose up` eventually 
      you can configure options passed to `docker-compose up`
    */
    
    dockerComposeTestCommandOptions := 
        DockerComposeTestCmd()
            .withOption(("-p", "myproject"))

So next time if i.e dockerCompose up gets invoked it will fallback to defined default options.

Health Check

dockerComposeTest command as stated before boots up containers and checks for health status of containers in 500ms interval, if all containers with health status support report healthy status it runs the tests. Notice: Plugin omits containers without health check(HealthCheck instruction) status.

Image tag substitution

dockerCompose .. up .., dockerComposeTest, dockerComposeQuickTest tasks support an extra option --tags service-name:image-tag, ..., this plugin uses this option to substitute image tags defined in the docker-compose file on the fly, so it's possible to run integration tests against different versions of services. Also there's a SBT setting key called dockerComposeTags you append tags you want to substitute to it, i.e :

    dockerComposeTags += ("service-name", "tag-name")

tags provided by cli supersede this option.

Container's ip/port as system property

In order to run integration test against running instance of the services their host/port address are required. i.e if the service name in docker-compose file is foobar and scale number of the service is 1 and the host port is 1234 you can get the public interface and container with following key: foobar_1_1234.

You can also access host name and public port of the container individually, just append _host or _port to the property key.

System.getProperty("foobar_1_1234") -> "interface:port" // foobar is the service name defined in docker-compose file, 1 is scale number and 1234 is host port of the container
System.getProperty("foobar_1_1234_host") -> "interface"
System.getProperty("foobar_1_1234_port") -> "port"

Since in most cases services expose only one port, and are not replicated you can access service host/port using service name only.

System.getProperty("foobar") -> "interface:port"
System.getProperty("foobar_host") -> "interface"
System.getProperty("foobar_port") -> "port"

If you can't guess the key just print all system properties, and find your key.

Configuration

There are a number of optional keys that can be set as well if you want to override the default setting.

    // Ignores all provided tasks in project scope ( useful for root projects in multi-project configuration )"
    dockerComposeIgnore := ???
    // Possible docker-compose file paths 
    dockerComposeFilePaths := ???
    dockerComposeHealthCheckDeadline := // Overall deadline for health-checking
    dockerComposeTags := // List of services and and their respective tag you'd like to override, --tags option in cli will supersede tags defined in this setting key.
    dockerComposeProjectName := 
    /* 
      Plugin runs you container in isolation,
      it does the isolation with help of `docker-compose` `-p` option.
      Plugin uses value of this setting as a default value to `-p` option.
     */

Concurrency Level

On multi-project config with too many projects is better to limit concurrency level of dockerComposeTest task, if you don't you'll get some crazy errors from docker-compose.

concurrentRestrictions in Global := Seq(
  Tags.limit(DockerComposeTest, 3)
)

Here I limited concurrency level of tasks using DockerComposeTest tag to 3.

sbt-docker-compose's People

Contributors

dariush-y avatar 2m avatar

Stargazers

Stephen Link avatar Tim Golding avatar Tom Richards avatar s3ni0r avatar Sean Glover avatar Jiro Kugiya avatar David Long avatar Richard Ashworth avatar Evgenii Tsvigun avatar Jakub Liska avatar Courtney Robinson avatar Roy de Bokx avatar Mark avatar Ivan Stanislavciuc avatar Joshua Junqueira avatar Emil Dafinov avatar Aleksandr Vinokurov avatar  avatar

Watchers

Aleksandr Vinokurov avatar James Cloos avatar Ivan Stanislavciuc avatar  avatar Roy de Bokx avatar

sbt-docker-compose's Issues

1.1.0 is not available in maven central

While the plugin can be found in maven central. the jar file and pom are not available. 1.0.0 is there

http://search.maven.org/#artifactdetails%7Ccom.github.ehsanyou%7Csbt-docker-compose%7C1.1.0%7Cjar

Example :

http://repo1.maven.org/maven2/com/github/ehsanyou/sbt-docker-compose_2.12_1.0/1.0.0/sbt-docker-compose-1.0.0.pom

http://repo1.maven.org/maven2/com/github/ehsanyou/sbt-docker-compose_2.12_1.0/1.1.0/sbt-docker-compose-1.1.0.pom

As a workaround one can do something like this

sbt 'set publishTo := Some("releases" at "https://your.repository/nexus/repository/hosted-foo-releases/")' "^publish"

To publish the artifact to your local repository

Unable to disable akka logging

While running tests from sbt, I start receiving log messages from the Actor after some time:

[INFO] [04/08/2019 14:06:38.335] [sbt-docker-compose-b0243dab-c9d4-4dd8-9467-ef5a05572a14-akka.actor.default-dispatcher-7] [akka://sbt-docker-compose-b0243dab-c9d4-4dd8-9467-ef5a05572a14/user/$b] Message [com.github.ehsanyou.sbt.docker.compose.HealthCheckActor$Protocol$ReturnTo] from Actor[akka://sbt-docker-compose-b0243dab-c9d4-4dd8-9467-ef5a05572a14/user/$b#-1605624210] to Actor[akka://sbt-docker-compose-b0243dab-c9d4-4dd8-9467-ef5a05572a14/user/$b#-1605624210] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

I have attempted to disable them by adding the following to my application.conf in src/it/resources, src/test/resources, and src/main/resources without any effect.

akka {
  stdout-loglevel = "OFF"
  loglevel = "OFF"
  log-dead-letters = off
  log-dead-letters-during-shutdown = off
}

Is it possible to disable the logging?

Error when path to docker compose path contains a space

If the docker compose file is located under a path that contains a space the build will fail.

Example :

[error] com.github.ehsanyou.sbt.docker.compose.DataTypes$InvalidExitCodeException: docker-compose -f /var/lib/jenkins/workspace/My Project/API - build and deploy/api-server-integration-tests/docker-compose.yml -p myprojectapiintegrationtests up -d command returned non-zero exit code.

Tab completion is broken by default

When I just add the plugin and type dock and TAB, I get

java.lang.RuntimeException: ./ ./ dockerComposeIgnore is undefined.
        at scala.sys.package$.error(package.scala:27)
        at sbt.Extracted.$anonfun$getOrError$1(Extracted.scala:119)
        at scala.Option.getOrElse(Option.scala:121)
        at sbt.Extracted.getOrError(Extracted.scala:119)
        at sbt.Extracted.getOrError(Extracted.scala:124)
        at sbt.Extracted.get(Extracted.scala:34)
        at com.github.ehsanyou.sbt.docker.compose.parsers.package$.$anonfun$dockerComposeTestParser$2(package.scala:52)
        at sbt.InputTask$.$anonfun$free$1(InputTask.scala:74)
        at sbt.InputTask.$anonfun$mapTask$1(InputTask.scala:19)
        at sbt.std.ParserInstance$.$anonfun$map$2(TaskMacro.scala:50)
        at sbt.InputTask.$anonfun$mapTask$1(InputTask.scala:19)
        at sbt.internal.Aggregation$.$anonfun$applyDynamicTasks$2(Aggregation.scala:164)
        at scala.collection.TraversableLike$WithFilter.$anonfun$map$2(TraversableLike.scala:739)
        at scala.collection.immutable.List.foreach(List.scala:389)
        at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:738)
        at sbt.internal.Aggregation$.applyDynamicTasks(Aggregation.scala:163)
        at sbt.internal.Aggregation$.evaluatingParser(Aggregation.scala:205)
        at sbt.internal.Act$.evaluate$1(Act.scala:431)
        at sbt.internal.Act$.$anonfun$actParser0$6(Act.scala:440)
        at sbt.internal.util.complete.BindParser.$anonfun$completions$3(Parser.scala:735)
        at sbt.internal.util.complete.Completions.$anonfun$flatMap$2(Completions.scala:28)
        at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)
        at scala.collection.immutable.HashSet$HashSet1.foreach(HashSet.scala:320)
        at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:976)
        at scala.collection.immutable.HashSet$HashTrieSet.foreach(HashSet.scala:976)
        at scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)
        at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)
        at scala.collection.AbstractTraversable.flatMap(Traversable.scala:104)
        at sbt.internal.util.complete.Completions.$anonfun$flatMap$1(Completions.scala:28)
        at sbt.internal.util.complete.Completions$$anon$1.get$lzycompute(Completions.scala:42)
        at sbt.internal.util.complete.Completions$$anon$1.get(Completions.scala:42)
        at sbt.internal.util.complete.Completions.$anonfun$$plus$plus$1(Completions.scala:20)
        at sbt.internal.util.complete.Completions$$anon$1.get$lzycompute(Completions.scala:42)

(tested on 1.0.0 as 1.1.0 is not published)

Support arbitrary SBT test configurations

I would like to create a new test configuration I consider distinct from integration tests, say ServiceTest:

# build.sbt
val ServiceTest = Configuration.of("ServiceTest", "servicetest").extend(IntegrationTest)

lazy val root = (project in file("."))
  .enablePlugins(DockerCompose)
  .configs(
    Configurations.IntegrationTest,
    ServiceTest
  )
  ...

However, when I try to run my custom task ServiceTest/test with dockerComposeTest I receive error:

sbt:my-project > dockerComposeTest ServiceTest/test
[error] Expected whitespace character
[error] Expected '/'
[error] Expected '-f'
[error] Expected '--file'
... redacted...
[error] Expected 'test'
[error] Expected 'testQuick'
[error] Expected 'testOnly'
[error] Expected 'it:test'
[error] Expected 'it:testQuick'
[error] Expected 'it:testOnly'
[error] dockerComposeTest ServiceTest/test

In other words, the only currently allowed test commands are test, testQuick, testOnly, it:test, it:testQuick, it:testOnly. This constrains the plugin's use cases and flexibility. It would be great to open this up to any test configurations and commands.

(as a secondary problem, It also hard-codes in the deprecated SBT 0.13 syntax - it:test should now be IntegrationTest/test.)

testOnly doesn't behave as expected

expected behavior:

sbt dockerComposeTest --no-ansi it:testOnly "*MyClass" --force-recreate

should run only the class given.

seen behavior:

this just tries to run the entire test suite

If I add a container_name to my docker-compose.yml file, then curiously sbt dockerComposeTest complains about that unrecognized field, BUT it actually behaves as expected, running only the test class provided. I'm not sure how to go about debugging here. Any advice?

dc file like so:

version: "3.7"

services:
  app:
    image: <my-image>
    container_name: cdp-etl-container # this effects sbt-docker-compose behavior
    working_dir: /var/app

when running with the extra container_name field above, we must remove all parameters passed to sbt dockerComposeTest like so:

sbt dockerComposeTest "testOnly *MyClass"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.