newrelic / centurion Goto Github PK

Hi,

I wanted to open the conversation here before committing to any work and send a PR. It would seem that some of the operations being performed currently via the CLI can be easily be shifted to regular calls to the remote API. Except for the console case this would make sense to me. Would this make sense or am I missing some reason why you chose the CLI? I suspect it was possibly for development speed, but I could be wrong and maybe there's a benefit. Let me know if it sounds good and I'll work on it.

Cheers.

It would be great if centurion supported an undeploy or stop/remove action. Specifically for when you need to retire an app/container or relocate it to a new host. And it could take an optional --host or --hosts option to filter which host you are targeting.

One use-case would be I might deploy a POC app to one shared docker host to test things out in a staging environment, but not want it to be long-lived. You'd have to run the docker stop / remove commands on the host instead of using centurion to remove / retire it.

Hi!

I have configured a server to use HTTPS on the docker API. I have added the certificates on ~/.docker/, and I can connect using docker. As an example:

tpiza@spring:~/projects/mobilestatus$ docker --tlsverify -H=sheriff.domain.com:2376 ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3e0adaab3d21 domain/humhub:latest "/apache-start.sh" 2 days ago Up 15 minutes 443/tcp, 0.0.0.0:8081->80/tcp humhub-humhub
497965b559cb domain/mariadb-replicator-db:latest "/docker-entrypoint. 2 days ago Up 3 hours 3306/tcp humhub-mariadb
1d45231852e0 tianon/exim4:latest "entrypoint.sh exim 2 days ago Up 3 hours 25/tcp exim4

But i'm completely unable to deploy/list anything using centurion 1.6.0. My configuration:

tpiza@spring:~/projects/mobilestatus$ cat config/centurion/redis.rake
namespace :environment do
task :common do
set :image, 'library/redis'
set :name, 'redis'
end

desc 'Staging environment'
task :staging => :common do
set_current_environment(:staging)
host '192.168.1.205:2375'
host_port 16379, container_port: 6379
end

desc 'Production environment'
task :production => :common do
set_current_environment(:production)
set :tlsverify, true
host 'sheriff.domain.com:2376'
host_port 6379, container_port: 6379
end
end

And the error (obviously I have generated the certs specifying the correct hostname, otherwise it will not work when i connect directly using the docker client :S):

tpiza@spring:~/projects/mobilestatus$ centurion -p redis -e production -a list
** Invoke environment:production (first_time)
** Invoke environment:common (first_time)
** Execute environment:common
** Execute environment:production
** Invoke centurion:setup (first_time)
** Invoke centurion:clean_environment (first_time)
** Execute centurion:clean_environment
** Execute centurion:setup
** Invoke list (first_time)
** Execute list
** Invoke list:tags (first_time)
** Execute list:tags
GET: "https://library/v1/repositories/redis/tags"
E, [2015-05-14T17:43:12.975252 #1199] ERROR -- : Couldn't communicate with Registry: getaddrinfo: >Name or service not known (SocketError)

** Invoke list:running_containers (first_time)
** Execute list:running_containers
I, [2015-05-14T17:43:12.975549 #1199] INFO -- : ----- Connecting to Docker on sheriff.domain.com -----
E, [2015-05-14T17:43:13.168049 #1199] ERROR -- : Couldn't communicate with Docker on sheriff.domain.com: hostname does not match the server certificate (OpenSSL::SSL::SSLError)
/usr/lib/ruby/1.9.1/openssl/ssl-internal.rb:129:in post_connection_check': hostname does not match the server certificate (OpenSSL::SSL::SSLError) (Excon::Errors::SocketError) from /usr/lib/ruby/gems/1.9.1/gems/excon-0.45.3/lib/excon/ssl_socket.rb:137:ininitialize'
from /usr/lib/ruby/gems/1.9.1/gems/excon-0.45.3/lib/excon/connection.rb:387:in new' from /usr/lib/ruby/gems/1.9.1/gems/excon-0.45.3/lib/excon/connection.rb:387:insocket'
from /usr/lib/ruby/gems/1.9.1/gems/excon-0.45.3/lib/excon/connection.rb:106:in request_call' from /usr/lib/ruby/gems/1.9.1/gems/excon-0.45.3/lib/excon/middlewares/mock.rb:47:inrequest_call'
from /usr/lib/ruby/gems/1.9.1/gems/excon-0.45.3/lib/excon/middlewares/instrumentor.rb:22:in request_call' from /usr/lib/ruby/gems/1.9.1/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:inrequest_call'
from /usr/lib/ruby/gems/1.9.1/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in request_call' from /usr/lib/ruby/gems/1.9.1/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:inrequest_call'
from /usr/lib/ruby/gems/1.9.1/gems/excon-0.45.3/lib/excon/connection.rb:233:in request' from /usr/lib/ruby/gems/1.9.1/gems/excon-0.45.3/lib/excon.rb:229:inget'
from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/centurion/docker_via_api.rb:19:in ps' from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/tasks/list.rake:45:inblock (3 levels) in <top (required)>'
from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/centurion/deploy_dsl.rb:6:in call' from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/centurion/deploy_dsl.rb:6:inblock (2 levels) in on_each_docker_host'
from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/centurion/docker_server_group.rb:22:in call' from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/centurion/docker_server_group.rb:22:inblock in each'
from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/centurion/docker_server_group.rb:20:in each' from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/centurion/docker_server_group.rb:20:ineach'
from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/centurion/deploy_dsl.rb:6:in block in on_each_docker_host' from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/centurion/deploy_dsl.rb:6:intap'
from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/centurion/deploy_dsl.rb:6:in on_each_docker_host' from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/tasks/list.rake:43:inblock (2 levels) in <top (required)>'
from /usr/lib/ruby/1.9.1/rake/task.rb:205:in call' from /usr/lib/ruby/1.9.1/rake/task.rb:205:inblock in execute'
from /usr/lib/ruby/1.9.1/rake/task.rb:200:in each' from /usr/lib/ruby/1.9.1/rake/task.rb:200:inexecute'
from /usr/lib/ruby/1.9.1/rake/task.rb:158:in block in invoke_with_call_chain' from /usr/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize'
from /usr/lib/ruby/1.9.1/rake/task.rb:151:in invoke_with_call_chain' from /usr/lib/ruby/1.9.1/rake/task.rb:144:ininvoke'
from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/capistrano_dsl.rb:88:in invoke' from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/tasks/list.rake:5:inblock in <top (required)>'
from /usr/lib/ruby/1.9.1/rake/task.rb:205:in call' from /usr/lib/ruby/1.9.1/rake/task.rb:205:inblock in execute'
from /usr/lib/ruby/1.9.1/rake/task.rb:200:in each' from /usr/lib/ruby/1.9.1/rake/task.rb:200:inexecute'
from /usr/lib/ruby/1.9.1/rake/task.rb:158:in block in invoke_with_call_chain' from /usr/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize'
from /usr/lib/ruby/1.9.1/rake/task.rb:151:in invoke_with_call_chain' from /usr/lib/ruby/1.9.1/rake/task.rb:144:ininvoke'
from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/lib/capistrano_dsl.rb:88:in invoke' from /usr/lib/ruby/gems/1.9.1/gems/centurion-1.6.0/bin/centurion:80:in<top (required)>'
from /usr/bin/centurion:23:in load' from /usr/bin/centurion:23:in

'

Did I have any mistake in my config/environment, or there is any real problem? Thanks a lot for your time!

I have something like this on my config/centurion/mysite.rake.

  task :common do
    set :image, 'https://registry.hub.docker.com/u/myorgn/myrepo'
  end

When I run centurion -p trendnu -e production It doesn't get the right URL.

** Invoke list:tags (first_time)
** Execute list:tags
GET: "https://registry.hub.docker.com/v1/repositories/https://registry.hub.docker.com/u/myorg/myrepo//tags"

If I try simply set :image, 'myorg/myrepo' then I get the following URL error

** Execute list:tags
GET: "https://pelcasandra/v1/myorg/myrepo/tags"

Is this a bug or I'm missing something? Thanks.

Now that there is Dogestry support, it would be good to have the ability to choose the means of deployment via an env var.

This is useful as a fall back mechanism, so if S3 is unavailable then someone can fall back to using quay.io or docker hub.

Right now you can only have environments from this list -

possible_environments = %w[development integration staging production local_integration]

We actually have 2 production environments and our teams deploy to both (two physical data centers). The problem is that I cant do that under 1 task in the rake file because each environment has its own set of docker hosts.

It would be nice if the environment list was customizable so I could do

desc 'Bar environment'
  task :bar => :common do
    set_current_environment(:bar)
    host "my-bar-docker-host"
  end

desc 'Foo environment'
  task :foo => :common do
    set_current_environment(:foo)
    host "my-foo-docker-host"
  end

Does that seem reasonable? if so I may work on submitting a PR for this.

Currently the command line interface for centurion will deploy either the provided tag, or default to 'latest'. For teams using CI, this becomes problematic when latest points to a potentially unstable image. This isn't a problem for anyone already familiar with the tags used by the team, but if a secondary team (DevOps, SRE, hosting, etc) need to deploy, they wouldn't know what tag to use.

My idea is to add a config value that allows each environment to optionally specify the tag that should be used if no tag is provided on the command line. This also leaves the flexibility for a team to have a deploy tag name that may not match the environment name for historical reasons. If no deploy tag is specified in the config, and none is provided on the command line, centurion could continue to use 'latest' as before.

Is there any way to roll back the changes?

Is there any way to do the zero downtime deployment with or without rolling releases?

Great work with the release of 1.8

With the new feature of 1.8
https://github.com/newrelic/centurion#exporting-environment-variables-locally

I can see this being a high risk. For instance, exporting production, forgetting they exported, and run rake db:drop or something else quite damaging. People dealing with multiple windows - panes - ...
This seems to be a hazard as there is nothing to tell me that I have certain environment variables set.

Alternatives I can think of to avoid this case.

1. Put a big red banner at the top of the terminal window to mention that you are affecting a certain environment on every command you run
2. Invoke the export for the scope of a single command. Allow a command to be passed when invoking the export and only do it for the one command. This limits the scope of the export and makes it less prone to human error (this does not prevent a person from invoking a terminal, and the environment will be set for as long as the session exists should they need that).

CC: @benders @relistan

These are merely thoughts, but may need some consideration.

Containers doing work that takes longer than 30 seconds to complete will get killed. Allow for a configurable stop timeout so containers that run longer jobs can exit cleanly.

We need to override the deployment stages to allow centurion to enable controlled container shutdown during a deploy.

For example, we have a service container that is running a set of background tasks. On a deploy, we need centurion to notify the service that a shutdown is requested. Then, centurion should monitor a status endpoint until the service indicates that all background processes have terminated. At that time, centurion will be clear to bring down the container and deploy the new image.

In order to increase the security of the deployment process
As a centurion instance
I want to use the --tlsverify flag
So that centurion is using my TLS certificates in order to establish the connections to the servers

All the procedure to make it work with the TLS certificates is well described here: http://sheerun.net/2014/05/17/remote-access-to-docker-with-tls/.

I am having an issue that looks similar to #60. But I'm not sure if it is the same. I have a fairly simple Centurion configuration:

namespace :environment do
  task :common do
    set :image, 'registry.hub.docker.com/polleverywhere/private_app'
    set :registry_user, ENV['CENTURION_REGISTRY_USER'] || 'robot'
    set :registry_password, ENV['CENTURION_REGISTRY_PASSWORD'] || 'p4ssw0rd'
  end

  desc 'Vagrant environment'
  task :vagrant => :common do
    host '172.17.9.100.xip.io'
  end
end

The tags listing works fine:

$ bundle exec centurion -p rails -e vagrant -a list
** Invoke environment:vagrant (first_time)
** Invoke environment:common (first_time)
** Execute environment:common
** Execute environment:vagrant
** Invoke list (first_time)
** Execute list
** Invoke list:tags (first_time)
** Execute list:tags
GET: "https://registry.hub.docker.com/v1/repositories/polleverywhere/private_app/tags"
    latest  -> 2801c5ed

** Invoke list:running_containers (first_time)
** Execute list:running_containers
I, [2014-12-14T20:22:19.332264 #15985]  INFO -- : ----- Connecting to Docker on 172.17.9.100.xip.io -----

But when I try to deploy, it fails to find the image:

$ bundle exec centurion -p rails -e vagrant -a deploy
** Invoke environment:vagrant (first_time)
** Invoke environment:common (first_time)
** Execute environment:common
** Execute environment:vagrant
** Invoke deploy (first_time)
** Execute deploy
** Invoke deploy:get_image (first_time)
** Execute deploy:get_image
** Invoke deploy:pull_image (first_time)
** Execute deploy:pull_image
I, [2014-12-14T20:24:57.435864 #16108]  INFO -- : Fetching image registry.hub.docker.com/polleverywhere/private_app:latest IN PARALLEL

I, [2014-12-14T20:24:57.436079 #16108]  INFO -- : Using CLI to pull
Pulling repository registry.hub.docker.com/polleverywhere/private_app
2014/12/14 20:24:59 Error: image polleverywhere/private_app not found
/Users/andy/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.3.1/lib/centurion/logging.rb:62:in `validate_status': The command failed with a non-zero exit status: 1. Command: 'docker -H=tcp://172.17.9.100.xip.io:2375 pull registry.hub.docker.com/polleverywhere/private_app:latest' (RuntimeError)
    from /Users/andy/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.3.1/lib/centurion/logging.rb:57:in `run_with_echo'
    from /Users/andy/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.3.1/lib/centurion/logging.rb:27:in `echo'
    from /Users/andy/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.3.1/lib/centurion/docker_via_cli.rb:16:in `pull'
    from /Users/andy/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.3.1/lib/tasks/deploy.rake:192:in `block (3 levels) in <top (required)>'
    from /Users/andy/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.3.1/lib/centurion/docker_server_group.rb:26:in `call'
    from /Users/andy/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.3.1/lib/centurion/docker_server_group.rb:26:in `block (2 levels) in each_in_parallel'

The image name on the CLI should be polleverywhere/private_app:latest instead of registry.hub.docker.com/polleverywhere/private_app:latest.

Bug or user error?

Given I'm execution the following command:

$ centurion -p myproject -e staging -a rolling_deploy

I'm facing the following error:

lib/centurion/deploy_dsl.rb:49:in `public_port_for': undefined method `values' for nil:NilClass (NoMethodError)

The first step called by the deploy tasks is the stop_containers which pass a second parameter with the output of fetch(:port_bindings).
This :port_bindings is not defined in my configuration but seems required (my config is the default one.

Can you please confirm there's something wrong or explain me what I'm missing ?

I've added a run_command action to some of our internal configurations; to keep from duplicating the code further (and give us a central place to make it work better), it oughta get moved into Centurion itself.

I'm having some problems getting an image from the public registry deployed with Centurion. I'm running rev aaaf36f and getting 503 errors from the registry when Centurion tries to list the available tags, but when I curl the same URI I get a successful response.

Command: bundle exec centurion -p rabbitmq -e local -a list

Config:

namespace :environment do
  task :common do
    set :image, 'registry.hub.docker.com/dockerfile/rabbitmq'

    host_port 5672, container_port: 5672
    host_port 15672, container_port: 15672
  end

  task :local => :common do
    set_current_environment(:local)

    host 'localhost'
  end
end

Output:

** Invoke environment:local (first_time)
** Invoke environment:common (first_time)
** Execute environment:common
** Execute environment:local
** Invoke list (first_time)
** Execute list
** Invoke list:tags (first_time)
** Execute list:tags
GET: "https://registry.hub.docker.com/v1/repositories/dockerfile/rabbitmq/tags"
E, [2014-11-15T10:46:23.198254 #32543] ERROR -- : Couldn't communicate with Registry: #<Excon::Response:0x007f1d46cfdf68 @data={:body=>"<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n\n", :headers=>{"Cache-Control"=>"no-cache", "Connection"=>"close", "Content-Type"=>"text/html"}, :status=>503, :reason_phrase=>"Service Unavailable", :remote_ip=>"162.242.195.83", :local_port=>52296, :local_address=>"10.20.0.11"}, @body="<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n\n", @headers={"Cache-Control"=>"no-cache", "Connection"=>"close", "Content-Type"=>"text/html"}, @status=503, @remote_ip="162.242.195.83", @local_port=52296, @local_address="10.20.0.11">

** Invoke list:running_containers (first_time)
** Execute list:running_containers
I, [2014-11-15T10:46:23.198515 #32543]  INFO -- : ----- Connecting to Docker on localhost -----

Curl:

$ curl -vvv https://registry.hub.docker.com/v1/repositories/dockerfile/rabbitmq/tags

* Hostname was NOT found in DNS cache
*   Trying 162.242.195.83...
* Connected to registry.hub.docker.com (162.242.195.83) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*    subject: OU=GT51013891; OU=See www.rapidssl.com/resources/cps (c)14; OU=Domain Control Validated - RapidSSL(R); CN=*.hub.docker.com
*    start date: 2014-10-25 15:58:21 GMT
*    expire date: 2015-05-23 20:41:27 GMT
*    subjectAltName: registry.hub.docker.com matched
*    issuer: C=US; O=GeoTrust Inc.; CN=RapidSSL SHA256 CA - G3
*    SSL certificate verify ok.
> GET /v1/repositories/dockerfile/rabbitmq/tags HTTP/1.1
> User-Agent: curl/7.35.0
> Host: registry.hub.docker.com
> Accept: */*
> 
< HTTP/1.1 200 OK
* Server nginx is not blacklisted
< Server: nginx
< Date: Sat, 15 Nov 2014 10:41:21 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: close
< Vary: Cookie
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=31536000
< 
* Closing connection 0
* SSLv3, TLS alert, Client hello (1):

We are starting to see more and more daemons that are not HTTP. I think we should allow optional port.

This may require adding a new feature: Allow custom check that is not HTTP based.

I don't see where the tag is being passed down to deploy. I've started to fix it here, but wanted to make sure it is really broken first.

I have the following project

namespace :environment do
  task :common do
    set :image, 'redis'
  end

  desc 'Staging environment'
  task staging: :common do
    set_current_environment(:staging)
    host_port 10_234, container_port: 9292
    host '127.0.0.1:4243'
    host '127.0.0.1:4244'
    set :container_hostname, ->(s) { s }
  end
end

I run the following command to deploy a specific tag.

$  centurion -e staging -a deploy -p example-app -t 2.6.17

I get the following output.

** Invoke deploy:determine_image_id_from_first_server (first_time)                                                                                                      [645/1859]
** Execute deploy:determine_image_id_from_first_server
** Invoke deploy:verify_image (first_time)
** Execute deploy:verify_image
** Invoke deploy:stop (first_time)
** Execute deploy:stop
** Invoke deploy:start_new (first_time)
** Execute deploy:start_new
/usr/local/Cellar/rbenv/0.4.0/versions/2.2.2/lib/ruby/gems/2.2.0/gems/centurion-1.7.4/lib/centurion/docker_via_api.rb:77:in `create_container': #<Excon::Response:0x007fed048351f0
 @data={:body=>"No such image: redis (tag: latest)\n", :headers=>{"Content-Type"=>"text/plain; charset=utf-8", "Date"=>"Tue, 09 Jun 2015 00:57:06 GMT", "Content-Length"=>"35"}, :
status=>404, :status_line=>"HTTP/1.1 404 Not Found\r\n", :reason_phrase=>"Not Found", :remote_ip=>"127.0.0.1", :local_port=>60632, :local_address=>"127.0.0.1"}, @body="No such im
age: redis (tag: latest)\n", @headers={"Content-Type"=>"text/plain; charset=utf-8", "Date"=>"Tue, 09 Jun 2015 00:57:06 GMT", "Content-Length"=>"35"}, @status=404, @remote_ip="127
.0.0.1", @local_port=60632, @local_address="127.0.0.1"> (RuntimeError)
        from /usr/local/opt/rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/centurion-1.7.4/lib/centurion/deploy.rb:104:in `start_new_container'
        from /usr/local/opt/rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/centurion-1.7.4/lib/tasks/deploy.rake:100:in `block (3 levels) in <top (required)>'
        from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.2/lib/ruby/gems/2.2.0/gems/centurion-1.7.4/lib/centurion/deploy_dsl.rb:8:in `call'
        from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.2/lib/ruby/gems/2.2.0/gems/centurion-1.7.4/lib/centurion/deploy_dsl.rb:8:in `block (2 levels) in on_each_docker_host'
        from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.2/lib/ruby/gems/2.2.0/gems/centurion-1.7.4/lib/centurion/docker_server_group.rb:22:in `call'
        from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.2/lib/ruby/gems/2.2.0/gems/centurion-1.7.4/lib/centurion/docker_server_group.rb:22:in `block in each'
        from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.2/lib/ruby/gems/2.2.0/gems/centurion-1.7.4/lib/centurion/docker_server_group.rb:20:in `each'
        from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.2/lib/ruby/gems/2.2.0/gems/centurion-1.7.4/lib/centurion/docker_server_group.rb:20:in `each'
        from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.2/lib/ruby/gems/2.2.0/gems/centurion-1.7.4/lib/centurion/deploy_dsl.rb:8:in `block in on_each_docker_host'
        from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.2/lib/ruby/gems/2.2.0/gems/centurion-1.7.4/lib/centurion/deploy_dsl.rb:8:in `tap'
        from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.2/lib/ruby/gems/2.2.0/gems/centurion-
1.7.4/lib/centurion/deploy_dsl.rb:8:in `on_each_docker_host'

Centurion should be able to manage which servers are selected to run a project at each deployment using at least a basic scheduling mechanism.

Write now, health checks are all using HTTP:
https://github.com/newrelic/centurion/blob/master/lib/centurion/deploy.rb#L61

It should allow for different schemas like https, or maybe event ftp or something else.

Sorry for I'm not familiar with "DSL items in Rake task". I'm not sure what's the reserved key there.
I wish to specify container's hostname, like docker do : -h hostname.
How could I let it happen? Thx in adv~

Can you add a configuration option to support url's for private registries?

All of the configuration for each project should be able to be fetched from etcd as well as from config files. We're working on implementing this.

Would someone could give me a hand?
I have a project named:poseidon, it can list private docker registry images successfully but I can't deploy it w/ messages as below:

$ bundle exec centurion -p poseidon -e staging -a deploy
logging.rb:54:in `popen': No such file ... (Errno::ENOENT)

I have no idea. Please give me a hint, thanks in advance~

Here's my project file:
namespace :environment do
task :common do
set :image, 'www.docker9.com.tw:8080/poseidon/all:1.0.175'
host '192.168.22.4'
end

desc 'Staging environment'
task :staging => :common do
set_current_environment(:staging)
env_vars YOUR_ENV: 'staging'
env_vars DOCKER_HOSTNAME: 'radosgw'
host_port 20996, container_port: 20996
# host '192.168.23.128'

command ['/poseidon/launcher']
# You can assign different docker daemon port. Example:
# host 'docker-server-staging-3.example.com:4243'

end

desc 'Production environment'
task :production => :common do
set_current_environment(:production)
env_vars YOUR_ENV: 'production'
# host_port 23235, container_port: 9293
# host 'docker-server-prod-1.example.com'
# host 'docker-server-prod-2.example.com'
# host_volume '/mnt/volume1', container_volume: '/mnt/volume1'
# command ['/bin/bash', '-c', '/path/to/server -e production']
end
end

Hi!
I'm trying to deploy using an private docker/registry-2.0 (docker-distribution)

Have I misconfigured my registry or is there simply no support for /v2/ api yet?

$ centurion -p myApp -e production 
** Invoke environment:production (first_time)
** Invoke environment:common (first_time)
** Execute environment:common
** Execute environment:production
** Invoke centurion:setup (first_time)
** Invoke centurion:clean_environment (first_time)
** Execute centurion:clean_environment
** Execute centurion:setup
** Invoke list (first_time)
** Execute list
** Invoke list:tags (first_time)
** Execute list:tags
GET: "https://registry.mydomain.com/v1/repositories/myapp/tags"
E, [2015-09-21T11:42:34.014356 #4959] ERROR -- : Couldn't communicate with Registry: #<Excon::Response:0x0000000195d188 @data={:body=>"<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx/1.9.4</center>\r\n</body>\r\n</html>\r\n", :headers=>{"Server"=>"nginx/1.9.4", "Date"=>"Mon, 21 Sep 2015 09:51:43 GMT", "Content-Type"=>"text/html", "Content-Length"=>"168", "Connection"=>"keep-alive"}, :status=>404, :status_line=>"HTTP/1.1 404 Not Found\r\n", :reason_phrase=>"Not Found"}, @body="<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx/1.9.4</center>\r\n</body>\r\n</html>\r\n", @headers={"Server"=>"nginx/1.9.4", "Date"=>"Mon, 21 Sep 2015 09:51:43 GMT", "Content-Type"=>"text/html", "Content-Length"=>"168", "Connection"=>"keep-alive"}, @status=404, @remote_ip="x.x.x.x", @local_port=39801, @local_address="x.x.x.x">

Note:
If i do try to replace the v1 to a v2 and visit the link manually:
https://registry.mydomain.com/v2/repositories/myapp/tags
I do indeed get to the place i should be, instead of a 404.

Now that Docker support docker exec functionality, deploy_console should use that rather than stealing a live node from the pool to replace with a console.

There's a weird interaction with the trollop gem's argument parsing; it tries to handle --no-xxxx arguments and gets it wrong when given a flag argument with a false default; --no-pull ends up setting opts[:nopull] to false.

Using the short form of this argument, -n, doesn't have this problem (but I still spent more than an hour on this because I lean toward always using long-form arguments when I embed command lines in scripts, to help readability).

In my config file, I have command set to be:

command ['/bin/bash', '-c', './docker_sourcing_run.sh']

but this doesn't seem to have any effects when I deploy. The container dies immediately because the "Cmd" seems to be "/bin/bash" in the log:

I, [2014-11-06T22:16:24.147296 #30221]  INFO -- : {"State"=>{"Running"=>true, "Paused"=>false, "Restarting"=>false, "Pid"=>30282, "ExitCode"=>0, "StartedAt"=>"2014-11-06T22:16:24.134526188Z", "FinishedAt"=>"0001-01-01T00:00:00Z"}, "ID"=>"b570360ab7b1097317e771c071a88ed02d860e4e793496bbfcbcfad9fd0dbf49", "Created"=>"2014-11-06T22:16:23.956575973Z", "Path"=>"/bin/bash", "Args"=>[], "Config"=>{"Hostname"=>"localhost", "Domainname"=>"", "User"=>"", "Memory"=>0, "MemorySwap"=>0, "CpuShares"=>0, "Cpuset"=>"", "AttachStdin"=>false, "AttachStdout"=>false, "AttachStderr"=>false, "PortSpecs"=>nil, "ExposedPorts"=>{"3306/tcp"=>{}, "8888/tcp"=>{}}, "Tty"=>false, "OpenStdin"=>false, "StdinOnce"=>false, "Env"=>["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"], "Cmd"=>["/bin/bash"], "Image"=>"43b4f9aba9cfcbcbf3d19246ae375d8e9747383d6ba3b0a9b19e16a1e726384a", "Volumes"=>{"/circleup/logs"=>{}, "/circleup/model/company-suitability"=>{}, "/circleup/sourcing"=>{}}, "WorkingDir"=>"/circleup/sourcing", "Entrypoint"=>nil, "NetworkDisabled"=>false, "OnBuild"=>nil, "SecurityOpt"=>nil}, "Image"=>"43b4f9aba9cfcbcbf3d19246ae375d8e9747383d6ba3b0a9b19e16a1e726384a", "NetworkSettings"=>{"IPAddress"=>"172.17.0.13", "IPPrefixLen"=>16, "MacAddress"=>"02:42:ac:11:00:0d", "Gateway"=>"172.17.42.1", "Bridge"=>"docker0", "PortMapping"=>nil, "Ports"=>{"3306/tcp"=>nil, "8888/tcp"=>[{"HostIp"=>"0.0.0.0", "HostPort"=>"8888"}]}}, "ResolvConfPath"=>"/var/lib/docker/containers/b570360ab7b1097317e771c071a88ed02d860e4e793496bbfcbcfad9fd0dbf49/resolv.conf", "HostnamePath"=>"/var/lib/docker/containers/b570360ab7b1097317e771c071a88ed02d860e4e793496bbfcbcfad9fd0dbf49/hostname", "HostsPath"=>"/var/lib/docker/containers/b570360ab7b1097317e771c071a88ed02d860e4e793496bbfcbcfad9fd0dbf49/hosts", "Name"=>"/stupefied_brown", "Driver"=>"aufs", "ExecDriver"=>"native-0.2", "MountLabel"=>"", "ProcessLabel"=>"", "AppArmorProfile"=>"", "RestartCount"=>0, "Volumes"=>{"/circleup/logs"=>"/circleup/logs", "/circleup/model/company-suitability"=>"/circleup/model/company-suitability", "/circleup/sourcing"=>"/circleup/sourcing"}, "VolumesRW"=>{"/circleup/logs"=>true, "/circleup/model/company-suitability"=>true, "/circleup/sourcing"=>true}, "HostConfig"=>{"Binds"=>["/circleup/model/company-suitability:/circleup/model/company-suitability", "/circleup/logs:/circleup/logs", "/circleup/sourcing:/circleup/sourcing"], "ContainerIDFile"=>"", "LxcConf"=>nil, "Privileged"=>false, "PortBindings"=>{"8888/tcp"=>[{"HostIp"=>"0.0.0.0", "HostPort"=>"8888"}]}, "Links"=>nil, "PublishAllPorts"=>false, "Dns"=>nil, "DnsSearch"=>nil, "ExtraHosts"=>nil, "VolumesFrom"=>nil, "Devices"=>nil, "NetworkMode"=>"", "CapAdd"=>nil, "CapDrop"=>nil, "RestartPolicy"=>{"Name"=>"", "MaximumRetryCount"=>0}}}

Am I doing something wrong here?

Hi,

this is me opening a ticket to question about some improvements before committing any work. If it makes sense I'd love to make them happen just don't want to come and send something mismatching the creator's intentions or philosophy.

I'd love to have a cleaner more summarized output by default that would include turning off some debugging output in rake, filtering some output in the CLI calls and only present them on errors for instance, adding some colors to highlight some things and shortening the log date included in every message.

Most stuff is usually shipped to log archiving or collected though systemd or similar so the date is added. Additionally quite often centurion is executed in a manual way on the developer's workstation so logs are not stored. Because of this I'd like to have a more readable mode by default or at least as an option.

Would this make sense? If yes, let me know and I'll see what I can work on.

Cheers.

Hello!

I have error while deploy with centurion.
Could not apply volumes of non-existent container "parent"
So I comment this line (https://github.com/newrelic/centurion/blob/master/lib/centurion/service.rb#L95) and problem resolved.

I'm wondering why "VolumesFrom" option has default value "parent".
Is it OK? If I remove VolumesFrom default option?

• docker version - 1.6.2

Howdy,

Just started to play with Centurion and am getting an error when trying to use the list action:

bundle exec centurion -p myproject -e staging -a list

Here is the error:

** Invoke environment:staging (first_time)
** Invoke environment:common (first_time)
** Execute environment:common
** Execute environment:staging
** Invoke list (first_time)
** Execute list
** Invoke list:tags (first_time)
** Execute list:tags
GET: "https://ianneub/v1/repositories/myproject/tags"
E, [2014-10-06T10:54:41.500188 #48677] ERROR -- : Couldn't communicate with Registry: getaddrinfo: nodename nor servname provided, or not known (SocketError)

** Invoke list:running_containers (first_time)
** Execute list:running_containers
I, [2014-10-06T10:54:41.500407 #48677]  INFO -- : ----- Connecting to Docker on localhost -----

Can I specify the registry somewhere? Or the credentials for the registry?

Here is my rake file:

namespace :environment do
  task :common do
    set :image, 'ianneub/myproject'

    host 'localhost:2375'
  end
   desc 'Staging environment'
  task :staging => :common do
    set_current_environment(:staging)
  end
   desc 'Production environment'
  task :production => :common do
    set_current_environment(:production)
  end
end

When i trigger the following cmd on my console:
bundle exec centurion -p radio-radio -e production -a deploy

I get the following output which is really senseless to me: (some one an idea?)
** Invoke environment:production (first_time)
** Invoke environment:common (first_time)
** Execute environment:common
** Execute environment:production
** Invoke deploy (first_time)
** Execute deploy
** Invoke deploy:get_image (first_time)
** Execute deploy:get_image
** Invoke deploy:pull_image (first_time)
** Execute deploy:pull_image
I, [2015-03-14T15:15:57.561132 #2996] INFO -- : Fetching image redis:latest IN PARALLEL

I, [2015-03-14T15:15:57.566596 #2996] INFO -- : Using CLI to pull
/Library/Ruby/Gems/2.0.0/gems/centurion-1.5.1/lib/centurion/logging.rb:43:in popen': No such file or directory - docker (Errno::ENOENT) from /Library/Ruby/Gems/2.0.0/gems/centurion-1.5.1/lib/centurion/logging.rb:43:inrun_without_echo'
from /Library/Ruby/Gems/2.0.0/gems/centurion-1.5.1/lib/centurion/logging.rb:25:in echo' from /Library/Ruby/Gems/2.0.0/gems/centurion-1.5.1/lib/centurion/docker_via_cli.rb:17:inpull'
from /Library/Ruby/Gems/2.0.0/gems/centurion-1.5.1/lib/tasks/deploy.rake:195:in block (3 levels) in <top (required)>' from /Library/Ruby/Gems/2.0.0/gems/centurion-1.5.1/lib/centurion/docker_server_group.rb:28:incall'
from /Library/Ruby/Gems/2.0.0/gems/centurion-1.5.1/lib/centurion/docker_server_group.rb:28:in `block (2 levels) in each_in_parallel'

In my rake file I would like to not specify the host_port value even if my container exposes a port. Is this possible? When I leave out that line I get the following error -

I, [2014-07-29T11:57:55.717050 #57151]  INFO -- : ----- Connecting to Docker on bld-docker-01 -----
/Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/centurion/deploy_dsl.rb:46:in `public_port_for': undefined method `values' for nil:NilClass (NoMethodError)
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/centurion/deploy.rb:9:in `stop_containers'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/tasks/deploy.rake:40:in `block (3 levels) in <top (required)>'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/centurion/deploy_dsl.rb:7:in `call'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/centurion/deploy_dsl.rb:7:in `block (2 levels) in on_each_docker_host'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/centurion/docker_server_group.rb:20:in `call'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/centurion/docker_server_group.rb:20:in `block in each'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/centurion/docker_server_group.rb:18:in `each'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/centurion/docker_server_group.rb:18:in `each'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/centurion/deploy_dsl.rb:7:in `block in on_each_docker_host'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/centurion/deploy_dsl.rb:6:in `tap'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/centurion/deploy_dsl.rb:6:in `on_each_docker_host'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/tasks/deploy.rake:40:in `block (2 levels) in <top (required)>'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:228:in `call'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:228:in `block in execute'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:223:in `each'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:223:in `execute'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:166:in `block in invoke_with_call_chain'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:159:in `invoke_with_call_chain'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:152:in `invoke'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/capistrano_dsl.rb:88:in `invoke'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/tasks/deploy.rake:7:in `block in <top (required)>'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:228:in `call'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:228:in `block in execute'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:223:in `each'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:223:in `execute'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:166:in `block in invoke_with_call_chain'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/monitor.rb:211:in `mon_synchronize'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:159:in `invoke_with_call_chain'
    from /Users/pairing/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/rake/task.rb:152:in `invoke'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/lib/capistrano_dsl.rb:88:in `invoke'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/gems/centurion-1.0.10/bin/centurion:74:in `<top (required)>'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/bin/centurion:23:in `load'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/bin/centurion:23:in `<main>'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/bin/ruby_executable_hooks:15:in `eval'
    from /Users/pairing/.rvm/gems/ruby-2.0.0-p353/bin/ruby_executable_hooks:15:in `<main>'

In trying to deploy RabbitMQ I attempted a rolling deploy, which fails with a status endpoint of / because its trying to do the status check on the AMQP port as well as the HTTP management port.

Removing the port mapping for the AMQP port means the status check completes successfully and the deploy can continue, but obviously also leaves the container pretty useless! Is there some way to specify that I only want a status check done on a specific port, ignoring the others?

Hi,

This may be similar to #76. I tried to set up credential on remote server but it didn't work.

Here is my configuration for my public image

namespace :environment do
  task :common do
    set :image, 'roonglit/docker-test'
  end

  desc 'Staging environment'
  task :staging => :common do
    set_current_environment(:staging)
    host 'my-host-here'
  end
end

when I call rolling_deploy action, it's working fine. I want to deploy my private image, so I change the configuration to:

namespace :environment do
  task :common do
    set :image, 'registry.hub.docker.com/hipflat/website'
    set :registry_user, 'my_username_here'
    set :registry_password, 'my_password_here'
  end

  desc 'Staging environment'
  task :staging => :common do
    set_current_environment(:staging)
    host 'my-host-here'
  end
end

when I call rolling_deploy action, I got an error.

* Invoke environment:staging (first_time)
** Invoke environment:common (first_time)
** Execute environment:common
** Execute environment:staging
** Invoke rolling_deploy (first_time)
** Execute rolling_deploy
** Invoke deploy:get_image (first_time)
** Execute deploy:get_image
** Invoke deploy:pull_image (first_time)
** Execute deploy:pull_image
I, [2015-01-23T10:49:05.880898 #955]  INFO -- : Fetching image registry.hub.docker.com/hipflat/website:latest IN PARALLEL

I, [2015-01-23T10:49:05.881224 #955]  INFO -- : Using CLI to pull
I, [2015-01-23T10:49:05.883343 #955]  INFO -- : Using CLI to pull
Pulling repository registry.hub.docker.com/hipflat/website
Pulling repository registry.hub.docker.com/hipflat/website
2015/01/23 10:49:09 Error: image hipflat/website:latest not found
2015/01/23 10:49:09 Error: image hipflat/website:latest not found
/Users/mac/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.4.1/lib/centurion/logging.rb:62:in `validate_status': The command failed with a non-zero exit status: 1. Command: 'docker -H=tcp://hipflat-website.cloudapp.net:2375 pull registry.hub.docker.com/hipflat/website:latest' (RuntimeError)
    from /Users/mac/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.4.1/lib/centurion/logging.rb:48:in `run_without_echo'
    from /Users/mac/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.4.1/lib/centurion/logging.rb:25:in `echo'
    from /Users/mac/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.4.1/lib/centurion/docker_via_cli.rb:17:in `pull'
    from /Users/mac/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.4.1/lib/tasks/deploy.rake:187:in `block (3 levels) in <top (required)>'
    from /Users/mac/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.4.1/lib/centurion/docker_server_group.rb:28:in `call'
    from /Users/mac/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.4.1/lib/centurion/docker_server_group.rb:28:in `block (2 levels) in each_in_parallel'

I tried using docker -H my-host-server:2375 pull hipflat/website, and it's working fine both my local pc, and my remote server (which I'm trying to deploy image with Centurion).

Could you give me some suggestions on what I need to do next?

For implement a zero downtime you already think use/integrate with something like this https://github.com/DevTable/gantryd ?

Its look to be a very good solution. You can just run a new container on each machine in parallel and let gantryd kill the old one.

There is a small issue when redeploying or stoping a container.

in deploy.rb

  def stop_containers(target_server, port_bindings, timeout = 30)
    public_port    = public_port_for(port_bindings)
    old_containers = target_server.find_containers_by_public_port(public_port)
    info "Stopping container(s): #{old_containers.inspect}"

    old_containers.each do |old_container|
      info "Stopping old container #{old_container['Id'][0..7]} (#{old_container['Names'].join(',')})"
      target_server.stop_container(old_container['Id'], timeout)
    end
  end

passes the 2 arguments old_container['Id'] and timeout to stop_container which only accepts one.

  def stop_container(container_id)
    path = "/v1.7/containers/#{container_id}/stop?t=30"
    response = Excon.post(
      @base_uri + path,
    )
    raise response.inspect unless response.status == 204
    true
  end

I modified it locally but just thought I would let you guys know just in case no one else has reported.

Thanks, keep up the awesome work!

The deployment is now working fine until it tries to start the image as it can't find the database. As I'm using Docker, the database is also a Docker image so I have to use the --links docker flag.

As of now I don't see anything in centurion to deploy the images to be linked. Is it something missing or did I missed something ?

Hi

Centurion looks really well designed, but am I correct in assuming that it doesn't allow one to deploy to machines behind firewalls?

I was wondering why do I have to have the gem on my production machines while I need it only for the deployment ?

Hello,

I found an error running 1.1.0 that I don't know how to solve, this is the last part of the stack trace:

I, [2014-09-13T15:25:54.391797 #61903]  INFO -- : ----- Connecting to Docker on public-1.node.staging -----
I, [2014-09-13T15:25:54.391962 #61903]  INFO -- : Creating new container for d556105c
I, [2014-09-13T15:25:54.983926 #61903]  INFO -- : Starting new container e0b2029c
/Volumes/Low/David/.dotfiles/tag-notrack/rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/centurion/docker_via_api.rb:85:in `start_container': Failed to start container! "Cannot start container e0b2029cd26e24e67a55235cd0a840cb6c074de78146dae5fbd18bd3cd25c676: exec: "/etc/cidfile": stat /etc/cidfile: no such file or directory (RuntimeError)
"
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/centurion/deploy.rb:157:in `start_container_with_config'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/centurion/deploy.rb:124:in `start_new_container'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/tasks/deploy.rake:49:in `block (3 levels) in <top (required)>'
        from /Volumes/Low/David/.dotfiles/tag-notrack/rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/centurion/deploy_dsl.rb:7:in `call'
        from /Volumes/Low/David/.dotfiles/tag-notrack/rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/centurion/deploy_dsl.rb:7:in `block (2 levels) in on_each_docker_host'
        from /Volumes/Low/David/.dotfiles/tag-notrack/rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/centurion/docker_server_group.rb:20:in `call'
        from /Volumes/Low/David/.dotfiles/tag-notrack/rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/centurion/docker_server_group.rb:20:in `block in each'
        from /Volumes/Low/David/.dotfiles/tag-notrack/rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/centurion/docker_server_group.rb:18:in `each'
        from /Volumes/Low/David/.dotfiles/tag-notrack/rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/centurion/docker_server_group.rb:18:in `each'
        from /Volumes/Low/David/.dotfiles/tag-notrack/rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/centurion/deploy_dsl.rb:7:in `block in on_each_docker_host'
        from /Volumes/Low/David/.dotfiles/tag-notrack/rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/centurion/deploy_dsl.rb:6:in `tap'
        from /Volumes/Low/David/.dotfiles/tag-notrack/rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/centurion/deploy_dsl.rb:6:in `on_each_docker_host'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/tasks/deploy.rake:48:in `block (2 levels) in <top (required)>'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:236:in `call'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:236:in `block in execute'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:231:in `each'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:231:in `execute'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:175:in `block in invoke_with_call_chain'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:168:in `invoke_with_call_chain'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:161:in `invoke'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/capistrano_dsl.rb:88:in `invoke'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/tasks/deploy.rake:8:in `block in <top (required)>'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:236:in `call'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:236:in `block in execute'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:231:in `each'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:231:in `execute'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:175:in `block in invoke_with_call_chain'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:168:in `invoke_with_call_chain'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/2.1.0/rake/task.rb:161:in `invoke'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/lib/capistrano_dsl.rb:88:in `invoke'
        from /Volumes/Low/David/.rbenv/versions/2.1.2/lib/ruby/gems/2.1.0/gems/centurion-1.1.0/bin/centurion:65:in `<top (required)>'
        from ./bin/centurion:16:in `load'
        from ./bin/centurion:16:in `<main>'

It seems related to not being capable of writing the cidfile, I'm getting through the code to find a solution an understand centurion's internals but I don't even know if this is an issue writing a file in localhost or the remote machine where the daemon's running. I'm glad to help solve the problem, but if there's anyone who can point me into the right direction, it's much appreciated.

Thanks!

Seems like that would be a more robust way of contacting docker hosts instead of relying on the docker cli to be installed. It also supports TLS out of the box which means we could add something to the DSL that said to use a cert when talking to the docker host (but not others).

Some saas analytics platforms, like New Relic, have an API that can be called during deployment of new versions. I think it would be cool if centurion had support for something like this. Maybe just a method that could be overridden to make the API call at start or end of deployment?

We use something like Phusion's Docker baseimage, which has an unconventional invocation:

docker run -ti phusion/baseimage /sbin/my_init -- bash -l

(the /sbin/my_init is necessary to start other services in the same container)

This isn't the only use-case for custom commands -- they could be useful for when running Docker images built with non-Dockerfile tools like Packer, where the metadata of CMD can't be stored.

Does this make sense? I'd be happy to try a PR if so. I image it could be a variable set just as the image name is currently configured.

Hello,

I'm having the following issue when I deploy with centurion 1.7.2, to an Ubuntu 14 Docker host, running Docker 1.7.1, and created using docker-machine).

I'm on MacOSX 10.10.4, and have ruby version 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]

/Users/{username}/bin/centurion/lib/centurion/docker_via_api.rb:33:in `inspect_image': #<Excon::Response:0x007fe7d20986b8 @data={:body=>"client is too old, minimum supported API version is 1.12, please upgrade your client to a newer version\n", :headers=>{"Content-Type"=>"text/plain; charset=utf-8", "Date"=>"Thu, 30 Jul 2015 21:21:39 GMT", "Content-Length"=>"104"}, :status=>400, :status_line=>"HTTP/1.1 400 Bad Request\r\n", :reason_phrase=>"Bad Request", :remote_ip=>"{ip address}", :local_port=>61525, :local_address=>"{ip address}"}, @body="client is too old, minimum supported API version is 1.12, please upgrade your client to a newer version\n", @headers={"Content-Type"=>"text/plain; charset=utf-8", "Date"=>"Thu, 30 Jul 2015 21:21:39 GMT", "Content-Length"=>"104"}, @status=400, @remote_ip="10.132.8.78", @local_port=61525, @local_address="10.130.200.109"> (RuntimeError)

Looking at the relevant bits of code in docker_via_api.rb on line 33, I see:

def inspect_image(image, tag = "latest")
    repository = "#{image}:#{tag}"
    path       = "/v1.7/images/#{repository}/json"

    response = Excon.get(
      @base_uri + path,
      tls_excon_arguments.merge(:headers => {'Accept' => 'application/json'})
    )
    #NOTE: It fails here -
    raise response.inspect unless response.status == 200
    JSON.load(response.body)
  end

It seems like the Excon library is receiving an exception from the docker server, because the docker server is looking for a user agent. Has anyone else seen this issue, and if so, how can it be fixed?

Thank you.

Centurion's rolling_deploy action iterates over the hosts: for each, it kills the old container, starts a new one, then waits for the new one to successfully health-check before moving on. With this paradigm, you clearly need to have at least two hosts running something to maintain uptime.... but if the second host is already down for some reason, having only two instances means deploying (or running a migration command) will blindly take your only other instance down.

Rolling deploy (and also anywhere we pick a host to do something on) ought to look for any "down" hosts and deploy to them first.

In docker_via_api.rb, Centurion explicitly calls v1.7 of the Docker API, which is supposed to return the JSON keys as snake_case. Using a specific version like this should protect us from any changes to the Docker API, like the one starting in v1.12, when the JSON keys began to be returned as CamelCase. (See docker docs)

However, when using a combination of Dogestry and Centurion, it's possible for Centurion to call v1.7 docker API endpoints and get back CamelCase, which results in Centurion crashing.

How?

When a docker image is exported from a docker host, the resulting tarball contains multiple directories, one for each layer. Each layer has a corresponding JSON file (See [docker docs for details](http://docs.docker.com/reference/api/docker_remote_api_v1.15/#load-a-tarball-with-a-set-of-images-and-tags-into-docker for details).) When the docker image is exported from a newer docker host, the keys in the JSON files are CamelCase.

Next, you import that tarball into a second docker host (using the /images/load API).

(Under the hood, dogestry uses these two docker API calls to export an image from one host and import it into another.)

Finally, you query the API in the second docker host for information about that image. Unfortunately, whether you use the v1.7 API, or the v1.15 API, the JSON response contains keys in CamelCase.

Here's an example of the wrong behavior:

An image (image_1) was exported (using /images/(name)/get) from a docker host with a server API of v1.12, which means that the JSON files have keys in CamelCase.

Then, we load the exported tarball (/images/load) into a docker host with a server API of v1.15

Now, let's hit two different versions of the API on docker host 'B'

# [WRONG] Calling /v1.7/images/ endpoint returns CamelCase.
# (This should return snake_case, which is all lowercase.)

$ curl http://DOCKER_HOST_B:2375/v1.7/images/<image_1)>/json
{"Architecture":"amd64","Author":"","Comment":"","Config":{"AttachStderr":false,...

# [CORRECT] Calling /v1.15/images endpoint returns CamelCase

$ curl http://DOCKER_HOST_B:2375/v1.15/images/<image_1)>/json
{"Architecture":"amd64","Author":"","Comment":"","Config":{"AttachStderr":false,...

Interestingly enough, importing a tarball that has JSON files with keys in snake_case results in correct responses from both versions of the API. Here's an example:

An image (image_2) was exported (/images/(name)/get) from docker host 'A' with a server API of v1.10, which means that the JSON files have keys in snake_case.

Then, we load the exported tarball (/images/load) into docker host 'B' with a server API of v1.15

Again, let's hit two different versions of the API on docker host 'B'

# [CORRECT] Calling /v1.7/images/ endpoint returns snake_case (begins with lowercase)

$ curl http://DOCKER_HOST_B:2375/v1.7/images/<image_2>/json
{"id":"<sha>","parent":"<sha>","created":"2015-01-08T23:15:07.130204289Z",

# [CORRECT] Calling /v1.15/images/ endpoint returns CamelCase (begins with uppercase)

$ curl http://DOCKER_HOST_B:2375/v1.15/images/<image_2>/json
{"Architecture":"amd64","Author":"","Comment":"","Config":{"AttachStderr":false,...

cc: @didip @relistan @amjith

I can get deploys to work:

https://gist.github.com/darron/e23184254b30af85d821

With a simple patch to centurionize here:

darron@ec3963d

But that causes image pulls to fail:

https://gist.github.com/darron/08f3957e78d8dd121ac2

If I rename to :image - then there's a different issue:

https://gist.github.com/darron/3af69d6f71533a2c6ecb

I've got a workaround for now:

darron@92635e5

This is what happens when I disable pulling, but use :image:

https://gist.github.com/darron/682d69d2a2f337d4087f

My config is here:

https://gist.github.com/darron/77e806a674ac623f3656

So it looks like theres something at least with :image and :image_id that needs to be cleared up - the authentication issues are mine I think.

Does that make sense?

This is all on current Docker:

[master] darron@~/Desktop/centurion_test: docker -v
Docker version 1.0.1, build 990021a
[master] darron@~/Desktop/centurion_test: docker -H=tcp://192.168.0.16:4243 info
Containers: 12
Images: 28
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 52
Execution Driver: native-0.2
Kernel Version: 3.13.0-29-generic
Username: octohost
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support
[master] darron@~/Desktop/centurion_test: docker -H=tcp://192.168.0.16:4243 -v
Docker version 1.0.1, build 990021a

As of right now we have to list out the hosts that we want to deploy to. In a lot of scenarios people that are using the cloud, might not be able to guarantee that the ip addresses of the hosts that they want to deploy to are going to be the same every time (think in the case of an autoscaling group).

I was thinking it would be a good idea to include hostgroup or host_collection option that is passed a lambda, and would return a collection of hosts. We could then propagate :hosts in the deploy_dsl.rb after calling the lambda. By doing it this way you wouldn't have to include specific support for any one cloud or service discover method. But it would allow for custom tooling out of the box to support any cloud.

I am willing to work on it, I might need some help writing appropriate rspec tests. I just wanted to get feedback first.

Hi Guys,

I'm interested if its possible to ssh tunnel through a jumphost and deploy to the fleet of machines.
Something like its in capistrano 2.15.xx.

Thanks