mesosphere-backup / deimos Goto Github PK

If task_info is used instead of executor_info (see here), then executor() return value is None, which makes the return value of needs_observer() True. This causes Mesos environment variables to not be set.

The code for that can be found here.

Is this the expected behaviour? It does not seem to be, because for me specifying task_info implies I want to create a new executor to work with it, rather than attaching to an existing one.

Also, the documentation makes reference to TaskInfo rather than ExecutorInfo here, which seems to suggest that this is the preferred way.

When a URI can't be fetch the error message incorrectly
added spaces, so the error ended up like this:

deimos.containerizer.place_uris() Failed while processing URI: h t t p : / / e x a m p l e . c o m

This is because the escape function takes an array, not
a string

I've seen this a few times when trying to launch a container from Marathon using deimos.

Jun 30 21:11:44 myserver deimos[8798]: deimos.cli() Unhandled failure in launch#012Traceback (most recent call last):#012  File "/usr/local/lib/python2.7/dist-packages/deimos/__init__.py", line 72, in cli#012    result = deimos.containerizer.stdio(containerizer, *argv[1:])#012  File "/usr/local/lib/python2.7/dist-packages/deimos/containerizer/__init__.py", line 97, in stdio#012    method(recordio.read(proto), *args[1:])#012  File "/usr/local/lib/python2.7/dist-packages/deimos/containerizer/docker.py", line 72, in launch#012    pre, image = re.split(r"^docker:///?", url)#012  File "/usr/lib/python2.7/re.py", line 167, in split#012    return _compile(pattern, flags).split(string, maxsplit)#012TypeError: expected string or buffer

When trying to use a request payload such as:

{
  "container": {
    "image": "docker:///jplock/breakerbox"
  },
  "id": "breakbox",
  "instances": "1",
  "cpus": ".5",
  "mem": "512",
  "uris": [ ],
  "cmd": ""
}

Can I use any docker image or do I have to use libmesos\ubuntu? That wasn't clear from the docs.

I'm having trouble using a container with an entrypoint defined because instead of passing the arguments as is, /bin/sh -c is being added to the cmd.

In my particular case, I am trying to run an image based off (flynn/slugrunner)[https://github.com/flynn/slugrunner/blob/master/Dockerfile]:

FROM progrium/cedarish
MAINTAINER Jonathan Rudenberg <[email protected]>

ADD ./runner/ /runner
ADD ./build/sdutil /bin/sdutil
ENTRYPOINT ["/runner/init"]

So if I wanted to run this container, I could specify something like the following:

docker run -e SLUG_URL=http://example.com/slug.tgz flynn/slugrunner start web

... which would end up calling something like this inside the container:

/runner/init start web

When /bin/sh -c is added to the cmd parameters passed to marathon, the container ends up trying to execute:

/runner/init /bin/sh -c 'start web'

Deimos always sets the workdir to /tmp/mesos-sandbox (via -w) which makes it hard to use containers that expect to be in the dir that is set via WORKDIR in Dockerfile.
It would be nice if Deimos could use WORKDIR instead.

That does not allow using local images, created by Dockerfile and not pushed yet to any repository - normal development cycle.

When is the image specificed in default under [containers.image] used? It seems to be ignored now, and the default image is instead something like debian:7. It also looks like I could change the default image by specifying distro and release under [docker.index], but I'm not sure that's the intention of those fields. Thanks!

If I installed deimos on mesos slaves, can I still use the mesos default shell executor?
If so, how to specify which executor to use?
Thanks!

Hi,

starting a simple container using Marathon/Deimos fails because for some reason it is fails on not being able to fetch a .jar file hosted on S3. The docker image is download correctly by a slave from the public docker repo, and can be run manually with no problems.
The app inside the container is a simple 'hello-world' type java app.

Details:

mesos: 0.19.1
deimos: 0.4.0
marathon: 0.6.0-1.0
ubuntu: 14.04 trusty

docker image: tnolet/hello1
Dockerfile:

FROM ubuntu:latest

MAINTAINER Tim Nolet

RUN apt-get update -y

RUN apt-get install -y --no-install-recommends openjdk-7-jre

ENV JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64

RUN apt-get install -y curl

RUN curl -sf -O https://s3-eu-west-1.amazonaws.com/deploy.magnetic.io/snapshots/dropwizard-0.0.1-SNAPSHOT.jar

RUN curl -sf -O https://s3-eu-west-1.amazonaws.com/deploy.magnetic.io/snapshots/hello-world.yml

EXPOSE 8080

EXPOSE 8081

ENV SERVICE hello:0.0.1:8080:8081

CMD java -jar dropwizard-0.0.1-SNAPSHOT.jar server hello-world.yml

task file:

{
    "container": {
    "image": "docker:///tnolet/hello1",
    "options" : []
  },
  "id": "hello1",
  "instances": "1",
  "cpus": ".5",
  "mem": "512",
  "uris": [],
  "cmd": ""
}

Error in stderr in mesos gui:
Error: Unable to access jarfile dropwizard-0.0.1-SNAPSHOT.jar

output from mesos.slave-INFO on slave:

I0731 13:09:21.673143  8814 slave.cpp:1664] Got registration for executor 'hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab' of framework 20140731-110416-606019500-5050-1090-0000
I0731 13:09:21.673703  8814 slave.cpp:1783] Flushing queued task hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab for executor 'hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab' of framewor
k 20140731-110416-606019500-5050-1090-0000
I0731 13:09:21.695307  8814 slave.cpp:2018] Handling status update TASK_RUNNING (UUID: 4e704272-eecd-4205-819c-a2eb63048c18) for task hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab o
f framework 20140731-110416-606019500-5050-1090-0000 from executor(1)@172.31.31.38:49678
I0731 13:09:21.695582  8814 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: 4e704272-eecd-4205-819c-a2eb63048c18) for task hello1.e0be7ca8-18b3-11e4-a08d
-0a4559673eab of framework 20140731-110416-606019500-5050-1090-0000
I0731 13:09:21.695897  8814 status_update_manager.cpp:373] Forwarding status update TASK_RUNNING (UUID: 4e704272-eecd-4205-819c-a2eb63048c18) for task hello1.e0be7ca8-18b3-11e4-a0
8d-0a4559673eab of framework 20140731-110416-606019500-5050-1090-0000 to [email protected]:5050
I0731 13:09:21.696854  8815 slave.cpp:2145] Sending acknowledgement for status update TASK_RUNNING (UUID: 4e704272-eecd-4205-819c-a2eb63048c18) for task hello1.e0be7ca8-18b3-11e4-
a08d-0a4559673eab of framework 20140731-110416-606019500-5050-1090-0000 to executor(1)@172.31.31.38:49678
I0731 13:09:21.702631  8812 status_update_manager.cpp:398] Received status update acknowledgement (UUID: 4e704272-eecd-4205-819c-a2eb63048c18) for task hello1.e0be7ca8-18b3-11e4-a
08d-0a4559673eab of framework 20140731-110416-606019500-5050-1090-0000
I0731 13:09:21.859962  8816 slave.cpp:2355] Monitoring executor 'hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab' of framework '20140731-110416-606019500-5050-1090-0000' in container 
'e4c7ca90-0dff-4492-b3c4-e6c7569f1eeb'
I0731 13:09:22.687067  8813 slave.cpp:2018] Handling status update TASK_FAILED (UUID: 2d405edf-32a0-493e-aea0-d2d8d4cc1f9c) for task hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab of
 framework 20140731-110416-606019500-5050-1090-0000 from executor(1)@172.31.31.38:49678
I0731 13:09:22.698246  8811 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: 2d405edf-32a0-493e-aea0-d2d8d4cc1f9c) for task hello1.e0be7ca8-18b3-11e4-a08d-
0a4559673eab of framework 20140731-110416-606019500-5050-1090-0000
I0731 13:09:22.699434  8811 status_update_manager.cpp:373] Forwarding status update TASK_FAILED (UUID: 2d405edf-32a0-493e-aea0-d2d8d4cc1f9c) for task hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab of framework 20140731-110416-606019500-5050-1090-0000 to [email protected]:5050
I0731 13:09:22.700186  8811 slave.cpp:2145] Sending acknowledgement for status update TASK_FAILED (UUID: 2d405edf-32a0-493e-aea0-d2d8d4cc1f9c) for task hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab of framework 20140731-110416-606019500-5050-1090-0000 to executor(1)@172.31.31.38:49678
I0731 13:09:22.709666  8815 status_update_manager.cpp:398] Received status update acknowledgement (UUID: 2d405edf-32a0-493e-aea0-d2d8d4cc1f9c) for task hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab of framework 20140731-110416-606019500-5050-1090-0000
I0731 13:09:23.060930  8814 slave.cpp:933] Got assigned task hello1.e3b9e259-18b3-11e4-a08d-0a4559673eab for framework 20140731-110416-606019500-5050-1090-0000
I0731 13:09:23.061293  8814 slave.cpp:1043] Launching task hello1.e3b9e259-18b3-11e4-a08d-0a4559673eab for framework 20140731-110416-606019500-5050-1090-0000
I0731 13:09:23.063863  8815 external_containerizer.cpp:433] Launching container 'cfb86a26-2821-49ce-95a0-3e4d0dfd8657'
I0731 13:09:23.080337  8814 slave.cpp:1153] Queuing task 'hello1.e3b9e259-18b3-11e4-a08d-0a4559673eab' for executor hello1.e3b9e259-18b3-11e4-a08d-0a4559673eab of framework '20140731-110416-606019500-5050-1090-0000
E0731 13:09:23.859387  8811 slave.cpp:2397] Termination of executor 'hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab' of framework '20140731-110416-606019500-5050-1090-0000' failed: External containerizer failed (status: 1)
I0731 13:09:23.859632  8811 slave.cpp:2552] Cleaning up executor 'hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab' of framework 20140731-110416-606019500-5050-1090-0000
I0731 13:09:23.860239  8811 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20140731-110416-606019500-5050-1090-2/frameworks/20140731-110416-606019500-5050-1090-0000/executors/hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab/runs/e4c7ca90-0dff-4492-b3c4-e6c7569f1eeb' for gc 6.99999004926815days in the future
I0731 13:09:23.860345  8811 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20140731-110416-606019500-5050-1090-2/frameworks/20140731-110416-606019500-5050-1090-0000/executors/hello1.e0be7ca8-18b3-11e4-a08d-0a4559673eab' for gc 6.99999004838222days in the future
I0731 13:09:23.871316  8816 external_containerizer.cpp:1040] Killed the following process tree/s:
[ 

]

Again. running the following command on the slave manually starts up the container with no problems:
sudo docker run -d -P tnolet/hello1

I have interest in contributing a patch to deimos that would allow for an add-on style similar to Heroku when local dependencies are needed.

As an example of what I had in mind, you create a rails container, but want nginx starting in front of it. Rather than install nginx inside the rails container, they should be 2 separate containers, linked together. Then, if a local memcached were desired, it could be linked into the rails container. I've given some thought to how the definition should work and would be interested in discussing it further if there's interest.

The most basic example would be something like the following

container: {
image: "docker:///nginx"
options: ["--graph", "[{
"key": "myrailsapp",
"image": "docker:///myrailsapp",
"options": [],
"links": ["memcached"],
"volumes-from": []},{
"key": "memcached",
"image": "docker:///memcached",
"options": [],
"links": [],
"volumes-from": []}]"]

Essentially a list of nodes with an adjacency list for links and volumes. The cluster of containers would start or fail as a single atomic unit. Let me know what you guys think!

If deimos fails to run a job (for instance, can't find the image to run), deimos won't remove the failed job from mesos, but leaves the job with staging status, and the resources allocated to this job won't be released.

I'm thinking the answer is no, but is there a way to explicitly use Deimos with Marathon only while using Mesos' default containerization for frameworks?

Howdy,

Id like to use deimos to create OpenVZ containers. Would you be open to me hacking up deimos to work with more than one containerizer? Ive looked over the code a good bit and it seems like a good bit of the code is abstracted in a way that it wouldnt be hard to accomplish said task. Whaddya think?

Also i can attempt to bribe you with beer and tapas!

Not sure if this is something unpushed or a WIP – but the containerizer is currently unusable with RC2 of the EC due to this. The slave comes up but then dies almost immediately due to the lack of this method.

I was recently evaluating Spark + Mesos + Deimos and run into the following issue. By default the Mesos scheduler backend in Spark sets the command to be executed as "cd %s*; ./sbin/spark-executor". This command is passed to Docker run arguments as "sh -c cd ..". However in my setup to enable networking inside the containers, the Docker run arguments need to be prefixed with "dhclient eth0". It would be great if Deimos could provide an option (perhaps in the deimos.cfg) to add prefixes to the Docker run arguments (resulting in sh -c dhclient eth0; ...; cd ..).

I have a question about deimos and centos 6.5 . I install mesos and docker but i have a problem with deimos . I check the source code and I found the problem (https://github.com/mesosphere/deimos/blob/master/deimos/docker.py#L109) I don’t know why in centos docker save the container information in /cgroups/. Any idea how to fix that ?

I'm trying to following this tutorial (http://mesosphere.io/learn/run-docker-on-mesosphere/) using the latest version of Docker (1.0.1). I can't seem to get Marathon to launch the container through deimos unless I launch it with the --privileged option.

{
  "container": {
    "image": "docker:///libmesos/ubuntu",
    "options": ["--privileged"]
  },
  "id": "ubuntu",
  "instances": "1",
  "cpus": ".5",
  "mem": "300",
  "uris": [ ],
  "cmd": "sleep 500"
}

Is this a known issue or am I executing it incorrectly. I'm launching docker as /usr/bin/docker -d -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock

When a URL has an authority component (docker://<authority>/<image>), Deimos should pass it as the registry to Docker (appending the default port if necessary, for Docker's benefit). For URLs without an authority (docker:///<image>), it should be possible to specify a default, to be used in place of the public registry at index.docker.io.

As per the README, I tried launching a container running a service with a port mapping like this, passing the option through as two separate list elements:

  "container": {
    "image": "docker:///libmesos/ubuntu",
    "options" : ["-p", "8090:8080"]
    },
... snipped

This causes Deimos to fail to launch the container. Merging the items works though!

  "container": {
    "image": "docker:///libmesos/ubuntu",
    "options" : ["-p 8090:8080"]
    },
... snipped

The deimos/VERSION file is missing (even though this is still unreleased) which makes installing via setup.py problematic.

• Check for containers without exit files that are no longer running
• Check for containers that are still running but have no deimos parent process
• Check that containers with an exit file have indeed exited
• &c

Hi,

what is the right way to get the service port of my app mapped to the random port marathon gives my container?

I can expose my port with expose and -P commands additionally but i still get a random port with a useless mapping.

So instead of

8080/tcp, 0.0.0.0:31013->31013/tcp

i would like to get

0.0.0.0:31013->8080/tcp

http://tools.ietf.org/html/rfc3986#section-3

docker:///ubuntu implies scheme=docker, authority=(null), path=ubuntu. This is valid, since the authority is actually implied. It's also valid for things like docker:///zaiste/postgresql, but with private registries, it's kinda wrong:

docker:///registry.borg.chi.shopify.com:5000/shopify:1234deadbeefcafe

If we dissect this, we get scheme=docker authority=null, path=registry.borg.chi.shopify.com:5000/shopify:1234deadbeefcafe. Really, it should be scheme=docker authority=registry.borg.chi.shopify.com:5000, path=shopify:1234deadbeefcafe. Thus the URL would look like docker://registry.borg.chi.shopify.com:5000/shopify:1234deadbeefcafe, with only 2 slashes after the scheme.

There are a couple places here and in the earlier docker-executor where you assume docker URLs should always begin with docker:///. It's a minor thing, since plugging in the theoretically-invalid URL works just fine in all cases, but it's probably easier to fix this now than it will be down the road.

Was looking around how to map multiple exposed ports to marathon/mesos ports using deimos. I was surprised by the following:

Starting a container that exposes ports 8080 and 8081, I (naively) assigned two ports in my json file:

{
    "container": {
    "image": "docker:///tnolet/hello:1.0",
    "options" : ["-P"]
  },
  "id": "hello2",
  "instances": "1",
  "cpus": ".5",
  "mem": "512",
  "uris": [],
  "cmd": "",
  "ports" : [40195,40196]
}

However, this resulted in deimos picking and auto incrementing two ports, 31267 and 31268:

Jul 31 17:17:17 ec2-54-84-57-179 deimos[5707]: deimos.docker.run() Port pairings (Mesos, Docker) // [(31627, 8080), (31628, 8081)]
Jul 31 17:17:17 ec2-54-84-57-179 deimos[5707]: deimos.containerizer.docker.launch() call // docker run --sig-proxy --rm --cidfile /tmp/deimos/mesos/54f421fd-0935-4f34-8be9-33a17862024e/cid -w /tmp/mesos-sandbox -v /tmp/deimos/mesos/54f421fd-0935-4f34-8be9-33a17862024e/fs:/tmp/mesos-sandbox -P -p 31627:8080 -p 31628:8081 -c 512 -m 512m -e PORT=31627 -e PORT0=31627 -e PORTS=31627,31628 -e PORT1=31628 tnolet/hello:1.0

If I leave the "port" key out of my json file, deimos only maps the lowest numbered port.
I love the auto incremented ports, but I'm not sure how this is triggered.

Seems to me that https://github.com/mesosphere/deimos/blob/master/deimos/containerizer/__init__.py#L97
and
https://github.com/mesosphere/deimos/blob/master/deimos/containerizer/__init__.py#L99

are missing the return statement and so the status code is not propagated.

Would it be possible to support Python 2.6? CentOS/RHEL 6 is still using it.

Is there any way to install on Debian? I tried compiling the latest version of Mesos myself, but I must still be missing something, because the command line options for mesos-slave aren't defined.

I am trying to pass the -e option for docker run via the deimos config file to set up some environment variables.

My config file looks like this

mesos@mesos-5:~$ cat /usr/local/etc/deimos.cfg 

[docker]
# These are options passed to Docker *before* the call to run/images/&c. This
# allows you to set the Docker host. Note that any key:value pair whatever can
# be set here. If the syntax of a key is such that it resembles a JSON list,
# the option will be passed multiple times, once for each item in the list.
host: ["unix:///var/run/docker.sock"]

[docker.index]
account_libmesos: libmesos
#account: guilhem 
index: index.docker.io

[containers.options]
# This array of options is passed to 'run' if there are none in the TaskInfo.
default: []
# These options are passed to 'run' in addition to those in the TaskInfo.
append: ['-e JAVA_HOME=java-6-openjdk-amd64']
# This flag causes all container options in the TaskInfo to be ignored.
ignore: false

[containers.image]
# The image to use when none is specified in the TaskInfo.
default: docker:///centos
# Causes Deimos to ignore the container image specified in the TaskInfo.
ignore: false

[uris]
# When false, Deimos will leave Tar and Zip archives as-is after download.
unpack: True

[log]
# Enable syslog and set its log level.
syslog:  DEBUG
# Enable console logging and set its log level.
console: DEBUG

[state]
root: /tmp/deimos

The output of deimos config is :

mesos@mesos-5:~$ deimos config
07:16:00.868 deimos.config.load_configuration() Loaded configuration from /usr/local/etc/deimos.cfg
07:16:00.868 deimos.config.load_configuration() Found: deimos.config.Docker(host=[u'unix:///var/run/docker.sock'])
07:16:00.869 deimos.config.load_configuration() Found: deimos.config.State(root='/tmp/deimos')
07:16:00.869 deimos.config.load_configuration() Found: deimos.config.Containers(image=deimos.config.Image(ignore=False, default='docker:///centos'), options=deimos.config.Options(ignore=False, default=[], append=[]))
07:16:00.869 deimos.config.load_configuration() Found: deimos.config.Log(syslog=10, console=10)
07:16:00.870 deimos.config.load_configuration() Found: deimos.config.DockerIndex(account_libmesos='libmesos', index='index.docker.io', account=None)
07:16:00.870 deimos.cli() Final configuration:
deimos.config.DockerIndex(account_libmesos='libmesos', index=None, account=None)
deimos.config.Log(syslog=10, console=10)
deimos.config.URIs(unpack=True)
deimos.config.State(root='/tmp/deimos')
deimos.config.Docker(host=[u'unix:///var/run/docker.sock'])
deimos.config.DockerIndex(account_libmesos='libmesos', index='index.docker.io', account=None)
deimos.config.Containers(image=deimos.config.Image(ignore=False, default='docker:///centos'), options=deimos.config.Options(ignore=False, default=[], append=[]))

Deimos should automatically GC the state directory, ideally at a regular interval. One option would be for every command to spawn a subprocess that attempts to lock the state directory, performs the GC operation (deimos state --rm) for stale containers, and then holds the lock for up to 5-10 minutes (to prevent the operation from running to frequently).

Previously with the mesos-docker executor, we had made some modifications that stored data about containers and their IP:PORT pairs in redis once they had been launched, and cleaned up redis once they were stopped or killed. This was being used for routing via openresty+lua redis. Since the switch to deimos, we have been looking for a way to accomplish the same thing.

One option would be to put the logic into our docker images and have them register/de-register themselves. This would work fine for our internal images that we build ourselves, but would not work with pre-built images from the docker index.

The options I see are:

1. Write something that keeps redis in sync using data from the marathon API/events
2. Write something similar to confd that writes out openresty/nginx upstreams from marathon API or zookeeper data.
3. Modify deimos to allow running a script or task after successful container launch, or shutdown.

I'm basically just looking for advice and best practices, and want to see if you guys had any plans for easier integration with service discovery before making any decisions about which way to go.

There's a large refactor that's gone on with the EC which deimos needs updating to match. Here's a few notable changes that I can see (against the 0.19.0 EC release candidate).

• Different protobuf messages
• New containers method to return list of running container IDs
• New recover method for recovering internal state
• New update method for updating a containers resources
  • This is an interesting one... deimos needs to (at minimum) support changing CPU and memory limits, though ports would be a good one.

There might be some improvements that can be made to how deimos works (and manages state) due to the introduction of the new recover method.

Would be interested in your thoughts @solidsnack – is this something you've started work on?

Recently I tried running deimos with mesos-slave-0.19.0 and got the following error:

Failed to perform recovery: Recover failed: External containerizer failed (status: 127)
To remedy this do as follows:
Step 1: rm -f /tmp/mesos/slave/work_dir/meta/slaves/latest
        This ensures slave doesn't recover old live executors.
Step 2: Restart the slave.

However the reason the slave would not start was due to the fact that deimos was installed at /usr/bin/deimos rather than /usr/local/bin/deimos (due to running on openSUSE rather than Ubuntu)

Is it possible to run any docker container available in the registry or only containers based upon libmesos/ubuntu?

For example, I have some containers that start their single internal process as soon as they are launched, so I'd like to do something like:

{
  "container": {
    "image": "docker:///jplock/rabbitmq",
    "options": []
  },
  "id": "rabbitmq",
  "instances": "1",
  "cpus": ".5",
  "mem": "300",
  "uris": [ ],
  "cmd": ""
}

with no cmd specified and no options provided. Should that work or what is the significance of libmesos/ubuntu? This container exists in the Docker registry at https://hub.docker.com/u/jplock/rabbitmq/ and I am able to run docker pull jplock/rabbitmq to download it. Thanks.

So, I was fiddling with Mesos on a single-node machine. I'd been using it with Spark and then found out about Deimos to utilize Docker as an external containerization medium.

It worked up until today. I've got this installed on an Ubuntu 14.04 64-bit installation in Virtualbox. Mesos master and slave are working. I also have Marathon running. Now, when I try to submit a Spark job, the first stage in the pipeline has a stderr output of the following:

Unable to find image 'libmesos/ubuntu:14.04' locally
Pulling repository libmesos/ubuntu
2014/07/10 10:46:10 Tag 14.04 not found in repository libmesos/ubuntu

I added to /etc/mesos-slave the file default_container_image with libmesos/ubuntu:13.10 inside it. After rebooting the VM, I can see that the mesos slave is started with that option:

--default_container_image=libmesos/ubuntu:13.10

So, I'm not sure if ubuntu:14.04 is suddenly being fed into the system or what, but I haven't touched anything aside from rebooting the image. I can't figure out how to get the Spark submission to not try to download libmesos/ubuntu:14.04 (and I'm not going through Marathon for this, obviously).

I'm trying to use mesos/marathon/deimos with an image hosted on a private registry, quay.io in my case. When I try to run the app, it fails with the following error:

Jul  2 19:09:38 vagrant deimos[1695]: deimos.containerizer.docker.launch()
Jul  2 19:09:38 vagrant deimos[1695]: deimos.state.lock() request // launch EX (60s)
Jul  2 19:09:38 vagrant deimos[1695]: deimos.state.lock() success // launch EX (60s)
Jul  2 19:09:38 vagrant deimos[1695]: deimos.containerizer.docker.launch() eid    = testapp-8_0-1404328178191
Jul  2 19:09:38 vagrant deimos[1695]: deimos.containerizer.docker.launch() mesos  = fdd21297-c501-4626-a757-d299e3f3097d
Jul  2 19:09:38 vagrant deimos[1695]: deimos.containerizer.docker.launch() image  = quay.io/myuser/testapp:latest
Jul  2 19:09:38 vagrant deimos[1695]: deimos.state.lock() request // observe EX|NB (60s)
Jul  2 19:09:38 vagrant deimos[1695]: deimos.state.lock() success // observe EX|NB (60s)
Jul  2 19:09:39 vagrant deimos[1695]: deimos.docker.pull() exit 1 // docker pull quay.io/myuser/testapp:latest
Jul  2 19:09:39 vagrant deimos[1695]: deimos.docker.pull() STDERR // 2014/07/02 19:09:39 HTTP code: 403
Jul  2 19:09:39 vagrant deimos[1695]: deimos.cli() Command '['docker', 'pull', u'quay.io/myuser/testapp:latest']' returned non-zero exit status 1

Which makes sense. My login shell can pull from the registry because I run docker login. which cached my credentials in ~/.dockercfg. It doesn't look like docker login is executed here, and even if it did my credentials have not been passed in.

Are there any known ways to allow deimos to authenticate with a private registry?

If I can help debug in any way let me know. I'm loving mesos/marathon/deimos so far!