merll / docker-fabric Goto Github PK

View Code? Open in Web Editor NEW

79.0 2.0 10.0 269 KB

Integration of Docker deployments into Fabric.

License: MIT License

Python 100.00%

fabric docker docker-client python docker-containers dockerfile

docker-fabric's Introduction

Docker-Fabric

Build Docker images, and run Docker containers in Fabric.

Project: https://github.com/merll/docker-fabric

Docs: https://docker-fabric.readthedocs.io/en/latest/

Overview

With a few preparations, Docker images can easily be generated and tested on development machines, and transferred on to a production environment. This package supports managing container configurations along with their dependencies within Fabric-based deployments. DockerFiles can also be easily implemented in Fabric tasks.

Local Docker clients can be controlled directly through docker-py. Remote Docker API services make use of Fabric's SSH connection.

API access

This project is based on Docker-Map, and adapts its container configuration methods.

As with Docker-Map, container configurations can be generated as objects, updated from Python dictionaries, or imported from YAML files in order to control remote clients via the API. Docker-Fabric includes the following enhancements:

Docker client

DockerFabricClient adds Fabric-like logging in the context of container instances on top of Fabric hosts, and enables automatic creation of tunnel connections for access to a remote Docker host using Fabric's SSH connection. By using the tool socat, the Docker client can access a remote service without re-configuration.

Client configuration

DockerClientConfiguration adds the capability of running containers to Fabric hosts with specific Docker settings for each, e.g. the version number.

Running container configurations

ContainerFabric is a simple wrapper that combines Docker-Map's DockerFabricClient, DockerClientConfiguration objects, and container maps.

Command-line based access

The following features are provided by running the appropriate commands on a remote Docker command line:

Copy resources from a container to a Fabric host.
Copy resources from a container and download them in a compressed tarball. The Docker Remote API currently does not support creating compressed tarballs.
Copy resources from a container and store them in a new blank image.
Generate a compressed image tarball. The Docker Remote API currently does not support creating compressed tarballs, but is capable of importing them.

Tasks

All essential container actions (create, start, stop, remove) and some advanced (e.g. update) can be triggered from the command line as Fabric tasks and executed on the remote service, e.g. via SSH.

Additionally the following tasks are included in this package, that can be run by Fabric directly:

check_version: Returns version information of the remote Docker service and provides useful insight if permissions are set up properly.
cleanup_containers: Removes all containers that have stopped.
cleanup_images: Removes all untagged images, that do not have a dependent container or other dependent images.
remove_all_containers: Stops and removes all containers on the remote Docker service.

Contributions

Thanks to lfasnacht for publishing an implementation for a local tunnel to a Fabric client in the pull request 939 of Fabric.

Further contributions are maintained in CONTRIBUTIONS.md of the project.

docker-fabric's People

Contributors

Stargazers

Watchers

Forkers

kenden primozs zalan-axis yonglehou pombredanne bountin pczerkas aflp91 happyshi0402 alexmziraik

docker-fabric's Issues

task list_containers does not return anything

The task list_containers does not list any containers:

$ echo "from dockerfabric import tasks as docker" > fabfile.py
$ fab -H 10.0.101.133 docker.list_containers
[10.0.101.133] Executing task 'docker.list_containers'
[10.0.101.133]
Id Names Image Command Ports Status Created

Done.

Note: list images is fine
$ fab -H 10.0.101.133 docker.list_images
[10.0.101.133] Executing task 'docker.list_images'
[10.0.101.133]
Id RepoTags ParentId Created VirtualSize Size
1183a43c2e94 stylight/ws-solr:latest 80ef0e18de7e 2015-02-27T12:39:25 1454511143 0
45bb9005020b stylight/wildfly:latest ec0bddcfa096 2015-02-11T09:24:48 1288633198 0

Done.

There IS a container runnning:
$ ssh 10.0.101.133
Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-46-generic x86_64)

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b6a28a641321 stylight/ws-solr:12 "/bin/sh -c '/usr/lo 2 days ago Up 8 hours 9999/tcp, 8787/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:9990->9990/tcp solr

lxc-docker vs. docker-engine

Docker's latest installation instructions instruct users to install docker-engine. This project sill references lxc-docker. Is there a reason this project doesn't reflect the latest instructions?

Documentation lacks more examples

I'd like very much to use docker-fabric in my projects (especially in testing) but I'm having a hard time trying to figure out how to use it. The examples in documentation are too basic and shallow to be useful. There are no unit tests and I couldn't find any projects or examples elsewhere using docker-fabric. I wonder if you could take some examples from fabric's docs and use them in a docker-fabric's context.

I miss an example that covers something from beginning to end such as:

build a docker image
create and start a container
run some commands in the container and several other tasks
stop and destroy the container

Regards,

André

_build_socat

This function was removed. Is it no longer necessary? Has it been replaced by something else? I honestly don't recall why I needed it, but I have a broken reference when I try to upgrade to the latest version of this package.

Invalid Hostname

When testing the following script as part of #13, I was getting an invalid hostname error.

from dockermap.map.container import ContainerMap
from dockermap.map.input import ContainerLink
from dockerfabric.apiclient import docker_fabric, container_fabric

maps = ContainerMap('test_map', {
    'host_root': '~',
    'u1': {
        'image': 'ubuntu:latest',
    }
})

container_fabric_inst = container_fabric(container_maps=maps)
container_fabric_inst.startup('u1')

unknown flag: --host-config

After upgrading to python 3.7, docker[tls]==3.7.1, docker-map==1.0.0 and docker-fabric==0.5.0, I'm running into the error unknown flag: --host-config when trying to create a container. I'm having trouble even finding the host-config option so I'm not clear what's causing this:

a py3 issue (e.g. the wrong flag is created)
a feature removed from docker
an issue in docker-py

Executing SSH command in running container?

Is it possible to execute a traditional fabric command within a running docker container? e.g. Make the Fabric run() execute inside Docker?

I'm looking for a tool that can SSH into and run commands inside a docker container. I'm familiar with Fabric, so I thought a tool named Docker-Fabric would be exactly what I needed. However, after reading through the documentation, there doesn't seem to be any Fabric-like functionality. This just appears to be a Python wrapper around the service-level components of Docker. Is this correct?

For example, I saw this doc page about SSH tunneling, but nothing there describes how to execute SSH commands inside the container. Is this outside the scope of the project?

Basic example does not work

I am following the documentation at:
https://docker-fabric.readthedocs.org/en/latest/installation.html

pip install docker-fabric
sudo apt-get install lxd-docker-1.5.0 (on host and remote machine)
sudo apt-get install socat (on host and remote machine)
usermod -aG docker jenkins
echo "from dockerfabric import tasks as docker" > fabfile.py

But I get the error:

$ fab docker.version
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/fabric/main.py", line 743, in main
    *args, **kwargs
  File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 424, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/dockerfabric/tasks.py", line 150, in version
    output = docker_fabric().version()
  File "/usr/local/lib/python2.7/dist-packages/dockerfabric/apiclient.py", line 45, in get_connection
    return self.get(key, DockerFabricClient, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/dockerfabric/base.py", line 15, in get
    e = d(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/dockerfabric/apiclient.py", line 94, in __init__
    local_port = get_local_port(init_local_port)
  File "/usr/local/lib/python2.7/dist-packages/dockerfabric/base.py", line 24, in get_local_port
    return int(init_port) + current_offset
TypeError: int() argument must be a string or a number, not 'NoneType'

I get the same with:

$ fab -H "localhost" docker.version
$ fab -H "10.0.101.133" docker.version

Note that SSH works:

$ ssh 10.0.101.133
Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-46-generic x86_64)
jenkins@test:~$

That's on Ubuntu 14.10

I have the following version installed:
docker-fabric (0.3.1)
docker-map (0.2.2)
docker-py (1.0.0)

Docker CLI Client Inheritance

Around line 85 of dockermap.map.client, a check is run looking for (among other things) a docker.Client. DockerFabricClient passes because it inherits (indirectly) from this class but the DockerCliClient does not (it only inherits from DockerUtilityMixin).

Traceback (most recent call last):
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/main.py", line 756, in main
    *args, **kwargs
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 426, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
    return self.wrapped(*args, **kwargs)
  File "/mnt/data/apex-server/build_deploy/src/fabfile.py", line 898, in setup_service_graylog
    startup_required_containers(db_client)
  File "/mnt/data/apex-server/build_deploy/src/fabfile.py", line 1903, in startup_required_containers
    container_fabric_inst = container_fabric(docker_client=docker_client, container_maps=[])
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockerfabric/api.py", line 48, in container_fabric
    return ContainerCliFabricClient(container_maps, docker_client, clients)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockerfabric/base.py", line 99, in __init__
    clients=current_clients)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 85, in __init__
    raise ValueError("Unexpected type of 'docker_client' argument: {0}".format(type(docker_client)))
ValueError: Unexpected type of 'docker_client' argument: <class 'dockerfabric.cli.DockerCliClient'>

Start with Recursive Dependencies

While debugging #12, I rolled back to docker-fabric==0.3.10 and docker-map==0.6.6. I ran into the following issue:

docker.errors.APIError: 500 Server Error: Internal Server Error ("{"message":"Cannot link to a non running container: /graylog_map.elasticsearch AS /graylog_map.graylog/elasticsearch"}")

In this case, I have a two-layer dependency: nginx => graylog => elasticsearch & mongo. When I didn't have nginx it worked correctly. When I manually coded the launch order, it worked correctly. I assume, therefore, that 0.3.10 is not recursively checking dependencies in the start logic.

NOTE: I can't replicate the issue in 0.4.0 until #12 is resolved so it could (at least theoretically) be fixed.

sudo access to socket

I'd like to use sudo to access the docker socket rather than the docker group. I'm happy to contribute a patch but want to make sure it's (1) compatible with the way docker-fabric already does things and (2) something that would be accepted once it was working. Any hint on the right part of the architecture to explore would also be much appreciated.

We got to this point because we wanted to use our LDAP to control who has docker access. We already control sudoers using LDAP groups and this article was the "duh" moment that we could do the same for docker with a specialzed sudoers line (and make it all transparent to our users). As a HIPAA-compliant organization, the security benefits that motivated the original article made it a "must have" for us.

Unfortunately, docker-fabric fails in this configuration and we want to fix it.

Regex for CLI parser is too narrow

This has been reported in #8, and is created as a new issue for tracking when it is fixed:

Removing the docker.Client check in #10 (and not adding it to the CLI class) has resolved the -H issue. The next issue I ran into was the CREATED_AT_PATTERN regex. Specifically, my time component was:

2016-12-06 18:09:58 -0500 EST

The regex pattern doesn't like the - timezone and can be fixed as:

CREATED_AT_PATTERN = re.compile(r'(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2}) .\d{4} \w+')

I tried (+|-) but it creates an additional, invalid group. The . will match anything but this isn't too risky since the rest of the pattern has to work.

API Connection Issue

(To ensure I'm not running into bugs in the CLI code, I switched back to the API. I also rebuilt the virtualenv from scratch so I didn't have lingering code modifications.)

Now I'm seeing the following (sanitized):

DEBUG:dockermap.map.client:Passing kwargs to client actions: {}
DEBUG:dockermap.map.state.base:Following dependency path for graylog_map.nginx.
DEBUG:dockermap.map.state.base:Dependency path at graylog_map.graylog, instances [None].
DEBUG:docker.auth.auth:Trying paths: ['/home/<domain>/<user>/.docker/config.json', '/home/<domain>/<user>/.dockercfg']
DEBUG:docker.auth.auth:Found file at path: /home/<domain>/<user>/.docker/config.json
DEBUG:docker.auth.auth:Found 'auths' section
DEBUG:docker.auth.auth:Found entry (registry=u'https://index.docker.io/v1/', username=u'<username>')
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1
DEBUG:paramiko.transport:[chan 19] Max packet in: 32768 bytes
ERROR:paramiko.transport:Secsh channel 19 open FAILED: Connection refused: Connect failed

As part of my testing, I opened another SSH session on the build machine (while the first was failed/frozen) and ran a successful call using the socat port:

$ docker -H localhost:22026 info
Containers: 5
 Running: 0
 Paused: 0
 Stopped: 5
...

I'm not even sure what to check next.

CLI Cient Authentication

The CLI introduces an extra wrinkle when it comes to authentication. Since we're not using the API, our local credentials doesn't get used by the client. Obviously, the simplest solution is to docker login on the client side, but that leaves a password hash in .docker/config.json so I'm trying to figure out if we can use the token infrastructure with the CLI.

CLI Client Error Parsing

+1 on the experimental CLI client

I'm testing it and ran into the following issue.

Traceback (most recent call last):
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/main.py", line 756, in main
    *args, **kwargs
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 426, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
    return self.wrapped(*args, **kwargs)
  File "/mnt/data/apex-server/build_deploy/src/fabfile.py", line 897, in setup_service_graylog
    startup_required_containers(db_client)
  File "/mnt/data/apex-server/build_deploy/src/fabfile.py", line 1908, in startup_required_containers
    container_fabric_inst.startup(container)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 227, in startup
    return self.run_actions('startup', container, instances=instances, map_name=map_name, **kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 110, in run_actions
    for states in state_generator.get_states(map_name or self._default_map, config_name, instances=instances):
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/state/base.py", line 221, in get_dependency_states
    client_names=client_names, is_dependency=True):
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/state/base.py", line 118, in generate_config_states
    instance_states = [i_state for i_state in _get_state(config_flags, instances)]
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/state/base.py", line 112, in _get_state
    client, item, c_flags)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/state/base.py", line 61, in get_container_state
    if container_name in self._policy.container_names[client_name]:
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/policy/cache.py", line 124, in __getitem__
    return self.refresh(item)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/policy/cache.py", line 137, in refresh
    val = self.item_class(client)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/policy/cache.py", line 15, in __init__
    self.refresh()
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/policy/cache.py", line 93, in refresh
    current_containers = self._client.containers(all=True)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockerfabric/cli.py", line 77, in containers
    return parse_containers_output(res)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/client/cli.py", line 166, in parse_containers_output
    _container_info(line) for line in out.splitlines() or ()
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/client/cli.py", line 72, in _container_info
    'Image': items[1],
IndexError: list index out of range

The problem is that (the repr of) the "resource" being returned is the following:

'invalid value "http://127.0.0.1:22026" for flag -H: Invalid bind address format: http://127.0.0.1:22026\r\nSee \'docker --help\'.'

Obviously, I'm going to do some digging to see if I can resolve, but I wanted to get the report out there ASAP in case you could shortcut the search.

I cant configure environment variable to container

Hi,
I'm trying to run a container that depends on some environment variables that I have to pass. I tried to add environment key like this but it seems not working.

env.docker_maps = ContainerMap('example_map', {
'repository': env.registry_prefix,
'host_root': env.host_root_path,
'excmd-service': {
'image': 'excmd:latest',
'instances': ('inst1'),
'environment': {
'MQ_IPADDRESS': env.app_mq_ip,
'DB_IPADDRESS': env.app_db_ip,
'MQ_TOPIC_NAME': env.MQ_TOPIC_NAME,
'DB_JOURNAL_KEYSPACE' : env.app_def_journal_kspace,
'DB_SNAPSHOT_KEYSPACE' : env.app_def_snapshot_kspace,
},

},

})

0.4.1: object has no attribute 'rsplit'

Upgraded. When I reached code to pull my repositories, I get the following:

DEBUG:docker.auth.auth:Found 'auths' section
DEBUG:docker.auth.auth:Found entry (registry=u'https://index.docker.io/v1/', username=u'<username>')
Traceback (most recent call last):
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/main.py", line 756, in main
    *args, **kwargs
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 426, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
    return self.wrapped(*args, **kwargs)
  File "/data/apex-server/build_deploy/src/fabfile.py", line 153, in full_setup
    setup_service_graylog(device_name=target, ldap=ldap)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 170, in __call__
    return self.run(*args, **kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
    return self.wrapped(*args, **kwargs)
  File "/data/apex-server/build_deploy/src/fabfile.py", line 812, in setup_service_graylog
    pull_required_containers()
  File "/data/apex-server/build_deploy/src/fabfile.py", line 1957, in pull_required_containers
    docker_client.pull(env.docker_maps.containers[container].image)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockerfabric/apiclient.py", line 249, in pull
    insecure_registry=c_insecure, **kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/client/base.py", line 184, in pull
    response = super(DockerClientWrapper, self).pull(repository, tag=tag, stream=stream, **kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/docker/api/image.py", line 152, in pull
    repository, tag = utils.parse_repository_tag(repository)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/docker/utils/utils.py", line 365, in parse_repository_tag
    parts = repo_name.rsplit('@', 1)
AttributeError: '_NotSet' object has no attribute 'rsplit'

Probably won't dig into the issue tonight, but wanted to at least get it reported.