merll / docker-fabric Goto Github PK

View Code? Open in Web Editor NEW

79.0 2.0 10.0 269 KB

Integration of Docker deployments into Fabric.

License: MIT License

Python 100.00%

fabric docker docker-client python docker-containers dockerfile

docker-fabric's Issues

CLI Client Error Parsing

+1 on the experimental CLI client

I'm testing it and ran into the following issue.

Traceback (most recent call last):
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/main.py", line 756, in main
    *args, **kwargs
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 426, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
    return self.wrapped(*args, **kwargs)
  File "/mnt/data/apex-server/build_deploy/src/fabfile.py", line 897, in setup_service_graylog
    startup_required_containers(db_client)
  File "/mnt/data/apex-server/build_deploy/src/fabfile.py", line 1908, in startup_required_containers
    container_fabric_inst.startup(container)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 227, in startup
    return self.run_actions('startup', container, instances=instances, map_name=map_name, **kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 110, in run_actions
    for states in state_generator.get_states(map_name or self._default_map, config_name, instances=instances):
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/state/base.py", line 221, in get_dependency_states
    client_names=client_names, is_dependency=True):
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/state/base.py", line 118, in generate_config_states
    instance_states = [i_state for i_state in _get_state(config_flags, instances)]
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/state/base.py", line 112, in _get_state
    client, item, c_flags)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/state/base.py", line 61, in get_container_state
    if container_name in self._policy.container_names[client_name]:
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/policy/cache.py", line 124, in __getitem__
    return self.refresh(item)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/policy/cache.py", line 137, in refresh
    val = self.item_class(client)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/policy/cache.py", line 15, in __init__
    self.refresh()
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/policy/cache.py", line 93, in refresh
    current_containers = self._client.containers(all=True)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockerfabric/cli.py", line 77, in containers
    return parse_containers_output(res)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/client/cli.py", line 166, in parse_containers_output
    _container_info(line) for line in out.splitlines() or ()
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/client/cli.py", line 72, in _container_info
    'Image': items[1],
IndexError: list index out of range

The problem is that (the repr of) the "resource" being returned is the following:

'invalid value "http://127.0.0.1:22026" for flag -H: Invalid bind address format: http://127.0.0.1:22026\r\nSee \'docker --help\'.'

Obviously, I'm going to do some digging to see if I can resolve, but I wanted to get the report out there ASAP in case you could shortcut the search.

Documentation lacks more examples

I'd like very much to use docker-fabric in my projects (especially in testing) but I'm having a hard time trying to figure out how to use it. The examples in documentation are too basic and shallow to be useful. There are no unit tests and I couldn't find any projects or examples elsewhere using docker-fabric. I wonder if you could take some examples from fabric's docs and use them in a docker-fabric's context.

I miss an example that covers something from beginning to end such as:

build a docker image
create and start a container
run some commands in the container and several other tasks
stop and destroy the container

Regards,

André

Docker CLI Client Inheritance

Around line 85 of dockermap.map.client, a check is run looking for (among other things) a docker.Client. DockerFabricClient passes because it inherits (indirectly) from this class but the DockerCliClient does not (it only inherits from DockerUtilityMixin).

Traceback (most recent call last):
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/main.py", line 756, in main
    *args, **kwargs
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 426, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
    return self.wrapped(*args, **kwargs)
  File "/mnt/data/apex-server/build_deploy/src/fabfile.py", line 898, in setup_service_graylog
    startup_required_containers(db_client)
  File "/mnt/data/apex-server/build_deploy/src/fabfile.py", line 1903, in startup_required_containers
    container_fabric_inst = container_fabric(docker_client=docker_client, container_maps=[])
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockerfabric/api.py", line 48, in container_fabric
    return ContainerCliFabricClient(container_maps, docker_client, clients)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockerfabric/base.py", line 99, in __init__
    clients=current_clients)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/map/client.py", line 85, in __init__
    raise ValueError("Unexpected type of 'docker_client' argument: {0}".format(type(docker_client)))
ValueError: Unexpected type of 'docker_client' argument: <class 'dockerfabric.cli.DockerCliClient'>

lxc-docker vs. docker-engine

Docker's latest installation instructions instruct users to install docker-engine. This project sill references lxc-docker. Is there a reason this project doesn't reflect the latest instructions?

Start with Recursive Dependencies

While debugging #12, I rolled back to docker-fabric==0.3.10 and docker-map==0.6.6. I ran into the following issue:

docker.errors.APIError: 500 Server Error: Internal Server Error ("{"message":"Cannot link to a non running container: /graylog_map.elasticsearch AS /graylog_map.graylog/elasticsearch"}")

In this case, I have a two-layer dependency: nginx => graylog => elasticsearch & mongo. When I didn't have nginx it worked correctly. When I manually coded the launch order, it worked correctly. I assume, therefore, that 0.3.10 is not recursively checking dependencies in the start logic.

NOTE: I can't replicate the issue in 0.4.0 until #12 is resolved so it could (at least theoretically) be fixed.

Basic example does not work

I am following the documentation at:
https://docker-fabric.readthedocs.org/en/latest/installation.html

pip install docker-fabric
sudo apt-get install lxd-docker-1.5.0 (on host and remote machine)
sudo apt-get install socat (on host and remote machine)
usermod -aG docker jenkins
echo "from dockerfabric import tasks as docker" > fabfile.py

But I get the error:

$ fab docker.version
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/fabric/main.py", line 743, in main
    *args, **kwargs
  File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 424, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/fabric/tasks.py", line 174, in run
    return self.wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/dockerfabric/tasks.py", line 150, in version
    output = docker_fabric().version()
  File "/usr/local/lib/python2.7/dist-packages/dockerfabric/apiclient.py", line 45, in get_connection
    return self.get(key, DockerFabricClient, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/dockerfabric/base.py", line 15, in get
    e = d(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/dockerfabric/apiclient.py", line 94, in __init__
    local_port = get_local_port(init_local_port)
  File "/usr/local/lib/python2.7/dist-packages/dockerfabric/base.py", line 24, in get_local_port
    return int(init_port) + current_offset
TypeError: int() argument must be a string or a number, not 'NoneType'

I get the same with:

$ fab -H "localhost" docker.version
$ fab -H "10.0.101.133" docker.version

Note that SSH works:

$ ssh 10.0.101.133
Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-46-generic x86_64)
jenkins@test:~$

That's on Ubuntu 14.10

I have the following version installed:
docker-fabric (0.3.1)
docker-map (0.2.2)
docker-py (1.0.0)

Invalid Hostname

When testing the following script as part of #13, I was getting an invalid hostname error.

from dockermap.map.container import ContainerMap
from dockermap.map.input import ContainerLink
from dockerfabric.apiclient import docker_fabric, container_fabric

maps = ContainerMap('test_map', {
    'host_root': '~',
    'u1': {
        'image': 'ubuntu:latest',
    }
})

container_fabric_inst = container_fabric(container_maps=maps)
container_fabric_inst.startup('u1')

I cant configure environment variable to container

Hi,
I'm trying to run a container that depends on some environment variables that I have to pass. I tried to add environment key like this but it seems not working.

env.docker_maps = ContainerMap('example_map', {
'repository': env.registry_prefix,
'host_root': env.host_root_path,
'excmd-service': {
'image': 'excmd:latest',
'instances': ('inst1'),
'environment': {
'MQ_IPADDRESS': env.app_mq_ip,
'DB_IPADDRESS': env.app_db_ip,
'MQ_TOPIC_NAME': env.MQ_TOPIC_NAME,
'DB_JOURNAL_KEYSPACE' : env.app_def_journal_kspace,
'DB_SNAPSHOT_KEYSPACE' : env.app_def_snapshot_kspace,
},

},

})

Executing SSH command in running container?

Is it possible to execute a traditional fabric command within a running docker container? e.g. Make the Fabric run() execute inside Docker?

I'm looking for a tool that can SSH into and run commands inside a docker container. I'm familiar with Fabric, so I thought a tool named Docker-Fabric would be exactly what I needed. However, after reading through the documentation, there doesn't seem to be any Fabric-like functionality. This just appears to be a Python wrapper around the service-level components of Docker. Is this correct?

For example, I saw this doc page about SSH tunneling, but nothing there describes how to execute SSH commands inside the container. Is this outside the scope of the project?

CLI Cient Authentication

The CLI introduces an extra wrinkle when it comes to authentication. Since we're not using the API, our local credentials doesn't get used by the client. Obviously, the simplest solution is to docker login on the client side, but that leaves a password hash in .docker/config.json so I'm trying to figure out if we can use the token infrastructure with the CLI.

task list_containers does not return anything

The task list_containers does not list any containers:

$ echo "from dockerfabric import tasks as docker" > fabfile.py
$ fab -H 10.0.101.133 docker.list_containers
[10.0.101.133] Executing task 'docker.list_containers'
[10.0.101.133]
Id Names Image Command Ports Status Created

Done.

Note: list images is fine
$ fab -H 10.0.101.133 docker.list_images
[10.0.101.133] Executing task 'docker.list_images'
[10.0.101.133]
Id RepoTags ParentId Created VirtualSize Size
1183a43c2e94 stylight/ws-solr:latest 80ef0e18de7e 2015-02-27T12:39:25 1454511143 0
45bb9005020b stylight/wildfly:latest ec0bddcfa096 2015-02-11T09:24:48 1288633198 0

Done.

There IS a container runnning:
$ ssh 10.0.101.133
Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-46-generic x86_64)

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b6a28a641321 stylight/ws-solr:12 "/bin/sh -c '/usr/lo 2 days ago Up 8 hours 9999/tcp, 8787/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:9990->9990/tcp solr

_build_socat

This function was removed. Is it no longer necessary? Has it been replaced by something else? I honestly don't recall why I needed it, but I have a broken reference when I try to upgrade to the latest version of this package.

API Connection Issue

(To ensure I'm not running into bugs in the CLI code, I switched back to the API. I also rebuilt the virtualenv from scratch so I didn't have lingering code modifications.)

Now I'm seeing the following (sanitized):

DEBUG:dockermap.map.client:Passing kwargs to client actions: {}
DEBUG:dockermap.map.state.base:Following dependency path for graylog_map.nginx.
DEBUG:dockermap.map.state.base:Dependency path at graylog_map.graylog, instances [None].
DEBUG:docker.auth.auth:Trying paths: ['/home/<domain>/<user>/.docker/config.json', '/home/<domain>/<user>/.dockercfg']
DEBUG:docker.auth.auth:Found file at path: /home/<domain>/<user>/.docker/config.json
DEBUG:docker.auth.auth:Found 'auths' section
DEBUG:docker.auth.auth:Found entry (registry=u'https://index.docker.io/v1/', username=u'<username>')
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 127.0.0.1
DEBUG:paramiko.transport:[chan 19] Max packet in: 32768 bytes
ERROR:paramiko.transport:Secsh channel 19 open FAILED: Connection refused: Connect failed

As part of my testing, I opened another SSH session on the build machine (while the first was failed/frozen) and ran a successful call using the socat port:

$ docker -H localhost:22026 info
Containers: 5
 Running: 0
 Paused: 0
 Stopped: 5
...

I'm not even sure what to check next.

unknown flag: --host-config

After upgrading to python 3.7, docker[tls]==3.7.1, docker-map==1.0.0 and docker-fabric==0.5.0, I'm running into the error unknown flag: --host-config when trying to create a container. I'm having trouble even finding the host-config option so I'm not clear what's causing this:

a py3 issue (e.g. the wrong flag is created)
a feature removed from docker
an issue in docker-py

0.4.1: object has no attribute 'rsplit'

Upgraded. When I reached code to pull my repositories, I get the following:

DEBUG:docker.auth.auth:Found 'auths' section
DEBUG:docker.auth.auth:Found entry (registry=u'https://index.docker.io/v1/', username=u'<username>')
Traceback (most recent call last):
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/main.py", line 756, in main
    *args, **kwargs
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 426, in execute
    results['<local-only>'] = task.run(*args, **new_kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
    return self.wrapped(*args, **kwargs)
  File "/data/apex-server/build_deploy/src/fabfile.py", line 153, in full_setup
    setup_service_graylog(device_name=target, ldap=ldap)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 170, in __call__
    return self.run(*args, **kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/fabric/tasks.py", line 173, in run
    return self.wrapped(*args, **kwargs)
  File "/data/apex-server/build_deploy/src/fabfile.py", line 812, in setup_service_graylog
    pull_required_containers()
  File "/data/apex-server/build_deploy/src/fabfile.py", line 1957, in pull_required_containers
    docker_client.pull(env.docker_maps.containers[container].image)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockerfabric/apiclient.py", line 249, in pull
    insecure_registry=c_insecure, **kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/dockermap/client/base.py", line 184, in pull
    response = super(DockerClientWrapper, self).pull(repository, tag=tag, stream=stream, **kwargs)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/docker/api/image.py", line 152, in pull
    repository, tag = utils.parse_repository_tag(repository)
  File "/mnt/data/.virtualenvs/build_deploy/local/lib/python2.7/site-packages/docker/utils/utils.py", line 365, in parse_repository_tag
    parts = repo_name.rsplit('@', 1)
AttributeError: '_NotSet' object has no attribute 'rsplit'

Probably won't dig into the issue tonight, but wanted to at least get it reported.

sudo access to socket

I'd like to use sudo to access the docker socket rather than the docker group. I'm happy to contribute a patch but want to make sure it's (1) compatible with the way docker-fabric already does things and (2) something that would be accepted once it was working. Any hint on the right part of the architecture to explore would also be much appreciated.

We got to this point because we wanted to use our LDAP to control who has docker access. We already control sudoers using LDAP groups and this article was the "duh" moment that we could do the same for docker with a specialzed sudoers line (and make it all transparent to our users). As a HIPAA-compliant organization, the security benefits that motivated the original article made it a "must have" for us.

Unfortunately, docker-fabric fails in this configuration and we want to fix it.

Regex for CLI parser is too narrow

This has been reported in #8, and is created as a new issue for tracking when it is fixed:

Removing the docker.Client check in #10 (and not adding it to the CLI class) has resolved the -H issue. The next issue I ran into was the CREATED_AT_PATTERN regex. Specifically, my time component was:

2016-12-06 18:09:58 -0500 EST

The regex pattern doesn't like the - timezone and can be fixed as:

CREATED_AT_PATTERN = re.compile(r'(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2}) .\d{4} \w+')

I tried (+|-) but it creates an additional, invalid group. The . will match anything but this isn't too risky since the rest of the pattern has to work.

merll / docker-fabric Goto Github PK

docker-fabric's Issues

Recommend Projects

Recommend Topics

Recommend Org