gliderlabs / connectable Goto Github PK

I have containers launched that are:

• linked to backends container
• include a BACKEND_ env var

And I consistently cannot reach the backends.

from backends container logs I see:
backends_1 | 2015/03/23 03:19:57 omni: unable to list containers
backends_1 | 2015/03/23 03:19:57 No backends! Closing the connection

Am I doing it wrong?

using docker 1.5 on osx+virtualbox:
docker@dev:$ docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
docker@dev:$

Thanks!
Tyson

Hey,

I'm wanting to use the consul service catalog with registrator not the key value store because of health checking and DNS goodness.

So I would run ambassadord with BACKEND_8080=httptest.services.consul to route to a service.

The really cool thing about the configBackends is the .WatchToUpdate() method because it caches the lookups. The DNS is good for database connections but for lots of HTTP its better to have resolved routes in memory.

A useful thing would be an in memory cache that changes with the service catalog not the key/value store.

I'm thinking there could be a 4th type of backend - like a srvBackends but that has a .WatchToUpdate() method listening to /v1/catalog/service/<service>.

This way the same consul: registrator path is used for both but the user can choose between current srvBackends routing (for things like long-lived database connections) and cached service routing (for things like HTTP connections).

it could be: BACKEND_8080=/httptest.services.consul (or something similar) to denote that the service cache mode should be used

Do you think this is useful? I'm happy to make a pull request if you do :)

We ran into #13 while using the latest Image pulled from Dockerhub. It looks like the latest code didn't make it into stage/ambassadord yet.

I tried to build it myself but my lacking Go karate together with our corporate proxy foiled that attempt. :(

Would you mind building the binary? Or give me a short rundown on how to gather all dependencies for building it?

Can you provide more run examples? How can I use connectable w/o Resolvable?

I can get omni mode working with my ambassadord container named backends.

However if I change the name to something else, like "myservices", the --setup-iptables container fails (even when I change the name of the linked container to match "myservices").

Reverting my container name to "backends" makes the --setup-iptables container work again.

I took a look at the code and it is not obvious to me where the "backends" container name becomes important. Some other names work, like "ba", but not "router" or "routes".

I'm also using fig, so my container names are not even directly backends, but rather "mydir_backends_1".

Is there something I'm missing?
Thanks

After upgrading to docker 1.8.2, when the container is first created its NetworkMode is empty. Only after starting does it get its value. I'm assuming both this and #23 are due to the network system changes, not sure if this is a docker bug, or if we should check for "" NetworkMode. In my fork, i changed connectable to detect "start" instead of "create" which fixes this bug, and has the advantage of working through container restarts.

When I use the wrong URI for etcd, get lots of these over and over and over again, and ambassadord can't connect to anything else.

Jul 25 21:24:47 core-01 bash[572]: 2014/07/25 21:24:46 configstore:
Jul 25 21:24:47 core-01 bash[572]: 2014/07/25 21:24:46 etcd: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]

When it's correct, this doesn't happen.

Been using docker 1.8 (1.8.1 but I believe this is a 1.8 change) and I had to change connectable to detect NetworkMode=("bridge" || "default") because of the new networking layer. Are you interested in these changes? Since docker 1.8 is now a stable release, I'd reckon supporting it is a priority.

Hi.

First, thanks for the great work.

I failed using ambassadord with Consul and registrator. I guess it is not an issue, and I did something wrong, but I cannot find in the documentation why it fails. So this is a call for help, or a full step by step tutorial ;)

Here are the steps, on a single host

Start Consul
sudo docker run -d -h consul-master -v /mnt:/data -p 8300:8300 -p 8301:8301 -p 8301:8301/udp -p 8302:8302 -p 8302:8302/udp -p 8400:8400 -p 8500:8500 -p 8600:53/udp progrium/consul -server -bootstrap

Start registrator
sudo docker run -d -v /var/run/docker.sock:/tmp/docker.sock -h $HOSTNAME progrium/registrator consul://54.77.108.120:8500

Start Redis
sudo docker run -d -p 6379 --name redis redis

Registrator makes the new redis service available in Consul.
curl http://127.0.0.1:8500/v1/catalog/service/redis

[{"Node":"consul-master","Address":"172.17.1.14","ServiceID":"ip-172-30-0-58:redis:6379","ServiceName":"redis","ServiceTags":null,"ServicePort":49157}]

Start ambassadord in point to point mode using SRV
sudo docker run --name amb -d progrium/ambassadord redis.service.consul

Start a redis-client, using docker links
sudo docker run --link amb:db redis redis-cli -h db -p 10000 ping

Error: Connection reset by peer

Checking redis is working
sudo docker run --link redis:db redis redis-cli -h db ping

PONG

Gathering the logs from ambassadord
sudo docker logs amb
2014/09/18 12:11:23 Ambassadord listening on 10000 using redis.service.consul ...
2014/09/18 12:12:29 dns: lookup redis.service.consul: no such host
2014/09/18 12:12:29 No backends! Closing the connection

Same goes for omnimod.

Thansk for your help

I've been working on building out a CoreOS cluster using ambassadord, and I ran into an error connecting to an etcd backend.

On the command:

docker pull progrium/ambassadord
docker run --name my-app-amb.1 progrium/ambassadord etcd://127.0.0.1:4001/services/my-app

I get the error:

2014/08/26 15:35:04 unrecognized config store backend: etcd

Looking in this repository, I see a few things that lead me to believe the build on docker hub is out of date:

1. etcd backends were added on July 8
2. The latest staged ambassadord was commited on June 30
3. The Dockerfile sources its executable from stage/

All of this has me thinking perhaps stage/ needs to be updated via make to refresh the bits on Docker hub. Does this seem right?

EDIT: Additonally, the hub page lists June 30 as its latest date.

connectable is presently failing during the iptables command:

$ docker logs connectable
2015/08/07 18:28:50 Connectable listening on 10000 ...
Self: 2011f14d0013ed337acf696b3def8737b45eb7d787b4de79e34b1294b8781e9d bridge
2015/08/07 18:28:51 non-zero exit: 1
$ docker logs thirsty_mcclintock
2015/08/07 18:28:51 listen tcp :10000: bind: address already in use

Given the following fig.yml:

app:
  build: .
  command: python run.py
  ports:
   - "5000"
  environment: 
    SERVICE_NAME: app

consul:
  image: progrium/consul
  ports:
    - "8400:8400"
    - "8500:8500"
    - "8600:53/udp"
  hostname: node1
  command: -server -bootstrap

registrator:
  image: progrium/registrator
  command: "consul://localdocker:8500"
  volumes:
    - /var/run/:/tmp/
  environment:
    SERVICE_NAME: registrator

ambassadord:
  image: progrium/ambassadord
  ports:
    - "80:10000"
  command: "app.service.consul"

I get the following problem with ambassadord when accessing http://localdocker/:

fig up ambassadord
Creating samantha_ambassadord_1...
Attaching to samantha_ambassadord_1
ambassadord_1 | 2014/10/08 23:37:11 Ambassadord listening on 10000 using app.service.consul ...
ambassadord_1 | 2014/10/08 23:37:15 dns: lookup app.service.consul on [10.0.2.3]:53: server misbehaving
ambassadord_1 | 2014/10/08 23:37:15 No backends! Closing the connection

Any clue would be excellent.

Hi,

I am trying to use this sidecar container in an architecture with Amazon EC2 Container Service (ECS). When I try to configure an ECS task definition with the label format connect[8081]=stock-price.service.consul it gives the following error message:

Docker label key connect[8081] contains invalid characters, does not match pattern ^[a-z0-9-.]+$

According to the docker spec the ECS service is correct in checking that the labels contain only lower-cased alphanumeric characters, dots and dashes (see: https://docs.docker.com/engine/userguide/labels-custom-metadata/). It seems however that docker engine is not strict in enforcing this.

Is there any chance this can be changed to support valid Docker tag as per the spec?

Im setting up fleet services in CoreOS. I'm using registrator + ambassadord to set up service discovery across multiple hosts.

From what I can tell I'm stuck trying to get a container to recognize whats in etcd.

Starting Registrator
docker run --rm --name registrator -v /var/run/docker.sock:/tmp/docker.sock progrium/registrator -ip=${COREOS_PRIVATE_IPV4} etcd://${COREOS_PRIVATE_IPV4}:4001/services

Starting Ambassadord
docker run --rm --name backends -v /var/run/docker.sock:/var/run/docker.sock progrium/ambassadord --omnimode

Starting Redis
docker run --name redis -p 6379:6379 -e SERVICE_NAME=redis dockerfile/redis

etcd status after registrator
$ etcdctl get /services/redis/core-01:redis:6379
$ 172.17.8.102:6739

Trying to PING the Redis container on a different host
docker run -it --rm --link backends:redis -e BACKEND_6379="etcd://${COREOS_PRIVATE_IPV4}/services/redis" dockerfile/redis bash -c 'redis-cli ping'

Result
Could not connect to Redis at 127.0.0.1:6379: Connection refused

The last docker run errors because its trying to connect to redis on localhost. Am I missing something? Do I have to build a container that supports ambassadord?

There is a memory leak in this app, I haven't been able to figure out where yet, but after several hours of operations at 10s of thousands if not 100s of thousands of connections, connectable just continues to consume memory.

I'm trying to track it down.

Started ambassadord in omni mode:

/usr/bin/docker run --name backends -v /var/run/docker.sock:/var/run/docker.sock progrium/ambassadord --omnimode
/usr/bin/docker run --rm=true --privileged --net container:backends progrium/ambassadord --setup-iptables

Then I tried to connect to something:

$ docker run --rm -it --link backends:backends -e BACKEND_4161="etcd://127.0.0.1:4001/services/foo-4161" busybox

/ # nc $BACKENDS_PORT_10000_TCP_ADDR 4161

Then I get this error from ambassadord, and the ambassador container exits:

2014/07/25 06:15:51 unrecognized config store backend: etcd

This is using the current docker image pushed to the registry. Maybe it's out of date? I'll try building it tomorrow.

For backends like etcd, typing in the backend name over and over again is tiresome and errorprone. Additionally, why should the container have to specify the servername?

So, if the backend environment variable starts with etcd:// or consul/etc, then it does the same thing that it does now. Otherwise, if its just a raw string, and --default-backend is set, then it appends the env variable to the key and does the lookup that way.

Thank you for a such a useful container. This is more of a how-to question. I have connected my mysql container to the ambassadord container through the ominmode option, with registrator and consul

docker run -v /var/run/docker.sock:/var/run/docker.sock --name backends -d progrium/ambassadord --omnimode
docker run --rm --privileged --net container:backends progrium/ambassadord --setup-iptables
docker run -p 9990:3306 -e SERVICE_NAME=mysql -e BACKEND_3306=mysql.service.consul -e MYSQL_USER=admin -e MYSQL_PASS=PassworD1 --link backends:backends -d tutum/mysql:5.6

Assuming the Private IP address for mysql host is 172.20.20.10. I have a 3-cluster consul servers, at 172.20.20.10,172.20.20.11, and 172.20.20.12.

How do I connect to the mysql instance from another host 172.20.20.11? I understand that I will be running another instance of ambassadord that connects to mysql amb or would I connecting to the consul server? Can you help me with the actual command that I should run in the other host to link to the mysql container. I am running the following, but it's not giving me the correct env variables set at the other host.

docker run --expose 3306 -e MYSQL_PORT_3306_TCP=tcp://172.20.20.10:3306 --name amb -d progrium/ambassadord --links

I wished there was a way to just register the external mysql host through consul services, something like:

docker run --expose 3306 -e backend_3306=mysql.service.consul --name amb -d progrium/ambassadord --links

Containers

• consul-server
• connectable (as shown in the documentation)
• redis-server

When I try and jump into a simple ubuntu container so I can verify the "connection" is happening, connectable always throws the error that it is unable to join a network because the container is not running.

docker run -it --rm -l connect[6379]=redis.service.consul ubuntu /bin/bash

I assume I am missing something, but cannot seem to figure it out yet.

Docker images from gliderlabs/connectable is broken .

Has to point to gliderlabs/connectable:master

Cheers

Kev

I think I must be missing something here. I thought ambassadord would help communicate across multiple (coreos) hosts.

So far I have the following coreos services setup:

https://gist.github.com/jeffutter/422ce28865aff09464a1

With this setup I can do the following on either host and it works:

core@core-02 ~ $ docker run --rm --link backends:backends -e "BACKEND_6379=etcd://172.17.8.101:4001/services/redis/redis" -ti redis:2.8 bash -c 'redis-cli -h backends ping'
PONG

However, the following also works, removing the utility of ambassador all-together:

core@core-02 ~ $ docker run --rm -ti redis:2.8 redis-cli -h $(etcdctl get services/redis/redis | cut -d ':' -f 1) ping
PONG

I don't see the purpose of ambassador in this setup. I assumed ambassador might work in a fashion where my docker containers could talk to ambassador via the internal docker network. Containers on the the same host would hit the local ambassador and be routed to other containers on the same host as needed. Containers on remote hosts would hit their local ambassador, which would forward to the remote ambassador (port 10000) and then forward on to the internal docker networks. Am I missing something as to how this is supposed to work?

Hi

I have two CoreOS clusters, one on Auzre and one on AWS.
They both run registrator, skydns. I am running a SDN (Weave).
I would like to be able to transparently discover services across clouds.

My idea is to use Ambassadord in omni mode, on each cloud, which has an IP on the SDN and can communicate with the Ambassadors on other clouds (and possibly DNS)

See diagram:https://dl.dropboxusercontent.com/u/31838796/MultiCloud-Networking%20and%20Discovery.svg

Is it possible to achieve this with Ambassadord?
What are your thoughts on the subject?

Since you already have the docker socket, you can do this. See moby/moby#7228 (comment).

This would make using ambassadord much easier. If you don't have the cycles to do this, I can take a look at it this weekend, perhaps this is a good time for me to finally learn go.

I wonder if this is the right setup for consul connectable service.

Host/VM 1 : Consul server
Host/VM 2 : Consul Agent (client mode) , registrator and connectable and All micro-services

So when I started my micro-services containers, I could see them in consul server.

Question :
If I have two micro-services.... how do I load balance them ?

What API (rest API) should I call to get the services in a round robin fashion ?

How does connetable helps ?

Since connectable is sitting on each host ...does it do roundrobin of micro-services distributed across the hosts ?

Praveen

Does ambassadord expose any critical information to the containers linking to it?

If I run docker run -d --name some-mongo --expose 27017 progrium/ambassadord mongo-host:27017
port 10000 is proxied to mongo-host:27017, but port 27017 of this container is not.

README.md says "Ambassador to fixed/resolved backend(s) using domain or IP and port. Listens on same port as destination."