crosbymichael / skydock Goto Github PK
View Code? Open in Web Editor NEWService discovery via DNS for docker
License: MIT License
Service discovery via DNS for docker
License: MIT License
Hello everyone, first of all, thanks for creating this awesome tool.
Let me explain the experiment I'm trying to accomplish; basically I want to spin up a set of services instances under different environments (with some sed'ing to create specific conf files out of some templates), let's say I run the skydock instances as follows:
docker run -d -v /var/run/docker.sock:/docker.sock --name skydock_qa --link skydns:skydns crosbymichael/skydock -ttl 30 -environment prod -s /docker.sock -domain company.local
docker run -d -v /var/run/docker.sock:/docker.sock --name skydock_uat --link skydns:skydns crosbymichael/skydock -ttl 30 -environment uat -s /docker.sock -domain company.local
What I have found (docker ps) is that only the second instance is active, while the first one is stopped, causing the services instances that attempt to reach each other using the "qa" env. to not be able to communicate between them.
Is this the intended behavior?
Should I approach it differently?
First I fired up those two instances.
docker run -d -p 8080:8080 -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain docker
docker run -d -v /var/run/docker.sock:/docker.sock --name skydock crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain docker -name skydns
And this setup is working as expected.
Now I want to spawn another production environment. This time I only fired another skydock container with the env production as follows.
docker run -d -v /var/run/docker.sock:/docker.sock --name skydock-prod crosbymichael/skydock -ttl 30 -environment prod -s /docker.sock -domain docker -name skydns
Querying the api doesn't show the production skydoc.
curl $(docker-ip):8080/skydns/services/
And now I am wondering on how to setup the production version of skydock.
Do I have to run in separate docker-host?
If I fire up in the same docker host, in which DNS url entry will the new containers be available?
Do I have to pass some flags/variables when I fire new containers to be available in the production env?
Hi,
I'm playing with Skydock and it looks good! Do you have planned to support alternative host -value?
For example: now it reports 172.17.0.XX as a host to Skydns, but I would like to use public ip address as a host value. What do you think?
I have just set this up for a development environment and have ran into a problem after 20 hours.
I grabbed specific logs for the skydock service and it shows a stop and die message for the not running redis instance. restarting redis instance fixes the issue.
skydock is not usable in this condition.
Somehow instances are failing TTL and not being put back into DNS.
First: Great work with SkyDock! 👍 I think service discovery via DNS is the way to go.
Second: Here is what I like to do:
$ docker run -d -p 8090:80 mattes/hello-world-nginx
# this works:
# from within docker container
$ docker run -it ubuntu /bin/bash
$ apt-get update && apt-get install —y curl
$ curl hello-world-nginx.dev.docker
Hello world!
# this works:
# from within boot2docker
$ boot2docker ssh
sudo vi /etc/resolv.conf
nameserver 172.17.42.1
$ curl hello-world-nginx.dev.docker
Hello world!
# from my mac (this won't work)
$ boot2docker ip
192.168.59.103
$ sudo sh -c "echo 'nameserver 192.168.59.103' >> /etc/resolver/docker"
# more info about /etc/resolver under mac: https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man5/resolver.5.html
$ curl hello-world-nginx.dev.docker
(hangs)
$ dig @192.168.59.103 +short hello-world-nginx.dev.docker
172.17.0.32
So the problem here is that hello-world-nginx.dev.docker
resolves to 172.17.0.32
. Which is actually correct, but makes no sense from the mac side.
Side note: I started skydock with docker run -p 53:53/udp ...
. So no binding to -p 172.17.42.1:53:53/udp
.
Any ideas how to make this work?
When adding images tagged with the following format, no DNS entry is added.
myregistry.mycompany.com/myrepository/imagename:tag
Hi, I was wondering whether SkyDns will also do the load-balancing like HA-Proxy?
Or do I still need to add HA-proxy layer?
I stopped my redis1, redis2 and redis3 containers. Also rm
ed the names.
The DNS entries are still appearing.
$ dig @172.17.42.1 redis.dev.docker
; <<>> DiG 9.8.1-P1 <<>> @172.17.42.1 redis.dev.docker
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63974
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;redis.dev.docker. IN A
;; ANSWER SECTION:
redis.dev.docker. 16 IN A 172.17.0.4
redis.dev.docker. 7 IN A 172.17.0.5
redis.dev.docker. 10 IN A 172.17.0.6
I have followed all instructions on the readme, but i cannot seem to reach the containers by hostname is there anyway to debug it where it is failing?
The ip is added on resolv.conf in the container and from there it does seem to work
I don't know if this is still an issue with the latest code.
But I received this error when using 60 sec TTL, no issue when reverting back to 30 sec TTL
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x40175c]
goroutine 8 [running]:
main.heartbeat(0xc200136d20, 0xa)
/home/ubuntu/docker/builder/tmp/skydock/main.go:126 +0x2fc
created by main.sendService
/home/ubuntu/docker/builder/tmp/skydock/main.go:179 +0x2b8
goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc200162068)
/usr/lib/go/src/pkg/runtime/zsema_linux_amd64.c:165 +0x2e
sync.(*WaitGroup).Wait(0xc200112180)
/usr/lib/go/src/pkg/sync/waitgroup.go:109 +0xf2
main.main()
/home/ubuntu/docker/builder/tmp/skydock/main.go:281 +0x5c1
goroutine 2 [syscall]:
goroutine 4 [syscall]:
os/signal.loop()
/usr/lib/go/src/pkg/os/signal/signal_unix.go:21 +0x1c
created by os/signal.init·1
/usr/lib/go/src/pkg/os/signal/signal_unix.go:27 +0x2f
goroutine 13 [chan receive]:
main.heartbeat(0xc200136dc0, 0xa)
/home/ubuntu/docker/builder/tmp/skydock/main.go:116 +0x18e
created by main.sendService
/home/ubuntu/docker/builder/tmp/skydock/main.go:179 +0x2b8
goroutine 14 [IO wait]:
net.runtime_pollWait(0x7f94aad93e60, 0x72, 0x0)
/usr/lib/go/src/pkg/runtime/znetpoll_linux_amd64.c:118 +0x82
net.(*pollDesc).WaitRead(0xc2001be3e0, 0xb, 0xc200111930)
/usr/lib/go/src/pkg/net/fd_poll_runtime.go:75 +0x31
net.(*netFD).Read(0xc2001be360, 0xc20016a000, 0x1000, 0x1000, 0x0, ...)
/usr/lib/go/src/pkg/net/fd_unix.go:195 +0x2b3
net.(*conn).Read(0xc2001620a0, 0xc20016a000, 0x1000, 0x1000, 0x0, ...)
/usr/lib/go/src/pkg/net/net.go:123 +0xc3
bufio.(*Reader).fill(0xc2001b6ae0)
/usr/lib/go/src/pkg/bufio/bufio.go:79 +0x10c
bufio.(*Reader).ReadSlice(0xc2001b6ae0, 0xa, 0x0, 0x0, 0x0, ...)
/usr/lib/go/src/pkg/bufio/bufio.go:262 +0x202
net/http.readLine(0xc2001b6ae0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/lib/go/src/pkg/net/http/chunked.go:91 +0x57
net/http.(*chunkedReader).beginChunk(0xc2001bd390)
/usr/lib/go/src/pkg/net/http/chunked.go:47 +0x28
net/http.(*chunkedReader).Read(0xc2001bd390, 0xc20017be00, 0x200, 0x200, 0x200, ...)
/usr/lib/go/src/pkg/net/http/chunked.go:65 +0x8f
net/http.(*body).Read(0xc2001bc7c0, 0xc20017be00, 0x200, 0x200, 0x0, ...)
/usr/lib/go/src/pkg/net/http/transfer.go:535 +0x94
encoding/json.(*Decoder).readValue(0xc2001b5540, 0xc2000b90a0, 0x7f94aad50010, 0x420853)
/usr/lib/go/src/pkg/encoding/json/stream.go:124 +0x46a
encoding/json.(*Decoder).Decode(0xc2001b5540, 0x6c3700, 0xc2001620c8, 0xc200167570, 0xc2001bc7c0, ...)
/usr/lib/go/src/pkg/encoding/json/stream.go:44 +0x60
github.com/crosbymichael/skydock/docker.func·002()
/home/ubuntu/go/src/github.com/crosbymichael/skydock/docker/client.go:186 +0x607
created by github.com/crosbymichael/skydock/docker.(*dockerClient).GetEvents
/home/ubuntu/go/src/github.com/crosbymichael/skydock/docker/client.go:196 +0xa8
goroutine 15 [chan receive]:
main.eventHandler(0xc2001ccc00, 0xc200112180)
/home/ubuntu/docker/builder/tmp/skydock/main.go:217 +0x6f
created by main.main
/home/ubuntu/docker/builder/tmp/skydock/main.go:277 +0x55f
goroutine 16 [chan receive]:
main.eventHandler(0xc2001ccc00, 0xc200112180)
/home/ubuntu/docker/builder/tmp/skydock/main.go:217 +0x6f
created by main.main
/home/ubuntu/docker/builder/tmp/skydock/main.go:277 +0x55f
goroutine 17 [chan receive]:
main.eventHandler(0xc2001ccc00, 0xc200112180)
/home/ubuntu/docker/builder/tmp/skydock/main.go:217 +0x6f
created by main.main
/home/ubuntu/docker/builder/tmp/skydock/main.go:277 +0x55f
goroutine 18 [chan receive]:
github.com/crosbymichael/skydock/docker.func·001()
/home/ubuntu/go/src/github.com/crosbymichael/skydock/docker/client.go:174 +0x54
created by github.com/crosbymichael/skydock/docker.func·002
/home/ubuntu/go/src/github.com/crosbymichael/skydock/docker/client.go:181 +0x551
If you tag the same commit with 2 differents image's name, when you start the containers with image A, docker may register image B un the Config section. It's not a problem except for skydock who thinks that the image is not tagged and returne a ErrImageNotTagged error.
$ docker images | grep d3b7d9f39158
foo-mailcatcher latest d3b7d9f39158 5 days ago 381.3 MB
bar-mailcatcher latest d3b7d9f39158 5 days ago 381.3 MB
$ docker run -d --name foo foo-mailcatcher
b9c81f106ad0509edc15d34305bb26ba766a465462d4a3a0f20f0f75e239e069
$ docker run -d --name bar bar-mailcatcher
80a7fe67d59d92b88d2121a75ce3b4e3bb9c19105e2f167bfaa79ceb288ca78a
$ docker inspect --format '{{ .Config.Image }}' foo
bar-mailcatcher
$ docker inspect --format '{{ .Config.Image }}' bar
bar-mailcatcher
The first image will raise a ErrImageNotTagged in skydns
[info] 1399526496 skydock: adding f6647e8847 (consulserver) to skydns
[error] 1399526496 skydock: error adding f6647e8847 to skydns: Invalid HTTP response
On ubuntu 12.4. Anything else you want to know, I'd be happy to try and find it.
When using a cluster of SkyDNS servers, the server used for the -http
parameter is also used for redirecting API requests on followers to the master.
This means that any API request performed on a follower will most likely fail, since skydock will not find the master.
When running a VPS and maxing out RAM and SWAP it might eventually cause skydns+skydock to stop resolving running containers. A manual docker restart skydock
will get the services back into skydns and everything resolves happily again after that.
Create a way for skydock+skydns stack to gracefully recover after being temporarily strangled.
My production box in this case is a 2x CPU, 4GB RAM, 4GB SWAP, Digital Ocean VPS running an apache+php container, a mysql container, a skydock container, a skydns container, and nginx on the host.
I noticed problems right at the end of the month (as site usage peaks) when I started getting 500 errors which required a manual docker restart skydock
to get skydns resolving properly again.
Correlating the logs and metrics led me to observe EOF in the skydock logs right when the 500 errors started with graphs showing high system load.
[error] 1399064111 skydock: EOF
[error] 1399064111 skydock: EOF
[error] 1399064111 skydock: EOF
[error] 1399064111 skydock: EOF
The best reference I could find was from docker irc chat where a rather overloaded VPS system threw the same errors.
The error seems like it is coming from heartbeat() -> updateService(). So perhaps the connection between skydock and skydns craps out allowing the services to time out?
I was able to repeat it two out of two tries on a fresh VPS test install with the latest docker/skydock/skydns stack. Increasing the TTL from 30 up to 300 made the system stay up longer, but eventually it hung long enough to crap out.
Stress testing provided by by starting multiple gitlab containers: thanks Ruby! :)
Steps to reproduce Skydock EOF and loss of registered services from skydns causing failure to resolve container names.
Digital Ocean Droplet
1x CPU
~512MB RAM
~512MB Swap
Ubuntu 12.04.4 LTS (GNU/Linux 3.8.0-29-generic x86_64)
Linux skydock-test 3.8.0-29-generic #42~precise1-Ubuntu SMP Wed Aug 14 16:19:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Installed cgroup-lite and lxc-docker from deb https://get.docker.io/ubuntu docker main
Docker Version:
Client version: 0.10.0
Client API version: 1.10
Go version (client): go1.2.1
Git commit (client): dc9c28f
Server version: 0.10.0
Server API version: 1.10
Git commit (server): dc9c28f
Go version (server): go1.2.1
Last stable version: 0.10.0
/etc/init/docker.conf -- DOCKER_OPTS="-dns 172.17.42.1 -bip 172.17.42.1/16"
ulimit -a
ulimit -a
as root:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 3781
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 3781
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$ docker pull crosbymichael/skydns
$ docker pull crosbymichael/skydock
$ docker pull crosbymichael/redis
$ docker pull sameersbn/gitlab:latest
$ docker run -d -p 172.17.42.1:53:53/udp --name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain docker
$ docker run -d -v /var/run/docker.sock:/docker.sock --name skydock crosbymichael/skydock -ttl 10 -environment dev -s /docker.sock -domain docker -name skydns
$ docker run -d --name redis1 crosbymichael/redis
$ docker run -d --name redis2 crosbymichael/redis
$ sudo sed -i '1s/^/nameserver 172.17.42.1\n/' /etc/resolv.conf
Everything starts out resolving fine.
$ ping redis.dev.docker
$ ping redis1.redis.dev.docker
$ ping redis2.redis.dev.docker
$ ping skydock.dev.docker
$ ping skydns.dev.docker
Keep spinning up gitlab containers to thrash the machine until DNS breaks.
$ docker run -d sameersbn/gitlab:latest
...
$ docker run -d sameersbn/gitlab:latest
[debug] 1399049971 skydock: received event (die) 5226e81f50219a1928f10bd50a9184d54f4b2c2d1e2aaf6503e2ec12c36e0c8f sameersbn/gitlab:latest
[info] 1399049971 skydock: removing 5226e81f50 from skydns
[error] 1399050033 skydock: read tcp 172.17.0.2:8080: connection reset by peer
[error] 1399050034 skydock: EOF
[error] 1399050034 skydock: EOF
[error] 1399050041 skydock: EOF
[debug] 1399050069 skydock: received event (die) 84e06d358662703ddadb7f1835ec7cfce0c317ece05888d625abfc36140b9d90 sameersbn/gitlab:latest
[info] 1399050069 skydock: removing 84e06d3586 from skydns
[debug] 1399050339 skydock: received event (die) 45ad35fad38dad6c56eda5513113fe70dcbff3afa32c61d4a807bdb8924e5f0f sameersbn/gitlab:latest
[info] 1399050339 skydock: removing 45ad35fad3 from skydns
2014/05/02 16:58:53 Received DNS Request for "redis1.redis.dev.docker." from "172.17.42.1:48430"
2014/05/02 16:58:59 Received DNS Request for "redis1.redis.dev.docker." from "172.17.42.1:56676"
2014/05/02 16:58:59 Calling 0 callback(s) for service 84e06d3586
2014/05/02 16:58:59 Removed Service: 84e06d3586
2014/05/02 16:58:59 Updated Service TTL: 87a190cf50 10
2014/05/02 16:58:59 Updated Service TTL: 4a52804fb8 10
2014/05/02 16:58:59 Updated Service TTL: ccb9859838 10
2014/05/02 16:58:59 Updated Service TTL: 8c65d2828b 10
2014/05/02 16:58:59 Updated Service TTL: 44f871be83 10
2014/05/02 16:58:59 Updated Service TTL: 5226e81f50 10
2014/05/02 16:58:59 Updated Service TTL: 1b16f8703a 10
2014/05/02 16:58:59 Calling 0 callback(s) for service 1b16f8703a
2014/05/02 16:58:59 Removed Service: 1b16f8703a
2014/05/02 16:58:59 Updated Service TTL: 45ad35fad3 10
2014/05/02 16:58:59 Calling 0 callback(s) for service 44f871be83
2014/05/02 16:58:59 Removed Service: 44f871be83
2014/05/02 16:58:59 Calling 0 callback(s) for service 5226e81f50
2014/05/02 16:58:59 Removed Service: 5226e81f50
2014/05/02 16:58:59 Calling 0 callback(s) for service 9233663fd1
2014/05/02 16:58:59 Removed Service: 9233663fd1
2014/05/02 16:59:00 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:44552"
2014/05/02 16:59:01 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 16:59:01 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:47767"
2014/05/02 16:59:01 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 16:59:02 Updated Service TTL: 45ad35fad3 10
2014/05/02 16:59:02 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:55471"
2014/05/02 16:59:02 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 16:59:02 Updated Service TTL: 87a190cf50 10
2014/05/02 16:59:03 Updated Service TTL: 4a52804fb8 10
2014/05/02 16:59:03 Updated Service TTL: ccb9859838 10
2014/05/02 16:59:03 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:59246"
2014/05/02 16:59:03 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 16:59:04 Updated Service TTL: 8c65d2828b 10
2014/05/02 16:59:06 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:41331"
2014/05/02 16:59:06 Error: Service does not exist in registry
2014/05/02 16:59:06 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:55604"
2014/05/02 16:59:06 Error: Service does not exist in registry
2014/05/02 16:59:08 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:52937"
2014/05/02 16:59:08 Error: Service does not exist in registry
2014/05/02 16:59:08 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:47870"
2014/05/02 16:59:08 Error: Service does not exist in registry
2014/05/02 16:59:10 Updated Service TTL: 45ad35fad3 10
2014/05/02 16:59:10 Updated Service TTL: 87a190cf50 10
2014/05/02 16:59:11 Updated Service TTL: 4a52804fb8 10
2014/05/02 16:59:11 Updated Service TTL: ccb9859838 10
2014/05/02 16:59:12 Updated Service TTL: 8c65d2828b 10
2014/05/02 16:59:14 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:47845"
2014/05/02 16:59:14 Error: Service does not exist in registry
2014/05/02 16:59:14 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:44293"
2014/05/02 16:59:14 Error: Service does not exist in registry
2014/05/02 16:59:18 Updated Service TTL: 45ad35fad3 10
2014/05/02 16:59:18 Updated Service TTL: 87a190cf50 10
2014/05/02 16:59:19 Updated Service TTL: ccb9859838 10
2014/05/02 16:59:19 Updated Service TTL: 4a52804fb8 10
2014/05/02 16:59:20 Updated Service TTL: 8c65d2828b 10
2014/05/02 16:59:30 Calling 0 callback(s) for service 4a52804fb8
2014/05/02 16:59:31 Removed Service: 4a52804fb8
2014/05/02 16:59:31 Updated Service TTL: 87a190cf50 10
2014/05/02 16:59:31 Updated Service TTL: ccb9859838 10
2014/05/02 16:59:31 Updated Service TTL: 8c65d2828b 10
2014/05/02 16:59:31 Calling 0 callback(s) for service ccb9859838
2014/05/02 16:59:31 Removed Service: ccb9859838
2014/05/02 16:59:31 Calling 0 callback(s) for service 8c65d2828b
2014/05/02 16:59:31 Removed Service: 8c65d2828b
2014/05/02 16:59:31 Updated Service TTL: 45ad35fad3 10
2014/05/02 16:59:31 Calling 0 callback(s) for service 45ad35fad3
2014/05/02 16:59:31 Removed Service: 45ad35fad3
2014/05/02 16:59:31 Calling 0 callback(s) for service 87a190cf50
2014/05/02 16:59:31 Removed Service: 87a190cf50
2014/05/02 16:59:35 Received DNS Request for "redis2.redis.dev.docker." from "172.17.42.1:55834"
2014/05/02 16:59:35 Error: Service does not exist in registry
$ docker stop skydns # nothing after a while hit <cntrl><c>
$ docker kill skydns # nothing, just hangs
2014/05/02 15:41:00 POST /v1.10/containers/87a190cf501c/stop?t=10
[/var/lib/docker|8d7c52cd] +job stop(87a190cf501c)
2014/05/02 15:41:10 Container 87a190cf501cdc2c1bd242be9b20ba56aeb2fa26bc0da481e11a23ab34d3a9a5 failed to exit within 10 seconds of SIGTERM - using the force
[/var/lib/docker|8d7c52cd] +job release_interface(87a190cf501cdc2c1bd242be9b20ba56aeb2fa26bc0da481e11a23ab34d3a9a5)
[/var/lib/docker|8d7c52cd] -job release_interface(87a190cf501cdc2c1bd242be9b20ba56aeb2fa26bc0da481e11a23ab34d3a9a5) = OK (0)
2014/05/02 16:14:48 POST /v1.10/containers/skydns/kill?signal=KILL
[/var/lib/docker|8d7c52cd] +job kill(skydns, KILL)
2014/05/02 16:14:58 Container ccb9859838ab failed to exit within 10 seconds of kill - trying direct SIGKILL
ps aux
root 3020 0.1 0.0 0 0 ? Zsl 12:23 0:20 [skydns] <defunct>
May be related to the kernel version
root@skydock-test:~# curl -X GET -L 172.17.0.2:8080/skydns/services/*
Service does not exist in registry
Same final result but it took more load and time before system hung long enough for problems to occur.
[debug] 1399064109 skydock: received event (create) 3c0272ebfc8df84d4250b9fbf6a3826e71291b674d46d913ae3fc3930b0e3651 sameersbn/gitlab:latest
[debug] 1399064109 skydock: received event (start) 3c0272ebfc8df84d4250b9fbf6a3826e71291b674d46d913ae3fc3930b0e3651 sameersbn/gitlab:latest
[info] 1399064109 skydock: adding 3c0272ebfc (gitlab) to skydns
[info] 1399064110 skydock: updating ttl for 42482f363d
[info] 1399064110 skydock: updating ttl for a8b74bf97f
[error] 1399064111 skydock: EOF
[error] 1399064111 skydock: EOF
[error] 1399064111 skydock: EOF
[error] 1399064111 skydock: EOF
2014/05/02 20:52:44 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 20:52:44 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:46400"
2014/05/02 20:52:44 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 20:52:51 Updated Service TTL: 2c1da75ec2 300
2014/05/02 20:52:54 Updated Service TTL: 894c4385c6 300
2014/05/02 20:53:07 Received DNS Request for "daisy.ubuntu.com." from "172.17.42.1:59028"
2014/05/02 20:53:08 Forwarded DNS Request "daisy.ubuntu.com." to "8.8.8.8:53"
2014/05/02 20:53:09 Received DNS Request for "daisy.ubuntu.com." from "172.17.42.1:45508"
2014/05/02 20:53:15 Error: Failure to Forward DNS Request "dial udp 8.8.8.8:53: i/o timeout"
2014/05/02 20:53:16 Received DNS Request for "redis1.redis.dev.docker." from "172.17.42.1:51719"
2014/05/02 20:55:08 Error: Failure to Forward DNS Request "dial udp 8.8.8.8:53: i/o timeout"
2014/05/02 20:55:08 Received DNS Request for "redis1.redis.dev.docker." from "172.17.42.1:45692"
2014/05/02 20:55:08 Received DNS Request for "redis1.redis.dev.docker." from "172.17.42.1:44059"
2014/05/02 20:55:09 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:39885"
2014/05/02 20:55:09 Received DNS Request for "daisy.ubuntu.com." from "172.17.42.1:38250"
2014/05/02 20:55:09 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 20:55:09 Forwarded DNS Request "daisy.ubuntu.com." to "8.8.8.8:53"
2014/05/02 20:55:09 Updated Service TTL: a978419b17 300
2014/05/02 20:55:09 Updated Service TTL: c20c19453e 300
2014/05/02 20:55:09 Updated Service TTL: fdda8dd6f2 300
2014/05/02 20:55:09 Updated Service TTL: 206af0ea2f 300
2014/05/02 20:55:09 Received DNS Request for "daisy.ubuntu.com." from "172.17.42.1:56460"
2014/05/02 20:55:09 Calling 0 callback(s) for service fdda8dd6f2
2014/05/02 20:55:09 Removed Service: fdda8dd6f2
2014/05/02 20:55:09 Added Service: {3c0272ebfc gitlab clever_fermi dev 172.17.0.7 22 300 2014-05-02 21:00:09.667536585 +0000 UTC map[]}
2014/05/02 20:55:09 Updated Service TTL: 8c1a3c129c 300
2014/05/02 20:55:09 Updated Service TTL: 198847fca4 300
2014/05/02 20:55:09 Updated Service TTL: 4b10f487ac 300
2014/05/02 20:55:09 Updated Service TTL: ca82643f15 300
2014/05/02 20:55:09 Updated Service TTL: b7dda2baa5 300
2014/05/02 20:55:09 Updated Service TTL: 4b59b2c0d3 300
2014/05/02 20:55:09 Updated Service TTL: 1c05e58b3c 300
2014/05/02 20:55:09 Updated Service TTL: 8b15f1d0b2 300
2014/05/02 20:55:09 Updated Service TTL: a7ae9c16b2 300
2014/05/02 20:55:09 Calling 0 callback(s) for service 2c1da75ec2
2014/05/02 20:55:09 Removed Service: 2c1da75ec2
2014/05/02 20:55:09 Forwarded DNS Request "daisy.ubuntu.com." to "8.8.8.8:53"
2014/05/02 20:55:09 Added Service: {de17559007 gitlab jolly_engelbart dev 172.17.0.21 22 300 2014-05-02 21:00:09.672444668 +0000 UTC map[]}
2014/05/02 20:55:09 Updated Service TTL: c0566e3579 300
2014/05/02 20:55:09 Calling 0 callback(s) for service 206af0ea2f
2014/05/02 20:55:09 Removed Service: 206af0ea2f
2014/05/02 20:55:09 Calling 0 callback(s) for service a978419b17
2014/05/02 20:55:09 Removed Service: a978419b17
2014/05/02 20:55:09 Calling 0 callback(s) for service c20c19453e
2014/05/02 20:55:09 Removed Service: c20c19453e
2014/05/02 20:55:09 Calling 0 callback(s) for service a7ae9c16b2
2014/05/02 20:55:09 Removed Service: a7ae9c16b2
2014/05/02 20:55:09 Calling 0 callback(s) for service c0566e3579
2014/05/02 20:55:09 Removed Service: c0566e3579
2014/05/02 20:55:09 Calling 0 callback(s) for service 8c1a3c129c
2014/05/02 20:55:09 Removed Service: 8c1a3c129c
2014/05/02 20:55:10 Received DNS Request for "daisy.ubuntu.com." from "172.17.42.1:57337"
2014/05/02 20:55:10 Received DNS Request for "4.0.17.172.in-addr.arpa." from "172.17.42.1:51288"
2014/05/02 20:55:10 Forwarded DNS Request "daisy.ubuntu.com." to "8.8.8.8:53"
2014/05/02 20:55:10 Forwarded DNS Request "4.0.17.172.in-addr.arpa." to "8.8.8.8:53"
2014/05/02 20:55:10 Updated Service TTL: 42482f363d 300
2014/05/02 20:55:10 Updated Service TTL: a8b74bf97f 300
Hi,
Is it possible to update the hostname of the container to reflect its DNS name?
I'm trying to setup a RabbitMQ cluster with a few containers, and they are very particular about hostnames.
Docker Version: 0.8.1
SkyDock version: Both the latest image from index, and the master branch tried.
SkyDNS version: latest image.
After a container's image ends up untagged (Primary reason, image being rebuilt), and then retagging it SkyDock does not seem to detect the new image name, I've tried the following:
The containers with retagged images:
4c6568f28cdc internal-old/nginx:latest /start 18 hours ago Up 13 minutes 80/tcp restore-test
8851c7056f5b internal-oldold/nginx:latest /start 18 hours ago Up 13 minutes 80/tcp satis-test
The log output from SkyDock on restart is
[info] 1394187898 skydock: adding 450666b2d4 (router) to skydns
[info] 1394187898 skydock: service already exists for 450666b2d4. Resetting ttl.
...
[info] 1394187898 skydock: adding f05a1bdd17 (mysql) to skydns
[info] 1394187898 skydock: service already exists for f05a1bdd17. Resetting ttl.
But no mention of nginx or the above containers.
Stop/starting the containers with retagged images:
[debug] 1394188038 skydock: received event (stop) 8851c7056f5b86fff433cb1ca041fa9888516bd4a0213bd8b350a5bf30b9706b internal-oldold/nginx:latest
[info] 1394188038 skydock: removing 8851c7056f from skydns
[debug] 1394188041 skydock: received event (start) 8851c7056f5b86fff433cb1ca041fa9888516bd4a0213bd8b350a5bf30b9706b internal-oldold/nginx:latest
But no mention of it being added or the name (nginx
) being detected.
Querying the SkyDNS server for any of these also returns no results.
Sorry for having a crappy subject line and combining two issues into one, but both fixes are needed to make skydock more usable in CoreOS, and they might be related...
I haven't found a way to start skydock and skydns in systemd reliably. I have skydock Requires=skydns.service
and After=skydns.service
but it looks like running docker start -a skydock
is not enough to guarantee the container is actually running. I get "Cannot link to a non running container". If I wait until the container is running (at least a few seconds after start -a skydns
) I get no problem linking. Maybe you have a better way of starting these services on boot. There is no treatment in the README.md. (Help!)
This is a new issue with docker 0.8.1, I think I read that the behavior of -link has changed: I get "failed to send 88a03c289d to skydns" in my journalctl logs. The full text with a few running containers:
[info] 1394224127 skydock: loading plugins from /plugins/default.js
[debug] 1394224127 skydock: starting restore of containers
[info] 1394224127 skydock: adding 0378034680 (skydock) to skydns
[error] 1394224127 skydock: failed to send 0378034680 to skydns on restore: Put http://:8080/skydns/services/0378034680: dial tcp :8080: connection refused
[info] 1394224127 skydock: adding 88a03c289d (ntp) to skydns
[error] 1394224127 skydock: failed to send 88a03c289d to skydns on restore: Put http://:8080/skydns/services/88a03c289d: dial tcp :8080: connection refused
[info] 1394224127 skydock: adding 47ba5e9b9b (skydns) to skydns
[error] 1394224127 skydock: failed to send 47ba5e9b9b to skydns on restore: Put http://:8080/skydns/services/47ba5e9b9b: dial tcp :8080: connection refused
[info] 1394224127 skydock: adding 763611feaf (registry) to skydns
[error] 1394224127 skydock: failed to send 763611feaf to skydns on restore: Put http://:8080/skydns/services/763611feaf: dial tcp :8080: connection refused
[debug] 1394224127 skydock: starting main process
It seems like skydock is not getting the address of skydns for some reason. I am not sure if the change in link semantics I heard about is responsible. This CoreOS release was only less than 24 hours ago, but Docker 0.8.1 has been out for a while, did you use it to test skydock already?
Hi,
I'm having trouble connecting to containers by hostname from the OS X side.
Is there anything I'm doing wrong here? Any help/ideas greatly appreciated!
Thx! Gijs
The following is what I know so far.
Running dig
from the OS X terminal replies as such:
$ dig elasticsearch.dev.docker.dev
; <<>> DiG 9.8.3-P1 <<>> elasticsearch.dev.docker.dev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23145
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;elasticsearch.dev.docker.dev. IN A
;; ANSWER SECTION:
elasticsearch.dev.docker.dev. 18 IN A 172.17.0.13
;; Query time: 0 msec
;; SERVER: 172.17.42.1#53(172.17.42.1)
;; WHEN: Mon Sep 8 13:02:02 2014
;; MSG SIZE rcvd: 90
And I can query the container using curl
using the IP address:
$ curl 172.17.0.13:9200
which returns:
{
"status" : 200,
"name" : "Ogre",
"version" : {
"number" : "1.2.2",
"build_hash" : "9902f08efc3ad14ce27882b991c4c56b920c9872",
"build_timestamp" : "2014-07-09T12:02:32Z",
"build_snapshot" : false,
"lucene_version" : "4.8"
},
"tagline" : "You Know, for Search"
}
But using curl
to query by domain...
$ curl http://elasticsearch.dev.docker.dev:9200
...gives:
curl: (6) Could not resolve host: elasticsearch.dev.docker.dev
Details:
172.17.0.3
and 172.17.42.1
alongside the DHCP-provided IP's to the DNS Servers in OS X's System Preferences under Network > Advanced
/etc/resolv.conf
:#
# Mac OS X Notice
#
# This file is not used by the host name and address resolution
# or the DNS query routing mechanisms used by most processes on
# this Mac OS X system.
#
# This file is automatically generated.
#
domain nens.local
nameserver 172.17.0.3
nameserver 172.17.42.1
nameserver 192.168.1.2
nameserver 192.168.1.4
$ sudo route -n add -net 172.17.0.0 10.2.0.10
$ docker logs skydns
:[debug] 1410171742 skydock: received event (start) 467159a7239d7314a8fa675f3ec781f30a6d4c69590c22c43e66b20f9a4e16fd gijs/elasticsearch:1.0
[info] 1410171742 skydock: adding 467159a723 (elasticsearch) to skydns
$ docker logs skydock
:2014/09/08 11:08:03 Received DNS Request for "elasticsearch.dev.docker.dev." from "10.2.0.1:61645"
$ curl -XGET http://172.17.0.10:8080/skydns/services/
gives:
[
{
"UUID": "8e04b49ff8",
"Name": "skydock",
"Version": "skydock",
"Environment": "dev",
"Region": "",
"Host": "172.17.0.11",
"Port": 80,
"TTL": 19,
"Expires": "2014-09-08T10:37:55.252560655Z"
},
{
"UUID": "3b24e79710",
"Name": "skydns",
"Version": "skydns",
"Environment": "dev",
"Region": "",
"Host": "172.17.0.10",
"Port": 53,
"TTL": 19,
"Expires": "2014-09-08T10:37:55.254102479Z"
},
{
"UUID": "08707bf365",
"Name": "pggis",
"Version": "pggis",
"Environment": "dev",
"Region": "",
"Host": "172.17.0.12",
"Port": 49156,
"TTL": 29,
"Expires": "2014-09-08T10:38:05.14786093Z"
},
{
"UUID": "467159a723",
"Name": "elasticsearch",
"Version": "elasticsearch",
"Environment": "dev",
"Region": "",
"Host": "172.17.0.13",
"Port": 49157,
"TTL": 14,
"Expires": "2014-09-08T10:37:49.989654756Z"
}
]
$ scutil --dns
:DNS configuration
resolver #1
search domain[0] : docker.dev
search domain[1] : dev
search domain[2] : nens.local
nameserver[0] : 172.17.0.3
nameserver[1] : 172.17.42.1
nameserver[2] : 192.168.1.2
nameserver[3] : 192.168.1.4
flags : Request A records
reach : Reachable
resolver #2
domain : local
options : mdns
timeout : 5
flags : Request A records
order : 300000
resolver #3
domain : 254.169.in-addr.arpa
options : mdns
timeout : 5
flags : Request A records
order : 300200
resolver #4
domain : 8.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records
order : 300400
resolver #5
domain : 9.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records
order : 300600
resolver #6
domain : a.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records
order : 300800
resolver #7
domain : b.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records
order : 301000
DNS configuration (for scoped queries)
resolver #1
search domain[0] : docker.dev
search domain[1] : dev
search domain[2] : nens.local
nameserver[0] : 172.17.0.3
nameserver[1] : 172.17.42.1
nameserver[2] : 192.168.1.2
nameserver[3] : 192.168.1.4
if_index : 4 (en0)
flags : Scoped, Request A records
reach : Reachable
2014/09/08 11:14:06 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
2014/09/08 11:14:07 Received DNS Request for "en.wikipedia.org." from "10.2.0.1:50015"
2014/09/08 11:14:07 Updated Service TTL: 467159a723 30
2014/09/08 11:14:08 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
2014/09/08 11:14:09 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
2014/09/08 11:14:10 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
2014/09/08 11:14:11 Error: Failure to Forward DNS Request "read udp 8.8.8.8:53: i/o timeout"
...however, the website did continue to work (wikipedia in this case)
I know that sykdns and skydock processes are running inside an LXC, but still, there is no need for them to really run as a root. Processes could drop to non-root user after opening sockets.
In your video at around 13:30, you use that skydnsctl
cli.
I couldn't find any such thing to install.
Can you make it clear on how can I get it and install?
I'm looking to balance load between containers of the same service. So let's say I have two api containers, api1 and api2. I would like to distribute load between these two containers.
What would be a good approach?
According to this there are quite a few considerations and client variations here. Does anyone know how skydns picks the order in which servers are listed?
I wouldn't want to wait until clients are unable to connect to api1 before moving some of the traffic to api2. An idea could be a -balance "[api,es,redis]" flag for skydock which would do a DELETE & PUT instead of PATCH when updating TTL. Depending on how skydns orders things.
Ideas? Thoughts? Am I missing something obvious?
This will help us filter when ppl run docker build
Sometimes, I don't want the image name to be used as the service name. Redis is a great example. I want to do this:
$ docker run -d --name redis-1 -e SERVICE_NAME=redis-cache redis
$ docker run -d --name redis-2 -e SERVICE_NAME=redis-cache redis
$ docker run -d --name redis-3 -e SERVICE_NAME=redis-cache redis
$
$ docker run -d --name redis-1 -e SERVICE_NAME=redis-session redis
$ docker run -d --name redis-2 -e SERVICE_NAME=redis-session redis
$ docker run -d --name redis-3 -e SERVICE_NAME=redis-session redis
$
$ host redis-cache.dev.skydns
redis-cache.dev.skydns has address 172.18.102.31
redis-cache.dev.skydns has address 172.18.102.30
redis-cache.dev.skydns has address 172.18.102.29
$
$ host redis-session.dev.skydns
redis-session.dev.skydns has address 172.18.102.18
redis-session.dev.skydns has address 172.18.102.14
redis-session.dev.skydns has address 172.18.102.6
Having to create new image tags for this seems like overkill.
This idea is stolen from: https://github.com/progrium/registrator
I've found that if I have containers just running latest, and then I update the image, thus untagging the image currently in use, Skydock does not add a DNS entry for it.
Not sure if this is solvable with Skydock or not outside of storing an image ID's tag for later.
I'm using upstart to manage my container services.
If I use upstart's restart command to restart the container most of the time I'll see the dns entry was removed from skydns but never added back in... at least not until I stop it again... wait... then start it back up.
Not sure if this would also be an issue with Docker's restart command.
if a container is started with a specific tag ( i had a container with a numeric tag 1.2.3), it looks like skydock doesnt catch the event..
If i start the same container without a tag (by giving it a tag - latest), it works fine..
Hi there,
I'm new to docker (and skydock obviously) and trying to get set up. In the README, you state the following:
First get the IP address of the docker0 bridge. We will need to know the IP of the bridge so that we can tell skydns to bind it's nameserver to that IP.
Is there a simple way to get the IP Address of the docker0
bridge? I can obviously spin up container and do ifconfig
, but wondering if docker had a command to get this information out so that in an upstart script it would be easy to run the skydns container on the right IP.
Thanks!
In particular, in my setup, it is running upstart, with which I am not familiar. Instructions for modifying either the Vagrantfile or the file system would be appreciated.
Running on a Mac, 10.9.1.
Vagrant 1.3.5
Virtual box 4.3.0 r89960
Using the docker image from the demo on the website. Hacked docker.conf until it didn't start on reboot & ran docker by hand with options from your example.
Sorry but I'm clueless about what's going on :/
$ docker run -d -v /var/run/docker.sock:/docker.sock -name skydock -link skydns:skydns crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain docker
0b18412a05ef15244e6d8f050dd71f8fb2b46c8ceb52165240c5303a28f97f1d
2014/01/29 23:51:18 Error: start: Cannot start container 0b18412a05ef15244e6d8f050dd71f8fb2b46c8ceb52165240c5303a28f97f1d: open /var/run/docker.sock: no such device or address
$ ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Jan 30 00:17 /var/run/docker.sock
$ docker -v
Docker version 0.7.1, build 88df052
$ docker images | grep crosbymichael
crosbymichael/skydock latest 2016b214a02c 7 days ago 137.2 MB
crosbymichael/skydns latest fbdbaca11389 9 days ago 139.6 MB
We're running our environment in a way that all our containers have already an unique name and thus the 'service' component in the domain name gets annoyingly in the way.
Could skydock offer a command line argument which registers the container without the service name component?
If my image came from a private registry, SkyDock don't add it to SkyDns
docker run -d --name apache my-private-registry.private.network/apache
[debug] 1415354852 skydock: received event (create) cd85709041e6e461b2565d1089ed26744ea7ede36076620cab892b875cc94a11 apache:latest
[debug] 1415354852 skydock: received event (start) cd85709041e6e461b2565d1089ed26744ea7ede36076620cab892b875cc94a11 apache:latest
docker pull my-private-registry.private.network/apache
docker run -d --name apache apache
[debug] 1415354888 skydock: received event (create) 22e4a95e1baada889a5f5e70bea40e2df250ba5e7e0233c60ab55ee2c8a967e8 apache:latest
[debug] 1415354888 skydock: received event (start) 22e4a95e1baada889a5f5e70bea40e2df250ba5e7e0233c60ab55ee2c8a967e8 apache:latest
[info] 1415354888 skydock: adding 22e4a95e1b (apache) to skydns
After thinking some more about multiple host support, I don't think there is an elegant DNS solution to the problem.
One idea I have is to allow listening on multiple docker hosts, e.g. supporting multiple -s parameters, and leaving the networking between the hosts to some other mechanism.
This would keep skydock simple, focused and clean.
A setup similar to http://goldmann.pl/blog/2014/01/21/connecting-docker-containers-on-multiple-hosts/ might work...? Or any other setup. The more flexible skydock would not make assumptions about it and simply get out of it's way.
Thoughts?
For some reason the SkyDNS domain names don't resolve when using anything but nslookup/dig/ping, any idea why? See output below:
[root@docker opt]# ping skydns.dev.dockerdev
PING skydns.dev.dockerdev (172.17.0.7) 56(84) bytes of data.
64 bytes from 172.17.0.7: icmp_seq=1 ttl=64 time=0.024 ms
64 bytes from 172.17.0.7: icmp_seq=2 ttl=64 time=0.055 ms
--- skydns.dev.dockerdev ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.024/0.039/0.055/0.016 ms
[root@docker opt]# dig skydns.dev.dockerdev
; <<>> DiG 9.9.2-P2 <<>> skydns.dev.dockerdev
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28414
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;skydns.dev.dockerdev. IN A
;; ANSWER SECTION:
skydns.dev.dockerdev. 23 IN A 172.17.0.7
;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Jan 30 22:26:27 2014
;; MSG SIZE rcvd: 76
However a curl or wget results in:
[root@docker opt]# curl http://skydns.dev.dockerdev:8080
curl: (6) Could not resolve host: skydns.dev.dockerdev
[root@docker opt]# wget skydns.dev.dockerdev
--2014-01-30 22:27:39-- http://skydns.dev.dockerdev/
Resolving skydns.dev.dockerdev (skydns.dev.dockerdev)... failed: Name or service not known.
wget: unable to resolve host address 'skydns.dev.dockerdev'
All that resolv.conf contains is
nameserver 127.0.0.1
And the VMs are created using:
docker run -d -p 53:53/udp -name skydns crosbymichael/skydns -nameserver 8.8.8.8:53 -domain dockerdev
docker run -d -v /var/run/docker.sock:/docker.sock -name skydock -link skydns:skydns crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain dockerdev
Putting a bit of load on my applications is causing the docker daemon to hang. It seems to be the binding to docker0 for skydns (-p 172.17.42.1:53:53/udp) that is causing the issues.
/var/log/upstart/docker.log says:
2014/01/30 12:20:52 Can't proxy a datagram to udp/172.17.0.2:53: dial udp 172.17.0.2:53: too many open files
...and goes on...
2014/01/30 12:20:54 http: Accept error: accept tcp [::]:4243: too many open files; retrying in 5ms
2014/01/30 12:20:54 http: Accept error: accept tcp [::]:4243: too many open files; retrying in 10ms
2014/01/30 12:20:54 http: Accept error: accept tcp [::]:4243: too many open files; retrying in 20ms
...same for the socket...
2014/01/30 12:24:29 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 5ms
2014/01/30 12:24:29 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 10ms
2014/01/30 12:24:29 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 20ms
2014/01/30 12:24:29 http: Accept error: accept unix /var/run/docker.sock: too many open files; retrying in 40ms
If I instead avoid the port binding for skydns, and rather pass
-dns $(docker inspect --format='{{.NetworkSettings.IPAddress}}' skydns)
to my containers, I can put as much load as I want on it without getting these errors.
Might be something specific to my setup, but I also tried a vanilla vm without any network tampering and saw the same.
Does skydock support reverse dns resolve ?
In my docker container, it prints that
bash-4.1# nslookup 172.17.0.7
Server: 172.17.42.1
Address: 172.17.42.1#53
** server can't find 7.0.17.172.in-addr.arpa.: NXDOMAIN
But for the hostname. it works
bash-4.1# nslookup s2.centos65.dev.docker
Server: 172.17.42.1
Address: 172.17.42.1#53
Name: s2.centos65.dev.docker
Address: 172.17.0.7
Using docker 0.9.1
Following the skydns/skydock setup I am able to
But from within a container with skydns as a server I'm unable to ping either a local domain or any other domain (unknown host google.com)
I've changed the host's /etc/resolv.conf to include skydns which allows the resolution of redis1.redis.dev.docker from the host, but the container still doesn't resolve (unknown host google.com)
I've also disabled SELinux
Changing the dns to 8.8.8.8 and ping works normally.
Testing under Ubuntu 13.04 everything works as expected.
Will test again once Docker 0.1.0 rpms are out as they include SELinux and other bug fixes relating to RHEL.
docker run -t -i crosbymichael/redis-cli -h redis1.redis.dev.docker
redis1.redis.dev.docker:6379>
I get no response from the terminal. ^C in the terminal kills the session to the redis host, but I'm not at all sure what's really happening there.
Running on a Mac, 10.9.1.
Vagrant 1.3.5
Virtual box 4.3.0 r89960
Using the docker image from the demo on the website. Hacked docker.conf until it didn't start on reboot & ran docker by hand with options from your example.
Say I'm using skydock
and running a few ruby web apps in docker:
app1.myrubyapp.production.myapp.com
app2.myrubyapp.production.myapp.com
app3.myrubyapp.production.myapp.com
I think I have two questions:
set $dns app-1.ruby.live.docker;
....
proxy_pass http://$dns:80;
Please excuse me if the questions are basic... I'm just getting into web dev stuff.
'nuff said.
/go/bin/skydock -ttl 30 -environment dev -s /docker.sock -domain docker -name skydns
constantly takes about 24-26% CPU usage. is that normal?
I'm testing this on an m1.small aws ec2.
Docker is configured not to resume container state (-r=false).
Container is started via upstart.
First is skydns, then skydock.
After that redis, mysql, fpm, and nginx
When rebooting, some of the containers are not registered by skydock.
Here is the log:
[info] 1391926655 skydock: loading plugins from /plugins/containerEnv.js
[debug] 1391926655 skydock: starting restore of containers
[info] 1391926655 skydock: adding 62cc880410 (skydock) to skydns
[info] 1391926655 skydock: adding 9ec3e07522 (skydns) to skydns
[debug] 1391926655 skydock: starting main process
[debug] 1391926656 skydock: received event (create) f8aa137b623fe5854037bf96f955df5f086b059bce69bcaff82fdff576b83837 atn-mysqld:latest
[debug] 1391926656 skydock: received event (create) 8a29be7375656d3ccc3ca44ac259b14b0df396dabb5c62731b280a9179f18ba9 atn-redis:latest
[debug] 1391926657 skydock: received event (create) e38405f92010678abc2fcc9139acf020db6d1a02d0fe3ab4bfb797f22a5a1b51 atn-fpm:latest
[debug] 1391926657 skydock: received event (create) 3ec3e1cc6bb27d68e886f74656fdd8180cc53a4747bd18ecfc088f91e5c554d4 atn-nginx:latest
[debug] 1391926657 skydock: received event (start) 8a29be7375656d3ccc3ca44ac259b14b0df396dabb5c62731b280a9179f18ba9 atn-redis:latest
[info] 1391926658 skydock: adding 8a29be7375 (redis) to skydns
[debug] 1391926658 skydock: received event (start) 3ec3e1cc6bb27d68e886f74656fdd8180cc53a4747bd18ecfc088f91e5c554d4 atn-nginx:latest
[info] 1391926658 skydock: adding 3ec3e1cc6b (www) to skydns
[debug] 1391926658 skydock: received event (start) e38405f92010678abc2fcc9139acf020db6d1a02d0fe3ab4bfb797f22a5a1b51 atn-fpm:latest
[info] 1391926658 skydock: adding e38405f920 (fpm) to skydns
After some time I stop the mysql container
[debug] 1391935317 skydock: received event (die) f8aa137b623fe5854037bf96f955df5f086b059bce69bcaff82fdff576b83837 atn-mysqld:latest
[info] 1391935317 skydock: removing f8aa137b62 from skydns
I'm not sure yet if this is skydock or docker fault.
Thank u for sharing this. Skydock looks promising.
will post details soon
I am using -environment server
and -domain example.com
and if I try to resolve server.example.com
I get back all IPs inside the environment, instead of real server.example.com
IP for that server. Even crazier, if I try to resolve example.com
, I get back 0.0.0.0
.
I think that everything from environment up (inclusive) should be recursed to provided DNS server.
Given that skydock has access to the Docker socket, it should be possible to query Docker for the IP address given an agreed-upon name of skydns
(or one configurable via an environment variable). That would alleviate the need to use container links and generally make setup simpler.
SkyDNS version 1 dev slowed down to a crawl, exiting new stuff is being added to SkyDNS version 2. Could you add support for this in skydock?
Not long ago I was able to get things to work but after reformating my machine I'm unable to get Skydock / SkyDNS working properly on an Ubuntu 14.04 host (dunno if it matters but it worked on my previous 13.10 installation). While I can't get things to work properly, I didn't change my /etc/default/docker
to include --dns
, and this is why you'll notice that I pass it around on the examples below.
These are the commands I ran on separate bash sessions and a "live demo" can be found on https://asciinema.org/a/9338
# SkyDNS
docker run -ti --rm \
-p 172.17.42.1:53:53/udp \
--dns 172.17.42.1 \
--name skydns \
crosbymichael/skydns -nameserver 8.8.8.8:53 -domain docker
# Skydock
docker run -ti --rm \
-v /var/run/docker.sock:/docker.sock \
--dns 172.17.42.1 \
--name skydock \
crosbymichael/skydock -ttl 30 -environment dev -s /docker.sock -domain docker -name skydns
# Redis server
docker run -ti --rm \
--name redis1 \
--dns 172.17.42.1 \
crosbymichael/redis
# Finnaly the redis-cli
docker run -ti --rm \
--dns 172.17.42.1 \
crosbymichael/redis-cli -h redis1.redis.dev.docker
# that gives me a
Could not connect to Redis at redis1.redis.dev.docker:6379: Temporary failure in name resolution
More information:
$ docker images | grep sky
crosbymichael/skydock latest e985023521f6 4 weeks ago 510.7 MB
crosbymichael/skydns next 79c99a4608ed 5 weeks ago 525 MB
crosbymichael/skydns latest 1923ce648d4c 3 months ago 137.5 MB
$ docker info
Containers: 1
Images: 270
Storage Driver: aufs
Root Dir: /home/fabio/docker/aufs
Dirs: 288
Execution Driver: native-0.1
Kernel Version: 3.13.0-24-generic
Username: fgrehm
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support
$ docker version
Client version: 0.10.0
Client API version: 1.10
Go version (client): go1.2.1
Git commit (client): dc9c28f
Server version: 0.10.0
Server API version: 1.10
Git commit (server): dc9c28f
Go version (server): go1.2.1
Last stable version: 0.10.0
If there's any other information you guys want from me to help debugging please LMK. Thanks in advance!
may you use a programming language please?, not a no-no 60 years deja-vue attempt of bullcrap.
Thank You.
https://www.youtube.com/watch?v=Nw42q1ofrV0
At 15:40 you have talked about redis-cli doing service discovery. Do you have the code for the redis-cli on the github ? Just wanted to understand how redis-cli is using skydock under the hoods..
I get nothing when I try pulling the published skydock/skydns images under Docker 0.9.
That is, starting them outputs absolutely nothing in logs. It seems to immediately return as well, even when -d isn't specified.
This is with a fresh install of Docker 0.9 on Ubuntu 13.10. A downgrade to 0.8.1 resolved everything.
I'll test further later, but I hit this when trying to roll out a fresh deployment, so figured I'd at least log it. I'll test the environment in Vagrant and add more details as I get them. Wish I had more time to provide better details now, but maybe someone else has encountered this and has more.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.