@fduranti if you have proxy for the docker registry, then no need to start Admiral in any specific way. Just when you register your registry (under Templates -> Manage Registries) provide the proxy URL, http://myproxyhost:80. Let me know if this worked for you

from admiral.

Hi I need a proxy server (we use trend micro) to be able to access Internet, my servers don't have direct internet access. It's not a registry proxy (like harbor that can proxy access to docker registry).
Without a proxy I get a timeout trying a query on registry or validating the registry:

This is a search:

Putting a proxy in the url will not work as it's just a proxy to navigate on internet not a registry proxy:

from admiral.

@fduranti Before we try anything can we confirm that these options are really correct:
-Dhttp.proxyPort=80
-Dhttps.proxyPort=80

My guess would be 443 for https.

from admiral.

@glechev they are correct we have 1 proxy server (Trend Micro IMVSA) that answer on port 80 for all protocols. I've also tested with a squid proxy just to be sure we use to do some test (also answer only on a single port for all protocol.
Normally on other Java Application it seems that i just need to configure those parameters but I'm not sure if the java program should be configured to use it.
Those are the option I pass:

XENON_OPTS="--sandbox=/opt/admiral/data"
JAVA_OPTS="-Dconfiguration.properties=/opt/admiral/etc/conf.properties -Dhttp.proxyHost=proxy -Dhttp.proxyPort=80 -Dhttps.proxyHost=proxy -Dhttps.proxyPort=80”
docker run -d -p 8282:8282 --name admiral -e XENON_OPTS="$XENON_OPTS" -e JAVA_OPTS="$JAVA_OPTS" -v /opt/admiral:/opt/admiral vmware/admiral:latest

I should also pass a -Dhttp.nonProxyHosts to have the proxy work correctly but at the moment it should at least work for registry access also without this option.

This is the java documentation related to proxy: http://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html

Doing a tcpdump on the docker network while trying a verify on the registry page i see a DNS query and then that the application try to access the registry directly so the http(s).proxy|port settings are not used.

14:47:52.254851 IP dnsserver.domain > 172.17.0.2.60470: 12741 5/0/0 CNAME elb-hub.us-east-1.aws.dckr.io., CNAME us-east-1-elbhub-1t5fblb53f6sl-411513349.us-east-1.elb.amazonaws.com., A 34.193.6.247, A 52.22.117.51, A 34.194.105.59 (211)
14:47:52.255371 IP 172.17.0.2.49180 > ec2-34-193-6-247.compute-1.amazonaws.com.https: Flags [S], seq 3453937841, win 29200, options [mss 1460,sackOK,TS val 2180386255 ecr 0,nop,wscale 7], length 0
14:47:53.255888 IP 172.17.0.2.49180 > ec2-34-193-6-247.compute-1.amazonaws.com.https: Flags [S], seq 3453937841, win 29200, options [mss 1460,sackOK,TS val 2180387256 ecr 0,nop,wscale 7], length 0

from admiral.

Some other discovery... setting the proxy via linux environonment variable seems to half-work.
I'm setting it running docker

PROXY="http://proxyserver.my.domain:80"
NOPROXY="my.domain,other.domain,localhost,127.0.0.1"
docker run -d -p $PORT:$PORT --name $ADM -e ADMIRAL_PORT=$PORT -e XENON_OPTS="$XENON_OPTS" -e JAVA_OPTS="$JAVA_OPTS" -e http_proxy="$PROXY" -e https_proxy="$PROXY" -e no_proxy="$NOPROXY" -v /opt/admiral:/opt/admiral vmware/admiral:latest

In this way container search work correctly. default registry verify return the same error as the screenshot I posted (and I don't see any log in the proxy log but i see direct traffic to docker registry on the tcpdump as before).
The only problem is that it seems admiral is not respecting no_proxy environment and try to use proxy also to:

1. access to docker hosts and sometimes it get errors
2. access to other admiral server in a node group when i try to use node group and admiral don't start because it cannot communicate.
3. communication to internal 172.x network (that should not be proxied but It's not possible to configure them on the no_proxy as it will not accept wildcard).

In the exclusion there's the my.domain but it seems that it's ignored as i see on the proxy logs connection that go to "server.my.domain"

1483973022.120 0 x.x.x.x TCP_DENIED/411 2040 GET http://docker.my.domain:4243/v1.19/containers/58b3e5afdc9a84bc223c3f50e44462e0c264bff231d4429cbcdb65c2b81603d6/json - NONE/- text/html
1483973023.111 0 x.x.x.x TCP_DENIED/400 4778 NONE error:unsupported-request-method - NONE/- text/html
1483973163.871 180836 x.x.x.x. TCP_MISS/504 1472 GET http://172.17.0.10:8284/core/node-groups/default - DIRECT/172.17.0.10 text/html

With my squid proxy that has also internal network access something works but using my real production proxy admiral don't even work.

As the proxy access is only needed by registry access the best would be to be able to configure it (or use the -D for java proxy only when registry access is needed.

from admiral.

@fduranti I've though about using the http(s)_proxy environment variable but I don't think java really takes it into account (or at least it's not documented anywhere), however you did describe change in behavior that makes me wonder if the issue is environmental. Can you try running admiral outside of a container? Here's an uber jar https://bintray.com/vmware/admiral/download_file?file_path=com%2Fvmware%2Fadmiral%2Fadmiral-host%2F0.9.1%2Fadmiral-host-0.9.1-uber-jar-with-agent.jar . To start it:

java -Dhttp.proxyHost= -Dhttp.proxyPort=80 -Dhttps.proxyHost= -Dhttps.proxyPort=80 -jar admiral-host-0.9.1-uber-jar-with-agent.jar --bindAddress=0.0.0.0 --port=8282

If this work we can focus on the container packaging.

from admiral.

@glechev
Same problem and same errors with the java as with the container.
It seems that java is not using http(s).proxy http(s).port http.noproxy

Started admiral with:

java -Dhttp.proxyHost=proxy.my.domain -Dhttp.proxyPort=80 -Dhttps.proxyHost=proxy.my.domain -Dhttps.proxyPort=80 -jar admiral-host-0.9.1-uber-jar-with-agent.jar --bindAddress=0.0.0.0 --port=8282 --sandbox=/opt/admiral/datajava

Trying a seach it's timing out:

[238][I][2017-01-09 16:03:03.890][8282/adapters/registry-service][processV1SearchRequest][Performing registry search: https://registry.hub.docker.com/v1/search?q=http]
[239][W][2017-01-09 16:03:34.915][8282/images][logFailures][Failed to perform registry search: connection timed out: registry.hub.docker.com/52.22.117.51:443]

Configuring (no_)http(s)_proxy at linux level and running admiral I can search template over registry but connection on excluded domain still go wrongly to proxy.

#env |grep proxy
http_proxy=http://proxy.my.domain:80
https_proxy=http://proxy.my.domain:80
no_proxy=my.domain,127.0.0.1

java -jar admiral-host-0.9.1-uber-jar-with-agent.jar --bindAddress=0.0.0.0 --port=8282 --sandbox=/opt/admiral/datajava

This is log on proxy (there are many) of connection to docker.my.domain that should go directly and not via proxy:

1483978083.740 x.x.x.x TCP_DENIED/411 2040 GET http://docker.my.domain:4243/v1.19/containers/cd4f3a49960a2910a278ee8d2172c3459ebae89a4058c7f78711eb10c1054930/json - NONE/- text/html

This is some error on the admiral logs related to connection to docker.my.domain:

[138][W][2017-01-09 16:08:03.755][8282/adapters/docker-service][fail][java.lang.Exception: Service http://docker.my.domain:4243/v1.19/containers/4d5c068b60c8dc2d8f5a1fc7e8c40f5edeb9cbfdd33aa669f1ec52f24975cf43/json returned error 411 for GET. id 4623; Reason:

<TITLE>ERROR: The requested URL could not be retrieved</TITLE> <STYLE type="text/css"></STYLE>

ERROR

The requested URL could not be retrieved

---

While trying to process the request:

GET /v1.19/containers/4d5c068b60c8dc2d8f5a1fc7e8c40f5edeb9cbfdd33aa669f1ec52f24975cf43/json HTTP/1.1
x-xenon-ctx-id: No callback set
x-xenon-auth-token: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJ4biIsInN1YiI6Ii9jb3JlL2F1dGh6L3N5c3RlbS11c2VyIiwiZXhwIjoyNTM0MDIyNzI0NDc1NDYwMDB9.BTYSRGaJpSJn945u3JIyJ8lee9T8H-50qKkv53i0syQ
Content-Length: 78
Content-Type: application/json
Connection: keep-alive
Referer: /
User-Agent: com.vmware.admiral.common.util.ServiceClientFactory
Accept: */*
Host: docker.my.domain

The following error was encountered:

• Invalid Request

Some aspect of the HTTP Request is invalid. Possible problems:

• Missing or unknown request method
• Missing URL
• Missing HTTP Identifier (HTTP/1.0)
• Request is too large
• Content-Length missing for POST or PUT requests
• Illegal character in hostname; underscores are not allowed

Your cache administrator is root.

---

Generated Mon, 09 Jan 2017 16:08:03 GMT by ITROMPX02 (squid/2.6.STABLE21)

at com.vmware.admiral.adapter.docker.service.AbstractDockerAdapterService.fail(AbstractDockerAdapterService.java:249)
at com.vmware.admiral.adapter.docker.service.DockerAdapterService.lambda$inspectContainer$58(DockerAdapterService.java:1093)
at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201)
at com.vmware.xenon.common.Operation.fail(Operation.java:1179)
at com.vmware.xenon.common.Operation.fail(Operation.java:1109)
at com.vmware.xenon.common.http.netty.NettyHttpServiceClient.fail(NettyHttpServiceClient.java:605)
at com.vmware.xenon.common.http.netty.NettyHttpServiceClient.lambda$doSendRequest$3(NettyHttpServiceClient.java:527)
at com.vmware.xenon.common.Operation.lambda$nestCompletion$1(Operation.java:1233)
at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201)
at com.vmware.xenon.common.Operation.fail(Operation.java:1179)
at com.vmware.xenon.common.Operation.fail(Operation.java:1109)
at com.vmware.xenon.common.http.netty.NettyHttpServerResponseHandler.checkResponseForError(NettyHttpServerResponseHandler.java:214)
at com.vmware.xenon.common.http.netty.NettyHttpServerResponseHandler.lambda$decodeResponseBody$0(NettyHttpServerResponseHandler.java:157)
at com.vmware.xenon.common.Operation.lambda$nestCompletion$1(Operation.java:1233)
at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201)
at com.vmware.xenon.common.Operation.complete(Operation.java:1105)
at com.vmware.xenon.common.Utils.decodeBody(Utils.java:861)
at com.vmware.xenon.common.Utils.decodeBody(Utils.java:838)
at com.vmware.xenon.common.http.netty.NettyHttpServerResponseHandler.decodeResponseBody(NettyHttpServerResponseHandler.java:163)
at com.vmware.xenon.common.http.netty.NettyHttpServerResponseHandler.completeRequest(NettyHttpServerResponseHandler.java:137)
at com.vmware.xenon.common.http.netty.NettyHttpServerResponseHandler.channelRead0(NettyHttpServerResponseHandler.java:59)
at com.vmware.xenon.common.http.netty.NettyHttpServerResponseHandler.channelRead0(NettyHttpServerResponseHandler.java:41)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:335)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:327)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:335)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:327)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:288)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:262)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:243)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:335)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:327)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1336)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:335)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:485)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:399)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ProtocolException: Service http://docker.my.domain:4243/v1.19/containers/4d5c068b60c8dc2d8f5a1fc7e8c40f5edeb9cbfdd33aa669f1ec52f24975cf43/json returned error 411 for GET. id 4623
... 39 more
]
[139][W][2017-01-09 16:08:03.757][8282/adapters/docker-service][fail][java.lang.Exception: Service http://docker.my.domain:4243/v1.19/containers/8c1172565ad15c2218ec717868d58e18ad9c796af2ebd140319cf353d604da27/json returned error 411 for GET. id 4622; Reason:

<TITLE>ERROR: The requested URL could not be retrieved</TITLE> <STYLE type="text/css"></STYLE>

ERROR

The requested URL could not be retrieved

---

While trying to process the request:

GET /v1.19/containers/8c1172565ad15c2218ec717868d58e18ad9c796af2ebd140319cf353d604da27/json HTTP/1.1
x-xenon-ctx-id: No callback set
x-xenon-auth-token: eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJ4biIsInN1YiI6Ii9jb3JlL2F1dGh6L3N5c3RlbS11c2VyIiwiZXhwIjoyNTM0MDIyNzI0NDc1NDYwMDB9.BTYSRGaJpSJn945u3JIyJ8lee9T8H-50qKkv53i0syQ
Content-Length: 78
Content-Type: application/json
Connection: keep-alive
Referer: /
User-Agent: com.vmware.admiral.common.util.ServiceClientFactory
Accept: */*
Host: docker.my.domain

The following error was encountered:

• Invalid Request

Some aspect of the HTTP Request is invalid. Possible problems:

• Missing or unknown request method
• Missing URL
• Missing HTTP Identifier (HTTP/1.0)
• Request is too large
• Content-Length missing for POST or PUT requests
• Illegal character in hostname; underscores are not allowed

Your cache administrator is root.

---

Generated Mon, 09 Jan 2017 16:08:03 GMT by ITROMPX02 (squid/2.6.STABLE21)

at com.vmware.admiral.adapter.docker.service.AbstractDockerAdapterService.fail(AbstractDockerAdapterService.java:249)
at com.vmware.admiral.adapter.docker.service.DockerAdapterService.lambda$inspectContainer$58(DockerAdapterService.java:1093)
at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201)
at com.vmware.xenon.common.Operation.fail(Operation.java:1179)
at com.vmware.xenon.common.Operation.fail(Operation.java:1109)
at com.vmware.xenon.common.http.netty.NettyHttpServiceClient.fail(NettyHttpServiceClient.java:605)
at com.vmware.xenon.common.http.netty.NettyHttpServiceClient.lambda$doSendRequest$3(NettyHttpServiceClient.java:527)
at com.vmware.xenon.common.Operation.lambda$nestCompletion$1(Operation.java:1233)
at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201)
at com.vmware.xenon.common.Operation.fail(Operation.java:1179)
at com.vmware.xenon.common.Operation.fail(Operation.java:1109)
at com.vmware.xenon.common.http.netty.NettyHttpServerResponseHandler.checkResponseForError(NettyHttpServerResponseHandler.java:214)
at com.vmware.xenon.common.http.netty.NettyHttpServerResponseHandler.lambda$decodeResponseBody$0(NettyHttpServerResponseHandler.java:157)
at com.vmware.xenon.common.Operation.lambda$nestCompletion$1(Operation.java:1233)
at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201)
at com.vmware.xenon.common.Operation.complete(Operation.java:1105)
at com.vmware.xenon.common.Utils.decodeBody(Utils.java:861)
at com.vmware.xenon.common.Utils.decodeBody(Utils.java:838)
at com.vmware.xenon.common.http.netty.NettyHttpServerResponseHandler.decodeResponseBody(NettyHttpServerResponseHandler.java:163)
at com.vmware.xenon.common.http.netty.NettyHttpServerResponseHandler.completeRequest(NettyHttpServerResponseHandler.java:137)
at com.vmware.xenon.common.http.netty.NettyHttpServerResponseHandler.channelRead0(NettyHttpServerResponseHandler.java:59)
at com.vmware.xenon.common.http.netty.NettyHttpServerResponseHandler.channelRead0(NettyHttpServerResponseHandler.java:41)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:335)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:327)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:335)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:327)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:428)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:288)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:262)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:243)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:335)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:327)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1336)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:335)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:544)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:485)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:399)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:371)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ProtocolException: Service http://docker.my.domain:4243/v1.19/containers/8c1172565ad15c2218ec717868d58e18ad9c796af2ebd140319cf353d604da27/json returned error 411 for GET. id 4622
... 39 more
]

Doing a wget on the same server port don't go through proxy:

#wget http://docker.my.domain:4243
--2017-01-09 17:12:11-- http://docker.my.domain:4243/
Resolving docker.my.domain (docker.my.domain)... x.x.x.x
Connecting to docker.my.domain (docker.my.domain)|x.x.x.x|:4243... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-01-09 17:12:11 ERROR 404: Not Found.

If i remove the my.domain from no_proxy wget go through proxy:

#unset no_proxy; wget http://docker.my.domain:4243
--2017-01-09 17:15:06-- http://docker.my.domain:4243/
Resolving itrompx01.q8int.com (proxy.my.domain)... y.y.y.y
Connecting to proxy.my.domain (proxy.my.domain)|y.y.y.y|:80... connected.
Proxy request sent, awaiting response... 404 Not Found
2017-01-09 17:15:06 ERROR 404: Not Found.

Doing a registry search work correctly:

[142][I][2017-01-09 16:10:14.113][8282/adapters/registry-service][processV1SearchRequest][Performing registry search: https://registry.hub.docker.com/v1/search?q=http]

from admiral.

Hi @fduranti,

Xenon, the library Admiral is based on uses Netty for http communication, and Netty does not support standard java http proxy arguments.
You can set environment variable http_proxy with value uri of your proxy. However it will be used for all traffic and there's no configuration to exclude given hosts.

In you case you can setup another container with configured proxy which will redirect the requests to the registry. Check https://github.com/schmunk42/docker-nginx-redirect and https://hub.docker.com/r/schmunk42/nginx-redirect/. Then when configure Admiral with the new registry put the address of this proxy container.

from admiral.

Seems like this feature was targeted to be addressed by March 1st. It's been raised as a blocker in a customer POC. Can you provide a status update please. Thx

from admiral.

If you need to have docker host to work behind proxy you can follow the this guide.
If you want the Admiral itself to be behind a proxy you need to edit this property file

from admiral.