Hello, Codenvy Team.
Please help me with the following issue.
I'm installing codenvy server on CentOS 7.2 x64 machine (actually it is Amazon EC2 instance).
I have sufficient number of processors and memory.
I'm using single node installation with defaults.
After the installation i can reach Codenvy by a link http://ip_address_of_codenvy_machine
But when i'm trying to create a workspace the process is hanging on the step of injecting and starting ws_agent. I attached the screenshot.
During the process of creation i can see that docker container is created. So, i increased the timeout to have enough time to investigate an issue (all customizations to codenvy config were done in Puppet) and then i opened a terminal to that container
docker exec -it <container_id> bash
I saw in the ws_agent (tomcat) log file (catalina.out) that the process is hanging after these lines
[STDOUT] 2016-07-18 09:29:15,275[ost-startStop-1] [INFO] [o.a.c.startup.HostConfig 1030] - Deploying web application directory /home/user/che/ws-agent/webapps/ROOT
[STDOUT] 2016-07-18 09:29:15,362[ost-startStop-1] [INFO] [o.a.c.startup.HostConfig 1142] - Deployment of web application directory /home/user/che/ws-agent/webapps/ROOT has finished in 87 ms
[STDOUT] 2016-07-18 09:29:15,368[main] [INFO] [o.a.c.http11.Http11NioProtocol 470] - Starting ProtocolHandler ["http-nio-4401"]
[STDOUT] 2016-07-18 09:29:15,373[main] [INFO] [o.a.catalina.startup.Catalina 642] - Server startup in 9850 ms
Then i decided to see what network ports are opened in the container
lsof -i
i can see these ones
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 22 user 23u IPv6 54794 0t0 TCP *:52129 (LISTEN)
java 22 user 55u IPv6 54796 0t0 TCP *:4401 (LISTEN)
java 22 user 59u IPv6 54798 0t0 TCP *:32002 (LISTEN)
java 22 user 60u IPv6 54799 0t0 TCP *:32102 (LISTEN)
java 22 user 69u IPv6 54070 0t0 TCP localhost:4405 (LISTEN)
java 22 user 230u IPv6 54067 0t0 TCP 060571bacb15:55640->codenvy:http (ESTABLISHED)
I have a working codenvy server in my VMWare home machine so i did the same thing there
and see the same ports + the port for terminal.
So the process is hanging on the command
sleep 5 && mkdir -p ~/che && rm -rf ~/che/* && unzip -q /mnt/che/ws-agent.zip -d ~/che/ws-agent && ~/che/ws-agent/bin/catalina.sh run
and never reaches this command
mkdir -p ~/che && cp /mnt/che/terminal -R ~/che && ~/che/terminal/che-websocket-terminal -addr :4411 -cmd /bin/bash -static ~/che/terminal/ -path '/[^/]+'
Okay after that thoughts and hours of reading issues here i decided to check firewall. I thought that may be ws_agent couldn't interract back with codenvy server. So i tried to connect from inside docker container
curl -v http://codenvy/api
and the result is
____________________________________________________--
- Hostname was NOT found in DNS cache
- Trying 172.17.42.1...
- Connected to codenvy (172.17.42.1) port 80 (#0)
GET /api HTTP/1.1
User-Agent: curl/7.38.0
Host: codenvy
Accept: /
< HTTP/1.1 302 Found
- Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< Location: /api/
< Transfer-Encoding: chunked
< Date: Mon, 18 Jul 2016 10:02:08 GMT
<
- Connection #0 to host codenvy left intact
After that i added logging chain to my HOST iptables but logging didn't show any blocked packets.
iptables -L -n --line-numbers
Chain INPUT (policy ACCEPT)
num target prot opt source destination
1 f2b-SSH tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22
2 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
3 ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
4 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
5 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
6 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:53
7 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:53
8 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:80
9 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:81
10 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:443
11 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:444
12 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:389
13 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:5000
14 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:8000
15 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:9000
16 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:10050
17 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:10051
18 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:27017
19 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:32001
20 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:32101
21 ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 state NEW udp dpt:161
22 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpts:32768:65535
23 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:8101
24 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:8140
25 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:32102
26 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:32202
27 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:5432
28 LOGGING all -- 0.0.0.0/0 0.0.0.0/0
29 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
1 DOCKER-ISOLATION all -- 0.0.0.0/0 0.0.0.0/0
2 DOCKER all -- 0.0.0.0/0 0.0.0.0/0
3 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
4 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
5 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
6 DOCKER all -- 0.0.0.0/0 0.0.0.0/0
7 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
8 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
9 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
10 ACCEPT all -- 0.0.0.0/0 172.17.0.0/16
11 LOGGING all -- 0.0.0.0/0 0.0.0.0/0
12 REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
Chain DOCKER (2 references)
num target prot opt source destination
1 ACCEPT tcp -- 0.0.0.0/0 172.17.0.1 tcp dpt:4411
2 ACCEPT tcp -- 0.0.0.0/0 172.17.0.1 tcp dpt:4403
3 ACCEPT tcp -- 0.0.0.0/0 172.17.0.1 tcp dpt:4401
4 ACCEPT tcp -- 0.0.0.0/0 172.17.0.1 tcp dpt:3306
5 ACCEPT tcp -- 0.0.0.0/0 172.17.0.1 tcp dpt:80
6 ACCEPT tcp -- 0.0.0.0/0 172.17.0.1 tcp dpt:22
Chain DOCKER-ISOLATION (1 references)
num target prot opt source destination
1 DROP all -- 0.0.0.0/0 0.0.0.0/0
2 DROP all -- 0.0.0.0/0 0.0.0.0/0
3 RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain LOGGING (2 references)
num target prot opt source destination
1 LOG all -- 0.0.0.0/0 0.0.0.0/0 limit: avg 10/min burst 5 LOG flags 0 level 7 prefix "DROP: "
2 DROP all -- 0.0.0.0/0 0.0.0.0/0
Chain f2b-SSH (1 references)
num target prot opt source destination
1 RETURN all -- 0.0.0.0/0 0.0.0.0/0
Please help!!!