AMQP connection fails on CentOS about cloudman HOT 8 CLOSED

It's also worth noting that when I attempt to go to the appropriate site (1.3.3.7:42284/cloud), I am 404'd

from cloudman.

For the URL access, you should use either 1.3.3.7:42284 or 1.3.3.7/cloud (nginx proxies that port to /cloud). And I guess the you got the networking figured out to be able to access this IP.

As far as the AMQP connection failure goes, did you run the cleanup step when building the AMI? On an instance where the connection fails, check if directory /var/lib/rabbitmq/mnesia exists (it should not). If it does, RabbitMQ won't start properly. The problem with the hostname service may also be causing this so might be best to figure that out first (for CentOS 7, it seem the following is all that's needed: hostnamectl set-hostname <new hostname>).
Finally, you can manually check the RabbitMQ server status with /usr/sbin/rabbitmqctl status. You can also look for logs in /var/log/rabbitmq/ to see if there are any more clues there.

from cloudman.

Accessing the Cloudman server:

I am 404'd when I access the ip/cloud but I reach a timeout when I go to to port 42284. Can you advise?

Hostname change

The problem is, I am not sure what <new hostname> should be set to after the line is changed in /etc/hosts where the error restarting hostname occurs. Is cloudman dependent on this hostname change when connecting to the rabbitmq server?

Rabbitmq/AMQP connection fails

The directory /var/lib/rabbitmq/mnesia doesn't exist. No logs in /var/log/rabbitmq. Yet my login (see below) reads root@ip-1-3-3-7. Status output below, similar to what I see when i try /usr/sbin/rabbitmq start_app

[root@ip-1-3-3-7 ~]> ls -la /var/lib/rabbitmq/
-r--------   1 rabbitmq rabbitmq   20 Oct 29 00:00 .erlang.cookie
[root@ip-1-3-3-7 ~]>/usr/sbin/rabbitmqctl status
Status of node 'rabbit@ip-1-3-3-7' ...
Error: unable to connect to node 'rabbit@ip-1-3-3-7': nodedown

DIAGNOSTICS
===========

nodes in question: ['rabbit@ip-1-3-3-7']

hosts, their running nodes and ports:
- ip-1-3-3-7: [{rabbitmqctl1651,43613}]

current node details:
- node name: 'rabbitmqctl1651@ip-1-3-3-7'
- home dir: /var/lib/rabbitmq

from cloudman.

It looks like my nginx was loading the wrong configuration. I made the necessary changes in my init script, loaded some extra resources into the /usr and /usr/local nginx/conf directories (loaded into /etc/nginx from galaxy-cloudman-playbook), restarted the server and the /cloud url was available. After saving the new AMI, I restarted via cloudlaunch, the issues with AMQP persist, and I cannot log in to /cloud with either ubuntu::password, galaxy::password, or ::password. @afgane is this related to commenting out _allow_password_login 10/25/15 in cm_autorun.py?

from cloudman.

I'd start with the hostname then. Try setting <new hostname> to ip-1-3-3-7 as that's what RabbitMQ is referencing. What does cat /etc/hosts return? What about the hostname command?

Regarding the login - it looks you're right. Nginx PAM system is used for auth and it is configured with common_auth, which just checks /ect/passwd. However, that's never set if the _allow_password_login. Sorry about that. To get around it, it may be sufficient to execute that routine up the ssh service reload line so the user passwords get set.

from cloudman.

Cloudman server issues on Centos

Okay, I fixed the _allow_password_login to work appropriately for my system. Additionally, I changed the service hostname restart command to be compatible with CentOS6. I confirm this by logging in via ssh with the galaxy user with the password passed through with Cloudlaunch. In my browser when I navigate to ip/cloud and try to access the server with the same credentials, it fails to authenticate fully and loops back to the same login prompt. I note in the Nginx error logs that the Galaxy application is not loaded onto /mnt by observing no such file or directory error for the favicon. I think that galaxy-cloudman-playbook is initially responsible for downloading Galaxy in the "galaxy" role. This ran fine when I initially created the "galaxyFS", and uploaded the tarball to S3, but this was never downloaded by Cloudlaunch or Cloudman. Am I misunderstanding something?

Is the solution to generating my AMI to simply load my galaxy software to /mnt/galaxy/galaxy-app? Which software is responsible for modifying Galaxy's config/galaxy.ini to change the host IP and port or should I simply use 0.0.0.0?

Nginx access.log

1.3.3.7 - - [05/Nov/2015:11:43:07 -0500] "GET /cloud HTTP/1.1" 401 596 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
1.3.3.7 - - [05/Nov/2015:11:43:40 -0500] "GET /favicon.ico HTTP/1.1" 404 197 "http://1.2.3.4/cloud" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"

Nginx error.log

2015/11/05 11:39:21 [notice] 10579#0: signal process started
2015/11/05 11:43:40 [error] 10580#0: *1 open() "/mnt/galaxy/galaxy-app/static/favicon.ico" failed (2: No such file or directory), client: 1.3.3.7, server: , request: "GET /favicon.ico HTTP/1.1", host: "1.2.3.4", referrer: "http://1.2.3.4/cloud"

RabbitMQ server startup failure

I am still experiencing a failure during rabbitmq server startup. I've noticed in /etc/passwd that the user rabbitmq exists while rabbit does not. Setting the environmental variable RABBITMQ_NODENAME to rabbitmq@ip-1-3-3-7 changes the debug output during rabbitmqctl start_app but the server still fails to start. Additionally, rabbitmq version is 3.1.5 and no sample configuration file can be found in /etc/rabbitmq or /usr/share/doc/rabbitmq-server-3.1.5. What version of rabbitmq should I be running? Would installing the latest stable version (3.5.6) be okay? Is there a sample configuration I can look at or is it using defaults? @afgane or @dannon Can you advise? Is there a gitter channel or something where I can speak with someone?

> rabbitmqctl start_app
Starting node 'rabbit@ip-1-3-3-7' ...
Error: unable to connect to node 'rabbit@ip-1-3-3-7': nodedown

DIAGNOSTICS
===========

nodes in question: ['rabbit@ip-1-3-3-7']

hosts, their running nodes and ports:
- ip-1-3-3-7: [{rabbitmqctl12426,46483}]

current node details:
- node name: 'rabbitmqctl12426@ip-1-3-3-7'
- home dir: /var/lib/rabbitmq

from cloudman.

Updating to the latest stable version of RabbitMQ server did the trick for me. The URL issue persists, but that's a separate issue.

from cloudman.

Glad to hear you sorted this out. Let us know as you have more questions.

from cloudman.