Ceilometer: Error communicating with http://controller:8777 [Errno 111] Connection refused about coa HOT 2 CLOSED

Reinstalled Ceilometer service and the corresponding three endpoints (internal,public and admin):

keystone endpoint-list

+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+
| id | region | publicurl | internalurl | adminurl | service_id |
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+
| 285ef1297d53464d8ce28bb0216a6878 | regionOne | http://controller:8774/v2/%(tenant_id)s | http://controller:8774/v2/%(tenant_id)s | http://controller:8774/v2/%(tenant_id)s | 57769e1cab174d2ab86d0bbf83b353cc |
| 63fa857ddc1441b687101410a127840d | regionOne | http://controller:9292 | http://controller:9292 | http://controller:9292 | eedef0e2c8d74e598e8a2b3ffda06fd4 |
| 922381868147415ca3d5eb4b096dbb5c | regionOne | http://swiftProxy:8080/v1/AUTH_%(tenant_id)s | http://swiftProxy:8080/v1/AUTH_%(tenant_id)s | http://swiftProxy:8080 | 7a0d02c3900142678905d6fb3913f294 |
| d1041200927e459c91047644931708a3 | regionOne | http://controller:8777 | http://controller:8777 | http://controller:8777 | 3283706e25af4c59a4537b96b24424fb |
| e5241b20f25847148e85d57b2feee030 | regionOne | http://controller:5000/v2.0 | http://controller:5000/v2.0 | http://controller:35357/v2.0 | 34392ff821624a07800d04df07720630 |
| fa875f79a0ab40e4b591726b851b234a | regionOne | http://controller:9696 | http://controller:9696 | http://controller:9696 | b12e3d7fdcc84d4a91bcd06745a994f1 |
| fdc0a86085ef4980a16c5bf710257104 | regionOne | http://controller:8776/v2/%(tenant_id)s | http://controller:8776/v2/%(tenant_id)s | http://controller:8776/v2/%(tenant_id)s | e290e4c059454cad974f3424362b084b |
| fed6aa2fc82f4c49ac4335e8849c7b30 | regionOne | http://controller:8776/v1/%(tenant_id)s | http://controller:8776/v1/%(tenant_id)s | http://controller:8776/v1/%(tenant_id)s | 9711ee66648243878763cc7c64bfc8e7 |
+----------------------------------+-----------+----------------------------------------------+----------------------------------------------+-----------------------------------------+----------------------------------+

Now I get a new error related to keystone authentication:

ceilometer alarm-list

WARNING (http:172) Request returned failure status.
Invalid OpenStack Identity credentials.

With the log:

tail /var/log/ceilometer/ceilometer-api.log

2016-12-15 03:32:03.048 1157 INFO urllib3.connectionpool [-] Starting new HTTPS connection (1): 127.0.0.1
2016-12-15 03:32:03.135 1157 WARNING keystoneclient.middleware.auth_token [-] Retrying on HTTP connection exception: [Errno 1] _ssl.c:510: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
2016-12-15 03:32:03.638 1157 INFO urllib3.connectionpool [-] Starting new HTTPS connection (1): 127.0.0.1
2016-12-15 03:32:03.653 1157 WARNING keystoneclient.middleware.auth_token [-] Retrying on HTTP connection exception: [Errno 1] _ssl.c:510: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
2016-12-15 03:32:04.656 1157 INFO urllib3.connectionpool [-] Starting new HTTPS connection (1): 127.0.0.1
2016-12-15 03:32:04.674 1157 WARNING keystoneclient.middleware.auth_token [-] Retrying on HTTP connection exception: [Errno 1] _ssl.c:510: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
2016-12-15 03:32:06.678 1157 INFO urllib3.connectionpool [-] Starting new HTTPS connection (1): 127.0.0.1
2016-12-15 03:32:06.691 1157 ERROR keystoneclient.middleware.auth_token [-] HTTP connection exception: [Errno 1] _ssl.c:510: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
2016-12-15 03:32:06.692 1157 WARNING keystoneclient.middleware.auth_token [-] Authorization failed for token
2016-12-15 03:32:06.692 1157 INFO keystoneclient.middleware.auth_token [-] Invalid user token - rejecting request

from coa.

Had to check the following:

1- Double Check the ceilometer configuration:
/etc/ceilometer/ceilometer.conf

[keystone_authtoken] auth_host =
controller auth_port = 35357
auth_protocol = http auth_uri =
http://controller:5000
admin_tenant_name = service
admin_user = ceilometer
admin_password = CEILOMETER_PASS

[service_credentials]
os_auth_url = http://controller:5000/v2.0
os_username = ceilometer
os_tenant_name = service
os_password = CEILOMETER_PASS

2- Use the logs to investigate the errors:

• /var/log/keystone/keystone-all.log
• /var/log/ceilometer/ceilometer-api.log

3- Check mongodb password, reset if needed

4- Check Ceilometer services:

export CEILO_SVCS='agent-central agent-notification ceilometer-api ceilometer-collector alarm-evaluator alarm-notifier' for svc in $CEILO_SVCS ; do sudo service ceilometer-$svc restart ; done

5- Check if the service and the endpoints (admin, internal and public) are correctly configured

6- [This solved my issue] Check the user ceilometer password, if needed delete and recreate the user and assign the appropriate role and roject, this solved my issue.

(Careful!! It was in a lab environment during first deployment, I guess it would be eventual consequences for your alarms created with the old user!! )

from coa.