Thanks for the report!
Unfortunately I can't reproduce this issue. Could you please provide the following informations:

• cat /etc/hostname
• cat /etc/hosts
• /opt/graylog2/embedded/bin/etcdctl ls /elasticsearch

from graylog2-images.

Sure, see below

--------------------------------------output--------------------------------------------------------------------------

administrator@ash-vsrv-net01:$ cat /etc/hostname
ash-vsrv-net01
administrator@ash-vsrv-net01:$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 ash-vsrv-net01

The following lines are desirable for IPv6 capable hosts

::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
administrator@ash-vsrv-net01:$ /opt/graylog2/embedded/bin/etcdctl ls
/elasticsearch
/elasticsearch/10.201.51.126
administrator@ash-vsrv-net01:$

On Wed Feb 11 2015 at 2:28:30 PM Marius Sturm [email protected]
wrote:

Thanks for the report!
Unfortunately I can't reproduce this issue. Could you please provide the
following informations:

• cat /etc/hostname
• cat /etc/hosts
• /opt/graylog2/embedded/bin/etcdctl ls /elasticsearch

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

from graylog2-images.

This looks all good. This could also help:

• ifconfig
• pgrep -a etcd
• cat /etc/graylog2/graylog2-settings.json

The error is an indicator that etcd is not running or can not be reached. But the last command shows that the service actually works. Does the error still persist or was it just a single apperance?

from graylog2-images.

See output below. I can reproduce the same original error with every
execution of 'sudo graylog2-ctl reconfigure' on this box.

------------------------------output-----------------------------------
root@ash-vsrv-net01:~# ifconfig
eth0 Link encap:Ethernet HWaddr 00:50:56:b3:77:8f
inet addr:10.201.51.126 Bcast:10.201.51.255 Mask:255.255.255.0
inet6 addr: fe80::250:56ff:feb3:778f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:77579 errors:0 dropped:10 overruns:0 frame:0
TX packets:17657 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:8440389 (8.4 MB) TX bytes:18782922 (18.7 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:840769 errors:0 dropped:0 overruns:0 frame:0
TX packets:840769 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:132974723 (132.9 MB) TX bytes:132974723 (132.9 MB)

root@ash-vsrv-net01:# pgrep -a etcd
892 /opt/graylog2/embedded/sbin/etcd -listen-client-urls=http://0.0.0.0:2379
,http://0.0.0.0:4001 -data-dir=/var/opt/graylog2/data/etcd
root@ash-vsrv-net01:# cat /etc/graylog2/graylog2-settings.json
{
"timezone": "EST",
"smtp_server": "smtp-relay.wilresearch.com",
"smtp_port": 587,
"smtp_user": "",
"smtp_password": "",
"master_node": "127.0.0.1",
"local_connect": false,
"current_address": "10.201.51.126",
"last_address": "10.201.51.126"
}
root@ash-vsrv-net01:~#

On Wed Feb 11 2015 at 2:50:11 PM Marius Sturm [email protected]
wrote:

This looks all good. This could also help:

• ifconfig
• pgrep -a etcd
• cat /etc/graylog2/graylog2-settings.json

The error is an indicator that etcd is not running or can not be reached.
But the last command shows that the service actually works. Does the error
still persist or was it just a single apperance?

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

from graylog2-images.

All your settings are perfectly fine and when I try to reproduce with the same I dont get that error :/

I have improved the output of the reconfigure run a little bit, could you please replace the file /opt/graylog2/embedded/cookbooks/graylog2/libraries/registry.rb with this one https://raw.githubusercontent.com/Graylog2/omnibus-graylog2/0.92/files/graylog2-cookbooks/graylog2/libraries/registry.rb

Afterwards re-run sudo graylog2-ctl reconfigure and then check the whole output for stack trace or errors. Maybe there are some informations.

Did you change anything in the /var/opt/graylog2/data directory? Like user rights or mount a bigger drive there or so?

from graylog2-images.

I made a backup of the registry.rb file and added your updated code to a
new registry.rb file and re-ran 'sudo graylog2-ctl reconfigure'. I still
received the same errors. Here is the fulloutput of the commmand:

P.S functionaly the GrayLog2 appliance seems to be working ok. Ive been
using it since it was initially deployed yesterday and I have no errors in
the web interface, only when executing 'sudo graylog2-ctl reconfigure'.
---------------------output--------------------

root@ash-vsrv-net01:/# sudo graylog2-ctl reconfigure
Starting Chef Client, version 12.0.3
Compiling Cookbooks...
Recipe: graylog2::default

• directory[/etc/graylog2] action create (up to date)
  Converging 119 resources
• directory[/etc/graylog2] action nothing (skipped due to action :nothing)
  Recipe: graylog2::users
• group[graylog2] action create (up to date)
• user[graylog2] action create (up to date)
• directory[/var/opt/graylog2] action create (up to date)
• template[/var/opt/graylog2/.gitconfig] action create (up to date)
• directory[/var/opt/graylog2/data] action create (up to date)
  Recipe: graylog2::authbind
• directory[/etc/authbind/byport] action create (up to date)
• file[/etc/authbind/byport/!514] action create (up to date)
  Recipe: runit::upstart
• cookbook_file[/etc/init/graylog2-runsvdir.conf] action create (up to
  date)
• execute[initctl status graylog2-runsvdir] action run
  • execute initctl status graylog2-runsvdir
• execute[initctl start graylog2-runsvdir] action run (skipped due to
  only_if)
  Recipe: timezone-ii::default
• apt_package[tzdata] action install (up to date)
  Recipe: timezone-ii::debian
• template[/etc/timezone] action create
  • update content in file /etc/timezone from 7456e2 to 156b95
    --- /etc/timezone 2015-02-11 17:23:05.988228736 -0500
    +++ /tmp/chef-rendered-template20150212-19893-1l3thv4 2015-02-12
    10:32:52.653161322 -0500
    @@ -1,2 +1,2 @@
    -SystemV/EST5
    +EST
• bash[dpkg-reconfigure tzdata] action nothing (skipped due to action
  :nothing)
  Recipe: graylog2::etcd
• directory[/var/log/graylog2/etcd] action create (up to date)
• directory[/var/opt/graylog2/data/etcd] action create (up to date)
• directory[/opt/graylog2/sv/etcd] action create (up to date)
• directory[/opt/graylog2/sv/etcd/log] action create (up to date)
• directory[/opt/graylog2/sv/etcd/log/main] action create (up to date)
• template[/opt/graylog2/sv/etcd/run] action create (up to date)
• template[/opt/graylog2/sv/etcd/log/run] action create (up to date)
• template[/var/log/graylog2/etcd/config] action create (up to date)
• ruby_block[reload etcd svlogd configuration] action nothing (skipped
  due to action :nothing)
• file[/opt/graylog2/sv/etcd/down] action delete (up to date)
• link[/opt/graylog2/init/etcd] action create (up to date)
• link[/opt/graylog2/service/etcd] action create (up to date)
• ruby_block[supervise_etcd_sleep] action run (skipped due to not_if)
• service[etcd] action nothing (skipped due to action :nothing)
• execute[/opt/graylog2/embedded/bin/graylog2-ctl start etcd] action run
  • execute /opt/graylog2/embedded/bin/graylog2-ctl start etcd
    Recipe: graylog2::elasticsearch
• directory[/var/log/graylog2/elasticsearch] action create (up to date)
• directory[/var/opt/graylog2/data/elasticsearch] action create (up to
  date)
• template[/opt/graylog2/conf/elasticsearch.yml] action create (up to
  date)
• directory[/opt/graylog2/sv/elasticsearch] action create (up to date)
• directory[/opt/graylog2/sv/elasticsearch/log] action create (up to date)
• directory[/opt/graylog2/sv/elasticsearch/log/main] action create (up to
  date)
• template[/opt/graylog2/sv/elasticsearch/run] action create (up to date)
• template[/opt/graylog2/sv/elasticsearch/log/run] action create (up to
  date)
• template[/var/log/graylog2/elasticsearch/config] action create (up to
  date)
• ruby_block[reload elasticsearch svlogd configuration] action nothing
  (skipped due to action :nothing)
• file[/opt/graylog2/sv/elasticsearch/down] action delete (up to date)
• link[/opt/graylog2/init/elasticsearch] action create (up to date)
• link[/opt/graylog2/service/elasticsearch] action create (up to date)
• ruby_block[supervise_elasticsearch_sleep] action run (skipped due to
  not_if)
• service[elasticsearch] action nothing (skipped due to action :nothing)
• execute[/opt/graylog2/embedded/bin/graylog2-ctl start elasticsearch]
  action run
  • execute /opt/graylog2/embedded/bin/graylog2-ctl start elasticsearch
• ruby_block[add node to cluster list] action
  run[2015-02-12T10:37:52-05:00] ERROR: Can not add node 10.201.51.126 to
  directory elasticsearch 500 "Internal Server Error"
  • execute the ruby block add node to cluster list
    Recipe: graylog2::mongodb
• directory[/var/log/graylog2/mongodb] action create (up to date)
• directory[/var/opt/graylog2/data/mongodb] action create (up to date)
• directory[/opt/graylog2/sv/mongodb] action create (up to date)
• directory[/opt/graylog2/sv/mongodb/log] action create (up to date)
• directory[/opt/graylog2/sv/mongodb/log/main] action create (up to date)
• template[/opt/graylog2/sv/mongodb/run] action create (up to date)
• template[/opt/graylog2/sv/mongodb/log/run] action create (up to date)
• template[/var/log/graylog2/mongodb/config] action create (up to date)
• ruby_block[reload mongodb svlogd configuration] action nothing (skipped
  due to action :nothing)
• file[/opt/graylog2/sv/mongodb/down] action delete (up to date)
• link[/opt/graylog2/init/mongodb] action create (up to date)
• link[/opt/graylog2/service/mongodb] action create (up to date)
• ruby_block[supervise_mongodb_sleep] action run (skipped due to not_if)
• service[mongodb] action nothing (skipped due to action :nothing)
• execute[/opt/graylog2/embedded/bin/graylog2-ctl start mongodb] action
  run
  • execute /opt/graylog2/embedded/bin/graylog2-ctl start mongodb
    Recipe: graylog2::graylog2-server
• directory[/var/log/graylog2/server] action create (up to date)
• directory[/var/opt/graylog2/data/spool] action create (up to date)
• file[/var/opt/graylog2/graylog2-server-node-id] action create (up to
  date)
• template[/opt/graylog2/conf/graylog2.conf] action create

  • update content in file /opt/graylog2/conf/graylog2.conf from 22196d
    to 962c17
    --- /opt/graylog2/conf/graylog2.conf 2015-02-11
    11:20:35.006599671 -0500
    +++ /tmp/chef-rendered-template20150212-19893-11q6751 2015-02-12
    10:37:53.178290431 -0500
    @@ -229,7 +229,7 @@
    transport_email_auth_username =
    transport_email_auth_password =
    transport_email_subject_prefix = [graylog2]
    -transport_email_from_email = graylog2@graylog2
    +transport_email_from_email = graylog2@ash-vsrv-net01

    Specify and uncomment this if you want to include links to the

    stream in your stream alert mails.

    This should define the fully qualified base url to your web

    interface exactly the same way as it is accessed by your users.

• directory[/opt/graylog2/sv/graylog2-server] action create (up to date)
• directory[/opt/graylog2/sv/graylog2-server/log] action create (up to
  date)
• directory[/opt/graylog2/sv/graylog2-server/log/main] action create (up
  to date)
• template[/opt/graylog2/sv/graylog2-server/run] action create (up to
  date)
• template[/opt/graylog2/sv/graylog2-server/log/run] action create (up to
  date)
• template[/var/log/graylog2/server/config] action create (up to date)
• ruby_block[reload graylog2-server svlogd configuration] action nothing
  (skipped due to action :nothing)
• file[/opt/graylog2/sv/graylog2-server/down] action delete (up to date)
• link[/opt/graylog2/init/graylog2-server] action create (up to date)
• link[/opt/graylog2/service/graylog2-server] action create (up to date)
• ruby_block[supervise_graylog2-server_sleep] action run (skipped due to
  not_if)
• service[graylog2-server] action nothing (skipped due to action :nothing)
• execute[/opt/graylog2/embedded/bin/graylog2-ctl start graylog2-server]
  action run
  • execute /opt/graylog2/embedded/bin/graylog2-ctl start graylog2-server
• ruby_block[add node to server list] action run

Error executing action `run` on resource 'ruby_block[add node to server

list]'

Net::HTTPFatalError
-------------------
500 "Internal Server Error"

Cookbook Trace:
---------------
/opt/graylog2/embedded/cookbooks/graylog2/libraries/registry.rb:15:in

`set_master'

/opt/graylog2/embedded/cookbooks/graylog2/recipes/graylog2-server.rb:71:in
`block (2 levels) in from_file'

Resource Declaration:
---------------------
# In

/opt/graylog2/embedded/cookbooks/graylog2/recipes/graylog2-server.rb

 69: ruby_block "add node to server list" do
 70:   block do
 71:     $registry.set_master
 72:     $registry.add_gl2_server(node['ipaddress'])
 73:     $registry.add_es_node(node['ipaddress'])
 74:   end
 75: end

Compiled Resource:
------------------
# Declared in

/opt/graylog2/embedded/cookbooks/graylog2/recipes/graylog2-server.rb:69:in
`from_file'

ruby_block("add node to server list") do
  action "run"
  retries 0
  retry_delay 2
  default_guard_interpreter :default
  block_name "add node to server list"
  declared_type :ruby_block
  cookbook_name :graylog2
  recipe_name "graylog2-server"
  block #<Proc:0x00000004474650@

/opt/graylog2/embedded/cookbooks/graylog2/recipes/graylog2-server.rb:70>
end

Recipe: timezone-ii::debian

• bash[dpkg-reconfigure tzdata] action run
  • execute "bash" "/tmp/chef-script20150212-19893-1q3dy6w"
    Recipe: graylog2::graylog2-server
• service[graylog2-server] action restart
  • restart service service[graylog2-server]

Running handlers:
[2015-02-12T10:42:58-05:00] ERROR: Running exception handlers
Running handlers complete
[2015-02-12T10:42:58-05:00] ERROR: Exception handlers complete
[2015-02-12T10:42:58-05:00] FATAL: Stacktrace dumped to
/opt/graylog2/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 10 resources updated in 608.559446104 seconds
[2015-02-12T10:42:58-05:00] ERROR: ruby_block[add node to server list](graylog2::graylog2-server line 69) had an error: Net::HTTPFatalError: 500
"Internal Server Error"
[2015-02-12T10:42:58-05:00] FATAL: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)
root@ash-vsrv-net01:/#

On Thu Feb 12 2015 at 2:10:34 AM Marius Sturm [email protected]
wrote:

All your settings are perfectly fine and when I try to reproduce with the
same I dont get that error :/

I have improved the out out of the reconfigure run a little bit, could
you please replace the file
/opt/graylog2/embedded/cookbooks/graylog2/libraries/registry.rb with this
one
https://raw.githubusercontent.com/Graylog2/omnibus-graylog2/0.92/files/graylog2-cookbooks/graylog2/libraries/registry.rb

Afterwards re-run sudo graylog2-ctl reconfigure and then check the whole
output for stack trace or errors. Maybe there are some informations.

Did you change anything in the /var/opt/graylog2/data directory? Like
user rights or mount a bigger drive there or so?

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

from graylog2-images.

I guess I can't really solve this problem. From the error message it is now pretty clear that the etcd service is not running properly. That is a small daemon that stores IP addresses of the Graylog cluster.
You can talk to the daemon via HTTP and exactly this doesn't work on your box.

You can check if there are any errors in the logs of etcd with sudo graylog2-ctl tail etcd. If yes
try to restart the service with sudo graylog2-ctl restart etcd.

In doubt you have to delete and re-create the VM but as you said Graylog is running and only the reconfigure throws an error.

from graylog2-images.

Not a problem and thanks for ll the feedback you have provided. Its very
much appreciated.

On Thu Feb 12 2015 at 11:09:58 AM Marius Sturm [email protected]
wrote:

I guess I can't really solve this problem. From the error message it is
now pretty clear that the etcd service is not running properly. That is a
small daemon that stores IP addresses of the Graylog cluster.
You can talk to the daemon via HTTP and exactly this doesn't work on your
box.

You can check if there are any errors in the logs of etcd with sudo
graylog2-ctl tail etcd. If yes
try to restart the service with sudo graylog2-ctl restart etcd.

In doubt you have to delete and re-create the VM but as you said Graylog
is running and only the reconfigure throws an error.

—
Reply to this email directly or view it on GitHub
#28 (comment)
.

from graylog2-images.