e.g. /pages/installation.html#ubuntu-14-04

Instead of just linking the DEB/RPM packages which create the repository configuration on the local machine, the documentation should also list the actual URL to the repository and link to the GnuPG public key being used to sign the packages.

This simplifies setting up these repositories in configuration management products like Puppet, Chef, Ansible, etc.

We should keep all release and upgrade notes in here to make them accessible and to find them easier than digging through old blog articles.

In section
https://github.com/Graylog2/documentation/blob/f9076e3460a39fe3e3d5a59d8a294afc522d2053/pages/installation/aws.rst#https

sudo graylog-ctl https://<public ip>:443/api

Should be

sudo graylog-ctl set-external-ip https://<public ip>:443/api

Also would be nice to specify if it could be a hostname instead of the IP or not.

When using some proxy in front of Graylog, some configuration might be necessary in order to make the websockets loading metrics to work. By default they use the /a/metrics route. We need to add documentation for this.

Hi,

i graylog seems to have issues when running in docker for windws (hyperv vm). following errors occure as soon as i open the webinterface:

graylog_1        | 2016-06-08 14:03:37,111 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://docker.local:12900/system/metrics/multiple on node <8055968d-b53e-46c7-8153-ff9810633a27>, caught exception: docker.local (class java.net.UnknownHostException)
graylog_1        | 2016-06-08 14:03:39,113 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://docker.local:12900/system/metrics/multiple on node <8055968d-b53e-46c7-8153-ff9810633a27>, caught exception: docker.local (class java.net.UnknownHostException)
graylog_1        | 2016-06-08 14:03:41,235 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://docker.local:12900/system/metrics/multiple on node <8055968d-b53e-46c7-8153-ff9810633a27>, caught exception: docker.local: unknown error (class java.net.UnknownHostException)
graylog_1        | 2016-06-08 14:03:43,122 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://docker.local:12900/system/metrics/multiple on node <8055968d-b53e-46c7-8153-ff9810633a27>, caught exception: docker.local (class java.net.UnknownHostException)
graylog_1        | 2016-06-08 14:03:45,105 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://docker.local:12900/system/metrics/multiple on node <8055968d-b53e-46c7-8153-ff9810633a27>, caught exception: docker.local (class java.net.UnknownHostException)
graylog_1        | 2016-06-08 14:03:47,125 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://docker.local:12900/system/metrics/multiple on node <8055968d-b53e-46c7-8153-ff9810633a27>, caught exception: docker.local (class java.net.UnknownHostException)
graylog_1        | 2016-06-08 14:03:49,109 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://docker.local:12900/system/metrics/multiple on node <8055968d-b53e-46c7-8153-ff9810633a27>, caught exception: docker.local (class java.net.UnknownHostException)

the docker-compose file i am using:

mongo:
  image: "mongo:3"
elasticsearch:
  image: "elasticsearch:2"
  command: "elasticsearch -Des.cluster.name='graylog'"
graylog:
  image: graylog2/server:2.0.0-1
  environment:
    GRAYLOG_PASSWORD_SECRET: somepasswordpepper
    GRAYLOG_ROOT_PASSWORD_SHA2: 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
    GRAYLOG_REST_TRANSPORT_URI: http://docker.local:12900
  links:
    - mongo
    - elasticsearch
  ports:
    - "9000:9000"
    - "12900:12900"
    - "5555:5555/udp"

The site in general is running, but it seems that i cannot successfully add inputs.
Was looking through several bug forums, and as far as i could find out, the GRAYLOG_REST_TRANSPORT_URI: http://docker.local:12900 must be a url the browser can access (which it is...)

We should make a list for all the widgets in the documentation.

Network-based inputs have configuration options for setting the receive buffer size which is dependent on the system limits.

We should add an informational section to the documentation which explains how to set those limits, similar to https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html

http://docs.graylog.org/en/2.0/pages/installation/operating_system_packages.html

URL tells us the version is 2.0, but the download .deb link is 1.3, and I test it on Ubuntu 14.04 following those steps, installation fails.

• https://www.playframework.com/documentation/2.3.x/ConfiguringHttps (old web interface)
• https://www.digitalocean.com/community/tutorials/java-keytool-essentials-working-with-java-keystores
• https://github.com/Graylog2/graylog2-server/blob/2.0.0-alpha.3/misc/graylog.conf#L57-L69
• https://github.com/Graylog2/graylog2-server/blob/2.0.0-alpha.3/misc/graylog.conf#L91-L103

In 'The manual setup' section, am I wrong that there appears to be a big leap between downloading and untarring the graylog.tar file and adding to the conf and running it from there?

If I follow the steps word for word, I have a graylog folder in my home directory, a conf file at /etc/graylog/server/server.conf (the parent folders have nothing in them after /etc) and I'm starting the server by manually running a script in ~/graylog-1.0.1/bin

That's quite a dearth of information information there...and a none functioning setup? I'm more confused than anything that this is the full documentation on installing a well used application.

So then, moving onto the ubuntu 14.04 installation instructions, I run them as instructed and I am left with an apparently installed instance - but no init.d script is in existence and I can do service greylog-server and web start and they do something...but, I have no idea what is supposed to happen from here, the next section is about setting up elasticsearch, and then receiving logs.

I have no graylog-ctl, I have nothing at 127.0.0.1:9000

I'm usually pretty resourceful, but these seem so utterly confusing I have no idea what I'm meant to do. I originally followed this, and had something working, but followed the advice to update the application.
https://www.digitalocean.com/community/tutorials/how-to-install-graylog2-and-centralize-logs-on-ubuntu-14-04

Sadly there are no issues for v1 of graylog, all old versions. So this is even more confusing.

On top of that you say
"It is important to remember that the quick setup app is not meant to create production ready setups. We strongly recommend to use one of the other installation methods for a Graylog setup that is intended to run in production."
So I don't even see the point in that? Sorry to be pessimistic but I've spent hours trying to crawl through this documentation and I'm no where closer to having this installed. It was the same when installing older versions, really obscure dependencies buried in some stack exchange article from 2012.

In the README file of greylog-setup-1.0.0, in "Graylog Server" section following command is missing a "-" switch for XX:+CMSConcurrentMTEnabled.

java -Xms1g -Xmx1g -XX:PermSize=128m -XX:MaxPermSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djava.net.preferIPv4Stack=true -Djava.library.path=lib/sigar -jar graylog/graylog.jar server -p run/graylog.pid -f conf/graylog.conf

In documentation/pages/installation/docker.rst "graylog-server" image on Docker-hub not-exist.

The https section in the aws installation has the following command:

sudo graylog-ctl https://<public ip>:443/api

But that command seems not to be valid. It results in: "I don't know that command."

I am running graylog version 2

The section of the document titled "Create a Stream Rule" reads as:
"Pick the Syslog UDP Input, and click Add stream rule."
[Image for picking the item & add stream rule button.]

Followed by:
"Then, type in the values shown below and hit save.

Then click I’m done!

We have just configured this stream to process in real time all the messages that come in from the security/authorization facility.

Now let’s create the alert."

The issue is that, at the line stating, "Then, type in the values show below and hit save", there is no image or values given.

I'm very much a novice & new to Graylog, but I tried several times to glean the information from text & sections following & didn't find that information to create the initial/test stream rule, in that location...

Refs Graylog2/graylog2-server#2343
Refs Graylog2/graylog2-server#2284

See https://www.loggly.com/blog/five-invaluable-techniques-to-improve-regex-performance/ for a good example.

https://www.elastic.co/guide/en/elasticsearch/reference/2.3/query-dsl-query-string-query.html#query-string-syntax

There are several new Kafka-based inputs in Graylog 1.1.x which need to be documented.

Refs Graylog2/graylog2-server#322 and Graylog2/graylog2-server#1165

I am logging this issue from "make latexpdf". This is on Fedora 22 OS with texlive packages all installed.

Chapter 5.
[43] 
  [44]
Underfull \hbox (badness 10000) in paragraph at lines 2376--2377
[]\T1/ptm/m/n/10 Change into the ex-tracted col-lec-tor di-rec-tory and cre-ate
 a col-lec-tor con-fig-u-ra-tion file in

  [45 <./coll
ector_win_install_1.png>] [46 <./collector_win_install_2.png>]

 
Underfull \hbox (badness 10000) in paragraph at lines 2432--2433
[]\T1/ptm/m/n/10 If you choose the op-er-at-ing sys-tem in-stal-la-tion method,
 the con-fig-u-ra-tion file de-faults to
[47 <./collector_win_install_3.png>]
Underfull \hbox (badness 10000) in paragraph at lines 2506--2507
[]\T1/ptm/m/n/10 Please make sure to es-cape the \T1/pcr/m/n/10 \ \T1/ptm/m/n/1
0 char-ac-ter in Win-dows paths: \T1/pcr/m/n/10 path = "C:\\Program

Underfull \hbox (badness 10000) in paragraph at lines 2513--2514
[]\T1/ptm/m/n/10 Please make sure to es-cape the \T1/pcr/m/n/10 \ \T1/ptm/m/n/1
0 char-ac-ter in Win-dows paths: \T1/pcr/m/n/10 path = "C:\\Program
[48]
Overfull \hbox (29.49223pt too wide) in paragraph at lines 2554--2555
[]\T1/ptm/m/n/10 A usual glob/wildcard string you know from other tools might b
e \T1/pcr/m/n/10 /var/log/apache2/**/*.{access,error}.log\T1/ptm/m/n/10 .

Underfull \hbox (badness 10000) in paragraph at lines 2554--2555
\T1/ptm/m/n/10 This means you are in-ter-ested in all log files which names end
 with \T1/pcr/m/n/10 .access.log \T1/ptm/m/n/10 or

Underfull \hbox (badness 10000) in paragraph at lines 2554--2555
\T1/pcr/m/n/10 .error.log \T1/ptm/m/n/10 and which are in a sub di-rec-tory of
\T1/pcr/m/n/10 /var/log/apache2\T1/ptm/m/n/10 . Ex-am-ple:
[49] [50] [51] [52]

! Package inputenc Error: Unicode char \u8:✔ not set up for use with LaTeX.

See the inputenc package documentation for explanation.
Type  H   for immediate help.
 ...

l.2877 \hline\end{tabulary}

?

Reference: http://tex.stackexchange.com/questions/83440/inputenc-error-unicode-char-u8-not-set-up-for-use-with-latex

There are several implementation details missing in the GELF specification, e. g. that using GELF via TCP mandates terminating messages with a null character (\0) or that compression cannot be used (due to the null character being used for message framing).

It might be good to refactor the documentation a bit, and to divide information a bit like

• Installing: "What takes to get it running"
• Configuring: "How to configure Graylog to accomplish stuff"
• Securing: "How to harden the setup"

Anyways, Securing Graylog should collect together for example

• Enabling audit logging of usage (I will make a pull request of this very shortly)
• Network traffic profile (internal in cluster, outside, firewall requirements, etc)
• Configuring for cryptographic compliance (this part exists, it's brand new)
• Configuring "real" certificates for REST API and inputs
• How to add a legal disclaimer to login page, and to footer (I am using Apache's mod_substitute, but it's a hack)
• How to configure authentication (the pluggable authentication is in development, I'm especially interested in kerberos/GSSAPI and TLS client certificate authentication)
• What security maintenance tasks should be performed in production environments
• Other pieces of security related configuration that are all around the current documentation set

• https://www.loggly.com/blog/nine-tips-configuring-elasticsearch-for-high-performance/

The installation docs are currently hardcoded to 0.92 versions. The old docs had a macro that was used and we should have that again.

https://t37.net/how-to-fix-your-elasticsearch-cluster-stuck-in-initializing-shards-mode.html

http://docs.graylog.org/en/1.3/pages/installation/graylog_ctl.html#configuration-commands

sudo graylog-ctl set-email-config <smtp server> [–port=<smtp port> –user=<username> –password=<password> –no-tls –no-ssl] 

For at least the port and no-tls and no-ssl parameters they seem to require two dashes instead of one.

we only have some minor information about how to raise heap for elasticsearch but not how this could be done for graylog server and why.

The Graylog 2.0 documentation requires Elasticsearch 2.1.x or later while it also advises to add script.disable_dynamic: true to the elasticsearch.yml file here. I did that using Elasticsearch 2.3.2 and it resulted in error:

Exception in thread "main" java.lang.IllegalArgumentException: script.disable_dynamic is not a supported setting, replace with fine-grained script settings. 
Dynamic scripts can be enabled for all languages and all operations by replacing `script.disable_dynamic: false` with `script.inline: on` and `script.indexed: on` in elasticsearch.yml

This issue says that such an error is thrown from ElasticSearch 2.x branch up. ElasticSearch docs on this topic are here. I am not sure if setting the most secure options:

script.inline: false
script.indexed: false
script.file: false

won't hinder Graylog communication to ElasticSearch, so I can't suggest specific values, but definitely that part of Graylog documentation should be updated.

For me it would be enough if the documentation said: "Make sure to add script.disable_dynamic: true to the elasticsearch.yml file if you use ElasticSearch <2.0 ..."

hello,
I have a problem when I use graylog
Is that when I want to use the data type conversion when there was a problem.
This page has introduced about how to make data type conversion, http://docs.graylog.org/en/2.0/pages/extractors.html
The page wrote:
Grok directly supports converting field values by adding ;datatype at the end of the pattern, like:

len=%{NUMBER:length;int} src=%{IP:srcip} sport=%{NUMBER:srcport} dst=%{IP:dstip} dport=%{NUMBER:dstport}

I write configuration is as follows:
%{NUMBER:byes;int}

but,I found the conversion is not successful !!!
The field is not of type int.
When I try to use the "Generate chart" drawing, tip:
Could not create field graph
Field graphs are only available for numeric fields.

I don't know what to do, I don't know why can't the converted format correct drawing.
I need your help! Please

Links like http://readthedocs.org/projects/graylog2-docs/downloads/epub/2.0/ point to 404. Documentation for 1.3 is correctly downloadable (which is awesome!)

http://docs.graylog.org/en/latest/pages/architecture.html#highly-available-setup-with-graylog-radio

When going to system/nodes/available input types in Graylog there is a Kafka documentation link linking to the input overview http://docs.graylog.org/en/1.0/pages/sending_data.html#json-path-from-http-api-input

I can not find any documentation on Kafka inputs at all. This either needs to be written or the link should be removed.

In the docs

every field you send and prefix with a _ (underscore) will be treated as an additional field. Allowed characters in field names are any word character (letter, number, underscore), dashes and dots. The verifying regular expression is: ^[w.-]*$

This is not what that regular expression does. Or maybe I'm misunderstanding something? Did you mean \w\.-? Also this regex does not force the initial underscores

I would be beneficial to have an official description of how to backup the graylog configuration that is persisted into mongodb ...

As a Graylog user would like to see better LDAP documentation. When I search for the word LDAP or Active Directory, I don't see any documentation that describes how to configure it etc.

Hi

I like to understand if this project will accept pull request for graylog related diagrams draw by tool like Inkscape,Dia ...

On several pages of the live documentation on docs.graylog.org double hypens "--" in source blocks are being rendered as an en dash "–" instead which looks like just a single hyphen, and when copied and pasted into a terminal is copied over as a single hyphen. The documentation source on Github correctly has double hyphens, rendering the documentation source with Sphinx correctly renders double hyphens, so not a problem with the source or config. It appears this is a known bug, and the live site is likely using an outdated version of smartypants and just needs to update their sphinx install.

Some examples of en dashes instead of double hyphens from http://docs.graylog.org/en/1.3/pages/installation/graylog_ctl.html

Live Site:
sudo graylog-ctl set-email-config <smtp server> [–port=<smtp port> –user=<username> –password=<password> –no-tls –no-ssl]
Should Read:
sudo graylog-ctl set-email-config <smtp server> [--port=<smtp port> --user=<username> --password=<password> --no-tls --no-ssl]

Live Site:

sudo parted -a optimal – /dev/sdb unit \
compact mkpart primary ext3 “1” “-1”

Should Read:

sudo parted -a optimal -- /dev/sdb unit \
compact mkpart primary ext3 "1" "-1"

Hi
When I send a log with default level { 0 or 5} it shows different log level in Graylog-web
Note

we use ruby module -- gelf notifier 

Now that you have released v1 of your product (congrats for that btw) I would really appreciate some official documentation for writing plugins. I have by now somehow managed to create all standard plugin types, but really would like to know the official way on this ought to be done.
One of my customers really complains that it is currently still very hard to start from scratch - especially for input (yes, there are examples by now) and rest resource (where I could not find an example) it was quite hard to find out.
In addition it would be beneficial to get more information about the built-in transports and codecs.
Or I can offer to write down something (already have some parts) if you tell me what you expect.
Documentation is key to get a community adding plugins so you can better compete with other products like splunk and logstash.
Otherwise just keep up the good work! Looking forward to seeing Graylog evolve! Ronald

Change http://docs.graylog.org/ to redirect always to the latest documentation.

https://gist.github.com/joschi/72fb7e75b171c10d3717

Looking at the current documentation for 1.3, I couldn't find anything regarding AND/OR stream rules. We should add some documentation for that.

Hi,

The only mention of configuring ES in the docker page is that one can pass in an ES_MEMORY field
http://docs.graylog.org/en/1.1/pages/installation/docker.html

I've started a docker Graylog2, and am getting the 'Elasticsearch nodes with too low open file limit' message. I can see there is a page for configuring ES:
http://docs.graylog.org/en/1.1/pages/configuring_es.html

But I'm not sure where elasticsearch.yml or graylog.conf is stored. Docker seems to be a black box. But maybe I'm misunderstanding docker. Must I 'go into' the container somehow, to find these files?

Is the docker install a viable production option? "We strongly recommend to use a dedicated Elasticsearch cluster for your Graylog setup." - (I'm using the one that comes with the docker file, so I presume I'm ok?).

Regards
Daniel

Pls add the note that there is the possibility to use mathematical functions in drools rules.

Bcs there are some users who want this feature but in Graylog itself, so they can get the field values in drools and add a new one with the outcome of the mathematical function.

https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping.html
https://www.elastic.co/guide/en/elasticsearch/reference/2.3/indices-templates.html