Hello, thanks for your feedback about #20 and supporting that feature.

i had tested your build option for solarisOS and i successfully finished making binary file from your project.

but i found that service couldn't be started while executing binary.

the error message was "Service command not found" something like that, and i assume that github.com/kardianos/service causes the problem.

the project says "Currently supports Windows XP+, Linux/(systemd | Upstart | SysV), and OSX/Launchd" and there no comment for solarisOS.

thank you.

Problem description

nxlog CE (as of 2.9.17) does not currently send milliseconds in the GELF timestamp field, so Graylog shows something like 2016-08-23 13:50:58.000. Without doing some special handling (e.g. we're extracting the full timestamp from the message on the Graylog side and overwriting timestamp) messages are out of order within a 1 second window. It may save others some time if this limitation is mentioned in the Collector Sidecar documentation.

See: https://nxlog.co/question/1855/gelf-timestamp-field-missing-millisecond-precision

Problem description

It would be nice to be able to include spinets in the inputs in graylog. like i show below, that way get can pull out some basic info like timestamps to start with.

example config

<Input in>
        Module  im_file
        File    "C:\\tmp\\\\example-log.txt"
        SavePos  TRUE
        Recursive TRUE
        InputType       multiline
        exec if $raw_event =~ /(?:\{"([0-9]+?), ([0-9]+?), "(.+?)", "(.+?)", "(?:.+?)","(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)New\sLogon:\s*(?:.+?)\n\s*Account\sName:\s*(.+?)\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)Source\sNetwork\sAddress:\s*([0-9\.]{7,15})\n(?:[a-zA-Z0-9:,"\n\r\s\f\t\-\{\}\.\(\)]+?)"\})/g; 

    $timestamp = $1; 
    $event = $2; 
    $status = $3; 
    $type = $4; 
    $short = $5; 
    $user = $6; 
    $source = $7; 

</Input>

Problem description

Using Graylog Collector Sidecar to configure NXLog on a CentOS 6.5 host.
Here is the resulting generated output in /etc/graylog/collector-sidecar/generated/nxlog.conf

<Output 57605344ed60c703b0fa7290>
Module om_tcp
Host scmgraylog-server.ghx.com
Port 12201
OutputType GELF_TCP
Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
Exec $gl2_source_collector = '6f145fef-5841-49ad-8b36-80c4a69a1568';
Exec $Hostname = hostname_fqdn();

Unfortunately, the Hostname gets set to localhost.localdomain on this host which makes searching by source in Graylog a little difficult. I think if $Hostname could be set to hostname() rather than hostname_fqdn() in the above generated file, the problem would be solved. Even if hostname_fqdn() returned the "real" FQDN of this host, I would rather have the option to set to the shortname.

Is this possible? Something I can fix with a Snippet?

Environment

• Sidecar Version: 0.0.8
• Graylog Version: 2.0.2
• Operating System: CentOS 6.5

Thanks.

as far as i am aware nxlog comes with its own log rotation engine. wouldn't it be nice if one could configure this via the sidecar?

Environment

• Sidecar Version: 0.5.0
• Graylog Version: 2.0.0 rc1

Problem description

Steps to reproduce the problem

1.sudo nxlog -v -f /etc/graylog/collector-sidecar/generated/nxlog.conf
2016-06-28 08:15:56 INFO configuration OK
2.tail -f /var/log/messages
Jun 28 08:13:46 M3 /usr/bin/graylog-collector-sidecar[17473]: time="2016-06-28T08:13:46Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
Jun 28 08:13:56 M3 /usr/bin/graylog-collector-sidecar[17473]: time="2016-06-28T08:13:56Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
Jun 28 08:14:06 M3 /usr/bin/graylog-collector-sidecar[17473]: time="2016-06-28T08:14:06Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
Jun 28 08:14:16 M3 /usr/bin/graylog-collector-sidecar[17473]: time="2016-06-28T08:14:16Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
Jun 28 08:14:26 M3 /usr/bin/graylog-collector-sidecar[17473]: time="2016-06-28T08:14:26Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
Jun 28 08:14:36 M3 /usr/bin/graylog-collector-sidecar[17473]: time="2016-06-28T08:14:36Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
Jun 28 08:14:46 M3 /usr/bin/graylog-collector-sidecar[17473]: time="2016-06-28T08:14:46Z" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
3.[root@M3 neal]$cat /etc/graylog/collector-sidecar/collector_sidecar.yml
server_url: http://172.15.5.174:12900
tls_skip_verify: false
node_id: graylog-collector-sidecar
collector_id: file:/etc/graylog/collector-sidecar/collector-id
tags:
- linux
- apache
log_path: /var/log/graylog/collector-sidecar
update_interval: 10
backends:
- name: nxlog
enabled: true
binary_path: /usr/bin/nxlog
configuration_path: /etc/graylog/collector-sidecar/generated/nxlog.conf

4.cat /etc/graylog/collector-sidecar/generated/nxlog.conf

This is a sample configuration file. See the nxlog reference manual about the

configuration options. It should be installed locally under

/usr/share/doc/nxlog-ce/ and is also available online at

http://nxlog.org/docs

Global directives

User root
Group root

LogFile /var/log/nxlog/nxlog.log
LogLevel INFO

Modules

Thankfully the Filename field was added in the release the other day, very useful, but it would be very nice to be able to add other arbitrary fields to an input to be able to tell them apart.

There's currently very little to be able to tell where messages came from, other than maintaining a all messages per host, and knowing what they do. Things like Application Name and Environment would save a lot of time and custom configuration.

The default configuration written for nxlog puts the modules directory as /usr/lib/nxlog/modules but my CentOS 6.6 installation has it in /var/libexec/nxlog/modules. If I change this and run nxlog manually to confirm then all is well.

Maybe this can be solved if there's a default nxlog config that could be pulled at startup to override these thing, binary location and UID/GID I imagine ought to be configurable too.

Problem description

Any trailing path in collector-sidecar's server_url is ignored. This prevents the use of a single port for Graylog Server web and API services on a proxy frontend.

Steps to reproduce the problem

1. Proxy web and API requests to Graylog Server through a single port, with the API available at "/api" (set with web_endpoint_uri in graylog-server).
2. Set the server_url for collector-sidecar to graylog-server's web_endpoint_uri (e.g., https://wat.tld/api)
3. collector-sidecar will send GET and PUT requests to https://wat.tld/ instead of https://wat.tld/api/.

Environment

• Sidecar Version: 0.0.8-1
• Graylog Version: 2.0.2-1
• Operating System: CentOS 7 x86_64
• Elasticsearch Version: 2.3.3-1
• MongoDB Version: 3.2.6-1

Problem description

I'm using nxlog and I've installed the graylog sidecar. I'm manually starting it with my configuration file so I can monitor it. Just after reporting that nxlog is starting it gives a 400 error related to the property tags.

Steps to reproduce the problem

1. Install nxlog
2. Install sidecar
3. Configure sidecar per graylog docs.
4. Manually start sidecar in the foreground so I can monitor
5. Results in error

yml config below:
collector_sidecar.txt

screenshot of error:

Environment

• Sidecar Version: 0.9-alpha-1
• Graylog Version: 2.0.3
• Operating System: Centos 6.8 x64
• Elasticsearch Version: 2.3.4-1
• MongoDB Version: 3.2.8-1

Problem description

When restarting the collector-sidecar service all monitored logs are completly resent to the graylog server when the "Read since start" checkbox is unchecked in the input section of the collector configuration whether or not you have "Save read position" checked.

I could be mistaken on the intent, but I would assume that checking "Save read position" and unchecking "Read since start" would send the entire log file on the first run and then only send changes from there on out.

When "read since start" is unchecked it appears that "save read position" has no effect.

Steps to reproduce the problem

In the collector configuration, inputs section.

1)Uncheck "Read since start"
2)Check "Save read position"
3)Restart the collector-sidecar service

The entire log file is sent every time the service is restarted.

Environment

• Sidecar Version: 0.0.8
• Graylog Version: 2.0.3
• Operating System: Centos 6
• Elasticsearch Version: 2.3
• MongoDB Version: 3.2.8

Problem description

generated nxlog.conf parse the collector-id including line break char

Steps to reproduce the problem

collector-id will add 0x0a at end after edited by vi/vim/nano

Environment

Graylog 2.0.1 (81e0187) on graylog (Oracle Corporation 1.8.0_77 on Linux 4.2.0-36-generic)

Problem description

On some of my collectors the "Show Messages" button that then querys the collector ID returns to resutls. I can server for the server the collector is installed on as the source and I see messages so I know the collector is working. I am using the NXLog backend and there seems to be no pattern bewteen the servers where the command works and where it doesnt. I cannot find any errors/messages on either server or client indicating a problem.

Steps to reproduce the problem

1. Install graylog collector sidecarwith nxlog as backend
2. send in logs and then find that you cannot view logs by collector ID.

Environment

• Sidecar Version: 0.8
• Graylog Version: 2.0.2
• Operating System: windows server 2012 (sidecar)/ CENTOS7 (Graylog)
• Elasticsearch Version: 2.3.2
• MongoDB Version: 3.2.6

Problem description

Inability to add "tail_files" flag to filebeats inputs in sidecar configuration.

While collecting large number of large log files that rotate hourly, after starting up the sidecar + filebeats, it burns down the whole GrayLog infrastructure. It starts to send huge amounts of data to graylog server (20k messages per second is a maximum our datastore is able to handle in testing environments, 40k on production) and causes graylog to be inaccessible untill it handles all the messages from all logfiles.

Steps to reproduce the problem

1. Collect large amount of logfiles with large number of entries.
2. Start up sidecar + collector
3. Profit...I mean, watch the infrastructure and data store burn :)

Environment

• Sidecar Version: 1.1.0 beta 1
• Graylog Version: 2.1.0 beta 1
• Operating System: RedHat 7
• Elasticsearch Version: 2.2.0
• MongoDB Version: 2.7

Problem description

nxlogs fill up your harddrive pretty quick with 2016-05-26 12:11:04 ERROR failed to open directory: /var/www/xxxxx: Permission denied entries when using a recursive file-input

Steps to reproduce the problem

define a collector entry with a wildcard-entry (/var/www/*.log) and enable recursion.

if nxlog can't access this path it puts an error into nxlog as well as in nxlog_stderr.log.

if you do this with some larger structure this can lead to logfiles using 60gb within 24 hours. so even logrotation doesnt help you much.

even if this is basically my own mistake it shouldnt break the system by filling its harddisks...

Environment

• Sidecar Version: 0.0.7
• Graylog Version: 2.0.1
• Operating System: ubuntu 14.04

Problem description

When I attempt to setup my collector_sidecar.yml file without a node_id entered graylog collector fails to start. I am given this error message when running the collector in debug mode, "Please provide a vaild node-id." I am also curious if there is something I do not understand about GitHub commits. I am using the installation for 0.0.8 downloaded from this link, https://github.com/Graylog2/collector-sidecar/releases, but it looks like that was released before the commit.

Steps to reproduce the problem

1. Install collector-sidecar on windows.
2. remove the node id from the collector-sidecar.yml file
3. attempt to start graylog collector

Environment

• Sidecar Version: 0.0.8
• Graylog Version: 2.0.2
• Operating System: windows 2003/2008/2012
• Elasticsearch Version:
• MongoDB Version:

This is the yml file I am using. I have also tried with the node id line removed completely.

############ collector_sidecar.yml

server_url: http://192.168.9.25:12900
collector_id: file:C:\Program Files\graylog\collector-sidecar\collector-id
node_id:
tags:
- IIS2003
log_path: C:\Program Files\graylog\collector-sidecar
update_interval: 10
backends:
- name: nxlog
enabled: true
binary_path: C:\Program Files\nxlog\nxlog.exe
configuration_path: C:\Program Files\nxlog\conf\nxlog.conf

##########END

Problem description

generated nxlog.conf parse the collector-id including line break char

Steps to reproduce the problem

collector-id will add 0x0a at end after edited by vi/vim/nano

Environment

Graylog 2.0.1 (81e0187) on graylog (Oracle Corporation 1.8.0_77 on Linux 4.2.0-36-generic)

Problem description

The graylog sidecar service eats up all ports, when listing ports using netstat the output is filled with:

Proto Local Address Foreign Address State
TCP x.x.x.x:52991 x.x.x.x:80 ESTABLISHED

this list continues to grow until all ports are exhausted.

The foreign address correlates with the API endpoint.

I have the same issue on all the machines i installed the Graylog Sidecar on.

Steps to reproduce the problem

Install Graylog collector, start service, wait.

Environment

• Sidecar Version: Graylog Collector Sidecar version 0.0.9 (0.0.9-beta.2)
• Graylog Version: Graylog 2.0.3 (f07c170) (Oracle Corporation 1.8.0_101 on Linux 4.4.0-31-generic)
• Operating System: Windows Server 2012 R2

Hi,
Is it possible to have windows installer file in msi format ?
It's more usefull to deploy with Active Directory.
Thank you

Problem description

To reduce the steps a user has to made till the Sidecar is ready to run, we could create a dialog in the Windows installer to ask for the most basic settings every installation needs. For most installations no further configuration would be needed.

From graylog-labs/graylog2-web-interface#1734.

@hryzec wrote:

After change of collector configuration input using options "Read since start" & "Save read position" Gaylog server seems to receive all the log messages at once causing Graylog to fill in disk journal and hang the processing until messages are processed. This however should not happen as it is expected from collector and nxLog to not process messages older than collector restart and older than collector read position.

Problem description

When using Sidecar Collector to centrally manage nxlog configuration files no logs are sent.

Steps to reproduce the problem

1. Deploy Graylog 2.0.3 ova
2. Create configuration and tag for Windows logs
3. Install Windows Server 2012R2
4. Disable firewall on server
5. Install NXlogCE 2.9.1504.
6. Uninstall NXlog service as per instructions
7. Install Collector Sidecar 0.0.8 x64
8. Install Collector Sidecar service as per instructions
9. Configure collector_sidecar.yml with the relevant ip address and tag as deployed in steps 1 and 2
10. Start Collector Sidecar service
11. Verify that the relevant nxlog.conf file is generated in /generated

No logs are shipped - this can be seen on a Wireshark capture on the Windows server.

1. copy the contents of /generated/nxlog.conf into /nxlog/conf/nxlog.conf
2. install the NXlog service and start it

No logs are shipped, again this can be seen in Wireshark

1. Stop the NXlog service, and edit /nxlog/conf/nxlog.conf. Change the GUID for the input and output to "in" and "out" respectively. Change the route in the same manner.
2. Start the NXlog service.

Logs are now shipped.

Environment

• Sidecar Version: 0.0.8
• Graylog Version: 2.0.3
• Operating System: Ubuntu
• Elasticsearch Version: ? (version in prebuilt OVA)
• MongoDB Version: ? (version in prebuilt OVA)

I can't find any sign of RPM packages for RHEL 6/7 are they coming soon? Building from source really isn't a good option for me.

https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#loadbalance

Environment

• Sidecar Version: 0.0.9-1
• Graylog Version: 2.1.1
• Operating System: Ubuntu 16.04

Problem description

In my opinion it makes sense to show centralized the tags which i add in the yml file in the "Collectors in Cluster" overview.
Now i have to check each of my machines.

Would it be possible to rework the inputs / outputs interface for just a generalized "options" text area? My use case is below. I realize I could do this with a query statement but NXLog's own functions make this easier and having an options area I could really do either/or since it would just place my text at this location. I could also see writing more advanced things here without having to resort to using a snippet for a full configuration. I hope I have explained this properly... Thoughts?

<Input WindowsEventLog>
    Module im_msvistalog
    PollInterval 1
    SavePos True
    ReadFromLast True
       Exec if ($Channel == "Security") drop();     <-- An options textarea that would place options here
</Input>

Problem description

We are trying to adjust graylog to our project and we think it's pretty helpful for us.

i think the collector-sidecar is great idea to gathering logs on the nodes file system.

you are providing packages for both and fortunately most developers are using ubuntu or centos .

i hopefully ask you to provide package for the solaris os.

actually we tried to build from source on the solaris os , it resulted to fail.

thank you.

Steps to reproduce the problem

1. ...

Environment

• Sidecar Version: 0.0.7
• Graylog Version: 2.0 rc
• Operating System: solaris
• Elasticsearch Version: 2.x
• MongoDB Version: 3.1.x

Problem description

Will Logstash backend be enhance? because nxlog backend configured by Collectors can not extract log at log client.

Problem description

winlogbeats configuraton for TLS output includes an empty certificate_authorities entry that causes it to fail. Excluding the CA entry when no CA file is configured will fix this issue

output:
  logstash:
    hosts:
    - graylog.example.com:5044
    tls:
      certificate_authorities:
      - ""
      insecure: true

Steps to reproduce the problem

1. Configure wnlogbeats collector output with "Enable TLS support" and "Insecure TLS connection" both enabled

Environment

• Sidecar Version: 0.0.9 (Windows amd64)
• Graylog Version: 2.1.0-1
• Operating System: Linux
• Elasticsearch Version:
• MongoDB Version:

Problem description

I added 2 tags (linux, apache) to the .yml file. These two i configured in graylog with 2 configurations (each tag one configuration) and i let each read a file. this works (i could see the content of both logs in Graylog) but i get every 10 seconds a syslog entry:

2016-05-18 14:51:10.000 test-debian
May 18 14:51:09 test-debian /usr/bin/graylog-collector-sidecar[3389]: time="2016-05-18T14:51:09+02:00" level=info msg="[RequestConfiguration] No configuration found for configured tags!"

2016-05-18 14:51:10.000 test-debian
May 18 14:51:09 test-debian graylog-collector-sidecar[3389]: time="2016-05-18T14:51:09+02:00" level=info msg="[RequestConfiguration] No configuration found for configured tags!"

Why?? What does it mean?

What is the Syntax in yml file for more than one tag?

tags:
- apache linux

or

tags:
- apache, linux

or

tags:
- apache
- linux

yaml sytax should be the last one with dash but log tells me that the config file is corrupt.
level=error msg="[nxlog] Error during configuration validation: exit status 1

Environment

• Sidecar Version: newest, 0.0.7
• Graylog Version: newest, 2.0.1
• Operating System: Debian Jessie
• Elasticsearch Version: 2.3

Problem description

Using multiple tags, e.g. one for logging a generic CentOS log files and another for logging Nginx logs doesn't work:

Jul 01 10:31:07 i-4cd9dfbf graylog-collector-sidecar[17877]: time="2016-07-01T10:31:07+03:00" level=error msg="[nxlog] Error during configuration validation: exit status 1"
Jul 01 10:31:07 i-4cd9dfbf graylog-collector-sidecar[17877]: time="2016-07-01T10:31:07+03:00" level=info msg="[nxlog] Collector configuration file is not valid, waiting for update..."

Nxlog itself does seem to think that configuration is OK:

[root@i-4cd9dfbf-goserver ~]# nxlog -v -f /etc/graylog/collector-sidecar/generated/nxlog.conf
2016-07-01 10:41:44 INFO configuration OK

Deleting either one of the configurations having a tag defined gets logging working again. Am I configuring the tags somehow incorrectly, each different tag I have is configured only in their own configuration - so each configuration has only one tag?

Attached the nxlog.conf from both cases, one configuration and one tag, and 2 configurations with one tag in each.
nxlog_one_tag.conf.txt
nxlog_multiple_tags.conf.txt

Steps to reproduce the problem

1. Define 2 tags in collector_sidecar.yml
2. Create 2 configurations in Graylog, each having one the tags defined in step 1
3. Logging doesn't work since sidecar thinks the Nxlog configuration is broken although Nxlog's own validation says the config is OK (there is some duplicated config options in the config file when multiple configurations/tags is setup)

Environment

• Sidecar Version: 0.0.8
• Graylog Version: 2.0.3
• Operating System: CentOS 7.2
• Elasticsearch Version: 2.3.3
• MongoDB Version: 2.6.12

if you have to define several rather similar inputs for a collector a clone (or copy) button would be nice.
i know there are wildcards but sometimes this simply does not cut the mustard.

Environment

• Sidecar Version: 0.5.0
• Graylog Version: 2.0.0 rc1

I installed the NXLOG Ubuntu package and tried to run the sidecar with -collector-path /usr/bin/nxlog.

This generates the following nxlog.conf:

define ROOT /usr/bin/nxlog
Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

Two problems:

• It uses the wrong path separator on Linux. (\)
• The ROOT is wrong for the following settings.

The separator issue should be easy to fix, but the second issue needs more thinking. When NXLOG gets installed via the DEB package, the directories are all over the place.

• The modules are in /usr/lib/nxlog/modules
• The binary is in /usr/bin/nxlog

The CacheDir, Pidfile, SpoolDir and LogFile directives should probably point to a configurable root directory. The binary path and the Moduledir should probably be separate settings. Maybe we also need something that detects the module dir etc.

Problem description

I added 2 tags (linux, apache) to the .yml file. These two i configured in graylog with 2 configurations (each tag one configuration) and i let each read a file. this works but i get every 10 seconds a syslog entry:

2016-05-18 14:51:10.000 test-debian
May 18 14:51:09 test-debian /usr/bin/graylog-collector-sidecar[3389]: time="2016-05-18T14:51:09+02:00" level=info msg="[RequestConfiguration] No configuration found for configured tags!"
2016-05-18 14:51:10.000 test-debian
May 18 14:51:09 test-debian graylog-collector-sidecar[3389]: time="2016-05-18T14:51:09+02:00" level=info msg="[RequestConfiguration] No configuration found for configured tags!"

What does it mean?

What is the Syntax in yml file for more than one tag?

tags:
- apache linux

or

tags:
- apache, linux

or

tags:
- apache
- linux

Steps to reproduce the problem

1. ...

Environment

• Sidecar Version: newest
• Graylog Version: newest
• Operating System: Debian Jessie
• Elasticsearch Version: 2.3
• MongoDB Version:

Adding Fields to systems not only Input Files

Need to add a Static Field to a Host, not only a Input

Steps to reproduce

At the moment you would have to add the Field to every InputFile you define for a Host, (eg. environment=development)
tedious in large environments with multiple Microservices per Host.

Suggestions

• Same functionality as in fileInputsToString() for outputsToString() So you could add "global" Fields
• Automatically add the tags from sidecar.yml as static fields

Environment

• Sidecar Version: 0.8
• Graylog Version: 2.0.2 (4da1379)
• Operating System: Ubuntu 14.04

Problem description

32bit/64bit versions of the Windows installer should use C:\Program Files as target, not C:\Program Files (x86).

Problem description

Having the backend process crash or beeing killed is not deteced by the sidecar service.
Leading to Lost logs

Steps to reproduce the problem

1. start sidecar service
2. stop backendprocess (killall nxlog)

Environment

• Sidecar Version: 0.8
• Graylog Version: 2.0.2 (4da1379)
• Operating System: Ubuntu 14.04

This may be related and could be solved by #13 but may need it's own solution.

Currenlty NXLog uses a short hostname when reporting which I am assuming gets used as the Source field in graylog, this is causing me some problems as the short hostnames are not unique across all my collectors. I currently can't see a way to work around it. NXLog does not let you configure this as a global option and suggests you override it by setting Exec $hostname... but as described in #13, I can't affect the content in these blocks.

It may be worth having a "snippets" section that will be included in each input for fields that you want to include in all.

A short term solution could be to just place Exec $Hostname = hostname_fqdn(); in all inputs, which I think should be the default anyway.

Problem description

while other regex' are noted without the surrounding slashes the multiline regex needs the slashes since nxlog wants them

Steps to reproduce the problem

define collector configuration
define output
define collector input using nxfile input
enable multiline
add a regex of your choice (without slashes)
save the input

examine the nxlog-logfile (usually /var/log/graylog/collector-sidecar/nxlog.log) for something like

2016-06-14 15:24:44 ERROR couldn't parse expression at line 11, character 14 in /etc/graylog/collector-sidecar/generated/nxlog.conf;syntax error, unexpected ^, expecting $end
2016-06-14 15:24:44 ERROR invalid expression in 'HeaderLine' at /etc/graylog/collector-sidecar/generated/nxlog.conf:11

as soon as you enclose the regex in slashes it works. but this is somewhat inconsistent to the other regex's used inside graylog. maybe the collector sidecar should do this for the user or there could at least be a hint inside the webinterface.

Environment

• Sidecar Version: 0.0.7 / 0.0.8
• Graylog Version: 2.0.1
• Operating System: Ubuntu 14.04

When the log or config don't exist, Sidecar should try to create them.

When creating a configuration for collector-sidecar with a file input type, it'd be good to be able to populate a field that stores the filename.
Right now, only the ID and type of the input are stored.

Problem description

can't use nxlog buffer when using collector side car

Environment

• Sidecar Version: collector-sidecar-0.0.7-1.i386.rpm
• Graylog OVA Version: v2.0.2
• Operating System: centos

RHEL 6.3

Sidecar is looking for NXlog modules in /usr/lib/ while RPM installs everything under /usr/libexec/

Name        : collector-sidecar            Relocations: / 
Version     : 0.0.7                             Vendor: graylog
Release     : 1                             Build Date: Fri 13 May 2016 12:32:04 PM CEST
Install Date: Mon 23 May 2016 10:47:52 AM CEST      Build Host: ubuntu
Group       : optional                      Source RPM: collector-sidecar-0.0.7-1.src.rpm
Size        : 11064832                         License: GPLv3
Signature   : (none)
Packager    : Graylog, Inc. <[email protected]>
URL         : https://graylog.org
Summary     : Graylog collector sidecar
Description :
Graylog collector sidecar

Name        : nxlog-ce                     Relocations: (not relocatable)
Version     : 2.9.1504                          Vendor: NXLog Ltd
Release     : 1                             Build Date: Tue 08 Dec 2015 02:07:49 PM CET
Install Date: Mon 23 May 2016 10:58:53 AM CEST      Build Host: localhost
Group       : System Environment/Daemons    Source RPM: nxlog-ce-2.9.1504-1.src.rpm
Size        : 5270620                          License: NXLog Public License
Signature   : (none)
URL         : http://nxlog.org
Summary     : nxlog is a modular, multi-threaded, high-performance log management solution
Description :

/var/log/graylog/collector-sidecar/nxlog.log:

2016-05-23 13:57:29 ERROR Failed to load module from /usr/lib/nxlog/modules/extension/xm_gelf.so, /usr/lib/nxlog/modules/extension/xm_gelf.so: cannot open shared object file: No such file or directory;DSO load failed

Installed:

# rpm -ql nxlog-ce | grep xm_gelf
/usr/libexec/nxlog/modules/extension/xm_gelf.so

When I start collector-sidecar on my vm whcih want to input log,
I see it auto connect to graylog but I cannot config it,

If I need click configurations,and then click on create configuration ?
If I do this but I got a 400 error.

Problem description

In my snippet I need to define an extension with the ShortMessageLength option. I get error messages saying the extension is already defined. Looking at the generated nxlog.conf I see the following:

define ROOT C:\Program Files (x86)\nxlog

<Extension gelf>
  Module xm_gelf
</Extension>

I can't seem to control the first 5 lines of this config file. Collector Sidecar generates it automatically. In the snippet I've written the following:

<Extension gelf>
  Module xm_gelf
  ShortMessageLength -1
</Extension>

This conflicts with the extension definition that sidecar creates automagically and generates configuration error messages in NXLOG and ignores the "ShortMessageLength -1" line

In other words my config file looks like this:

<Extension gelf>
  Module xm_gelf
</Extension>

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel DEBUG

<Extension gelf>
  Module xm_gelf
  ShortMessageLength -1
</Extension>

Instead of this:

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log
LogLevel DEBUG

<Extension gelf>
  Module xm_gelf
  ShortMessageLength -1
</Extension>

I can run nxlog on it's own and my config file works fine. The first 5 lines that graylog side adds to my config breaks it.

Steps to reproduce the problem

Create a new collector configuration and just use a custom snippet.

Environment

• Sidecar Version: 0.0.8 x64
• Graylog Version: Graylog 2.0.3 (f07c170)
• Operating System: Red Hat Enterprise Linux Server release 6.8 (Santiago)
• Elasticsearch Version: 2.3.3
• MongoDB Version: 3.2.8

logrotate module defined twice in generated nxlog.conf

1. ...

Environment

• Sidecar Version: 0.9
• Graylog Version: 2.1.0
• Operating System: OVA
• Elasticsearch Version: OVA
• MongoDB Version: OVA

All our Graylog servers are behind a single load balancer, thus the output for every input will be one of three options.
When selecting an output for an input, might be helpful to have a drop-down with global ones with one option being 'new'?

Also, as mentioned in the Google group, having the option to add a custom snippet to a specific input will be hugely helpful in our setup.
This encompasses mostly Exec statements, breaking up the log entry into key/value pairs via regex and adding some custom fields.

Maybe in the output as well seeing we add some specific fields/values to the log based on app name, environment etc.

This is similar to Graylog2/graylog-plugin-collector@be22699, but only for sidecar

Problem description

[root@myfreebsd ~/work/src/collector-sidecar]# gmake build
go build -ldflags "-s -X github.com/Graylog2/collector-sidecar/common.GitRevision=d4c2910" -v -i -o graylog-collector-sidecar
github.com/Graylog2/collector-sidecar/vendor/github.com/cloudfoundry/gosigar
#github.com/Graylog2/collector-sidecar/vendor/github.com/cloudfoundry/gosigar
/usr/local/go/src/github.com/Graylog2/collector-sidecar/vendor/github.com/cloudfoundry/gosigar/sigar_unix.go:18: invalid operation: int64(stat.Blocks) * uint64(bsize) (mismatched types int64 and uint64)
/usr/local/go/src/github.com/Graylog2/collector-sidecar/vendor/github.com/cloudfoundry/gosigar/sigar_unix.go:19: invalid operation: int64(stat.Bfree) * uint64(bsize) (mismatched types int64 and uint64)
/usr/local/go/src/github.com/Graylog2/collector-sidecar/vendor/github.com/cloudfoundry/gosigar/sigar_unix.go:20: invalid operation: int64(stat.Bavail) * uint64(bsize) (mismatched types int64 and uint64)
/usr/local/go/src/github.com/Graylog2/collector-sidecar/vendor/github.com/cloudfoundry/gosigar/sigar_unix.go:23: stat.Free undefined (type syscall.Statfs_t has no field or method Free)
gmake: *** [Makefile:53: build] Error 2

Steps to reproduce the problem

1. ... hi im try to install collector sidecar in freebsd 10.2 amd64. im success to install go1.7 and glide newer version glide-v0.12.2-freebsd-amd64.tar.gz .
   im compile using GMAKE no MAKE coz in distribution UNIX.
   when im try GMAKE BUILD i have error in above..

anyone can help me ???

Environment

• Sidecar Version: ...
• Graylog Version: 2.1.0
• Operating System: Freebsd 10.2 64amd
• Elasticsearch Version: 2.4.0
• MongoDB Version: 3.2.9

Problem description

I'm trying to setup collector sidecar with a nxlog backend to ship logs using TLS.

I've enabled TLS on my TCP GELF input, specifying the cert and private key. I then went on to configure the collector, only to find out that there are no TLS settings on the collector config.
nxlog.log on the client show the following error: "ERROR om_tcp detected a connection error;Connection reset by peer".

Am I missing something? Do I need to write a custom nxlog snippet to enable the ssl module?
Here is an example config just to make my question clear: #https://www.loggly.com/docs/nxlog-tls-configuration/

Steps to reproduce the problem

1. Enable TLS on a GELF TCP input
2. Setup client OS with collector sidecar and nxlog backend
3. Configure collector in Graylog GUI (system\collectors --> collectors)
4. I used a nxlog file input and a TCP GELF ouput.
5. Go to the client, wait for the sidecar to apply new collector config and watch nxlog.log.

Environment

• Sidecar Version: 0.0.8
• Graylog Version: 2.0.1
• Operating System: centos 6