papertrail / remote_syslog Goto Github PK

I wanted to check with you before working on a pull request to fix this, to make sure there's not something I'm missing here.

It seems the only way to make remote_syslog work with upstart is to pass the "no detach" option. I'm having problems with the init script and I can't reproduce them reliably so I'm switching to upstart. I would like to keep using monit to send alerts when remote_syslog goes down (it's been unreliable), so I need a PID file to be present.

When not running in normal mode Servolux::Server#startup creates the PID file as expected. However it's not created by RemoteSyslog::Agent#run in "no detach" mode. Is this by design?

Thanks in advance.

If you have multiple files in different directories with the same name, there is no way to differentiate between them right now.

There should be a way to specify what program name is used for each file.

I have three log files listed in my yaml config file. I would like to be able to specify each to have it's own facility, e.g. local3, local4, local5.

Right now the only way I can think this would work would be running 3 separate remote_syslog processes, each with it's own set facility and log file.

Testing remote_syslog we discovered that it was holding onto the file that was just rotated by logrotate. This should be clearer in the README.

Hi,

We use remote_syslog in UDP mode and see incomplete messages on the receiving end. The messages get chopped at 1024 characters.

Looking at:
https://github.com/papertrail/remote_syslog/blob/master/lib/remote_syslog/agent.rb#L112 I see a max_message_size at 1024 is enforced for UDP.

Why is this?

Our Apache accesslogging contains cookies and useragents, so we need to be able to transfer long lines. My guesstimation: ~ 8kb long.

I know about the TCP mode in remote_syslog, but I prefer the fire-and-forget feature of UDP.

Any help is much appreciated.

Regards,
Renzo

My log file has errors that look like:

[ 7:35:58.807470188] [FATAL]
Moped::Errors::QueryFailure (The operation: #<Moped::Protocol::Query
  @length=106
  @request_id=145
...

but on Papertrail, only the [FATAL] comes through. The same happens for error messages with newlines, it seems to not send the content after the newline.

HI,

First, great tool!

I got a little issue sending over a log file form a Rails App. The rails app has a normal \n character, but on the receiving end (netcat) i see <13> (as a string of 4 chars). Which make the log on the receiving end one long line.

example:

<13>Jan 16 14:52:31 RUBY-ON-RAILS development.log: <13>Jan 16 14:52:31 RUBY-ON-RAILS development.log: <13>

The syslog protocol does not allow for tabs to be sent, so they'll need to be converted to spaces before sending.

Is this a good place to ask questions? I apologize if not, but I couldn't find any other channel.

The README file states that there is support for Windows deployments, however after installing the gem and trying to run on Windows I get:

ERROR -- : Unhandled EventMachine Exception: EventMachine::Unsupported: no file watching support on this system

TIA

remote_syslog segfaults, openssl is installed with rvm and I am running ruby 1.9.2-p320
Openssl is installed and I have apps using SSL functionality.

I am running debian 6.0.5

any ideas?

/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/i686-linux/openssl.so: [BUG] Segmentation fault
ruby 1.9.2p320 (2012-04-20 revision 35421) [i686-linux]

-- control frame ----------
c:0036 p:-40232738 s:0130 b:0130 l:000129 d:000129 TOP
c:0035 p:---- s:0128 b:0128 l:000127 d:000127 CFUNC :require
c:0034 p:0036 s:0124 b:0124 l:000123 d:000123 METHOD /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36
c:0033 p:0011 s:0117 b:0117 l:000116 d:000116 TOP /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/openssl.rb:17
c:0032 p:---- s:0115 b:0115 l:000114 d:000114 FINISH
c:0031 p:---- s:0113 b:0113 l:000112 d:000112 CFUNC :require
c:0030 p:0036 s:0109 b:0109 l:000108 d:000108 METHOD /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36
c:0029 p:0013 s:0102 b:0102 l:000101 d:000101 TOP /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/securerandom.rb:37
c:0028 p:---- s:0100 b:0100 l:000099 d:000099 FINISH
c:0027 p:---- s:0098 b:0098 l:000097 d:000097 CFUNC :require
c:0026 p:0036 s:0094 b:0094 l:000093 d:000093 METHOD /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36
c:0025 p:0061 s:0087 b:0087 l:000086 d:000086 TOP /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/resolv.rb:7
c:0024 p:---- s:0085 b:0085 l:000084 d:000084 FINISH
c:0023 p:---- s:0083 b:0083 l:000082 d:000082 CFUNC :require
c:0022 p:0036 s:0079 b:0079 l:000078 d:000078 METHOD /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36
c:0021 p:0384 s:0072 b:0072 l:000071 d:000071 TOP /usr/local/rvm/gems/ruby-1.9.2-p320/gems/eventmachine-1.0.0.rc.4/lib/eventmachine.rb:39
c:0020 p:---- s:0070 b:0070 l:000069 d:000069 FINISH
c:0019 p:---- s:0068 b:0068 l:000067 d:000067 CFUNC :require
c:0018 p:0157 s:0064 b:0064 l:000063 d:000063 METHOD /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:55
c:0017 p:0023 s:0057 b:0057 l:000056 d:000056 TOP /usr/local/rvm/gems/ruby-1.9.2-p320/gems/remote_syslog-1.6.5/lib/remote_syslog/reader.rb:2
c:0016 p:---- s:0054 b:0054 l:000053 d:000053 FINISH
c:0015 p:---- s:0052 b:0052 l:000051 d:000051 CFUNC :require
c:0014 p:0157 s:0048 b:0048 l:000047 d:000047 METHOD /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:55
c:0013 p:0021 s:0041 b:0041 l:000040 d:000040 TOP /usr/local/rvm/gems/ruby-1.9.2-p320/gems/remote_syslog-1.6.5/lib/remote_syslog.rb:5
c:0012 p:---- s:0039 b:0039 l:000038 d:000038 FINISH
c:0011 p:---- s:0037 b:0037 l:000036 d:000036 CFUNC :require
c:0010 p:0157 s:0033 b:0033 l:000032 d:000032 METHOD /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:55
c:0009 p:0011 s:0026 b:0026 l:000025 d:000025 TOP /usr/local/rvm/gems/ruby-1.9.2-p320/gems/remote_syslog-1.6.5/bin/remote_syslog:3
c:0008 p:---- s:0024 b:0024 l:000023 d:000023 FINISH
c:0007 p:---- s:0022 b:0022 l:000021 d:000021 CFUNC :load
c:0006 p:0127 s:0018 b:0018 l:000e64 d:0010c8 EVAL /usr/local/rvm/gems/ruby-1.9.2-p320/bin/remote_syslog:19
c:0005 p:---- s:0015 b:0015 l:000014 d:000014 FINISH
c:0004 p:---- s:0013 b:0013 l:000012 d:000012 CFUNC :eval
c:0003 p:0101 s:0007 b:0007 l:000e64 d:000788 EVAL /usr/local/rvm/gems/ruby-1.9.2-p320/bin/ruby_noexec_wrapper:14
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH

c:0001 p:0000 s:0002 b:0002 l:000e64 d:000e64 TOP

-- Ruby level backtrace information ----------------------------------------
/usr/local/rvm/gems/ruby-1.9.2-p320/bin/ruby_noexec_wrapper:14:in <main>' /usr/local/rvm/gems/ruby-1.9.2-p320/bin/ruby_noexec_wrapper:14:ineval'
/usr/local/rvm/gems/ruby-1.9.2-p320/bin/remote_syslog:19:in <main>' /usr/local/rvm/gems/ruby-1.9.2-p320/bin/remote_syslog:19:inload'
/usr/local/rvm/gems/ruby-1.9.2-p320/gems/remote_syslog-1.6.5/bin/remote_syslog:3:in <top (required)>' /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:55:inrequire'
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:55:in require' /usr/local/rvm/gems/ruby-1.9.2-p320/gems/remote_syslog-1.6.5/lib/remote_syslog.rb:5:in<top (required)>'
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:55:in require' /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:55:inrequire'
/usr/local/rvm/gems/ruby-1.9.2-p320/gems/remote_syslog-1.6.5/lib/remote_syslog/reader.rb:2:in <top (required)>' /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:55:inrequire'
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:55:in require' /usr/local/rvm/gems/ruby-1.9.2-p320/gems/eventmachine-1.0.0.rc.4/lib/eventmachine.rb:39:in<top (required)>'
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/resolv.rb:7:in <top (required)>' /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/securerandom.rb:37:in<top (required)>'
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in require' /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/openssl.rb:17:in <top (required)>' /usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:inrequire'
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'

-- C level backtrace information -------------------------------------------
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/libruby.so.1.9(rb_vm_bugreport+0x76) [0xb76c47d6]
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/libruby.so.1.9(+0x58861) [0xb758e861]
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/libruby.so.1.9(rb_bug+0x33) [0xb758f5c3]
/usr/local/rvm/rubies/ruby-1.9.2-p320/lib/libruby.so.1.9(+0x119013) [0xb764f013]
[0xb776c40c]
/lib/i386-linux-gnu/libc.so.6(strcmp+0x10) [0xb73ddf30]
/usr/local/rvm/usr/lib/libcrypto.so.0.9.8(+0x422d2) [0xb6ff12d2]

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

Aborted

Consider the following directory stucture:
/path/to/logs/myapp
/path/to/logs/myapp-2013-09-10
/path/to/logs/myapp-2013-09-09

If we watch /path/to/logs/myapp/*.log and the directory is moved to /path/to/logs/myapp-2013-09-11 and a new directory is created called /path/to/logs/myapp remote_syslog will keep the files in /path/to/logs/myapp-2013-09-11 open and start watching the logs in the newly created /path/to/logs/myapp

Eventually the process will run out of available files and stop working.

How can we get around this problem, besides restarting the service daily?

I noticed the copytruncate option in the readme but the examples don't show how to use it; is it just copytruncate: true?

I tried checking the source code to see how it was used, but it doesn't even seem to be checked for: https://github.com/papertrail/remote_syslog/blob/master/lib/remote_syslog/cli.rb#L206

I also searched the code for 'copytruncate' but only found it in the readme.

I just installed the 1.6.2 remote_syslog gem, and am getting this error when I try to run it:

Invalid gemspec in [/var/lib/gems/1.8/specifications/remote_syslog-1.6.2.gemspec]: invalid date format in specification: "2012-04-05 00:00:00.000000000Z"
/usr/lib/ruby/vendor_ruby/1.8/rubygems.rb:926:in report_activate_error': Could not find RubyGem remote_syslog (>= 0) (Gem::LoadError) from /usr/lib/ruby/vendor_ruby/1.8/rubygems.rb:244:inactivate_dep'
from /usr/lib/ruby/vendor_ruby/1.8/rubygems.rb:236:in activate' from /usr/lib/ruby/vendor_ruby/1.8/rubygems.rb:1307:ingem'
from /usr/local/bin/remote_syslog:18

The line in the gemspec is:
s.date = %q{2012-04-05 00:00:00.000000000Z}

SO basically I am getting the following error but ONLY on my ActionCable server, not on my actual rails web server, Im not quite sure whats going on but it seems somewhat similar to Issue #6 ??
ArgumentError: Tag may not contain spaces
tag= at /usr/local/bundle/gems/syslog_protocol-0.9.2/lib/syslog_protocol/packet.rb:52
initialize at /usr/local/bundle/gems/remote_syslog_logger-1.0.3/lib/remote_syslog_logger/udp_sender.rb:20
new at /usr/local/bundle/gems/remote_syslog_logger-1.0.3/lib/remote_syslog_logger.rb:9

Is this kind of CPU usage expected?

I find it quite high.

I have several instances running, because I have several applications and I want their log separated by host in PaperTrail. The configuration is very basic.

For example, here is the command line of a process:

/usr/bin/ruby1.9.1 /usr/local/bin/remote_syslog -D --pid-file /var/run/remote_syslog_demo1.pid -c /etc/logs/demo1.yml

and here is the YML config file:

files: 
  - /home/web/demo1/data/logs/error.log
  - /home/web/demo1/data/logs/worker.log

hostname: prod.demo1

destination:
  host: logs.papertrailapp.com
  port: XXXX

As you can see, the logs here are empty (on other apps, logs are very small for now):

$ ll /home/web/demo1/data/logs/
-rwxrwxrwx 1 root root 0 Oct 24 17:37 error.log
-rwxrwxrwx 1 root root 0 Oct 24 17:37 worker.log

The server is Ubuntu 13.04, 4 cores (hosted at digital ocean).

This looks like the same issue from #6. However I'm on 1.3.0 so I don't understand what the problem is.

our log file names are ie org.bookshare.service.task.AudioConverterTask08_03_2011.log

root@ip-10-114-58-13:~# /var/lib/gems/1.8/bin/remote_syslog -D -p 514 /var/log/scheduler/audio/*
/var/lib/gems/1.8/gems/syslog_protocol-0.9.0/lib/syslog_protocol/packet.rb:44:in tag=': Tag must not be longer than 32 characters (ArgumentError) from /var/lib/gems/1.8/gems/remote_syslog-1.3.0/lib/remote_syslog/reader.rb:31:ininitialize'
from /var/lib/gems/1.8/gems/eventmachine-tail-0.6.1/lib/em/filetail.rb:444:in new' from /var/lib/gems/1.8/gems/eventmachine-tail-0.6.1/lib/em/filetail.rb:444:infile_tail'
from /var/lib/gems/1.8/gems/remote_syslog-1.3.0/lib/remote_syslog/cli.rb:164:in start' from /var/lib/gems/1.8/gems/remote_syslog-1.3.0/lib/remote_syslog/cli.rb:162:ineach'
from /var/lib/gems/1.8/gems/remote_syslog-1.3.0/lib/remote_syslog/cli.rb:162:in start' from /var/lib/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:incall'
from /var/lib/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in run_machine' from /var/lib/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:inrun'
from /var/lib/gems/1.8/gems/remote_syslog-1.3.0/lib/remote_syslog/cli.rb:155:in start' from /var/lib/gems/1.8/gems/remote_syslog-1.3.0/lib/remote_syslog/cli.rb:146:inrun'
from /var/lib/gems/1.8/gems/remote_syslog-1.3.0/lib/remote_syslog/cli.rb:17:in process!' from /var/lib/gems/1.8/gems/remote_syslog-1.3.0/bin/remote_syslog:6 from /var/lib/gems/1.8/bin/remote_syslog:19:inload'
from /var/lib/gems/1.8/bin/remote_syslog:19

We've ran into many issues where eventmachine-tail leaves file descriptors open when it shouldn't, doesn't sense when files are rotated, and issues with systems that don't have epoll or kqueue.

It would be nice to move to a simpler mechanism that more closely mapped to how /usr/bin/tail works.

Running on several boxes, all have similar memory usage reported by top.

Is this expected?

Hello,

It seems the following request causes remote_syslog 1.3.1 to crash hard every time:

Started GET "/transit?utf8=%E2%9C%93&decorator_id=4c25091aff30612085000036&supplier_id=16&deliver_by=2011-08-16&commit=Submit" for 0.0.0.0 at 2011-08-12 21:29:41 -0400 
  Processing by TransitController#index as HTML
  Parameters: {"utf8"=>"✓", "decorator_id"=>"4c25091aff30612085000036", "supplier_id"=>"16", "deliver_by"=>"2011-08-16", "commit"=>"Submit"}
Rendered transit/index.haml within layouts/application (1007.5ms)
Completed 200 OK in 1818ms (Views: 1011.4ms)

Things to note:

• The only thing I sanitized above was the IP (0.0.0.0).
• I'm using the --tls option.
• Ruby 1.9.2-p180

Thanks,
Chris

Hey there,

How feasible would it be to have a configurable compression feature for remote_syslog, in order to reduce the amount of bandwidth consumed, at the cost of slightly higher CPU usage?

I was reading that rsyslogd does support compressed messages, so wanted to see if this was something you had considered at all?

Thanks!

does not complain either

The usage of exclude_patterns in the config yaml is confusing. A clear example or instruction should be provided to users.

Quote from master/examples/log_files.yml.example.advanced

exclude_patterns:
  - exclude this
  - \d+ things

This sample/instruction is kidding on the format.

It seems the exclude_patterns does not support either the plain path or the Regex pattern, as I tested.
The following config failed to exclude the log from the file at /LOG/FOLDER/tmp-stderr.log.

files:
  - /LOG/FOLDER/*.log
destination:
  host: #########
  port: #########
  protocol: tls
exclude_patterns:
  - .*tmp.+\.log$
  - /LOG/FOLDER/tmp-stderr.log

Please let me know the mistake I might have made in the test.

(Looks like http://help.papertrailapp.com/discussions/problems/1750-remote_syslog-terminating - but nothing here on GH.)

Running remote_syslog with ruby 1.8.7 (2012-06-29 patchlevel 370) [x86_64-linux] on RHEL6.

Daemon crashes with this debug logging:

# Logfile created on Mon Aug 12 09:39:49 +0200 2013 by logger.rb/1.2.6
D, [2013-08-12T09:39:49.722363 #5760] DEBUG -- : About to fork ...
D, [2013-08-12T09:39:49.724257 #5762] DEBUG -- : Server "remote_syslog" creating pid file "/var/run/remote_syslog_apache_access.pid"
D, [2013-08-12T09:39:49.724465 #5762] DEBUG -- : Starting
D, [2013-08-12T09:39:49.724947 #5760] DEBUG -- : Waiting for "remote_syslog" to startup.
D, [2013-08-12T09:39:49.727405 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.727568 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.727715 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.727887 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.728071 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.728234 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.728396 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.728541 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.728685 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.728809 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.728935 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.729100 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.741770 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.741942 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.742103 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.742246 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.742400 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.742546 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.742678 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.742796 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.742913 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.743062 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.743181 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.743304 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.743471 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.743604 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.743739 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.743857 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.744012 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.744142 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.744261 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.744404 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.744563 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T09:39:49.744682 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
I, [2013-08-12T09:39:50.025176 #5760]  INFO -- : Server has daemonized.
D, [2013-08-12T10:48:22.754135 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T11:07:53.560893 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-12T13:55:30.222148 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:00:05.574444 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:00:05.574773 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:00:05.574959 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:00:05.575198 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:00:45.608892 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:00:55.618261 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:00:55.618513 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:00:55.618911 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:00:55.619114 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:01:05.626917 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:01:15.639680 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:01:15.640017 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:01:25.646126 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:01:25.646538 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:01:45.658514 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:02:25.678513 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:02:35.682057 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:02:45.685721 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:02:45.685953 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:04:25.734267 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:04:25.734709 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:11:06.055067 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
D, [2013-08-13T00:13:26.150143 #5762] DEBUG -- : Watching /var/log/httpd/SOME_VHOST/access.log with EventMachineReader
terminate called after throwing an instance of 'std::runtime_error'
  what():  attempted to remove invalid watch signature

Config file:

files:
  - /var/log/httpd/*/access.log
destination:
  host: LOGSTASH_HOST
  port: 5002

Gem list:

*** LOCAL GEMS ***
em-resolv-replace (1.1.3)
eventmachine (0.12.10)
eventmachine-tail (0.6.4)
file-tail (1.0.12)
json (1.5.5)
remote_syslog (1.6.14)
servolux (0.10.0)
stomp (1.2.2)
syslog_protocol (0.9.2)
tins (0.8.3)

If you need more info, please let me know.
We're running remote_syslog on 100+ machines and notice frequent crashes, which is causing us to loose data to logstash.

Hi

I'm having an issue starting up remote_syslog on an ubuntu 11.10 server. I can start it successfully with any normal user from the command line, but when i try to do this with the root user, the remote_syslog process seems to start up but after checking it with ps -ef | grep remote_syslog the process doesnt show up. It dies after starting.
What i have noticed is that if i run it as root locally from a graphical console, it works fine.
The version of remote_syslog is 1.6.0

Thanks

I planned to work on this feature and issue a pull request if you guys were interested.

My plan was to add a command line option as well as configuration file support to take a list of blobs, similar to the includes. The files that match the exclude blobs would be removed from the include list (thus being processed after the includes)

How does this sound? Any thoughts before I get started?

for example if my log says: http://url/path?a=1&api_key=somethinghere&b=1 replace it with http://url/path?a=1&api_key=so*************&b=1 before sending it to the remote log server.

Howdy!

I found this project through someone posting on the logstash-users mailing list.

We should figure out some kind of grand unification for your project and my project (syslog-shipper). There's so much in common it's quite silly maintaining both separately, I'd say!

Thoughts?

Would be a much neater dependency graph to install it on a remote node.

Go just requires you to distribute a single binary executable.

Avoids ruby/python/whatever dependency hell.

I set the copytruncate option on the logrotate config file, and when run logroate for that file, remote_syslog seems will no longer send the UDP message after truncate.

remote_syslog: 1.6.14
ruby: ruby-2.0.0-p0
os: ubuntu 12.04
logrotate: 3.7.8

BTW, i try to the remote_syslog2, the GO version, and it works ok~

We noticed that remote_syslog is keeping logfiles open that have been deleted.
An example lsof:
ruby 4894 root 10r REG 253,2 151396 65567 /path/to/catalina.out (deleted)
ruby 4894 root 11r DIR 0,10 0 649 inotify
ruby 4894 root 12r REG 253,2 151396 65567 /path/to/catalina.out (deleted)
ruby 4894 root 13r REG 253,2 13646 65556 /path/to/catalina.out (deleted)
ruby 4894 root 14r REG 253,7 29238 507906 /path/to/catalina.out

Is there a way, besides restarting the service, to prevent this or get rid of them?

This is particularly useful when remote_syslog is run as a daemon. You could do /etc/init.d/remote_syslog start, and then immediately check for errors using something like: tail /var/log/remote_syslog.log. These errors should also be transmitted to the remote syslog server.

It wouldn't catch everything, but it's better than nothing. One particular issue is where a glob has been used, and the user that remote_syslog runs as has no access to any of the files in that directory.

Right now TLS is only needed after daemonizing, which means that if libopenssl Ruby support is not installed, it will silently fail only when used with TLS. This fixes it in Ubuntu:

sudo apt-get install libssl-dev libopenssl-ruby

remote_syslog should try to detect and react to this before daemonizing if TLS is enabled, since at that point it's guaranteed to fail. Thanks to @torbjokv for helping find this.

Like: Watching 4 files/globs. Sending to X:Y (UDP) as www42.

     if @tls
       connection = TlsEndpoint.new(@dest_host, @dest_port)
     else
       connection = UdpEndpoint.new(@dest_host, @dest_port)
     end

is there a technical reason for this?

:)

Remove the quotes around EXTRAOPTIONS so that multiple parameters aren't treated as one. Change:
$prog -c $config --pid-file $pid_file "$EXTRAOPTIONS"

To

$prog -c $config --pid-file $pid_file $EXTRAOPTIONS

Would be handy if you could pass log lines through a custom function before submitting them. For example for anonymizing sensitive data / PII, etc

Show a friendly error to the user when they make a typo e.g. adding extra spaces to the beginning of a line.

Hey is remote_syslog still under active development?

Try to detect this case (and any other easy daemonization-specific failures):

write(2, "Permission denied - /var/run/remote_syslog.pid", 46) = 46

Thanks to @cluesque for finding and reporting this problem.

When installing on Ubuntu 11.04:

sudo gem install remote_syslog

Links are not made from /usr/bin/ making the remote_syslog command unavailable without using a full path.

remote_syslog: #<Errno::ENOENT: No such file or directory @ rb_sysopen - ssl/papertrail.crt

I have the papertrail certificate in my application's code and I'm attempting to access it by giving the relative path to the file. I can't give an absolute path, since my deploy directory depends on the environment, application, etc. Anyway, hardcoding paths sucks. I think it has something to do with this comment regarding Daemonize setting the path to /
https://github.com/papertrail/remote_syslog/blob/master/lib/remote_syslog/cli.rb#L171

The upcase calls for facility and severity prevent them from matching the hashes in RemoteSyslog::Levels.

trying to start remote_syslogs using the init script from the examples:
(https://github.com/papertrail/remote_syslog/blob/master/examples%2Fremote_syslog.init.d)

keep getting this error.

root@mycomp:~# service remote_syslog start
Starting remote_syslog: /usr/lib/ruby/1.9.1/rubygems/dependency.rb:247:in `to_specs': Could not find servolux (>= 0) amongst [remote_syslog-1.6.14] (Gem::LoadError)
        from /usr/lib/ruby/1.9.1/rubygems/specification.rb:777:in `block in activate_dependencies'
        from /usr/lib/ruby/1.9.1/rubygems/specification.rb:766:in `each'
        from /usr/lib/ruby/1.9.1/rubygems/specification.rb:766:in `activate_dependencies'
        from /usr/lib/ruby/1.9.1/rubygems/specification.rb:750:in `activate'
        from /usr/lib/ruby/1.9.1/rubygems.rb:1232:in `gem'
        from /usr/local/bin/remote_syslog:22:in `<main>'

If i just run directly from the command line, it works fine:

root@mycomp:~# remote_syslog
Watching 1 files/globs. Sending to logs.papertrailapp.com:12345 (UDP).

We're seeing an issue where long log lines (above 8k characters or so) get wrapped. Unfortunately, the wrapped portion of the line isn't sent to papertrail with the hostname setting. So, for a particular server with a hostname set in log_files.yml, the first 8k characters of the log line will be sent to the names system, while the remaining characters will be sent to a system names after the IP of the server (or whatever the default hostname is).

I receive this error on startup. I'm can't find where tag is being defined. remote_syslog was installed on Ubuntu 11.04 using gem install remote_syslog

Thanks.

Traceback below.

/usr/lib/ruby/gems/1.8/gems/syslog_protocol-0.9.0/lib/syslog_protocol/packet.rb:44:in `tag=': Tag must not be longer than 32 characters (ArgumentError)
    from /usr/lib/ruby/gems/1.8/gems/remote_syslog-1.1.0/lib/remote_syslog/reader.rb:32:in `initialize'
    from /usr/lib/ruby/gems/1.8/gems/eventmachine-tail-0.6.1/lib/em/filetail.rb:444:in `new'
    from /usr/lib/ruby/gems/1.8/gems/eventmachine-tail-0.6.1/lib/em/filetail.rb:444:in `file_tail'
    from /usr/lib/ruby/gems/1.8/gems/remote_syslog-1.1.0/bin/remote_syslog:103:in `remote_syslog_daemon'
    from /usr/lib/ruby/gems/1.8/gems/remote_syslog-1.1.0/bin/remote_syslog:101:in `each'
    from /usr/lib/ruby/gems/1.8/gems/remote_syslog-1.1.0/bin/remote_syslog:101:in `remote_syslog_daemon'
    from /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `call'
    from /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
    from /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
    from /usr/lib/ruby/gems/1.8/gems/remote_syslog-1.1.0/bin/remote_syslog:98:in `remote_syslog_daemon'
    from /usr/lib/ruby/gems/1.8/gems/daemons-1.1.3/lib/daemons/application.rb:249:in `call'
    from /usr/lib/ruby/gems/1.8/gems/daemons-1.1.3/lib/daemons/application.rb:249:in `start_proc'
    from /usr/lib/ruby/gems/1.8/gems/daemons-1.1.3/lib/daemons/application.rb:260:in `call'
    from /usr/lib/ruby/gems/1.8/gems/daemons-1.1.3/lib/daemons/application.rb:260:in `start_proc'
    from /usr/lib/ruby/gems/1.8/gems/daemons-1.1.3/lib/daemons/application.rb:293:in `start'
    from /usr/lib/ruby/gems/1.8/gems/daemons-1.1.3/lib/daemons/controller.rb:73:in `run'
    from /usr/lib/ruby/gems/1.8/gems/daemons-1.1.3/lib/daemons.rb:195:in `run_proc'
    from /usr/lib/ruby/gems/1.8/gems/daemons-1.1.3/lib/daemons/cmdline.rb:109:in `call'
    from /usr/lib/ruby/gems/1.8/gems/daemons-1.1.3/lib/daemons/cmdline.rb:109:in `catch_exceptions'
    from /usr/lib/ruby/gems/1.8/gems/daemons-1.1.3/lib/daemons.rb:194:in `run_proc'
    from /usr/lib/ruby/gems/1.8/gems/remote_syslog-1.1.0/bin/remote_syslog:97:in `remote_syslog_daemon'
    from /usr/lib/ruby/gems/1.8/gems/remote_syslog-1.1.0/bin/remote_syslog:117
    from /usr/bin/remote_syslog:19:in `load'
    from /usr/bin/remote_syslog:19

I know it's deprecated, however a recent change to the gem's dependency has broken this gem like so:

lib/ruby/gems/1.9.1/gems/remote_syslog-1.6.14/lib/remote_syslog/cli.rb:185:in `basename': can't convert Servolux::PidFile into String (TypeError)

A solution we found was to install servolux version 0.10.0 before installing remote_syslog gem, as there's no pinning of dependency at all in this gem.