remote_syslog 1.6.14 keeps deleted logfiles open about remote_syslog HOT 6 OPEN

This happens when the files are symbolic links too, which means we can't stop logging by just removing a symbolic link but instead have to restart the service.

from remote_syslog.

Still happens in remote_syslog (1.6.15)

from remote_syslog.

This seems to be an interaction with logrotate.

$ sudo lsof | grep deleted
# many other remote_syslog files that are deleted
# ...
remote_sy 26216 26218       root   25r      REG              202,1 391179605     176530 /var/log/nginx/core.access.log.1 (deleted)

The default logrotate conf that ubuntu 14.04 ships

/var/log/nginx/*.log {
    weekly
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    prerotate
        if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
            run-parts /etc/logrotate.d/httpd-prerotate; \
        fi \
    endscript
    postrotate
        [ -s /run/nginx.pid ] && kill -USR1 `cat /run/nginx.pid`
    endscript
}

and my log_files.yml looks like:

files:
   - # stuff
   - /var/log/nginx/core.access.log
destination:
   host: logs.papertrailapp.com
   port: #####

from remote_syslog.

I think @ChrisLundquist's problem is unrelated to @rolandkool's. @ChrisLundquist: check out copytruncate; that may prevent the leftover open file descriptors.

from remote_syslog.

@troy Thanks for the speedy reply!

Yeah, my goal in the above was to give to simplest way to replicate an issue. You're right that the default nginx logrotate configuration doesn't use copytruncate. I inherited this setup, so it is possible there was documentation that got missed in the implementation. It seems to me that using papertrail with nginx is a pretty common use case. In my ideal world, remote_syslog would work with a vanilla sudo apt-get install nginx.

My rails app logs use copytruncate and see a similar issue.

$ sudo lsof | grep deleted
remote_sy 26216             root   23r      REG              202,1   2295124      86059 /path/to/rails/app/e2e590b3d1b4942191c7fd20fbc74997d8b9d63c/build/log/production.log (deleted)

$ cat /etc/logrotate.d/core_service
/path/to/rails/app/current/log/*.log {
  daily
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
  copytruncate
}

This could be related to the symlink issue, but I'm not entirely sure. current is a symlink, but it looks like File.expand_path is called to resolve *.log to an absolute path.

from remote_syslog.

@ChrisLundquist no problem for the quick reply. I'm going to transition your last email into a thread in our support system (so we can explore your environment more specifically), so please expect an email from support@ late Wednesday or Thursday US time.

from remote_syslog.