didfet / logstash-forwarder-java Goto Github PK

Currently paths like "/test/*/.log" are not supported. Please add wildcard support to directories.

Add an option to output logs to rotating files instead of stdout.

Hello,

Is there any possibilities to add subfields like this exemple :
"fields": { "fields" : { "mail" : "yes" } }

To result => fields.mail : yes instead of mail:yes

Thanks for your work by the way !
Alexis

Hey,

When trying to quiet a recurrent warning only the message gets removed and the stack trace persists.

In my case it happens while logstash-forwarder-java keeps on trying to read an empty file which causes a recurrent warning. It becomes a problem with a lot of empty files and can cause a FS full.

Also wanted to thank you a lot for your program !
Have a good one,
Paul

Log output in normal mode :

2015-06-26 11:00:36,536 WARN FileReader - Exception raised while reading file : /home/jqm/jqm/logs/0000411411.stdout.log
java.io.IOException: Seek failed
at info.fetter.logstashforwarder.FileReader.isCompressedFile(FileReader.java:87)
at info.fetter.logstashforwarder.FileReader.readFile(FileReader.java:73)
at info.fetter.logstashforwarder.FileReader.readFiles(FileReader.java:53)
at info.fetter.logstashforwarder.FileWatcher.readFiles(FileWatcher.java:103)
at info.fetter.logstashforwarder.Forwarder.infiniteLoop(Forwarder.java:93)
at info.fetter.logstashforwarder.Forwarder.main(Forwarder.java:82)

Log output in quiet mode :

java.io.IOException: Seek failed
at info.fetter.logstashforwarder.FileReader.isCompressedFile(FileReader.java:87)
at info.fetter.logstashforwarder.FileReader.readFile(FileReader.java:73)
at info.fetter.logstashforwarder.FileReader.readFiles(FileReader.java:53)
at info.fetter.logstashforwarder.FileWatcher.readFiles(FileWatcher.java:103)
at info.fetter.logstashforwarder.Forwarder.infiniteLoop(Forwarder.java:93)
at info.fetter.logstashforwarder.Forwarder.main(Forwarder.java:82)

Please forgive me so bad in English.
I have tried to run this on aix6.1，but it did not send anything .
I know the first warn is OK .

Add tail option to start reading files from the end instead of the beginning.

Create a CommandLineManager class to simplify Forwarder and other classes

To improve performance in FileReader

    private String readLine(RandomAccessFile reader) throws IOException {
        StringBuffer sb  = new StringBuffer();
        int ch;
        boolean seenCR = false;
        while((ch=reader.read()) != -1) {
            switch(ch) {
            case '\n':
                return sb.toString();
            case '\r':
                seenCR = true;
                break;
            default:
                if (seenCR) {
                    sb.append('\r');
                    seenCR = false;
                }
                sb.append((char)ch); // add character, not its ascii value
            }
        }
        return null;
    }

Running on java 1.6

Debug information:
*** ClientHello, TLSv1
RandomCookie: GMT: 1495540085 bytes = { 77, 137, 247, 232, 125, 176, 57, 204, 233, 35, 120, 161, 136, 132, 154, 1, 139, 148, 39, 24, 153, 1
Session ID: {}
Cipher Suites: [SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_
Compression Methods: { 0 }

main, WRITE: TLSv1 Handshake, length = 75
main, WRITE: SSLv2 client hello message, length = 101
main, received EOFException: error
main, handling exception: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
main, SEND TLSv1 ALERT: fatal, description = handshake_failure
main, WRITE: TLSv1 Alert, length = 2
main, called closeSocket()
adding as trusted cert:
Subject: CN=vuis-test
Issuer: CN=vuis-test
Algorithm: RSA; Serial number: 0xbb69e8a36dc43861
Valid from Mon Feb 08 17:01:33 MSK 2016 until Thu Feb 05 17:01:33 MSK 2026

2017-05-23 14:54:05,086 ERROR Forwarder - Failed to connect to server vuis-test:5000 :
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:817)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1165)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1149)
at info.fetter.logstashforwarder.protocol.LumberjackClient.(LumberjackClient.java:92)
at info.fetter.logstashforwarder.Forwarder.connectToServer(Forwarder.java:130)
at info.fetter.logstashforwarder.Forwarder.main(Forwarder.java:89)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:333)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
... 6 more

Hello,

I found a lot of similar warnings in the log:

2020-06-04 13:51:36,600 WARN FileReader - Exception raised while reading file : /bank/log/tploader.log.0603-21:05:40:313584
java.io.IOException: Seek failed
	at info.fetter.logstashforwarder.util.RandomAccessFile.length(RandomAccessFile.java:427)
	at info.fetter.logstashforwarder.util.RandomAccessFile.isEmpty(RandomAccessFile.java:458)
	at info.fetter.logstashforwarder.FileReader.readFile(FileReader.java:82)
	at info.fetter.logstashforwarder.FileReader.readFiles(FileReader.java:56)
	at info.fetter.logstashforwarder.FileWatcher.readFiles(FileWatcher.java:118)
	at info.fetter.logstashforwarder.Forwarder.infiniteLoop(Forwarder.java:188)
	at info.fetter.logstashforwarder.Forwarder.main(Forwarder.java:104)

It runs on Java 1.6.

Can you tell me why?

When the connection to the logstash server fails, the events in the event list are sent twice.

Firstly, I believe this tool would help me a lot, because I have aix os...

But When I came to use it to collect my log, an null pointer exception occured. Can anyone helps out?

First： AIX version
6100-04-06-1034

Second：Java Version
java version "1.6.0"
Java(TM) SE Runtime Environment (build pap6460sr14-20130705_01(SR14))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 jvmap6460sr14-20130704_155156 (JIT enabled, AOT enabled)
J9VM - 20130704_155156
JIT - r9_20130517_38390
GC - GA24_Java6_SR14_20130704_1138_B155156)
JCL - 20130618_01

Config file：

SADB01:/home/cmbcsa/logstash$ cat logforwarder.conf
{
"network": {
"servers": [ "40.32.64.52:5043"],
"ssl certificate": "/home/cmbcsa/logstash/logstash.pub",
"ssl key": "/home/cmbcsa/logstash/logstash.key"
},
"files": [
{
"paths": [ "/home/cmbcsa/logstash/1.txt" ],
"fields": { "type": "SADB" }
}
]
}

Finally command used：
java -jar logstash-forwarder-java-0.2.2.jar -debugwatcher -debug -trace -config logforwarder.conf

2015-07-21 23:16:25,436 DEBUG FileWatcher - Loading saved states
2015-07-21 23:16:25,661 WARN FileWatcher - Could not load saved states : .logstash-forwarder-java (路径名中的文件或目录不存在。)
2015-07-21 23:16:25,799 INFO FileWatcher - Watching file : /home/cmbcsa/logstash/1.txt
2015-07-21 23:16:25,815 DEBUG FileWatcher - Initializing FileWatcher
2015-07-21 23:16:25,815 DEBUG FileWatcher - File /home/cmbcsa/logstash/1.txt has been truncated or created, not retrieving pointer
2015-07-21 23:16:25,833 INFO Forwarder - Trying to connect to 40.32.64.52:5043
2015-07-21 23:16:26,215 ERROR Forwarder - Failed to connect to server 40.32.64.52:5043 : java.lang.NullPointerException
2015-07-21 23:16:26,216 INFO Forwarder - Trying to connect to 40.32.64.52:5043
2015-07-21 23:16:26,216 ERROR Forwarder - Failed to connect to server 40.32.64.52:5043 : java.lang.NullPointerException
2015-07-21 23:16:26,216 INFO Forwarder - Trying to connect to 40.32.64.52:5043
2015-07-21 23:16:26,216 ERROR Forwarder - Failed to connect to server 40.32.64.52:5043 : java.lang.NullPointerException
2015-07-21 23:16:26,216 INFO Forwarder - Trying to connect to 40.32.64.52:5043
2015-07-21 23:16:26,216 ERROR Forwarder - Failed to connect to server 40.32.64.52:5043 : java.lang.NullPointerException
2015-07-21 23:16:26,216 INFO Forwarder - Trying to connect to 40.32.64.52:5043
2015-07-21 23:16:26,217 ERROR Forwarder - Failed to connect to server 40.32.64.52:5043 : java.lang.NullPointerException
2015-07-21 23:16:26,217 INFO Forwarder - Trying to connect to 40.32.64.52:5043
2015-07-21 23:16:26,217 ERROR Forwarder - Failed to connect to server 40.32.64.52:5043 : java.lang.NullPointerException
2015-07-21 23:16:26,217 INFO Forwarder - Trying to connect to 40.32.64.52:5043
2015-07-21 23:16:26,217 ERROR Forwarder - Failed to connect to server 40.32.64.52:5043 : java.lang.NullPointerException

Could you add a command line option for users to customize sincedb filename? For some reason I may run two logstash-forwarder-java processes, if I run it in the same directory, the first .logstash-forwarder-java will be overwittern.

Hello,

I have an issue with deleted file. Due to log rollback (like log4j rollback), some files a deleted (with linux mv) but are always open (lsof tag them as "deleted"). The problem is that i have a "too many open file" error from logstash forwarder when the number of open file reach the system max open file (ulimit -n).

logstash forwarder (elastic branch) has create a field in order to solve this problem : dead time (https://discuss.elastic.co/t/too-many-open-files/29969 for example).

I tried this option but got an error (field unknown).

Is there an existing way to solve this problem or could you add the filed "dead time" like in logstash forwarder java?

Thanks!

Files not written for > 24h disappear from sincedb. When they are written again, old events are resent.

It seems that java version has the same bug elastic/logstash-forwarder#200

When a log file is rotated the logstash-forwarder agent keeps the deleted file opened until you kill its service.
This problem obviously causes an unhealthy situation of a consistently decreasing free space on the filesystem and currently my patch solution is scheduled restart for logstash-forwarder by a cron.

$ps -ef | grep forwarder | grep -v grep
wasadm 5981 1 0 Apr 6 ? 177:33 /IBM/WebSphere/AppServer/java/bin/IA64N/java -jar logstash-forwarder-java-0.2.3.jar -config config/ibm-was-system-logs-forwarde
wasadm 24530 1 31 Apr 26 ? 563:04 /IBM/WebSphere/AppServer/java/bin/IA64N/java -jar logstash-forwarder-java-0.2.3.jar -config config/app-albo-logs-forwarder.conf

$/usr/local/bin/lsof -a +L1 /LOGS | grep -wc 5981
1818

$/usr/local/bin/lsof -a +L1 /LOGS | grep -wc 24530
1022

In FileWatcher.java, the newWatchMap file table might get cleared before a new file added at the FileAdderThread has been processed.

Hi,

I don't know if this repo is still maintained, but I downloaded it, and try to run.
But first:

OS: AIX 7100-04-03-1642
JAVA: java version "1.7.0" Java(TM) SE Runtime Environment (build pap6470sr9fp60-20161021_01(SR9 FP60)) IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc64-64 Compressed References 20161005_321282 (JIT enabled, AOT enabled) J9VM - R26_Java726_SR9_20161005_1259_B321282 JIT - tr.r11_20161001_125404 GC - R26_Java726_SR9_20161005_1259_B321282_CMPRSS J9CL - 20161005_321282)

So jar is starting but, it can't find file called .logstash-forwarder-java,

[myAIX /gitlab/logstash-forwarder-java-0.2.4]# java -jar logstash-forwarder-java-0.2.4.jar -config pwd/config.json -debug 2017-03-02 16:11:40,329 DEBUG FileWatcher - Loading saved states 2017-03-02 16:11:40,642 WARN FileWatcher - Could not load saved states : .logstash-forwarder-java (A file or directory in the path name does not exist.) java.io.FileNotFoundException: .logstash-forwarder-java (A file or directory in the path name does not exist.) at java.io.FileInputStream.<init>(FileInputStream.java:158) at com.fasterxml.jackson.core.JsonFactory.createJsonParser(JsonFactory.java:768) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1897) at info.fetter.logstashforwarder.Registrar.readStateFromJson(Registrar.java:35) at info.fetter.logstashforwarder.Registrar.readStateFromJson(Registrar.java:40) at info.fetter.logstashforwarder.FileWatcher.setSincedb(FileWatcher.java:375) at info.fetter.logstashforwarder.Forwarder.main(Forwarder.java:78) java.lang.NullPointerException at info.fetter.logstashforwarder.Event.<init>(Event.java:37) at info.fetter.logstashforwarder.Forwarder.main(Forwarder.java:83)

So, I created it manualy, add JSON entries , {[]}, but next problem occur with parse:
2017-03-02 16:15:59,946 DEBUG FileWatcher - Loading saved states 2017-03-02 16:16:00,357 WARN FileWatcher - Could not load saved states : Can not deserialize instance of info.fetter.logstashforwarder.FileState[] out of START_OBJECT token at [Source: .logstash-forwarder-java; line: 1, column: 1] com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of info.fetter.logstashforwarder.FileState[] out of START_OBJECT token at [Source: .logstash-forwarder-java; line: 1, column: 1] at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:164) at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:599) at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:593) at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.handleNonArray(ObjectArrayDeserializer.java:220) at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:131) at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:18) at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2793) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1897) at info.fetter.logstashforwarder.Registrar.readStateFromJson(Registrar.java:35) at info.fetter.logstashforwarder.Registrar.readStateFromJson(Registrar.java:40) at info.fetter.logstashforwarder.FileWatcher.setSincedb(FileWatcher.java:375) at info.fetter.logstashforwarder.Forwarder.main(Forwarder.java:78) java.lang.NullPointerException at info.fetter.logstashforwarder.Event.<init>(Event.java:37) at info.fetter.logstashforwarder.Forwarder.main(Forwarder.java:83)

So how you can help with that?

To be fully compatible with logstash-forwarder.

There's certainly a small memory leak in FileWatcher processModifications() method :

for(File file : newWatchMap.keySet()) {
            if(logger.isTraceEnabled()) {
                logger.trace("Refreshing file : " + file.getCanonicalPath());
            }
            FileState state = newWatchMap.get(file);
            FileState oldState = state.getOldFileState();
            if(oldState == null) {
                logger.trace("File has been truncated or created, not retrieving pointer");
            } else {
                logger.trace("File has not been truncated or created, retrieving pointer");
                state.setPointer(oldState.getPointer());
                try {
                    oldState.getRandomAccessFile().close();
                } catch(Exception e) {}
            }
}

Could you add a section in the readme that talks about the environments you've tested or this is known to work in?

OS (eg AIX Versions and maintenance levels)
Java versions (etc IBM, Oracle, OpenJava, etc.)

I'm very interested in anything you've done in the AIX/IBM Java area with this and to what scale you've tested (eg how many monitored files on a host, log volumes per instance, etc.). I've got clients with ~500 monitored log files per AIX LPAR and would love to know how this works compared to current log shippers we use.

When log files are rotated, they can be compressed. So it is important to recognize them in order not to read them.

Hello,

When I execute the .jar appear the next message in the log, please help me

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Layout
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.Layout
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more

Hello. I'm using 0.2.4 version on SuSe 11.4 and Java 8u102.

On one of the servers java, after ~ 11 seconds of working, LSF Java process terminates without any exception in logs.

First of all I thought about OOM kill, but in GC log there's no problem at all.

So maybe there's a need to add some try/catch statement to detect such kind of problems.

Application log --> logstash-forwarder-java.log.txt

Xloggc --> lsf.gc.txt

It looks like there are some character encoding issues when reading the file character by character, storing in StringBuilder and then converting to byte array.

There's no timer between reconnections when the server doesn't respond.

Hi again,

Now I have problem as topic says.
When I run LSF the Exception is thrown, but I don't know why, no trace, no debug available.

Here are my run:

[aixlab /gitlab/logstash-forwarder-java-0.2.4]# java -jar logstash-forwarder-java-0.2.4.jar -trace -sincedb sincedb.json  -config config.json --debug
2017-03-09 15:36:51,692 DEBUG FileWatcher - Loading saved states
java.lang.NullPointerException
        at info.fetter.logstashforwarder.Event.<init>(Event.java:37)
        at info.fetter.logstashforwarder.Forwarder.main(Forwarder.java:83)

Config:

{
  "network": {
    "servers": [ "logserver:5044" ]
  },

  "files": [
    {
      "paths": [
        "/var/log/sudo.log"
      ]
    }
  ]
}

I manualy create sincedb.json because it won's created at start as you said in my last issue.

[{
   "directory": "/var/log",
   "fileName": "sudo.log",
   "signature": 10,
   "signatureLength": 2,
   "pointer": 1
}]

Can you advice about this issue ?

Add a feature to compute a checksum for each line. It could then be used as a unique event id in elasticsearch.