TimeStamp exception in valid timestamp about kinesis-agent-windows HOT 15 OPEN

Does it still update the internally known mapping once it encounters a new set of #Fields in the log?

I think it would make sense to update the mapping after reading that line.

from kinesis-agent-windows.

@johnkeates Can you post your configuration for this source?

from kinesis-agent-windows.

@dhhoang This is the source definition:

{
            "Directory": "D:\\Logs\\MY_MACHINE\\W3SVC1",
            "FileNameFilter": "*.log",
            "Id": "MyIISLogSource",
	    "TimeZoneKind": "UTC",
            "SourceType": "W3SVCLogSource"
}

from kinesis-agent-windows.

Looking at an identical source with a lower volume of logs, that one does work with no errors. From the logs I would expect a full StackTrace but there doesn't seem to be one, NLog.xml on Debug level doesn't add anything either (but I suspected as much from looking at the sources).

I don't understand why it would not have a record instance at that point (that's what that instance pointer refers to, right?).

The log data is exactly the same as all other IIS logs I have, no difference in line endings or binary formatting or anything like that.

from kinesis-agent-windows.

@johnkeates Can you verify that in your log files, there's a header line that specifies the fields according to https://docs.microsoft.com/en-us/previous-versions/iis/6.0-sdk/ms525807(v=vs.90)#w3c-extended-log-file-format

The header line usually starts with "#Fields:", for ex #Fields: time c-ip cs-method cs-uri-stem sc-status cs-version

from kinesis-agent-windows.

@dhhoang yes, it exists, but it doesn't always seem to be at the top of the file

from kinesis-agent-windows.

@johnkeates could you provide some sample lines with that line included?

from kinesis-agent-windows.

@dhhoang Here is a sample from a log from today:

2020-11-01 23:59:59 W3SVC1 IISW002 10.2.29.4 POST /r.messageservice/messageService.svc/sampler/ApplicationResponse - 80 - 10.2.29.2 HTTP/1.1 - - - internal.domain.hidden 201 0 0 206 3660 0
2020-11-01 23:59:59 W3SVC1 IISW002 10.2.29.4 POST /r.messageservice/messageService.svc/sampler/ApplicationResponse - 80 - 10.2.29.2 HTTP/1.1 - - - internal.domain.hidden 201 0 0 206 3660 0
#Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2020-11-02 00:00:00
#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken
2020-11-02 00:00:00 W3SVC1 IISW002 10.2.29.4 PUT /common.rest/carrier.svc/ApplicationResponse - 80 sa-car 10.2.29.2 HTTP/1.1 GuzzleHttp/6.5.1+curl/7.61.1+PHP/7.2.24 - - internal.domain.hidden 202 0 0 206 3989 15
2020-11-02 00:00:00 W3SVC1 IISW002 10.2.29.4 POST /R.messageservice/MessageService.svc/sampler/ApplicationResponse - 80 - 10.2.29.2 HTTP/1.1 Java/1.8.0_65 - - private.domain.hidden 201 0 0 206 4740 15

Above and below it are actual log lines with the correct fields

from kinesis-agent-windows.

@johnkeates Does the "#Fields" line line repeat in your log? If so, does it repeat daily?
The agent seems to have hit a case where it's not able to figure out the field mapping from the "#Fields" line if it's not at the start of the log file.

from kinesis-agent-windows.

@dhhoang Yes, it does repeat in the logs. In the file used in the example it was on line 100 and on line 1149769.

from kinesis-agent-windows.

@johnkeates it looks like if the #Fields line is not at the top of the file, the agent is unable to figure out the mapping and hence unable to parse the subsequent lines.
Would you want to have the option to specify a default field mapping?

from kinesis-agent-windows.

@dhhoang A default field mapping would be great; in theory the mapping of a file is known thread of time for the W3SVC log format anyway. There are multiple versions of IIS using slightly more or less fields, so being able to configure a default to match is a great method to solve this.

Does it still update the internally known mapping once it encounters a new set of #Fields in the log?

from kinesis-agent-windows.

The fix is in progress. I'll update this issue accordingly.

from kinesis-agent-windows.

Thanks for the update!

from kinesis-agent-windows.

@johnkeates Please give a new beta https://github.com/awslabs/kinesis-agent-windows/releases/tag/1.2.3.3 a try and see if it fixes your issue

from kinesis-agent-windows.