NLog 4.5.0 RTM has been released for NetCore: https://www.nuget.org/packages/NLog/

It would be nice to be able to turn the xml into Json so the fields are extracted from the security logs.
https://stackoverflow.com/questions/15675352/regex-convert-xml-to-json
https://stackoverflow.com/questions/814001/how-to-convert-json-to-xml-or-xml-to-json-in-c/814027#814027
currently using the json export will just put all the fields in one big string.
Using an xml to json converter will allow for more automatic field extraction.

I have known over the years in working with the FileSystemWatcher that it is not totally reliable and in may be more to do with the underlying OS, than the library itself. Such as missing events, and or just stop working and needing to restart it. This is even more true when monitoring UNC paths. I have read several articles that suggest using a polling mechanism to check for changes to a file every X seconds in place of using FSW would be more reliable, especially for UNC paths. Or, at the very least if using the FSW, you need to have a separate "watcher" to make sure FSW is still running, and if not, restart it.

Has this been taken into consideration with the design of the DirectorySource class?

I want to stream data present in a .txt file from local folder to kinesis stream using kinesis agent.

I have installed the kinesis agent and configured appsettings.json file.

When I start the service , I get following error

2023-02-08 16:24:01.8310 default ERROR Unable to load event source myKinesisSource System.NullReferenceException: Object reference not set to an instance of an object.
at Amazon.KinesisTap.Filesystem.FileSystemFactory.CreateFsSource(IPlugInContext context)
at Amazon.KinesisTap.Filesystem.FileSystemFactory.Amazon.KinesisTap.Core.IFactory<Amazon.KinesisTap.Core.ISource>.CreateInstance(String entry, IPlugInContext context)**
at Amazon.KinesisTap.Hosting.Session.LoadEventSources()
**2023-02-08 16:24:01.8666 default ERROR Unable to connect source myKinesisSource to sink myKinesisStreamSink.
2023-02-08 16:24:01.8722 default ERROR SourceRef myKinesisSource not found for pipe id myKinesisStreamPipe

appsettings.json configuration file:

{
"Sources": [
{
"Id": "myKinesisSource",
"SourceType": "DirectorySource",
"Directory": "C:\sales_file\",
"FileNameFilter": "*.txt"
}
],
"Sinks": [
{
"Id": "myKinesisStreamSink",
"SinkType": "KinesisStream",
"StreamName": streamname,
"AccessKey": accesskey,
"SecretKey": secretkey,
"Region": region
}
],
"Pipes": [
{
"Id": "myKinesisStreamPipe",
"SourceRef": "myKinesisSource",
"SinkRef": "myKinesisStreamSink"
}
],
"Telemetrics":
{
"off": "true"
}
}

I'm new to kinesis streaming and data agent concept. Any help on this is highly appreciated.

I got an error when I use the configuration below.
The encoding 'shift_jis' is very popular in japan.

Version: AWSKinesisTap.1.2.5.1.msi
OS: Windows Server 2019(1809)

Error
2022-12-16 14:31:51.0682 default ERROR Unable to load event source DHCPServerLog System.ArgumentException: 'shift_jis' is not a supported encoding name. For information on defining a custom encoding, see the documentation for the Encoding.RegisterProvider method. (Parameter 'name')

Configuration
"Sources": [ { "Id": "DHCPServerLog", "SourceType": "DirectorySource", "Directory": "C:\\Windows\\System32\\dhcp", "FileNameFilter": "Dhcp*SrvLog-*.log", "InitialPosition": "Bookmark", "Encoding": "shift_jis", "RecordParser": "SingleLine" } ],

Hi,

We send records from the different source files to the different regions KinesisStream with the same name. And sometimes record is sent to the wrong region.
Is it possible that the problem is in the configuration?

{
"Sources": [
{
"Id": "JsonLogSource",
"SourceType": "DirectorySource",
"RecordParser": "SingleLineJson",
"Directory": "C:\\us-west-1",
"FileNameFilter": "*.log",
"InitialPosition": 0
},
{
"Id": "JsonLogSource2",
"SourceType": "DirectorySource",
"RecordParser": "SingleLineJson",
"Directory": "C:\\eu-west-2",
"FileNameFilter": "*.log",
"InitialPosition": 0
},
{
"Id": "JsonLogSource3",
"SourceType": "DirectorySource",
"RecordParser": "SingleLineJson",
"Directory": "C:\\eu-central-1",
"FileNameFilter": "*.log",
"InitialPosition": 0
}
],
"Sinks": [
{
"Id": "US1FirehoseLogStream",
"SinkType": "KinesisStream",
"StreamName": "kstream",
"Region": "us-west-1",
"AccessKey": "...",
"SecretKey": "..."
},
{
"Id": "EU1FirehoseLogStream",
"SinkType": "KinesisStream",
"StreamName": "kstream",
"Region": "eu-west-2",
"AccessKey": "...",
"SecretKey": "..."
},
{
"Id": "EU2FirehoseLogStream",
"SinkType": "KinesisStream",
"StreamName": "kstream",
"Region": "eu-central-1",
"AccessKey": "...",
"SecretKey": "..."
}
],
"Pipes": [
{
"Id": "JsonLogSourceToFirehoseLogStream",
"SourceRef": "JsonLogSource",
"SinkRef": "US1FirehoseLogStream"
},
{
"Id": "JsonLogSourceToFirehoseLogStream2",
"SourceRef": "JsonLogSource2",
"SinkRef": "EU1FirehoseLogStream"
},
{
"Id": "JsonLogSourceToFirehoseLogStream3",
"SourceRef": "JsonLogSource3",
"SinkRef": "EU2FirehoseLogStream"
}
]
}

Is the ability to suppress event types supported in the ka4v? example from MS docs below, it's not indicated in the docs how/if the syntax would apply to source query config.

Why must we go through PowerShell? I'd like to be able to download an EXE and go from there.

Is there currently a Windows Sysmon Source Declaration? Are there any plans to develop one?

The Sysmon collector allows for several endpoint related events to be streamed.

https://docs.microsoft.com/en-us/sysinternals/downloads/sysmon

Thank you in advance.

I've setup a Kinesis agent on EC2 instance having windows operating system. Its getting me the following error for multiple sinks I've specifiedin my appsetting file. This error occurs randomly. There doesn't seem to be any connection lost issue since its running on EC2 instance also, the clock is not slow.

Kindly guide what could be wrong here and what can be done to avoid it?

2019-06-09 18:25:31.1244 Amazon.KinesisTap.Hosting.LogManager ERROR KinesisFirehoseSink client dev_dw_fh_s3_dl_inventtrans_snk exception: Amazon.KinesisFirehose.AmazonKinesisFirehoseException: Signature expired: 20190609T232013Z is now earlier than 20190609T232031Z (20190609T232531Z - 5 min.) ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.HttpRequest.d__16.MoveNext()
--- End of inner exception stack trace ---
at Amazon.Runtime.Internal.HttpRequest.d__16.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.HttpHandler1.<InvokeAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.Unmarshaller.d__31.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__51.MoveNext()
--- End of inner exception stack trace ---
at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)
at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
at Amazon.Runtime.Internal.ErrorHandler.d__51.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.CredentialsRetriever.d__71.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.RetryHandler.<InvokeAsync>d__101.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Amazon.Runtime.Internal.RetryHandler.d__101.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.CallbackHandler.d__91.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__51.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.Runtime.Internal.MetricsHandler.d__1`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Amazon.KinesisTap.AWS.KinesisFirehoseSink.d__7.MoveNext()

For AWS Compute Optimizer optimizer to be able to evaluate the Available MBytes metric it must be published to CWAgent namespace and the InstanceId dimension. I tried to configure the KTAP agent config to only publish to the InstanceId dimension but it always does the the InstanceId & Category dimension.

https://docs.aws.amazon.com/compute-optimizer/latest/ug/metrics.html?icmpid=docs_co_help_panel#cw-agent

Here is my KTAP agent config:

{
  "Sources": [
    {
      "Categories": [
        {
          "Counters": [
            {
              "Unit": "Megabytes",
              "Counter": "Available MBytes"
            }
          ],
          "Category": "Memory"
        }
      ],
      "Id": "PerformanceCounter",
      "SourceType": "WindowsPerformanceCounterSource"
    }
  ],
  "Sinks": [
    {
      "Id": "CloudWatchSink",
      "Interval": "60",
      "Namespace": "CWAgent",
      "SinkType": "CloudWatch",
      "Dimensions": "InstanceId={instance_id}",
      "Region": "US-WEST-2"
    }
  ],
  "Pipes": [
    {
      "SourceRef": "PerformanceCounter",
      "Id": "PerformanceCounterToCloudWatch",
      "SinkRef": "CloudWatchSink"
    }
  ],
  "SelfUpdate": 0
}

I am in Powershell and trying to execute Build.ps1. The error is that nuget.exe can not be found as part of a CmdLet, Function or script. But nuget.exe is in the directory from which I execute build.ps1

Hi everyone,

I'm looking for using the Kinesis agent to do something that may or may not be useful to other users and it is related to the issue raised in this #17 some time ago.

Let me try to describe the scenario: We have a variety of clients that are generating data, mainly into files, in a variety of formats. Those data could be logs, test results, metrics, whatever, but all those data is being created today into separated files, I don't have only one file that is constantly appended.

Our main idea on using Kinesis was to have it running on-premises "collecting" all those data and sending them to Amazon to a S3 bucket for instance, and having different consumers (whatever they are/would be) able to read this data and do whatever it's necessary to do. It is basically a producer-consumer scenario, Kinesis agent here would act as the man-in-the-middle between the producer and the consumer. What we want is to use the agent as a relay running in the clients, sending data to AWS, but the functionality that I'm looking for is to have the agent deleting delete or moving the files to a configurable destination, after transfer them to AWS.

Does anyone knows if this feature exists or if it is a good one for future releases ?

EventID has an important meaning in ETWEvent, but this is not acquired as data. I want to get analysis log of WindowsDNS server. However, the lack of EventIDs in the JSON data sent by the Agent makes analysis difficult.

1. kinesis-agent-windows/Amazon.KinesisTap.Windows/EtwEvent.cs

   Line 69 in ac0ee50

   EventName = traceData.EventName;

2. https://github.com/microsoft/perfview/blob/9dd10abfa9bbce7df9b14331841dba08c2774b13/src/TraceEvent/TraceEvent.cs#L670
3. https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn800669(v=ws.11)?redirectedfrom=MSDN#audit-and-analytic-event-logging

Using the latest KinesisTap release on IIS8 Logs I see failures to parse the lines:

2020-10-22 00:00:00.9431 MyIISLogSource ERROR 2020-10-21 21:59:01 W3SVC1 MY_MACHINE 12.29.33.35 GET /version.html - 80 - 12.29.153.3 HTTP/1.1 - - - intservices.exp.companyinternal.net 200 0 0 1014 87 0

2020-10-22 00:00:00.9431 MyIISLogSource ERROR Failed to get time stamp in D:\Logs\MY_MACHINE\W3SVC1\iis-2020-10-21.log, 45270226: System.NullReferenceException: Object reference not set to an instance of an object.
@stacktrace_ref 144 12802 1089308128

Hi,
The agent perfectly reads data from the input files; but is there a way to configure the agent to delete the input files once the data is processed and sent to the destination?

Thanks in advance.

Can you add support for configurable log group retention in the agent config like the CW agent now supports?

https://aws.amazon.com/about-aws/whats-new/2022/02/amazon-cloudwatch-agent-log-retention/

In complex designs it would be nice to specify the PartitionKey per source to be able to identify the source of the message. I see this was already added to the Linux agent in another branch: https://github.com/awslabs/amazon-kinesis-agent/pull/198/files/9fc16d7fa689aa310150cdcae293441855226918

Could this be added to the Windows agent as well?

Are there any plans to switch to .NET Core, so that we can have one self contained agent that runs on all operating systems and platforms? I don't understand why there are 2 implementations of the agent and i am afraid that they will behave different.

Best regards,
Thomas

Hi,

I just can't install the agent I have tried every method and I keep getting errors.
I've tried digging into them but I can't find anything...

Could you help me out ?
Thanks !