Giter Club home page Giter Club logo

windows-container-tools's Introduction

Build Status Join the chat at https://gitter.im/Microsoft/windows-containers-tools

Overview

Windows Container Tools is a collection of tools to augment the Windows Container experience.

For general discussions and issues surrounding Windows Containers please visit https://github.com/microsoft/Windows-Containers.

Tool List

Contributing

This project welcomes contributions and suggestions. See details on how to contribute in CONTRIBUTING.md.

windows-container-tools's People

Contributors

algamaes avatar bobsira avatar charitykathure avatar iankingori avatar lucillex avatar manoj-kadam avatar microsoft-github-operations[bot] avatar microsoftopensource avatar mloskot avatar pengzhisun avatar pl4nty avatar profnandaa avatar ulfaxelssoncab avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

windows-container-tools's Issues

[Discussion] Certificate Tool

Process isolated containers is an isolation mode of containers that have loose sandbox capabilities in the sense they do not have serviceable security boundaries. The containers are isolated in that they share different network stacks, trace trees, and certificate stores but the sandboxing capabilities are not strong enough that someone can intentionally or unintentionally get past the sandbox and access other containers being hosted on the same VM.

Containers have two default users that they can run - container user and container administrator. The container user is a low-privilege mode that can do a lot less than a container administrator. However, a container administrator user is almost like an administrator in the underlying host because the container kernel is shared with the underlying VM kernel.

Generally, admin privileges are mostly needed at startup when the service needs to read/set registry settings, access the certificate store, download and install certificates, and configure monitoring agents. We've heard from customer engagement that having a tool for Windows Containers to handle processes needed to be done in higher level privileges such as certificate installation before dropping to low-privilege would provide value to users.

We would love to hear your feedback on this.

Windows nodes are failing to start containerd task when calling LogMonitor.exe

I am running a windows containerd nodepool on the following OS Distro:
AKSWindows-2019-containerd-17763.3287.220810 which I believe to be the latest
image

And I am getting the following error on the pod events:
Warning Failed 31m (x5 over 33m) kubelet Error: failed to start containerd task "sm": hcs::System::CreateProcess sm: This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.: unknown

The Dockerfile has the following parameter that probably is not ingested by the LogMonitor
ENTRYPOINT ["C:\scripts\logging\LogMonitor.exe", "C:\scripts\startup.ps1"]

So then I tried running logmonitor.exe from inside the pod, it throws the same incompatibility error.

PS C:\scripts\logging> .\LogMonitor.exe
Program 'LogMonitor.exe' failed to run: T**he specified executable is not a valid application for this OS platform.At line:1 char:1**
+ .\LogMonitor.exe
+ ~~~~~~~~~~~~~~~~.
At line:1 char:1
+ .\LogMonitor.exe
+ ~~~~~~~~~~~~~~~~
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed

Can you please help me on this?

TdhGetEventInformation returns 1168 for manifest based events

Hello , I am no expert in ETW but I know I have an event stream that has some custom properties and I want to use this tool to print them out to STDOUT.

I running the code from master to debug the issue and am getting the 1168 error while querying for information via TdhGetEventInformation

image

The configuration file is verry simple :

{
  "LogConfig": {
    "sources": [
      {
        "type": "ETW",
        "providers": [
          {
            "ProviderGuid": "AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAA",
            "level": "Error"
          }
        ]
      }
    ]
  }
}

Do I need to do something extra for manifest based/ extra properties event ?

IIS Certificate binding

I was watching the ignite video and found myself here. There was a brief mention of certificates in that talk and maybe this could be a good place to bring it up.

I was looking for guidance on certificate management with IIS inside windows containers. This is what I'd got to thus far for my Dockerfile to use central certificate store and volume mapping a pfx secret for k8s into C:\inetpub\certs:

RUN $userPass = [guid]::newguid(); \
    # Admin user that can be used for enabling cert provider
    New-LocalUser -Name CertUser -Password (ConvertTo-SecureString $userPass -AsPlainText -Force); \
    Add-LocalGroupMember -Group Administrators -Member CertUser; \
    # Enable Central Certificate Store for IIS
    Enable-WindowsOptionalFeature -Online -FeatureName IIS-CertProvider; \
    # Enable the central certificate store and tell it where certificates are stored
    New-Item -ItemType Directory -Path C:\inetpub\certs | Out-Null; \
    Enable-WebCentralCertProvider -CertStoreLocation C:\inetpub\certs -UserName CertUser -Password $userPass; \
    Import-Module WebAdministration; \
    # Add a central certificate store ssl binding for https
    New-WebBinding -Name 'Default Web Site' -SslFlags 3 -Protocol https -IP '*' -Port 443 -HostHeader '*'; \
    New-Item -Path IIS:\SslBindings\!443!* -SslFlags 3

This is still in development from my side and I'm still validating it's working state etc.

Guidance, tools or anything else that would enable us to have SSL on IIS in our container at runtime while potentially simplifying this would be a great resource.

[LOGMONITOR] INFO: Entrypoint processs exit code: 0

Whenever I see below log, the pod in Kubernetes will crash in seconds and will restart, it happened intermittently (like 40% chance) after pod started around 3 mins. And keep going until things go to stable

Log: =====================================
...
Stopping service 'w3svc'

ERROR: Failed to stop or query status of service 'w3svc' error [800705b4]
[2020-08-20T14:10:32.000Z][LOGMONITOR] INFO: Entrypoint processs exit code: 0

LogMonitorConfig.json: =====================================
{
"LogConfig": {
"sources": [
{
"type": "EventLog",
"startAtOldestRecord": true,
"eventFormatMultiLine": false,
"channels": [
{
"name": "application",
"level": "Information"
}
]
},
{
"type": "File",
"directory": "$directory",
"filter": "*.log",
"includeSubdirectories": true
},
{
"type": "ETW",
"providers": [
{
"providerName": "IIS: WWW Server",
"ProviderGuid": ""$ProviderGuid",
"level": "Information"
},
{
"providerName": "Microsoft-Windows-IIS-Logging",
"ProviderGuid": "$ProviderGuid",
"level": "Information",
"keywords": "0xFF"
}
]
}
]
}
}

Dockerfile: =====================================
...

Log monitor

RUN [System.Diagnostics.EventLog]::CreateEventSource('TEST.TEST.TEST', 'Application')
RUN $version=$env:LOG_MONITOR_VERSION ; mkdir C:\LogMonitor\ ;[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]'Tls12' ; curl https://github.com/microsoft/windows-container-tools/releases/download/v${version}/LogMonitor.exe -o C:\LogMonitor\LogMonitor.exe
COPY /pipeline/LogMonitorConfig.json /LogMonitor/LogMonitorConfig.json

...
SHELL ["C:\LogMonitor\LogMonitor.exe"]
ENTRYPOINT cmd /c "powershell C:\startup\entrypoint.ps1 & C:\ServiceMonitor.exe w3svc"

[BUG] If level is omitted, the tool does not display all events in the named channel, spanning various levels

According to the documentation for Windows Event Channels:

https://github.com/microsoft/windows-container-tools/tree/main/LogMonitor/docs#event-log-monitoring

"level (Optional): This string field specifies the verboseness of the events collected. These include Critical, Error, Warning, and Information. If the level is not specified, the tool displays all events in the named channel, spanning various levels."

Taking a peek at current code base, it seems level is not optional and to get all logging, the level should be "Verbose"?

UTF8 configuration files are not supported

When creating a configuration file with the following code you get the following error when running the LogMonitor.exe program.

Error:
[2020-11-10T23:42:02.000Z][LOGMONITOR] ERROR: Failed to read json configuration file. JsonFileParser: Error reading a valid value. [2020-11-10T23:42:02.000Z][LOGMONITOR] ERROR: Invalid configuration file.

Powershell repro:
`
$var = @"
{
"LogConfig": {
"sources": [
{
"type": "EventLog",
"startAtOldestRecord": true,
"eventFormatMultiLine": false,
"channels": [
{
"name": "system",
"level": "Error"
}
]
}
]
}
}
"@

$var | Out-File -FilePath "C:\LogMonitor\LogMonitorConfig.json" -Encoding utf8

Only when you use -Encoding ascii is when you're able to get LogMonitor.exe to read the file

LogMonitor does not capture STDOUT

LogMonitor is useful for routing Event Logs, ETW Providers, and Custom App logs to STDOUT for log collection but it doesn't capture STDOUT logs as well. I want to include STDOUT/STDERR logs in addition to the sources supported so that I have better log coverage.

Blank lines in output

It looks like LogMonitor produces 2 CLRF entries for each log line, this could be a bug with conversion to UTF-16.
image
Input log file: log.input.txt
Output log file, produced by .\LogMonitor.exe | Out-File log.output.txt: log.output.txt

Empty (null) messages in EventLog

I get empty messages like this:

<Source>EventLog</Source><Time>2020-02-10T17:34:10.000Z</Time><LogEntry><Channel
>Application</Channel><Level>Error</Level><EventId>0</EventId><Message>(null)</M
essage></LogEntry>

ASP.NET app logs to EventLog like this:

EventLog myLog = new EventLog(); myLog.Source = "myapp"; myLog.Log = "Application"; DateTime now = DateTime.Now; myLog.WriteEntry((now.ToString() + "." + now.Millisecond + ": " + msg), type);

[BUG] process monitor does not support multi-byte characters

Describe the bug
The current process monitor does not support multi-byte characters, e.g.

2023-02-27 03:50:18 - DEBUG: Unicode character 161:¡''
2023-02-27 03:50:18 - DEBUG: Unicode character 162:'¢'
2023-02-27 03:50:18 - DEBUG: Unicode character 163:'£'
2023-02-27 03:50:18 - DEBUG: Unicode character 164:'¤'
2023-02-27 03:50:18 - DEBUG: Unicode character 165:'Â¥'
2023-02-27 03:50:18 - DEBUG: Unicode character 166:'¦'
2023-02-27 03:50:18 - DEBUG: Unicode character 167:'§'
2023-02-27 03:50:18 - DEBUG: Unicode character 168:'¨'
2023-02-27 03:50:18 - DEBUG: Unicode character 169:'©'
2023-02-27 03:50:18 - DEBUG: Unicode character 170:'ª'
2023-02-27 03:50:18 - DEBUG: Unicode character 171:'«'
2023-02-27 03:50:18 - DEBUG: Unicode character 172:'¬'

Expected behavior
Print out the characters exactly as produced by the process.
em.

Configuration
-Tool: Log Monitor
-Version: 2.0.0-rc

Additional Info
Current work-around in v2/main is to print out - for each byte in place of the character, see PR #122

Examples missing

I can read in the README.md:

The repo includes several sample config files for key Windows Container scenarios. For more detail on how to author the config file, see the wiki.

I can't find the examples in the repo. Where are they?

maintenance: update the README

Include, how to contribute, etc.

How to contribute

  • Include details on CLA
  • branching guideline + forking
  • linting instructions and styling guidelines
  • how to follow up on stalled PR
  • PR template

Can't monitor a file at C:

{
    "type": "File",
    "directory": "C:\\",
    "filter": "*.log",
    "includeSubdirectories": false
}

or

{
    "type": "File",
    "directory": "C:",
    "filter": "*.log",
    "includeSubdirectories": false
}

Does not seem to work, and produces error:

[LOGMONITOR] ERROR: Failed to monitor log directory changes. Log directory: \\?\C:, Error: 87
[2020-12-09T04:48:35.000Z][LOGMONITOR] ERROR: Failed to start log file monitor. Log files in a directory \\?\C: will not be monitored. Error: 87

Can not build x32 Version ?

I tried to compile the x32 version, but got the following error message.

\logmonitor\main.cpp(276): error C2664: 'BOOL SetConsoleCtrlHandler(PHANDLER_ROUTINE,BOOL)': 無法將引數 1 從 'BOOL (__cdecl *)(DWORD)' 轉換為 'PHANDLER_ROUTINE'

Chinese character ends in gibberish

We are running Windows container in vms and using logmonitor in container collecting log files containing Chinese. When running docker logs, the English words shows fine, but the Chinese words are showing as gibberish. While in the container, when we open the origin logfiles, both English and Chinese are fine. Please help us.

VM OS Version: Window Server 2019 Version 1809(OS Build 17763.1397)
Container BaseImage Version: Build 10.0.17763.1007

[feature request] Setting to add log file name to every line streamed out

There is this configuration property "includeFileNames": true which tells LogMonitor to print log file name at the beginning of each batch of printed log lines.

An extension to "includeFileNames": true would be useful, e.g. "prependLogLinesWithFileNames": true, which toggles LogMonitor to prepend every line with source log file name.

LogMonitor with named provider without GUID?

This is a feature request to allow access to log equivalent to this

Get-WinEvent -ProviderName "My.Custom.Service.1.2.3"

I have a custom Windows service and I can use that PowerShell cmdlet to access the logs.

However, if I feed the LogMonitor with the following bit in my LogMonitorConfig.json:

{
  "type": "ETW",
  "eventFormatMultiLine": false,
  "providers": [
    {
      "providerName": "My.Custom.Service.1.2.3",
      "level": "Information"
    }
  ]
}

then it complains about the invalid provider:

[2022-07-29T14:24:36.000Z][LOGMONITOR] ERROR: Invalid providers.

I read the source code and I see it is probing every named provider for GUID

for (auto provider : Providers)
{
if (provider.ProviderGuidStr.empty())
{
if (!provider.ProviderName.empty())
{
wstring providerName(provider.ProviderName);
transform(
providerName.begin(), providerName.end(),
providerName.begin(),
towlower);
providersWithoutGuid[providerName] = (provider);
}
}
else
{
ValidProviders.push_back(provider);
}
}
//
// Return now if there aren't providers that need to obtain theirs GUID.
//
if (providersWithoutGuid.empty())
{
return status;
}

It would be nice to allow logs access by named providers only, without GUID.

Setup Question

I used this config file https://github.com/microsoft/windows-container-tools/blob/master/LogMonitor/src/LogMonitor/sample-config-files/IIS/LogMonitorConfig.json and my dockerfile looks as below:

ARG REGISTRY_NAME=acr.azurecr.io
FROM ${REGISTRY_NAME}/os:baseline

COPY ["publish", "/www/os"]

ADD https://build.blob.core.windows.net/logmonitor/LogMonitorConfig.json /LogMonitor/LogMonitorConfig.json
ADD https://build.blob.core.windows.net/logmonitor/LogMonitor.exe /LogMonitor/LogMonitor.exe

WORKDIR /LogMonitor
SHELL ["C:\\LogMonitor\\LogMonitor.exe", "powershell.exe"]

EXPOSE 85

# Start IIS Remote Management and monitor IIS
ENTRYPOINT ["C:\\ServiceMonitor.exe", "w3svc"]

Still I am just seeing The selected container has not logged any messages yet. as part of pod logs.. Am I missing any additional setup or config?

Please help !
@algamaes

How non entry point (random) process running inside Windows Container could write to docker default log output (stdout)?

Hi, first of all, thank's for this project. It's awesome to finally bring Windows Applications to container.

So, I have one Windows Container with various processes, .NET and Win32 legacy applications.

First scenario

Entry point calls myConfigurator.exe, who calls first.exe, who creates a pool of second.exe process

Second scenario

Entry point calls myConfigurator.exe, who calls ServiceMonitor.exe, who initialize my IIS ASP.NET Web Application, who creates a pool of second.exe process

Problem

I'm already using log4net appender to send all logs to elasticsearch, and it's all good.
But, I need to send logs to default docker log too, so datacenter operators can see logs the same way they do with standard/compliant containers, using docker logs, or portainer or any other container tool.

Question

  • How random processes running inside Windows Container could write to default log output?
  • How can I use LogMonitor to address those scenarios?

AB#14 Request for a generic Service Monitor to define as Entrypoint

Hello,

This is a request for a "generic" equivalent to IIS Service Monitor : https://github.com/microsoft/IIS.ServiceMonitor

Within my organisation, we have containerized a WCF service application.
This application is runned as a self-hosted Windows NT service.
We do not need IIS (we want to stay in this self-hosted mode and maintain our image as small as possible).

In this context, we would like to use some "NT Service Monitor" tool as Docker ENTRYPOINT.
The important point here is that we want to benefit of the full capacities of the existing "IIS Service Monitor" tool (see above link), but not only with IIS/w3svc services...

By "full capacities", I mean passing the environments variables to the monitored service.

Also, we would like to monitor multiples services (in some mode, our container has to run -and monitor- 2 NT Services in a single docker instance).

By now, our entry point is a PS script.
Here is an overview of the script content :

  1. copy entrypoint "Process" env vars to the "Machine" level (so that future services can "inherit" them)
  2. start service(s)
  3. monitor service(s)

We use IIS ServiceMonitor.exe only for the third step (even if it can only monitor our "main" service, even if we run multiple.).
We had to code the whole process on own, whereas ServiceMonor.exe can handle step "1)" for IIS App pools...

(Note: For that first step, we had no choice to copy env variables globally (machine scope), an alternative to pass it only to the our monitored services/processes scope would be appreciate...)

To sum up, we think a unique tool should exists to :

  • Start and monitor one (or mutiple) windows services (or process) (not only IIS Apps)
  • Pass the "Dockerfile" env vars to monitored processes and services.
  • Define and optional "exitpoint command" that would be run if some monitored process exits, before the container instance fail (a timeout would be necessary, as this extra command should not hang the failed instance forever).
    (This last point is not mandatory, but it would be appreciate for diagnostic purpose. When our docker instance fail for an obscure reason, we would like to collect some "diagnostic report" (windows events, log files, etc.) and send it to a persistent repository).

All of this should be define as a single ENTYPOINT command on that tool.
A single tool would avoid some custom scripting for folks like us who have windows application containerized, but not using IIS...

Thank you for that opportunity to send this request,
Geoffrey

Log Monitor Feedback and Future Features

Hi everyone! I'm Lucille, and I'm a PM on the Windows Container team working on Log Monitor.

Our team would love to hear from the community on how to improve Log Monitor. We are currently planning for the next versions of Log Monitor and would like to hear feedback.

Here are some guiding questions we would love to hear your thoughts on:

  1. How are you currently using Log Monitor? What problem is it solving for you?
  2. What issues are you facing surrounding Log Monitor?
  3. What features would you like to see?

maintenance: add code linter and checks

Why

We need standardized code for better maintenance, hence a linter will be a necessity.
I see that the current code already has some minor code styling violations that will also need to be fixed.

Options

See tip from @chrischiedo in the comment.

TODO

  • add automated linting checks
  • update the README with the code styling guidelines

[BUG] LogMonitor requires Path to folder to exist before it is started for file logging.

Describe the bug
The Log file handler won't work unless the folder to monitor already exists before LogMonitor.exe is started

To Reproduce
Steps to reproduce the behavior:

  1. Create a log file configuration for a folder that doesn't exist in the docker image, and that is created by the process started.
  2. Launch the docker image with the Logmonitor starting the process that will create the folder
  3. LogMonitor will log an error like "
[2023-03-22T14:30:37.000Z][LOGMONITOR] ERROR: Failed to open log directory handle. Directory: \\?\C:\Logs\AppLogs Error=0
--
  1. If any entry is then written to a file in the c:\logs\applogs\ folder, LogMonitor will not log those errors.

Expected behavior
I would expect LogMonitor to recognize that while the folder doesn't exist when it was launched, it might exist in the future, and log based on it being created later.
OR
Documentation somewhere in the LogMonitor git repo that explicitly outlines this as a hard dependency ("The path to the log files must exist before LogMonitor is started, or it won't be monitored")

Screenshots
If applicable, add screenshots to help explain your problem.

Configuration
-Tool: LogMonitor
-Version: 1.2

Additional context
Add any other context about the problem here.

How to use Logmon when running as ContainerUser

Discussed in #142

Originally posted by rigdal May 15, 2023
Running apps in containers as a non-root user is standard with containers nowadays.

I am new to Windows containers, but it appears that I should leverage USER ContainerUser in my Dockerfile for running as non-root.

However, I cannot seem to start Logmon or maybe there is a better spot to have my USER layer in my Dockerfile. Here is the output I get upon startup.

_npc-oes-webapp  | [2023-05-15T16:48:49.000Z][LOGMONITOR] ERROR: Failed to start ETW trace. Error: 5
npc-oes-webapp  | [2023-05-15T16:48:49.000Z][LOGMONITOR] ERROR: Failed to start ETW trace session. Error: 5
npc-oes-webapp  | [2023-05-15T16:48:49.000Z][LOGMONITOR] ERROR: Failed to start ETW monitor. Error: 5
npc-oes-webapp  | 
npc-oes-webapp  | ERROR: Failed to stop or query status of service 'w3svc' error [80070005]
npc-oes-webapp  | [2023-05-15T16:48:49.000Z][LOGMONITOR] ERROR: Failed to open log directory handle. Directory: \\?\c:\inetpub\logs Error=5
npc-oes-webapp  | [2023-05-15T16:48:49.000Z][LOGMONITOR] ERROR: Failed to start log file monitor. Log files in a directory \\?\c:\inetpub\logs will not be monitored. Error: 5
npc-oes-webapp  | [2023-05-15T16:48:49.000Z][LOGMONITOR] INFO: Entrypoint processs exit code: -2147024891
npc-oes-webapp  | [2023-05-15T16:48:49.000Z][LOGMONITOR] WARNING: The given session is not running. Error: 4201
npc-oes-webapp  | [2023-05-15T16:48:49.000Z][LOGMONITOR] ERROR: Failed to wait for log file monitor to stop. Log directory: \\?\c:\inetpub\logs Error: 2147942406_

[BUG] Debug Assertion Failed! for debug build

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Build Log Monitor in debug mode
  2. Click on CTRL + C
  3. See error

Expected behavior
The compiler raises a Debug Assertion Failed error. I've attached the screen shot.

Configuration
Any version of Log Monitor

Debug

chore: fix CppUnitTest.h not found issue

Error C1083 Cannot open include file: 'CppUnitTest.h': No such file or directory LogMonitorTests c:\dev\windows-containers\windows-container-tools\logmonitor\logmonitortests\pch.h 17

No way to identify the source of messages

There's no way to tell from the output what the source of a message is. For example, if I'm capturing messages from the event viewer logs:

    "type": "EventLog",
    "startAtOldestRecord": true,
    "eventFormatMultiLine": false,
    "channels": [
      {
        "name": "system",
        "level": "Information"
      },
      {
        "name": "application",
        "level": "Error"
      }

The output messages don't identify if the message is from the application or system log. Or, if I'm capturing from log files:

  {
    "type": "File",
    "directory": "c:\\inetpub\\logs",
    "filter": "*.log",
    "includeSubdirectories": true
  },

The messages don't identify which log the message came from. It would be very helpful if some additional information was added to the message indicating the source. In my case, I am capturing information from a set of windows services which each have their own log file. The log file messages can all look very similar and it's impossible to identify where they're coming from. The log file name would help.

Failed to format ETW event property. Unexpected length of 0 for intype 300 and outtype 0

To repro the issue you can tougle the Bluetooth on/off with the listen to bthport provider mentioned below.

iceAddress>233717558281702
[2022-11-25T03:09:40.000Z][LOGMONITOR] ERROR: Failed to format ETW event property. Unexpected length of 0 for intype 300 and outtype 0
[2022-11-25T03:09:40.000Z][LOGMONITOR] ERROR: Failed to query ETW event property length. Error: 15005l
[2022-11-25T03:09:40.000Z][LOGMONITOR] ERROR: Failed to format ETW event user data.
[2022-11-25T03:09:40.000Z][LOGMONITOR] ERROR: Failed to format ETW event property. Unexpected length of 0 for intype 300 and outtype 0

{
"LogConfig": {
"sources": [
{
"type": "ETW",
"eventFormatMultiLine": false,
"providers": [
{
"providerName": "BthPort",
"providerGuid": "A8DD90AF-85F0-40B1-B022-4F54961E8AE5",
"level": "Verbose"
}
]
}
]
}
}

[BUG] 30 second delay

I'm using LogMonitor to monitor a single file, client.log, in a directory, running in a container orchestrated by Kubernetes, in Azure.

The application that the container runs will output roughly 2 lines to client.log every second when idling and potentially thousands of lines every second when active, and keep an open handle to the file at all times.

I'm finding the container output will be up to 30 seconds behind the contents of the file. This observable when comparing the log file entry timestamp from inside the file (right hand column of ss) and the timestamp from the container output log (left hand column):

Screenshot 2023-03-06 170021

I've also noticed that if I connect into the running container, and run a get-content over the file, the container logs immediately catch up, but will then go back to waiting 30 seconds. If I do a get-content -wait on the file, the container logs steam real-time.

This is causing problems when viewing the container output through Azure Log Analytics, as Log Analytics gets an inaccurate "Time-Generated" timestamp, making ordering in Log Analytics wrong, making viewing the logs and troubleshooting confusing and very difficult.

Is this a configurable option, or a known issue, and is there a solution or workaround?

Thanks.

Error when config file is in `utf-8`

  • Version: 1.1 release
  • steps to reproduce:
    • Create a simple config file using utf-8 encoding. In a binary file viewer file will have two special characters in the beginning.

image

  • Run LogMonitor.exe /CONFIG yourConfigWithUTF.json yourApp.exe
  • Error:
[2021-05-12T13:58:35.000Z][LOGMONITOR] ERROR: Failed to read json configuration file. JsonFileParser: Error reading a valid value.                                                                         

[2021-05-12T13:58:35.000Z][LOGMONITOR] ERROR: Invalid configuration file. 

LogMonitor documentation

hey microsoft team! i'm interested in trying Log Monitor for a windows container project. is there any additional documentation on the tool outside of the windows docker container logs troubleshoot page or the example LogMonitorConfig.json?

a LogMonitorConfig schema would be particularly helpful. it'd also be helpful to see what Log Monitor output looks like. i'm not entirely sure how messages from different inputs could be differentiated in log monitors stdout.

End of Life?

Is this project actively maintained? Microsoft container documentation points to this project as a recommended approach, but the project doesn't appear to be maintained.

File watcher hangs

I have a windows docker image and started using this to stream logs from my application log foler to std out. It all work fines but sometimes it stops pushing logs to std out.
At first i thought it might be no new logs but when i open the files , new lines are continuously getting added by the actual application but Logmonitor doesnot puts those new lines anymore. Everytime i have to restart the container. My configuration json file is pulling logs from Eventvwr and log folder. The eventvwr keeps working though.

What could be the possible reasons? Can there be a possibility of file locking which is not allowing the logmonitor to read new lines? Need some urgent guidance on this.

maintenance: update README to include local linting instructions

Update README to include local linting instructions and our coding guidelines.

  • Set up local linting on Windows, using cpplint (wasn't straight forward)
  • Write instructions for Linting from *nix OSes (eg. Ubuntu) or using WSL on Windows.
  • Add the command in the build/make file for better automation

image

XML tag names should not include spaces

The XML that LogMonitor generates has many tag names with spaces, like <Dest IPv6 Address>. Unfortunately that is not valid in XML, and the spaces must be removed - e.g. <DestIPv6Address> - before it is possible to view the XML in a viewer such as Chrome.

I suggest you ensure this in the FormatData code by eliminating spaces when converting from properties to tags.

BTW, some other things needed for viewing XML in the Chrome viewer:

  • add an enclosing tag at the start and end of each line, e.g. <Log>...</Log>

  • add an enclosing tag around the whole file, e.g.

    <Logs>
    ...
    </Logs>
    
  • add standard XML preamble, e.g. <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.