Support needed for RFC5424 syslog format about oms-agent-for-linux HOT 11 CLOSED

Hi @MarianoMunoz , when you say OMS Agent does not seem to allow them to be installed - is there a specific error message when installing the plugin?

from oms-agent-for-linux.

Hi @agup006 , I was attempting to install newsyslog input plugin ( https://github.com/athenahealth/fluent-plugin-newsyslog ) using the standard fluent-gem command ( sudo /opt/microsoft/omsagent/ruby/bin/fluent-gem install fluent-plugin-newsyslog ) on an ubuntu 14.04 LTS VM, so that I could parse RFC5424 logs with the OMS agent.

There is no error when installing the gem, but the path where the plug in files are installed ( /opt/microsoft/omsagent/ruby/lib/ruby/gems/2.2.0/gems/fluent-plugin-newsyslog-0.1.1/lib/fluent/plugin ) appears to be different from where the OMS agent expects to find it ( /opt/microsoft/omsagent/ruby/lib/ruby/gems/2.2.0/gems/fluentd-0.12.14/lib/fluent/plugin ). I don't know if this is result of different default paths being used.

I update the OMS agent configuration at /etc/opt/microsoft/omsagent/conf/omsagent.conf with the following to use the new input plugin.

When I restart the service ( sudo /opt/microsoft/omsagent/bin/service_control restart ), I get an error that is a result of missing dependencies. Are there different paths where the OMS agent is looking for gems? Is there some other method to install plug ins like this one?

Is there any planned support for RFC5424 logs with the Linux OMS agent and Log Analytics platform?

from oms-agent-for-linux.

@MarianoMunoz The version of fluentd that we use is configured to use /opt/microsoft/omsagent/plugin for it's plugins. Furthermore, we have an .so file for a module, Libomi.so, that lives in the same directory.

You might try adding some soft links in the /opt/microsoft/omsagent/plugin for the dependencies that are missing.

If that doesn't work, let me know, and I'll loop someone in that understands the gem architecture of Ruby better than I do.

Having the precise error message that you're getting when trying to start up OMS would be helpful as well.

from oms-agent-for-linux.

@jeffaco I was able to get the OMS agent to start without any errors by dropping the 2 ruby files ( here - https://github.com/athenahealth/fluent-plugin-newsyslog/tree/master/lib/fluent/plugin ) into the /opt/microsoft/omsagent/plugin path. I had to modify the "in_newsyslog.rb" file at line 47 from require 'fluent/plugin/parser_newsyslog' to require_relative 'parser_newsyslog' so that it would see the parser. This removed the previous error I received when restarting the agent.

I pointed my java application that is using log4j2 to send RFC5424 formatted logs at port 25224. I am not seeing those messages in the log analytics platform yet, but I do see the heartbeat message, meaning the OMS agent is now running. I will continue to troubleshoot, but I have made some progress thanks to your input.

I do not see any errors in the /var/opt/microsoft/omsagent/log/omsagent.log

2016-06-18 07:31:13 +0000 [info]: reading config file path="/etc/opt/microsoft/omsagent/conf/omsagent.conf"
2016-06-18 07:31:13 +0000 [info]: starting fluentd-0.12.14 without supervision
2016-06-18 07:31:13 +0000 [info]: gem 'fluentd' version '0.12.14'
2016-06-18 07:31:13 +0000 [info]: adding filter pattern="oms.syslog." type="filter_syslog"
2016-06-18 07:31:13 +0000 [info]: adding match pattern="oms.blob." type="out_oms_blob"
2016-06-18 07:31:13 +0000 [info]: adding match pattern="oms.** docker.**" type="out_oms"
2016-06-18 07:31:13 +0000 [info]: adding source type="exec"
2016-06-18 07:31:13 +0000 [info]: adding source type="exec"
2016-06-18 07:31:13 +0000 [info]: adding source type="oms_omi"
2016-06-18 07:31:13 +0000 [info]: adding source type="oms_omi"
2016-06-18 07:31:13 +0000 [info]: adding source type="oms_omi"
2016-06-18 07:31:13 +0000 [info]: adding source type="oms_omi"
2016-06-18 07:31:13 +0000 [info]: adding source type="newsyslog"
2016-06-18 07:31:13 +0000 [info]: using configuration file:

I will continue to experiment with the configuration and attempt to send an RFC5424 formatted message through the linux Logger utility.

from oms-agent-for-linux.

@MarianoMunoz It can sometimes take up to 15 minutes for messages that get posted to the OMS server to make it through the pipeline and show up in search results. Did you wait that long?

Other things of interest for syslog:

• If you're using syslog, TCP/IP vs. UDP matters. You need to be consistent.
• For the standard syslog daemon on RedHat 7 or CentOS 7, we drop a file into /etc/rsyslog.d/95-omsagent.conf to configure syslog itself to send OMS messages. Other platforms have similar hooks. That gets OMS the actual messages.
• If worse comes to worse, you could turn on diagnostics, or you can run the entire server in diagnostics mode which will allow you to trace messages. That's a standard fluentd thing, so look for the startup parameters to fluentd, stop the server, and start by hand if you'd like with special options. See fluentd for more information.
• You can enable diagnostics in the output plugin to insure the messages are going out to OMS.

Note (this is important): OMS has particular formats for specific data types. Thus, your RFC 5424 message would need to get reformatted to look like a standard syslog message in order for OMS to ingest it properly. We are working on a feature, custom logs, that will give you a lot more flexibility in that area, but I'm not sure of the release cycle for that feature.

If you formatted a message improperly, it would be sent up to the OMS server, but OMS would discard it. So you may want to compare how your RFC 5424 messages look (see what the output plugin is sending) compared to standard syslog messages, and if necessary, reformat them as appropriate before sending to the output plugin.

Hopefully there's enough detail here for you to diagnose the issue and fix it.

Hope this helps,

/Jeff

from oms-agent-for-linux.

@jeffaco Thank you for the information. I waited several hours for the logs to appear. The only message I see is the hourly heartbeat from the OMS agent. I am attempting to bypass the rsyslog service built into Ubuntu and deliver messages directly to the OMS agent. It drops a valuable part of the log message when it converts it to RFC3164 and then forwards to the OMS agent. I have successfully sent messages to the agent directly using the linux logger utility without issue, so I don't want to use a middle man service if direct messaging is supported.

I will look into the fluentd diagnostics for the output plugin. I am still in the process of learning about the tool. I will report my results later.

Is there some location with diagnostics I can check in the OMS portal to see if the messages are being dropped? I have not found one.
Is there some documentation available for the log analytics api?

from oms-agent-for-linux.

@MarianoMunoz Your best bet, I think, is to compare debug output from fluentd logging when working via rsyslog with the debug output when not working from RFC5424. If you make the RFC5424 messages look the same, you should be in good shape.

In fact, you can even create a formatter plugin to not lose valuable portions of log messages (although, until custom logs come along, it may be hard to search for those portions).

There are limited diagnostics (at least from the portal) if OMS ingestion has a problem, although @robbiezhang is far more familiar with this area than I am.

from oms-agent-for-linux.

It can sometimes take up to 15 minutes for messages that get posted to the OMS server to make it through the pipeline and show up in search results.

@jeffaco We wanted to use OMS as a debug tool in order to see what's happening in our MS Azure IaaS environment based on Linux VMs. The OMS overview page states

The OMS Agent for Linux enables rich and real-time analytics for operational data (Syslog, Performance, Alerts, Inventory) from Linux servers

Now I'm confused. A 15 minutes delay is far from real-time. Is there any way to speed up the process through the pipeline?

Thanks in advance for your support.

from oms-agent-for-linux.

@ghislinc I said It can sometimes take up to 15 minutes (worst case). It's often much faster.

@agup006 Can speak more about accurately about actual OMS performance characteristics.

from oms-agent-for-linux.

@ghislinc - During our initial private preview Linux ingestion could take up to 15 minutes to full take fruition in Log Analytics Search.

Since that private preview, we have made numerous service and agent improvements to rapidly increase our reliability and time for message to be available in search. Additionally, you can view the average time for a batch of messages [30-45 seconds across our Linux customer] to be processed within the OMS Portal under usage.

@MarianoMunoz - we are investigating additional syslog RFCs to support, and have placed this item in our backlog. In the interim our Custom JSON pipeline might be able to help bring this data into Log Analytics: https://github.com/Microsoft/OMS-Agent-for-Linux/blob/master/docs/OMS-Agent-for-Linux.md#custom-json-data-sources. I am going to close this issue and mark as a feature request.

Thanks,
Anurag

from oms-agent-for-linux.

Hello all,

Is there any update about this thread ?
I was faced to the same issue MarianoMunoz is encountering, and i can't make the OMS agent start after adding the 'newsyslog' type. The OMS Agent service fails to start.

from oms-agent-for-linux.