This is exported from internal ticket

Background

Same as open-telemetry/opentelemetry-collector#575 (comment) we are fetching scrape target metadata using relabeled job and instance and it will break like #190

Fix

Using relabel magic like #191 , though this time we need to prepend it in relabel_config

I deployed an EC2 Windows instance this week, including https://s3.amazonaws.com/amazoncloudwatch-agent/windows/amd64/latest/amazon-cloudwatch-agent.msi
The instance started up OK, but CPU was running between 40% and 80% which caused our auto scale group to scale up to the maximum. The Windows system logs include thousands of events like

The CWAgent Otel Collector service terminated unexpectedly. It has done this 11646 time(s). The following corrective action will be taken in 2000 milliseconds: Restart the service.
The CWAgent Otel Collector service terminated with the following error: An exception occurred in the service when handling the control request.
The CWAgent Otel Collector service entered the stopped state.

I have these log entries in CloudWatch logs, so that agent and connector are working correctly.

All our other environments are running CWAGENT_VERSION 1.247346.1b249759 or a previous one, and none have this problem (I see from the release notes that the Otel collector has only been added with this release). This is Windows 2019 built from the latest CIS hardened AMI running .Net classic with all windows and dotNet patches up to date.

The full event logs:

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Service Control Manager' Guid='{555908d1-a6d7-4695-8e1e-26931d2012f4}' EventSourceName='Service Control Manager'/><EventID Qualifiers='49152'>7023</EventID><Version>0</Version><Level>2</Level><Task>0</Task><Opcode>0</Opcode><Keywords>0x8080000000000000</Keywords><TimeCreated SystemTime='2021-01-05T08:37:08.735977500Z'/><EventRecordID>86168</EventRecordID><Correlation/><Execution ProcessID='608' ThreadID='2640'/><Channel>System</Channel><Computer>EC2AMAZ-XXXXXXX</Computer><Security/></System><EventData><Data Name='param1'>CWAgent Otel Collector</Data><Data Name='param2'>%%1064</Data><Binary>430057004100670065006E0074004F00740065006C0043006F006C006C006500630074006F0072000000</Binary></EventData><RenderingInfo Culture='en-US'><Message>The CWAgent Otel Collector service terminated with the following error: An exception occurred in the service when handling the control request.</Message><Level>Error</Level><Task></Task><Opcode></Opcode><Channel></Channel><Provider>Microsoft-Windows-Service Control Manager</Provider><Keywords><Keyword>Classic</Keyword></Keywords></RenderingInfo></Event>	
<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Service Control Manager' Guid='{555908d1-a6d7-4695-8e1e-26931d2012f4}' EventSourceName='Service Control Manager'/><EventID Qualifiers='49152'>7031</EventID><Version>0</Version><Level>2</Level><Task>0</Task><Opcode>0</Opcode><Keywords>0x8080000000000000</Keywords><TimeCreated SystemTime='2021-01-05T08:37:08.735977500Z'/><EventRecordID>86169</EventRecordID><Correlation/><Execution ProcessID='608' ThreadID='4220'/><Channel>System</Channel><Computer>EC2AMAZ-XXXXXXX</Computer><Security/></System><EventData><Data Name='param1'>CWAgent Otel Collector</Data><Data Name='param2'>11646</Data><Data Name='param3'>2000</Data><Data Name='param4'>1</Data><Data Name='param5'>Restart the service</Data><Binary>430057004100670065006E0074004F00740065006C0043006F006C006C006500630074006F0072000000</Binary></EventData><RenderingInfo Culture='en-US'><Message>The CWAgent Otel Collector service terminated unexpectedly. It has done this 11646 time(s). The following corrective action will be taken in 2000 milliseconds: Restart the service.</Message><Level>Error</Level><Task></Task><Opcode></Opcode><Channel></Channel><Provider>Microsoft-Windows-Service Control Manager</Provider><Keywords><Keyword>Classic</Keyword></Keywords></RenderingInfo></Event>	
<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Service Control Manager' Guid='{555908d1-a6d7-4695-8e1e-26931d2012f4}' EventSourceName='Service Control Manager'/><EventID Qualifiers='16384'>7036</EventID><Version>0</Version><Level>4</Level><Task>0</Task><Opcode>0</Opcode><Keywords>0x8080000000000000</Keywords><TimeCreated SystemTime='2021-01-05T08:37:08.735977500Z'/><EventRecordID>86167</EventRecordID><Correlation/><Execution ProcessID='608' ThreadID='2640'/><Channel>System</Channel><Computer>EC2AMAZ-XXXXXXX</Computer><Security/></System><EventData><Data Name='param1'>CWAgent Otel Collector</Data><Data Name='param2'>stopped</Data><Binary>430057004100670065006E0074004F00740065006C0043006F006C006C006500630074006F0072002F0031000000</Binary></EventData><RenderingInfo Culture='en-US'><Message>The CWAgent Otel Collector service entered the stopped state.</Message><Level>Information</Level><Task></Task><Opcode></Opcode><Channel></Channel><Provider>Microsoft-Windows-Service Control Manager</Provider><Keywords><Keyword>Classic</Keyword></Keywords></RenderingInfo></Event>

I have added new file json to the script and executed make build .. build went to no success with the error as below :

check_secrets failed
make: *** [check_secrets] Error 1

We have installed the agent on a device located on a wifi network that redirects to a login page before allowing access to the network.

What I believe is happening - First the agent checks to see if it's an EC2 instance by trying to hit the meta-data service, but the local network is redirecting this request to a login page. The entire contents of this page is then saved as the device hostname.

We are using the hostname within our log-group name. So then an error is thrown when trying to create the log group because the name limit is 512 characters and the content of the web page is much longer than that.

Potential solution - Parse the returned page to make sure it is the ec2 meta-data service that is responding.

Telegraf has native support for capturing metrics on NVIDIA backed GPU instances through the inputs.nvidia_smi plugin
https://github.com/influxdata/telegraf/tree/master/plugins/inputs/nvidia_smi

It would be great if we have this integrated into the CW agent.

We have the cloudwatch agent up and running and we can see the logs coming through. After several hours (and I've been told even a few days on some occasions, the cloudwatch agent stops sending logs to AWS.

I am not sure if this is relevent, but I do see within the cloudwatch logs:

021-03-04T22:51:54Z W! [outputs.cloudwatchlogs] Invalid SequenceToken used, will use new token and retry: The given sequenceToken is invalid. The next expected sequenceToken is: 49612984946004412055118931556677438474248778094089797074
2021-03-04T22:51:54Z W! [outputs.cloudwatchlogs] Retried 0 time, going to sleep 125.739149ms before retrying.

That said, the logs originally update just fine and the timing of the warning does not coincide with the failure to update.

If there is any way to run this with debugging please let me know and I will gladly provide that information.

Ubuntu 20.04.2 LTS
Linux ip-10-65-53-134 5.4.0-1037-aws #39-Ubuntu SMP Thu Jan 14 02:56:06 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent --version
CWAgent/1.247347.5 (go1.15.6; linux; amd64) No Build Date

Source location:
https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

Similar to how the agent supports EC2 instance ID as an append-able dimension, it should also support ECS/Fargate tasks IDs. I'd like to be able to do something like "TaskId":"${aws:ECSTaskId}".

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html#CloudWatch-Agent-Configuration-File-Metricssection

I have a few things that I'd really like to ignore in my cluster that runs Jenkins/CI jobs...I don't need CPU metrics and junk for every builder pod that comes along. Since CW charges by the metric...that gets expensive.

I know my way around Go, if somebody wants to point me at the right spot I'd be happy to develop and test a change on our CI cluster. I'm looking at a number of different spots in the code right now, but I think I should be catching the pod as early as possible in the pipeline. Maybe right at the Watch but I'm just looking to see if I can figure out a field selector that will work...

Hi! Thank you for the software.
I'd like to ask what was the reason to forbid specifying custom global dimensions? I would like to add a global custom dimension environment name: <env name> for ALL metrics from cloudwatch agent for example. Yes, I can add it manually for most of agent inputs, but I can't do this for statsd and collectd, for example.

I'm talking about this part:

append_dimensions – Optional. Adds Amazon EC2 metric dimensions to all metrics collected by the agent. The only supported key-value pairs are shown in the following list. Any other key-value pairs are ignored.

"ImageID":"${aws:ImageId}" sets the instance's AMI ID as the value of the ImageID dimension.
"InstanceId":"${aws:InstanceId}" sets the instance's instance ID as the value of the InstanceID dimension.
"InstanceType":"${aws:InstanceType}" sets the instance's instance type as the value of the InstanceType dimension.
"AutoScalingGroupName":"${aws:AutoScalingGroupName}" sets the instance's Auto Scaling group name as the value of the AutoScalingGroupName dimension.

May be you are planning to add this feature or you can suggest any workaround.
Thanks!

Currently metrics scraped from a prometheus exporter which are not typed are dropped by this agent.

There are use cases (such as scraping a prometheus server federate endpoint) where the the TYPE is not set. It would be benefical to be able to override or set the TYPE for metrics to get around this limitation.

Here is an example of this functionality in the datadog agent.
https://github.com/DataDog/integrations-core/pull/7071/files

after installing the repository in aws linux machine , there will be no config.json file created in the location /opt/aws/amazon-cloudwatch-agent/bin..

I would like to request for running some command so that i can pass the file location as the argument , so the config .json file will be created in the required location ..

I have an EC2(ubuntu 18.04) with CloudWatch Agent (CWAgent) and collectd installed.

I'd like to send some customized metrics from collectd to CWAgent but failed, I have no idea now to debug, could you please guide me how to do that?

Some clues:

• metrics data from collectd can be found in local path (/var/lib/collectd/rrd/ip-10-x-x-x.us-west-2.compute.internal/postgresql-my)
• CWAgent build-in metrics can be found in CloudWatch Metrics dashboard. (IAM role is configured correctly)
• CWAgent daemon runs correctly

  # netstat -tunlp
  
  udp        0      0 127.0.0.1:25826         0.0.0.0:*                           24347/amazon-cloudw
  

Configs:

• /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

{
        "agent": {
                "metrics_collection_interval": 60,
                "run_as_user": "cwagent"
        },
        "metrics": {
                "append_dimensions": {
                        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
                        "ImageId": "${aws:ImageId}",
                        "InstanceId": "${aws:InstanceId}",
                        "InstanceType": "${aws:InstanceType}"
                },
                "metrics_collected": {
                        "collectd": {
                                "metrics_aggregation_interval": 60
                        },
                        "disk": {
                                "measurement": [
                                        "used_percent",
                                        "inodes_free"
                                ],
                                "metrics_collection_interval": 60,
                                "resources": [
                                        "*"
                                ]
                        },
                        "mem": {
                                "measurement": [
                                        "mem_used_percent"
                                ],
                                "metrics_collection_interval": 60
                        }
                }
        }
}

• /etc/collectd/collectd.conf

...
LoadPlugin network
...
LoadPlugin postgresql
...

<Plugin network>
        Server "127.0.0.1" "25826"
</Plugin>
...
<Plugin postgresql>
        <Query connection_count>
                Statement "SELECT sum(numbackends) AS connection_count FROM pg_stat_database;"
                <Result>
                        Type gauge
                        InstancePrefix "connection_count"
                        ValuesFrom "connection_count"
                </Result>
        </Query>
        <Database my>
                Interval 60
                Host "localhost"
                Port 5432
                User "master"
                Password "MYPASSWORD"

                Query backends
                Query connection_count
        </Database>
</Plugin>

Logs:

• /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

2020/10/21 09:21:50 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2020/10/21 09:21:50 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2020/10/21 09:21:50 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json ...
2020/10/21 09:21:50 I! Detected runAsUser: cwagent
2020/10/21 09:21:50 I! Change ownership to cwagent:cwagent
2020/10/21 09:21:50 I! Set HOME: /home/cwagent
2020-10-21T09:21:50Z I! Starting AmazonCloudWatchAgent 1.247345.36
2020-10-21T09:21:50Z I! Loaded inputs: disk mem socket_listener
2020-10-21T09:21:50Z I! Loaded aggregators:
2020-10-21T09:21:50Z I! Loaded processors: delta ec2tagger
2020-10-21T09:21:50Z I! Loaded outputs: cloudwatch
2020-10-21T09:21:50Z I! Tags enabled: host=ip-10-x-x-x
2020-10-21T09:21:50Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-10-x-x-x", Flush Interval:1s
2020-10-21T09:21:50Z I! [logagent] starting
2020-10-21T09:21:50Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started initialization.
2020-10-21T09:21:50Z I! cloudwatch: get unique roll up list []
2020-10-21T09:21:50Z I! [inputs.socket_listener] Listening on udp://127.0.0.1:25826
2020-10-21T09:21:50Z I! cloudwatch: publish with ForceFlushInterval: 1m0s, Publish Jitter: 37s
2020-10-21T09:21:50Z I! [processors.ec2tagger] ec2tagger: Initial retrieval of tags succeded
2020-10-21T09:21:50Z I! [processors.ec2tagger] ec2tagger: EC2 tagger has started, finished initial retrieval of tags and Volumes

• /var/log/syslog

collectd[23736]: plugin_load: plugin "syslog" successfully loaded.
collectd[23736]: plugin_load: plugin "battery" successfully loaded.
collectd[23736]: plugin_load: plugin "cpu" successfully loaded.
collectd[23736]: plugin_load: plugin "df" successfully loaded.
collectd[23736]: plugin_load: plugin "disk" successfully loaded.
collectd[23736]: plugin_load: plugin "entropy" successfully loaded.
collectd[23736]: plugin_load: plugin "interface" successfully loaded.
collectd[23736]: plugin_load: plugin "irq" successfully loaded.
collectd[23736]: plugin_load: plugin "load" successfully loaded.
collectd[23736]: plugin_load: plugin "memory" successfully loaded.
collectd[23736]: plugin_load: plugin "network" successfully loaded.
collectd[23736]: plugin_load: plugin "postgresql" successfully loaded.
collectd[23736]: plugin_load: plugin "processes" successfully loaded.
collectd[23736]: plugin_load: plugin "swap" successfully loaded.
collectd[23736]: plugin_load: plugin "users" successfully loaded.
collectd[23752]: plugin_load: plugin "syslog" successfully loaded.
collectd[23752]: plugin_load: plugin "battery" successfully loaded.
collectd[23752]: plugin_load: plugin "cpu" successfully loaded.
collectd[23752]: plugin_load: plugin "df" successfully loaded.
collectd[23752]: plugin_load: plugin "disk" successfully loaded.
collectd[23752]: plugin_load: plugin "entropy" successfully loaded.
collectd[23752]: plugin_load: plugin "interface" successfully loaded.
collectd[23752]: plugin_load: plugin "irq" successfully loaded.
collectd[23752]: plugin_load: plugin "load" successfully loaded.
collectd[23752]: plugin_load: plugin "memory" successfully loaded.
collectd[23752]: plugin_load: plugin "network" successfully loaded.
collectd[23752]: plugin_load: plugin "postgresql" successfully loaded.
collectd[23752]: plugin_load: plugin "processes" successfully loaded.
collectd[23752]: plugin_load: plugin "swap" successfully loaded.
collectd[23752]: plugin_load: plugin "users" successfully loaded.
collectd[23752]: Systemd detected, trying to signal readyness.
collectd[23752]: Initialization complete, entering read-loop.
collectd[23752]: Successfully connected to database my (user master) at server localhost:5432 (server version: 12.0.4, protocol version: 3, pid: 8692)

In the screen shot below The columns are the following

Timestamp , Ingestion Time , Message , EventID

Data continued to be posted by the agent to cloudwatch, but the timestamp was stuck at 09:01:25 for several hours. At midnight UTC, the problem self corrected. Note, we do not rotate logs at midnight.

If you restart the cloudwatch agent, the timestamp is fixed until it randomly stops changing. Our current work around is to run a daemon which restarts the cloudwatch agent every 15 minutes to ensure our logs are correctly timestamped in AWS.

As you can see in the screenshot below, the actual log time and the ingest time all agree. It is just the reported cloudwatch timestamp that is off.

If there is any way to run this with debugging please let me know and I will gladly provide that information.

Ubuntu 20.04.2 LTS
Linux ip-10-65-53-134 5.4.0-1037-aws #39-Ubuntu SMP Thu Jan 14 02:56:06 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent --version
CWAgent/1.247347.5 (go1.15.6; linux; amd64) No Build Date

Source location:
https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

I'm using the Cloudwatch Agent version 1.247347.5b250583 on a Ubuntu 20.04 EC2 instance. I created the JSON config file using the wizard and manually edited it to include the datetime_format property on my collected logs:

{
   "agent":{ ... },
   "logs":{
      "logs_collected":{
         "files":{
            "collect_list":[
               {
                  "file_path":"/var/log/nginx/error.log",
                  "log_group_name":"nginx_error",
                  "log_stream_name":"{instance_id}",
                  "datetime_format":"%Y/%m/%d %H:%M:%S"
               },
               {
                  "file_path":"/var/log/apache2/error.log",
                  "log_group_name":"apache_error",
                  "log_stream_name":"{instance_id}",
                  "datetime_format":"%a %b %d %H:%M:%S.%f %Y"
               }
            ]
         }
      }
   },
   "metrics":{ ... }
}

I get the following error when trying to start the agent:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

****** processing amazon-cloudwatch-agent ******
/opt/aws/amazon-cloudwatch-agent/bin/config-downloader --output-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --download-source file:/opt/aws/amazon-cloudwatch-agent/bin/config.json --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp
Start configuration validation...
/opt/aws/amazon-cloudwatch-agent/bin/config-translator --input /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json --input-dir /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d --output /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml --mode ec2 --config /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml --multi-config default
2021/03/10 09:03:55 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp ...
2021/03/10 09:03:55 Invalid Json input schema.
2021/03/10 09:03:55 Under path : /logs/logs_collected/files/collect_list/6 | Error : Additional property datetime_format is not allowed
2021/03/10 09:03:55 Configuration validation first phase failed. Agent version: 1.0. Verify the JSON input is only using features supported by this version.

Am I doing something wrong or is this a bug?

My ECS tasks that use cloudwatch agent on EC2 instances work great. The exact same container run on Fargate detect themselves as OnPrem and then break instance role permissions by adding these lines to the toml config:
profile = "AmazonCloudWatchAgent"
shared_credential_file = "/root/.aws/credentials"

As a temporary measure, is there a way to tell cloudwatch agent not to add those 2 lines?

Thanks in advance!

This would be an easy to implement (almost identical make target as the darwin tar one) and very useful addition for systems with deb and rpm package support (distroless images).

It's just a minor issue for people using macOS.

Mac is not using the GNU version of coreutils, See https://stackoverflow.com/questions/7216358/date-command-on-os-x-doesnt-have-iso-8601-i-option

In the Makefile

BUILD = $(shell date --iso-8601=seconds)
LDFLAGS = -s -w
LDFLAGS +=  -X github.com/aws/amazon-cloudwatch-agent/cfg/agentinfo.VersionStr=${VERSION}
LDFLAGS +=  -X github.com/aws/amazon-cloudwatch-agent/cfg/agentinfo.BuildStr=${BUILD}

Example output

$ make build

date: illegal option -- -
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]

And I think it might be better to use Simply expanded variables BUILD := $(shell date --iso-8601=seconds) so all the binaries have the same BuildStr

btw: There is no GitHub Issue Templates

This is diverged from aws-observability/aws-otel-collector#338 and aws-observability/aws-otel-collector#450 cloudwatch agent bundles otel collector as composite agent When otel collector exit with non zero code, Windows will restart it. The (hacky) solution in our otel distro is exit 0 on config error.

The better solution is to provide extra config/script when packaging otel collector in composite agent so we can remove the hack in otel collector.

The Microsoft Windows CloudWatch agent does not support the timezone property within the Windows Event Log collect list unlike the generic log collect list.

Desired Configuration

"windows_events": {
    "collect_list": [
        {
            "event_format": "xml",
            "event_levels": [
                "INFORMATION",
                "WARNING",
                "ERROR",
                "CRITICAL"
            ],
            "event_name": "Application",
            "log_group_name": "/c/windows/system32/winevt/logs/application",
            "timezone": "UTC"
        }
    ]
}

The CloudWatch agent service being unable to start with the Log file for the local Windows instance when the timezone property is specified within the windows_event property:

2020/08/27 14:03:27 Under path : /logs/logs_collected/windows_events/collect_list/0 | Error : Additional property timezone is not allowed". 

This is exported from internal ticket

Background

If user replace metric name directly (i.e. target and source are both __name__

- source_labels: [ __name__ ]
   target_label: __name__
   regex: ielts_score
   action: replace
   replacement: celpip_score

They will see error like

D! receive metric batch with 3 prometheus metrics
2021-03-19T03:34:26Z E! metricsHandler NO metaData for celpip_score | 10.0.0.1:15090 | foo-stats 
2021-03-19T03:34:26Z E! metric_type ERROR: celpip_score|foo-stats|10.0.0.1:15090|  

Root Cause

cwagent is trying to get metric type from the scrape target using relabeled name and it can't find one.

• cwagent get type from the metric cache (actually it's just scrape.Target) https://github.com/aws/amazon-cloudwatch-agent/blob/master/plugins/inputs/prometheus_scraper/metrics_type_handler.go#L176-L190
• the name used in cache is from parser in scrapeLoop.append, which is whatever exposed in the http
  response http://github.com/prometheus/prometheus/blob/8d2a8f493905e46fe6181e8c1b79ccdfcbdb57fc/scrape/scrape.go#L1282-L1282
• cwagent lookup cache using name after relabel http://github.com/aws/amazon-cloudwatch-agent/blob/master/plugins/inputs/prometheus_scraper/metrics_receiver.go#L60-L60

Fix

One solution is keep the original metric name as some internal label so it can survive relabel on __name__ See https://github.com/aws/amazon-cloudwatch-agent/compare/master...pingleig:prometheus-relabel-name?expand=1

Currently run EC2 servers on Ubuntu x86, either version 20.04 or 18.04
I have other services that require x86 and not ARM, one example is AWS Inspector agent, this only works on x86.

Do I need to build from source in order to utilize this tool?

The amazon-cloudwatch-agent-ctl contains the next piece of the code for detecting of systemd [ "$(systemctl | grep -c '\-\.mount')" = 1 ].

    # detect which init system is in use
    if [ "$(/sbin/init --version 2>/dev/null | grep -c upstart)" = 1 ]; then
        SYSTEMD='false'
    elif [ "$(systemctl | grep -c '\-\.mount')" = 1 ]; then
        SYSTEMD='true'
    elif [ -f /etc/init.d/cron ] && [ ! -h /etc/init.d/cron ]; then
        echo "sysv-init is not supported" >&2
        exit 1
    else
        echo "unknown init system" >&2
        exit 1
    fi

Unfortunately, it didn't work on my laptop; when I run the command systemctl | grep '\-\.mount' it shows at least two entities in the list

$ systemctl | grep '\-\.mount'
  -.mount                                                                                  loaded active     mounted   Root Mount                                                                
  tmp-.mount_jetbraP6HI4I.mount                                                            loaded active     mounted   /tmp/.mount_jetbraP6HI4I

There is a solution for systemd detection here https://superuser.com/a/1183819/457737 ps --no-headers -o comm 1, let's use it?

This is exported from internal ticket

TL;DR

The latest image is released, if you were using temp image from this comment #188 (comment) please update to the latest tag.

If the error message W! No pod metric collected, metrics count is still 7 is containerd socket mounted? https://github.com/aws/amazon-cloudwatch-agent/issues/188leads you to this issue

• make sure you have updated the yaml to mount the containerd socket into the cloudwatch-agent pod
• the path for containerd socket may not be in the standard location. e.g. bottlerocket uses /run/dockershim.sock instead of /run/containerd/containerd.sock

Background

We were relying on pause container to have POD for detecting pod, which is the case for docker but not for containerd containerd/cri#922 (comment)

User will not see pod metrics in container insight dashboard and they will find the following log which is introduced in #171

The root cause is we are expecting containerName == 'POD' to mark a path as pod

Fix

• code should be similar to DataDog/integrations-core#2283
• manifest need to mount containerd sock into cwagent daemonset container

Release

The fix will be included in next release, the release date is not determined (yet).

Follow up on #188

Background

• Upstream issue google/cadvisor#2785
• PR google/cadvisor#2872 and google/cadvisor#2936

When using containerd runtime, you will not see container filesystem metrics in structured log.
Node filesystem metrics is still there, and there is no pod filesystem metrics regardless of runtime.

    "device",
    "fstype",
    "container_filesystem_available",
    "container_filesystem_capacity",
    "container_filesystem_usage",
    "container_filesystem_utilization"

Fix

That metrics is supported in both docker and crio and the rough logic is get rootfs dir for the container, base on the storage driver do some calculation, a common helper is also used to simply walk the dir and call Stat on every file (which is also how size limit is enforced by kubelet for emptydir volume).

We might be able to contribute to upstream

I'm using Microsoft AD FS and enabled trace logging as per https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/troubleshooting/ad-fs-tshoot-logging#trace-log however CWAgent is unable to monitor this event log (AD FS Tracing/Debug). I'm not sure if its related to the type of file (%SystemRoot%\System32\Winevt\Logs\AD FS Tracing%4Debug.etl) or the structure of the log that CWAgent doesn't understand. When enabling debugging in the agent there is no hint of any issues, no hint either that the agent is failing/able to read this event log and additionally the CW log group does not get created. Monitoring a regular event log works fine, config.json included below. Any ideas here?

{
"agent": {
"debug": true
},
"logs": {
"logs_collected": {
"windows_events": {
"collect_list": [
{
"event_format": "xml",
"event_levels": [
"WARNING",
"ERROR",
"CRITICAL",
"INFORMATION",
"VERBOSE"
],
"event_name": "AD FS Tracing/Debug", <<<< not monitoring any logs
"log_group_name": "{instance_id}",
"log_stream_name": "Trace-debug"
},
{
"event_format": "xml",
"event_levels": [
"WARNING",
"ERROR",
"CRITICAL",
"INFORMATION",
"VERBOSE"
],
"event_name": "AD FS/Admin", <<<< successfully monitoring logs
"log_group_name": "{instance_id}",
"log_stream_name": "ADFS-admin"
}
]
}
}
}
}

Summary

make release command fails at go test.

Issue details

I cloned the code on new Amazon EC2 instances and followed the building steps at README.md, but make release fails at go test.

$ make release
...(snip)...
# github.com/aws/amazon-cloudwatch-agent/translator/util
translator/util/platform_windows.go:4:1: +build comment must appear before package clause and be followed by a blank line
...(snip)...
ok  	github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/util	0.004s
=== RUN   TestCreds
--- PASS: TestCreds (0.00s)
=== RUN   TestHostName
--- PASS: TestHostName (0.00s)
=== RUN   TestIpAddress
--- PASS: TestIpAddress (0.00s)
PASS
ok  	github.com/aws/amazon-cloudwatch-agent/translator/translate/util	0.003s
FAIL	github.com/aws/amazon-cloudwatch-agent/translator/util [build failed]
?   	github.com/aws/amazon-cloudwatch-agent/translator/util/ec2util	[no test files]
?   	github.com/aws/amazon-cloudwatch-agent/translator/util/ecsutil	[no test files]
?   	github.com/aws/amazon-cloudwatch-agent/translator/util/httpclient	[no test files]
FAIL
make: *** [Makefile:102: test] Error 2

I've already checked the similar issue #123, but sudo yum install -y gcc didn't resolve this issue.
I've already verified building Agent by make build command succeeds itself.

Woud you check this issue and fix it?
If you need more information about my verification details or something is wrong with my verification, please let me know it.

Thanks,

My verfication environments:

[code version]

• commit: b6df94b
  (The current latest commit on master branch)

[OS]
I tried both of two OS below, but the result was the same.

• Amazon Linux 2

$ cat /etc/image-id
image_name="amzn2-ami-hvm"
image_version="2"
image_arch="x86_64"
image_file="amzn2-ami-hvm-2.0.20210219.0-x86_64.xfs.gpt"
image_stamp="2d25-8dac"
image_date="20210219214335"
recipe_name="amzn2 ami"
recipe_id="2388cb54-f1aa-0862-e76d-dca6-4b2b-8167-d379c05d"

• Red Hat Enterprise Linux 8.0

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.0 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.0"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.0 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.0:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.0
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.0"

[Go version]

• 1.16

$ go version
go version go1.16 linux/amd64

If the configuration file ends up with '.tmp' suffix, CloudWatch agent will try to locate it with '.tmp.tmp' suffix. This is problematic in Windows, because you might automate the creation of the config file (Since it can be temporary from the automatic deployment POV) and use New-TemporaryFile which will add the .tmp suffix.

This is a tracker to direct discussion of existing issues into one place. There is no detailed timeline (yet). Please leave feedback if you are planning to use it or have concern for breaking changes (if any).

Background

EKS Graviton support is GA and cloudwatch-agent already has an arm64 build (not containerized). Fludentd has arm64 dameonset build.

Related Issues

• Arm64 support, including multi-arch amazon/cloudwatch-agent
• EKS: ARM64 support for "Container Insights"
• Added NodeSelector to prevent it from running on windows Nodes We are likely to update label selector because fluentd daemonset does not use multi arch image.

Not related issues

• #104 Raspberry pi we are not creating new arm build

Design

• Provide a multi arch image on dockerhub like xray-dameon
  • Existing manifest should still work in theory i.e. pull amd64 on amd64 host, haven't tested it with dockerhub though.
• Provide new repo/tag that contains arm64 amd64 for people using registry/runtime that does not support multi arch image NOTE: Amazon ECR supports it

TODO

• provide full snippet for workaround
• official arm64 image
• Update related manifest in We recommend users to use fluent-bit which has a multi-arch image, for fluentd users need update the image tag manually since the official daemonset image is x86 only.
  • this repo
  • example
  • doc

Workaround

It is possible to build arm image by yourself from official arm64 packages. Running docker buildx or creating an ARM instance on ec2 as a builder box should both work.

You need to modify example manifests to make sure you are

• Using arm64 version of fluentd image
• Using node selector arch=arm64 if your eks cluster is mixing amd64 and arm64 node groups. Even if you build a mult-arch image for cloudwatch-agent, fluentd is still specifying arch using image tag.

I have cloudwatch-agent setup to scrape Prometheus metrics from my pods, however only one of the pods actually has an endpoint exposing those metrics. I am getting everything showing up in Cloudwatch correctly, but the logfiles for cloudwatch-agent are filling up very quickly with

2020-12-08T17:56:40Z D! receive metric batch with 0 prometheus metrics
2020-12-08T17:56:40Z E! Failed to get Job Name and Instance ID from Prometheus metrics.

With debug mode disabled only the 2nd line shows up, but it shows up a few times every second so that's about all you see in the logs.

Is this as simple as adding a check for count > 0 somewhere before decorating? I don't mind making a PR if that is all that is needed.

Hello,

Telegraf supports monitoring the status (running, stopped etc.) of windows services through their "inputs.win_services"-plugin:
https://github.com/influxdata/telegraf/tree/release-1.16/plugins/inputs/win_services
Since CWA is based on Telegraf I wonder if this can be supported by CWA as well?

It would be really helpful to send the status of windows services as metrics to CW using CWA to further integrate against various AWS-services. This can of course be accomplished by sending custom metrics to CW by scripting it but would be great to have it natively in CWA especially since it already seems to be supported by Telegraf.

Any idea if this is on the roadmap and if so if there is an ETA on it?

Best Regards

when pulled the code on aws ec2 machine , make build executed without issues but no rpm deb files are generated in the folders as mentioned in the read me files. after executing the make release resulted in failure , some of the steps were failed as shown below & machine details were provided after the fail steps

FAIL github.com/aws/amazon-cloudwatch-agent/awscsm/csm [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/awscsm/sdkmetricsdataplane [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/cfg/aws [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/cfg/commonconfig [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/cmd/amazon-cloudwatch-agent-config-wizard [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/cmd/config-translator [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/internal/ecsservicediscovery [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/internal/k8sCommon/k8sclient [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/internal/k8sCommon/k8sutil [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/internal/k8sCommon/kubeletutil [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/internal/publisher [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/internal/structuredlogscommon [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/logger [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/metric/distribution/regular [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/metric/distribution/seh1 [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/inputs/awscsm [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/inputs/cadvisor/extractors [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/inputs/demo [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/inputs/k8sapiserver [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/inputs/logfile [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/inputs/logfile/globpath [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/inputs/prometheus_scraper [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/inputs/statsd [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/inputs/statsd/graphite [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/outputs/awscsm [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/outputs/awscsm/providers [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/outputs/cloudwatch [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/outputs/cloudwatchlogs [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/parsers/awscsm [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/parsers/emf [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/processors/delta [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/processors/ec2tagger [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/processors/ecsdecorator [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/processors/emfProcessor [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/processors/k8sdecorator/stores [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/plugins/processors/k8sdecorator/structuredlogsadapter [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/profiler [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/data [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/data/config [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/data/config/logs [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/data/config/metric [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/data/config/metric/collectd [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/data/config/metric/linux [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/data/config/metric/statsd [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/data/config/metric/windows [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/agentconfig [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/basicInfo [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/collectd [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/defaultConfig [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/migration [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/migration/linux [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/migration/windows [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/question [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/question/events [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/question/logs [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/question/metrics [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/serialization [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/ssm [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/processors/statsd [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/tool/util [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/config [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/jsonconfig [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/toenvconfig [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/totomlconfig [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/agent [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/csm [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/globaltags [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/logs [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/logs/logs_collected/files/collect_list [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/logs/logs_collected/windows_events [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/logs/logs_collected/windows_events/collect_list [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/logs/metrics_collected [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/logs/metrics_collected/emf [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/logs/metrics_collected/prometheus/ecsservicediscovery/taskdefinition [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/append_dimensions [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metric_decoration [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/agentInternal [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/collectd [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/cpu [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/disk [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/diskio [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/ethtool [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/mem [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/net [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/netstat [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/processes [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/procstat [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/statsd [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/metrics_collect/swap [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/rollup_dimensions [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/metrics/util [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/translate/util [build failed]
FAIL github.com/aws/amazon-cloudwatch-agent/translator/util [build failed]
make: *** [test] Error 2

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

We are running containers using the ECS optimized AMI and have installed the cloudwatch agent. The cloudwatch agent is configured to emit the TCP established metric netstat_tcp_established, however the metric does not account for TCP connections made by containers. Can the cloudwatch agent support accounting for containers for netstat metrics?

Hello folks, are there any plans to add auto-update of the CloudWatch agent, similar to what the SSM agent does?

https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent-automatic-updates.html

Hello -

We are running this agent to make private connections through VPCE with Cloudwatch service. We configured the "endpoint_override" values and it's definitely attempting to POST data to the endpoint.

The problem is that the CWAgent log says:

RequestError: send request failed
caused by: Post "https://vpce-1234567890kdjfdkjf.logs.us-gov-west-1.vpce.amazonaws.com/": x509: certificate is valid for logs.us-gov-west-1.amazonaws.com, logs-fips.us-gov-west-1.amazon.com, not vpce-1234567890kdjfdkjf.logs.us-gov-west-1.vpce.amazonaws.com

Is there a way to relax TLS verification? maybe some insecure_skip_verify setting somplace?

Thanks,
Keith

Description

This is a known issue when uploading Windows Event Logs to CloudWatch by using windows_events in CloueWatch Agent. Some of Windows Events will be failed to send in the following condition.

When an operation is performed in Windows OS. A bunch of event logs will appear in Event Viewer with Timestamp, event ID, Source service. For example, Chrome application update events could have several related source service registered, (e.g. gupdate, gupdatem). If any of them is not registered itself correctly, it will block the whole event logs being retrieved from CWAgent and upload to CloudWatch.

This is basically some question in regards to documentation on how to monitor EC2 instance deploying cw agent as ECS daemon task.

I have this currently working but not in an optimal way. Is not clear to me or had found any docs on the best practice to do achieve this.

I would like to enable ethtool Network performance metrics but not sure how can I achieve this.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-network-performance.html

With telegraf one can do something like

 environment:
      HOST_ETC: /hostfs/etc
      HOST_PROC: /hostfs/proc
      HOST_SYS: /hostfs/sys
      HOST_MOUNT_PREFIX: /hostfs
    volumes:
      - /etc/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /:/hostfs:ro
      - /etc:/hostfs/etc:ro
      - /proc:/hostfs/proc:ro
      - /sys:/hostfs/sys:ro
      - /var/run/utmp:/var/run/utmp:ro

Is this something supported by cw agent version of telegraf ? or how can this be possible.

The most relevant doc I had found so far is https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/deploy-container-insights-ECS-instancelevel.html

Either way .. I am not getting any metrics related to ethtool being from host or container so I guess that is seeing container networking and as it does not see ENA interface reports no metrics.. is this possible to achieve and if so how?

"metrics": {
    "append_dimensions": {
      "InstanceId": "${aws:InstanceId}"
    },
    "metrics_collected": {
      "ethtool": {
        "interface_include": [
          "ens5"
        ],
        "metrics_include": [
          "rx_packets",
          "tx_packets",
          "bw_in_allowance_exceeded",
          "bw_out_allowance_exceeded",
          "conntrack_allowance_exceeded",
          "linklocal_allowance_exceeded",
          "pps_allowance_exceeded"
         ]
    }
 }

Thanks

In our organization we provide and enforce usage of preconfigured AMIs. As part of those AMIs we want to include CloudWatch agent with a default configuration. The goal is to have out of the box monitoring and logging for every EC2 instance, which is launched with any of the AMIs, while also giving app teams something to start with, not having to install and configure the defaults first.

We are struggling to combine this default configuration with the possibility for an app team to later on modify the configuration to add their own logs, metrics or aggregation dimensions.
We would like to separate default configuration from custom configuration, in order to be able to roll out new default configurations without breaking/overwriting the app teams custom configuration.

Our key problem is: We can not preconfigure the AMI with one configuration file and let a projects load only an extension (covering e.g. only their application specific logs) with another configuration file. You cannot have 2 configuration files (the second only containing "extensions"). You have to integrate extensions by a project into the default configuration file.

Our current solution for this is to split the configuration into multiple files, which we then merge back together (e.g. via shell scripts). I have attached a screenshot highlighting the different parts in that we split the configuration file.

We would like to have the possibility to simply include optional configuration files into the main config file (e.g. with a cwl-config.d directory as you might know it from other configurations). This would enable us to use the standard while achieving all of our requirements, instead of building our own solution, which requires the users to follow exactly our documentation (instead of relying on AWS documentation and standards).

Hi,

I use ssm to configure cloudwatch using AmazonCloudWatch-ManageAgent.

First with configure action, and the following configuration from parameter store :

{
	"agent": {
		"logfile": "c:\\ProgramData\\Amazon\\AmazonCloudWatchAgent\\Logs\\amazon-cloudwatch-agent.log"
	},
	"metrics": {
	    "append_dimensions": {
			"InstanceId": "${aws:InstanceId}"
		},
		"metrics_collected": {
			"Memory": {
				"metrics_collection_interval": 5,
				"measurement": [
					"% Committed Bytes In Use"
				]
			},
			"procstat": [{
				"exe": "KAVFS",
				"measurement": [
					"pid_count"
				]
			}],
			"LogicalDisk": {
				"measurement": [{
					"name": "% Free Space",
					"unit": "Percent"
				}],
				"resources": [
					"*"
				]
			}
		}
	}
}

Then, I use the append action to add some metrics :

{
"metrics": {
    "metrics_collected": {
			"procstat": [
                                {
				"exe": "W3SVC",
				"measurement": [
					"pid_count"
				]
				},
            	{
				"exe": "IISADMIN",
				"measurement": [
					"pid_count"
				]
			}]
}
}
}

Therefore the agent crash trying to merging the configuration

2020/12/01 14:40:53 Reading json config file path: C:\ProgramData\Amazon\AmazonCloudWatchAgent\Configs\ssm__infr-adm-edfx_metrics_iis.tmp ...

2020/12/01 14:40:53 Reading json config file path: C:\ProgramData\Amazon\AmazonCloudWatchAgent\Configs\ssm__infr-adm-edfx_metrics_standardWindows ...

2020/12/01 14:40:53 Failed to merge multiple json config files.

2020/12/01 14:40:53 Under path : /metrics/metrics_collected/procstat | Error : Different values are specified for procstat

How to handle this ?

Hi!

I'm using this project in order to monitor all of our AWS servers the same way.

We have some EKS self-managed nodes deployed and I was interested in grabbing the disk size of any EBS volumes that would be mounted on a node.
It appears that cloudwatch agent does not see the volumes and thus does not export it to cloudwatch metrics. This is deploying on https://aws.amazon.com/marketplace/pp/B07M68CJS5?ref_=srh_res_product_title which is hardened amazon-linux-2 instance.

I tried to read through the code but could not understand where the logic that grabs the mount points. I ended up reading gopsutil which I suspect it is what is used here but nothing that could indicate that specific volumes are filtered.

This is an example of my configuration specific to metrics:

     "disk": {
        "measurement": [
          "used_percent"
        ],
        "drop_device": true,
        "resources": [
          "*"
        ],
        "ignore_file_system_types": [
          "sysfs",
          "devtmpfs",
          "tmpfs"
        ]
      }
    },

And it works great, I do see the % used in cloudwatch every minute. But I do not see the other mount points. Specifically:

# grep nvme /proc/self/mountinfo
59 0 259:1 / / rw,noatime shared:1 - xfs /dev/nvme0n1p1 rw,attr2,inode64,noquota
863 59 259:1 /var/lib/kubelet/pods/f2e4450b-f54a-4828-9d65-709eb3d47731/volumes/kubernetes.io~configmap/prometheus-config/..2021_04_15_20_58_12.954665680/prometheus.yml /var/lib/kubelet/pods/f2e4450b-f54a-4828-9d65-709eb3d47731/volume-subpaths/prometheus-config/prometheus/1 rw,noatime shared:1 - xfs /dev/nvme0n1p1 rw,attr2,inode64,noquota
1132 59 259:3 / /var/lib/kubelet/plugins/kubernetes.io/aws-ebs/mounts/aws/us-east-1a/vol-0f38122ac3fb9338d rw,relatime shared:188 - ext4 /dev/nvme1n1 rw,data=ordered
1182 59 259:3 / /var/lib/kubelet/pods/7bf88b2b-fc75-4fdd-8aa2-5d74a98a5e17/volumes/kubernetes.io~aws-ebs/pvc-9487c72d-42d2-4079-b804-61f45b0e423e rw,relatime shared:188 - ext4 /dev/nvme1n1 rw,data=ordered
2154 59 259:6 / /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-e61cc91c-2d71-4ee5-adbf-ca2a64006293/globalmount rw,relatime shared:253 - ext4 /dev/nvme4n1 rw,data=ordered
2159 59 259:6 / /var/lib/kubelet/pods/02e68d50-5a2f-4442-8be7-cac7b8d610e2/volumes/kubernetes.io~csi/pvc-e61cc91c-2d71-4ee5-adbf-ca2a64006293/mount rw,relatime shared:253 - ext4 /dev/nvme4n1 rw,data=ordered
2164 59 259:4 / /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-37fc4d8f-0a28-47a4-9b06-f8e9198d4e4c/globalmount rw,relatime shared:256 - ext4 /dev/nvme2n1 rw,data=ordered
2169 59 259:4 / /var/lib/kubelet/pods/ea168af4-c1b5-48fe-9918-7b005c8d66ee/volumes/kubernetes.io~csi/pvc-37fc4d8f-0a28-47a4-9b06-f8e9198d4e4c/mount rw,relatime shared:256 - ext4 /dev/nvme2n1 rw,data=ordered
2200 59 259:5 / /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3cbd32f0-abab-4cde-81cc-dfed92188fe0/globalmount rw,relatime shared:259 - ext4 /dev/nvme3n1 rw,data=ordered
2205 59 259:5 / /var/lib/kubelet/pods/17c774a1-0886-4eb9-85c2-0d1f1eff0483/volumes/kubernetes.io~csi/pvc-3cbd32f0-abab-4cde-81cc-dfed92188fe0/mount rw,relatime shared:259 - ext4 /dev/nvme3n1 rw,data=ordered
2221 59 259:7 / /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-8ec14585-5d1d-4d35-b896-375660bf6541/globalmount rw,relatime shared:268 - ext4 /dev/nvme5n1 rw,data=ordered
2263 59 259:7 / /var/lib/kubelet/pods/3c1e612f-014b-483c-af16-a102beaf025a/volumes/kubernetes.io~csi/pvc-8ec14585-5d1d-4d35-b896-375660bf6541/mount rw,relatime shared:268 - ext4 /dev/nvme5n1 rw,data=ordered
2505 59 259:7 /postgres /var/lib/kubelet/pods/3c1e612f-014b-483c-af16-a102beaf025a/volume-subpaths/pvc-8ec14585-5d1d-4d35-b896-375660bf6541/db/0 rw,relatime shared:268 - ext4 /dev/nvme5n1 rw,data=ordered
2640 59 259:4 /postgres /var/lib/kubelet/pods/ea168af4-c1b5-48fe-9918-7b005c8d66ee/volume-subpaths/pvc-37fc4d8f-0a28-47a4-9b06-f8e9198d4e4c/db/0 rw,relatime shared:256 - ext4 /dev/nvme2n1 rw,data=ordered
2799 59 259:8 / /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-3d7dd169-f9fc-4050-8df7-7f1dc1df0415/globalmount rw,relatime shared:307 - ext4 /dev/nvme6n1 rw,data=ordered
2804 59 259:8 / /var/lib/kubelet/pods/ff2e7af6-bcc0-4985-822f-280636a48a07/volumes/kubernetes.io~csi/pvc-3d7dd169-f9fc-4050-8df7-7f1dc1df0415/mount rw,relatime shared:307 - ext4 /dev/nvme6n1 rw,data=ordered

I'm grabbing /proc/self/mountinfo as an example here because that's what I read in the gopsutil project.

Those mount points are done from the aws ebs daemonsets on kubernetes and are not part of the /etc/fstab file because they come and go.

I know that I can grab the same metrics using other tools, but I chose this tool to be deployed everywhere as it works out of the box for cloudwatch.

Thank you!

Hi there,

Would it be possible to expose allowed_pending_messages configuration option in the CW Agent configuration (here:

We are running into this issue:

2020-10-21T19:46:50Z E! Error: statsd message queue full. We have dropped XXX messages so far. You may want to increase allowed_pending_messages in the config

And would like to increase that value.

According to this README it should be possible to modify that value through the config file, but I'm not sure if that is available in the CW Agent as well (looks like that README is from telegraf).

Thanks!

We would like to collect the performance logs from Kubernetes, like Container Insights already does, but to only create the log group and not create custom metrics from these logs. The custom metrics are expensive, and we are not using them.

If only there was an option to allow disabling these custom metrics by making this line conditional ...

How should I configure the CloudWatch Agent to make sure all collected data pushed away when I stop the agent?
I tried many options, but wasn't able to make it work as expected.

We use metrics to collect statistics in our application. Thus, we expect the metrics to be precise enough.
We have several ASG clusters with installed CloudWatch Agent with StatsD plugin on each EC2. There are several modules sending metrics to the agent. We found some metrics lost when ASG scales down. During investigation and some manual tests we found the agent doesn't publish collected data even if the amazon-cloudwatch-agent service stopped gracefully.

Can it be just incorrect configuration or WAD feature?

My configuration:

{
  "agent": {
    "metrics_collection_interval": 60,
    "omit_hostname": true,
    "logfile": "/var/log/amazon-cloudwatch-agent.log"
  },
  "metrics": {
    "namespace": "MyNameSpace",
    "append_dimensions": {
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}"
    },
    "metrics_collected": {
      "statsd": {
        "metrics_collection_interval": 60,
        "metrics_aggregation_interval": 0
      },
      "mem": {
        "metrics_collection_interval": 60,
        "metrics_aggregation_interval": 0,
        "measurement": [
          "used_percent"
        ]
      }
    }
  }
}

For manual test we used:

• start the agent service
• send 1000 data points for a single metric by netcat
• wait about 20 seconds
• stop the agent service

Thanks.

I discovered today that the Amazon Cloud Watch Agent is continuously pulling custom log files causing sustained CPU usage on our AWS servers. Shouldn't the agent pull logs only at each collection interval? Instead, I see it constant as if it is stuck in an endless loop.

In cases where we are collecting IISLogs, and IIS has generated tens of thousands of log files in different directories, this causes the Cloud Watch Agent to produce sustained, high CPU usage as it is constantly perform a recursive directory listing and reading the most recent files. This can be seen in process monitor and it doesn't matter what log files are specified.

Here is am example config file: (But, I've confirmed it does not matter what log files are specified, even if it is a single file)
{ "logs": { "logs_collected": { "files": { "collect_list": [ { "file_path": "D:\\inetpub\\logs\\LogFiles\\**.log", "log_group_name": "IISLogs", "log_stream_name": "{instance_id}", "timezone":"UTC", "timestamp_format":"%Y-%m-%d %H:%M:%S" } ] } } } }

Even if I add a metrics_collection_interval it ignores it.

Looking back in our cloudwatch metrics, it appears this problem has been occurring for a long time - at least several months.
Agent Version 1.247345.34

Describe the bug
I've installed (or updated) the Amazon CloudWatch agent to version amazon-cloudwatch-agent-1.247345.31-1.amzn2.x86_64 via package-manager. Once you request the current agent version via amazon-cloudwatch-agent-ctl you getting no value for key "version" within JSON.

CloudWatch version number
1.247345.31-1

Platform/OS/Hardware/Device
Linux/Amazon Linux 2/EC2

To Reproduce (observed behavior)

• install 1.247345.31-1 on Amazon Linux 2 (please use exact this version - see 'Additional context') via yum
• issue the command on the shell:
  /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status

{
  "status": "running",
  "starttime": "2020-09-20T16:17:38+0000",
  "version": ""
}

Expected behavior
I get the currently installed agent version within the returned JSON by using amazon-cloudwatch-agent-ctl.

Additional context
The above-named package is the current [checked at 21.09.2020] package within Amazon Linux 2 Repository "amzn2-core". Currently, all local packages/versions within my OS are managed by yum. Having this said, a potential fix needs to be available via Repository "amzn2-core" as well.

I'm using CloudFormation to create an EC2 Instance with CloudWatch Agent and a CloudWatch Alarm if the disk_used_percent is above a certain threshold, or data is missing (assuming that means the instance is hosed).

The alarm unfortunately triggers at first because there is a delay before CloudWatch Agent sends the first metrics. I set force_flush_interval to 0 but it still appears to wait for the first metrics_collection_interval to elapse before collecting and sending the metrics, so in my UserData script I have to sleep 70 before calling cfn-signal on the instance, so that the alarm doesn't get created until data is present.

On top of that, I only need a 5 minute data interval, so to avoid pointless wait during deployment I have to start the agent with a 1 minute interval, sleep, and then reconfigure the agent to use a 5 minute interval.

This has been tedious and time-consuming to set up. I wish there were a config option to collect and send metrics immediately on startup, or at least the subset of metrics that can be collected instantaneously.

This code should be !out.hasMeta && !out.hasSuperMeta based on the comment that it should skip if there are no meta characters. I'm not sure if this really affects anything since the Match() method would still react correctly.

I’d like to run this on my raspberry pi, which is not using arm64, but just arm. Can arm also be a supported platform?

We're trying to get the AWS Cloudwatch Agent working on an on-prem Windows system (no IMDS) with temporary IAM role credentials. We are having difficulty because Amazon Cloudwatch Agent doesn't fallow the default credential provider chain.

We have the ability to retrieve temporary 1 hour IAM credentials and refresh them at will. We're using weep, which has the ability to emulate the ECS credential provider. Unfortunately, amazon-cloudwatch-agent doesn't honor the AWS_CONTAINER_CREDENTIALS_FULL_URI environment variable.

We can also emulate the instance metadata service on our system, but in Windows we can't find a way to translate the 169.254.169.254 IMDS address to 127.0.0.1 . We can't point it to a custom metadata IP because this address is hardcoded in amazon-cloudwatch-agent.

We also tried to use the AWS-supported credential_process flow, but amazon-cloudwatch-agent doesn't honor this field.

The other option we're trying is less ideal, it's basically writing credentials to file and refreshing that periodically to keep them fresh on disk.

Is it possible to add a fallback to amazon-cloudwatch-agent so that it abides by the default credential provider chain, like the AWS Go SDK? Or is there a better way to do what we are trying to accomplish?

Thank you!

Right now new CW agent is available for all kind Operating Systems for AMD64 & ARM64 architectures as listed in the article

However, if I tried to install the same in my Alpine base image, I am not yet successful, I tried to compile it from source:

This attempt failed as Alpine uses the musl libc and the OpenRC init system.
Is there anyway I can use Unified CW agent in my alpine based system?

I've started cloudwatch agent，and I configured port 8125,then what path can I get the data of metrics from my browser。