As far as I understand, in order for this agent to recognize the log file, you must specify the file name in the conf file. However, this approach, as I see it, requires me to change the conf every time I add a new custom log file in my log directory.

Is it possible to provide the ability to specify a log directory and have the agent auto discover the .log files within the specified directory? The configuration settings should be propagated to each found log file. The log_stream_name should take the configured standard and append the actual found log file name.

All the ingredients exist... the Docker server API 9+, log-driver options,.. heck you can even log directly into the ECS ec2 instance and manually run a docker container with the awslog driver. Is there anything beyond the UI adjustment?

Hello,

We are running our application in AWS ECS and we are trying to collect our container logs via TCP port using gelf log driver. But if I used TCP in --log-driver-options --gelf-address tcp://ipaddress:port my container is not starting.

docker: Error response from daemon: gelf: endpoint needs to be UDP.

Kindly suggest how to use TCP in gelf-address option.

My docker version is 1.12

Regards,
Raja

I have the following directory structure:

• /opt/workspace/module1/logs/module1.log
• /opt/workspace/module2/logs/module2.log
• /opt/workspace/module3/logs/module3.log

Is there a good way for AWS CloudWatch to auto discover all the log files under the parent workspace folder ?

What we are trying to achieve is have a single log group with multiple log streams per module
so there would be 3 log streams under 1 log group for the above

• module1 log stream
• module2 log stream
• module3 log stream

We do not want to register a new log stream every time there is a new module
If this is not supported out of the box could you please let me know what is the best way to automate this?

Please note I have already checked JIRA, this is not my use case: [https://github.com//issues/1]

Hello,

I am using awslogs driver to collect my ECS container logs and I enabled Lambda (LogsToElasticsearch_mytestloges) to stream my logs to AWS elasticsearch, but it is not taking ECS Service name, Container instance name, task definition and Docker image name.

The below are my log sample on AWS Elasticsearch. Here I cant find docker image name, ECS Service name and task definition.

{
"_index": "cwl-2017.01.03",
"_type": "mytestloges",
"_id": "33081504929870395169036308436466460047129742806680469507",
"_score": 1,
"_source": {
"Message": 1482993472485,
"Request": "request",
"Rsponse": "request received",
"Method": "GET",
"Info": "INFO",
"Information": "services-shelters",
"URL": "/favicon.ico",
"Plugin": "0|app |",
"@id": "33081504929870395169036308436466460047129742806680469507",
"@timestamp": "2017-01-03T06:46:25.785Z",
"@message": ""0|app |" 1482993472485 INFO "services-shelters" request "GET" "/favicon.ico" "request received"",
"@owner": "855158544446",
"@log_group": "mytestloges",
"@log_stream": "nodejs-app/blue-green-task/9f158062-346b-4a07-b2d3-e6bdd9321dee"
},
"fields": {
"@timestamp": [
1483425985785
]
}
}

Someone kindly suggest how to proceed further.

Regards,
Raja

In gunicorn to specify syslog you give it the syslog address. How to you find the syslog url for the cloudwatch container

This is how you specify the syslog in aws coud

syslog_addr
--log-syslog-to SYSLOG_ADDR
udp://localhost:514
Address to send syslog messages.

Address is a string of the form:

unix://PATH#TYPE : for unix domain socket. TYPE can be stream for the stream driver or dgram for the dgram driver. stream is the default.
udp://HOST:PORT : for UDP sockets
tcp://HOST:PORT : for TCP sockets

Hello,

I am using awslogs driver to collect my ECS container logs and I enabled Lambda (LogsToElasticsearch_mytestloges) to stream my logs to AWS elasticsearch, but it is not taking ECS Service name, Container instance name, task definition and Docker image name.

The below are my log sample on AWS Elasticsearch. Here I cant find docker image name, ECS Service name and task definition.

{
"_index": "cwl-2017.01.03",
"_type": "mytestloges",
"_id": "33081504929870395169036308436466460047129742806680469507",
"_score": 1,
"_source": {
"Message": 1482993472485,
"Request": "request",
"Rsponse": "request received",
"Method": "GET",
"Info": "INFO",
"Information": "services-shelters",
"URL": "/favicon.ico",
"Plugin": "0|app |",
"@id": "33081504929870395169036308436466460047129742806680469507",
"@timestamp": "2017-01-03T06:46:25.785Z",
"@message": ""0|app |" 1482993472485 INFO "services-shelters" request "GET" "/favicon.ico" "request received"",
"@owner": "855158544446",
"@log_group": "mytestloges",
"@log_stream": "nodejs-app/blue-green-task/9f158062-346b-4a07-b2d3-e6bdd9321dee"
},
"fields": {
"@timestamp": [
1483425985785
]
}
}

Someone kindly suggest how to proceed further.

Regards,
Raja

Hi team,

I am not sure this is the correct place to ask my request. I am trying to put some pattern while passing my Cloudwatch logs to AWS ES. Let's assume my logs look like below lines.

Running:on:http://localhost:8081
Running:on:http://localhost:8081
Running:on:http://localhost:8081
Running:on:http://localhost:8081
Running:on:http://localhost:8081
Running:on:http://localhost:8081

How can I split using colon-delimited filter in AWS Cloudwatch Filter pattern. Kindly someone suggest how to fix this.

Regards,
Raja

Since you are not running cron, you won't have logrotate, and the logs will eventually take up a LOT of disk space in the container. Right? IT would be really nice if we could just forward the logs off in the rsyslog config rather than writing them to a file then loading them back up.

Hi,

I ran into an issue but am not sure if is a bug.

I have TaskDefinition #1 that was configured to send container log to cloudwatch log group name LG1. The stream prefix was created successfully for TaskDefinition #1. Then a second TaskDefinition failed to create the stream prefix to the same log group "LG1" with error message "LG1 already exists in stack reference the arn of TaskDefinition #1".

I wonder if TaskDefinition #2 cannot be added to the existing LG1.

Thanks In Advance,
DT

Hi Team,

It is a kind question. I am just curious to know why logstash is the best tool to send logs to elasticsearch. We are running our all applications in AWS so, we are planning to use AWS Cloudwarch logs to collect and send application logs to elasticsearch. please give some suggestion for best practices.

Regards,
Raja