umputun / docker-logger Goto Github PK

I have a docker-compose.yml without TIME_ZONE, but I get this first lines in docker-logger log:

timezone=America/Chicago
Thu Apr  4 04:54:44 CDT 2019
docker-logger v1.2.0-fb4ef2c-20190328T09:57:08

And logger is writing Chicago's time, however according to README default time zone should be UTC.

Probably the reason is in this line: baseimage/base.alpine/Dockerfile

Currently, it uses dep. Going to switch to go mod in vendor mode

Hi! Thanks for radio-t and this great tool

Given I use environment LOG_FILES=true
When I deploy logger and service using docker swarm
Then docker ps shows random names (for each deploy swarm starts a container with random postfix: new log file is created)
And I got a million of small log files
-rw-r--r-- 1 root root 502 Aug 28 11:20 citlk_prod_back.1.h9laxuf7hhuuw4yznoigua1sv.err
-rw-r--r-- 1 root root 502 Aug 27 16:31 citlk_prod_back.1.hqf074zou6m0qwbq1f1kjup84.err
-rw-r--r-- 1 root root 502 Aug 20 15:17 citlk_prod_back.1.jeea6umrmwbph7prkxqa2gihu.err
-rw-r--r-- 1 root root 502 Aug 27 17:35 citlk_prod_back.1.kbcf9mef0yzyxr092i40cteqp.err

I would like to log to single file with COMPOSE_PROJECT_NAME + service name as log file name
I cant't find any naming setting for swarm that can solve this
I would like to try fix this or find a workaround

• Do you use this tool in production?
• Do you have any clues how to get service name or compose project name from dockerEvent.Actor.Attributes here
  

  docker-logger/app/discovery/events.go

  Line 80 in 0d623a5

  containerName := strings.TrimPrefix(dockerEvent.Actor.Attributes["name"], "/")

  
  or any idea what should be done to have single service log file and rotation
  ?

There are import of winio module at

But i can't find any referneces to it. Is it usefull?

Currently, both are part of main and both do too much to be tested efficiently. We need to figure a good way to test them and split if needed.

Hello
I'm using docker image umputun/docker-logger:latest
"Created": "2019-06-18T17:22:53.675688126Z"
"Image": "sha256:ceb65856493ad141b17534346a21f357513852f852c8e5fb4524b12ae54e2f6d"

Trying to configure includes to have only specific containers, but option INCLUDE_PATTERN seems doesn't work
More over, the option EXCLUDE_PATTERN seems does not work also

Now I have BOTH options configured simultaneously in .yml file and the service doesn't reports any error or warning.
It looks like it completelly ignores them and collects logs for all running containers

Current configuration

        environment:
            - INCLUDE_PATTERN="ib.*"
            - EXCLUDE_PATTERN="docker-logger"

docker logs docker-logger

2020/11/11 12:48:39.251 [INFO] {app/main.go:63 main.main} options: {DockerHost:unix:///var/run/docker.sock EnableSyslog:false SyslogHost:127.0.0.1:514 SyslogPrefix:docker/ EnableFiles:true MaxFileSize:10 MaxFilesCount:5 MaxFilesAge:10 MixErr:true FilesLocation:logs Excludes:[] Includes:[] ExtJSON:false Dbg:true}
2020/11/11 12:48:39.251 [DEBUG] {discovery/events.go:40 discovery.NewEventNotif} create events notif, excludes: [], includes: []
2020/11/11 12:48:39.260 [DEBUG] {discovery/events.go:112 discovery.(*EventNotif).emitRunningContainers} total containers = 17
...
2020/11/11 12:48:39.261 [DEBUG] {app/main.go:116 main.runEventLoop.func1} streaming for 17 containers

Can not find any information about MHOST env. What is it's purpose? Is it necessary in a generic case?

docker restart [my container name] leads to all records are logged twice.

I suppose, two events start + restart are cause of this problem.

This is output of docker events --filter 'container=[my container name]':

2019-06-14T17:37:51.073152530+07:00 container kill 29b0325002d7d22a9b3ea886cb0e13439c67218c3b8259737b3ca16e1fb17a1b (com.docker.compose.config-hash=27db521962a7ab8da02b7af15bb23cff77bad720769ebaddac7e886d291756e5, com.docker.compose.container-number=1, com.docker.compose.oneoff=False, com.docker.compose.project=[my project name], com.docker.compose.service=[my service name], com.docker.compose.version=1.17.0, image=openjdk:8u151-jre-slim, name=[my container name], signal=15)
2019-06-14T17:37:51.928043548+07:00 container die 29b0325002d7d22a9b3ea886cb0e13439c67218c3b8259737b3ca16e1fb17a1b (com.docker.compose.config-hash=27db521962a7ab8da02b7af15bb23cff77bad720769ebaddac7e886d291756e5, com.docker.compose.container-number=1, com.docker.compose.oneoff=False, com.docker.compose.project=[my project name], com.docker.compose.service=[my service name], com.docker.compose.version=1.17.0, exitCode=143, image=openjdk:8u151-jre-slim, name=[my container name])
2019-06-14T17:37:52.151926158+07:00 container stop 29b0325002d7d22a9b3ea886cb0e13439c67218c3b8259737b3ca16e1fb17a1b (com.docker.compose.config-hash=27db521962a7ab8da02b7af15bb23cff77bad720769ebaddac7e886d291756e5, com.docker.compose.container-number=1, com.docker.compose.oneoff=False, com.docker.compose.project=[my project name], com.docker.compose.service=[my service name], com.docker.compose.version=1.17.0, image=openjdk:8u151-jre-slim, name=[my container name])
2019-06-14T17:37:52.634706721+07:00 container start 29b0325002d7d22a9b3ea886cb0e13439c67218c3b8259737b3ca16e1fb17a1b (com.docker.compose.config-hash=27db521962a7ab8da02b7af15bb23cff77bad720769ebaddac7e886d291756e5, com.docker.compose.container-number=1, com.docker.compose.oneoff=False, com.docker.compose.project=[my project name], com.docker.compose.service=[my service name], com.docker.compose.version=1.17.0, image=openjdk:8u151-jre-slim, name=[my container name])
2019-06-14T17:37:52.635993651+07:00 container restart 29b0325002d7d22a9b3ea886cb0e13439c67218c3b8259737b3ca16e1fb17a1b (com.docker.compose.config-hash=27db521962a7ab8da02b7af15bb23cff77bad720769ebaddac7e886d291756e5, com.docker.compose.container-number=1, com.docker.compose.oneoff=False, com.docker.compose.project=[my project name], com.docker.compose.service=[my service name], com.docker.compose.version=1.17.0, image=openjdk:8u151-jre-slim, name=[my container name])

This is log of docker-logger container:

2019/06/14 10:37:51.928 [INFO]  new event {ContainerID:29b0325002d7d22a9b3ea886cb0e13439c67218c3b8259737b3ca16e1fb17a1b ContainerName:[my container name] Group: TS:2019-07-02 12:06:19.928043548 +0000 UTC Status:false} 
2019/06/14 10:37:51.928 [WARN]  stream from 29b0325002d7d22a9b3ea886cb0e13439c67218c3b8259737b3ca16e1fb17a1b terminated with error context canceled                                                                    
2019/06/14 10:37:52.152 [INFO]  new event {ContainerID:29b0325002d7d22a9b3ea886cb0e13439c67218c3b8259737b3ca16e1fb17a1b ContainerName:[my container name] Group: TS:2019-07-02 12:06:20.151926158 +0000 UTC Status:false} 
2019/06/14 10:37:52.635 [INFO]  new event {ContainerID:29b0325002d7d22a9b3ea886cb0e13439c67218c3b8259737b3ca16e1fb17a1b ContainerName:[my container name] Group: TS:2019-07-02 12:06:20.634706721 +0000 UTC Status:true}  
2019/06/14 10:37:52.635 [INFO]  loggers created for logs/[my container name].log, max.size=10M, max.files=10, max.days=30                                                                                                 
2019/06/14 10:37:52.635 [INFO]  start log streamer for [my container name]                                                                                                                                                
2019/06/14 10:37:52.636 [INFO]  new event {ContainerID:29b0325002d7d22a9b3ea886cb0e13439c67218c3b8259737b3ca16e1fb17a1b ContainerName:[my container name] Group: TS:2019-07-02 12:06:20.635993651 +0000 UTC Status:true}  
2019/06/14 10:37:52.637 [INFO]  loggers created for logs/[my container name].log, max.size=10M, max.files=10, max.days=30                                                                                                 
2019/06/14 10:37:52.637 [INFO]  start log streamer for [my container name]                                                                                                                                                

What do you think about option reverse to EXCLUDE, INCLUDE for example.
INCLUDE options will get a list of containers' names, which logs will be collected.
This option will be useful if you host several docker-compose projects at the same hosts. In this case, you should exclude all other containers in every project for now.
I don't want to maintain a fork, but can make this feature, open PR and finish a review, so I can start after maintainer accept that option.

We deploy new releases in a new container with docker-compose --scale (out -> test -> in).

Then it creates a new container with a number in the name, like app_api_13 and next deploy scale out to app_api_14, and delete old app_api_13 at scale in ..... the log file follow the new containers name after each deploy, then we have lot of log files:

app_api_10.log  app_api_11.log  app_api_12.log  app_api_13.log  app_api_14.log  app_api_9.log  

when just last log app_api_14.log is the container running.

Is there are way to avoid number in log and maintain just one log, avoid number?

For some users it will be very useful to have a single log file for a container.

To reproduce the error, try the following test case:

• start docker-logger
• open log file for any container via vim
• edit log file
• save changes
  and then log file stops recording
  and i don't have any error in docker-logger.log

Add new parameter -j / --json (env JSON) to convert messages into JSON format

type jMsg struct {
	Msg       string    `json:"msg"`
	Container string    `json:"container"`
	Group     string    `json:"group"`
	TS        time.Time `json:"ts"`
	Host      string    `json:"host"`
}

By default, it will be off.

Currently, this project can't be compiled at windows cause no syslog module available at this operating system. https://github.com/golang/go/blob/master/src/log/syslog/syslog.go#L5

Now time the easiest way is comment syslog import and usage. As options, it is available to make syslog disabled for windows with compiler directives as I think.