Draft code which can get a long xml but without the log sent by e2e tests. Need to investigate how to search the results. Opened the issue in splunk repo: splunk/docker-splunk#625

func validateTestLogsInSplunk(url string, token string, testLog string) {
	// Use REST API to retrieve the logs. https://docs.splunk.com/Documentation/Splunk/9.1.1/RESTREF/RESTsearch#search.2Fjobs
	tr := &http.Transport{
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}
	client := &http.Client{Transport: tr}
	searchQuery := "search *" // Get the latest of all the log events.
	req, err := http.NewRequest("POST", fmt.Sprintf("%s/services/search/jobs", url), bytes.NewBufferString(searchQuery))
	gomega.Expect(err).ShouldNot(gomega.HaveOccurred())
	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
	req.Header.Set("Authorization", fmt.Sprintf("Splunk %s", token)) // https://docs.splunk.com/Documentation/Splunk/9.1.1/Security/UseAuthTokens
	resp, err := client.Do(req)
	gomega.Expect(err).ShouldNot(gomega.HaveOccurred())
	defer resp.Body.Close()
	body, err := io.ReadAll(resp.Body)
	gomega.Expect(err).ShouldNot(gomega.HaveOccurred())
	fmt.Println(string(body))
}

See https://github.com/moby/moby/releases/tag/v20.10.25

During my test against awslogs shim logger, the last matched line cannot be logged when multiline pattern is set.

For example, when the multiline pattern is ^INFO, and the log printed by container is

INFO a
INFO b
INFO c

Only INFO a and INFO b can be passed to awslogs.

This may be caused by test set up or the bug of the feature itself.

Need more investigation. Some potential directions to investigate:

• Try to reproduce with Nerdctl by specifying the shim logger binary path.
• The shim logger depends on the log drivers in Moby. So we can try to reproduce by Docker with its native awslogs driver.
• The logic here seems processing event first with buffer and then append to the buffer. So it is possible that the last line's content is attached to the buffer finally but does not have chance to be processed.
• The current awslogs test infra uses AWS local stack , so may have feature gaps.

Unit test is failing, log message here:

2023-07-12T21:57:19.9885430Z     --- FAIL: TestGetDockerConfigs/NoError (0.01s)
2023-07-12T21:57:19.9895872Z         args_test.go:278: assertion failed: env0=envValue0 (testContainerEnvSlice[i] string) != env1=envValue1 (args.ContainerEnv[i] string)
2023-07-12T21:57:19.9959246Z FAIL

This is happening because the implementation of ContainerEnv is converting a map to a slice which does not guarantee ordering: https://github.com/aws/amazon-ecs-shim-loggers-for-containerd/blob/e1ce0e61f9b35a41eb7112b9b5862fb8c79d600b/args.go#L97

While for the unit test, it is ordered: testContainerEnv = "{\"env0\":\"envValue0\",\"env1\":\"envValue1\"}"

https://github.com/aws/amazon-ecs-shim-loggers-for-containerd/blob/e1ce0e61f9b35a41eb7112b9b5862fb8c79d600b/args_test.go#L40

Since the implementation does not care about the order, we should rewrite the test to either change the loop to make a map instead and check the contents match.

Supporting Long Log Lines in Fargate PV 1.4

Background

Related Issues

• fluent/fluent-bit#4309
• aws/containers-roadmap#1550

How Container Logs are processed with FireLens

When the fluentd log driver is used (ex: Amazon ECS FireLens), the following is the data-flow:

App container stdout/stderr => container runtime (containerd) => fluentd log driver (shim logger package) => unix socket 
  => Fluent Bit Forward input

The key take-away here is that logs do not go straight to Fluent Bit, they pass through the container runtime and log driver first.

Multiline Log Use Case: Long Log Lines

The other major multiline logging use case is long log lines. Monitoring is critical to modern containerized applications, and we have seen many customers that produce very large and verbose log events. Structured logging, where logs are generated by the app with a set schema/format (usually JSON) is also very common. Many customers, will log huge amounts of information in a single JSON log event- 1 MB log events are not unheard of.

These large logs are emitted by the application in a single line of code/single print statement. However, the container runtime must process them, and most container runtimes, including Docker and Containerd, will split log lines when they are greater than 16KB. In the case of log files, each 16KB chunk of a split log is written to a new log line. In the case of the Fluentd Docker Log driver, each 16KB chunk of data is a separate event.

For the Fluentd Docker Log Driver, a key will be set to note that the message is partial. Below are real split docker logs. A large log line was sent to both stdout and stderr, which are separate pipes, so each is split and creates a series of messages. Notice that stderr and stdout have different values for partial_id.

This is because stderr and stdout

{"source": "stdout", "log": "...", "partial_message": "true", "partial_id": "dc37eb08b4242c41757d4cd995d983d1cdda4589193755a22fcf47a638317da0", "partial_ordinal": "1", "partial_last": "false", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig"}]
{"container_name": "/hopeful_taussig", "source": "stderr", "log": "...", "partial_message": "true", "partial_id": "ecccce95711776e6a06d631af8e9227686446814eba7a87cb59b36bbaaad8b58", "partial_ordinal": "1", "partial_last": "false", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2"}]
{"partial_last": "false", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig", "source": "stdout", "log": "...", "partial_message": "true", "partial_id": "dc37eb08b4242c41757d4cd995d983d1cdda4589193755a22fcf47a638317da0", "partial_ordinal": "2"}]
{"log": "...", "partial_message": "true", "partial_id": "dc37eb08b4242c41757d4cd995d983d1cdda4589193755a22fcf47a638317da0", "partial_ordinal": "3", "partial_last": "false", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig", "source": "stdout"}]
{"source": "stderr", "log": "...", "partial_message": "true", "partial_id": "ecccce95711776e6a06d631af8e9227686446814eba7a87cb59b36bbaaad8b58", "partial_ordinal": "2", "partial_last": "false", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig"}]
{"partial_id": "ecccce95711776e6a06d631af8e9227686446814eba7a87cb59b36bbaaad8b58", "partial_ordinal": "3", "partial_last": "false", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig", "source": "stderr", "log": "...", "partial_message": "true"}]
{"container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig", "source": "stdout", "log": "...", "partial_message": "true", "partial_id": "dc37eb08b4242c41757d4cd995d983d1cdda4589193755a22fcf47a638317da0", "partial_ordinal": "4", "partial_last": "false"}]
{"partial_last": "false", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig", "source": "stderr", "log": "...", "partial_message": "true", "partial_id": "ecccce95711776e6a06d631af8e9227686446814eba7a87cb59b36bbaaad8b58", "partial_ordinal": "4"}]
{"container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig", "source": "stdout", "log": "...", "partial_message": "true", "partial_id": "dc37eb08b4242c41757d4cd995d983d1cdda4589193755a22fcf47a638317da0", "partial_ordinal": "5", "partial_last": "true"}]
{"log": "...", "partial_message": "true", "partial_id": "ecccce95711776e6a06d631af8e9227686446814eba7a87cb59b36bbaaad8b58", "partial_ordinal": "5", "partial_last": "true", "container_id": "a96998303938eab6087a7f8487ca40350f2c252559bc6047569a0b11b936f0f2", "container_name": "/hopeful_taussig", "source": "stderr"}]

The following partial metadata fields should be present in split container logs:

• partial_message: boolean to signal if the message is partial or not.
• partial_id: all parts of a split message get the same unique ID
• partial_ordinal: counts up from 1, ordering the parts of the split message.
• partial_last: is this the last split message or not.

AWS for Fluent Bit issue: aws/aws-for-fluent-bit#25

Problem Statement

In Fargate PV 1.4, which uses this containerd shim logger package, long log messages are currenly split, but the partial metadata keys are not added to the outputted logs.

These partial metadata keys are standard and customers expect them to be added. PV 1.3 correctly adds these partial metadata keys.

Implementation

Currently, the shim logger package does split long log lines and does track whether messages are partial or not: https://github.com/aws/amazon-ecs-shim-loggers-for-containerd/blob/master/logger/common.go#L234

However, the shim logger code does not properly set the partial metadata attributes in the message structure: https://github.com/moby/moby/blob/master/daemon/logger/copier.go#L114

Implementation will set the partial metadata fields and generate partial IDs.

Testing Plan

Unit Test Cases

TODO

Manual Test Cases

Created a test logger container which outputs 10 long log lines of a configurable length, 5 to stdout and 5 to stderr.

The logs can be sent to a local Fluent Bit and outputted to a file after Fluent Bit re-joins the split messages.

Created a validation script to read the outputted file and validate that all log lines were properly re-joined.

TODO: add scripts and results

During #41, I found the Windows platform test and linting was not being ran. I was able to get test work, but linting was more effort and I did not want to block addressing other CI issues.

This task would be to enable linting on the Windows platform and address any issues found.

io/ioutil has been marked deprecated starting Go1.16.

Recommend moving to bulitin os package.

The github.com/pkg/errors package has been archived.

Recommend moving to standard library errors.

Mainly my goal is to update the build to use a more modern version of the compiler. It looks like Go is offering support for past two versions available. Although I think an argument could be made for building against Go 1.16 and 1.17 as well for folks who haven't upgraded their toolchain yet.