Log aggregation/archive tools about infrastructure HOT 4 CLOSED

If we're going with Prometheus/Grafana, I vote for Loki/Promtail as well. That way we can have..... wait for it..... a single pane of glass.

/me waits for @NeilHanlon to drink

But seriously, keep the stack small and light as possible. Unless there's a good technical reason to go with something else. That technical reason just needs to outweigh the very real costs (time, money, operational complexity) in running separate logging/monitoring solutions.

from infrastructure.

As it stand now, we are planning to use an elk stack.

from infrastructure.

I would recommend for log shipping that we use fluentd as it connects across any system you could think of with routing based on tags and everything else And can change if the tech stack updates at any point. To the point in the slack chat, the dev/ops team manages the input definition and then the monitoring/security team cares about the output configuration.

Security team using ELK ✅
Ops team with Loki/grafana ✅
Need to archive logs for audit purposes S3 connector ✅

from infrastructure.

After talking with @NeilHanlon, as it stands right now, we are planning to use an elk stack. At the current time, there is no reason to run multiple logging infrastructures. An ELK stack is not that hard to admin or provision that it would warrant multiple logging clusters. That may change in the future and please feel free to discuss it more in the infra channel, but at the moment that is the plan. Logging and auditing will be one of the very first items stood up, and having a solution the existing core infra and security teams are familiar with, and trust are a much higher priority at the moment.

From my discussion with @NeilHanlon we're going to close this issue, since it's not an active issue being worked on. But again, feel free to discuss more in the infra channel.

from infrastructure.