[Feature] Delay reporting errors after startup about nomarch HOT 5 CLOSED

This should be resolved by PR #5 which has been merged.

from nomarch.

Good call. We've run into the same problem at work :)

I like being able to skip the first x seconds after startup rather than tying it to the max_seconds_to_reach_end. It might take multiple minutes for an event to make it all the way through the pipeline during particularly bursty moments and max_seconds_to_reach_end has to take that into account, but just skipping the first 10 seconds after deployment would remove that flood of false negatives under common load.

One question I have is whether you'd agree that the best thing to do would be to just not log the outcome of the events reported during that period. This seems to make the most sense to me. It would keep logs and the logic around alarms simple. But an alternative I could see someone wanting is to log it with a specific message that mentions whether we think it made it or not, but we don't know because it was within that startup period.

from nomarch.

One question I have is whether you'd agree that the best thing to do would be to just not log the outcome of the events reported during that period. This seems to make the most sense to me.

Personally that is how I would like it done, so there is just a simple filter needed to find the logs, and not much thought has to go into is it fully operational yet/or am i waiting for delay still?

an alternative I could see someone wanting is to log it with a specific message that mentions whether we think it made it or not, but we don't know because it was within that startup period.

If you were to implement this would it be a third event state? Complete, Did not complete, and Unknown
Because if that is the case then i think you could implement both so you have insight of events during startup, without having to alarm

from nomarch.

If you were to implement [logging unknown events] would it be a third event state?

Probably. I think there would be two different ways to implement it without adding a bunch of stuff we don't want in the main loop. One would just be to have another branch in the statement we already have during expiry that checks if we are in that time window, and if we are then print something different that signifies it's unknown. Another option would be to move the logging into a more structured format (probably json) and have a flag that signals whether the service is warmed up or not.

I think either of those implementation paths are reasonable...but at the end of the day I think it keeps things easier to reason about if we just drop the events for that duration. The default config option would be zero seconds so that people new to the project don't think their thing is broken when it doesn't log anything for a bit.

from nomarch.

I think either of those implementation paths are reasonable...but at the end of the day I think it keeps things easier to reason about if we just drop the events for that duration. The default config option would be zero seconds so that people new to the project don't think their thing is broken when it doesn't log anything for a bit.

Totally reasonable. I think dropping them outright and maybe another proposal later with more states is a clear path

from nomarch.