Brief errors are not triggering notifications. about tattle HOT 8 CLOSED

There are ways to do this with graphite functions before you get to Tattle.

Sent from my iPhone. Thanks Steve.

On Dec 15, 2012, at 9:38 PM, John Gietzen [email protected] wrote:

We sometimes have errors that last only for a few seconds (CPU usage above the error threshold, for example). We would like to be notified of these issues, even if they aren't still in error when processor.php is being called.

Rather than checking the last value, we would prefer if Graphite-Tattle checked all values since the last time processor.php was run.

This is related to #33.

—
Reply to this email directly or view it on GitHub.

from tattle.

Our servers only ship their CPU usage to graphite once an hour. How exactly would we have Tattle warn us about something that happened 30 minutes ago?

As far as I can tell, this is impossible at this time, even with the ability to run graphite functions.

Plus, we would prefer to not have to resort to graphite gymnastics to work around an issue with Tattle. (We were hoping to allow non-tech-savvy users to create alerts.)

from tattle.

Ok so you have as set of issues.

1. You're sampling every 10 seconds, but there's an assumption built into Tattle that samples are every minute, so things like moving average look at 1/6th the samples you expect. That's really a labeling issue, Tattle asks for the sample size in minutes, then passes that to the moving average as the sample window. Which is reasonable if tattle scans 1/minute.
2. You'd like maximum and minimum built into Tattle. Perhaps 5% and 95% make sense as well.
3. You're only shipping stats to Graphite once an hour, so you want max/min over 60 minutes.

Not the primary Tattle maintainer but #2 seems reasonable to do. #1 would mean splitting the current field into time window and sample size.

I might do #2 for my own purposes, #1 I'll leave on the table for others as I don't see a driving need.

Sent from my iPad

On Dec 16, 2012, at 9:38 AM, John Gietzen [email protected] wrote:

Our servers only ship their CPU usage to graphite once an hour. How exactly would we have Tattle warn us about something that happened 30 minutes ago?

As far as I can tell, this is impossible at this time.

Plus, we would prefer to not have to resort to graphite gymnastics to work around an issue with Tattle. (We were hoping to allow non-tech-savvy users to create alerts.)

—
Reply to this email directly or view it on GitHub.

from tattle.

1. The default for graphite is 10 second buckets. The label should certainly be changed to say "samples".
2. We don't need maximum or minimum. We need tattle to run once per minute, but report on everything that happened over the last minute.
3. Again, we don't want max/min. If the error threshold is broken twice over the last hour, we expect two notifications.

I would be changing the code myself, but my employer has a lot of red tape before I'm allowed to work on an open source project.

Again, this is related to #33, since both of these issues are related to Tattle only looking at the last sample. Ideally, tattle would look at all samples since it last ran.

This pseudo code would fix both issues (I'm not a PHP programmer, so I apologize for it being shitty):

$lastNotNullTimeStampForThisMetric = /* get it from the database? */;
$data = $obj->getDataSince($lastTimeProcessorRan);
foreach ($datapoint in $data) {
    if ($datapoint.value == null) continue;

    $lastNotNullTimeStampForThisMetric = $datapoint.timestamp;

    /* here you would process each datapoint as per normal */
}

/* save $lastNotNullTimeStampForThisMetric to the database? */

from tattle.

I forgot to mention that Ignoring nulls is the correct thing to do, since that means "no data" in Graphite.

If someone wanted to change the behavior, there are certainly graphite functions that would allow them to be explicit about what they wanted: keepLastValue or transformNull(foo,0) for example.

from tattle.

Yeah, turns out max/min didn't do what you want anyways, what I was thinking was a "movingMaximum" which doesn't exist.

from tattle.

I'm not sure that ignoring nulls is the right decision since stoping graphite feeding can be an issue that we want to be warned of

from tattle.

I think some users may want for some alerts the current behavior, some other users may want for some other alerts a keepLastValue() behavior (which is more or less @otac0n's pseudocode in #35), and others even transformNull() behavior (as @otac0n states in #33).

The keepLastValue and transformNull value can be implemented by hand on an alert per alert basis, and morover this way one can add a parameter (both keepLastValue and transformNull need one).

Therefore, if nobody else reopens the point and is able to provide a consistent way to enhance this for everyone, I close related issues.

Sorry not to have a better idea.

from tattle.