chaos-genius / chaos_genius Goto Github PK

This includes:

• Anomaly UI
• Anomaly drill downs
• Anomaly computation - non-distributed
• Anomaly severity scoring - baseline
• DB storage of anomalies
• API to serve anomalies from DB
• Dataset driven anomaly testing

Some things to understand include:

• #71
• #72
• Configurable execution modes that be picked via config at every level of execution pipeline
• Task expiry

Benchmarking test:

1. Load 10, 1000, 100000 randomly generated series of length 100 and 500
2. Fit prediction models on them. There will be multiple prediction libraries including prophet, neural-prophet, statsmodels.
3. Return a predicted value, store in Redis key

Metrics & measures to evaluate:

1. Performance in terms of series fitted & predicted per min for single node setup
2. Performance in terms of series fitted & predicted per min for multiple (2, 5) node setup
3. Effect of performance & memory footprint when passing multiple ML library context
4. Number of failed/ expired jobs in each run
5. Median, 90th, 99th percentile task execution time

Currently, when we have numerical/binned sub-dimensions, the generated query string uses the binned column name instead of the sub-dimension name.

Query strings for numerical columns currently look like this:

12.0 < Quantity_binned <= 2880.0

They should look like this:

12.0 < Quantity <= 2880.0

Need to complete the following:

• Multidimensional RCA
• Single dimensional RCA
• Overlap focussed table for multidimensional RCA
• Overall group metrics

JS mockups to be built by @keshprad and @varunp2k

Full UI to be built by @sketchbrahma team.

Pick info from DB & store results in DB for each KPI processed

Internal server error and route error should be handled gracefully.

Close the db connection during the request teardown.

Unable to use a count aggregation for the get_waterfall_and_impact_table functions

out = get_waterfall_and_impact_table(
    ecom_df.query('20110820 <= date < 20110827'),
    ecom_df.query('20110827 <= date < 20110903'),
    ['Country', 'PeriodOfDay', 'DayOfWeek'],
    'num_purchases',
    agg = 'count',
    n = [1, 2, 3],
    K=5,
    plot_in_mpl=True
)

Two exceptions are raised:
KeyError:

ValueError:

^ I think the problem is that count is used as one of the column names in compare_subgroups

@varunp2k

Look at the point near Feb 28th. The point is inside CI, but marked as an anomaly. This is only apparent at lower Confidence Levels (like 0.8 or 0.7).

Until we aren't doing OSS we can integrate sentry for the backend application.

The current implementation of Anomaly Detection works well for a demo, but needs to address these issues for scalability.

• Training on the last 89 data points and predicting an anomaly for the next immediate data point.
• Filtering various sub-dimensional groups.
• For drill-downs in anomaly, we need to add sorting for various sub-dimensional series.
• Testing with different frequencies of data (like an hourly series)
• Testing and implementing at least another model (apart from Prophet)

For the get_waterfall_and_impact_table, I've set K=3 max sub-groups; however, there are 4 subgroups displayed + an others sub-group if needed.

In the single_dim function, this doesn't happen. Setting K=3 max subgroups, gives the intended 3 sub-groups + an others sub-group if needed. Here's the exact same situation as above, except with a single dimension as 'main_cat'

Test it out

Distribute the anomaly detection

• Sensitivity UI for users setting up anomaly (high, medium, low)
• Drills-down
• Data quality

Categorical columns using count aggregation are allowed KPI.
RCA get_panel_metrics doesn't check whether the column is categorical and attempts min, max, mean, etc. It should only perform count and null_count.

• Shopify
• Stripe
• Quickbooks
• Zendesk

More details to be defined by alerting architecture

This is the more basic version that has a CSV/ JSON as an output

While calculating the best subgroup combination, there is a zero division error because len(indices) == 0 on line 440

Set indices is defined here, on line 420. I printed both indices & curr_filter_string. These are the values just before the error:

indices = set()
curr_filter_string = "1027.304 < `pressure` <= 1030.316 and 8.323 < `visibility` <= 9.142 and 0.814 < `humidity` <= 0.869"

The query with curr_filter_string returns 0 rows. A case is needed to handle this.

Here's the full error message: