flanksource / canary-checker Goto Github PK

On hover / click of the check show the description/log entry

Run OPA policy against all objects defined in a namespace
Input:

namespace: *
objects:
- deployments
- pods
- service
opa:
- fromBundle: http://path/to/opa/bundle.zip
- fromConfigMap: policies
- fromValue: {...}

Metrics:
Passing / Failing objects

Change all the references from moshloop to flanksource

• Report current pod phase if not running + conditions/events

A canary that can be used to check that backups are being made and that they are non-empty

Input:

bucket:
accessKey:
secretKey:
endpoint:
objectPath: # glob path to restricy matches to a subet
readWrite: false
maxAge: # maximum allowed age of matched objects
minSize: # min size of of most recent matched object

Output: Pass/Failnvalid

Metrics:

• Total Size
• Object Count
• Last write time

Each check should allow specifying a tag e.g. tag: critical which then gets pushed through to prometheus.

Input

host: ldaps://hostname:636
username: admin
password: admin
bindDn: 
userSearch: 

Output: Pass/Fail/Invalid

Metrics:

• Lookup time
• Record count (min,max,avg)

Similar to pod check, but run first create a new namespace, and then delete it afterwards.

Input:

# If DNS, test connectivity for each returned record
endpoints: []
# max response time
thresholdMillis: 300
# max percentage packet loss
packetLossThreshold: 0.01

Metrics (per endpoint)

• latency
• packetLoss

Also noticed some unused things here
https://github.com/flanksource/canary-checker/blob/master/checks/namespace.go#L58

checks/namespace.go:58
deadline := time.Now().Add(time.Duration(config.Interval) * time.Second)
this deadline is calculated, but is not used in the Check()
same for pod.go
https://github.com/flanksource/canary-checker/blob/master/checks/pod.go#L75
checks/pod.go:75
deadline := time.Now().Add(time.Duration(config.Interval) * time.Second)

Moshe Immerman
Yeh - it needs to be normalized
the deadline is used here: https://github.com/flanksource/canary-checker/blob/master/checks/namespace.go#L170
checks/namespace.go:170
deadline := time.Now().Add(time.Duration(check.Deadline) * time.Millisecond)

although that should probably be a httpTimeout
deadlines should probably be based on the interval, and implemented at a higher level
including context cancellation

i.e. if an aggregation endpoint becomes unreachable the UI currently does not display anything

• Specified endpoints with latency tolerancy

Choose where to run the checks from - can be a single node or multiple
If multiple - all checks must pass, and a prometheus metric is exported per node

Connect and then close a connection to a port

Also need to look at the tooltips - Currently they overlap, and they should probably update a single window rather than create and fade

Reuse pod names (maybe in a rolling fashion) to prevent too many metrics series from being created which can bloat prometheus

i.e. when hovering over an individual check, display a Prometheus graph of latency/failure/success in the last 1h|3h|6h|12h|1d|3d|1w

Evaluate:

• https://tech.shutterstock.com/rickshaw/
• https://omnipotent.net/jquery.sparkline/#s-docs
• http://gionkunz.github.io/chartist-js/index.html
• other ?

Test that a persistent volume of specified class & size can be:

• Created
• Folders created
• Files created, updated, read and deleted
• Folders Deleted
• Deleted and cleaned up

Input:

# If DNS, test connectivity for each returned record
endpoints: 
   - https://google.com
   - http://google.com:443
# max response time
thresholdMillis: 300
responseCodes: [201,200,301]
responseContent: "OK"
maxSSLExpiry: 60d

Metrics (per endpoint)

• response_time
• response_code
• ssl_certificate_expiry

Input:

• DNS Server host and port
• DNS Query
• DNS Query Type (default to A)
• Min number of records to return
• Exact reply match
• Timeout

Output: Pass/Fail/Invalid

Metrics:

• Lookup time
• Record count (min,max,avg)

Input:

bucket:
accessKey:
secretKey:
endpoint:
objectPath:
readWrite: false

Output: Pass/Failnvalid

Metrics:

• Lookup time
• List / Update / Read

apiVersion: canary.flanksource.com/v1
kind: CanaryCheck
spec:
    interval: 60s
    secretRefs:
        - secrets
    config: #json.rawMessage

The controller should update the status for each CRD after each check and fire events for failures, to enable kubectl get canary

We can either deploy a new canary-checker instance per CRD and aggregate results or run all the canaries from the controller,

Currently, the UI just displays a Red or Green square with no visual indicator of age or duration.

Possible designs:

Now  ---------------      5m Ago
✓.    ✗ ✓     ✗              ✗

5h Ago ---------------    2d Ago
    ✓.    ✗ ✓     ✗               ✓   

Could use red/green circles and then the size of the circle to indicate the duration

i.e.

999/1000 checks = 99.9%

Test that a docker image can be pulled

Input:

image: docker.io/busybox
username: 
password:
expectedDigest:  abcdef123
expectedSize: 200m

Metrics (per endpoint)

• pull_time
• layers
• size

Input

connection: user=pqgotest dbname=pqgotest sslmode=verify-full
query:  SELECT 1
results: 1

Output: Pass/Fail/Invalid

Metrics:

• response_time
• Record count (min,max,avg)

/metrics

Export a snapshot of metrics created by all configured checks

If the request takes longer than the threshold, then the check should fail

Given the pod spec, create a Pod, service and ingress, schedule it and then confirm that it serves results correctly via the ingress

# Full pod spec
spec:
   ....
thresholdMillis: 300
expectedContent: OK

Metrics:

• schedule_time (from Pending to ContainerCreating)
• creation_time (from ContainerCreatning to Running)
• destroy_time (from Terminating to Terminated)
• ingress_time (from Running to Ingress check succeeds)
• serve_time (Response time via ingress)

Create a self-reloading status page that displays a summary from all configured checks.