Garbage collect old Argo workflows / Argo UI unresponsive about testing HOT 23 CLOSED

Argo v2.2.0 now supports setting a TTLAfterSeconds flag.
https://github.com/argoproj/argo/blob/7864ad36788dc78d035d59ddb27ecd979f7216f4/pkg/apis/workflow/v1alpha1/types.go

We need to do a couple things

1. Upgrade Argo
2. Start setting that on our workflows.
3. Consider adding an admission controller to ensure its set.

from testing.

Navigating to a specific workflow doesn't seem to work either
http://testing-argo.kubeflow.org/timeline/kubeflow-test-infra/kubeflow-presubmit-tf-serving-image-387-12405f1-511-aec5;tab=workflow

from testing.

I tried manually deleting the older workflows as follows

Dump all workflows sorted by age to a file

kubectl get wf -a --sort-by='.metadata.creationTimestamp'  > /tmp/wf.list

Manually delete the lines corresponding to workflows I don't want to delete.

Then do

cat /tmp/wf.list | awk '{print $1}' | xargs kubectl delete wf 

from testing.

Deleting the workflows appears to have worked so it appears to be an issue with the UI not performing well when there are lots of workflows.

/cc @jessesuen @alexmt

from testing.

Argo does not have a GC policy/configuration. It should though.

from testing.

UI is having problems again.

Currently 697 workflows

wf.list.txt

Delete workflows more than a day old

from testing.

@jlewi, unfortunately, Argo GC is still not implemented. UI is not performing well since it does not have pagination (because kubernetes api does not have pagination yet)

As a workaround, you might try argo v2.1.0-alpha1 +. UI had been migrated to React and can render much more workflows.

from testing.

@alexmt Great thank you so much.

from testing.

1330 workflows

Deleted workflows older than 3 days to make UI more responsive.

from testing.

deleted 1391 workflows older than 3d

from testing.

Deleted 307 workflows; older than 3d

from testing.

1726 workflows
Deleting 1601 workflows older >=3d

from testing.

Running get wf was having problems on my local machine but when I ran it from within cluster it succeeded. I think its probably because there's so much K8s master traffic.

from testing.

The number of workflows is currently so large that I'm having trouble getting them and deleting them.

Using labels to subselect some of the workflows seems to help.
Unfortunately we don't necessarily have good labels on most of our workflows.

from testing.

I deleted the namespace because deleting individual workflows wasn't working.

kubectl delete namespace kubeflow-test-infra

from testing.

Waiting for the namespace to be deleted. Hopefully that succeeds.

from testing.

Namespace was successfully deleted.

Now redeploy test infra

kubectl create namespace kubeflow-test-infra
ks apply kubeflow-ci -c argo

ksonnet app is here:
https://github.com/kubeflow/testing/tree/master/test-infra

from testing.

Argo UI is backup

from testing.

I had to delete and recreate the nfs component

ks delete kubeflow-ci nfs-external
ks apply kubeflow-ci nfs-external

from testing.

Will be fixed by #300 which includes a cron job to GC Argo workflows.

from testing.

See #324 seems like there's a problem with our cron job and we aren't GC'ing the resources.

from testing.

#267 Issue to monitor cron jobs.

from testing.

Haven't had problems in a while so closing this issue.

from testing.