Override Vs Overlay and CRD rejection issue about apps HOT 9 CLOSED

But what if I want to add another dashboard? The way I understand it I would need to override the kustomization here.

Not necessarily. There's a more convenient way. You can use overlays to further modify/add additional resources. Please take a look on how we approached this in the odh-dashboard deployment here:

We're adding "new" overlays used by the ODH operator. These overlays can introduce new resources as well as modify the base. In the case of odh-dasboard mentioned above, we're using both, we're adding 2 new overlays, both of which are then used in the kfdef:

• custom-image which overlays the base with a custom image -modifies the base
• cluster-role which adds a resource

But still, the issue is not solved by this.

from apps.

I think the trend here is to stick with overrides.

If we're in an agreement on this, let's stick with overrides. I'll update the docs here to take a hard stance that any extension to odh should be done via overrides.

We also need issues to move grafana dashaboards and kafkatopics into overrides and update docs for those accordingly. If there's anything else that should be restructured, please note them here. Link to this discussion as well in the PR.

I'll leave this up for a day or two in-case others want to chime in as well.

from apps.

I'm not quite seeing the issue with the override -- isn't the whole point of the ODH operator that you can use it to deploy your own manifests? So we'd change https://github.com/operate-first/apps/blob/master/odh/base/monitoring/kfdef.yaml#L48 to point at our own repo. We could have a fork of the upstream manifests that we sync on a regular basis (perhaps with every upstream release?) and keep adding our own changes onto it.

from apps.

Honestly, I'm not sure about this. It's feels like we're trying to engineer a workaround for a workaround for bug in kubernetes.

I also don't see much of a problem with using overrides besides the observability. Which is a bug on kubeflow/kubernetes side.

All the resources we're talking about here are connected to the ODH deployment. And ODH is meant to be deployed by the ODH operator. But again, if overrides are the problem we can maintain a odh-manifests fork as Anish said. But I would rather see us keeping things in slim overrides and pushing them to upstream ODH as soon as we can.

from apps.

It is my understanding that the point of override is so that we don't have to fork the whole repo.

One of my major concerns are actually resolved with what @tumido has suggested above.

I still think it's a bit unfortunate that we can't see some these resources in ArgoCD. And I expect we're going to encounter issues with odh taking a while to reconcile changes. But that's also something we should look to explore and report upstream.

I'll +1 for continuing to enforce everything via overrides then.

from apps.

Also note that this is only a problem for first deploys, once the CRDs are available, then using overlays doesn't result in any errors/issues.

One solution I can think of is using waves, but with autosync turned on. We set KFDefs to always be deployed first, then everything else including CRs second, or later. Then when one tries to deploy components from this repo for the first time, they will see errors in the beginning, while ArgoCD (and thus ODH) deploys kfdef first. Since auto-sync is enabled, it will eventually resolve itself once ODH properly reconciles the CRDs.

The problem is of course, with this situation you'd see some errors in the beginning while you wait for ODH to reconcile.

from apps.

Ah that's good to know, thanks @tumido .

from apps.

I'm fine with that too. Both overrides and fork seem reasonable to me, and fwiw I don't think a fork would be that bad to keep running.

from apps.

I'll speculate here a bit. I have no intention to make this issue philosophical though, so take this comment with a grain of salt 😄

I think the idea why to use overrides is also a bit deeper than that. It introduces a bit of discomfort which may motivate us to keep the changeset minimal and push us to stay closer to upstream. This can maybe make us contribute the overridden content back to upstream to lower the maintenance costs.

This may come more obvious when compared to the "forked odh-manifests case". It's really easy to diverge the repo trees while it's very comfortable to contribute downstream only. And we have no motivation to push our changes upstream. We'd just "merge/rebase" each ODH release and that's it.

from apps.