Proxy pod appears running even if delegate pod isn't running yet about admiralty HOT 13 CLOSED

Multicluster-scheduler is now a virtual-kubelet provider. A proxy pod's now simply reflects its delegate's status, so it appears pending until its delegate runs. Also, there is no pod limit on the virtual node. The startegic merge patch workaround is no longer needed (unless we realize we need it for something else in the future).

from admiralty.

Hello, we're running into this issue because the proxy pods will fill up the worker nodes to the maximum number of pods per node (110), at which point the delegate pods become unschedulable on that node. Is there a possibility of implementing a fix, or hints on implementing a fix ourselves? Thanks!

from admiralty.

Alternatively, is there a way to modify the setup so that the proxy pods run on master nodes rather than worker nodes?

from admiralty.

Hi @danikam, the fix that I suggested above when I initially wrote up this issue would actually create a new problem. The delegate pod could be scheduled, run to completion, while its proxy pod would remain unschedulable, which would break the feedback loop. So, for now, I'd like to let the proxy pods start before the delegate pods.

Your particular issue is that proxy pods and delegate pods compete for a starved resource. As you mentioned, scheduling proxy pods to a subset of nodes would unlock your system. This would be done by the mutating admission webhook: https://github.com/admiraltyio/multicluster-scheduler/blob/master/pkg/webhooks/proxypod/proxypod.go. At first, to test the idea, you could hack a hardcoded nodeSelector on the proxy pod . We could then add a configuration value, e.g., as an annotation on the source pod (e.g., multicluster.admiralty.io/proxy-pod-node-selector).

from admiralty.

Hi @adrienjt, many thanks for your quick reply! I like the idea of hardcoding the NodeSelector as as quick test, in fact I've been looking for way to do that today. I'm wondering if you could please point me to where the NodeSelector would need to be added (sorry, I'm new to this .go language)? I'm also not quite sure how to implement edits to a .go file in my cluster for testing

from admiralty.

Go is a compiled language, so you can't just edit a file in a running container. You'd need to pull the repo to your machine and add this line at the end of the Handle function:

proxyPod.Spec.NodeSelector = map[string]string{"node-role.kubernetes.io/master":""}

Then, you'd build an image of the pod admission controller and push it to a registry that you and your cluster can push/pull to/from:

# (inspired from build/build.sh)
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o "_out/cmd/pod-admission-controller/manager" "admiralty.io/multicluster-scheduler/cmd/pod-admission-controller"
cp build/Dockerfile _out/
docker build -t "$YOUR_IMG_REGISTRY/multicluster-scheduler-pod-admission-controller:$SOME_TAG" --build-arg target="cmd/pod-admission-controller" _out

Then you'd replace the deployed image:

kubectl edit deployment pod-admission-controller -n multicluster-scheduler-agent
... 

from admiralty.

Thank you for these very clear instructions! Works perfectly :) Just working on adding a NoSchedule toleration for the master nodes.

from admiralty.

Got the toleration working finally :) I've attached the updated proxypod.go with your suggested 'nodeSelector hack'.
proxypod.txt

from admiralty.

I'm glad it's working, and thank you. If you'd like this merged, it would have to be configurable. Understandably, not all users want their proxy pods to be scheduled to master nodes. I'm thinking of a general solution that would allow users to apply a custom strategic merge patch to source pods when they become proxy pods. The patch could be stored as an annotation on each source pod (e.g., multicluster.admiralty.io/proxy-pod-strategic-merge-patch`), or in a ConfigMap that would be mounted inside the webhook container, to be applied to all source pods. The former is likely easier to implement at first. Let me know if you'd like to own this feature and submit a pull request.

Applied to the example of the README, here's what the patch would look like as an annotation in your case:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 10
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        multicluster.admiralty.io/elect: ""
        multicluster.admiralty.io/proxy-pod-strategic-merge-patch: |
          spec:
            nodeSelector:
              node-role.kubernetes.io/master:""
            tolerations:
            - key: node-role.kubernetes.io/master
              operator: Exists
              effect: NoSchedule
    spec:
      containers:
      - name: nginx
        image: nginx
        resources:
          requests:
            cpu: 100m
            memory: 32Mi
        ports:
        - containerPort: 80

from admiralty.

Sure, I can try to work on this strategic merge patch you suggest to make it a more generally usable feature (though I may take some time to get up to speed with the implementation details)

from admiralty.

Thank you! Let me know if you have any questions along the way. You should be able to call the StrategicMergePatch function of apimachinery, with a bit of marshaling/unmarshaling to/from JSON, directly from the Handle function in the webhook.

from admiralty.

Awesome, thank you for implementing this so quickly! And sorry to be so tardy with the workaround from my end :S Will try this out on the UVic clusters asap

from admiralty.

You're welcome @danikam and no worries at all, our discussions have helped a lot in designing this new feature. Thanks for trying it out, I look forward to your feedback.

from admiralty.