Comments (10)
Hi @metheu
You can use the kubectl edit nodeclaim mynodeclaim --subresource='status'
But here is the ChatGPT-generated script to fix all of them:
from kubernetes import client, config
from kubernetes.client.rest import ApiException
import copy
import sys
def patch_nodeclaim_conditions():
# Load the kube config from the default location
config.load_kube_config()
# Create an API client
api_instance = client.CustomObjectsApi()
group = "karpenter.sh"
version = "v1"
plural = "nodeclaims"
try:
# List all NodeClaim resources at the cluster level
nodeclaims = api_instance.list_cluster_custom_object(
group=group,
version=version,
plural=plural
)
for nodeclaim in nodeclaims['items']:
name = nodeclaim['metadata']['name']
conditions = nodeclaim.get('status', {}).get('conditions', [])
# Create a deep copy of the conditions to modify
new_conditions = copy.deepcopy(conditions)
modified = False
for condition in new_conditions:
if 'reason' not in condition:
condition['reason'] = condition['type']
modified = True
# Patch the resource if any modifications were made
if modified:
patch_body = {
"status": {
"conditions": new_conditions
}
}
# Patching the status subresource
print(patch_body)
api_instance.patch_cluster_custom_object_status(
group=group,
version=version,
plural=plural,
name=name,
body=patch_body
)
print(f"Patched NodeClaim {name}")
print(f"=====================")
# sys.exit()
else:
print(f"No patch needed for NodeClaim {name}")
print(f"=====================")
except ApiException as e:
print(f"Exception when calling CustomObjectsApi->list_cluster_custom_object: {e}")
if __name__ == "__main__":
patch_nodeclaim_conditions()
and another one to add the missing message field:
from kubernetes import client, config
from kubernetes.client.rest import ApiException
import copy
import sys
def patch_nodeclaim_conditions():
# Load the kube config from the default location
config.load_kube_config()
# Create an API client
api_instance = client.CustomObjectsApi()
group = "karpenter.sh"
version = "v1"
plural = "nodeclaims"
try:
# List all NodeClaim resources at the cluster level
nodeclaims = api_instance.list_cluster_custom_object(
group=group,
version=version,
plural=plural
)
for nodeclaim in nodeclaims['items']:
name = nodeclaim['metadata']['name']
conditions = nodeclaim.get('status', {}).get('conditions', [])
# Create a deep copy of the conditions to modify
new_conditions = copy.deepcopy(conditions)
modified = False
for condition in new_conditions:
if 'message' not in condition:
condition['message'] = ""
modified = True
# Patch the resource if any modifications were made
if modified:
patch_body = {
"status": {
"conditions": new_conditions
}
}
# Patching the status subresource
print(patch_body)
api_instance.patch_cluster_custom_object_status(
group=group,
version=version,
plural=plural,
name=name,
body=patch_body
)
print(f"Patched NodeClaim {name}")
print(f"=====================")
#sys.exit()
else:
print(f"No patch needed for NodeClaim {name}")
print(f"=====================")
except ApiException as e:
print(f"Exception when calling CustomObjectsApi->list_cluster_custom_object: {e}")
if __name__ == "__main__":
patch_nodeclaim_conditions()
from karpenter-provider-aws.
Looks quite similar to #6326. I believe this is the same issue and gets resolved by upgrading the CRDs to the latest version as described here: https://karpenter.sh/docs/upgrading/upgrade-guide/#crd-upgrades
from karpenter-provider-aws.
Yes, it looks similar, indeed. But I performed the CRDs upgrade to the latest version right before upgrading the karpenter.. So it's happening with the latest CRDs.
So it looks like now I need to edit all existing nodeClaims and add the missing field to make karpenter happy.
from karpenter-provider-aws.
I tried upgrading CRDs once again after the karpenter upgrade and now got this error while upgrading CRDs:
$ kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.37.0/pkg/apis/crds/karpenter.sh_nodepools.yaml
The CustomResourceDefinition "nodepools.karpenter.sh" is invalid: status.storedVersions[1]: Invalid value: "v1": must appear in spec.versions
$ kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.37.0/pkg/apis/crds/karpenter.sh_nodeclaims.yaml
The CustomResourceDefinition "nodeclaims.karpenter.sh" is invalid: status.storedVersions[1]: Invalid value: "v1": must appear in spec.versions
$ kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.37.0/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml
customresourcedefinition.apiextensions.k8s.io/ec2nodeclasses.karpenter.k8s.aws configured
from karpenter-provider-aws.
I fixed all nodeClaims manually with a script populating the missing "reason" and "message" fields.
But still, there is smth wrong with the CRD upgrade. I wonder whether we should re-create the CRDs to fix the CRD apply issue now? ...
from karpenter-provider-aws.
I fixed all nodeClaims manually with a script populating the missing "reason" and "message" fields. But still, there is smth wrong with the CRD upgrade. I wonder whether we should re-create the CRDs to fix the CRD apply issue now? ...
Hey @project-administrator , how did you managed to update the status field of the nodeclaim? I too upgraded the crds first then ran the karpetner upgrade - seems to be related.
from karpenter-provider-aws.
I fixed all nodeClaims manually with a script populating the missing "reason" and "message" fields. But still, there is smth wrong with the CRD upgrade. I wonder whether we should re-create the CRDs to fix the CRD apply issue now? ...
Hey @project-administrator , how did you managed to update the status field of the nodeclaim? I too upgraded the crds first then ran the karpetner upgrade - seems to be related.
Update: We have karpenter crds added separately as an argocd application with the replace flag but it seems that they were not upgraded. Manually applying the crds as per docs fixed the issue.
from karpenter-provider-aws.
hi im also running into issue with error
The CustomResourceDefinition "nodepools.karpenter.sh" is invalid: status.storedVersions[1]: Invalid value: "v1": must appear in spec.versions
The CustomResourceDefinition "nodeclaims.karpenter.sh" is invalid: status.storedVersions[1]: Invalid value: "v1": must appear in spec.versions
The CustomResourceDefinition "ec2nodeclasses.karpenter.k8s.aws" is invalid: status.storedVersions[1]: Invalid value: "v1": must appear in spec.versions
after applying crd using this guide https://karpenter.sh/preview/upgrading/upgrade-guide/
my eks version is 1.30
installing 0.37 karpenter
from karpenter-provider-aws.
hi im also running into issue with error
The CustomResourceDefinition "nodepools.karpenter.sh" is invalid: status.storedVersions[1]: Invalid value: "v1": must appear in spec.versions
The CustomResourceDefinition "nodeclaims.karpenter.sh" is invalid: status.storedVersions[1]: Invalid value: "v1": must appear in spec.versions
The CustomResourceDefinition "ec2nodeclasses.karpenter.k8s.aws" is invalid: status.storedVersions[1]: Invalid value: "v1": must appear in spec.versionsafter applying crd using this guide https://karpenter.sh/preview/upgrading/upgrade-guide/
my eks version is 1.30
installing 0.37 karpenter
@rocklim hey, did you try to reapply the cards for nodeclaim after upgrade? It fixed the issue in my case.
from karpenter-provider-aws.
@rocklim hey, did you try to reapply the cards for nodeclaim after upgrade? It fixed the issue in my case.
I did this and it fixed the issue by forcing all nodes to be recycled, so be careful if you have any services that would be interrupted.
from karpenter-provider-aws.
Related Issues (20)
- Migrating pods from One NodePool to Another HOT 1
- `make setup` has been deleted but still exists in the document
- GPU workload pod failed to schedule due to low-priority gpu placeholder daemonset HOT 2
- Allocatable memory less than actual memory during scheduling (trn1.32x) HOT 2
- Upgrade process for Karpenter HOT 1
- Support pod PID limits through NodePool spec.template.spec.kubelet
- EC2 capacity blocks for ML in Karpenter Nodepools
- Pods were CrashLoopBackOff in provisioned node by karpenter(rare occurrences)
- Get used AMI-Images by metrics
- EC2 Instances configuration messed up when spawned by Karpenter, they fail to register to the cluster
- Metric karpenter_interruption_received_messages(message_type="SpotInterruptionKind") is not accurate
- karpenter unable to spin up nodes for statefulset
- Manage CRDs with Kustomize
- Karpenter frequently pulls up nodes and pods cannot be scheduled HOT 1
- no service port 8443 found for service "karpenter" after migrating from cluster autoscaler HOT 1
- Option to disable setting default blockDeviceMappings HOT 1
- Resolving STS Credentials with I/O Timeout - use karpenter DNS and no CoreDNS?
- Karpenter metrics documentation should be improved
- Node Metrics included nodes not managed by Karpenter
- Distribute workloads across multiple Availability Zones to mitigate the risk of single-zone failure. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from karpenter-provider-aws.