Comments (3)
nm pod的yaml如下
apiVersion: v1
kind: Pod
metadata:
annotations:
cni.projectcalico.org/containerID: cd1d69000010b7f504918b6337f51073545cddd5b1aece4ea080c62c46c69932
cni.projectcalico.org/podIP: 10.244.168.31/32
cni.projectcalico.org/podIPs: 10.244.168.31/32
yarn.hadoop.apache.org/node-id: hadoop-yarn-nm-0:8041
creationTimestamp: "2024-06-06T13:12:53Z"
generateName: hadoop-yarn-nm-
labels:
app.kubernetes.io/component: node-manager
app.kubernetes.io/instance: hadoop-yarn
app.kubernetes.io/name: hadoop-yarn
controller-revision-hash: hadoop-yarn-nm-5b5cc8688c
koordinator.sh/qosClass: BE
statefulset.kubernetes.io/pod-name: hadoop-yarn-nm-0
name: hadoop-yarn-nm-0
namespace: hadoop-yarn
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: StatefulSet
name: hadoop-yarn-nm
uid: bc2740be-043a-4068-af0a-5852059548f0
resourceVersion: "1763244"
uid: 5e73961d-1f00-461d-860e-7e33726b24f5
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:- labelSelector:
matchLabels:
app.kubernetes.io/component: node-manager
app.kubernetes.io/instance: hadoop-yarn
app.kubernetes.io/name: hadoop-yarn
topologyKey: kubernetes.io/hostname
containers:
- labelSelector:
- command:
- /bin/bash
- /tmp/hadoop-config/bootstrap.sh
- -d
env: - name: YARN_ROLE
value: yarn-nm - name: NM_INIT_CPU_CORES
value: "10" - name: NM_INIT_MEMORY_MB
value: "10240" - name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace - name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: registry.cn-beijing.aliyuncs.com/koordinator-sh/apache-hadoop:3.3.3-v1.1
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /node
port: 8042
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
name: yarn-nm
readinessProbe:
failureThreshold: 3
httpGet:
path: /node
port: 8042
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
resources:
limits:
kubernetes.io/batch-cpu: 2k
kubernetes.io/batch-memory: 2Gi
requests:
kubernetes.io/batch-cpu: 1k
kubernetes.io/batch-memory: 1Gi
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts: - mountPath: /tmp/hadoop-config
name: hadoop-config - mountPath: /host-cgroup/
name: host-cgroup-root - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-8j9mw
readOnly: true
- command:
- /bin/bash
- /tmp/hadoop-config/bootstrap.sh
- -d
env: - name: YARN_ROLE
value: hdfs-dn
image: registry.cn-beijing.aliyuncs.com/koordinator-sh/apache-hadoop:3.3.3-v1.1
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /
port: 9864
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
name: hdfs-dn
readinessProbe:
failureThreshold: 3
httpGet:
path: /
port: 8042
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 2
resources:
limits:
kubernetes.io/batch-cpu: 1k
kubernetes.io/batch-memory: 1Gi
requests:
kubernetes.io/batch-cpu: 1k
kubernetes.io/batch-memory: 1Gi
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts: - mountPath: /tmp/hadoop-config
name: hadoop-config - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-8j9mw
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostname: hadoop-yarn-nm-0
nodeName: 10.10.10.22
priorityClassName: koord-batch
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: yarn-nodemanager
serviceAccountName: yarn-nodemanager
setHostnameAsFQDN: false
subdomain: node-manager
terminationGracePeriodSeconds: 0
volumes:
- configMap:
defaultMode: 420
name: hadoop-yarn
name: hadoop-config - hostPath:
path: /sys/fs/cgroup/
type: ""
name: host-cgroup-root - name: kube-api-access-8j9mw
projected:
defaultMode: 420
sources:- serviceAccountToken:
expirationSeconds: 3607
path: token - configMap:
items:- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- key: ca.crt
- downwardAPI:
items:- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- fieldRef:
- serviceAccountToken:
- lastProbeTime: null
lastTransitionTime: "2024-06-06T13:12:53Z"
status: "True"
type: Initialized - lastProbeTime: null
lastTransitionTime: "2024-06-06T13:13:13Z"
status: "True"
type: Ready - lastProbeTime: null
lastTransitionTime: "2024-06-06T13:13:13Z"
status: "True"
type: ContainersReady - lastProbeTime: null
lastTransitionTime: "2024-06-06T13:12:53Z"
status: "True"
type: PodScheduled
containerStatuses: - containerID: docker://9f129c606eafcb12c9bf07199b99b5c4884a64fa3fe78241e3d7d2004034a022
image: registry.cn-beijing.aliyuncs.com/koordinator-sh/apache-hadoop:3.3.3-v1.1
imageID: docker://sha256:d998b66c0037312d7b0740edbd8a49b1c746f52e52ec4578c05c4b0b966f79fc
lastState: {}
name: hdfs-dn
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-06-06T13:12:54Z" - containerID: docker://21387e35289058b43ba0c87c7848beb61faa3d082f366c6099b711d46fce4df3
image: registry.cn-beijing.aliyuncs.com/koordinator-sh/apache-hadoop:3.3.3-v1.1
imageID: docker://sha256:d998b66c0037312d7b0740edbd8a49b1c746f52e52ec4578c05c4b0b966f79fc
lastState: {}
name: yarn-nm
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-06-06T13:12:54Z"
hostIP: 10.10.10.22
phase: Running
podIP: 10.244.168.31
podIPs: - ip: 10.244.168.31
qosClass: BestEffort
startTime: "2024-06-06T13:12:53Z"
from koordinator.
the annotation yarn.hadoop.apache.org/node-id: hadoop-yarn-nm-0:8041
must be equal to node id in YARN.
this annotation is patched in start script in demo chart: https://github.com/koordinator-sh/charts/blob/main/hadoop-yarn/v3.3.3/templates/hadoop-configmap.yaml#L83
if you have changed the chart config such as nm pod template, remember to change this script also.
from koordinator.
是的,我改过一个配置,当时主机名申请67个字符报错超过64个字符,我就把setHostnameAsFQDN改成了false,然后就导致了这个问题。刚我把部署的namespace从hadoop-yarn改成yarn就没有报错了
from koordinator.
Related Issues (20)
- [BUG]koord-manager failed to renew lease HOT 1
- [BUG]koord-manager failed to renew lease HOT 5
- [BUG] Koordinator doesn't support multiple card sharing HOT 8
- [BUG] hadoop-yarn rm get /proxy/application_xxx java.net.BindException: Cannot assign requested address (Bind failed) HOT 2
- [proposal] koord-descheduler: add a new arg evictAllBarePods to MigrationController HOT 1
- [question] elastic quota allow-lent-resource 没有生效 HOT 1
- [BUG] koord-manager metrics externtal exporter error
- [question]reservation对象和namespace配额疑问 HOT 1
- [question]koordinator yarn跑实时流计算作业 HOT 2
- [BUG] 添加了 Reservation Affinity 的 Pod 通过了调度器 Filte 阶段,但是节点上实际没有 Reservation 可以满足资源条件 HOT 2
- [BUG] 添加了 Reservation Affinity 的 Pod 通过了调度器 Filte 阶段,但是节点上实际没有 Reservation 可以满足资源条件
- [question] resource reservation does not consider scale up via cluster-autoscaler HOT 4
- [proposal] runtime hook should support initContainers
- [BUG] EnqueueRequestForNode should not only watche update of node's allocatable HOT 4
- [BUG] The parent queue has over the maximum HOT 1
- [BUG] unexpected CPU eviction HOT 2
- [proposal] koord-descheduler: Optimize the reassessment of single-node resource left HOT 1
- elastic quota 弹性抢占疑问[question] HOT 4
- [dependency] Upgrade github.com/containerd/nri to v0.6.1 HOT 2
- [proposal] Coscheduling move some BeforePreFilter quick check into PreEnqueueExtension HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from koordinator.