在kuboard监控套件安装时,prometheus-k8两个pod状态一直为Pending,文中提供分析及解决方法。
0. ENV
kubernetes V1.23.5;
Kuboard V3.5.0.1;
资源层监控套件system-monitor.addons.kuboard.cn v3.1.7;
1. 问题现象
安装kuboard套件时,prometheus-k8s两个pod不能启动,状态为Pending
[root@rh-master01 ~]# kubectl get pod -n kuboardNAME READY STATUS RESTARTS AGE...prometheus-k8s-0 0/2 Pending 0 61sprometheus-k8s-1 0/2 Pending 0 61sprometheus-operator-5d7cc5dc4d-wv2ws 2/2 Running 0 103ssystem-monitor-config-864f784987-fhbzg 1/1 Running 0 62s
2. 问题分析
查看其中一个pod(prometheus-k8s-0)描述
[root@rh-master01 ~]# kubectl describe pod prometheus-k8s-0 -n kuboard...Warning FailedScheduling 114s default-scheduler 0/6 nodes are available: 6 pod has unbound immediate PersistentVolumeClaims.Warning FailedScheduling 35s (x1 over 112s) default-scheduler 0/6 nodes are available: 6 pod has unbound immediate PersistentVolumeClaims.
上述提示告警,当前6个节点上的6个pod的未绑定(unbound)。
查看pvc状态,两个PVC确实正常挂载,状态为Pending:
[root@rh-master01 ~]# kubectl get pvc -n kuboard | grep kuboardkuboard-kube-prometheus0 Pending 5skuboard-kube-prometheus1 Pending 5s
3. 问题原因
pod依赖的PVC(存储)未挂载,需要确保PVC挂载,待问题进一步分析。
4. 解决方法
删除并重建kuboard依赖的PVC。
1) 删除原有StorageClass、PV、PVC
[root@rh-master01 kuboard]# kubectl delete -f kuboard-kube-prometheusV1.yaml
2) 修改配置
确保SotrageClass、PV、PVC名称统一,并正常挂载。
vim kuboard-kube-prometheusV2.yaml #新建V2进行修改,调整PVC相关的配置,确保StorageClass、PV、PVC的storageClassName名称保持相同1 apiVersion: storage.k8s.io/v12 kind: StorageClass3 metadata:4 name: kuboard-kube-prometheus #StorageClass名称为kuboard-kube-prometheus5 provisioner: kubernetes.io/no-provisioner6 reclaimPolicy: Retain7 volumeBindingMode: Immediate89 ---10 apiVersion: v111 kind: PersistentVolume12 metadata:13 name: kuboard-kube-prometheus014 spec:15 accessModes:16 - ReadWriteMany17 capacity:18 storage: 40Gi19 nfs:20 path: vm/dev-nfs/kuboard_pv/prometheus-k8s-db-prometheus-k8s-021 server: 192.18.80.15922 persistentVolumeReclaimPolicy: Retain23 storageClassName: kuboard-kube-prometheus #PV的storageClassName保持和StorageClass名称相同,均为kuboard-kube-prometheus24 volumeMode: Filesystem...43 ---44 apiVersion: v145 kind: PersistentVolumeClaim46 metadata:47 labels:48 name: prometheus-k8s-db-prometheus-k8s-049 name: prometheus-k8s-db-prometheus-k8s-050 namespace: kuboard51 spec:52 accessModes:53 - ReadWriteMany54 resources:55 requests:56 storage: 40Gi57 storageClassName: kuboard-kube-prometheus #PVC的storageClassName一行,确保和StorageClass、PV的storageClassName名称相同...
3) 再次创建StorageClass、PV、PVC
[root@rh-master01 kuboard]# kubectl apply -f kuboard-kube-prometheusV2.yamlstorageclass.storage.k8s.io/kuboard-kube-prometheus createdpersistentvolume/kuboard-kube-prometheus0 createdpersistentvolume/kuboard-kube-prometheus1 createdpersistentvolumeclaim/prometheus-k8s-db-prometheus-k8s-0 createdpersistentvolumeclaim/prometheus-k8s-db-prometheus-k8s-1 created
4) prometheus-k8s两个pod运行正常
稍等片刻后,pod运行正常,问题解决。
[root@rh-master01 ~]# k get pod -n kuboardNAME READY STATUS RESTARTS AGE...prometheus-k8s-0 2/2 Running 0 37mprometheus-k8s-1 2/2 Running 0 37mprometheus-operator-5d7cc5dc4d-wv2ws 2/2 Running 0 38m
5. 小结
在kuboard监控套件安装时,prometheus-k8两个pod运行异常,通过分析,PVC未正常挂载,统一SotrageClass、PV、PVC名称,并重建后,pod运行正常。
-- 完 --
更多精彩,敬请期待
不足之处,还望抛转。
作者:王坤,微信公众号:rundba,欢迎转载,转载请注明出处。
如需公众号转发,请联系wx:landnow。

往期推荐
文章转载自rundba,如果涉嫌侵权,请发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。




