kubernetes配置之十四：kubernetes资源指标及HPA控制器

运维扫盲人 2020-11-26

774

1、kubernetes资源监控概述

1.1 资源监控及其Heapster

要实现kubernetes集群的监控必须要有一些生成、收集、存储资源指标的工具和手段；
Heapster是早期实现资源指标的方法，弊端在于仅仅支持CPU一项指标；
kubernetes在kubelet程序中集成了cAdvisor，可以收集监控CPU、内存、网络吞吐量及文件系统使用情况指标，通过http://Node_IP:4194提供Web UI，cAdvisor的问题在于仅能收集单个节点及其Pod的资源指标，且kubeadm部署的集群默认未开启此功能；
Heapster使用InfluxDB作为后端存储、Grafana为可视化接口，Heapster从各节点的cAdvisor采集数据并存储于InfluxDB中，用Grafana进行展示；
鉴于Heapster的各类限制，从1.7版本以后kubernetes聚合扩展了资源指标API和自定义指标API；

1.2 自定义指标概述

是以扩展方式提供的API；
提供用户自行按需扩展指标的接口;

1.3 资源指标概述

是以扩展方式提供的API；
主要提供核心系统组件使用，如调度程序、HPA、kubectl top命令等等，只提供核心系统指标；
不适用第三方监控系统使用，例如promethues；
同时使用自定义指标和资源指标的组件是HPAv2；
Heapster提供核心指标API的功能被聚合方式的指标API服务器metrics-server所取代；

2、资源指标与metrics-server实现

2.1 部署metrics-server

2.1.1 在未部署metrics-server时"top node"会报错

[root@k8s-master-01 ~]# kubectl top node
error: Metrics API not available

2.1.2 使用在线资源部署metrcis-server(失败)

[root@k8s-master-01 HPA]# yum install -y git
[root@k8s-master-01 HPA]# git clone https://github.com/kubernetes-incubator/metrics-server.git
Cloning into 'metrics-server'...
remote: Enumerating objects: 58, done.
remote: Counting objects: 100% (58/58), done.
remote: Compressing objects: 100% (55/55), done.
remote: Total 12635 (delta 20), reused 17 (delta 2), pack-reused 12577
Receiving objects: 100% (12635/12635), 12.56 MiB | 8.00 KiB/s, done.
Resolving deltas: 100% (6601/6601), done.
[root@k8s-master-01 HPA]# ls
metrics-server
[root@k8s-master-01 base]# kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yam
#由于镜像下载失败，改用0.3.6版本部署

2.1.3 部署v0.3.6版本的metrics-server

#https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/metrics-server
[root@k8s-master-01 ~]# wget https://github.com/kubernetes-sigs/metrics-server/archive/v0.3.6.tar.gz
[root@k8s-master-01 ~]# tar xf v0.3.6.tar.gz
[root@k8s-master-01 ~]# cd metrics-server-0.3.6/deploy/1.8+/
[root@k8s-master-01 1.8+]# pwd
/root/metrics-server-0.3.6/deploy/1.8+
[root@k8s-master-01 1.8+]# ll
total 28
-rw-rw-r-- 1 root root  393 Oct 14  2019 aggregated-metrics-reader.yaml
-rw-rw-r-- 1 root root  308 Oct 14  2019 auth-delegator.yaml
-rw-rw-r-- 1 root root  329 Oct 14  2019 auth-reader.yaml
-rw-rw-r-- 1 root root  298 Oct 14  2019 metrics-apiservice.yaml
-rw-rw-r-- 1 root root 1085 Nov 22 15:33 metrics-server-deployment.yaml
-rw-rw-r-- 1 root root  291 Oct 14  2019 metrics-server-service.yaml
-rw-rw-r-- 1 root root  517 Oct 14  2019 resource-reader.yaml
[root@k8s-master-01 1.8+]# kubectl apply -f root/metrics-server-0.3.6/deploy/1.8+/
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
Warning: rbac.authorization.k8s.io/v1beta1 RoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 RoleBinding
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
Warning: apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created

注意：下载后的metrics-service-deployment.yaml文件需要修改；

[root@k8s-master-01 1.8+]# cat metrics-server-deployment.yaml 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        #修改为本地的可用镜像
        image: mirrorgooglecontainers/metrics-server-amd64:v0.3.6
        #或者
        iamge: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64
        imagePullPolicy: IfNotPresent
        #添加command内容
        command:
        - metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalDNS,InternalIP,ExternalDNS,ExternalIP,Hostname
        #将原内容注释掉
        #image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        #imagePullPolicy: Always
        volumeMounts:
        - name: tmp-dir
          mountPath: tmp

[root@k8s-master-01 1.8+]# cat metrics-apiservice.yaml 
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

[root@k8s-master-01 1.8+]# cat metrics-server-service.yaml 
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443

[root@k8s-master-01 1.8+]# cat aggregated-metrics-reader.yaml 
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:aggregated-metrics-reader
  labels:
    rbac.authorization.k8s.io/aggregate-to-view: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list", "watch"]

[root@k8s-master-01 1.8+]# cat auth-delegator.yaml 
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

[root@k8s-master-01 1.8+]# cat auth-reader.yaml 
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

[root@k8s-master-01 1.8+]# cat resource-reader.yaml 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

2.1.4 查看部署后各资源对象的状态

[root@k8s-master-01 1.8+]# kubectl get apiservice | grep metrics
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        4m37s
[root@k8s-master-01 1.8+]# kubectl api-versions | grep metrics.k8s
metrics.k8s.io/v1beta1
#APIService成功创建了对应的群组
[root@k8s-master-01 1.8+]# kubectl get pods -n kube-system -l k8s-app=metrics-server -o wide
NAME                              READY   STATUS    RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES
metrics-server-8665c8fbd9-txlnr   1/1     Running   0          2m41s   10.244.3.12   k8s-worker-02   <none>           <none>
[root@k8s-master-01 1.8+]# kubectl logs metrics-server-8665c8fbd9-txlnr -n kube-system
I1122 07:35:06.404183       1 serving.go:312] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
I1122 07:35:07.332599       1 secure_serving.go:116] Serving securely on [::]:443

2.2 使用top测试资源指标

2.2.1 使用"top node"命令已无报错

[root@k8s-master-01 1.8+]# kubectl top node
NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-master-01   223m         5%     1562Mi          42%       
k8s-master-02   171m         4%     1528Mi          41%       
k8s-worker-01   90m          2%     533Mi           14%       
k8s-worker-02   66m          1%     821Mi           22%

2.2.2 使用资源URL路径查看资源指标

[root@k8s-master-01 1.8+]# yum install -y jq
[root@k8s-master-01 1.8+]# kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq
{
  "kind": "NodeMetricsList",
  "apiVersion": "metrics.k8s.io/v1beta1",
  "metadata": {
    "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
  },
  "items": [
    {
      "metadata": {
        "name": "k8s-worker-01",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/k8s-worker-01",
        "creationTimestamp": "2020-11-22T07:45:18Z"
      },
      "timestamp": "2020-11-22T07:45:05Z",
      "window": "30s",
      "usage": {
        "cpu": "91433503n",
        "memory": "548340Ki"
      }
    },
    {
      "metadata": {
        "name": "k8s-worker-02",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/k8s-worker-02",
        "creationTimestamp": "2020-11-22T07:45:18Z"
      },
      "timestamp": "2020-11-22T07:45:02Z",
      "window": "30s",
      "usage": {
        "cpu": "88106006n",
        "memory": "841060Ki"
      }
    },
    {
      "metadata": {
        "name": "k8s-master-01",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/k8s-master-01",
        "creationTimestamp": "2020-11-22T07:45:18Z"
      },
      "timestamp": "2020-11-22T07:45:02Z",
      "window": "30s",
      "usage": {
        "cpu": "233161058n",
        "memory": "1568216Ki"
      }
    },
    {
      "metadata": {
        "name": "k8s-master-02",
        "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/k8s-master-02",
        "creationTimestamp": "2020-11-22T07:45:18Z"
      },
      "timestamp": "2020-11-22T07:45:01Z",
      "window": "30s",
      "usage": {
        "cpu": "185202863n",
        "memory": "1542340Ki"
      }
    }
  ]
}
#使用URL路径也可查看每个Pod的指标状态;
[root@k8s-master-01 1.8+]# kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq

注意:jq是专用于处理JSON数据的命令行工具.

3、自定义指标与prometheus实现

3.1 自定义指标

除了资源指标之外,我们还需要获取很多指标,自定义指标允许API请求任意指标,其指标API要特定于相应的后端监视系统,prometheus是一个相应的监控系统.

3.2 部署prometheus监控系统

[root@k8s-master-01 HPA]# git clone https://github.com/kubernetes
Cloning into 'kubernetes'...
remote: Enumerating objects: 218, done.
remote: Counting objects: 100% (218/218), done.
remote: Compressing objects: 100% (127/127), done.
remote: Total 1169214 (delta 112), reused 91 (delta 91), pack-reused 1168996
Receiving objects: 100% (1169214/1169214), 710.78 MiB | 2.74 MiB/s, done.
Resolving deltas: 100% (839462/839462), done.
Checking out files: 100% (22748/22748), done.

注意：经过试验,https://github.com/kubernetes/kubernetes/已经没有promethues的资源文件.

[root@k8s-master-01 HPA]# git clone https://github.com/iKubernetes/k8s-prom.git
Cloning into 'k8s-prom'...
remote: Enumerating objects: 49, done.
remote: Total 49 (delta 0), reused 0 (delta 0), pack-reused 49
Unpacking objects: 100% (49/49), done.
[root@k8s-master-01 HPA]# tree k8s-prom
k8s-prom
├── k8s-prometheus-adapter
│   ├── custom-metrics-apiserver-auth-delegator-cluster-role-binding.yaml
│   ├── custom-metrics-apiserver-auth-reader-role-binding.yaml
│   ├── custom-metrics-apiserver-deployment.yaml
│   ├── custom-metrics-apiserver-resource-reader-cluster-role-binding.yaml
│   ├── custom-metrics-apiserver-service-account.yaml
│   ├── custom-metrics-apiserver-service.yaml
│   ├── custom-metrics-apiservice.yaml
│   ├── custom-metrics-cluster-role.yaml
│   ├── custom-metrics-resource-reader-cluster-role.yaml
│   └── hpa-custom-metrics-cluster-role-binding.yaml
├── kube-state-metrics
│   ├── kube-state-metrics-deploy.yaml
│   ├── kube-state-metrics-rbac.yaml
│   └── kube-state-metrics-svc.yaml
├── namespace.yaml
├── node_exporter
│   ├── node-exporter-ds.yaml
│   └── node-exporter-svc.yaml
├── podinfo
│   ├── podinfo-deploy.yaml
│   └── podinfo-svc.yaml
├── prometheus
│   ├── prometheus-cfg.yaml
│   ├── prometheus-deploy.yaml
│   ├── prometheus-rbac.yaml
│   └── prometheus-svc.yaml
└── README.md


5 directories, 23 files

注意：https://github.com/iKubernetes/为最新的promethues链接.

3.2.1 生成集群资源状态指标

kube-state-metrics能够从kubernetes API Server上收集大多数资源对象的状态信息并转化为指标数据。
kube-state-metrics主要负责为CronJob、Deployment、PersistentVolumeClaim等各类资源的对象生成指标数据；

[1] 创建专用的名称空间

[root@k8s-master-01 k8s-prom]# kubectl apply -f namespace.yaml 
namespace/prom created
[root@k8s-master-01 k8s-prom]# kubectl get namespaces
NAME              STATUS   AGE
prom              Active   7s

[2] 创建kube-state-metrics deployment

[root@k8s-worker-02 ~]# docker  pull registry.cn-hangzhou.aliyuncs.com/kubernets-imags/kube-state-metrics:v1.3.0
[root@k8s-worker-02 ~]# docker tag registry.cn-hangzhou.aliyuncs.com/kubernets-imags/kube-state-metrics:v1.3.0 gcr.io/google_containers/kube-state-metrics-amd64:v1.3.1


[root@k8s-master-01 kube-state-metrics]# kubectl apply -f kube-state-metrics/
[root@k8s-master-01 k8s-prom]# cat kube-state-metrics/kube-state-metrics-deploy.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: prom
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-state-metrics
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: gcr.io/google_containers/kube-state-metrics-amd64:v1.3.1
        ports:
        - containerPort: 8080

[3] 创建kube-state-metrics svc

[root@k8s-master-01 k8s-prom]# cat kube-state-metrics/kube-state-metrics-svc.yaml 
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  name: kube-state-metrics
  namespace: prom
  labels:
    app: kube-state-metrics
spec:
  ports:
  - name: kube-state-metrics
    port: 8080
    protocol: TCP
  selector:
    app: kube-state-metrics

[4] 创建kube-state-metrics rbac

[root@k8s-master-01 k8s-prom]# cat kube-state-metrics/kube-state-metrics-rbac.yaml 
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-state-metrics
  namespace: prom
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kube-state-metrics
rules:
- apiGroups: [""]
  resources: ["nodes", "pods", "services", "resourcequotas", "replicationcontrollers", "limitranges", "persistentvolumeclaims", "persistentvolumes", "namespaces", "endpoints"]
  verbs: ["list", "watch"]
- apiGroups: ["extensions"]
  resources: ["daemonsets", "deployments", "replicasets"]
  verbs: ["list", "watch"]
- apiGroups: ["apps"]
  resources: ["statefulsets"]
  verbs: ["list", "watch"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["list", "watch"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kube-state-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-state-metrics
subjects:
- kind: ServiceAccount
  name: kube-state-metrics
  namespace: prom

[5] 查看各kube-state-metrics 资源状态

[root@k8s-master-01 kube-state-metrics]# kubectl get pods -n prom -o wide
NAME                                  READY   STATUS    RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES
kube-state-metrics-7b99db8cdb-22vdz   1/1     Running   0          6m28s   10.244.3.13   k8s-worker-02   <none>           <none>
[root@k8s-master-01 kube-state-metrics]# kubectl get svc -n prom -o wide
NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE   SELECTOR
kube-state-metrics   ClusterIP   10.100.241.139   <none>        8080/TCP   33m   app=kube-state-metrics

[6] 测试指标获取情况

[root@k8s-worker-01 ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2
[root@k8s-worker-01 ~]# docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 k8s.gcr.io/pause:3.2
[root@k8s-master-01 kube-state-metrics]# kubectl run client-$RANDOM --image=cirros -n prom -it -- sh
/ # curl kube-state-metrics:8080/metrics | tail
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5333  100  5333    0     0  1302k      0 --:--:-- --:--:-- --:--:-- 1736k
kube_namespace_status_phase{namespace="default",phase="Active"} 1
kube_namespace_status_phase{namespace="default",phase="Terminating"} 0
kube_namespace_status_phase{namespace="kube-node-lease",phase="Active"} 1
kube_namespace_status_phase{namespace="kube-node-lease",phase="Terminating"} 0
kube_namespace_status_phase{namespace="kube-public",phase="Active"} 1
kube_namespace_status_phase{namespace="kube-public",phase="Terminating"} 0
kube_namespace_status_phase{namespace="kube-system",phase="Active"} 1
kube_namespace_status_phase{namespace="kube-system",phase="Terminating"} 0
kube_namespace_status_phase{namespace="prom",phase="Active"} 1
kube_namespace_status_phase{namespace="prom",phase="Terminating"} 0

3.2.2 部署Node exporter

prometheus通过HTTP周期性的抓取指标数据，监控目标上用于接收并相应数据抓取请求的组件统称为exporter；
node_exporter是prometheus提供的专用的监控类程序；
node_exporter不是必须组件,也可通过kubelet或者cAdvsior提供监控指标；

[1] 部署Node export的DaemonSet

[root@k8s-master-01 k8s-prom]# kubectl apply -f node_exporter/node-exporter-ds.yaml
[root@k8s-master-01 k8s-prom]# cat node_exporter/node-exporter-ds.yaml 
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: prometheus-node-exporter
  namespace: prom
  labels:
    app: prometheus
    component: node-exporter
spec:
  selector:
    matchLabels:
      app: prometheus
      component: node-exporter
  template:
    metadata:
      name: prometheus-node-exporter
      labels:
        app: prometheus
        component: node-exporter
    spec:
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      containers:
      - image: prom/node-exporter:v0.15.2
        name: prometheus-node-exporter
        ports:
        - name: prom-node-exp
          containerPort: 9100
          hostPort: 9100
      hostNetwork: true
      hostPID: true

[2] 部署Node export的svc

[root@k8s-master-01 k8s-prom]# kubectl apply -f node_exporter/node-exporter-svc.yaml
[root@k8s-master-01 k8s-prom]# cat node_exporter/node-exporter-svc.yaml 
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'
  name: prometheus-node-exporter
  namespace: prom
  labels:
    app: prometheus
    component: node-exporter
spec:
  clusterIP: None
  ports:
    - name: prometheus-node-exporter
      port: 9100
      protocol: TCP
  selector:
    app: prometheus
    component: node-exporter
  type: ClusterIP

[3] 查看各资源状态

[root@k8s-master-01 k8s-prom]# kubectl get svc -n prom -o wide
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE     SELECTOR
prometheus-node-exporter   ClusterIP   None             <none>        9100/TCP   3m57s   app=prometheus,component=node-exporter
[root@k8s-master-01 k8s-prom]# kubectl get svc -n prom -o wide
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE     SELECTOR
prometheus-node-exporter   ClusterIP   None             <none>        9100/TCP   4m16s   app=prometheus,component=node-exporter

[4]测试

[root@k8s-master-01 k8s-prom]# curl http://172.17.61.220:9100/metrics | tail
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  110k  100  110k    0     0  6765k      0 --:--:-- --:--:-- --:--:-- 7369k
process_open_fds 8
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.0027008e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.60604524865e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2.05336576e+08

3.2.3 部署告警系统Alertmanager

为prometheus根据高进规则将告警信息发送给alertmanager插件，alertmanager插件再做去重、分组处理，而后发送给告警接收端，包括email、slack

[root@k8s-master-01 alertmanager]# ls root/kubernetes-config/prometheus/k8s-prom/alertmanager
alertmanager-configmap.yaml     alertmanager-rule.yaml  alertmanager-deployment.yaml  
[root@k8s-master-01 alertmanager]# kubectl apply -f root/kubernetes-config/prometheus/k8s-prom/alertmanager
configmap/alertmanager-config unchanged
deployment.apps/alertmanager unchanged
service/alertmanager unchanged
configmap/prometheus-rules unchanged

[1] 部署Alertmanager deployment

[root@k8s-master-01 alertmanager]# cat alertmanager-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: alertmanager
  namespace: prom
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alertmanager
  template:
    metadata:
      labels:
        app: alertmanager
    spec:
      containers:
      - name: alertmanager
        image: prom/alertmanager:v0.14.0
        args:
          - "--config.file=/etc/alertmanager/config.yml"
          - "--storage.path=/alertmanager"
        ports:
        - name: alertmanager
          containerPort: 9093
        volumeMounts:
        - name: alertmanager-cm
          mountPath: etc/alertmanager
      volumes:
      - name: alertmanager-cm
        configMap:
          name: alertmanager-config

[2] 部署Alertmanager svc

apiVersion: v1
kind: Service
metadata:
  name: alertmanager
  namespace: prom
spec:
  selector:
    app: alertmanager
  ports:
    - port: 80
      targetPort: 9093
  type: "ClusterIP"

[3] 部署Alertmanager configmap

[root@k8s-master-01 alertmanager]# cat alertmanager-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: prom
data:
  config.yml: |-
    global:
      # 在没有报警的情况下声明为已解决的时间
      resolve_timeout: 5m
      # 配置邮件发送信息
      smtp_smarthost: 'smtp.163.com:25'
      smtp_from: 'xxx@163.com'
      smtp_auth_username: 'xxx@163.com'
      smtp_auth_password: 'xxxxxx'
      smtp_require_tls: false
      # 所有报警信息进入后的根路由，用来设置报警的分发策略
    route:
      # 这里的标签列表是接收到报警信息后的重新分组标签，例如，接收到的报警信息里面有许多具有 cluster=A 和 alertname=LatncyHigh 这样的标签的报警信息将会批量被聚合到一个分组里面
      group_by: ['alertname', 'cluster']
      # 当一个新的报警分组被创建后，需要等待至少group_wait时间来初始化通知，这种方式可以确保您能有足够的时间为同一分组来获取多个警报，然后一起触发这个报警信息。
      group_wait: 30s
      # 当第一个报警发送后，等待'group_interval'时间来发送新的一组报警信息。
      group_interval: 5m
      # 如果一个报警信息已经发送成功了，等待'repeat_interval'时间来重新发送他们
      repeat_interval: 5m
      # 默认的receiver：如果一个报警没有被一个route匹配，则发送给默认的接收器
      receiver: default
    receivers:
    - name: 'default'
      email_configs:
      - to: 'xxxx@qq.com'
        send_resolved: true

[4] 添加告警规则rule

告警规则为configMap资源对象，供prometheus使用；

[root@k8s-master-01 alertmanager]# cat alertmanager-rule.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-rules
  namespace: prom
data:
  # 通用角色
  general.rules: |
    groups:
    - name: general.rules
      rules:
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: error 
        annotations:
          summary: "Instance {{ $labels.instance }} 停止工作"
          description: "{{ $labels.instance }} job {{ $labels.job }} 已经停止5分钟以上."
  # Node对所有资源的监控
  node.rules: |
    groups:
    - name: node.rules
      rules:
      - alert: NodeFilesystemUsage
        expr: 100 - (node_filesystem_free_bytes{fstype=~"ext4|xfs"}  node_filesystem_size_bytes{fstype=~"ext4|xfs"} * 100) > 80 
        for: 1m
        labels:
          severity: warning 
        annotations:
          summary: "Instance {{ $labels.instance }} : {{ $labels.mountpoint }} 分区使用率过高"
          description: "{{ $labels.instance }}: {{ $labels.mountpoint }} 分区使用大于80% (当前值: {{ $value }})"


      - alert: NodeMemoryUsage
        expr: 100 - (node_memory_MemFree_bytes+node_memory_Cached_bytes+node_memory_Buffers_bytes)  node_memory_MemTotal_bytes * 100 > 80
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Instance {{ $labels.instance }} 内存使用率过高"
          description: "{{ $labels.instance }}内存使用大于80% (当前值: {{ $value }})"


      - alert: NodeCPUUsage    
        expr: 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance) * 100) > 60 
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "Instance {{ $labels.instance }} CPU使用率过高"       
          description: "{{ $labels.instance }}CPU使用大于60% (当前值: {{ $value }})"

[5] 修改prometheus configmap使得prometheus可以与alertmanager通信

[root@k8s-master-01 alertmanager]# cat  ../prometheus/prometheus-cfg.yaml  | egrep -v '^#|^[[:space:]]*#' 
---
kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    app: prometheus
  name: prometheus-config
  namespace: prom
data:
  prometheus.yml: |


    global:
      scrape_interval: 15s
      scrape_timeout: 10s
      evaluation_interval: 1m


    scrape_configs:
    - job_name: 'kubernetes-apiservers'


      kubernetes_sd_configs:
      - role: endpoints


      scheme: https


      tls_config:
        ca_file: var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: var/run/secrets/kubernetes.io/serviceaccount/token


      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https


    - job_name: 'kubernetes-nodes'


      scheme: https


      tls_config:
        ca_file: var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: var/run/secrets/kubernetes.io/serviceaccount/token


      kubernetes_sd_configs:
      - role: node


      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: api/v1/nodes/${1}/proxy/metrics


    - job_name: 'kubernetes-cadvisor'


      scheme: https


      tls_config:
        ca_file: var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: var/run/secrets/kubernetes.io/serviceaccount/token


      kubernetes_sd_configs:
      - role: node


      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: api/v1/nodes/${1}/proxy/metrics/cadvisor


    - job_name: 'kubernetes-service-endpoints'


      kubernetes_sd_configs:
      - role: endpoints


      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name


    - job_name: 'kubernetes-pods'
      honor_labels: false
      kubernetes_sd_configs:
      - role: pod
      tls_config:
        insecure_skip_verify: true
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (.+)
    alerting:
      alertmanagers:
        - static_configs:
            - targets: ["alertmanager:80"]
    rule_files:
      - etc/config/rules/*.rules

注意：prometheus-config中的alertmanager:80为alermanager的svc名称，通过该名称与alertmanager通信；

[6] 修改prometheus deployment

为了保证prometheus能够加载到告警规则，该实例中将prometheus-rules直接挂载在prometheus的容器上，也可使用statefulset来加载prometheus-rules；

[root@k8s-master-01 alertmanager]# cat ../prometheus/prometheus-deploy.yaml 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-server
  namespace: prom
  labels:
    app: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
      component: server
    #matchExpressions:
    #- {key: app, operator: In, values: [prometheus]}
    #- {key: component, operator: In, values: [server]}
  template:
    metadata:
      labels:
        app: prometheus
        component: server
      annotations:
        prometheus.io/scrape: 'false'
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus
        image: prom/prometheus:v2.2.1
        imagePullPolicy: Always
        command:
          - prometheus
          - --config.file=/etc/prometheus/prometheus.yml
          - --storage.tsdb.path=/prometheus
          - --storage.tsdb.retention=720h
        ports:
        - containerPort: 9090
          protocol: TCP
        resources:
          limits:
            memory: 2Gi
        volumeMounts:
        - mountPath: etc/prometheus/prometheus.yml
          name: prometheus-config
          subPath: prometheus.yml
        - mountPath: prometheus/
          name: prometheus-storage-volume
        - mountPath: etc/config/rules
          name: prometheus-alertmanage
      volumes:
        - name: prometheus-config
          configMap:
            name: prometheus-config
            items:
              - key: prometheus.yml
                path: prometheus.yml
                mode: 0644
        - name: prometheus-alertmanage
          configMap:
            name: prometheus-rules
        - name: prometheus-storage-volume
          emptyDir: {}

[7] 验证rule是否生效

[root@k8s-master-01 alertmanager]# kubectl exec prometheus-server-75d864d655-h5cf7 -n prom -- ls etc/config/rules
general.rules
node.rules
[root@k8s-master-01 alertmanager]# kubectl get pod -n prom
NAME                                        READY   STATUS        RESTARTS   AGE
alertmanager-6656585ccd-2xp95               1/1     Running       0          3h5m
custom-metrics-apiserver-55c849446d-g2kn6   1/1     Running       0          5h41m
custom-metrics-apiserver-d6b994c76-tdmm2    1/1     Running   0          2d19h
kube-state-metrics-7b99db8cdb-22vdz         1/1     Running       0          2d21h
prometheus-node-exporter-42z4b              1/1     Running       0          5h42m
prometheus-node-exporter-cdllq              1/1     Running   0          2d19h
prometheus-node-exporter-gmnmv              1/1     Running       0          5h42m
prometheus-node-exporter-jd7qt              1/1     Running       0          5h42m
prometheus-server-75d864d655-h5cf7          1/1     Running       0          118m
[root@k8s-master-01 alertmanager]# kubectl get svc -n prom
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
alertmanager               ClusterIP   10.97.198.54     <none>        80/TCP           3h5m
custom-metrics-apiserver   ClusterIP   10.96.103.132    <none>        443/TCP          5h41m
kube-state-metrics         ClusterIP   10.100.241.139   <none>        8080/TCP         2d21h
prometheus-node-exporter   ClusterIP   None             <none>        9100/TCP         5h42m

3.2.4 部署prometheus服务器

[root@k8s-master-01 k8s-prom]# kubectl apply -f prometheus/
configmap/prometheus-config created
deployment.apps/prometheus-server created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/prometheus created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
service/prometheus created

[1]部署prometheus Depolyment

[root@k8s-master-01 k8s-prom]# cat prometheus/prometheus-deploy.yaml 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-server
  namespace: prom
  labels:
    app: prometheus
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
      component: server
    #matchExpressions:
    #- {key: app, operator: In, values: [prometheus]}
    #- {key: component, operator: In, values: [server]}
  template:
    metadata:
      labels:
        app: prometheus
        component: server
      annotations:
        prometheus.io/scrape: 'false'
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus
        image: prom/prometheus:v2.2.1
        imagePullPolicy: Always
        command:
          - prometheus
          - --config.file=/etc/prometheus/prometheus.yml
          - --storage.tsdb.path=/prometheus
          - --storage.tsdb.retention=720h
        ports:
        - containerPort: 9090
          protocol: TCP
        resources:
          limits:
            memory: 2Gi
        volumeMounts:
        - mountPath: etc/prometheus/prometheus.yml
          name: prometheus-config
          subPath: prometheus.yml
        - mountPath: prometheus/
          name: prometheus-storage-volume
      volumes:
        - name: prometheus-config
          configMap:
            name: prometheus-config
            items:
              - key: prometheus.yml
                path: prometheus.yml
                mode: 0644
        - name: prometheus-storage-volume
          emptyDir: {}

[2]部署prometheus configmap

[root@k8s-master-01 k8s-prom]# cat prometheus/prometheus-cfg.yaml 
---
kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    app: prometheus
  name: prometheus-config
  namespace: prom
data:
  prometheus.yml: |
    # A scrape configuration for running Prometheus on a Kubernetes cluster.
    # This uses separate scrape configs for cluster components (i.e. API server, node)
    # and services to allow each to use different authentication configs.
    #
    # Kubernetes labels will be added as Prometheus labels on metrics via the
    # `labelmap` relabeling action.
    #
    # If you are using Kubernetes 1.7.2 or earlier, please take note of the comments
    # for the kubernetes-cadvisor job; you will need to edit or remove this job.


    # Scrape config for API servers.
    #
    # Kubernetes exposes API servers as endpoints to the default/kubernetes
    # service so this uses `endpoints` role and uses relabelling to only keep
    # the endpoints associated with the default/kubernetes service using the
    # default named port `https`. This works for single API server deployments as
    # well as HA API server deployments.
    global:
      scrape_interval: 15s
      scrape_timeout: 10s
      evaluation_interval: 1m


    scrape_configs:
    - job_name: 'kubernetes-apiservers'


      kubernetes_sd_configs:
      - role: endpoints


      # Default to scraping over https. If required, just disable this or change to
      # `http`.
      scheme: https


      # This TLS & bearer token file config is used to connect to the actual scrape
      # endpoints for cluster components. This is separate to discovery auth
      # configuration because discovery & scraping are two separate concerns in
      # Prometheus. The discovery auth config is automatic if Prometheus runs inside
      # the cluster. Otherwise, more config options have to be provided within the
      # <kubernetes_sd_config>.
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        # If your node certificates are self-signed or use a different CA to the
        # master CA, then disable certificate verification below. Note that
        # certificate verification is an integral part of a secure infrastructure
        # so this should only be disabled in a controlled environment. You can
        # disable certificate verification by uncommenting the line below.
        #
        # insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token


      # Keep only the default/kubernetes service endpoints for the https port. This
      # will add targets for each API server which Kubernetes adds an endpoint to
      # the default/kubernetes service.
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https


    # Scrape config for nodes (kubelet).
    #
    # Rather than connecting directly to the node, the scrape is proxied though the
    # Kubernetes apiserver.  This means it will work if Prometheus is running out of
    # cluster, or can't connect to nodes for some other reason (e.g. because of
    # firewalling).
    - job_name: 'kubernetes-nodes'


      # Default to scraping over https. If required, just disable this or change to
      # `http`.
      scheme: https


      # This TLS & bearer token file config is used to connect to the actual scrape
      # endpoints for cluster components. This is separate to discovery auth
      # configuration because discovery & scraping are two separate concerns in
      # Prometheus. The discovery auth config is automatic if Prometheus runs inside
      # the cluster. Otherwise, more config options have to be provided within the
      # <kubernetes_sd_config>.
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token


      kubernetes_sd_configs:
      - role: node


      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics


    # Scrape config for Kubelet cAdvisor.
    #
    # This is required for Kubernetes 1.7.3 and later, where cAdvisor metrics
    # (those whose names begin with 'container_') have been removed from the
    # Kubelet metrics endpoint.  This job scrapes the cAdvisor endpoint to
    # retrieve those metrics.
    #
    # In Kubernetes 1.7.0-1.7.2, these metrics are only exposed on the cAdvisor
    # HTTP endpoint; use "replacement: /api/v1/nodes/${1}:4194/proxy/metrics"
    # in that case (and ensure cAdvisor's HTTP server hasn't been disabled with
    # the --cadvisor-port=0 Kubelet flag).
    #
    # This job is not necessary and should be removed in Kubernetes 1.6 and
    # earlier versions, or it will cause the metrics to be scraped twice.
    - job_name: 'kubernetes-cadvisor'


      # Default to scraping over https. If required, just disable this or change to
      # `http`.
      scheme: https


      # This TLS & bearer token file config is used to connect to the actual scrape
      # endpoints for cluster components. This is separate to discovery auth
      # configuration because discovery & scraping are two separate concerns in
      # Prometheus. The discovery auth config is automatic if Prometheus runs inside
      # the cluster. Otherwise, more config options have to be provided within the
      # <kubernetes_sd_config>.
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token


      kubernetes_sd_configs:
      - role: node


      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor


    # Scrape config for service endpoints.
    #
    # The relabeling allows the actual service scrape endpoint to be configured
    # via the following annotations:
    #
    # * `prometheus.io/scrape`: Only scrape services that have a value of `true`
    # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need
    # to set this to `https` & most likely set the `tls_config` of the scrape config.
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: If the metrics are exposed on a different port to the
    # service then set this appropriately.
    - job_name: 'kubernetes-service-endpoints'


      kubernetes_sd_configs:
      - role: endpoints


      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name


    # Example scrape config for pods
    #
    # The relabeling allows the actual pod scrape endpoint to be configured via the
    # following annotations:
    #
    # * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the
    # pod's declared ports (default is a port-free target if none are declared).
    - job_name: 'kubernetes-pods'
      # if you want to use metrics on jobs, set the below field to
      # true to prevent Prometheus from setting the `job` label
      # automatically.
      honor_labels: false
      kubernetes_sd_configs:
      - role: pod
      # skip verification so you can do HTTPS to pods
      tls_config:
        insecure_skip_verify: true
      # make sure your labels are in order
      relabel_configs:
      # these labels tell Prometheus to automatically attach source
      # pod and namespace information to each collected sample, so
      # that they'll be exposed in the custom metrics API automatically.
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      # these labels tell Prometheus to look for
      # prometheus.io/{scrape,path,port} annotations to configure
      # how to scrape
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (.+)

[3]部署prometheus svc

[root@k8s-master-01 k8s-prom]# cat prometheus/prometheus-svc.yaml 
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: prom
  labels:
    app: prometheus
spec:
  type: NodePort
  ports:
    - port: 9090
      targetPort: 9090
      nodePort: 30090
      protocol: TCP
  selector:
    app: prometheus
    component: server

[4]部署prometheus rbac

[root@k8s-master-01 k8s-prom]# cat prometheus/prometheus-rbac.yaml 
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: prom
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: prom

[5] 查看prometheus 各组件状态并验证

[root@k8s-master-01 k8s-prom]# kubectl get pods -n prom -o wide
NAME                                  READY   STATUS              RESTARTS   AGE    IP              NODE            NOMINATED NODE   READINESS GATES
prometheus-server-59db489b4f-6xfrk    1/1     Running             0          3m6s   10.244.3.19     k8s-worker-02   <none>           <none>
[root@k8s-master-01 k8s-prom]# kubectl get svc -n prom -o wide
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE     SELECTOR
prometheus                 NodePort    10.102.109.46    <none>        9090:30090/TCP   9m18s   app=prometheus,component=server

http://172.17.61.220:30090/targets

3.2.5 部署自定义适配器k8s-prometheus-adapter

prometheus提供的PromQL无法直接作为自定义指标数据使用,它并非kubernetes系统的聚合API服务器,需要定一个中间层--kubernetes custom metrics adapter for prometheus,其中k8s-prometheus-adapter是一个很好的实现;

[1] 应用对应项目下的配置文件

[root@k8s-master-01 k8s-prom]# kubectl apply -f k8s-prometheus-adapter/
clusterrolebinding.rbac.authorization.k8s.io/custom-metrics:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/custom-metrics-auth-reader created
deployment.apps/custom-metrics-apiserver created
clusterrolebinding.rbac.authorization.k8s.io/custom-metrics-resource-reader created
serviceaccount/custom-metrics-apiserver created
service/custom-metrics-apiserver created
Warning: apiregistration.k8s.io/v1beta1 APIService is deprecated in v1.19+, unavailable in v1.22+; use apiregistration.k8s.io/v1 APIService
apiservice.apiregistration.k8s.io/v1beta1.custom.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/custom-metrics-server-resources created
clusterrole.rbac.authorization.k8s.io/custom-metrics-resource-reader created
clusterrolebinding.rbac.authorization.k8s.io/hpa-controller-custom-metrics created

[2] 为自定义API Sever定义自签证书

[root@k8s-master-01 k8s-prom]# cd /etc/kubernetes/pki/
[root@k8s-master-01 pki]# (umask 077; openssl genrsa -out serving.key 2048)
Generating RSA private key, 2048 bit long modulus
...................................+++++
.........................................+++++
e is 65537 (0x10001)
[root@k8s-master-01 pki]#  openssl req -new -key serving.key -out serving.csr -subj "/CN=serving"
[root@k8s-master-01 pki]# openssl x509 -req -in serving.csr -CA ./ca.crt -CAkey ./ca.key -CAcreateserial -out serving.crt -days 3650
Signature ok
subject=/CN=serving
Getting CA Private Key
[root@k8s-master-01 pki]# kubectl create secret generic cm-adapter-serving-certs --from-file=serving.crt=./serving.crt --from-file=serving.key -n prom
secret/cm-adapter-serving-certs created

[3] 在线下载的两个配置文件有错误需要替换（注意）

[root@k8s-master-01 k8s-prometheus-adapter]# wget https://raw.githubusercontent.com/DirectXMan12/k8s-prometheus-adapter/master/deploy/manifests/custom-metrics-config-map.yaml
[root@k8s-master-01 k8s-prometheus-adapter]# wget https://raw.githubusercontent.com/DirectXMan12/k8s-prometheus-adapter/master/deploy/manifests/custom-metrics-apiserver-deployment.yaml

[root@k8s-master-01 k8s-prometheus-adapter]# cat custom-metrics-apiserver-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: custom-metrics-apiserver
  name: custom-metrics-apiserver
  namespace: prom
spec:
  replicas: 1
  selector:
    matchLabels:
      app: custom-metrics-apiserver
  template:
    metadata:
      labels:
        app: custom-metrics-apiserver
      name: custom-metrics-apiserver
    spec:
      serviceAccountName: custom-metrics-apiserver
      containers:
      - name: custom-metrics-apiserver
        image: directxman12/k8s-prometheus-adapter-amd64
        args:
        - --secure-port=6443
        - --tls-cert-file=/var/run/serving-cert/serving.crt
        - --tls-private-key-file=/var/run/serving-cert/serving.key
        - --logtostderr=true
        - --prometheus-url=http://prometheus.prom.svc:9090/
        - --metrics-relist-interval=1m
        - --v=10
        - --config=/etc/adapter/config.yaml
        ports:
        - containerPort: 6443
        volumeMounts:
        - mountPath: /var/run/serving-cert
          name: volume-serving-cert
          readOnly: true
        - mountPath: /etc/adapter/
          name: config
          readOnly: true
        - mountPath: /tmp
          name: tmp-vol
      volumes:
      - name: volume-serving-cert
        secret:
          secretName: cm-adapter-serving-certs
      - name: config
        configMap:
          name: adapter-config
      - name: tmp-vol
        emptyDir: {}

[root@k8s-master-01 k8s-prometheus-adapter]# cat custom-metrics-config-map.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: adapter-config
  namespace: prom
data:
  config.yaml: |
    rules:
    - seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
      seriesFilters: []
      resources:
        overrides:
          namespace:
            resource: namespace
          pod_name:
            resource: pod
      name:
        matches: ^container_(.*)_seconds_total$
        as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[1m])) by (<<.GroupBy>>)
    - seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
      seriesFilters:
      - isNot: ^container_.*_seconds_total$
      resources:
        overrides:
          namespace:
            resource: namespace
          pod_name:
            resource: pod
      name:
        matches: ^container_(.*)_total$
        as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[1m])) by (<<.GroupBy>>)
    - seriesQuery: '{__name__=~"^container_.*",container_name!="POD",namespace!="",pod_name!=""}'
      seriesFilters:
      - isNot: ^container_.*_total$
      resources:
        overrides:
          namespace:
            resource: namespace
          pod_name:
            resource: pod
      name:
        matches: ^container_(.*)$
        as: ""
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}) by (<<.GroupBy>>)
    - seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
      seriesFilters:
      - isNot: .*_total$
      resources:
        template: <<.Resource>>
      name:
        matches: ""
        as: ""
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
    - seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
      seriesFilters:
      - isNot: .*_seconds_total
      resources:
        template: <<.Resource>>
      name:
        matches: ^(.*)_total$
        as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
    - seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
      seriesFilters: []
      resources:
        template: <<.Resource>>
      name:
        matches: ^(.*)_seconds_total$
        as: ""
      metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
    resourceRules:
      cpu:
        containerQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)
        nodeQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>, id='/'}[1m])) by (<<.GroupBy>>)
        resources:
          overrides:
            instance:
              resource: node
            namespace:
              resource: namespace
            pod_name:
              resource: pod
        containerLabel: container_name
      memory:
        containerQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>}) by (<<.GroupBy>>)
        nodeQuery: sum(container_memory_working_set_bytes{<<.LabelMatchers>>,id='/'}) by (<<.GroupBy>>)
        resources:
          overrides:
            instance:
              resource: node
            namespace:
              resource: namespace
            pod_name:
              resource: pod
        containerLabel: container_name
      window: 1m
    externalRules:
    - seriesQuery: '{__name__=~"^.*_queue_(length|size)$",namespace!=""}'
      resources:
        overrides:
          namespace:
            resource: namespace
      name:
        matches: ^.*_queue_(length|size)$
        as: "$0"
      metricsQuery: max(<<.Series>>{<<.LabelMatchers>>})
    - seriesQuery: '{__name__=~"^.*_queue$",namespace!=""}'
      resources:
        overrides:
          namespace:
            resource: namespace
      name:
        matches: ^.*_queue$
        as: "$0"
      metricsQuery: max(<<.Series>>{<<.LabelMatchers>>})

[4] 新注册的API Server会生成新的API群组

[root@k8s-master-01 k8s-prometheus-adapter]# kubectl api-versions | grep custom
custom.metrics.k8s.io/v1beta1

[5] 使用自定义的API查看指定的指标值

[root@k8s-master-01 k8s-prometheus-adapter]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"  |  jq '.resources[].name'
"jobs.batch/go_memstats_buck_hash_sys_bytes"
"namespaces/process_virtual_memory_bytes"
"pods/go_memstats_lookups"
"jobs.batch/http_requests"
"pods/go_memstats_mspan_inuse_bytes"
"pods/http_requests_bucket"
"namespaces/http_requests_count"
"pods/http_requests_count"
"jobs.batch/kube_endpoint_address_available"
"pods/go_memstats_alloc_bytes"
"namespaces/go_memstats_last_gc_time_seconds"
"pods/go_memstats_other_sys_bytes"
"pods/scrape_samples_scraped"
"jobs.batch/kube_endpoint_address_not_ready"
"namespaces/kube_namespace_annotations"
"namespaces/http_requests"

rate(container_cpu_usage_seconds_total{image!=" ",pod_name!=" "}[1m])

sum by (pod)(rate(container_cpu_usage_seconds_total{image!=" ", pod_name!=" "}[1m] ))

3.3 部署grafana展示界面

3.3.1 部署grafana deployment

[root@k8s-master-01 grafana]# cat grafana-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana-core
  namespace: prom
  labels:
    app: grafana
    component: core
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
        component: core
    spec:
      containers:
      - image: grafana/grafana:5.0.4
        name: grafana-core
        imagePullPolicy: IfNotPresent
        # env:
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        env:
          # The following env variables set up basic auth twith the default admin user and admin password.
          - name: GF_AUTH_BASIC_ENABLED
            value: "true"
          - name: GF_AUTH_ANONYMOUS_ENABLED
            value: "false"
          # - name: GF_AUTH_ANONYMOUS_ORG_ROLE
          #   value: Admin
          # does not really work, because of template variables in exported dashboards:
          # - name: GF_DASHBOARDS_JSON_ENABLED
          #   value: "true"
        readinessProbe:
          httpGet:
            path: /login
            port: 3000
          # initialDelaySeconds: 30
          # timeoutSeconds: 1
        volumeMounts:
        - name: grafana-persistent-storage
          mountPath: /var
      volumes:
      - name: grafana-persistent-storage
        emptyDir: {}

3.3.2 部署grafana svc

[root@k8s-master-01 grafana]# cat grafana-svc.yaml 
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: prom
  labels:
    app: grafana
    component: core
spec:
  type: NodePort
  selector:
    app: grafana
    component: core
  ports:
  - protocol: TCP
    port: 3000
    targetPort: 3000
    nodePort: 32220
http://172.17.61.220:32220

[root@k8s-master-01 grafana]# kubectl get svc -n prom
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
grafana                    NodePort    10.101.121.39    <none>        3000:32220/TCP   11h

3.3.3 grafana展示界面配置

# User:admin
# Password:admin

#配置prometheus源数据，https://prometheus:9090

#grafana官方站点有丰富的dashboard，可以自行下载导入。
https://grafana.com/grafana/dashboards/162

<-------------------------------------------------------------------------->

Note：整套Prometheus创建完成后的所有资源状态：

[root@k8s-master-01 ~]# kubectl get deployment -n prom  -o wide
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS                 IMAGES                                                     SELECTOR
alertmanager               1/1     1            1           21h     alertmanager               prom/alertmanager:v0.14.0                                  app=alertmanager
custom-metrics-apiserver   1/1     1            1           24h     custom-metrics-apiserver   directxman12/k8s-prometheus-adapter-amd64                  app=custom-metrics-apiserver
grafana-core               1/1     1            1           12h     grafana-core               grafana/grafana:5.0.4                                      app=grafana
kube-state-metrics         1/1     1            1           3d15h   kube-state-metrics         gcr.io/google_containers/kube-state-metrics-amd64:v1.3.1   app=kube-state-metrics
prometheus-server          1/1     1            1           20h     prometheus                 prom/prometheus:v2.2.1                                     app=prometheus,component=server
[root@k8s-master-01 ~]# kubectl get daemonset -n prom  -o wide
NAME                       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE   CONTAINERS                 IMAGES                       SELECTOR
prometheus-node-exporter   4         4         4       4            4           <none>          24h   prometheus-node-exporter   prom/node-exporter:v0.15.2   app=prometheus,component=node-exporter
[root@k8s-master-01 ~]# kubectl get pod -n prom  -o wide
NAME                                        READY   STATUS    RESTARTS   AGE     IP              NODE            NOMINATED NODE   READINESS GATES
alertmanager-6656585ccd-2xp95               1/1     Running   0          21h     10.244.3.80     k8s-worker-02   <none>           <none>
custom-metrics-apiserver-55c849446d-g2kn6   1/1     Running   0          24h     10.244.3.69     k8s-worker-02   <none>           <none>
grafana-core-657675c8d9-8frnw               1/1     Running   0          12h     10.244.3.99     k8s-worker-02   <none>           <none>
kube-state-metrics-7b99db8cdb-22vdz         1/1     Running   0          3d15h   10.244.3.13     k8s-worker-02   <none>           <none>
prometheus-0                                2/2     Running   0          19h     10.244.3.86     k8s-worker-02   <none>           <none>
prometheus-node-exporter-42z4b              1/1     Running   0          24h     172.17.61.223   k8s-worker-02   <none>           <none>
prometheus-node-exporter-cn5tl              1/1     Running   0          9m25s   172.17.61.222   k8s-worker-01   <none>           <none>
prometheus-node-exporter-gmnmv              1/1     Running   0          24h     172.17.61.220   k8s-master-01   <none>           <none>
prometheus-node-exporter-jd7qt              1/1     Running   0          24h     172.17.61.221   k8s-master-02   <none>           <none>
prometheus-server-75d864d655-h5cf7          1/1     Running   0          20h     10.244.3.85     k8s-worker-02   <none>           <none>
[root@k8s-master-01 ~]# kubectl get svc -n prom  -o wide
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE     SELECTOR
alertmanager               ClusterIP   10.97.198.54     <none>        80/TCP           21h     app=alertmanager
custom-metrics-apiserver   ClusterIP   10.96.103.132    <none>        443/TCP          24h     app=custom-metrics-apiserver
grafana                    NodePort    10.101.121.39    <none>        3000:32220/TCP   12h     app=grafana,component=core
kube-state-metrics         ClusterIP   10.100.241.139   <none>        8080/TCP         3d16h   app=kube-state-metrics
prometheus                 NodePort    10.99.13.9       <none>        9090:30090/TCP   20h     app=prometheus,component=server
prometheus-node-exporter   ClusterIP   None             <none>        9100/TCP         24h     app=prometheus,component=node-exporter
[root@k8s-master-01 ~]# kubectl get configmap -n prom  -o wide
NAME                  DATA   AGE
adapter-config        1      24h
alertmanager-config   1      21h
prometheus-config     1      20h
prometheus-rules      2      21h

<-------------------------------------------------------------------------->

4、HPA控制器

目前HPA分为两个版本v1、v2；

[root@k8s-master-01 HPA]# kubectl api-versions  | grep autoscal
autoscaling/v1
autoscaling/v2beta1
autoscaling/v2beta2

4.1 HPA v1控制器

HPAv1默认使用Heapster进行数据采集，只能根据CPU指标进行缩放;

4.1.1 定义具有资源限制参数的depolyment

[root@k8s-master-01 HPA]# cat hpa-v1-deployment_web.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: java-demo
  template:
    metadata:
      labels:
        app: java-demo
    spec:
      containers:
      - image: lizhenliang/java-demo
        name: java
        resources: 
           requests:
             memory: "300Mi"
             cpu: "250m"
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: NodePort
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
    nodePort: 32227
  selector:
    app: java-demo

4.1.2 创建HPAv1

[root@k8s-master-01 HPA]# cat hpa-v1.yaml 
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  maxReplicas: 5
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  targetCPUUtilizationPercentage: 2
[root@k8s-master-01 HPA]# kubectl get hpa
NAME              REFERENCE                TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
metrics-app-hpa   Deployment/metrics-app   500m/800m   2         10        2          9h
web               Deployment/web           1%/2%       1         5         5          16m
[root@k8s-master-01 HPA]# kubectl get hpa web -o yaml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  ...
  managedFields:
  - apiVersion: autoscaling/v1
    fieldsType: FieldsV1
   ...
  - apiVersion: autoscaling/v1
    fieldsType: FieldsV1
    ...
  name: web
  namespace: default
spec:
  maxReplicas: 5
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  targetCPUUtilizationPercentage: 2
status:
  currentCPUUtilizationPercentage: 1
  currentReplicas: 5
  desiredReplicas: 5
  lastScaleTime: "2020-11-22T23:43:25Z"

4.1.3 压力测试

[root@k8s-master-01 HPA]# yum install httpd-tools
[root@k8s-master-01 HPA]# ab -n 80000 -c 800  http://localhost:32227/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/


Benchmarking localhost (be patient)
Completed 8000 requests
Completed 16000 requests
Completed 24000 requests
Completed 32000 requests
Completed 40000 requests
Completed 48000 requests
Completed 56000 requests
Completed 64000 requests
Completed 72000 requests
Completed 80000 requests
Finished 80000 requests
[root@k8s-master-01 HPA]# kubectl get pods
NAME                            READY   STATUS        RESTARTS   AGE
web-cc846d4fb-dx5bc             0/1     Pending       0          69s
web-cc846d4fb-f69fq             1/1     Running       0          22m
web-cc846d4fb-q9629             0/1     Pending       0          53s
web-cc846d4fb-qn7xr             1/1     Running       0          69s
web-cc846d4fb-qwst2             0/1     Pending       0          69s
[root@k8s-master-01 HPA]# kubectl get hpa
NAME              REFERENCE                TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
web               Deployment/web           17%/2%      1         5         5          28m
[root@k8s-master-01 HPA]# kubectl describe hpa web
...
Events:
  Type     Reason                        Age                 From                       Message
  ----     ------                        ----                ----                       -------
  Normal   SuccessfulRescale             3m15s               horizontal-pod-autoscaler  New size: 1; reason: All metrics below target
  Normal   SuccessfulRescale             119s (x2 over 12m)  horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  Normal   SuccessfulRescale             103s (x2 over 11m)  horizontal-pod-autoscaler  New size: 5; reason: cpu resource utilization (percentage of request) above target

4.2 HPA v2控制器

HPAv2从metrics-server中请求资源指标，从k8s-promehteus-adapter一类自定义API中获取自定义指标数据，从而可以基于核心指标CPU和内存资源以及任意自定义指标占用情况进行弹性伸缩；

4.2.1 定义Deployment Web服务器

#metrics-app可以输出特定的资源指标供HPA监控
[root@k8s-master-01 HPA]# cat hpa-v2-deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: metrics-app
  name: metrics-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: metrics-app
  template:
    metadata:
      labels:
        app: metrics-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "80"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - image: ikubernetes/metrics-app
        name: metrics-app
        ports:
        - name: web
          containerPort: 80
        resources:
          requests:
            cpu: 200m
            memory: 256Mi
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: metrics-app
  labels:
    app: metrics-app
spec:
  ports:
  - name: web
    port: 80
    targetPort: 80
  selector:
    app: metrics-app
[root@k8s-master-01 HPA]# kubectl apply -f hpa-v2-deployment.yaml 
deployment.apps/metrics-app created
service/metrics-app created
[root@k8s-master-01 HPA]# kubectl get svc 
NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
kubernetes           ClusterIP   10.96.0.1        <none>        443/TCP          2d6h
metrics-app          ClusterIP   10.106.192.12    <none>        80/TCP           20s
myapp                ClusterIP   10.109.50.132    <none>        80/TCP           49m
myapp-svc-nodeport   NodePort    10.101.208.103   <none>        80:32223/TCP     25m
podinfo              NodePort    10.109.61.187    <none>        9898:31198/TCP   155m
#测试metrics-app输出资源指标的情况
[root@k8s-master-01 HPA]# curl 10.106.192.12/metrics
# HELP http_requests_total The amount of requests in total
# TYPE http_requests_total counter
http_requests_total 11
# HELP http_requests_per_second The amount of requests per second the latest ten seconds
# TYPE http_requests_per_second gauge
http_requests_per_second 0.6

4.2.2 使用HPA实时监控Deployment控制器

[root@k8s-master-01 HPA]# cat hpa-v2.yaml 
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: metrics-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: metrics-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      #HPA监控metics-app的http_requests_per_second指标
      metricName: http_requests_per_second
      targetAverageValue: 800m 
      #800m即0.8个/秒

[root@k8s-master-01 HPA]# kubectl get hpa metrics-app-hpa -o yaml
  - apiVersion: autoscaling/v2beta1
    fieldsType: FieldsV1
    fieldsV1:
  ...
spec:
  maxReplicas: 10
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: metrics-app
status:
  currentReplicas: 2
  desiredReplicas: 2
  lastScaleTime: "2020-11-22T14:57:31Z"

4.2.3 进行压力测试HPA的弹性缩放效果

[root@k8s-master-01 HPA]# kubectl run client -it --image=cirros --rm -- /bin/sh
/ # while true; do curl http://metrics-app; let i++; sleep 0.$RANDOM; done
[root@k8s-master-01 HPA]# kubectl get hpa
NAME              REFERENCE                 TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
metrics-app-hpa   Deployment/metrics-app    675m/800m       2         10        7          2m10s
[root@k8s-master-01 HPA]# kubectl describe hpa metrics-app-hpa
Name:                                  metrics-app-hpa
Namespace:                             default
Labels:                                <none>
Annotations:                           <none>
CreationTimestamp:                     Sun, 22 Nov 2020 22:45:25 +0800
Reference:                             Deployment/metrics-app
Metrics:                               ( current / target )
  "http_requests_per_second" on pods:  500m / 800m
Min replicas:                          2
Max replicas:                          10
Deployment pods:                       7 current / 7 desired
Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from pods metric http_requests_per_second
  ScalingLimited  False   DesiredWithinRange   the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  65s   horizontal-pod-autoscaler  New size: 4; reason: pods metric http_requests_per_second above target
  Normal  SuccessfulRescale  50s   horizontal-pod-autoscaler  New size: 6; reason: pods metric http_requests_per_second above target
  Normal  SuccessfulRescale  35s   horizontal-pod-autoscaler  New size: 7; reason: pods metric http_requests_per_second above target
#通过查看HPA的详情可以观察到HPA的弹性缩放效果已经生效；

数据库

文章转载自运维扫盲人，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

kubernetes配置之十四：kubernetes资源指标及HPA控制器

评论