kubernetes集群快速安装部署

大侠之运维 2022-01-06

294

主要内容：
1.前置环境准备
2.k8s环境搭建
3.mertics-server部署
4.dashboard部署
5.k8s命令补全

前言

2022年开始，每周一个关于kubernetes技术分享，也记录下关于k8s的一个学习或者实践历程。

2022年的第一周是进入新公司的第三周，在熟悉新环境的同时，利用一些空余时间，做一些简单的实践，本周的话，就从环境的安装开始吧。

其实现在安装一套k8s环境已经很简单了，没有之前几年开始兴起的时候，各种镜像源无法获取的问题，本文的话，是利用虚拟机环境，采用kubeadm方式安装的1.23.0版本。

虚拟机环境是已经准备好的三台2c4G的centos7.9，有关虚拟机的问题，可以直接在后台问，看到会第一时间回复的。

安装环境

ip	操作系统	配置
192.168.25.128	centos7.9	2C4G
192.168.25.129	centos7.9	2C4G
192.168.25.130	centos7.9	2C4G

安装版本

kubernetes版本	1.23.0
docker版本	20.10.12

前置工作

这部分主要涉及一些正式安装的之前的一些前置化操作，尤其是虚拟机环境，很多的基础命令没有，大致分为如下几步：

下面几步在三台主机上都要操作

systemctl stop firewalld && systemctl disable firewalld
setenforce 0 && sed -i "s/SELINUX=enforcing$/SELINUX=disabled/g" etc/selinux/config
swapoff -a && sed -i "s/\/dev\/mapper\/centos-swap/\#\/dev\/mapper\/centos-swap/g" etc/fstab

yum 源替换安装基础包

curl -o etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
yum clean all
yum makecache
yum install -y yum-utils device-mapper-persistent-data lvm2 vim chrony
#同步各个节点时间
systemctl start chronyd.service && systemctl enable chronyd.service

新增docker源，安装docker环境





yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce
systemctl start docker && systemctl enable docker
#安装完docker后需要调整镜像源、修改cgroup驱动为systemd
tee etc/docker/daemon.json <<-'EOF'
{
 "registry-mirrors": ["https://pcy9sknd.mirror.aliyuncs.com"],
 "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
systemctl daemon-reload && systemctl restart docker

新增k8s安装包源，安装kube系列包

tee etc/yum.repos.d/kubernetes.repo <<-'EOF'
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubelet kubeadm kubectl --nogpgcheck
systemctl start kubelet && systemctl enable kubelet

导入IPVS模块

cat > etc/sysconfig/modules/ipvs.modules <<EOF
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 etc/sysconfig/modules/ipvs.modules && bash

修改brige规则

cat > etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF
sysctl --system

开启iptables的FORWARD转发链

iptables -P FORWARD ACCEPT
sed -i '/ExecStart/a ExecStartPost=/sbin/iptables -P FORWARD ACCEPT' usr/lib/systemd/system/docker.service
修改/etc/hosts文件
tee etc/hosts  <<-'EOF'
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.25.128 k8s-master
192.168.25.129 k8s-node1
192.168.25.130 k8s-node2
EOF

#master节点添加解析，后面会用到
18.199.108.133 raw.githubusercontent.com

搭建kubernetes集群

master初始化

cat > master-init.yaml << EOF
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
 advertiseAddress: 192.168.25.128
 bindPort: 6443
---
apiVersion: kubeadm.k8s.io/v1beta2
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.23.0
networking:
 dnsDomain: cluster.local
 podSubnet: 10.244.0.0/16
EOF

然后执行初始化命令

kubeadmin init --config==master-init.yaml

关于podSubnet这个配置采用不同的网络插件会有不同，flannel的话，默认采用10.244.0.0/16,采用calico的话默认192.168.0.0/16

查看输出结果如下，说明初始化成功

Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:




  mkdir -p $HOME/.kube
  sudo cp -i etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config




Alternatively, if you are the root user, you can run:




  export KUBECONFIG=/etc/kubernetes/admin.conf




You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/




Then you can join any number of worker nodes by running the following on each as root:




kubeadm join 192.168.25.128:6443 --token e6zejn.ofv83pndgkvq2xq9 \
        --discovery-token-ca-cert-hash sha256:e771115aab28654d85c2a73d08e6007587f05b7f945b27aa2aa6243984459f0d

然后根据提示进行操作

  mkdir -p $HOME/.kube
  sudo cp -i etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

这部分的的话是用来加入集群使用的

kubeadm join 192.168.25.128:6443 --token e6zejn.ofv83pndgkvq2xq9 \
        --discovery-token-ca-cert-hash sha256:e771115aab28654d85c2a73d08e6007587f05b7f945b27aa2aa6243984459f0d

网络插件初始化

搭建的集群是用的的flannel，用到的yaml文件如下

可以通过如下链接获取

https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: psp.flannel.unprivileged
  annotations:
    seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
    seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
    apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
    apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
  privileged: false
  volumes:
  - configMap
  - secret
  - emptyDir
  - hostPath
  allowedHostPaths:
  - pathPrefix: "/etc/cni/net.d"
  - pathPrefix: "/etc/kube-flannel"
  - pathPrefix: "/run/flannel"
  readOnlyRootFilesystem: false
  # Users and groups
  runAsUser:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  # Privilege Escalation
  allowPrivilegeEscalation: false
  defaultAllowPrivilegeEscalation: false
  # Capabilities
  allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
  defaultAddCapabilities: []
  requiredDropCapabilities: []
  # Host namespaces
  hostPID: false
  hostIPC: false
  hostNetwork: true
  hostPorts:
  - min: 0
    max: 65535
  # SELinux
  seLinux:
    # SELinux is unused in CaaSP
    rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
rules:
- apiGroups: ['extensions']
  resources: ['podsecuritypolicies']
  verbs: ['use']
  resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: flannel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: flannel
subjects:
- kind: ServiceAccount
  name: flannel
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: flannel
  namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "flannel",
          "delegate": {
            "hairpinMode": true,
            "isDefaultGateway": true
          }
        },
        {
          "type": "portmap",
          "capabilities": {
            "portMappings": true
          }
        }
      ]
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds
  namespace: kube-system
  labels:
    tier: node
    app: flannel
spec:
  selector:
    matchLabels:
      app: flannel
  template:
    metadata:
      labels:
        tier: node
        app: flannel
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
      hostNetwork: true
      priorityClassName: system-node-critical
      tolerations:
      - operator: Exists
        effect: NoSchedule
      serviceAccountName: flannel
      initContainers:
      - name: install-cni-plugin
        image: rancher/mirrored-flannelcni-flannel-cni-plugin:v1.0.0
        command:
        - cp
        args:
        - -f
        - flannel
        - opt/cni/bin/flannel
        volumeMounts:
        - name: cni-plugin
          mountPath: opt/cni/bin
      - name: install-cni
        image: quay.io/coreos/flannel:v0.15.1
        command:
        - cp
        args:
        - -f
        - etc/kube-flannel/cni-conf.json
        - etc/cni/net.d/10-flannel.conflist
        volumeMounts:
        - name: cni
          mountPath: etc/cni/net.d
        - name: flannel-cfg
          mountPath: etc/kube-flannel/
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.15.1
        command:
        - opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        resources:
          requests:
            cpu: "100m"
            memory: "50Mi"
          limits:
            cpu: "100m"
            memory: "50Mi"
        securityContext:
          privileged: false
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: run
          mountPath: run/flannel
        - name: flannel-cfg
          mountPath: etc/kube-flannel/
      volumes:
      - name: run
        hostPath:
          path: run/flannel
      - name: cni-plugin
        hostPath:
          path: opt/cni/bin
      - name: cni
        hostPath:
          path: etc/cni/net.d
      - name: flannel-cfg
        configMap:
          name: kube-flannel-cfg

通过 kubectl apply -f kube-flannel.yaml 安装

安装完后pod及集群状态

kubectl get pods -n kube-system
NAME                                            READY   STATUS    RESTARTS   AGE
coredns-65c54cc984-rwzvn                        1/1     Running   0          6m3s
coredns-65c54cc984-shkx2                        1/1     Running   0          6m3s
etcd-localhost.localdomain                      1/1     Running   0          6m18s
kube-apiserver-localhost.localdomain            1/1     Running   0          6m18s
kube-controller-manager-localhost.localdomain   1/1     Running   0          6m17s
kube-flannel-ds-2br8l                           1/1     Running   0          87s
kube-proxy-6c4w8                                1/1     Running   0          6m3s
kube-scheduler-localhost.localdomain            1/1     Running   0          6m18s
kubectl get nodes
NAME                    STATUS   ROLES                  AGE    VERSION
localhost.localdomain   Ready    control-plane,master   7m2s   v1.23.1

node节点加入

根据master的提示进行node接入

kubeadm join 192.168.25.128:6443 --token e6zejn.ofv83pndgkvq2xq9 \
        --discovery-token-ca-cert-hash sha256:e771115aab28654d85c2a73d08e6007587f05b7f945b27aa2aa6243984459f0d

查看状态

基础资源监控metrics-server

可以查看nodes、pod资源使用情况，后续的HPA的实现也依赖

mkdir metrics-server
cd metrics-server
curl -OL  https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml

修改部分配置

image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6
        imagePullPolicy: IfNotPresent
        args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - --kubelet-insecure-tls
          - --kubelet-preferred-address-types=InternalIP

执行 kubectl apply -f coments.yaml

查看资源情况

简单dashboard搭建配置

curl -OL https://raw.githubusercontent.com/kubernetes/dashboard/v2.4.0/aio/deploy/recommended.yaml

修改暴露服务方式为nodeport

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  selector:
    k8s-app: kubernetes-dashboard


下面为建立相关用户信息
[root@k8s-master dashboard]# cat dashboard-adminuser.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
[root@k8s-master dashboard]# cat dashboard-ClusterRoleBinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard

分别创建，打开https://192.168.25.128:30001/ 页面如下

通过如下命令获取密钥

kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"

K8S命令补全

yum -y install bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo “source <(kubectl completion bash)” >> ~/.bashrc

本文暂时到这里，更多关于k8s资料，可以直接在后台回复k8s获取

后面会讲关于Prometheus及alertmanager内容

文章转载自大侠之运维，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。