基于 K8S 线上 1.27.4 环境部署 Gitlab

运维狗工作日记 2023-11-06

ZHDYA(七哥)，8年互联网一线K8S从业者，专注分享K8S生态为主的实战经验，助力大家更高效、可靠地构建和管理容器!

最近更新：K8S架构、公有云、持久化存储、HELM、CICD、负载均衡、监控告警、可观察性、服务治理、架构探索等文章。

一、系统环境二、部署服务2.1、部署Redis2.2、部署Postgresql2.3、部署GitLab三、访问Gitlab四、故障解决

一、系统环境

Gitlab版本：16.1.0
Redis版本：6.2
Postgresql版本：14.0
Kubernetes版本：1.27.4
Helm 安装方式：https://docs.gitlab.com/charts/installation/deployment.html

二、部署服务

2.1、部署Redis

# cat redis.yaml
kind: Service
apiVersion: v1
metadata:
  name: redis
  namespace: devops
  labels:
    name: redis
spec:
  type: ClusterIP
  ports:
    - name: redis
      protocol: TCP
      port: 6379
      targetPort: redis
  selector:
    name: redis
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-data
  namespace: devops
  labels:
    app: redis
spec:
  storageClassName: nfs-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: redis
  namespace: devops
  labels:
    name: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      name: redis
  template:
    metadata:
      name: redis
      labels:
        name: redis
    spec:
      containers:
      - name: redis
        image: 'redis:6.2'
        ports:
        - name: redis
          containerPort: 6379
          protocol: TCP
        volumeMounts:
          - name: redis-persistent-storage
            mountPath: /var/lib/redis
        livenessProbe:
          exec:
            command:
              - redis-cli
              - ping
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          exec:
            command:
              - redis-cli
              - ping
          initialDelaySeconds: 5
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
      volumes:
      - name: redis-persistent-storage
        persistentVolumeClaim:
          claimName: redis-data

2.2、部署Postgresql

# cat postgresql.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgresql-data
  namespace: devops
spec:
  storageClassName: nfs-storage
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
kind: Service
apiVersion: v1
metadata:
  name: postgresql
  namespace: devops
  labels:
    name: postgresql
spec:
  ports:
    - name: postgres
      protocol: TCP
      port: 5432
      targetPort: postgres
  selector:
    name: postgresql
  type: ClusterIP
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: postgresql
  namespace: devops
  labels:
    name: postgresql
spec:
  replicas: 1
  selector:
    matchLabels:
      name: postgresql
  template:
    metadata:
      name: postgresql
      labels:
        name: postgresql
    spec:
      containers:
      - name: postgresql
        image: postgres:14.0
        ports:
        - name: postgres
          containerPort: 5432
        env:
        - name: POSTGRES_USER
          value: admin
        - name: POSTGRES_PASSWORD
          value: test-pwd
        - name: POSTGRES_DB
          value: postgresdb
        - name: DB_EXTENSION
          value: 'pg_trgm,btree_gist'
        resources:
          requests:
            cpu: 2
            memory: 2Gi
          limits:
            cpu: 2
            memory: 2Gi
        livenessProbe:
          exec:
            command: ["pg_isready","-h","localhost","-U","postgres"]
          initialDelaySeconds: 30
          timeoutSeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        readinessProbe:
          exec:
            command: ["pg_isready","-h","localhost","-U","postgres"]
          initialDelaySeconds: 5
          timeoutSeconds: 1
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: postgresql-data

变量说明：

参数名称	默认值	描述
DB_USER	-	创建一个数据库用户
DB_PASS	-	指定创建的用户的密码
DB_NAME	-	创建一个数据库并指定库名
DB_EXTENSION	-	指定安装的扩展包

创建gitlab所需库，命令如下：

[root@k8s01 gitlab]# kubectl exec -it -n devops postgresql-86c4647487-7kg7f bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@postgresql-74755f975c-7whcv:/# psql -h localhost -U admin --password -p 5432 postgresdb 
Password: 
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.

postgresdb=# create user gitlab with password '123456';             
CREATE ROLE

postgresdb=# CREATE DATABASE gitlab;     
CREATE DATABASE

postgresdb=# GRANT ALL PRIVILEGES ON DATABASE gitlab TO gitlab;                  
GRANT

postgresdb=# ALTER ROLE gitlab CREATEROLE SUPERUSER;
ALTER ROLE

2.3、部署GitLab

Redis和Postgres部署完成之后就开始部署核心应用GitLab

# cat gitlab.yaml
kind: Service
apiVersion: v1
metadata:
  name: gitlab
  namespace: devops
  labels:
    name: gitlab
spec:
  ports:
    - name: http
      protocol: TCP
      port: 80
    - name: ssh
      protocol: TCP
      port: 22
      targetPort: ssh
  type: ClusterIP
  selector:
    name: gitlab
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: gitlab-data
  namespace: devops
  labels:
    app: gitlab
spec:
  storageClassName: nfs-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: gitlab
  namespace: devops
  labels:
    name: gitlab
spec:
  replicas: 1
  selector:
    matchLabels:
      name: gitlab
  template:
    metadata:
      name: gitlab
      labels:
        name: gitlab
    spec:
      containers:
        - name: gitlab
          image: sameersbn/gitlab:16.1.0
          ports:
            - name: ssh
              containerPort: 22
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          env:
            - name: TZ
              value: Asia/Shanghai
            - name: GITLAB_TIMEZONE
              value: Beijing
            - name: GITLAB_SECRETS_DB_KEY_BASE
              value: long-and-random-alpha-numeric-string
            - name: GITLAB_SECRETS_SECRET_KEY_BASE
              value: long-and-random-alpha-numeric-string
            - name: GITLAB_SECRETS_OTP_KEY_BASE
              value: long-and-random-alpha-numeric-string
            - name: GITLAB_ROOT_PASSWORD
              value: S6n5Y7b81wRrJnKv
            - name: GITLAB_ROOT_EMAIL
              value: xxx@163.com
            - name: GITLAB_HOST
              value: 'gitlab.kubernets.cn'
            - name: GITLAB_PORT
              value: '80'
            - name: GITLAB_SSH_PORT
              value: '22'
            - name: GITLAB_NOTIFY_ON_BROKEN_BUILDS
              value: 'true'
            - name: GITLAB_NOTIFY_PUSHER
              value: 'false'
            - name: GITLAB_BACKUP_SCHEDULE
              value: daily
            - name: GITLAB_BACKUP_TIME
              value: 01:00
            - name: DB_TYPE
              value: postgres
            - name: DB_HOST
              value: postgresql.devops.svc
            - name: DB_PORT
              value: '5432'
            - name: DB_USER
              value: gitlab
            - name: DB_PASS
              value: '123456'
            - name: DB_NAME
              value: gitlab
            - name: REDIS_HOST
              value: redis.devops.svc
            - name: REDIS_PORT
              value: '6379'
          livenessProbe:
            httpGet:
              path: /
              port: 80
              scheme: HTTP
            initialDelaySeconds: 300
            timeoutSeconds: 5
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /
              port: 80
              scheme: HTTP
            initialDelaySeconds: 5
            timeoutSeconds: 30
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          volumeMounts:
            - name: gitlab-persistent-storage
              mountPath: /home/git/data
            - name: localtime
              mountPath: /etc/localtime
      volumes:
        - name: gitlab-persistent-storage
          persistentVolumeClaim:
            claimName: gitlab-data
        - name: localtime
          hostPath:
            path: /etc/localtime

参数说明：

参数名称	默认值	描述
GITLAB_TIMEZONE	UTC	指定时区
GITLAB_SECRETS_DB_KEY_BASE	-	用于加密数据库中的CI机密变量以及导入凭据。如果丢失或旋转了此机密，则将无法使用现有的CI机密
GITLAB_SECRETS_SECRET_KEY_BASE	-	用于密码重置链接和其他“标准”身份验证功能。如果丢失或旋转了此机密，电子邮件中的密码重置令牌将重置
GITLAB_SECRETS_OTP_KEY_BASE	-	用于加密数据库中的2FA机密。如果您丢失或旋转了此机密，则您的所有用户都将无法使用 2FA 登录
GITLAB_ROOT_PASSWORD	S6n5Y7b81wRrJnKv	指定 root 用户在首次运行时的密码（注意：GitLab 要求长度至少为8个字符）
GITLAB_ROOT_EMAIL	admin@example.com	指定 root 用户在首次运行时的电子邮件
GITLAB_HOST	localhost	指定 GitLab 服务器的主机名，默认为localhost，修改此参数可用配置Gitlab库中的克隆地址
GITLAB_PORT	80	指定 GitLab 服务器的端口号，修改此参数可用配置 Gitlab 库中的克隆地址的端口号
GITLAB_SSH_PORT	$GITLAB_SSH_LISTEN_PORT	指定 ssh 端口号
GITLAB_NOTIFY_ON_BROKEN_BUILDS	true	启用或禁用通知的电子邮件
GITLAB_NOTIFY_PUSHER	true	将推送程序添加到构建通知电子邮件的收件人列表中
GITLAB_NOTIFY_PUSHER	false	将推送程序添加到构建通知电子邮件的收件人列表中
GITLAB_BACKUP_SCHEDULE	daily weekly monthly disable	备份方式
GITLAB_BACKUP_TIME	01:00	备份时间
DB_TYPE	postgres	指定数据库类型
DB_HOST	localhost	指定数据库主机地址（k8s service地址）
DB_PORT	5432	指定数据库服务器端口
DB_USER	root	指定数据库用户名
DB_PASS	-	指定数据库密码
DB_NAME	gitlabhq_production	指定数据库名
REDIS_HOST	localhost	指定 Redis 的主机地址
REDIS_PORT	6379	指定 Redis 端口

三、访问Gitlab

上面已经成功配置了Gitlab，可以通过Ingress配置域名访问；

# cat gitlab-ing.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gitlab-ing
  namespace: devops
spec:
  ingressClassName: nginx
  rules:
  - host: gitlab.kubernets.cn
    http:
      paths:
      - backend:
          service:
            name: gitlab
            port:
              number: 80
        path: /
        pathType: Prefix

默认的管理员用户root，密码在部署Gitlab的 yaml 文件的环境变量中进行了定义；

四、故障解决

1）报错

error: RPC failed; result=22, HTTP code = 413
fatal: The remote end hung up unexpectedly

通过 ingress-nginx 配置 ing 的上传文件大小限制参数 client_max_body_size

# cat nginx-ingress-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.8.2
    helm.sh/chart: ingress-nginx-4.7.2
data:
  proxy-body-size: "500m"