ZHDYA(七哥)，9年互联网一线K8S从业者，专注分享K8S生态为主的实战经验，助力大家更高效、可靠地构建和管理容器!
最近更新：K8S架构、公有云、持久化存储、HELM、CICD、负载均衡、监控告警、可观察性、服务治理、架构探索等文章。
1. 什么是 Velero?
Velero 是一个 Vmware 大厂用 Go 语言开发的开源工具，用于备份和恢复 Kubernetes 集群中的资源和持久化数据。
它支持在公有云平台或本地运行，提供备份集群数据、集群资源迁移到其他集群、以及将生产集群复制到开发和测试集群的能力。
2. Velero 与 ETCD 快照备份的区别
备份粒度：ETCD 快照是全局备份，而 Velero 可以对 Kubernetes 集群内对象级别进行备份，支持按 Type、Namespace、Label 等对象进行分类备份或恢复。 恢复影响：ETCD 快照恢复会影响其他 namespace 中的 pod 运行服务，而 Velero 可以针对性地恢复，不影响其他服务。 存储位置：Velero 支持多种对象存储，如 MiniO、Ceph、OSS、S3等存储，而 ETCD 快照是一个本地文件。 备份方式：Velero 支持增量备份，只备份变化的数据，而 ETCD 快照通常是全量备份。
3. Velero 整体架构
Velero 由客户端和服务端两部分组成。客户端是本地运行的命令行工具，服务端运行在 Kubernetes 集群中，负责执行具体的备份和恢复操作。
4. Velero 备份工作流程
本地 Velero 客户端发送备份指令。 Kubernetes 集群内创建一个 Backup 对象。 BackupController 监测 Backup 对象并开始备份过程。 BackupController 向 API Server 查询相关数据。 BackupController 将查询到的数据备份到远端的对象存储（MiniO）。
5. 对象存储MiniO
当前 K8S 集群中，已经有完整 MiniO 服务，部署步骤略；
[root@master1 root]# kgp -A|grep minio
minio-operator console-64fff96878-vtbrm 1/1 Running 0 315d
minio-operator minio-operator-774875b47c-5gg58 1/1 Running 0 315d
minio-operator minio-operator-774875b47c-9hw72 1/1 Running 0 125d
minio-tenant-1 minio-1-ss-0-0 2/2 Running 0 309d
minio-tenant-1 minio-1-ss-0-1 2/2 Running 0 125d
minio-tenant-1 minio-1-ss-0-2 2/2 Running 0 140d
在 MiniO 集群中创建 S3桶
# mc mb minio/velero-k8s
6. 部署Velero
安装 Velero 客户端（最新版 1.15）
# wget https://mirror.ghproxy.com/https://github.com/vmware-tanzu/velero/releases/download/v1.14.0/velero-v1.14.0-linux-amd64.tar.gz
# tar -zxvf velero-v1.14.0-linux-amd64.tar.gz
# cp velero-v1.14.0-linux-amd64/velero usr/local/bin/
# velero version
创建 Velero 使用的 MiniO 密钥
[root@master1 root]# cat >>credentials-velero <EOF
[default]
aws_access_key_id = admin
aws_secret_access_key = XXXXXXX
EOF
创建 Velero namespace
# kubectl create ns velero
在K8S中安装 Velero 服务端
velero install \
# 指定存储提供商，minio用的是s3，所以这里是aws
--provider aws \
# 指定aws插件
--plugins velero/velero-plugin-for-aws:v1.10.1 \
# 指定velero镜像
--image velero/velero:v1.14.1 \
# 限制agent CPU和内存资源
--node-agent-pod-cpu-limit 2 \
--node-agent-pod-mem-limit 2048Mi \
# 指定命名空间
--namespace velero \
# 指定bucket名称
--bucket velero-k8s \
# 默认以文件形式备份持久化卷
--default-volumes-to-fs-backup \
# 启用node agent 进行备份和恢复操作，--use-restic已启用，从v1.10开始默认使用kopia备份
--use-node-agent \
# 指定S3存储桶凭证
--secret-file ./credentials-velero \
# 禁用卷快照功能，只使用文件系统级别的备份
--use-volume-snapshots=false \
# 配置备份存储位置的详细信息，其中<minio-api-url>填写minio的api地址，如192.168.10.25:9000
--backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://<minio-api-url>
7. 无状态备份
#【1】在demo/demoapp 创建app
[root@master1 root]# kubectl create ns demo
namespace/demo created
[root@master1 root]# kubectl create deployment demoapp --image=ikubernetes/demoapp:v1.0 --replicas=3 -n demo
deployment.apps/demoapp created
[root@master1 root]# kubectl get pods -n demo
NAME READY STATUS RESTARTS AGE
demoapp-7c58cd6bb-cs2fc 1/1 Running 0 49s
demoapp-7c58cd6bb-r4rcb 1/1 Running 0 49s
demoapp-7c58cd6bb-w7ckj 1/1 Running 0 49s
#【2】创建一个备份 - 只备份 demo名称空间
[root@master1 root]# velero backup create demo --include-namespaces demo
Backup request "demo" submitted successfully.
Run `velero backup describe demo` or `velero backup logs demo` for more details.
[root@master1 root]# velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
demo Completed 0 0 2024-11-21 06:46:30 +0000 UTC 29d default <none>
#【3】模拟误删除demo名称空间 进行恢复操作
[root@master1 root]# kubectl delete ns demo
[root@master1 root]# kubectl get ns #确定无demo名称空间
NAME STATUS AGE
cilium-secrets Active 20d
default Active 21d
kube-node-lease Active 21d
kube-public Active 21d
kube-system Active 21d
metallb-system Active 20d
velero Active 40m
#进行恢复
[root@master1 root]# velero restore create --from-backup demo
Restore request "demo-20240321065255" submitted successfully.
Run `velero restore describe demo-20240321065255` or `velero restore logs demo-20240321065255` for more details.
[root@master1 root]# velero restore get
NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR
demo-20240321065255 demo Completed 2024-11-21 06:52:55 +0000 UTC 2024-11-21 06:52:55 +0000 UTC 0 2 2024-11-21 06:52:55 +0000 UTC <none>
[root@master1 root]# kubectl get ns #已恢复demo名称空间
NAME STATUS AGE
cilium-secrets Active 20d
default Active 21d
demo Active 23s
kube-node-lease Active 21d
kube-public Active 21d
kube-system Active 21d
metallb-system Active 20d
velero Active 42m
[root@master1 root]# kubectl get pods -n demo #pod重新拉起
NAME READY STATUS RESTARTS AGE
demoapp-7c58cd6bb-cs2fc 1/1 Running 0 36s
demoapp-7c58cd6bb-r4rcb 1/1 Running 0 36s
demoapp-7c58cd6bb-w7ckj 1/1 Running 0 36s
8. 有状态备份
背景：创建一个有状态redis应用，底层数据持久化使用NFS，验证数据的备份恢复
#【1】验证集群storageclass
[root@master1 root]# kg sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-csi (default) fuseim.pri/ifs Delete Immediate false 466d
#【2】创建demo/redis 使用nfs-pvc持久化
[root@master1 root]# cat redis-with-nfs-pvc.yaml
apiVersion: v1
kind: Pod
metadata:
name: redis-with-nfs-pvc
spec:
containers:
- name: redis
image: redis:7-alpine
ports:
- containerPort: 6379
name: redis
volumeMounts:
- mountPath: data
name: data-storage
volumes:
- name: data-storage
persistentVolumeClaim:
claimName: nfs-csi
[root@master1 root]# kubectl apply -f redis-with-nfs-pvc.yaml -n demo
pod/redis-with-nfs-pvc created
[root@master1 root]# kubectl get pods -n demo
NAME READY STATUS RESTARTS AGE
redis-with-nfs-pvc 1/1 Running 0 51s
#【3】测试写数据
[root@master1 root]# kubectl exec -it redis-with-nfs-pvc -n demo -- bin/sh
/data # redis-cli
127.0.0.1:6379> set mykey "BIRKHOFF 2024-11-21"
OK
127.0.0.1:6379> BGSAVE
Background saving started
127.0.0.1:6379> exit
/data # ls
dump.rdb
#【4】备份数据 - redis-backup
[root@master1 root]# velero backup create redis-backup --include-namespaces demo --default-volumes-to-fs-backup
Backup request "redis-backup" submitted successfully.
Run `velero backup describe redis-backup` or `velero backup logs redis-backup` for more details.
[root@master1 root]# velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
demo Completed 0 0 2024-11-21 06:46:30 +0000 UTC 29d default <none>
redis-backup Completed 0 0 2024-11-21 08:01:48 +0000 UTC 29d default <none>
[root@master1 root]# velero backup describe redis-backup --details
Name: redis-backup
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/resource-timeout=10m0s
velero.io/source-cluster-k8s-gitversion=v1.29.2
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=29
Phase: Completed
Namespaces:
Included: demo
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: <none>
Or label selector: <none>
Storage Location: default
Velero-Native Snapshot PVs: auto
Snapshot Move Data: false
Data Mover: velero
TTL: 720h0m0s
CSISnapshotTimeout: 10m0s
ItemOperationTimeout: 4h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2024-11-21 08:01:48 +0000 UTC
Completed: 2024-11-21 08:01:52 +0000 UTC
Expiration: 2024-12-20 08:01:48 +0000 UTC
Total items to be backed up: 23
Items backed up: 23
Resource List:
apiextensions.k8s.io/v1/CustomResourceDefinition:
- ciliumendpoints.cilium.io
apps/v1/Deployment:
- demo/demoapp
apps/v1/ReplicaSet:
- demo/demoapp-7c58cd6bb
cilium.io/v2/CiliumEndpoint:
- demo/demoapp-7c58cd6bb-cs2fc
- demo/demoapp-7c58cd6bb-r4rcb
- demo/demoapp-7c58cd6bb-w7ckj
- demo/redis-with-nfs-pvc
v1/ConfigMap:
- demo/kube-root-ca.crt
v1/Event:
- demo/nfs-pvc.17beb8890487bc71
- demo/nfs-pvc.17beb88905411edd
- demo/nfs-pvc.17beb8890aa62b1b
- demo/redis-with-nfs-pvc.17beb88c587fdc81
- demo/redis-with-nfs-pvc.17beb88c78733a92
- demo/redis-with-nfs-pvc.17beb88c78e45daf
- demo/redis-with-nfs-pvc.17beb88c7abb19b1
v1/Namespace:
- demo
v1/PersistentVolume:
- pvc-6a17223d-6628-4eb4-877c-954e9b0eef09
v1/PersistentVolumeClaim:
- demo/nfs-pvc
v1/Pod:
- demo/demoapp-7c58cd6bb-cs2fc
- demo/demoapp-7c58cd6bb-r4rcb
- demo/demoapp-7c58cd6bb-w7ckj
- demo/redis-with-nfs-pvc
v1/ServiceAccount:
- demo/default
Backup Volumes:
Velero-Native Snapshots: <none included>
CSI Snapshots: <none included>
Pod Volume Backups - kopia:
Completed:
demo/redis-with-nfs-pvc: data-storage
HooksAttempted: 0
HooksFailed: 0
#【5】模拟故障 删除ns ：demo
[root@master1 root]# kubectl delete ns demo
namespace "demo" deleted
[root@master1 root]# kubectl get pv
No resources found
[root@master1 root]# velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
demo Completed 0 0 2024-11-21 06:46:30 +0000 UTC 29d default <none>
redis-backup Completed 0 0 2024-11-21 08:01:48 +0000 UTC 29d default <none>
[root@master1 root]# velero restore create --from-backup redis-backup
Restore request "redis-backup-20240321080651" submitted successfully.
Run `velero restore describe redis-backup-20240321080651` or `velero restore logs redis-backup-20240321080651` for more details.
[root@master1 /root]# kubectl get pvc
No resources found in default namespace.
root@ubuntu-k8s-master01:~# kubectl get pvc -n demo
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
nfs-pvc Bound pvc-5628dd5b-5360-4e33-93b6-8ce42e2da07e 3Gi RWX nfs-csi <unset> 9s
[root@master1 /root]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
pvc-5628dd5b-5360-4e33-93b6-8ce42e2da07e 3Gi RWX Delete Bound demo/nfs-pvc nfs-csi <unset> 24s
[root@master1 /root]# kubectl get pods -n demo
NAME READY STATUS RESTARTS AGE
redis-with-nfs-pvc 1/1 Running 0 2m13s
#【6】-验证数据恢复情况
#NFS 服务确认
[root@k8s-harbor ~]# cd /data/nfs/
[root@k8s-harbor nfs]# ll
total 0
drwxr-xr-x 3 polkitd root 74 Mar 21 16:08 pvc-5628dd5b-5360-4e33-93b6-8ce42e2da07e
#服务中数据确认
[root@master1 /root]# kubectl exec -it redis-with-nfs-pvc -n demo -- /bin/sh
Defaulted container "redis" out of: redis, restore-wait (init)
/data # ls
BGSAVE dump.rdb exit set
/data # redis-cli
127.0.0.1:6379> get mykey
"BIRKHOFF 2024-11-21"
10. 总结
Velero 是一个强大的 Kubernetes 集群备份和恢复工具，它提供了细粒度的备份选项，支持多种存储解决方案，并允许跨集群迁移。与 ETCD 快照备份相比，Velero 提供了更多的灵活性和选择性，也让他成为 Kubernetes 集群数据保护的理想选择。
如上述步骤，大家可以轻松地安装和使用 Velero 来保护的 Kubernetes 集群数据，赶紧搜藏验证起来！！
● 🔥【实战】K8S云原生企业实战训练营课程大纲(直播+录播)
● 🌟【Free】负载均衡Ingress-nginx最佳实践
● 😉【加群】扫码，备注：进群
关注七哥，带你领略不一样的K8S！
文章转载自运维狗工作日记，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。
