prometheus-operator + thanos + S3 做prometheus 持久化存储

我的监控架构图

thanos 核心组件(目前用到的)

Sidecar 的作用

收集Prometheus信息上传到S3 并且供给 Store Gateway 查询

Store Gateway 的作用

当Query 请求Prometheus 之前的数据的时候会从S3 上拉取 让Query 查询

Query 的作用

整合的查询接口 提供给Grafana 出图

AWS 配置

创建一个用户 配置对应的IAM权限，创建一个ack,官方给的最小权限，我试过了权限不够，所以给了"s3:*",这个权限是没问题的

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<bucket-name>/*",
                "arn:aws:s3:::<bucket-name>"
            ]
        }
    ]
}

部署的yaml 文件

因为我用的 Prometheus-Operator 会修改一下 Prometheus-promethtus.yaml 的配置
保证thanos 跟Prometheus在一个ns

prometheus-prometheus.yaml

更新一下 添加sidecar，注意添加的都要加上

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  baseImage: quay.io/prometheus/prometheus
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  replicas: 1
  resources:
    requests:
      cpu: 1
      memory: 2Gi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  enableAdminAPI: true   ### 添加的开启admin api 权限
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.11.0
  additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yaml
  thanos:    ### 添加的
    baseImage: quay.io/thanos/thanos   ### 添加的
    version: v0.11.0   ### 添加的
    objectStorageConfig:   ### 添加的
     key: thanos.yaml   ### 添加的
     name: thanos-objstore-config   ### 添加的

thanos-config.yaml

日志存储桶的ack

type: s3
config:
  bucket: xxx  # S3 存储的名字
  endpoint: s3.us-east-2.amazonaws.com # S3 存储的端点，可以去AWS 官网查
  access_key: xxx  # ACKID
  secret_key: xxx  # ACK密码　

addthanosconfig.sh

让thanos-config.yaml生效

#!/bin/bash
kubectl -n monitoring delete secret thanos-objstore-config
kubectl -n monitoring create secret generic thanos-objstore-config --from-file=thanos.yaml=thanos-config.yaml

thanos-query-deployment.yaml

• --store= 这个通过服务发现链接两个svc

apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-query
  namespace: monitoring
  labels:
    app: thanos-query
spec:
  replicas: 1
  selector:
    matchLabels:
      app: thanos-query
  template:
    metadata:
      labels:
        app: thanos-query
    spec:
      containers:
      - name: thanos-query
        image: quay.io/thanos/thanos:v0.11.0
        args:
        - query
        - --log.level=debug
        - --query.replica-label=prometheus_replica
        - --query.replica-label=thanos_ruler_replica
        - --store=dnssrv+thanos-store:10901
        - --store=dnssrv+thanos-store-sidecar:10901
        ports:
        - name: http
          containerPort: 10902
        - name: grpc
          containerPort: 10901

thanos-query-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: thanos-query
  namespace: monitoring
  labels:
    app: thanos-query
spec:
  selector:
    app: thanos-query
  ports:
  - name: http
    port: 9090
    targetPort: http

thanos-query-alb.yaml

创建一个内网alb 给grafana 使用

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: thanos-query-ingress
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/certificate-arn: xxxxxxxxxxxxxx
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    external-dns.alpha.kubernetes.io/hostname: thanos.xxx.com
  labels:
    k8s-app: thanos-query-ingress
spec:
  rules:
    - host: thanos.xxx.com
    - http:
        paths:
          - path: /*
            backend:
              serviceName: thanos-query
              servicePort: 9090

thanos-store-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: thanos-store
  namespace: monitoring
  labels:
    app: thanos-store
spec:
  replicas: 1
  selector:
    matchLabels:
      app: thanos-store
  serviceName: thanos-store
  template:
    metadata:
      labels:
        app: thanos-store
        thanos-store-api: "true"
    spec:
      containers:
        - name: thanos
          image: thanosio/thanos:v0.11.0
          args:
          - "store"
          - "--log.level=debug"
          - "--data-dir=/data"
          - "--objstore.config-file=/etc/secret/thanos.yaml"
          - "--index-cache-size=500MB"
          - "--chunk-pool-size=500MB"
          ports:
          - name: http
            containerPort: 10902
          - name: grpc
            containerPort: 10901
          livenessProbe:
            httpGet:
              port: 10902
              path: /-/healthy
          readinessProbe:
            httpGet:
              port: 10902
              path: /-/ready
          volumeMounts:
            - name: object-storage-config
              mountPath: /etc/secret
              readOnly: false
      volumes:
        - name: object-storage-config
          secret:
            secretName: thanos-objstore-config

thanos-store-service.yaml

暴露store的svc给query服务发现用

apiVersion: v1
kind: Service
metadata:
  name: thanos-store
  namespace: monitoring
spec:
  clusterIP: None
  ports:
  - name: grpc
    port: 10901
    targetPort: grpc
  selector:
    thanos-store-api: "true"

thanos-store-sidecar-service.yaml

暴露sidecar的svc给query服务发现用

apiVersion: v1
kind: Service
metadata:
  name: thanos-store-sidecar
  namespace: monitoring
spec:
  clusterIP: None
  ports:
  - name: grpc
    port: 10901
    targetPort: grpc
  selector:
    app: prometheus

感谢阳神的文档。附加一个链接，全是各种理论的，我是看这个搞出来的
https://www.qikqiak.com/k8strain/monitor/thanos/