0

日志聚合系统Grafana Loki

FlyZhi 2021-03-16
1867

Loki是受Prometheus启发水平可扩展、高度可用的多租户日志聚合系统。

Grafana Loki是一套组件,这些组件可以组合成一个功能齐全的日志堆栈。与其他日志记录系统不同,Loki是基于仅索引有关日志的元数据的想法而构建的:标签(就像Prometheus标签一样)。日志数据本身被压缩然后并存储在对象存储(例如S3或GCS)中的块中,甚至存储在文件系统上的本地。小索引和高度压缩的块简化了操作,并显着降低了Loki的成本。

Grafana Loki 由3部分组成:

  • loki: 日志记录引擎,负责存储日志和处理查询

  • promtail: 代理,负责收集日志并将其发送给 loki 

  • Grafana: UI 

1.安装Loki

    # unzip loki-linux-amd64.zip
    # mv loki-linux-amd64 usr/local/bin/loki
    # loki --version
    loki, version 2.2.0 (branch: HEAD, revision: a27c7991)
    build user: root@b6becd8d0173
    build date: 2021-03-11T14:05:11Z
    go version: go1.15.3
    platform: linux/amd64

    2.配置Loki

      # vim loki-local-config.yaml 
      auth_enabled: false
      # 通过X-Scope-OrgID标头启用身份验证,如果为true,则必须存在。如果为false,则OrgID将始终设置为“ fake”。

      server:
      http_listen_port: 3100
      grpc_listen_port: 9096

      ingester:
      # 配置ingester的生命周期将如何运行以及在何处注册以进行发现。
      wal:
      enabled: true
      # 允许写入WAL
      dir: tmp/wal
      # 存储和/或恢复WAL数据的目录
      lifecycler:
      address: 127.0.0.1
      ring:
      kvstore:
      # 用于ring的后端存储.支持的值:consul, etcd,inmemory, memberlist
      store: inmemory
      replication_factor: 1
      # 要写入和读取的ingesters值
      final_sleep: 0s
      # 退出前的睡眠时间,以确保指标已被删除
      chunk_idle_period: 1h
      # 此时未接收到新日志的任何区块都将被刷新
      max_chunk_age: 1h
      # 内存中时间序列块的最大持续时间(默认值为1h)
      chunk_target_size: 1048576
      # 块的目标_compressed_大小(以字节为单位)
      chunk_retain_period: 30s
      # 刷新后应在内存中保留多长时间
      max_transfer_retries: 0
      # 禁用块传输

      schema_config:
      # 块索引架构的配置
      configs:
      - from: 2020-10-24
      # 应该创建索引存储桶的第一天的日期
      store: boltdb-shipper
      # 用于索引的存储
      object_store: filesystem
      # 哪个存储用于块。如果省略,则默认为与store相同的值。
      schema: v11
      # 要使用的架构版本,当前建议的架构是v11
      index:
      # 配置索引的更新和存储方式
      prefix: index_
      # 所有期间表的表前缀
      period: 24h
      # 表周期

      storage_config:
      # 从给定日期开始配置模式
      boltdb_shipper:
      active_index_directory: tmp/loki/boltdb-shipper-active
      # 接收程序将在其中写入boltdb文件的目录
      cache_location: tmp/loki/boltdb-shipper-cache
      # 用于还原boltDB文件以进行查询的缓存位置
      cache_ttl: 24h
      # 在缓存中还原的用于查询的boltDB文件的TTL
      shared_store: filesystem
      # 用于保存boltdb文件的共享存储。支持的类型:gcs、s3、azure、filesystem
      filesystem:
      # 配置在本地文件系统上存储块。仅当配置中存在文件系统时才需要必需字段
      directory: tmp/loki/chunks
      # 用于存储块的目录

      compactor:
      # 压缩
      working_directory: tmp/loki/boltdb-shipper-compactor
      shared_store: filesystem

      limits_config:
      reject_old_samples: true
      # 旧样品是否将被拒绝
      reject_old_samples_max_age: 168h
      # 拒绝前的最大接受样品时间

      chunk_store_config:
      max_look_back_period: 0s
      # 限制可以查询多长时间的数据

      table_manager:
      retention_deletes_enabled: false
      # Master 'on-switch' 删除表保留
      retention_period: 0s
      # 在删除前表将保留多久(0s禁用删除)
      • Loki配置示例:

      https://raw.githubusercontent.com/grafana/loki/master/cmd/loki/loki-local-config.yaml

      • Loki配置参数:

      https://grafana.com/docs/loki/latest/configuration/

      3.启用Loki

        # vim etc/systemd/system/loki.service
        [Unit]
        Description=Grafana Loki
        Documentation=https://github.com/grafana/loki

        [Service]
        ExecStart=/usr/local/bin/loki \
        -config.file=/etc/loki/loki-local-config.yaml

        [Install]
        WantedBy=multi-user.target

        # systemctl daemon-reload
        # systemctl start loki
        # systemctl status loki

        4.安装Promtail

        Promtail是将本地日志内容发送到Loki实例的代理。通常将其部署到具有需要监视的应用程序的每台计算机上。

        Promtail主要:

        • 发现目标

        • 将标签附加到日志流

        • 将它们推送到Loki实例

        当前,Promtail可以从两个来源跟踪日志:本地日志文件和systemd日志(仅在AMD64计算机上)。

          # unzip promtail-linux-amd64.zip
          # mv promtail-linux-amd64 usr/local/bin/promtail
          # promtail --version
          promtail, version 2.2.0 (branch: HEAD, revision: a27c7991)
          build user: root@b6becd8d0173
          build date: 2021-03-11T14:05:11Z
          go version: go1.15.3
          platform: linux/amd64

          5.配置Promtail

            # vim promtail-local-config.yaml
            server:
            # 为Promtail配置服务器
            http_listen_port: 9080
            # HTTP服务监听端口(0表示随机端口)
            grpc_listen_port: 0
            # gRPC服务侦听端口(0表示随机端口)

            positions:
            # 配置Promtail保存文件的位置,该位置指示Promtail读入文件的程度。重新启动Promtail时需要使用它,以允许它从上次中断的地方继续。
            filename: tmp/positions.yaml

            clients:
            # 配置Promtail如何连接到Loki实例
            - url: http://localhost:3100/loki/api/v1/push

            scrape_configs:
            # 配置Promtail使用指定的发现方法从一系列目标中抓取日志
            - job_name: system
            # 在Promtail UI中标识此抓取配置的名称
            static_configs:
            # 抓取的静态目标
            - targets:
            - localhost
            # 将发现配置为查看当前机器
            labels:
            job: varlogs
            __path__: var/log/*log
            # 定义一个要抓取的文件和一组可选的附加标签,以应用于__path__中的文件定义的所有流
            - job_name: messages
            static_configs:
            - targets:
            - localhost
            labels:
            job: messages-logs
            __path__: var/log/messages
            - job_name: journal
            journal:
            max_age: 12h
            # 从进程开始,读取并发送最近12小时的数据至Loki
            labels:
            job: systemd-journal
            relabel_configs:
            # 描述如何重新标记目标以确定是否应处理它们
            - source_labels: ['__journal__systemd_unit']
            # 源标签从现有标签中选择值。它们的内容使用配置的分隔符连接起来,并与replace、keep和drop操作的配置正则表达式匹配。
            target_label: 'unit'
            # 替换操作中将结果值写入的标签。替换操作是必需的。正则表达式捕获组可用。
            • Promtail配置示例:

            https://raw.githubusercontent.com/grafana/loki/master/cmd/promtail/promtail-local-config.yaml

            • Promtail配置参数:

            https://grafana.com/docs/loki/latest/clients/promtail/configuration/

            6.启用Promtail

              # vim etc/systemd/system/promtail.service
              [Unit]
              Description= Loki Promtail
              Documentation=https://github.com/grafana/loki

              [Service]
              ExecStart=/usr/local/bin/promtail \
              -config.file=/etc/loki/promtail-local-config.yaml

              [Install]
              WantedBy=multi-user.target

              # systemctl daemon-reload
              # systemctl start promtail
              # systemctl status promtail

              7.配置kubernetes下的promtail

              a.创建serviceaccount

                # vim promtail-serviceaccount.yaml 
                apiVersion: v1
                kind: ServiceAccount
                metadata:
                name: promtail
                # kubectl create -f promtail-serviceaccount.yaml
                serviceaccount/promtail created


                b.创建clusterrole

                  # vim promtail-clusterrole.yaml 
                  apiVersion: rbac.authorization.k8s.io/v1
                  kind: ClusterRole
                  metadata:
                  name: promtail
                  rules:
                  - apiGroups: [""]
                  resources:
                  - nodes
                  - services
                  - pods
                  verbs:
                  - get
                  - watch
                  - list
                  # kubectl create -f promtail-clusterrole.yaml
                  clusterrole.rbac.authorization.k8s.io/promtail created


                  c.创建clusterrolebinding

                    # vim promtail-clusterrolebinding.yaml 
                    apiVersion: rbac.authorization.k8s.io/v1
                    kind: ClusterRoleBinding
                    metadata:
                    name: promtail
                    subjects:
                    - kind: ServiceAccount
                    name: promtail
                    namespace: default
                    roleRef:
                    kind: ClusterRole
                    name: promtail
                    apiGroup: rbac.authorization.k8s.io
                    # kubectl create -f promtail-clusterrolebinding.yaml
                    clusterrolebinding.rbac.authorization.k8s.io/promtail created


                    d.创建configmap

                      # vim promtail-configmap.yaml
                      apiVersion: v1
                      kind: ConfigMap
                      metadata:
                      name: promtail-config
                      data:
                      promtail.yaml: |-


                      positions:
                      filename: /tmp/positions.yaml


                      scrape_configs:
                      - job_name: kubernetes-pods-name
                      pipeline_stages:
                      - docker: {}
                      kubernetes_sd_configs:
                      - role: pod
                      relabel_configs:
                      - source_labels:
                      - __meta_kubernetes_pod_label_name
                      target_label: __service__
                      - source_labels:
                      - __meta_kubernetes_pod_node_name
                      target_label: __host__
                      - action: drop
                      regex: ''
                      source_labels:
                      - __service__
                      - action: labelmap
                      regex: __meta_kubernetes_pod_label_(.+)
                      - action: replace
                      replacement: $1
                      separator: /
                      source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                      target_label: job
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_namespace
                      target_label: namespace
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_pod_name
                      target_label: pod
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_pod_container_name
                      target_label: container
                      - replacement: /var/log/pods/*$1/*.log
                      separator: /
                      source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                      target_label: __path__
                      - job_name: kubernetes-pods-app
                      pipeline_stages:
                      - docker: {}
                      kubernetes_sd_configs:
                      - role: pod
                      relabel_configs:
                      - action: drop
                      regex: .+
                      source_labels:
                      - __meta_kubernetes_pod_label_name
                      - source_labels:
                      - __meta_kubernetes_pod_label_app
                      target_label: __service__
                      - source_labels:
                      - __meta_kubernetes_pod_node_name
                      target_label: __host__
                      - action: drop
                      regex: ''
                      source_labels:
                      - __service__
                      - action: labelmap
                      regex: __meta_kubernetes_pod_label_(.+)
                      - action: replace
                      replacement: $1
                      separator: /
                      source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                      target_label: job
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_namespace
                      target_label: namespace
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_pod_name
                      target_label: pod
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_pod_container_name
                      target_label: container
                      - replacement: /var/log/pods/*$1/*.log
                      separator: /
                      source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                      target_label: __path__
                      - job_name: kubernetes-pods-direct-controllers
                      pipeline_stages:
                      - docker: {}
                      kubernetes_sd_configs:
                      - role: pod
                      relabel_configs:
                      - action: drop
                      regex: .+
                      separator: ''
                      source_labels:
                      - __meta_kubernetes_pod_label_name
                      - __meta_kubernetes_pod_label_app
                      - action: drop
                      regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
                      source_labels:
                      - __meta_kubernetes_pod_controller_name
                      - source_labels:
                      - __meta_kubernetes_pod_controller_name
                      target_label: __service__
                      - source_labels:
                      - __meta_kubernetes_pod_node_name
                      target_label: __host__
                      - action: drop
                      regex: ''
                      source_labels:
                      - __service__
                      - action: labelmap
                      regex: __meta_kubernetes_pod_label_(.+)
                      - action: replace
                      replacement: $1
                      separator: /
                      source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                      target_label: job
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_namespace
                      target_label: namespace
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_pod_name
                      target_label: pod
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_pod_container_name
                      target_label: container
                      - replacement: /var/log/pods/*$1/*.log
                      separator: /
                      source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                      target_label: __path__
                      - job_name: kubernetes-pods-indirect-controller
                      pipeline_stages:
                      - docker: {}
                      kubernetes_sd_configs:
                      - role: pod
                      relabel_configs:
                      - action: drop
                      regex: .+
                      separator: ''
                      source_labels:
                      - __meta_kubernetes_pod_label_name
                      - __meta_kubernetes_pod_label_app
                      - action: keep
                      regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
                      source_labels:
                      - __meta_kubernetes_pod_controller_name
                      - action: replace
                      regex: '([0-9a-z-.]+)-[0-9a-f]{8,10}'
                      source_labels:
                      - __meta_kubernetes_pod_controller_name
                      target_label: __service__
                      - source_labels:
                      - __meta_kubernetes_pod_node_name
                      target_label: __host__
                      - action: drop
                      regex: ''
                      source_labels:
                      - __service__
                      - action: labelmap
                      regex: __meta_kubernetes_pod_label_(.+)
                      - action: replace
                      replacement: $1
                      separator: /
                      source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                      target_label: job
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_namespace
                      target_label: namespace
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_pod_name
                      target_label: pod
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_pod_container_name
                      target_label: container
                      - replacement: /var/log/pods/*$1/*.log
                      separator: /
                      source_labels:
                      - __meta_kubernetes_pod_uid
                      - __meta_kubernetes_pod_container_name
                      target_label: __path__
                      - job_name: kubernetes-pods-static
                      pipeline_stages:
                      - docker: {}
                      kubernetes_sd_configs:
                      - role: pod
                      relabel_configs:
                      - action: drop
                      regex: ''
                      source_labels:
                      - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_pod_label_component
                      target_label: __service__
                      - source_labels:
                      - __meta_kubernetes_pod_node_name
                      target_label: __host__
                      - action: drop
                      regex: ''
                      source_labels:
                      - __service__
                      - action: labelmap
                      regex: __meta_kubernetes_pod_label_(.+)
                      - action: replace
                      replacement: $1
                      separator: /
                      source_labels:
                      - __meta_kubernetes_namespace
                      - __service__
                      target_label: job
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_namespace
                      target_label: namespace
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_pod_name
                      target_label: pod
                      - action: replace
                      source_labels:
                      - __meta_kubernetes_pod_container_name
                      target_label: container
                      - replacement: /var/log/pods/*$1/*.log
                      separator: /
                      source_labels:
                      - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
                      - __meta_kubernetes_pod_container_name
                      target_label: __path__
                      # kubectl create -f promtail-configmap.yaml
                      configmap/promtail-config created


                      e.创建daemonset

                        # vim promtail-daemonset.yaml 
                        apiVersion: apps/v1
                        kind: DaemonSet
                        metadata:
                        name: promtail-daemonset
                        spec:
                        selector:
                        matchLabels:
                        name: promtail
                        template:
                        metadata:
                        labels:
                        name: promtail
                        spec:
                        serviceAccount: promtail
                        serviceAccountName: promtail
                        containers:
                        - name: promtail-container
                        image: grafana/promtail:2.2.0
                        args:
                        - -config.file=/etc/promtail/promtail.yaml
                        - -client.url=http://192.168.100.132:3100/loki/api/v1/push
                        - -server.http-listen-port=8090
                        volumeMounts:
                        - name: promtail-config
                        mountPath: /etc/promtail
                        - name: containers
                        mountPath: /var/log/containers
                        readOnly: true
                        - name: docker
                        mountPath: /var/lib/docker/containers
                        readOnly: true
                        - name: pods
                        mountPath: /var/log/pods
                        readOnly: true
                        hostNetwork: true
                        hostPID: true
                        volumes:
                        - name: containers
                        hostPath:
                        path: /var/log/containers
                        - name: docker
                        hostPath:
                        path: /var/lib/docker/containers
                        - name: pods
                        hostPath:
                        path: /var/log/pods
                        - name: promtail-config
                        configMap:
                        name: promtail-config
                        # kubectl create -f promtail-daemonset.yaml
                        daemonset.apps/promtail-daemonset created


                        f.查看pod

                          # kubectl get pod
                          NAME READY STATUS RESTARTS AGE
                          promtail-daemonset-6qg2b 1/1 Running 0 69s
                          promtail-daemonset-krk66 1/1 Running 0 69s
                          promtail-daemonset-qfnhj 1/1 Running 0 69s


                          8.安装Grafana

                            # vim /etc/yum.repos.d/grafana.repo
                            [grafana]
                            name=grafana
                            baseurl=https://packages.grafana.com/oss/rpm
                            repo_gpgcheck=1
                            enabled=1
                            gpgcheck=1
                            gpgkey=https://packages.grafana.com/gpg.key
                            sslverify=1
                            sslcacert=/etc/pki/tls/certs/ca-bundle.crt
                            # yum -y install grafana
                            # systemctl daemon-reload
                            # systemctl start grafana-server
                            # systemctl status grafana-server


                            9.查看loki信息

                                 登录grafana,在Configuration页“Add data source”添加loki,之后进入“Explore”选择数据源和以及相应的“Log labels”即可查看。



                            「喜欢文章,快来给作者赞赏墨值吧」
                            文章转载自FlyZhi,如果涉嫌侵权,请发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。

                            评论