暂无图片
暂无图片
暂无图片
暂无图片
暂无图片
首页 / 基于AWS EKS的K8S实践 - 通过 Agent收集日志

基于AWS EKS的K8S实践 - 通过 Agent收集日志

程序员修炼笔记 2023-08-08
604



Hi~朋友，关注置顶防止错过消息


基于AWS EKS的K8S实践系列文章是基于企业级的实战文章，一些设置信息需要根据公司自身要求进行设置，如果大家有问题讨论或咨询可以加我微信（公众号后台回复 程序员修炼笔记 可获取联系方式）。

基于SideCar的容器

基于SideCar的容器灵活程度相对较高，但每个Pod多一个容器也是对资源的消耗。

基于Agent的容器收集方案

基于Agent的日志虽然将所有配置都放在了一个ConfigMap中，可能配置会比较冗长（通过filebeat.autodiscover进行解决），但由于他是DaemoSet的形式，将会极大程度的降低资源的损耗。

配置kube-system的filebeat-config ConfigMap

修改ConfigMap，配置filebeat-config，这里相当于修改filebeat的配置文件如下：

由于我们线上开了证书认证，因此我这里还需要存储一下证书，用于DaemonSet 在连接的时候使用，另外我们这里选择的 input 是 container，可以看到在我这里我分别采集了我们 nginx 的日志（我们的 nginx 日志是个 json 格式）和 xxx-app 的日志， 如果你们的日志格式都很统一，且日志处理方式也类似，推荐使用filebeat.autodiscover，这个可能会大大减少你的配置文件的书写。

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: filebeat-config
  5. namespace: kube-system
  6.   labels:
  7.     k8s-app: filebeat
  8. data:
  9.   elasticsearch-ca.pem: |-
  10. -----BEGIN CERTIFICATE-----
  11.     XXXXXXX
  12. -----END CERTIFICATE-----
  13.   filebeat.yml: |-
  14.     filebeat.inputs:
  15. - type: container
  16.       ignore_older: 1h
  17.       paths:
  18. - /var/log/containers/ingress-nginx-controller*.log
  19.       fields:
  20.         project: k8s-nginx
  21.       json.keys_under_root: true
  22.       json.add_error_key: true
  23.       json.overwrite_keys: true
  24.       json.expand_keys: true

  26. - type: container
  27.       paths:
  28. - "/var/log/containers/xxx-app*.log"
  29.       multiline:
  30.         type: pattern
  31.         pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3}'
  32.         negate: true
  33.         match: after
  34.       fields:
  35.         project: xxx-app
  36.       pipeline: application-service-log-pipeline

  38.     output.elasticsearch:
  39.       hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  40.       username: ${ELASTICSEARCH_USERNAME}
  41.       password: ${ELASTICSEARCH_PASSWORD}
  42.       allow_older_versions: "true"
  43.       protocol: "https"
  44.       ssl.certificate_authorities: /etc/filebeat/certs/elasticsearch-ca.pem
  45.       indices:
  46. - index: "%{[fields.project]}-%{+yyyy.MM.dd}"

修改DaemonSet

这里我主要修改了污点容忍，这是为了让我 DaemonSet 的 FileBeat 能够在有污点的节点上运行，另外还需要设置ES 的用户名、密码以及端口，以及SSL 证书，如下(下面的示例只是我这里和官网上配置不通的地方，完整的配置可以见文章末尾)：

  1. apiVersion: apps/v1
  2. kind: DaemonSet
  3. metadata:
  4.   name: filebeat
  5. namespace: kube-system
  6.   labels:
  7.     k8s-app: filebeat
  8. spec:
  9. ...
  10.     spec:
  11. ...
  12.       tolerations:
  13. - key: subnet-type.kubernetes.io
  14. operator: Equal
  15.           value: public
  16.           effect: NoSchedule
  17.       containers:
  18. - name: filebeat
  19.           image: docker.elastic.co/beats/filebeat:8.6.1
  20.           args: [
  21. "-c", "/etc/filebeat.yml",
  22. "-e",
  23. ]
  24.           env:
  25. - name: ELASTICSEARCH_HOST
  26.               value:  xxx
  27. - name: ELASTICSEARCH_PORT
  28.               value: "9200"
  29. - name: ELASTICSEARCH_USERNAME
  30.               value: xxxx
  31. - name: ELASTICSEARCH_PASSWORD
  32.               value: xxxxx
  33. ...
  34.           volumeMounts:
  35. - name: config
  36.               mountPath: /etc/filebeat/certs/elasticsearch-ca.pem
  37.               readOnly: true
  38.               subPath: elasticsearch-ca.pem
  39. ...
  40.       volumes:
  41. - name: config
  42.           configMap:
  43.             defaultMode: 0640
  44.             name: filebeat-config
  45. ...

完整 FileBeat DaemonSet 的部署 Yaml

下面的 Yaml把上面我提到的地方按照你们自己的配置修改以后，直接运行就可以将 FileBeat 进行部署，部署成功以后如果有新的日志采集需要加入，只需要修改 ConfigMap，然后重新部署 DaemonSet 即可。

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4.   name: filebeat
  5. namespace: kube-system
  6.   labels:
  7.     k8s-app: filebeat

  9. ---

  11. apiVersion: rbac.authorization.k8s.io/v1
  12. kind: ClusterRole
  13. metadata:
  14.   name: filebeat
  15.   labels:
  16.     k8s-app: filebeat
  17. rules:
  18. - apiGroups: [""] # "" indicates the core API group
  19.     resources:
  20. - namespaces
  21. - pods
  22. - nodes
  23.     verbs:
  24. - get
  25. - watch
  26. - list
  27. - apiGroups: ["apps"]
  28.     resources:
  29. - replicasets
  30.     verbs: ["get", "list", "watch"]
  31. - apiGroups: ["batch"]
  32.     resources:
  33. - jobs
  34.     verbs: ["get", "list", "watch"]

  36. ---
  37. apiVersion: rbac.authorization.k8s.io/v1
  38. kind: Role
  39. metadata:
  40.   name: filebeat
  41. # should be the namespace where filebeat is running
  42. namespace: kube-system
  43.   labels:
  44.     k8s-app: filebeat
  45. rules:
  46. - apiGroups:
  47. - coordination.k8s.io
  48.     resources:
  49. - leases
  50.     verbs: ["get", "create", "update"]

  52. ---
  53. apiVersion: rbac.authorization.k8s.io/v1
  54. kind: Role
  55. metadata:
  56.   name: filebeat-kubeadm-config
  57. namespace: kube-system
  58.   labels:
  59.     k8s-app: filebeat
  60. rules:
  61. - apiGroups: [""]
  62.     resources:
  63. - configmaps
  64.     resourceNames:
  65. - kubeadm-config
  66.     verbs: ["get"]

  68. ---
  69. apiVersion: rbac.authorization.k8s.io/v1
  70. kind: ClusterRoleBinding
  71. metadata:
  72.   name: filebeat
  73. subjects:
  74. - kind: ServiceAccount
  75.     name: filebeat
  76. namespace: kube-system
  77. roleRef:
  78.   kind: ClusterRole
  79.   name: filebeat
  80.   apiGroup: rbac.authorization.k8s.io

  82. ---
  83. apiVersion: rbac.authorization.k8s.io/v1
  84. kind: RoleBinding
  85. metadata:
  86.   name: filebeat
  87. namespace: kube-system
  88. subjects:
  89. - kind: ServiceAccount
  90.     name: filebeat
  91. namespace: kube-system
  92. roleRef:
  93.   kind: Role
  94.   name: filebeat
  95.   apiGroup: rbac.authorization.k8s.io

  97. ---
  98. apiVersion: rbac.authorization.k8s.io/v1
  99. kind: RoleBinding
  100. metadata:
  101.   name: filebeat-kubeadm-config
  102. namespace: kube-system
  103. subjects:
  104. - kind: ServiceAccount
  105.     name: filebeat
  106. namespace: kube-system
  107. roleRef:
  108.   kind: Role
  109.   name: filebeat-kubeadm-config
  110.   apiGroup: rbac.authorization.k8s.io

  112. ---
  113. apiVersion: v1
  114. kind: ConfigMap
  115. metadata:
  116.   name: filebeat-config
  117. namespace: kube-system
  118.   labels:
  119.     k8s-app: filebeat
  120. data:
  121.   elasticsearch-ca.pem: |-
  122. -----BEGIN CERTIFICATE-----
  123.     XXXXXXX
  124. -----END CERTIFICATE-----
  125.   filebeat.yml: |-
  126.     filebeat.inputs:
  127. - type: container
  128.       ignore_older: 1h
  129.       paths:
  130. - /var/log/containers/ingress-nginx-controller*.log
  131.       fields:
  132.         project: k8s-nginx
  133.       json.keys_under_root: true
  134.       json.add_error_key: true
  135.       json.overwrite_keys: true
  136.       json.expand_keys: true

  138. - type: container
  139.       paths:
  140. - "/var/log/containers/xxx-app*.log"
  141.       multiline:
  142.         type: pattern
  143.         pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}\:\d{2}\:\d{2}\.\d{3}'
  144.         negate: true
  145.         match: after
  146.       fields:
  147.         project: xxx-app
  148.       pipeline: application-service-log-pipeline

  150.     output.elasticsearch:
  151.       hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  152.       username: ${ELASTICSEARCH_USERNAME}
  153.       password: ${ELASTICSEARCH_PASSWORD}
  154.       allow_older_versions: "true"
  155.       protocol: "https"
  156.       ssl.certificate_authorities: /etc/filebeat/certs/elasticsearch-ca.pem
  157.       indices:
  158. - index: "%{[fields.project]}-%{+yyyy.MM.dd}"

  160. ---
  161. apiVersion: apps/v1
  162. kind: DaemonSet
  163. metadata:
  164.   name: filebeat
  165. namespace: kube-system
  166.   labels:
  167.     k8s-app: filebeat
  168. spec:
  169.   selector:
  170.     matchLabels:
  171.       k8s-app: filebeat
  172. template:
  173.     metadata:
  174.       labels:
  175.         k8s-app: filebeat
  176.     spec:
  177.       serviceAccountName: filebeat
  178.       terminationGracePeriodSeconds: 30
  179.       hostNetwork: true
  180.       dnsPolicy: ClusterFirstWithHostNet
  181.       tolerations:
  182. - key: subnet-type.kubernetes.io
  183. operator: Equal
  184.           value: public
  185.           effect: NoSchedule
  186.       containers:
  187. - name: filebeat
  188.           image: docker.elastic.co/beats/filebeat:8.6.1
  189.           args: [
  190. "-c", "/etc/filebeat.yml",
  191. "-e",
  192. ]
  193.           env:
  194. - name: ELASTICSEARCH_HOST
  195.               value:  xxx
  196. - name: ELASTICSEARCH_PORT
  197.               value: "9200"
  198. - name: ELASTICSEARCH_USERNAME
  199.               value: xxxx
  200. - name: ELASTICSEARCH_PASSWORD
  201.               value: xxxxx
  202. - name: NODE_NAME
  203.               valueFrom:
  204.                 fieldRef:
  205.                   fieldPath: spec.nodeName
  206.           securityContext:
  207.             runAsUser: 0
  208. # If using Red Hat OpenShift uncomment this:
  209. #privileged: true
  210.           resources:
  211.             limits:
  212.               memory: 200Mi
  213.             requests:
  214.               cpu: 100m
  215.               memory: 100Mi
  216.           volumeMounts:
  217. - name: config
  218.               mountPath: /etc/filebeat.yml
  219.               readOnly: true
  220.               subPath: filebeat.yml
  221. - name: config
  222.               mountPath: /etc/filebeat/certs/elasticsearch-ca.pem
  223.               readOnly: true
  224.               subPath: elasticsearch-ca.pem
  225. - name: data
  226.               mountPath: /usr/share/filebeat/data
  227. - name: varlibdockercontainers
  228.               mountPath: /var/lib/docker/containers
  229.               readOnly: true
  230. - name: varlog
  231.               mountPath: /var/log
  232.               readOnly: true
  233.       volumes:
  234. - name: config
  235.           configMap:
  236.             defaultMode: 0640
  237.             name: filebeat-config
  238. - name: varlibdockercontainers
  239.           hostPath:
  240.             path: /var/lib/docker/containers
  241. - name: varlog
  242.           hostPath:
  243.             path: /var/log
  244. # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
  245. - name: data
  246.           hostPath:
  247. # When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
  248.             path: /var/lib/filebeat-data
  249.             type: DirectoryOrCreate



【知识星球招募】


  知识星球正在初期搭建中，在星球中可以互相提问，分享经验，我也会在星球中分享基础技术架构的全流程搭建，包括但不仅限于DevOps，监控、Flink开发，Java开发（JVM、多线程、MySQL数据库）等相关技术类文章，期待大家的加入




elasticsearchk8saws科技新闻filebeat
文章转载自程序员修炼笔记，如果涉嫌侵权，请发送邮件至：contact@modb.pro进行举报，并提供相关证据，一经查实，墨天轮将立刻删除相关内容。

评论