暂无图片
暂无图片
暂无图片
暂无图片
暂无图片

k8s搭建elk日志收集平台

IT那活儿 2026-01-21
50

点击上方“IT那活儿”公众号--专注于企业全栈运维技术分享,不管IT什么活儿,干就完了!!!


网 址
官网:https://www.elastic.co/cn/elastic-cloud-kubernetes
部署:https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html
#部署crd
kubectl create -f https://download.elastic.co/downloads/eck/2.12.1/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/2.1


创建es集群
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: es-cluster
spec:
version: 8.13.2
  nodeSets:
  - name: default
    count:1
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
      name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: nfs-csi-es
EOF

es采用的认证机制。所以需要在secret中取出密码:
kubectl get secret es-cluster-es-elastic-user -o yaml
data:
  elastic: XXXX   #这段是密码

#解密查看密码,可定义为环境变量后调用:
echo XXXXXX  | base64 -d
#使用账号密码登陆,如果有ns记得加上,查看是否部署成功:
curl -u"ekastic:XXXXX" -k https://es-cluster-es-http.NAMESAPCE.svc:9200

创建kibana
cat <<EOF | kubectl apply -f -
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
version: 8.13.2
  count:1
  elasticsearchRef:
    name: es-cluster    #      和es集群名称一致
  http:                 #关闭默认的https。
    tls:
      selfSignedCertificate:
        disabled: true
  podTemplate:     #设置pod模板资源,添加了标签和资源使用限制
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        resources:
          limits:
            memory: 1Gi
            cpu: 1
EOF

#可创建一个ingress暴露kibana,绑定的service名称为kibana-kb-http。

部署filebeat
filebeat要在每台服务器都创建,所以需要以daemonset方式部署:
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
  namespace: elastic-system
spec:
type: filebeat
version: 8.13.2
  elasticsearchRef:       #关联es和kibana
    name: "es-cluster"
  kibanaRef:
    name: "kibana"
  config:
    filebeat:
      autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}       #node名称
          hints:
            enabled: true
            default_config:
              type: container
              paths:
              - var/log/containers/*${data.kubernetes.container.id}.log
        processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
        - drop_event.when:    
            or:              #忽略掉一些namespace下的日志收集
            - equals:      
kubernetes.namespace: "kube-system"
            - equals:
kubernetes.namespace: "logging"
            - equals:
kubernetes.namespace: "ingress-nginx"
            - equals:
kubernetes.namespace: "kube-node-lease"
            - equals:
kubernetes.namespace: "elastic-system"
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
automountServiceAccountToken: true
terminationGracePeriodSeconds:30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        containers:
        - name: filebeat
          securityContext:
            runAsUser: 0
            # If using Red Hat OpenShift uncomment this:
            #privileged: true
          volumeMounts:
          - name: varlogcontainers
            mountPath: var/log/containers
          - name: varlogpods
            mountPath: var/log/pods
          - name: varlibdockercontainers
            mountPath: var/lib/docker/containers
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
        volumes:
        - name: varlogcontainers
          hostPath:
            path: var/log/containers
        - name: varlogpods
          hostPath:
            path: var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: var/lib/docker/containers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  - nodes
verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: elastic-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: elastic-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
#查看是否收到了日志
curl -u'slastic:XXXX' -k https://es-cluster.NAMESPACE.svc:9200/_cat/indices

到此搭建成功。

END


本文作者:刘 营(上海新炬中北团队)

本文来源:“IT那活儿”公众号

文章转载自IT那活儿,如果涉嫌侵权,请发送邮件至:contact@modb.pro进行举报,并提供相关证据,一经查实,墨天轮将立刻删除相关内容。

评论