[k8s] Kubernetes 로그 - EFK 구축

Kubernetes 로그 - EFK 개요

이전 포스트에서 EFK에 대해 간략하게 설명하였습니다. 이제부터 EFK를 직접 구축해 보겠습니다.

Fluent Bit 구성

fluent bit는 helm을 통해 설치했습니다.
helm install을 수행하기 전 values.yaml에서 config 부분을 수정해야 합니다.
fluent bit는 Daemonset으로 각 노드별로 생성됩니다.

helm repo add fluent https://fluent.github.io/helm-charts
helm fetch fluent/fluent-bit --untar
helm install fluent-bit fluent/fluent-bit -f values.yaml # values.yaml 파일 수정 후 실행

config:
  service: |
    [SERVICE]
        Daemon Off
        Flush {{ .Values.flush }}
        Log_Level {{ .Values.logLevel }}
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On

  inputs: |
    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Parser            cri
        Tag               kube.*
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   Off

    [INPUT]
        Name            systemd
        Tag             host.*
        Systemd_Filter  _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail  On

  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Merge_Log           On
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

  outputs: |
    [OUTPUT]
        Name          es
        Match         *
        Host          elasticsearch.logging
        Port          9200
        Logstash_Format Off

 

Elasticsearch 구성

Elasticsearch는 8.0 이후 버전부터는 security가 기본적으로 enabled 상태입니다. security를 사용하지 않을 경우 7.17.0 버전을 사용하시면 됩니다. 그리고 Elasticsearch와 Kibana 연동 시 같은 버전으로 진행하셔야 합니다.
아래 코드는 현재 Minimal Security까지만 되어있으며, Alert 기능까지는 사용할 수 있도록 설정되어 있습니다.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: elasticsearch-service-account
  namespace: logging

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elasticsearch-cluster-role
  namespace: logging
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/exec", "services", "endpoints"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "replicasets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources: ["jobs", "cronjobs"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["extensions"]
    resources: ["ingresses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["monitoring.coreos.com"]
    resources: ["servicemonitors", "prometheuses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["logging.k8s.io"]
    resources: ["logs"]
    verbs: ["get", "list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elasticsearch-cluster-role-binding
  namespace: logging
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: elasticsearch-cluster-role
subjects:
  - kind: ServiceAccount
    name: elasticsearch-service-account
    namespace: logging

apiVersion: v1
kind: ConfigMap
metadata:
  name: elasticsearch-config
  namespace: logging
data:
  elasticsearch.yml: |
    network.host: 0.0.0.0
    http.port: 9200
    discovery.seed_hosts: ["elasticsearch-discovery"]
    xpack.security.enabled: true
    xpack.license.self_generated.type: basic
    xpack.security.authc.api_key.enabled: true

apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: logging
  labels:
    app: elasticsearch
spec:
  replicas: 1
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      serviceAccountName: elasticsearch-service-account
      containers:
        - name: elasticsearch
          image: elasticsearch:7.17.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9200
            - containerPort: 9300
          env:
            - name: discovery.type
              value: "single-node"
            - name: ES_JAVA_OPTS
              value: "-Xms500m -Xmx1000m"
          volumeMounts:
            - name: elasticsearch-config
              mountPath: /usr/share/elasticsearch/config/elasticsearch.yml
              subPath: elasticsearch.yml
            - name: elasticsearch-data
              mountPath: /usr/share/elasticsearch/data
      volumes:
        - name: elasticsearch-config
          configMap:
            name: elasticsearch-config
        - name: elasticsearch-data
          emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  namespace: logging
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  ports:
    - name: elasticsearch
      port: 9200
      protocol: TCP
      targetPort: 9200
    - name: elasticsearch-ssl
      port: 9300
      protocol: TCP
      targetPort: 9300

Kibana 구성

Kibana를 통해 Slack 알림까지 연동하고 싶었으나 이 기능은 유료인 경우에만 사용이 가능합니다.
Elasticsearch의 Security를 Basic security +  SSL/TLS로 설정하면 14일 trial 버전으로 사용해 보실 수 있습니다.

apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana-config
  namespace: logging
  labels:
    app: kibana
data:
  kibana.yml: |
    server.host: "0.0.0.0"
    elasticsearch.hosts: [ "http://elasticsearch.logging:9200" ]
    server.publicBaseUrl: ${{ secrets.SERVER_NAME }}
    elasticsearch.username: "${ ELASTICSEARCH_USERNAME }"
    elasticsearch.password: "${ ELASTICSEARCH_PASSWORD }"
    xpack.encryptedSavedObjects.encryptionKey: ${{ vars.ENCRYPTEDSAVEDOBJECTS }}
    xpack.reporting.encryptionKey: ${{ vars.REPORTING }}
    xpack.security.encryptionKey: ${{ vars.SECURITY }}
    xpack.security.enabled: true

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kibana
  namespace: logging
  labels:
    app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      name: kibana
      labels:
        app: kibana
    spec:
      containers:
        - name: kibana
          image: kibana:7.17.0
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5601
              protocol: TCP
          volumeMounts:
            - mountPath: /usr/share/kibana/config/kibana.yml
              name: config
              subPath: kibana.yml
          env:
            - name: ELASTICSEARCH_URL
              value: http://elasticsearch.logging:9200
            - name: discovery.type
              value: single-node
            - name: ELASTICSEARCH_USERNAME
              value: ${{ secrets.ELASTICSEARCH_USERNAME }}
            - name: ELASTICSEARCH_PASSWORD
              value: ${{ secrets.ELASTICSEARCH_PASSWORD }}
            - name: SERVER_NAME
              value: ${{ secrets.SERVER_NAME }}
      volumes:
        - name: config
          configMap:
            name: kibana-config

---
apiVersion: v1
kind: Service
metadata:
  name: kibana
  namespace: logging
  labels:
    app: kibana
spec:
  selector:
    app: kibana
  ports:
    - nodePort: 30651
      port: 5601
      protocol: TCP
      targetPort: 5601
  type: NodePort

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kibana-ingress
  namespace: logging
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
  labels:
    name: kibana-ingress
spec:
  ingressClassName: "alb-ingress-class"
  rules:
    - host: ${{ secrets.SERVER_NAME }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kibana
                port:
                  number: 5601

AWS환경이 아닌 경우 Ingress Class를 설치 후 진행하셔야 합니다.
저는 현재 AWS의 EKS 환경에서 구축하였기 때문에 annotations부분을 작성하였습니다.
SSL과 TLS 설정은 계속 막혀서 추후 성공하면 올리겠습니다.🥲
다음은 PLG 개요 및 구축과 slack 알림 연동에 대해서 올리겠습니다.