Kubernetes Filebeat收集日志

Kubernetes Filebeat收集日志

之前是针对标准输出进行采集，现在来看一下针对于容器当中的日志，是在pod当中添加一个日志采集器，这里部署一个应用，单独部署一个容器，这个容器是filebeat日志采集器，这一块就通过emptydir来实现数据的共享。

filebeat的配置放在configmap当中，指明了日志采集的路径在哪，这个日志没有在标准输出当中，虽然是个nginx镜像。所以这里的日志还是存放在容器当中的。

apiVersion: apps/v1kind: Deploymentmetadata: name: app-log-logfilespec: replicas: 3 selector: matchLabels: project: microservice app: nginx-logfile template: metadata: labels: project: microservice app: nginx-logfile spec: containers: # 应用容器 - name: nginx image: lizhenliang/nginx-php # 将数据卷挂载到日志目录 volumeMounts: - name: nginx-logs mountPath: /usr/local/nginx/logs # 日志采集器容器 - name: filebeat image: elastic/filebeat:7.9.2 args: [ "-c", "/etc/filebeat.yml", "-e", ] resources: requests: cpu: 100m memory: 100Mi limits: memory: 500Mi securityContext: runAsUser: 0 volumeMounts: # 挂载filebeat配置文件 - name: filebeat-config mountPath: /etc/filebeat.yml subPath: filebeat.yml # 将数据卷挂载到日志目录 - name: nginx-logs mountPath: /usr/local/nginx/logs # 数据卷共享日志目录 volumes: - name: nginx-logs emptyDir: {} - name: filebeat-config configMap: name: filebeat-nginx-config---apiVersion: v1kind: Servicemetadata: name: app-log-logfilespec: ports: - port: 80 protocol: TCP targetPort: 80 selector: project: microservice app: nginx-logfile---apiVersion: v1kind: ConfigMapmetadata: name: filebeat-nginx-config data: # 配置文件保存在ConfigMap filebeat.yml: |- filebeat.inputs: - type: log paths: - /usr/local/nginx/logs/access.log # tags: ["access"] fields_under_root: true fields: project: microservice app: nginx setup.ilm.enabled: false setup.template.name: "nginx-access" setup.template.pattern: "nginx-access-*" output.elasticsearch: hosts: ['elasticsearch.ops:9200'] index: "nginx-access-%{+yyyy.MM.dd}"#这里是给项目打上了标签，标明日志来源 fields: project: microservice app: nginx

[root@master elk]# kubectl run -it busybox --image=busybox:1.28.4 --rm shIf you don't see a command prompt, try pressing enter./ # nslookup elasticsearch.opsServer: 169.254.25.10Address 1: 169.254.25.10Name: elasticsearch.opsAddress 1: 10.233.51.153 elasticsearch.ops.svc.cluster.local/ # [root@master ~]# kubectl get svc -n opsNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEelasticsearch ClusterIP 10.233.51.153 9200/TCP 12hkibana NodePort 10.233.17.6 5601:30601/TCP 12h

现在日志采集器采集到日志之后会推送到elastic里面

[root@master elk]# kubectl get pod -n opsNAME READY STATUS RESTARTS AGEapp-log-logfile-75b98c44bd-dmkhn 2/2 Running 0 13melasticsearch-549b496f94-rzt85 1/1 Running 1 13hfilebeat-d9hm5 1/1 Running 1 13hfilebeat-j49lr 1/1 Running 1 13hkibana-5c7bd6f4c5-ccnvt 1/1 Running 1 13h[root@master elk]# kubectl get svc -n ops NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEapp-log-logfile ClusterIP 10.233.1.237 80/TCP 15melasticsearch ClusterIP 10.233.51.153 9200/TCP 13hkibana NodePort 10.233.17.6 5601:30601/TCP 13h[root@master elk]# curl 10.233.1.237/status.htmlok

filebeat: Container ID: docker://eeff7cfd77b7aebb62e5161e0828ea020b018fb2d92f84aae0df758a4d296e11 Image: elastic/filebeat:7.9.2 Image ID: docker-pullable://elastic/filebeat@sha256:4276cd8246821c085d1abb4547ab3bbb66f9f818e927a5270ffa1d4fb16ebead Port: Host Port: Args: -c /etc/filebeat.yml -e State: Running Started: Sat, 25 Sep 2021 11:09:39 +0800 Ready: True Restart Count: 0 Limits: memory: 500Mi Requests: cpu: 100m memory: 100Mi Environment: Mounts: /etc/filebeat.yml from filebeat-config (rw,path="filebeat.yml") /usr/local/nginx/logs from nginx-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-sqqw5 (ro)[root@master elk]# kubectl exec -it app-log-logfile-75b98c44bd-dmkhn -c filebeat -n ops shkubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.sh-4.2# cd /usr/local/nginx/logssh-4.2# lsaccess.log error.logsh-4.2# cat access.log 10.233.70.0 - - [25/Sep/2021:11:21:30 +0800] "GET / HTTP/1.1" 403 146 "-" "curl/7.29.0"10.233.70.0 - - [25/Sep/2021:11:25:36 +0800] "GET / HTTP/1.1" 403 146 "-" "curl/7.29.0"10.233.70.0 - - [25/Sep/2021:11:25:59 +0800] "GET /status.html HTTP/1.1" 200 3 "-" "curl/7.29.0"

可以看到索引了

project和app标志了项目的来源

所以，不同的应用名称索引名称和标签都需要修改！

output.elasticsearch: hosts: ['elasticsearch.ops:9200'] index: "nginx-access-%{+yyyy.MM.dd}" fields: project: microservice app: nginx namespace: ops deployment: app-log-logfile

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。