非json格式(低效)的app日志监控 见 grok_plain_text 分支
main分支只维护json格式的app日志
- MacOS 15.6
- Rancher Desktop 1.19
- 本地 K8s 1.33 集群 (2 worker nodes)
- 本地 Harbor 服务器 (harbor.local:80/library/order-service:latest)
- Elastic Stack 8.17
- 收集 非 json 格式的app日志，通过 Filebeat 收集，Logstash清洗, 并存储到 ES
- 使用k8s代替docker compose
- 优化manifest组织结构和使用独立logging命名空间
- json 格式的 app日志 (with logstash-logback-encoder)
- 以 sidecar 模式收集app日志 (with filebeat ndjson parser)
- 收集worker node的日志
- 收集中间件的日志
- 使用kafka做日志缓冲
mvn clean package docker compose up --build k8s的yaml manifest可以使用 kompose 辅助生成.
- 针对 Filebeat, 需要删除生成的 deployment + pvc, 修改为
daemonset, 让 Filebeat 在每个 Node 上运行一个 Pod，读取/var/log/containers/*.log（K8s 标准日志路径），并自动注入 Pod 元数据（namespace、pod name、labels）。 - 给Filebeat准备service account, 让它可以读取Node 资源, 比如nodeName (add_kubernetes_metadata需要用到)
$ mvn clean package $ docker build -t order-service . $ push order-service $ k get nodes NAME STATUS ROLES AGE VERSION kubeadm-master Ready control-plane 74d v1.33.4 kubeadm-worker01 Ready <none> 66d v1.33.4 kubeadm-worker02 Ready <none> 66d v1.33.4 kubectl apply -f k8s-manifest/logging -n logging kubectl apply -f k8s-manifest/apps -n apps $ k get po -o wide -n logging NAME READY STATUS RESTARTS AGE IP NODE elasticsearch-5599b96998-59fj4 1/1 Running 0 18s 10.244.1.220 kubeadm-worker01 filebeat-2dmfz 1/1 Running 0 18s 10.244.1.217 kubeadm-worker01 filebeat-hjdf4 1/1 Running 0 18s 10.244.2.164 kubeadm-worker02 kibana-d9c96cd58-xd75t 1/1 Running 0 18s 10.244.2.165 kubeadm-worker02 logstash-757b57ff9b-dlrd4 1/1 Running 0 18s 10.244.1.218 kubeadm-worker01 $ k get po -o wide -n apps NAME READY STATUS RESTARTS AGE IP NODE order-service-59678cc76-629w5 1/1 Running 0 24s 10.244.2.166 kubeadm-worker02# 端口转发 kubectl port-forward svc/order-service 8081:8080 -n apps kubectl port-forward svc/kibana 5601:5601 -n logging # 造http请求 $ siege -c1 -d5 -t60M http://localhost:8081/order/2 # 调式 filebeat 和 logstash的配置文件 (修改cm后,删除pod才能生效)# 不可以使用kubectl rollout restart, 因为cm的修改不会触发pod重启 $ kubectl apply -f k8s-manifest/logging $ kubectl delete po -l k8s-pod=filebeat -n logging $ kubectl delete po -l k8s-app=logstash -n loggingcurl -s http://localhost:80/order/2 > /dev/null && tail -n 1 /var/log/app/app.log | jq .{"@timestamp": "2025-09-18T15:16:21.047+0800", "appname": "demo-elk", "logger": "a.s.e.OrderController", "level": "INFO", "thread": "http-nio-80-exec-1", "message": "正在处理订单, orderNumber=ORD1758179781047", "traceId": "57e3d72813f08837ca80ed152227f7bd", "spanId": "74a75cccd86a0be7" }注释掉 filebeat-cm-sidecar.yaml 中的 output.logstash, 修改为 output.console, 可以看到被filebeat处理过的spring boot日志输出到控制台
k logs -n apps order-service-57bfd65dc-pk8sq -c sidecar{"@timestamp": "2025-09-18T11:53:11.175Z", "@metadata":{"beat": "filebeat", "type": "_doc", "version": "8.17.4" }, "host":{"name": "order-service-57bfd65dc-pk8sq" }, ... }{"_index": "app-logs-2025.09.18","_id": "9UzqXJkBA6sL6YX_QSxn","_version": 1,"_source": {"@version": "1","appname": "demo-elk","thread": "http-nio-80-exec-2","logger": "a.s.e.OrderController","traceId": "9e33ddd0a226348c7000b77b363429cb","spanId": "88aeb3ccce9e7fd8","level": "INFO","tags": ["beats_input_codec_plain_applied","parsed-success"],"@timestamp": "2025-09-18T13:01:07.875Z","message": "正在处理订单, orderNumber=ORD1758200467875"},"fields": {// OMITTED}}