- EFK
- 1. 日志系统
- 2. 部署ElasticSearch
- 2.1 创建handless服务
- 2.2 创建sts
- 3. 部署kibana
- 4. 部署ilogtail(docker-compose)
- 4.1 编写docker-compose
- 4.2 配置ilogtail采集
- 4.3 查看容器采集的日志
- 4.4 采集容器标准输出日志(可选)
- 4.5 查看采集的容器日志
- 5. 部署kafka
- 5.1 kafka介绍
- 5.2 部署kafka(docker-compose)
- 5.3 部署kafdrop(kafka的web界面)
- 5.4 ilogtail将日志写入到kafka
- 6. 部署logstash
- 6.1 部署logstash(docker-compose)
- 6.2 输出日志到es
- 6.3 到kibana查看
- 6.3.1 查看索引
EFK
这是一个日志收集系统,日志收集属于可观测性体系
可观测性体系
-
监控
- 基础设施的维度
- USE方法
- CPU:
- 利用率
- 负载
- 内存:
- 利用率
- 饱和度
- 错误率
- 网卡:
- 利用率
- 饱和度
- 错误率
- CPU:
- USE方法
- 应用程序的维度
- RED方法
- 基础设施的维度
-
日志
- 操作系统维度
- 应用维度
- 通过日志的错误进一步完善监控
- 通过日志排查问题
- 行为分析
-
链路追踪
1. 日志系统
- ELK
- ElasticSearch :日志存储系统
- LogStash:日志采集器
- Kibana:日志查询分析系统
ELK现在用的少,原因是
- jruby(java+ruby)
- 语法复杂:重量级日志采集
- 性能差
-
EFK
- ElasticSearch
- Fluneted:日志采集器
- Kibana
-
PLG
- Promtail :日志采集器
- Loki:日志存储系统
- Grafana:日志查询分析系统
我们这里部署的架构是
ilogtail ---> kafka ---> logstash ---> elasticsearch ---> kibana
使用ilogtail采集日志写入到kafka消息队列里,再由logstash从消息队列里读取日志写入到 es,最后再由kibana做展示
至于第三个环节为什么是logstash而不是ilogtail是因为,ilogtail要往es里面写日志会需要配置es的认证密码,但我们是没有给es配置用户名和密码的,所以采用logstash
2. 部署ElasticSearch
2.1 创建handless服务
[root@master EFK]# vim es-svc.yaml
kind: Service
apiVersion: v1
metadata:name: elasticsearchnamespace: logginglabels:app: elasticsearch
spec:selector:app: elasticsearchclusterIP: Noneports:- port: 9200name: rest- port: 9300name: inter-node
2.2 创建sts
[root@master EFK]# vim es-sts.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:name: esnamespace: logging
spec:serviceName: elasticsearchreplicas: 1selector:matchLabels:app: elasticsearchtemplate:metadata:labels:app: elasticsearchspec:initContainers:- name: initc1image: busyboxcommand: ["sysctl","-w","vm.max_map_count=262144"]securityContext:privileged: true- name: initc2image: busyboxcommand: ["sh","-c","ulimit -n 65536"]securityContext:privileged: true- name: initc3image: busyboxcommand: ["sh","-c","chmod 777 /data"]volumeMounts:- name: datamountPath: /datacontainers:- name: elasticsearchimage: swr.cn-east-3.myhuaweicloud.com/hcie_openeuler/elasticsearch:7.17.1resources:limits:cpu: 1000mrequests:cpu: 100mports:- containerPort: 9200name: restprotocol: TCP- containerPort: 9300name: inter-nodeprotocol: TCPvolumeMounts:- name: datamountPath: /usr/share/elasticsearch/dataenv:- name: cluster.namevalue: k8s-logs- name: node.namevalueFrom:fieldRef:fieldPath: metadata.name- name: cluster.initial_master_nodesvalue: "es-0"- name: discovery.zen.minimum_master_nodesvalue: "2"- name: discovery.seed_hostsvalue: "elasticsearch"- name: ES_JAVA_OPTSvalue: "-Xms512m -Xmx512m"- name: network.hostvalue: "0.0.0.0"volumeClaimTemplates:- metadata:name: datalabels:app: elasticsearchspec:accessModes: ["ReadWriteOnce"]resources:requests:storage: 10Gi
应用yaml文件
[root@master EFK]# kubectl create ns logging
[root@master EFK]# kubectl apply -f .
service/elasticsearch create
statefulset.apps/es create
[root@master EFK]# kubectl get pods -n logging
NAME READY STATUS RESTARTS AGE
es-0 1/1 Running 0 46s
pod显示running就是部署好了
3. 部署kibana
我直接将所有需要的资源放在一个yaml文件里面
apiVersion: v1
kind: ConfigMap
metadata:namespace: loggingname: kibana-configlabels:app: kibana
data:kibana.yml: |server.name: kibanaserver.host: "0.0.0.0"i18n.locale: zh-CNelasticsearch:hosts: ${ELASTICSEARCH_HOSTS}
---
apiVersion: v1
kind: Service
metadata:name: kibananamespace: logginglabels:app: kibana
spec:ports:- port: 5601type: NodePortselector:app: kibana---
apiVersion: apps/v1
kind: Deployment
metadata:name: kibananamespace: logginglabels:app: kibana
spec:selector:matchLabels:app: kibanatemplate:metadata:labels:app: kibanaspec:containers:- name: kibanaimage: swr.cn-east-3.myhuaweicloud.com/hcie_openeuler/kibana:7.17.1imagePullPolicy: IfNotPresentresources:limits:cpu: 1requests:cpu: 1env:- name: ELASTICSEARCH_URLvalue: http://elasticsearch:9200 # 写handless的名字- name: ELASTICSEARCH_HOSTSvalue: http://elasticsearch:9200 # 写handless的名字ports:- containerPort: 5601volumeMounts:- name: configmountPath: /usr/share/kibana/config/kibana.ymlreadOnly: truesubPath: kibana.ymlvolumes: - name: configconfigMap:name: kibana-config
查看端口并访问
[root@master EFK]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 17m
kibana NodePort 10.104.94.122 <none> 5601:30980/TCP 4m30s
kibana的nodeport端口是30980,我们来访问
这样就算部署好了,接下来需要部署日志采集工具
4. 部署ilogtail(docker-compose)
因为Fluentd配置复杂,所以我这里采用ilogtail来采集日志
- ilogtail配置简单
- 阿里开源,界面中文
我们先使用docker-compose的方式部署,最后整个平台搭建起来之后我们再将ilogtail部署到k8s集群里
4.1 编写docker-compose
[root@master ilogtail]# vim docker-compose.yaml
version: "3"
services:ilogtail:container_name: ilogtailimage: sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:2.0.4network_mode: hostvolumes:- /:/logtail_host:ro- /var/run:/var/run- ./checkpoing:/usr/local/ilogtail/checkpoint- ./config:/usr/local/ilogtail/config/local
- /:我们将宿主机的整个 / ,目录挂载到容器里面,方便容器读取日志
- checkpoint:这个相当于一个指针,指向当前读取到哪一行日志了,如果ilogtail被重启了它可以根据这个checkpoint来回到上一次读取的地方
- config:这个就是放采集的配置文件的
启动容器
[root@master ilogtail]# docker-compose up -d
[root@master ilogtail]# docker ps |grep ilogtail
eac545d4da87 sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:2.0.4 "/usr/local/ilogtail…" 10 seconds ago Up 9 seconds ilogtail
这样容器就启动了
4.2 配置ilogtail采集
[root@master ilogtail]# cd config/
[root@master config]# vim sample-stdout.yaml
enable: true
inputs:- Type: input_file # 文件输入类型FilePaths: - /logtail_host/var/log/messages
flushers:- Type: flusher_stdout # 标准输出流输出类型OnlyStdout: true
[root@master config]# docker restart ilogtail
-
/logtail_host/var/log/messages:这里是这个地址的原因是我们将宿主机的/,挂载到了容器内的logtail_host,所以我们宿主机产生的日志会在容器的/logtail_host/var/log/messages这个目录下
-
配置文件写好之后我们还需要重启容器让他读取配置,所以有一个restart
4.3 查看容器采集的日志
[root@master config]# docker logs ilogtail2024-06-30 11:16:25 {"content":"Jun 30 19:16:22 master dockerd[1467]: time=\"2024-06-30T19:16:22.251108165+08:00\" level=info msg=\"handled exit event processID=9a8df40981b3609897794e50aeb2bde805eab8a75334266d7b5c2899f61d486e containerID=61770e8f88e3c6a63e88f2a09d2683c6ccce1e13f6d4a5b6f79cc4d49094bab4 pid=125402\" module=libcontainerd namespace=moby","__time__":"1719746182"}
2024-06-30 11:16:25 {"content":"Jun 30 19:16:23 master kubelet[1468]: E0630 19:16:23.594557 1468 kubelet_volumes.go:245] \"There were many similar errors. Turn up verbosity to see them.\" err=\"orphaned pod \\\"9d5ae64f-1341-4c15-b70f-1c8f71efc20e\\\" found, but error not a directory occurred when trying to remove the volumes dir\" numErrs=2","__time__":"1719746184"}
可以看到,宿主机的日志已经被成功采集了,宿主机的日志会被封装到content里,如果没有看到输出的日志的话,需要进入到容器内部查看一个叫做ilogtail.LOG的文件,而不能使用docker logs ilogtail
4.4 采集容器标准输出日志(可选)
[root@master config]# cp sample-stdout.yaml docker-stdout.yaml
# 为了避免同时输出到标准输出而导致的日志杂乱,我们临时将sample-stdout关掉
[root@master config]# cat sample-stdout.yaml
enable: false # 将这里改为false
inputs:- Type: input_file # 文件输入类型FilePaths: - /logtail_host/var/log/messages
flushers:- Type: flusher_stdout # 标准输出流输出类型OnlyStdout: true
[root@master config]# cat docker-stdout.yaml
enable: true
inputs:- Type: service_docker_stdout Stdout: true # 采集标准输出Stderr: false # 不采集错误输出
flushers:- Type: flusher_stdout OnlyStdout: true
[root@master config]# docker restart ilogtail
ilogtail
4.5 查看采集的容器日志
2024-06-30 11:24:13 {"content":"2024-06-30 11:24:10 {\"content\":\"2024-06-30 11:24:07 {\\\"content\\\":\\\"2024-06-30 11:24:04.965 [INFO][66] felix/summary.go 100: Summarising 12 dataplane reconciliation loops over 1m3.4s: avg=3ms longest=12ms ()\\\",\\\"_time_\\\":\\\"2024-06-30T11:24:04.965893702Z\\\",\\\"_source_\\\":\\\"stdout\\\",\\\"_container_ip_\\\":\\\"192.168.200.200\\\",\\\"_image_name_\\\":\\\"calico/node:v3.23.5\\\",\\\"_container_name_\\\":\\\"calico-node\\\",\\\"_pod_name_\\\":\\\"calico-node-hgqzr\\\",\\\"_namespace_\\\":\\\"kube-system\\\",\\\"_pod_uid_\\\":\\\"4d0d950c-346a-4f81-817c-c19526700542\\\",\\\"__time__\\\":\\\"1719746645\\\"}\",\"_time_\":\"2024-06-30T11:24:07.968118197Z\",\"_source_\":\"stdout\",\"_container_ip_\":\"192.168.200.200\",\"_image_name_\":\"sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:2.0.4\",\"_container_name_\":\"ilogtail\",\"__time__\":\"1719746647\"}","_time_":"2024-06-30T11:24:10.971474647Z","_source_":"stdout","_container_ip_":"192.168.200.200","_image_name_":"sls-opensource-registry.cn-shanghai.cr.aliyuncs.com/ilogtail-community-edition/ilogtail:2.0.4","_container_name_":"ilogtail","__time__":"1719746650"}
能够正常看见日志就说明日志采集没有问题,接下来我们部署kafka,用来接收ilogtail的日志,注意将日志采集关掉,不然你的虚拟机磁盘很快就会满
5. 部署kafka
kafka作为消息队列,会有消费者和生产者,生产者在这里就是ilogtail,也就是将日志写入到kafka,消费者就是logstash,从kafka里面读取日志写入到es
5.1 kafka介绍
Apache kafka是分布式的,基于发布/订阅的容错消息系统,主要特性如下
-
高吞吐,低延迟:可以做到每秒百万级的吞吐量,并且延迟低(其他的消息队列基本也都可以)
-
持久性,可靠性:消息会被持久化到本地磁盘,支持数据备份防止数据丢失,并且可以配置消息有效期,以便消费者可以多次消费
-
kafka官方不支持docker部署,我们可以使用第三方的镜像
5.2 部署kafka(docker-compose)
version: '3'
services:zookeeper:image: quay.io/3330878296/zookeeper:3.8network_mode: hostcontainer_name: zookeeper-testvolumes:- zookeeper_vol:/data- zookeeper_vol:/datalog- zookeeper_vol:/logskafka:image: quay.io/3330878296/kafka:2.13-2.8.1network_mode: hostcontainer_name: kafkaenvironment:KAFKA_ADVERTISED_HOST_NAME: "192.168.200.200"KAFKA_ZOOKEEPER_CONNECT: "192.168.200.200:2181"KAFKA_LOG_DIRS: "/kafka/logs"volumes:- kafka_vol:/kafkadepends_on:- zookeeper
volumes:zookeeper_vol: {}kafka_vol: {}
- KAFKA_LOG_DIRS: "/kafka/logs":这个地方需要注意,在kafka的名词里面,他把数据叫做日志,这个地方看似是定义的日志目录,其实是kafka的数据目录
5.3 部署kafdrop(kafka的web界面)
[root@master kafka]# docker run -d --rm -p 9000:9000 \-e KAFKA_BROKERCONNECT=192.168.200.200:9092 \-e SERVER_SERVLET_CONTEXTPATH="/" \quay.io/3330878296/kafdrop
部署好之后就可以使用web界面查看了,部署web界面的原因是我们将日志写入到kafka之后可以直接使用web界面查看也没有写入进去,比kafka命令行更加的直观
在浏览器输入ip:9000
5.4 ilogtail将日志写入到kafka
[root@master config]# cd /root/ilogtail/config
[root@master config]# cp sample-stdout.yaml kafka.yaml
[root@master config]# vim kafka.yaml
enable: true
inputs:- Type: input_file FilePaths:- /logtail_host/var/log/messages
flushers:- Type: flusher_kafka_v2 Brokers:- 192.168.200.200:9092Topic: KafkaTopic
[root@master config]# docker restart ilogtail
ilogtail
这个时候我们再回到web界面就会出现一个topic
点进去可以查看有哪些日志被写入进去了
能看见日志就没问题了,接下来部署logstash
6. 部署logstash
logstash会从kafka读取消息然后写入到es里面去
6.1 部署logstash(docker-compose)
[root@master ~]# mkdir logstash
[root@master ~]# cd logstash
[root@master logstash]# vim docker-compose.yaml
version: '3'
services:logstash:image: quay.io/3330878296/logstash:8.10.1container_name: logstashnetwork_mode: hostenvironment:LS_JAVA_OPTS: "-Xmx1g -Xms1g"volumes:- /etc/localtime:/etc/localtime:ro- /apps/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml- /apps/logstash/pipeline:/usr/share/logstash/pipeline- /var/log:/var/log
- config里面放的是logstash本身的配置文件
- pipeline里面放的是采集/输出日志的规则
docker-compose写好之后先不要着急启动,因为我们给他挂载的配置文件还没有启动
现在编写配置文件
[root@master logstash]# mkdir /apps/logstash/{config,pipeline}
[root@master logstash]# cd /apps/logstash/config/
[root@master config]# vim logstash.yml
pipeline.workers: 2
pipeline.batch.size: 10
pipeline.batch.delay: 5
config.reload.automatic: true
config.reload.interval: 60s
写好这个文件之后我们启动这个logstash容器
[root@master logstash]# /root/logstash
[root@master logstash]# docker-compose up -d
[root@master logstash]# docker ps |grep logstash
60dfde4df40d quay.io/3330878296/logstash:8.10.1 "/usr/local/bin/dock…" 2 minutes ago Up 2 minutes logstash
启动之后就没问题了
6.2 输出日志到es
Logstash官方文档地址
我们要使用logstash输出日志到es的话就需要到pipeline里面去写一些规则
[root@master EFK]# cd /apps/logstash/pipeline/
[root@master pipeline]# vim logstash.conf
input {kafka {# 指定kafka地址bootstrap_servers => "192.168.200.200:9092"# 从哪些topic获取数据,要写已经存在topictopics => ["KafkaTopic"]# 从哪个地方开始读取,earliest是从头开始读取auto_offset_reset => "earliest"codec => "json"# 当一个logstash中有多个input插件时,建议每个插件定义一个id# id => "kubernetes"# group_id => "kubernetes"}
}filter {json {source => "event.original"}mutate {remove_field => ["event.original","event"]}
}output {elasticsearch {hosts => ["http://192.168.200.200:9200"]index => "kubernetes-logs-%{+YYYY.mm}"}
}
- hosts => ["http://192.168.200.200:9200"]:这个地方的9200,因为我们的logstash是用docker部署的,但是es是部署在k8s集群内部的,所以这个地方9200端口是通不了的,所以我们需要给k8s的es创建一个nodeport类型的svc,来让docker可以访问到
[root@master EFK]# kubectl expose pod es-0 --type NodePort --port 9200 --target-port 9200
service/es-0 exposed
[root@master EFK]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 3h38m
es-0 NodePort 10.97.238.173 <none> 9200:32615/TCP 2s
kibana NodePort 10.106.1.52 <none> 5601:30396/TCP 3h38m
这里他将9200映射到了本地的32615端口,所以我们将logstash的地址改到32615
output {elasticsearch {hosts => ["http://192.168.200.200:32615"]index => "kubernetes-logs-%{+YYYY.mm}"}
}
然后重启logstash
[root@master pipeline]# docker restart logstash
6.3 到kibana查看
6.3.1 查看索引
-
点击stack management
-
点击索引管理,会看到有索引存在就是正常
-
点击索引模式,创建索引
- 进入discover