1 默认配置1.1 filebeatfilebeat-7.17.yml,从网关中下载k8s的配置指定es和kibana的配置通过kibana查询可以查询到日志了但此时还不知道具体怎么用。1.2 kibana在Discover中创建索引格式filebeat-*得到如下图可以看出acc-statement-server的日志最多。但里面的字段太多了下面应该怎么看呢再跟进查看日志filebeat应该是每一样记录一次这个浪费了很多存储空间。另外排查问题也并不好查。2 多行合并输出如果使用默认的配置每一行日志就会产生一条记录。2.1 filebeat增加多行规则匹配设置索引符合条件走自己的索引否则则为默认索引output.elasticsearch: hosts: [10.101.10.2:9200,10.101.10.3:9200,10.101.10.4:9200] username: ${ELASTICSEARCH_USERNAME} password: ${ELASTICSEARCH_PASSWORD} indices: - index: acc-accountbook-server-%{yyyy.MM.dd} when.contains: kubernetes.container.name: acc-accountbook-server - index: acc-analysis-server-%{yyyy.MM.dd} when.contains: kubernetes.container.name: acc-analysis-server index: filebeat-7.17.25-%{yyyy.MM.dd}在kibana中跟进日志发现少部分日志输出成功大多失败这是什么原因呢为什么有些可以添加数据有些不能呢调试发现我在索引前面加上了eayc就可以了看来问题就出现在索引策略2.2 logback上面的时间分割是需要logback配置与之对应。如我的系统日志打印出来的是这个那么filebeat中就无法实现多行合并了。如下图修改logback配置。2.3 pipeline在elasticsearch中创建pipeline,参考了【ELK】到【EFK】,【Filebeat】收集【SpringBoot】日志但最终还是放弃了还是按照k8s自带的格式这样便于处理不需要非得自定义格式。PUT _ingest/pipeline/eayc_log_pipeline { description: 岁月云日志管道, processors: [ { remove: { field: agent.name, ignore_missing: true } }, { remove: { field: agent.ephemeral_id, ignore_missing: true } }, { remove: { field: log.file.path, ignore_missing: true } }, { remove: { field: input.type, ignore_missing: true } }, { remove: { field: kubernetes.node.labels.kubernetes_io/hostname, ignore_missing: true } }, { remove: { field: kubernetes.labels.k8s_kuboard_cn/layer, ignore_missing: true } }, { remove: { field: kubernetes.deployment.name, ignore_missing: true } }, { remove: { field: container.runtime, ignore_missing: true } }, { remove: { field: ecs.version, ignore_missing: true } }, { remove: { field: host.architecture, ignore_missing: true } }, { remove: { field: host.containerized, ignore_missing: true } }, { remove: { field: host.mac, ignore_missing: true } }, { remove: { field: host.os.codename, ignore_missing: true } }, { remove: { field: host.os.name, ignore_missing: true } }, { remove: { field: host.os.platform, ignore_missing: true } }, { remove: { field: kubernetes.labels.k8s_kuboard_cn/name, ignore_missing: true } }, { remove: { field: kubernetes.labels.pod-template-hash, ignore_missing: true } }, { remove: { field: kubernetes.namespace_uid, ignore_missing: true } }, { remove: { field: kubernetes.node.labels.beta_kubernetes_io/arch, ignore_missing: true } }, { remove: { field: kubernetes.node.labels.beta_kubernetes_io/os, ignore_missing: true } }, { remove: { field: log.flags, ignore_missing: true } }, { remove: { field: log.offset, ignore_missing: true } }, { remove: { field: kubernetes.container.id, ignore_missing: true } }, { remove: { field: kubernetes.pod.uid, ignore_missing: true } } ] }创建索引策略PUT _ilm/policy/eayc_log_policy { policy: { phases: { hot: { min_age: 0ms, actions: { rollover: { max_size: 50gb, max_age: 30d } } }, delete: { min_age: 90d, actions: { delete: {} } } } } }2.4 elasticsearch在k8s中启动filebeat中查看filebeat的日志发现2024-10-29T07:55:33.935Z ERROR [elasticsearch] elasticsearch/client.go:226 failed to perform any bulk index operations: 500 Internal Server Error: {error:{root_cause:[{type:illegal_state_exception,reason:There are no ingest nodes in this cluster, unable to forward request to an ingest node.}],type:illegal_state_exception,reason:There are no ingest nodes in this cluster, unable to forward request to an ingest node.},status:500}则需要在elasticsearch.yml中增加配置node.roles: [ingest]创建组合模板PUT _component_template/filebeat_settings { template: { settings: { number_of_shards: 1, number_of_replicas: 1 } } } PUT _component_template/filebeat_mappings { template: { mappings: { properties: { timestamp: { type: date }, message: { type: text } } } } } PUT _component_template/eayc_mappings { template: { mappings: { properties: { timestamp: { type: date }, message: { type: text }, custom_field: { type: keyword } } } } } PUT _component_template/acc_mappings { template: { mappings: { properties: { timestamp: { type: date }, message: { type: text }, custom_field: { type: keyword } } } } } PUT _index_template/filebeat { index_patterns: [filebeat-*], composed_of: [filebeat_settings, filebeat_mappings], priority: 100, template: { settings: { index.lifecycle.name: eayc_log_policy, index.lifecycle.rollover_alias: filebeat-write } } } PUT _index_template/eayc { index_patterns: [eayc-*], composed_of: [filebeat_settings, eayc_mappings], priority: 100, template: { settings: { index.lifecycle.name: eayc_log_policy, index.lifecycle.rollover_alias: filebeat-write } } } PUT _index_template/acc { index_patterns: [acc-*], composed_of: [filebeat_settings, acc_mappings], priority: 100, template: { settings: { index.lifecycle.name: eayc_log_policy, index.lifecycle.rollover_alias: filebeat-write } } }接着再看acc添加进去了再看日志数据出来了3 环境相关3.1 cgroup绝对性的控制CPU/内存在一个范围内使用cgroup来实现。下面是centos7.9[rootmaster1 ~]# cat /proc/mounts | grep cgroup tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent/usr/lib/systemd/systemd-cgroups-agent,namesystemd 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_prio,net_cls 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0下面是rockylinux9.5使用的是较新的内核默认启用了 cgroup v2可能是filebeat7.17不能兼容。下载安装8.17就没有报这个错误了。[rootmaster8 ~]# cat /proc/mounts | grep cgroup cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 none /run/calico/cgroup cgroup2 rw,relatime,nsdelegate,memory_recursiveprot 0 03.2 挂载目录将版本从8.17回退到8.10针对docker的目录如果docker的目录发生改变则下面两个地方更改在Kubernetes中Filebeat 通常从宿主机上的/var/log/containers目录获取日志3.3 日志没有写入elastic这个仍然是filebeat与elastic版本造成增加allow_older_versions: true配置即可