OpenTelemetry OBI - 非侵入式监控各种语言应用的神器 0. 前言最近我写了几篇文章用零侵入方式使用OpenTelemetry监控Java、Node.JS和Python的应用。有人问我那GoLang的应用和C/C的应用怎么办要监控C/C还想零侵入确实有点难度。对于GoLang来说如果运行在Kubernetes内还有办法而独立运行也很难零侵入。但是有一种最强的OpenTelemetry监控手段与变成语言无关那就是本文描述的OBI方式。虽然监控得到的数据种类有限但是耦合性最小最简单方便。本文最早写作于2026年1月我今天根据OpenTelemetry的最新文档对文章进行了一些微量改写。1. 为什么需要非侵入式监控“非侵入式监控(non-intrusive observability)”解决了分布式系统在真实生产环境中的结构性问题。侵入式监控通常的做法是在代码里手动加监控SDK的代码。典型的做法是使用各种编程语言的Open Telemetry SDK把监控代码嵌入应用程序。这样做增加了开发的成本并引入了过度耦合的问题。一些微侵入式监控不需要修改用户代码但是需要调整配置和部署方式有时需要重启动。这类微侵入式监控有时也被归入非侵入式监控大体包括以下的方式静态或者动态加载Agent修改 Deployment注入 sidecar改变网络路径激活软件框架内嵌的监控配置但是仍然有不少应用无法使用非侵入式监控比如老系统第三方闭源系统商业软件系统使用多种语言且版本复杂。某些编程语言(比如独立运行的GoLang应用、C/C的应用等)而SRE的一个基本原则是不能因为监控引入生产事故。这样就需要零侵入式监控。基于eBPF的OpenTelemetry eBPF Instrumentation(OBI)就是这样一种方式。比如C/C的应用只能采用本文描述的OBI方式。而GoLang应用如果不希望修改任何代码大体有两种方式一是运行在Kubernetes集群通过使用OpenTelemetry Operator (采用了自动挂接Sidecar使用增强的eBPF的系统来启动应用。我计划以后会写一篇文章描述)。另一种就是采用本文描述的OBI方式。2. 什么是OpenTelemetry OBIOpenTelemetry eBPF Instrumentation (OBI)不依赖应用代码或 SDK完全零代码自动监控/插桩(instrumentation)。它利用 Linux 内核的eBPF(extended Berkeley Packet Filter)机制拦截系统调用例如网络 I/O在内核空间解析协议和事件生成 Trace Span 和 Metrics通过用户态组件发送数据到OTLP端口(比如OpenTelemetry Collector)这种方式让 OBI 可被透明应用于任意语言/运行时无需修改源码或重启。以下是OBI的工作原理eBPF 程序加载与挂载。OBI在启动时会执行将一组eBPF程序注入内核。使用kprobe拦截内核函数(如 tcp_sendmsg, recvmsg)获取网络吞吐和延迟。使用uprobe拦截用户态库函数(如 OpenSSL 的 SSL_read 或 Go 的 http.Handler)在不进入内核的情况下获取加密前的明文数据。使用 tracepoints / BPF类型(BTF) 提供更高层结构信息以减少手工探针解析复杂度。这些探针在内核空间运行可以非常高效捕获进程产生的内核事件而不进入应用层执行。协议级解析。在内核捕获网络 I/O 时识别协议层(HTTP, gRPC 等)包括从数据包中提取URI/方法状态码主机/端口Request/Response 时长等元数据。协议识别和解析是通过分析内核缓冲区中的数据结构和报文模式来完成而不是从应用代码调用链的角度。Trace Context传播机制。自动分布式追踪的核心是传播trace context。OBI 能自动读取和注入W3C traceparent header对于明文 HTTP它会直接在内核层写入header。在 TCP/IP 层它甚至会修改报文使下游服务继续传递已有context。这种实现利用eBPF结合Linux Traffic Control (TC)链接和packet manipulation。虽然 eBPF 程序跑在内核里但 OBI 还有一个用户态进程从内核 eBPF 程序收集事件(通过 ring buffer、perf 等机制)执行序列化并推送到 OTLP endpoint / Collector管理配置、动态调整 probe 等。这是一种典型的内核探针 用户态处理模式。OBI生成的监控数据包括以下类型Span(Tracing)。每次请求/响应会生成一个 Span。如果已有 upstream trace context则继续该 trace。否则自动生成按照 W3C 规范的新 trace context。这保证了与应用级 OpenTelemetry SDK 生成的 traces 能够端到端联通。RED Metrics(Rate, Error, Duration)和TCP/IP Layer Metrics。OBI 在内核自动记录请求数(Rate)、错误数(Error基于状态码)、响应延迟(Duration)。产出的 metrics 遵循 OpenTelemetry 及 Prometheus 兼容格式。Logs: stdout/stderr 捕获3. OpenTelemetry OBI作为监控系统的特点性能优势。eBPF 程序运行内核空间避免应用层 hook 或代理转发因此CPU 开销显著低于传统用户态 agent且无需增加额外网络 hop。这使得在高流量场景下能较低成本获取遥测。由于 OBI 在内核层捕获系统调用理论上支持所有运行在 Linux 上的语言但目前官方重点验证和优化的语言包括Java、.NETC#、Go(原生支持最好)、Python、Node.js、Ruby、C / C、Rust...协议覆盖与可扩展性。目前官方文档列出的协议包括 HTTP/S 和 gRPC 等基本服务。对于加密流量(TLS/SSL)OBI 可以在不解密流量的情况下观测到事务的元数据(例如通过分析 TLS 握手后的明文协议信息)这对于追踪 HTTPS 服务很有帮助。社区文档显示项目正在快速扩展覆盖更多协议和数据库层遥测。OBI 的协议识别逻辑基于“模式识别和报文结构”不断扩展到更多的协议(HTTP/1.1, HTTP/2, gRPC (Alpha), Kafka, Redis, SQL, MongoDB...)。以下是OBI的系统和权限要求Linux 内核5.8(或 Red Hat Enterprise Linux 的 4.18 回溯补丁)架构x86_64 或 arm64权限root 或特定 capabilities(如 CAP_DAC_READ_SEARCH、CAP_SYS_PTRACE、CAP_PERFMON、CAP_BPF)。网络观测需额外 CAP_NET_ADMIN。测试过的发行版Ubuntu 20.04、CentOS 7、Red Hat Enterprise Linux 8、Debian 11 等。与 OpenTelemetry SDK 的协同。当 OBI 在运行时检测到应用自身已有 OpenTelemetry SDK sending spans/metricsOBI 会关闭重复的signals以避免数据冲突或重复发送。这种 “智能协同” 保证不干扰已有的应用级观测逻辑。OBI不擅长的领域包括自定义业务 span、方法级 tracing(例如 Java 方法级别)、业务属性(如 order_id)、内部函数调用栈。如果你需要这些仍需OpenTelemetry SDK、Agents、手动 instrumentation。4. 搭建一个OpenTelemetry OBI演示系统4.1 搭建一个支持OTLP端口的后端监控系统目前所有的一流商业应用监控系统(APM)都支持OpenTelemetry定义的OTLP端口。支持OTLP端口的开源或免费系统也很多。OpenTelemetry的后端工具就是支持OpenTelemetry metrics/traces/logs/profiles的数据库和UI的工具集。最常见的做法还是使用OpenTelemetry Collector来连接不同的后端工具。假如您是初学者或者系统很小可以直接使用基于Docker的Grafana LGTM几乎是一键安装完成非常简单易行。前提是需要您有个支持Docker的环境。假设您想把LGTM安装到/opt/lgtm目录 (任何目录均可)下面是命令(假设在Linux系统)docker pull grafana/otel-lgtmmkdir/opt/lgtmcd/opt/lgtm wget https://raw.githubusercontent.com/grafana/docker-otel-lgtm/main/run-lgtm.shchmodx run-lgtm.sh sed -is/3000:3000/3100:3000/run-lgtm.sh注意最后一行命令因为LGTM的Grafana的默认对外端口是3000这个端口经常和一些应用程序冲突我就改成了3100.下面是启动LGTM的方法就一个命令cd/opt/lgtm ./run-lgtm.shLGTM要监听以下的端口4317/4318 是OTLP端口用来接受metrics/traces/logs数据3100 是Grafana UI的端口9090 是Prometheus的端口用于调试4040 是Pyroscope接受Profiles的端口将来会整合进入4317/4318后面我用OpenTelemetry OBI从宿主机的应用以及Kubernetes集群的应用发送OTLP信号数据(包括OpenTelemetry metrcs, traces, logs)给Grafana LGTM Stack。其实无论无论是在传统宿主机上的应用还是Kubernetes集群中运行的应用无论是什么语言最终展现出的metrics和traces的格式是相同的当然如果是在Kubernetes上的应用可以选择增加Kubernetes相关的元数据作为属性(比如命名空间pod名称等)。使用“http://localhost:3100”就可以访问LGTM的Grafana UI界面(如果是远程的话用主机名或者IP替换localhost)使用admin/admin登录。虽然LGTM支持各种类型的数据metrics/traces/logs/profiles但是OBI目前主要收集metrics/traces所以我们的Grafana Dashboard只展现这两种数据。以下是我做的Dashboard的截图下面是该dashboard的源码以下是在Explorer上配置metrics的页面以下是查看Trace(追踪)的页面4.2 怎样从宿主机的应用上收集监控数据首先要下载OBI工具过去是要到一个容器里面提取现在已经可以直接下载了。到 OBI 的发行网页(https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation/releases)根据您的平台选择最新的发行版来下载。比如对于最常见的平台(amd64) 和当前最新的版本好v0.10.0 下载obi-v0.10.0-linux-amd64.tar.gz。配饰OBI的方式非常多。可以采用环境变量也可以用配置文件。对于这个简单的测试我使用了以下的配置文件obi.yamldiscovery: instrument: - open_ports: 8080,9090,28080 log_level: DEBUG ebpf: context_propagation: all otel_traces_export: endpoint: http://localhost:4318 otel_metrics_export: endpoint: http://loalhost:4318注意这个配置文件说明我要监控8080、9090和28080三个端口的应用。localhost是Grafana LGTM Stack所在的宿主机地址如果是其它IP地址(比如192.168.108.128)直接修改即可。以下是运行OBI的命令(在应用所在的Linux主机上运行注意应用的端口必须在 obi.yaml 设置的范围之内)sudo ./obi --config./obi.yaml至于配置OBI的详细说明请参照官方文档https://opentelemetry.io/docs/zero-code/obi/configure/为了让OpenTelemetry很好的区分不同的应用请在启动应用之前设置OTEL_SERVICE_NAME环境变量。例如exportOTEL_SERVICE_NAMEmy-godemo ./app从本文的dashboard里可以看到我在这个演示系统里有两个应用一个叫做my-godemo一个叫做my-service。4.3 怎样从Kubernetes集群中运行的应用上收集监控数据以下是我执行的命令包括权限的配置(ServiceAccount和ClusterRoleBinding)以及建立一个运行OBI的DaemonsetapiVersion: v1 kind: ServiceAccount metadata: name: obi --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: obi rules: - apiGroups: [apps] resources: [replicasets] verbs: [list, watch] - apiGroups: [] resources: [pods, services, nodes] verbs: [list, watch] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: obi subjects: - kind: ServiceAccount name: obi namespace: default roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: obi --- apiVersion: apps/v1 kind: DaemonSet metadata: name: obi labels: app: obi spec: selector: matchLabels: app: obi template: metadata: labels: app: obi spec: hostPID: true # Required to access the processes on the host serviceAccountName: obi # required if you want kubernetes metadata decoration containers: - name: autoinstrument image: otel/ebpf-instrument:main securityContext: privileged: true env: # Select the executable by its name instead of OTEL_EBPF_OPEN_PORT - name: OTEL_EBPF_OPEN_PORT value: 8080,28080 - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://192.168.108.128:4318 # required if you want kubernetes metadata decoration - name: OTEL_EBPF_KUBE_METADATA_ENABLE value: true注意这个配置文件说明我要监控8080、9090和28080三个端口的应用。192.168.108.128是Grafana LGTM Stack所在的宿主机地址。至于配置OBI的详细说明请参照官方文档obi-v0.10.0-linux-amd64.tar.gz4.4 简单介绍一下Dashboard的做法我简单介绍一下本文的Dashboard第一行左图是描述吞吐量主要的PromQL是:sumby(http_route, http_response_status_code, service_name) ( rate(http_server_request_duration_seconds_count{service_name!~lgtm|}[2m]) )第一行右图是描述HTTP服务器延迟时间(90%)主要的PromQL是:histogram_quantile(0.90, sumby(le, http_route, service_name) (rate(http_server_request_duration_seconds_bucket{service_name!~lgtm|}[2m])))第二行左图是请求包大小主要的PromQL是:histogram_quantile(0.90, sumby(le, http_route, service_name) (rate(http_server_request_body_size_bytes_bucket{service_name!~lgtm|}[2m])))第二行右图是返回包大小主要的PromQL是:histogram_quantile(0.90, sumby(le, http_route, service_name) (rate(http_server_response_body_size_bytes_bucket{service_name!~lgtm|}[2m])))第三行左图是失败次数/秒主要的PromQL是:sumby(client, server) (rate(traces_service_graph_request_failed_total[2m]))第三行右图是总流量主要的PromQL是:sumby(service) (rate(traces_spanmetrics_size_total[2m]))第四行左图是描述服务之间的调用关系。第四行右图是所有Traces。以下是整个Dashboard的代码可以直接导入。{ annotations: { list: [ { builtIn: 1, datasource: { type: grafana, uid: -- Grafana -- }, enable: true, hide: true, iconColor: rgba(0, 211, 255, 1), name: Annotations Alerts, type: dashboard } ] }, editable: true, fiscalYearStartMonth: 0, graphTooltip: 0, links: [], panels: [ { datasource: { type: prometheus, uid: prometheus }, fieldConfig: { defaults: { color: { mode: palette-classic }, custom: { axisBorderShow: false, axisCenteredZero: false, axisColorMode: text, axisLabel: , axisPlacement: auto, barAlignment: 0, barWidthFactor: 0.6, drawStyle: line, fillOpacity: 0, gradientMode: none, hideFrom: { legend: false, tooltip: false, viz: false }, insertNulls: false, lineInterpolation: linear, lineWidth: 1, pointSize: 5, scaleDistribution: { type: linear }, showPoints: never, showValues: false, spanNulls: false, stacking: { group: A, mode: none }, thresholdsStyle: { mode: off } }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] }, unit: reqps }, overrides: [] }, gridPos: { h: 8, w: 12, x: 0, y: 0 }, id: 2, options: { legend: { calcs: [], displayMode: list, placement: bottom, showLegend: true }, tooltip: { hideZeros: false, mode: single, sort: none } }, pluginVersion: 12.4.1, targets: [ { datasource: { type: prometheus, uid: prometheus }, editorMode: code, expr: sum by (http_route, http_response_status_code, service_name) (\r\n rate(http_server_request_duration_seconds_count{service_name!~\lgtm|\}[2m])\r\n)\r\n, instant: true, legendFormat: __auto, range: true, refId: A } ], title: 吞吐量, type: timeseries }, { datasource: { type: prometheus, uid: prometheus }, fieldConfig: { defaults: { color: { mode: palette-classic }, custom: { axisBorderShow: false, axisCenteredZero: false, axisColorMode: text, axisLabel: , axisPlacement: auto, barAlignment: 0, barWidthFactor: 0.6, drawStyle: line, fillOpacity: 0, gradientMode: none, hideFrom: { legend: false, tooltip: false, viz: false }, insertNulls: false, lineInterpolation: linear, lineWidth: 1, pointSize: 5, scaleDistribution: { type: linear }, showPoints: never, showValues: false, spanNulls: false, stacking: { group: A, mode: none }, thresholdsStyle: { mode: off } }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] }, unit: s }, overrides: [] }, gridPos: { h: 8, w: 12, x: 12, y: 0 }, id: 1, options: { legend: { calcs: [], displayMode: list, placement: bottom, showLegend: true }, tooltip: { hideZeros: false, mode: single, sort: none } }, pluginVersion: 12.4.1, targets: [ { datasource: { type: prometheus, uid: prometheus }, editorMode: code, exemplar: false, expr: histogram_quantile(0.90, sum by (le, http_route, service_name) (rate(http_server_request_duration_seconds_bucket{service_name!~\lgtm|\}[2m])))\r\n, instant: true, legendFormat: __auto, range: true, refId: A } ], title: 延迟时间, type: timeseries }, { datasource: { type: prometheus, uid: prometheus }, fieldConfig: { defaults: { color: { mode: palette-classic }, custom: { axisBorderShow: false, axisCenteredZero: false, axisColorMode: text, axisLabel: , axisPlacement: auto, barAlignment: 0, barWidthFactor: 0.6, drawStyle: line, fillOpacity: 0, gradientMode: none, hideFrom: { legend: false, tooltip: false, viz: false }, insertNulls: false, lineInterpolation: linear, lineWidth: 1, pointSize: 5, scaleDistribution: { type: linear }, showPoints: never, showValues: false, spanNulls: false, stacking: { group: A, mode: none }, thresholdsStyle: { mode: off } }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] }, unit: decbytes }, overrides: [] }, gridPos: { h: 8, w: 12, x: 0, y: 8 }, id: 5, options: { legend: { calcs: [], displayMode: list, placement: bottom, showLegend: true }, tooltip: { hideZeros: false, mode: single, sort: none } }, pluginVersion: 12.4.1, targets: [ { datasource: { type: prometheus, uid: prometheus }, editorMode: code, expr: histogram_quantile(0.90, sum by (le, http_route, service_name) (rate(http_server_request_body_size_bytes_bucket{service_name!~\lgtm|\}[2m]))), instant: true, key: Q-33194457-04d2-4472-b32e-942fd5e718af-0, legendFormat: __auto, range: true, refId: A } ], title: 请求包大小, type: timeseries }, { datasource: { type: prometheus, uid: prometheus }, fieldConfig: { defaults: { color: { mode: palette-classic }, custom: { axisBorderShow: false, axisCenteredZero: false, axisColorMode: text, axisLabel: , axisPlacement: auto, barAlignment: 0, barWidthFactor: 0.6, drawStyle: line, fillOpacity: 0, gradientMode: none, hideFrom: { legend: false, tooltip: false, viz: false }, insertNulls: false, lineInterpolation: linear, lineWidth: 1, pointSize: 5, scaleDistribution: { type: linear }, showPoints: never, showValues: false, spanNulls: false, stacking: { group: A, mode: none }, thresholdsStyle: { mode: off } }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] }, unit: decbytes }, overrides: [] }, gridPos: { h: 8, w: 12, x: 12, y: 8 }, id: 6, options: { legend: { calcs: [], displayMode: list, placement: bottom, showLegend: true }, tooltip: { hideZeros: false, mode: single, sort: none } }, pluginVersion: 12.4.1, targets: [ { datasource: { type: prometheus, uid: prometheus }, editorMode: code, expr: histogram_quantile(0.90, sum by (le, http_route, service_name) (rate(http_server_response_body_size_bytes_bucket{service_name!~\lgtm|\}[2m])))\r\n, instant: true, legendFormat: __auto, range: true, refId: A } ], title: 返回包大小, type: timeseries }, { datasource: { type: prometheus, uid: prometheus }, fieldConfig: { defaults: { color: { mode: palette-classic }, custom: { axisBorderShow: false, axisCenteredZero: false, axisColorMode: text, axisLabel: , axisPlacement: auto, barAlignment: 0, barWidthFactor: 0.6, drawStyle: line, fillOpacity: 0, gradientMode: none, hideFrom: { legend: false, tooltip: false, viz: false }, insertNulls: false, lineInterpolation: linear, lineWidth: 1, pointSize: 5, scaleDistribution: { type: linear }, showPoints: never, showValues: false, spanNulls: false, stacking: { group: A, mode: none }, thresholdsStyle: { mode: off } }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] }, unit: reqps }, overrides: [] }, gridPos: { h: 8, w: 12, x: 0, y: 16 }, id: 7, options: { legend: { calcs: [], displayMode: list, placement: bottom, showLegend: true }, tooltip: { hideZeros: false, mode: single, sort: none } }, pluginVersion: 12.4.1, targets: [ { datasource: { type: prometheus, uid: prometheus }, editorMode: code, expr: sum by (client, server) (rate(traces_service_graph_request_failed_total[2m])), instant: true, legendFormat: __auto, range: true, refId: A } ], title: 失败次数/秒, type: timeseries }, { datasource: { type: prometheus, uid: prometheus }, fieldConfig: { defaults: { color: { mode: palette-classic }, custom: { axisBorderShow: false, axisCenteredZero: false, axisColorMode: text, axisLabel: , axisPlacement: auto, barAlignment: 0, barWidthFactor: 0.6, drawStyle: line, fillOpacity: 0, gradientMode: none, hideFrom: { legend: false, tooltip: false, viz: false }, insertNulls: false, lineInterpolation: linear, lineWidth: 1, pointSize: 5, scaleDistribution: { type: linear }, showPoints: never, showValues: false, spanNulls: false, stacking: { group: A, mode: none }, thresholdsStyle: { mode: off } }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] }, unit: Bps }, overrides: [] }, gridPos: { h: 8, w: 12, x: 12, y: 16 }, id: 8, options: { legend: { calcs: [], displayMode: list, placement: bottom, showLegend: true }, tooltip: { hideZeros: false, mode: single, sort: none } }, pluginVersion: 12.4.1, targets: [ { datasource: { type: prometheus, uid: prometheus }, editorMode: code, expr: sum by (service) (rate(traces_spanmetrics_size_total[2m])), instant: true, legendFormat: __auto, range: true, refId: A } ], title: 总流量, type: timeseries }, { datasource: { type: tempo, uid: tempo }, fieldConfig: { defaults: {}, overrides: [] }, gridPos: { h: 8, w: 12, x: 0, y: 24 }, id: 9, options: { edges: {}, layoutAlgorithm: layered, nodes: {}, zoomMode: cooperative }, pluginVersion: 12.4.1, targets: [ { datasource: { type: tempo, uid: tempo }, key: Q-b5c0b572-5c72-48f7-a7fe-eefba712e204-0, limit: 20, metricsQueryType: range, queryType: serviceMap, refId: A, serviceMapUseNativeHistograms: false, tableType: traces } ], title: 服务调用图, type: nodeGraph }, { datasource: { type: tempo, uid: tempo }, fieldConfig: { defaults: { custom: { align: auto, cellOptions: { type: auto }, footer: { reducers: [] }, inspect: false }, mappings: [], thresholds: { mode: absolute, steps: [ { color: green, value: 0 }, { color: red, value: 80 } ] } }, overrides: [] }, gridPos: { h: 8, w: 12, x: 12, y: 24 }, id: 4, options: { cellHeight: sm, showHeader: true }, pluginVersion: 12.4.1, targets: [ { datasource: { type: tempo, uid: tempo }, filters: [ { id: 3aaf9885, operator: , scope: span }, { id: service-name, isCustomValue: false, operator: !, scope: resource, tag: service.name, value: [ lgtm ], valueType: string } ], key: Q-a3d09f2b-d713-4e41-964b-de64d135e60a-0, limit: 20, metricsQueryType: range, queryType: traceqlSearch, refId: A, serviceMapUseNativeHistograms: false, tableType: traces } ], title: Traces, type: table } ], preload: false, schemaVersion: 42, tags: [], templating: { list: [] }, time: { from: now-1h, to: now }, timepicker: {}, timezone: browser, title: OBI dashboard, uid: ad8krnh, version: 18, weekStart: }5. 总结OpenTelemetry OBI 不是要取代用户现有的所有监控 SDK而是作为一种强大的补充将可观测性的覆盖范围扩展到以前难以触及的角落。对于程序员它意味着更少的代码侵入和更快的反馈循环对于系统管理员它意味着一个安全、统一且易于运维的观测层。结合两者你才能构建起一个真正无死角的全栈可观测性体系。