添加qiming-rcoder模块
This commit is contained in:
Codex
449
qiming-rcoder/docker/rcoder-agent-runner/ebpf-tools/README.md
Normal file
449
qiming-rcoder/docker/rcoder-agent-runner/ebpf-tools/README.md
Normal file
@@ -0,0 +1,449 @@
|
||||
# eBPF 诊断工具
|
||||
|
||||
本目录包含用于监控和诊断 `agent_runner` 及其子进程性能的 eBPF 工具。
|
||||
|
||||
## 📊 监控架构总览
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ 进程性能监控完整方案 │
|
||||
├─────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ CPU 性能监控（持续） │ │
|
||||
│ │ │ │
|
||||
│ │ Grafana Alloy (eBPF) → Pyroscope Server → Web UI (4040) │ │
|
||||
│ │ - 97 Hz 采样率 │ │
|
||||
│ │ - 每 15 秒发送数据 │ │
|
||||
│ │ - 自动发现进程 │ │
|
||||
│ │ - 支持历史查询 │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ ↓ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ 进程指标监控（持续） │ │
|
||||
│ │ │ │
|
||||
│ │ Alloy Process Exporter → Prometheus → Grafana Dashboard │ │
|
||||
│ │ - CPU、内存、I/O、FD、线程数 │ │
|
||||
│ │ - 15 秒采集间隔 │ │
|
||||
│ │ - 时序数据存储 │ │
|
||||
│ │ - Dashboard 可视化 (3000) │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ ↓ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Off-CPU 阻塞监控（定期） │ │
|
||||
│ │ │ │
|
||||
│ │ offcputime-bpfcc → SVG 火焰图文件 │ │
|
||||
│ │ - 每 60 秒生成一次 │ │
|
||||
│ │ - 显示阻塞堆栈 │ │
|
||||
│ │ - 识别 I/O、锁、等待等阻塞 │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ ↓ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ 系统调用监控（持续） │ │
|
||||
│ │ │ │
|
||||
│ │ syscount-bpfcc → 统计文件 │ │
|
||||
│ │ execsnoop-bpfcc → 进程创建日志 │ │
|
||||
│ │ opensnoop-bpfcc → 文件访问日志 │ │
|
||||
│ │ - 每 60 秒统计一次 │ │
|
||||
│ │ - 持续追踪进程和文件访问 │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ ↓ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ 手动诊断工具（按需使用） │ │
|
||||
│ │ │ │
|
||||
│ │ diag-tool.sh - 综合诊断工具 │ │
|
||||
│ │ auto-flamegraph.sh - 自动火焰图生成 │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
ebpf-tools/
|
||||
├── README.md # 本文档
|
||||
├── alloy-config.alloy # Grafana Alloy 配置（CPU 监控 + 进程指标导出）
|
||||
├── diag-tool.sh # 手动诊断工具
|
||||
├── auto-flamegraph.sh # 自动火焰图生成
|
||||
├── offcpu-monitor.sh # Off-CPU 阻塞监控
|
||||
└── syscall-monitor.sh # 系统调用监控
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 工具详解
|
||||
|
||||
### 1. alloy-config.alloy - Grafana Alloy 配置
|
||||
|
||||
**用途**: 持续 CPU 性能监控 + 进程指标导出
|
||||
|
||||
**工作原理**:
|
||||
- 使用 eBPF 自动发现 `agent_runner` 及其子进程
|
||||
- 以 97 Hz 频率采样 CPU 性能数据，发送到 Pyroscope
|
||||
- 采集进程指标（CPU、内存、I/O、FD、线程数），发送到 Prometheus
|
||||
- 在 Web UI 中实时查看：Pyroscope (4040) + Grafana (3000)
|
||||
|
||||
**采集的指标**:
|
||||
| 指标类别 | 指标名称 | 说明 |
|
||||
|---------|---------|------|
|
||||
| **CPU** | `process_cpu_seconds_total` | CPU 时间（累计） |
|
||||
| **内存** | `process_resident_memory_bytes` | 常驻内存（RSS） |
|
||||
| **内存** | `process_virtual_memory_bytes` | 虚拟内存（VSZ） |
|
||||
| **FD** | `process_open_fds` | 打开的文件描述符 |
|
||||
| **I/O** | `process_read_bytes_total` | 读取字节数 |
|
||||
| **I/O** | `process_write_bytes_total` | 写入字节数 |
|
||||
| **线程** | `process_num_threads` | 线程数量 |
|
||||
| **上下文切换** | `process_context_switches_total` | 上下文切换次数 |
|
||||
|
||||
**进程标签**:
|
||||
- `process_pid`: 进程 PID
|
||||
- `process_name`: 进程名称
|
||||
- `process_exe`: 完整执行路径
|
||||
- `parent_pid`: 父进程 PID
|
||||
- `project_id`: 项目 ID
|
||||
- `container_id`: 容器 ID
|
||||
|
||||
**环境变量**:
|
||||
| 变量 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `ENABLE_ALLOY` | - | 是否启用（由 ebpf-debug feature 控制） |
|
||||
| `PYROSCOPE_URL` | http://pyroscope:4040 | Pyroscope Server 地址 |
|
||||
| `PROMETHEUS_URL` | http://prometheus:9090/api/v1/write | Prometheus 写入地址 |
|
||||
|
||||
**查看数据**:
|
||||
```bash
|
||||
# 1. Pyroscope Web UI (CPU 性能)
|
||||
open http://localhost:4040
|
||||
|
||||
# 2. Grafana Dashboard (进程指标)
|
||||
open http://localhost:3000
|
||||
# 登录: admin / admin
|
||||
# 查找: "Agent Runner 进程监控"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. offcpu-monitor.sh - Off-CPU 阻塞监控
|
||||
|
||||
**用途**: 定期生成 Off-CPU 阻塞火焰图，分析进程阻塞原因
|
||||
|
||||
**工作原理**:
|
||||
- 每 60 秒自动运行一次（可配置）
|
||||
- 对 `agent_runner` 及其所有子进程进行采样
|
||||
- 使用 `offcputime-bpfcc` 捕获阻塞堆栈
|
||||
- 生成 SVG 火焰图文件，自动清理旧文件（最多保留 50 个）
|
||||
|
||||
**环境变量**:
|
||||
| 变量 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `ENABLE_OFFCPUTIME` | - | 是否启用（由 ebpf-debug feature 控制） |
|
||||
| `OFFCPU_DURATION` | 30 | 每次采样时长（秒） |
|
||||
| `OFFCPU_INTERVAL` | 60 | 生成间隔（秒） |
|
||||
| `MAX_OFFCPU_FILES` | 50 | 最多保留文件数量 |
|
||||
|
||||
**输出文件**:
|
||||
```
|
||||
/app/container-logs/diag/
|
||||
├── offcpu-monitor.log # 监控日志
|
||||
├── offcpu-agent_runner-1-20250111_143025.svg # 主进程阻塞火焰图
|
||||
└── offcpu-claude-code-acp-123-*.svg # 子进程阻塞火焰图
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. syscall-monitor.sh - 系统调用监控
|
||||
|
||||
**用途**: 监控进程的系统调用活动，包括进程创建、文件访问和系统调用统计
|
||||
|
||||
**工作原理**:
|
||||
- 每 60 秒统计一次系统调用（可配置）
|
||||
- 后台持续追踪进程创建 (`execsnoop-bpfcc`)
|
||||
- 后台持续追踪文件访问 (`opensnoop-bpfcc`)
|
||||
- 定期生成系统调用统计报告
|
||||
|
||||
**采集的数据**:
|
||||
| 工具 | 数据 | 说明 |
|
||||
|------|------|------|
|
||||
| `syscount-bpfcc` | 系统调用统计 | 每个系统调用的次数和耗时 |
|
||||
| `execsnoop-bpfcc` | 进程创建日志 | 新进程的创建时间和命令行 |
|
||||
| `opensnoop-bpfcc` | 文件访问日志 | 文件打开/关闭操作 |
|
||||
|
||||
**环境变量**:
|
||||
| 变量 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `ENABLE_SYSCALL_MONITOR` | - | 是否启用（由 ebpf-debug feature 控制） |
|
||||
| `SAMPLE_DURATION` | 30 | 每次采样时长（秒） |
|
||||
| `GENERATE_INTERVAL` | 60 | 生成间隔（秒） |
|
||||
|
||||
**输出文件**:
|
||||
```
|
||||
/app/container-logs/diag/
|
||||
├── syscall-monitor.log # 监控日志
|
||||
├── syscall-count-agent_runner-1-*.txt # 系统调用统计
|
||||
├── execsnoop-*.log # 进程创建日志
|
||||
└── opensnoop-*.log # 文件访问日志
|
||||
```
|
||||
|
||||
**日志说明**:
|
||||
- **控制台输出**: 每次采样只输出一行汇总日志
|
||||
- **文件日志**: 详细日志写入 `syscall-monitor.log`
|
||||
|
||||
---
|
||||
|
||||
### 4. diag-tool.sh - 手动诊断工具
|
||||
|
||||
手动触发的 eBPF 诊断工具，用于在怀疑有性能问题时主动采集数据。
|
||||
|
||||
**用法**:
|
||||
```bash
|
||||
diag-tool.sh {offcpu|flame|profile|all} <pid> [duration]
|
||||
```
|
||||
|
||||
**命令**:
|
||||
| 命令 | 说明 |
|
||||
|------|------|
|
||||
| `offcpu <pid> [duration]` | 分析 off-cpu 堆栈，默认 30 秒 |
|
||||
| `flame <pid> [duration]` | 生成火焰图，默认 30 秒 |
|
||||
| `profile <pid> [duration]` | CPU 性能分析，默认 30 秒 |
|
||||
| `all <pid>` | 综合诊断（包含所有分析） |
|
||||
|
||||
**快捷命令**:
|
||||
- `e-offcpu` - 等同于 `diag-tool.sh offcpu`
|
||||
- `e-flame` - 等同于 `diag-tool.sh flame`
|
||||
- `e-profile` - 等同于 `diag-tool.sh profile`
|
||||
- `e-all` - 等同于 `diag-tool.sh all`
|
||||
|
||||
---
|
||||
|
||||
### 5. auto-flamegraph.sh - 自动火焰图生成
|
||||
|
||||
持续在后台运行的火焰图生成工具，自动监控 `agent_runner` 及其所有子进程。
|
||||
|
||||
**工作原理**:
|
||||
1. 自动检测 `agent_runner` 进程 PID
|
||||
2. 递归获取所有子进程 PID
|
||||
3. 使用 bpftrace 采样性能数据（默认 30 秒）
|
||||
4. 生成火焰图 SVG 文件
|
||||
5. 每 60 秒重复一次
|
||||
|
||||
---
|
||||
|
||||
## 🎯 使用场景
|
||||
|
||||
### 场景 1: 持续监控 CPU 性能
|
||||
|
||||
**目标**: 在 Web UI 中实时查看 agent_runner 性能数据
|
||||
|
||||
**步骤**:
|
||||
1. 确保容器已启动（Alloy 自动运行）
|
||||
2. 打开 http://localhost:4040 (Pyroscope)
|
||||
3. 选择应用 `agent_runner`
|
||||
4. 按需过滤标签（如 `process_name="agent_runner"`）
|
||||
5. 查看火焰图、时序数据、Top 函数
|
||||
|
||||
**适用问题**:
|
||||
- CPU 使用率异常
|
||||
- 函数调用热点分析
|
||||
- 性能回归检测
|
||||
|
||||
---
|
||||
|
||||
### 场景 2: 查看进程指标趋势
|
||||
|
||||
**目标**: 监控内存、I/O、FD 等进程指标的趋势
|
||||
|
||||
**步骤**:
|
||||
1. 打开 http://localhost:3000 (Grafana)
|
||||
2. 登录（admin / admin）
|
||||
3. 查找 "Agent Runner 进程监控" Dashboard
|
||||
4. 选择 `project_id`、`instance`、`process_name` 过滤
|
||||
5. 查看各面板数据
|
||||
|
||||
**可用面板**:
|
||||
- 概览: RSS/VSZ 内存、CPU 使用率、文件描述符
|
||||
- 内存趋势: RSS 和 VSZ 的时间序列图
|
||||
- I/O 监控: 读取/写入速率
|
||||
- 上下文切换: 自愿/非自愿切换速率
|
||||
- 线程详情: 线程数量、FD 使用率
|
||||
- 缺页错误: 次要/主要缺页错误速率
|
||||
|
||||
---
|
||||
|
||||
### 场景 3: 分析进程阻塞问题
|
||||
|
||||
**目标**: 找出进程为什么被阻塞（等待 I/O、锁等）
|
||||
|
||||
**步骤**:
|
||||
1. 等待 offcpu-monitor 自动生成火焰图（或手动触发）
|
||||
2. 导出 SVG 文件到本地
|
||||
3. 在浏览器中打开火焰图
|
||||
4. 查找宽的阻塞堆栈
|
||||
|
||||
```bash
|
||||
# 导出最新的 Off-CPU 火焰图
|
||||
docker cp <container>:/app/container-logs/diag/offcpu-*.svg ./
|
||||
open offcpu-*.svg
|
||||
```
|
||||
|
||||
**适用问题**:
|
||||
- `new_session` 超时
|
||||
- 进程响应缓慢
|
||||
- I/O 阻塞
|
||||
- 锁竞争
|
||||
|
||||
---
|
||||
|
||||
### 场景 4: 分析系统调用模式
|
||||
|
||||
**目标**: 了解进程的系统调用行为，找出系统调用热点
|
||||
|
||||
**步骤**:
|
||||
1. 查看系统调用统计日志
|
||||
2. 分析哪些系统调用最频繁
|
||||
3. 查看 execsnoop/opensnoop 日志了解进程和文件访问
|
||||
|
||||
```bash
|
||||
# 查看系统调用统计
|
||||
docker exec <container> cat /app/container-logs/diag/syscall-count-*.txt | head -20
|
||||
|
||||
# 查看进程创建日志
|
||||
docker exec <container> tail -f /app/container-logs/diag/execsnoop-*.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 场景 5: 手动诊断已知问题
|
||||
|
||||
```bash
|
||||
# 进入容器
|
||||
docker exec -it <container> bash
|
||||
|
||||
# 诊断 agent_runner
|
||||
e-all $(pgrep agent_runner)
|
||||
|
||||
# 导出结果
|
||||
docker cp <container>:/app/container-logs/diag ./diag-results
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📈 火焰图分析
|
||||
|
||||
### CPU 火焰图（Alloy + Pyroscope）
|
||||
|
||||
**如何阅读**:
|
||||
- **横轴**: CPU 时间占比（越宽表示占用越多）
|
||||
- **纵轴**: 调用堆栈（从上到下是调用关系）
|
||||
- **颜色**: 暖色调表示热点函数
|
||||
|
||||
**典型问题识别**:
|
||||
| 现象 | 可能原因 |
|
||||
|------|----------|
|
||||
| 某个函数占据大部分宽度 | CPU 密集型计算 |
|
||||
| 深层函数很宽 | 递归调用或深层嵌套 |
|
||||
| 出现 `syscall` 大量时间 | 系统调用开销 |
|
||||
| 出现 `sleep`/`usleep` | 主动休眠或等待 |
|
||||
|
||||
### Off-CPU 火焰图（offcputime-bpfcc）
|
||||
|
||||
**如何阅读**:
|
||||
- **横轴**: 阻塞时间占比（越宽表示阻塞越久）
|
||||
- **纵轴**: 阻塞时的调用堆栈
|
||||
- **颜色**: 暖色调表示阻塞热点
|
||||
|
||||
**典型问题识别**:
|
||||
| 现象 | 可能原因 |
|
||||
|------|----------|
|
||||
| `schedule()` 占据大量时间 | 进程被调度出去（CPU 竞争） |
|
||||
| `do_wait()`/`wait_event()` | 等待事件或信号 |
|
||||
| `__sock_sendmsg()`/`__sock_recvmsg()` | 网络 I/O 阻塞 |
|
||||
| `blk_mq_submit_bio()` | 磁盘 I/O 阻塞 |
|
||||
| `futex_wait()` | 锁等待（互斥锁） |
|
||||
|
||||
---
|
||||
|
||||
## 🔍 故障排查
|
||||
|
||||
### Grafana Dashboard 显示 "No Data"
|
||||
|
||||
```bash
|
||||
# 1. 检查 Prometheus 中是否有数据
|
||||
curl -s 'http://localhost:9091/api/v1/query?query=process_resident_memory_bytes'
|
||||
|
||||
# 2. 检查标签值
|
||||
curl -s 'http://localhost:9091/api/v1/label/project_id/values'
|
||||
|
||||
# 3. 确认 agent_runner 容器正在运行
|
||||
docker ps | grep agent_runner
|
||||
```
|
||||
|
||||
### 变量下拉框为空
|
||||
|
||||
```bash
|
||||
# 检查标签值是否存在
|
||||
curl -s 'http://localhost:9091/api/v1/label/project_id/values'
|
||||
|
||||
# 如果没有数据，说明没有 agent_runner 容器在运行
|
||||
# 启动一个 agent_runner 容器后再检查
|
||||
```
|
||||
|
||||
### Pyroscope Web UI 无数据
|
||||
|
||||
```bash
|
||||
# 1. 检查 Pyroscope Server
|
||||
docker ps | grep pyroscope
|
||||
docker logs rcoder-pyroscope
|
||||
|
||||
# 2. 检查 Alloy 是否发送数据
|
||||
docker exec <container> grep "collected profiles" /app/container-logs/diag/alloy.log
|
||||
|
||||
# 3. 检查网络连接
|
||||
docker exec <container> curl http://pyroscope:4040
|
||||
```
|
||||
|
||||
### Off-CPU 火焰图未生成
|
||||
|
||||
```bash
|
||||
# 1. 检查 offcputime-bpfcc 是否可用
|
||||
docker exec <container> which offcputime-bpfcc
|
||||
|
||||
# 2. 检查监控进程
|
||||
docker exec <container> ps aux | grep offcpu-monitor
|
||||
|
||||
# 3. 查看监控日志
|
||||
docker exec <container> tail -f /app/container-logs/diag/offcpu-monitor.log
|
||||
```
|
||||
|
||||
### 系统调用监控无输出
|
||||
|
||||
```bash
|
||||
# 1. 检查 syscount-bpfcc 是否可用
|
||||
docker exec <container> which syscount-bpfcc
|
||||
|
||||
# 2. 检查监控脚本是否运行
|
||||
docker exec <container> ps aux | grep syscall-monitor
|
||||
|
||||
# 3. 查看监控日志
|
||||
docker exec <container> tail -f /app/container-logs/diag/syscall-monitor.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 安全注意事项
|
||||
|
||||
⚠️ **eBPF 工具需要容器特权模式运行**，仅在受信任的调试环境使用！
|
||||
|
||||
生产环境请使用 `make docker-build-agent-production` 构建无 eBPF 工具的镜像。
|
||||
|
||||
---
|
||||
|
||||
## 📚 相关文档
|
||||
|
||||
- [Grafana Alloy Documentation](https://grafana.com/docs/alloy/latest/)
|
||||
- [Pyroscope Documentation](https://pyroscope.io/docs/)
|
||||
- [Prometheus Documentation](https://prometheus.io/docs/)
|
||||
- [Grafana Documentation](https://grafana.com/docs/)
|
||||
- [Brendan Gregg's FlameGraph](https://github.com/brendangregg/FlameGraph)
|
||||
- [bpftrace 参考指南](https://bpftrace.dev/)
|
||||
- 主项目文档: `/docker/README.md`
|
||||
@@ -0,0 +1,142 @@
|
||||
// ============================================================================
|
||||
// Grafana Alloy 配置文件 - 用于监控 agent_runner 进程
|
||||
// ============================================================================
|
||||
// 此配置文件实现：
|
||||
// 1. 自动发现 agent_runner 及其子进程
|
||||
// 2. 使用 eBPF 进行 CPU 性能剖析
|
||||
// 3. 将性能数据发送到 Pyroscope Server
|
||||
// ============================================================================
|
||||
|
||||
// ============================================================================
|
||||
// 进程发现 - 发现所有运行中的进程
|
||||
// ============================================================================
|
||||
discovery.process "all" {
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// 进程过滤 - 只保留 agent_runner 相关进程
|
||||
// ============================================================================
|
||||
discovery.relabel "agent_runner_filter" {
|
||||
targets = discovery.process.all.targets
|
||||
|
||||
// 规则 1: 保留 agent_runner 主进程及其所有子进程
|
||||
// 匹配条件：进程名是 agent_runner 或者父进程 PID 是 1
|
||||
rule {
|
||||
source_labels = ["__meta_process_exe", "__meta_process_ppid"]
|
||||
regex = ".*/agent_runner|;1"
|
||||
action = "keep"
|
||||
}
|
||||
|
||||
// 规则 2: 添加进程 PID 标签
|
||||
rule {
|
||||
source_labels = ["__meta_process_pid"]
|
||||
target_label = "process_pid"
|
||||
}
|
||||
|
||||
// 规则 3: 添加进程名称标签
|
||||
rule {
|
||||
source_labels = ["__meta_process_comm"]
|
||||
target_label = "process_name"
|
||||
}
|
||||
|
||||
// 规则 4: 添加完整执行路径标签
|
||||
rule {
|
||||
source_labels = ["__meta_process_exe"]
|
||||
target_label = "process_exe"
|
||||
}
|
||||
|
||||
// 规则 5: 添加父进程 PID 标签
|
||||
rule {
|
||||
source_labels = ["__meta_process_ppid"]
|
||||
target_label = "parent_pid"
|
||||
}
|
||||
|
||||
// 规则 6: 添加命令行参数标签
|
||||
rule {
|
||||
source_labels = ["__meta_process_cmdline"]
|
||||
target_label = "process_cmdline"
|
||||
}
|
||||
|
||||
// 规则 7: 添加环境标签
|
||||
rule {
|
||||
target_label = "env"
|
||||
replacement = "dev"
|
||||
}
|
||||
|
||||
// 规则 8: 添加容器 ID 标签（从环境变量动态组合）
|
||||
rule {
|
||||
target_label = "container_id"
|
||||
replacement = "agent-${env:USER_ID}"
|
||||
}
|
||||
|
||||
// 规则 9: 添加应用名称标签
|
||||
rule {
|
||||
target_label = "job"
|
||||
replacement = "agent_runner"
|
||||
}
|
||||
|
||||
// 规则 10: 添加实例标签（从环境变量动态组合）
|
||||
rule {
|
||||
target_label = "instance"
|
||||
replacement = "agent-${env:USER_ID}"
|
||||
}
|
||||
|
||||
// 添加集群标签
|
||||
rule {
|
||||
target_label = "cluster"
|
||||
replacement = "rcoder"
|
||||
}
|
||||
|
||||
// 添加数据中心标签
|
||||
rule {
|
||||
target_label = "datacenter"
|
||||
replacement = "local"
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// 写入 Pyroscope Server
|
||||
// ============================================================================
|
||||
pyroscope.write "remote" {
|
||||
endpoint {
|
||||
url = "http://pyroscope:4040"
|
||||
}
|
||||
|
||||
external_labels = {
|
||||
"env" = "dev",
|
||||
"datacenter" = "local",
|
||||
"cluster" = "rcoder",
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// eBPF 性能分析组件
|
||||
// ============================================================================
|
||||
pyroscope.ebpf "agent_runner" {
|
||||
forward_to = [pyroscope.write.remote.receiver]
|
||||
|
||||
targets = discovery.relabel.agent_runner_filter.output
|
||||
|
||||
sample_rate = 97
|
||||
collect_interval = "15s"
|
||||
|
||||
collect_user_profile = true
|
||||
collect_kernel_profile = false
|
||||
|
||||
python_enabled = true
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
// Prometheus Remote Write
|
||||
// ============================================================================
|
||||
prometheus.remote_write "metrics" {
|
||||
endpoint {
|
||||
url = "http://prometheus:9090/api/v1/write"
|
||||
}
|
||||
|
||||
external_labels = {
|
||||
"env" = "dev",
|
||||
"datacenter" = "local",
|
||||
"cluster" = "rcoder",
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,240 @@
|
||||
#!/bin/bash
|
||||
# eBPF 自动持续火焰图生成脚本
|
||||
# 以 agent_runner 为入口，监控其所有子进程，定期生成火焰图
|
||||
|
||||
set -e
|
||||
|
||||
DIAG_OUTPUT_DIR="${DIAG_OUTPUT_DIR:-/app/container-logs/diag}"
|
||||
MONITOR_LOG="$DIAG_OUTPUT_DIR/auto-flamegraph.log"
|
||||
SAMPLE_DURATION=${SAMPLE_DURATION:-30} # 每次采样时长（秒），默认 30 秒
|
||||
GENERATE_INTERVAL=${GENERATE_INTERVAL:-60} # 生成火焰图间隔（秒），默认 60 秒
|
||||
MAX_FLAMEFILES=${MAX_FLAMEFILES:-50} # 最多保留火焰图文件数量
|
||||
|
||||
mkdir -p "$DIAG_OUTPUT_DIR"
|
||||
|
||||
log() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$MONITOR_LOG"
|
||||
}
|
||||
|
||||
# 获取 agent_runner 的 PID
|
||||
get_agent_runner_pid() {
|
||||
pgrep -f "agent_runner" | head -1
|
||||
}
|
||||
|
||||
# 获取 agent_runner 及其所有子进程的 PID 列表
|
||||
get_process_tree_pids() {
|
||||
local parent_pid=$1
|
||||
if [ -z "$parent_pid" ]; then
|
||||
return
|
||||
fi
|
||||
|
||||
# 输出父进程 PID
|
||||
echo "$parent_pid"
|
||||
|
||||
# 递归获取所有子进程
|
||||
local child_pids=$(pgrep -P "$parent_pid" 2>/dev/null || true)
|
||||
for child_pid in $child_pids; do
|
||||
get_process_tree_pids "$child_pid"
|
||||
done
|
||||
}
|
||||
|
||||
# 生成火焰图
|
||||
generate_flamegraph() {
|
||||
local timestamp=$(date '+%Y%m%d_%H%M%S')
|
||||
local bpftrace_data="$DIAG_OUTPUT_DIR/profile-${timestamp}.bt"
|
||||
local folded_output="$DIAG_OUTPUT_DIR/profile-${timestamp}.folded"
|
||||
local flamegraph_svg="$DIAG_OUTPUT_DIR/flamegraph-${timestamp}.svg"
|
||||
local agent_pid=$1
|
||||
|
||||
log "🔥 开始生成火焰图 (agent_runner PID: $agent_pid)..."
|
||||
|
||||
# 获取进程树 PID 列表
|
||||
local pids=$(get_process_tree_pids "$agent_pid" | tr '\n' ' ')
|
||||
log "📋 监控进程树: $pids"
|
||||
|
||||
# 构建 bpftrace 脚本，监控所有相关进程
|
||||
local pid_filter=""
|
||||
for pid in $pids; do
|
||||
if [ -n "$pid_filter" ]; then
|
||||
pid_filter="$pid_filter || "
|
||||
fi
|
||||
pid_filter="${pid_filter}pid == $pid"
|
||||
done
|
||||
|
||||
log "📊 采样 ${SAMPLE_DURATION} 秒..."
|
||||
|
||||
# 使用 bpftrace 采样（后台运行）
|
||||
timeout ${SAMPLE_DURATION}s bpftrace -e "
|
||||
profile:hz:99 /($pid_filter) && comm != \"bpftrace\"/ {
|
||||
@[ustack] = count();
|
||||
}
|
||||
" 2>/dev/null | sort -rn -k2 > "$bpftrace_data" || true
|
||||
|
||||
if [ ! -s "$bpftrace_data" ]; then
|
||||
log "⚠️ 未采集到性能数据（进程可能已结束）"
|
||||
rm -f "$bpftrace_data"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log "📊 采样完成，共 $(wc -l < "$bpftrace_data") 个堆栈样本"
|
||||
|
||||
# 调试：保存原始数据样本（前 10 行）
|
||||
head -10 "$bpftrace_data" > "$DIAG_OUTPUT_DIR/debug-bpftrace-sample.txt"
|
||||
log "📋 原始数据样本已保存到: $DIAG_OUTPUT_DIR/debug-bpftrace-sample.txt"
|
||||
|
||||
# 转换为 FlameGraph 格式
|
||||
log "🔄 转换为火焰图格式..."
|
||||
|
||||
# bpftrace 实际输出格式（多行）:
|
||||
# ]: 10
|
||||
# Attaching 1 probe...
|
||||
# @[
|
||||
# 0x634c5c
|
||||
# 0x634758
|
||||
# ...
|
||||
#
|
||||
# 使用 awk 解析多行格式
|
||||
awk '
|
||||
BEGIN {
|
||||
count = 0
|
||||
in_stack = 0
|
||||
stack = ""
|
||||
}
|
||||
/^]:/ {
|
||||
# 提取计数值: ]: 10
|
||||
count = $2
|
||||
next
|
||||
}
|
||||
/^\@\[/ {
|
||||
# 开始新的堆栈
|
||||
in_stack = 1
|
||||
stack = ""
|
||||
next
|
||||
}
|
||||
/^$/ {
|
||||
# 空行结束当前堆栈
|
||||
if (in_stack && stack != "" && count > 0) {
|
||||
# 移除末尾分号并输出
|
||||
gsub(/;$/, "", stack)
|
||||
print stack " " count
|
||||
}
|
||||
in_stack = 0
|
||||
stack = ""
|
||||
count = 0
|
||||
next
|
||||
}
|
||||
{
|
||||
# 跳过 Attaching 消息等非堆栈行
|
||||
if (in_stack && $0 !~ /^Attaching/) {
|
||||
# 提取地址（缩进或未缩进的十六进制地址）
|
||||
if (match($0, /0x[0-9a-f]+/)) {
|
||||
addr = substr($0, RSTART, RLENGTH)
|
||||
if (stack != "") {
|
||||
stack = stack ";"
|
||||
}
|
||||
stack = stack addr
|
||||
}
|
||||
}
|
||||
}
|
||||
END {
|
||||
# 处理最后一个堆栈
|
||||
if (in_stack && stack != "" && count > 0) {
|
||||
gsub(/;$/, "", stack)
|
||||
print stack " " count
|
||||
}
|
||||
}
|
||||
' "$bpftrace_data" > "$folded_output"
|
||||
|
||||
# 检查转换结果
|
||||
if [ ! -s "$folded_output" ]; then
|
||||
log "⚠️ 火焰图格式转换失败，保存原始数据用于调试"
|
||||
cp "$bpftrace_data" "$DIAG_OUTPUT_DIR/debug-${timestamp}.bt"
|
||||
log "📋 请检查以下文件以诊断问题:"
|
||||
log " - 原始数据: $DIAG_OUTPUT_DIR/debug-${timestamp}.bt"
|
||||
log " - 数据样本: $DIAG_OUTPUT_DIR/debug-bpftrace-sample.txt"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log "📊 转换完成，共 $(wc -l < "$folded_output") 个有效堆栈"
|
||||
|
||||
# 调试：保存转换后的样本（前 5 行）
|
||||
head -5 "$folded_output" > "$DIAG_OUTPUT_DIR/debug-folded-sample.txt"
|
||||
log "📋 转换后样本已保存到: $DIAG_OUTPUT_DIR/debug-folded-sample.txt"
|
||||
|
||||
# 生成火焰图
|
||||
log "🎨 生成 SVG 火焰图..."
|
||||
if command -v flamegraph.pl &> /dev/null; then
|
||||
flamegraph.pl \
|
||||
--title="agent_runner 进程树火焰图 (${timestamp})" \
|
||||
--width=1600 \
|
||||
--height=800 \
|
||||
"$folded_output" > "$flamegraph_svg"
|
||||
|
||||
if [ -s "$flamegraph_svg" ]; then
|
||||
log "✅ 火焰图已生成: $flamegraph_svg"
|
||||
log "💡 复制到宿主机查看: docker cp <container>:$flamegraph_svg ./"
|
||||
|
||||
# 清理临时文件
|
||||
rm -f "$bpftrace_data" "$folded_output"
|
||||
|
||||
# 清理旧火焰图文件（保留最新的 MAX_FLAMEFILES 个）
|
||||
cleanup_old_flamegraphs
|
||||
return 0
|
||||
else
|
||||
log "❌ 火焰图生成失败"
|
||||
return 1
|
||||
fi
|
||||
else
|
||||
log "❌ flamegraph.pl 未安装，无法生成火焰图"
|
||||
log "📋 原始数据保存在: $bpftrace_data"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# 清理旧火焰图文件
|
||||
cleanup_old_flamegraphs() {
|
||||
local flame_count=$(ls -1 "$DIAG_OUTPUT_DIR"/flamegraph-*.svg 2>/dev/null | wc -l)
|
||||
|
||||
if [ "$flame_count" -gt "$MAX_FLAMEFILES" ]; then
|
||||
local delete_count=$((flame_count - MAX_FLAMEFILES))
|
||||
log "🗑️ 清理 ${delete_count} 个旧火焰图文件..."
|
||||
|
||||
ls -1t "$DIAG_OUTPUT_DIR"/flamegraph-*.svg | tail -n "$delete_count" | while read -r file; do
|
||||
log " 删除: $(basename "$file")"
|
||||
rm -f "$file"
|
||||
done
|
||||
fi
|
||||
}
|
||||
|
||||
# 主监控循环
|
||||
monitor_loop() {
|
||||
log "🚀 eBPF 自动火焰图生成已启动"
|
||||
log "📋 配置: 采样时长=${SAMPLE_DURATION}s, 生成间隔=${GENERATE_INTERVAL}s"
|
||||
log "💡 火焰图将保存到: $DIAG_OUTPUT_DIR/flamegraph-*.svg"
|
||||
|
||||
while true; do
|
||||
# 获取 agent_runner PID
|
||||
local agent_pid=$(get_agent_runner_pid)
|
||||
|
||||
if [ -n "$agent_pid" ]; then
|
||||
log "✅ 检测到 agent_runner 进程 (PID: $agent_pid)"
|
||||
generate_flamegraph "$agent_pid"
|
||||
else
|
||||
log "⚠️ 未检测到 agent_runner 进程，等待中..."
|
||||
fi
|
||||
|
||||
log "⏰ ${GENERATE_INTERVAL} 秒后生成下一张火焰图..."
|
||||
sleep "$GENERATE_INTERVAL"
|
||||
done
|
||||
}
|
||||
|
||||
# 启动监控
|
||||
case "$1" in
|
||||
start)
|
||||
monitor_loop
|
||||
;;
|
||||
*)
|
||||
echo "用法: $0 start"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
255
qiming-rcoder/docker/rcoder-agent-runner/ebpf-tools/diag-tool.sh
Normal file
255
qiming-rcoder/docker/rcoder-agent-runner/ebpf-tools/diag-tool.sh
Normal file
@@ -0,0 +1,255 @@
|
||||
#!/bin/bash
|
||||
# eBPF 诊断工具快捷脚本
|
||||
# 使用方法: diag-tool.sh {offcpu|flame|profile|all} <pid> [duration]
|
||||
|
||||
DIAG_OUTPUT_DIR="${DIAG_OUTPUT_DIR:-/app/container-logs/diag}"
|
||||
mkdir -p "$DIAG_OUTPUT_DIR"
|
||||
|
||||
log_info() {
|
||||
echo "$(date '+%Y-%m-%d %H:%M:%S') INFO ℹ️ $*"
|
||||
}
|
||||
|
||||
log_success() {
|
||||
echo "$(date '+%Y-%m-%d %H:%M:%S') INFO ✓ $*"
|
||||
}
|
||||
|
||||
log_error() {
|
||||
echo "$(date '+%Y-%m-%d %H:%M:%S') ERROR ❌ $*"
|
||||
}
|
||||
|
||||
# 检查 eBPF 工具是否可用
|
||||
check_ebpf_tools() {
|
||||
if ! command -v bpftrace &> /dev/null; then
|
||||
log_error "bpftrace 未安装，请检查 Dockerfile 中 INSTALL_EBPF_TOOLS 参数"
|
||||
return 1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# off-cpu 分析 - 定位进程阻塞位置
|
||||
diag_offcpu() {
|
||||
local pid=$1
|
||||
local duration=${2:-30}
|
||||
|
||||
if [ -z "$pid" ]; then
|
||||
log_error "缺少 PID 参数"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if ! kill -0 "$pid" 2>/dev/null; then
|
||||
log_error "进程 $pid 不存在"
|
||||
return 1
|
||||
fi
|
||||
|
||||
check_ebpf_tools || return 1
|
||||
|
||||
log_info "分析进程 $pid 的 CPU 性能堆栈 (${duration}s)..."
|
||||
log_info "输出文件: $DIAG_OUTPUT_DIR/offcpu-${pid}.txt"
|
||||
log_info "提示: 这将显示进程在哪些函数上花费最多 CPU 时间"
|
||||
log_info "注意: 如果进程 CPU 使用率低，可能需要更长的采样时间"
|
||||
|
||||
# 使用 bpftrace 进行 CPU 性能分析（后台运行）
|
||||
timeout ${duration}s bpftrace -e "profile:hz:99 /pid == $pid/ && comm != \"bpftrace\"/ { @[ustack] = count(); }" \
|
||||
2>/dev/null > "$DIAG_OUTPUT_DIR/offcpu-${pid}.bpftrace" &
|
||||
local bpftrace_pid=$!
|
||||
|
||||
# 等待采样完成
|
||||
sleep $duration
|
||||
wait $bpftrace_pid 2>/dev/null
|
||||
|
||||
# 处理输出
|
||||
if [ -s "$DIAG_OUTPUT_DIR/offcpu-${pid}.bpftrace" ]; then
|
||||
cat "$DIAG_OUTPUT_DIR/offcpu-${pid}.bpftrace" | sort -rn -k2 | head -50 > "$DIAG_OUTPUT_DIR/offcpu-${pid}.txt"
|
||||
log_success "性能分析完成: $DIAG_OUTPUT_DIR/offcpu-${pid}.txt"
|
||||
echo ""
|
||||
echo "📊 Top 10 CPU 消耗堆栈:"
|
||||
head -10 "$DIAG_OUTPUT_DIR/offcpu-${pid}.txt"
|
||||
else
|
||||
log_error "性能分析失败（未收集到数据，进程可能空闲或采样时间过短）"
|
||||
log_info "建议: 使用更长的采样时间（如 60 秒）或在进程高负载时分析"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# 生成火焰图
|
||||
diag_flame() {
|
||||
local pid=$1
|
||||
local duration=${2:-30}
|
||||
|
||||
if [ -z "$pid" ]; then
|
||||
log_error "缺少 PID 参数"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if ! kill -0 "$pid" 2>/dev/null; then
|
||||
log_error "进程 $pid 不存在"
|
||||
return 1
|
||||
fi
|
||||
|
||||
check_ebpf_tools || return 1
|
||||
|
||||
# 检查 FlameGraph 工具
|
||||
if ! command -v flamegraph.pl &> /dev/null; then
|
||||
log_error "flamegraph.pl 未安装，请检查 FlameGraph 工具安装"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if ! command -v stackcollapse-perf.pl &> /dev/null; then
|
||||
log_error "stackcollapse-perf.pl 未安装，请检查 FlameGraph 工具安装"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "生成进程 $pid 的火焰图 (${duration}s)..."
|
||||
log_info "输出文件: $DIAG_OUTPUT_DIR/flame-${pid}.svg"
|
||||
log_info "提示: 火焰图可直观显示 CPU 性能瓶颈"
|
||||
|
||||
# 使用 bpftrace 收集数据到临时文件（后台运行）
|
||||
timeout ${duration}s bpftrace -e "profile:hz:99 /pid == $pid/ && comm != \"bpftrace\"/ { @[ustack] = count(); }" \
|
||||
2>/dev/null > "$DIAG_OUTPUT_DIR/flame-${pid}.bpftrace" &
|
||||
local bpftrace_pid=$!
|
||||
|
||||
# 等待采样完成
|
||||
sleep $duration
|
||||
wait $bpftrace_pid 2>/dev/null
|
||||
|
||||
# 处理数据生成火焰图
|
||||
if [ -s "$DIAG_OUTPUT_DIR/flame-${pid}.bpftrace" ]; then
|
||||
cat "$DIAG_OUTPUT_DIR/flame-${pid}.bpftrace" | \
|
||||
stackcollapse-perf.pl 2>/dev/null | \
|
||||
flamegraph.pl > "$DIAG_OUTPUT_DIR/flame-${pid}.svg" 2>/dev/null
|
||||
fi
|
||||
|
||||
if [ $? -eq 0 ] && [ -s "$DIAG_OUTPUT_DIR/flame-${pid}.svg" ]; then
|
||||
log_success "火焰图生成完成: $DIAG_OUTPUT_DIR/flame-${pid}.svg"
|
||||
log_info "将 SVG 文件复制到宿主机查看: docker cp <container>:$DIAG_OUTPUT_DIR/flame-${pid}.svg ./"
|
||||
else
|
||||
log_error "火焰图生成失败（未收集到数据，进程可能空闲或采样时间过短）"
|
||||
log_info "建议: 使用更长的采样时间（如 60 秒）或在进程高负载时分析"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# CPU 性能分析
|
||||
diag_profile() {
|
||||
local pid=$1
|
||||
local duration=${2:-30}
|
||||
|
||||
if [ -z "$pid" ]; then
|
||||
log_error "缺少 PID 参数"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if ! kill -0 "$pid" 2>/dev/null; then
|
||||
log_error "进程 $pid 不存在"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "分析进程 $pid 的 CPU 性能 (${duration}s)..."
|
||||
log_info "输出文件: $DIAG_OUTPUT_DIR/profile-${pid}.txt"
|
||||
|
||||
# 使用 bpftrace 进行性能分析（后台运行）
|
||||
timeout ${duration}s bpftrace -e "profile:hz:99 /pid == $pid/ && comm != \"bpftrace\"/ { @[ustack] = count(); }" \
|
||||
2>/dev/null > "$DIAG_OUTPUT_DIR/profile-${pid}.bpftrace" &
|
||||
local bpftrace_pid=$!
|
||||
|
||||
# 等待采样完成
|
||||
sleep $duration
|
||||
wait $bpftrace_pid 2>/dev/null
|
||||
|
||||
# 处理输出
|
||||
if [ -s "$DIAG_OUTPUT_DIR/profile-${pid}.bpftrace" ]; then
|
||||
cat "$DIAG_OUTPUT_DIR/profile-${pid}.bpftrace" > "$DIAG_OUTPUT_DIR/profile-${pid}.txt"
|
||||
log_success "性能分析完成: $DIAG_OUTPUT_DIR/profile-${pid}.txt"
|
||||
else
|
||||
log_error "性能分析失败（未收集到数据，进程可能空闲或采样时间过短）"
|
||||
log_info "建议: 使用更长的采样时间（如 60 秒）或在进程高负载时分析"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# 综合诊断
|
||||
diag_all() {
|
||||
local pid=$1
|
||||
|
||||
if [ -z "$pid" ]; then
|
||||
log_error "缺少 PID 参数"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if ! kill -0 "$pid" 2>/dev/null; then
|
||||
log_error "进程 $pid 不存在"
|
||||
return 1
|
||||
fi
|
||||
|
||||
log_info "综合诊断进程 $pid..."
|
||||
mkdir -p "$DIAG_OUTPUT_DIR/all-${pid}"
|
||||
|
||||
# 保存当前输出目录
|
||||
local old_output_dir="$DIAG_OUTPUT_DIR"
|
||||
export DIAG_OUTPUT_DIR="$DIAG_OUTPUT_DIR/all-${pid}"
|
||||
|
||||
# 执行各项诊断
|
||||
diag_offcpu $pid 30
|
||||
diag_flame $pid 30
|
||||
diag_profile $pid 30
|
||||
|
||||
# 恢复输出目录
|
||||
export DIAG_OUTPUT_DIR="$old_output_dir"
|
||||
|
||||
log_success "诊断完成，结果保存在: $DIAG_OUTPUT_DIR/all-${pid}/"
|
||||
echo ""
|
||||
echo "📊 诊断结果:"
|
||||
echo " - off-cpu 堆栈: $DIAG_OUTPUT_DIR/all-${pid}/offcpu-${pid}.txt"
|
||||
echo " - 火焰图: $DIAG_OUTPUT_DIR/all-${pid}/flame-${pid}.svg"
|
||||
echo " - CPU 性能: $DIAG_OUTPUT_DIR/all-${pid}/profile-${pid}.txt"
|
||||
echo ""
|
||||
echo "💡 导出所有诊断数据:"
|
||||
echo " docker cp <container>:$DIAG_OUTPUT_DIR/all-${pid} ./diag-results"
|
||||
}
|
||||
|
||||
# 显示用法
|
||||
show_usage() {
|
||||
echo "eBPF 诊断工具"
|
||||
echo ""
|
||||
echo "用法: $0 {offcpu|flame|profile|all} <pid> [duration]"
|
||||
echo ""
|
||||
echo "命令:"
|
||||
echo " offcpu <pid> [duration] - 分析 off-cpu 堆栈（定位阻塞位置），默认 30 秒"
|
||||
echo " flame <pid> [duration] - 生成火焰图，默认 30 秒"
|
||||
echo " profile <pid> [duration] - CPU 性能分析，默认 30 秒"
|
||||
echo " all <pid> - 综合诊断（包含所有分析）"
|
||||
echo ""
|
||||
echo "示例:"
|
||||
echo " $0 offcpu \$(pgrep agent_runner) # 分析 agent_runner 阻塞位置"
|
||||
echo " $0 flame \$(pgrep agent_runner) 60 # 生成 60 秒火焰图"
|
||||
echo " $0 all \$(pgrep agent_runner) # 综合诊断"
|
||||
echo ""
|
||||
echo "输出目录: $DIAG_OUTPUT_DIR"
|
||||
echo ""
|
||||
echo "环境变量:"
|
||||
echo " DIAG_OUTPUT_DIR - 自定义输出目录（默认: /app/container-logs/diag）"
|
||||
echo ""
|
||||
echo "快捷命令:"
|
||||
echo " e-offcpu <pid> - 等同于 diag-tool.sh offcpu"
|
||||
echo " e-flame <pid> - 等同于 diag-tool.sh flame"
|
||||
echo " e-profile <pid> - 等同于 diag-tool.sh profile"
|
||||
echo " e-all <pid> - 等同于 diag-tool.sh all"
|
||||
}
|
||||
|
||||
case "$1" in
|
||||
offcpu)
|
||||
diag_offcpu "$2" "${3:-30}"
|
||||
;;
|
||||
flame)
|
||||
diag_flame "$2" "${3:-30}"
|
||||
;;
|
||||
profile)
|
||||
diag_profile "$2" "${3:-30}"
|
||||
;;
|
||||
all)
|
||||
diag_all "$2"
|
||||
;;
|
||||
*)
|
||||
show_usage
|
||||
;;
|
||||
esac
|
||||
@@ -0,0 +1,120 @@
|
||||
#!/bin/bash
|
||||
# offcputime 自动监控脚本
|
||||
# 专门捕获进程阻塞点堆栈
|
||||
|
||||
set -e
|
||||
|
||||
DIAG_OUTPUT_DIR="${DIAG_OUTPUT_DIR:-/app/container-logs/diag}"
|
||||
OFFCPU_LOG="$DIAG_OUTPUT_DIR/offcpu-monitor.log"
|
||||
SAMPLE_DURATION=${OFFCPU_DURATION:-30} # 每次采样时长
|
||||
GENERATE_INTERVAL=${OFFCPU_INTERVAL:-60} # 生成间隔（秒，默认 1 分钟）
|
||||
MAX_OFFCPU_FILES=${MAX_OFFCPU_FILES:-50} # 最多保留火焰图文件数量
|
||||
|
||||
mkdir -p "$DIAG_OUTPUT_DIR"
|
||||
|
||||
log() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$OFFCPU_LOG"
|
||||
}
|
||||
|
||||
# 静默日志（只写入文件，不输出到控制台）
|
||||
log_silent() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$OFFCPU_LOG"
|
||||
}
|
||||
|
||||
# 获取进程树 PID
|
||||
get_process_tree_pids() {
|
||||
local parent_pid=$1
|
||||
if [ -z "$parent_pid" ]; then
|
||||
return
|
||||
fi
|
||||
|
||||
echo "$parent_pid"
|
||||
|
||||
local child_pids=$(pgrep -P "$parent_pid" 2>/dev/null || true)
|
||||
for child_pid in $child_pids; do
|
||||
get_process_tree_pids "$child_pid"
|
||||
done
|
||||
}
|
||||
|
||||
# 清理旧火焰图文件
|
||||
cleanup_old_offcpu_files() {
|
||||
local offcpu_count=$(ls -1 "$DIAG_OUTPUT_DIR"/offcpu-*.svg 2>/dev/null | wc -l)
|
||||
|
||||
if [ "$offcpu_count" -gt "$MAX_OFFCPU_FILES" ]; then
|
||||
local delete_count=$((offcpu_count - MAX_OFFCPU_FILES))
|
||||
log_silent "🗑️ 清理 ${delete_count} 个旧火焰图文件..."
|
||||
|
||||
ls -1t "$DIAG_OUTPUT_DIR"/offcpu-*.svg 2>/dev/null | tail -n "$delete_count" | while read -r file; do
|
||||
log_silent " 删除: $(basename "$file")"
|
||||
rm -f "$file"
|
||||
done
|
||||
fi
|
||||
}
|
||||
|
||||
# 生成 off-cpu 火焰图（静默模式）
|
||||
generate_offcpu_flamegraph() {
|
||||
local timestamp=$(date '+%Y%m%d_%H%M%S')
|
||||
local agent_pid=$(pgrep -f "agent_runner" | head -1)
|
||||
|
||||
if [ -z "$agent_pid" ]; then
|
||||
log_silent "⚠️ 未检测到 agent_runner 进程"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# 获取进程树 PID
|
||||
local pids=$(get_process_tree_pids "$agent_pid" | tr '\n' ' ')
|
||||
local pid_count=$(echo $pids | wc -w)
|
||||
local success_count=0
|
||||
|
||||
# 为每个 PID 生成 off-cpu 火焰图
|
||||
for pid in $pids; do
|
||||
local comm=$(ps -p "$pid" -o comm= 2>/dev/null || echo "unknown")
|
||||
local output_file="$DIAG_OUTPUT_DIR/offcpu-${comm}-${pid}-${timestamp}.svg"
|
||||
|
||||
# 使用 offcputime-bpfcc 生成火焰图（完全静默）
|
||||
timeout ${SAMPLE_DURATION}s offcputime-bpfcc \
|
||||
-p "$pid" \
|
||||
-f "$output_file" \
|
||||
--full-stacks 2>/dev/null || true
|
||||
|
||||
if [ -s "$output_file" ]; then
|
||||
((success_count++))
|
||||
# 只记录到文件，不输出到控制台
|
||||
log_silent " ✅ $comm ($pid): 已保存火焰图"
|
||||
else
|
||||
rm -f "$output_file"
|
||||
fi
|
||||
done
|
||||
|
||||
# 清理旧火焰图文件
|
||||
cleanup_old_offcpu_files
|
||||
|
||||
# 只在汇总时输出一行日志
|
||||
log "🔍 Off-CPU 阻塞分析完成: ${success_count}/${pid_count} 个进程"
|
||||
}
|
||||
|
||||
# 主监控循环
|
||||
monitor_loop() {
|
||||
log "🚀 Off-CPU 阻塞监控已启动 (每 ${GENERATE_INTERVAL} 秒采样一次)"
|
||||
log "📝 详细日志: $OFFCPU_LOG"
|
||||
|
||||
local iteration=0
|
||||
while true; do
|
||||
((iteration++))
|
||||
generate_offcpu_flamegraph
|
||||
# 只记录到文件，不输出到控制台
|
||||
log_silent "⏰ 第 ${iteration} 次采样完成，${GENERATE_INTERVAL} 秒后进行下一次..."
|
||||
sleep "$GENERATE_INTERVAL"
|
||||
done
|
||||
}
|
||||
|
||||
# 启动监控
|
||||
case "$1" in
|
||||
start)
|
||||
monitor_loop
|
||||
;;
|
||||
*)
|
||||
echo "用法: $0 start"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
@@ -0,0 +1,137 @@
|
||||
#!/bin/bash
|
||||
# 系统调用监控脚本
|
||||
# 使用 bpfcc-tools 追踪进程的系统调用活动
|
||||
|
||||
set -e
|
||||
|
||||
DIAG_OUTPUT_DIR="${DIAG_OUTPUT_DIR:-/app/container-logs/diag}"
|
||||
SYSLOG="$DIAG_OUTPUT_DIR/syscall-monitor.log"
|
||||
SAMPLE_DURATION=${SAMPLE_DURATION:-30} # 每次采样时长（秒）
|
||||
GENERATE_INTERVAL=${GENERATE_INTERVAL:-60} # 生成间隔（秒，默认 1 分钟）
|
||||
|
||||
mkdir -p "$DIAG_OUTPUT_DIR"
|
||||
|
||||
log() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$SYSLOG"
|
||||
}
|
||||
|
||||
# 静默日志（只写入文件，不输出到控制台）
|
||||
log_silent() {
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" >> "$SYSLOG"
|
||||
}
|
||||
|
||||
# 获取进程树 PID
|
||||
get_process_tree_pids() {
|
||||
local parent_pid=$1
|
||||
if [ -z "$parent_pid" ]; then
|
||||
return
|
||||
fi
|
||||
|
||||
echo "$parent_pid"
|
||||
|
||||
local child_pids=$(pgrep -P "$parent_pid" 2>/dev/null || true)
|
||||
for child_pid in $child_pids; do
|
||||
get_process_tree_pids "$child_pid"
|
||||
done
|
||||
}
|
||||
|
||||
# 系统调用计数统计（静默模式）
|
||||
collect_syscall_counts() {
|
||||
local pids="$1"
|
||||
local timestamp=$(date '+%Y%m%d_%H%M%S')
|
||||
local success_count=0
|
||||
local total_count=0
|
||||
|
||||
for pid in $pids; do
|
||||
((total_count++))
|
||||
local comm=$(ps -p "$pid" -o comm= 2>/dev/null || echo "unknown")
|
||||
local output_file="$DIAG_OUTPUT_DIR/syscall-count-${comm}-${pid}-${timestamp}.txt"
|
||||
|
||||
# 使用 syscount-bpfcc 统计系统调用（完全静默）
|
||||
timeout ${SAMPLE_DURATION}s syscount-bpfcc -p "$pid" -s 2>/dev/null > "$output_file" || true
|
||||
|
||||
if [ -s "$output_file" ]; then
|
||||
((success_count++))
|
||||
# 只记录到文件，不输出到控制台
|
||||
log_silent " ✅ $comm ($pid): 已保存统计"
|
||||
else
|
||||
rm -f "$output_file"
|
||||
fi
|
||||
done
|
||||
|
||||
# 只在汇总时输出一行日志
|
||||
log "🔍 系统调用统计完成: ${success_count}/${total_count} 个进程"
|
||||
}
|
||||
|
||||
# 进程创建追踪（静默启动）
|
||||
trace_process_creation() {
|
||||
local output_file="$DIAG_OUTPUT_DIR/execsnoop-$(date '+%Y%m%d_%H%M%S').log"
|
||||
|
||||
# 后台运行 execsnoop-bpfcc（完全静默）
|
||||
execsnoop-bpfcc -t -n 1 > "$output_file" 2>/dev/null &
|
||||
local execsnoop_pid=$!
|
||||
|
||||
log_silent "✅ execsnoop-bpfcc 已启动 (PID: $execsnoop_pid)"
|
||||
echo "$execsnoop_pid"
|
||||
}
|
||||
|
||||
# 文件访问追踪（静默启动）
|
||||
trace_file_access() {
|
||||
local output_file="$DIAG_OUTPUT_DIR/opensnoop-$(date '+%Y%m%d_%H%M%S').log"
|
||||
|
||||
# 后台运行 opensnoop-bpfcc（完全静默）
|
||||
opensnoop-bpfcc -t -n 1 > "$output_file" 2>/dev/null &
|
||||
local opensnoop_pid=$!
|
||||
|
||||
log_silent "✅ opensnoop-bpfcc 已启动 (PID: $opensnoop_pid)"
|
||||
echo "$opensnoop_pid"
|
||||
}
|
||||
|
||||
# 主监控循环
|
||||
monitor_loop() {
|
||||
log "🚀 系统调用监控已启动 (每 ${GENERATE_INTERVAL} 秒采样一次)"
|
||||
log "📝 详细日志: $SYSLOG"
|
||||
|
||||
# 启动持续追踪（静默）
|
||||
local execsnoop_pid=$(trace_process_creation)
|
||||
local opensnoop_pid=$(trace_file_access)
|
||||
|
||||
# 清理函数
|
||||
cleanup() {
|
||||
log "🛑 停止系统调用监控..."
|
||||
kill "$execsnoop_pid" "$opensnoop_pid" 2>/dev/null || true
|
||||
log "✅ 系统调用监控已停止"
|
||||
}
|
||||
|
||||
trap cleanup EXIT TERM INT
|
||||
|
||||
# 定期生成系统调用统计
|
||||
local iteration=0
|
||||
while true; do
|
||||
((iteration++))
|
||||
local agent_pid=$(pgrep -f "agent_runner" | head -1)
|
||||
|
||||
if [ -n "$agent_pid" ]; then
|
||||
local pids=$(get_process_tree_pids "$agent_pid" | tr '\n' ' ')
|
||||
# 每次只输出一行汇总日志
|
||||
collect_syscall_counts "$pids"
|
||||
else
|
||||
log_silent "⚠️ 未检测到 agent_runner 进程"
|
||||
fi
|
||||
|
||||
# 只记录到文件，不输出到控制台
|
||||
log_silent "⏰ 第 ${iteration} 次采样完成，${GENERATE_INTERVAL} 秒后进行下一次..."
|
||||
sleep "$GENERATE_INTERVAL"
|
||||
done
|
||||
}
|
||||
|
||||
# 启动监控
|
||||
case "$1" in
|
||||
start)
|
||||
monitor_loop
|
||||
;;
|
||||
*)
|
||||
echo "用法: $0 start"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
Reference in New Issue
Block a user