添加qiming-rcoder模块
This commit is contained in:
329
qiming-rcoder/.qoder/repowiki/zh/content/故障排除/安全考虑.md
Normal file
329
qiming-rcoder/.qoder/repowiki/zh/content/故障排除/安全考虑.md
Normal file
@@ -0,0 +1,329 @@
|
||||
# 安全考虑
|
||||
|
||||
<cite>
|
||||
**本文引用的文件**
|
||||
- [docker/Dockerfile](file://docker/Dockerfile)
|
||||
- [scripts/setup-network-isolation.sh](file://scripts/setup-network-isolation.sh)
|
||||
- [docker/docker-compose.yml](file://docker/docker-compose.yml)
|
||||
- [config.yml](file://config.yml)
|
||||
- [crates/rcoder/src/main.rs](file://crates/rcoder/src/main.rs)
|
||||
- [crates/agent_runner/src/main.rs](file://crates/agent_runner/src/main.rs)
|
||||
- [crates/rcoder/src/config.rs](file://crates/rcoder/src/config.rs)
|
||||
- [crates/agent_runner/src/config.rs](file://crates/agent_runner/src/config.rs)
|
||||
- [crates/rcoder/src/router.rs](file://crates/rcoder/src/router.rs)
|
||||
- [crates/agent_runner/src/router.rs](file://crates/agent_runner/src/router.rs)
|
||||
- [crates/rcoder/src/handler/chat_handler.rs](file://crates/rcoder/src/handler/chat_handler.rs)
|
||||
- [crates/agent_runner/src/handler/chat_handler.rs](file://crates/agent_runner/src/handler/chat_handler.rs)
|
||||
- [crates/rcoder/src/middleware/tracing_middleware.rs](file://crates/rcoder/src/middleware/tracing_middleware.rs)
|
||||
- [crates/agent_runner/src/middleware/tracing_middleware.rs](file://crates/agent_runner/src/middleware/tracing_middleware.rs)
|
||||
- [crates/docker_manager/src/manager.rs](file://crates/docker_manager/src/manager.rs)
|
||||
- [crates/shared_types/src/multi_image_config.rs](file://crates/shared_types/src/multi_image_config.rs)
|
||||
- [crates/shared_types/src/service_config.rs](file://crates/shared_types/src/service_config.rs)
|
||||
- [crates/pingora-proxy/src/service.rs](file://crates/pingora-proxy/src/service.rs)
|
||||
- [crates/agent_runner/src/utils/system_prompt.rs](file://crates/agent_runner/src/utils/system_prompt.rs)
|
||||
</cite>
|
||||
|
||||
## 目录
|
||||
1. [简介](#简介)
|
||||
2. [项目结构与安全边界](#项目结构与安全边界)
|
||||
3. [核心安全组件](#核心安全组件)
|
||||
4. [架构总览](#架构总览)
|
||||
5. [详细组件安全分析](#详细组件安全分析)
|
||||
6. [依赖关系与耦合分析](#依赖关系与耦合分析)
|
||||
7. [性能与安全权衡](#性能与安全权衡)
|
||||
8. [故障排查与安全审计](#故障排查与安全审计)
|
||||
9. [结论](#结论)
|
||||
|
||||
## 简介
|
||||
本章节系统阐述 RCoder 在部署与运行中的安全实践,围绕容器隔离、挂载权限控制、镜像来源验证、API 访问控制、输入验证与防滥用、敏感信息管理、网络隔离脚本、日志与审计、漏洞扫描与最小权限原则等方面展开。文档旨在帮助运维与开发人员在保障功能完备的同时,提升系统的安全性与可审计性。
|
||||
|
||||
## 项目结构与安全边界
|
||||
- 容器运行时:通过 Docker Compose 启动主服务与代理服务,主服务暴露端口并可选启用反向代理;Agent Runner 作为容器内服务参与聊天处理。
|
||||
- 网络隔离:提供宿主机侧网络隔离脚本,基于 iptables 限制容器对内网地址段的访问。
|
||||
- 配置管理:支持命令行参数、环境变量与配置文件的多级覆盖;敏感信息建议通过环境变量注入,避免明文写入配置文件。
|
||||
- 日志与可观测性:内置 OpenTelemetry 与结构化日志,支持按天滚动与 trace_id 传播,便于审计与定位问题。
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "宿主机"
|
||||
DC["Docker Compose<br/>rcoder 服务"]
|
||||
NI["网络隔离脚本<br/>iptables 规则"]
|
||||
LOG["日志目录<br/>logs/"]
|
||||
end
|
||||
subgraph "容器内"
|
||||
RC["rcoder 主服务"]
|
||||
AR["agent_runner 服务"]
|
||||
PM["Pingora 反向代理"]
|
||||
DM["Docker 管理器"]
|
||||
end
|
||||
DC --> RC
|
||||
RC --> PM
|
||||
RC --> DM
|
||||
RC --> AR
|
||||
NI -.-> DC
|
||||
LOG -.-> RC
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [docker/docker-compose.yml](file://docker/docker-compose.yml#L1-L37)
|
||||
- [crates/rcoder/src/main.rs](file://crates/rcoder/src/main.rs#L210-L272)
|
||||
- [crates/agent_runner/src/main.rs](file://crates/agent_runner/src/main.rs#L120-L178)
|
||||
- [crates/pingora-proxy/src/service.rs](file://crates/pingora-proxy/src/service.rs#L556-L596)
|
||||
|
||||
章节来源
|
||||
- [docker/docker-compose.yml](file://docker/docker-compose.yml#L1-L37)
|
||||
- [crates/rcoder/src/main.rs](file://crates/rcoder/src/main.rs#L210-L272)
|
||||
- [crates/agent_runner/src/main.rs](file://crates/agent_runner/src/main.rs#L120-L178)
|
||||
|
||||
## 核心安全组件
|
||||
- 容器隔离与能力控制:容器启动时丢弃网络相关能力,禁用特权模式,降低逃逸风险。
|
||||
- 网络隔离脚本:在宿主机 iptables 上为 rcoder- 前缀网络添加规则,阻止访问常见内网地址段。
|
||||
- 配置与环境变量:支持环境变量覆盖端口、工作目录、网络模式、自动清理、容器 TTL 等,敏感信息建议通过环境变量注入。
|
||||
- 日志与追踪:结构化 JSON 日志按天滚动,trace_id 传播,便于审计与关联分析。
|
||||
- 反向代理与健康检查:Pingora 代理提供后端健康检查与统计接口,便于监控与故障定位。
|
||||
|
||||
章节来源
|
||||
- [crates/docker_manager/src/manager.rs](file://crates/docker_manager/src/manager.rs#L147-L165)
|
||||
- [scripts/setup-network-isolation.sh](file://scripts/setup-network-isolation.sh#L1-L112)
|
||||
- [crates/rcoder/src/config.rs](file://crates/rcoder/src/config.rs#L212-L239)
|
||||
- [crates/agent_runner/src/config.rs](file://crates/agent_runner/src/config.rs#L134-L171)
|
||||
- [crates/rcoder/src/main.rs](file://crates/rcoder/src/main.rs#L274-L321)
|
||||
- [crates/agent_runner/src/main.rs](file://crates/agent_runner/src/main.rs#L181-L232)
|
||||
- [crates/pingora-proxy/src/service.rs](file://crates/pingora-proxy/src/service.rs#L556-L596)
|
||||
|
||||
## 架构总览
|
||||
下图展示 RCoder 主服务、Agent Runner、Pingora 代理与 Docker 管理器之间的交互,以及网络隔离脚本对宿主机网络的影响。
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client as "客户端"
|
||||
participant RC as "rcoder 主服务"
|
||||
participant PM as "Pingora 代理"
|
||||
participant AR as "agent_runner 服务"
|
||||
participant DM as "Docker 管理器"
|
||||
participant NI as "网络隔离脚本"
|
||||
Client->>RC : "POST /chat"
|
||||
RC->>DM : "获取/创建容器"
|
||||
DM-->>RC : "返回容器信息"
|
||||
RC->>AR : "转发请求到容器内 /chat"
|
||||
AR-->>RC : "返回处理结果"
|
||||
RC-->>Client : "返回响应"
|
||||
Note over NI,PM : "宿主机 iptables 限制容器访问内网"
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [crates/rcoder/src/handler/chat_handler.rs](file://crates/rcoder/src/handler/chat_handler.rs#L323-L431)
|
||||
- [crates/agent_runner/src/handler/chat_handler.rs](file://crates/agent_runner/src/handler/chat_handler.rs#L174-L321)
|
||||
- [crates/docker_manager/src/manager.rs](file://crates/docker_manager/src/manager.rs#L105-L165)
|
||||
- [scripts/setup-network-isolation.sh](file://scripts/setup-network-isolation.sh#L70-L95)
|
||||
|
||||
章节来源
|
||||
- [crates/rcoder/src/router.rs](file://crates/rcoder/src/router.rs#L52-L84)
|
||||
- [crates/agent_runner/src/router.rs](file://crates/agent_runner/src/router.rs#L40-L70)
|
||||
|
||||
## 详细组件安全分析
|
||||
|
||||
### 容器隔离与挂载权限控制
|
||||
- 能力与特权控制
|
||||
- 容器启动时丢弃网络原始与管理能力,禁用特权模式,降低容器逃逸与网络嗅探风险。
|
||||
- 挂载策略
|
||||
- 默认挂载类型为 bind,部分服务挂载路径允许只读或额外绑定,需谨慎评估挂载路径与权限。
|
||||
- 网络隔离脚本
|
||||
- 通过查找 rcoder- 前缀网络,为每个网络子网添加 DROP 规则,阻止访问常见内网地址段;规则在系统重启后会丢失,需持久化或开机重载。
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start(["容器启动"]) --> DropCaps["丢弃网络相关能力<br/>禁用特权模式"]
|
||||
DropCaps --> Mounts["绑定挂载配置<br/>只读/额外挂载"]
|
||||
Mounts --> NetIsolate["宿主机 iptables 规则<br/>阻止访问内网地址段"]
|
||||
NetIsolate --> Run(["容器运行"])
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [crates/docker_manager/src/manager.rs](file://crates/docker_manager/src/manager.rs#L147-L165)
|
||||
- [scripts/setup-network-isolation.sh](file://scripts/setup-network-isolation.sh#L70-L95)
|
||||
|
||||
章节来源
|
||||
- [crates/docker_manager/src/manager.rs](file://crates/docker_manager/src/manager.rs#L105-L165)
|
||||
- [scripts/setup-network-isolation.sh](file://scripts/setup-network-isolation.sh#L1-L112)
|
||||
|
||||
### 镜像来源验证与多镜像配置
|
||||
- 多镜像配置
|
||||
- 支持按服务类型选择镜像,包含全局默认与架构特定镜像,策略为仅使用服务特定配置。
|
||||
- 缓存与选择
|
||||
- 提供镜像缓存配置,支持 TTL 与最大条目数;镜像选择策略为 ServiceOnly。
|
||||
- 建议
|
||||
- 通过 registry 前缀与服务特定镜像保证来源可控;结合镜像签名与拉取策略,进一步强化来源可信度。
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
class MultiImageConfig {
|
||||
+services
|
||||
+global_defaults
|
||||
+selection_strategy
|
||||
+cache_config
|
||||
+validate()
|
||||
+hash_key()
|
||||
}
|
||||
class ServiceConfig {
|
||||
+service_type
|
||||
+image/arm64/amd64
|
||||
+mounts
|
||||
+environment
|
||||
+resource_limits
|
||||
+get_image_for_platform(platform)
|
||||
+validate()
|
||||
}
|
||||
MultiImageConfig --> ServiceConfig : "包含"
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [crates/shared_types/src/multi_image_config.rs](file://crates/shared_types/src/multi_image_config.rs#L43-L110)
|
||||
- [crates/shared_types/src/multi_image_config.rs](file://crates/shared_types/src/multi_image_config.rs#L361-L406)
|
||||
- [crates/shared_types/src/service_config.rs](file://crates/shared_types/src/service_config.rs#L153-L188)
|
||||
|
||||
章节来源
|
||||
- [crates/shared_types/src/multi_image_config.rs](file://crates/shared_types/src/multi_image_config.rs#L43-L110)
|
||||
- [crates/shared_types/src/multi_image_config.rs](file://crates/shared_types/src/multi_image_config.rs#L361-L406)
|
||||
- [crates/shared_types/src/service_config.rs](file://crates/shared_types/src/service_config.rs#L153-L188)
|
||||
|
||||
### API 端点访问控制、输入验证与防滥用
|
||||
- 访问控制
|
||||
- 当前未实现显式的鉴权/授权中间件;建议在反向代理层或网关处增加认证与授权策略。
|
||||
- 输入验证
|
||||
- 主服务与 Agent Runner 均对请求进行基本校验(如 prompt 非空),并在业务层进行并发控制与会话清理。
|
||||
- 防滥用
|
||||
- 通过并发请求限制与会话缓存清理,避免同一项目下的并发冲突;建议引入速率限制中间件或上游限流策略。
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client as "客户端"
|
||||
participant RC as "rcoder 主服务"
|
||||
participant AR as "agent_runner 服务"
|
||||
Client->>RC : "POST /chat"
|
||||
RC->>RC : "校验请求参数"
|
||||
RC->>AR : "转发请求"
|
||||
AR->>AR : "并发检查/会话清理"
|
||||
AR-->>RC : "返回结果"
|
||||
RC-->>Client : "返回响应"
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [crates/rcoder/src/handler/chat_handler.rs](file://crates/rcoder/src/handler/chat_handler.rs#L108-L170)
|
||||
- [crates/agent_runner/src/handler/chat_handler.rs](file://crates/agent_runner/src/handler/chat_handler.rs#L174-L238)
|
||||
|
||||
章节来源
|
||||
- [crates/rcoder/src/handler/chat_handler.rs](file://crates/rcoder/src/handler/chat_handler.rs#L108-L170)
|
||||
- [crates/agent_runner/src/handler/chat_handler.rs](file://crates/agent_runner/src/handler/chat_handler.rs#L174-L238)
|
||||
|
||||
### 配置文件与敏感信息管理
|
||||
- 配置优先级
|
||||
- 命令行参数 > 环境变量 > 配置文件 > 默认配置;Docker 配置支持环境变量覆盖。
|
||||
- 敏感信息建议
|
||||
- 将密钥、令牌等敏感信息通过环境变量注入,避免写入 config.yml;必要时使用密钥管理服务或容器编排平台的机密存储。
|
||||
- 配置文件生成
|
||||
- 首次启动会生成默认配置文件,建议在部署时以只读方式挂载或通过环境变量覆盖。
|
||||
|
||||
章节来源
|
||||
- [crates/rcoder/src/config.rs](file://crates/rcoder/src/config.rs#L253-L332)
|
||||
- [crates/agent_runner/src/config.rs](file://crates/agent_runner/src/config.rs#L110-L191)
|
||||
- [config.yml](file://config.yml#L1-L161)
|
||||
|
||||
### 网络隔离脚本实施与影响
|
||||
- 规则生效范围
|
||||
- 为 rcoder- 前缀网络添加 DROP 规则,阻止访问常见内网地址段;规则在系统重启后会丢失,需持久化。
|
||||
- 实施建议
|
||||
- 将脚本加入系统启动项或 systemd 服务;定期检查规则有效性;结合防火墙策略与网络命名空间进一步加固。
|
||||
|
||||
章节来源
|
||||
- [scripts/setup-network-isolation.sh](file://scripts/setup-network-isolation.sh#L1-L112)
|
||||
|
||||
### 日志与追踪、健康检查与可观测性
|
||||
- 日志
|
||||
- 结构化 JSON 日志按天滚动,支持 trace_id 传播,便于审计与跨服务关联。
|
||||
- 追踪
|
||||
- 中间件生成并传播 trace_id,便于端到端链路追踪。
|
||||
- 健康检查
|
||||
- Pingora 代理提供健康检查循环与统计接口,便于监控后端健康状态。
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant RC as "rcoder 主服务"
|
||||
participant PM as "Pingora 代理"
|
||||
participant HC as "健康检查循环"
|
||||
RC->>PM : "启动代理服务"
|
||||
PM->>HC : "启动健康检查循环"
|
||||
HC->>HC : "周期性检查后端连通性"
|
||||
PM-->>RC : "健康状态快照/统计"
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [crates/rcoder/src/main.rs](file://crates/rcoder/src/main.rs#L168-L209)
|
||||
- [crates/agent_runner/src/main.rs](file://crates/agent_runner/src/main.rs#L80-L121)
|
||||
- [crates/pingora-proxy/src/service.rs](file://crates/pingora-proxy/src/service.rs#L556-L596)
|
||||
|
||||
章节来源
|
||||
- [crates/rcoder/src/main.rs](file://crates/rcoder/src/main.rs#L274-L321)
|
||||
- [crates/agent_runner/src/main.rs](file://crates/agent_runner/src/main.rs#L181-L232)
|
||||
- [crates/pingora-proxy/src/service.rs](file://crates/pingora-proxy/src/service.rs#L556-L596)
|
||||
|
||||
### 最小权限原则与系统提示词约束
|
||||
- 最小权限
|
||||
- 容器禁用特权模式并丢弃网络相关能力,减少潜在攻击面。
|
||||
- 系统提示词约束
|
||||
- Agent Runner 的系统提示词包含多项安全禁令,禁止内网探测、反向 Shell、框架转换等行为,有助于降低风险。
|
||||
|
||||
章节来源
|
||||
- [crates/docker_manager/src/manager.rs](file://crates/docker_manager/src/manager.rs#L147-L165)
|
||||
- [crates/agent_runner/src/utils/system_prompt.rs](file://crates/agent_runner/src/utils/system_prompt.rs#L96-L115)
|
||||
|
||||
## 依赖关系与耦合分析
|
||||
- 组件耦合
|
||||
- rcoder 主服务依赖 Docker 管理器进行容器生命周期管理;与 Pingora 代理耦合用于后端发现与健康检查;与 Agent Runner 通过容器内通信协作。
|
||||
- 外部依赖
|
||||
- Docker API、iptables、Pingora 代理、OpenTelemetry/Tracing。
|
||||
- 潜在风险
|
||||
- Docker socket 挂载与权限配置不当可能导致容器逃逸;网络隔离规则缺失可能使容器访问内网。
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
RC["rcoder 主服务"] --> DM["Docker 管理器"]
|
||||
RC --> PM["Pingora 代理"]
|
||||
RC --> AR["agent_runner 服务"]
|
||||
NI["网络隔离脚本"] -.-> RC
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [crates/rcoder/src/main.rs](file://crates/rcoder/src/main.rs#L111-L158)
|
||||
- [crates/agent_runner/src/main.rs](file://crates/agent_runner/src/main.rs#L120-L178)
|
||||
- [scripts/setup-network-isolation.sh](file://scripts/setup-network-isolation.sh#L52-L95)
|
||||
|
||||
章节来源
|
||||
- [crates/rcoder/src/main.rs](file://crates/rcoder/src/main.rs#L111-L158)
|
||||
- [crates/agent_runner/src/main.rs](file://crates/agent_runner/src/main.rs#L120-L178)
|
||||
|
||||
## 性能与安全权衡
|
||||
- 性能
|
||||
- 容器能力限制与网络隔离会带来一定开销,但显著降低逃逸与横向移动风险。
|
||||
- 安全
|
||||
- 建议在生产环境启用严格的镜像来源验证、最小权限与网络隔离;结合速率限制与上游防护,平衡吞吐与安全。
|
||||
|
||||
[本节为通用指导,无需引用具体文件]
|
||||
|
||||
## 故障排查与安全审计
|
||||
- 日志审计
|
||||
- 检查 logs 目录下的 JSON 日志,结合 trace_id 进行关联分析;关注健康检查失败与代理统计异常。
|
||||
- 网络隔离验证
|
||||
- 使用脚本列出当前规则,确认 rcoder- 网络的 DROP 规则已生效;必要时重新加载规则。
|
||||
- 配置核验
|
||||
- 确认环境变量覆盖生效,敏感信息未出现在配置文件中;检查 Docker 配置与挂载路径。
|
||||
- 漏洞扫描
|
||||
- 建议对镜像进行静态扫描,识别高危漏洞;对运行时容器进行定期扫描与基线核查。
|
||||
|
||||
章节来源
|
||||
- [crates/rcoder/src/main.rs](file://crates/rcoder/src/main.rs#L274-L321)
|
||||
- [scripts/setup-network-isolation.sh](file://scripts/setup-network-isolation.sh#L97-L112)
|
||||
- [crates/rcoder/src/config.rs](file://crates/rcoder/src/config.rs#L253-L332)
|
||||
|
||||
## 结论
|
||||
RCoder 在容器隔离、网络隔离、日志与追踪方面具备良好基础。建议在生产环境中补充鉴权/授权、速率限制、镜像来源验证与漏洞扫描机制,严格使用环境变量管理敏感信息,并完善网络隔离规则的持久化与监控,以实现更全面的安全保障与可审计性。
|
||||
189
qiming-rcoder/.qoder/repowiki/zh/content/故障排除/常见问题.md
Normal file
189
qiming-rcoder/.qoder/repowiki/zh/content/故障排除/常见问题.md
Normal file
@@ -0,0 +1,189 @@
|
||||
# 常见问题
|
||||
|
||||
<cite>
|
||||
**本文档中引用的文件**
|
||||
- [README.md](file://README.md)
|
||||
- [config.yml](file://config.yml)
|
||||
- [docker-compose.yml](file://docker/docker-compose.yml)
|
||||
- [main.rs](file://crates/rcoder/src/main.rs)
|
||||
- [docker_container_agent.rs](file://crates/rcoder/src/proxy_agent/docker_container_agent.rs)
|
||||
- [port_manager.rs](file://crates/rcoder/src/proxy_agent/port_manager.rs)
|
||||
- [server.rs](file://crates/pingora-proxy/src/server.rs)
|
||||
- [config.rs](file://crates/pingora-proxy/src/config.rs)
|
||||
- [CLAUDE.md](file://CLAUDE.md)
|
||||
- [install.md](file://install.md)
|
||||
</cite>
|
||||
|
||||
## 目录
|
||||
1. [代理启动失败](#代理启动失败)
|
||||
2. [端口冲突问题](#端口冲突问题)
|
||||
3. [Docker权限错误](#docker权限错误)
|
||||
4. [SSE流中断](#sse流中断)
|
||||
5. [Claude Code代理集成限制](#claude-code代理集成限制)
|
||||
|
||||
## 代理启动失败
|
||||
|
||||
代理启动失败通常由配置错误或依赖服务未就绪导致。根据`crates/pingora-proxy/src/server.rs`中的实现,代理服务在启动时会进行配置验证。
|
||||
|
||||
**根本原因分析**:
|
||||
- 配置文件`config.yml`中`proxy_config.listen_port`设置为0或无效值
|
||||
- 环境变量`RCODER_PORT`与代理端口冲突
|
||||
- Docker socket未正确挂载,导致容器无法通信
|
||||
|
||||
**调试步骤**:
|
||||
1. 检查日志中是否出现"配置验证失败"错误
|
||||
2. 验证`config.yml`中`proxy_config.listen_port`是否为有效端口
|
||||
3. 确认Docker socket路径是否正确挂载
|
||||
|
||||
**解决方案**:
|
||||
```yaml
|
||||
# config.yml
|
||||
proxy_config:
|
||||
listen_port: 8088 # 确保端口大于0
|
||||
default_backend_port: 8087 # 与主服务端口一致
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [server.rs](file://crates/pingora-proxy/src/server.rs#L27-L54)
|
||||
- [config.rs](file://crates/pingora-proxy/src/config.rs#L50-L68)
|
||||
- [config.yml](file://config.yml#L16)
|
||||
|
||||
## 端口冲突问题
|
||||
|
||||
端口冲突主要发生在多个服务尝试绑定同一端口时。系统通过`port_manager.rs`管理端口分配。
|
||||
|
||||
**根本原因分析**:
|
||||
- 多个RCoder实例同时运行
|
||||
- 端口范围(8000-9999)内所有端口已被占用
|
||||
- 容器未正确清理,遗留端口映射
|
||||
|
||||
**调试步骤**:
|
||||
1. 检查`netstat -an | grep <port>`确认端口占用情况
|
||||
2. 查看日志中"端口范围内无可用端口"错误
|
||||
3. 使用`docker ps`检查是否有遗留容器
|
||||
|
||||
**解决方案**:
|
||||
```bash
|
||||
# 修改端口范围
|
||||
cargo run --bin rcoder -- --port 9000 --proxy-port 9001
|
||||
```
|
||||
|
||||
或在代码中调整端口管理器范围:
|
||||
```rust
|
||||
// port_manager.rs
|
||||
impl PortManager {
|
||||
pub fn new() -> Self {
|
||||
Self::with_range(10000, 11999) // 扩大端口范围
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [port_manager.rs](file://crates/rcoder/src/proxy_agent/port_manager.rs#L32-L49)
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L223-L225)
|
||||
|
||||
## Docker权限错误
|
||||
|
||||
Docker权限错误通常与socket访问权限或挂载配置有关。
|
||||
|
||||
**根本原因分析**:
|
||||
- Docker socket未挂载到容器
|
||||
- 用户不在docker组中
|
||||
- 路径解析器初始化失败
|
||||
|
||||
**调试步骤**:
|
||||
1. 检查`docker-compose.yml`中socket挂载配置
|
||||
2. 验证用户权限:`groups $USER | grep docker`
|
||||
3. 测试Docker API连通性:`curl --unix-socket /var/run/docker.sock http://localhost/info`
|
||||
|
||||
**解决方案**:
|
||||
确保`docker-compose.yml`包含正确配置:
|
||||
```yaml
|
||||
services:
|
||||
rcoder:
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock
|
||||
- ./project_workspace:/app/project_workspace
|
||||
```
|
||||
|
||||
并添加用户到docker组:
|
||||
```bash
|
||||
sudo usermod -aG docker $USER
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L49-L82)
|
||||
- [docker-compose.yml](file://docker/docker-compose.yml#L11-L15)
|
||||
|
||||
## SSE流中断
|
||||
|
||||
SSE(Server-Sent Events)流中断会影响实时进度更新功能。
|
||||
|
||||
**根本原因分析**:
|
||||
- 客户端连接超时设置过短
|
||||
- 代理配置未正确转发SSE连接
|
||||
- 服务器资源不足导致连接被重置
|
||||
|
||||
**调试步骤**:
|
||||
1. 检查客户端是否设置`Connection: keep-alive`
|
||||
2. 验证Nginx等反向代理的超时配置
|
||||
3. 监控服务器资源使用情况
|
||||
|
||||
**解决方案**:
|
||||
在客户端增加重连机制:
|
||||
```javascript
|
||||
const eventSource = new EventSource('/agent/progress/session123');
|
||||
eventSource.onmessage = (event) => {
|
||||
console.log(event.data);
|
||||
};
|
||||
eventSource.onerror = () => {
|
||||
setTimeout(() => {
|
||||
// 重新连接
|
||||
eventSource.close();
|
||||
}, 5000);
|
||||
};
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [README.md](file://README.md#L254-L266)
|
||||
|
||||
## Claude Code代理集成限制
|
||||
|
||||
Claude Code代理集成存在特定限制和配置要求。
|
||||
|
||||
**根本原因分析**:
|
||||
- CLI工具未正确安装
|
||||
- API密钥未配置
|
||||
- 环境变量未正确传递
|
||||
|
||||
**调试步骤**:
|
||||
1. 验证Claude CLI安装:`claude --version`
|
||||
2. 检查环境变量:`echo $ANTHROPIC_API_KEY`
|
||||
3. 查看日志中代理初始化错误
|
||||
|
||||
**解决方案**:
|
||||
1. 安装CLI工具:
|
||||
```bash
|
||||
npm install -g @anthropic-ai/claude-code
|
||||
```
|
||||
|
||||
2. 配置环境变量:
|
||||
```bash
|
||||
export ANTHROPIC_API_KEY="your-api-key"
|
||||
export ANTHROPIC_MODEL="claude-3-sonnet-20240229"
|
||||
```
|
||||
|
||||
3. 在`config.yml`中确保代理配置正确:
|
||||
```yaml
|
||||
default_agent: "Claude"
|
||||
```
|
||||
|
||||
**规避方法**:
|
||||
- 使用环境变量而非配置文件存储API密钥
|
||||
- 实现密钥轮换机制
|
||||
- 添加代理健康检查
|
||||
|
||||
**Section sources**
|
||||
- [CLAUDE.md](file://CLAUDE.md#L109-L132)
|
||||
- [install.md](file://install.md#L5-L8)
|
||||
- [config.yml](file://config.yml#L5)
|
||||
335
qiming-rcoder/.qoder/repowiki/zh/content/故障排除/性能优化.md
Normal file
335
qiming-rcoder/.qoder/repowiki/zh/content/故障排除/性能优化.md
Normal file
@@ -0,0 +1,335 @@
|
||||
# 性能优化
|
||||
|
||||
<cite>
|
||||
**本文引用的文件**
|
||||
- [lib.rs](file://crates/pingora-proxy/src/lib.rs)
|
||||
- [config.rs](file://crates/pingora-proxy/src/config.rs)
|
||||
- [pingora_server.rs](file://crates/pingora-proxy/src/pingora_server.rs)
|
||||
- [server.rs](file://crates/pingora-proxy/src/server.rs)
|
||||
- [cleanup_task.rs](file://crates/agent_runner/src/proxy_agent/cleanup_task.rs)
|
||||
- [system_prompt.rs](file://crates/agent_runner/src/utils/system_prompt.rs)
|
||||
- [session_cache.rs](file://crates/agent_runner/src/service/session_cache.rs)
|
||||
- [agent_session_notification.rs](file://crates/agent_runner/src/handler/agent_session_notification.rs)
|
||||
- [generate-flamegraph.sh](file://docker/scripts/generate-flamegraph.sh)
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs)
|
||||
- [proxy_handler_api.rs](file://crates/agent_runner/src/handler/proxy_handler_api.rs)
|
||||
- [service.rs](file://crates/pingora-proxy/src/service.rs)
|
||||
</cite>
|
||||
|
||||
## 目录
|
||||
1. [引言](#引言)
|
||||
2. [项目结构](#项目结构)
|
||||
3. [核心组件](#核心组件)
|
||||
4. [架构总览](#架构总览)
|
||||
5. [详细组件分析](#详细组件分析)
|
||||
6. [依赖分析](#依赖分析)
|
||||
7. [性能考量](#性能考量)
|
||||
8. [故障排查指南](#故障排查指南)
|
||||
9. [结论](#结论)
|
||||
10. [附录](#附录)
|
||||
|
||||
## 引言
|
||||
本文件聚焦RCoder系统在高并发、低延迟场景下的性能优化策略,围绕以下主题展开:
|
||||
- Pingora反向代理的性能优势与配置参数对吞吐量的影响
|
||||
- 代理会话的资源清理机制(cleanup_task.rs)如何避免内存泄漏和连接堆积
|
||||
- 系统提示词(system_prompt.rs)优化建议以降低AI推理开销
|
||||
- Tokio运行时调优、连接池配置、缓存策略与SSE流压缩等实践
|
||||
- 基于实际压测数据与火焰图(generate-flamegraph.sh)的优化效果验证
|
||||
|
||||
## 项目结构
|
||||
RCoder系统由“代理层”和“业务层”组成:
|
||||
- 代理层:基于Cloudflare Pingora的高性能反向代理,支持HTTP/1.1与HTTP/2、健康检查、连接池与负载均衡
|
||||
- 业务层:Agent Runner提供统一SSE流、会话缓存、清理任务与代理会话管理;RCoder侧提供SSE代理与后端健康状态查询
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "代理层"
|
||||
PLib["pingora-proxy 库<br/>lib.rs"]
|
||||
PConf["配置<br/>config.rs"]
|
||||
PServer["服务器管理<br/>pingora_server.rs"]
|
||||
PRun["服务器运行器<br/>server.rs"]
|
||||
PMetrics["指标记录<br/>service.rs"]
|
||||
end
|
||||
subgraph "业务层"
|
||||
ARMain["Agent Runner 入口<br/>main.rs"]
|
||||
Clean["清理任务<br/>cleanup_task.rs"]
|
||||
Sess["会话缓存<br/>session_cache.rs"]
|
||||
SSE["SSE处理器<br/>agent_session_notification.rs"]
|
||||
ProxyAPI["代理API<br/>proxy_handler_api.rs"]
|
||||
end
|
||||
PLib --> PServer
|
||||
PLib --> PRun
|
||||
PConf --> PServer
|
||||
PServer --> PMetrics
|
||||
ARMain --> PServer
|
||||
ARMain --> Clean
|
||||
ARMain --> Sess
|
||||
Sess --> SSE
|
||||
ProxyAPI --> PServer
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [lib.rs](file://crates/pingora-proxy/src/lib.rs#L1-L120)
|
||||
- [config.rs](file://crates/pingora-proxy/src/config.rs#L1-L95)
|
||||
- [pingora_server.rs](file://crates/pingora-proxy/src/pingora_server.rs#L1-L120)
|
||||
- [server.rs](file://crates/pingora-proxy/src/server.rs#L1-L120)
|
||||
- [service.rs](file://crates/pingora-proxy/src/service.rs#L90-L140)
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L1-L120)
|
||||
- [cleanup_task.rs](file://crates/agent_runner/src/proxy_agent/cleanup_task.rs#L1-L120)
|
||||
- [session_cache.rs](file://crates/agent_runner/src/service/session_cache.rs#L1-L120)
|
||||
- [agent_session_notification.rs](file://crates/agent_runner/src/handler/agent_session_notification.rs#L100-L170)
|
||||
- [proxy_handler_api.rs](file://crates/agent_runner/src/handler/proxy_handler_api.rs#L40-L80)
|
||||
|
||||
章节来源
|
||||
- [lib.rs](file://crates/pingora-proxy/src/lib.rs#L1-L120)
|
||||
- [server.rs](file://crates/pingora-proxy/src/server.rs#L1-L120)
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L1-L120)
|
||||
|
||||
## 核心组件
|
||||
- Pingora反向代理库:提供高性能、异步I/O、HTTP/1.1与HTTP/2支持、健康检查与连接池
|
||||
- 代理服务器管理器:封装Pingora Server启动、监听与关闭信号处理
|
||||
- 代理服务器运行器:提供便捷的代理实例创建与负载均衡算法选择
|
||||
- 会话缓存与SSE:基于环形缓冲区与mpsc通道的极简设计,支持心跳与取消令牌
|
||||
- 清理任务:基于RAII的闲置Agent清理,定期扫描并移除超时闲置Agent,避免资源泄漏
|
||||
- 系统提示词:通过结构化提示词配置降低AI推理冗余,提升稳定性与一致性
|
||||
|
||||
章节来源
|
||||
- [lib.rs](file://crates/pingora-proxy/src/lib.rs#L1-L120)
|
||||
- [server.rs](file://crates/pingora-proxy/src/server.rs#L1-L120)
|
||||
- [pingora_server.rs](file://crates/pingora-proxy/src/pingora_server.rs#L1-L120)
|
||||
- [session_cache.rs](file://crates/agent_runner/src/service/session_cache.rs#L1-L120)
|
||||
- [cleanup_task.rs](file://crates/agent_runner/src/proxy_agent/cleanup_task.rs#L1-L120)
|
||||
- [system_prompt.rs](file://crates/agent_runner/src/utils/system_prompt.rs#L1-L120)
|
||||
|
||||
## 架构总览
|
||||
RCoder在高并发场景下的关键路径:
|
||||
- 客户端请求经Pingora代理到达业务层路由
|
||||
- 业务层根据会话ID将SSE流推送至客户端
|
||||
- 代理层维护后端健康状态与连接池,保障低延迟转发
|
||||
- 清理任务周期性回收闲置Agent,避免内存与连接堆积
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as "客户端"
|
||||
participant P as "Pingora代理"
|
||||
participant R as "Agent Runner"
|
||||
participant S as "会话缓存/Worker"
|
||||
participant T as "清理任务"
|
||||
C->>P : "HTTP 请求"
|
||||
P->>R : "转发到业务路由"
|
||||
R->>S : "推送SSE消息"
|
||||
S-->>R : "实时消息/心跳"
|
||||
R-->>C : "SSE流"
|
||||
T->>R : "周期清理闲置Agent"
|
||||
R-->>T : "清理完成"
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [pingora_server.rs](file://crates/pingora-proxy/src/pingora_server.rs#L38-L120)
|
||||
- [agent_session_notification.rs](file://crates/agent_runner/src/handler/agent_session_notification.rs#L390-L483)
|
||||
- [session_cache.rs](file://crates/agent_runner/src/service/session_cache.rs#L140-L222)
|
||||
- [cleanup_task.rs](file://crates/agent_runner/src/proxy_agent/cleanup_task.rs#L278-L310)
|
||||
|
||||
## 详细组件分析
|
||||
|
||||
### Pingora反向代理性能与配置
|
||||
- 性能优势
|
||||
- 基于Pingora的高性能异步I/O与HTTP/1.1、HTTP/2支持,适合高并发与低延迟场景
|
||||
- 内置健康检查与连接池,减少后端抖动带来的延迟
|
||||
- 负载均衡算法可选轮询或一致性哈希,满足不同流量特征
|
||||
- 关键配置参数
|
||||
- 监听端口、默认后端端口、后端主机、端口参数名、详细日志开关
|
||||
- 负载均衡算法(轮询/一致性哈希)影响后端命中与抖动
|
||||
- 吞吐量影响
|
||||
- 合理设置监听端口与后端端口,避免端口冲突与回退路径
|
||||
- 启用健康检查与连接池可显著降低后端异常导致的失败率与重试成本
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start(["代理启动"]) --> LoadCfg["加载配置<br/>监听端口/后端端口/主机/参数名"]
|
||||
LoadCfg --> LB["选择负载均衡算法<br/>轮询/一致性哈希"]
|
||||
LB --> HC["健康检查与连接池"]
|
||||
HC --> Run["运行服务器"]
|
||||
Run --> Metrics["记录指标<br/>成功率/平均响应时间"]
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [config.rs](file://crates/pingora-proxy/src/config.rs#L1-L95)
|
||||
- [server.rs](file://crates/pingora-proxy/src/server.rs#L120-L155)
|
||||
- [service.rs](file://crates/pingora-proxy/src/service.rs#L90-L140)
|
||||
|
||||
章节来源
|
||||
- [lib.rs](file://crates/pingora-proxy/src/lib.rs#L1-L120)
|
||||
- [config.rs](file://crates/pingora-proxy/src/config.rs#L1-L95)
|
||||
- [server.rs](file://crates/pingora-proxy/src/server.rs#L120-L155)
|
||||
- [service.rs](file://crates/pingora-proxy/src/service.rs#L90-L140)
|
||||
|
||||
### 代理会话资源清理机制(cleanup_task.rs)
|
||||
- 设计要点
|
||||
- 基于RAII的清理:从全局映射移除Agent即触发资源自动释放
|
||||
- 定时扫描:按闲置超时与清理间隔执行
|
||||
- 清理范围:闲置Agent、孤立SSE会话与消息
|
||||
- 关键流程
|
||||
- 先清理孤立SSE会话与消息,再扫描Idle状态Agent
|
||||
- 通过全局映射与上下文映射同步清理,避免悬挂引用
|
||||
- 统计清理数量与剩余状态,便于监控
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Tick["定时Tick"] --> Scan["扫描闲置Agent"]
|
||||
Scan --> |Idle且超时| Remove["从映射移除<br/>触发RAII清理"]
|
||||
Scan --> |非Idle| Skip["跳过"]
|
||||
Remove --> Stats["更新统计"]
|
||||
Skip --> Stats
|
||||
Stats --> Done["完成一轮清理"]
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [cleanup_task.rs](file://crates/agent_runner/src/proxy_agent/cleanup_task.rs#L156-L241)
|
||||
- [cleanup_task.rs](file://crates/agent_runner/src/proxy_agent/cleanup_task.rs#L243-L310)
|
||||
|
||||
章节来源
|
||||
- [cleanup_task.rs](file://crates/agent_runner/src/proxy_agent/cleanup_task.rs#L1-L310)
|
||||
|
||||
### 系统提示词优化建议(system_prompt.rs)
|
||||
- 目标:降低AI推理开销,提升一致性与安全性
|
||||
- 建议
|
||||
- 明确框架识别优先级与路由模式(hash模式),避免重复检索与转换
|
||||
- 严格约束安全与框架转换禁令,减少无效推理分支
|
||||
- 限定允许的操作范围,减少无关工具调用
|
||||
- 将MCP工具使用规则前置,减少上下文膨胀
|
||||
- 效果:通过结构化提示词减少模型输出冗余,缩短推理时间,提高稳定性
|
||||
|
||||
章节来源
|
||||
- [system_prompt.rs](file://crates/agent_runner/src/utils/system_prompt.rs#L1-L260)
|
||||
- [system_prompt.rs](file://crates/agent_runner/src/utils/system_prompt.rs#L260-L407)
|
||||
|
||||
### Tokio运行时调优与连接池配置
|
||||
- 运行时布局
|
||||
- 主异步运行时:启动清理任务、代理服务器与HTTP服务
|
||||
- 单线程LocalSet运行时:承载非Send的Agent Worker,避免跨线程迁移成本
|
||||
- 连接池与健康检查
|
||||
- 代理层内置连接池与健康检查,按配置启动健康检查循环
|
||||
- 通过后端健康快照与后端列表接口暴露状态,便于运维观测
|
||||
- 建议
|
||||
- 根据CPU核数与I/O密集度调整主运行时线程数
|
||||
- 为I/O密集型任务分配更多线程,为CPU密集型任务使用LocalSet隔离
|
||||
- 合理设置健康检查间隔与超时,避免过度探测
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant M as "主运行时"
|
||||
participant L as "LocalSet运行时"
|
||||
participant P as "Pingora服务器"
|
||||
participant HC as "健康检查循环"
|
||||
M->>M : "启动清理任务"
|
||||
M->>P : "启动代理服务器"
|
||||
M->>HC : "启动健康检查循环"
|
||||
L->>L : "运行Agent Worker!Send"
|
||||
HC-->>M : "后端健康状态快照"
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L29-L120)
|
||||
- [pingora_server.rs](file://crates/pingora-proxy/src/pingora_server.rs#L60-L120)
|
||||
- [proxy_handler_api.rs](file://crates/agent_runner/src/handler/proxy_handler_api.rs#L40-L80)
|
||||
|
||||
章节来源
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L29-L120)
|
||||
- [pingora_server.rs](file://crates/pingora-proxy/src/pingora_server.rs#L60-L120)
|
||||
- [proxy_handler_api.rs](file://crates/agent_runner/src/handler/proxy_handler_api.rs#L40-L80)
|
||||
|
||||
### 缓存策略与SSE流压缩
|
||||
- 会话缓存(session_cache.rs)
|
||||
- 基于环形缓冲区(ringbuf)与mpsc通道,支持心跳过滤与实时推送
|
||||
- 通过共享状态直接设置当前连接,避免命令传递开销
|
||||
- 主动关闭SSE连接时使用取消令牌与显式关闭发送端,确保客户端及时断开
|
||||
- SSE流(agent_session_notification.rs)
|
||||
- SSE代理直连容器SSE端点,按双换行符分割事件,透传原始SSE数据
|
||||
- 心跳定时器与取消令牌结合,保证连接生命周期可控
|
||||
- 压缩建议
|
||||
- SSE事件透传原始数据,建议在上游服务端启用Gzip/Deflate压缩
|
||||
- 对于大消息,可考虑分片与增量推送,减少单事件体积
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Push["推送消息到SessionData"] --> Buffer["环形缓冲区"]
|
||||
Buffer --> Realtime["实时推送到当前连接"]
|
||||
Realtime --> |失败| Drop["清空当前连接状态"]
|
||||
Close["主动关闭SSE连接"] --> Cancel["触发取消令牌"]
|
||||
Close --> Drop2["显式关闭发送端"]
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [session_cache.rs](file://crates/agent_runner/src/service/session_cache.rs#L140-L222)
|
||||
- [agent_session_notification.rs](file://crates/agent_runner/src/handler/agent_session_notification.rs#L390-L483)
|
||||
|
||||
章节来源
|
||||
- [session_cache.rs](file://crates/agent_runner/src/service/session_cache.rs#L1-L222)
|
||||
- [agent_session_notification.rs](file://crates/agent_runner/src/handler/agent_session_notification.rs#L100-L170)
|
||||
- [agent_session_notification.rs](file://crates/agent_runner/src/handler/agent_session_notification.rs#L390-L483)
|
||||
|
||||
## 依赖分析
|
||||
- 组件耦合
|
||||
- 代理层与业务层通过Pingora服务接口解耦,业务层仅依赖会话缓存与SSE处理器
|
||||
- 清理任务与会话缓存相互配合,避免资源泄漏
|
||||
- 外部依赖
|
||||
- Pingora提供高性能代理能力
|
||||
- Tokio运行时与OpenTelemetry用于可观测性
|
||||
- 潜在风险
|
||||
- 若清理任务配置不当,可能导致连接堆积或频繁重建
|
||||
- SSE代理需关注上游SSE端点可用性与事件格式一致性
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
P["Pingora 服务器"] --> R["Agent Runner"]
|
||||
R --> S["会话缓存"]
|
||||
R --> C["清理任务"]
|
||||
R --> SSE["SSE处理器"]
|
||||
P --> API["代理API"]
|
||||
```
|
||||
|
||||
图表来源
|
||||
- [pingora_server.rs](file://crates/pingora-proxy/src/pingora_server.rs#L38-L120)
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L120-L179)
|
||||
- [proxy_handler_api.rs](file://crates/agent_runner/src/handler/proxy_handler_api.rs#L40-L80)
|
||||
|
||||
章节来源
|
||||
- [pingora_server.rs](file://crates/pingora-proxy/src/pingora_server.rs#L38-L120)
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L120-L179)
|
||||
- [proxy_handler_api.rs](file://crates/agent_runner/src/handler/proxy_handler_api.rs#L40-L80)
|
||||
|
||||
## 性能考量
|
||||
- 高并发与低延迟
|
||||
- 使用Pingora的HTTP/2与连接池,减少握手与队头阻塞
|
||||
- 会话缓存采用环形缓冲区与共享状态,降低锁竞争与拷贝
|
||||
- 资源管理
|
||||
- RAII清理避免悬挂Agent与会话,降低内存与文件描述符占用
|
||||
- SSE连接主动关闭与心跳维持,避免连接堆积
|
||||
- 可观测性
|
||||
- 指标记录与健康检查快照,便于定位瓶颈
|
||||
- 火焰图脚本用于CPU热点分析,指导进一步优化
|
||||
|
||||
## 故障排查指南
|
||||
- 火焰图分析
|
||||
- 使用脚本采集进程采样并生成SVG火焰图,定位热点函数与阻塞调用
|
||||
- 关注cleanup_task、docker_manager相关调用与tokio运行时相关函数
|
||||
- 常见问题
|
||||
- SSE连接长时间不释放:检查取消令牌与发送端关闭逻辑
|
||||
- 代理后端不稳定:查看健康检查快照与后端列表
|
||||
- 清理任务未生效:确认清理间隔与闲置超时配置
|
||||
|
||||
章节来源
|
||||
- [generate-flamegraph.sh](file://docker/scripts/generate-flamegraph.sh#L1-L59)
|
||||
- [agent_session_notification.rs](file://crates/agent_runner/src/handler/agent_session_notification.rs#L390-L483)
|
||||
- [proxy_handler_api.rs](file://crates/agent_runner/src/handler/proxy_handler_api.rs#L40-L80)
|
||||
|
||||
## 结论
|
||||
RCoder通过Pingora代理的高性能转发、会话缓存与SSE流的极简设计、以及基于RAII的清理机制,在高并发与低延迟场景下实现了稳定的吞吐表现。结合Tokio运行时调优、连接池与健康检查、提示词优化与火焰图分析,可进一步降低AI推理开销与系统延迟,持续提升整体性能与可靠性。
|
||||
|
||||
## 附录
|
||||
- 实践建议
|
||||
- 启用HTTP/2与连接池,合理设置健康检查间隔
|
||||
- 优化提示词结构,减少模型冗余输出
|
||||
- 使用火焰图定位热点,针对性优化关键路径
|
||||
- 通过指标与日志持续监控系统状态
|
||||
617
qiming-rcoder/.qoder/repowiki/zh/content/故障排除/故障排除.md
Normal file
617
qiming-rcoder/.qoder/repowiki/zh/content/故障排除/故障排除.md
Normal file
@@ -0,0 +1,617 @@
|
||||
# 故障排除
|
||||
|
||||
<cite>
|
||||
**本文档引用的文件**
|
||||
- [main.rs](file://crates/rcoder/src/main.rs)
|
||||
- [config.rs](file://crates/rcoder/src/config.rs)
|
||||
- [health_handler.rs](file://crates/rcoder/src/handler/health_handler.rs)
|
||||
- [tracing_middleware.rs](file://crates/rcoder/src/middleware/tracing_middleware.rs)
|
||||
- [manager.rs](file://crates/docker_manager/src/manager.rs)
|
||||
- [container_stop.rs](file://crates/docker_manager/src/container_stop.rs)
|
||||
- [types.rs](file://crates/docker_manager/src/types.rs)
|
||||
- [cleanup_task.rs](file://crates/rcoder/src/proxy_agent/cleanup_task.rs)
|
||||
- [app_error.rs](file://crates/shared_types/src/model/app_error.rs)
|
||||
</cite>
|
||||
|
||||
## 目录
|
||||
1. [常见问题及解决方案](#常见问题及解决方案)
|
||||
2. [日志分析](#日志分析)
|
||||
3. [性能优化](#性能优化)
|
||||
4. [安全考虑](#安全考虑)
|
||||
5. [配置选项与参数](#配置选项与参数)
|
||||
6. [组件关系](#组件关系)
|
||||
|
||||
## 常见问题及解决方案
|
||||
|
||||
### Docker连接问题
|
||||
当服务无法连接到Docker守护进程时,系统会提供详细的错误信息和解决建议。主要问题包括:
|
||||
|
||||
- **Docker socket路径错误**:检查环境变量`DOCKER_SOCKET_PATH`是否正确设置
|
||||
- **权限不足**:确保容器有权限访问Docker API
|
||||
- **Docker服务未运行**:验证Docker是否正在运行
|
||||
|
||||
**解决方案**:
|
||||
1. 检查`docker-compose.yml`中的配置,确保正确挂载Docker socket
|
||||
2. 验证Docker socket文件是否存在
|
||||
3. 检查用户是否在docker组中
|
||||
4. 测试Docker API连通性
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([启动服务]) --> CheckDockerSocket["检查DOCKER_SOCKET_PATH环境变量"]
|
||||
CheckDockerSocket --> ValidateSocket["验证Docker socket文件存在"]
|
||||
ValidateSocket --> TestAPI["测试Docker API连通性"]
|
||||
TestAPI --> CheckPermissions["检查用户权限"]
|
||||
CheckPermissions --> Success["服务启动成功"]
|
||||
CheckDockerSocket --> |路径未设置| UseDefault["使用默认路径: /var/run/docker.sock"]
|
||||
UseDefault --> ValidateSocket
|
||||
TestAPI --> |连接失败| ShowHelp["显示详细的配置帮助信息"]
|
||||
ShowHelp --> Stop["停止服务启动"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L48-L81)
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L322-L351)
|
||||
|
||||
### 容器清理失败
|
||||
在服务启动和关闭时,系统会尝试清理遗留的容器。可能会遇到以下问题:
|
||||
|
||||
- **409冲突错误**:容器已在删除过程中,这是正常情况
|
||||
- **容器不存在**:容器已被其他进程清理
|
||||
- **权限不足**:无法访问或删除容器
|
||||
|
||||
系统采用智能清理策略,忽略409冲突错误和容器不存在的情况,只记录真正的清理失败。
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([启动清理]) --> FindContainers["查找匹配模式的容器"]
|
||||
FindContainers --> CheckExist["检查容器是否存在"]
|
||||
CheckExist --> |存在| StopContainer["停止并删除容器"]
|
||||
CheckExist --> |不存在| Skip["跳过,记录为已清理"]
|
||||
StopContainer --> CheckError["检查错误类型"]
|
||||
CheckError --> |409冲突| Skip["跳过,记录为已清理"]
|
||||
CheckError --> |其他错误| LogError["记录失败,计入失败统计"]
|
||||
CheckError --> |成功| LogSuccess["记录成功"]
|
||||
LogSuccess --> NextContainer["处理下一个容器"]
|
||||
LogError --> NextContainer
|
||||
Skip --> NextContainer
|
||||
NextContainer --> |还有容器| CheckExist
|
||||
NextContainer --> |无容器| Complete["清理完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [container_stop.rs](file://crates/docker_manager/src/container_stop.rs#L82-L182)
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L131-L157)
|
||||
|
||||
### 服务启动失败
|
||||
服务启动时可能遇到多种问题,系统提供了详细的错误处理和恢复机制。
|
||||
|
||||
**常见问题**:
|
||||
- 配置文件加载失败
|
||||
- Docker管理器初始化失败
|
||||
- 端口绑定失败
|
||||
- 依赖服务未就绪
|
||||
|
||||
**解决方案**:
|
||||
1. 检查配置文件是否存在且格式正确
|
||||
2. 验证Docker服务是否正常运行
|
||||
3. 确认端口未被其他进程占用
|
||||
4. 检查依赖服务的状态
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([服务启动]) --> LoadConfig["加载配置文件"]
|
||||
LoadConfig --> |成功| InitDocker["初始化Docker管理器"]
|
||||
LoadConfig --> |失败| UseDefault["使用默认配置"]
|
||||
UseDefault --> InitDocker
|
||||
InitDocker --> |成功| CheckContainers["检查并清理遗留容器"]
|
||||
InitDocker --> |失败| Stop["停止服务启动"]
|
||||
CheckContainers --> |成功| StartProxy["启动代理服务"]
|
||||
CheckContainers --> |失败| LogWarn["记录警告,继续启动"]
|
||||
StartProxy --> |成功| StartServer["启动HTTP服务器"]
|
||||
StartProxy --> |失败| LogError["记录错误"]
|
||||
StartServer --> Complete["服务启动完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L31-L271)
|
||||
- [config.rs](file://crates/rcoder/src/config.rs#L253-L332)
|
||||
|
||||
## 日志分析
|
||||
|
||||
### 日志配置
|
||||
系统使用`tracing`库进行日志记录,支持多种输出格式和级别。
|
||||
|
||||
**日志配置特点**:
|
||||
- 按天滚动保存日志文件
|
||||
- 保留最近5天的日志
|
||||
- 同时输出到控制台和文件
|
||||
- 文件日志采用JSON格式,便于后续分析
|
||||
- 控制台日志采用简洁格式
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([日志初始化]) --> CreateDir["创建logs目录"]
|
||||
CreateDir --> SetupRolling["设置按天滚动的文件appender"]
|
||||
SetupRolling --> CreateFileLayer["创建文件日志层"]
|
||||
CreateFileLayer --> CreateConsoleLayer["创建控制台日志层"]
|
||||
CreateConsoleLayer --> SetPropagator["设置全局文本传播器"]
|
||||
SetPropagator --> InitSubscriber["初始化tracing subscriber"]
|
||||
InitSubscriber --> Complete["日志系统初始化成功"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L274-L318)
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L182-L229)
|
||||
|
||||
### 日志级别
|
||||
系统支持多种日志级别,便于不同场景下的调试和监控。
|
||||
|
||||
**日志级别**:
|
||||
- `error`:记录错误信息,需要立即关注
|
||||
- `warn`:记录警告信息,可能影响系统稳定性
|
||||
- `info`:记录重要事件和状态变化
|
||||
- `debug`:记录调试信息,用于问题排查
|
||||
- `trace`:记录详细跟踪信息,用于深度分析
|
||||
|
||||
**日志级别配置**:
|
||||
- 通过环境变量`RUST_LOG`设置
|
||||
- 支持按模块设置不同级别
|
||||
- 默认级别为`info`
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([日志级别]) --> Error["error: 记录错误信息"]
|
||||
Error --> Warn["warn: 记录警告信息"]
|
||||
Warn --> Info["info: 记录重要事件"]
|
||||
Info --> Debug["debug: 记录调试信息"]
|
||||
Debug --> Trace["trace: 记录详细跟踪信息"]
|
||||
Trace --> Complete["完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L308-L314)
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L217-L225)
|
||||
|
||||
### 日志内容
|
||||
日志记录了系统运行过程中的关键信息,便于问题排查和性能分析。
|
||||
|
||||
**日志内容包括**:
|
||||
- 服务启动和关闭信息
|
||||
- 配置加载和应用信息
|
||||
- 容器创建和销毁信息
|
||||
- 请求处理信息
|
||||
- 错误和异常信息
|
||||
|
||||
**日志字段**:
|
||||
- `timestamp`:时间戳
|
||||
- `level`:日志级别
|
||||
- `target`:日志来源
|
||||
- `message`:日志消息
|
||||
- `trace_id`:请求跟踪ID
|
||||
- `method`:HTTP方法
|
||||
- `uri`:请求URI
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([日志内容]) --> ServiceInfo["服务信息: 启动、关闭、状态"]
|
||||
ServiceInfo --> ConfigInfo["配置信息: 加载、应用、验证"]
|
||||
ConfigInfo --> ContainerInfo["容器信息: 创建、启动、停止、销毁"]
|
||||
ContainerInfo --> RequestInfo["请求信息: 处理、响应、错误"]
|
||||
RequestInfo --> ErrorInfo["错误信息: 异常、堆栈跟踪"]
|
||||
ErrorInfo --> Complete["完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L36-L271)
|
||||
- [tracing_middleware.rs](file://crates/rcoder/src/middleware/tracing_middleware.rs#L75-L128)
|
||||
|
||||
## 性能优化
|
||||
|
||||
### 容器清理优化
|
||||
系统实现了高效的容器清理机制,避免资源泄漏和性能下降。
|
||||
|
||||
**清理策略**:
|
||||
- 定时清理闲置容器
|
||||
- 启动时清理遗留容器
|
||||
- 关闭时清理所有容器
|
||||
- 并发清理多个容器
|
||||
|
||||
**优化措施**:
|
||||
- 使用`force remove`避免竞态条件
|
||||
- 过滤409冲突错误
|
||||
- 并发处理多个容器
|
||||
- 添加超时机制
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([容器清理]) --> CheckIdle["检查容器是否闲置"]
|
||||
CheckIdle --> |是| StopContainer["停止并删除容器"]
|
||||
CheckIdle --> |否| Skip["跳过"]
|
||||
StopContainer --> CheckError["检查错误类型"]
|
||||
CheckError --> |409冲突| Skip["跳过"]
|
||||
CheckError --> |其他错误| LogError["记录错误"]
|
||||
CheckError --> |成功| LogSuccess["记录成功"]
|
||||
LogSuccess --> NextContainer["处理下一个容器"]
|
||||
LogError --> NextContainer
|
||||
Skip --> NextContainer
|
||||
NextContainer --> |还有容器| CheckIdle
|
||||
NextContainer --> |无容器| Complete["清理完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [cleanup_task.rs](file://crates/rcoder/src/proxy_agent/cleanup_task.rs#L248-L429)
|
||||
- [container_stop.rs](file://crates/docker_manager/src/container_stop.rs#L82-L182)
|
||||
|
||||
### 请求处理优化
|
||||
系统使用`axum`框架处理HTTP请求,具有高性能和低延迟的特点。
|
||||
|
||||
**优化措施**:
|
||||
- 使用异步处理
|
||||
- 连接复用
|
||||
- 批量处理
|
||||
- 缓存机制
|
||||
|
||||
**性能指标**:
|
||||
- 请求处理时间
|
||||
- 并发连接数
|
||||
- 内存使用
|
||||
- CPU使用
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([请求处理]) --> ReceiveRequest["接收HTTP请求"]
|
||||
ReceiveRequest --> ParseRequest["解析请求"]
|
||||
ParseRequest --> ProcessRequest["处理请求"]
|
||||
ProcessRequest --> GenerateResponse["生成响应"]
|
||||
GenerateResponse --> SendResponse["发送响应"]
|
||||
SendResponse --> Complete["完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L222-L264)
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L132-L171)
|
||||
|
||||
### 资源管理优化
|
||||
系统实现了高效的资源管理机制,避免资源泄漏和性能下降。
|
||||
|
||||
**资源管理**:
|
||||
- 内存管理
|
||||
- CPU管理
|
||||
- 网络管理
|
||||
- 存储管理
|
||||
|
||||
**优化措施**:
|
||||
- 资源限制
|
||||
- 资源回收
|
||||
- 资源监控
|
||||
- 资源优化
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([资源管理]) --> Memory["内存管理: 分配、释放、监控"]
|
||||
Memory --> CPU["CPU管理: 调度、限制、监控"]
|
||||
CPU --> Network["网络管理: 连接、带宽、监控"]
|
||||
Network --> Storage["存储管理: 读写、缓存、监控"]
|
||||
Storage --> Complete["完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [manager.rs](file://crates/docker_manager/src/manager.rs#L147-L175)
|
||||
- [types.rs](file://crates/docker_manager/src/types.rs#L51-L60)
|
||||
|
||||
## 安全考虑
|
||||
|
||||
### 容器安全
|
||||
系统在容器创建和运行时实施了多种安全措施。
|
||||
|
||||
**安全措施**:
|
||||
- 禁用`NET_RAW`和`NET_ADMIN`能力
|
||||
- 禁用特权模式
|
||||
- 使用非root用户运行
|
||||
- 限制资源使用
|
||||
- 网络隔离
|
||||
|
||||
**安全配置**:
|
||||
- `cap_drop`:移除危险能力
|
||||
- `privileged`:禁用特权模式
|
||||
- `user`:指定运行用户
|
||||
- `memory`:限制内存使用
|
||||
- `nano_cpus`:限制CPU使用
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([容器安全]) --> DropCapabilities["移除NET_RAW和NET_ADMIN能力"]
|
||||
DropCapabilities --> DisablePrivileged["禁用特权模式"]
|
||||
DisablePrivileged --> LimitResources["限制资源使用"]
|
||||
LimitResources --> NetworkIsolation["网络隔离"]
|
||||
NetworkIsolation --> Complete["完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [manager.rs](file://crates/docker_manager/src/manager.rs#L153-L165)
|
||||
- [types.rs](file://crates/docker_manager/src/types.rs#L51-L60)
|
||||
|
||||
### 网络安全
|
||||
系统在网络安全方面实施了多种措施,防止未授权访问和攻击。
|
||||
|
||||
**安全措施**:
|
||||
- 使用安全的网络配置
|
||||
- 限制端口暴露
|
||||
- 使用防火墙规则
|
||||
- 监控网络流量
|
||||
- 防止DDoS攻击
|
||||
|
||||
**网络配置**:
|
||||
- `network_mode`:设置网络模式
|
||||
- `port_bindings`:限制端口映射
|
||||
- `host_config`:配置主机安全
|
||||
- `networking_config`:配置网络安全
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([网络安全]) --> SecureNetwork["使用安全的网络配置"]
|
||||
SecureNetwork --> LimitPorts["限制端口暴露"]
|
||||
LimitPorts --> Firewall["使用防火墙规则"]
|
||||
Firewall --> MonitorTraffic["监控网络流量"]
|
||||
MonitorTraffic --> PreventDDoS["防止DDoS攻击"]
|
||||
PreventDDoS --> Complete["完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [manager.rs](file://crates/docker_manager/src/manager.rs#L178-L206)
|
||||
- [types.rs](file://crates/docker_manager/src/types.rs#L24-L26)
|
||||
|
||||
### 数据安全
|
||||
系统在数据安全方面实施了多种措施,保护用户数据和系统数据。
|
||||
|
||||
**安全措施**:
|
||||
- 数据加密
|
||||
- 访问控制
|
||||
- 数据备份
|
||||
- 数据审计
|
||||
- 数据隔离
|
||||
|
||||
**数据保护**:
|
||||
- `encryption`:数据加密
|
||||
- `authentication`:身份验证
|
||||
- `authorization`:授权
|
||||
- `backup`:数据备份
|
||||
- `audit`:数据审计
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([数据安全]) --> EncryptData["数据加密"]
|
||||
EncryptData --> AccessControl["访问控制"]
|
||||
AccessControl --> BackupData["数据备份"]
|
||||
BackupData --> AuditData["数据审计"]
|
||||
AuditData --> IsolateData["数据隔离"]
|
||||
IsolateData --> Complete["完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [config.rs](file://crates/rcoder/src/config.rs#L43-L44)
|
||||
- [types.rs](file://crates/docker_manager/src/types.rs#L15-L16)
|
||||
|
||||
## 配置选项与参数
|
||||
|
||||
### 主要配置选项
|
||||
系统提供了多种配置选项,便于用户根据需求进行定制。
|
||||
|
||||
**配置文件**:`config.yml`
|
||||
|
||||
**主要配置项**:
|
||||
- `default_agent`:默认使用的AI代理类型
|
||||
- `projects_dir`:项目工作目录
|
||||
- `port`:主服务端口
|
||||
- `proxy_config`:反向代理配置
|
||||
- `docker_config`:Docker配置
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([配置选项]) --> DefaultAgent["default_agent: 默认AI代理类型"]
|
||||
DefaultAgent --> ProjectsDir["projects_dir: 项目工作目录"]
|
||||
ProjectsDir --> Port["port: 主服务端口"]
|
||||
Port --> ProxyConfig["proxy_config: 反向代理配置"]
|
||||
ProxyConfig --> DockerConfig["docker_config: Docker配置"]
|
||||
DockerConfig --> Complete["完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [config.rs](file://crates/rcoder/src/config.rs#L38-L50)
|
||||
- [config.rs](file://crates/agent_runner/src/config.rs#L39-L49)
|
||||
|
||||
### 反向代理配置
|
||||
反向代理配置允许用户自定义代理服务的行为。
|
||||
|
||||
**配置项**:
|
||||
- `listen_port`:代理服务监听端口
|
||||
- `default_backend_port`:默认后端服务端口
|
||||
- `backend_host`:后端服务主机地址
|
||||
- `port_param`:端口参数名称
|
||||
- `health_check`:健康检查配置
|
||||
|
||||
**健康检查配置**:
|
||||
- `enabled`:是否启用健康检查
|
||||
- `interval_seconds`:检查间隔(秒)
|
||||
- `timeout_seconds`:超时时间(秒)
|
||||
- `healthy_threshold`:健康阈值
|
||||
- `unhealthy_threshold`:不健康阈值
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([代理配置]) --> ListenPort["listen_port: 监听端口"]
|
||||
ListenPort --> DefaultBackend["default_backend_port: 默认后端端口"]
|
||||
DefaultBackend --> BackendHost["backend_host: 后端主机"]
|
||||
BackendHost --> PortParam["port_param: 端口参数"]
|
||||
PortParam --> HealthCheck["health_check: 健康检查"]
|
||||
HealthCheck --> Enabled["enabled: 是否启用"]
|
||||
HealthCheck --> Interval["interval_seconds: 间隔"]
|
||||
HealthCheck --> Timeout["timeout_seconds: 超时"]
|
||||
HealthCheck --> Healthy["healthy_threshold: 健康阈值"]
|
||||
HealthCheck --> Unhealthy["unhealthy_threshold: 不健康阈值"]
|
||||
Unhealthy --> Complete["完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [config.rs](file://crates/rcoder/src/config.rs#L68-L80)
|
||||
- [config.rs](file://crates/agent_runner/src/config.rs#L61-L73)
|
||||
|
||||
### Docker配置
|
||||
Docker配置允许用户自定义容器的行为。
|
||||
|
||||
**配置项**:
|
||||
- `multi_image_config`:多镜像配置
|
||||
- `network_mode`:网络模式
|
||||
- `work_dir`:工作目录
|
||||
- `auto_cleanup`:自动清理
|
||||
- `container_ttl_seconds`:容器存活时间(秒)
|
||||
|
||||
**多镜像配置**:
|
||||
- `services`:服务配置
|
||||
- `global_defaults`:全局默认配置
|
||||
- `selection_strategy`:选择策略
|
||||
- `cache_config`:缓存配置
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([Docker配置]) --> MultiImage["multi_image_config: 多镜像配置"]
|
||||
MultiImage --> NetworkMode["network_mode: 网络模式"]
|
||||
NetworkMode --> WorkDir["work_dir: 工作目录"]
|
||||
WorkDir --> AutoCleanup["auto_cleanup: 自动清理"]
|
||||
AutoCleanup --> ContainerTTL["container_ttl_seconds: 容器存活时间"]
|
||||
ContainerTTL --> Complete["完成"]
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [config.rs](file://crates/rcoder/src/config.rs#L82-L96)
|
||||
- [types.rs](file://crates/docker_manager/src/types.rs#L176-L195)
|
||||
|
||||
## 组件关系
|
||||
|
||||
### 核心组件关系
|
||||
系统由多个核心组件组成,它们之间有明确的关系和依赖。
|
||||
|
||||
**主要组件**:
|
||||
- `rcoder`:主服务
|
||||
- `agent_runner`:代理运行器
|
||||
- `docker_manager`:Docker管理器
|
||||
- `pingora-proxy`:反向代理
|
||||
- `shared_types`:共享类型
|
||||
|
||||
**组件关系**:
|
||||
- `rcoder`依赖`docker_manager`和`pingora-proxy`
|
||||
- `agent_runner`依赖`docker_manager`
|
||||
- `rcoder`和`agent_runner`共享`shared_types`
|
||||
- `pingora-proxy`提供反向代理功能
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
class Rcoder {
|
||||
+start()
|
||||
+load_config()
|
||||
+init_docker_manager()
|
||||
+start_proxy()
|
||||
}
|
||||
class AgentRunner {
|
||||
+start()
|
||||
+load_config()
|
||||
+run_agent()
|
||||
}
|
||||
class DockerManager {
|
||||
+create_container()
|
||||
+stop_container()
|
||||
+get_container_info()
|
||||
}
|
||||
class PingoraProxy {
|
||||
+start()
|
||||
+add_backend()
|
||||
+health_check()
|
||||
}
|
||||
class SharedTypes {
|
||||
+AgentType
|
||||
+ContainerInfo
|
||||
+AppError
|
||||
}
|
||||
Rcoder --> DockerManager : "使用"
|
||||
Rcoder --> PingoraProxy : "使用"
|
||||
AgentRunner --> DockerManager : "使用"
|
||||
Rcoder --> SharedTypes : "共享"
|
||||
AgentRunner --> SharedTypes : "共享"
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L1-L271)
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L1-L179)
|
||||
|
||||
### 服务启动流程
|
||||
服务启动时,各个组件按照特定顺序初始化和启动。
|
||||
|
||||
**启动流程**:
|
||||
1. 初始化遥测系统
|
||||
2. 解析命令行参数
|
||||
3. 加载配置文件
|
||||
4. 创建项目工作目录
|
||||
5. 初始化Docker管理器
|
||||
6. 启动代理服务
|
||||
7. 启动HTTP服务器
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Main as "main函数"
|
||||
participant Telemetry as "遥测系统"
|
||||
participant Config as "配置系统"
|
||||
participant Docker as "Docker管理器"
|
||||
participant Proxy as "代理服务"
|
||||
participant Server as "HTTP服务器"
|
||||
Main->>Telemetry : init_telemetry()
|
||||
Telemetry-->>Main : 初始化成功
|
||||
Main->>Config : load_config_with_args()
|
||||
Config-->>Main : 返回配置
|
||||
Main->>Main : create_dir_all(projects_dir)
|
||||
Main->>Docker : init_global_docker_manager()
|
||||
Docker-->>Main : 初始化成功
|
||||
Main->>Proxy : start()
|
||||
Proxy-->>Main : 启动成功
|
||||
Main->>Server : serve()
|
||||
Server-->>Main : 服务器运行
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L31-L271)
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L30-L179)
|
||||
|
||||
### 请求处理流程
|
||||
HTTP请求的处理流程涉及多个组件的协作。
|
||||
|
||||
**处理流程**:
|
||||
1. 接收HTTP请求
|
||||
2. 通过追踪中间件处理
|
||||
3. 路由到相应处理器
|
||||
4. 处理请求
|
||||
5. 生成响应
|
||||
6. 返回响应
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client as "客户端"
|
||||
participant Server as "HTTP服务器"
|
||||
participant Middleware as "追踪中间件"
|
||||
participant Router as "路由"
|
||||
participant Handler as "处理器"
|
||||
Client->>Server : HTTP请求
|
||||
Server->>Middleware : 调用中间件
|
||||
Middleware->>Middleware : 生成trace_id
|
||||
Middleware->>Middleware : 创建span
|
||||
Middleware->>Router : 调用下一个处理器
|
||||
Router->>Handler : 路由到处理器
|
||||
Handler->>Handler : 处理请求
|
||||
Handler-->>Router : 返回响应
|
||||
Router-->>Middleware : 返回响应
|
||||
Middleware-->>Server : 返回响应
|
||||
Server-->>Client : HTTP响应
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [tracing_middleware.rs](file://crates/rcoder/src/middleware/tracing_middleware.rs#L72-L128)
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L220-L221)
|
||||
284
qiming-rcoder/.qoder/repowiki/zh/content/故障排除/日志分析.md
Normal file
284
qiming-rcoder/.qoder/repowiki/zh/content/故障排除/日志分析.md
Normal file
@@ -0,0 +1,284 @@
|
||||
# 日志分析
|
||||
|
||||
<cite>
|
||||
**本文档引用的文件**
|
||||
- [chat_handler.rs](file://crates/agent_runner/src/handler/chat_handler.rs)
|
||||
- [agent_service.rs](file://crates/agent_runner/src/proxy_agent/agent_service.rs)
|
||||
- [tracing_middleware.rs](file://crates/agent_runner/src/middleware/tracing_middleware.rs)
|
||||
- [chat_handler.rs](file://crates/rcoder/src/handler/chat_handler.rs)
|
||||
- [tracing_middleware.rs](file://crates/rcoder/src/middleware/tracing_middleware.rs)
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs)
|
||||
- [main.rs](file://crates/rcoder/src/main.rs)
|
||||
- [manager.rs](file://crates/docker_manager/src/manager.rs)
|
||||
- [container_manager.rs](file://crates/rcoder/src/service/container_manager.rs)
|
||||
</cite>
|
||||
|
||||
## 目录
|
||||
1. [日志结构与关键日志点](#日志结构与关键日志点)
|
||||
2. [Tracing与OpenTelemetry集成](#tracing与opentelemetry集成)
|
||||
3. [关键操作日志记录模式](#关键操作日志记录模式)
|
||||
4. [性能瓶颈与错误识别](#性能瓶颈与错误识别)
|
||||
5. [日志级别配置与过滤](#日志级别配置与过滤)
|
||||
6. [Docker容器日志集成](#docker容器日志集成)
|
||||
7. [端到端问题定位](#端到端问题定位)
|
||||
|
||||
## 日志结构与关键日志点
|
||||
|
||||
RCoder系统采用结构化日志记录,结合`tracing`和`OpenTelemetry`实现分布式追踪。日志系统在`main.rs`中初始化,配置了文件和控制台双输出。文件日志采用JSON格式,便于后续分析和监控系统集成。
|
||||
|
||||
日志文件按天滚动保存,保留最近5天的日志文件,文件名前缀分别为`agent-runner`和`rcoder`,分别对应不同的服务组件。日志目录自动创建于项目根目录下的`logs`文件夹中。
|
||||
|
||||
关键日志点包括:
|
||||
- HTTP请求的开始和结束
|
||||
- 容器创建、启动和停止
|
||||
- 代理服务的状态转换
|
||||
- 会话管理操作
|
||||
- 错误和异常处理
|
||||
|
||||
**Section sources**
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L182-L231)
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L275-L320)
|
||||
|
||||
## Tracing与OpenTelemetry集成
|
||||
|
||||
系统通过`tracing_middleware.rs`实现了HTTP请求追踪中间件,为每个请求生成唯一的`trace_id`,并创建相应的span用于日志跟踪。中间件自动从请求头中提取`trace_id`,支持`x-trace-id`、`x-request-id`、`traceparent`等常见追踪头。
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client as "客户端"
|
||||
participant Rcoder as "RCoder服务"
|
||||
participant Agent as "AgentRunner服务"
|
||||
Client->>Rcoder : POST /chat (带trace_id)
|
||||
Rcoder->>Rcoder : 生成/提取trace_id
|
||||
Rcoder->>Rcoder : 记录请求开始
|
||||
Rcoder->>Agent : 转发请求 (传递trace_id)
|
||||
Agent->>Agent : 记录代理处理
|
||||
Agent-->>Rcoder : 返回响应
|
||||
Rcoder->>Rcoder : 记录请求完成
|
||||
Rcoder-->>Client : 返回响应
|
||||
```
|
||||
|
||||
**Diagram sources**
|
||||
- [tracing_middleware.rs](file://crates/agent_runner/src/middleware/tracing_middleware.rs#L1-L179)
|
||||
- [tracing_middleware.rs](file://crates/rcoder/src/middleware/tracing_middleware.rs#L1-L179)
|
||||
|
||||
**Section sources**
|
||||
- [tracing_middleware.rs](file://crates/agent_runner/src/middleware/tracing_middleware.rs#L1-L179)
|
||||
- [tracing_middleware.rs](file://crates/rcoder/src/middleware/tracing_middleware.rs#L1-L179)
|
||||
|
||||
## 关键操作日志记录模式
|
||||
|
||||
### HTTP入口日志记录
|
||||
|
||||
在`chat_handler.rs`中,HTTP请求的处理过程有详细的日志记录。当处理聊天请求时,系统会记录请求的开始、项目ID的生成、会话状态的检查等关键信息。
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Start([开始处理聊天请求]) --> Validate["验证请求参数"]
|
||||
Validate --> CheckStatus["检查Agent状态"]
|
||||
CheckStatus --> RemoveSession["移除旧会话"]
|
||||
RemoveSession --> CreateWorkspace["创建项目工作目录"]
|
||||
CreateWorkspace --> SelectAgent["选择Agent类型"]
|
||||
SelectAgent --> SendTask["发送本地任务"]
|
||||
SendTask --> AwaitResponse["等待代理响应"]
|
||||
AwaitResponse --> HandleResult["处理响应结果"]
|
||||
HandleResult --> End([返回响应])
|
||||
style Start fill:#4CAF50,stroke:#388E3C
|
||||
style End fill:#4CAF50,stroke:#388E3C
|
||||
```
|
||||
|
||||
**Diagram sources**
|
||||
- [chat_handler.rs](file://crates/agent_runner/src/handler/chat_handler.rs#L176-L320)
|
||||
|
||||
**Section sources**
|
||||
- [chat_handler.rs](file://crates/agent_runner/src/handler/chat_handler.rs#L176-L320)
|
||||
|
||||
### 代理服务日志记录
|
||||
|
||||
在`agent_service.rs`中,代理服务的启动和管理有明确的日志记录。系统通过`AcpAgentService` trait定义了统一的接口,不同类型的代理(如Claude、Codex)实现该接口。
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
class AcpAgentService {
|
||||
<<trait>>
|
||||
+start_agent_service(chat_prompt, model_provider) Result~AcpConnectionInfo~
|
||||
+agent_type_name() &'static str
|
||||
}
|
||||
class AgentType {
|
||||
+Claude
|
||||
+Codex
|
||||
}
|
||||
AcpAgentService <|-- AgentType : "实现"
|
||||
AgentType --> "ClaudeCodeAgent" : "调用"
|
||||
AgentType --> "CodexAgent" : "调用"
|
||||
```
|
||||
|
||||
**Diagram sources**
|
||||
- [agent_service.rs](file://crates/agent_runner/src/proxy_agent/agent_service.rs#L7-L62)
|
||||
|
||||
**Section sources**
|
||||
- [agent_service.rs](file://crates/agent_runner/src/proxy_agent/agent_service.rs#L7-L62)
|
||||
|
||||
## 性能瓶颈与错误识别
|
||||
|
||||
### 性能瓶颈识别
|
||||
|
||||
通过分析日志中的时间戳和处理时长,可以识别系统性能瓶颈。关键的性能指标包括:
|
||||
|
||||
- HTTP请求处理时间
|
||||
- 容器创建和启动时间
|
||||
- 代理服务响应时间
|
||||
- 网络通信延迟
|
||||
|
||||
日志中的emoji标记有助于快速识别不同类型的日志:
|
||||
- 🚀 表示开始处理
|
||||
- ✅ 表示成功完成
|
||||
- ❌ 表示错误或失败
|
||||
- ⚠️ 表示警告或需要注意的情况
|
||||
|
||||
### 错误堆栈分析
|
||||
|
||||
系统在关键错误点记录详细的错误信息,包括错误代码、消息和堆栈跟踪。例如,在`chat_handler.rs`中,当代理处理失败时,系统会记录错误详情:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[收到代理执行结果] --> B{结果是否成功?}
|
||||
B --> |是| C[检查响应错误]
|
||||
C --> D{有错误?}
|
||||
D --> |是| E[记录错误日志]
|
||||
D --> |否| F[记录成功日志]
|
||||
B --> |否| G[记录接收失败日志]
|
||||
style E fill:#f44336,stroke:#d32f2f
|
||||
style G fill:#f44336,stroke:#d32f2f
|
||||
```
|
||||
|
||||
**Diagram sources**
|
||||
- [chat_handler.rs](file://crates/agent_runner/src/handler/chat_handler.rs#L289-L318)
|
||||
|
||||
**Section sources**
|
||||
- [chat_handler.rs](file://crates/agent_runner/src/handler/chat_handler.rs#L289-L318)
|
||||
|
||||
## 日志级别配置与过滤
|
||||
|
||||
### 日志级别配置
|
||||
|
||||
系统通过`tracing_subscriber::EnvFilter`配置日志级别,支持环境变量配置。默认配置如下:
|
||||
|
||||
- `rcoder=debug`:RCoder组件的调试级别
|
||||
- `tower_http=debug`:HTTP中间件的调试级别
|
||||
- `axum_tracing_opentelemetry=info`:OpenTelemetry追踪的普通级别
|
||||
|
||||
### 日志过滤技巧
|
||||
|
||||
可以通过环境变量`RUST_LOG`来动态调整日志级别,例如:
|
||||
|
||||
```bash
|
||||
# 只显示错误日志
|
||||
export RUST_LOG=error
|
||||
|
||||
# 显示RCoder组件的调试日志
|
||||
export RUST_LOG=rcoder=debug
|
||||
|
||||
# 显示所有组件的详细日志
|
||||
export RUST_LOG=debug
|
||||
```
|
||||
|
||||
**Section sources**
|
||||
- [main.rs](file://crates/agent_runner/src/main.rs#L219-L221)
|
||||
- [main.rs](file://crates/rcoder/src/main.rs#L310-L311)
|
||||
|
||||
## Docker容器日志集成
|
||||
|
||||
### 容器管理日志
|
||||
|
||||
`docker_manager`模块提供了完整的容器管理功能,包括创建、启动、停止和清理容器。每个操作都有相应的日志记录:
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Rcoder as "RCoder服务"
|
||||
participant Docker as "Docker守护进程"
|
||||
Rcoder->>Docker : 创建容器
|
||||
Docker-->>Rcoder : 返回容器ID
|
||||
Rcoder->>Rcoder : 记录容器创建
|
||||
Rcoder->>Docker : 启动容器
|
||||
Docker-->>Rcoder : 启动成功
|
||||
Rcoder->>Rcoder : 记录容器启动
|
||||
Rcoder->>Docker : 检查容器健康
|
||||
Docker-->>Rcoder : 健康状态
|
||||
Rcoder->>Rcoder : 记录健康检查
|
||||
```
|
||||
|
||||
**Diagram sources**
|
||||
- [manager.rs](file://crates/docker_manager/src/manager.rs#L81-L352)
|
||||
|
||||
**Section sources**
|
||||
- [manager.rs](file://crates/docker_manager/src/manager.rs#L81-L352)
|
||||
|
||||
### 容器日志获取
|
||||
|
||||
系统提供了获取容器日志的功能,可以通过`get_container_logs`方法获取指定容器的日志:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[获取容器日志] --> B{容器是否存在?}
|
||||
B --> |是| C[调用Docker API]
|
||||
C --> D[获取日志流]
|
||||
D --> E[合并日志内容]
|
||||
E --> F[返回日志字符串]
|
||||
B --> |否| G[返回错误]
|
||||
style G fill:#f44336,stroke:#d32f2f
|
||||
```
|
||||
|
||||
**Diagram sources**
|
||||
- [manager.rs](file://crates/docker_manager/src/manager.rs#L588-L623)
|
||||
|
||||
**Section sources**
|
||||
- [manager.rs](file://crates/docker_manager/src/manager.rs#L588-L623)
|
||||
|
||||
## 端到端问题定位
|
||||
|
||||
### 请求生命周期追踪
|
||||
|
||||
通过`trace_id`可以追踪请求从HTTP入口到代理执行的完整生命周期。系统在`chat_handler.rs`中实现了请求转发,保持了`trace_id`的传递:
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client as "客户端"
|
||||
participant Rcoder as "RCoder服务"
|
||||
participant Container as "Agent容器"
|
||||
Client->>Rcoder : 发送请求 (带trace_id)
|
||||
Rcoder->>Rcoder : 记录请求开始
|
||||
Rcoder->>Container : 转发请求 (传递trace_id)
|
||||
Container->>Container : 处理请求
|
||||
Container-->>Rcoder : 返回响应
|
||||
Rcoder->>Rcoder : 记录请求完成
|
||||
Rcoder-->>Client : 返回响应
|
||||
```
|
||||
|
||||
**Diagram sources**
|
||||
- [chat_handler.rs](file://crates/rcoder/src/handler/chat_handler.rs#L110-L320)
|
||||
|
||||
**Section sources**
|
||||
- [chat_handler.rs](file://crates/rcoder/src/handler/chat_handler.rs#L110-L320)
|
||||
|
||||
### 状态转换异常检测
|
||||
|
||||
系统通过日志记录关键状态的转换,可以检测异常状态。例如,在容器管理中,系统会检查容器的健康状态:
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Idle
|
||||
Idle --> Creating : "创建容器"
|
||||
Creating --> Running : "启动成功"
|
||||
Creating --> Error : "启动失败"
|
||||
Running --> Stopped : "停止容器"
|
||||
Running --> Error : "健康检查失败"
|
||||
Stopped --> Idle : "清理完成"
|
||||
Error --> Idle : "错误处理"
|
||||
```
|
||||
|
||||
**Diagram sources**
|
||||
- [manager.rs](file://crates/docker_manager/src/manager.rs#L758-L800)
|
||||
|
||||
**Section sources**
|
||||
- [manager.rs](file://crates/docker_manager/src/manager.rs#L758-L800)
|
||||
Reference in New Issue
Block a user