添加qiming-rcoder模块
This commit is contained in:
Codex
390
qiming-rcoder/k8s/README.md
Normal file
390
qiming-rcoder/k8s/README.md
Normal file
@@ -0,0 +1,390 @@
|
||||
# RCoder K8s 部署配置
|
||||
|
||||
本目录提供 **三种部署方式**，按场景选择：
|
||||
|
||||
| 场景 | 方式 | 入口 |
|
||||
|------|------|------|
|
||||
| 日常开发 / 在线环境 | **Kustomize** | `deploy-dev.sh` / `deploy-prod.sh` |
|
||||
| 参数化部署 / 对外交付 | **Helm** (双栈并存) | `helm install ... k8s/helm/rcoder` |
|
||||
| 政企客户 / 完全断网 | **离线 Bundle** | `make k8s-offline-bundle` → `install.sh` |
|
||||
|
||||
另外还有可选的 **本地镜像加速**（`register2/`），面向多节点 k3s / 频繁构建场景，详见下文"本地镜像加速"一节。
|
||||
|
||||
## 目录结构
|
||||
|
||||
```
|
||||
k8s/
|
||||
├── manifests/ # Kustomize manifests (日常开发 + 在线部署)
|
||||
│ ├── base/ # 基础配置 (不可直接部署, 需走 overlay)
|
||||
│ │ ├── storage/ # postgresql / minio / juicefs-init / juicefs-pvc
|
||||
│ │ └── rcoder/ # rcoder Deployment/Service/NetworkPolicy/PDB/SA+ClusterRole
|
||||
│ └── overlays/
|
||||
│ ├── dev/ # 开发环境 (nuwax-rcoder-dev, NodePort 30080)
|
||||
│ └── prod/ # 生产环境 (nuwax-rcoder-prod, NodePort 30081)
|
||||
│
|
||||
├── helm/ # Helm chart (对外交付 + 离线部署源)
|
||||
│ └── rcoder/
|
||||
│ ├── Chart.yaml
|
||||
│ ├── values.yaml # 默认值 (共享)
|
||||
│ ├── values-dev.yaml # dev 覆盖
|
||||
│ ├── values-prod.yaml # prod 覆盖
|
||||
│ ├── values-offline.yaml # 离线 registry 覆盖
|
||||
│ └── templates/ # K8s 模板
|
||||
│
|
||||
├── offline/ # 离线 bundle 素材
|
||||
│ ├── images.txt # 镜像清单 (单一数据源)
|
||||
│ ├── install.sh # 离线安装脚本 (direct / registry 双模式)
|
||||
│ ├── rewrite-registry.sh # 重打 tag + 推送私有 registry
|
||||
│ └── README.md # 交付给政企客户的部署手册
|
||||
│
|
||||
├── register2/ # 本地私有镜像仓库 (docker compose)
|
||||
│ ├── docker-compose.yml # registry:2 容器 + 数据卷
|
||||
│ └── README.md # 作 k3s mirror / 离线 push 目标的用法
|
||||
│
|
||||
├── deploy-dev.sh # [在线] Kustomize 开发环境一键部署
|
||||
├── deploy-prod.sh # [在线] Kustomize 生产环境一键部署 (含占位符密码守卫)
|
||||
├── undeploy.sh # [在线] 清理部署 (ENV=dev|prod)
|
||||
│
|
||||
├── scripts/ # 辅助脚本
|
||||
│ ├── deploy-juicefs.sh # 仅部署存储层 (ENV=dev|prod)
|
||||
│ ├── test-chat.sh # 功能冒烟测试 (NAMESPACE 可覆盖)
|
||||
│ ├── install-k3s-registry-mirrors-cn.sh # K3s 节点镜像加速配置 (支持 REGISTRY_HOST)
|
||||
│ └── k3s-registries-cn.yaml # 镜像加速配置模板
|
||||
│
|
||||
└── README.md
|
||||
```
|
||||
|
||||
dev 和 prod 可以并存于同一个集群：
|
||||
- 集群级资源（StorageClass / ClusterRoleBinding）命名独立：`juicefs-sc-dev` / `juicefs-sc-prod`、`rcoder-pods-crb-dev` / `rcoder-pods-crb-prod`
|
||||
- NodePort 错开：dev=30080 / prod=30081
|
||||
- ClusterRole `rcoder-pods-clusterrole` 是唯一共享的集群级资源（两个 overlay 写入完全一致，幂等）
|
||||
|
||||
---
|
||||
|
||||
## 快速开始
|
||||
|
||||
### 开发环境
|
||||
|
||||
```bash
|
||||
# 首次完整部署（含 Longhorn、JuiceFS CSI、open-iscsi 检查）
|
||||
./k8s/deploy-dev.sh
|
||||
|
||||
# 常用
|
||||
kubectl apply -k k8s/manifests/overlays/dev # 部署/更新
|
||||
kubectl delete -k k8s/manifests/overlays/dev # 删除
|
||||
kubectl get all -n nuwax-rcoder-dev # 查看状态
|
||||
k8s/scripts/test-chat.sh # 冒烟测试
|
||||
```
|
||||
|
||||
### 生产环境
|
||||
|
||||
> **重要**：首次部署前必须替换 `k8s/manifests/overlays/prod/credentials.yaml` 和 `juicefs-secret.yaml` 里的 `CHANGE-ME-BEFORE-DEPLOY` 占位符；`deploy-prod.sh` 有守卫检查，发现占位符会拒绝部署。若使用 SealedSecret / ExternalSecret 在集群外注入，设置 `FORCE_PROD_DEPLOY=1` 跳过。
|
||||
|
||||
```bash
|
||||
# 首次完整部署
|
||||
./k8s/deploy-prod.sh
|
||||
|
||||
# 跳过占位符检查（仅当通过外部密钥注入方案时）
|
||||
FORCE_PROD_DEPLOY=1 ./k8s/deploy-prod.sh
|
||||
|
||||
# 常用
|
||||
kubectl apply -k k8s/manifests/overlays/prod
|
||||
kubectl get all -n nuwax-rcoder-prod
|
||||
NAMESPACE=nuwax-rcoder-prod k8s/scripts/test-chat.sh
|
||||
```
|
||||
|
||||
### 清理
|
||||
|
||||
```bash
|
||||
ENV=dev ./k8s/undeploy.sh # 清理 dev
|
||||
ENV=prod ./k8s/undeploy.sh # 清理 prod
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Makefile 快捷命令
|
||||
|
||||
在项目根目录执行：
|
||||
|
||||
```bash
|
||||
# 镜像
|
||||
make dev-build-k8s # 构建并推送 K8s 镜像 (含 k3s 本地 import + rollout restart)
|
||||
|
||||
# 部署/清理（默认走 overlay）
|
||||
make deploy-dev # kubectl apply -k overlays/dev
|
||||
make deploy-prod # kubectl apply -k overlays/prod
|
||||
make undeploy-dev # 清理 dev
|
||||
make undeploy-prod # 清理 prod
|
||||
|
||||
# 开发闭环
|
||||
make dev-up-k8s # 部署 dev
|
||||
make dev-restart-k8s # 重新构建镜像 + 重部署 dev
|
||||
make dev-down-k8s # 清理 dev
|
||||
make dev-logs-k8s # 跟随 rcoder 日志
|
||||
|
||||
# 本地验证（不部署）
|
||||
make kustomize-build # 分别 build base / dev / prod
|
||||
|
||||
# 离线部署 (政企内网)
|
||||
make k8s-offline-bundle # 打包完整离线包 (images + helm + manifests)
|
||||
make k8s-offline-import BUNDLE=<tgz> INSTALL_ARGS="--mode=direct --env=dev"
|
||||
make k8s-offline-images-list # 打印所有离线依赖镜像清单
|
||||
make k8s-offline-clean # 清理构建产物
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Helm 部署 (参数化 / 对外交付)
|
||||
|
||||
Helm chart 与 Kustomize **双栈并存、长期维护**。修改 `manifests/base/*.yaml` 时请同步更新 `helm/rcoder/templates/*` 对应模板。
|
||||
|
||||
```bash
|
||||
# dev (本地或在线环境)
|
||||
helm install rcoder-dev k8s/helm/rcoder \
|
||||
--namespace nuwax-rcoder-dev --create-namespace \
|
||||
-f k8s/helm/rcoder/values-dev.yaml
|
||||
|
||||
# 同集群已有 Kustomize 部署的话, 加 --set rcoder.clusterRole.create=false
|
||||
# (避免 Helm 尝试接管 Kustomize 已创建的 ClusterRole)
|
||||
helm install rcoder-helm-test k8s/helm/rcoder \
|
||||
--namespace nuwax-rcoder-helm-test --create-namespace \
|
||||
-f k8s/helm/rcoder/values-dev.yaml \
|
||||
--set rcoder.service.nodePort=30082 \
|
||||
--set juicefs.storageClass.name=juicefs-sc-helm-test \
|
||||
--set rcoder.clusterRole.create=false
|
||||
|
||||
# prod (密码通过 --set 或 external secret 注入)
|
||||
helm install rcoder-prod k8s/helm/rcoder \
|
||||
--namespace nuwax-rcoder-prod --create-namespace \
|
||||
-f k8s/helm/rcoder/values-prod.yaml \
|
||||
--set credentials.postgresql.password=<real> \
|
||||
--set credentials.minio.rootPassword=<real>
|
||||
|
||||
# 升级
|
||||
helm upgrade rcoder-dev k8s/helm/rcoder -f k8s/helm/rcoder/values-dev.yaml
|
||||
|
||||
# 卸载
|
||||
helm uninstall rcoder-dev --namespace nuwax-rcoder-dev
|
||||
```
|
||||
|
||||
### 渲染对比 (开发调试)
|
||||
|
||||
```bash
|
||||
helm template rcoder-dev k8s/helm/rcoder -f k8s/helm/rcoder/values-dev.yaml > /tmp/helm-dev.yaml
|
||||
kubectl kustomize k8s/manifests/overlays/dev > /tmp/kustomize-dev.yaml
|
||||
diff <(grep -E 'image:|nodePort:|storageClassName:' /tmp/helm-dev.yaml | sort -u) \
|
||||
<(grep -E 'image:|nodePort:|storageClassName:' /tmp/kustomize-dev.yaml | sort -u)
|
||||
```
|
||||
|
||||
### 集群级资源命名
|
||||
|
||||
Helm 为每个 release 自动生成独立的集群级资源名：
|
||||
|
||||
| 资源 | dev release (`rcoder-dev`) | prod release (`rcoder-prod`) |
|
||||
|------|---------------------------|------------------------------|
|
||||
| StorageClass | `juicefs-sc-dev` (显式 override) | `juicefs-sc-prod` (显式 override) |
|
||||
| ClusterRoleBinding | `rcoder-dev-pods-crb` (自动) | `rcoder-prod-pods-crb` (自动) |
|
||||
| ClusterRole | `rcoder-pods-clusterrole` (共享, `resource-policy: keep`) |
|
||||
|
||||
---
|
||||
|
||||
## 离线部署 (政企内网, 完全断网)
|
||||
|
||||
当客户内网无公网访问时，使用 **Bundle 模式**：在有网的构建机打包所有镜像、helm charts、manifests，产出单个 `tar.gz`；离线机器上解压运行 `install.sh` 完成部署。
|
||||
|
||||
### 构建 bundle (有网构建机)
|
||||
|
||||
```bash
|
||||
# 拉取所有镜像 + 打包 helm charts + 下载 Longhorn/JuiceFS CSI manifest + 压缩
|
||||
make k8s-offline-bundle
|
||||
|
||||
# 输出: dist/rcoder-offline-<version>-<arch>.tar.gz (约 1.7-2 GB, 共 27 个镜像)
|
||||
ls -lh dist/rcoder-offline-*.tar.gz
|
||||
|
||||
# 查看镜像清单
|
||||
make k8s-offline-images-list
|
||||
```
|
||||
|
||||
### 离线机器部署
|
||||
|
||||
```bash
|
||||
# 方式 1: 用 make 一键跑完
|
||||
make k8s-offline-import BUNDLE=/path/to/rcoder-offline-xxx.tar.gz \
|
||||
INSTALL_ARGS="--mode=direct --env=dev"
|
||||
|
||||
# 方式 2: 手工解压 + 跑 install.sh
|
||||
tar xzf rcoder-offline-xxx.tar.gz -C /opt/rcoder-offline
|
||||
cd /opt/rcoder-offline
|
||||
sudo bash install.sh --mode=direct --env=dev
|
||||
```
|
||||
|
||||
### 两种导入模式
|
||||
|
||||
| 模式 | 镜像落盘方式 | 适用场景 |
|
||||
|------|------------|----------|
|
||||
| `--mode=direct` | `ctr image import` 到节点 containerd | 单节点 / 小集群 / 无私有 registry |
|
||||
| `--mode=registry` | `docker load` + re-tag + push | 多节点 / 已有 Harbor/Nexus / 客户自建 registry |
|
||||
|
||||
详见 [`offline/README.md`](./offline/README.md)（客户交付文档）。
|
||||
|
||||
---
|
||||
|
||||
## 本地镜像加速（多节点 / 频繁构建场景）
|
||||
|
||||
当本地 k3s 有多个节点，或者你频繁 push 测试镜像又不想每次走公网时，启用 `register2/` 作本地 registry mirror，所有节点直接拉局域网镜像（千兆）。
|
||||
|
||||
### 架构
|
||||
|
||||
```
|
||||
k3s/containerd (所有节点)
|
||||
│
|
||||
├── (1) http://<你的机器>:5000 ← register2 (本目录, 私有推送)
|
||||
├── (2) http://<你的机器>:5002 ← registry-cache (可选 pull-through)
|
||||
├── (3) https://docker.m.daocloud.io / ... ← 公网 cn mirror 回退
|
||||
└── (4) 默认源 (docker.io / registry.k8s.io) 兜底
|
||||
```
|
||||
|
||||
containerd 按顺序尝试 endpoint，命中（200）即返回，失败（404/超时）自动回退下一个。
|
||||
|
||||
### 启动本地 registry
|
||||
|
||||
```bash
|
||||
cd k8s/register2
|
||||
docker compose up -d
|
||||
# 验证:
|
||||
curl -s http://<你的机器>:5000/v2/_catalog
|
||||
```
|
||||
|
||||
### 一键配置 k3s 节点
|
||||
|
||||
```bash
|
||||
# 在每个 k3s 节点上执行 (REGISTRY_HOST 按实际填)
|
||||
sudo REGISTRY_HOST=192.168.32.228:5000 \
|
||||
k8s/scripts/install-k3s-registry-mirrors-cn.sh
|
||||
|
||||
# 多个 registry 串联 (比如同时有 registry2 和 pull-through cache)
|
||||
sudo REGISTRY_HOST=192.168.32.228:5000,192.168.32.228:5002 \
|
||||
k8s/scripts/install-k3s-registry-mirrors-cn.sh
|
||||
```
|
||||
|
||||
脚本会：
|
||||
- 备份现有 `/etc/rancher/k3s/registries.yaml`
|
||||
- 写入包含本地 registry + 公网 cn mirror 回退的完整配置（含 `registry.k8s.io`/`ghcr.io`/`quay.io` 全套）
|
||||
- 自动加 "自指 mirror" + `insecure_skip_verify` 允许 `crictl pull <host>/foo` 走 plain HTTP
|
||||
- 自动 `systemctl restart k3s` 或 `k3s-agent`
|
||||
- 用 `crictl pull docker.io/rancher/mirrored-pause:3.6` 验证
|
||||
|
||||
### 本机 docker daemon 配置（允许 push 到 HTTP registry）
|
||||
|
||||
```bash
|
||||
sudo tee /etc/docker/daemon.json <<EOF
|
||||
{
|
||||
"insecure-registries": ["192.168.32.228:5000"]
|
||||
}
|
||||
EOF
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
|
||||
详见 [`register2/README.md`](./register2/README.md)。
|
||||
|
||||
### 与离线 bundle 的联动
|
||||
|
||||
配好本地 registry 后，在本机就能测通 bundle 的 `registry 模式`：
|
||||
|
||||
```bash
|
||||
tar xzf dist/rcoder-offline-*.tar.gz -C /tmp/offline-test
|
||||
cd /tmp/offline-test
|
||||
docker load -i images/all-images.tar
|
||||
bash rewrite-registry.sh \
|
||||
--registry 192.168.32.228:5000/rcoder \
|
||||
--thirdparty-registry 192.168.32.228:5000/thirdparty \
|
||||
--images-file images/images.txt
|
||||
# 之后 helm install 时 k3s 自动从 192.168.32.228:5000 拉
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 存储架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Kubernetes 集群 │
|
||||
│ ┌──────────────────────────────────────────────────────┐ │
|
||||
│ │ JuiceFS CSI Driver (kube-system) │ │
|
||||
│ └──────────────────────────────────────────────────────┘ │
|
||||
│ ↓ │
|
||||
│ ┌────────────────┐ ┌────────────────┐ │
|
||||
│ │ rcoder 主服务 │ │ Agent Runner │ │
|
||||
│ │ (Deployment) │ │ (动态创建 Pod) │ │
|
||||
│ └────────┬───────┘ └────────┬───────┘ │
|
||||
│ └──────────┬───────────┘ │
|
||||
│ ↓ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ JuiceFS 卷 │ ← ReadWriteMany 共享 │
|
||||
│ └───────┬──────┘ │
|
||||
└──────────────────────┼──────────────────────────────────────┘
|
||||
↓
|
||||
┌───────────────────────────────────────────────────────────────┐
|
||||
│ MinIO (S3) ← JuiceFS 数据存储（per-namespace） │
|
||||
│ PostgreSQL ← JuiceFS 元数据存储（per-namespace） │
|
||||
└───────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
每个 namespace 独立拥有 PostgreSQL 和 MinIO 实例，metadata 和 object 存储互不干扰。
|
||||
|
||||
---
|
||||
|
||||
## 环境要求
|
||||
|
||||
- Kubernetes 1.14+（本仓库在 K3s 1.34.x 上测试）
|
||||
- `kubectl` 已配置 kubeconfig
|
||||
- `helm` 3.x（用于首次安装 JuiceFS CSI Driver + Helm 部署路径）
|
||||
- Longhorn 存储（为 PostgreSQL / MinIO 提供 PV；`deploy-*.sh` 会自动安装）
|
||||
- `open-iscsi`（Longhorn 依赖；`deploy-*.sh` 会自动安装）
|
||||
- `docker`（仅在有网构建机 `make k8s-offline-bundle` 或本地 push 到 register2 时需要）
|
||||
|
||||
国内节点建议先配置 k3s 镜像加速（纯公网 cn mirror 版）：
|
||||
```bash
|
||||
sudo k8s/scripts/install-k3s-registry-mirrors-cn.sh
|
||||
```
|
||||
|
||||
有本地 registry 时用 `REGISTRY_HOST=...`（见 "本地镜像加速" 一节）。
|
||||
|
||||
---
|
||||
|
||||
## 故障排查
|
||||
|
||||
```bash
|
||||
# Pod 状态
|
||||
kubectl get pods -n nuwax-rcoder-dev # 或 -prod / -helm-test / <your-ns>
|
||||
|
||||
# 事件
|
||||
kubectl get events -n nuwax-rcoder-dev --sort-by='.lastTimestamp'
|
||||
|
||||
# RCoder 日志
|
||||
kubectl logs -n nuwax-rcoder-dev -l app=rcoder --tail=200 -f
|
||||
|
||||
# PVC 绑定情况
|
||||
kubectl get pvc -n nuwax-rcoder-dev
|
||||
kubectl describe pvc rcoder-workspace -n nuwax-rcoder-dev
|
||||
|
||||
# StorageClass
|
||||
kubectl get sc
|
||||
|
||||
# 镜像加速是否生效 (与 kubelet 同链路验证, 不要用 k3s ctr images pull)
|
||||
sudo crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock \
|
||||
pull docker.io/rancher/mirrored-pause:3.6
|
||||
|
||||
# 本地 registry 命中情况
|
||||
curl -s http://192.168.32.228:5000/v2/_catalog
|
||||
curl -s http://192.168.32.228:5002/v2/_catalog # pull-through 缓存
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 历史说明
|
||||
|
||||
此前版本曾提供 `k8s/helm/rcoder/` Helm chart 作为另一条部署路径；**2025 年后改为 Kustomize + Helm 双栈并存**：
|
||||
- Kustomize 用于开发和已有部署，保持轻量；
|
||||
- Helm 用于参数化对外交付和离线 bundle；
|
||||
- 两者共享同一份 K8s 资源定义（修改时需同步）。
|
||||
331
qiming-rcoder/k8s/deploy-dev.sh
Normal file
331
qiming-rcoder/k8s/deploy-dev.sh
Normal file
@@ -0,0 +1,331 @@
|
||||
#!/bin/bash
|
||||
# ============================================================
|
||||
# RCoder K8s 开发环境部署脚本 (Kustomize)
|
||||
# 用于日常开发测试验证
|
||||
# ============================================================
|
||||
|
||||
set -e
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# 路径解析：支持从任意 cwd 调用脚本
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
NAMESPACE="${NAMESPACE:-nuwax-rcoder-dev}"
|
||||
KUSTOMIZE_DIR="${KUSTOMIZE_DIR:-${SCRIPT_DIR}/manifests/overlays/dev}"
|
||||
JUICEFS_CE_MOUNT_IMAGE="${JUICEFS_CE_MOUNT_IMAGE:-juicedata/mount:ce-v1.3.1}"
|
||||
DNS_CHECK_IMAGE="${DNS_CHECK_IMAGE:-busybox:1.36}"
|
||||
|
||||
echo -e "${GREEN}=========================================="
|
||||
echo " RCoder K8s 开发环境部署 (Kustomize)"
|
||||
echo "==========================================${NC}"
|
||||
|
||||
ensure_juicefs_ce_image() {
|
||||
echo -e "${YELLOW}配置 JuiceFS CSI 使用 CE Mount 镜像: ${JUICEFS_CE_MOUNT_IMAGE}${NC}"
|
||||
|
||||
local node_ds=""
|
||||
if kubectl get daemonset/juicefs-csi-node -n kube-system &> /dev/null; then
|
||||
node_ds="juicefs-csi-node"
|
||||
elif kubectl get daemonset/juicefs-csi-driver-node -n kube-system &> /dev/null; then
|
||||
node_ds="juicefs-csi-driver-node"
|
||||
fi
|
||||
|
||||
# 兼容不同版本 CSI 驱动：同时设置旧变量与 CE 专用变量
|
||||
if [ -n "$node_ds" ]; then
|
||||
kubectl -n kube-system set env "daemonset/${node_ds}" -c juicefs-plugin \
|
||||
JUICEFS_CE_MOUNT_IMAGE="${JUICEFS_CE_MOUNT_IMAGE}" \
|
||||
JUICEFS_MOUNT_IMAGE="${JUICEFS_CE_MOUNT_IMAGE}" >/dev/null
|
||||
fi
|
||||
|
||||
kubectl -n kube-system set env statefulset/juicefs-csi-controller -c juicefs-plugin \
|
||||
JUICEFS_CE_MOUNT_IMAGE="${JUICEFS_CE_MOUNT_IMAGE}" \
|
||||
JUICEFS_MOUNT_IMAGE="${JUICEFS_CE_MOUNT_IMAGE}" >/dev/null
|
||||
|
||||
# 同步更新驱动 ConfigMap（部分版本从这里读取 mountImage）
|
||||
kubectl -n kube-system patch configmap juicefs-csi-driver-config --type merge \
|
||||
-p "{\"data\":{\"config.yaml\":\"enableNodeSelector: false\\nmountImage: ${JUICEFS_CE_MOUNT_IMAGE}\\nmountPodPatch:\\n\"}}" >/dev/null || true
|
||||
}
|
||||
|
||||
check_cluster_dns() {
|
||||
echo -e "${GREEN}检查集群 DNS 基线...${NC}"
|
||||
local check_name="rcoder-dns-check-$(date +%s)"
|
||||
if kubectl run "${check_name}" --restart=Never --rm -i -n "${NAMESPACE}" \
|
||||
--image="${DNS_CHECK_IMAGE}" --command -- \
|
||||
sh -lc "nslookup kubernetes.default.svc.cluster.local >/dev/null && nslookup postgresql.${NAMESPACE}.svc.cluster.local >/dev/null && nslookup minio-service.${NAMESPACE}.svc.cluster.local >/dev/null" >/dev/null 2>&1; then
|
||||
echo -e "${GREEN}✅ 集群 DNS 解析正常${NC}"
|
||||
else
|
||||
echo -e "${RED}❌ 集群 DNS 解析异常：无法解析 Kubernetes/业务 Service 域名${NC}"
|
||||
echo -e "${YELLOW}建议先修复 CoreDNS 后再继续（例如重启 coredns / 检查 coredns addon）${NC}"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ============================================================
|
||||
# 步骤 1: 检查 K8s 集群
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}[1/6] 检查 K8s 集群...${NC}"
|
||||
|
||||
if ! command -v kubectl &> /dev/null; then
|
||||
echo -e "${RED}Error: kubectl not found${NC}"
|
||||
echo "请安装 kubectl: https://kubernetes.io/docs/tasks/tools/"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! kubectl cluster-info &> /dev/null; then
|
||||
echo -e "${RED}Error: 无法连接到 K8s 集群${NC}"
|
||||
echo ""
|
||||
echo -e "${YELLOW}请先部署 K3s 集群 (中国镜像):${NC}"
|
||||
echo ""
|
||||
echo " curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -"
|
||||
echo ""
|
||||
echo " # 安装完成后, 配置 kubectl:"
|
||||
echo " mkdir -p ~/.kube"
|
||||
echo " sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config"
|
||||
echo " chmod 600 ~/.kube/config"
|
||||
echo ""
|
||||
echo " # 重新运行此脚本"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}✅ K8s 集群连接正常${NC}"
|
||||
kubectl get nodes
|
||||
|
||||
# ============================================================
|
||||
# 步骤 1.5: 检查/安装 open-iscsi (Longhorn 依赖)
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}[1.5/6] 检查 open-iscsi 依赖...${NC}"
|
||||
|
||||
install_open_iscsi() {
|
||||
echo -e "${YELLOW}正在检测操作系统...${NC}"
|
||||
|
||||
# 检测操作系统类型
|
||||
if [ -f /etc/os-release ]; then
|
||||
. /etc/os-release
|
||||
OS_ID="$ID"
|
||||
OS_VERSION="$VERSION_ID"
|
||||
else
|
||||
OS_ID="unknown"
|
||||
fi
|
||||
|
||||
echo -e "${YELLOW}检测到操作系统: $OS_ID${NC}"
|
||||
|
||||
case "$OS_ID" in
|
||||
ubuntu|debian)
|
||||
echo -e "${YELLOW}安装 open-iscsi...${NC}"
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq open-iscsi > /dev/null 2>&1
|
||||
;;
|
||||
centos|rhel|rocky|almalinux)
|
||||
echo -e "${YELLOW}安装 iscsi-initiator-utils...${NC}"
|
||||
yum install -y -q iscsi-initiator-utils > /dev/null 2>&1
|
||||
;;
|
||||
sles|opensuse-leap|opensuse-tumbleweed)
|
||||
echo -e "${YELLOW}安装 open-iscsi...${NC}"
|
||||
zypper install -y -q open-iscsi > /dev/null 2>&1
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}不支持的操作系统: $OS_ID${NC}"
|
||||
echo -e "${YELLOW}请手动安装 open-iscsi 或 iscsi-initiator-utils${NC}"
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
|
||||
echo -e "${YELLOW}启动 iscsid 服务...${NC}"
|
||||
systemctl enable --now iscsid 2>/dev/null || true
|
||||
systemctl enable --now iscsid.socket 2>/dev/null || true
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# 检查 iscsiadm 是否存在
|
||||
if command -v iscsiadm &> /dev/null; then
|
||||
echo -e "${GREEN}✅ open-iscsi 已安装: $(iscsiadm --version 2>&1 | head -1)${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}⚠️ open-iscsi 未安装${NC}"
|
||||
echo -e "${YELLOW}Longhorn 需要 open-iscsi 来提供 iSCSI 存储${NC}"
|
||||
|
||||
# 尝试自动安装
|
||||
if [ "$EUID" -eq 0 ]; then
|
||||
install_open_iscsi
|
||||
if [ $? -eq 0 ]; then
|
||||
echo -e "${GREEN}✅ open-iscsi 安装成功${NC}"
|
||||
fi
|
||||
else
|
||||
echo -e "${YELLOW}请使用 sudo 运行此脚本，或手动安装:${NC}"
|
||||
echo -e " ${CYAN}Ubuntu/Debian: sudo apt install open-iscsi${NC}"
|
||||
echo -e " ${CYAN}CentOS/RHEL: sudo yum install iscsi-initiator-utils${NC}"
|
||||
echo -e " ${CYAN}SUSE: sudo zypper install open-iscsi${NC}"
|
||||
echo ""
|
||||
echo -e "${YELLOW}安装完成后，重新运行此脚本${NC}"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# 验证 iscsiadm 可用
|
||||
if ! command -v iscsiadm &> /dev/null; then
|
||||
echo -e "${RED}❌ open-iscsi 安装失败${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}✅ open-iscsi 检查完成${NC}"
|
||||
|
||||
# ============================================================
|
||||
# 步骤 2: 部署 Longhorn 存储
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}[2/6] 检查/部署 Longhorn 存储...${NC}"
|
||||
|
||||
if kubectl get sc longhorn &> /dev/null; then
|
||||
echo -e "${GREEN}✅ Longhorn 已安装${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}Longhorn 未安装，正在部署...${NC}"
|
||||
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
|
||||
echo -e "${GREEN}⏳ 等待 Longhorn 就绪...${NC}"
|
||||
|
||||
# 等待 Longhorn Manager 就绪
|
||||
kubectl wait --for=condition=ready pod -l app=longhorn-manager \
|
||||
-n longhorn-system --timeout=300s 2>/dev/null || true
|
||||
|
||||
# 等待 Longhorn UI 就绪
|
||||
kubectl wait --for=condition=ready pod -l app=longhorn-ui \
|
||||
-n longhorn-system --timeout=300s 2>/dev/null || true
|
||||
|
||||
echo -e "${GREEN}✅ Longhorn 部署完成${NC}"
|
||||
fi
|
||||
|
||||
kubectl get sc | grep longhorn || echo -e "${YELLOW}Warning: Longhorn StorageClass 未就绪${NC}"
|
||||
|
||||
# ============================================================
|
||||
# 步骤 3: 部署 JuiceFS CSI
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}[3/6] 检查/部署 JuiceFS CSI Driver...${NC}"
|
||||
|
||||
if kubectl get ds juicefs-csi-driver-node -n kube-system &> /dev/null || kubectl get ds juicefs-csi-node -n kube-system &> /dev/null; then
|
||||
echo -e "${GREEN}✅ JuiceFS CSI Driver 已安装${NC}"
|
||||
ensure_juicefs_ce_image
|
||||
else
|
||||
echo -e "${YELLOW}JuiceFS CSI Driver 未安装，正在部署...${NC}"
|
||||
|
||||
if ! command -v helm &> /dev/null; then
|
||||
echo -e "${YELLOW}Helm 未安装，跳过 JuiceFS CSI 部署${NC}"
|
||||
echo "请手动安装: helm repo add juicedata https://juicedata.github.io/charts"
|
||||
else
|
||||
helm repo add juicedata https://juicedata.github.io/charts 2>/dev/null || true
|
||||
helm repo update
|
||||
helm install juicefs-csi-driver juicedata/juicefs-csi-driver \
|
||||
--namespace kube-system \
|
||||
--set webhook.enabled=false
|
||||
|
||||
echo -e "${GREEN}⏳ 等待 JuiceFS CSI Driver 就绪...${NC}"
|
||||
kubectl wait --for=condition=ready pod -l app=juicefs-csi-driver-node \
|
||||
-n kube-system --timeout=120s 2>/dev/null || true
|
||||
|
||||
ensure_juicefs_ce_image
|
||||
echo -e "${GREEN}✅ JuiceFS CSI Driver 部署完成${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# 步骤 4: 使用 Kustomize 部署 RCoder
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}[4/6] 部署 RCoder 到 namespace: $NAMESPACE${NC}"
|
||||
|
||||
# 检查 kubectl kustomize 插件
|
||||
if ! kubectl kustomize --help &> /dev/null; then
|
||||
echo -e "${RED}Error: kubectl kustomize 插件未找到 (需要 kubectl 1.14+)${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 部署 (分两阶段避免 JuiceFS CSI mount 期间 postgresql 未就绪导致 60s retry 噪音)
|
||||
echo -e "${YELLOW}阶段 1/2: 先部署存储层 + RBAC + Secret...${NC}"
|
||||
kubectl apply -k "$KUSTOMIZE_DIR"
|
||||
|
||||
echo -e "${YELLOW}等待 PostgreSQL 就绪 (JuiceFS CSI 依赖)...${NC}"
|
||||
kubectl wait --for=condition=ready pod -l app=postgresql -n "$NAMESPACE" --timeout=180s 2>&1 || {
|
||||
echo -e "${RED}❌ PostgreSQL 未在 180s 内就绪${NC}"
|
||||
kubectl get pods -n "$NAMESPACE"
|
||||
exit 1
|
||||
}
|
||||
|
||||
echo -e "${YELLOW}等待 MinIO 就绪...${NC}"
|
||||
kubectl wait --for=condition=ready pod -l app=minio -n "$NAMESPACE" --timeout=180s 2>&1 || {
|
||||
echo -e "${RED}❌ MinIO 未在 180s 内就绪${NC}"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# 如果 rcoder pod 已经创建但还卡在 mount retry，删掉让它重建 (clean mount)
|
||||
RCODER_POD_READY=$(kubectl get pod -n "$NAMESPACE" -l app=rcoder -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}' 2>/dev/null | grep -c True)
|
||||
if [ "${RCODER_POD_READY:-0}" -eq 0 ]; then
|
||||
echo -e "${YELLOW}阶段 2/2: 重建 rcoder pod 以触发干净 mount...${NC}"
|
||||
kubectl delete pod -n "$NAMESPACE" -l app=rcoder --ignore-not-found --wait=false
|
||||
fi
|
||||
|
||||
echo -e "${YELLOW}等待 rcoder Deployment 就绪...${NC}"
|
||||
kubectl rollout status deploy/rcoder -n "$NAMESPACE" --timeout=180s
|
||||
|
||||
check_cluster_dns
|
||||
|
||||
echo -e "${GREEN}✅ RCoder 部署完成${NC}"
|
||||
|
||||
# ============================================================
|
||||
# 步骤 5: 验证部署
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}[5/6] 验证部署状态...${NC}"
|
||||
|
||||
echo ""
|
||||
echo "--- Pods ---"
|
||||
kubectl get pods -n "$NAMESPACE"
|
||||
|
||||
echo ""
|
||||
echo "--- Services ---"
|
||||
kubectl get svc -n "$NAMESPACE"
|
||||
|
||||
echo ""
|
||||
echo "--- StorageClass ---"
|
||||
kubectl get sc | grep -E "longhorn|juicefs"
|
||||
|
||||
echo ""
|
||||
echo "--- PVC ---"
|
||||
kubectl get pvc -n "$NAMESPACE"
|
||||
|
||||
# 获取访问信息
|
||||
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}' 2>/dev/null || echo "localhost")
|
||||
SVC_TYPE=$(kubectl get svc rcoder -n "$NAMESPACE" -o jsonpath='{.spec.type}' 2>/dev/null)
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}=========================================="
|
||||
echo " 部署完成!"
|
||||
echo "==========================================${NC}"
|
||||
echo ""
|
||||
echo -e "访问方式:"
|
||||
|
||||
if [ "$SVC_TYPE" = "NodePort" ]; then
|
||||
NODE_PORT=$(kubectl get svc rcoder -n "$NAMESPACE" -o jsonpath='{.spec.ports[0].nodePort}')
|
||||
echo -e " ${GREEN}http://${NODE_IP}:${NODE_PORT}/health${NC}"
|
||||
echo -e " ${GREEN}http://${NODE_IP}:${NODE_PORT}/chat${NC}"
|
||||
elif [ "$SVC_TYPE" = "LoadBalancer" ]; then
|
||||
EXT_IP=$(kubectl get svc rcoder -n "$NAMESPACE" -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || echo "<pending>")
|
||||
echo -e " ${GREEN}http://${EXT_IP}:8087/health${NC}"
|
||||
else
|
||||
echo -e " ${GREEN}kubectl port-forward svc/rcoder 8087:8087 -n $NAMESPACE${NC}"
|
||||
echo -e " 然后访问: ${GREEN}http://localhost:8087/health${NC}"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo -e "Kustomize 管理命令:"
|
||||
echo -e " ${YELLOW}kubectl apply -k $KUSTOMIZE_DIR${NC} # 部署/更新"
|
||||
echo -e " ${YELLOW}kubectl delete -k $KUSTOMIZE_DIR${NC} # 删除"
|
||||
echo -e " ${YELLOW}kubectl get all -n $NAMESPACE${NC} # 查看状态"
|
||||
echo ""
|
||||
echo -e "Longhorn 控制台:"
|
||||
echo -e " ${YELLOW}kubectl port-forward svc/longhorn-frontend 8080:80 -n longhorn-system${NC}"
|
||||
echo -e " 访问: ${GREEN}http://localhost:8080${NC}"
|
||||
echo ""
|
||||
353
qiming-rcoder/k8s/deploy-prod.sh
Normal file
353
qiming-rcoder/k8s/deploy-prod.sh
Normal file
@@ -0,0 +1,353 @@
|
||||
#!/bin/bash
|
||||
# ============================================================
|
||||
# RCoder K8s 生产环境部署脚本 (Kustomize)
|
||||
# 部署目标: namespace = nuwax-rcoder-prod, overlay = manifests/overlays/prod
|
||||
# 支持 K3s / 标准 K8s 集群
|
||||
# ============================================================
|
||||
|
||||
set -e
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# 路径解析：支持从任意 cwd 调用脚本
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
NAMESPACE="${NAMESPACE:-nuwax-rcoder-prod}"
|
||||
KUSTOMIZE_DIR="${KUSTOMIZE_DIR:-${SCRIPT_DIR}/manifests/overlays/prod}"
|
||||
JUICEFS_CE_MOUNT_IMAGE="${JUICEFS_CE_MOUNT_IMAGE:-juicedata/mount:ce-v1.3.1}"
|
||||
DNS_CHECK_IMAGE="${DNS_CHECK_IMAGE:-busybox:1.36}"
|
||||
|
||||
echo -e "${GREEN}=========================================="
|
||||
echo " RCoder K8s 生产环境部署 (Kustomize)"
|
||||
echo "==========================================${NC}"
|
||||
echo -e " namespace: ${NAMESPACE}"
|
||||
echo -e " overlay: ${KUSTOMIZE_DIR}"
|
||||
echo ""
|
||||
|
||||
# ============================================================
|
||||
# 生产环境安全闸门: 拒绝使用占位符密码部署
|
||||
# 允许用 FORCE_PROD_DEPLOY=1 绕过（用于 CI 在外部注入 Secret 的场景）
|
||||
# ============================================================
|
||||
check_prod_credentials() {
|
||||
local overlay_dir="$1"
|
||||
local files=("${overlay_dir}/credentials.yaml" "${overlay_dir}/juicefs-secret.yaml")
|
||||
local found=0
|
||||
for f in "${files[@]}"; do
|
||||
if [ -f "$f" ] && grep -q "CHANGE-ME-BEFORE-DEPLOY" "$f"; then
|
||||
echo -e "${RED}❌ 检测到占位符密码未替换: $f${NC}" >&2
|
||||
found=1
|
||||
fi
|
||||
done
|
||||
if [ "$found" -eq 1 ]; then
|
||||
echo "" >&2
|
||||
echo -e "${YELLOW}生产环境禁止使用 CHANGE-ME-BEFORE-DEPLOY 占位符部署。${NC}" >&2
|
||||
echo -e "${YELLOW}处理方式（二选一）:${NC}" >&2
|
||||
echo -e " 1) 把上述文件里的占位符替换为真实强密码后再部署" >&2
|
||||
echo -e " 2) 如果你用 SealedSecret / ExternalSecret 从外部注入密钥，" >&2
|
||||
echo -e " 请设置 FORCE_PROD_DEPLOY=1 并确保上线后真正的 Secret 已就绪" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
if [ "${FORCE_PROD_DEPLOY:-0}" != "1" ]; then
|
||||
check_prod_credentials "${KUSTOMIZE_DIR}"
|
||||
fi
|
||||
|
||||
ensure_juicefs_ce_image() {
|
||||
echo -e "${YELLOW}配置 JuiceFS CSI 使用 CE Mount 镜像: ${JUICEFS_CE_MOUNT_IMAGE}${NC}"
|
||||
|
||||
local node_ds=""
|
||||
if kubectl get daemonset/juicefs-csi-node -n kube-system &> /dev/null; then
|
||||
node_ds="juicefs-csi-node"
|
||||
elif kubectl get daemonset/juicefs-csi-driver-node -n kube-system &> /dev/null; then
|
||||
node_ds="juicefs-csi-driver-node"
|
||||
fi
|
||||
|
||||
if [ -n "$node_ds" ]; then
|
||||
kubectl -n kube-system set env "daemonset/${node_ds}" -c juicefs-plugin \
|
||||
JUICEFS_CE_MOUNT_IMAGE="${JUICEFS_CE_MOUNT_IMAGE}" \
|
||||
JUICEFS_MOUNT_IMAGE="${JUICEFS_CE_MOUNT_IMAGE}" >/dev/null
|
||||
fi
|
||||
|
||||
kubectl -n kube-system set env statefulset/juicefs-csi-controller -c juicefs-plugin \
|
||||
JUICEFS_CE_MOUNT_IMAGE="${JUICEFS_CE_MOUNT_IMAGE}" \
|
||||
JUICEFS_MOUNT_IMAGE="${JUICEFS_CE_MOUNT_IMAGE}" >/dev/null
|
||||
|
||||
kubectl -n kube-system patch configmap juicefs-csi-driver-config --type merge \
|
||||
-p "{\"data\":{\"config.yaml\":\"enableNodeSelector: false\\nmountImage: ${JUICEFS_CE_MOUNT_IMAGE}\\nmountPodPatch:\\n\"}}" >/dev/null || true
|
||||
}
|
||||
|
||||
check_cluster_dns() {
|
||||
echo -e "${GREEN}检查集群 DNS 基线...${NC}"
|
||||
local check_name="rcoder-dns-check-$(date +%s)"
|
||||
if kubectl run "${check_name}" --restart=Never --rm -i -n "${NAMESPACE}" \
|
||||
--image="${DNS_CHECK_IMAGE}" --command -- \
|
||||
sh -lc "nslookup kubernetes.default.svc.cluster.local >/dev/null && nslookup postgresql.${NAMESPACE}.svc.cluster.local >/dev/null && nslookup minio-service.${NAMESPACE}.svc.cluster.local >/dev/null" >/dev/null 2>&1; then
|
||||
echo -e "${GREEN}✅ 集群 DNS 解析正常${NC}"
|
||||
else
|
||||
echo -e "${RED}❌ 集群 DNS 解析异常：无法解析 Kubernetes/业务 Service 域名${NC}"
|
||||
echo -e "${YELLOW}建议先修复 CoreDNS 后再继续（例如重启 coredns / 检查 coredns addon）${NC}"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ============================================================
|
||||
# 步骤 1: 检查 K8s 集群
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}[1/6] 检查 K8s 集群...${NC}"
|
||||
|
||||
if ! command -v kubectl &> /dev/null; then
|
||||
echo -e "${RED}Error: kubectl not found${NC}"
|
||||
echo "请安装 kubectl: https://kubernetes.io/docs/tasks/tools/"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! kubectl cluster-info &> /dev/null; then
|
||||
echo -e "${RED}Error: 无法连接到 K8s 集群${NC}"
|
||||
echo ""
|
||||
echo -e "${YELLOW}请先部署 K3s 集群 (中国镜像):${NC}"
|
||||
echo ""
|
||||
echo " curl -sfL https://rancher-mirror.rancher.cn/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -"
|
||||
echo ""
|
||||
echo " # 安装完成后, 配置 kubectl:"
|
||||
echo " mkdir -p ~/.kube"
|
||||
echo " sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config"
|
||||
echo " chmod 600 ~/.kube/config"
|
||||
echo ""
|
||||
echo " # 重新运行此脚本"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}✅ K8s 集群连接正常${NC}"
|
||||
kubectl get nodes
|
||||
|
||||
# ============================================================
|
||||
# 步骤 1.5: 检查/安装 open-iscsi (Longhorn 依赖)
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}[1.5/6] 检查 open-iscsi 依赖...${NC}"
|
||||
|
||||
install_open_iscsi() {
|
||||
echo -e "${YELLOW}正在检测操作系统...${NC}"
|
||||
|
||||
if [ -f /etc/os-release ]; then
|
||||
. /etc/os-release
|
||||
OS_ID="$ID"
|
||||
OS_VERSION="$VERSION_ID"
|
||||
else
|
||||
OS_ID="unknown"
|
||||
fi
|
||||
|
||||
echo -e "${YELLOW}检测到操作系统: $OS_ID${NC}"
|
||||
|
||||
case "$OS_ID" in
|
||||
ubuntu|debian)
|
||||
echo -e "${YELLOW}安装 open-iscsi...${NC}"
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq open-iscsi > /dev/null 2>&1
|
||||
;;
|
||||
centos|rhel|rocky|almalinux)
|
||||
echo -e "${YELLOW}安装 iscsi-initiator-utils...${NC}"
|
||||
yum install -y -q iscsi-initiator-utils > /dev/null 2>&1
|
||||
;;
|
||||
sles|opensuse-leap|opensuse-tumbleweed)
|
||||
echo -e "${YELLOW}安装 open-iscsi...${NC}"
|
||||
zypper install -y -q open-iscsi > /dev/null 2>&1
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}不支持的操作系统: $OS_ID${NC}"
|
||||
echo -e "${YELLOW}请手动安装 open-iscsi 或 iscsi-initiator-utils${NC}"
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
|
||||
echo -e "${YELLOW}启动 iscsid 服务...${NC}"
|
||||
systemctl enable --now iscsid 2>/dev/null || true
|
||||
systemctl enable --now iscsid.socket 2>/dev/null || true
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
if command -v iscsiadm &> /dev/null; then
|
||||
echo -e "${GREEN}✅ open-iscsi 已安装: $(iscsiadm --version 2>&1 | head -1)${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}⚠️ open-iscsi 未安装${NC}"
|
||||
echo -e "${YELLOW}Longhorn 需要 open-iscsi 来提供 iSCSI 存储${NC}"
|
||||
|
||||
if [ "$EUID" -eq 0 ]; then
|
||||
install_open_iscsi
|
||||
if [ $? -eq 0 ]; then
|
||||
echo -e "${GREEN}✅ open-iscsi 安装成功${NC}"
|
||||
fi
|
||||
else
|
||||
echo -e "${YELLOW}请使用 sudo 运行此脚本，或手动安装:${NC}"
|
||||
echo -e " Ubuntu/Debian: sudo apt install open-iscsi"
|
||||
echo -e " CentOS/RHEL: sudo yum install iscsi-initiator-utils"
|
||||
echo -e " SUSE: sudo zypper install open-iscsi"
|
||||
echo ""
|
||||
echo -e "${YELLOW}安装完成后，重新运行此脚本${NC}"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
if ! command -v iscsiadm &> /dev/null; then
|
||||
echo -e "${RED}❌ open-iscsi 安装失败${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}✅ open-iscsi 检查完成${NC}"
|
||||
|
||||
# ============================================================
|
||||
# 步骤 2: 部署 Longhorn 存储
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}[2/6] 检查/部署 Longhorn 存储...${NC}"
|
||||
|
||||
if kubectl get sc longhorn &> /dev/null; then
|
||||
echo -e "${GREEN}✅ Longhorn 已安装${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}Longhorn 未安装，正在部署...${NC}"
|
||||
kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
|
||||
echo -e "${GREEN}⏳ 等待 Longhorn 就绪...${NC}"
|
||||
|
||||
kubectl wait --for=condition=ready pod -l app=longhorn-manager \
|
||||
-n longhorn-system --timeout=300s 2>/dev/null || true
|
||||
kubectl wait --for=condition=ready pod -l app=longhorn-ui \
|
||||
-n longhorn-system --timeout=300s 2>/dev/null || true
|
||||
|
||||
echo -e "${GREEN}✅ Longhorn 部署完成${NC}"
|
||||
fi
|
||||
|
||||
kubectl get sc | grep longhorn || echo -e "${YELLOW}Warning: Longhorn StorageClass 未就绪${NC}"
|
||||
|
||||
# ============================================================
|
||||
# 步骤 3: 部署 JuiceFS CSI
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}[3/6] 检查/部署 JuiceFS CSI Driver...${NC}"
|
||||
|
||||
if kubectl get ds juicefs-csi-driver-node -n kube-system &> /dev/null || kubectl get ds juicefs-csi-node -n kube-system &> /dev/null; then
|
||||
echo -e "${GREEN}✅ JuiceFS CSI Driver 已安装${NC}"
|
||||
ensure_juicefs_ce_image
|
||||
else
|
||||
echo -e "${YELLOW}JuiceFS CSI Driver 未安装，正在部署...${NC}"
|
||||
|
||||
if ! command -v helm &> /dev/null; then
|
||||
echo -e "${RED}Error: Helm not found（安装 JuiceFS CSI Driver 需要 Helm）${NC}"
|
||||
echo "请安装 Helm: https://helm.sh/docs/intro/install/"
|
||||
exit 1
|
||||
else
|
||||
helm repo add juicedata https://juicedata.github.io/charts 2>/dev/null || true
|
||||
helm repo update
|
||||
helm install juicefs-csi-driver juicedata/juicefs-csi-driver \
|
||||
--namespace kube-system \
|
||||
--set webhook.enabled=false
|
||||
|
||||
echo -e "${GREEN}⏳ 等待 JuiceFS CSI Driver 就绪...${NC}"
|
||||
kubectl wait --for=condition=ready pod -l app=juicefs-csi-driver-node \
|
||||
-n kube-system --timeout=120s 2>/dev/null || true
|
||||
|
||||
ensure_juicefs_ce_image
|
||||
echo -e "${GREEN}✅ JuiceFS CSI Driver 部署完成${NC}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ============================================================
|
||||
# 步骤 4: 使用 Kustomize 部署 RCoder (prod overlay)
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}[4/6] 部署 RCoder 到 namespace: $NAMESPACE${NC}"
|
||||
|
||||
if ! kubectl kustomize --help &> /dev/null; then
|
||||
echo -e "${RED}Error: kubectl kustomize 插件未找到 (需要 kubectl 1.14+)${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 分两阶段部署：等 PostgreSQL/MinIO 就绪再让 rcoder 起，避免 JuiceFS CSI mount 的 60s retry 噪音
|
||||
echo -e "${YELLOW}阶段 1/2: apply manifests...${NC}"
|
||||
kubectl apply -k "$KUSTOMIZE_DIR"
|
||||
|
||||
echo -e "${YELLOW}等待 PostgreSQL 就绪 (JuiceFS CSI 依赖)...${NC}"
|
||||
kubectl wait --for=condition=ready pod -l app=postgresql -n "$NAMESPACE" --timeout=180s 2>&1 || {
|
||||
echo -e "${RED}❌ PostgreSQL 未在 180s 内就绪${NC}"
|
||||
kubectl get pods -n "$NAMESPACE"
|
||||
exit 1
|
||||
}
|
||||
|
||||
echo -e "${YELLOW}等待 MinIO 就绪...${NC}"
|
||||
kubectl wait --for=condition=ready pod -l app=minio -n "$NAMESPACE" --timeout=180s 2>&1 || {
|
||||
echo -e "${RED}❌ MinIO 未在 180s 内就绪${NC}"
|
||||
exit 1
|
||||
}
|
||||
|
||||
RCODER_POD_READY=$(kubectl get pod -n "$NAMESPACE" -l app=rcoder -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}' 2>/dev/null | grep -c True)
|
||||
if [ "${RCODER_POD_READY:-0}" -eq 0 ]; then
|
||||
echo -e "${YELLOW}阶段 2/2: 重建 rcoder pod 以触发干净 mount...${NC}"
|
||||
kubectl delete pod -n "$NAMESPACE" -l app=rcoder --ignore-not-found --wait=false
|
||||
fi
|
||||
|
||||
echo -e "${YELLOW}等待 rcoder Deployment 就绪...${NC}"
|
||||
kubectl rollout status deploy/rcoder -n "$NAMESPACE" --timeout=180s
|
||||
|
||||
check_cluster_dns
|
||||
|
||||
echo -e "${GREEN}✅ RCoder 部署完成${NC}"
|
||||
|
||||
# ============================================================
|
||||
# 步骤 5: 验证部署
|
||||
# ============================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}[5/6] 验证部署状态...${NC}"
|
||||
|
||||
echo ""
|
||||
echo "--- Pods ---"
|
||||
kubectl get pods -n "$NAMESPACE"
|
||||
|
||||
echo ""
|
||||
echo "--- Services ---"
|
||||
kubectl get svc -n "$NAMESPACE"
|
||||
|
||||
echo ""
|
||||
echo "--- StorageClass ---"
|
||||
kubectl get sc | grep -E "longhorn|juicefs"
|
||||
|
||||
echo ""
|
||||
echo "--- PVC ---"
|
||||
kubectl get pvc -n "$NAMESPACE"
|
||||
|
||||
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}' 2>/dev/null || echo "localhost")
|
||||
SVC_TYPE=$(kubectl get svc rcoder -n "$NAMESPACE" -o jsonpath='{.spec.type}' 2>/dev/null)
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}=========================================="
|
||||
echo " 部署完成!"
|
||||
echo "==========================================${NC}"
|
||||
echo ""
|
||||
echo -e "访问方式:"
|
||||
|
||||
if [ "$SVC_TYPE" = "NodePort" ]; then
|
||||
NODE_PORT=$(kubectl get svc rcoder -n "$NAMESPACE" -o jsonpath='{.spec.ports[0].nodePort}')
|
||||
echo -e " ${GREEN}http://${NODE_IP}:${NODE_PORT}/health${NC}"
|
||||
echo -e " ${GREEN}http://${NODE_IP}:${NODE_PORT}/chat${NC}"
|
||||
elif [ "$SVC_TYPE" = "LoadBalancer" ]; then
|
||||
EXT_IP=$(kubectl get svc rcoder -n "$NAMESPACE" -o jsonpath='{.status.loadBalancer.ingress[0].ip}' 2>/dev/null || echo "<pending>")
|
||||
echo -e " ${GREEN}http://${EXT_IP}:8087/health${NC}"
|
||||
else
|
||||
echo -e " ${GREEN}kubectl port-forward svc/rcoder 8087:8087 -n $NAMESPACE${NC}"
|
||||
echo -e " 然后访问: ${GREEN}http://localhost:8087/health${NC}"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo -e "Kustomize 管理命令:"
|
||||
echo -e " ${YELLOW}kubectl apply -k $KUSTOMIZE_DIR${NC} # 部署/更新"
|
||||
echo -e " ${YELLOW}kubectl delete -k $KUSTOMIZE_DIR${NC} # 删除"
|
||||
echo -e " ${YELLOW}kubectl get all -n $NAMESPACE${NC} # 查看状态"
|
||||
echo ""
|
||||
echo -e "Longhorn 控制台:"
|
||||
echo -e " ${YELLOW}kubectl port-forward svc/longhorn-frontend 8080:80 -n longhorn-system${NC}"
|
||||
echo -e " 访问: ${GREEN}http://localhost:8080${NC}"
|
||||
echo ""
|
||||
617
qiming-rcoder/k8s/docs/ARCHITECTURE.md
Normal file
617
qiming-rcoder/k8s/docs/ARCHITECTURE.md
Normal file
@@ -0,0 +1,617 @@
|
||||
# RCoder K8s 部署方案 —— 架构详解
|
||||
|
||||
> 本文档详细说明 `rcoder` 项目在 Kubernetes 上的完整部署架构。
|
||||
> 代码仓库: `/home/swufe/gitworkspace/rcoder/k8s/`
|
||||
|
||||
---
|
||||
|
||||
## 1. 整体架构
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────────────────┐
|
||||
│ Kubernetes 集群 │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ Ingress Controller (可选) │ │
|
||||
│ │ className 可配置 (nginx/traefik/alb) │ │
|
||||
│ └──────────────────────────────────────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ rcoder Service (NodePort 30080) │ │
|
||||
│ │ Deployment / Pod (rcoder 主服务) │ │
|
||||
│ │ │ │
|
||||
│ │ • REST API :8087 (health / chat / agent 管理) │ │
|
||||
│ │ • Pingora :8088 (内部反向代理,给动态 agent-runner Pod 用) │ │
|
||||
│ │ │ │
|
||||
│ │ 动态创建 Agent Runner Pod ←── K8s API (不挂 docker.sock) │ │
|
||||
│ └──────────────────────────────┬───────────────────────────────────────┘ │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌──────────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ JuiceFS StorageClass (RWX, 跨节点共享) │ │
|
||||
│ │ juicefs-sc-{release} │ │
|
||||
│ │ │ │
|
||||
│ │ PVC: rcoder-workspace (50Gi) │ │
|
||||
│ │ ├── /app/project_workspace ← rcoder 主服务读写 │ │
|
||||
│ │ └── /app/computer-project-workspace ← computer agent runner 用 │ │
|
||||
│ └───────────────────────────────┬──────────────────────────────────────┘ │
|
||||
└──────────────────────────────────┼──────────────────────────────────────────┘
|
||||
│
|
||||
┌───────────────────┼────────────────────┐
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────────────────────┐ ┌────────────────┐ ┌─────────────────────────────┐
|
||||
│ PostgreSQL │ │ MinIO │ │ JuiceFS CSI Driver │
|
||||
│ StatefulSet (RWO) │ │ StatefulSet │ │ (kube-system) │
|
||||
│ │ │ (RWO) │ │ │
|
||||
│ • JuiceFS 元数据存储 │ │ │ │ • csi-provisioner │
|
||||
│ • DB: juicefs │ │ bucket: juicefs│ │ • csi-node-driver-registrar │
|
||||
│ • Longhorn / local-path │ │ │ │ • juicefs-plugin (DaemonSet) │
|
||||
│ • 10Gi PVC │ │ Longhorn / │ │ │
|
||||
│ │ │ local-path │ │ │
|
||||
└──────────────────────────┘ │ 30Gi PVC │ └─────────────────────────────┘
|
||||
└────────────────┘
|
||||
▲ │
|
||||
│ │
|
||||
└───────────────────────┬─────────────────┘
|
||||
│
|
||||
┌────────▼────────┐
|
||||
│ Longhorn / │
|
||||
│ local-path │
|
||||
│ (块存储 RWO) │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. 存储架构详解
|
||||
|
||||
### 2.1 存储分层
|
||||
|
||||
| 层次 | 存储类型 | 访问模式 | 使用场景 | 典型大小 |
|
||||
|------|---------|---------|---------|---------|
|
||||
| **共享文件系统** | JuiceFS (CSI) | RWX 跨节点 | rcoder workspace, project_workspace, computer_workspace | 50Gi |
|
||||
| **对象存储** | MinIO (S3) | RWO 单节点 | JuiceFS 数据块后端 + 应用文件存储 | 30Gi+ |
|
||||
| **元数据库** | PostgreSQL | RWO 单节点 | JuiceFS 文件元数据 (inode/权限/目录结构) | 10Gi |
|
||||
| **块存储** | Longhorn / local-path | RWO 单节点 | PostgreSQL 数据盘, MinIO 数据盘 | 按需 |
|
||||
|
||||
### 2.2 JuiceFS 数据流
|
||||
|
||||
```
|
||||
用户/应用 Pod ──写入──▶ JuiceFS FUSE Mount (CSI Driver)
|
||||
│
|
||||
▼
|
||||
PostgreSQL (元数据)
|
||||
• 文件名 / inode 编号
|
||||
• 权限 / 所有者
|
||||
• 目录结构
|
||||
• 硬链接 / 符号链接
|
||||
│
|
||||
▼
|
||||
MinIO S3 (数据块)
|
||||
• chunk-xxx 文件
|
||||
• juicefs.db 内部索引
|
||||
```
|
||||
|
||||
**为什么这样分层：**
|
||||
- **JuiceFS = 共享层**：提供 POSIX RWX，多节点 rcoder Pod 和动态创建的 agent-runner Pod 可以同时读写同一份项目文件
|
||||
- **PostgreSQL = 元数据**：文件系统"目录结构/文件名/权限"存这里，JuiceFS mount pod 重启不丢文件索引
|
||||
- **MinIO = 数据块后端**：文件内容 chunk 存在 MinIO bucket，不存在节点本地，JuiceFS mount pod 重启不丢数据
|
||||
- **Longhorn/local-path = 数据库盘**：MySQL/Redis/Milvus/ES 不能用 JuiceFS（延迟太高），强制走块存储
|
||||
|
||||
### 2.3 Longhorn vs local-path
|
||||
|
||||
| 维度 | local-path (k3s 内置) | Longhorn |
|
||||
|------|----------------------|---------|
|
||||
| 多节点副本 | ❌ 单节点独占 | ✅ 3 副本同步分布不同节点 |
|
||||
| 扩容 | ❌ 手动迁数据 | ✅ UI/CLI 秒扩 |
|
||||
| 快照/回滚 | ❌ | ✅ |
|
||||
| 备份到 S3 | ❌ | ✅ |
|
||||
| UI 管理 | ❌ | ✅ (:9000) |
|
||||
| 资源开销 | ~0 | ~500MB agent/node |
|
||||
| 适用场景 | 单节点 dev | 多节点 prod |
|
||||
|
||||
**扩展策略**：
|
||||
- 当前 `values.yaml` 里 `storageClass: local-path`（dev够用）
|
||||
- 未来多节点集群装好 Longhorn 后，只需把 `storageClass` 改为 `longhorn`，所有 StatefulSet 的 PVC 自动用 Longhorn
|
||||
|
||||
### 2.4 JuiceFS Secret 关键配置
|
||||
|
||||
```yaml
|
||||
# juicefs-secret (渲染后)
|
||||
name: "rcoder-juicefs"
|
||||
metaurl: "postgres://juicefs:<pass>@postgresql.<ns>:5432/juicefs"
|
||||
storage: "minio"
|
||||
bucket: "http://minio-service.<ns>:9000/juicefs"
|
||||
access-key: "<minio-root-user>"
|
||||
secret-key: "<minio-root-password>"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 核心组件
|
||||
|
||||
### 3.1 rcoder 主服务 (Deployment)
|
||||
|
||||
**关键环境变量**：
|
||||
|
||||
| 环境变量 | 值 | 作用 |
|
||||
|---------|---|------|
|
||||
| `CONTAINER_RUNTIME` | `kubernetes` | 激活 K8s API 模式，创建动态 Pod 而非 Docker 容器 |
|
||||
| `RCODER_K8S_NAMESPACE` | `{{ .Release.Namespace }}` | rcoder 通过此 namespace 创建 agent-runner Pod |
|
||||
| `RCODER_K8S_STORAGE_CLASS` | `juicefs-sc-{{ .Release.Name }}` | 动态 Pod 挂 JuiceFS PVC 用的 SC 名 |
|
||||
| `RCODER_DOCKER_IMAGE` | `rcoder-k8s:latest` | agent-runner Pod 的基础镜像 |
|
||||
| `RCODER_DOCKER_IMAGE_COMPUTER` | `rcoder-computer-agent-runner:latest` | computer agent runner 镜像 |
|
||||
|
||||
**挂载的卷**：
|
||||
|
||||
| 卷名 | 来源 | 挂载路径 | 用途 |
|
||||
|------|------|---------|------|
|
||||
| rcoder-config | ConfigMap | `/app/config.yml` | rcoder 运行时配置 |
|
||||
| rcoder-workspace | JuiceFS PVC (RWX) | `/app/project_workspace` (subPath: `project_workspace`) | 项目工作区 |
|
||||
| rcoder-workspace | JuiceFS PVC (RWX) | `/app/computer-project-workspace` (subPath: `computer_workspace`) | Computer agent 工作区 |
|
||||
|
||||
**权限模型**：
|
||||
- ServiceAccount: `rcoder-pods-sa`
|
||||
- ClusterRole: `rcoder-pods-clusterrole` (集群级，多 release 共享)
|
||||
- `pods`: create / delete / get / list / watch / patch / update
|
||||
- `pods/log`: get / list
|
||||
- `pods/exec`: create
|
||||
- `pods/status`: get
|
||||
- `persistentvolumeclaims`: get / list / watch / create / delete
|
||||
|
||||
### 3.2 动态 Agent Runner Pod
|
||||
|
||||
rcoder 主服务通过 K8s API 动态创建的临时 Pod：
|
||||
|
||||
```
|
||||
rcoder 主服务 (Deployment)
|
||||
│
|
||||
├── 收到 /chat 请求
|
||||
│
|
||||
├── 调用 K8s API 创建 Pod:
|
||||
│ • name: rcoder-agent-runner-<session-id>
|
||||
│ • image: rcoder-computer-agent-runner
|
||||
│ • env: RCODER_K8S_NAMESPACE / RCODER_DOCKER_IMAGE_* 等
|
||||
│ • volumes:
|
||||
│ - project_workspace (JuiceFS RWX, subPath)
|
||||
│ - computer_workspace (JuiceFS RWX, subPath)
|
||||
│ - emptyDir cache (可重建)
|
||||
│
|
||||
└── Pod 内 gRPC 服务，rcoder 主服务通过 serviceDNS:50051 连接
|
||||
```
|
||||
|
||||
**与 docker-compose 模式的区别**：
|
||||
|
||||
| 维度 | docker-compose (Docker) | K8s |
|
||||
|------|------------------------|-----|
|
||||
| 容器创建 | docker CLI / Docker API | K8s API |
|
||||
| 通信方式 | Docker 内部网络 + 端口映射 | K8s ClusterIP Service DNS |
|
||||
| 工作空间 | hostPath bind-mount | JuiceFS RWX PVC |
|
||||
| 清理 | docker stop + rm | K8s Pod delete |
|
||||
| socket | /var/run/docker.sock | 不需要 |
|
||||
|
||||
### 3.3 PostgreSQL (StatefulSet)
|
||||
|
||||
- **用途**：JuiceFS 元数据存储
|
||||
- **数据库**：`juicefs` (JuiceFS 自动建)
|
||||
- **用户**：`juicefs` (来自 credentials Secret)
|
||||
- **存储**：Longhorn / local-path 块存储 PVC (RWO)
|
||||
- **initContainers**: 清理 `lost+found` 目录避免 PG 初始化失败
|
||||
|
||||
### 3.4 MinIO (StatefulSet)
|
||||
|
||||
- **用途**：JuiceFS S3 数据块后端
|
||||
- **bucket**：`juicefs` (由 minio-init Job 创建)
|
||||
- **凭据**：来自 `rcoder-credentials` Secret (`MINIO_ROOT_USER` / `MINIO_ROOT_PASSWORD`)
|
||||
- **存储**：Longhorn / local-path 块存储 PVC (RWO)
|
||||
- **健康检查**：
|
||||
- Liveness: `/minio/health/live`
|
||||
- Readiness: `/minio/health/ready`
|
||||
|
||||
### 3.5 minio-init Job
|
||||
|
||||
- **触发时机**：`helm.sh/hook: post-install,post-upgrade`
|
||||
- **作用**：确保 `juicefs` bucket 存在
|
||||
- **策略**：`helm.sh/hook-delete-policy: before-hook-creation` 每次 upgrade 重新运行
|
||||
- **镜像**：`minio/mc` (MinIO Client)
|
||||
|
||||
---
|
||||
|
||||
## 4. 部署方式
|
||||
|
||||
### 4.1 三种部署方式对比
|
||||
|
||||
| 方式 | 入口 | 适用场景 | 维护者 |
|
||||
|------|------|---------|--------|
|
||||
| **Kustomize** | `deploy-dev.sh` / `deploy-prod.sh` | 日常开发 / 在线环境快速迭代 | 开发团队 |
|
||||
| **Helm** | `helm install ... k8s/helm/rcoder` | 参数化部署 / 对外交付 | 交付/运维 |
|
||||
| **Offline Bundle** | `make k8s-offline-bundle` → `install.sh` | 政企客户完全断网 | 交付 |
|
||||
|
||||
### 4.2 Kustomize 部署流程
|
||||
|
||||
```
|
||||
deploy-dev.sh 顺序执行:
|
||||
|
||||
[1/6] 检查 K8s 集群 + kubectl
|
||||
│
|
||||
[1.5/6] 检查/安装 open-iscsi (Longhorn 依赖)
|
||||
│
|
||||
[2/6] 部署 Longhorn (若未安装)
|
||||
│ └── kubectl apply -f https://raw.githubusercontent.com/longhorn/longhorn/master/deploy/longhorn.yaml
|
||||
│
|
||||
[3/6] 部署 JuiceFS CSI Driver (若未安装)
|
||||
│ └── helm install juicefs-csi-driver juicedata/juicefs-csi-driver --namespace kube-system
|
||||
│
|
||||
[4/6] Kustomize 部署应用 (manifests/overlays/dev)
|
||||
│ └── kubectl apply -k manifests/overlays/dev
|
||||
│ 顺序:
|
||||
│ 1. namespace.yaml
|
||||
│ 2. storage/ (postgresql → minio → minio-init → juicefs-secret → juicefs-sc → juicefs-pvc)
|
||||
│ 3. rcoder/ (SA → ClusterRole/CRB → ConfigMap → Deployment → Service → NetworkPolicy → PDB)
|
||||
│
|
||||
[5/6] 验证部署状态
|
||||
│
|
||||
[6/6] 输出访问信息
|
||||
```
|
||||
|
||||
### 4.3 Helm 部署
|
||||
|
||||
```bash
|
||||
# dev
|
||||
helm install rcoder-dev k8s/helm/rcoder \
|
||||
--namespace nuwax-rcoder-dev --create-namespace \
|
||||
-f k8s/helm/rcoder/values-dev.yaml
|
||||
|
||||
# prod (密码通过 --set 注入)
|
||||
helm install rcoder-prod k8s/helm/rcoder \
|
||||
--namespace nuwax-rcoder-prod --create-namespace \
|
||||
-f k8s/helm/rcoder/values-prod.yaml \
|
||||
--set credentials.postgresql.password=<real> \
|
||||
--set credentials.minio.rootPassword=<real>
|
||||
|
||||
# 同集群 Kustomize + Helm 并存 (Helm 接管 ClusterRole)
|
||||
helm install rcoder-helm k8s/helm/rcoder \
|
||||
--set rcoder.clusterRole.create=false
|
||||
```
|
||||
|
||||
### 4.4 离线部署
|
||||
|
||||
**构建 (有网机器)**：
|
||||
```bash
|
||||
make k8s-offline-bundle
|
||||
# 产出: dist/rcoder-offline-<ver>-<arch>.tar.gz (~1.7-2GB, 27个镜像)
|
||||
```
|
||||
|
||||
**安装 (客户内网, 两种模式)**：
|
||||
|
||||
| 模式 | 镜像导入方式 | 适用场景 |
|
||||
|------|------------|---------|
|
||||
| `--mode=direct` | `ctr image import` 到节点 containerd | 单节点 / 小集群, 无私有 registry |
|
||||
| `--mode=registry` | re-tag + push 到客户 Harbor/Nexus/ACR | 多节点, 已有私有 registry |
|
||||
|
||||
**跳过选项**：
|
||||
|
||||
| 参数 | 何时使用 |
|
||||
|------|---------|
|
||||
| `--skip-longhorn` | 集群已有 Ceph/NFS/CSI-local-path |
|
||||
| `--skip-juicefs-csi` | 集群已有 JuiceFS CSI 或用其他 RWX 方案 |
|
||||
| `--skip-image-import` | 镜像已手工导入完成 |
|
||||
|
||||
---
|
||||
|
||||
## 5. 多环境共存
|
||||
|
||||
同一集群可同时跑 dev / test / prod，通过 release 名字 + namespace 天然隔离：
|
||||
|
||||
| 资源 | dev (nuwax-rcoder-dev) | prod (nuwax-rcoder-prod) |
|
||||
|------|----------------------|--------------------------|
|
||||
| Namespace | `nuwax-rcoder-dev` | `nuwax-rcoder-prod` |
|
||||
| StorageClass | `juicefs-sc-dev` | `juicefs-sc-prod` |
|
||||
| ClusterRoleBinding | `rcoder-dev-pods-crb` | `rcoder-prod-pods-crb` |
|
||||
| ClusterRole | `rcoder-pods-clusterrole` (共享, `helm.sh/resource-policy: keep`) | 同 |
|
||||
| PostgreSQL | `postgresql` StatefulSet | 同名 (不同 namespace) |
|
||||
| MinIO | `minio` StatefulSet | 同名 (不同 namespace) |
|
||||
| rcoder Deployment | `rcoder` Deployment | 同名 (不同 namespace) |
|
||||
| NodePort | 30080 | 30081 |
|
||||
|
||||
---
|
||||
|
||||
## 6. 镜像清单
|
||||
|
||||
**离线包共 27 个镜像，分 5 类**：
|
||||
|
||||
### 6.1 RCoder 自有镜像
|
||||
|
||||
| 镜像 | 用途 | 架构 |
|
||||
|------|------|------|
|
||||
| `rcoder` | rcoder 主服务 (docker-compose 用, 不进 K8s) | amd64 + arm64 |
|
||||
| `rcoder-k8s` | rcoder K8s 变体 (`CARGO_FEATURES=kubernetes`) | amd64 + arm64 |
|
||||
| `rcoder-agent-runner` | 动态 agent-runner 基础镜像 | amd64 + arm64 |
|
||||
|
||||
### 6.2 存储层
|
||||
|
||||
| 镜像 | 用途 | 版本 |
|
||||
|------|------|------|
|
||||
| `postgres:16-alpine` | JuiceFS 元数据 | 16-alpine |
|
||||
| `minio/minio` | S3 对象存储 + JuiceFS bucket | RELEASE.2024-12-18T13-15-44Z |
|
||||
| `minio/mc` | MinIO Client (init Job 用) | RELEASE.2024-11-21T17-21-54Z |
|
||||
| `busybox:1.36` | init container / DNS check | 1.36 |
|
||||
|
||||
### 6.3 JuiceFS CE
|
||||
|
||||
| 镜像 | 用途 |
|
||||
|------|------|
|
||||
| `juicedata/mount:ce-v1.3.1` | JuiceFS FUSE Mount Pod 镜像 |
|
||||
| `juicedata/juicefs-csi-driver:v0.31.3` | CSI Driver 主镜像 |
|
||||
| `juicedata/csi-dashboard:v0.31.3` | CSI Dashboard |
|
||||
|
||||
### 6.4 JuiceFS CSI Sidecars
|
||||
|
||||
| 镜像 | 版本 |
|
||||
|------|------|
|
||||
| `registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0` |
|
||||
| `registry.k8s.io/sig-storage/csi-provisioner:v3.6.0` |
|
||||
| `registry.k8s.io/sig-storage/csi-resizer:v1.9.0` |
|
||||
| `registry.k8s.io/sig-storage/livenessprobe:v2.11.0` |
|
||||
|
||||
### 6.5 Longhorn v1.7.2
|
||||
|
||||
| 镜像 | 用途 |
|
||||
|------|------|
|
||||
| `longhornio/longhorn-manager:v1.7.2` | Longhorn 控制平面 |
|
||||
| `longhornio/longhorn-engine:v1.7.2` | 存储引擎 |
|
||||
| `longhornio/longhorn-ui:v1.7.2` | Web UI |
|
||||
| `longhornio/longhorn-instance-manager:v1.7.2` | 实例管理 |
|
||||
| `longhornio/longhorn-share-manager:v1.7.2` | NFS / iSCSI 共享 |
|
||||
| `longhornio/backing-image-manager:v1.7.2` | 镜像管理 |
|
||||
| `longhornio/longhorn-cli:v1.7.2` | CLI |
|
||||
| `longhornio/support-bundle-kit:v0.0.45` | 诊断工具 |
|
||||
| `longhornio/csi-attacher:v4.7.0` | CSI attacher |
|
||||
| `longhornio/csi-provisioner:v4.0.1-20241007` | CSI provisioner |
|
||||
| `longhornio/csi-resizer:v1.12.0` | CSI resizer |
|
||||
| `longhornio/csi-snapshotter:v7.0.2-20241007` | CSI snapshotter |
|
||||
| `longhornio/csi-node-driver-registrar:v2.12.0` | CSI node driver |
|
||||
| `longhornio/livenessprobe:v2.14.0` | Liveness probe |
|
||||
|
||||
---
|
||||
|
||||
## 7. 文件结构
|
||||
|
||||
```
|
||||
k8s/
|
||||
├── docs/
|
||||
│ └── ARCHITECTURE.md ← 本文档
|
||||
│
|
||||
├── README.md ← 总览 + 快速开始
|
||||
├── nuwax-platform-k8s-plan.md ← 迁移计划 (build-agent-docker 视角)
|
||||
│
|
||||
├── deploy-dev.sh ← Kustomize dev 部署 (一键)
|
||||
├── deploy-prod.sh ← Kustomize prod 部署 (含密码守卫)
|
||||
├── undeploy.sh ← 清理脚本
|
||||
│
|
||||
├── helm/ ← Helm chart (对外交付 + 离线源)
|
||||
│ └── rcoder/
|
||||
│ ├── Chart.yaml
|
||||
│ ├── .helmignore
|
||||
│ ├── values.yaml ← 默认值 (dev + prod 共享)
|
||||
│ ├── values-dev.yaml ← dev 覆盖 (NodePort 30080, local-path)
|
||||
│ ├── values-prod.yaml ← prod 覆盖 (密码 CHANGE-ME)
|
||||
│ ├── values-offline.yaml ← 离线 registry 覆盖
|
||||
│ └── templates/
|
||||
│ ├── _helpers.tpl ← 镜像拼装/SC名/ClusterRole名 helpers
|
||||
│ ├── NOTES.txt
|
||||
│ ├── storage/
|
||||
│ │ ├── credentials-secret.yaml
|
||||
│ │ ├── juicefs-secret.yaml
|
||||
│ │ ├── juicefs-storageclass.yaml
|
||||
│ │ ├── juicefs-pvc.yaml ← RWX PVC (JuiceFS SC)
|
||||
│ │ ├── postgresql-statefulset.yaml
|
||||
│ │ ├── postgresql-service.yaml
|
||||
│ │ ├── minio-statefulset.yaml
|
||||
│ │ ├── minio-service.yaml
|
||||
│ │ └── minio-init-job.yaml ← 创建 juicefs bucket
|
||||
│ └── rcoder/
|
||||
│ ├── clusterrole.yaml ← 集群级, helm.sh/resource-policy: keep
|
||||
│ ├── clusterrolebinding.yaml
|
||||
│ ├── serviceaccount.yaml
|
||||
│ ├── deployment.yaml ← CONTAINER_RUNTIME=kubernetes
|
||||
│ ├── service.yaml
|
||||
│ ├── configmap.yaml ← config.yml
|
||||
│ ├── networkpolicy.yaml
|
||||
│ └── pdb.yaml
|
||||
│
|
||||
├── manifests/ ← Kustomize (日常开发用)
|
||||
│ ├── base/
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ ├── namespace.yaml
|
||||
│ │ ├── storage/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ ├── juicefs-pvc.yaml
|
||||
│ │ │ ├── postgresql-deployment.yaml
|
||||
│ │ │ ├── minio-deployment.yaml
|
||||
│ │ │ └── minio-init-job.yaml
|
||||
│ │ └── rcoder/
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ ├── rcoder-deployment.yaml
|
||||
│ │ ├── rcoder-service.yaml
|
||||
│ │ ├── rcoder-configmap.yaml
|
||||
│ │ ├── rcoder-networkpolicy.yaml
|
||||
│ │ ├── rcoder-pdb.yaml
|
||||
│ │ └── serviceaccount.yaml
|
||||
│ ├── overlays/
|
||||
│ │ ├── dev/
|
||||
│ │ │ ├── kustomization.yaml
|
||||
│ │ │ ├── clusterrolebinding.yaml
|
||||
│ │ │ ├── credentials.yaml
|
||||
│ │ │ ├── juicefs-secret.yaml
|
||||
│ │ │ ├── juicefs-storageclass.yaml
|
||||
│ │ │ └── rcoder-configmap.yaml
|
||||
│ │ └── prod/
|
||||
│ │ ├── kustomization.yaml
|
||||
│ │ ├── clusterrolebinding.yaml
|
||||
│ │ ├── credentials.yaml
|
||||
│ │ ├── juicefs-secret.yaml
|
||||
│ │ ├── juicefs-storageclass.yaml
|
||||
│ │ └── rcoder-configmap.yaml
|
||||
│ ├── _deprecated/
|
||||
│ │ └── nfs/
|
||||
│ │ ├── nfs-server.yaml
|
||||
│ │ └── nfs-subdir-provisioner.yaml
|
||||
│ └── JUICEFS_DEPLOYMENT.md ← JuiceFS CSI 手动部署指南
|
||||
│
|
||||
├── offline/ ← 离线部署工具
|
||||
│ ├── images.txt ← 27 个镜像清单 (版本 pin)
|
||||
│ ├── install.sh ← 主安装脚本 (direct / registry 双模式)
|
||||
│ ├── rewrite-registry.sh ← registry 模式 re-tag + push
|
||||
│ └── README.md ← 客户交付手册
|
||||
│
|
||||
├── register2/ ← 本地私有镜像仓库 (可选)
|
||||
│ ├── docker-compose.yml ← registry:2 + 数据卷
|
||||
│ ├── registry-config.yml
|
||||
│ └── README.md
|
||||
│
|
||||
└── scripts/
|
||||
├── deploy-juicefs.sh ← 仅部署 JuiceFS CSI (不装 Longhorn)
|
||||
├── test-chat.sh ← 冒烟测试脚本
|
||||
├── install-k3s-registry-mirrors-cn.sh ← K3s 镜像加速配置
|
||||
└── k3s-registries-cn.yaml ← 镜像加速配置模板
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 关键设计决策
|
||||
|
||||
### 8.1 rcoder-k8s 独立镜像
|
||||
|
||||
**方式 A（采用）**：新增 `rcoder-k8s` 独立镜像 tag
|
||||
- `Dockerfile` 加 `ARG CARGO_FEATURES`
|
||||
- `make build-rcoder-k8s` 传 `CARGO_FEATURES=kubernetes`
|
||||
- 不挂 docker.sock，通过 K8s API 创建动态 agent-runner Pod
|
||||
- docker-compose 用的 `rcoder` 镜像不受影响
|
||||
|
||||
**未采用方式 B**：让 `rcoder` 镜像始终含 kubernetes feature
|
||||
- docker-compose 老用户也会带上 K8s 代码（轻微浪费）
|
||||
|
||||
### 8.2 集群级资源命名策略
|
||||
|
||||
| 资源类型 | 命名策略 | 理由 |
|
||||
|---------|---------|------|
|
||||
| StorageClass | `juicefs-sc-{release}` | 每个 release 独立，避免多环境冲突 |
|
||||
| ClusterRole | `rcoder-pods-clusterrole` (固定) | 规则相同，共享最简；`helm.sh/resource-policy: keep` 防误删 |
|
||||
| ClusterRoleBinding | `{release}-pods-crb` | release 独立 |
|
||||
|
||||
### 8.3 密码管理
|
||||
|
||||
| 环境 | 策略 |
|
||||
|------|------|
|
||||
| dev | 明文默认值（`CHANGE-ME`），开箱即用 |
|
||||
| prod | `CHANGE-ME-BEFORE-DEPLOY` 占位符，`deploy-prod.sh` 守卫拒绝部署 |
|
||||
| 进阶 | 推荐 `--set` / SealedSecret / ExternalSecret 注入 |
|
||||
|
||||
### 8.4 JuiceFS vs 其他 RWX 方案
|
||||
|
||||
| 方案 | 优势 | 劣势 |
|
||||
|------|------|------|
|
||||
| JuiceFS + MinIO + PG | K8s 原生，跨节点 POSIX，CSI 集成 | 需要 MinIO + PG 两个有状态组件 |
|
||||
| NFS Server | 简单 | 单点，无副本，无 CSI 原生集成 |
|
||||
| CephFS | 成熟，副本分布 | 需要 Ceph 集群，运维复杂 |
|
||||
| Longhorn NFS | Longhorn 一个组件搞定 | Longhorn NFS 共享盘性能一般 |
|
||||
|阿里云 NAS/EFS | 云上托管 | 绑定云厂商 |
|
||||
|
||||
选择 JuiceFS 的核心原因：RCoder 的核心价值是跨 agent-runner Pod 共享项目文件，JuiceFS 提供 POSIX RWX + K8s CSI 原生支持，是最轻量的自建方案。
|
||||
|
||||
---
|
||||
|
||||
## 9. 环境要求
|
||||
|
||||
| 组件 | 版本要求 | 说明 |
|
||||
|------|---------|------|
|
||||
| Kubernetes | 1.19+ | 已在 K3s 1.34.x 测试通过 |
|
||||
| kubectl | 与集群匹配 | `kubectl cluster-info` 能通 |
|
||||
| helm | 3.x | 用于 Helm 部署路径 |
|
||||
| docker | 最新 | 仅 `make k8s-offline-bundle` 或 push 到 register2 时需要 |
|
||||
| k3s / nerdctl / ctr | 最新 | 仅 direct 模式离线部署需要 |
|
||||
| open-iscsi | 任意 | Longhorn 依赖 (`apt install open-iscsi`) |
|
||||
|
||||
---
|
||||
|
||||
## 10. 运维常用命令
|
||||
|
||||
```bash
|
||||
# 查看所有资源
|
||||
kubectl get all -n nuwax-rcoder-dev
|
||||
|
||||
# 查看 Pod 日志
|
||||
kubectl logs -n nuwax-rcoder-dev -l app=rcoder --tail=200 -f
|
||||
|
||||
# 查看 JuiceFS 挂载
|
||||
kubectl exec -n nuwax-rcoder-dev deploy/rcoder -- df -h | grep juicefs
|
||||
|
||||
# 进入 Pod 调试
|
||||
kubectl exec -it -n nuwax-rcoder-dev deploy/rcoder -- sh
|
||||
|
||||
# 查看 PVC 状态
|
||||
kubectl get pvc -n nuwax-rcoder-dev
|
||||
|
||||
# 查看 StorageClass
|
||||
kubectl get sc | grep -E "longhorn|juicefs"
|
||||
|
||||
# 重启 rcoder deployment
|
||||
kubectl rollout restart deploy/rcoder -n nuwax-rcoder-dev
|
||||
kubectl rollout status deploy/rcoder -n nuwax-rcoder-dev
|
||||
|
||||
# Longhorn UI
|
||||
kubectl port-forward svc/longhorn-frontend 8080:80 -n longhorn-system
|
||||
|
||||
# 清理 dev 环境
|
||||
ENV=dev ./undeploy.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. 故障排查
|
||||
|
||||
### Pod 一直 Pending
|
||||
|
||||
```bash
|
||||
kubectl describe pod -n nuwax-rcoder-dev <pod-name>
|
||||
|
||||
# 常见原因:
|
||||
# - ImagePullBackOff: 节点 containerd 没有镜像 (direct 模式漏了某节点)
|
||||
# - ContainerCreating: Longhorn / JuiceFS CSI 未就绪
|
||||
```
|
||||
|
||||
### JuiceFS 挂载失败
|
||||
|
||||
```bash
|
||||
# CSI Driver 日志
|
||||
kubectl logs -n kube-system -l app=juicefs-csi-driver-node --tail=50
|
||||
|
||||
# 检查 PostgreSQL / MinIO 是否就绪
|
||||
kubectl get pods -n nuwax-rcoder-dev -l app=postgresql
|
||||
kubectl get pods -n nuwax-rcoder-dev -l app=minio
|
||||
|
||||
# JuiceFS mount pod 是否 Running
|
||||
kubectl get pods -n nuwax-rcoder-dev | grep juicefs
|
||||
```
|
||||
|
||||
### Longhorn 无法启动
|
||||
|
||||
```bash
|
||||
# 检查 open-iscsi
|
||||
ssh <node> "sudo apt install -y open-iscsi && sudo systemctl enable --now iscsid"
|
||||
kubectl delete pod -n longhorn-system -l app=longhorn-manager --force
|
||||
```
|
||||
|
||||
### rcoder 健康检查失败
|
||||
|
||||
```bash
|
||||
# 查看 rcoder 日志 (RUST_LOG=info)
|
||||
kubectl logs -n nuwax-rcoder-dev -l app=rcoder --tail=100
|
||||
|
||||
# 验证 K8s API 连通性 (rcoder pod 内)
|
||||
kubectl exec -it -n nuwax-rcoder-dev deploy/rcoder -- sh
|
||||
kubectl auth can-i create pods -n nuwax-rcoder-dev --as=system:serviceaccount:nuwax-rcoder-dev:rcoder-pods-sa
|
||||
```
|
||||
15
qiming-rcoder/k8s/helm/rcoder/.helmignore
Normal file
15
qiming-rcoder/k8s/helm/rcoder/.helmignore
Normal file
@@ -0,0 +1,15 @@
|
||||
# Patterns to ignore when building packages.
|
||||
.DS_Store
|
||||
.git/
|
||||
.gitignore
|
||||
.vscode/
|
||||
.idea/
|
||||
*.tmproj
|
||||
*.bak
|
||||
*.swp
|
||||
*.orig
|
||||
*~
|
||||
# OWNERS file
|
||||
OWNERS
|
||||
# Helm
|
||||
.helmignore
|
||||
18
qiming-rcoder/k8s/helm/rcoder/Chart.yaml
Normal file
18
qiming-rcoder/k8s/helm/rcoder/Chart.yaml
Normal file
@@ -0,0 +1,18 @@
|
||||
apiVersion: v2
|
||||
name: rcoder
|
||||
description: RCoder AI 驱动开发平台 (ACP-based, Rust) —— 含 RCoder 主服务 + PostgreSQL + MinIO + JuiceFS StorageClass
|
||||
type: application
|
||||
version: 1.0.0
|
||||
appVersion: "latest"
|
||||
kubeVersion: ">=1.19.0-0"
|
||||
keywords:
|
||||
- rcoder
|
||||
- acp
|
||||
- ai
|
||||
- agent
|
||||
- juicefs
|
||||
- rust
|
||||
maintainers:
|
||||
- name: RCoder Team
|
||||
annotations:
|
||||
category: AI
|
||||
39
qiming-rcoder/k8s/helm/rcoder/templates/NOTES.txt
Normal file
39
qiming-rcoder/k8s/helm/rcoder/templates/NOTES.txt
Normal file
@@ -0,0 +1,39 @@
|
||||
RCoder {{ .Chart.AppVersion }} 已安装到 namespace "{{ .Release.Namespace }}" (release: {{ .Release.Name }}).
|
||||
|
||||
环境: {{ .Values.environment }}
|
||||
副本数: {{ .Values.rcoder.replicas }}
|
||||
NodePort: {{ .Values.rcoder.service.nodePort }}
|
||||
StorageClass: {{ include "rcoder.storageClassName" . }}
|
||||
|
||||
1. 查看部署状态：
|
||||
kubectl get pods -n {{ .Release.Namespace }}
|
||||
kubectl get svc -n {{ .Release.Namespace }}
|
||||
kubectl get pvc -n {{ .Release.Namespace }}
|
||||
|
||||
2. 等待就绪（首次部署 PostgreSQL + MinIO + JuiceFS mount 约需 2-5 分钟）：
|
||||
kubectl rollout status deploy/rcoder -n {{ .Release.Namespace }} --timeout=300s
|
||||
|
||||
3. 访问服务 (NodePort):
|
||||
{{- if eq .Values.rcoder.service.type "NodePort" }}
|
||||
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
|
||||
curl http://${NODE_IP}:{{ .Values.rcoder.service.nodePort }}/health
|
||||
{{- else }}
|
||||
kubectl port-forward svc/rcoder {{ .Values.rcoder.service.port }}:{{ .Values.rcoder.service.port }} -n {{ .Release.Namespace }}
|
||||
curl http://localhost:{{ .Values.rcoder.service.port }}/health
|
||||
{{- end }}
|
||||
|
||||
4. 查看日志：
|
||||
kubectl logs -n {{ .Release.Namespace }} -l app=rcoder -f
|
||||
|
||||
{{- if eq .Values.environment "prod" }}
|
||||
|
||||
⚠️ 生产环境安全提醒：
|
||||
- credentials.postgresql.password / credentials.minio.rootPassword 不应使用默认值
|
||||
- 建议使用 --set 动态注入或 SealedSecret / ExternalSecret 方案
|
||||
- 当前密码: {{ .Values.credentials.postgresql.password }} （如为 CHANGE-ME-BEFORE-DEPLOY 请立即替换）
|
||||
{{- end }}
|
||||
|
||||
升级: helm upgrade {{ .Release.Name }} k8s/helm/rcoder -f <values>
|
||||
卸载: helm uninstall {{ .Release.Name }} --namespace {{ .Release.Namespace }}
|
||||
（注意：StorageClass 与 ClusterRoleBinding 是集群级资源，会随 release 删除；
|
||||
PostgreSQL / MinIO 的 PVC 由 reclaimPolicy 决定是否保留数据）
|
||||
140
qiming-rcoder/k8s/helm/rcoder/templates/_helpers.tpl
Normal file
140
qiming-rcoder/k8s/helm/rcoder/templates/_helpers.tpl
Normal file
@@ -0,0 +1,140 @@
|
||||
{{/*
|
||||
RCoder chart 公共 helper
|
||||
*/}}
|
||||
|
||||
{{/* 应用全名 (Release + chart) */}}
|
||||
{{- define "rcoder.fullname" -}}
|
||||
{{- if .Values.fullnameOverride -}}
|
||||
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
|
||||
{{- else -}}
|
||||
{{- $name := default .Chart.Name .Values.nameOverride -}}
|
||||
{{- if contains $name .Release.Name -}}
|
||||
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
|
||||
{{- else -}}
|
||||
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* Chart 标识 */}}
|
||||
{{- define "rcoder.chart" -}}
|
||||
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* 通用标签 (所有资源都带) */}}
|
||||
{{- define "rcoder.labels" -}}
|
||||
helm.sh/chart: {{ include "rcoder.chart" . }}
|
||||
app.kubernetes.io/name: rcoder
|
||||
app.kubernetes.io/instance: {{ .Release.Name }}
|
||||
app.kubernetes.io/managed-by: {{ .Release.Service }}
|
||||
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
|
||||
environment: {{ .Values.environment | quote }}
|
||||
{{- end -}}
|
||||
|
||||
{{/* 应用层的 selector label (app=rcoder 与 Kustomize 完全一致, 保证 Service 选得到 pod) */}}
|
||||
{{- define "rcoder.selectorLabels" -}}
|
||||
app: rcoder
|
||||
component: rcoder-main
|
||||
{{- end -}}
|
||||
|
||||
{{/* ============================================================
|
||||
镜像地址拼装
|
||||
rcoder-own image -> {{ .Values.global.imageRegistry }}/{repo}:{tag}
|
||||
third-party image -> {{ .Values.global.thirdPartyRegistry }}/{repo}:{tag}
|
||||
(若 thirdPartyRegistry 为空, 保留原始 repo)
|
||||
============================================================ */}}
|
||||
|
||||
{{/* 自有镜像: 入参 dict {repository, tag, chartContext} */}}
|
||||
{{- define "rcoder.ownImage" -}}
|
||||
{{- $repo := .repository -}}
|
||||
{{- $tag := .tag -}}
|
||||
{{- $registry := .ctx.Values.global.imageRegistry -}}
|
||||
{{- if empty $tag -}}{{- $tag = .ctx.Chart.AppVersion -}}{{- end -}}
|
||||
{{- if $registry -}}
|
||||
{{- printf "%s/%s:%s" $registry $repo $tag -}}
|
||||
{{- else -}}
|
||||
{{- printf "%s:%s" $repo $tag -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* 第三方镜像: 入参 dict {repository, tag, ctx} */}}
|
||||
{{- define "rcoder.thirdPartyImage" -}}
|
||||
{{- $repo := .repository -}}
|
||||
{{- $tag := .tag -}}
|
||||
{{- $registry := .ctx.Values.global.thirdPartyRegistry -}}
|
||||
{{- if $registry -}}
|
||||
{{- printf "%s/%s:%s" $registry $repo $tag -}}
|
||||
{{- else -}}
|
||||
{{- printf "%s:%s" $repo $tag -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* 快捷: rcoder 主服务 image */}}
|
||||
{{- define "rcoder.image" -}}
|
||||
{{- include "rcoder.ownImage" (dict "repository" .Values.rcoder.image.repository "tag" .Values.rcoder.image.tag "ctx" .) -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* 快捷: agent-runner image (供 rcoder env var 使用) */}}
|
||||
{{- define "rcoder.agentRunnerImage" -}}
|
||||
{{- include "rcoder.ownImage" (dict "repository" .Values.agentRunner.image.repository "tag" .Values.agentRunner.image.tag "ctx" .) -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* 镜像仓库前缀 (config.yml 里 docker_config.multi_image_config.global_defaults.registry_prefix 使用) */}}
|
||||
{{- define "rcoder.registryPrefix" -}}
|
||||
{{- default "" .Values.global.imageRegistry -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* ============================================================
|
||||
StorageClass 名 (集群级资源, Release 之间必须独立)
|
||||
- 用户显式指定 juicefs.storageClass.name -> 照用
|
||||
- 否则回退到 "juicefs-sc-{release}"
|
||||
============================================================ */}}
|
||||
{{- define "rcoder.storageClassName" -}}
|
||||
{{- if .Values.juicefs.storageClass.name -}}
|
||||
{{- .Values.juicefs.storageClass.name -}}
|
||||
{{- else -}}
|
||||
{{- printf "juicefs-sc-%s" .Release.Name -}}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* ============================================================
|
||||
ClusterRoleBinding 名 (集群级资源, 同上)
|
||||
============================================================ */}}
|
||||
{{- define "rcoder.clusterRoleBindingName" -}}
|
||||
{{- printf "%s-pods-crb" .Release.Name -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* ClusterRole 共享名 —— 多个 release 绑到同一个 ClusterRole, 规则相同故无冲突 */}}
|
||||
{{- define "rcoder.clusterRoleName" -}}
|
||||
rcoder-pods-clusterrole
|
||||
{{- end -}}
|
||||
|
||||
{{/* imagePullSecrets 渲染块 */}}
|
||||
{{- define "rcoder.imagePullSecrets" -}}
|
||||
{{- if .Values.global.imagePullSecrets }}
|
||||
imagePullSecrets:
|
||||
{{- range .Values.global.imagePullSecrets }}
|
||||
- name: {{ .name }}
|
||||
{{- end }}
|
||||
{{- end -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* PostgreSQL service 主机名 (namespace 内) */}}
|
||||
{{- define "rcoder.postgresqlHost" -}}
|
||||
{{- printf "postgresql.%s.svc.cluster.local" .Release.Namespace -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* MinIO service 主机名 */}}
|
||||
{{- define "rcoder.minioHost" -}}
|
||||
{{- printf "minio-service.%s.svc.cluster.local" .Release.Namespace -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* JuiceFS metaurl */}}
|
||||
{{- define "rcoder.juicefsMetaurl" -}}
|
||||
{{- printf "postgres://%s:%s@postgresql.%s:5432/juicefs" .Values.credentials.postgresql.user .Values.credentials.postgresql.password .Release.Namespace -}}
|
||||
{{- end -}}
|
||||
|
||||
{{/* JuiceFS bucket URL */}}
|
||||
{{- define "rcoder.juicefsBucketUrl" -}}
|
||||
{{- printf "http://minio-service.%s:9000/%s" .Release.Namespace .Values.juicefs.bucket -}}
|
||||
{{- end -}}
|
||||
@@ -0,0 +1,29 @@
|
||||
{{- if .Values.rcoder.clusterRole.create }}
|
||||
# 集群级 ClusterRole —— 多 release 共享同一个 (规则与 namespace 无关, 重复 apply 幂等)
|
||||
# 与 Kustomize 并存时, 把 .Values.rcoder.clusterRole.create 置 false, 让 Kustomize 持有所有权
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: {{ include "rcoder.clusterRoleName" . }}
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
annotations:
|
||||
# 避免多 release 互相 helm uninstall 时删除共享 ClusterRole
|
||||
"helm.sh/resource-policy": keep
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["pods"]
|
||||
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/log"]
|
||||
verbs: ["get", "list"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/exec"]
|
||||
verbs: ["create"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/status"]
|
||||
verbs: ["get"]
|
||||
- apiGroups: [""]
|
||||
resources: ["persistentvolumeclaims"]
|
||||
verbs: ["get", "list", "watch", "create", "delete"]
|
||||
{{- end }}
|
||||
@@ -0,0 +1,15 @@
|
||||
# 集群级 ClusterRoleBinding —— 每个 release 独立, 解决同集群多 release 冲突
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: {{ include "rcoder.clusterRoleBindingName" . }}
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: rcoder-pods-sa
|
||||
namespace: {{ .Release.Namespace }}
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: {{ include "rcoder.clusterRoleName" . }}
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
@@ -0,0 +1,70 @@
|
||||
# ConfigMap: 渲染 config.yml —— 保持与 k8s/manifests/overlays 下 rcoder-configmap.yaml 结构对齐
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: rcoder-config
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
data:
|
||||
config.yml: |
|
||||
default_agent: {{ .Values.config.defaultAgent | quote }}
|
||||
|
||||
projects_dir: {{ .Values.config.projectsDir | quote }}
|
||||
port: {{ .Values.config.port }}
|
||||
|
||||
cleanup_config:
|
||||
enabled: {{ .Values.config.cleanup.enabled }}
|
||||
idle_timeout_seconds: {{ .Values.config.cleanup.idleTimeoutSeconds }}
|
||||
cleanup_interval_seconds: {{ .Values.config.cleanup.cleanupIntervalSeconds }}
|
||||
docker_stop_timeout_seconds: {{ .Values.config.cleanup.dockerStopTimeoutSeconds }}
|
||||
|
||||
proxy_config:
|
||||
listen_port: {{ .Values.config.proxy.listenPort }}
|
||||
default_backend_port: {{ .Values.config.proxy.defaultBackendPort }}
|
||||
backend_host: {{ .Values.config.proxy.backendHost | quote }}
|
||||
port_param: {{ .Values.config.proxy.portParam | quote }}
|
||||
health_check:
|
||||
enabled: {{ .Values.config.proxy.healthCheck.enabled }}
|
||||
interval_seconds: {{ .Values.config.proxy.healthCheck.intervalSeconds }}
|
||||
timeout_seconds: {{ .Values.config.proxy.healthCheck.timeoutSeconds }}
|
||||
healthy_threshold: {{ .Values.config.proxy.healthCheck.healthyThreshold }}
|
||||
unhealthy_threshold: {{ .Values.config.proxy.healthCheck.unhealthyThreshold }}
|
||||
|
||||
docker_config:
|
||||
multi_image_config:
|
||||
global_defaults:
|
||||
registry_prefix: {{ include "rcoder.registryPrefix" . | quote }}
|
||||
services:
|
||||
rcoder:
|
||||
service_type: "RCoder"
|
||||
arm64_image: {{ include "rcoder.image" . | quote }}
|
||||
amd64_image: {{ include "rcoder.image" . | quote }}
|
||||
default_image: {{ include "rcoder.image" . | quote }}
|
||||
image_tag_prefix: "rcoder"
|
||||
enabled: true
|
||||
computer-agent-runner:
|
||||
service_type: "ComputerAgentRunner"
|
||||
arm64_image: {{ include "rcoder.agentRunnerImage" . | quote }}
|
||||
amd64_image: {{ include "rcoder.agentRunnerImage" . | quote }}
|
||||
default_image: {{ include "rcoder.agentRunnerImage" . | quote }}
|
||||
image_tag_prefix: "rcoder-computer-agent-runner"
|
||||
enabled: true
|
||||
selection_strategy: "ServiceOnly"
|
||||
cache_config:
|
||||
enabled: {{ .Values.config.dockerConfig.cache.enabled }}
|
||||
ttl_seconds: {{ .Values.config.dockerConfig.cache.ttlSeconds }}
|
||||
max_entries: {{ .Values.config.dockerConfig.cache.maxEntries }}
|
||||
|
||||
network_base_name: {{ .Values.config.dockerConfig.networkBaseName | quote }}
|
||||
network_mode: {{ .Values.config.dockerConfig.networkMode | quote }}
|
||||
work_dir: {{ .Values.config.dockerConfig.workDir | quote }}
|
||||
auto_cleanup: {{ .Values.config.dockerConfig.autoCleanup }}
|
||||
container_ttl_seconds: {{ .Values.config.dockerConfig.containerTtlSeconds }}
|
||||
|
||||
api_key_auth:
|
||||
enabled: {{ .Values.config.apiKeyAuth.enabled }}
|
||||
api_key: {{ .Values.config.apiKeyAuth.apiKey | quote }}
|
||||
|
||||
kubernetes_config:
|
||||
storage_class: {{ include "rcoder.storageClassName" . | quote }}
|
||||
@@ -0,0 +1,65 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: rcoder
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
spec:
|
||||
replicas: {{ .Values.rcoder.replicas }}
|
||||
selector:
|
||||
matchLabels:
|
||||
{{- include "rcoder.selectorLabels" . | nindent 6 }}
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 8 }}
|
||||
{{- include "rcoder.selectorLabels" . | nindent 8 }}
|
||||
spec:
|
||||
terminationGracePeriodSeconds: {{ .Values.rcoder.terminationGracePeriodSeconds }}
|
||||
serviceAccountName: rcoder-pods-sa
|
||||
{{- include "rcoder.imagePullSecrets" . | nindent 6 }}
|
||||
containers:
|
||||
- name: rcoder
|
||||
image: {{ include "rcoder.image" . }}
|
||||
imagePullPolicy: {{ .Values.global.imagePullPolicy }}
|
||||
command: ["/app/bin/rcoder"]
|
||||
args: ["--port", "{{ .Values.rcoder.service.port }}"]
|
||||
ports:
|
||||
- containerPort: {{ .Values.rcoder.service.port }}
|
||||
env:
|
||||
- name: CONTAINER_RUNTIME
|
||||
value: "kubernetes"
|
||||
- name: RCODER_K8S_NAMESPACE
|
||||
value: {{ .Release.Namespace | quote }}
|
||||
- name: RUST_LOG
|
||||
value: {{ .Values.rcoder.rustLog | quote }}
|
||||
- name: RCODER_K8S_STORAGE_CLASS
|
||||
value: {{ include "rcoder.storageClassName" . | quote }}
|
||||
- name: RCODER_DOCKER_IMAGE
|
||||
value: {{ include "rcoder.image" . | quote }}
|
||||
- name: RCODER_DOCKER_IMAGE_COMPUTER
|
||||
value: {{ include "rcoder.agentRunnerImage" . | quote }}
|
||||
resources:
|
||||
{{- toYaml .Values.rcoder.resources | nindent 12 }}
|
||||
livenessProbe:
|
||||
{{- toYaml .Values.rcoder.livenessProbe | nindent 12 }}
|
||||
readinessProbe:
|
||||
{{- toYaml .Values.rcoder.readinessProbe | nindent 12 }}
|
||||
volumeMounts:
|
||||
- name: rcoder-config
|
||||
mountPath: /app/config.yml
|
||||
subPath: config.yml
|
||||
- name: project-workspace
|
||||
mountPath: /app/project_workspace
|
||||
subPath: project_workspace
|
||||
- name: project-workspace
|
||||
mountPath: /app/computer-project-workspace
|
||||
subPath: computer_workspace
|
||||
volumes:
|
||||
- name: rcoder-config
|
||||
configMap:
|
||||
name: rcoder-config
|
||||
- name: project-workspace
|
||||
persistentVolumeClaim:
|
||||
claimName: rcoder-workspace
|
||||
@@ -0,0 +1,62 @@
|
||||
{{- if .Values.rcoder.networkPolicy.enabled }}
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: rcoder-network-policy
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: rcoder
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
# 同 namespace 入站流量
|
||||
- from:
|
||||
- podSelector: {}
|
||||
ports:
|
||||
- { protocol: TCP, port: {{ .Values.rcoder.service.port }} }
|
||||
- { protocol: TCP, port: {{ .Values.config.proxy.listenPort }} }
|
||||
egress:
|
||||
# DNS
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- { protocol: UDP, port: 53 }
|
||||
# K8s API Server
|
||||
- to:
|
||||
- namespaceSelector: {}
|
||||
podSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/service-name: kubernetes
|
||||
ports:
|
||||
- { protocol: TCP, port: 443 }
|
||||
# PostgreSQL
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: postgresql
|
||||
ports:
|
||||
- { protocol: TCP, port: {{ .Values.postgresql.service.port }} }
|
||||
# MinIO
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: minio
|
||||
ports:
|
||||
- { protocol: TCP, port: {{ .Values.minio.service.apiPort }} }
|
||||
- { protocol: TCP, port: {{ .Values.minio.service.consolePort }} }
|
||||
# 动态 agent pod (rcoder -> agent gRPC/HTTP)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
managed-by: rcoder-runtime
|
||||
ports:
|
||||
- { protocol: TCP, port: 50051 }
|
||||
- { protocol: TCP, port: 8086 }
|
||||
{{- end }}
|
||||
14
qiming-rcoder/k8s/helm/rcoder/templates/rcoder/pdb.yaml
Normal file
14
qiming-rcoder/k8s/helm/rcoder/templates/rcoder/pdb.yaml
Normal file
@@ -0,0 +1,14 @@
|
||||
{{- if .Values.rcoder.pdb.enabled }}
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: rcoder-pdb
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
spec:
|
||||
maxUnavailable: {{ .Values.rcoder.pdb.maxUnavailable }}
|
||||
selector:
|
||||
matchLabels:
|
||||
app: rcoder
|
||||
{{- end }}
|
||||
19
qiming-rcoder/k8s/helm/rcoder/templates/rcoder/service.yaml
Normal file
19
qiming-rcoder/k8s/helm/rcoder/templates/rcoder/service.yaml
Normal file
@@ -0,0 +1,19 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: rcoder
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
spec:
|
||||
type: {{ .Values.rcoder.service.type }}
|
||||
# component=rcoder-main 把动态创建的 rcoder-agent / computer-agent-runner pod
|
||||
# (它们也带 app=rcoder 标签) 排除在外, 避免 NodePort 把流量路由到不监听 8087 的 agent pod
|
||||
selector:
|
||||
{{- include "rcoder.selectorLabels" . | nindent 4 }}
|
||||
ports:
|
||||
- port: {{ .Values.rcoder.service.port }}
|
||||
targetPort: {{ .Values.rcoder.service.port }}
|
||||
{{- if eq .Values.rcoder.service.type "NodePort" }}
|
||||
nodePort: {{ .Values.rcoder.service.nodePort }}
|
||||
{{- end }}
|
||||
@@ -0,0 +1,7 @@
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: rcoder-pods-sa
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
@@ -0,0 +1,13 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: rcoder-credentials
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
type: Opaque
|
||||
stringData:
|
||||
POSTGRES_USER: {{ .Values.credentials.postgresql.user | quote }}
|
||||
POSTGRES_PASSWORD: {{ .Values.credentials.postgresql.password | quote }}
|
||||
MINIO_ROOT_USER: {{ .Values.credentials.minio.rootUser | quote }}
|
||||
MINIO_ROOT_PASSWORD: {{ .Values.credentials.minio.rootPassword | quote }}
|
||||
@@ -0,0 +1,15 @@
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: rcoder-workspace
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
app: rcoder
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
spec:
|
||||
# JuiceFS 原生支持 RWX, 多节点 rcoder pod 共享工作空间
|
||||
accessModes: ["ReadWriteMany"]
|
||||
storageClassName: {{ include "rcoder.storageClassName" . | quote }}
|
||||
resources:
|
||||
requests:
|
||||
storage: {{ .Values.workspace.size | quote }}
|
||||
@@ -0,0 +1,26 @@
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: juicefs-secret
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
type: Opaque
|
||||
stringData:
|
||||
# JuiceFS CE 必需字段 (小写键名)
|
||||
name: "rcoder-juicefs"
|
||||
metaurl: {{ include "rcoder.juicefsMetaurl" . | quote }}
|
||||
storage: "minio"
|
||||
bucket: {{ include "rcoder.juicefsBucketUrl" . | quote }}
|
||||
access-key: {{ .Values.credentials.minio.rootUser | quote }}
|
||||
secret-key: {{ .Values.credentials.minio.rootPassword | quote }}
|
||||
|
||||
# 兼容字段 (保留给脚本/日志使用)
|
||||
METAURL: {{ include "rcoder.juicefsMetaurl" . | quote }}
|
||||
MINIO_ACCESS_KEY: {{ .Values.credentials.minio.rootUser | quote }}
|
||||
MINIO_SECRET_KEY: {{ .Values.credentials.minio.rootPassword | quote }}
|
||||
MINIO_ADDRESS: {{ printf "minio-service.%s:%d" .Release.Namespace (int .Values.minio.service.apiPort) | quote }}
|
||||
|
||||
# 其他配置
|
||||
APPEND_FSYNC: {{ .Values.juicefs.meta.appendFsync | quote }}
|
||||
CACHE_SIZE: {{ .Values.juicefs.meta.cacheSize | quote }}
|
||||
@@ -0,0 +1,25 @@
|
||||
{{- if .Values.juicefs.storageClass.create }}
|
||||
# JuiceFS StorageClass —— 集群级资源, 每个 release 独立命名
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: {{ include "rcoder.storageClassName" . }}
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
annotations:
|
||||
storageclass.kubernetes.io/is-default-class: "false"
|
||||
provisioner: csi.juicefs.com
|
||||
parameters:
|
||||
csi.storage.k8s.io/provisioner-secret-name: juicefs-secret
|
||||
csi.storage.k8s.io/provisioner-secret-namespace: {{ .Release.Namespace }}
|
||||
csi.storage.k8s.io/controller-expand-secret-name: juicefs-secret
|
||||
csi.storage.k8s.io/controller-expand-secret-namespace: {{ .Release.Namespace }}
|
||||
csi.storage.k8s.io/node-publish-secret-name: juicefs-secret
|
||||
csi.storage.k8s.io/node-publish-secret-namespace: {{ .Release.Namespace }}
|
||||
juicefs/controllerQuotaSet: "false"
|
||||
juicefs/mountImage: {{ include "rcoder.thirdPartyImage" (dict "repository" .Values.juicefs.mountImage.repository "tag" .Values.juicefs.mountImage.tag "ctx" .) | quote }}
|
||||
juicefs/mountOptions: {{ .Values.juicefs.storageClass.mountOptions | quote }}
|
||||
reclaimPolicy: {{ .Values.juicefs.storageClass.reclaimPolicy }}
|
||||
allowVolumeExpansion: true
|
||||
volumeBindingMode: Immediate
|
||||
{{- end }}
|
||||
@@ -0,0 +1,47 @@
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: minio-init
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
annotations:
|
||||
# 每次 helm upgrade 强制重新运行 (确保 bucket 一定存在, 即使 Job 已清理)
|
||||
helm.sh/hook: post-install,post-upgrade
|
||||
helm.sh/hook-weight: "5"
|
||||
helm.sh/hook-delete-policy: before-hook-creation
|
||||
spec:
|
||||
ttlSecondsAfterFinished: 300
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
{{- include "rcoder.imagePullSecrets" . | nindent 6 }}
|
||||
containers:
|
||||
- name: mc
|
||||
image: {{ include "rcoder.thirdPartyImage" (dict "repository" .Values.minio.initImage.repository "tag" .Values.minio.initImage.tag "ctx" .) }}
|
||||
imagePullPolicy: {{ .Values.global.imagePullPolicy }}
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
echo "Waiting for MinIO to be ready..."
|
||||
until mc alias set myminio http://minio:{{ .Values.minio.service.apiPort }} ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD}; do
|
||||
echo "MinIO not ready, retrying..."
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "MinIO is ready, creating bucket..."
|
||||
mc mb myminio/{{ .Values.minio.bucket }} --ignore-existing
|
||||
mc ls myminio/
|
||||
echo "MinIO initialization completed!"
|
||||
env:
|
||||
- name: MINIO_ROOT_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: rcoder-credentials
|
||||
key: MINIO_ROOT_USER
|
||||
- name: MINIO_ROOT_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: rcoder-credentials
|
||||
key: MINIO_ROOT_PASSWORD
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
# Headless Service for StatefulSet
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: minio
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
app: minio
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
spec:
|
||||
clusterIP: None
|
||||
selector:
|
||||
app: minio
|
||||
ports:
|
||||
- name: api
|
||||
port: {{ .Values.minio.service.apiPort }}
|
||||
targetPort: {{ .Values.minio.service.apiPort }}
|
||||
- name: console
|
||||
port: {{ .Values.minio.service.consolePort }}
|
||||
targetPort: {{ .Values.minio.service.consolePort }}
|
||||
---
|
||||
# ClusterIP Service for internal access
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: minio-service
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
app: minio
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
spec:
|
||||
selector:
|
||||
app: minio
|
||||
ports:
|
||||
- name: api
|
||||
port: {{ .Values.minio.service.apiPort }}
|
||||
targetPort: {{ .Values.minio.service.apiPort }}
|
||||
- name: console
|
||||
port: {{ .Values.minio.service.consolePort }}
|
||||
targetPort: {{ .Values.minio.service.consolePort }}
|
||||
@@ -0,0 +1,83 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: minio-pvc
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
app: minio
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
{{- if .Values.minio.storage.storageClass }}
|
||||
storageClassName: {{ .Values.minio.storage.storageClass | quote }}
|
||||
{{- end }}
|
||||
resources:
|
||||
requests:
|
||||
storage: {{ .Values.minio.storage.size | quote }}
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: minio
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
app: minio
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
spec:
|
||||
serviceName: minio
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: minio
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: minio
|
||||
{{- include "rcoder.labels" . | nindent 8 }}
|
||||
spec:
|
||||
{{- include "rcoder.imagePullSecrets" . | nindent 6 }}
|
||||
containers:
|
||||
- name: minio
|
||||
image: {{ include "rcoder.thirdPartyImage" (dict "repository" .Values.minio.image.repository "tag" .Values.minio.image.tag "ctx" .) }}
|
||||
imagePullPolicy: {{ .Values.global.imagePullPolicy }}
|
||||
ports:
|
||||
- containerPort: {{ .Values.minio.service.apiPort }}
|
||||
name: api
|
||||
- containerPort: {{ .Values.minio.service.consolePort }}
|
||||
name: console
|
||||
env:
|
||||
- name: MINIO_ROOT_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: rcoder-credentials
|
||||
key: MINIO_ROOT_USER
|
||||
- name: MINIO_ROOT_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: rcoder-credentials
|
||||
key: MINIO_ROOT_PASSWORD
|
||||
args:
|
||||
- server
|
||||
- /data
|
||||
- --console-address
|
||||
- ":{{ .Values.minio.service.consolePort }}"
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /minio/health/live
|
||||
port: {{ .Values.minio.service.apiPort }}
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /minio/health/ready
|
||||
port: {{ .Values.minio.service.apiPort }}
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
volumeMounts:
|
||||
- name: minio-data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: minio-data
|
||||
persistentVolumeClaim:
|
||||
claimName: minio-pvc
|
||||
@@ -0,0 +1,15 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: postgresql
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
app: postgresql
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
spec:
|
||||
selector:
|
||||
app: postgresql
|
||||
ports:
|
||||
- name: postgres
|
||||
port: {{ .Values.postgresql.service.port }}
|
||||
targetPort: {{ .Values.postgresql.service.port }}
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: postgresql-pvc
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
app: postgresql
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
{{- if .Values.postgresql.storage.storageClass }}
|
||||
storageClassName: {{ .Values.postgresql.storage.storageClass | quote }}
|
||||
{{- end }}
|
||||
resources:
|
||||
requests:
|
||||
storage: {{ .Values.postgresql.storage.size | quote }}
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: postgresql
|
||||
namespace: {{ .Release.Namespace }}
|
||||
labels:
|
||||
app: postgresql
|
||||
{{- include "rcoder.labels" . | nindent 4 }}
|
||||
spec:
|
||||
serviceName: postgresql
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: postgresql
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: postgresql
|
||||
{{- include "rcoder.labels" . | nindent 8 }}
|
||||
spec:
|
||||
{{- include "rcoder.imagePullSecrets" . | nindent 6 }}
|
||||
initContainers:
|
||||
- name: cleanup
|
||||
image: {{ include "rcoder.thirdPartyImage" (dict "repository" .Values.postgresql.image.repository "tag" .Values.postgresql.image.tag "ctx" .) }}
|
||||
imagePullPolicy: {{ .Values.global.imagePullPolicy }}
|
||||
command: ["rm", "-rf", "/var/lib/postgresql/data/lost+found"]
|
||||
volumeMounts:
|
||||
- name: postgres-data
|
||||
mountPath: /var/lib/postgresql/data
|
||||
containers:
|
||||
- name: postgresql
|
||||
image: {{ include "rcoder.thirdPartyImage" (dict "repository" .Values.postgresql.image.repository "tag" .Values.postgresql.image.tag "ctx" .) }}
|
||||
imagePullPolicy: {{ .Values.global.imagePullPolicy }}
|
||||
ports:
|
||||
- containerPort: {{ .Values.postgresql.service.port }}
|
||||
name: postgres
|
||||
env:
|
||||
- name: POSTGRES_DB
|
||||
value: "juicefs"
|
||||
- name: POSTGRES_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: rcoder-credentials
|
||||
key: POSTGRES_USER
|
||||
- name: POSTGRES_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: rcoder-credentials
|
||||
key: POSTGRES_PASSWORD
|
||||
livenessProbe:
|
||||
exec:
|
||||
command: ["pg_isready", "-U", {{ .Values.credentials.postgresql.user | quote }}]
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
exec:
|
||||
command: ["pg_isready", "-U", {{ .Values.credentials.postgresql.user | quote }}]
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
volumeMounts:
|
||||
- name: postgres-data
|
||||
mountPath: /var/lib/postgresql/data
|
||||
volumes:
|
||||
- name: postgres-data
|
||||
persistentVolumeClaim:
|
||||
claimName: postgresql-pvc
|
||||
25
qiming-rcoder/k8s/helm/rcoder/values-dev.yaml
Normal file
25
qiming-rcoder/k8s/helm/rcoder/values-dev.yaml
Normal file
@@ -0,0 +1,25 @@
|
||||
# ============================================================================
|
||||
# Dev 环境覆盖
|
||||
# 使用: helm install rcoder-dev k8s/helm/rcoder -f k8s/helm/rcoder/values-dev.yaml \
|
||||
# --namespace nuwax-rcoder-dev --create-namespace
|
||||
# ============================================================================
|
||||
|
||||
environment: dev
|
||||
|
||||
rcoder:
|
||||
replicas: 1
|
||||
service:
|
||||
nodePort: 30080
|
||||
|
||||
juicefs:
|
||||
storageClass:
|
||||
name: juicefs-sc-dev
|
||||
|
||||
# Dev 默认值 —— 方便本地联调，生产部署前必须改
|
||||
credentials:
|
||||
postgresql:
|
||||
user: juicefs
|
||||
password: juicefs_password
|
||||
minio:
|
||||
rootUser: minioadmin
|
||||
rootPassword: minioadmin
|
||||
31
qiming-rcoder/k8s/helm/rcoder/values-offline.yaml
Normal file
31
qiming-rcoder/k8s/helm/rcoder/values-offline.yaml
Normal file
@@ -0,0 +1,31 @@
|
||||
# ============================================================================
|
||||
# 离线部署覆盖
|
||||
#
|
||||
# 使用场景：政企内网，无公网访问，所有镜像已 re-tag 推送到客户私有 registry。
|
||||
#
|
||||
# 如果走 "ctr image import 到每个节点" 模式，节点 containerd 里会以原始 repo:tag
|
||||
# 保存镜像，此时不需要改 global.imageRegistry —— 保留默认值即可，配合
|
||||
# imagePullPolicy: IfNotPresent 直接用本地缓存。
|
||||
#
|
||||
# 如果走 "re-tag + push 到私有 registry" 模式，把下面两个字段覆盖为实际地址：
|
||||
#
|
||||
# global:
|
||||
# imageRegistry: harbor.internal/rcoder # RCoder 自有镜像仓库
|
||||
# thirdPartyRegistry: harbor.internal/thirdparty # 第三方镜像 (postgres/minio/juicedata)
|
||||
#
|
||||
# 使用:
|
||||
# helm install rcoder k8s/helm/rcoder \
|
||||
# -f k8s/helm/rcoder/values-dev.yaml \
|
||||
# -f k8s/helm/rcoder/values-offline.yaml \
|
||||
# --set global.imageRegistry=harbor.internal/rcoder \
|
||||
# --set global.thirdPartyRegistry=harbor.internal/thirdparty
|
||||
# ============================================================================
|
||||
|
||||
global:
|
||||
# 示例值, 客户按需改
|
||||
imageRegistry: harbor.internal/rcoder
|
||||
thirdPartyRegistry: harbor.internal/thirdparty
|
||||
imagePullPolicy: IfNotPresent
|
||||
# 如果私有 registry 需要认证，在集群里创建 Secret 后填:
|
||||
# imagePullSecrets: [{name: harbor-pull-secret}]
|
||||
imagePullSecrets: []
|
||||
30
qiming-rcoder/k8s/helm/rcoder/values-prod.yaml
Normal file
30
qiming-rcoder/k8s/helm/rcoder/values-prod.yaml
Normal file
@@ -0,0 +1,30 @@
|
||||
# ============================================================================
|
||||
# Prod 环境覆盖
|
||||
#
|
||||
# ⚠️ 上线前必须替换 credentials.* 里的 CHANGE-ME-BEFORE-DEPLOY 占位符。
|
||||
# 推荐做法：不把真实密码写进这里，改用 --set 或 SealedSecret/ExternalSecret。
|
||||
#
|
||||
# 使用: helm install rcoder-prod k8s/helm/rcoder -f k8s/helm/rcoder/values-prod.yaml \
|
||||
# --namespace nuwax-rcoder-prod --create-namespace \
|
||||
# --set credentials.postgresql.password=<real> \
|
||||
# --set credentials.minio.rootPassword=<real>
|
||||
# ============================================================================
|
||||
|
||||
environment: prod
|
||||
|
||||
rcoder:
|
||||
replicas: 2
|
||||
service:
|
||||
nodePort: 30081
|
||||
|
||||
juicefs:
|
||||
storageClass:
|
||||
name: juicefs-sc-prod
|
||||
|
||||
credentials:
|
||||
postgresql:
|
||||
user: juicefs
|
||||
password: CHANGE-ME-BEFORE-DEPLOY
|
||||
minio:
|
||||
rootUser: CHANGE-ME-BEFORE-DEPLOY
|
||||
rootPassword: CHANGE-ME-BEFORE-DEPLOY
|
||||
176
qiming-rcoder/k8s/helm/rcoder/values.yaml
Normal file
176
qiming-rcoder/k8s/helm/rcoder/values.yaml
Normal file
@@ -0,0 +1,176 @@
|
||||
# ============================================================================
|
||||
# RCoder Helm values - 默认（共享）配置
|
||||
#
|
||||
# 使用方式：
|
||||
# helm install rcoder-dev k8s/helm/rcoder -f k8s/helm/rcoder/values-dev.yaml
|
||||
# helm install rcoder-prod k8s/helm/rcoder -f k8s/helm/rcoder/values-prod.yaml
|
||||
#
|
||||
# 离线部署叠加 values-offline.yaml 覆盖 global.imageRegistry。
|
||||
# ============================================================================
|
||||
|
||||
global:
|
||||
# 所有镜像的仓库前缀。离线部署时覆盖为 harbor.internal/rcoder 等私有地址。
|
||||
# 应用到: rcoder / rcoder-agent-runner / postgres / minio / mc / juicedata/mount
|
||||
# 说明: 为空时使用每个镜像的原始 repo；非空时会被拼到 repo 前面形成 "{registry}/{repo}"。
|
||||
imageRegistry: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev"
|
||||
imagePullPolicy: IfNotPresent
|
||||
imagePullSecrets: []
|
||||
# 第三方镜像的 registry 前缀。离线/私有场景用于把 postgres / minio/minio / minio/mc /
|
||||
# juicedata/mount 这些上游镜像指向客户私有仓库。为空时使用原始 repo。
|
||||
thirdPartyRegistry: ""
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# 应用部署
|
||||
# ----------------------------------------------------------------------------
|
||||
environment: dev # dev | prod —— 仅作为 label 和默认后缀
|
||||
|
||||
rcoder:
|
||||
image:
|
||||
repository: rcoder
|
||||
tag: "" # 空时继承 .Chart.appVersion
|
||||
replicas: 1
|
||||
resources:
|
||||
requests:
|
||||
memory: 512Mi
|
||||
cpu: 500m
|
||||
limits:
|
||||
memory: 2Gi
|
||||
cpu: 2000m
|
||||
service:
|
||||
type: NodePort
|
||||
port: 8087
|
||||
nodePort: 30080
|
||||
terminationGracePeriodSeconds: 60
|
||||
rustLog: info
|
||||
livenessProbe:
|
||||
httpGet: { path: /health, port: 8087 }
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
readinessProbe:
|
||||
httpGet: { path: /health, port: 8087 }
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 2
|
||||
# NetworkPolicy 是否启用 (cluster 不支持 NetworkPolicy 时置 false)
|
||||
networkPolicy:
|
||||
enabled: true
|
||||
# PodDisruptionBudget
|
||||
pdb:
|
||||
enabled: true
|
||||
maxUnavailable: 0
|
||||
# 是否创建共享的 ClusterRole (rcoder-pods-clusterrole)
|
||||
# - true (默认): chart 拥有该 ClusterRole
|
||||
# - false: 当集群里已有其他方式创建的同名 ClusterRole (如 Kustomize 并存) 时置 false
|
||||
clusterRole:
|
||||
create: true
|
||||
|
||||
# 动态 agent pod 使用的镜像 (通过 env var 注入，不会直接创建 pod)
|
||||
agentRunner:
|
||||
image:
|
||||
repository: rcoder-agent-runner
|
||||
tag: "" # 空时继承 .Chart.appVersion
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# 存储层
|
||||
# ----------------------------------------------------------------------------
|
||||
postgresql:
|
||||
image:
|
||||
repository: postgres
|
||||
tag: 16-alpine
|
||||
storage:
|
||||
size: 10Gi
|
||||
# 默认 local-path（与 Kustomize base 保持一致, k3s 自带）。
|
||||
# Longhorn / 离线集群可按需改为 longhorn / "" (空 => 集群默认)。
|
||||
storageClass: local-path
|
||||
service:
|
||||
port: 5432
|
||||
|
||||
minio:
|
||||
image:
|
||||
repository: minio/minio
|
||||
tag: latest
|
||||
initImage:
|
||||
repository: minio/mc
|
||||
tag: latest
|
||||
storage:
|
||||
size: 30Gi
|
||||
storageClass: local-path
|
||||
service:
|
||||
apiPort: 9000
|
||||
consolePort: 9001
|
||||
bucket: juicefs # 需与 juicefs.bucket 对应
|
||||
|
||||
juicefs:
|
||||
mountImage:
|
||||
repository: juicedata/mount
|
||||
tag: ce-v1.3.1
|
||||
# StorageClass 是集群级资源; Helm release 之间必须独立命名避免冲突
|
||||
storageClass:
|
||||
create: true
|
||||
# 留空则自动使用 "juicefs-sc-{release}"
|
||||
name: ""
|
||||
reclaimPolicy: Retain
|
||||
mountOptions: "cache-size=1024,allow-other"
|
||||
# juicefs-secret 里会使用到, 决定 mount 时的元数据连接 / 对象存储地址
|
||||
meta:
|
||||
appendFsync: "1"
|
||||
cacheSize: "1024"
|
||||
# bucket URL 指向 namespace 内的 minio-service (自动拼装, 无需修改)
|
||||
bucket: juicefs
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# 凭据 (Secret rcoder-credentials)
|
||||
# ----------------------------------------------------------------------------
|
||||
credentials:
|
||||
postgresql:
|
||||
user: juicefs
|
||||
password: CHANGE-ME
|
||||
minio:
|
||||
rootUser: minioadmin
|
||||
rootPassword: minioadmin
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# 工作空间 PVC (ReadWriteMany via JuiceFS)
|
||||
# ----------------------------------------------------------------------------
|
||||
workspace:
|
||||
size: 50Gi
|
||||
|
||||
# ----------------------------------------------------------------------------
|
||||
# RCoder 应用配置 (渲染为 ConfigMap 里的 config.yml)
|
||||
# ----------------------------------------------------------------------------
|
||||
config:
|
||||
defaultAgent: Claude
|
||||
projectsDir: /app/project_workspace
|
||||
port: 8087
|
||||
cleanup:
|
||||
enabled: true
|
||||
idleTimeoutSeconds: 600
|
||||
cleanupIntervalSeconds: 300
|
||||
dockerStopTimeoutSeconds: 30
|
||||
proxy:
|
||||
listenPort: 8088
|
||||
defaultBackendPort: 8087
|
||||
backendHost: 127.0.0.1
|
||||
portParam: port
|
||||
healthCheck:
|
||||
enabled: true
|
||||
intervalSeconds: 5
|
||||
timeoutSeconds: 1
|
||||
healthyThreshold: 2
|
||||
unhealthyThreshold: 3
|
||||
dockerConfig:
|
||||
networkBaseName: agent-network
|
||||
networkMode: bridge
|
||||
workDir: /app
|
||||
autoCleanup: true
|
||||
containerTtlSeconds: 3600
|
||||
cache:
|
||||
enabled: true
|
||||
ttlSeconds: 3600
|
||||
maxEntries: 50
|
||||
apiKeyAuth:
|
||||
enabled: false
|
||||
apiKey: YOUR_API_KEY_HERE
|
||||
282
qiming-rcoder/k8s/manifests/JUICEFS_DEPLOYMENT.md
Normal file
282
qiming-rcoder/k8s/manifests/JUICEFS_DEPLOYMENT.md
Normal file
@@ -0,0 +1,282 @@
|
||||
# JuiceFS + MinIO + PostgreSQL 部署指南
|
||||
|
||||
## 架构
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Kubernetes 集群 │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────────────┐ │
|
||||
│ │ JuiceFS CSI Driver │ │
|
||||
│ │ (StorageClass: juicefs-sc) │ │
|
||||
│ └──────────────────────────────────────────────────────┘ │
|
||||
│ ↓ │
|
||||
│ ┌────────────────┐ ┌────────────────┐ │
|
||||
│ │ rcoder 主服务 │ │ Agent Runner │ │
|
||||
│ │ Pod (Deployment)│ │ Pod (动态创建) │ │
|
||||
│ └────────┬───────┘ └────────┬───────┘ │
|
||||
│ │ │ │
|
||||
│ └──────────┬───────────┘ │
|
||||
│ ↓ │
|
||||
│ ┌──────────────┐ │
|
||||
│ │ JuiceFS 卷 │ ← 共享文件系统 (RWX) │
|
||||
│ └───────┬──────┘ │
|
||||
└──────────────────────┼────────────────────────────────────┘
|
||||
↓
|
||||
┌───────────────────────────────────────────────────────────────┐
|
||||
│ MinIO (S3) ←──── 数据存储 │
|
||||
│ PostgreSQL ← 元数据存储 │
|
||||
└───────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 文件结构
|
||||
|
||||
```
|
||||
manifests/
|
||||
├── 00-namespace.yaml # Namespace
|
||||
├── 01-storage/ # 存储层
|
||||
│ ├── postgresql-deployment.yaml
|
||||
│ ├── minio-deployment.yaml
|
||||
│ ├── minio-init-job.yaml
|
||||
│ ├── juicefs-secret.yaml
|
||||
│ ├── juicefs-storageclass.yaml
|
||||
│ └── juicefs-pvc.yaml
|
||||
├── 02-rcoder/ # 应用层
|
||||
│ ├── serviceaccount.yaml
|
||||
│ ├── rcoder-configmap.yaml
|
||||
│ ├── rcoder-deployment.yaml
|
||||
│ ├── rcoder-service.yaml
|
||||
│ ├── rcoder-networkpolicy.yaml
|
||||
│ └── rcoder-pdb.yaml
|
||||
└── _deprecated/nfs/ # 废弃的 NFS 配置
|
||||
```
|
||||
|
||||
## 部署顺序
|
||||
|
||||
### 第一步：部署 JuiceFS CSI Driver
|
||||
|
||||
```bash
|
||||
# 使用 Helm 部署
|
||||
helm repo add juicefs https://juicefs.github.io/charts
|
||||
helm repo update
|
||||
|
||||
helm install juicefs-csi-driver juicefs/juicefs-csi-driver \
|
||||
--namespace kube-system \
|
||||
--set webhook.enabled=false \
|
||||
--set csiSidecarImage.registry=docker.io \
|
||||
--set csiSidecarImage.repository=juicedata/csi-sidecar \
|
||||
--set csiSidecarImage.tag=v1.3.8 \
|
||||
--set image.registry=docker.io \
|
||||
--set image.repository=juicedata/juicefs-csi-driver \
|
||||
--set image.tag=v1.1.0 \
|
||||
--set nodeDaemonImage.registry=docker.io \
|
||||
--set nodeDaemonImage.repository=juicedata/juicefs-csi-driver \
|
||||
--set nodeDaemonImage.tag=v1.1.0
|
||||
```
|
||||
|
||||
验证：
|
||||
|
||||
```bash
|
||||
kubectl get pods -n kube-system | grep juicefs
|
||||
# 预期：juicefs-csi-driver-xxx 运行正常
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 第二步：部署 PostgreSQL
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/01-storage/postgresql-deployment.yaml
|
||||
```
|
||||
|
||||
验证：
|
||||
|
||||
```bash
|
||||
kubectl get pods -n nuwax-rcoder | grep postgresql
|
||||
kubectl logs -n nuwax-rcoder -l app=postgresql --tail=50
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 第三步：部署 MinIO
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/01-storage/minio-deployment.yaml
|
||||
```
|
||||
|
||||
验证：
|
||||
|
||||
```bash
|
||||
kubectl get pods -n nuwax-rcoder | grep minio
|
||||
# 访问 MinIO Console: http://<minio-ip>:9001
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 第四步：初始化 MinIO Bucket
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/01-storage/minio-init-job.yaml
|
||||
|
||||
# 等待完成
|
||||
kubectl wait --for=condition=complete job/minio-init -n nuwax-rcoder --timeout=120s
|
||||
|
||||
# 验证
|
||||
kubectl exec -it -n nuwax-rcoder deploy/minio -- mc ls myminio/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 第五步：创建 JuiceFS Secret
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/01-storage/juicefs-secret.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 第六步：创建 JuiceFS StorageClass
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/01-storage/juicefs-storageclass.yaml
|
||||
```
|
||||
|
||||
验证：
|
||||
|
||||
```bash
|
||||
kubectl get sc | grep juicefs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 第七步：部署 RCoder 相关资源
|
||||
|
||||
```bash
|
||||
# 部署 rcoder PVC (使用 JuiceFS)
|
||||
kubectl apply -f manifests/01-storage/juicefs-pvc.yaml
|
||||
|
||||
# 验证 PVC
|
||||
kubectl get pvc -n nuwax-rcoder
|
||||
# 状态应为 Bound
|
||||
|
||||
# 部署其他 rcoder 资源
|
||||
kubectl apply -f manifests/00-namespace.yaml
|
||||
kubectl apply -f manifests/02-rcoder/serviceaccount.yaml
|
||||
kubectl apply -f manifests/02-rcoder/rcoder-configmap.yaml
|
||||
kubectl apply -f manifests/02-rcoder/rcoder-deployment.yaml
|
||||
kubectl apply -f manifests/02-rcoder/rcoder-service.yaml
|
||||
kubectl apply -f manifests/02-rcoder/rcoder-networkpolicy.yaml
|
||||
kubectl apply -f manifests/02-rcoder/rcoder-pdb.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 第八步：验证整个系统
|
||||
|
||||
```bash
|
||||
# 检查所有 Pod
|
||||
kubectl get pods -n nuwax-rcoder
|
||||
|
||||
# 查看 JuiceFS 挂载情况
|
||||
kubectl exec -it -n nuwax-rcoder deploy/rcoder -- df -h | grep juicefs
|
||||
|
||||
# 测试写入
|
||||
kubectl exec -it -n nuwax-rcoder deploy/rcoder -- sh -c "echo 'test' > /app/project_workspace/test.txt"
|
||||
|
||||
# 查看日志
|
||||
kubectl logs -n nuwax-rcoder -l app=rcoder --tail=50
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 存储结构
|
||||
|
||||
部署后，数据存储结构如下：
|
||||
|
||||
```
|
||||
PostgreSQL (元数据):
|
||||
├── juicefs database
|
||||
│ ├── inodes
|
||||
│ ├── chunks
|
||||
│ ├── del-files
|
||||
│ └── sessions
|
||||
|
||||
MinIO (S3 数据):
|
||||
├── juicefs bucket
|
||||
│ └── juicefs/
|
||||
│ ├── juicefs.db ← JuiceFS 内部数据库
|
||||
│ └── chunk-xxx ← 数据块
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 卸载顺序
|
||||
|
||||
```bash
|
||||
# 1. 删除 rcoder 资源
|
||||
kubectl delete -f manifests/02-rcoder/rcoder-deployment.yaml
|
||||
kubectl delete -f manifests/02-rcoder/rcoder-configmap.yaml
|
||||
kubectl delete -f manifests/01-storage/juicefs-pvc.yaml
|
||||
|
||||
# 2. 删除 JuiceFS 相关
|
||||
kubectl delete -f manifests/01-storage/juicefs-storageclass.yaml
|
||||
kubectl delete -f manifests/01-storage/juicefs-secret.yaml
|
||||
|
||||
# 3. 删除存储层
|
||||
kubectl delete -f manifests/01-storage/minio-deployment.yaml
|
||||
kubectl delete -f manifests/01-storage/postgresql-deployment.yaml
|
||||
|
||||
# 4. 删除 JuiceFS CSI Driver (可选)
|
||||
helm uninstall juicefs-csi-driver -n kube-system
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 常见问题
|
||||
|
||||
### 1. PVC 一直 Pending
|
||||
|
||||
```bash
|
||||
# 检查 JuiceFS CSI Driver 状态
|
||||
kubectl get pods -n kube-system | grep juicefs
|
||||
|
||||
# 检查 StorageClass
|
||||
kubectl get sc juicefs-sc -o yaml
|
||||
|
||||
# 检查 Secret
|
||||
kubectl get secret juicefs-secret -n nuwax-rcoder -o yaml
|
||||
```
|
||||
|
||||
### 2. MinIO Bucket 创建失败
|
||||
|
||||
```bash
|
||||
# 确认 MinIO 运行正常
|
||||
kubectl logs -n nuwax-rcoder -l app=minio --tail=20
|
||||
|
||||
# 手动进入 Pod 测试
|
||||
kubectl exec -it -n nuwax-rcoder deploy/minio -- sh
|
||||
mc alias set myminio http://localhost:9000 minioadmin minioadmin
|
||||
mc mb myminio/juicefs
|
||||
```
|
||||
|
||||
### 3. JuiceFS 挂载失败
|
||||
|
||||
```bash
|
||||
# 查看 Node Pod 日志
|
||||
kubectl get pods -n kube-system -o wide | grep juicefs
|
||||
|
||||
# 查看挂载详情
|
||||
kubectl describe pod -n kube-system -l app=juicefs-csi-driver-node
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 资源需求
|
||||
|
||||
| 组件 | CPU | Memory | Storage |
|
||||
|------|-----|--------|---------|
|
||||
| PostgreSQL | 500m | 1Gi | 20Gi |
|
||||
| MinIO | 500m | 1Gi | 100Gi+ |
|
||||
| JuiceFS CSI | 100m | 128Mi | - |
|
||||
|
||||
总计：约 2 核 + 2.5Gi + 120Gi 存储
|
||||
106
qiming-rcoder/k8s/manifests/_deprecated/nfs/nfs-server.yaml
Normal file
106
qiming-rcoder/k8s/manifests/_deprecated/nfs/nfs-server.yaml
Normal file
@@ -0,0 +1,106 @@
|
||||
# ============================================================
|
||||
# NFS Server (内建模式)
|
||||
# 集群内自建 NFS Server，数据持久化到 local-path PV
|
||||
#
|
||||
# ⚠️ 注意: OrbStack/Mac 桌面 K8s 环境不支持此方式，请使用外部 NFS Server
|
||||
#
|
||||
# 如使用外部 NFS Server，跳过此文件，直接部署 nfs-subdir-provisioner.yaml
|
||||
# ============================================================
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: nfs-storage
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: nfs-server
|
||||
namespace: nuwax-nfs-storage
|
||||
spec:
|
||||
clusterIP: None
|
||||
selector:
|
||||
app: nfs-server
|
||||
ports:
|
||||
- name: nfs
|
||||
port: 2049
|
||||
targetPort: 2049
|
||||
- name: mountd
|
||||
port: 20048
|
||||
targetPort: 20048
|
||||
- name: rpcbind
|
||||
port: 111
|
||||
targetPort: 111
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: nfs-server
|
||||
namespace: nuwax-nfs-storage
|
||||
spec:
|
||||
serviceName: nfs-server
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: nfs-server
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: nfs-server
|
||||
spec:
|
||||
# 亲和性: NFS Server 调度到有 local-path 存储的节点
|
||||
affinity:
|
||||
nodeAffinity:
|
||||
requiredDuringSchedulingIgnoredDuringExecution:
|
||||
nodeSelectorTerms:
|
||||
- matchExpressions:
|
||||
- key: kubernetes.io/hostname
|
||||
operator: Exists
|
||||
containers:
|
||||
- name: nfs-server
|
||||
image: itsthenetwork/nfs-server-alpine:latest
|
||||
env:
|
||||
- name: SHARED_DIRECTORY
|
||||
value: "/exports"
|
||||
- name: SYNC
|
||||
value: "true"
|
||||
ports:
|
||||
- name: nfs
|
||||
containerPort: 2049
|
||||
protocol: TCP
|
||||
- name: mountd
|
||||
containerPort: 20048
|
||||
protocol: TCP
|
||||
- name: rpcbind
|
||||
containerPort: 111
|
||||
protocol: TCP
|
||||
securityContext:
|
||||
runAsUser: 0
|
||||
resources:
|
||||
requests:
|
||||
cpu: "100m"
|
||||
memory: "128Mi"
|
||||
limits:
|
||||
cpu: "500m"
|
||||
memory: "512Mi"
|
||||
volumeMounts:
|
||||
- name: nfs-data
|
||||
mountPath: /exports
|
||||
volumes:
|
||||
- name: nfs-data
|
||||
persistentVolumeClaim:
|
||||
claimName: nfs-server-pvc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: nfs-server-pvc
|
||||
namespace: nuwax-nfs-storage
|
||||
labels:
|
||||
app: nfs-server
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 100Gi
|
||||
@@ -0,0 +1,135 @@
|
||||
# ============================================================
|
||||
# NFS Subdir External Provisioner + StorageClass
|
||||
#
|
||||
# 两种使用方式:
|
||||
# 模式一(内建 NFS): nfs-server.yaml 部署后，使用此文件
|
||||
# 模式二(外部 NFS): 直接修改 NFS_SERVER/NFS_PATH 环境变量指向外部 NFS
|
||||
#
|
||||
# 参考: https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner
|
||||
# ============================================================
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: nfs-storage
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: nfs-client-provisioner
|
||||
namespace: nuwax-nfs-storage
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: nfs-client-provisioner-cluster-runner
|
||||
rules:
|
||||
- apiGroups: [""]
|
||||
resources: ["nodes"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["persistentvolumes"]
|
||||
verbs: ["get", "list", "watch", "create", "delete"]
|
||||
- apiGroups: [""]
|
||||
resources: ["persistentvolumeclaims"]
|
||||
verbs: ["get", "list", "watch", "update"]
|
||||
- apiGroups: ["storage.k8s.io"]
|
||||
resources: ["storageclasses"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["events"]
|
||||
verbs: ["create", "update", "patch"]
|
||||
- apiGroups: ["storage.k8s.io"]
|
||||
resources: ["csinodeinfos"]
|
||||
verbs: ["get", "list", "watch"]
|
||||
- apiGroups: ["coordination.k8s.io"]
|
||||
resources: ["leases"]
|
||||
verbs: ["get", "create", "update"]
|
||||
---
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: nfs-client-provisioner-cluster-runner-binding
|
||||
roleRef:
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
kind: ClusterRole
|
||||
name: nfs-client-provisioner-cluster-runner
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: nfs-client-provisioner
|
||||
namespace: nuwax-nfs-storage
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: nfs-client-provisioner
|
||||
namespace: nuwax-nfs-storage
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: nfs-client-provisioner
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: nfs-client-provisioner
|
||||
spec:
|
||||
serviceAccountName: nfs-client-provisioner
|
||||
containers:
|
||||
- name: nfs-client-provisioner
|
||||
image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
|
||||
volumeMounts:
|
||||
- name: nfs-provisioner
|
||||
mountPath: /persistentvolumes
|
||||
env:
|
||||
- name: PROVISIONER_NAME
|
||||
value: rcoder-nfs
|
||||
# ⚠️ 修改此处指向外部 NFS Server
|
||||
# 内建模式: nfs-server.nfs-storage.svc.cluster.local (nfs-server.yaml 部署后)
|
||||
# 外部模式: 替换为客户提供的 NFS 地址，如 192.168.1.100
|
||||
- name: NFS_SERVER
|
||||
value: nfs-server.nfs-storage.svc.cluster.local
|
||||
- name: NFS_PATH
|
||||
value: /exports
|
||||
- name: NFS_MOUNT_OPTIONS
|
||||
value: "vers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2"
|
||||
- name: ON_DELETE_POLICY
|
||||
value: "delete"
|
||||
- name: ARCHIVE_ON_DELETE
|
||||
value: "false"
|
||||
resources:
|
||||
requests:
|
||||
cpu: "50m"
|
||||
memory: "64Mi"
|
||||
limits:
|
||||
cpu: "200m"
|
||||
memory: "256Mi"
|
||||
volumes:
|
||||
- name: nfs-provisioner
|
||||
nfs:
|
||||
# ⚠️ 修改此处指向外部 NFS Server
|
||||
server: nfs-server.nfs-storage.svc.cluster.local
|
||||
path: /exports
|
||||
---
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: rcoder-nfs
|
||||
annotations:
|
||||
storageclass.kubernetes.io/is-default-class: "false"
|
||||
provisioner: rcoder-nfs
|
||||
parameters:
|
||||
pathPattern: "${.PVC.namespace}/${.PVC.name}"
|
||||
onDelete: "delete"
|
||||
archiveOnDelete: "false"
|
||||
mountOptions:
|
||||
- vers=4.1
|
||||
- rsize=1048576
|
||||
- wsize=1048576
|
||||
- hard
|
||||
- timeo=600
|
||||
- retrans=2
|
||||
- noresvport
|
||||
reclaimPolicy: Delete
|
||||
volumeBindingMode: WaitForFirstConsumer
|
||||
allowVolumeExpansion: true
|
||||
10
qiming-rcoder/k8s/manifests/base/kustomization.yaml
Normal file
10
qiming-rcoder/k8s/manifests/base/kustomization.yaml
Normal file
@@ -0,0 +1,10 @@
|
||||
# Kustomize base - 基础配置
|
||||
# 使用方式: kubectl apply -k manifests/base
|
||||
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- storage
|
||||
- rcoder
|
||||
5
qiming-rcoder/k8s/manifests/base/namespace.yaml
Normal file
5
qiming-rcoder/k8s/manifests/base/namespace.yaml
Normal file
@@ -0,0 +1,5 @@
|
||||
# Namespace for RCoder
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: nuwax-rcoder
|
||||
15
qiming-rcoder/k8s/manifests/base/rcoder/kustomization.yaml
Normal file
15
qiming-rcoder/k8s/manifests/base/rcoder/kustomization.yaml
Normal file
@@ -0,0 +1,15 @@
|
||||
# RCoder application layer - 应用层配置
|
||||
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
# 注意:
|
||||
# - rcoder-configmap.yaml 由各环境 overlay 提供（storage_class 需随环境变化）
|
||||
# - ClusterRoleBinding 由各环境 overlay 提供（避免集群级命名冲突）
|
||||
|
||||
resources:
|
||||
- serviceaccount.yaml
|
||||
- rcoder-deployment.yaml
|
||||
- rcoder-service.yaml
|
||||
- rcoder-networkpolicy.yaml
|
||||
- rcoder-pdb.yaml
|
||||
@@ -0,0 +1,85 @@
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: rcoder
|
||||
namespace: nuwax-rcoder
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: rcoder
|
||||
component: rcoder-main
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: rcoder
|
||||
component: rcoder-main
|
||||
spec:
|
||||
# Graceful shutdown: allow 60s for containers to clean up
|
||||
terminationGracePeriodSeconds: 60
|
||||
serviceAccountName: rcoder-pods-sa
|
||||
containers:
|
||||
- name: rcoder
|
||||
image: nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder:latest
|
||||
# IfNotPresent: 节点已缓存就直接用，避免每次启动都去 registry 验 token/manifest
|
||||
# 缺点: latest tag 内容更新后需 kubectl rollout restart 才会拉新 image
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["/app/bin/rcoder"]
|
||||
args: ["--port", "8087"]
|
||||
ports:
|
||||
- containerPort: 8087
|
||||
env:
|
||||
- name: CONTAINER_RUNTIME
|
||||
value: "kubernetes"
|
||||
- name: RCODER_K8S_NAMESPACE
|
||||
value: "nuwax-rcoder"
|
||||
- name: RUST_LOG
|
||||
value: "info"
|
||||
# JuiceFS 存储配置 (StorageClass 名称由各环境 overlay 覆盖)
|
||||
- name: RCODER_K8S_STORAGE_CLASS
|
||||
value: "juicefs-sc-placeholder"
|
||||
- name: RCODER_DOCKER_IMAGE
|
||||
value: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder:latest"
|
||||
- name: RCODER_DOCKER_IMAGE_COMPUTER
|
||||
value: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder-agent-runner:latest"
|
||||
resources:
|
||||
requests:
|
||||
memory: "512Mi"
|
||||
cpu: "500m"
|
||||
limits:
|
||||
memory: "2Gi"
|
||||
cpu: "2000m"
|
||||
# Health probes for K8s liveness/readiness detection
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8087
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health
|
||||
port: 8087
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 2
|
||||
volumeMounts:
|
||||
- name: rcoder-config
|
||||
mountPath: /app/config.yml
|
||||
subPath: config.yml
|
||||
- name: project-workspace
|
||||
mountPath: /app/project_workspace
|
||||
subPath: project_workspace
|
||||
- name: project-workspace
|
||||
mountPath: /app/computer-project-workspace
|
||||
subPath: computer_workspace
|
||||
volumes:
|
||||
- name: rcoder-config
|
||||
configMap:
|
||||
name: rcoder-config
|
||||
- name: project-workspace
|
||||
persistentVolumeClaim:
|
||||
claimName: rcoder-workspace
|
||||
@@ -0,0 +1,70 @@
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: rcoder-network-policy
|
||||
namespace: nuwax-rcoder
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app: rcoder
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress:
|
||||
# 允许来自同 namespace 的流量
|
||||
- from:
|
||||
- podSelector: {}
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 8087
|
||||
- protocol: TCP
|
||||
port: 8088
|
||||
egress:
|
||||
# DNS 解析
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
# K8s API Server
|
||||
- to:
|
||||
- namespaceSelector: {}
|
||||
podSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/service-name: kubernetes
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 443
|
||||
# PostgreSQL 访问
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: postgresql
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 5432
|
||||
# MinIO 访问
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
app: minio
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 9000
|
||||
- protocol: TCP
|
||||
port: 9001
|
||||
# Agent Pod gRPC 访问 (rcoder -> agent 通信)
|
||||
# 用 managed-by: rcoder-runtime 标签同时匹配两类动态创建的 pod：
|
||||
# - service_type=rcoder (legacy /chat 创建的 rcoder-agent pod)
|
||||
# - service_type=computer-agent-runner (/computer/chat 创建的 computer-agent-runner pod)
|
||||
- to:
|
||||
- podSelector:
|
||||
matchLabels:
|
||||
managed-by: rcoder-runtime
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 50051
|
||||
- protocol: TCP
|
||||
port: 8086
|
||||
10
qiming-rcoder/k8s/manifests/base/rcoder/rcoder-pdb.yaml
Normal file
10
qiming-rcoder/k8s/manifests/base/rcoder/rcoder-pdb.yaml
Normal file
@@ -0,0 +1,10 @@
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: rcoder-pdb
|
||||
namespace: nuwax-rcoder
|
||||
spec:
|
||||
maxUnavailable: 0
|
||||
selector:
|
||||
matchLabels:
|
||||
app: rcoder
|
||||
16
qiming-rcoder/k8s/manifests/base/rcoder/rcoder-service.yaml
Normal file
16
qiming-rcoder/k8s/manifests/base/rcoder/rcoder-service.yaml
Normal file
@@ -0,0 +1,16 @@
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: rcoder
|
||||
namespace: nuwax-rcoder
|
||||
spec:
|
||||
type: NodePort
|
||||
# 用 component=rcoder-main 把动态创建的 rcoder-agent / computer-agent-runner pod
|
||||
# （它们也带 app=rcoder 标签）排除在外，避免 NodePort 把流量路由到不监听 8087 的 agent pod
|
||||
selector:
|
||||
app: rcoder
|
||||
component: rcoder-main
|
||||
ports:
|
||||
# nodePort 由各环境 overlay 指定（避免同集群下 dev/prod 冲突）
|
||||
- port: 8087
|
||||
targetPort: 8087
|
||||
35
qiming-rcoder/k8s/manifests/base/rcoder/serviceaccount.yaml
Normal file
35
qiming-rcoder/k8s/manifests/base/rcoder/serviceaccount.yaml
Normal file
@@ -0,0 +1,35 @@
|
||||
# ServiceAccount + ClusterRole for RCoder to manage Pods
|
||||
# 注意: ClusterRoleBinding 由各环境 overlay 提供（避免集群级命名冲突）
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ServiceAccount
|
||||
metadata:
|
||||
name: rcoder-pods-sa
|
||||
namespace: nuwax-rcoder
|
||||
---
|
||||
# ClusterRole for Pod management（集群共享，dev/prod 可复用同一个 ClusterRole）
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: rcoder-pods-clusterrole
|
||||
rules:
|
||||
# Pod 管理
|
||||
- apiGroups: [""]
|
||||
resources: ["pods"]
|
||||
verbs: ["create", "delete", "get", "list", "watch", "patch", "update"]
|
||||
# Pod 日志
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/log"]
|
||||
verbs: ["get", "list"]
|
||||
# Pod 执行命令（用于调试）
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/exec"]
|
||||
verbs: ["create"]
|
||||
# Pod 状态
|
||||
- apiGroups: [""]
|
||||
resources: ["pods/status"]
|
||||
verbs: ["get"]
|
||||
# PVC 管理（动态创建/删除工作空间存储）
|
||||
- apiGroups: [""]
|
||||
resources: ["persistentvolumeclaims"]
|
||||
verbs: ["get", "list", "watch", "create", "delete"]
|
||||
19
qiming-rcoder/k8s/manifests/base/storage/juicefs-pvc.yaml
Normal file
19
qiming-rcoder/k8s/manifests/base/storage/juicefs-pvc.yaml
Normal file
@@ -0,0 +1,19 @@
|
||||
# ============================================================
|
||||
# RCoder JuiceFS PVC
|
||||
# 替代原来的 NFS PVC
|
||||
# ============================================================
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: rcoder-workspace
|
||||
namespace: nuwax-rcoder
|
||||
labels:
|
||||
app: rcoder
|
||||
spec:
|
||||
# ReadWriteMany: 支持 K8s 多节点同时读写 (JuiceFS 特性)
|
||||
accessModes: ["ReadWriteMany"]
|
||||
# storageClassName 由各环境 overlay 覆盖（dev: juicefs-sc-dev, prod: juicefs-sc-prod）
|
||||
storageClassName: juicefs-sc-placeholder
|
||||
resources:
|
||||
requests:
|
||||
storage: 50Gi
|
||||
15
qiming-rcoder/k8s/manifests/base/storage/kustomization.yaml
Normal file
15
qiming-rcoder/k8s/manifests/base/storage/kustomization.yaml
Normal file
@@ -0,0 +1,15 @@
|
||||
# Storage layer - 存储层配置
|
||||
# 注意:
|
||||
# - juicefs-secret 由各环境 overlay 提供 (dev/prod)
|
||||
# - juicefs-storageclass 由各环境 overlay 提供（集群级资源需按环境独立命名，避免冲突）
|
||||
# - 不再使用 juicefs-init Job：JuiceFS CSI driver 首次挂载时会用 Secret 里的
|
||||
# name/metaurl/storage/bucket/access-key/secret-key 自动执行 format，Job 是冗余的
|
||||
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
resources:
|
||||
- postgresql-deployment.yaml
|
||||
- minio-deployment.yaml
|
||||
- minio-init-job.yaml
|
||||
- juicefs-pvc.yaml
|
||||
112
qiming-rcoder/k8s/manifests/base/storage/minio-deployment.yaml
Normal file
112
qiming-rcoder/k8s/manifests/base/storage/minio-deployment.yaml
Normal file
@@ -0,0 +1,112 @@
|
||||
# ============================================================
|
||||
# MinIO for JuiceFS Data Storage
|
||||
# JuiceFS S3 兼容对象存储
|
||||
# ============================================================
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: minio-pvc
|
||||
namespace: nuwax-rcoder
|
||||
labels:
|
||||
app: minio
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
storageClassName: "local-path" # 单节点环境使用 local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 30Gi # 单节点环境使用较小存储
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: minio
|
||||
namespace: nuwax-rcoder
|
||||
spec:
|
||||
serviceName: minio
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: minio
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: minio
|
||||
spec:
|
||||
containers:
|
||||
- name: minio
|
||||
image: minio/minio:latest
|
||||
ports:
|
||||
- containerPort: 9000
|
||||
name: api
|
||||
- containerPort: 9001
|
||||
name: console
|
||||
env:
|
||||
- name: MINIO_ROOT_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: rcoder-credentials
|
||||
key: MINIO_ROOT_USER
|
||||
- name: MINIO_ROOT_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: rcoder-credentials
|
||||
key: MINIO_ROOT_PASSWORD
|
||||
args:
|
||||
- server
|
||||
- /data
|
||||
- --console-address
|
||||
- ":9001"
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /minio/health/live
|
||||
port: 9000
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /minio/health/ready
|
||||
port: 9000
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
volumeMounts:
|
||||
- name: minio-data
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: minio-data
|
||||
persistentVolumeClaim:
|
||||
claimName: minio-pvc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: minio
|
||||
namespace: nuwax-rcoder
|
||||
spec:
|
||||
selector:
|
||||
app: minio
|
||||
ports:
|
||||
- name: api
|
||||
port: 9000
|
||||
targetPort: 9000
|
||||
- name: console
|
||||
port: 9001
|
||||
targetPort: 9001
|
||||
clusterIP: None # Headless service for StatefulSet
|
||||
---
|
||||
# Service for K8s internal access (ClusterIP)
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: minio-service
|
||||
namespace: nuwax-rcoder
|
||||
spec:
|
||||
selector:
|
||||
app: minio
|
||||
ports:
|
||||
- name: api
|
||||
port: 9000
|
||||
targetPort: 9000
|
||||
- name: console
|
||||
port: 9001
|
||||
targetPort: 9001
|
||||
47
qiming-rcoder/k8s/manifests/base/storage/minio-init-job.yaml
Normal file
47
qiming-rcoder/k8s/manifests/base/storage/minio-init-job.yaml
Normal file
@@ -0,0 +1,47 @@
|
||||
# ============================================================
|
||||
# MinIO Bucket 初始化 Job
|
||||
# 创建 JuiceFS 所需的 bucket
|
||||
# ============================================================
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: minio-init
|
||||
namespace: nuwax-rcoder
|
||||
spec:
|
||||
ttlSecondsAfterFinished: 300 # Job 完成后 5 分钟自动清理
|
||||
template:
|
||||
spec:
|
||||
restartPolicy: OnFailure
|
||||
containers:
|
||||
- name: mc
|
||||
image: minio/mc:latest
|
||||
command:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
# 等待 MinIO 就绪
|
||||
echo "Waiting for MinIO to be ready..."
|
||||
until mc alias set myminio http://minio:9000 ${MINIO_ROOT_USER} ${MINIO_ROOT_PASSWORD}; do
|
||||
echo "MinIO not ready, retrying..."
|
||||
sleep 2
|
||||
done
|
||||
|
||||
echo "MinIO is ready, creating bucket..."
|
||||
# 创建 JuiceFS 专用 bucket
|
||||
mc mb myminio/juicefs --ignore-existing
|
||||
|
||||
# 验证
|
||||
mc ls myminio/
|
||||
|
||||
echo "MinIO initialization completed!"
|
||||
env:
|
||||
- name: MINIO_ROOT_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: rcoder-credentials
|
||||
key: MINIO_ROOT_USER
|
||||
- name: MINIO_ROOT_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: rcoder-credentials
|
||||
key: MINIO_ROOT_PASSWORD
|
||||
@@ -0,0 +1,97 @@
|
||||
# ============================================================
|
||||
# PostgreSQL for JuiceFS Metadata
|
||||
# JuiceFS 元数据存储
|
||||
# ============================================================
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: postgresql-pvc
|
||||
namespace: nuwax-rcoder
|
||||
labels:
|
||||
app: postgresql
|
||||
spec:
|
||||
accessModes: ["ReadWriteOnce"]
|
||||
storageClassName: "local-path" # 单节点环境使用 local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 10Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
metadata:
|
||||
name: postgresql
|
||||
namespace: nuwax-rcoder
|
||||
spec:
|
||||
serviceName: postgresql
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: postgresql
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: postgresql
|
||||
spec:
|
||||
initContainers:
|
||||
- name: cleanup
|
||||
image: postgres:16-alpine
|
||||
command: ["rm", "-rf", "/var/lib/postgresql/data/lost+found"]
|
||||
volumeMounts:
|
||||
- name: postgres-data
|
||||
mountPath: /var/lib/postgresql/data
|
||||
containers:
|
||||
- name: postgresql
|
||||
image: postgres:16-alpine
|
||||
ports:
|
||||
- containerPort: 5432
|
||||
name: postgres
|
||||
env:
|
||||
- name: POSTGRES_DB
|
||||
value: "juicefs"
|
||||
- name: POSTGRES_USER
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: rcoder-credentials
|
||||
key: POSTGRES_USER
|
||||
- name: POSTGRES_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: rcoder-credentials
|
||||
key: POSTGRES_PASSWORD
|
||||
livenessProbe:
|
||||
exec:
|
||||
command:
|
||||
- pg_isready
|
||||
- -U
|
||||
- juicefs
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
exec:
|
||||
command:
|
||||
- pg_isready
|
||||
- -U
|
||||
- juicefs
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 5
|
||||
volumeMounts:
|
||||
- name: postgres-data
|
||||
mountPath: /var/lib/postgresql/data
|
||||
volumes:
|
||||
- name: postgres-data
|
||||
persistentVolumeClaim:
|
||||
claimName: postgresql-pvc
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: postgresql
|
||||
namespace: nuwax-rcoder
|
||||
spec:
|
||||
selector:
|
||||
app: postgresql
|
||||
ports:
|
||||
- name: postgres
|
||||
port: 5432
|
||||
targetPort: 5432
|
||||
@@ -0,0 +1,17 @@
|
||||
# ============================================================
|
||||
# ClusterRoleBinding - Dev 环境专用
|
||||
# 把 base 里共享的 ClusterRole 绑定到 dev namespace 的 ServiceAccount
|
||||
# 集群级资源独立命名，避免与 prod 冲突
|
||||
# ============================================================
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: rcoder-pods-crb-dev
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: rcoder-pods-sa
|
||||
namespace: nuwax-rcoder-dev
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: rcoder-pods-clusterrole
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
16
qiming-rcoder/k8s/manifests/overlays/dev/credentials.yaml
Normal file
16
qiming-rcoder/k8s/manifests/overlays/dev/credentials.yaml
Normal file
@@ -0,0 +1,16 @@
|
||||
# ============================================================
|
||||
# 存储层凭据 Secret - Dev 环境
|
||||
# 维护 PostgreSQL / MinIO 账号密码
|
||||
# 注意：同步维护 juicefs-secret.yaml 里的 metaurl / MINIO_ACCESS_KEY 等字段
|
||||
# ============================================================
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: rcoder-credentials
|
||||
namespace: nuwax-rcoder-dev
|
||||
type: Opaque
|
||||
stringData:
|
||||
POSTGRES_USER: "juicefs"
|
||||
POSTGRES_PASSWORD: "juicefs_password"
|
||||
MINIO_ROOT_USER: "minioadmin"
|
||||
MINIO_ROOT_PASSWORD: "minioadmin"
|
||||
28
qiming-rcoder/k8s/manifests/overlays/dev/juicefs-secret.yaml
Normal file
28
qiming-rcoder/k8s/manifests/overlays/dev/juicefs-secret.yaml
Normal file
@@ -0,0 +1,28 @@
|
||||
# ============================================================
|
||||
# JuiceFS Secret - Dev 环境专用
|
||||
# 覆盖 base 中的 juicefs-secret
|
||||
# ============================================================
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: juicefs-secret
|
||||
namespace: nuwax-rcoder-dev
|
||||
type: Opaque
|
||||
stringData:
|
||||
# JuiceFS CE 必需字段（小写键名）
|
||||
name: "rcoder-juicefs"
|
||||
metaurl: "postgres://juicefs:juicefs_password@postgresql.nuwax-rcoder-dev:5432/juicefs"
|
||||
storage: "minio"
|
||||
bucket: "http://minio-service.nuwax-rcoder-dev:9000/juicefs"
|
||||
access-key: "minioadmin"
|
||||
secret-key: "minioadmin"
|
||||
|
||||
# 兼容字段（保留给现有脚本/日志使用）
|
||||
METAURL: "postgres://juicefs:juicefs_password@postgresql.nuwax-rcoder-dev:5432/juicefs"
|
||||
MINIO_ACCESS_KEY: "minioadmin"
|
||||
MINIO_SECRET_KEY: "minioadmin"
|
||||
MINIO_ADDRESS: "minio-service.nuwax-rcoder-dev:9000"
|
||||
|
||||
# 其他配置
|
||||
APPEND_FSYNC: "1"
|
||||
CACHE_SIZE: "1024"
|
||||
@@ -0,0 +1,24 @@
|
||||
# ============================================================
|
||||
# JuiceFS StorageClass - Dev 环境专用
|
||||
# 集群级资源独立命名，避免与 prod 冲突
|
||||
# ============================================================
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: juicefs-sc-dev
|
||||
annotations:
|
||||
storageclass.kubernetes.io/is-default-class: "false"
|
||||
provisioner: csi.juicefs.com
|
||||
parameters:
|
||||
csi.storage.k8s.io/provisioner-secret-name: juicefs-secret
|
||||
csi.storage.k8s.io/provisioner-secret-namespace: nuwax-rcoder-dev
|
||||
csi.storage.k8s.io/controller-expand-secret-name: juicefs-secret
|
||||
csi.storage.k8s.io/controller-expand-secret-namespace: nuwax-rcoder-dev
|
||||
csi.storage.k8s.io/node-publish-secret-name: juicefs-secret
|
||||
csi.storage.k8s.io/node-publish-secret-namespace: nuwax-rcoder-dev
|
||||
juicefs/controllerQuotaSet: "false"
|
||||
juicefs/mountImage: "juicedata/mount:ce-v1.3.1"
|
||||
juicefs/mountOptions: "cache-size=1024,allow-other"
|
||||
reclaimPolicy: Retain
|
||||
allowVolumeExpansion: true
|
||||
volumeBindingMode: Immediate
|
||||
90
qiming-rcoder/k8s/manifests/overlays/dev/kustomization.yaml
Normal file
90
qiming-rcoder/k8s/manifests/overlays/dev/kustomization.yaml
Normal file
@@ -0,0 +1,90 @@
|
||||
# Dev environment overlay
|
||||
# 使用方式: kubectl apply -k manifests/overlays/dev
|
||||
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: nuwax-rcoder-dev
|
||||
|
||||
resources:
|
||||
- ../../base
|
||||
# dev 独立的集群级资源（避免与 prod 冲突）
|
||||
- juicefs-storageclass.yaml
|
||||
- clusterrolebinding.yaml
|
||||
# dev 独立的 namespace 内资源
|
||||
- rcoder-configmap.yaml
|
||||
- credentials.yaml
|
||||
- juicefs-secret.yaml
|
||||
|
||||
# 开发环境标签
|
||||
# - includeSelectors 必须保持 false：否则 environment label 会被注入到
|
||||
# NetworkPolicy 的 egress podSelector，误伤 K8s API Server 和动态创建的 Agent Runner Pod。
|
||||
# - includeTemplates: true 让 Pod 模板也带上 environment label，方便 kubectl 按环境筛选。
|
||||
labels:
|
||||
- pairs:
|
||||
environment: dev
|
||||
includeSelectors: false
|
||||
includeTemplates: true
|
||||
|
||||
# 开发环境副本数
|
||||
replicas:
|
||||
- name: rcoder
|
||||
count: 1
|
||||
|
||||
# 开发环境镜像 tag
|
||||
images:
|
||||
- name: nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder
|
||||
newName: nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/nuwax-test/rcoder-k8s
|
||||
newTag: latest
|
||||
- name: nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder-agent-runner
|
||||
newName: nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/nuwax-test/rcoder-computer-agent-runner
|
||||
newTag: latest
|
||||
|
||||
patches:
|
||||
# NodePort: 30080（dev 专用）
|
||||
- patch: |-
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: rcoder
|
||||
spec:
|
||||
ports:
|
||||
- port: 8087
|
||||
targetPort: 8087
|
||||
nodePort: 30080
|
||||
target:
|
||||
kind: Service
|
||||
name: rcoder
|
||||
|
||||
# PVC 指向 dev 专属 StorageClass
|
||||
- patch: |-
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: rcoder-workspace
|
||||
spec:
|
||||
storageClassName: juicefs-sc-dev
|
||||
target:
|
||||
kind: PersistentVolumeClaim
|
||||
name: rcoder-workspace
|
||||
|
||||
# rcoder deployment 指向 dev 环境的 namespace 和 StorageClass
|
||||
- patch: |-
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: rcoder
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: rcoder
|
||||
env:
|
||||
- name: RCODER_K8S_NAMESPACE
|
||||
value: nuwax-rcoder-dev
|
||||
- name: RCODER_K8S_STORAGE_CLASS
|
||||
value: juicefs-sc-dev
|
||||
target:
|
||||
kind: Deployment
|
||||
name: rcoder
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# ============================================================
|
||||
# RCoder ConfigMap - Dev 环境专用
|
||||
# storage_class 指向 juicefs-sc-dev
|
||||
# ============================================================
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: rcoder-config
|
||||
data:
|
||||
config.yml: |
|
||||
# RCoder 配置文件
|
||||
default_agent: "Claude"
|
||||
|
||||
projects_dir: "/app/project_workspace"
|
||||
port: 8087
|
||||
|
||||
cleanup_config:
|
||||
enabled: true
|
||||
idle_timeout_seconds: 600
|
||||
cleanup_interval_seconds: 300
|
||||
docker_stop_timeout_seconds: 30
|
||||
|
||||
proxy_config:
|
||||
listen_port: 8088
|
||||
default_backend_port: 8087
|
||||
backend_host: "127.0.0.1"
|
||||
port_param: "port"
|
||||
health_check:
|
||||
enabled: true
|
||||
interval_seconds: 5
|
||||
timeout_seconds: 1
|
||||
healthy_threshold: 2
|
||||
unhealthy_threshold: 3
|
||||
|
||||
docker_config:
|
||||
multi_image_config:
|
||||
global_defaults:
|
||||
registry_prefix: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev"
|
||||
services:
|
||||
rcoder:
|
||||
service_type: "RCoder"
|
||||
arm64_image: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder:latest"
|
||||
amd64_image: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder:latest"
|
||||
default_image: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder:latest"
|
||||
image_tag_prefix: "rcoder"
|
||||
enabled: true
|
||||
computer-agent-runner:
|
||||
service_type: "ComputerAgentRunner"
|
||||
arm64_image: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder-agent-runner:latest"
|
||||
amd64_image: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder-agent-runner:latest"
|
||||
default_image: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder-agent-runner:latest"
|
||||
image_tag_prefix: "rcoder-computer-agent-runner"
|
||||
enabled: true
|
||||
selection_strategy: "ServiceOnly"
|
||||
cache_config:
|
||||
enabled: true
|
||||
ttl_seconds: 3600
|
||||
max_entries: 50
|
||||
|
||||
network_base_name: "agent-network"
|
||||
network_mode: "bridge"
|
||||
work_dir: "/app"
|
||||
auto_cleanup: true
|
||||
container_ttl_seconds: 3600
|
||||
|
||||
api_key_auth:
|
||||
enabled: false
|
||||
api_key: "YOUR_API_KEY_HERE"
|
||||
|
||||
kubernetes_config:
|
||||
storage_class: "juicefs-sc-dev"
|
||||
@@ -0,0 +1,14 @@
|
||||
# ClusterRoleBinding - 本地测试环境
|
||||
# 把 base 里共享的 ClusterRole 绑定到 local namespace 的 ServiceAccount
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: rcoder-pods-crb-local
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: rcoder-pods-sa
|
||||
namespace: rcoder-local
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: rcoder-pods-clusterrole
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
12
qiming-rcoder/k8s/manifests/overlays/local/credentials.yaml
Normal file
12
qiming-rcoder/k8s/manifests/overlays/local/credentials.yaml
Normal file
@@ -0,0 +1,12 @@
|
||||
# 存储层凭据 Secret - 本地测试环境
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: rcoder-credentials
|
||||
namespace: rcoder-local
|
||||
type: Opaque
|
||||
stringData:
|
||||
POSTGRES_USER: "juicefs"
|
||||
POSTGRES_PASSWORD: "juicefs_password"
|
||||
MINIO_ROOT_USER: "minioadmin"
|
||||
MINIO_ROOT_PASSWORD: "minioadmin"
|
||||
@@ -0,0 +1,91 @@
|
||||
# Local overlay - OrbStack 本地测试
|
||||
# 用法: kubectl apply -k k8s/manifests/overlays/local
|
||||
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: rcoder-local
|
||||
|
||||
resources:
|
||||
- ../../base
|
||||
- clusterrolebinding.yaml
|
||||
- credentials.yaml
|
||||
- rcoder-configmap.yaml
|
||||
|
||||
labels:
|
||||
- pairs:
|
||||
environment: local
|
||||
includeSelectors: false
|
||||
includeTemplates: true
|
||||
|
||||
replicas:
|
||||
- name: rcoder
|
||||
count: 1
|
||||
|
||||
images:
|
||||
- name: nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder
|
||||
newName: rcoder-local
|
||||
newTag: latest
|
||||
- name: nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder-agent-runner
|
||||
newName: rcoder-agent-runner-local
|
||||
newTag: latest
|
||||
|
||||
patches:
|
||||
# PVC 使用 local-path（OrbStack 默认 StorageClass）
|
||||
# local-path 只支持 ReadWriteOnce，本地测试改为单节点读写
|
||||
- patch: |-
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: rcoder-workspace
|
||||
spec:
|
||||
storageClassName: local-path
|
||||
accessModes:
|
||||
- ReadWriteOnce
|
||||
target:
|
||||
kind: PersistentVolumeClaim
|
||||
name: rcoder-workspace
|
||||
|
||||
# Deployment 环境变量适配
|
||||
- patch: |-
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: rcoder
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: rcoder
|
||||
env:
|
||||
- name: RCODER_K8S_NAMESPACE
|
||||
value: rcoder-local
|
||||
- name: RCODER_K8S_STORAGE_CLASS
|
||||
value: local-path
|
||||
- name: RCODER_K8S_PVC_ACCESS_MODE
|
||||
value: ReadWriteOnce
|
||||
- name: RUST_LOG
|
||||
value: "debug"
|
||||
- name: RCODER_DOCKER_IMAGE
|
||||
value: "rcoder-local:latest"
|
||||
- name: RCODER_DOCKER_IMAGE_COMPUTER
|
||||
value: "rcoder-agent-runner-local:latest"
|
||||
target:
|
||||
kind: Deployment
|
||||
name: rcoder
|
||||
|
||||
# Service 使用 NodePort（本地访问）
|
||||
- patch: |-
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: rcoder
|
||||
spec:
|
||||
type: NodePort
|
||||
ports:
|
||||
- port: 8087
|
||||
targetPort: 8087
|
||||
nodePort: 30087
|
||||
target:
|
||||
kind: Service
|
||||
name: rcoder
|
||||
@@ -0,0 +1,50 @@
|
||||
# RCoder ConfigMap - 本地测试环境
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: rcoder-config
|
||||
data:
|
||||
config.yml: |
|
||||
default_agent: "Claude"
|
||||
|
||||
projects_dir: "/app/project_workspace"
|
||||
port: 8087
|
||||
|
||||
cleanup_config:
|
||||
enabled: true
|
||||
idle_timeout_seconds: 600
|
||||
cleanup_interval_seconds: 300
|
||||
docker_stop_timeout_seconds: 30
|
||||
|
||||
docker_config:
|
||||
multi_image_config:
|
||||
global_defaults:
|
||||
registry_prefix: ""
|
||||
services:
|
||||
rcoder:
|
||||
service_type: "RCoder"
|
||||
arm64_image: "rcoder-local:latest"
|
||||
amd64_image: "rcoder-local:latest"
|
||||
default_image: "rcoder-local:latest"
|
||||
image_tag_prefix: "rcoder"
|
||||
enabled: true
|
||||
computer-agent-runner:
|
||||
service_type: "ComputerAgentRunner"
|
||||
arm64_image: "rcoder-agent-runner-local:latest"
|
||||
amd64_image: "rcoder-agent-runner-local:latest"
|
||||
default_image: "rcoder-agent-runner-local:latest"
|
||||
image_tag_prefix: "rcoder-agent-runner"
|
||||
enabled: true
|
||||
selection_strategy: "ServiceOnly"
|
||||
|
||||
network_base_name: "agent-network"
|
||||
network_mode: "bridge"
|
||||
work_dir: "/app"
|
||||
auto_cleanup: true
|
||||
container_ttl_seconds: 3600
|
||||
|
||||
api_key_auth:
|
||||
enabled: false
|
||||
|
||||
kubernetes_config:
|
||||
storage_class: "local-path"
|
||||
@@ -0,0 +1,17 @@
|
||||
# ============================================================
|
||||
# ClusterRoleBinding - Prod 环境专用
|
||||
# 把 base 里共享的 ClusterRole 绑定到 prod namespace 的 ServiceAccount
|
||||
# 集群级资源独立命名，避免与 dev 冲突
|
||||
# ============================================================
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRoleBinding
|
||||
metadata:
|
||||
name: rcoder-pods-crb-prod
|
||||
subjects:
|
||||
- kind: ServiceAccount
|
||||
name: rcoder-pods-sa
|
||||
namespace: nuwax-rcoder-prod
|
||||
roleRef:
|
||||
kind: ClusterRole
|
||||
name: rcoder-pods-clusterrole
|
||||
apiGroup: rbac.authorization.k8s.io
|
||||
16
qiming-rcoder/k8s/manifests/overlays/prod/credentials.yaml
Normal file
16
qiming-rcoder/k8s/manifests/overlays/prod/credentials.yaml
Normal file
@@ -0,0 +1,16 @@
|
||||
# ============================================================
|
||||
# 存储层凭据 Secret - Prod 环境
|
||||
# ⚠️ 上线前必须替换为真实强密码，并同步修改 juicefs-secret.yaml 的 metaurl / MINIO_ACCESS_KEY 等字段
|
||||
# 推荐做法：不把真实密码提交到 git，改用 SealedSecret / ExternalSecret / 部署时注入
|
||||
# ============================================================
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: rcoder-credentials
|
||||
namespace: nuwax-rcoder-prod
|
||||
type: Opaque
|
||||
stringData:
|
||||
POSTGRES_USER: "juicefs"
|
||||
POSTGRES_PASSWORD: "prodPg2024"
|
||||
MINIO_ROOT_USER: "minioadmin"
|
||||
MINIO_ROOT_PASSWORD: "prodMinio2024"
|
||||
@@ -0,0 +1,29 @@
|
||||
# ============================================================
|
||||
# JuiceFS Secret - Prod 环境专用
|
||||
# ⚠️ 上线前必须把 metaurl/access-key/secret-key/MINIO_* 的明文密码
|
||||
# 替换为 credentials.yaml 中设置的真实密码，保持两处一致。
|
||||
# ============================================================
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: juicefs-secret
|
||||
namespace: nuwax-rcoder-prod
|
||||
type: Opaque
|
||||
stringData:
|
||||
# JuiceFS CE 必需字段（小写键名）
|
||||
name: "rcoder-juicefs"
|
||||
metaurl: "postgres://juicefs:prodPg2024@postgresql.nuwax-rcoder-prod:5432/juicefs"
|
||||
storage: "minio"
|
||||
bucket: "http://minio-service.nuwax-rcoder-prod:9000/juicefs"
|
||||
access-key: "minioadmin"
|
||||
secret-key: "prodMinio2024"
|
||||
|
||||
# 兼容字段（保留给现有脚本/日志使用）
|
||||
METAURL: "postgres://juicefs:prodPg2024@postgresql.nuwax-rcoder-prod:5432/juicefs"
|
||||
MINIO_ACCESS_KEY: "minioadmin"
|
||||
MINIO_SECRET_KEY: "prodMinio2024"
|
||||
MINIO_ADDRESS: "minio-service.nuwax-rcoder-prod:9000"
|
||||
|
||||
# 其他配置
|
||||
APPEND_FSYNC: "1"
|
||||
CACHE_SIZE: "1024"
|
||||
@@ -0,0 +1,24 @@
|
||||
# ============================================================
|
||||
# JuiceFS StorageClass - Prod 环境专用
|
||||
# 集群级资源独立命名，避免与 dev 冲突
|
||||
# ============================================================
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: juicefs-sc-prod
|
||||
annotations:
|
||||
storageclass.kubernetes.io/is-default-class: "false"
|
||||
provisioner: csi.juicefs.com
|
||||
parameters:
|
||||
csi.storage.k8s.io/provisioner-secret-name: juicefs-secret
|
||||
csi.storage.k8s.io/provisioner-secret-namespace: nuwax-rcoder-prod
|
||||
csi.storage.k8s.io/controller-expand-secret-name: juicefs-secret
|
||||
csi.storage.k8s.io/controller-expand-secret-namespace: nuwax-rcoder-prod
|
||||
csi.storage.k8s.io/node-publish-secret-name: juicefs-secret
|
||||
csi.storage.k8s.io/node-publish-secret-namespace: nuwax-rcoder-prod
|
||||
juicefs/controllerQuotaSet: "false"
|
||||
juicefs/mountImage: "juicedata/mount:ce-v1.3.1"
|
||||
juicefs/mountOptions: "cache-size=1024,allow-other"
|
||||
reclaimPolicy: Retain
|
||||
allowVolumeExpansion: true
|
||||
volumeBindingMode: Immediate
|
||||
87
qiming-rcoder/k8s/manifests/overlays/prod/kustomization.yaml
Normal file
87
qiming-rcoder/k8s/manifests/overlays/prod/kustomization.yaml
Normal file
@@ -0,0 +1,87 @@
|
||||
# Production environment overlay
|
||||
# 使用方式: kubectl apply -k manifests/overlays/prod
|
||||
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
|
||||
namespace: nuwax-rcoder-prod
|
||||
|
||||
resources:
|
||||
- ../../base
|
||||
# prod 独立的集群级资源（避免与 dev 冲突）
|
||||
- juicefs-storageclass.yaml
|
||||
- clusterrolebinding.yaml
|
||||
# prod 独立的 namespace 内资源
|
||||
- rcoder-configmap.yaml
|
||||
- credentials.yaml
|
||||
- juicefs-secret.yaml
|
||||
|
||||
# 生产环境标签（含义同 dev 注释）
|
||||
labels:
|
||||
- pairs:
|
||||
environment: production
|
||||
includeSelectors: false
|
||||
includeTemplates: true
|
||||
|
||||
# 生产环境副本数
|
||||
replicas:
|
||||
- name: rcoder
|
||||
count: 2
|
||||
|
||||
# 生产环境镜像 tag
|
||||
images:
|
||||
- name: nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder
|
||||
newName: nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/nuwax-test/rcoder-k8s
|
||||
newTag: latest
|
||||
- name: nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder-agent-runner
|
||||
newName: nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/nuwax-test/rcoder-computer-agent-runner
|
||||
newTag: latest
|
||||
|
||||
patches:
|
||||
# NodePort: 30081（prod 专用，与 dev 30080 错开）
|
||||
- patch: |-
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: rcoder
|
||||
spec:
|
||||
ports:
|
||||
- port: 8087
|
||||
targetPort: 8087
|
||||
nodePort: 30081
|
||||
target:
|
||||
kind: Service
|
||||
name: rcoder
|
||||
|
||||
# PVC 指向 prod 专属 StorageClass
|
||||
- patch: |-
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: rcoder-workspace
|
||||
spec:
|
||||
storageClassName: juicefs-sc-prod
|
||||
target:
|
||||
kind: PersistentVolumeClaim
|
||||
name: rcoder-workspace
|
||||
|
||||
# rcoder deployment 指向 prod 环境的 namespace 和 StorageClass
|
||||
- patch: |-
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: rcoder
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: rcoder
|
||||
env:
|
||||
- name: RCODER_K8S_NAMESPACE
|
||||
value: nuwax-rcoder-prod
|
||||
- name: RCODER_K8S_STORAGE_CLASS
|
||||
value: juicefs-sc-prod
|
||||
target:
|
||||
kind: Deployment
|
||||
name: rcoder
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
# ============================================================
|
||||
# RCoder ConfigMap - Prod 环境专用
|
||||
# storage_class 指向 juicefs-sc-prod
|
||||
# ============================================================
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: rcoder-config
|
||||
data:
|
||||
config.yml: |
|
||||
# RCoder 配置文件
|
||||
default_agent: "Claude"
|
||||
|
||||
projects_dir: "/app/project_workspace"
|
||||
port: 8087
|
||||
|
||||
cleanup_config:
|
||||
enabled: true
|
||||
idle_timeout_seconds: 600
|
||||
cleanup_interval_seconds: 300
|
||||
docker_stop_timeout_seconds: 30
|
||||
|
||||
proxy_config:
|
||||
listen_port: 8088
|
||||
default_backend_port: 8087
|
||||
backend_host: "127.0.0.1"
|
||||
port_param: "port"
|
||||
health_check:
|
||||
enabled: true
|
||||
interval_seconds: 5
|
||||
timeout_seconds: 1
|
||||
healthy_threshold: 2
|
||||
unhealthy_threshold: 3
|
||||
|
||||
docker_config:
|
||||
multi_image_config:
|
||||
global_defaults:
|
||||
registry_prefix: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev"
|
||||
services:
|
||||
rcoder:
|
||||
service_type: "RCoder"
|
||||
arm64_image: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder:latest"
|
||||
amd64_image: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder:latest"
|
||||
default_image: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder:latest"
|
||||
image_tag_prefix: "rcoder"
|
||||
enabled: true
|
||||
computer-agent-runner:
|
||||
service_type: "ComputerAgentRunner"
|
||||
arm64_image: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder-agent-runner:latest"
|
||||
amd64_image: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder-agent-runner:latest"
|
||||
default_image: "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder-agent-runner:latest"
|
||||
image_tag_prefix: "rcoder-computer-agent-runner"
|
||||
enabled: true
|
||||
selection_strategy: "ServiceOnly"
|
||||
cache_config:
|
||||
enabled: true
|
||||
ttl_seconds: 3600
|
||||
max_entries: 50
|
||||
|
||||
network_base_name: "agent-network"
|
||||
network_mode: "bridge"
|
||||
work_dir: "/app"
|
||||
auto_cleanup: true
|
||||
container_ttl_seconds: 3600
|
||||
|
||||
api_key_auth:
|
||||
enabled: false
|
||||
api_key: "YOUR_API_KEY_HERE"
|
||||
|
||||
kubernetes_config:
|
||||
storage_class: "juicefs-sc-prod"
|
||||
417
qiming-rcoder/k8s/nuwax-platform-k8s-plan.md
Normal file
417
qiming-rcoder/k8s/nuwax-platform-k8s-plan.md
Normal file
@@ -0,0 +1,417 @@
|
||||
# Nuwax Platform K8s 部署方案
|
||||
|
||||
本文档记录 **`build-agent-docker` 项目** 从 docker-compose 迁移到 K8s 部署的完整方案。rcoder 项目的 Helm chart 是其中一个组件的**复制来源**,不做二次开发。
|
||||
|
||||
> 代码仓库:`/home/swufe/gitworkspace/build-agent-docker`
|
||||
> 分支:`dev-k8s`
|
||||
> 对应 rcoder Helm chart:`agents-a24a3ab5f6/k8s/helm/rcoder/`(模板复制基础)
|
||||
|
||||
---
|
||||
|
||||
## 1. 背景 / 为什么迁移
|
||||
|
||||
### 1.1 docker-compose 的两个痛点
|
||||
|
||||
1. **hostPath 限制扩容节点**
|
||||
- 所有数据卷 bind-mount 到宿主机目录
|
||||
- 新加节点时文件系统不共享,rcoder 动态创建的 agent-runner 容器只能跑在同一台机器
|
||||
- 单点故障、IO 瓶颈、无弹性
|
||||
|
||||
2. **政企客户需要 K8s 原生部署**
|
||||
- 很多客户已有自建 K8s 集群
|
||||
- 方案必须兼容**标准 K8s 1.25+**(k3s 只作 dev 便利)
|
||||
- 必须支持离线部署
|
||||
|
||||
### 1.2 目标
|
||||
|
||||
- **全栈 Helm chart** (`k8s/helm/nuwax-platform/`),支持 dev / test / prod / offline 四环境
|
||||
- **JuiceFS + PostgreSQL + MinIO** K8s 原生文件服务栈(替代 hostPath)
|
||||
- **保留现有 workflow**:`make dev` → `make push-test` → `make push-prod` 不变,新增 `make k8s-deploy-*` 对接
|
||||
- **政企离线 bundle**(照抄 rcoder offline 模式,自包含所有镜像)
|
||||
- **双栈并存**:docker-compose 继续可用,不 break 现有流程
|
||||
|
||||
---
|
||||
|
||||
## 2. 架构
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Ingress (可选, className 可配) │
|
||||
│ └── / → frontend svc:80 │
|
||||
│ │
|
||||
│ NodePort (dev 便利 / 未配域名时兜底): │
|
||||
│ frontend :30080 MinIO API :30900 Console :30901 │
|
||||
│ │
|
||||
│ 集群内部 (ClusterIP): │
|
||||
│ backend:8080(主 REST API) │
|
||||
│ +18082(Netty 代理 /page/) │
|
||||
│ +18085(computer 路由,noVNC/websockify/audio) │
|
||||
│ +18086(模型代理) │
|
||||
│ +18087(沙盒代理) │
|
||||
│ mcp-proxy:8089 / rcoder:8086 │
|
||||
│ mysql:3306 / redis:6379 / milvus:19530 / es:9200 │
|
||||
│ │
|
||||
│ 文件服务(K8s 原生,跨节点): │
|
||||
│ ├── JuiceFS StorageClass (juicefs-sc-<release>, RWX) │
|
||||
│ │ 元数据: PostgreSQL StatefulSet (块存储 PVC) │
|
||||
│ │ 数据块: MinIO StatefulSet (块存储 PVC) │
|
||||
│ │ bucket juicefs (JuiceFS chunks) │
|
||||
│ │ bucket nuwax-app (应用对象存储) │
|
||||
│ └── RWX PVC (挂 JuiceFS SC): │
|
||||
│ backend-upload / rcoder-workspace / │
|
||||
│ rcoder-computer-workspace │
|
||||
│ 动态 agent-runner Pod 也挂同一组 │
|
||||
│ │
|
||||
│ 数据库自有块存储 (global.blockStorageClass / 集群默认): │
|
||||
│ mysql / redis / milvus / elasticsearch-ik StatefulSet │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**分档原则**: 数据库类用**块存储**(RWO,低延迟),共享文件类用 **JuiceFS**(RWX,跨节点)。两档都可通过 values 切换到客户自有 SC。
|
||||
|
||||
---
|
||||
|
||||
## 3. 关键决策 + 理由
|
||||
|
||||
### 3.1 Helm chart 大一统(不用 Kustomize)
|
||||
|
||||
| 维度 | 选择 | 理由 |
|
||||
|------|------|------|
|
||||
| 部署方式 | **单一 Helm chart**,4 个 values 文件 | 8+ 组件 + 多环境用 Kustomize overlay 会很臃肿;Helm values 参数化更友好,对外交付好用 |
|
||||
| 已有 K8s 资产 | 推倒重建(tekton / deploy / overlays / 旧 helm 全删) | 现有都是半成品,有硬编码密码/hostPath/和 GitHub Actions 重复的 Tekton |
|
||||
|
||||
### 3.2 rcoder:新增 `rcoder-k8s` 镜像(feature flag 编译)
|
||||
|
||||
rcoder 源码里 `kubernetes` feature 已经存在:
|
||||
|
||||
```toml
|
||||
# crates/rcoder/Cargo.toml
|
||||
[features]
|
||||
default = []
|
||||
kubernetes = ["docker_manager/kubernetes"] # CONTAINER_RUNTIME=kubernetes 生效
|
||||
```
|
||||
|
||||
**方式 A(采用)**: 新增独立 image tag `rcoder-k8s`
|
||||
- `Dockerfile` 加 `ARG CARGO_FEATURES`,`make build-rcoder-k8s` 传 `CARGO_FEATURES=kubernetes`
|
||||
- 不挂 docker.sock,通过 K8s API 创建动态 agent-runner Pod
|
||||
- docker-compose 用的 `rcoder` 镜像不受影响
|
||||
|
||||
**方式 B(未采用)**: 让 `rcoder` 镜像始终含 kubernetes feature
|
||||
- 镜像多几 MB,但逻辑简单
|
||||
- docker-compose 老用户场景也会带上 K8s 代码(轻微浪费)
|
||||
|
||||
### 3.3 Elasticsearch IK 分词插件:定制镜像
|
||||
|
||||
docker-compose 通过挂载 `docker/config/elasticsearch/plugins/` 使用 IK 插件,K8s 不能挂主机目录,所以:
|
||||
|
||||
- 新建 `build_config/elasticsearch-ik/Dockerfile`:4 行 `FROM elasticsearch:9.2.1` + `COPY analysis-ik-9.2.1/`
|
||||
- 复制 docker-compose 已用的 IK 发行包到 `build_config/elasticsearch-ik/`(维护时双份同步,IK 本身 ~1.5MB)
|
||||
- K8s values `elasticsearch.image.repository: elasticsearch-ik`
|
||||
|
||||
### 3.4 文件服务:**JuiceFS + PG + MinIO,不用 Longhorn**
|
||||
|
||||
rcoder chart 有 Longhorn,但 nuwax-platform 不带:
|
||||
- 数据库用集群默认块存储 SC 就够(用户自有 longhorn/ceph-rbd/local-path 都行)
|
||||
- 共享文件用 JuiceFS 完全覆盖跨节点读写
|
||||
- 少装一个 Longhorn 减少离线 bundle 体积(~5GB → ~3GB)
|
||||
|
||||
**MinIO 只起一个实例**,开**两个 bucket**:
|
||||
- `juicefs` — JuiceFS 数据块
|
||||
- `nuwax-app` — 应用对象存储(backend 的文件等)
|
||||
|
||||
### 3.5 兼容标准 K8s 1.25+
|
||||
|
||||
明确 **不依赖 k3s 特有机制**:
|
||||
- StorageClass 空串 → 走集群默认,不强制 `local-path`
|
||||
- Ingress Controller 由用户提供(`className` 参数化,默认 `nginx`,可改 `traefik` / `alb` / `istio`)
|
||||
- 镜像拉取走公共或客户私有 registry,不依赖 k3s registries.yaml
|
||||
|
||||
k3s 只作为 dev 便利路径(开发者自己本地起一套测),生产一律走标准 K8s。
|
||||
|
||||
### 3.6 对外暴露:只 frontend + MinIO API
|
||||
|
||||
| 服务 | 是否对外 | 方式 |
|
||||
|-----|--------|------|
|
||||
| frontend | ✓ | Ingress(配域名时)或 NodePort(dev) |
|
||||
| **MinIO API** | **✓** | NodePort 30900(应用要直连 S3 协议) |
|
||||
| MinIO Console | (可选) | NodePort 30901(管理员) |
|
||||
| backend / mcp-proxy / rcoder | ✗ | ClusterIP 内部 |
|
||||
| mysql / redis / milvus / es | ✗ | ClusterIP 内部 |
|
||||
| postgresql(JuiceFS 元数据) | ✗ | ClusterIP 内部 |
|
||||
|
||||
### 3.7 多环境同集群共存
|
||||
|
||||
同一 K8s 集群可同时跑 dev / test / prod 三个 release,天然隔离:
|
||||
- **namespace 级资源** → 各自 namespace 隔离
|
||||
- **集群级资源**(冲突风险):
|
||||
- `ClusterRole` 名字固定 `nuwax-rcoder-pods-clusterrole`(规则相同,多 release 共享,`helm.sh/resource-policy: keep`)
|
||||
- `ClusterRoleBinding` 名字 `{release}-rcoder-pods-crb`(Release 独立)
|
||||
- `StorageClass` 名字 `juicefs-sc-{release}`(Release 独立)
|
||||
|
||||
### 3.8 不在范围内的
|
||||
|
||||
| 组件 | 为什么不做 |
|
||||
|-----|-----------|
|
||||
| Doris | 未正式使用,后续用到再加 |
|
||||
| log-platform | 废弃 |
|
||||
| video-analysis | 废弃 |
|
||||
| subapp-deployer | 容器里 npm install 安装,不需要独立部署 |
|
||||
| agent-client | 不需要 K8s 部署 |
|
||||
|
||||
---
|
||||
|
||||
## 4. 实现(3 个 PR + 1 个 fix commit)
|
||||
|
||||
Git 提交历史(`build-agent-docker` 仓库 `dev-k8s` 分支):
|
||||
|
||||
```
|
||||
1563052 refactor(build): remove dev-parallel target, rely on serial make dev
|
||||
15ca818 fix(k8s): post-review fixes to make deployment actually work
|
||||
77d33af feat(k8s): offline bundle tooling + production hardening (PR3)
|
||||
138b313 feat(k8s): complete nuwax-platform Helm chart with K8s-native file service (PR2)
|
||||
edf1f7d refactor(k8s): rebuild k8s/ directory with Helm chart skeleton (PR1)
|
||||
```
|
||||
|
||||
### PR1 — 清理 + Helm 骨架 (`edf1f7d`)
|
||||
|
||||
- 删除 `k8s/{tekton,deploy,overlays,helm/nuwax-platform}`(195 文件)
|
||||
- 新建 `k8s/helm/nuwax-platform/` 骨架:Chart.yaml + values + `_helpers.tpl` + NOTES + 空目录占位
|
||||
- 重写 `makefiles/10-k8s.mk`:`k8s-lint` / `k8s-render-*`
|
||||
- **验证**:helm lint 4 环境通过,render 空骨架
|
||||
|
||||
### PR2 — 全量模板 + 构建系统 (`138b313`)
|
||||
|
||||
**构建系统**:
|
||||
- `build_config/rcoder/Dockerfile` 加 `ARG CARGO_FEATURES`
|
||||
- 新建 `build_config/elasticsearch-ik/Dockerfile` + 复制 IK 插件
|
||||
- `makefiles/02-build.mk` 加 `build-rcoder-k8s-{amd64,arm64}` / `build-elasticsearch-ik-{amd64,arm64}`,挂到现有 `build-{amd64,arm64}-only`(→ `make dev` 自动覆盖)
|
||||
- `makefiles/05-push.mk` 加 `push-rcoder-k8s` / `push-elasticsearch-ik`,挂到 `push-custom-images`(→ `make push-test`/`push-prod` 自动覆盖)
|
||||
|
||||
**Helm 模板**(33 个):
|
||||
- `rcoder/` 8 个(复制自 rcoder chart,去 docker.sock + 换 image 为 rcoder-k8s)
|
||||
- `frontend/` 4 个(nginx.conf 里 `backend:` 通过 replace 自动变 `{release}-backend:`)
|
||||
- `backend/` 4 个(5 个端口 + ConfigMap 化 application-external.yml + entrypoint.sh)
|
||||
- `mcp-proxy/` 3 个
|
||||
- `storage/` 15 个(MySQL/Redis/Milvus/ES/PG/MinIO + JuiceFS Secret/SC + credentials)
|
||||
- `shared-pvcs/` 3 个 RWX PVC
|
||||
- `ingress.yaml` 1 个(条件渲染)
|
||||
|
||||
**Make 部署目标**:
|
||||
- `k8s-install-juicefs-csi`(集群级,一次性)
|
||||
- `k8s-deploy-{dev,test,prod}` + preflight
|
||||
- `k8s-undeploy-*` / `k8s-purge-*` / `k8s-status-*` / `k8s-logs-<env>-<component>`
|
||||
|
||||
### PR3 — 离线 bundle + 生产加固 (`77d33af`)
|
||||
|
||||
**离线**:
|
||||
- `k8s/offline/images.txt` — ~20 个镜像 pin 版本
|
||||
- `k8s/offline/install.sh` — direct / registry 双模式,direct 自动探测 k3s ctr / nerdctl / ctr
|
||||
- `k8s/offline/rewrite-registry.sh` — 重 tag 推到客户 registry
|
||||
- `k8s/offline/README.md` — 客户交付手册
|
||||
- `makefiles/10-k8s.mk` 加 `k8s-offline-bundle` / `k8s-offline-import`
|
||||
|
||||
**加固**:
|
||||
- `templates/common/network-policy.yaml` — prod 默认 deny-all + 显式放行
|
||||
- values-prod 开启 NetworkPolicy + Ingress + TLS + PDB
|
||||
|
||||
**文档**:`k8s/README.md` 覆盖 dev/test/prod/offline 四场景
|
||||
|
||||
### Fix commit — 部署前自查发现的 5 处 bug (`15ca818`)
|
||||
|
||||
| # | 问题 | 修法 |
|
||||
|---|-----|-----|
|
||||
| 1 | backend 等 Redis 时 `redis-cli ping` 未带 `-a $REDIS_PASSWORD` → NOAUTH,entrypoint exit 1 | `configmap.yaml` replace 加专项规则 |
|
||||
| 2 | `manage_jwt_secret()` 覆盖 env,Pod 重启换 JWT → session 失效 | Secret 投影为 `/app/config/jwt/jwt_secret_key.txt`,脚本走"读文件"分支 |
|
||||
| 3 | rcoder 缺 nginx 子应用 conf + 404/502 + 3 个 workspace 子目录 + 脚手架模板 + agent-runner 覆盖目录 | 加 2 个 ConfigMap(nginx conf / binaryData 模板 zip)+ subPath 切分 workspace PVC + emptyDir 给 agent-runner |
|
||||
| 4 | ES readinessProbe 硬编码 Basic base64 → 改密码后 401 | 改 exec curl -u `elastic:$ELASTIC_PASSWORD`,从 Secret 动态注入 |
|
||||
| 5 | values-prod.yaml 6 处 CHANGE-ME 需人工替换 | 改成和 `docker/.env` 一致的默认密码(`root` / `admin123` / `123456` / `elastic123` / `minioadmin`)开箱即用 |
|
||||
| 6 | 文档补漏 | prod namespace PSS ≤ baseline(Milvus 需要 seccomp Unconfined)+ CNI 支持矩阵 |
|
||||
|
||||
### 精简 commit — 移除 `dev-parallel` (`1563052`)
|
||||
|
||||
- 并行 docker buildx 会偶发 BuildKit cache 竞态,生产/复现难
|
||||
- `make dev` 串行路径已覆盖所有 K8s 镜像,移除 `dev-parallel` 及其子目标
|
||||
- 保留 `setup-parallel`(git clone/pull 无竞态)
|
||||
|
||||
---
|
||||
|
||||
## 5. 使用流程
|
||||
|
||||
### 5.1 本地 dev(k3s / kind)
|
||||
|
||||
```bash
|
||||
cd build-agent-docker
|
||||
|
||||
# 1. 构建全部镜像 + 推到测试仓库(含 rcoder-k8s + elasticsearch-ik)
|
||||
make dev
|
||||
|
||||
# 2. 装 JuiceFS CSI Driver(集群级,一次性)
|
||||
make k8s-install-juicefs-csi
|
||||
|
||||
# 3. 部署到 nuwax-dev namespace
|
||||
make k8s-deploy-dev
|
||||
make k8s-status-dev
|
||||
|
||||
# 4. 访问
|
||||
curl http://<node-ip>:30080/ # frontend
|
||||
curl http://<node-ip>:30080/api/health # 经 nginx 到 backend
|
||||
|
||||
# 验证 JuiceFS 跨 Pod 共享
|
||||
kubectl exec -n nuwax-dev deploy/nuwax-dev-backend -- sh -c 'echo test > /app/upload/x'
|
||||
kubectl exec -n nuwax-dev deploy/nuwax-dev-rcoder -- cat /app/upload/x # test
|
||||
```
|
||||
|
||||
### 5.2 test / prod
|
||||
|
||||
```bash
|
||||
# 改 values-test.yaml / values-prod.yaml(blockStorageClass / ingress hosts / 域名)
|
||||
kubectl config use-context <test-cluster>
|
||||
make k8s-install-juicefs-csi
|
||||
make k8s-deploy-test
|
||||
|
||||
# prod(镜像切生产仓库)
|
||||
make push-prod
|
||||
kubectl config use-context <prod-cluster>
|
||||
make k8s-install-juicefs-csi
|
||||
make k8s-deploy-prod
|
||||
```
|
||||
|
||||
### 5.3 政企离线
|
||||
|
||||
```bash
|
||||
# 构建机(有网)
|
||||
make push-prod # 先把镜像推到 /nuwax
|
||||
make k8s-offline-bundle # dist/nuwax-offline-<ver>-<arch>.tar.gz (~3GB)
|
||||
|
||||
# 客户机器(断网)
|
||||
scp dist/nuwax-offline-*.tar.gz offline-node:/tmp/
|
||||
ssh offline-node
|
||||
cd /tmp && tar xzf nuwax-offline-*.tar.gz -C nuwax-offline && cd nuwax-offline
|
||||
bash install.sh --mode=direct --env=prod # 单节点
|
||||
# 或多节点:
|
||||
bash install.sh --mode=registry --registry=harbor.internal/nuwax --env=prod
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 镜像清单 + 用途
|
||||
|
||||
| 镜像 | docker-compose | K8s | 说明 |
|
||||
|-----|:---:|:---:|------|
|
||||
| agent-platform-front | ✓ | ✓ | 前端 nginx |
|
||||
| agent-platform-backend | ✓ | ✓ | Spring Boot 后端 |
|
||||
| mcp-proxy | ✓ | ✓ | Rust MCP 代理 |
|
||||
| rcoder | ✓ | ✗ | docker-compose 用,挂 docker.sock |
|
||||
| **rcoder-k8s** | ✗ | ✓ | **K8s 用,feature=kubernetes 编译,K8s API 起 Pod** |
|
||||
| rcoder-computer-agent-runner | ✓ | ✓ | 动态 agent-runner 基础镜像 |
|
||||
| agent-client | ✓ | ✗ | 不纳入 K8s |
|
||||
| elasticsearch(官方) | ✓ | ✗ | docker-compose 用 |
|
||||
| **elasticsearch-ik** | ✗ | ✓ | **K8s 用,含 IK 中文分词插件** |
|
||||
| mysql:8.0 | ✓ | ✓ | 第三方 |
|
||||
| redis:7.0 | ✓ | ✓ | 第三方 |
|
||||
| milvusdb/milvus:v2.5.8 | ✓ | ✓ | 第三方 |
|
||||
| minio:RELEASE-xxx | ✓ | ✓ | JuiceFS 后端 + 应用对象存储(2 bucket) |
|
||||
| postgres:16-alpine | ✗ | ✓ | JuiceFS 元数据(K8s 专用) |
|
||||
| busybox:1.36 | ✓ | ✓ | init container |
|
||||
| juicedata/juicefs-csi-driver:v0.31.3 | ✗ | ✓ | K8s 专用,集群级组件 |
|
||||
| juicedata/mount:ce-v1.3.1 | ✗ | ✓ | JuiceFS CSI 运行时 |
|
||||
| registry.k8s.io/sig-storage/csi-* 4 个 | ✗ | ✓ | JuiceFS CSI sidecars |
|
||||
|
||||
---
|
||||
|
||||
## 7. 文件结构
|
||||
|
||||
```
|
||||
build-agent-docker/
|
||||
├── build_config/
|
||||
│ ├── rcoder/
|
||||
│ │ └── Dockerfile # 加 ARG CARGO_FEATURES,rcoder-k8s 传 kubernetes
|
||||
│ └── elasticsearch-ik/ # 【新增】
|
||||
│ ├── Dockerfile # 4 行 FROM + COPY
|
||||
│ └── elasticsearch-analysis-ik-9.2.1/
|
||||
│
|
||||
├── makefiles/
|
||||
│ ├── 00-vars.mk # + RCODER_K8S_IMAGE / ELASTICSEARCH_IK_IMAGE
|
||||
│ ├── 02-build.mk # + build-rcoder-k8s-* / build-elasticsearch-ik-*
|
||||
│ ├── 05-push.mk # + push-rcoder-k8s / push-elasticsearch-ik
|
||||
│ └── 10-k8s.mk # 整体重写: k8s-lint/render/deploy/offline
|
||||
│
|
||||
└── k8s/ # 【整体重建】
|
||||
├── README.md # 四场景操作指南 + 故障排查
|
||||
├── helm/
|
||||
│ └── nuwax-platform/
|
||||
│ ├── Chart.yaml
|
||||
│ ├── values.yaml # 默认
|
||||
│ ├── values-dev.yaml # 本地 k3s / NodePort / 小资源
|
||||
│ ├── values-test.yaml # 自建 K8s / 中等规模
|
||||
│ ├── values-prod.yaml # 副本 2+ / Ingress+TLS / NetworkPolicy
|
||||
│ ├── values-offline.yaml # 私有 registry 占位
|
||||
│ ├── configs/ # .Files.Get 读的配置文件
|
||||
│ │ ├── nginx.conf # (backend: 主机自动替换)
|
||||
│ │ ├── application-external.yml
|
||||
│ │ ├── docker-entrypoint.sh
|
||||
│ │ ├── mcp_config.yml
|
||||
│ │ ├── mysql.cnf / redis.conf / milvus/*.yaml / elasticsearch/*
|
||||
│ │ ├── sub_app_multi_apps.conf # rcoder 内置 nginx 子应用路由
|
||||
│ │ ├── init_mysql{,_data}.sql
|
||||
│ │ ├── nginx/custom/{404,502}.html
|
||||
│ │ └── rcoder-templates/ # react-vite-template.zip, vue3-vite-template.zip
|
||||
│ └── templates/
|
||||
│ ├── _helpers.tpl
|
||||
│ ├── NOTES.txt
|
||||
│ ├── ingress.yaml
|
||||
│ ├── frontend/ (4) # Deploy + Service + nginx CM + 404/502 CM
|
||||
│ ├── backend/ (4) # Deploy + Service + config CM + pdb
|
||||
│ ├── mcp-proxy/ (3)
|
||||
│ ├── rcoder/ (10) # 8 复制自 rcoder chart + nginx CM + 模板 CM
|
||||
│ ├── storage/ (17) # 6 组中间件 + PG + MinIO + JuiceFS
|
||||
│ ├── shared-pvcs/ (3) # backend-upload / rcoder-workspace ×2
|
||||
│ └── common/ (1) # network-policy (prod)
|
||||
│
|
||||
└── offline/ # 离线 bundle 源素材
|
||||
├── images.txt # 20+ 镜像清单
|
||||
├── install.sh # direct / registry 双模式
|
||||
├── rewrite-registry.sh
|
||||
└── README.md # 客户交付手册
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 已知限制 / 运维注意
|
||||
|
||||
| 项 | 详情 |
|
||||
|----|------|
|
||||
| K3s flannel 不支持 NetworkPolicy | prod 启用 NetworkPolicy 需用 Calico / Cilium / Antrea / kube-router |
|
||||
| prod namespace PSS 级别 | 必须 ≤ baseline(Milvus 需要 seccompProfile: Unconfined,restricted 会拒) |
|
||||
| rcoder chart 双份维护 | `build-agent-docker/.../templates/rcoder/` 和 rcoder 仓库 `k8s/helm/rcoder/` 同步更新 |
|
||||
| ES IK 插件目录双份 | `docker/config/elasticsearch/plugins/` 和 `build_config/elasticsearch-ik/` 同步 |
|
||||
| JuiceFS 上的 DB 性能 | MySQL/Redis/Milvus/ES 严禁用 JuiceFS,模板里已强制块存储 PVC |
|
||||
| 密码管理 | dev/prod values 里是明文默认值(开箱即用);生产环境建议 External Secrets / sealed-secrets |
|
||||
| direct 模式镜像导入 | 多节点集群每个节点都要运行 install.sh,或推荐 registry 模式一次推送全集群可用 |
|
||||
|
||||
---
|
||||
|
||||
## 9. 验证矩阵
|
||||
|
||||
| 检查项 | 命令 | 预期 |
|
||||
|-------|------|------|
|
||||
| Helm chart 语法 | `make k8s-lint` | 4 环境都 pass |
|
||||
| 模板渲染 | `make k8s-render-all` | 4 环境输出有效 YAML |
|
||||
| dev 资源数 | `make k8s-render-dev \| grep -c "^kind:"` | 46 |
|
||||
| prod 资源数 | `make k8s-render-prod \| grep -c "^kind:"` | 54(含 5 NetworkPolicy + Ingress + 2 PDB) |
|
||||
| 镜像构建全量 | `make -n dev \| grep -c "docker buildx build"` | 20(9 镜像 × 2 架构 + 基础镜像) |
|
||||
| K8s 集群端到端 | `make k8s-deploy-dev && curl http://<ip>:30080/` | HTTP 200 |
|
||||
| JuiceFS 跨 Pod 共享 | 见 §5.1 最后两条 kubectl exec | `test` |
|
||||
| 离线 bundle 构建 | `make k8s-offline-bundle` | `dist/nuwax-offline-*.tar.gz` 产出 |
|
||||
| 离线 bundle 内容 | `tar tzf dist/nuwax-offline-*.tar.gz \| head` | install.sh + images + charts + values |
|
||||
|
||||
---
|
||||
|
||||
## 10. 术语
|
||||
|
||||
- **release name**: Helm 安装时的名字,和 namespace 同名,形如 `nuwax-dev` / `nuwax-test` / `nuwax-prod`
|
||||
- **block storage**: 块存储(RWO, 单 Pod 读写),用于数据库
|
||||
- **shared storage**: 共享文件存储(RWX, 多 Pod 读写),用于上传文件 / workspace
|
||||
- **JuiceFS CSI**: 集群级 CSI driver,不在 nuwax-platform chart 里,单独 `make k8s-install-juicefs-csi`
|
||||
- **direct 模式**: 离线部署时把镜像直接导入每个节点 containerd,单节点 / 小集群用
|
||||
- **registry 模式**: 离线部署时把镜像推到客户私有 registry,多节点推荐
|
||||
215
qiming-rcoder/k8s/offline/README.md
Normal file
215
qiming-rcoder/k8s/offline/README.md
Normal file
@@ -0,0 +1,215 @@
|
||||
# RCoder 离线部署指南
|
||||
|
||||
本文档面向政企/内网客户：**目标机器无公网访问，所有镜像与依赖必须随部署包交付**。
|
||||
|
||||
---
|
||||
|
||||
## 目录内容（解压 `rcoder-offline-*.tar.gz` 后）
|
||||
|
||||
```
|
||||
.
|
||||
├── install.sh # 主安装脚本
|
||||
├── rewrite-registry.sh # registry 模式: re-tag + push 辅助脚本
|
||||
├── README.md # 本文件
|
||||
├── values-dev.yaml # RCoder dev 环境 values
|
||||
├── values-prod.yaml # RCoder prod 环境 values
|
||||
├── values-offline.yaml # 离线 registry 覆盖模板
|
||||
├── images/
|
||||
│ ├── all-images.tar # 所有镜像的 docker save 产物
|
||||
│ └── images.txt # 镜像清单
|
||||
├── charts/
|
||||
│ └── rcoder-*.tgz # RCoder Helm chart
|
||||
├── longhorn/
|
||||
│ └── longhorn-v*.yaml # Longhorn 全量 manifest
|
||||
├── juicefs-csi/
|
||||
│ └── juicefs-csi-v*.yaml # JuiceFS CSI Driver 官方 manifest
|
||||
└── manifests/ # 原始 Kustomize 资产 (参考, 不必使用)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 前置条件
|
||||
|
||||
在离线机器上，提前准备好：
|
||||
|
||||
| 组件 | 版本 | 说明 |
|
||||
|------|------|------|
|
||||
| K8s | 1.19+ | 已部署, kubeconfig 已配置 |
|
||||
| kubectl | 与集群匹配 | `kubectl cluster-info` 能通 |
|
||||
| helm | 3.x | 用于 `helm upgrade --install` |
|
||||
| docker | 最新 | **仅 registry 模式需要** |
|
||||
| k3s / nerdctl / ctr | 最新 | **仅 direct 模式需要** |
|
||||
| open-iscsi | 任意 | Longhorn 依赖 (Ubuntu: `apt install open-iscsi`) |
|
||||
|
||||
---
|
||||
|
||||
## 两种部署模式
|
||||
|
||||
### 模式 A: direct（推荐单节点 / 小集群）
|
||||
|
||||
直接把镜像 import 到节点 containerd，**无需私有 registry**。适合 k3s 单节点或 3-5 节点小集群。
|
||||
|
||||
多节点集群需要**在每个节点上依次执行**镜像导入。
|
||||
|
||||
```bash
|
||||
# 1. 解压 bundle (可通过 make k8s-offline-import 一步完成)
|
||||
tar xzf rcoder-offline-<version>-<arch>.tar.gz -C /opt/rcoder-offline
|
||||
cd /opt/rcoder-offline
|
||||
|
||||
# 2. 一键安装
|
||||
sudo ./install.sh --mode=direct --env=dev
|
||||
|
||||
# 生产环境
|
||||
sudo ./install.sh --mode=direct --env=prod \
|
||||
--set credentials.postgresql.password='<prod-pg-pass>' \
|
||||
--set credentials.minio.rootPassword='<prod-minio-pass>'
|
||||
```
|
||||
|
||||
> **多节点注意**: 每个节点都要执行 `sudo k3s ctr -n k8s.io image import images/all-images.tar`
|
||||
> 或把 bundle scp 到每个节点各跑一次 `--skip-juicefs-csi --skip-longhorn` 只做镜像导入。
|
||||
|
||||
### 模式 B: registry（推荐多节点 / 生产）
|
||||
|
||||
把镜像重新打 tag 推送到客户的私有 registry（Harbor / Nexus / 阿里云 ACR 内网实例），K8s 按常规流程拉取。
|
||||
|
||||
**前置: 集群节点能访问私有 registry，并配置好拉取凭据（如需要）。**
|
||||
|
||||
```bash
|
||||
# 1. 解压
|
||||
tar xzf rcoder-offline-<version>-<arch>.tar.gz -C /opt/rcoder-offline
|
||||
cd /opt/rcoder-offline
|
||||
|
||||
# 2. 如果私有 registry 需要认证, 先登录 docker daemon
|
||||
docker login harbor.internal
|
||||
|
||||
# 3. 安装 (一条命令完成推送+部署)
|
||||
./install.sh \
|
||||
--mode=registry \
|
||||
--registry=harbor.internal/rcoder \
|
||||
--thirdparty-registry=harbor.internal/thirdparty \
|
||||
--env=prod
|
||||
```
|
||||
|
||||
推送完成后，镜像会以如下命名出现在 registry：
|
||||
|
||||
| 原镜像 | 推送目标 |
|
||||
|--------|---------|
|
||||
| `nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder:latest` | `harbor.internal/rcoder/rcoder:latest` |
|
||||
| `postgres:16-alpine` | `harbor.internal/thirdparty/postgres:16-alpine` |
|
||||
| `longhornio/longhorn-manager:v1.7.2` | `harbor.internal/thirdparty/longhornio/longhorn-manager:v1.7.2` |
|
||||
| `registry.k8s.io/sig-storage/csi-provisioner:v5.1.0` | `harbor.internal/thirdparty/sig-storage/csi-provisioner:v5.1.0` |
|
||||
|
||||
如果集群需要 imagePullSecret:
|
||||
```bash
|
||||
kubectl -n nuwax-rcoder-prod create secret docker-registry harbor-pull \
|
||||
--docker-server=harbor.internal \
|
||||
--docker-username=<user> \
|
||||
--docker-password=<pass>
|
||||
# 重新安装时:
|
||||
./install.sh ... --set 'global.imagePullSecrets[0].name=harbor-pull'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 常见操作
|
||||
|
||||
### 查看部署状态
|
||||
```bash
|
||||
kubectl get pods -n nuwax-rcoder-dev
|
||||
kubectl get svc -n nuwax-rcoder-dev
|
||||
kubectl rollout status deploy/rcoder -n nuwax-rcoder-dev
|
||||
```
|
||||
|
||||
### 访问服务
|
||||
```bash
|
||||
# NodePort
|
||||
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
|
||||
curl http://${NODE_IP}:30080/health # dev
|
||||
curl http://${NODE_IP}:30081/health # prod
|
||||
|
||||
# 或 port-forward
|
||||
kubectl port-forward -n nuwax-rcoder-dev svc/rcoder 8087:8087
|
||||
curl http://localhost:8087/health
|
||||
```
|
||||
|
||||
### 卸载
|
||||
```bash
|
||||
helm uninstall rcoder-dev --namespace nuwax-rcoder-dev
|
||||
|
||||
# 彻底删除 (包含 PVC)
|
||||
kubectl delete pvc --all -n nuwax-rcoder-dev
|
||||
kubectl delete namespace nuwax-rcoder-dev
|
||||
```
|
||||
|
||||
### 升级
|
||||
收到新 bundle 后直接重新跑 `install.sh`（`helm upgrade --install` 会自动升级）：
|
||||
```bash
|
||||
./install.sh --mode=registry --registry=harbor.internal/rcoder --env=prod
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 跳过选项
|
||||
|
||||
| 参数 | 用途 |
|
||||
|------|------|
|
||||
| `--skip-longhorn` | 集群已有其他 CSI (Ceph/NFS/CSI-local-path)，不装 Longhorn |
|
||||
| `--skip-juicefs-csi` | 集群已有 JuiceFS CSI 或用其他 RWX 方案 |
|
||||
| `--skip-image-import` | 镜像已手工导入，调试用 |
|
||||
|
||||
---
|
||||
|
||||
## 多节点镜像批量导入（direct 模式）
|
||||
|
||||
`install.sh` 只在执行节点导入镜像。多节点集群建议写一个辅助脚本循环每个节点：
|
||||
|
||||
```bash
|
||||
NODES=(node1 node2 node3)
|
||||
for n in "${NODES[@]}"; do
|
||||
scp images/all-images.tar root@$n:/tmp/
|
||||
ssh root@$n 'sudo k3s ctr -n k8s.io image import /tmp/all-images.tar && rm /tmp/all-images.tar'
|
||||
done
|
||||
# 然后在控制平面节点跑:
|
||||
./install.sh --mode=direct --env=prod --skip-image-import
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 故障排查
|
||||
|
||||
### Pod 长时间 Pending
|
||||
```bash
|
||||
kubectl describe pod -n nuwax-rcoder-dev <pod-name>
|
||||
# 常见原因:
|
||||
# - ImagePullBackOff -> 节点 containerd 没有镜像 (direct 模式漏了某节点)
|
||||
# - 长时间 ContainerCreating -> Longhorn / JuiceFS CSI 未就绪
|
||||
```
|
||||
|
||||
### JuiceFS 挂载失败
|
||||
```bash
|
||||
kubectl logs -n kube-system -l app=juicefs-csi-driver-node --tail=50
|
||||
# 检查 postgresql / minio 是否就绪
|
||||
kubectl get pods -n nuwax-rcoder-dev -l app=postgresql
|
||||
kubectl get pods -n nuwax-rcoder-dev -l app=minio
|
||||
```
|
||||
|
||||
### Longhorn 无法启动
|
||||
```bash
|
||||
# 常见原因: open-iscsi 未安装
|
||||
ssh <node> "sudo apt install -y open-iscsi && sudo systemctl enable --now iscsid"
|
||||
kubectl delete pod -n longhorn-system -l app=longhorn-manager --force
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 版本信息
|
||||
|
||||
本 bundle 固定了以下版本：
|
||||
- RCoder: (由构建时的 git tag 决定, 见 `images/images.txt`)
|
||||
- PostgreSQL: 16-alpine
|
||||
- MinIO: 固定 release tag
|
||||
- JuiceFS Mount: ce-v1.3.1
|
||||
- JuiceFS CSI: v0.31.3 (官方 deploy/k8s.yaml; helm 仓库已停更)
|
||||
- Longhorn: v1.7.2
|
||||
|
||||
升级任意组件需要**重新构建 bundle**（`make k8s-offline-bundle`），本离线包不能在线更新。
|
||||
52
qiming-rcoder/k8s/offline/images.txt
Normal file
52
qiming-rcoder/k8s/offline/images.txt
Normal file
@@ -0,0 +1,52 @@
|
||||
# ============================================================================
|
||||
# 离线部署镜像清单 —— 所有 make k8s-offline-bundle 会拉取 + 打包的镜像
|
||||
#
|
||||
# 格式: 一行一个 <repo>:<tag>; # 开头为注释, 空行忽略
|
||||
# 原则: 版本必须 pin, 避免离线包随时间漂移
|
||||
#
|
||||
# 升级 Longhorn / JuiceFS CSI 时, 同步更新本文件 + longhorn-$VERSION.yaml
|
||||
# ============================================================================
|
||||
|
||||
# ==== RCoder 自有镜像 ====
|
||||
nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder:latest
|
||||
nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder-agent-runner:latest
|
||||
|
||||
# ==== 存储层 ====
|
||||
postgres:16-alpine
|
||||
minio/minio:RELEASE.2024-12-18T13-15-44Z
|
||||
minio/mc:RELEASE.2024-11-21T17-21-54Z
|
||||
busybox:1.36
|
||||
|
||||
# ==== JuiceFS CE Mount ====
|
||||
juicedata/mount:ce-v1.3.1
|
||||
|
||||
# ==== JuiceFS CSI Driver v0.31.3 (通过官方 deploy/k8s.yaml 安装) ====
|
||||
# 清单来源: curl https://raw.githubusercontent.com/juicedata/juicefs-csi-driver/v0.31.3/deploy/k8s.yaml | grep image:
|
||||
juicedata/juicefs-csi-driver:v0.31.3
|
||||
juicedata/csi-dashboard:v0.31.3
|
||||
|
||||
# ==== JuiceFS CSI sidecars (版本由 juicefs-csi k8s.yaml 决定) ====
|
||||
registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0
|
||||
registry.k8s.io/sig-storage/csi-provisioner:v3.6.0
|
||||
registry.k8s.io/sig-storage/csi-resizer:v1.9.0
|
||||
registry.k8s.io/sig-storage/livenessprobe:v2.11.0
|
||||
|
||||
# ==== Longhorn v1.7.2 核心组件 ====
|
||||
# 清单来源: curl https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn-images.txt
|
||||
# 升级时必须重新拉取 longhorn-images.txt 并同步 tag
|
||||
longhornio/longhorn-manager:v1.7.2
|
||||
longhornio/longhorn-engine:v1.7.2
|
||||
longhornio/longhorn-ui:v1.7.2
|
||||
longhornio/longhorn-instance-manager:v1.7.2
|
||||
longhornio/longhorn-share-manager:v1.7.2
|
||||
longhornio/backing-image-manager:v1.7.2
|
||||
longhornio/longhorn-cli:v1.7.2
|
||||
longhornio/support-bundle-kit:v0.0.45
|
||||
|
||||
# ==== Longhorn CSI sidecars (独立于 Longhorn 主镜像, 版本随 Longhorn 升级变化) ====
|
||||
longhornio/csi-attacher:v4.7.0
|
||||
longhornio/csi-provisioner:v4.0.1-20241007
|
||||
longhornio/csi-resizer:v1.12.0
|
||||
longhornio/csi-snapshotter:v7.0.2-20241007
|
||||
longhornio/csi-node-driver-registrar:v2.12.0
|
||||
longhornio/livenessprobe:v2.14.0
|
||||
277
qiming-rcoder/k8s/offline/install.sh
Normal file
277
qiming-rcoder/k8s/offline/install.sh
Normal file
@@ -0,0 +1,277 @@
|
||||
#!/bin/bash
|
||||
# ============================================================================
|
||||
# RCoder 离线部署脚本
|
||||
#
|
||||
# 使用场景: 政企内网, 无外网访问
|
||||
#
|
||||
# 运行流程:
|
||||
# 1. 将镜像导入到节点 containerd (direct 模式) 或推到私有 registry (registry 模式)
|
||||
# 2. 安装 Longhorn 存储 (可跳过)
|
||||
# 3. 安装 JuiceFS CSI Driver (可跳过)
|
||||
# 4. 安装 RCoder Helm chart
|
||||
#
|
||||
# 依赖 (离线机器上必须预装):
|
||||
# - kubectl (配置好 kubeconfig)
|
||||
# - helm 3.x
|
||||
# - docker (registry 模式) 或 k3s/nerdctl (direct 模式)
|
||||
# ============================================================================
|
||||
set -e
|
||||
|
||||
# ---- 默认参数 ----
|
||||
MODE="direct" # direct | registry
|
||||
REGISTRY="" # registry 模式下必填, 例: harbor.internal
|
||||
THIRDPARTY_REGISTRY="" # 第三方镜像目标 registry, 留空同 REGISTRY
|
||||
ENV="dev" # dev | prod
|
||||
RELEASE_NAME="" # 留空 => rcoder-$ENV
|
||||
SKIP_LONGHORN=0
|
||||
SKIP_JUICEFS_CSI=0
|
||||
SKIP_IMAGE_IMPORT=0 # 调试用: 跳过镜像导入
|
||||
NAMESPACE="" # 留空 => nuwax-rcoder-$ENV
|
||||
|
||||
# ---- 颜色 ----
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; NC='\033[0m'
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
用法: $0 [选项]
|
||||
|
||||
选项:
|
||||
--mode=MODE direct (默认) 或 registry
|
||||
--registry=URL 私有 registry 地址 (registry 模式必填)
|
||||
--thirdparty-registry=URL 第三方镜像目标 (留空同 --registry)
|
||||
--env=ENV dev (默认) 或 prod
|
||||
--release=NAME Helm release 名 (默认 rcoder-\$ENV)
|
||||
--namespace=NS 目标 namespace (默认 nuwax-rcoder-\$ENV)
|
||||
--skip-longhorn 跳过 Longhorn 安装 (集群已有 CSI 时用)
|
||||
--skip-juicefs-csi 跳过 JuiceFS CSI 安装
|
||||
--skip-image-import 跳过镜像导入 (调试/镜像已就位时用)
|
||||
-h|--help 显示本帮助
|
||||
|
||||
示例:
|
||||
# 最常见: 断网机器直接用节点 containerd 本地缓存
|
||||
$0 --mode=direct --env=dev
|
||||
|
||||
# 有私有 registry: 先推送再部署
|
||||
$0 --mode=registry --registry=harbor.internal/rcoder --env=prod
|
||||
|
||||
# 集群已有存储, 只装 RCoder 本身
|
||||
$0 --mode=direct --env=dev --skip-longhorn --skip-juicefs-csi
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
# ---- 参数解析 ----
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--mode=*) MODE="${1#*=}" ;;
|
||||
--registry=*) REGISTRY="${1#*=}" ;;
|
||||
--thirdparty-registry=*) THIRDPARTY_REGISTRY="${1#*=}" ;;
|
||||
--env=*) ENV="${1#*=}" ;;
|
||||
--release=*) RELEASE_NAME="${1#*=}" ;;
|
||||
--namespace=*) NAMESPACE="${1#*=}" ;;
|
||||
--skip-longhorn) SKIP_LONGHORN=1 ;;
|
||||
--skip-juicefs-csi) SKIP_JUICEFS_CSI=1 ;;
|
||||
--skip-image-import) SKIP_IMAGE_IMPORT=1 ;;
|
||||
-h|--help) usage ;;
|
||||
*) echo -e "${RED}未知参数: $1${NC}"; usage ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
[ -z "$RELEASE_NAME" ] && RELEASE_NAME="rcoder-$ENV"
|
||||
[ -z "$NAMESPACE" ] && NAMESPACE="nuwax-rcoder-$ENV"
|
||||
[ -z "$THIRDPARTY_REGISTRY" ] && THIRDPARTY_REGISTRY="$REGISTRY"
|
||||
|
||||
if [ "$MODE" = "registry" ] && [ -z "$REGISTRY" ]; then
|
||||
echo -e "${RED}错误: registry 模式必须指定 --registry=URL${NC}"; exit 1
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}=========================================="
|
||||
echo " RCoder 离线部署"
|
||||
echo "==========================================${NC}"
|
||||
echo " 模式: $MODE"
|
||||
echo " 环境: $ENV"
|
||||
echo " Release: $RELEASE_NAME"
|
||||
echo " Namespace: $NAMESPACE"
|
||||
[ "$MODE" = "registry" ] && echo " Registry: $REGISTRY"
|
||||
[ "$MODE" = "registry" ] && echo " 3rd-party: $THIRDPARTY_REGISTRY"
|
||||
echo " Longhorn: $([ $SKIP_LONGHORN = 1 ] && echo '跳过' || echo '安装')"
|
||||
echo " JuiceFS CSI: $([ $SKIP_JUICEFS_CSI = 1 ] && echo '跳过' || echo '安装')"
|
||||
echo ""
|
||||
|
||||
# ---- 前置检查 ----
|
||||
for cmd in kubectl helm; do
|
||||
if ! command -v $cmd &>/dev/null; then
|
||||
echo -e "${RED}错误: 未安装 $cmd${NC}"; exit 1
|
||||
fi
|
||||
done
|
||||
if ! kubectl cluster-info &>/dev/null; then
|
||||
echo -e "${RED}错误: 无法连接 K8s 集群, 检查 kubeconfig${NC}"; exit 1
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# 步骤 1: 导入镜像
|
||||
# ============================================================================
|
||||
if [ "$SKIP_IMAGE_IMPORT" = "0" ]; then
|
||||
echo -e "${GREEN}[1/4] 导入镜像...${NC}"
|
||||
IMAGES_TAR="$SCRIPT_DIR/images/all-images.tar"
|
||||
if [ ! -f "$IMAGES_TAR" ]; then
|
||||
echo -e "${RED}错误: 未找到 $IMAGES_TAR${NC}"; exit 1
|
||||
fi
|
||||
|
||||
case "$MODE" in
|
||||
direct)
|
||||
# 优先使用 k3s ctr, 回退到 nerdctl, 再回退到 ctr
|
||||
if command -v k3s &>/dev/null; then
|
||||
echo " 使用 k3s ctr 导入 (namespace: k8s.io)..."
|
||||
sudo k3s ctr -n k8s.io image import "$IMAGES_TAR"
|
||||
elif command -v nerdctl &>/dev/null; then
|
||||
echo " 使用 nerdctl 导入..."
|
||||
sudo nerdctl --namespace k8s.io load -i "$IMAGES_TAR"
|
||||
elif command -v ctr &>/dev/null; then
|
||||
echo " 使用 ctr 导入 (namespace: k8s.io)..."
|
||||
sudo ctr -n k8s.io image import "$IMAGES_TAR"
|
||||
else
|
||||
echo -e "${RED}错误: direct 模式需要 k3s / nerdctl / ctr 其中之一${NC}"; exit 1
|
||||
fi
|
||||
echo -e "${GREEN} ✅ 镜像已导入节点 containerd${NC}"
|
||||
echo -e "${YELLOW} ⚠️ 如果是多节点集群, 需要在每个节点上运行 ctr image import${NC}"
|
||||
;;
|
||||
registry)
|
||||
if ! command -v docker &>/dev/null; then
|
||||
echo -e "${RED}错误: registry 模式需要 docker${NC}"; exit 1
|
||||
fi
|
||||
echo " 加载到 docker daemon..."
|
||||
docker load -i "$IMAGES_TAR"
|
||||
echo " 重打 tag 并推送到 $REGISTRY..."
|
||||
bash "$SCRIPT_DIR/rewrite-registry.sh" \
|
||||
--registry "$REGISTRY" \
|
||||
--thirdparty-registry "$THIRDPARTY_REGISTRY" \
|
||||
--images-file "$SCRIPT_DIR/images/images.txt"
|
||||
;;
|
||||
esac
|
||||
else
|
||||
echo -e "${YELLOW}[1/4] 跳过镜像导入 (--skip-image-import)${NC}"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# 步骤 2: 安装 Longhorn
|
||||
# ============================================================================
|
||||
echo -e "${GREEN}[2/4] Longhorn...${NC}"
|
||||
if [ "$SKIP_LONGHORN" = "1" ]; then
|
||||
echo " 跳过 (--skip-longhorn)"
|
||||
elif kubectl get sc longhorn &>/dev/null; then
|
||||
echo -e "${GREEN} ✅ 已安装${NC}"
|
||||
else
|
||||
LONGHORN_YAML=$(ls "$SCRIPT_DIR/longhorn/longhorn-"*.yaml 2>/dev/null | head -1)
|
||||
if [ -z "$LONGHORN_YAML" ]; then
|
||||
echo -e "${RED}错误: 未找到 longhorn/longhorn-*.yaml${NC}"; exit 1
|
||||
fi
|
||||
|
||||
# registry 模式: 先把 manifest 里的 longhornio/* 改成 $THIRDPARTY_REGISTRY/longhornio/*
|
||||
APPLY_YAML="$LONGHORN_YAML"
|
||||
if [ "$MODE" = "registry" ]; then
|
||||
APPLY_YAML="/tmp/longhorn-rewritten.yaml"
|
||||
sed -E "s#(image:\s*)longhornio/#\1${THIRDPARTY_REGISTRY}/longhornio/#g" \
|
||||
"$LONGHORN_YAML" > "$APPLY_YAML"
|
||||
fi
|
||||
|
||||
echo " 部署 Longhorn..."
|
||||
kubectl apply -f "$APPLY_YAML"
|
||||
echo " 等待 Longhorn manager 就绪 (最长 5 分钟)..."
|
||||
kubectl wait --for=condition=ready pod -l app=longhorn-manager \
|
||||
-n longhorn-system --timeout=300s || true
|
||||
echo -e "${GREEN} ✅ Longhorn 已安装${NC}"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# 步骤 3: 安装 JuiceFS CSI Driver (使用官方 deploy/k8s.yaml, 不走 helm)
|
||||
# ============================================================================
|
||||
echo -e "${GREEN}[3/4] JuiceFS CSI Driver...${NC}"
|
||||
if [ "$SKIP_JUICEFS_CSI" = "1" ]; then
|
||||
echo " 跳过 (--skip-juicefs-csi)"
|
||||
elif kubectl get ds -n kube-system juicefs-csi-driver-node &>/dev/null \
|
||||
|| kubectl get ds -n kube-system juicefs-csi-node &>/dev/null; then
|
||||
echo -e "${GREEN} ✅ 已安装${NC}"
|
||||
else
|
||||
JUICEFS_YAML=$(ls "$SCRIPT_DIR/juicefs-csi/juicefs-csi-"*.yaml 2>/dev/null | head -1)
|
||||
if [ -z "$JUICEFS_YAML" ]; then
|
||||
echo -e "${RED}错误: 未找到 juicefs-csi/juicefs-csi-*.yaml${NC}"; exit 1
|
||||
fi
|
||||
|
||||
# registry 模式: manifest 里所有 image 前面加上私有 registry 前缀
|
||||
APPLY_YAML="$JUICEFS_YAML"
|
||||
if [ "$MODE" = "registry" ]; then
|
||||
APPLY_YAML="/tmp/juicefs-csi-rewritten.yaml"
|
||||
sed -E \
|
||||
-e "s#(image:\s*)juicedata/#\1${REGISTRY}/juicedata/#g" \
|
||||
-e "s#(image:\s*)registry\.k8s\.io/sig-storage/#\1${THIRDPARTY_REGISTRY}/sig-storage/#g" \
|
||||
"$JUICEFS_YAML" > "$APPLY_YAML"
|
||||
fi
|
||||
|
||||
echo " 部署 JuiceFS CSI ($(basename $JUICEFS_YAML))..."
|
||||
kubectl apply -f "$APPLY_YAML"
|
||||
|
||||
# 设置 mount 镜像 (CE 版本, 与 RCoder 使用的 juicefs 客户端对齐)
|
||||
MOUNT_IMG="${THIRDPARTY_REGISTRY:+$THIRDPARTY_REGISTRY/}juicedata/mount:ce-v1.3.1"
|
||||
for ds in juicefs-csi-driver-node juicefs-csi-node; do
|
||||
kubectl -n kube-system get daemonset/$ds &>/dev/null && \
|
||||
kubectl -n kube-system set env daemonset/$ds -c juicefs-plugin \
|
||||
JUICEFS_CE_MOUNT_IMAGE="$MOUNT_IMG" \
|
||||
JUICEFS_MOUNT_IMAGE="$MOUNT_IMG" >/dev/null 2>&1 || true
|
||||
done
|
||||
kubectl -n kube-system get statefulset/juicefs-csi-controller &>/dev/null && \
|
||||
kubectl -n kube-system set env statefulset/juicefs-csi-controller -c juicefs-plugin \
|
||||
JUICEFS_CE_MOUNT_IMAGE="$MOUNT_IMG" \
|
||||
JUICEFS_MOUNT_IMAGE="$MOUNT_IMG" >/dev/null 2>&1 || true
|
||||
|
||||
echo " 等待 CSI 就绪 (最长 2 分钟)..."
|
||||
for label in "app=juicefs-csi-driver-node" "app=juicefs-csi-node"; do
|
||||
kubectl wait --for=condition=ready pod -l "$label" -n kube-system --timeout=120s 2>/dev/null && break || true
|
||||
done
|
||||
echo -e "${GREEN} ✅ JuiceFS CSI 已安装${NC}"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# 步骤 4: 安装 RCoder
|
||||
# ============================================================================
|
||||
echo -e "${GREEN}[4/4] 安装 RCoder...${NC}"
|
||||
|
||||
RCODER_CHART=$(ls "$SCRIPT_DIR/charts/rcoder-"*.tgz 2>/dev/null | head -1)
|
||||
if [ -z "$RCODER_CHART" ]; then
|
||||
echo -e "${RED}错误: 未找到 charts/rcoder-*.tgz${NC}"; exit 1
|
||||
fi
|
||||
|
||||
VALUES_ARGS=(-f "$SCRIPT_DIR/values-$ENV.yaml")
|
||||
if [ "$MODE" = "registry" ]; then
|
||||
VALUES_ARGS+=(
|
||||
--set "global.imageRegistry=$REGISTRY"
|
||||
--set "global.thirdPartyRegistry=$THIRDPARTY_REGISTRY"
|
||||
)
|
||||
fi
|
||||
|
||||
echo " helm upgrade --install $RELEASE_NAME ..."
|
||||
helm upgrade --install "$RELEASE_NAME" "$RCODER_CHART" \
|
||||
--namespace "$NAMESPACE" \
|
||||
--create-namespace \
|
||||
"${VALUES_ARGS[@]}"
|
||||
|
||||
echo -e "${GREEN} 等待 rcoder deployment 就绪 (最长 5 分钟)...${NC}"
|
||||
kubectl rollout status deploy/rcoder -n "$NAMESPACE" --timeout=300s || true
|
||||
|
||||
# ============================================================================
|
||||
# 收尾
|
||||
# ============================================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}=========================================="
|
||||
echo " 部署完成!"
|
||||
echo "==========================================${NC}"
|
||||
kubectl get pods -n "$NAMESPACE"
|
||||
echo ""
|
||||
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}' 2>/dev/null || echo "<node-ip>")
|
||||
NODE_PORT=$(kubectl get svc rcoder -n "$NAMESPACE" -o jsonpath='{.spec.ports[0].nodePort}' 2>/dev/null || echo "<nodeport>")
|
||||
echo "访问: curl http://${NODE_IP}:${NODE_PORT}/health"
|
||||
echo "日志: kubectl logs -n $NAMESPACE -l app=rcoder -f"
|
||||
echo "卸载: helm uninstall $RELEASE_NAME --namespace $NAMESPACE"
|
||||
73
qiming-rcoder/k8s/offline/rewrite-registry.sh
Normal file
73
qiming-rcoder/k8s/offline/rewrite-registry.sh
Normal file
@@ -0,0 +1,73 @@
|
||||
#!/bin/bash
|
||||
# ============================================================================
|
||||
# 把 docker daemon 里已加载的镜像 re-tag 并推送到私有 registry
|
||||
#
|
||||
# 输入: images.txt (一行一个镜像)
|
||||
# 行为:
|
||||
# - RCoder 自有镜像 (nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/*)
|
||||
# -> 推到 $REGISTRY/{name}:{tag}
|
||||
# - 其他 (postgres / minio / juicedata / sig-storage / longhornio 等)
|
||||
# -> 推到 $THIRDPARTY_REGISTRY/{repo}:{tag}
|
||||
#
|
||||
# 依赖: docker (镜像已 load 在本地 daemon)
|
||||
# ============================================================================
|
||||
set -e
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; NC='\033[0m'
|
||||
|
||||
REGISTRY=""
|
||||
THIRDPARTY_REGISTRY=""
|
||||
IMAGES_FILE=""
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--registry) REGISTRY="$2"; shift 2 ;;
|
||||
--thirdparty-registry) THIRDPARTY_REGISTRY="$2"; shift 2 ;;
|
||||
--images-file) IMAGES_FILE="$2"; shift 2 ;;
|
||||
*) echo "未知参数: $1"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[ -z "$REGISTRY" ] && { echo -e "${RED}--registry 必填${NC}"; exit 1; }
|
||||
[ -z "$THIRDPARTY_REGISTRY" ] && THIRDPARTY_REGISTRY="$REGISTRY"
|
||||
[ -z "$IMAGES_FILE" ] && { echo -e "${RED}--images-file 必填${NC}"; exit 1; }
|
||||
[ ! -f "$IMAGES_FILE" ] && { echo -e "${RED}未找到 $IMAGES_FILE${NC}"; exit 1; }
|
||||
|
||||
# RCoder 自有镜像的 registry 前缀 (用于识别 + 剥离)
|
||||
RCODER_OWN_PREFIX="nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/"
|
||||
|
||||
echo -e "${GREEN}Re-tag + push 镜像到私有 registry${NC}"
|
||||
echo " 自有镜像 -> $REGISTRY/"
|
||||
echo " 第三方 -> $THIRDPARTY_REGISTRY/"
|
||||
|
||||
while IFS= read -r line; do
|
||||
# 跳过注释和空行
|
||||
line="${line%%#*}"; line="${line##[[:space:]]}"; line="${line%%[[:space:]]}"
|
||||
[ -z "$line" ] && continue
|
||||
|
||||
SRC="$line"
|
||||
DST=""
|
||||
|
||||
if [[ "$SRC" == "$RCODER_OWN_PREFIX"* ]]; then
|
||||
# 自有镜像: 剥离前缀, 推到 REGISTRY
|
||||
STRIPPED="${SRC#$RCODER_OWN_PREFIX}"
|
||||
DST="$REGISTRY/$STRIPPED"
|
||||
else
|
||||
# 第三方: 保留完整 repo 路径, 推到 THIRDPARTY_REGISTRY
|
||||
# 处理 registry.k8s.io/sig-storage/xxx —— 去掉 registry.k8s.io/ 保留 sig-storage/xxx
|
||||
REPO="$SRC"
|
||||
case "$REPO" in
|
||||
registry.k8s.io/*) REPO="${REPO#registry.k8s.io/}" ;;
|
||||
docker.io/*) REPO="${REPO#docker.io/}" ;;
|
||||
quay.io/*) REPO="${REPO#quay.io/}" ;;
|
||||
esac
|
||||
DST="$THIRDPARTY_REGISTRY/$REPO"
|
||||
fi
|
||||
|
||||
echo -e "${YELLOW} $SRC${NC}"
|
||||
echo " => $DST"
|
||||
docker tag "$SRC" "$DST"
|
||||
docker push "$DST"
|
||||
done < "$IMAGES_FILE"
|
||||
|
||||
echo -e "${GREEN}✅ 所有镜像已推送到私有 registry${NC}"
|
||||
110
qiming-rcoder/k8s/register2/README.md
Normal file
110
qiming-rcoder/k8s/register2/README.md
Normal file
@@ -0,0 +1,110 @@
|
||||
## registry2（本地私有镜像仓库）
|
||||
|
||||
本目录用 docker compose 起一个 `registry:2` 容器，作为局域网内的镜像中转站——既供 docker push 测试用，也可配到 k3s 节点作首选 mirror。
|
||||
|
||||
### 地址
|
||||
- **私有仓库**：`192.168.32.228:5000`（本机启动后局域网可达）
|
||||
- **Web UI**：`http://192.168.32.228:5080`（浏览器访问；用于查看仓库/镜像/Tag 等）
|
||||
|
||||
### 启动/停止
|
||||
|
||||
```bash
|
||||
cd /home/swufe/gitworkspace/rcoder/k8s/register2
|
||||
docker compose up -d
|
||||
docker compose ps
|
||||
docker compose logs -f
|
||||
docker compose down
|
||||
```
|
||||
|
||||
数据落地在同目录下的 `./data/` （compose 已声明 volume）。
|
||||
|
||||
---
|
||||
|
||||
## 作为 k3s 的镜像 mirror
|
||||
|
||||
### 架构
|
||||
|
||||
```
|
||||
k3s/containerd (两个节点)
|
||||
│
|
||||
├──(1st) http://192.168.32.228:5000 ← registry2 (本目录, 私有推送)
|
||||
├──(2nd) http://192.168.32.228:5002 ← registry-cache (pull-through, 可选)
|
||||
├──(3rd) https://docker.m.daocloud.io / ... ← 公网 cn mirror 回退
|
||||
└──(4th) https://*.docker.io / registry.k8s.io (默认兜底)
|
||||
```
|
||||
|
||||
containerd 按顺序尝试 endpoint，404 自动回退到下一个。
|
||||
|
||||
### 一键配置两个节点
|
||||
|
||||
```bash
|
||||
# 1. 启动 registry2
|
||||
cd k8s/register2 && docker compose up -d
|
||||
|
||||
# 2. 给每个 k3s 节点配 mirror (用 REGISTRY_HOST 环境变量)
|
||||
sudo REGISTRY_HOST=192.168.32.228:5000 \
|
||||
k8s/scripts/install-k3s-registry-mirrors-cn.sh
|
||||
|
||||
# 脚本会:
|
||||
# - 备份已有 /etc/rancher/k3s/registries.yaml (如有)
|
||||
# - 写入新配置 (本地 registry 作首选, 公网 cn mirror 兜底)
|
||||
# - 自动 restart k3s 或 k3s-agent
|
||||
# - 用 crictl pull 验证 mirror 是否生效
|
||||
|
||||
# 3. 配置本机 docker daemon 允许 push 到 HTTP registry
|
||||
# 编辑 /etc/docker/daemon.json:
|
||||
# {
|
||||
# "insecure-registries": ["192.168.32.228:5000"]
|
||||
# }
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
|
||||
### 多个 registry 串联
|
||||
|
||||
如果同时有 registry2 (5000, 私有存储) 和 registry-cache (5002, pull-through 缓存)：
|
||||
|
||||
```bash
|
||||
sudo REGISTRY_HOST=192.168.32.228:5000,192.168.32.228:5002 \
|
||||
k8s/scripts/install-k3s-registry-mirrors-cn.sh
|
||||
```
|
||||
|
||||
### 验证
|
||||
|
||||
```bash
|
||||
# 1) 推个镜像进去看 5000 是否可用
|
||||
docker tag busybox:1.36 192.168.32.228:5000/busybox:test
|
||||
docker push 192.168.32.228:5000/busybox:test
|
||||
curl -s http://192.168.32.228:5000/v2/_catalog
|
||||
|
||||
# 2) k3s 节点上: 用 crictl 拉 (与 kubelet 一致的链路)
|
||||
sudo crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock \
|
||||
pull 192.168.32.228:5000/busybox:test
|
||||
|
||||
# 3) 验证 mirror 路由 (通过 docker.io 间接拉)
|
||||
sudo crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock \
|
||||
pull docker.io/rancher/mirrored-pause:3.6
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 与离线 bundle 的联动
|
||||
|
||||
配好 registry mirror 后，bundle 的 `--mode=registry` 可以直接推到 5000：
|
||||
|
||||
```bash
|
||||
tar xzf dist/rcoder-offline-*.tar.gz -C /tmp/offline-test
|
||||
cd /tmp/offline-test
|
||||
docker load -i images/all-images.tar
|
||||
bash rewrite-registry.sh \
|
||||
--registry 192.168.32.228:5000/rcoder \
|
||||
--thirdparty-registry 192.168.32.228:5000/thirdparty \
|
||||
--images-file images/images.txt
|
||||
|
||||
# 之后 helm install 时 k3s 会自动从本地 registry 拉, 走内网千兆
|
||||
helm install rcoder-offline k8s/helm/rcoder \
|
||||
-f k8s/helm/rcoder/values-dev.yaml \
|
||||
-f k8s/helm/rcoder/values-offline.yaml \
|
||||
--set global.imageRegistry=192.168.32.228:5000/rcoder \
|
||||
--set global.thirdPartyRegistry=192.168.32.228:5000/thirdparty \
|
||||
--namespace nuwax-rcoder-offline-test --create-namespace
|
||||
```
|
||||
26
qiming-rcoder/k8s/register2/docker-compose.yml
Normal file
26
qiming-rcoder/k8s/register2/docker-compose.yml
Normal file
@@ -0,0 +1,26 @@
|
||||
services:
|
||||
registry2:
|
||||
image: registry:2
|
||||
container_name: registry2
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "5000:5000"
|
||||
environment:
|
||||
REGISTRY_STORAGE_DELETE_ENABLED: "true"
|
||||
volumes:
|
||||
- ./data:/var/lib/registry
|
||||
- ./registry-config.yml:/etc/docker/registry/config.yml:ro
|
||||
|
||||
registry-ui:
|
||||
image: joxit/docker-registry-ui:latest
|
||||
container_name: registry2-ui
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "5080:80"
|
||||
environment:
|
||||
SINGLE_REGISTRY: "true"
|
||||
REGISTRY_TITLE: "registry2"
|
||||
# UI 在浏览器端直接请求 registry，这里必须是浏览器可达的地址
|
||||
REGISTRY_URL: "http://192.168.32.228:5000"
|
||||
depends_on:
|
||||
- registry2
|
||||
14
qiming-rcoder/k8s/register2/registry-config.yml
Normal file
14
qiming-rcoder/k8s/register2/registry-config.yml
Normal file
@@ -0,0 +1,14 @@
|
||||
version: 0.1
|
||||
log:
|
||||
fields:
|
||||
service: registry2
|
||||
storage:
|
||||
filesystem:
|
||||
rootdirectory: /var/lib/registry
|
||||
http:
|
||||
addr: :5000
|
||||
headers:
|
||||
Access-Control-Allow-Origin: ["*"]
|
||||
Access-Control-Allow-Methods: ["HEAD", "GET", "OPTIONS", "DELETE"]
|
||||
Access-Control-Allow-Headers: ["Authorization", "Accept", "Cache-Control", "Content-Type", "Origin", "User-Agent"]
|
||||
Access-Control-Expose-Headers: ["Docker-Content-Digest"]
|
||||
101
qiming-rcoder/k8s/scripts/deploy-juicefs.sh
Normal file
101
qiming-rcoder/k8s/scripts/deploy-juicefs.sh
Normal file
@@ -0,0 +1,101 @@
|
||||
#!/bin/bash
|
||||
# ============================================================
|
||||
# JuiceFS + MinIO + PostgreSQL 部署脚本
|
||||
# 使用 Kustomize overlay (dev / prod) 部署，避免集群级资源冲突
|
||||
# 用法 (可在任意目录运行):
|
||||
# ENV=dev k8s/scripts/deploy-juicefs.sh # 部署到 nuwax-rcoder-dev（默认）
|
||||
# ENV=prod k8s/scripts/deploy-juicefs.sh # 部署到 nuwax-rcoder-prod
|
||||
# ============================================================
|
||||
|
||||
set -e
|
||||
|
||||
ENV="${ENV:-dev}"
|
||||
|
||||
case "$ENV" in
|
||||
dev|prod) ;;
|
||||
*)
|
||||
echo "Error: ENV 必须是 dev 或 prod，当前值: $ENV"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# 路径解析：脚本位于 k8s/scripts/，manifests 位于 k8s/manifests/
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
K8S_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
|
||||
NAMESPACE="${NAMESPACE:-nuwax-rcoder-${ENV}}"
|
||||
MANIFESTS_DIR="${K8S_ROOT}/manifests"
|
||||
OVERLAY_DIR="${MANIFESTS_DIR}/overlays/${ENV}"
|
||||
|
||||
echo "=========================================="
|
||||
echo " RCoder - JuiceFS 存储部署 (${ENV})"
|
||||
echo " namespace: ${NAMESPACE}"
|
||||
echo " overlay: ${OVERLAY_DIR}"
|
||||
echo "=========================================="
|
||||
|
||||
# 检查 kubectl
|
||||
if ! command -v kubectl &> /dev/null; then
|
||||
echo "Error: kubectl not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 检查 kubectl kustomize 插件
|
||||
if ! kubectl kustomize --help &> /dev/null; then
|
||||
echo "Error: kubectl kustomize not found (需要 kubectl 1.14+)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 部署 JuiceFS CSI Driver
|
||||
echo ""
|
||||
echo "[1/6] 检查 JuiceFS CSI Driver..."
|
||||
if ! kubectl get daemonset juicefs-csi-driver-node -n kube-system &> /dev/null; then
|
||||
echo "JuiceFS CSI Driver 未部署，是否部署? (y/n)"
|
||||
read -r response
|
||||
if [[ "$response" =~ ^[Yy]$ ]]; then
|
||||
helm repo add juicefs https://juicefs.github.io/charts || true
|
||||
helm repo update
|
||||
helm install juicefs-csi-driver juicefs/juicefs-csi-driver \
|
||||
--namespace kube-system \
|
||||
--set webhook.enabled=false
|
||||
echo "等待 JuiceFS CSI Driver 就绪..."
|
||||
kubectl rollout status daemonset/juicefs-csi-driver-node -n kube-system --timeout=120s
|
||||
fi
|
||||
fi
|
||||
|
||||
# 使用 Kustomize overlay 部署
|
||||
echo ""
|
||||
echo "[2/6] 使用 Kustomize overlay 部署..."
|
||||
kubectl apply -k "${OVERLAY_DIR}"
|
||||
|
||||
# 等待存储层就绪
|
||||
echo ""
|
||||
echo "[3/6] 等待 PostgreSQL 就绪..."
|
||||
kubectl wait --for=condition=ready pod -l app=postgresql -n ${NAMESPACE} --timeout=180s 2>/dev/null || echo "⚠️ PostgreSQL 等待超时"
|
||||
|
||||
echo ""
|
||||
echo "[4/6] 等待 MinIO 就绪..."
|
||||
kubectl wait --for=condition=ready pod -l app=minio -n ${NAMESPACE} --timeout=180s 2>/dev/null || echo "⚠️ MinIO 等待超时"
|
||||
|
||||
echo ""
|
||||
echo "[5/6] 等待 MinIO Bucket 初始化..."
|
||||
kubectl wait --for=condition=complete job/minio-init -n ${NAMESPACE} --timeout=120s 2>/dev/null || echo "⚠️ MinIO 初始化等待超时"
|
||||
|
||||
# 验证部署
|
||||
echo ""
|
||||
echo "[6/6] 验证部署状态..."
|
||||
echo ""
|
||||
echo "StorageClass:"
|
||||
kubectl get sc | grep juicefs || echo " (未找到)"
|
||||
|
||||
echo ""
|
||||
echo "PVC:"
|
||||
kubectl get pvc -n ${NAMESPACE}
|
||||
|
||||
echo ""
|
||||
echo "Pods:"
|
||||
kubectl get pods -n ${NAMESPACE}
|
||||
|
||||
echo ""
|
||||
echo "=========================================="
|
||||
echo " 部署完成！"
|
||||
echo "=========================================="
|
||||
192
qiming-rcoder/k8s/scripts/install-k3s-registry-mirrors-cn.sh
Normal file
192
qiming-rcoder/k8s/scripts/install-k3s-registry-mirrors-cn.sh
Normal file
@@ -0,0 +1,192 @@
|
||||
#!/usr/bin/env bash
|
||||
# 为 K3s 配置中国大陆常用的 containerd 镜像加速（docker.io / registry.k8s.io / ghcr.io / quay.io）
|
||||
# 用法（在 K3s 节点上）:
|
||||
# sudo k8s/scripts/install-k3s-registry-mirrors-cn.sh
|
||||
#
|
||||
# 可选: 本地内网 registry 作首选 mirror (命中时走千兆内网)
|
||||
# sudo REGISTRY_HOST=192.168.32.228:5000 k8s/scripts/install-k3s-registry-mirrors-cn.sh
|
||||
# # 多个 registry 用逗号隔开, 按顺序作为 endpoint:
|
||||
# sudo REGISTRY_HOST=192.168.32.228:5000,192.168.32.228:5002 k8s/scripts/install-k3s-registry-mirrors-cn.sh
|
||||
#
|
||||
# 可选：禁止回退到 Docker Hub / GHCR 官方地址（对已配置 mirrors 的仓库不再走官方 endpoint）
|
||||
# sudo env INSTALL_DISABLE_REGISTRY_FALLBACK=1 k8s/scripts/install-k3s-registry-mirrors-cn.sh
|
||||
#
|
||||
# 完成后会重启 k3s，短暂影响调度中的 Pod。
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m'
|
||||
|
||||
if [[ "${EUID:-}" -ne 0 ]]; then
|
||||
echo -e "${RED}请使用 root 运行: sudo $0${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
SRC="${SCRIPT_DIR}/k3s-registries-cn.yaml"
|
||||
DEST="/etc/rancher/k3s/registries.yaml"
|
||||
|
||||
if [[ ! -f "$SRC" ]]; then
|
||||
echo -e "${RED}找不到模板: $SRC${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
install -d /etc/rancher/k3s
|
||||
|
||||
# 备份现有配置 (如果有)
|
||||
if [[ -f "$DEST" ]]; then
|
||||
BACKUP="${DEST}.bak.$(date +%s)"
|
||||
cp -a "$DEST" "$BACKUP"
|
||||
echo -e "${YELLOW}已备份现有配置:${NC} $BACKUP"
|
||||
fi
|
||||
|
||||
# ---- 可选: 生成带本地 registry 的配置 ----
|
||||
if [[ -n "${REGISTRY_HOST:-}" ]]; then
|
||||
echo -e "${GREEN}使用本地 registry 前缀:${NC} $REGISTRY_HOST"
|
||||
|
||||
IFS=',' read -ra LOCAL_REGS <<< "$REGISTRY_HOST"
|
||||
|
||||
# 纯粹用 printf 累加真实换行 (不依赖命令替换保留 \n)
|
||||
_emit_endpoints() {
|
||||
for r in "${LOCAL_REGS[@]}"; do
|
||||
printf ' - "http://%s"\n' "$r"
|
||||
done
|
||||
}
|
||||
_emit_self_mirrors() {
|
||||
for r in "${LOCAL_REGS[@]}"; do
|
||||
printf ' "%s":\n endpoint:\n - "http://%s"\n' "$r" "$r"
|
||||
done
|
||||
}
|
||||
_emit_self_configs() {
|
||||
for r in "${LOCAL_REGS[@]}"; do
|
||||
printf ' "%s":\n tls:\n insecure_skip_verify: true\n' "$r"
|
||||
done
|
||||
}
|
||||
|
||||
{
|
||||
cat <<EOF
|
||||
# 由 install-k3s-registry-mirrors-cn.sh 生成 (REGISTRY_HOST=${REGISTRY_HOST})
|
||||
# 文档: https://docs.k3s.io/installation/private-registry
|
||||
mirrors:
|
||||
"docker.io":
|
||||
endpoint:
|
||||
EOF
|
||||
_emit_endpoints
|
||||
cat <<'EOF'
|
||||
- "https://docker.m.daocloud.io"
|
||||
- "https://docker.1ms.run"
|
||||
|
||||
"registry.k8s.io":
|
||||
endpoint:
|
||||
EOF
|
||||
_emit_endpoints
|
||||
cat <<'EOF'
|
||||
- "https://registry.aliyuncs.com/google_containers"
|
||||
- "https://k8s.m.daocloud.io"
|
||||
|
||||
"ghcr.io":
|
||||
endpoint:
|
||||
EOF
|
||||
_emit_endpoints
|
||||
cat <<'EOF'
|
||||
- "https://ghcr.m.daocloud.io"
|
||||
|
||||
"quay.io":
|
||||
endpoint:
|
||||
EOF
|
||||
_emit_endpoints
|
||||
cat <<'EOF'
|
||||
- "https://quay.m.daocloud.io"
|
||||
|
||||
"nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com":
|
||||
endpoint:
|
||||
EOF
|
||||
_emit_endpoints
|
||||
cat <<'EOF'
|
||||
- "https://nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com"
|
||||
|
||||
# 自指: 允许 "crictl pull <host>/foo" 走 plain HTTP
|
||||
EOF
|
||||
_emit_self_mirrors
|
||||
cat <<'EOF'
|
||||
|
||||
configs:
|
||||
EOF
|
||||
_emit_self_configs
|
||||
} > "$DEST"
|
||||
|
||||
else
|
||||
cp -a "$SRC" "$DEST"
|
||||
fi
|
||||
chmod 0644 "$DEST"
|
||||
|
||||
echo -e "${GREEN}已写入:${NC} $DEST"
|
||||
echo -e "${YELLOW}内容预览:${NC}"
|
||||
cat "$DEST"
|
||||
|
||||
# 可选：对已在 registries.yaml 中声明的仓库，不再回退到官方默认 endpoint（见 K3s 文档 Default Endpoint Fallback）
|
||||
if [[ "${INSTALL_DISABLE_REGISTRY_FALLBACK:-0}" == "1" ]]; then
|
||||
install -d /etc/rancher/k3s/config.yaml.d
|
||||
DROPIN="/etc/rancher/k3s/config.yaml.d/99-rcoder-disable-default-registry-endpoint.yaml"
|
||||
cat >"$DROPIN" <<'EOF'
|
||||
disable-default-registry-endpoint: true
|
||||
EOF
|
||||
chmod 0644 "$DROPIN"
|
||||
echo ""
|
||||
echo -e "${GREEN}已写入 K3s 配置片段:${NC} $DROPIN"
|
||||
echo -e "${YELLOW}（仅对 registries.yaml 里配置过 mirrors 的仓库生效；避免回退到被墙地址导致长时间卡住）${NC}"
|
||||
fi
|
||||
|
||||
if systemctl is-active --quiet k3s 2>/dev/null; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}正在重启 k3s 使镜像配置生效...${NC}"
|
||||
systemctl restart k3s
|
||||
echo -e "${GREEN}k3s 已重启。${NC}"
|
||||
elif systemctl is-active --quiet k3s-agent 2>/dev/null; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}正在重启 k3s-agent...${NC}"
|
||||
systemctl restart k3s-agent
|
||||
echo -e "${GREEN}k3s-agent 已重启。${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}未检测到 k3s / k3s-agent 服务处于 active，请手动重启:${NC}"
|
||||
echo " systemctl restart k3s"
|
||||
echo " # 或 agent 节点:"
|
||||
echo " systemctl restart k3s-agent"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo -e "${YELLOW}说明:${NC} ${GREEN}k3s ctr images pull${NC} 往往 ${RED}不会${NC} 使用 /etc/rancher/k3s/registries.yaml，"
|
||||
echo " 仍会直连 registry-1.docker.io，因此在国内容易误报「镜像加速没生效」。"
|
||||
echo -e " 与 kubelet/CRI 一致的正确验证方式是用 ${GREEN}crictl pull${NC}："
|
||||
|
||||
RUNTIME_SOCK=""
|
||||
for s in /run/k3s/containerd/containerd.sock /var/run/k3s/containerd/containerd.sock; do
|
||||
if [[ -S "$s" ]]; then
|
||||
RUNTIME_SOCK="$s"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ -n "$RUNTIME_SOCK" ]]; then
|
||||
echo ""
|
||||
echo -e "${GREEN}验证拉取（推荐）:${NC}"
|
||||
echo " sudo crictl --runtime-endpoint unix://${RUNTIME_SOCK} pull docker.io/rancher/mirrored-pause:3.6"
|
||||
if command -v crictl &>/dev/null; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}正在尝试自动拉取 pause 镜像（失败则忽略，你可手动执行上一行）...${NC}"
|
||||
if crictl --runtime-endpoint "unix://${RUNTIME_SOCK}" pull docker.io/rancher/mirrored-pause:3.6; then
|
||||
echo -e "${GREEN}crictl 拉取成功：镜像加速已对 CRI/kubelet 生效。${NC}"
|
||||
else
|
||||
echo -e "${RED}crictl 拉取失败，请把完整终端输出保存后排查（勿用 k3s ctr 判断）。${NC}"
|
||||
fi
|
||||
else
|
||||
echo -e "${YELLOW}未安装 crictl，可安装后再验证。${NC}"
|
||||
fi
|
||||
else
|
||||
echo ""
|
||||
echo -e "${GREEN}验证拉取（推荐）:${NC}"
|
||||
echo " sudo crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock pull docker.io/rancher/mirrored-pause:3.6"
|
||||
fi
|
||||
63
qiming-rcoder/k8s/scripts/k3s-registries-cn.yaml
Normal file
63
qiming-rcoder/k8s/scripts/k3s-registries-cn.yaml
Normal file
@@ -0,0 +1,63 @@
|
||||
# 复制到节点: /etc/rancher/k3s/registries.yaml 后执行: sudo systemctl restart k3s
|
||||
#
|
||||
# 注意: 「k3s ctr images pull」通常不走本文件；验证请用 crictl（与 kubelet 一致），例如:
|
||||
# sudo crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock pull docker.io/rancher/mirrored-pause:3.6
|
||||
#
|
||||
# 文档: https://docs.k3s.io/installation/private-registry
|
||||
#
|
||||
# ============================================================================
|
||||
# 可选: 内网本地 registry 加速
|
||||
# ----------------------------------------------------------------------------
|
||||
# 如果你在局域网起了 registry:2 (见 k8s/register2/docker-compose.yml), 建议把它
|
||||
# 作为首选 mirror —— 命中时走内网 (~千兆), 未命中自动回退到公网 cn mirror。
|
||||
#
|
||||
# 使用 install-k3s-registry-mirrors-cn.sh 时, 设置 REGISTRY_HOST=<ip>:<port> 会
|
||||
# 自动生成包含本地 registry 的完整配置; 或者手动把下方注释掉的 `- "http://..."`
|
||||
# 行取消注释并改成你的 registry 地址。
|
||||
# ============================================================================
|
||||
|
||||
mirrors:
|
||||
"docker.io":
|
||||
endpoint:
|
||||
# - "http://192.168.32.228:5000" # 本地私有 (按需 push 加速专用镜像)
|
||||
# - "http://192.168.32.228:5002" # 本地 pull-through 缓存
|
||||
- "https://docker.m.daocloud.io"
|
||||
- "https://docker.1ms.run"
|
||||
|
||||
"registry.k8s.io":
|
||||
endpoint:
|
||||
# - "http://192.168.32.228:5000"
|
||||
- "https://registry.aliyuncs.com/google_containers"
|
||||
- "https://k8s.m.daocloud.io"
|
||||
|
||||
"ghcr.io":
|
||||
endpoint:
|
||||
# - "http://192.168.32.228:5000"
|
||||
- "https://ghcr.m.daocloud.io"
|
||||
|
||||
"quay.io":
|
||||
endpoint:
|
||||
# - "http://192.168.32.228:5000"
|
||||
- "https://quay.m.daocloud.io"
|
||||
|
||||
# RCoder 自有阿里云镜像 —— 本地 registry 命中时走内网
|
||||
# "nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com":
|
||||
# endpoint:
|
||||
# - "http://192.168.32.228:5000"
|
||||
# - "https://nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com"
|
||||
|
||||
# 自指 mirror: 允许 "crictl pull 192.168.32.228:5000/foo" 直接走 plain HTTP
|
||||
# "192.168.32.228:5000":
|
||||
# endpoint:
|
||||
# - "http://192.168.32.228:5000"
|
||||
# "192.168.32.228:5002":
|
||||
# endpoint:
|
||||
# - "http://192.168.32.228:5002"
|
||||
|
||||
# configs:
|
||||
# "192.168.32.228:5000":
|
||||
# tls:
|
||||
# insecure_skip_verify: true
|
||||
# "192.168.32.228:5002":
|
||||
# tls:
|
||||
# insecure_skip_verify: true
|
||||
57
qiming-rcoder/k8s/scripts/test-chat.sh
Normal file
57
qiming-rcoder/k8s/scripts/test-chat.sh
Normal file
@@ -0,0 +1,57 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
NAMESPACE="${NAMESPACE:-nuwax-rcoder-dev}"
|
||||
|
||||
echo "=== Testing RCoder in K8s ==="
|
||||
|
||||
# 获取 NodePort
|
||||
NODE_PORT=$(kubectl get svc rcoder -n "${NAMESPACE}" -o jsonpath='{.spec.ports[0].nodePort}')
|
||||
echo "NodePort: $NODE_PORT"
|
||||
|
||||
# 获取节点 IP（优先使用第一个 IPv4，避免 dual-stack 返回多个地址导致 URL 拼接失败）
|
||||
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}' | awk '{print $1}')
|
||||
echo "Node IP: $NODE_IP"
|
||||
|
||||
BASE_URL="http://${NODE_IP}:${NODE_PORT}"
|
||||
|
||||
echo ""
|
||||
echo "Testing /health endpoint..."
|
||||
curl -s "${BASE_URL}/health" | jq .
|
||||
|
||||
echo ""
|
||||
echo "Testing /chat endpoint..."
|
||||
CHAT_RESP=$(curl -s -X POST "${BASE_URL}/chat" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"prompt": "hello"}')
|
||||
echo "${CHAT_RESP}" | jq .
|
||||
PROJECT_ID=$(echo "${CHAT_RESP}" | jq -r '.data.project_id // empty')
|
||||
|
||||
echo ""
|
||||
echo "Testing /computer/chat endpoint..."
|
||||
USER_ID="k8s-test-user"
|
||||
COMPUTER_RESP=$(curl -s -X POST "${BASE_URL}/computer/chat" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"user_id\":\"${USER_ID}\",\"prompt\":\"hello from k8s\"}")
|
||||
echo "${COMPUTER_RESP}" | jq .
|
||||
COMPUTER_PROJECT_ID=$(echo "${COMPUTER_RESP}" | jq -r '.data.project_id // empty')
|
||||
|
||||
echo ""
|
||||
echo "Verifying created pods by labels..."
|
||||
if [ -n "${PROJECT_ID}" ]; then
|
||||
kubectl get pods -n "${NAMESPACE}" -l "project_id=${PROJECT_ID}" -o wide
|
||||
fi
|
||||
kubectl get pods -n "${NAMESPACE}" -l "user_id=${USER_ID}" -o wide
|
||||
|
||||
echo ""
|
||||
echo "Verifying pod status API..."
|
||||
curl -s "${BASE_URL}/computer/pod/status?user_id=${USER_ID}" | jq .
|
||||
if [ -n "${PROJECT_ID}" ]; then
|
||||
curl -s "${BASE_URL}/computer/pod/status?project_id=${PROJECT_ID}" | jq .
|
||||
fi
|
||||
if [ -n "${COMPUTER_PROJECT_ID}" ]; then
|
||||
curl -s "${BASE_URL}/computer/pod/status?project_id=${COMPUTER_PROJECT_ID}" | jq .
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "=== Test complete ==="
|
||||
30
qiming-rcoder/k8s/undeploy.sh
Normal file
30
qiming-rcoder/k8s/undeploy.sh
Normal file
@@ -0,0 +1,30 @@
|
||||
#!/bin/bash
|
||||
# ============================================================
|
||||
# K8s 资源清理脚本 (Kustomize)
|
||||
# 用法:
|
||||
# ENV=dev ./undeploy.sh # 清理开发环境 (默认)
|
||||
# ENV=prod ./undeploy.sh # 清理生产环境
|
||||
# ============================================================
|
||||
|
||||
set -e
|
||||
|
||||
ENV="${ENV:-dev}"
|
||||
|
||||
case "$ENV" in
|
||||
dev|prod) ;;
|
||||
*)
|
||||
echo "Error: ENV 必须是 dev 或 prod，当前值: $ENV"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# 路径解析：支持从任意 cwd 调用
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
NAMESPACE="${NAMESPACE:-nuwax-rcoder-${ENV}}"
|
||||
OVERLAY_DIR="${SCRIPT_DIR}/manifests/overlays/${ENV}"
|
||||
|
||||
echo "=== 使用 Kustomize 清理 RCoder 资源 (${ENV} 环境, namespace: ${NAMESPACE}) ==="
|
||||
|
||||
kubectl delete -k "${OVERLAY_DIR}" --ignore-not-found
|
||||
|
||||
echo "=== RCoder 资源已清理 ==="
|
||||
Reference in New Issue
Block a user