添加qiming-rcoder模块
This commit is contained in:
215
qiming-rcoder/k8s/offline/README.md
Normal file
215
qiming-rcoder/k8s/offline/README.md
Normal file
@@ -0,0 +1,215 @@
|
||||
# RCoder 离线部署指南
|
||||
|
||||
本文档面向政企/内网客户:**目标机器无公网访问,所有镜像与依赖必须随部署包交付**。
|
||||
|
||||
---
|
||||
|
||||
## 目录内容(解压 `rcoder-offline-*.tar.gz` 后)
|
||||
|
||||
```
|
||||
.
|
||||
├── install.sh # 主安装脚本
|
||||
├── rewrite-registry.sh # registry 模式: re-tag + push 辅助脚本
|
||||
├── README.md # 本文件
|
||||
├── values-dev.yaml # RCoder dev 环境 values
|
||||
├── values-prod.yaml # RCoder prod 环境 values
|
||||
├── values-offline.yaml # 离线 registry 覆盖模板
|
||||
├── images/
|
||||
│ ├── all-images.tar # 所有镜像的 docker save 产物
|
||||
│ └── images.txt # 镜像清单
|
||||
├── charts/
|
||||
│ └── rcoder-*.tgz # RCoder Helm chart
|
||||
├── longhorn/
|
||||
│ └── longhorn-v*.yaml # Longhorn 全量 manifest
|
||||
├── juicefs-csi/
|
||||
│ └── juicefs-csi-v*.yaml # JuiceFS CSI Driver 官方 manifest
|
||||
└── manifests/ # 原始 Kustomize 资产 (参考, 不必使用)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 前置条件
|
||||
|
||||
在离线机器上,提前准备好:
|
||||
|
||||
| 组件 | 版本 | 说明 |
|
||||
|------|------|------|
|
||||
| K8s | 1.19+ | 已部署, kubeconfig 已配置 |
|
||||
| kubectl | 与集群匹配 | `kubectl cluster-info` 能通 |
|
||||
| helm | 3.x | 用于 `helm upgrade --install` |
|
||||
| docker | 最新 | **仅 registry 模式需要** |
|
||||
| k3s / nerdctl / ctr | 最新 | **仅 direct 模式需要** |
|
||||
| open-iscsi | 任意 | Longhorn 依赖 (Ubuntu: `apt install open-iscsi`) |
|
||||
|
||||
---
|
||||
|
||||
## 两种部署模式
|
||||
|
||||
### 模式 A: direct(推荐单节点 / 小集群)
|
||||
|
||||
直接把镜像 import 到节点 containerd,**无需私有 registry**。适合 k3s 单节点或 3-5 节点小集群。
|
||||
|
||||
多节点集群需要**在每个节点上依次执行**镜像导入。
|
||||
|
||||
```bash
|
||||
# 1. 解压 bundle (可通过 make k8s-offline-import 一步完成)
|
||||
tar xzf rcoder-offline-<version>-<arch>.tar.gz -C /opt/rcoder-offline
|
||||
cd /opt/rcoder-offline
|
||||
|
||||
# 2. 一键安装
|
||||
sudo ./install.sh --mode=direct --env=dev
|
||||
|
||||
# 生产环境
|
||||
sudo ./install.sh --mode=direct --env=prod \
|
||||
--set credentials.postgresql.password='<prod-pg-pass>' \
|
||||
--set credentials.minio.rootPassword='<prod-minio-pass>'
|
||||
```
|
||||
|
||||
> **多节点注意**: 每个节点都要执行 `sudo k3s ctr -n k8s.io image import images/all-images.tar`
|
||||
> 或把 bundle scp 到每个节点各跑一次 `--skip-juicefs-csi --skip-longhorn` 只做镜像导入。
|
||||
|
||||
### 模式 B: registry(推荐多节点 / 生产)
|
||||
|
||||
把镜像重新打 tag 推送到客户的私有 registry(Harbor / Nexus / 阿里云 ACR 内网实例),K8s 按常规流程拉取。
|
||||
|
||||
**前置: 集群节点能访问私有 registry,并配置好拉取凭据(如需要)。**
|
||||
|
||||
```bash
|
||||
# 1. 解压
|
||||
tar xzf rcoder-offline-<version>-<arch>.tar.gz -C /opt/rcoder-offline
|
||||
cd /opt/rcoder-offline
|
||||
|
||||
# 2. 如果私有 registry 需要认证, 先登录 docker daemon
|
||||
docker login harbor.internal
|
||||
|
||||
# 3. 安装 (一条命令完成推送+部署)
|
||||
./install.sh \
|
||||
--mode=registry \
|
||||
--registry=harbor.internal/rcoder \
|
||||
--thirdparty-registry=harbor.internal/thirdparty \
|
||||
--env=prod
|
||||
```
|
||||
|
||||
推送完成后,镜像会以如下命名出现在 registry:
|
||||
|
||||
| 原镜像 | 推送目标 |
|
||||
|--------|---------|
|
||||
| `nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder:latest` | `harbor.internal/rcoder/rcoder:latest` |
|
||||
| `postgres:16-alpine` | `harbor.internal/thirdparty/postgres:16-alpine` |
|
||||
| `longhornio/longhorn-manager:v1.7.2` | `harbor.internal/thirdparty/longhornio/longhorn-manager:v1.7.2` |
|
||||
| `registry.k8s.io/sig-storage/csi-provisioner:v5.1.0` | `harbor.internal/thirdparty/sig-storage/csi-provisioner:v5.1.0` |
|
||||
|
||||
如果集群需要 imagePullSecret:
|
||||
```bash
|
||||
kubectl -n nuwax-rcoder-prod create secret docker-registry harbor-pull \
|
||||
--docker-server=harbor.internal \
|
||||
--docker-username=<user> \
|
||||
--docker-password=<pass>
|
||||
# 重新安装时:
|
||||
./install.sh ... --set 'global.imagePullSecrets[0].name=harbor-pull'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 常见操作
|
||||
|
||||
### 查看部署状态
|
||||
```bash
|
||||
kubectl get pods -n nuwax-rcoder-dev
|
||||
kubectl get svc -n nuwax-rcoder-dev
|
||||
kubectl rollout status deploy/rcoder -n nuwax-rcoder-dev
|
||||
```
|
||||
|
||||
### 访问服务
|
||||
```bash
|
||||
# NodePort
|
||||
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')
|
||||
curl http://${NODE_IP}:30080/health # dev
|
||||
curl http://${NODE_IP}:30081/health # prod
|
||||
|
||||
# 或 port-forward
|
||||
kubectl port-forward -n nuwax-rcoder-dev svc/rcoder 8087:8087
|
||||
curl http://localhost:8087/health
|
||||
```
|
||||
|
||||
### 卸载
|
||||
```bash
|
||||
helm uninstall rcoder-dev --namespace nuwax-rcoder-dev
|
||||
|
||||
# 彻底删除 (包含 PVC)
|
||||
kubectl delete pvc --all -n nuwax-rcoder-dev
|
||||
kubectl delete namespace nuwax-rcoder-dev
|
||||
```
|
||||
|
||||
### 升级
|
||||
收到新 bundle 后直接重新跑 `install.sh`(`helm upgrade --install` 会自动升级):
|
||||
```bash
|
||||
./install.sh --mode=registry --registry=harbor.internal/rcoder --env=prod
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 跳过选项
|
||||
|
||||
| 参数 | 用途 |
|
||||
|------|------|
|
||||
| `--skip-longhorn` | 集群已有其他 CSI (Ceph/NFS/CSI-local-path),不装 Longhorn |
|
||||
| `--skip-juicefs-csi` | 集群已有 JuiceFS CSI 或用其他 RWX 方案 |
|
||||
| `--skip-image-import` | 镜像已手工导入,调试用 |
|
||||
|
||||
---
|
||||
|
||||
## 多节点镜像批量导入(direct 模式)
|
||||
|
||||
`install.sh` 只在执行节点导入镜像。多节点集群建议写一个辅助脚本循环每个节点:
|
||||
|
||||
```bash
|
||||
NODES=(node1 node2 node3)
|
||||
for n in "${NODES[@]}"; do
|
||||
scp images/all-images.tar root@$n:/tmp/
|
||||
ssh root@$n 'sudo k3s ctr -n k8s.io image import /tmp/all-images.tar && rm /tmp/all-images.tar'
|
||||
done
|
||||
# 然后在控制平面节点跑:
|
||||
./install.sh --mode=direct --env=prod --skip-image-import
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 故障排查
|
||||
|
||||
### Pod 长时间 Pending
|
||||
```bash
|
||||
kubectl describe pod -n nuwax-rcoder-dev <pod-name>
|
||||
# 常见原因:
|
||||
# - ImagePullBackOff -> 节点 containerd 没有镜像 (direct 模式漏了某节点)
|
||||
# - 长时间 ContainerCreating -> Longhorn / JuiceFS CSI 未就绪
|
||||
```
|
||||
|
||||
### JuiceFS 挂载失败
|
||||
```bash
|
||||
kubectl logs -n kube-system -l app=juicefs-csi-driver-node --tail=50
|
||||
# 检查 postgresql / minio 是否就绪
|
||||
kubectl get pods -n nuwax-rcoder-dev -l app=postgresql
|
||||
kubectl get pods -n nuwax-rcoder-dev -l app=minio
|
||||
```
|
||||
|
||||
### Longhorn 无法启动
|
||||
```bash
|
||||
# 常见原因: open-iscsi 未安装
|
||||
ssh <node> "sudo apt install -y open-iscsi && sudo systemctl enable --now iscsid"
|
||||
kubectl delete pod -n longhorn-system -l app=longhorn-manager --force
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 版本信息
|
||||
|
||||
本 bundle 固定了以下版本:
|
||||
- RCoder: (由构建时的 git tag 决定, 见 `images/images.txt`)
|
||||
- PostgreSQL: 16-alpine
|
||||
- MinIO: 固定 release tag
|
||||
- JuiceFS Mount: ce-v1.3.1
|
||||
- JuiceFS CSI: v0.31.3 (官方 deploy/k8s.yaml; helm 仓库已停更)
|
||||
- Longhorn: v1.7.2
|
||||
|
||||
升级任意组件需要**重新构建 bundle**(`make k8s-offline-bundle`),本离线包不能在线更新。
|
||||
52
qiming-rcoder/k8s/offline/images.txt
Normal file
52
qiming-rcoder/k8s/offline/images.txt
Normal file
@@ -0,0 +1,52 @@
|
||||
# ============================================================================
|
||||
# 离线部署镜像清单 —— 所有 make k8s-offline-bundle 会拉取 + 打包的镜像
|
||||
#
|
||||
# 格式: 一行一个 <repo>:<tag>; # 开头为注释, 空行忽略
|
||||
# 原则: 版本必须 pin, 避免离线包随时间漂移
|
||||
#
|
||||
# 升级 Longhorn / JuiceFS CSI 时, 同步更新本文件 + longhorn-$VERSION.yaml
|
||||
# ============================================================================
|
||||
|
||||
# ==== RCoder 自有镜像 ====
|
||||
nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder:latest
|
||||
nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/rcoder-agent-runner:latest
|
||||
|
||||
# ==== 存储层 ====
|
||||
postgres:16-alpine
|
||||
minio/minio:RELEASE.2024-12-18T13-15-44Z
|
||||
minio/mc:RELEASE.2024-11-21T17-21-54Z
|
||||
busybox:1.36
|
||||
|
||||
# ==== JuiceFS CE Mount ====
|
||||
juicedata/mount:ce-v1.3.1
|
||||
|
||||
# ==== JuiceFS CSI Driver v0.31.3 (通过官方 deploy/k8s.yaml 安装) ====
|
||||
# 清单来源: curl https://raw.githubusercontent.com/juicedata/juicefs-csi-driver/v0.31.3/deploy/k8s.yaml | grep image:
|
||||
juicedata/juicefs-csi-driver:v0.31.3
|
||||
juicedata/csi-dashboard:v0.31.3
|
||||
|
||||
# ==== JuiceFS CSI sidecars (版本由 juicefs-csi k8s.yaml 决定) ====
|
||||
registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0
|
||||
registry.k8s.io/sig-storage/csi-provisioner:v3.6.0
|
||||
registry.k8s.io/sig-storage/csi-resizer:v1.9.0
|
||||
registry.k8s.io/sig-storage/livenessprobe:v2.11.0
|
||||
|
||||
# ==== Longhorn v1.7.2 核心组件 ====
|
||||
# 清单来源: curl https://raw.githubusercontent.com/longhorn/longhorn/v1.7.2/deploy/longhorn-images.txt
|
||||
# 升级时必须重新拉取 longhorn-images.txt 并同步 tag
|
||||
longhornio/longhorn-manager:v1.7.2
|
||||
longhornio/longhorn-engine:v1.7.2
|
||||
longhornio/longhorn-ui:v1.7.2
|
||||
longhornio/longhorn-instance-manager:v1.7.2
|
||||
longhornio/longhorn-share-manager:v1.7.2
|
||||
longhornio/backing-image-manager:v1.7.2
|
||||
longhornio/longhorn-cli:v1.7.2
|
||||
longhornio/support-bundle-kit:v0.0.45
|
||||
|
||||
# ==== Longhorn CSI sidecars (独立于 Longhorn 主镜像, 版本随 Longhorn 升级变化) ====
|
||||
longhornio/csi-attacher:v4.7.0
|
||||
longhornio/csi-provisioner:v4.0.1-20241007
|
||||
longhornio/csi-resizer:v1.12.0
|
||||
longhornio/csi-snapshotter:v7.0.2-20241007
|
||||
longhornio/csi-node-driver-registrar:v2.12.0
|
||||
longhornio/livenessprobe:v2.14.0
|
||||
277
qiming-rcoder/k8s/offline/install.sh
Normal file
277
qiming-rcoder/k8s/offline/install.sh
Normal file
@@ -0,0 +1,277 @@
|
||||
#!/bin/bash
|
||||
# ============================================================================
|
||||
# RCoder 离线部署脚本
|
||||
#
|
||||
# 使用场景: 政企内网, 无外网访问
|
||||
#
|
||||
# 运行流程:
|
||||
# 1. 将镜像导入到节点 containerd (direct 模式) 或推到私有 registry (registry 模式)
|
||||
# 2. 安装 Longhorn 存储 (可跳过)
|
||||
# 3. 安装 JuiceFS CSI Driver (可跳过)
|
||||
# 4. 安装 RCoder Helm chart
|
||||
#
|
||||
# 依赖 (离线机器上必须预装):
|
||||
# - kubectl (配置好 kubeconfig)
|
||||
# - helm 3.x
|
||||
# - docker (registry 模式) 或 k3s/nerdctl (direct 模式)
|
||||
# ============================================================================
|
||||
set -e
|
||||
|
||||
# ---- 默认参数 ----
|
||||
MODE="direct" # direct | registry
|
||||
REGISTRY="" # registry 模式下必填, 例: harbor.internal
|
||||
THIRDPARTY_REGISTRY="" # 第三方镜像目标 registry, 留空同 REGISTRY
|
||||
ENV="dev" # dev | prod
|
||||
RELEASE_NAME="" # 留空 => rcoder-$ENV
|
||||
SKIP_LONGHORN=0
|
||||
SKIP_JUICEFS_CSI=0
|
||||
SKIP_IMAGE_IMPORT=0 # 调试用: 跳过镜像导入
|
||||
NAMESPACE="" # 留空 => nuwax-rcoder-$ENV
|
||||
|
||||
# ---- 颜色 ----
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; NC='\033[0m'
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
用法: $0 [选项]
|
||||
|
||||
选项:
|
||||
--mode=MODE direct (默认) 或 registry
|
||||
--registry=URL 私有 registry 地址 (registry 模式必填)
|
||||
--thirdparty-registry=URL 第三方镜像目标 (留空同 --registry)
|
||||
--env=ENV dev (默认) 或 prod
|
||||
--release=NAME Helm release 名 (默认 rcoder-\$ENV)
|
||||
--namespace=NS 目标 namespace (默认 nuwax-rcoder-\$ENV)
|
||||
--skip-longhorn 跳过 Longhorn 安装 (集群已有 CSI 时用)
|
||||
--skip-juicefs-csi 跳过 JuiceFS CSI 安装
|
||||
--skip-image-import 跳过镜像导入 (调试/镜像已就位时用)
|
||||
-h|--help 显示本帮助
|
||||
|
||||
示例:
|
||||
# 最常见: 断网机器直接用节点 containerd 本地缓存
|
||||
$0 --mode=direct --env=dev
|
||||
|
||||
# 有私有 registry: 先推送再部署
|
||||
$0 --mode=registry --registry=harbor.internal/rcoder --env=prod
|
||||
|
||||
# 集群已有存储, 只装 RCoder 本身
|
||||
$0 --mode=direct --env=dev --skip-longhorn --skip-juicefs-csi
|
||||
EOF
|
||||
exit 0
|
||||
}
|
||||
|
||||
# ---- 参数解析 ----
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--mode=*) MODE="${1#*=}" ;;
|
||||
--registry=*) REGISTRY="${1#*=}" ;;
|
||||
--thirdparty-registry=*) THIRDPARTY_REGISTRY="${1#*=}" ;;
|
||||
--env=*) ENV="${1#*=}" ;;
|
||||
--release=*) RELEASE_NAME="${1#*=}" ;;
|
||||
--namespace=*) NAMESPACE="${1#*=}" ;;
|
||||
--skip-longhorn) SKIP_LONGHORN=1 ;;
|
||||
--skip-juicefs-csi) SKIP_JUICEFS_CSI=1 ;;
|
||||
--skip-image-import) SKIP_IMAGE_IMPORT=1 ;;
|
||||
-h|--help) usage ;;
|
||||
*) echo -e "${RED}未知参数: $1${NC}"; usage ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
[ -z "$RELEASE_NAME" ] && RELEASE_NAME="rcoder-$ENV"
|
||||
[ -z "$NAMESPACE" ] && NAMESPACE="nuwax-rcoder-$ENV"
|
||||
[ -z "$THIRDPARTY_REGISTRY" ] && THIRDPARTY_REGISTRY="$REGISTRY"
|
||||
|
||||
if [ "$MODE" = "registry" ] && [ -z "$REGISTRY" ]; then
|
||||
echo -e "${RED}错误: registry 模式必须指定 --registry=URL${NC}"; exit 1
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}=========================================="
|
||||
echo " RCoder 离线部署"
|
||||
echo "==========================================${NC}"
|
||||
echo " 模式: $MODE"
|
||||
echo " 环境: $ENV"
|
||||
echo " Release: $RELEASE_NAME"
|
||||
echo " Namespace: $NAMESPACE"
|
||||
[ "$MODE" = "registry" ] && echo " Registry: $REGISTRY"
|
||||
[ "$MODE" = "registry" ] && echo " 3rd-party: $THIRDPARTY_REGISTRY"
|
||||
echo " Longhorn: $([ $SKIP_LONGHORN = 1 ] && echo '跳过' || echo '安装')"
|
||||
echo " JuiceFS CSI: $([ $SKIP_JUICEFS_CSI = 1 ] && echo '跳过' || echo '安装')"
|
||||
echo ""
|
||||
|
||||
# ---- 前置检查 ----
|
||||
for cmd in kubectl helm; do
|
||||
if ! command -v $cmd &>/dev/null; then
|
||||
echo -e "${RED}错误: 未安装 $cmd${NC}"; exit 1
|
||||
fi
|
||||
done
|
||||
if ! kubectl cluster-info &>/dev/null; then
|
||||
echo -e "${RED}错误: 无法连接 K8s 集群, 检查 kubeconfig${NC}"; exit 1
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# 步骤 1: 导入镜像
|
||||
# ============================================================================
|
||||
if [ "$SKIP_IMAGE_IMPORT" = "0" ]; then
|
||||
echo -e "${GREEN}[1/4] 导入镜像...${NC}"
|
||||
IMAGES_TAR="$SCRIPT_DIR/images/all-images.tar"
|
||||
if [ ! -f "$IMAGES_TAR" ]; then
|
||||
echo -e "${RED}错误: 未找到 $IMAGES_TAR${NC}"; exit 1
|
||||
fi
|
||||
|
||||
case "$MODE" in
|
||||
direct)
|
||||
# 优先使用 k3s ctr, 回退到 nerdctl, 再回退到 ctr
|
||||
if command -v k3s &>/dev/null; then
|
||||
echo " 使用 k3s ctr 导入 (namespace: k8s.io)..."
|
||||
sudo k3s ctr -n k8s.io image import "$IMAGES_TAR"
|
||||
elif command -v nerdctl &>/dev/null; then
|
||||
echo " 使用 nerdctl 导入..."
|
||||
sudo nerdctl --namespace k8s.io load -i "$IMAGES_TAR"
|
||||
elif command -v ctr &>/dev/null; then
|
||||
echo " 使用 ctr 导入 (namespace: k8s.io)..."
|
||||
sudo ctr -n k8s.io image import "$IMAGES_TAR"
|
||||
else
|
||||
echo -e "${RED}错误: direct 模式需要 k3s / nerdctl / ctr 其中之一${NC}"; exit 1
|
||||
fi
|
||||
echo -e "${GREEN} ✅ 镜像已导入节点 containerd${NC}"
|
||||
echo -e "${YELLOW} ⚠️ 如果是多节点集群, 需要在每个节点上运行 ctr image import${NC}"
|
||||
;;
|
||||
registry)
|
||||
if ! command -v docker &>/dev/null; then
|
||||
echo -e "${RED}错误: registry 模式需要 docker${NC}"; exit 1
|
||||
fi
|
||||
echo " 加载到 docker daemon..."
|
||||
docker load -i "$IMAGES_TAR"
|
||||
echo " 重打 tag 并推送到 $REGISTRY..."
|
||||
bash "$SCRIPT_DIR/rewrite-registry.sh" \
|
||||
--registry "$REGISTRY" \
|
||||
--thirdparty-registry "$THIRDPARTY_REGISTRY" \
|
||||
--images-file "$SCRIPT_DIR/images/images.txt"
|
||||
;;
|
||||
esac
|
||||
else
|
||||
echo -e "${YELLOW}[1/4] 跳过镜像导入 (--skip-image-import)${NC}"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# 步骤 2: 安装 Longhorn
|
||||
# ============================================================================
|
||||
echo -e "${GREEN}[2/4] Longhorn...${NC}"
|
||||
if [ "$SKIP_LONGHORN" = "1" ]; then
|
||||
echo " 跳过 (--skip-longhorn)"
|
||||
elif kubectl get sc longhorn &>/dev/null; then
|
||||
echo -e "${GREEN} ✅ 已安装${NC}"
|
||||
else
|
||||
LONGHORN_YAML=$(ls "$SCRIPT_DIR/longhorn/longhorn-"*.yaml 2>/dev/null | head -1)
|
||||
if [ -z "$LONGHORN_YAML" ]; then
|
||||
echo -e "${RED}错误: 未找到 longhorn/longhorn-*.yaml${NC}"; exit 1
|
||||
fi
|
||||
|
||||
# registry 模式: 先把 manifest 里的 longhornio/* 改成 $THIRDPARTY_REGISTRY/longhornio/*
|
||||
APPLY_YAML="$LONGHORN_YAML"
|
||||
if [ "$MODE" = "registry" ]; then
|
||||
APPLY_YAML="/tmp/longhorn-rewritten.yaml"
|
||||
sed -E "s#(image:\s*)longhornio/#\1${THIRDPARTY_REGISTRY}/longhornio/#g" \
|
||||
"$LONGHORN_YAML" > "$APPLY_YAML"
|
||||
fi
|
||||
|
||||
echo " 部署 Longhorn..."
|
||||
kubectl apply -f "$APPLY_YAML"
|
||||
echo " 等待 Longhorn manager 就绪 (最长 5 分钟)..."
|
||||
kubectl wait --for=condition=ready pod -l app=longhorn-manager \
|
||||
-n longhorn-system --timeout=300s || true
|
||||
echo -e "${GREEN} ✅ Longhorn 已安装${NC}"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# 步骤 3: 安装 JuiceFS CSI Driver (使用官方 deploy/k8s.yaml, 不走 helm)
|
||||
# ============================================================================
|
||||
echo -e "${GREEN}[3/4] JuiceFS CSI Driver...${NC}"
|
||||
if [ "$SKIP_JUICEFS_CSI" = "1" ]; then
|
||||
echo " 跳过 (--skip-juicefs-csi)"
|
||||
elif kubectl get ds -n kube-system juicefs-csi-driver-node &>/dev/null \
|
||||
|| kubectl get ds -n kube-system juicefs-csi-node &>/dev/null; then
|
||||
echo -e "${GREEN} ✅ 已安装${NC}"
|
||||
else
|
||||
JUICEFS_YAML=$(ls "$SCRIPT_DIR/juicefs-csi/juicefs-csi-"*.yaml 2>/dev/null | head -1)
|
||||
if [ -z "$JUICEFS_YAML" ]; then
|
||||
echo -e "${RED}错误: 未找到 juicefs-csi/juicefs-csi-*.yaml${NC}"; exit 1
|
||||
fi
|
||||
|
||||
# registry 模式: manifest 里所有 image 前面加上私有 registry 前缀
|
||||
APPLY_YAML="$JUICEFS_YAML"
|
||||
if [ "$MODE" = "registry" ]; then
|
||||
APPLY_YAML="/tmp/juicefs-csi-rewritten.yaml"
|
||||
sed -E \
|
||||
-e "s#(image:\s*)juicedata/#\1${REGISTRY}/juicedata/#g" \
|
||||
-e "s#(image:\s*)registry\.k8s\.io/sig-storage/#\1${THIRDPARTY_REGISTRY}/sig-storage/#g" \
|
||||
"$JUICEFS_YAML" > "$APPLY_YAML"
|
||||
fi
|
||||
|
||||
echo " 部署 JuiceFS CSI ($(basename $JUICEFS_YAML))..."
|
||||
kubectl apply -f "$APPLY_YAML"
|
||||
|
||||
# 设置 mount 镜像 (CE 版本, 与 RCoder 使用的 juicefs 客户端对齐)
|
||||
MOUNT_IMG="${THIRDPARTY_REGISTRY:+$THIRDPARTY_REGISTRY/}juicedata/mount:ce-v1.3.1"
|
||||
for ds in juicefs-csi-driver-node juicefs-csi-node; do
|
||||
kubectl -n kube-system get daemonset/$ds &>/dev/null && \
|
||||
kubectl -n kube-system set env daemonset/$ds -c juicefs-plugin \
|
||||
JUICEFS_CE_MOUNT_IMAGE="$MOUNT_IMG" \
|
||||
JUICEFS_MOUNT_IMAGE="$MOUNT_IMG" >/dev/null 2>&1 || true
|
||||
done
|
||||
kubectl -n kube-system get statefulset/juicefs-csi-controller &>/dev/null && \
|
||||
kubectl -n kube-system set env statefulset/juicefs-csi-controller -c juicefs-plugin \
|
||||
JUICEFS_CE_MOUNT_IMAGE="$MOUNT_IMG" \
|
||||
JUICEFS_MOUNT_IMAGE="$MOUNT_IMG" >/dev/null 2>&1 || true
|
||||
|
||||
echo " 等待 CSI 就绪 (最长 2 分钟)..."
|
||||
for label in "app=juicefs-csi-driver-node" "app=juicefs-csi-node"; do
|
||||
kubectl wait --for=condition=ready pod -l "$label" -n kube-system --timeout=120s 2>/dev/null && break || true
|
||||
done
|
||||
echo -e "${GREEN} ✅ JuiceFS CSI 已安装${NC}"
|
||||
fi
|
||||
|
||||
# ============================================================================
|
||||
# 步骤 4: 安装 RCoder
|
||||
# ============================================================================
|
||||
echo -e "${GREEN}[4/4] 安装 RCoder...${NC}"
|
||||
|
||||
RCODER_CHART=$(ls "$SCRIPT_DIR/charts/rcoder-"*.tgz 2>/dev/null | head -1)
|
||||
if [ -z "$RCODER_CHART" ]; then
|
||||
echo -e "${RED}错误: 未找到 charts/rcoder-*.tgz${NC}"; exit 1
|
||||
fi
|
||||
|
||||
VALUES_ARGS=(-f "$SCRIPT_DIR/values-$ENV.yaml")
|
||||
if [ "$MODE" = "registry" ]; then
|
||||
VALUES_ARGS+=(
|
||||
--set "global.imageRegistry=$REGISTRY"
|
||||
--set "global.thirdPartyRegistry=$THIRDPARTY_REGISTRY"
|
||||
)
|
||||
fi
|
||||
|
||||
echo " helm upgrade --install $RELEASE_NAME ..."
|
||||
helm upgrade --install "$RELEASE_NAME" "$RCODER_CHART" \
|
||||
--namespace "$NAMESPACE" \
|
||||
--create-namespace \
|
||||
"${VALUES_ARGS[@]}"
|
||||
|
||||
echo -e "${GREEN} 等待 rcoder deployment 就绪 (最长 5 分钟)...${NC}"
|
||||
kubectl rollout status deploy/rcoder -n "$NAMESPACE" --timeout=300s || true
|
||||
|
||||
# ============================================================================
|
||||
# 收尾
|
||||
# ============================================================================
|
||||
echo ""
|
||||
echo -e "${GREEN}=========================================="
|
||||
echo " 部署完成!"
|
||||
echo "==========================================${NC}"
|
||||
kubectl get pods -n "$NAMESPACE"
|
||||
echo ""
|
||||
NODE_IP=$(kubectl get nodes -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}' 2>/dev/null || echo "<node-ip>")
|
||||
NODE_PORT=$(kubectl get svc rcoder -n "$NAMESPACE" -o jsonpath='{.spec.ports[0].nodePort}' 2>/dev/null || echo "<nodeport>")
|
||||
echo "访问: curl http://${NODE_IP}:${NODE_PORT}/health"
|
||||
echo "日志: kubectl logs -n $NAMESPACE -l app=rcoder -f"
|
||||
echo "卸载: helm uninstall $RELEASE_NAME --namespace $NAMESPACE"
|
||||
73
qiming-rcoder/k8s/offline/rewrite-registry.sh
Normal file
73
qiming-rcoder/k8s/offline/rewrite-registry.sh
Normal file
@@ -0,0 +1,73 @@
|
||||
#!/bin/bash
|
||||
# ============================================================================
|
||||
# 把 docker daemon 里已加载的镜像 re-tag 并推送到私有 registry
|
||||
#
|
||||
# 输入: images.txt (一行一个镜像)
|
||||
# 行为:
|
||||
# - RCoder 自有镜像 (nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/*)
|
||||
# -> 推到 $REGISTRY/{name}:{tag}
|
||||
# - 其他 (postgres / minio / juicedata / sig-storage / longhornio 等)
|
||||
# -> 推到 $THIRDPARTY_REGISTRY/{repo}:{tag}
|
||||
#
|
||||
# 依赖: docker (镜像已 load 在本地 daemon)
|
||||
# ============================================================================
|
||||
set -e
|
||||
|
||||
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; NC='\033[0m'
|
||||
|
||||
REGISTRY=""
|
||||
THIRDPARTY_REGISTRY=""
|
||||
IMAGES_FILE=""
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--registry) REGISTRY="$2"; shift 2 ;;
|
||||
--thirdparty-registry) THIRDPARTY_REGISTRY="$2"; shift 2 ;;
|
||||
--images-file) IMAGES_FILE="$2"; shift 2 ;;
|
||||
*) echo "未知参数: $1"; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[ -z "$REGISTRY" ] && { echo -e "${RED}--registry 必填${NC}"; exit 1; }
|
||||
[ -z "$THIRDPARTY_REGISTRY" ] && THIRDPARTY_REGISTRY="$REGISTRY"
|
||||
[ -z "$IMAGES_FILE" ] && { echo -e "${RED}--images-file 必填${NC}"; exit 1; }
|
||||
[ ! -f "$IMAGES_FILE" ] && { echo -e "${RED}未找到 $IMAGES_FILE${NC}"; exit 1; }
|
||||
|
||||
# RCoder 自有镜像的 registry 前缀 (用于识别 + 剥离)
|
||||
RCODER_OWN_PREFIX="nuwax-docker-images-registry.cn-hangzhou.cr.aliyuncs.com/dev/"
|
||||
|
||||
echo -e "${GREEN}Re-tag + push 镜像到私有 registry${NC}"
|
||||
echo " 自有镜像 -> $REGISTRY/"
|
||||
echo " 第三方 -> $THIRDPARTY_REGISTRY/"
|
||||
|
||||
while IFS= read -r line; do
|
||||
# 跳过注释和空行
|
||||
line="${line%%#*}"; line="${line##[[:space:]]}"; line="${line%%[[:space:]]}"
|
||||
[ -z "$line" ] && continue
|
||||
|
||||
SRC="$line"
|
||||
DST=""
|
||||
|
||||
if [[ "$SRC" == "$RCODER_OWN_PREFIX"* ]]; then
|
||||
# 自有镜像: 剥离前缀, 推到 REGISTRY
|
||||
STRIPPED="${SRC#$RCODER_OWN_PREFIX}"
|
||||
DST="$REGISTRY/$STRIPPED"
|
||||
else
|
||||
# 第三方: 保留完整 repo 路径, 推到 THIRDPARTY_REGISTRY
|
||||
# 处理 registry.k8s.io/sig-storage/xxx —— 去掉 registry.k8s.io/ 保留 sig-storage/xxx
|
||||
REPO="$SRC"
|
||||
case "$REPO" in
|
||||
registry.k8s.io/*) REPO="${REPO#registry.k8s.io/}" ;;
|
||||
docker.io/*) REPO="${REPO#docker.io/}" ;;
|
||||
quay.io/*) REPO="${REPO#quay.io/}" ;;
|
||||
esac
|
||||
DST="$THIRDPARTY_REGISTRY/$REPO"
|
||||
fi
|
||||
|
||||
echo -e "${YELLOW} $SRC${NC}"
|
||||
echo " => $DST"
|
||||
docker tag "$SRC" "$DST"
|
||||
docker push "$DST"
|
||||
done < "$IMAGES_FILE"
|
||||
|
||||
echo -e "${GREEN}✅ 所有镜像已推送到私有 registry${NC}"
|
||||
Reference in New Issue
Block a user