背景

目前公司采用 protocol buffer 作为 IDL，虽然可以根据 API 定义，轻松生成客户端和服务端的代码。但是对于跨项目的接口，会增加项目之间的耦合性。例如 A 服务对外提供了一个接口，B 服务去调用。那么就需要根据 A 服务的 proto 文件，生成客户端代码，并拷贝给 B。如果联调期间，A 服务改动了该接口，还需重复前面的步骤，非常繁琐

方案

常见的几种解决方案，煎鱼大佬已经描述得很详细了（真是头疼，Proto 代码到底放哪里？）

• 方案一：api 大仓 + git submodule（b 站）
• 方案二：api 大仓 + git submodule + 每个项目生成代码专有仓库
• 方案三：每个项目都有一个 api 仓库，包含生成的代码
• 方案四：api 大仓 + api 生成代码的集中仓库

具体方案的优缺点 yuyy 博主已经写清楚了。 权衡了下，和博主一样选择方案四。

具体实现

DRONE 文件

$ tree -L 1
.
├── Dockerfile
├── Makefile
├── README.md
├── apis-go.gen.yml
├── apis-go.sh
├── apis-swagger.gen.yml
├── apis-swagger.sh
├── auth-center
├── budget-center
├── common
├── consume-order
├── consume-quota-center
├── consume-rule-to-third
├── datacenter
├── fino-multi-env-center
├── finobase
├── finoconsume
├── invoice
├── mng-center
├── mq-center
├── notify-center
├── org-arch-center
├── org-asset-center
├── org-order-center
├── org-recharge-center
├── org-settle-center
├── pubsvc
├── pushcenter
├── right-recharge
├── snowflake
├── task-center
├── third-consume-order
└── timer

26 directories, 7 files

.drone.yaml

kind: pipeline
type: docker
name: apis

workspace:
  base: /app
  path: ${DRONE_REPO_NAME}

steps:
  - name: 检查proto文件
    image: reg.xxxx.com/golang/apis-generate-go:1.0.0
    pull: if-not-exists
    volumes:
      - name: buf-cache
        path: /app/buf/.cache
    commands:
      - buf lint

  - name: 编译proto文件
    image: reg.xxxx.com/golang/apis-generate-go:1.0.0
    pull: if-not-exists
    volumes:
      - name: buf-cache
        path: /app/buf/.cache
    environment:
      BUF_CACHE_DIR: /app/buf/.cache
      TARGET_REPO: apis-go
      TARGET_REPO_ADDR: git@gogs.xxxx.com:fino/apis-go.git
      SSH_PRIVATE_KEY:
        from_secret: ssh_private_key
    commands:
      - sh ./apis-go.sh

  - name: 生成swagger文件
    image: reg.xxxx.com/golang/apis-generate-go:1.0.0
    pull: if-not-exists
    volumes:
      - name: buf-cache
        path: /app/buf/.cache
    environment:
      BUF_CACHE_DIR: /app/buf/.cache
      TARGET_REPO: apis-swagger
      TARGET_REPO_ADDR: git@gogs.xxxx.com:fino/apis-swagger.git
      SSH_PRIVATE_KEY:
        from_secret: ssh_private_key
    commands:
      - sh ./apis-swagger.sh

  - name: 通知
    image: plugins/webhook
    pull: if-not-exists
    settings:
      urls: https://oapi.dingtalk.com/robot/send?access_token=xxxx
      content_type: application/json
      template: |
        {
            "msgtype": "text",
            "text": {
                "content": "Proto \n > 构建结果: {{ build.status }} \n > 代码分支: {{ build.branch }} \n > 编译详情: {{ build.link }} \n > 提交信息: {{ build.message }} \n > 提交发起: {{ build.author }} "
            }
        }

volumes: # 定义流水线挂载目录，用于共享数据
  - name: buf-cache
    host:
      path: /home/docker/drone/buf/.cache # 从宿主机中挂载的目录

ssh_private_key 经过echo '私钥文件' | base64生成，并配置在 DRONE 的 Secrets 上 buf-cache 是 drone 挂载硬盘，设置 buf 缓存，加快构建速度 reg.xxxx.com/golang/apis-generate-go:1.0.0 封装的一个包含 buf 命令的镜像

apis-generate-go DOCKERFILE 基础镜像

FROM golang:1.21-alpine

RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apk/repositories && \
    apk add --no-cache git protobuf curl openssh-server openssh-client && \
    git --version && \
    protoc --version && \
    ssh-keygen -A

RUN BIN="/usr/local/bin" && \
    VERSION="1.26.1" && \
    curl -sSL \
    "https://mirror.ghproxy.com/https://github.com/bufbuild/buf/releases/download/v${VERSION}/buf-$(uname -s)-$(uname -m)" \
    -o "${BIN}/buf" && \
    chmod +x "${BIN}/buf"

RUN go env -w GOPROXY=https://goproxy.cn,direct && \
    go install google.golang.org/protobuf/cmd/protoc-gen-go@latest && \
    go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest && \
    go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2@latest && \
    go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway@latest && \
    go install github.com/go-kratos/kratos/cmd/protoc-gen-go-http/v2@latest

构建指令 docker build --platform linux/amd64 -t reg.xxxx.com/golang/apis-generate-go:1.0.0

编译 proto 文件

apis-go.gen.yml

version: v1
managed:
  enabled: true

plugins:
  - plugin: buf.build/protocolbuffers/go:v1.33.0 # 生成go代码
    out: apis-go
    opt: paths=source_relative
  - plugin: buf.build/grpc/go # 生成grpc代码
    out: apis-go
    opt: paths=source_relative,require_unimplemented_servers=false
  - plugin: buf.build/bufbuild/validate-go
    out: apis-go
    opt: paths=source_relative
  - plugin: go-http # 生成http-gateway，本地插件（kratos）
    out: apis-go
    opt:
      - paths=source_relative

apis-go.sh

#!/bin/bash

echo "---- buf generate ----"
echo "+ buf generate"
buf generate --template apis-go.gen.yml && echo "buf generate success" || exit

echo "---- 配置 git ssh，实现免密提交到 ${TARGET_REPO} 仓库 ----"
eval "$(ssh-agent -s)"
ssh-add <(echo "${SSH_PRIVATE_KEY}" | base64 -d)
mkdir -p ~/.ssh
touch ~/.ssh/config
{
    echo "StrictHostKeyChecking no"
    echo "UserKnownHostsFile /dev/null"
} >>~/.ssh/config

echo "---- 配置 git 用户信息为当前触发流水线的用户 ----"
if [ -z "${DRONE_COMMIT_AUTHOR}" ]; then
  echo "无法获取提交用户git信息，请检查仓库邮箱是否和gogs对应"
  exit
fi
echo "git提交用户：${DRONE_COMMIT_AUTHOR}"
git config --global user.name "${DRONE_COMMIT_AUTHOR}"
git config --global user.email "${DRONE_COMMIT_AUTHOR_EMAIL}"

echo "---- git clone ${TARGET_REPO} 仓库，只拉取指定分支的最后一次 commit ----"
mkdir -p /app
cd /app || exit
echo "git clone --branch ${DRONE_BRANCH} ${TARGET_REPO_ADDR}"
if git clone --branch "${DRONE_BRANCH}" "${TARGET_REPO_ADDR}" > /dev/null 2>&1; then
  echo "远程仓库 ${TARGET_REPO} 已存在，直接拉取"
else
  echo "远程仓库 ${TARGET_REPO} 不存在，基于 master 分支创建"
  echo "+ git clone ${TARGET_REPO_ADDR}"
  git clone "${TARGET_REPO_ADDR}" || exit
  cd "${TARGET_REPO}" || exit
  echo "+ git checkout -b ${DRONE_BRANCH}"
  git checkout -b "${DRONE_BRANCH}"
fi

echo "---- 拷贝go文件到 ${TARGET_REPO} 仓库 ----"
cd /app/"${TARGET_REPO}" || exit
echo "删除并复制文件"

echo "+ find /app/${TARGET_REPO}/* -mindepth 1 -type d -exec rm -rf {} +"
find /app/"${TARGET_REPO}"/* -type d -exec rm -rf {} + || exit
echo "+ cp -rf"
cp -rf /app/"${DRONE_REPO_NAME}"/apis-go/* /app/"${TARGET_REPO}" || exit
cp /app/"${DRONE_REPO_NAME}"/.apis-go.drone.yml /app/"${TARGET_REPO}"/.drone.yml || exit

echo "+ go mod tidy"
go mod tidy > /dev/null 2>&1

echo "---- 提交到 ${TARGET_REPO} 仓库 ----"
git fetch --unshallow
git add . || exit
printf "+ git commit\n"
git commit -m "${DRONE_COMMIT_MESSAGE}"
printf "+ git push\n"
git push --set-upstream origin "${DRONE_BRANCH}"

echo "---- 同步成功 ----"

生成 swagger

apis-swagger.gen.yml

version: v1
managed:
  enabled: true

plugins:
  - plugin: buf.build/grpc-ecosystem/openapiv2 # 生成openapi文档
    out: apis-swagger
    opt:
      - disable_default_responses=false
      - json_names_for_fields=false

.apis-swagger.sh

#!/bin/bash

echo "---- buf generate ----"
echo "+ buf generate"
buf generate --template apis-swagger.gen.yml && echo "buf generate success" || exit

echo "----- 获取项目列表 -----"
PROJECTS=$(find /app/apis-swagger/* -mindepth 0 -type d) || exit
if [ -z "${PROJECTS}" ]; then
  echo "没有项目，跳过上传"
  exit
fi

for PROJECT in ${PROJECTS}; do
  echo "---- 项目：${PROJECT} ----"
  if [ -f "${PROJECT}/token" ]; then
    echo "-- 读取token文件 ---"
    token=$(cat "${PROJECT}/token")

    echo "-- 获取项目下的文件 ---"
    echo "+ ls ${PROJECT}"
    FILES=$(ls "${PROJECT}") || exit

    for FILE in ${FILES}; do
      # 文化名不是token
      if [ "${FILE}" == "${PROJECT}/token" ]; then
        continue
      fi
      echo "+ curl import_data ${FILE}"
      fileContent=$(cat "${FILE}") || exit
      curl -s --location --request POST 'http://yapi.xxxx.com/api/open/import_data' \
      --header 'Connection: keep-alive' \
      --header 'Content-Type: application/x-www-form-urlencoded' \
      --data-urlencode 'type=swagger' \
      --data-urlencode 'merge=good' \
      --data-urlencode "token=${token}" \
      --data-urlencode "json=$fileContent" || exit
    done
    printf '\n'
  else
    echo "不存在token文件，不执行yapi导入操作"
  fi
done

参考

1. Gitlab CI/CD 实践六：统一管理 protocol buffer，API 大仓设计与实现
2. 真是头疼，Proto 代码到底放哪里？