为 GitLab CI 配置基于 GKE Autopilot 的 Runner

https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke

GKE Autopilot Cluster ¹ 免费额度²为每个帐号1集群，运行费另计。

创建集群

https://console.cloud.google.com/kubernetes/auto/add

获取凭据

打开 Cloud Shell 粘贴一把梭，开头变量要改。³。

CLUSTER_NAME=anything
GOOGLE_CLOUD_PROJECT=project-slug
GOOGLE_CLOUD_REGION=europe-central2


gcloud iam service-accounts create gitlab-ci-runner
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
    --member=serviceAccount:gitlab-ci-runner@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com \
    --role=roles/container.developer
gcloud iam service-accounts keys create gsa-key.json \
    --iam-account=gitlab-ci-runner@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com
CLUSTER_CA_CERT=$(gcloud container clusters describe $CLUSTER_NAME --zone=$GOOGLE_CLOUD_REGION --format="value(masterAuth.clusterCaCertificate)")
CLUSTER_ENDPOINT=$(gcloud container clusters describe $CLUSTER_NAME --zone=$GOOGLE_CLOUD_REGION --format="value(endpoint)")


cat > kubeconfig.yaml <EOF
apiVersion: v1
kind: Config
clusters:
- name: $CLUSTER_NAME
  cluster:
    server: https://$CLUSTER_ENDPOINT
    certificate-authority-data: $CLUSTER_CA_CERT
users:
- name: gitlab-ci-runner
  user:
    auth-provider:
      name: gcp
contexts:
- context:
    cluster: $CLUSTER_NAME
    user: gitlab-ci-runner
  name: $CLUSTER_NAME-gitlab-ci
current-context: $CLUSTER_NAME-gitlab-ci
EOF

完了之后下载 gsa-key.json 和 kubeconfig.yaml 两个文件，拷到 runner 上。

配 GitLab CI Runner

环境变量：

KUBECONFIG=path/to/kubeconfig.yaml
GOOGLE_APPLICATION_CREDENTIALS=path/to/gsa-key.json

配置文件： ⁴ ⁵ ⁶ ⁷ ⁸

# concurrent = 1
# log_level = "debug"

[[runners]]
  # name = "..."
  # url = "https://.../ci"
  # token = "..."
  executor = "kubernetes"

  [runners.kubernetes]
    image = "ubuntu:bionic"
    namespace = "default"
    privileged = false
    # helper_image = ""
    # helper_image_flavor = "ubuntu"
    pull_policy = "always"

    cpu_limit = "2"
    cpu_limit_overwrite_max_allowed = "4"
    cpu_request = "2"
    cpu_request_overwrite_max_allowed = "4"
    helper_cpu_limit = "250m"
    helper_cpu_limit_overwrite_max_allowed = "1"
    helper_cpu_request = "250m"
    helper_cpu_request_overwrite_max_allowed = "1"
    service_cpu_limit = "1"
    service_cpu_limit_overwrite_max_allowed = "2"
    service_cpu_request = "1"
    service_cpu_request_overwrite_max_allowed = "2"
    memory_limit = "8Gi"
    memory_limit_overwrite_max_allowed = "32Gi"
    memory_request = "8Gi"
    memory_request_overwrite_max_allowed = "32Gi"
    helper_memory_limit = "512Mi"
    helper_memory_limit_overwrite_max_allowed = "4Gi"
    helper_memory_request = "512Mi"
    helper_memory_request_overwrite_max_allowed = "4Gi"
    service_memory_limit = "2Gi"
    service_memory_limit_overwrite_max_allowed = "8Gi"
    service_memory_request = "2Gi"
    service_memory_request_overwrite_max_allowed = "8Gi"
    ephemeral_storage_limit = "10Gi"
    ephemeral_storage_limit_overwrite_max_allowed = "10Gi"
    ephemeral_storage_request = "10Gi"
    ephemeral_storage_request_overwrite_max_allowed = "10Gi"
    helper_ephemeral_storage_limit = "10Gi"
    helper_ephemeral_storage_limit_overwrite_max_allowed = "10Gi"
    helper_ephemeral_storage_request = "10Gi"
    helper_ephemeral_storage_request_overwrite_max_allowed = "10Gi"
    service_ephemeral_storage_limit = "10Gi"
    service_ephemeral_storage_limit_overwrite_max_allowed = "10Gi"
    service_ephemeral_storage_request = "10Gi"
    service_ephemeral_storage_request_overwrite_max_allowed = "10Gi"

    [runners.kubernetes.pod_annotations]
      # for img
      "container.apparmor.security.beta.kubernetes.io/build" = "unconfined"
      "container.seccomp.security.alpha.kubernetes.io/build" = "unconfined"

其中 pod_annotations 是给 img 构建用的。

里面 /build 是说 pod 里面的容器名字，这个参考 k8s executor 的文档就知道了：

The build container is build

The helper container is helper The services containers are svc-X where X is [0-9]+

问题

Pod Unschedulable

Waiting for pod default/runner-b005efda-project-170-concurrent-0c5lg8 to be running, status is Pending Unschedulable: "0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory."

无视，等 node 启动就行了，要刷一个屏幕那么多

无法使用 dind

用 img 可解。

但是 img 需要一个 overlayfs 或者 overlayfs-fuse。没有的话就会疯狂复制文件。然后超过 10GiB 的限制，失败。

接下来的想法

用 https://docs.gitlab.com/runner/executors/custom.html 组合一些乱七八糟的容器服务搭一个。

再在 GKE 上弄一些 kaniko 或者别的不太需要空间的方案。

其它

⁹