====== 为 GitLab CI 配置基于 GKE Autopilot 的 Runner ======

https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke

GKE Autopilot Cluster (([[https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview|GKE Autopilot Overview]])) 免费额度(([[https://cloud.google.com/kubernetes-engine/pricing#autopilot_mode|GKE Autopilot Pricing]]))为每个帐号1集群，运行费另计。

===== 创建集群 =====

https://console.cloud.google.com/kubernetes/auto/add

===== 获取凭据 =====

打开 Cloud Shell 粘贴一把梭，开头变量要改。(([[https://cloud.google.com/kubernetes-engine/docs/how-to/api-server-authentication|GKE How to Authenticating to the Kubernetes API server]]))。


<code bash>
CLUSTER_NAME=anything
GOOGLE_CLOUD_PROJECT=project-slug
GOOGLE_CLOUD_REGION=europe-central2


gcloud iam service-accounts create gitlab-ci-runner
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
    --member=serviceAccount:gitlab-ci-runner@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com \
    --role=roles/container.developer
gcloud iam service-accounts keys create gsa-key.json \
    --iam-account=gitlab-ci-runner@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com
CLUSTER_CA_CERT=$(gcloud container clusters describe $CLUSTER_NAME --zone=$GOOGLE_CLOUD_REGION --format="value(masterAuth.clusterCaCertificate)")
CLUSTER_ENDPOINT=$(gcloud container clusters describe $CLUSTER_NAME --zone=$GOOGLE_CLOUD_REGION --format="value(endpoint)")


cat > kubeconfig.yaml <<EOF
apiVersion: v1
kind: Config
clusters:
- name: $CLUSTER_NAME
  cluster:
    server: https://$CLUSTER_ENDPOINT
    certificate-authority-data: $CLUSTER_CA_CERT
users:
- name: gitlab-ci-runner
  user:
    auth-provider:
      name: gcp
contexts:
- context:
    cluster: $CLUSTER_NAME
    user: gitlab-ci-runner
  name: $CLUSTER_NAME-gitlab-ci
current-context: $CLUSTER_NAME-gitlab-ci
EOF
</code>

完了之后下载 ''gsa-key.json'' 和 ''kubeconfig.yaml'' 两个文件，拷到 runner 上。

===== 配 GitLab CI Runner =====

环境变量：

<code bash>
KUBECONFIG=path/to/kubeconfig.yaml
GOOGLE_APPLICATION_CREDENTIALS=path/to/gsa-key.json
</code>

配置文件：
(([[https://docs.gitlab.com/runner/configuration/advanced-configuration.html|GitLab Runners Advanced Config]]))
(([[https://docs.gitlab.com/runner/executors/kubernetes.html|GitLab Runners Kubernetes Executor]]))
(([[https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#allowable_resource_ranges|Autopilot Allowable Resource Ranges]]))
(([[https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-overview#default_container_resource_requests|Autopilot Default container resource requests]]))
(([[https://docs.gitlab.com/ee/api/runners.html#register-a-new-runner|Register a new runner - Runners API - GitLab Documentation]]))

<file toml config.toml>
# concurrent = 1
# log_level = "debug"

[[runners]]
  # name = "..."
  # url = "https://.../ci"
  # token = "..."
  executor = "kubernetes"

  [runners.kubernetes]
    image = "ubuntu:bionic"
    namespace = "default"
    privileged = false
    # helper_image = ""
    # helper_image_flavor = "ubuntu"
    pull_policy = "always"

    cpu_limit = "2"
    cpu_limit_overwrite_max_allowed = "4"
    cpu_request = "2"
    cpu_request_overwrite_max_allowed = "4"
    helper_cpu_limit = "250m"
    helper_cpu_limit_overwrite_max_allowed = "1"
    helper_cpu_request = "250m"
    helper_cpu_request_overwrite_max_allowed = "1"
    service_cpu_limit = "1"
    service_cpu_limit_overwrite_max_allowed = "2"
    service_cpu_request = "1"
    service_cpu_request_overwrite_max_allowed = "2"
    memory_limit = "8Gi"
    memory_limit_overwrite_max_allowed = "32Gi"
    memory_request = "8Gi"
    memory_request_overwrite_max_allowed = "32Gi"
    helper_memory_limit = "512Mi"
    helper_memory_limit_overwrite_max_allowed = "4Gi"
    helper_memory_request = "512Mi"
    helper_memory_request_overwrite_max_allowed = "4Gi"
    service_memory_limit = "2Gi"
    service_memory_limit_overwrite_max_allowed = "8Gi"
    service_memory_request = "2Gi"
    service_memory_request_overwrite_max_allowed = "8Gi"
    ephemeral_storage_limit = "10Gi"
    ephemeral_storage_limit_overwrite_max_allowed = "10Gi"
    ephemeral_storage_request = "10Gi"
    ephemeral_storage_request_overwrite_max_allowed = "10Gi"
    helper_ephemeral_storage_limit = "10Gi"
    helper_ephemeral_storage_limit_overwrite_max_allowed = "10Gi"
    helper_ephemeral_storage_request = "10Gi"
    helper_ephemeral_storage_request_overwrite_max_allowed = "10Gi"
    service_ephemeral_storage_limit = "10Gi"
    service_ephemeral_storage_limit_overwrite_max_allowed = "10Gi"
    service_ephemeral_storage_request = "10Gi"
    service_ephemeral_storage_request_overwrite_max_allowed = "10Gi"

    [runners.kubernetes.pod_annotations]
      # for img
      "container.apparmor.security.beta.kubernetes.io/build" = "unconfined"
      "container.seccomp.security.alpha.kubernetes.io/build" = "unconfined"
</file>

其中 `pod_annotations` 是给 [[https://github.com/genuinetools/img|img]] 构建用的。

里面 `/build` 是说 pod 里面的容器名字，这个参考 k8s executor 的文档就知道了：

> The build container is ''build''
> The helper container is ''helper''
> The services containers are ''svc-X'' where X is ''[0-9]+''

===== 问题 =====

==== Pod Unschedulable ====

> Waiting for pod default/runner-b005efda-project-170-concurrent-0c5lg8 to be running, status is Pending
> 	Unschedulable: "0/2 nodes are available: 2 Insufficient cpu, 2 Insufficient memory."

无视，等 node 启动就行了，要刷一个屏幕那么多

==== 无法使用 dind ====

用 img 可解。

但是 img 需要一个 overlayfs 或者 overlayfs-fuse。没有的话就会疯狂复制文件。然后超过 10GiB 的限制，失败。

===== 接下来的想法 =====

用 [[https://docs.gitlab.com/runner/executors/custom.html]] 组合一些乱七八糟的容器服务搭一个。

再在 GKE 上弄一些 kaniko 或者别的不太需要空间的方案。

===== 其它 =====

(([[https://gitlab.com/gitlab-org/gitlab-runner|GitLab Runner Project on GitLab]]))

{{tag>gitlab gitlab-ci}}