Using GitLab CI/CD with a Kubernetes cluster (FREE)

• Introduced in GitLab 14.1.
• The pre-configured KUBECONFIG was introduced in GitLab 14.2.
• Introduced the ci_access attribute in GitLab 14.3.
• The ability to authorize groups was introduced in GitLab 14.3.
• Moved to GitLab Free in 14.5.
• Support for Omnibus installations was introduced in GitLab 14.5.
• The ability to switch between certificate-based clusters and agents was introduced in GitLab 14.9. The certificate-based cluster context is always called gitlab-deploy.
• Renamed from CI/CD tunnel to CI/CD workflow in GitLab 14.9.

You can use a GitLab CI/CD workflow to safely deploy to and update your Kubernetes clusters.

To do so, you must first install an agent in your cluster. When done, you have a Kubernetes context and can run Kubernetes API commands in your GitLab CI/CD pipeline.

To ensure access to your cluster is safe:

• Each agent has a separate context (kubecontext).
• Only the project where the agent is configured, and any additional projects you authorize, can access the agent in your cluster.

You do not need to have a runner in the cluster with the agent.

GitLab CI/CD workflow steps

To update a Kubernetes cluster by using GitLab CI/CD, complete the following steps.

1. Ensure you have a working Kubernetes cluster and the manifests are in a GitLab project.
2. In the same GitLab project, register and install the GitLab agent.
3. Update your .gitlab-ci.yml file to select the agent's Kubernetes context and run the Kubernetes API commands.
4. Run your pipeline to deploy to or update the cluster.

If you have multiple GitLab projects that contain Kubernetes manifests:

1. Install the GitLab agent in its own project, or in one of the GitLab projects where you keep Kubernetes manifests.
2. Authorize the agent to access your GitLab projects.
3. Optional. For added security, use impersonation.
4. Update your .gitlab-ci.yml file to select the agent's Kubernetes context and run the Kubernetes API commands.
5. Run your pipeline to deploy to or update the cluster.

Authorize the agent

If you have multiple GitLab projects, you must authorize the agent to access the project where you keep your Kubernetes manifests. You can authorize the agent to access individual projects, or authorize a group or subgroup, so all projects within have access. For added security, you can also use impersonation.

Authorization configuration can take one or two minutes to propagate.

Authorize the agent to access your projects

Introduced in GitLab 14.4.

To authorize the agent to access the GitLab project where you keep Kubernetes manifests:

1. On the top bar, select Menu > Projects and find the project that contains the agent configuration file (config.yaml).

2. Edit the config.yaml file. Under the ci_access keyword, add the projects attribute.

3. For the id, add the path:

   ci_access:
     projects:
       - id: path/to/project

   • The Kubernetes projects must be in the same group hierarchy as the project where the agent's configuration is.
   • You can install additional agents into the same cluster to accommodate additional hierarchies.
   • You can authorize up to 100 projects.

All CI/CD jobs now include a KUBECONFIG with contexts for every shared agent connection. Choose the context to run kubectl commands from your CI/CD scripts.

Authorize the agent to access projects in your groups

Introduced in GitLab 14.3.

To authorize the agent to access all of the GitLab projects in a group or subgroup:

1. On the top bar, select Menu > Projects and find the project that contains the agent configuration file (config.yaml).

2. Edit the config.yaml file. Under the ci_access keyword, add the groups attribute.

3. For the id, add the path:

   ci_access:
     groups:
       - id: path/to/group/subgroup

   • The Kubernetes projects must be in the same group hierarchy as the project where the agent's configuration is.
   • You can install additional agents into the same cluster to accommodate additional hierarchies.
   • All of the subgroups of an authorized group also have access to the same agent (without being specified individually).
   • You can authorize up to 100 groups.

All the projects that belong to the group and its subgroups are now authorized to access the agent. All CI/CD jobs now include a KUBECONFIG with contexts for every shared agent connection. Choose the context to run kubectl commands from your CI/CD scripts.

Update your .gitlab-ci.yml file to run kubectl commands

In the project where you want to run Kubernetes commands, edit your project's .gitlab-ci.yml file.

In the first command under the script keyword, set your agent's context. Use the format path/to/agent/repository:agent-name. For example:

deploy:
  image:
    name: bitnami/kubectl:latest
    entrypoint: ['']
  script:
    - kubectl config get-contexts
    - kubectl config use-context path/to/agent/repository:agent-name
    - kubectl get pods

If you are not sure what your agent's context is, open a terminal and connect to your cluster. Run kubectl config get-contexts.

Environments with both certificate-based and agent-based connections

When you deploy to an environment that has both a certificate-based cluster (deprecated) and an agent connection:

• The certificate-based cluster's context is called gitlab-deploy. This context is always selected by default.
• In GitLab 14.9 and later, agent contexts are included in the KUBECONFIG. You can select them by using kubectl config use-context path/to/agent/repository:agent-name.
• In GitLab 14.8 and earlier, you can still use agent connections, but for environments that already have a certificate-based cluster, the agent connections are not included in the KUBECONFIG.

To use an agent connection when certificate-based connections are present, you can manually configure a new kubectl configuration context. For example:

deploy:
  variables:
    KUBE_CONTEXT: my-context # The name to use for the new context
    AGENT_ID: 1234 # replace with your agent's numeric ID
    K8S_PROXY_URL: https://<KAS_DOMAIN>/k8s-proxy/ # For agent server (KAS) deployed in Kubernetes cluster (for gitlab.com use kas.gitlab.com); replace with your URL
    # K8S_PROXY_URL: https://<GITLAB_DOMAIN>/-/kubernetes-agent/k8s-proxy/ # For agent server (KAS) in Omnibus
    # ... any other variables you have configured
  before_script:
    - kubectl config set-credentials agent:$AGENT_ID --token="ci:${AGENT_ID}:${CI_JOB_TOKEN}"
    - kubectl config set-cluster gitlab --server="${K8S_PROXY_URL}"
    - kubectl config set-context "$KUBE_CONTEXT" --cluster=gitlab --user="agent:${AGENT_ID}"
    - kubectl config use-context "$KUBE_CONTEXT"
  # ... rest of your job configuration

Restrict project and group access by using impersonation (PREMIUM)

Introduced in GitLab 14.5.

By default, your CI/CD job inherits all the permissions from the service account used to install the agent in the cluster. To restrict access to your cluster, you can use impersonation.

To specify impersonations, use the access_as attribute in your agent configuration file and use Kubernetes RBAC rules to manage impersonated account permissions.

You can impersonate:

• The agent itself (default).
• The CI/CD job that accesses the cluster.
• A specific user or system account defined within the cluster.

Authorization configuration can take one or two minutes to propagate.

Impersonate the agent

The agent is impersonated by default. You don't need to do anything to impersonate it.

Impersonate the CI/CD job that accesses the cluster

To impersonate the CI/CD job that accesses the cluster, under the access_as key, add the ci_job: {} key-value.

When the agent makes the request to the actual Kubernetes API, it sets the impersonation credentials in the following way:

• UserName is set to gitlab:ci_job:<job id>. Example: gitlab:ci_job:1074499489.

• Groups is set to:

  • gitlab:ci_job to identify all requests coming from CI jobs.

  • The list of IDs of groups the project is in.

  • The project ID.

  • The slug of the environment this job belongs to.

    Example: for a CI job in group1/group1-1/project1 where:

    • Group group1 has ID 23.
    • Group group1/group1-1 has ID 25.
    • Project group1/group1-1/project1 has ID 150.
    • Job running in a prod environment.

  Group list would be [gitlab:ci_job, gitlab:group:23, gitlab:group:25, gitlab:project:150, gitlab:project_env:150:prod].

• Extra carries extra information about the request. The following properties are set on the impersonated identity:

Example config.yaml to restrict access by the CI/CD job's identity:

ci_access:
  projects:
    - id: path/to/project
      access_as:
        ci_job: {}

Example RBAC to restrict CI/CD jobs

The following RoleBinding resource restricts all CI/CD jobs to view rights only.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ci-job-view
roleRef:
  name: view
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
subjects:
  - name: gitlab:ci_job
    kind: Group

Impersonate a static identity

For a given connection, you can use a static identity for the impersonation.

Under the access_as key, add the impersonate key to make the request using the provided identity.

The identity can be specified with the following keys:

• username (required)
• uid
• groups
• extra

See the official Kubernetes documentation for details.

Related topics

• Self-paced classroom workshop (Uses AWS EKS, but you can use for other Kubernetes clusters)

Troubleshooting

kubectl commands not supported

The commands kubectl exec, kubectl cp, kubectl attach, kubectl run --attach=true and kubectl port-forward are not supported. Anything that uses these API endpoints does not work, because they use the deprecated SPDY protocol. An issue exists to add support for these commands.

Grant write permissions to ~/.kube/cache

Tools like kubectl, Helm, kpt, and kustomize cache information about the cluster in ~/.kube/cache. If this directory is not writable, the tool fetches information on each invocation, making interactions slower and creating unnecessary load on the cluster. For the best experience, in the image you use in your .gitlab-ci.yml file, ensure this directory is writable.

Enable TLS

If you are on a self-managed GitLab instance, ensure your instance is configured with Transport Layer Security (TLS).

If you attempt to use kubectl without TLS, you might get an error like:

$ kubectl get pods
error: You must be logged in to the server (the server has asked for the client to provide credentials)