Kubernetes, GitLab and DigitalOcean

Prerequisites

Before you can link your cluster to GitLab, you need to have a DigitalOcean account and access to GitLab. It doesn’t matter what version of GitLab you are using. It can be your own installation or gitlab.com.

Kubernetes

Getting Kubernetes on DigitalOcean is fairly straight forward. You can add a new cluster in the cloud interface. Click on Create and head for Clusters. The wizard will guide you through the process. If you plan to use Knative, make sure your cluster has at least 3 nodes, 6 vCPUs and 22.5GB memory.

Install doctl

doctl is the command line interface to interact with DigitalOcean, similar to cli-tools from other cloud providers. The installation instructions are available on GitHub. You need this to get the kubectl configuration for Kubernetes.

When doctl is installed you can authenticate with DigitalOcean.

1. Create an API-token.

2. Run doctl auth init. It will ask you for the token you’ve created.

You now have access to your project at DigitalOcean.

Install kubectl

kubectl is used to interact with Kubernetes. You can find the installation instructions for all operating systems on the kubernetes.io website.

When you’ve got kubectl installed, you can use doctl to get and save the Kubernetes configuration from DigitalOcean.

doctl kubernetes cluster kubeconfig save <cluster>
doctl kubernetes cluster kubeconfig save <cluster>

DigitalOcean rotates all cluster certificates every 7 days. And although this is a good thing, it means you will have to update your configuration occasionally too.

Add Kubernetes to GitLab

Credits to Stepan Kuzmin for his gist and GitLab for the video, most of which you can find below.

Add an existing Kubernetes cluster

When you’re adding a Kubernetes cluster to your project or installation, add an existing one. The steps below will help you get an API URL, CA Certificate and Service Token.

Create a service account:

kubectl create -f - <<EOF
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: gitlab
    namespace: default
EOF
kubectl create -f - <<EOF
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: gitlab
    namespace: default
EOF

Create the cluster role giving the gitlab-account cluster-admin privileges.

kubectl create -f - <<EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-cluster-admin
subjects:
- kind: ServiceAccount
  name: gitlab
  namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
EOF
kubectl create -f - <<EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-cluster-admin
subjects:
- kind: ServiceAccount
  name: gitlab
  namespace: default
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
EOF

Get the API URL.

kubectl cluster-info | grep 'Kubernetes master' | awk '/http/ {print $NF}'
kubectl cluster-info | grep 'Kubernetes master' | awk '/http/ {print $NF}'

Get the CA Certificate.

kubectl get secret $(kubectl get secrets | awk '/gitlab/ {print $1}') -o jsonpath="{['data']['ca\.crt']}" | base64 -D
kubectl get secret $(kubectl get secrets | awk '/gitlab/ {print $1}') -o jsonpath="{['data']['ca\.crt']}" | base64 -D

Get the Service Token.

kubectl get secret $(kubectl get secrets | awk '/gitlab/ {print $1}') -o jsonpath="{['data']['token']}" | base64 -D
kubectl get secret $(kubectl get secrets | awk '/gitlab/ {print $1}') -o jsonpath="{['data']['token']}" | base64 -D

Make sure you keep RBAC-enabled cluster and GitLab-managed enabled.

Install helper applications

As soon as the cluster is added you will be able to add a number of helper applications that GitLab provides.

1. Install Helm Tiller

2. Install Ingress*

3. Install Cert-Manager

4. Install Prometheus

If you want to run your CI jobs on Kubernetes you can install the runner. I won’t cover this, as there are a few caveats when you do, Docker in Docker being one of them, but it’s still pretty cool.

Once the helper applications are installed you should see the cluster health in your dashboard. This means Prometheus is running and things should be in good health.

Setting up your app in GitLab

It’s time to deploy our application. Let’s start with a basic .gitlab-ci.yml.

variables:
 K8S_NAMESPACE: $CI_PROJECT_PATH_SLUG-$CI_COMMIT_REF_SLUG
 
stages:
  - deploy
 
deploy-k8s:
  image: dtzar/helm-kubectl
  stage: deploy
  before_script:
    - apk add -u gettext
    - kubectl create secret docker-registry regcred --docker-server=$CI_REGISTRY --docker-username="$CI_DEPLOY_USER" --docker-password="$CI_DEPLOY_PASSWORD" --docker-email="$GITLAB_USER_EMAIL" --dry-run=true -o yaml | kubectl apply -f -
  environment:
    name: test
    url: test.yourapp.com
  script:
    - VERSION="${CI_COMMIT_TAG:-none}" envsubst < k8s.yml | kubectl apply -f -
variables:
 K8S_NAMESPACE: $CI_PROJECT_PATH_SLUG-$CI_COMMIT_REF_SLUG
 
stages:
  - deploy
 
deploy-k8s:
  image: dtzar/helm-kubectl
  stage: deploy
  before_script:
    - apk add -u gettext
    - kubectl create secret docker-registry regcred --docker-server=$CI_REGISTRY --docker-username="$CI_DEPLOY_USER" --docker-password="$CI_DEPLOY_PASSWORD" --docker-email="$GITLAB_USER_EMAIL" --dry-run=true -o yaml | kubectl apply -f -
  environment:
    name: test
    url: test.yourapp.com
  script:
    - VERSION="${CI_COMMIT_TAG:-none}" envsubst < k8s.yml | kubectl apply -f -

In this deployment we’re using a few predefined variables.

$KUBE_NAMESPACE — is the namespace we get from GitLab.
$GITLAB_USER_EMAIL — is your e-mail address.
$CI_DEPLOY_USER — is the user name of the read only deploy token.
$CI_DEPLOY_PASSWORD — is the key of the deploy token.

These last two variables need some further explanation.

Deploy tokens

In most environments your app sits in a private registry. GitLab creates tokens that can be used for temporary access. Although Kubernetes could use these, it would break whenever you start scaling your nodes. It wouldn’t be able to use the expired token and Kubernetes will not be able to launch new pods.

This is why we’re using Deploy Tokens. A deploy token is persistent. To use them, you need to create a secret in Kubernetes.

1. Create a read only Deploy Token, use gitlab-deploy-token as its name. It only needs access to read the registry. The username is irrelevant.

2. If you have named the token gitlab-deploy-token, your CI job will find the username and token automatically.

Our before_script creates a secret for you with the name regcred. In most cases there is no need to change this name as we’re deploying each app in its own namespace, but feel free to do. You will need it in your deployment. It should be added to the spec of your deployment template like so.

...
imagePullSecrets:
- name: regcred
...
...
imagePullSecrets:
- name: regcred
...

Preparing for deployment

Once the deploy token is configured and the actual job is kicked off, there are a few things happening that need an explanation. You may want to keep a copy of the gist open while reading the below.

The kubectl commands are different from what you would expect if you have worked with Kubernetes before. We’re using --dry-run=true to get the YAML output and apply this instead.

... --dry-run=true -o yaml | kubectl apply -f -
... --dry-run=true -o yaml | kubectl apply -f -

You may wonder why. Running kubectl create would have created the same resource, right? Although strictly true, Kubernetes will throw errors when you try to create that same resource. This is why you need apply instead of create.

Deploying your app

The script to deploy your application is pretty straight forward:

envsubst < k8s.yml | kubectl apply -f -
envsubst < k8s.yml | kubectl apply -f -

We have prefixed it in the gist with a VERSION variable as we use this to label our objects. It helps to quickly track what version a deployment is running.

More on envsubst below.

Your deployment

I won’t go into the details of setting up a deployment, service or ingress. It’s up to you how you configure your deployments. There are some things that may help you get up to speed faster though. I will cover these below.

Variables

envsubst replaces environment variables in standard input. This means we can create YAML-files like this:

...
  labels:
    app: your-app
    commit: ${CI_COMMIT_SHORT_SHA}
    version: ${VERSION}
...
...
  labels:
    app: your-app
    commit: ${CI_COMMIT_SHORT_SHA}
    version: ${VERSION}
...

And pass them to envsubst with envsubst < k8s.yml. This will be copied to standard output and can then be used to create your resources.

We’re using a single file for all environments. We’re keeping it DRY.

Ingress load balancing

Remember the applications you have deployed?

It includes an ingress controller that is linked to a load balancer. To use it, you have to annotate the ingress object you’re creating.

...
metadata:
  name: ingress-your-app
  annotations:
    kubernetes.io/ingress.class: "nginx"
...
...
metadata:
  name: ingress-your-app
  annotations:
    kubernetes.io/ingress.class: "nginx"
...

This will route traffic from the ingress controller to your ingress object, to your service and your pods.

Other useful annotations can be found here.

SSL and Let's Encrypt

Remember the Cert-Manager too?

It can automatically assign and renew certificates from Let’s Encrypt. To use it, you have to annotate the ingress object and add tls to your spec.

...
metadata:
  name: ingress-your-app
  annotations:
    ...
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    ...
...
spec:
  tls:
  - hosts:
    - test.yourapp.com
    secretName: test-yourapp-com
...
metadata:
  name: ingress-your-app
  annotations:
    ...
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    ...
...
spec:
  tls:
  - hosts:
    - test.yourapp.com
    secretName: test-yourapp-com

It can take a while for your certificate to be available. Be patient and make sure your DNS has been set — use a wildcard if you’re using review apps in GitLab to make things easier.

Other useful annotations can be found here.

Monitoring

Last but not least, Prometheus.

Make sure your deployment, service and ingress are prefixed with the environment slug like so:

...
metadata:
  name: ${CI_ENVIRONMENT_SLUG}-your-app
...
...
metadata:
  name: ${CI_ENVIRONMENT_SLUG}-your-app
...

This makes sure GitLab knows where to pick up data from Prometheus.

If you’re on the silver or equivalent plans and up and you would like to use the deploy boards feature, make sure your annotations are set up on the deployment itself AND the spec template:

...
metadata:
...
  annotations:
    app.gitlab.com/env: $CI_ENVIRONMENT_SLUG
    app.gitlab.com/app: $CI_PROJECT_PATH_SLUG
...
...
metadata:
...
  annotations:
    app.gitlab.com/env: $CI_ENVIRONMENT_SLUG
    app.gitlab.com/app: $CI_PROJECT_PATH_SLUG
...

The 'magic' namespace

If you’re a little familiar with Kubernetes you might have noticed .gitlab-ci.yml has not been configured to create a namespace. Its creation is hidden from your CI job.

On the first deployment of your app, GitLab will create the $KUBE_NAMESPACE, service accounts and access tokens. Subsequent jobs will use these credentials to deploy your application.

There’s a downside to this.

If you decide to delete this namespace, your jobs will fail. GitLab stores the namespace in a database and once it’s there, it is assumed that it is never deleted. There is no reality check — and that’s a mistake.

In case you do decide to delete the namespace and want to start over, you need to remove the entry from GitLab’s database.

# gitlab-rails db
gitlabhq_production=> select * from clusters_kubernetes_namespaces;
gitlabhq_production=> delete from clusters_kubernetes_namespaces where namespace = '<namespace>';
# gitlab-rails db
gitlabhq_production=> select * from clusters_kubernetes_namespaces;
gitlabhq_production=> delete from clusters_kubernetes_namespaces where namespace = '<namespace>';

Wrapping up: the Kubernetes dashboard

The Kubernetes dashboard is a web-based user interface. It makes it easier to troubleshoot your application(s). You can install the dashboard in the kube-system namespace with the following command:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

Read more on how to connect to it on the Web UI documentation pages.

Closing words

Thanks for reading this. I hope this article was useful and helps you get the most out of your GitLab installation together with DigitalOcean.