View source for Equipment/Blanton/Kubernetes

= Intro =

This page provides a bit of a braindump on the Kubernetes setup.

There is (currently) one master node on [[Equipment/Blanton|Blanton]]. I have thought about adding one on Landin, but it would require careful thought, since an even number of masters is generally discouraged since it can lead to split brain scenarios. Ideally we'd have a third machine to keep the number of masters odd.

{| class="wikitable"
! Node
! Role
! Location
! Notes
|- 
| kube-master || Master || [[Equipment/Blanton|Blanton]] || 4 cores, 4GB of RAM
|-
| kube-node-blanton || Worker || [[Equipment/Blanton|Blanton]] || 8 cores, 8GB of RAM
|-
| kube-node-landin || Worker || [[Equipment/Landin|Landin]] ||  8 cores, 8GB of RAM
|-
| kube-node || Worker || [[Equipment/Blanton|Blanton]] || 8 cores, 8GB of RAM<br> Older worker, mostly providing redundant capacity when one of the others are down for maintainence
|}

As of writing, all of our containerised services can run on one of the nodes.

= General Notes =
I did try doing something with docker-compose, but the networking got unwealdy fast, and I realised I was about to create something not unlike Kubernetes but badly in a bunch of scripts!
A big sticking point of what took so long to get this working was the dual stack IPv4 and IPv6 support needed to fit into the rest of the hackspace environment,

A few quick notes:

* Networking is provided by Calico
* LoadBalancer requests are serviced by metallb
** If you want both IPv4 and IPv6 you will need to create two LoadBalancer instances pointing to the same service
* nginx-ingress is configured to support HTTP/HTTPS services
* cert-manager is configured to issue LetsEncrypt certificates automatically, assuming DNS entries are already in place (as would be needed for a regular VM wanting a cert)
** Mark your ingress with the annotation ''cert-manager.io/cluster-issuer: "letsencrypt-prod"''
* there's a single-node glusterfs "cluster" providing storage
* While it's all currently on Blanton, if there was another box (or ideally two) available, it would be possible to make this much more resilient
* It's running a bleeding edge version of cert-issuer and ingress-nginx because I updated to 1.22 before things were ready :-)

MetalLB is configured to allocate IP addresses in the ranges 10.0.21.128/25 and 2a00:1d40:1843:182:f000::/68 - it uses layer 2 ARP to advertise these on the LAN.

= Accessing the cluster =

'''Currently (useful) access is limited to those in the Admins ldap group'''. After a bit more testing though, I'd like to create a members namespace for members to play in.

First, go get kubectl (preferably version 1.22.4 which, at the time of writing this, is the version the cluster runs) https://kubernetes.io/docs/tasks/tools/

Now go get the binary for your OS from https://github.com/londonhackspace/kube-auth-handler/releases/ and put it somewhere on your PATH as <code>ldap-kube-auth</code>

'''Note: This is only tested on Linux so far, but should work on any OS supported by kubectl, possibly with some tweaks.'''

Now stick the following in $HOME/.kube/config (if you already have a config in there, you'll need to merge them)

<pre>
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM1ekNDQWMrZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1EVXdOVEUyTXpFeE5sb1hEVE14TURVd016RTJNekV4Tmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTnZQClVia1htT21JcnRiSHhZZHdBd3ZIaUoxVmhORUFkMXpiNlVKdXI5MFJSNHJ1VTlIRjZIaWpVd3Z3YzQ4MGh0K1UKY3F0dlZ6TG55L29OWTBNM01DQkxTRU90VndiMlQ0RXB3SUtVQ3drdENOcXV0MTA4eGhvRjlFOUp5NnhPWG9FMQp6c0xMeWQwcm1WNzFmU1NsUGg1RG9SRWoxSnNUZFdKQitJL21MNDhDd2k2aENEaXdnU25ZeFBEWitCaEFvTkVNCkxiMm1rdDVFdXVXUmVhME40bGl0cThpWktYZy9VcEh2aUJvaGxWLzMvemJWWnZnY3RyTlBrcWJvZDdUM2pYa1oKZnNwR1VjU1g1aVRKMUJpUHowZ0dlNEJ0SndvT2tVdGNjRUk0M0o1aXhQMi9hWVYwcURCd1B3MWpmVUFDaGZ5RAptbkdUVUVCSUNzYWVnY3UweFVVQ0F3RUFBYU5DTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZGNlZkVFFoOWVGdzUzMWEydEh4S3ZmWkxqMDZNQTBHQ1NxR1NJYjMKRFFFQkN3VUFBNElCQVFDWVpBcDRqdWl5MjdoK041aXl5RVczNFYzMExRc2pxOHpWdlF5OHFORVJtUS9IRDVnQQpQQkFlWFpPdXZHaklHcVB2TS9BeWZYempPL25MK09aS1Vvay91eWFVUkxHYU1rSzBqVzVMb3ZOU28zVFMwTEU4CnJBQlRjeUdLYW40Tk9zN2FoSWM4SGdxa2lNOUR2cVhiRGlwbC9pbG95VnJCRDh3bzZ3RVBPSG1tUGZJdUJZYnUKUzNZY0JXRW90bVpSNWlxTWxWdDRIUXR0MUYzMmZyS21lcmFPNFdUM2hIMkRWNTQ3enlSYmF4cktOZ1VJL0FJQgoycG53dktYNlUyNFM0Z0lUbW8zTUZzNDNoTzk4NDU2YXYrVExrTkNrYVhCVktMaUtMU2NZMXJnQlFzN0JvSHp4CnR1aU5nQ241dnVLRTBWU0diK2FhTTgzdVl6Q001TjVuVm9xbAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://kube-master.lan.london.hackspace.org.uk:6443
  name: lhs
contexts:
- context:
    cluster: lhs
    namespace: default
    user: lhsuser
  name: lhs
current-context: lhs
kind: Config
preferences: {}
users:
- name: lhsuser
  user:
    exec:
      command: /usr/local/bin/ldap-kube-auth
      env:
      - name: AUTH_CLUSTER
        value: LHS
      - name: AUTH_URL
        value: https://kube-auth.lan.london.hackspace.org.uk/getToken
      interactiveMode: Always
      apiVersion: "client.authentication.k8s.io/v1"
</pre>

now you should be able to run commands:
<pre>
$ kubectl get pods
kubectl LDAP login helper
Logging into LHS
Press enter for defaults
Username (michael): mich181189
Password: NAME                         READY   STATUS    RESTARTS   AGE
dockerreg-8568799fdb-vwtgg   1/1     Running   0          173m
</pre>

== The slightly gory details ==

Kubernetes does not natively support LDAP authentication, so we use webhook tokens (https://kubernetes.io/docs/reference/access-authn-authz/authentication/#webhook-token-authentication). To make this easier to use, there is an credential plugin (https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins)

The sources for the server and client implementations of this are on Github: https://github.com/londonhackspace/kube-auth-handler

Tokens are valid for 6 hours, and get cached in $HOME/.kube/LHS_cache (assuming the above config is used) - delete this file if you think the cache is causing you problems. The caching is very necessary otherwise you get a prompt (or sometimes multiple prompts!) per command. Server-side, while JWTs might be popular, this service just generates random strings and stashes them in redis for later lookup

Kubernetes is configured to check these tokens against a web service running in the cluster. This is done with the following chunk of kubeadm-config

<pre>
apiVersion: v1
data:
  ClusterConfiguration: |
    apiServer:
      extraArgs:
        authentication-token-webhook-config-file: /etc/kubernetes/auth/ldap-auth.yaml
        authentication-token-webhook-version: v1
</pre>

and a patch passed to kubeadm to get it to mount /etc/kubernetes/auth into the apiserver container. If this patch is not used, it will cause the API server to refuse to start! (see the upgrade instructions below for how to specify the patch directory to kubeadm upgrade)

The server deployment is in the (private) github repo kubernetes-config - this is just a fairly standard kubernetes service deployment.

= Instructions Braindump For Admins =

== Adding a node ==
Kubernetes mostly requires a basic OS install, but there are a few steps you need to make sure you do correctly. 

A key point here is that until recently, Kubernetes didn’t support nodes with swap enabled. These instructions therefore do not have swap. (I’m still not entirely convinced it’s a good idea!)

# Install latest Debian on a VM (without swap) with SSH and standard system utilities
# Stick your SSH key into /root/.ssh/authorized_keys and /home/<you>/.ssh/authorized_keys
# Add to the lhs-hosts section of Ansible (all nodes starting with kube-* get some basic kubernetes requirements installed) and deploy to it
# On the master, run <pre>kubeadm token create</pre> to get a token
# On the master run to get the cert hash:
#: <pre>
#:: openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
#:: openssl dgst -sha256 -hex | sed 's/^.* //'</pre>
# On the new node, run <pre>kubeadm join --token <token> kube-master.lan.london.hackspace.org.uk:6443 --discovery-token-ca-cert-hash sha256:<hash></pre>

== Draining a node for mantainence ==
It's a good idea to shift work off a node when you're about do to anything to it (upgrade, reboot, e.t.c)

# run <code>kubectl drain <node> --ignore-daemonsets</code>

when you're done with the maintainence:

# run <code> kubectl uncordon <node></code>

You might want to delete some pods that are running on remaining nodes so they get restarted more evenly spread across the nodes. Alternatively, you might want to just wait for usual updates and stuff to restart pods

== Removing a node ==

# Drain the node
#: <pre> kubectl drain <node></pre>
# Delete the node record from Kubernetes
#: <pre> kubectl delete node <node></pre>
# Probably delete the VM or something - it's done now

== Upgrading Kubernetes ==
Full instructions here: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/ READ THEM!

It’s perhaps worth ignoring the 1.x.0 releases, since experience suggests things like metallb and callico might not yet support it in a stable version, which is a recipe for pain.

# On the master node, run <code>apt-cache madison kubeadm</code>  to find a version to update to
# On the master node, run:
#: <pre>sudo apt-get install -y --allow-change-held-packages kubeadm=<your-chosen-version></pre>
#: <pre>sudo kubeadm upgrade plan</pre>
#: <pre>sudo kubeadm upgrade apply --patches=/etc/kubernetes/patches v<your-chosen-version></pre>
#: <pre>sudo apt-get install -y --allow-change-held-packages kubelet=<your-chosen-version> kubectl=<your-chosen-version></pre>


On each node:

# Drain the node
# Run:
#: <pre>sudo apt-get install -y --allow-change-held-packages kubeadm=<your-chosen-version></pre>
#: <pre>sudo kubeadm upgrade node</pre>
#: <pre>sudo apt-get install -y --allow-change-held-packages kubelet=<your-chosen-version> kubectl=<your-chosen-version></pre>
# run <code>kubectl get nodes</code> wherever you normally run kubectl to make sure the node is running the expected version
# Uncordon node

== Fixing Screwups ==
=== Re-adding a node you removed by mistake ===
If you accidentaly run <pre>kubectl delete node <node></pre> when you didn't mean to, don't panic - the workload should be shifted automatically to
a remaining node. here's how to re-add the one you just removed:

# Run <code>kubeadm reset</code> on the affected node
# Run <code>kubeadm join</code> as if it was a new node