Monday, October 28, 2019

k8s cluster on raspberry pi 4


(Note: All information is from the internet)

A personal note about installing k8s on Raspberry Pi4

A messy setup before got the cluster case..

In my setup, the cluster run on an isolated LAN  connected via a fast switch. The cluster can be reached from outside via the Wifi connection.

A) On Master Node

1.  install docker and change dockerd cgroup to systemd
curl -sSL get.docker.com | sh && \
sudo usermod pi -aG docker

    After install the docker, update the line /lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --exec-opt native.cgroupdriver=systemd

2. Add a default route so that kubeadm doesn't complains, because
   - I use a isolated network 10.10.10.0/16 for inter-nodes communication.
   - Connection to Internet using wifi wlan0 interface 192.168.1.0/24 network
   - this is to avoid the cert generate using the 192.168.1.x IP and services bound to that... wlan0
   # sudo ip route add default via 10.10.10.10
   Further connection has to jump from other 10.10.10.x nodes (via 192.168.1.11 / 10.10.10.11 to 10.10.10.10)

3. kubeadm reset
  kubeadm reset is not necessary if you do a fresh installation

4. Aim to use flannel as CNI, so start with
   $ sudo kubeadm init --control-plane-endpoint 10.10.10.10 --pod-network-cidr=10.244.0.0/16

5. remove & recreate the .kube
  rm -rf $HOME/.kube
  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config


6. Next keep the message of installation below:
---
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join 10.10.10.10:6443 --token gavux7.hgj1dk2xpfuulggg \
    --discovery-token-ca-cert-hash sha256:248dea9ecc6691b8d2738fd3571a560e468caadb3ab19a56b494652912373b9e \
    --control-plane

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.10.10.10:6443 --token gavux7.hgj1dk2xpfuulggg \
    --discovery-token-ca-cert-hash sha256:248dea9ecc6691b8d2738fd3571a560e468caadb3ab19a56b494652912373b9e
---


7. install flannel
   Since we need to connect to Internet, we need to disable to default route set in step (2)
   One of the easier way is "reboot" and back to normal state before continue

   $ kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created

   Check there is a new network interface create


8. Check the installation
pi@rbpi4-n0:~/system_setup $ kubectl get nodes
NAME       STATUS   ROLES    AGE     VERSION
rbpi4-n0   Ready    master   8m34s   v1.16.2
pi@rbpi4-n0:~/system_setup $


9. Setup Worker nodes in section (B)

10. Setup dashboard
$  kube apply -f kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta5/aio/deploy/recommended.yaml

11. Check status
pi@rbpi4-n0:~/system_setup $ kubectl get pods --all-namespaces -o wide
NAMESPACE              NAME                                         READY   STATUS              RESTARTS   AGE     IP             NODE       NOMINATED NODE   READINESS GATES
kube-system            coredns-5644d7b6d9-rbdqr                     1/1     Running             0          22m     10.244.0.2     rbpi4-n0   <none>           <none>
kube-system            coredns-5644d7b6d9-xpftj                     1/1     Running             0          22m     10.244.0.3     rbpi4-n0   <none>           <none>
kube-system            etcd-rbpi4-n0                                1/1     Running             1          21m     192.168.1.10   rbpi4-n0   <none>           <none>
kube-system            kube-apiserver-rbpi4-n0                      1/1     Running             1          21m     192.168.1.10   rbpi4-n0   <none>           <none>
kube-system            kube-controller-manager-rbpi4-n0             1/1     Running             1          21m     192.168.1.10   rbpi4-n0   <none>           <none>
kube-system            kube-flannel-ds-arm-4cmf2                    1/1     Running             0          14m     192.168.1.10   rbpi4-n0   <none>           <none>
kube-system            kube-flannel-ds-arm-9vgkb                    1/1     Running             0          6m20s   192.168.1.11   rbpi4-n1   <none>           <none>
kube-system            kube-flannel-ds-arm-znm4h                    1/1     Running             0          2m27s   192.168.1.12   rbpi4-n2   <none>           <none>
kube-system            kube-proxy-kffjb                             1/1     Running             0          2m27s   192.168.1.12   rbpi4-n2   <none>           <none>
kube-system            kube-proxy-qxt92                             1/1     Running             1          22m     192.168.1.10   rbpi4-n0   <none>           <none>
kube-system            kube-proxy-rm97m                             1/1     Running             0          6m20s   192.168.1.11   rbpi4-n1   <none>           <none>
kube-system            kube-scheduler-rbpi4-n0                      1/1     Running             1          21m     192.168.1.10   rbpi4-n0   <none>           <none>
kubernetes-dashboard   dashboard-metrics-scraper-566cddb686-x5k9c   0/1     ContainerCreating   0          94s     <none>         rbpi4-n1   <none>           <none>
kubernetes-dashboard   kubernetes-dashboard-7b5bf5d559-vkknw        0/1     ContainerCreating   0          95s     <none>         rbpi4-n1   <none>           <none>
pi@rbpi4-n0:~/system_setup $


B) On Worker Node


1) Reset and start again
pi@rbpi4-n1:~ $ sudo kubeadm reset
[reset] WARNING: Changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] Are you sure you want to proceed? [y/N]: y
[preflight] Running pre-flight checks
W1028 18:51:58.108270   24830 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory
[reset] No etcd config found. Assuming external etcd
[reset] Please, manually reset etcd to prevent further issues
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes /var/lib/cni]

 To avoid complication, advise to reboot the node and start again. Clean the /etc/cabinet files except the manifest

2) Join the cluster as worker node (not control plane node)

$ sudo -i
# root@rbpi4-n1:~# kubeadm join 10.10.10.10:6443 --token gavux7.hgj1dk2xpfuulggg \
>     --discovery-token-ca-cert-hash sha256:248dea9ecc6691b8d2738fd3571a560e468caadb3ab19a56b494652912373b9e
[preflight] Running pre-flight checks
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 19.03.4. Latest validated version: 18.09
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.16" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster

(Repeat the same for all worker node)

KBTalking Pro Quickstart guide

Found a legacy card while doing the housekeeping... If you have a KBTalking Pro USB + Bluetooth mechanical keyboards and you have...