Kubernetes on Novena

Kubernetes is a high-availability clustering package. It is available for ARM, but mostly is used on x86-64. These steps detail how to use it on Novena.

These steps assume you're starting from a base Novena image, such as the one found here on the Kosagi wiki.

Installing Kubernetes

Some steps I've taken to install Kubernetes on Debian. Adapted from https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/ for Novena.

0. Remove software that comes installed by default on Novena, but is unnecessary for servers.

apt-get install -y network-manager
apt-get remove -y --auto-remove --purge hexchat pidgin bluez bluez-tools bluez-hcidump \
       enigmail hexchat icedove iceweasel pavucontrol x11-xserver-utils xscreensaver \
       libbluetooth3 keychain xserver-xorg-video-modesetting arandr android-tools-adb \
       android-tools-fastboot android-tools-fsutils xfce4-goodies xfce4-power-manager \
       xfce4-mixer xfce4-terminal mousepad orage dbus-x11 irssi synaptic qalc libqt5gui5 \
       libqt5core5a libqt5widgets5 x11-apps x11-session-utils xbitmaps xfce4 xfce4-appfinder \
       xfce4-notifyd xfce4-session xfce4-settings xfdesktop4 xfdesktop4-data xfonts-100dpi \
       xfonts-75dpi xfonts-scalable xfwm4 xfwm4-themes xinit xorg xorg-docs-core \
       libdrm-armada2-dbg libetnaviv-dev libetnaviv-dbg novena-disable-ssp novena-eeprom-gui \
       novena-firstrun xorg-novena xserver-xorg-video-armada xserver-xorg-video-armada-dbg \
       xserver-xorg-video-armada-etnaviv

1. Update the key.

wget http://repo.novena.io/debian/pool/main/k/kosagi-repo/kosagi-repo_1.2-r1_all.deb
sudo dpkg -i kosagi-repo_1.2-r1_all.deb
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade
apt-get install apt-transport-https # Used for accessing the newer repos

2. At this point, it's probably a good idea to move to a SATA disk.

dd if=/dev/mmcblk0 of=/dev/sda bs=1M count=1
fdisk /dev/sda
# Re-create partitions 2 and 3.  Make partition 2 at least 6GB
# for swap (type 82), and make partition 3 the rest of the disk.
# Type "x" for Expert mode, then "i", and change the ID to 0x4e6f7653.
# Then "r" to return to the main menu, and "w" to write it to disk.
mkfs.ext4 /dev/sda3 # or install btrfs-tools or xfsprogs and make a different type
mount /dev/sda3 /mnt
rsync -avxHAX --progress / /mnt/
novena-eeprom -w -f es8328,pcie,gbit,hdmi,eepromoops,sataroot
reboot

3. Install Docker (borrowed from https://github.com/hypriot/image-builder-odroid-c1/blob/master/builder/chroot-script.sh). If you need a specific version, you can list them with:

apt-cache madison docker-engine

wget -q https://packagecloud.io/gpg.key -O - | apt-key add -
echo 'deb https://packagecloud.io/Hypriot/Schatzkiste/debian/ jessie main' > /etc/apt/sources.list.d/hypriot.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 2C52609D
echo 'deb [arch=armhf] https://apt.dockerproject.org/repo debian-jessie main' > /etc/apt/sources.list.d/docker.list
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install \
   docker-engine=1.12.6-0~debian-jessie \
   docker-compose="1.9.0-23" \
   docker-machine="0.9.0-39"

4. Install kubeadm

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install -y kubeadm

5. Run kubeadm on the master device to start the cluster.

kubeadm init --pod-network-cidr 10.244.0.0/16

It will output something like the following:

Your Kubernetes master has initialized successfully!

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
    http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node:

kubeadm join --token=964a50.d8053ed3de195a11 10.0.245.169

6. Join the cluster from other machines.

kubeadm join --token=964a50.d8053ed3de195a11  10.0.245.169

7. Ensure cAdvisor doesn't start up. It's nice to have, but it leaks lots of information.

printf '[Service]\nEnvironment="KUBELET_EXTRA_ARGS=--cadvisor-port=0"\n' > /etc/systemd/system/kubelet.service.d/05-disable-cadvisor.conf
systemctl daemon-reload
systemctl restart kubelet

8. Install Flannel on the Master, which will manage the network for us.

curl -sSL https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

9. Wait for all pods to be Running:

kubectl get po --all-namespaces