Difference between revisions of "Kubernetes on Novena"

From Studio Kousagi Wiki
Jump to: navigation, search
(→‎Installing Kubernetes)
 
(26 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
Kubernetes is a high-availability clustering package.  It is available for ARM, but mostly is used on x86-64.  These steps detail how to use it on Novena.
 
Kubernetes is a high-availability clustering package.  It is available for ARM, but mostly is used on x86-64.  These steps detail how to use it on Novena.
 +
 +
These steps assume you're starting from a base Novena image, such as the one found here on the Kosagi wiki.
  
 
==Installing Kubernetes==
 
==Installing Kubernetes==
  
 
Some steps I've taken to install Kubernetes on Debian.  Adapted from https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/ for Novena.
 
Some steps I've taken to install Kubernetes on Debian.  Adapted from https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/ for Novena.
 +
 +
*Before you begin*: Write http://repo.novena.io/novena/images/novena-mmc-disk-r1.img to an SD card, e.g. using "dd".
 +
 +
0. Remove software that comes installed by default on Novena, but is unnecessary for servers.
 +
apt-get install -y network-manager
 +
apt-get remove -y --auto-remove --purge hexchat pidgin bluez bluez-tools bluez-hcidump \
 +
        enigmail hexchat icedove iceweasel pavucontrol x11-xserver-utils xscreensaver \
 +
        libbluetooth3 keychain xserver-xorg-video-modesetting arandr android-tools-adb \
 +
        android-tools-fastboot android-tools-fsutils xfce4-goodies xfce4-power-manager \
 +
        xfce4-mixer xfce4-terminal mousepad orage dbus-x11 irssi synaptic qalc libqt5gui5 \
 +
        libqt5core5a libqt5widgets5 x11-apps x11-session-utils xbitmaps xfce4 xfce4-appfinder \
 +
        xfce4-notifyd xfce4-session xfce4-settings xfdesktop4 xfdesktop4-data xfonts-100dpi \
 +
        xfonts-75dpi xfonts-scalable xfwm4 xfwm4-themes xinit xorg xorg-docs-core \
 +
        libdrm-armada2-dbg libetnaviv-dev libetnaviv-dbg novena-disable-ssp novena-eeprom-gui \
 +
        novena-firstrun xorg-novena xserver-xorg-video-armada xserver-xorg-video-armada-dbg \
 +
        xserver-xorg-video-armada-etnaviv lightdm gstreamer1.0-x ghostscript firefox-esr \
 +
        sane-utils imagemagick-common eject emacs24 emacs libgstreamer-plugins-base1.0-0 \
 +
        libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good hunspell-en-us \
 +
        libgstreamer-plugins-base1.0-0 libgstreamer1.0-0 libavresample2 libavutil54 \
 +
        libavcodec56 libavformat56 libsane libswscale3 libgs9 libgs9-common libjasper1 \
 +
        libjbig2dec0 libfreetype6
 +
  
 
1. Update the key.
 
1. Update the key.
−
  wget http://repo.novena.io/debian/pool/main/k/kosagi-repo/kosagi-repo_1.2-r1_all.deb
+
wget http://repo.novena.io/debian/pool/main/k/kosagi-repo/kosagi-repo_1.2-r1_all.deb
−
  sudo dpkg -i kosagi-repo_1.2-r1_all.deb
+
sudo dpkg -i kosagi-repo_1.2-r1_all.deb
−
  apt-get update
+
apt-get update
−
  DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade
+
DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
−
  apt-get install apt-transport-https # Used for accessing the newer repos
+
apt-get install -y apt-transport-https # Used for accessing the newer repos
  
 
2. At this point, it's probably a good idea to move to a SATA disk.
 
2. At this point, it's probably a good idea to move to a SATA disk.
−
  dd if=/dev/mmcblk0 of=/dev/sda bs=1M count=1
+
dd if=/dev/mmcblk0 of=/dev/sda bs=1M count=1
−
  fdisk /dev/sda
+
fdisk /dev/sda
−
  # Re-create partitions 2 and 3.  Make partition 2 at least 6GB
+
# Re-create partitions 2 and 3.  Make partition 2 at least 6GB
−
  # for swap (type 82), and make partition 3 the rest of the disk.
+
# for swap (type 82), and make partition 3 the rest of the disk.
−
  # Type "x" for Expert mode, then "i", and change the ID to 0x4e6f7653.
+
# Type "x" for Expert mode, then "i", and change the ID to 0x4e6f7653.
−
  # Then "r" to return to the main menu, and "w" to write it to disk.
+
# Then "r" to return to the main menu, and "w" to write it to disk.
−
  mkfs.ext4 /dev/sda3 # or install btrfs-tools or xfsprogs and make a different type
+
apt-get install xfsprogs # btrfs isn't well supported yet by kubeadm
−
  mount /dev/sda3 /mnt
+
mkfs.xfs /dev/sda3 # or install btrfs-tools or xfsprogs and make a different type
−
  rsync -avxHAX --progress / /mnt/
+
mount /dev/sda3 /mnt
−
  novena-eeprom -w -f es8328,pcie,gbit,hdmi,eepromoops,sataroot
+
rsync -avxHAX --progress / /mnt/
−
  reboot
+
# Update /mnt/etc/fstab to point swap to /dev/sda2
 +
novena-eeprom -w -f es8328,pcie,gbit,hdmi,eepromoops,sataroot
 +
reboot
  
−
3. Install Docker (borrowed from https://github.com/hypriot/image-builder-odroid-c1/blob/master/builder/chroot-script.sh)
+
3. Install Docker (borrowed from https://github.com/hypriot/image-builder-odroid-c1/blob/master/builder/chroot-script.sh).  If you need a specific version, you can list them with:
−
  wget -q https://packagecloud.io/gpg.key -O - | apt-key add -
+
apt-cache madison docker-engine
−
  echo 'deb https://packagecloud.io/Hypriot/Schatzkiste/debian/ jessie main' > /etc/apt/sources.list.d/hypriot.list
+
 
−
  apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 2C52609D
+
wget -q https://packagecloud.io/gpg.key -O - | apt-key add -
−
  echo 'deb [arch=armhf] https://apt.dockerproject.org/repo debian-jessie main' > /etc/apt/sources.list.d/docker.list
+
echo 'deb https://packagecloud.io/Hypriot/Schatzkiste/debian/ jessie main' > /etc/apt/sources.list.d/hypriot.list
−
  apt-get update
+
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 2C52609D
−
  export DOCKER_ENGINE_VERSION="17.03.0~ce-0~debian-jessie"
+
echo 'deb [arch=armhf] https://apt.dockerproject.org/repo debian-jessie main' > /etc/apt/sources.list.d/docker.list
−
  export DOCKER_COMPOSE_VERSION="1.9.0-23"
+
apt-get update
−
  export DOCKER_MACHINE_VERSION="0.9.0-39"
+
DEBIAN_FRONTEND=noninteractive apt-get install \
−
  DEBIAN_FRONTEND=noninteractive apt-get install \
+
     docker-engine=1.12.6-0~debian-jessie \
−
    libltdl7 \
+
     docker-compose="1.9.0-23" \
−
     docker-engine="${DOCKER_ENGINE_VERSION}" \
+
     docker-machine="0.9.0-39"
−
     docker-compose="${DOCKER_COMPOSE_VERSION}" \
 
−
     docker-machine="${DOCKER_MACHINE_VERSION}"
 
  
 
4. Install kubeadm
 
4. Install kubeadm
−
  curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
+
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
−
  echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
+
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
−
  apt-get update && apt-get install -y kubeadm
+
apt-get update && apt-get install -y kubeadm
  
−
5. Run kubeadm on the master device to start the cluster.  As of this moment, we need to skip preflight checks due to changes in Docker version numbering.
+
5. Run kubeadm on the master device to start the cluster.
−
  kubeadm init --pod-network-cidr 10.244.0.0/16 --skip-preflight-checks
+
  kubeadm init --pod-network-cidr 10.244.0.0/16
  
 
It will output something like the following:
 
It will output something like the following:
−
  Your Kubernetes master has initialized successfully!
+
Your Kubernetes master has initialized successfully!
−
 
+
−
  You should now deploy a pod network to the cluster.
+
You should now deploy a pod network to the cluster.
−
  Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
+
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
−
      http://kubernetes.io/docs/admin/addons/
+
    http://kubernetes.io/docs/admin/addons/
−
 
+
−
  You can now join any number of machines by running the following on each node:
+
You can now join any number of machines by running the following on each node:
−
 
+
−
  kubeadm join --token=964a50.d8053ed3de195a11 10.0.245.169
+
kubeadm join --token=964a50.d8053ed3de195a11 10.0.245.169
  
−
6. Join the cluster from other machines.  We still need to skip preflight checks.
+
6. Ensure cAdvisor doesn't start up.  It's nice to have, but it leaks lots of information.
−
  kubeadm join --token=964a50.d8053ed3de195a11 --skip-preflight-checks 10.0.245.169
+
printf '[Service]\nEnvironment="KUBELET_EXTRA_ARGS=--cadvisor-port=0"\n' > /etc/systemd/system/kubelet.service.d/05-disable-cadvisor.conf
 +
systemctl daemon-reload
 +
systemctl restart kubelet
  
−
7. Install Flannel on the Master, which will manage the network for us.
+
7. Install Flannel on the Master, which will manage the network for us.  We need to specify version 0.7.0 because newer versions aren't compatible with Docker's new iptables rules.
−
  curl -sSL https://rawgit.com/coreos/flannel/master/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
+
kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
 +
curl -sSL https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -
  
 
8. Wait for all pods to be Running:
 
8. Wait for all pods to be Running:
−
  kubectl get po --all-namespaces
+
kubectl get po --all-namespaces
 +
 
 +
 
 +
===Joining machines to the cluster===
 +
 
 +
1. Join the cluster from other machines.
 +
kubeadm join --token=964a50.d8053ed3de195a11  10.0.245.169
 +
 
 +
2. Copy over /etc/fstab
 +
  10.0.246.13:/data    /data                nfs4      defaults              0  0
 +
 
 +
3. Ensure public SSH keys are copied to all devices
 +
 
 +
4. Set PasswordAuthentication to "no" in /etc/ssh/sshd_config
  
 
==Using Kubernetes==
 
==Using Kubernetes==
 +
 +
1. Deploy the dashboard.
 +
curl -sSL https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml | sed "s/amd64/arm/g" | kubectl create -f -
 +
 +
2. Deploy Heapster, to get more information on the dashboard:
 +
kubectl create -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
 +
curl -sSL https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/standalone/heapster-controller.yaml | sed "s/amd64/arm/g" | kubectl create -f -
 +
 +
curl -sSL https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml | sed "s/amd64/arm/g" | kubectl create -f -
 +
curl -sSL https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml | sed "s/amd64/arm/g" | kubectl create -f -
 +
curl -sSL https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml | sed "s/amd64/arm/g" | kubectl create -f -
 +
 +
==Resetting==
 +
 +
To reset things, use "kubeadm reset", and clean up after the program.
 +
 +
1. Reset
 +
kubeadm reset
 +
 +
2. Clean up
 +
ip link delete flannel.1; rm -rf /var/lib/cni /run/flannel /etc/cni /var/lib/etcd; ifconfig cni0 down; brctl delbr cni0

Latest revision as of 01:05, 21 June 2017

Kubernetes is a high-availability clustering package. It is available for ARM, but mostly is used on x86-64. These steps detail how to use it on Novena.

These steps assume you're starting from a base Novena image, such as the one found here on the Kosagi wiki.

Installing Kubernetes

Some steps I've taken to install Kubernetes on Debian. Adapted from https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/ for Novena.

0. Remove software that comes installed by default on Novena, but is unnecessary for servers. 

apt-get install -y network-manager
apt-get remove -y --auto-remove --purge hexchat pidgin bluez bluez-tools bluez-hcidump \
       enigmail hexchat icedove iceweasel pavucontrol x11-xserver-utils xscreensaver \
       libbluetooth3 keychain xserver-xorg-video-modesetting arandr android-tools-adb \
       android-tools-fastboot android-tools-fsutils xfce4-goodies xfce4-power-manager \
       xfce4-mixer xfce4-terminal mousepad orage dbus-x11 irssi synaptic qalc libqt5gui5 \
       libqt5core5a libqt5widgets5 x11-apps x11-session-utils xbitmaps xfce4 xfce4-appfinder \
       xfce4-notifyd xfce4-session xfce4-settings xfdesktop4 xfdesktop4-data xfonts-100dpi \
       xfonts-75dpi xfonts-scalable xfwm4 xfwm4-themes xinit xorg xorg-docs-core \
       libdrm-armada2-dbg libetnaviv-dev libetnaviv-dbg novena-disable-ssp novena-eeprom-gui \
       novena-firstrun xorg-novena xserver-xorg-video-armada xserver-xorg-video-armada-dbg \
       xserver-xorg-video-armada-etnaviv lightdm gstreamer1.0-x ghostscript firefox-esr \
       sane-utils imagemagick-common eject emacs24 emacs libgstreamer-plugins-base1.0-0 \
       libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good hunspell-en-us \
       libgstreamer-plugins-base1.0-0 libgstreamer1.0-0 libavresample2 libavutil54 \
       libavcodec56 libavformat56 libsane libswscale3 libgs9 libgs9-common libjasper1 \
       libjbig2dec0 libfreetype6


1. Update the key. 

wget http://repo.novena.io/debian/pool/main/k/kosagi-repo/kosagi-repo_1.2-r1_all.deb
sudo dpkg -i kosagi-repo_1.2-r1_all.deb
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get -y dist-upgrade
apt-get install -y apt-transport-https # Used for accessing the newer repos

2. At this point, it's probably a good idea to move to a SATA disk. 

dd if=/dev/mmcblk0 of=/dev/sda bs=1M count=1
fdisk /dev/sda
# Re-create partitions 2 and 3.  Make partition 2 at least 6GB
# for swap (type 82), and make partition 3 the rest of the disk.
# Type "x" for Expert mode, then "i", and change the ID to 0x4e6f7653.
# Then "r" to return to the main menu, and "w" to write it to disk.
apt-get install xfsprogs # btrfs isn't well supported yet by kubeadm
mkfs.xfs /dev/sda3 # or install btrfs-tools or xfsprogs and make a different type
mount /dev/sda3 /mnt
rsync -avxHAX --progress / /mnt/
# Update /mnt/etc/fstab to point swap to /dev/sda2
novena-eeprom -w -f es8328,pcie,gbit,hdmi,eepromoops,sataroot
reboot

3. Install Docker (borrowed from https://github.com/hypriot/image-builder-odroid-c1/blob/master/builder/chroot-script.sh). If you need a specific version, you can list them with: 

apt-cache madison docker-engine

wget -q https://packagecloud.io/gpg.key -O - | apt-key add -
echo 'deb https://packagecloud.io/Hypriot/Schatzkiste/debian/ jessie main' > /etc/apt/sources.list.d/hypriot.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 2C52609D
echo 'deb [arch=armhf] https://apt.dockerproject.org/repo debian-jessie main' > /etc/apt/sources.list.d/docker.list
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install \
   docker-engine=1.12.6-0~debian-jessie \
   docker-compose="1.9.0-23" \
   docker-machine="0.9.0-39"

4. Install kubeadm 

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install -y kubeadm

5. Run kubeadm on the master device to start the cluster. 

kubeadm init --pod-network-cidr 10.244.0.0/16

It will output something like the following: 

Your Kubernetes master has initialized successfully!

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
    http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node:

kubeadm join --token=964a50.d8053ed3de195a11 10.0.245.169

6. Ensure cAdvisor doesn't start up. It's nice to have, but it leaks lots of information. 

printf '[Service]\nEnvironment="KUBELET_EXTRA_ARGS=--cadvisor-port=0"\n' > /etc/systemd/system/kubelet.service.d/05-disable-cadvisor.conf
systemctl daemon-reload
systemctl restart kubelet

7. Install Flannel on the Master, which will manage the network for us. We need to specify version 0.7.0 because newer versions aren't compatible with Docker's new iptables rules. 

kubectl create -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel-rbac.yml
curl -sSL https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml | sed "s/amd64/arm/g" | kubectl create -f -

8. Wait for all pods to be Running: 

kubectl get po --all-namespaces


Joining machines to the cluster

1. Join the cluster from other machines. 

kubeadm join --token=964a50.d8053ed3de195a11  10.0.245.169

2. Copy over /etc/fstab 

 10.0.246.13:/data    /data                nfs4       defaults              0  0

3. Ensure public SSH keys are copied to all devices

4. Set PasswordAuthentication to "no" in /etc/ssh/sshd_config

Using Kubernetes

1. Deploy the dashboard. 

curl -sSL https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml | sed "s/amd64/arm/g" | kubectl create -f -

2. Deploy Heapster, to get more information on the dashboard: 

kubectl create -f https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/rbac/heapster-rbac.yaml
curl -sSL https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/standalone/heapster-controller.yaml | sed "s/amd64/arm/g" | kubectl create -f -

curl -sSL https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/influxdb.yaml | sed "s/amd64/arm/g" | kubectl create -f -
curl -sSL https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/grafana.yaml | sed "s/amd64/arm/g" | kubectl create -f -
curl -sSL https://raw.githubusercontent.com/kubernetes/heapster/master/deploy/kube-config/influxdb/heapster.yaml | sed "s/amd64/arm/g" | kubectl create -f -

Resetting

To reset things, use "kubeadm reset", and clean up after the program.

1. Reset 

kubeadm reset

2. Clean up 

ip link delete flannel.1; rm -rf /var/lib/cni /run/flannel /etc/cni /var/lib/etcd; ifconfig cni0 down; brctl delbr cni0
Retrieved from "https://kosagi.com/w/index.php?title=Kubernetes_on_Novena&oldid=1928"