At the beginning, there was this little idea to have a Kubernetes cluster to take with me to conferences and learn more about the inner workings of Kubernetes. And then, the journey began. This post tells the story, gives some insights and link but doesn't dive into deep technical details.
The Hardware
My goal was to find a cheap ARM board with at least 2 GB RAM. There are not many around: Either they are not cheap (less than $40) or they don't have more than 1 GB RAM. After some time, I found the Orange Pi WinPlus board which I ordered at Ali Express. A few weeks later they arrived and the challenge to get them working began.
First I had to find a working operating system which has proven not to be that easy. The software offered by Orange Pi is not at a good quality, either messed with unnecessary software, many errors during boot, very outdated, ... To the rescue came Armbian which offers support for this board.
After having found a working Linux distribution I discovered the next pitfall: All boards have the exact same MAC address. There goes my idea of distributing the IPs easily via DHCP and not touching every SD card. So I had to customize each SD card to set an individual MAC address during boot. Some boards just stopped responding to network connections after some time, booting another with the same SD card was stable. So at least two boards are having hardware stability issues. I was expecting that, so I initially ordered more than I actually needed.
The HDMI touchscreen didn't work with the boards as the Kernel is still at 3.10 and the resolutions are compiled in, of course the needed resolution of 1024×600 was missing. This is one of the reasons I'm using an additional RaspberryPI for the display and as a management board (DHCP, NAT, Resolver, Display, Jumphost, Ansible ...).
TL;DR: Here is a list of hardware I ordered for this project:
- 8 x Orange Pi WinPlus
- 10 X Kingston microSDHC UHS-I mit Adapter (16GB, Class 10)
- 3 x Intenso Slim Line 32GB USB 3.0 (32GB, USB 3.0)
- 7 x Value USB 2.0 Kabel (A - Micro, 0.15m)
- 8 x Wirewin Netzwerkkabel (0.15m, F/FTP, Kat. 6, Flachband)
- 1 x D-Link DGS-1100-08/E
- 1 x ICY BOX IB-AC6110 (10Ports)
- 1 x Raspberry Pi 3 Model B (ARMv8)
- 1 x Waveshare Display 7inch HDMI LCD (C), 1024×600, IPS
The Network
I wanted to have a separate network for the Kubernetes cluster and make use of VLANs. The D-Link switch supports VLANs and can be configured via a web interface. There are two VLANs configured: A LAN and a WAN. All Orange Pis are connected to the LAN VLAN, the RaspberryPI to a VLAN tagged interface which carries both VLANs and one port is for the WAN link.
The RaspberryPI runs Pi-Hole which offers a resolver and easy DHCP server configuration. All boards are registered with their MAC address in the DHCP server, so they always get the same IP address and hostname assigned. This enables easy hostname resolving in the local network.
The Software
The initial idea was to get OpenShift running on this cluster. There is no official ARM build of OpenShift, so I did some experimental cross-building by myself. While the crossbuild worked and the produced binary could be executed on the boards, properly starting the API server was not possible and lead into the following bug-report which hasn't been fixed until the time of this writing: API server does not start with "tls: DialWithDialer timed out" on arm64 #19290. So I decided to go for plain Kubernetes, which has proven to work. Nevertheless, I wasn't able to deploy Kubernetes 1.10 which has some similar issues to OpenShift (TLS timeouts) while Kubernetes 1.9 works like a charm.
The Automation
For installing Kubernetes on ARM boards I discovered rak8s which is a set of nice Ansible roles to automatically bootstrap a Kubernetes cluster on RaspberryPI boards, based on kubeadm. On the shoulders of this work I extended the roles to support Orange PIs and automate many aspects of my little project: github.com/tobru/rak8s#appuio2go. My changes are not documented and the commit messages also don't properly state what I was doing - sorry for that! Hopefully it gives a least a bit of impression.
The Applications
A Kubernetes cluster without any deployed applications is not very interesting. So I deployed several applications:
- Traefik as ingress controller
- ngrok as reverse tunnel
- Helm to deploy more applications
- Kubernetes Dashboard
- kube-ops-view
- Prometheus, Grafana, serveral exporters
The Persistent Storage
For persistent storage I wanted to give rook.io a try, which is currently supporting Ceph as storage backend. The three USB sticks are meant as storage backend. Installing and configuring rook worked and the Ceph cluster started and synced. But then I discovered the the Armbian 3.10 Kernel doesn't ship the RBD Kernel module, so no storage could be mounted.
The next try was OpenEBS, but I wasn't able to get it running on this cluster.
In The Wild
The cluster got a name: APPUiOli. It's first public exposure was at the DevOpsDays conference in Zürich-Winterthur. We deployed podstalk to the cluster and offered a public URL via ngrok so everyone could connect to the cluster. On a big screen we displayed the Grafana dashboard.
Resources
The whole project wouldn't have been possible without the help of many great sources, here are some of them:
- https://github.com/rak8s/rak8s
- https://gist.github.com/alexellis/fdbc90de7691a1b9edb545c17da2d975
- https://github.com/ReSearchITEng/kubeadm-playbook/
- https://github.com/luxas/kubeadm-workshop
- https://github.com/alexellis/docker-arm/tree/master/images/armhf
- https://github.com/carlosedp/prometheus-operator-ARM
