There are several options to run Kubernetes locally to get some experience with Kubernetes as developer. For example Minikube, MicroK8s and MiniShift. These options however are not representative for a real environment. They for example usually do not have master and slave nodes. Running locally requires quite different configuration compared to running multiple nodes in VMs. Think for example about how to deal with storage and a container registry which you want to share over the different nodes. Installing a full blown environment requires a lot of work and resources. Using a cloud service usually is not free and you usually have less to no control over the environment Kubernetes is running in.
In this blog I'll describe a 'middle way'. Get an easy to manage small multi node Kubernetes environment running in different VMs. You can use this environment for example to learn what the challenges of clusters are and how to deal with them efficiently.
It uses the work done here with some minor additions to get a dashboard ready.
Getting the host ready
As host OS I used Cent OS 7 (on bare metal). CentOS 8 introduces some major changes such as Podman instead of Docker so I did not want to take any risks and decided to stick with this commonly used open source OS compiled from Red Hat sources. I do recommend sticking to a single partition for a local development environment to make it more easy for yourself. Also I used a minimal desktop environment with administrative tools.
Also create a user which can do sudo to execute various commands in the following steps.
Install QEMU/KVM + libvirt
We are going to use QEMU/KVM and access it through libvirt. Why? Because I want to approach bare metal performance as much as I can and QEMU/KVM does a good job at that. See for example this performance comparison of bare metal vs KVM vs Virtualbox. KVM greatly outperforms Virtualbox and approaches bare metal speeds in quite some tests. I do like the Virtualbox GUI though but I can live with the Virtual Machine Manager.
The following will do the trick on CentOS 7
sudo yum install qemu-kvm qemu-img virt-manager libvirt libvirt-python libvirt-client virt-install virt-viewer bridge-utils gcc git make
Install Vagrant and required plugins
Vagrant is used to create the virtual machines for the master and nodes. Vagrant can easily be installed from here. It even has a CentOS specific RPM which is nice.
With Vagrant I'm going to use two plugins. vagrant-libvirt and vagrant-sshfs. The first plugin allows vagrant to manage QEMU/KVM VMs through libvirt. The second plugin will be used for shared folders. Why sshfs? Mainly because libvirt alternatives for shared folders such as NFS and 9p were more difficult to set-up and I wanted to be able to provide the same shared storage to all VMs.
vagrant plugin install vagrant-libvirt
vagrant plugin install vagrant-sshfs
Install Kubernetes
Install kubectl
First install kubectl on the host. This is described in detail here.
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubectl
Create the VMs
Execute the following under a normal user (you also installed the Vagrant plugins under this user).
git clone https://github.com/MaartenSmeets/k8s-vagrant-multi-node.git
cd k8s-vagrant-multi-node
mkdir data/shared
make up -j 3 BOX_OS=centos VAGRANT_DEFAULT_PROVIDER=libvirt NODE_CPUS=1 NODE_COUNT=2 MASTER_CPUS=2
This process will take a while. You can follow the progress by looking in the Virtual Machine Manager and in the console.
The command will also ask a couple of times for the user password. Be ready to input this since if you wait too long, it will timeout and the build will fail. If it fails, clean up using
make clean VAGRANT_DEFAULT_PROVIDER=libvirt
If at the end you only see a single node, you can do the following to create a second node
make start-node-2 NODE_CPUS=1 VAGRANT_DEFAULT_PROVIDER=libvirt BOX_OS=centos
There are many more useful commands available in the Makefile so do check them out! Read the documentation here.
Check you can access the environment by:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 15h v1.18.2
node1 Ready <none> 15h v1.18.2
node2 Ready <none> 15h v1.18.2
Configure Kubernetes
We now have a master and two nodes running.
Install a dashboard
The environment does not have an out of the box dashboard like OKD (open source OpenShift). Even though the make scripts allow you to add the dashboard during the creation of the nodes, I prefer to this afterwards so I know what I'm doing.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0/aio/deploy/recommended.yaml
Now you have a dashboard but no user who can browse resources. In order to give an admin user the required privileges, I did the following (based on this).
cat > dashboard-adminuser.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system
EOF
kubectl apply -f dashboard-adminuser.yaml
cat > admin-role-binding.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kube-system
EOF
kubectl apply -f admin-role-binding.yaml
Now you can obtain a token by:
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
Start a local proxy:
kubectl proxy
Now you can use the token to login to:
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
and open the dashboard
Additional notes
This environment is not done yet.
- Every node uses its own local registry. The Makefile used does provide ways to load the same image to the different registries but what I actually want is all the nodes to use the same registry.
- There is no shared PersistentVolume available within the Kubernetes environment which can be used. Shared storage. Preparations for that have been made though since /shared in every VM is mounted to the same local folder.
- We have not installed anything in the environment yet but a small dashboard. I want to have Jenkins running inside and be able to deploy applications.
No comments:
Post a Comment