Yet Another Kubernetes Intro - Part 5 - Deployments
It is time to have a look at the last pod management related resource type in Kubernetes, the Deployment. In part 3 I talked about using ReplicaSets to run multiple pods in our cluster, which is all nice and dandy until you want to update the service you are running. With pure RSs, this can get a bit complicated…
When rolling out a new version, we generally want to do this without downtime. This means replacing a few pods at the time, so that there are always pods up and running, as well as making sure that the new version is behaving as it should, and not taking down the whole system. Doing this manually is a bit tedious, and probably quite error prone. That is why K8s has the Deployment resource to help out.
K8s deployments support 2 types of update strategies, Recreate and RollingUpdate. Recreate just tears down the existing pods and puts up new ones, leaving us with downtime while the new pods come online. Using a RollingUpdate strategy on the other hand, allows us to roll out an update of a pod in the cluster, replacing a few pods at the time. Because of this, a rolling update is preferred in most situations.
But before we go into how we set one of these up, let’s have a quick look at how they work, at a high level.
Deployments
When you add a deployment resource to your cluster, it will create a ReplicaSet to manage the pods that the deployment defines. And the RS in turn, schedules the pods. So it is pretty much just a nice layer on top of ReplicaSets. However, the “magic” comes in when you deploy a new version of your deployment, with the update strategy set to perform a rolling update.
Note: Not all changes to your deployment will cause an update. For example, changing the replica count does not require an update. It will just cause the RS to be reconfigured. But every time you change your pod template, it will cause an update.
Rolling updates
When making a change that causes an update to happen, the first thing that happens is that the deployment (or rather the controller managing the deployment, but that is semantics) will set up a new RS to handle the new, updated pods. It then scales up that RS to create one or more pods depending on the configuration and existing replica count.
Once the first group of new pods have come online, the previous RS is scaled down to compensate for the newly created pods. Then the new RS is scaled up, and the old one scaled down. This dance then goes on until the old RS is scaled to 0, and the new RS has been scaled up to the full number of requested replicas. Along the way, the deployment can be requested to monitor the new pods to make sure that the new pods stay online for at least X amount of time before being considered available. This makes sure that any configuration problems, or container issues that would cause a pod to fail, doesn’t replicate throughout your cluster. Otherwise it would be very easy to take down your application with a small config mistake for example.
Configuration
The rolling update can be configured in a few ways. First of all, as just mentioned, we can have the deployment monitor the new pods for a while before carrying on with the roll out. This is a nice little insurance to make sure that we aren’t rolling out a botched pod that will cause the application to fail. Or in the words of the infinitely wise @DevOps_Borat
On top of that, we can define how many pods that can be allowed to be unavailable at any point. Either in fixed numbers, or a percentage of the total desired replica count. So for example, if we tell it that we want a maximum of 3 pods unavailable, it means that even if we have expressed a desire to have 12 pods, during the update, the cluster is allowed to go down as low as 9.
We can also define a maximum “surge”. This tells the deployment how many extra pods that can be allowed. For example, setting it to 3 with a desired state of 12, means that during the update, the cluster can scale up to a maximum of 15 pods.
Using these two configuration values, we can configure how fast the update is allowed to roll through the cluster. The reason for these for having these settings is that since the deployment uses 2 ReplicaSets, scaling up one and down the other, there is no way to make sure that there is no overlap of pods between the sets. So this allows there to be a certain fluctuation in the pod count of the combined RS.
Defining a deployment
As with most K8s resources, creating the deployment definition in a YAML file is recommended. This gives us a way to version it using source control etc.
A spec looks something like this
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
spec:
replicas: 3
selector:
matchLabels:
app: hello-world
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
minReadySeconds: 10
template:
metadata:
labels:
app: hello-world
spec:
containers:
- name: hello-world
image: zerokoll.azurecr.io/zerokoll/helloworld:v1
As you can see, the deployment spec is pretty similar to a RS definition, which kind of makes sense as it is pretty much just a controlling layer on top of an RS.
The above example defines a deployment called hello-world that should have 3 replicas. It should also use a rolling update strategy with maximum deviation of 1 pod both up and down from the desired 3, allowing the pods to be ready for 10 seconds before continuing the rollout. And as it spins up new pods, it uses the defined template.
That’s all there is to it! And after adding it to the cluster like this
kubectl apply -f .\hello-world-deployment.yml
we can see that it creates 1 RS and 3 pods
kubectl get all
NAME READY STATUS RESTARTS AGE
pod/hello-world-5f4c77d94-hphp5 1/1 Running 0 68s
pod/hello-world-5f4c77d94-jxbht 1/1 Running 0 68s
pod/hello-world-5f4c77d94-mvltd 1/1 Running 0 68s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6d8h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello-world 3/3 3 3 68s
NAME DESIRED CURRENT READY AGE
replicaset.apps/hello-world-5f4c77d94 3 3 3 68s
Performing a rolling update
One the deployment is up and running, we can do a rolling update by simply changing the pod template to use another version like this
…
spec:
containers:
- name: hello-world
image: zerokoll.azurecr.io/zerokoll/helloworld:v2
and then re-deploy the deployment using kubectl apply. While the update is rolling out, we can monitor it in a couple of different ways. First of all, we can obviously have a look at the resources in the cluster
kubectl get all
NAME READY STATUS RESTARTS AGE
pod/hello-world-5f4c77d94-hphp5 0/1 Terminating 0 5m28s
pod/hello-world-5f4c77d94-jxbht 1/1 Running 0 5m28s
pod/hello-world-5f4c77d94-mvltd 1/1 Running 0 5m28s
pod/hello-world-6669969979-pxf64 1/1 Running 0 2s
pod/hello-world-6669969979-rdmqs 1/1 Running 0 2s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6d8h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello-world 4/3 2 2 5m28s
NAME DESIRED CURRENT READY AGE
replicaset.apps/hello-world-5f4c77d94 2 2 2 5m28s
replicaset.apps/hello-world-6669969979 2 2 2 2s
As you can see from the output, doing the update has caused a second RS to be created. You can also see that the new RS (hello-world-6669969979) has already scaled up to 2 pods, and the original one (hello-world-5f4c77d94) has started scaling down by one.
Once the update has rolled through, the output looks like this
kubectl get all
NAME READY STATUS RESTARTS AGE
pod/hello-world-6669969979-4zf5n 1/1 Running 0 2m3s
pod/hello-world-6669969979-pxf64 1/1 Running 0 2m14s
pod/hello-world-6669969979-rdmqs 1/1 Running 0 2m14s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6d8h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello-world 3/3 3 3 7m40s
NAME DESIRED CURRENT READY AGE
replicaset.apps/hello-world-5f4c77d94 0 0 0 7m40s
replicaset.apps/hello-world-6669969979 3 3 3 2m14s
The new RS is running at full speed and the old one has scaled down to 0. However, the old RS is still in place, and isn’t removed. Instead, the deployment can keep track of the ReplicaSets, and pods, from a specific deployment, by adding a label to them called pod-template-hash. The value for this label is a hash of the pod template. This allows the deployment to make sure that the created label selectors for the ReplicaSets aren’t overlapping.
Besides just looking at the resources in the cluster, you can also monitor the rolling update by running
kubectl rollout status deployment.v1.apps/hello-world
This returns locks up your terminal during the update, keeping track of the progress as it is rolling through the cluster. The output looks something like this
Waiting for deployment "hello-world" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "hello-world" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "hello-world" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "hello-world" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "hello-world" rollout to finish: 2 of 3 updated replicas are available...
Waiting for deployment "hello-world" rollout to finish: 2 of 3 updated replicas are available...
deployment "hello-world" successfully rolled out
Finally, you can always use kubectl get and kubectl describe to get more information about your deployment. For example, running kubectl describe for your deployment will get you the following information.
kubectl describe deployment/hello-world
Name: hello-world
Namespace: default
CreationTimestamp: Tue, 28 Jan 2020 23:36:20 +0100
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 2
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"hello-world","namespace":"default"},"spec":{"minReadySeco..."
Selector: app=hello-world
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 10
RollingUpdateStrategy: 1 max unavailable, 1 max surge
Pod Template:
Labels: app=hello-world
Containers:
hello-world:
Image: zerokoll.azurecr.io/zerokoll/helloworld:v2
Port: <none>
Host Port: <none>
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: hello-world-6669969979 (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 3d12h deployment-controller Scaled up replica set hello-world-5f4c77d94 to 3
Normal ScalingReplicaSet 3d12h deployment-controller Scaled up replica set hello-world-6669969979 to 1
Normal ScalingReplicaSet 3d12h deployment-controller Scaled down replica set hello-world-5f4c77d94 to 2
Normal ScalingReplicaSet 3d12h deployment-controller Scaled up replica set hello-world-6669969979 to 2
Normal ScalingReplicaSet 3d12h deployment-controller Scaled down replica set hello-world-5f4c77d94 to 0
Normal ScalingReplicaSet 3d12h deployment-controller Scaled up replica set hello-world-6669969979 to 3
In this massive output you can see the pod template, the current status of the deployment, the replica count, the name of the old and new RS etc. And you can also see the historical events for the deployment to keep track of what has happened over time.
Revision history
As you make changes to the deployments you are using, and rolling out new version, a history of the old version is recorded. This allows us to roll back a failed update in a very simple way. Just remember that the revision history is limited. But on the other hand, in most cases you only rollback the latest version anyway.
To look at the history of my deployment, you can run the following command
kubectl rollout history deployment.v1.apps/hello-world
deployment.apps/hello-world
REVISION CHANGE-CAUSE
1 <none>
2 <none>
As you can see, in this case there are 2 revisions. The first one was the initial deployment, and the second one is when we upgraded to version 2. However, the CHANGE-CAUSE column is pretty useless right now. This is because I forgot to add the parameter --record then applying the deployment spec. Including the --record parameter like this
kubectl apply -f=hello-world-deployment.yaml --record=true
would have caused the change to be recorded in a resource annotation called kubernetes.io/change-cause. And this in turn would have changed the empty output from the kubectl rollout history command above, to something like this
deployment.apps/hello-world
REVISION CHANGE-CAUSE
1 kubectl.exe apply --filename=.\hello-world-deployment.yml --record=true
2 kubectl.exe apply --filename=.\hello-world-deployment.yml --record=true
Not a whole lot better as such. But at least it is some information about what caused the changed. And if you had made manual updates to your deployment using the CLI instead of a YAML file, the commands you ran would show up in the change cause. For example, running the following command to roll out the v1 image again
kubectl set image deployment/hello-world hello-world=zerokoll.azurecr.io/zerokoll/helloworld:v1 --record
causes the rollout history to look like this
kubectl rollout history deployment.v1.apps/hello-world
deployment.apps/hello-world
REVISION CHANGE-CAUSE
2 kubectl.exe apply --filename=.\hello-world-deployment.yml --record=true
3 kubectl.exe set image deployment/hello-world hello-world=zerokoll.azurecr.io/zerokoll/helloworld:v1 --record=true
This is obviously a bit more useful, but I still recommend using YAML files!
Note: As you can see, the full revision history is not kept. However, the number of revisions to keep can be defined in the deployment by setting the .spec.revisionHistoryLimit property for the deployment.
If you want more information about a specific revision, you can run
kubectl rollout history deployment.v1.apps/hello-world --revision=2
deployment.apps/hello-world with revision #2
Pod Template:
Labels: app=hello-world
pod-template-hash=6669969979
Annotations: kubernetes.io/change-cause: kubectl.exe apply --filename=.\hello-world-deployment.yml --record=true
Containers:
hello-world:
Image: zerokoll.azurecr.io/zerokoll/helloworld:v2
Port: <none>
Host Port: <none>
Environment: <none>
Mounts: <none>
Volumes: <none>
This shows the pod template that was being used for that specific revision.
And if you ever want to roll back to a previous revision, you can easily do this by running
kubectl rollout undo deployment.v1.apps/hello-world
This will roll back the last revision. And if you add --to-revision=<REVISION ID>, you can roll back to a specific revision.
Managing roll outs
During the roll out a new version, you can pause the rollout by running
kubectl rollout pause deployment.v1.apps/hello-world
This will pause the rollout and allow you to make manual changes to the deployment using kubectl. And when you are done changing the deployment that you are currently in the process of rolling out, you can run
kubectl rollout resume deployment.v1.apps/hello-world
to resume the rollout with the new configuration.
However, as mentioned several times before, it is better to make your changes in a YAML file so that it can be recorded in source control. So if you find that you want to make a change to a deployment in the middle of a rollout, you can just update the deployment spec and apply it again. This will cause a “rollover”. Basically that means that the deployment will stop the current rollout, and spin up yet another RS. It will then create new pods in the new RS, replacing both the original, and semi rolled out pods, with new pods in this new RS.
I think that is pretty much all had to say about deployments this time. To sum it up, they are the way you should define your replicated pods in your cluster. They allow you to create replicated pods using an RS, but adding a nice way to roll out updates to your cluster.
The next part is available here, and talks about configuration.
