At Kohort, we publish messages to queues which in turn are consumed by Kubernetes (k8s) pods. As these messages are consumed, they increase the average CPU usage of these pods which in turn triggers Horizontal Pod Autoscaling (HPA) which then creates more pods. Simply put, if the average CPU usage increases, then the number of pods increases and vice versa.

‍

During a recent change in architecture we offloaded a substantial amount of computational work required in some of our longer-running processes from the pods to our data warehouse. This decrease in computational effort on the pods introduced an unforeseen consequence. K8s would detect the decrease in average CPU utilisation and consequently it would prematurely remove these seemingly idle pods (resulting in a lot of re-processing).

‍

Solution

After investigating multiple strategies we came to two conclusions. Firstly, we have to look into metrics other than CPU usage for scaling our resources and secondly, some of our processes are better suited for k8s jobs (a k8s job has the advantage that it does not have to be externally monitored to determine when the job is done, it merely exists when it has completed its workload).

We discovered that KEDA solves both these requirements. Not only does KEDA allow us to scale resources based on several other scalers, it also allows us to start either k8s jobs or pods.

Implementation

Setting up KEDA proved to be a very simple process. With a basic helm chart installation all the required KEDA components and plugins were ready for action. After that we configured a ScaledJob (for our solution we are using the KEDA Amazon SQS scaler) and updated our code so that it will work within the context of a k8s job instead of a k8s deployment. Finally, we’ve set the queueLength to 1; this allows us to deploy a single k8s job per message in the queue. Now we have a one-to-one mapping between messages and jobs.

Known Issues

KEDA has been serving us well for the last couple of months with very few problems. On occasion KEDA seems to “lock up” after a ScaledJob is updated; a simple re-running of the deployment CI/CD job resolves the issue. Unfortunately, we have not yet been able to identify the root cause of this issue.

--

In summary, we are extremely happy with KEDA. It made the transition to using k8s jobs much easier than we expected.

‍