Handling multiple environments with Helm
Have you tried Helm?
It provisions all your kubernetes applications beautifully, but I’ve always been mildly frustrated that it doesn’t support multiple product environments by default using its single values file.
This is a ubiquitous problem — nobody likes to have the same code copied with different values for multiple environments. It is clunky and makes template changes hard in the future, with the need to update these separately for each environment’s source code. So I decided to roll up my sleeves and do something about it.
In this post, I am going to cover how I’ve achieved support for multiple kubernetes environments using Helm.
Each kubernetes environment requires handling the below aspects. Generally, all non secret values should be put in kubernetes configmap(s) and all secret values should be in kubernetes secrets.
- Environment specific constants like urls etc. - mostly non secret values which can be safely checked in to git (Kubernetes Configmap)
- Environment specific secret values like db username and passwords etc. - can’t be checked into git (Kubernetes Secrets)
- Updating kubernetes deployments (pods) when configmap / secrets gets updated
Here’s the project structure at a glance.
The docker folder was created for generating a simple docker image, which will demonstrate my approach.
The environments folder is where we’ll need to put all our environment specific files.
Every environment has 3 files inside the environments folder. These are the Helm values file, configmap file and secrets file. The secrets files won’t be checked into git and only available to our CI/CD pipelines.
FYI: This is a sample project that’s meant to give you an understanding of how this works. Feel free to make changes as needed for your application. I can’t explain the Helm structure and its way of working because it’s out of scope for this post. If you want to check out the source code, you can view it here.
Both the configmap.<env>.yaml and secrets.<env>.yaml files are simple key value defined files. Except that the value of each key in the secrets file has to be base64 encoded. All key value pairs should be defined as KEY: VALUE with a newline between each key pair.
Helm requires values files to be supplied for the helm install/upgrade command. But we’ve got two values files at the root and environments folder levels. And by default, Helm will read the root level values file, even if you don’t specify any values file explicitly.
Ports and probes that are common to all the environments and they should be in the root level values file. This way, we won’t need to make changes in multiple places. And all environment specific values should reside in the values.<env>.yaml file. I won’t be focussing on how to define values for Helm here, so if you need any help, please go through this official documentation.
So you’ve defined all the yaml files. Now what?
Once you’re done updating both the configmap and secret yaml files, they won’t be directly applied to the cluster. Instead you need to carry out these steps for creating and modifying configmap and secrets.
Configmap: Create the configmap.yaml file under the templates directory and paste the below content.
Secrets: Create the secrets.yaml file under the templates directory and paste the below content.
Once you’ve created both these files under templates, both the kubernetes configmap and secrets files will be created and updated on the cluster from the next CI/CD pipeline run.
Here’s how you can update all your deployments.
Add both the annotations mentioned below to all your deployments and whenever the configmap or secrets files are updated, your pods will be restarted. Once any file changes, kubernetes will try to redeploy all deployments which are configured with below annotations at once. Please make sure that you have at least two replicas for each deployment, for high availability.
How to setup and run Helm commands.
Usually a Helm upgrade or install requires the release_name, chart_folder and other necessary flags. You’ll need to run your Helm commands with explicit flag -f or --values, with the value as the path to your environment specific values file. And make sure you add your application specific other Helm flags if required. Here’s a master command that will handle everything from Helm release installs to upgrades, if the release already exists.
helm upgrade --install --create-namespace -n <namespace> -f environments/values.<environment>.yaml <your_release_name> <your_chart_path>Note: Although you’ve defined the namespace value inside the yaml file, you will need to provide it with your command explicitly as Helm needs this namespace to save and retrieve Helm releases.
And we’re done!
With this article, I hope I’ve clarified how you can make Helm work with your multiple environment setup. Extend this work with your own customisations wherever applicable and drop a comment if you need any clarifications!
Hello Montreal! We’ll be at the Startupfest ’22 all three days, so drop by and say hi. We’d love to see what you’re building, and early birds get a free (limited) design audit or tech consultation. DM us on Twitter (@tarkalabs) and let’s talk.
