Having a Kubernetes cluster is the first step towards building a more resilient and scalable infrastructure. While building a Kubernetes cluster is becoming easier every day, building a solid deployment pipeline to deploy our applications onto can feel a bit scary. It shouldn’t be. Let’s see what we need to deploy our applications onto a cluster in a safe, secure, reliable and repeatable way.
1. Be a Good Citizen
Fewer Kubernetes cluster are easier to manage and cost less. The other side of this equation is that you’re most likely going to share this cluster with others in your team or company. This means being more careful when deploying apps or changing configuration. Role Based Access Control (RBAC) can be a useful feature to control access for different parties based on namespaces but it’s not going to be enough in cases when the users of a cluster should be given higher privileges so they can make changes across the cluster (like adding new storage types).
It helps to remember that while protecting your cluster from malicious actions is very important, in this case you are mostly looking for ways to give developers control they need to do their work while preventing the cluster to break for others. That’s why tools like Copper can be useful additions to your toolbox. With Copper you can write “policies” that stop bad configuration from hitting your cluster.
2. Find a Home for Your Configuration Files
Kubernetes can be (and should be) configured with configuration files that are kept in version control and can be applied to the cluster at any time without worry. Some of these configuration files will hold cluster configurations (like your RBAC configuration or some of your cluster services) and some of those will be specific to each application that runs on the cluster. Application specific configuration files themselves can belong to different services. The code for each one of these services usually lives in a different git repository itself. All of this means you’re going to end up with at least a few git repositories to hold your cluster configuration files. Some teams would like to keep application configuration files next to the application services themselves and while that usually works, it still leaves you to find a repository for your cluster-wide configuration files.
One note: while git is an amazing source control system, it’s not known for fine grained user access control over individual files, so if you choose to store application configuration files next to your code, you need to make sure you’re comfortable with them being visible to and modified by developer teams that work on the service.
3. Keep Your Secrets Safe
Kubernetes can store your secrets in a Secret storage and give it to individual services as tempfs files or environment variables. However the secrets themselves can be exposed to the outside world if you keep them in a Kubernetes configuration file. Some teams decided to keep the secrets out of configuration files to work around this problem. This is not a good idea as it breaks the sync between code and configuration. There are solutions available to make this work: encrypting secret configuration files or using placeholders for tools like SOPS. Skycap uses fine grained ACLs to grant “write-only” access rights to users so you can still keep secrets in a Kubernetes configuration file without compromising their values. This gives developers the control they need to make changes to the configuration file without weakening your security.
4. Don’t Use Magic
I think Kubernetes should be a magic free zone. It is complicated enough as the operating system of your infrastructure to confuse most of us without the added complexity of magic! The most maintainable systems are the ones that can be understood and the most understandable systems are the ones that follow conventions and allow the least exceptions to the rules.
To make the most of Kubernetes, sometimes you need to make changes to a configuration file depending on where it’s applied. For example, you might want to make sure all your services in production log to your logging facility. This might not be a requirement for non-production environments which leaves you with 3 options:
- Create 2 separate configuration files: one for production and one for non-production environments with the production configuration files containing modifications that ship the logs over from STDOUT to a logging facility.
- Ship the logs your logging facility for all environments
- Make changes to your configuration files on the fly or behind the scenes to inject what’s needed to change the behavior in production but no other environment.
If you ask me, the second option is the best one! While it might end up to be very expensive to send the logs from all environments to an external log processing facility, at least unlike solution 1 it is maintainable and unlike solution 3 it’s no obfuscated and doesn’t cause confusion 6 months later when everyone has forgotten how 1 environment behaves differently.
More advanced networking or security solutions like Istio require injection of a “sidecar” container in every pod to work and while it is tempting to inject those configurations somewhere in the cluster, it’s better to find a solution that changes the configuration files before they are applied to the cluster so you can see them, inspect them and most importantly see how they are explicitly changed by a converter. Tools like Alterant can help with scenarios like this by converting the files without applying them directly to the cluster.
5. Solidify Your Best Practices
Over time, you will develop best practices for deploying and maintaining your applications and cluster. Many of these practices are applicable to different applications and your team will thank you if they could reuse your experience every time they are going to interact with Kubernetes. For example, making sure all your Kubernetes assets (deployments, replica sets, services, etc) are annotated properly so they can be tracked in production or used for cost allocation later on, is a great idea. By creating templates for different Kubernetes configuration files that can be reused by your team every time, practices like this can propagate your team easier and become part of your DevOps discipline.
Helm as a package manager for Kubernetes works well for off the shell package and products like Skycap are great for building and maintaining a library of reusable Kubernetes configuration files.
Summary
Kubernetes is a great tool to improve your infrastructure and DevOps practices in many different ways, from security to scalability and reliability. Extracting those values out of this great tool requires practice, experience and using the best tools for the job.