Overview
We all love Docker for how it simplifies the development workflow. With Docker, it is possible to run different components of an application inside different containers and run all of them on a development laptop. Docker Compose takes this one step further by simplifying the task of starting and stopping all those containers together in one single file.
Going beyond development into production, Docker Compose can still be a very useful tool to keep Docker hosting simple. If you intend to use Docker compose for hosting your app in production as well, the following tips might help you save some headache and time.
Tip #1: Keep your compose file the same across dev and production
Let’s start with this point. While Docker suggests having different Docker Compose files in development and production, we have seen many cases that this has caused issues. By keeping your development and production files separate, at the best, you end up reapplying all modifications in your development Compose file to your production one. This is a manual and error-prone process. In many cases, some changes are missed and your app breaks just as it goes to production. This approach also ignores other environments than development and production, like Staging or QA. Any new environment will add to this complexity. As a rule, you should always try to keep only a single Docker Compose file for all environments. Some of the tips below will elaborate more on some of the ways of doing so.
Tip #2 Keep your compose file version controlled and next to the app
Your Docker Compose file is there to provide a description of your application components. This means any changes in your application should be reflected in your Docker Compose file. Changes to the port of a service or a new service added to the application should be made in your Docker Compose file as well. That's why the best place for your Docker Compose file is next to your app and version controlled in lockstep with your application.
This might not be as straight-forward for an application delivered as micro-services, but even in a completely decoupled application consisting of completely different git repositories, there is a core repository that can be a good home for your Compose file. Putting your Compose file in the repository you should follow the same rules you would with the code: no secrets should be put in source control. Luckily, Docker Compose allows you to refer to environment variables in service settings which is where you can store secrets for your operations with a format like ENV_VAR=${ENV_VAR}
Another option is to use Docker secrets if that can be also used in your development workflow.
Tip #3 Know the gotchas between Compose and Swarm (creation order, retries, volumes)
Docker’s aim is to keep the same Compose format usable for both Docker Compose and Docker Swarm. If you are thinking of using Docker Compose for your development and Docker Swarm for production, you can use Docker Compose Version 3 format. However, this compatibility is not complete: Some elements like depends_on are ignored by Swarm. Also, some Swarm attributes are not supported by Compose (like deploy). While most of those incompatibilities are ok to live by, they can cause a serious headache for you in some cases like service startup order: While Compose relies on depends_on attribute, Swarm doesn’t honor service dependency and simply starts services randomly and expects them to find each other eventually. This can mean changes in your app to ensure resilience during the startup phase (see Tip #7).
Tip #4 Keep images explicitly versioned
This might seem obvious but keeping images explicitly versioned in your Compose file saves a lot of hassle later on. By default, Docker uses latest as the tag for an image when no tag is specified. While this behavior works fine in development and is very convenient (since you don’t need to change the Compose file every time you make a change to a service), it makes tracking of changes difficult and running in production indeterministic. It is best to explicitly specify the version of each service in the Compose file. You can use the ${ENV} format to get this version from an external source like your git ref or another parameter to avoid changing your Compose file every time.
Tip #5 Keep databases out of containers
In the micro-services / containerized crazy world we live in now, this tip is borderline heresy. You are supposed to keep everything in containers after all. However, until we have a reliable and developer environment-friendly storage solution for containers, running containers in databases means more pain than gain. Most applications use 2 to 3 different “supporting” component like databases or message buses which require persistent storage and most of us use stable versions of those: MySQL or RabbitMQ to name a few. Hosting and high availability for those components is a much older problem than container scheduling, storage persistence and high availability and as such there is much more information around on how to build a reliable MySQL Master / Slave cluster natively than running it in a container. Moreover, most databases are available as services like RDS which are scalable and reliable.
Now, I know my top Tip #1 was to keep your production and development the same but here I am going to make an exception. While we are going to keep the Docker Compose files the same across all environments, we are going to run databases in containers in development while using native databases for production. This is achievable by using environment variables to point to the right database service. For example, in a fully containerized system hosted in Docker Compose, you can reach the database server by using its service name. When your database is running as a standalone service, you can make sure the environment variable for the database address is pointing to the native database service. This leaves a second database service running inside of a container which is not used in production, but we can find other ways not to start it (labels) or shut it down after it started (starting the DB through a shim that returns an exit code depending on the environment). While none of those solutions are ideal, the absence of a good storage solution is not going to last for long. That’s why at Cloud 66 we use a file called service.yml which is similar to Docker Compose file but has databases and other data sources listed separately.
Tip #6 Use the same ports and load balancers
In a development environment, you don’t need to run load balancers in front of your services because you’re not going to have multiple containers for the same service. However, this is different in production. With the exception of single container services (persisted services), you are very likely to run multiple instances of the same service on each node (server). This means the load needs to be balanced across your containers using a load balancer. This is achieved by using Nginx or HAProxy in most cases. Following Tip #1, I think it’s a good idea to have load balancers running in your development environment as well. This is not to allow multiple containers running at the same time, but to keep your port numbers for services consistent and avoid making those parameterized.
Tip #7 Always use defined volumes
Docker Compose allows you to define volumes for each service the same way they are defined with the docker run command. While this is possible it is not advisable. The best way to use volumes in Docker Compose is to define them a named volumes on top of your Compose file and use those names. This would allow your Ops team to replace the underlying storage for the volume with a SAN or NAS or other types of storage while during development you will just use a local volume mounted onto the container.
Tip #8 Keep your environment variables in one place
I’m not a fan of how Docker Compose handles environment variables. You can import them from a file like production.env or explicitly state them in your Compose file. In Compose file you can use the $ syntax to refer to process environment variables while the .env file doesn’t support that. This makes the task of finding out where the environment variables are coming from difficult and opaque. I would always advise on explicitly referring to all the required environment variables for a service at the point service is defined and use the $ syntax to pull them in based on the environment. Don’t use .env files. They make your life easier in the short term and cause hair loss in the long term.
Tip #9 Use strong conventions
With so many building blocks of a containerized application getting lost is very easy: git repositories, Dockerfiles, images, and services they all point to the same part of your application. While it is easy to get started with pulling the code out of your git, building it on your laptop into a Docker image and then adding that as a service to Docker compose, it will start to get confusing very quickly. Therefore this Tip has some “sub-tips”:
Tip #9.1 Make your containers traceable to codebase
You might use multiple git repos or a single git repo with multiple Dockerfiles or start commands for your services. While these are all ok, make sure you follow a naming convention that allows you to look at a running container and tell you exactly how that container was built, which git repo and using which Dockerfile.
Tip #9.2 Make your containers traceable to git commit
This is not a repeat of the last tip and not to iterate the importance of it. This tip is about git refs and making sure you know how to trace a running container back to the git ref (git hash or tag) that was used to build it. With Docker, you now have 2 repositories to trace code through: one is your git repo and the other one is your Docker repo. This makes tracing containers back to the git ref less transparent. To avoid this use a container native CI/CD tool that is integrated with your Orchestrator (shameless plug: use Cloud 66). Short of that, use the git ref as the tag for the container running.
Tip #10 Make sure your services return the correct exit code
We all use logging or console output to show errors in our apps. In a containerized world, exit codes have a special place. Orchestrators (like Compose, Swarm, Kubernetes or Cloud 66 Maestro) use exit codes to tell if a service started or failed to start. The change of behavior in starting containers between Compose and Swarm means exist codes are even more important. While Docker Compose allows defining service dependencies, Docker Swarm (and Kubernetes) keep trying to start a crashing container (that exits with a code other than 0) several times. This means you need to make sure not only errors are logged but the exit codes are accurate as they will show up in the orchestrator’s logs and can give you valuable clues as to what’s going on with your application.
Summary
While this list is not exhaustive, it covers some of the gotchas we’ve seen helping our customers use Cloud 66 Skycap and Maestro. I will be updating this list in future and would love to hear your thoughts on it.