Debugging Docker containers

It's been really exciting to see people adopting Docker in production these past few months. Working with our users to help them get up and running has been challenging at times but very satisfying, and I'd like to write about some common issues when you start using Docker in production. It can be challenging at first, but you'll soon be reaping the benefits of this amazing technology.

Does it work locally?

One big benefit of containers is that they're static, so you can finally achieve consistency across environments. If you're having issues with a specific image, the first step is to try running it locally. Running Docker in development is very straightforward, and will make troubleshooting much easier.

If you can build your image locally, you should be able to start your container by running docker run -it <image_name>:<tag> <command>. The command used will be determined by the service running in the container, and any output will put you on the right track to determining what the problem is.

Does it work on Cloud 66?

The first step here is to check if you've given your service a start command, either in their Dockerfile or in the service configuration used by Cloud 66 to start a container. If you've covered this, these are the steps I'd take to troubleshoot:

Automatic (live)

Use the Cloud 66 toolbelt to run your container (with all necessary parameters automatically loaded):

cx run -s <stack_name> --server <server_name> --service <service_name>

This command will start a new container from your image with the benefit of having the required environment variables loaded, as well as assigning it a network and DNS, and then attach to it. You will now be able to see any error messages from startup, which will help you troubleshoot.

You can optionally add '/bin/bash' to the end of that command to start a Bash shell in the container, which helps debug a container from the inside. These commands are provided for you to copy & paste if you visit your Stack page -> Docker server -> Server name -> View running containers (in the right menu).

Manual (retrospective)

1. Start by SSHing to your server.
2. We automatically output the logs for your container start process to /var/log/containers on your host, and the files are named in the following convention: <service>_start_errors.log. These logs files will contain any output created by your start command, and are a good starting point in understanding why your container won't start.
3. Check for any running containers with docker ps - if the container isn't running there, there may be an issue preventing it from starting.
4. In this case, we can run docker ps -a to check for old containers that have failed to start (or are no longer running). Take note of the container ID for your latest deployment, as we'll need it in the next step.
5. Now let's check the log of that failed container by running docker logs <container-id>, which will output any error messages for why the container couldn't start.

In general, most issues you'll face have likely been experienced by others. A quick Google search of any error message will lead you in the right direction, and feel free to reach out to us if you're stuck!