To succeed with Docker and running Docker in production the key element is your image. The image - effectively the blueprint for running your containers, needs to be as good as your code.
For this post, I'll be building my container using the one process per container practice. In almost all cases, you should only run a single process in a single container. Decoupling applications into multiple containers will make it much more easier to scale horizontally and reuse containers.
The evil Dockerfile
The container produced by the image your Dockerfile defines should be as ephemeral as possible. By “ephemeral” we mean it can be stopped and destroyed, with a new one built and put back in its place with the absolute minimum of set-up and configuration.
I and lots of our customers have a love/hate relationship with creating images and getting to grips with the evil Dockerfile. But why so evil?
A Dockerfile describes the runtime environment of your single process (= your app). If you go down the path of complete containerization - which on the surface may seem fun and exciting, you will struggle with maintaining your Dockerfile and DockerCompose files a lot.
As developers, we're not always familiar with the SecOps (security and operations) side of application management, which is one of the primary reasons customers build, deploy and manage their apps using Cloud 66. The frustrating part of the evil Dockerfile is all about SecOps.
The not so good Docker flow
For instance, you probably need different environments and stuff like compile time dependencies for development purposes, but are your Dockerfiles also reusable in production? Before you know it, you'll be building different Dockerfiles, makefiles, scripts and then trying to engineer your way out of it all, leaving a lot of trash and vulnerabilities inside those containers.
Have you ever run the command docker history command? Running# docker history <image> will show all the file system layers of your image. All the trash and vulnerabilities are still there, and it will be your responsibility to clean up your mess.
Take this Dockerfile as an example:
#the go base image
FROM golang:1.6
#setup source and excutable directories
ARG SRC_DIR=/usr/src/secretapp
ARG EXEC_DIR=/usr/bin
#copy the sourcecode
COPY src $SRC_DIR
#compile the code
WORKDIR $SRC_DIR
RUN go build -v -ldflags "-s" -a -installsuffix cgo -o secretapp
#copy binary and remove source
RUN cp secretapp $EXEC_DIR
RUN rm -rf $SRC_DIR
#run the process as a non-root user
USER daemon
CMD ["/usr/bin/secretapp"]
And this very secret source code nobody should copy:
package main
import (
"fmt"
"time"
"os"
)
func main() {
for {
fmt.Println("Protect all your base @", os.Getenv("HOSTNAME"))
time.Sleep(1000 * time.Millisecond)
}
}
When you build the image docker build -t cloud66/bloghabitus:full, we use compile time and runtime dependencies in one image.
Run the application:
$ docker run -it --rm cloud66/bloghabitus:full
Protect all your base @ 1daf9bdbe0cf
Protect all your base @ 1daf9bdbe0cf
Protect all your base @ 1daf9bdbe0cf
Protect all your base @ 1daf9bdbe0cf
It's working great. Done? Not yet. Run the docker history command and see for yourself:
$ docker run -it --rm cloud66/bloghabitus:full
...
fc2743fd7227 About an hour ago /bin/sh -c #(nop) COPY dir:b09c67f26c315df980 187 B
...
What's this line doing here? It that my source code?
Yup, but if I want to access the source code it's not there right?
$ docker run -it --rm cloud66/bloghabitus:full cat /usr/src/secretapp/main.go
cat: /usr/src/secretapp/main.go: No such file or directory
Check, but it's still somewhere in the Docker image layers.
Layers?
A layered file system
Yes! Docker uses a layered file system, which is what makes Docker as a technology really exciting. Each Docker image references a list of read-only layers that represent filesystem differences. Layers are stacked on top of each other to form a base for a container’s root filesystem.
Deleting files just creates an extra layer, which represents the filesystem differences. It's not gone!
I can run for example a container using the hash of the layer before the source code is deleted.
$ docker run -it --rm fc2743fd7227 cat /usr/src/secretapp/main.go
...
fmt.Println("Protect all your base @", os.Getenv("HOSTNAME"))
...
BAM! I just got pawned by a hax0r if they were to gain access to my (private) image repo or host system.
Squash those flies
One way to deal with this security hole is to squash all those layers into one big fat filesystem. The downside is, you're losing the speed with which you can re-deploy your apps. This is the case because you're no longer only just downloading the layers which have been changed.
To squash the image you can use docker-squash.
$ docker save cloud66/bloghabitus:full | sudo docker-squash -t cloud66/bloghabitus:full -verbose | docker load
You only got one layer and no access to the source code anymore, and no layers can be found before the deletion of the source code.
$ docker run -it --rm cloud66/bloghabitus:full cat /usr/src/secretapp/main.go
cat: /usr/src/secretapp/main.go: No such file or directory
OK - great! But that's not all folks. Take a look at the image security scan of the base image FROM golang:1.6 you're using?
Doesn't look good. Probably a lot of stuff you don't use, but those security issues can and WILL be exploited by clever hackers.
With tools like bench and twistlock you can explore vulnerabilities in your images, running containers and adjust your Dockerfiles accordingly.
In the end, we want the smallest attack footprint and no leaking of secrets like source code, private keys and other things, which will harm your business.
The solution to love Dockerfiles and Docker again applies to all programming languages also interpreted languages like Ruby or Node, which maybe you want to encrypt the source code in another build step.
Solutions to love Dockerfiles again
Well we've been there, done that and came up with a solution. Have a look at Habitus, an open-source build flow tool for Docker. It's worth noting, you can also use Habitus on Buildgrid, the automated image build feature of Cloud 66.
Basically, we have two build steps:
- compile the binary
- get the binary in the smallest possible image with zero-to-no vulnerabilities.
For Habitus to work we need a build.yml and the Dockerfiles.
build:
version: 2016-03-14
steps:
compile:
name: cloud66/bloghabitus_compile
dockerfile: Dockerfile.compile
artifacts:
- /usr/bin/secretapp
runtime_minimal:
name: cloud66/bloghabitus:minimal
dockerfile: Dockerfile.minimal.production
depends_on:
- compile
runtime_alpine:
name: cloud66/bloghabitus:alpine
dockerfile: Dockerfile.alpine.production
depends_on:
- compile
Dockerfile.compile
#the base image
FROM golang:1.6
#setup source and excutable directories
ARG SRC_DIR=/usr/src/secretapp
ARG EXEC_DIR=/usr/bin
#copy the sourcecode
COPY src $SRC_DIR
#compile the code
WORKDIR $SRC_DIR
RUN go build -v -ldflags "-s" -a -installsuffix cgo -o secretapp
#copy binary
RUN cp secret app$EXEC_DIR
Dockerfile.alpine.production
is a Dockerfile using the Alpine Linux distribution. This one is very small (5MB) and has a package manager, and can be used if you need to install some tools you need in production.
#the base image
FROM alpine:3.4
#copy compiled binary
ADD secretapp /secretapp
#run the process as a non-root user
USER daemon
CMD ["/secretapp"]
Dockerfile.minimal.production is a Dockerfile using scratch. The scratch base image has no tools, and can only run a process you provide. 0 vulnerabilities.
#the base image
FROM scratch
#copy compiled binary
ADD secretapp /
#run app
CMD ["/secretapp"]
Now we can run Habitus.
$ habitus --use-tls=false -host=unix:///var/run/docker.sock -secrets=false -keep-all
2016/09/07 14:37:52 ▶ Using 'xxx/cloud66_blogpost/build.yml' as build file
2016/09/07 14:37:52 ▶ Building 3 steps
2016/09/07 14:37:52 ▶ Step 0 - compile: cloud66/bloghabitus_compile
2016/09/07 14:37:52 ▶ Step 1 - runtime_minimal: cloud66/bloghabitus:minimal
2016/09/07 14:37:52 ▶ Step 2 - runtime_alpine: cloud66/bloghabitus:alpine
2016/09/07 14:37:52 ▶ Parallel build for cloud66/bloghabitus_compile
...
Done. We've constructed and welded the smallest possible image.
$ docker history cloud66/bloghabitus:alpine
IMAGE CREATED CREATED BY SIZE COMMENT
1d53f6955785 3 hours ago /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/secr 0 B
923d1d2b86a0 3 hours ago /bin/sh -c #(nop) USER [daemon] 0 B
d300bc5be88f 3 hours ago /bin/sh -c #(nop) ADD file:193739de4959f603ab 1.59 MB
184352182c50 3 months ago /bin/sh -c #(nop) ADD file:f6b806676d872f26a4 4.797 MB
$ docker history cloud66/bloghabitus:minimal
IMAGE CREATED CREATED BY SIZE COMMENT
1e042380fdc8 3 hours ago /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/secr 0 B
59b3b818a30b 3 hours ago /bin/sh -c #(nop) ADD file:193739de4959f603ab 1.59 MB
Summary
Habitus will help you to create, construct and weld the perfect secure container. Docker images are the key ingredient of successfully running Docker in production. Always take care of what you deploy inside a container and always check the sources (base images) to make sure you remove the things you don't need.
With Habitus and the marriage with Buildgrid you can now build, deploy and manage your applications with ease. And don't forget we also provide high-end security of your Docker cluster nodes using ActiveProtect and very strong firewall rules.