Create, construct and weld the perfect secure container with Habitus

alt

To succeed with Docker and running Docker in production the key element is your image. The image - effectively the blueprint for running your containers, needs to be as good as your code.

For this post, I'll be building my container using the one process per container practice. In almost all cases, you should only run a single process in a single container. Decoupling applications into multiple containers will make it much more easier to scale horizontally and reuse containers.

The evil Dockerfile

The container produced by the image your Dockerfile defines should be as ephemeral as possible. By “ephemeral” we mean it can be stopped and destroyed, with a new one built and put back in its place with the absolute minimum of set-up and configuration.

I and lots of our customers have a love/hate relationship with creating images and getting to grips with the evil Dockerfile. But why so evil?

A Dockerfile describes the runtime environment of your single process (= your app). If you go down the path of complete containerization - which on the surface may seem fun and exciting, you will struggle with maintaining your Dockerfile and DockerCompose files a lot.

As developers, we're not always familiar with the SecOps (security and operations) side of application management, which is one of the primary reasons customers build, deploy and manage their apps using Cloud 66. The frustrating part of the evil Dockerfile is all about SecOps.

The not so good Docker flow

For instance, you probably need different environments and stuff like compile time dependencies for development purposes, but are your Dockerfiles also reusable in production? Before you know it, you'll be building different Dockerfiles, makefiles, scripts and then trying to engineer your way out of it all, leaving a lot of trash and vulnerabilities inside those containers.

Have you ever run the command docker history command? Running
# docker history <image> will show all the file system layers of your image. All the trash and vulnerabilities are still there, and it will be your responsibility to clean up your mess.

Take this Dockerfile as an example:

#the go base image
FROM golang:1.6

#setup source and excutable directories
ARG SRC_DIR=/usr/src/secretapp  
ARG EXEC_DIR=/usr/bin

#copy the sourcecode
COPY src $SRC_DIR

#compile the code
WORKDIR $SRC_DIR  
RUN go build -v -ldflags "-s" -a -installsuffix cgo -o secretapp

#copy binary and remove source
RUN cp secretapp $EXEC_DIR  
RUN rm -rf $SRC_DIR

#run the process as a non-root user
USER daemon  
CMD ["/usr/bin/secretapp"]  

And this very secret source code nobody should copy:

package main

import (  
    "fmt"
    "time"
    "os"
)

func main() {  
    for {
        fmt.Println("Protect all your base @", os.Getenv("HOSTNAME"))
        time.Sleep(1000 * time.Millisecond)
    }
}

When you build the image docker build -t cloud66/bloghabitus:full, we use compile time and runtime dependencies in one image.

Run the application:

$ docker run -it --rm cloud66/bloghabitus:full
Protect all your base @ 1daf9bdbe0cf  
Protect all your base @ 1daf9bdbe0cf  
Protect all your base @ 1daf9bdbe0cf  
Protect all your base @ 1daf9bdbe0cf  

It's working great. Done? Not yet. Run the docker history command and see for yourself:

$ docker run -it --rm cloud66/bloghabitus:full
...
fc2743fd7227        About an hour ago   /bin/sh -c #(nop) COPY dir:b09c67f26c315df980   187 B  
...

What's this line doing here? It that my source code?

Yup, but if I want to access the source code it's not there right?

$ docker run -it --rm cloud66/bloghabitus:full cat /usr/src/secretapp/main.go
cat: /usr/src/secretapp/main.go: No such file or directory  

Check, but it's still somewhere in the Docker image layers.

Layers?

alt

A layered file system

Yes! Docker uses a layered file system, which is what makes Docker as a technology really exciting. Each Docker image references a list of read-only layers that represent filesystem differences. Layers are stacked on top of each other to form a base for a container’s root filesystem.

Deleting files just creates an extra layer, which represents the filesystem differences. It's not gone!

I can run for example a container using the hash of the layer before the source code is deleted.

$ docker run -it --rm fc2743fd7227 cat /usr/src/secretapp/main.go
...
   fmt.Println("Protect all your base @", os.Getenv("HOSTNAME")) 
...

BAM! I just got pawned by a hax0r if they were to gain access to my (private) image repo or host system.

Squash those flies

One way to deal with this security hole is to squash all those layers into one big fat filesystem. The downside is, you're losing the speed with which you can re-deploy your apps. This is the case because you're no longer only just downloading the layers which have been changed.

To squash the image you can use docker-squash.

$ docker save cloud66/bloghabitus:full | sudo docker-squash -t cloud66/bloghabitus:full -verbose | docker load

You only got one layer and no access to the source code anymore, and no layers can be found before the deletion of the source code.

$ docker run -it --rm cloud66/bloghabitus:full cat /usr/src/secretapp/main.go
cat: /usr/src/secretapp/main.go: No such file or directory  

OK - great! But that's not all folks. Take a look at the image security scan of the base image FROM golang:1.6 you're using?

alt

Doesn't look good. Probably a lot of stuff you don't use, but those security issues can and WILL be exploited by clever hackers.

With tools like bench and twistlock you can explore vulnerabilities in your images, running containers and adjust your Dockerfiles accordingly.

In the end, we want the smallest attack footprint and no leaking of secrets like source code, private keys and other things, which will harm your business.

The solution to love Dockerfiles and Docker again applies to all programming languages also interpreted languages like Ruby or Node, which maybe you want to encrypt the source code in another build step.

Solutions to love Dockerfiles again

Well we've been there, done that and came up with a solution. Have a look at Habitus, an open-source build flow tool for Docker. It's worth noting, you can also use Habitus on Buildgrid, the automated image build feature of Cloud 66.

Basically, we have two build steps:
1. compile the binary
2. get the binary in the smallest possible image with zero-to-no vulnerabilities.

For Habitus to work we need a build.yml and the Dockerfiles.

build:  
  version: 2016-03-14
  steps:
    compile:
      name: cloud66/bloghabitus_compile
      dockerfile: Dockerfile.compile
      artifacts:
        - /usr/bin/secretapp
    runtime_minimal:
      name: cloud66/bloghabitus:minimal
      dockerfile: Dockerfile.minimal.production
      depends_on:
        - compile
    runtime_alpine:
      name: cloud66/bloghabitus:alpine
      dockerfile: Dockerfile.alpine.production
      depends_on:
        - compile

Dockerfile.compile

#the base image
FROM golang:1.6


#setup source and excutable directories
ARG SRC_DIR=/usr/src/secretapp  
ARG EXEC_DIR=/usr/bin

#copy the sourcecode
COPY src $SRC_DIR

#compile the code
WORKDIR $SRC_DIR  
RUN go build -v -ldflags "-s" -a -installsuffix cgo -o secretapp

#copy binary
RUN cp secret app$EXEC_DIR  

Dockerfile.alpine.production is a Dockerfile using the Alpine Linux distribution. This one is very small (5MB) and has a package manager, and can be used if you need to install some tools you need in production.

alt

#the base image
FROM alpine:3.4

#copy compiled binary
ADD secretapp /secretapp

#run the process as a non-root user
USER daemon  
CMD ["/secretapp"]  

Dockerfile.minimal.production is a Dockerfile using scratch. The scratch base image has no tools, and can only run a process you provide. 0 vulnerabilities.

#the base image
FROM scratch

#copy compiled binary
ADD secretapp /

#run app
CMD ["/secretapp"]  

Now we can run Habitus.

$ habitus --use-tls=false -host=unix:///var/run/docker.sock -secrets=false -keep-all
2016/09/07 14:37:52 ▶ Using 'xxx/cloud66_blogpost/build.yml' as build file  
2016/09/07 14:37:52 ▶ Building 3 steps  
2016/09/07 14:37:52 ▶ Step 0 - compile: cloud66/bloghabitus_compile  
2016/09/07 14:37:52 ▶ Step 1 - runtime_minimal: cloud66/bloghabitus:minimal  
2016/09/07 14:37:52 ▶ Step 2 - runtime_alpine: cloud66/bloghabitus:alpine  
2016/09/07 14:37:52 ▶ Parallel build for cloud66/bloghabitus_compile  
...

Done. We've constructed and welded the smallest possible image.

$ docker history cloud66/bloghabitus:alpine
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT  
1d53f6955785        3 hours ago         /bin/sh -c #(nop)  CMD ["/bin/sh" "-c" "/secr   0 B  
923d1d2b86a0        3 hours ago         /bin/sh -c #(nop)  USER [daemon]                0 B  
d300bc5be88f        3 hours ago         /bin/sh -c #(nop) ADD file:193739de4959f603ab   1.59 MB  
184352182c50        3 months ago        /bin/sh -c #(nop) ADD file:f6b806676d872f26a4   4.797 MB

$ docker history cloud66/bloghabitus:minimal
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT  
1e042380fdc8        3 hours ago         /bin/sh -c #(nop)  CMD ["/bin/sh" "-c" "/secr   0 B  
59b3b818a30b        3 hours ago         /bin/sh -c #(nop) ADD file:193739de4959f603ab   1.59 MB  

Summary

Habitus will help you to create, construct and weld the perfect secure container. Docker images are the key ingredient of successfully running Docker in production. Always take care of what you deploy inside a container and always check the sources (base images) to make sure you remove the things you don't need.

With Habitus and the marriage with Buildgrid you can now build, deploy and manage your applications with ease. And don't forget we also provide high-end security of your Docker cluster nodes using ActiveProtect and very strong firewall rules.

Daniël van Gils

Daniël van Gils is a developer advocate at Cloud 66. He helps other developers craft web apps and container based architectures with ♥, to deploy on any server or public cloud.

Amsterdam & London
Subscribe and get updates

Thank you! We keep in touch.

Have feedback? Please get in touch @cloud66 on Twitter.

Try Cloud 66 for Free, No credit card required