To succeed with Docker and running Java containers in production the key element is your image. The image - effectively the blueprint for running your containers, needs to be as good as your code.
For this post, I'll be building my container using the one Java process per container practice. In almost all cases, you should only run a single process in a single container. Decoupling applications into multiple containers will make it much easier to scale horizontally and reuse containers.
I'm advocating the need for the RIGHT production image a lot during my talks and workshops. I'm speaking about this subject during Dutch Container Day, Devops Pro Russia and All Day Devops.
The evil Dockerfile
The container produced by the image your Dockerfile defines should be as ephemeral as possible. By “ephemeral” we mean it can be stopped and destroyed, with a new one built and put back in its place with the absolute minimum of set-up and configuration.
I and lots of our customers have a love/hate relationship with creating images and getting to grips with the evil Dockerfile. But why so evil?
A Dockerfile describes the runtime environment of your single process (= your app). If you go down the path of complete containerization - which on the surface may seem fun and exciting, you will struggle with maintaining your Dockerfile and DockerCompose files a lot.
As developers, we're not always familiar with the SecOps (security and operations) side of application management, which is one of the primary reasons customers build, deploy and manage their apps using Cloud 66. The frustrating part of the evil Dockerfile is all about SecOps.
The not so good Docker flow
For instance, you probably need different environments and stuff like Java compile time dependencies for development purposes (using a pom.xml and maven), but are your Dockerfiles also reusable in production? Do we need all those Java sources inside a production ready container? The answer? No, you don't!
Before you know it, you'll be building different Dockerfiles, makefiles, scripts and then trying to engineer your way out of it all, leaving a lot of trash and vulnerabilities inside those containers.
Let me show me an example!
Have you ever run the command docker history command? Running# docker history <image> will show all the file system layers of your image. All the trash and vulnerabilities are still there, and it will be your responsibility to clean up your mess.
Take this Dockerfile to build and run a Java application as an example:
FROM maven:3-jdk-8-alpine
# creating the source dir
ENV APPLICATION_DIR /app
ENV SOURCE_DIR /src
RUN mkdir $APPLICATION_DIR && mkdir $SOURCE_DIR
WORKDIR $SOURCE_DIR
# selectively add the POM file
COPY pom.xml $SOURCE_DIR
# get all the downloads out of the way
RUN mvn dependency:resolve
# copy the source code
COPY . $SOURCE_DIR
# package the project
RUN mvn package
RUN cp target/artifact-1.0.0.jar $APPLICATION_DIR
# remove source
RUN rm -rf $SOURCE_DIR
# run the API
WORKDIR $APPLICATION_DIR
EXPOSE 8080
CMD java -jar artifact-1.0.0.jar
And this very secret source code nobody should copy:
package secret;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class VerySecret {
}
When you build the image docker build -t cloud66/bloghabitus_java:1.0.0, we use compile time and runtime dependencies in one image.
Run the application:
$ docker run -it --rm cloud66/bloghabitus_java:1.0.0
VerySecretJavaApplication runs on port 5000 @ 1daf9bdbe0cf
It's working great. Done? Not yet. Run the docker history command and see for yourself:
$ docker run -it --rm cloud66/bloghabitus_java:1.0.0
...
fc2743fd7227 About an hour ago /bin/sh -c #(nop) ADD dir:b09c67f26c315df980 187 B
...
What's this line doing here? It that my source code?
Yup, but if I want to access the source code it's not there right?
$ docker run -it --rm cloud66/bbloghabitus_java:1.0.0 cat /src/main/java/secret/VerySecret.java
cat: /src/main/java/secret/VerySecret.java: No such file or directory
Check, but it's still somewhere in the Docker image layers.
Layers?
A layered file system
Yes! Docker uses a layered file system, which is what makes Docker as a technology really exciting. Each Docker image references a list of read-only layers that represent filesystem differences. Layers are stacked on top of each other to form a base for a container’s root filesystem.
Deleting files just creates an extra layer, which represents the filesystem differences. It's not gone!
I can run for example a container using the hash of the layer before the source code is deleted.
$ docker run -it --rm fc2743fd7227 cat /src/main/java/secret/VerySecret.java
...
public class VerySecret {
}
...
BAM! I just got pawned by a hax0r if they were to gain access to my (private) image repo or host system.
Squash those flies
One way to deal with this security hole is to squash all those layers into one big fat filesystem. The downside is, you're losing the speed with which you can re-deploy your apps. This is the case because you're no longer only just downloading the layers which have been changed.
To squash the image you can use docker-squash.
$ docker save cloud66/bloghabitus_java:1.0.0 | sudo docker-squash -t cloud66/bloghabitus_java:1.0.1 -verbose | docker load
You only got one layer and no access to the source code anymore, and no layers can be found before the deletion of the source code.
$ docker run -it --rm cloud66/bloghabitus_java:1.0.1 cat /src/main/java/secret/VerySecret.java
cat: /src/main/java/secret/VerySecret.java: No such file or directory
OK - great! But that's not all folks. Take a look at the image security scan of the base image FROM maven:3-jdk-8-alpine you're using?
Doesn't look good. Probably a lot of stuff you don't use, but those security issues can and WILL be exploited by clever hackers.
With tools like bench and twistlock you can explore vulnerabilities in your images, running containers and adjust your Dockerfiles accordingly. In the end, we want the smallest attack footprint and no leaking of secrets like source code, private keys and other things, which will harm your business. The solution to love Dockerfiles and Docker again applies to all programming languages also interpreted languages like Ruby or Node, which maybe you want to encrypt the source code in another build step.
Solutions to love Dockerfiles again
Well we've been there, done that and came up with a solution. Have a look at Habitus, an open-source build flow tool for Docker. It's worth noting, you can also use Habitus on Buildgrid, the automated image build feature of Cloud 66.
Basically, we have two build steps:
- compile the binary (step = builder)
- get the binary in the smallest possible image with zero-to-no vulnerabilities. (step = production)
For Habitus to work we need a build.yml and the Dockerfiles.
build:
version: 2016-03-14 # version of the build schema.
steps:
builder:
name: builder/java
dockerfile: Dockerfile.build
artifacts:
- /app/artifact-1.0.0.jar
production:
name: example/java
dockerfile: Dockerfile.production
depends_on:
- builder
We can still use the Dockerfile we created in the last example. Let me rename that to Dockerfile.compile
FROM maven:3-jdk-8-alpine
# creating the source dir
ENV APPLICATION_DIR /app
ENV SOURCE_DIR /src
RUN mkdir $APPLICATION_DIR && mkdir $SOURCE_DIR
WORKDIR $SOURCE_DIR
# selectively add the POM file
COPY pom.xml $SOURCE_DIR
# get all the downloads out of the way
RUN mvn dependency:resolve
# copy the source code
COPY . $SOURCE_DIR
# package the project
RUN mvn package
RUN cp target/artifact-1.0.0.jar $APPLICATION_DIR
# remove source
RUN rm -rf $SOURCE_DIR
# run the API
WORKDIR $APPLICATION_DIR
EXPOSE 8080
CMD java -jar artifact-1.0.0.jar
Dockerfile.alpine.production
is a Dockerfile using the Alpine Linux distribution. This one is very small (5MB) and has a package manager, and can be used if you need to install some tools you need in production.
#the base image
FROM openjdk:8u92-jre-alpine
WORKDIR /app
#copy compiled binary
COPY artifact-1.0.0.jar /app/artifact-1.0.0.jar
#run the Java process as a non-root user
USER daemon
CMD java -jar artifact-1.0.0.jar
Now we can run Habitus.
$ habitus --use-tls=false -host=unix:///var/run/docker.sock -secrets=false -keep-all
2016/09/07 14:37:52 ▶ Using 'xxx/cloud66_blogpost/build.yml' as build file
2016/09/07 14:37:52 ▶ Building 3 steps
2016/09/07 14:37:52 ▶ Step 0 - builder: builder/java
2016/09/07 14:37:52 ▶ Step 1 - production: example/java
...
Done. We've constructed and welded the smallest possible image.
Summary
Habitus will help you to create, construct and weld the perfect secure Java image. Docker images are the key ingredient of successfully running Docker in production. Always take care of what you deploy inside a container and always check the sources (base images) to make sure you remove the things you don't need.
With Habitus and the marriage with Buildgrid you can now build, deploy and manage your Java applications with ease on our Cloud 66 for Docker platform. And don't forget we also provide high-end security of your Docker cluster nodes using ActiveProtect and very strong firewall rules.
I'm not done advocating the need for the RIGHT production image. I'm going to speak about this subject during Dutch Container Day, Devops Pro Russia and All Day Devops