Honey I Shrunk The Container

(or how I found out about microcontainers and why I think you should use them)

Context

About 18 months ago I was asked to help an ISV partner who wanted to migrate their product to cloud.

At the time they had a very traditional software industry business model - sell the customer a license to run their software, along with a subscription for ongoing support.

However they wanted to be able to move to a SaaS model where the customer would consume their software as a service running in the cloud.
Their stack consisted of:

• a Ruby on Rails application
• Postgres
• Redis

Their initial attempt at creating a SaaS offering had been to run this as a set of VMs on top of IaaS.
While the software ran just fine, they soon saw that this approach wasn't going to give them a successful SaaS business.

If any part of the application were experiencing high demand, the only way to scale it was to start another instance of the whole application.

Since they were packaging the application as a VM, this meant that they would then be spinning up a whole new VM, just to scale out the one function.

Scaling at the VM level was not only slow, it was also inefficient and meant that their IaaS costs were too high for the business model to be higher.

Clearly they needed to find a way to do more responsive, fine - grained scaling.

Splitting the Monolith

To achieve fine grained scaling, we decided to split the application into individual microservices so that each could be scaled out independently.

To improve startup time and make more efficient use of the IaaS platform we decided to run each microservice instance in it's own Docker container.

Once these changes had been made, and we had tested the new implementation the partner was confident that they could go to market with an offer that was attractive to customers in terms of both

• Service Level
• Price

The Challenge

A short while later the partner got back in contact with me. They had indeed got a customer interested...

...but...

...the customer's security department had some questions that needed to be answered before they would allow the business to use the partners SaaS service, and the partner wanted my help to answer these.

Some of these questions were "the usual suspects" - where are your data centres, how secure is the network etc. and for these there are standard answers.

However there were some other questions about the container based deployment:

• How will you secure the containers?
• How will you manage vulnerabilities that may occur in the software or the container image?
• How will you patch the containers?
• How will you build the images?

All good questions, but ones I didn't have a standard answer to.

The problem is rather neatly summed up by one of the Google Developer Advocates:

“Docker makes building containers a breeze. Just put a standard Dockerfile into your folder, run the docker ‘build’ command, and shazam! Your container image is built! The downside of this simplicity is that it’s easy to build huge containers full of things you don’t need—including potential security holes.”

- Sandeep Dinesh, Google Developer Advocate

So I decided to reach out to Oracle's cloud operations (CloudOps) team to see if they had come across these challenges and if they had any advice.

The answer that I got back was "microcontainers".

Microcontainers

So what is a microcontainer? It's a container that includes only a single executable and it's dependencies.

That helps in two ways:

1. Reduced attack surface - by stripping the image down to the minimum required to run the executable, we reduce the opportunities for attack. And if an attacker does manage to get access to the running container, they'll find a lot less tools in the container for them to misuse.
2. Certainty over vulnerabilities - if your image includes only the components your executable requires to run, you know that if a vulnerability is found in one of those components, you absolutely have to fix it. On the other hand, if it isn't required by your executable, it won't be included in the image and you don't have to worry.

As well as the security benefits, microcontainers are, well, smaller - which makes them easier and quicker to move around.

We made this case to the customer's security department, who accepted that the microcontainer approach addressed their concerns. I'm pleased to say that they've been using the partners SaaS offering, hosted on Oracle's cloud, for a few months now.

So the next question is how to create microcontainers?

There are a few ways to do it.  The original way was to use the builder pattern.  While it works, it is a bit clunky and so Docker added support for multi stage builds from version 17.05.

For a detailed discussion of the builder pattern and multi - stage builds, see here.

The problem with both the builder pattern and multi - stage builds is that you're responsible for doing your own dependency management.  While that might not be an issue if your build produces a statically linked binary, it can definitely cause challenges with a dynamic language such as Ruby.

To help with this, the Oracle CloudOps team have created a tool for producing microcontainer images.  It's called Smith.

Smith

Smith is an open - source microcontainer builder.

You can find/download/fork it on github here.

It has two modes:

• Building microcontainer images from "scratch" (Yum repositories and RPM files).  This is the way it is most commonly used internally at Oracle.
• Shrinking an existing image.  This is the way I've used it most.

In either case, we create a smith.yaml file, which defines:

• the type of the image source(s)
• the location of the image source(s)
• the paths to include in the image
• the command to run

We can also create a subdirectory rootfs - the contents of which will be placed in the root directory of the image.

Smith produces images in OCI format.  Since Docker doesn't run an OCI image (yet) you'll need to push this image to a Docker repository before running it.

Build from "scratch"

If we have a file rootfs/read/data:

Then the smith.yaml to create a "Hello World!" microcontainer is as follows:

Smith will examine /usr/bin/cat/ for it's dependencies and then recursively explore the dependency tree before producing a new image with a single layer, containing everything you need (and nothing you don't) to run cat against /read/data and print Hello World!

Shrinking an existing image

In the case of shrinking an existing image, we can either download the image (in OCI format) and point to specify it's path, or point directly to it's URL in the smith.yaml file:

This approach shrank my image from 934MB down to 84MB with all dependencies in place. I gave up with the multi stage build and manual dependency resolution when the image size was 120MB and it still wasn't working!

Exercises

For a set of step - by - step exercises, see here.

Summary

However you choose to build them, I hope you can see the advantages of microcontainers in terms of security and agility.

And if you're interested, please get involved with Smith, either on github or via Slack.