During the past year I supported several clients in their journey toward Containerized Delivery on the Microsoft stack. In this article I’d like to share eight practices I learned while practicing Containerized Delivery on the Microsoft stack using Docker, both in a Greenfield and in a Brownfield situation.
PRACTICE 1: Small, reusable image layers
Once you start containerizing.NET workloads, you need to decide how to modularize your container images. A good starting point is to review your architecture and determine which parts of your application landscape you need to selectively scale or release independently. In fact, each container image you create should be self-contained and must be able to run on its own. There is also another important aspect you have to think about: container image layering.
As you may or may not know, container images are the blueprint for your containers. Images consist of image layers. Each image layer is created during the Docker build process as the resulting artifact of a set of instructions (e.g., creating a directory, enabling Windows features) specified within the Docker file. This process is shown in Figure 1 – Docker Image build process.
Figure 1 – Docker Image build process
The nice thing about Docker is that this image-layering principle is reused to optimize the performance and speed of Docker. Once Docker notices that a given layer is already available within the image layer cache on your local machine, it will not download, rebuild or add this layer again. For example, if you have two ASP.NET container images – one for Website 1 and one for Website 2 – Docker will reuse the ASP.NET, IIS and OS layers both at container runtime and in the container image cache. This is shown in Figure 2 – How image layering works for containers as well as for container images.
Figure 2 – How image layering works for containers as well as for container images
If you implement your container image layers in a smart way, you’ll see an increase in the performance of your containerized workload and the speed of their delivery. Moreover, you’ll see a decrease in the amount of storage your container images require.
The following practices are related to container image layers:
• Sequence of layers: Try to order and structure the layering of your container images in such a way that you reuse layers as much as possible. Figure 3 shows how I achieved this for one of my clients by creating different images.
Figure 3 – Real-world example of reusing image layers for different images
• Combining actions in a single instruction: try to combine multiple actions (e.g., by enabling a Windows feature, creating a filesystem directory, etc.) in a single Docker instruction as much as possible. By default, Docker will create a separate container image layer from each individual Docker file instruction. If you don’t need a separate image layer for later use, combine multiple actions in a single instruction line to avoid overhead in the storage of image layers.
For example instead of:
SHELL [“powershell”, “-Command”, “$ErrorActionPreference = ‘Stop’;”]
RUN Add-WindowsFeature NET-Framework-45-Core
RUN Add-WindowsFeature NET-Framework-45-ASPNET
combine both Add-WindowsFeature actions into a single instruction:
SHELL [“powershell”, “-Command”, “$ErrorActionPreference = ‘Stop’;”]
RUN Add-WindowsFeature NET-Framework-45-Core,NET-Framework-45-ASPNET
Another example of running multiple PowerShell commands in one instruction is:
RUN Invoke-WebRequest “https://aka.ms/InstallAzureCliWindows” -OutFile az.msi -UseBasicParsing; `
Start-Process msiexec.exe -ArgumentList ‘/i’, ‘C:\az.msi’, ‘/quiet’, ‘/norestart’ -NoNewWindow -Wait; `
Remove-Item az.msi; `
$env:PATH = $env:AZ_PATH + $env:PATH; `
[Environment]::SetEnvironmentVariable(‘PATH’, $env:PATH, [EnvironmentVariableTarget]::Machine)
PRACTICE 2: Multi-stage builds
New since Docker 17.05 is the concept of multi-stage builds. As described in the previous practice, container images can be rather large because of a large number of image layers. Multi-stage builds help you to reduce the size of your images in an easy way. Instead of an image that contains all in-between image-layers, the multi-stage build implementation makes it possible to copy resulting image content of a given image into another image you define.
Another great benefit of the multi-stage build is that it minimizes your attack surface by removing all in-between layers and installations. And last but not least, multi-stage builds enable easy creation of debug and testing images from your production images by putting the resulting content of your production image into a default debugging and testing image.
Before Docker 17.05 this could also be achieved by making use of the Docker builder-pattern(1) . In the new multi-stage solution we can just add a second FROM statement to our existing Docker file and instruct that image to copy the content of the earlier stage into the image of this second stage by making use of the –from argument within the COPY instruction. As shown in the following example, the content of the first image definition (called temp) is copied into the second image definition (called final). As a result, the image size of the final image is 315 MB smaller than the original temp image! Looking at the number of image layers, we see a reduction of 15 out of 20 image layers(2)!
## First Stage – Install WebDeploy, create disk structure, deploy MVCMusicStore website
FROM microsoft/aspnet AS temp
RUN mkdir c:\install c:\MvcMusicstore
ADD WebDeploy_2_10_amd64_en-US.msi /install/WebDeploy_2_10_amd64_en-US.msi
WORKDIR /install
RUN msiexec.exe /i c:\install\WebDeploy_2_10_amd64_en-US.msi /qn
WORKDIR /MvcMusicStore
ADD fixAcls.ps1 /MvcMusicStore/fixAcls.ps1
ADD MvcMusicstore.zip /MvcMusicStore/MvcMusicStore.zip
ADD MvcMusicStore.deploy.cmd /MvcMusicStore/MvcMusicStore.deploy.cmd
ADD MvcMusicStore.SetParameters.xml /MvcMusicStore/MvcMusicStore.SetParameters.xml
RUN powershell.exe -executionpolicy bypass .\fixAcls.ps1
RUN MvcMusicStore.deploy.cmd, /Y
## Second Stage – Creates final image
FROM microsoft/aspnet AS final
COPY –from=temp c:/inetpub/wwwroot c:/inetpub/wwwroot
EXPOSE 80
[1] https://blog.alexellis.io/3-steps-to-msbuild-with-docker/
[2] http://fluentbytes.com/optimizing-your-windows-docker-images-with-multi-staged-docker-builds/
PRACTICE 3: Keep your Windows Containers up-to-date
If you work with or start working with Windows containers you may wonder how to implement an update strategy to deal with Windows updates. Because containers are meant to be stateless and immutable, you can’t run Windows Update within your container. So what is the solution?
In contrast to Linux container images where people can create a container image from scratch, each Windows container image you create should be based on one of the base images (currently nanoserver(3) or windowsservercore(4) ) that were created by Microsoft. Similar to Windows Updates, those base images are updated by Microsoft on a regular basis in order to roll out the latest security and bug fixes. This is one of the reasons why you should always use one of Microsoft’s base images instead of creating your own. As part of the containerized delivery way of- working, you have to put a process in place to ensure that you update your image references to the latest Windows base images on a regular basis.
I recommend my clients to use the docker pull command to automatically pull the latest Microsoft base images each time they have to build an image which is directly dependent on it. This ensures that you always build those container images using the latest OS base images and including all security and bug fixes. For all other container images that depend on those “internal base images”, you have to put a process in place to ensure that they always use one of the most recent internal base images.
Most of the times when I discuss this approach, I get the question: “So we have to download that 10 Gigs of base image each time for each update?” Luckily this is not the case. When you execute the docker inspect command against the base images, you will see that they consist of two layers: one big base layer that will be used for a longer period of time and another smaller update layer that contains small patches and that is updated constantly by Microsoft. So updating to a newer Windows base image version is not painful as it is only pulling the latest update layer from Docker Hub.
PRACTICE 4: Group Managed Service Accounts
A lot of existing .NET applications make use of Domain Accounts for authentication, e.g., connecting with SQL Server via Windows Integrated Authentication. The fact that Windows containers cannot be domain-joined may surprise you, but this is by design. Joining a computer to the AD is something that we do for long-term registration. Containers come and go very frequently on a machine and this is not what AD is designed for. Consequently, this could lead to registration issues. Luckily the Windows server team already had a solution in-place to make it possible to make use of Domain Accounts for external authentication from within a container. This solution is called Group Managed Service Accounts (gMSA).
[3] https://hub.docker.com/r/microsoft/nanoserver/
[4] https://hub.docker.com/r/microsoft/windowsservercore/
The idea behind this solution is to create a so-called gMSA within Active Directory where you also specify which container hosts have access to this account. This can be a list of container hosts or a security group containing all container hosts that should be able to run a container using this account. The next step is to prepare(5) the container hosts and test whether they can access the gMSA. After that, you have to install and make use of the CredentialSpec PowerShell module to store the required gMSA details in a correct (JSON credential spec) format on your container hosts.
Figure 4- gMSA container authentication flow
Once the required gMSA details have been stored, you can make use of the stored gMSA credentials spec to authenticate external services under the given gMSA from within your container. The only thing you have to ensure within your container is that your services and other processes that need to access the external resource run as ‘Local System’ or ‘Network Service’. When you start your container on the container host you can execute a docker run command with the –security-opt “credentialspec=file://WebApplication1.json” argument in order to pass the appropriate credential spec on to your container. At the same time, the applications within your container can access the external resource. For an extensive walk-through, please refer to http://bit.ly/2tWGloy.
PRACTICE 5: Secure Containerized Delivery
Securing your container infrastructure and deployments is an important aspect of Containerized Delivery. There are a lot of aspects(6) to keep in mind here, so I will highlight the most important ones.
• Harden your images, containers, daemons and hosts
When you set up your containerized infrastructure, it is important that you harden your infrastructure elements against threats. To help you with this, the Center for Internet Security has published a Docker benchmark that includes configuration and hardening guidelines for containers, images, and hosts. Have a look at this benchmark at https://www.cisecurity.org/benchmark/docker/.
[5] https://github.com/Microsoft/Virtualization-Documentation/tree/live/windows-server-container-tools/ServiceAccounts
[6] https://www.twistlock.com/resources/secops-guide-container-security/
Based on this benchmark, there’s a Linux container available that checks for dozens of common best-practices around deploying Docker containers in production in a scripted way. Unfortunately, this implementation is not available for Windows hosts right now, but if you make use of Linux container hosts for your ASP.NET core applications, you should definitely check this implementation at https://github.com/docker/docker-bench-security.
One important aspect of hardening your container hosts is protecting your Docker daemon with TLS. A great, fast and simple way to achieve this is to use Stefan Scherer’s dockertls-windows container(7) . This container generates all TLS certificates you’ll need to access the secured container daemon. Save the .pem files in a central, secure location so that you can use the content of those files once you want to access the secured Docker daemon. If you make use of VSTS for CI/CD, you can store the contents of the various .pem files directly in the Docker host Service endpoint.
• Know the origin and content of your images
As mentioned in practice 3, there are two Microsoft base images that all Windows container images should derive from. However, there are also a lot of other public container images available on DockerHub, such as microsoft/iis, microsoft/powershell and even images from other publishers. Using those out-of-the-box images accelerates the development of your systems. However, making use of public image definitions can expose your production landscape to high risk. For example, how can you make sure that those images do not contain any vulnerabilities? How do you ensure that the owner of the image will maintain the image definition over time in case of vulnerabilities and exploits that are discovered? It is important to know the origin and content of the images you consume.
Luckily there are many tools available to help you fill this gap. For example, you can make use of Docker Notary to check the authenticity of images, or Docker Security Scan to scan for any vulnerabilities within your image. You can also use other solutions such as Aqua(8) and Twistlock(9) . Whatever tool you use, make sure that you put a process in place that forces you to only use scanned public images and trusted origins.
For existing internal images, it is important that you perform regular checks and actively maintain those images with regard to vulnerabilities and exploits. For new internal images it is important that you reduce the attack surface as much as possible. Many of the out-of-the-box images from DockerHub, such as the microsoft/iis and microsoft/aspnet enable too many features for your workload. At one of my clients, this was the reason that we decided to create our own internal IIS base image with only those Windows features and services enabled that we really needed. For example, the default IIS image enables all Web-Server sub-features. The image we created enables only some of the sub-features, e.g., Web-Static-Content, Web-Http-Logging, Web-Stat-Compression, and Web-Dyn-Compression. By creating our own internal IIS image we made our workload more secure and achieved better performance. To find out which Windows features you really need, take a look at the PowerShell Get-WindowsOptionalFeature –Online and Get-WindowsFeature commandlets.
[7] https://stefanscherer.github.io/protecting-a-windows-2016-docker-engine-with-tls/
[8] https://www.aquasec.com/
[9] https://www.twistlock.com
PRACTICE 6: Dealing with secrets
Before the container era, we used to put our secrets (i.e.,credentials, certificates, connection strings) at a given location in the file system, alongside our application files. Within a containerized world, there is a problem with this approach. Because container images contain both the application as registry, environment variables and other file system content, this approach would mean that each team member can see the secrets by spinning up a new container based on this image. Dealing with secrets of containerized applications therefore means that you need to specify your secrets on container initialization and store them outside your container images. But how can you achieve this?
Before we look at the solution for dealing with secrets, you have to know exactly what you need to declare as secrets. Many values in configuration files are not secrets, e.g., endpoints, whereas passwords and SSL certificates are definitely secrets. It is important to be aware of this separation between secrets and configuration settings because in an ideal world you will manage both in a different way.
Looking at configuration settings, there are several ways to manage them. The option I like most is to make use of a configuration container in which all configuration settings (e.g., endpoints) are stored. At the time of container initialization, you can make use of this container to get the right endpoints for your application, for instance a service bus topic endpoint or an external SMS endpoint. The nice thing about a configuration container is that you don’t have to change the content of all other containers to deal with configurations over multiple environments like Dev, Test and Production. By making use of Docker Compose you can define this configuration store as a separate service and use its Docker DNS name to get the latest configuration settings from that store.
Until the beginning of this year, the most frequently used solution for dealing with secrets was either to make use of volume mappings or to make use of environment variables that contain the actual secrets. However, neither of these options was very secure. In the case of environment variables, your secrets are accessible by any process in the container, preserved in intermediate layers of an image, visible in docker inspect, and shared with any container linked to the container. In the case of volume mappings, the disadvantage is that you are making your containers dependent on th
e content of a data volume and this means that this container becomes unnecessarily stateful instead of stateless.
Luckily, since the beginning of this year, the most applicable option is to make use of the secrets management solution of the different cluster implementations, e.g., Kubernetes Secrets or Docker Secrets (Docker 17.05 for Windows containers). The nice thing about secret management at the cluster level is that secrets are automatically distributed across the container hosts. Another benefit is that the same secret name can be used across multiple clusters. If you have a separate Development, Test and Acceptance cluster, you can reuse the secret name, and your containers only need to know the name of the secret in order to function in all three environments. Creating those secrets in your container cluster environment can be orchestrated by the tools you are using for your delivery pipeline, e.g.,VSTS.
PRACTICE 7: Explicit dependency management
A lot of Docker and Compose files I have seen contain a reference to images, without specifying the version of the image, or with the default “latest” tag. Not explicitly specifying the version of an image can result in unexpected behavior and this is very risky.
Figure 5 – Docker file without version tag
The reason for this is that every Docker host has its own local Docker image cache. Once an image-tag combination is found in your local cache, Docker will consume the cached blob on docker run instead of downloading the blob from the container registry. Making use of the “latest” tag could result in the fact that you get an older version of an image than the image you actually want. As shown in Figure 6 a docker run cornellk/test command on Host01 will run version 10.0.01 instead of the download of image version 10.0.02 from the registry.
Figure 6 – The latest” tag issue
You can test this scenario yourself by comparing the result of a docker run and docker pull command on an outdated image in the local cache. As you see in Figure 7, docker pull downloads newer image layers from the registry where docker run just starts a new container based on the outdated cached image.
Figure 7 – Docker run vs docker pull in case of implicit versioning
Using the “latest” tag makes your build-and-release process non-repeatable. To avoid this, all you need to do is explicitly reference which version of an image you want.
PRACTICE 8: Environment-as-code pipeline and individual pipelines
Containerized Delivery means that you treat your application stack as cattle instead of pets(10) . This means that you deal with the immutability and stateless characteristics of containers. The nice thing about those characteristics is that they make your containerized application stack reproducible and scalable. Instead of upgrading, you replace your application stack.
You can leverage from the above characteristics by extending your existing set of individual delivery pipelines with a combined pipeline that can deploy your entire application stack to production. You’ll still have a separate commit (build) stage for each individual application, but the other delivery pipeline stages (e.g., automated acceptance test and user acceptance test stages) are combined stages within this pipeline. At the end of the pipeline you deploy your entire application stack to production. As a result, you can choose wether you want to deploy your entire application stack at once, or whether you just want to deploy the newest version of a separate application.
Figure 8 – Delivery pipelines in a containerized world
[10] https://www.theregister.co.uk/2013/03/18/servers_pets_or_cattle_cern/