The third pillar of an Engineering Culture
An Engineering Culture consists of eight pillars. The third pillar is Smooth Delivery. When developing an application, you want to get that into the hands of your users. That is the ultimate place where your changes deliver business value. You want to do that as quickly, safely, and reliably as possible. Delivering the application can be as simple as copying a file from one location to another. It can also be a lot more complex when, for example, the delivery contains multiple containerized microservices.
When developing an application, you want to get that into the hands of your users. That is the ultimate place where your changes deliver business value. You want to do that as quickly, safely, and reliably as possible. Delivering the application can be as simple as copying a file from one location to another. It can also be a lot more complex when, for example, the delivery contains multiple containerized microservices.
You can use tools like Azure DevOps pipelines or GitHub workflows to build these deployment pipelines. These tools support writing the pipelines as code. Even the pipelines can be versioned, reviewed, and tested with all added benefits. The pipeline itself needs to be secure and compliant. People, for example, should not be able to remove a pipeline or be able to remove any of its runs. That both would impact traceability, etc. Xpirit colleagues have written a tool that can be used to make and keep Azure DevOps pipelines compliant and secure.
Any delivery process starts with building a new version, no matter what you deliver. Smooth delivery also begins here, in your build stage of the pipeline. When delivering software, this is where you compile your software and run tests against it. Smooth delivery isn’t just about all-green unit tests. It, for example, also includes checking your code for vulnerabilities and code smells. Various tools like Blackduck, WhiteSource, SonarCloud, or Snyk can help you find vulnerabilities and code smells.
These tools are known as Dynamic Application Security Testing (DAST) or Static Application Security Testing (SAST). We try to run various tests and validations as early as in the pipeline. We call that approach ‘shift left’ because it emphasizes the left side of a typical pipeline. That way, we get the feedback as soon as possible on the increment we plan to put in the hands of our users.
After the build stage, the following stages in the pipeline are used to get further confidence in the change made. A stage could push the software to a test environment to further be explored. Another stage could be used to run a nightly performance test of the changes made that day. The final stage is used to push the change into production. Exposing the changes to users can be made more resilient and reliable by using various release strategies. Examples are the rolling deployment, Blue-Green deployment, or Canary deployment.
Infrastructure as Code
Delivery pipelines aren’t there just to deploy your application. The software runs on infrastructure, whether that is in the cloud or on-premises. The benefits that we get from automated pipelines that deliver software, like traceability, repeatability, and reliability, can also be achieved for infrastructure. This is where Infrastructure as Code (IaC) comes in. Using IaC, you can deploy your complete infrastructure the same way you deploy your software.
Writing Infrastructure as Code can be one using various tools, each having its pros and cons. You can use a scripting language like PowerShell or Bash and use the Azure CLI, for example. Often a more declarative approach is preferred. That is where tools like Bicep or Terraform come in. There are even options to write infrastructure in C# using Pulumi. No matter what tool you use, you will get the advantages that IaC brings.
As with delivering software, security is a big item in delivering infrastructure. You can again use tools to discover potential security issues early in the pipeline. For Terraform, there is TFSec. For Bicep, you can use the ARM Template Toolkit. Both these tools will give a warning when you, for example, try to open a vulnerable connection to the internet. Running these tools is not enough. You will also need to ensure that no one can change the infrastructure, at least in production.
The infrastructure as Code in source control should always be leading. This is often done by splitting the production environment from other environments. You then make sure that only the deployment tool, e.g. Azure DevOps, can make changes in production. Anyone else gets read permissions at most.
If you have all of this in place, you are doing excellent. If you want, you can take it to the next level. How about separating the release from the deployment? That needs a little explanation. Imagine the situation in which you’ve been building a feature, and it’s ready to be released. However, another department of the organization is not quite ready for it.
Wouldn’t it be great if you could just hide this particular feature and still be able to push other features and bug fixes to production? This is where Feature Toggles come in. Basic Feature Toggles are nothing more than an “if-statement” in your code. If the feature toggle is “on” you can use the feature, if the toggle is “off” you cannot use the feature, and its hidden from the users.
Have you ever heard about “Testing on Production”? It’s a release strategy in which you take the concept of the feature toggle and make it more advanced. Instead of enabling or disabling a feature, what if you could expose a new feature to only fractions of live production traffic. You could release a feature when it’s done for 70% and ask the users for feedback before you finalize it, giving them a better product.
Finally, you could introduce what is called Chaos Engineering. It’s about how your application behaves during an incident or unexpected behavior. Does it recover, or will it crash and burn? You will make a hypothesis on how your application will react to, for example, an outage. Next, you will create that chaos to see if you are correct. It will help you increase the reliability and quality of your application.
Let's talk Engineering Culture