Imagine this, You are working as IT operations in a large organization. You are the one who holds the keys to all kinds of resources that other people would like to use. So these people come to you with questions and requests which you handle with care to give them the best results possible. After a while the services you provide become more and more popular. The result, a queue forms at your desk every day, your mailbox explodes and people are getting mad they have to wait for a long time until you are able to help them.
What would you do?
Birth of Shadow IT
Handing everyone the keys to the castle seems the simplest option that most developers would probably choose. If they have to wait really long or if you make your process really complicated or frustrating this will lead to shadow IT where people will just create their own castle.
Having your own castle as a team might be nice and fun in the beginning but castles needs maintenance and you need to make sure that security is in place yourself. When more and more people or teams start using your castle you’ll have to make sure certain rules are in place and often someone will take the role of being the dedicated operations person for the new castle. All queries and problems are sent to this person who holds the key to the castle and we’re back where we started.
A frustration a lot of IT operations people have is that they get a lot of questions or problems that miss a lot of information to be able to handle the request. (for the developer readers: sounds familiar?) This leads to ticketing systems or forms that people need to fill in to do their requests. Although this makes it easier for the operations to handle the request it moves the problem to the person who does the request who might not know all the things needed to get things done. He’s only concerned in getting the end result. Let’s take a look at an example: Requesting a firewall change.
A development team is building an application that has to communicate with another system on a certain port. This port is blocked by the corporate firewall so they ask operations to open the port.
What would you do if you were the firewall admin?
In 99% of all enterprise organizations IT operations is not the deciding authority of which ports should be opened or should stay closed. So in their ticketing system they add something where you need to prove you have the seal of approval by the security authority that opening this port is not a security risk and it is really necessary.
When you think about this what is your actual role here? In the end you’ll just be the person opening the port as soon as a valid request comes in and that shouldn’t be the case.
The DevOps approach
But what would actually work? Handing over the keys is obviously not an option but having dedicated people to perform certain tasks is also a non-sustainable solution. The only reasonable thing to do is create Self-Service automation. Let people perform the tasks they need but let them do it themselves. Not by giving them the keys, but by giving them a prefab entrance to the castle that can be tracked, traced and audited without having to bother anyone.
Does this mean we don’t need IT Operations? Certainly not, we definitely need people with IT operations skills, only the Devs won’t need them. They become …Invisible.
Self-Service is something that many people do not think of in the beginning. It seems scary because you are “out of control”. But the reality is that you actually have a lot more control. If you have an automated process that always runs in the same manner, the enables people to do things themselves in one consistent way, it is much better than following a manual process and keeping people in line (literally) to follow these processes.
And just like many other things, moving to Self-Service is mainly a mindset. To get started use this step-by step approach
- Create awareness about Self-Service.
- IT operations needs to start automating their processes
- Developers and IT operations can work together in building a solution that works for both of them.
- Automate everything
- At first this will be a bunch of scripts that are still run by IT operations. But this will speed up the process and they will get more time to do more automation
- Create a Self-Service UI/API
- Glue the scripts together and let people use the functionality themselves. Either using an UI or an API.
- Move from Control to Report
- Create reports and alerts to monitor the changes and alert if strange things happen
Choosing the right tool for the job
Automation can help us improve the speed and quality of how we handle common requests. What kind of automation would you prefer?
- Integration in a ticketing system
- Web forms with questions + scripts that execute on them
- Email system that handles mails with scripts
- Chat bot
- Solution “as code” using GIT to automate your workflow
- Other
When we try to automate processes a common problem is that we take the current process and switch out the manual steps to an automated counterpart. This will often result in an suboptimal process that mimics the old process which was made from the operations point of view. In the digital revolution that is currently happening in companies is that we focus on client value, improving processes so they give our customers the best user experience. When automating these operations processes we should also pick the right tool that fits our customers. Are they business users? Then maybe a chat bot or web form works great. Are they developers? Then trying to automate the process using the systems they use like git works best. It all depends on your customers.
A real world example
One of the things that we currently see a lot is the management of Azure. In many organizations there is still A central authority that manages the Azure subscriptions. In some cases, teams have their own subscription where they can do everything they want, but the more they move towards production the less permissions they (should) have. The setup of many organizations is like the picture below. Teams are Contributor or Reader meaning that a lot of security related actions cannot be done by themselves.
But how do you manage all these resources? And how can IT-Operations avoid being the Single Point of Failure Contact to set all security related permissions.
In the spirit of the article we have created a library that enables DevOps teams to handle these request in a Self-Service manner. We have called the library Azure Security As Code (Asac) and it can be found on GitHub.
This library allows you to store the security settings in a YAML file. This can then be maintained in a Git Repository. By means of a pull request, developers can update their own security settings. This will then automatically have a review attached. Then the library allows you to apply the new settings to Azure again. All done automatically, a ll done by 1 service account with sufficient rights. Basically, the prefab entrance to the castle.
Conclusion
DevOps is more than teaming up Developers and IT-Operations. And it is also more than just automating everything. DevOps is also a mindset change. One of the most important changes that teams should adopt is move away from the silos. Focus on enabling others instead of optimizing your own process. By implementing Self-Service for teams, the role of IT-Operations becomes a different one. One that is more focused on building the right solution and approach so that others can succeed. Time that is spent on solving Developer permission issues, can now be spent on streamlining the process. And the developers, they will feel empowered and will most certainly be very willing to help the IT-Operation to write the automation. And that is what DevOps is all about!
This article is written by Rene van Osnabrugge & Geert van der Cruijsen