Automation: The Best Roads Are the Curvy Ones
As part of my day job, I spend a fair amount of time obsessing about automation and specifically about why more people don't have more of their infrastructure automated. For a set of technologies that are near uniformly lauded, actual broad deployments are still relatively modest.
Below are a couple of different ways we use to frame the automation journey—one view looks at technical scope and the other is more concerned with process and organization. In both cases, difficulty increases as you move to the right: you go from a single engineer running some scripts to make their life easier on the left to progressively more expansive automation on the right that benefits the entire organization. While the broader scope inevitably delivers greater benefits for the organization, the increase in people and processes touched also makes it that much harder. It's easy to get early automation wins, but the road quickly becomes curvy and, so, here are some thoughts on how to navigate those curves.
I see five areas where automation plans run the risk of running off the road—they are all a mix of people, process and tech issues, and they are all ultimately navigable. Over this post and the next, I'll discuss those areas and how to address them.
Trust
When folks think of automation, cautionary tales like the rise of SkyNet or The Sorcerer's Apprentice color their thinking. It's an understandable concern. While there is no substitute for training and operational rigor, the technology itself can help mitigate risk and provide more predictable outcomes. One such area is having a single source of truth. Imagine if different airlines used different definitions of “North” for navigation—it would certainly make flying even more of an adventure. The same is true when not everyone (humans and tools) agrees on how things are and what is supposed to be happening. Regardless of if you are using imperative or declarative automation tools, everyone needs to agree on which direction “up” is. Tools are starting to emerge which attempt to tackle this problem. For example, within Cisco Network Services Orchestrator (NSO) we have a configuration data base, creatively called CDB, that maintains state info for every device under NSO's control. Because CDB is ACID compliant and supports features like two-phase commit, it provides a reliable view of infrastructure which translates to more predictable and trusted automation. Another aspect of trust is knowing your tools are doing what you think they are doing. For example, NSO provides the ability to dry-run a config change to see exactly what config changes are being made, and in case of the inevitable “oops”, NSO can also rollback to a prior config. These types of capabilities go a long way to increasing trust and expanded use of tooling. We have done a fair bit of work on this topic including a white paper on single source of truth in partnership with Enterprise Management Associates and a podcast on the topic with the crew at PacketPushers.
Tools
A few years ago, you might have argued that there was a dearth of automation tools in the networking space. Today, that is a hard argument to make as there are now myriad automation tools and vendors making the products more amenable to programmatic control. Even the most ardent CLI-hugger is making furtive trips to DevNet. This bounty introduces a couple of new complications. The first is that now there are more options, and everyone has their favorite. This is not inherently bad as each tool has its strengths and is suited for certain purposes, but it becomes a complication when you need all these folks together. To illustrate the second complication, let me introduce you to the eierlegende Wollmilchsau, which translates to "egg laying wool milk sow," and is a wonderous animal that gives you eggs, milk, bacon and a soft comfy sweater to boot (I love the Lego-like approach to making new words for new concepts that German gives you). Some folks end up looking for the automation tool equivalent and this kind of quest either results in doing nothing while you wait for a do-it-all automation tool to come along or taking your favorite tool and forcing it to handle tasks it was never meant to handle.
How do you get past this? The first thing to do is recognize there is no magical pig in your future. I think NSO is an amazing tool, but I will be the first to tell you it is not going to address every need for every stakeholder. The reality is you will need to thoughtfully select a handful of tools to work in concert to build a flexible and scalable automation framework. The key role NSO plays in this strategy is its ability to serve as a bridge across tools, technologies and domains. Because of its multi-vendor cross-domain ability, NSO allows your domain teams to keep using their favorite tools and still be able to build integrated automation workflows. At the other end of the stack, NSO allows developers to use their preferred interfaces like Java, Python and RESTCONF or even other tools like Ansible. This way, you can allow flexibility and autonomy for both operations and developers, without giving up the ability to automate your infrastructure in a simple and holistic manner.
