Actual broad deployments are still relatively modest for automation.

November 3, 2020

8 Min Read
Automation: The Best Roads Are the Curvy Ones

As part of my day job, I spend a fair amount of time obsessing about automation and specifically about why more people don't have more of their infrastructure automated. For a set of technologies that are near uniformly lauded, actual broad deployments are still relatively modest.

Figure 1:

Below are a couple of different ways we use to frame the automation journey—one view looks at technical scope and the other is more concerned with process and organization. In both cases, difficulty increases as you move to the right: you go from a single engineer running some scripts to make their life easier on the left to progressively more expansive automation on the right that benefits the entire organization. While the broader scope inevitably delivers greater benefits for the organization, the increase in people and processes touched also makes it that much harder. It's easy to get early automation wins, but the road quickly becomes curvy and, so, here are some thoughts on how to navigate those curves.

Figure 2:

Figure 3:

I see five areas where automation plans run the risk of running off the road—they are all a mix of people, process and tech issues, and they are all ultimately navigable. Over this post and the next, I'll discuss those areas and how to address them.

Trust

When folks think of automation, cautionary tales like the rise of SkyNet or The Sorcerer's Apprentice color their thinking. It's an understandable concern. While there is no substitute for training and operational rigor, the technology itself can help mitigate risk and provide more predictable outcomes. One such area is having a single source of truth. Imagine if different airlines used different definitions of “North” for navigation—it would certainly make flying even more of an adventure. The same is true when not everyone (humans and tools) agrees on how things are and what is supposed to be happening. Regardless of if you are using imperative or declarative automation tools, everyone needs to agree on which direction “up” is. Tools are starting to emerge which attempt to tackle this problem. For example, within Cisco Network Services Orchestrator (NSO) we have a configuration data base, creatively called CDB, that maintains state info for every device under NSO's control. Because CDB is ACID compliant and supports features like two-phase commit, it provides a reliable view of infrastructure which translates to more predictable and trusted automation. Another aspect of trust is knowing your tools are doing what you think they are doing. For example, NSO provides the ability to dry-run a config change to see exactly what config changes are being made, and in case of the inevitable “oops”, NSO can also rollback to a prior config. These types of capabilities go a long way to increasing trust and expanded use of tooling. We have done a fair bit of work on this topic including a white paper on single source of truth in partnership with Enterprise Management Associates and a podcast on the topic with the crew at PacketPushers.

Tools

A few years ago, you might have argued that there was a dearth of automation tools in the networking space. Today, that is a hard argument to make as there are now myriad automation tools and vendors making the products more amenable to programmatic control. Even the most ardent CLI-hugger is making furtive trips to DevNet. This bounty introduces a couple of new complications. The first is that now there are more options, and everyone has their favorite. This is not inherently bad as each tool has its strengths and is suited for certain purposes, but it becomes a complication when you need all these folks together. To illustrate the second complication, let me introduce you to the eierlegende Wollmilchsau, which translates to "egg laying wool milk sow," and is a wonderous animal that gives you eggs, milk, bacon and a soft comfy sweater to boot (I love the Lego-like approach to making new words for new concepts that German gives you). Some folks end up looking for the automation tool equivalent and this kind of quest either results in doing nothing while you wait for a do-it-all automation tool to come along or taking your favorite tool and forcing it to handle tasks it was never meant to handle.

How do you get past this? The first thing to do is recognize there is no magical pig in your future. I think NSO is an amazing tool, but I will be the first to tell you it is not going to address every need for every stakeholder. The reality is you will need to thoughtfully select a handful of tools to work in concert to build a flexible and scalable automation framework. The key role NSO plays in this strategy is its ability to serve as a bridge across tools, technologies and domains. Because of its multi-vendor cross-domain ability, NSO allows your domain teams to keep using their favorite tools and still be able to build integrated automation workflows. At the other end of the stack, NSO allows developers to use their preferred interfaces like Java, Python and RESTCONF or even other tools like Ansible. This way, you can allow flexibility and autonomy for both operations and developers, without giving up the ability to automate your infrastructure in a simple and holistic manner.

Processes

Not surprisingly, you need processes before you can automate them, which is often the first hurdle customers encounter: there is no agreed upon way to do things. How Lilly configures an L3 VPN might be different from how Pat does it, but a team needs to converge on the best way to do something and that can lead to some, um, passionate discussions. As you work to automate more of your operational and business processes, the same dynamic plays out repeatedly as you get more teams and more stakeholders involved, so it's important to set expectations with participants and stakeholders up front that the road ahead will be a bit bumpy.

There is often an urge to justify automation by showing a massive win on your first outing. Resist that urge. The worst thing you can do is try to automate an entire business process at one time. Instead, take a business workflow and break it into smaller steps and work at automating each step. Implement things, get quick wins to build confidence, and learn how to do it better next time. Over time, start stitching these smaller efforts together. This gives your team time to learn the tools, test the processes, and learn how to work together.

Fair warning: not every one of your process can be automated. The way your org accomplishes a task may not lend itself to automation and trying to force it to be automated will likely not end well, especially when you are starting out. However, bear in mind, a market peer that can automate something you are still doing manually will inevitably gain a competitive advantage over you. Long term it's worth striving on re-working your processes so that they are all automatable.

Org structure

Conway's Law states that "[a]ny organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." This is not great for product design and it's equally undesirable for workflow design. If teams only interact through their leaders, it not only slows down design and implementation of new processes, its fatal to fast and effective troubleshooting, problem resolution and improvement of those process once they are in production. We've also shown how tools like NSO can provide a technology solution to keep the peace, but it's also imperative that there are short, clear communication pathways between the folks that have operational responsibility.

Beyond the individual operational domains involved in a process or workflow, there also needs to be someone that has the mandate to provide governance and coordination across these various teams—think conductor for an orchestra. Again, this role is not only important when developing and implementing automation workflows, but also once they are in production, this person (or team) are often on the front line for user experience issues—someone needs to own the big picture.

People

This last one is for the leaders reading this: your people will continue to be a critical part of your automation strategy. With that in mind, make them part of the process—your automation projects feel like something done with them not to them. The folks that live with your infrastructure day-to-day know where the opportunities are, where the easy wins are, and where the landmines are buried. Just as important, train your teams, not just on the technologies, but also on the mindset. Concepts like DevOps, Continuous Integration/Continuous Delivery (CI/CD) and infrastructure-as-code are as much about mindset and culture as they are about a specific toolset.

Ironically, this is an area where contractor and professional services come in handy—they can help bridge your operational expertise bringing in best practices until your own team comes up to speed.

Figure 4:

In closing

Organization-scale automation is absolutely a navigable journey and it's most definitely worth making the trip. But, like anything worthwhile, it will take some work. I hope these two posts have given you some things to think about. In terms of other resources, check out our Network Automation Deployment Model (NADM). The NSO team has spent close to a decade automating some of the most demanding networks on the planet and the model was a way to share their experiences and best practices back with the community.

– Omar Sultan, Leader, NSO/ESC Product Management, Cisco Systems

This content is sponsored by Cisco.

Cisco Systems Inc. (Nasdaq: CSCO)

Subscribe and receive the latest news from the industry.
Join 62,000+ members. Yes it's completely free.

You May Also Like