Verizon's wireline unit is developing its own machine learning internally using open source tools and 'agile' development methods.

October 2, 2018

7 Min Read
Verizon: Vendor AI Not Ready for Prime Time

For all the buzz around artificial intelligence, vendors have yet to produce practical approaches that live up to that hype, says a Verizon executive engaged with his own team developing internal approaches that ultimately will use AI.

Matt Tegerdine, director of network performance analytics for Verizon Communications Inc. (NYSE: VZ)'s wireline side, says there is no shortage of vendors willing to pitch AI products that they claim will dramatically cut network operations costs. But in his experience, most of those claims fall apart in the face of the size and complexity of the Verizon network and the amount of data it generates.

So instead, Tegerdine is leading an internal team that is gradually building machine learning systems to solve specific business problems, using mostly open source tools and their own internal development processes. By adopting McKinsey & Co. concepts of creating six-person units that combine a specific set of skills, including knowledge of network operations, Verizon is able to create its own practical approaches, the first of which is an effort to improve customer service and reduce truck rolls.

Figure 1: Verizon's Matt Tegerdine Verizon's Matt Tegerdine

Verizon has been on a five-year journey to build machine learning into operations, "almost like simple biology, where you start with a simple cell and then you create more cells on that working together and then you have different organisms that then work in concert into the point where you evolve into that artificial intelligence," Tegerdine explains.

That doesn't mean vendors aren't pitching him their products.

"I've talked to a lot of different vendors that come in, and the pitch is a bit of a fallacy," he says. "They say, 'Hey, we're going to come in. We're going to install artificial intelligence and machine learning. We've got 50 different patented algorithms, and within three weeks we're going to save you 75% of your cost.' I mean, these aren't exaggerated numbers. This was an actual pitch that I listened to at one point."

But given the size and complexity of the Verizon network and the amount of data, vendor product pitches "are simply not true," Tegerdine says. The hard piece isn't getting the AI right, it's understanding the network.

"Even with my team, that's one of the biggest challenges I have when I bring new people onto the team," he explains. "It's not really the data science or the data analytical tools or the statistical knowledge. That stuff is not easy, but it's easier to teach. It's an easier deficiency to make up for. The real hard part for me is understanding the Verizon network. Because you can sit there and look at data and let the models do their thing, and it spits out the insights. But if you don't understand the network, you really don't understand what you're potentially looking at."

How will service providers enable automated and efficient network operations to support NFV & SDN? Find the answers at Light Reading's Software-Defined Operations & the Autonomous Network event in London, November 7-8. Take advantage of this opportunity to learn from and network with industry experts – communications service providers get in free!

And that is a prime reason why Verizon is working internally to build up its AI capabilities, he says. "It doesn't mean it won't change in the future, but as it stands right now we don't use vendors, and we don't see a lot of value at this point."

Tegerdine says one area with a lot of promise today is in customer service, focusing on call center technicians and creating a model that looks at test results following a customer's complaint and makes a recommendation, particularly hoping to avoid a dispatch or truck roll, which is always costly.

"It's a very key place for us," he says. "There's a cost to the company but there's also a cost to the customer and their satisfaction at this point. If a problem can possibly be fixed remotely, the customer can have [satisfaction] immediately instead of waiting for dispatch and someone to show up at their house."

Next page: A layered approach to building out AI What Verizon's internally developed model is doing is learning to become a good technician, making recommendations and determining which are right or wrong. The goal is to continue adding layers onto the process to have the model look further upstream into the network and then, with the move to software-defined networking, to potentially make changes to the network, as needed, on an automated basis.

"If we have to adjust something or tweak something or change a routing, then this virtual technician that we're growing essentially can take over," Tegerdine says. "That's our path to get toward artificial intelligence. And we have a lot of models, we are doing a lot of prediction right now. There is going to be some methodology where we start tying these models together and start layering them and getting them to work together and then build artificial intelligence on top of that to pull in all the data."

Tegerdine is clear in saying it's very early days for true AI within Verizon.

Figure 2:

"And I would even go so far to say that what we have in Verizon is probably, it's a lot of stuff like chat bots and very primitive sort of customer-facing piece at this point," he says. "It doesn't mean that there's not value there, but let's be really clear on that definition. There's no great intelligence behind the scenes at this point that's reconfiguring, managing the network, making adjustments. That's not there."

Tegerdine's team is run like a startup inside Verizon, and he says its mindset is "more that West Coast Silicon Valley mindset inside the team where we're very innovative and fast, tackling problems from different directions." The concept was brought to Tegerdine's group from Verizon Wireless by [now chief network operations manager and acting CTO] Kyle Malady, but the wireline team had to figure out how to replicate what wireless was doing "with less people and less money, essentially," Tegerdine admits.

The teams he assembles combine the known characteristics of data scientists, as laid out by companies such as Price Waterhouse Cooper and implemented by Boston Consultant Group. So each team includes business or domain expertise as well as expertise in statistics, programming, database technology and visual art and design.

"You need to be pretty proficient in each one of those five pillars to be a really good data scientist and that person is very hard to find," Tegerdine says. Instead he adopted the BCG approach of hiring people with specific expertise and assembling that team, including those who understood network operations. That means combining recent college grads with folks who have decades of network experience but aren't afraid to try to learn how to code and other skills.

"When we tackle a problem it's with a six-person team," Tegerdine says.

A key member of that team is what he calls a data translator, who is typically brought in from the outside, with some data scientist's skills but also "the ability to speak to the business," and they act as a bridge between those with data skills and those mostly steeped in networks. Each team also has a business owner, the person from the operations side who understands the benefits of what's happening, and a product owner, as defined in "agile" processes, that ultimately represents the end user.

One of the processes still being developed is how problems that this group will tackle are identified. Initially, Tegerdine says, the operations folks pulled into his team had an ample supply of issues they could identify that could be tackled but as their time away from the day-to-day operations extends, that has faded. Ideally, problems will be brought forth by the business units, he says, but that is still a work-in-progress.

Among the things the team has worked on to date are predictive models for equipment failure -- identifying routers, etc., before they cause outages -- as well as self-diagnostics and healing logic inside home routers to prevent problems before consumers become aware of them.

"We also can look at long-term fiber degradation," he comments. "It allows us to get way ahead of getting fibers replaced, making sure that we look at the data over time and we see when it is out of spec, or when we can predict it to be our of spec and get that work done well in advance before anything significant could potentially happen."

— Carol Wilson, Editor-at-Large, Light Reading

Subscribe and receive the latest news from the industry.
Join 62,000+ members. Yes it's completely free.

You May Also Like