TalkTalk Puts Network & IT Convergence to Work
James Crawshaw, Senior Analyst – Service Provider IT and Automation, Heavy Reading
We recently met with Colin Whitbread, TalkTalk's chief operations officer for technology, to learn more about the company's strategy for network modernization and automation.
Whitbread leads a team responsible for the operator's public (i.e. customer-serving) network and associated services and operations. With a role that spans both the network and associated IT elements of operations, Whitbread has a strong interest in the convergence of these traditionally separate domains.
TalkTalk provides broadband, telephony, pay-TV and mobile services to around 4 million customers in the UK. In the consumer broadband market TalkTalk has the fourth-largest customer base, behind incumbent telco BT, satellite TV operator Sky, and cable operator Virgin Media. TalkTalk also competes in the enterprise market where its circa 60,000 Ethernet circuits represent a market share of 10-12%.
Moving DNS and security to the edge
One major initiative Whitbread's team has been working on involves migrating DNS and associated security controls away from disparate central data centers/IT servers and closer to the customer. Around five years ago TalkTalk began a major upgrade of its BRAS edge routers in order to support the ever-increasing broadband traffic on its network. The solution involved replacing 100-plus edge routers with around 60 devices with significantly greater capacity.
Subsequently, Whitbread's team was looking at their ageing DNS infrastructure, which comprised over 100 servers spread over six data centers, considering whether it was time to upgrade to new servers to support the increase in DNS requests from web browsing customers. But rather than upgrading the existing DNS servers, the capabilities of TalkTalk's new edge routers provided for an innovative idea: "Wouldn't it be better to have something integral within the network?" The idea was to decommission the existing 100-plus DNS servers and move this functionality onto a line card running on the 60-plus BRAS/edge routers. By placing DNS software closer to the customer, TalkTalk would significantly reduce the latency of IP address lookups leading to a better customer experience.
Along with its new edge routers, TalkTalk had purchased modular line cards with processing and storage that allowed them to run additional services on the routers. They had originally intended to use them for caching content at the edge; instead they decided to put their DNS software on these cards. After an evaluation process TalkTalk chose a new DNS software supplier, Nominum (the existing solution had been based on open source components). The Nominum solution also came with cyber-threat protection and could additionally replace a URL blocking service (Homesafe) which was being provided on legacy Huawei Technologies Co. Ltd. infrastructure.
Lab tests were carried out to ensure the DNS and security applications would run satisfactorily on the line cards and support arrangements were established between the router vendor and the DNS supplier to ensure no finger pointing in the case of problems. The decision to run not only DNS on the edge router cards, but also the customer cyber-threat protection service required some significant OSS and BSS adaptation which took around nine months to complete.
By running DNS and cyber-threat protection on the edge router cards TalkTalk was able to retire around 200 servers that had previously been running these functions. Additionally, TalkTalk's new DNS and cyber threat protection vendor supplied a new cybersecurity application (that protects the network and subscribers against DDoS, DNS tunnelling, botnets, etc.) running on the same cards. This now provides TalkTalk's security operations center with a wealth of information that they can use to keep their domestic and corporate customers safe.
Organizationally, this convergence of network (edge router) and IT (DNS, security) capability can pose issues. "Who looks after the compute card sitting on the router on which the DNS and security software runs? Is that the network organization or IT?" Whitbread's approach has been to move some key personnel from the IT department to the network team to look after this specific DNS and security capability. This has helped to break down some of the cultural barriers between network and IT staff that often exists in telco organisations.
Next page: Early days for automation
Early days for automation
We also asked Whitbread about how TalkTalk is tackling the challenge of automation in network operations.
"Automation is a significant issue for us," he says, "and I've personally been championing automation since I joined the company six years ago."
One example Whitbread cites is that historically the process of configuring an Ethernet circuit for TalkTalk's enterprise customers had been highly manual. Given that TalkTalk has been adding around 8,000 new Ethernet and Ethernet in the First Mile circuits each year, provisioning was becoming a bottleneck.
Nowadays, TalkTalk uses a service provisioning and activation software platform that automates the processes associated with the creation and activation Ethernet services across the entire fulfillment stack including order management, resource inventory and service activation. Whitbread adds, "This has brought down the cost to serve our customers and the timeframe for delivery of new services."
Whitbread sees a huge number of additional opportunities for automation in network operations.
"In the NOC [Network Operations Center] we've got first line engineers investigating alarms and then escalating issues to second line engineers to fix the problem. All that can be automated with scripts."
That's not to say automation equals job cuts. Instead, Whitbread sees automation as a way to augment the workforce, freeing up their time from mundane activities so they can focus on more value-added, complex tasks instead.
One example of automation TalkTalk has already implemented takes advantage of SDN. Historically, when a link in the IP network went down it would take the engineers as much as an hour to reroute the traffic. This could have been done automatically with a traffic optimizer which TalkTalk had purchased to automate the creation of traffic-engineering paths over IP/MPLS WANs. However, TalkTalk first needed to get all of its IP routers onto the same version of software in order to use this particular vendor's controller. When completed, this software upgrade will enable the automatic rerouting all IP traffic in the case of a link or node failure.
"It will take away all the manual tasks of rerouting and hence improve the service to customers," notes Whitbread, "but it takes time and a lot of effort to get these things to work effectively. During the same period, we also replaced much of our caching capability out to the edge of the network, also to improve our customer experience."
Whitbread sees plenty of scope to further increase the level of network automation. "There are more and more suppliers knocking on my door every day offering new ways to automate. And we have plenty of automation tools already. It is a question of finding budget for investment [for new tools] and finding the time to implement and utilise our existing tools more thoroughly."
— James Crawshaw, Senior Analyst, Heavy Reading