Light Reading

The 'Big Data' Challenge


"Big Data" was a much-discussed concept in IT management circles last year. But the effect of Big Data -- the non-stop massive growth of information -- has been a challenge for service providers' data infrastructure and technology teams for quite some time.

Tier 1 providers have experienced billions of call-detail records (CDRs) every day for years. In response, custom database technologies are now being developed and refined. Petabyte-scale database systems holding years of call-detail records and Internet protocol detail records (IPDRs) are now considered the norm at many Tier 1 providers.

The Big Data effect has proven even more complex and challenging with the advent of more stringent compliance regulations. A single global provider must proactively configure, manage and retain specific subscriber records for a designated time period. Additionally, the regulatory regimes of different nations impose different timeframes, requiring different data types in different formats. Specific response times to access that data vary across borders as well. In the past year alone, Japan and India have imposed new regulatory initiatives upon providers, regardless of size and subscriber level.

Fulfilling lawful intercept requirements demands substantial investments by providers in storage, database and search technologies -- and, of course, trained personnel with the skills necessary to deploy these solutions. Lawful intercept involves both real-time monitoring and access to data records. For example, law enforcement may obtain judicial warrants to tap a suspect's landlines, cellphones and email accounts, and procure copies of their call, SMS/MMS and other communication records. Having this data readily available in the correct format and delivered in a timely manner creates ongoing pressure to make the data center run efficiently and cost effectively. The possibility of legal fines only compounds the pressure.

Today communications service providers (CSPs) must innovate to achieve scale and better IT economics. Interoperability with other global partners, providers and suppliers is important but not enough. The CSP must be prepared to use technology, network architecture and standardization to deal with these pressures.

A recent Gartner industry report, Predicts 2011: CSPs Must Rethink Business Paradigms to Meet Market Challenges, cited falling margins of CSPs in mobile and fixed combined services sectors in development markets. There are very few exceptions to this trend. The causes of the drop include Internet Protocol (IP) substitution, lower average revenue per unit (ARPU) from new customers, increasing competition, regulators getting a better view of non-competitive price elements and the need to invest in expensive broadband networks with long payback cycles.

Improving the customer experience and service level is absolutely a necessary part of staying ahead. New partnerships are formed daily to provide new applications and mobile capabilities to drive new channels of revenue. However, all of the front-end customer-facing products and services that are rolled out will only be sustainable if back-office and data infrastructure are capable of robust management and increasing scale in the coming years.

A service provider IT issue
Let's consider the data problem from the perspective of today's IT team.

Within the life-cycle of a single customer's communication "transactions" in a given billing cycle, core data (essentially call data detail records and Internet Protocol detail records) that are captured, collated and stored are immediately historical in nature -- once the "transaction" takes place, the data does not undergo any further change. This is quite different from traditional business transactions that are captured in an online transactional database system, such as a retail purchase. This customer communication data is then processed by a series of applications that allow various IT and business functions, including network capacity planning, traffic analysis, user trending, revenue assurance, billing, digital advertising and more.

Some of these OSS and BSS applications leverage the same database system; in other cases, the data is moved around with varying degrees of latency. Data sub-sets are formed depending on application function and requirements, and data warehouses are fed with customer activity in order to perform trending analysis and determine the overall value of a customer at any point in time. Currently, many CSPs retain much of this data in a traditional database such as Oracle. For many of these functions, the data does not need to change; therefore it can be stored in a database whose primary purpose is long-term retention.

Given the growth in data volume and complexity across diverse systems and applications, organizations are forced to retain the data for as little time as possible, largely because systems have grown bloated -- to the point where the sheer size of the data harms performance. For a large provider, even the critical function of replicating data from a complex billing application in order to provide continuous uptime can cause significant technology challenges due to the volume of the data sets.

Specialized databases
However, new database technologies are emerging that promise to solve this problem. Specialized databases with the ability to significantly de-duplicate and reduce this data to a much smaller footprint are being deployed by providers. When you consider a 100-terabyte system reduced to a storage footprint of 5 terabytes, you can begin to imagine the cost savings not only in storage but also hardware capacity. Imagine a petabyte environment reduced to 100 terabytes. These specialized data-retention databases -- built for the purpose of storing data online for many years -- can also perform fast queries using standard SQL or various business intelligence and analytics solutions. With a database solution focused on Big Data retention that enables significant cost reduction, CSPs can store critical customer activity data for much longer periods -- even beyond what is required by regulators. By having access to larger historical data sets, CSPs can gain greater insights into customer behavior and even discover new opportunities to further segment the market and create new services with a wider range of price points.

Today's R&D teams now employ data scientists who dissect and mine usage data to detect behavior patterns in order to predict key market drivers such as what causes churn. Being able to predict customer behavior is much more powerful to the business than analyzing what happened and why it happened after the fact! Once churned, it costs the business as much as six times more to win that customer loyalty back. Those data scientists don't need SQL statements or business intelligence tools -- they must have continuous access to the raw data, the detailed historical xDR data, in order to determine market and behavior patterns. Giving them a dedicated Big Data retention database is the best way to achieve that. By looking inward and discovering new technologies and new approaches to managing core customer and transaction data, CSPs can discover new service offerings -- even new markets.

The ocean of Big Data faced by today's CSP does not have to reach a boiling point. Applying innovative data-retention database capabilities across the infrastructure will significantly reduce cost, and help manage scale and complexity.

— Dr. Hossein Eslambolchi is the former CTO of AT&T and chairman and CEO of 2020 Venture Partners

(0)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
More Blogs from Column
Cord cutting is a real and present threat to pay-TV, but it has been an erosion, not a landslide.
They go together hand-in-hand in many networks but the relationship is strained – is there a way to improve the efficiency of an OpenStack and OVS deployment?
The comms industry is rallying to the cause of open, independent interoperability testing.
Data analytics and revenue assurance are growing in strategic importance for the cable industry.
The introduction of IoT applications is supposed to make life easier for homo sapiens. But things are off to a bad start, according to Accedian's Scott Sumner.
From The Founder
Light Reading's conference in November will attempt to answer all of the big questions around white box networks. No pressure...
Flash Poll
Live Streaming Video
CLOUD / MANAGED SERVICES: Prepping Ethernet for the Cloud
Moderator: Ray LeMaistre Panelists: Jeremy Bye, Leonard Sheahan
LRTV Custom TV
ZTE in Budapest ITU 2015

10|13|15   |   03:26   |   (0) comments

ZTE Chief Architect David Huo discusses the company's progress on the 5G front.
Telecom Innovators Video Showcase
Close-up on ConfD

10|12|15   |   10.21   |   (0) comments

Tail-f's Renée Robinson-Stromberg tells Steve Saunders about the powerful ConfD management interface.
Women in Comms Introduction Videos
Women in Comms: Highlights From Dallas

10|12|15   |   2:23   |   (1) comment

The best soundbites, quotes and words of wisdom from leading women from Intel, AT&T, Verizon and Genband at our recent WiC breakfast in Dallas.
Telecom Innovators Video Showcase
NetNumber Founder on Managing Signaling Control

10|12|15   |   6:36   |   (0) comments

NetNumber Founder and Chief Strategy Officer Doug Ranalli describes the essential complexity of real-world signaling-control and how NetNumber enables carriers to bring signaling-control "under-control". Learn why virtualization alone isn't the answer.
LRTV Documentaries
Verizon Gets Proactive on App Performance

10|12|15   |   04:50   |   (0) comments

SDN is turning traditional service models around to allow Verizon to measure and deliver performance at the application layer. As Shawn Hakl, VP of enterprise networking and managed solutions for Verizon, explains, the carrier had to develop new skill sets and change some of its internal operations, but the payoff was happier enterprise customers.
Telecom Innovators Video Showcase
Tail-f, Cisco & What the Future Holds

10|9|15   |   8:17   |   (0) comments

Steve Saunders meets with Tail-f's Director of Technology, Carl Moberg, in Stockholm to discuss becoming part of Cisco, ETSI MANO, virtualization and the need to combine science and business in order to create opportunities for service providers.
LRTV Interviews
Broadband Forum Embraces SDN & NFV

10|9|15   |   02:42   |   (1) comment

At Gigabit Europe 2015, Robin Mersh and Kevin Foster from the Broadband Forum explain how the industry body is adapting to meet the SDN, NFV and cloud needs of the access network sector.
LRTV Interviews
Top Tips for FTTH Operators

10|8|15   |   02:26   |   (0) comments

At Gigabit Europe 2015, Ventura Team co-founder Richard Jones talks about some of the key business case considerations for FTTH network operators.
LRTV Interviews
M-net Calls for FTTx Unity

10|8|15   |   03:45   |   (0) comments

At the Gigabit Europe event, Jörn Schoof from M-net, the Munich city network operator, calls for industry collaboration on fiber broadband access rollouts.
LRTV Documentaries
The Business Case Challenge for NFV

10|7|15   |   03:47   |   (0) comments

Virtual CPE is one of the early success stories for network functions virtualization, as service providers are finding flexible, programmable CPE solves a lot of logistics problems and reduces their cost. But even here, Masergy Communications faced a business case challenge, says CTO Tim Naramore.
LRTV Interviews
JT Offers Some Gigabit Lessons

10|7|15   |   4:08   |   (1) comment

Barna Kutvolgyi, managing director, Global Consumer, at JT, the incumbent operator on the island of Jersey, talks about how other service providers can learn from his company's gigabit broadband rollout experiences.
LRTV Interviews
AT&T's Chiosi on the Potential of Open Source

10|6|15   |   06:27   |   (0) comments

AT&T Distinguished Network Architect Margaret T. Chiosi talks to Light Reading's Carol Wilson about the potential for open source technology to liberate communications service providers.
Upcoming Live Events
October 14-15, 2015, New Orleans Ernest N. Morial Convention Center, New Orleans, LA
November 5, 2015, Hilton Santa Clara, Santa Clara, CA
November 17, 2015, Santa Clara, California
December 1, 2015, The Westin Times Square, New York City
December 2, 2015, The Westin Times Square, New York City
All Upcoming Live Events
Network appliances have a strong value proposition in today's networks and will continue to do so in the NFV and SDN-enabled networks of tomorrow.
Hot Topics
Dell Buys EMC for $67B in Biggest Tech Deal Ever
Mari Silbey, Senior Editor, Cable/Video, 10/12/2015
M&A Speculation Swirls Around Juniper
Ray Le Maistre, Editor-in-chief, 10/6/2015
Cord Cutting? 'Fraid so.
Brett Sappington, 10/7/2015
Cisco Makes 'Martian' Connection
Mitch Wagner, West Coast Bureau Chief, Light Reading, 10/9/2015
Like Us on Facebook
Twitter Feed
Webinar Archive
BETWEEN THE CEOs - Executive Interviews
With so many new and exciting communications technologies now under development, it's easy to get caught up in the industry's escalating hype cycle. That's why the ...
Last week saw a big day in the 15-year history of Light Reading when Editor-in-Chief Ray Le Maistre and I were invited to interview the Deputy Chairman and Rotating ...
Cats with Phones
"What?! I'm on with Finisar about their stock price tanking" Click Here
Live Digital Audio

Think NFV is just about virtualization? Think again!

Network architects are learning that there's a lot more to the technology than first thought – more complexity, that is; but also, more potential benefits.

On May 29th 1 PM ET, Steve Saunders, founder and CEO of Light Reading, will be drilling into the "pains and gains" of NFV with Saar Gillai, SVP & GM, HP Communications Solutions Business at Hewlett-Packard Co. (NYSE: HPQ) (HP). He has defined a four-step NFV model describing a sequence of technology innovation. It's a must-read doc for any network architect looking to get to grips with their NFV migration strategy. Join us for the interview, and the chance to ask Saar your NFV questions directly!