& cplSiteName &

The 'Big Data' Challenge


"Big Data" was a much-discussed concept in IT management circles last year. But the effect of Big Data -- the non-stop massive growth of information -- has been a challenge for service providers' data infrastructure and technology teams for quite some time.

Tier 1 providers have experienced billions of call-detail records (CDRs) every day for years. In response, custom database technologies are now being developed and refined. Petabyte-scale database systems holding years of call-detail records and Internet protocol detail records (IPDRs) are now considered the norm at many Tier 1 providers.

The Big Data effect has proven even more complex and challenging with the advent of more stringent compliance regulations. A single global provider must proactively configure, manage and retain specific subscriber records for a designated time period. Additionally, the regulatory regimes of different nations impose different timeframes, requiring different data types in different formats. Specific response times to access that data vary across borders as well. In the past year alone, Japan and India have imposed new regulatory initiatives upon providers, regardless of size and subscriber level.

Fulfilling lawful intercept requirements demands substantial investments by providers in storage, database and search technologies -- and, of course, trained personnel with the skills necessary to deploy these solutions. Lawful intercept involves both real-time monitoring and access to data records. For example, law enforcement may obtain judicial warrants to tap a suspect's landlines, cellphones and email accounts, and procure copies of their call, SMS/MMS and other communication records. Having this data readily available in the correct format and delivered in a timely manner creates ongoing pressure to make the data center run efficiently and cost effectively. The possibility of legal fines only compounds the pressure.

Today communications service providers (CSPs) must innovate to achieve scale and better IT economics. Interoperability with other global partners, providers and suppliers is important but not enough. The CSP must be prepared to use technology, network architecture and standardization to deal with these pressures.

A recent Gartner industry report, Predicts 2011: CSPs Must Rethink Business Paradigms to Meet Market Challenges, cited falling margins of CSPs in mobile and fixed combined services sectors in development markets. There are very few exceptions to this trend. The causes of the drop include Internet Protocol (IP) substitution, lower average revenue per unit (ARPU) from new customers, increasing competition, regulators getting a better view of non-competitive price elements and the need to invest in expensive broadband networks with long payback cycles.

Improving the customer experience and service level is absolutely a necessary part of staying ahead. New partnerships are formed daily to provide new applications and mobile capabilities to drive new channels of revenue. However, all of the front-end customer-facing products and services that are rolled out will only be sustainable if back-office and data infrastructure are capable of robust management and increasing scale in the coming years.

A service provider IT issue
Let's consider the data problem from the perspective of today's IT team.

Within the life-cycle of a single customer's communication "transactions" in a given billing cycle, core data (essentially call data detail records and Internet Protocol detail records) that are captured, collated and stored are immediately historical in nature -- once the "transaction" takes place, the data does not undergo any further change. This is quite different from traditional business transactions that are captured in an online transactional database system, such as a retail purchase. This customer communication data is then processed by a series of applications that allow various IT and business functions, including network capacity planning, traffic analysis, user trending, revenue assurance, billing, digital advertising and more.

Some of these OSS and BSS applications leverage the same database system; in other cases, the data is moved around with varying degrees of latency. Data sub-sets are formed depending on application function and requirements, and data warehouses are fed with customer activity in order to perform trending analysis and determine the overall value of a customer at any point in time. Currently, many CSPs retain much of this data in a traditional database such as Oracle. For many of these functions, the data does not need to change; therefore it can be stored in a database whose primary purpose is long-term retention.

Given the growth in data volume and complexity across diverse systems and applications, organizations are forced to retain the data for as little time as possible, largely because systems have grown bloated -- to the point where the sheer size of the data harms performance. For a large provider, even the critical function of replicating data from a complex billing application in order to provide continuous uptime can cause significant technology challenges due to the volume of the data sets.

Specialized databases
However, new database technologies are emerging that promise to solve this problem. Specialized databases with the ability to significantly de-duplicate and reduce this data to a much smaller footprint are being deployed by providers. When you consider a 100-terabyte system reduced to a storage footprint of 5 terabytes, you can begin to imagine the cost savings not only in storage but also hardware capacity. Imagine a petabyte environment reduced to 100 terabytes. These specialized data-retention databases -- built for the purpose of storing data online for many years -- can also perform fast queries using standard SQL or various business intelligence and analytics solutions. With a database solution focused on Big Data retention that enables significant cost reduction, CSPs can store critical customer activity data for much longer periods -- even beyond what is required by regulators. By having access to larger historical data sets, CSPs can gain greater insights into customer behavior and even discover new opportunities to further segment the market and create new services with a wider range of price points.

Today's R&D teams now employ data scientists who dissect and mine usage data to detect behavior patterns in order to predict key market drivers such as what causes churn. Being able to predict customer behavior is much more powerful to the business than analyzing what happened and why it happened after the fact! Once churned, it costs the business as much as six times more to win that customer loyalty back. Those data scientists don't need SQL statements or business intelligence tools -- they must have continuous access to the raw data, the detailed historical xDR data, in order to determine market and behavior patterns. Giving them a dedicated Big Data retention database is the best way to achieve that. By looking inward and discovering new technologies and new approaches to managing core customer and transaction data, CSPs can discover new service offerings -- even new markets.

The ocean of Big Data faced by today's CSP does not have to reach a boiling point. Applying innovative data-retention database capabilities across the infrastructure will significantly reduce cost, and help manage scale and complexity.

— Dr. Hossein Eslambolchi is the former CTO of AT&T and chairman and CEO of 2020 Venture Partners

(0)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
Educational Resources
sponsor supplied content
Educational Resources Archive
More Blogs from Column
5G could ride the traditional wireless hype cycle, or – quite possibly – break the chain, suggests Nokia's North American CTO.
NFV feels like it's going nowhere fast, suggests Napatech's Dan Joe Barry.
The NG-PON2 standard is the key to the future of ultra-broadband, believes Calix's Alan DiCicco.
Amazing new 5G innovations are coming below 6GHz -- it's not just about mmWave.
5G is about more than just faster mobile broadband, notes Tom Sawanobori, SVP and chief technology officer at the CTIA.
Light Reading’s Upskill U is a FREE, interactive, online educational resource that delivers must-have education on themes that relate to the overall business transformation taking place in the communications industry.
Friday, October 21, 1:00PM EDT
Security: Evolving the Data Center
Rasool Kareem Irfan, Head, Telecom & Infrastructure Security Practice, Tata Communications Transformation Services Ltd (TCTS)
Wednesday, October 26, 1:00PM EDT
Security: Tackling DDoS
Gary Sockrider, Principal Security Technologist, Arbor Networks
Friday, October 28, 1:00PM EDT
Security: The Plusses and Minuses of Open Source Software
Nick Feamster, Acting Director, Center for Information Technology Policy, Princeton University
Wednesday, November 16, 1:00PM EST
SDN 101
in association with:
From The Founder
Light Reading today starts a new voyage as part of a larger Enterprise.
Flash Poll
Live Streaming Video
Charting the CSP's Future
Six different communications service providers join to debate their visions of the future CSP, following a landmark presentation from AT&T on its massive virtualization efforts and a look back on where the telecom industry has been and where it's going from two industry veterans.
LRTV Custom TV
Next-Generation Technology Beyond DOCSIS 3.1

10|20|16   |     |   (0) comments

At SCTE 2016, Huawei's Liu Jianhua speaks with Alan Breznick for an exclusive interview.
LRTV Custom TV
Hybrid Video Solutions to Change TV, Change Future

10|20|16   |     |   (0) comments

At SCTE 2016, Huawei's Ian Locke speaks with Alan Breznick for an exclusive interview.
LRTV Custom TV
Huawei Future-Oriented Giga Coax Network

10|20|16   |     |   (0) comments

At SCTE 2016, Huawei's Allen Wang speaks with Alan Breznick for an exclusive interview.
LRTV Custom TV
Huawei at SCTE 2016

10|20|16   |     |   (0) comments

Join Alan Breznick of Light Reading and take a sneak peek at the Huawei booth at SCTE 2016.
LRTV Custom TV
Assuring Network Quality in a Rapidly Changing Environment

10|20|16   |     |   (0) comments

As the rate of change and complexity increases in agile networks, the importance of introducing DevOps methodologies for integrating active test and assurance solutions throughout the full service lifecycle becomes critical to ensure that customers are experiencing the service quality they demand. The industry landscape is changing, and software-based test and ...
Telecom Innovators Video Showcase
A10 Networks on Service Providers' Industry Needs

10|20|16   |     |   (0) comments

Light Reading's Steve Saunders hears how A10 enables service providers to accelerate, secure and optimize their application delivery to drive down costs, enhance service availability, and better respond to customer requirements, so they can improve customer satisfaction, monetize their network, and grow revenues.
LRTV Custom TV
New NFV Use Cases for Cable TV

10|19|16   |     |   (0) comments

A large number of NFV use cases are focused on the enterprise domain, looking at virtualization of customer-premises equipment (CPE). To date, there has been little focus on the use cases and business case for virtualization of the video content delivery networks required to deliver unicast and streaming video to consumers. Amdocs commissioned Analysys Mason to ...
Women in Comms Introduction Videos
Meet the Future Workforce: New Faces, Expectations & Motivations

10|19|16   |   5:33   |   (1) comment

Millennials and their younger peers, Gen Z, expect more out of their network and more out of their work. Intel's Lynn Comp shares how the industry can prepare for this new generation of workers.
LRTV Custom TV
ZTE Global Services User Congress 2016 Highlights

10|19|16   |     |   (0) comments

ZTE held its 2nd Global Service User Conference in Dusseldorf on October 13-14. Representatives from network operators, leading industry analysts and ZTE senior expertsattended the event, exploring the best practice in managed services and the vision to transform network operations into the operations center of the future (OpCF) in the software-defined networking ...
LRTV Custom TV
Cliff Grossner on Cloud & Network Synergy From Carrier Service

10|18|16   |     |   (0) comments

Local carriers offer the collaborated cloud and network service that benefits from their understanding of the regulations operating in different vertical markets.
In this interview, Cliff Grossner from IHS Technology talks about how this advanced service can support business agility and flexibility.
LRTV Custom TV
VeEX's Plan for DOCSIS Deployment

10|17|16   |     |   (0) comments

Join VeEX's CEO Cyrille Morelle and Light Reading's Alan Breznick as they discuss VeEX's new products at SCTE 2016 at Philadelphia.
LRTV Custom TV
Smith Micro's Carrier-Grade WiFi Component

10|17|16   |     |   (0) comments

Join Carol Wilson of Light Reading as she interviews the CTO of Smith Micro, Dave Sperling.
Upcoming Live Events
November 3, 2016, The Montcalm Marble Arch, London
November 30, 2016, The Westin Times Square, New York City
December 1, 2016, The Westin Times Square, New York, NY
December 6-8, 2016, The Westin Excelsior, Rome
May 16-17, 2017, Austin Convention Center, Austin, TX
All Upcoming Live Events
Hot Topics
Amazon Unlimited, Alexa, AI & How to Cut Off Google at Its Knees
Brian Santo, Senior editor, Test & Measurement / Components, Light Reading, 10/14/2016
WiCipedia: Following Women on Twitter... and on Stage
Eryn Leavens, Special Features & Copy Editor, 10/14/2016
US Issues Total Flight Ban on Samsung Note 7
Dan Jones, Mobile Editor, 10/15/2016
Oh Snap! Qualcomm Unveils X50, Its First 5G Modem
Dan Jones, Mobile Editor, 10/17/2016
Verizon Wants More Data on Yahoo Breach
Alan Breznick, Cable/Video Practice Leader, Light Reading, 10/20/2016
Like Us on Facebook
Twitter Feed
BETWEEN THE CEOs - Executive Interviews
Join us for an in-depth interview between Steve Saunders of Light Reading and Alexis Black Bjorlin of Intel as they discuss the release of the company's Silicon Photonics platform, its performance, long-term prospects, customer expectations and much more.
There's no question that, come 2020, 5G technology will turn the world's conception of what mobile networking is on its head. Within the world of 5G development, Dr. ...
Animals with Phones
Live Digital Audio

A vital part of increasing the number of women in comms is transforming the ways companies can support and empower women. While progressive company policies that support both men and women in achieving work-life balance are a step in the right direction, creating a company culture that supports those policies can at times be more challenging.

During this show, we'll talk to Lynn Comp, Senior Director of Industry and Sales Enabling (ISE) in the Network Platforms Group at Intel, about why those challenges exist and how companies can overcome them. She'll provide insight into how Intel has worked to create a culture that supports work-life balance, and provide steps and guidance for other companies wishing to do the same. We will also leave plenty of time to get your questions answered live on the air.