Light Reading

The 'Big Data' Challenge

Column
Column
Column
12/5/2011
50%
50%

"Big Data" was a much-discussed concept in IT management circles last year. But the effect of Big Data -- the non-stop massive growth of information -- has been a challenge for service providers' data infrastructure and technology teams for quite some time.

Tier 1 providers have experienced billions of call-detail records (CDRs) every day for years. In response, custom database technologies are now being developed and refined. Petabyte-scale database systems holding years of call-detail records and Internet protocol detail records (IPDRs) are now considered the norm at many Tier 1 providers.

The Big Data effect has proven even more complex and challenging with the advent of more stringent compliance regulations. A single global provider must proactively configure, manage and retain specific subscriber records for a designated time period. Additionally, the regulatory regimes of different nations impose different timeframes, requiring different data types in different formats. Specific response times to access that data vary across borders as well. In the past year alone, Japan and India have imposed new regulatory initiatives upon providers, regardless of size and subscriber level.

Fulfilling lawful intercept requirements demands substantial investments by providers in storage, database and search technologies -- and, of course, trained personnel with the skills necessary to deploy these solutions. Lawful intercept involves both real-time monitoring and access to data records. For example, law enforcement may obtain judicial warrants to tap a suspect's landlines, cellphones and email accounts, and procure copies of their call, SMS/MMS and other communication records. Having this data readily available in the correct format and delivered in a timely manner creates ongoing pressure to make the data center run efficiently and cost effectively. The possibility of legal fines only compounds the pressure.

Today communications service providers (CSPs) must innovate to achieve scale and better IT economics. Interoperability with other global partners, providers and suppliers is important but not enough. The CSP must be prepared to use technology, network architecture and standardization to deal with these pressures.

A recent Gartner industry report, Predicts 2011: CSPs Must Rethink Business Paradigms to Meet Market Challenges, cited falling margins of CSPs in mobile and fixed combined services sectors in development markets. There are very few exceptions to this trend. The causes of the drop include Internet Protocol (IP) substitution, lower average revenue per unit (ARPU) from new customers, increasing competition, regulators getting a better view of non-competitive price elements and the need to invest in expensive broadband networks with long payback cycles.

Improving the customer experience and service level is absolutely a necessary part of staying ahead. New partnerships are formed daily to provide new applications and mobile capabilities to drive new channels of revenue. However, all of the front-end customer-facing products and services that are rolled out will only be sustainable if back-office and data infrastructure are capable of robust management and increasing scale in the coming years.

A service provider IT issue
Let's consider the data problem from the perspective of today's IT team.

Within the life-cycle of a single customer's communication "transactions" in a given billing cycle, core data (essentially call data detail records and Internet Protocol detail records) that are captured, collated and stored are immediately historical in nature -- once the "transaction" takes place, the data does not undergo any further change. This is quite different from traditional business transactions that are captured in an online transactional database system, such as a retail purchase. This customer communication data is then processed by a series of applications that allow various IT and business functions, including network capacity planning, traffic analysis, user trending, revenue assurance, billing, digital advertising and more.

Some of these OSS and BSS applications leverage the same database system; in other cases, the data is moved around with varying degrees of latency. Data sub-sets are formed depending on application function and requirements, and data warehouses are fed with customer activity in order to perform trending analysis and determine the overall value of a customer at any point in time. Currently, many CSPs retain much of this data in a traditional database such as Oracle. For many of these functions, the data does not need to change; therefore it can be stored in a database whose primary purpose is long-term retention.

Given the growth in data volume and complexity across diverse systems and applications, organizations are forced to retain the data for as little time as possible, largely because systems have grown bloated -- to the point where the sheer size of the data harms performance. For a large provider, even the critical function of replicating data from a complex billing application in order to provide continuous uptime can cause significant technology challenges due to the volume of the data sets.

Specialized databases
However, new database technologies are emerging that promise to solve this problem. Specialized databases with the ability to significantly de-duplicate and reduce this data to a much smaller footprint are being deployed by providers. When you consider a 100-terabyte system reduced to a storage footprint of 5 terabytes, you can begin to imagine the cost savings not only in storage but also hardware capacity. Imagine a petabyte environment reduced to 100 terabytes. These specialized data-retention databases -- built for the purpose of storing data online for many years -- can also perform fast queries using standard SQL or various business intelligence and analytics solutions. With a database solution focused on Big Data retention that enables significant cost reduction, CSPs can store critical customer activity data for much longer periods -- even beyond what is required by regulators. By having access to larger historical data sets, CSPs can gain greater insights into customer behavior and even discover new opportunities to further segment the market and create new services with a wider range of price points.

Today's R&D teams now employ data scientists who dissect and mine usage data to detect behavior patterns in order to predict key market drivers such as what causes churn. Being able to predict customer behavior is much more powerful to the business than analyzing what happened and why it happened after the fact! Once churned, it costs the business as much as six times more to win that customer loyalty back. Those data scientists don't need SQL statements or business intelligence tools -- they must have continuous access to the raw data, the detailed historical xDR data, in order to determine market and behavior patterns. Giving them a dedicated Big Data retention database is the best way to achieve that. By looking inward and discovering new technologies and new approaches to managing core customer and transaction data, CSPs can discover new service offerings -- even new markets.

The ocean of Big Data faced by today's CSP does not have to reach a boiling point. Applying innovative data-retention database capabilities across the infrastructure will significantly reduce cost, and help manage scale and complexity.

— Dr. Hossein Eslambolchi is the former CTO of AT&T and chairman and CEO of 2020 Venture Partners

(0)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
Educational Resources
sponsor supplied content
Educational Resources Archive
More Blogs from Column
In the second part of a two-part series, Paul Robinson explains why the move to live IP production is an opportunity to significantly change the viewing experience, and more importantly, open up new revenue and business opportunities.
One thing is certain: the coming technology changes will affect everyone to varying degrees, including content providers, service providers and equipment makers.
It's time for our industry to call BS on the US government's position on Chinese telco manufacturers.
There are important considerations for companies to keep in mind when moving Domain Name System (DNS) infrastructure to an NFV implementation.
The lies, damned lies and statistics of real open access network operations.
From The Founder
Light Reading sits down at CES with the head of Cisco's service provider video business, Conrad Clemson, to discuss how NFV and cloud security relate to video, the challenge of managing 4K/8K traffic, the global expansion of Netflix and virtual reality.
Flash Poll
Live Streaming Video
CLOUD / MANAGED SERVICES: Prepping Ethernet for the Cloud
Moderator: Ray LeMaistre Panelists: Jeremy Bye, Leonard Sheahan
LRTV Custom TV
Hosting in Ireland, Past & Present

2|10|16   |   16:07   |   (0) comments


Garry Connolly, president of Host in Ireland, presents the keynote at Light Reading's 2020 Vision Executive Summit in Dublin.
Women in Comms Introduction Videos
What's Hot in Mobile Commerce?

2|10|16   |   12:18   |   (1) comment


Claire Maslen, financial services relationship manager at the GSMA, talks about the development of the digital commerce sector and the types of relationships that mobile operators are developing to further their m-commerce strategies.
LRTV Documentaries
EANTC Tests Nokia IP Routing & Mobile Gateway VNFs for Real World Deployment

2|9|16   |   5:08   |   (0) comments


Nokia obtained validation of its virtualized router and virtualized mobile gateway capabilities through rigorous testing performed by EANTC. The results set a new industry benchmark for outstanding performance, scalability, resiliency and manageability. Nokia VNFs are ready for telco cloud deployment, so that service providers can accelerate mobile, business and ...
Between the CEOs
CEO Chat With Level 3's Jack Waters

2|8|16   |   26:15   |   (1) comment


Light Reading CEO and founder Steve Saunders sits down with Level 3 Communications' CTO Jack Waters to discuss hot topics like virtualization, 4K and the future of telecom...
LRTV Custom TV
The Composable Telco

2|8|16   |   24:46   |   (0) comments


Heavy Reading's Principal Analyst Caroline Chappell presents the keynote at Light Reading's 2020 Vision Executive Summit in Dublin.
LRTV Custom TV
Join Us at the Digital Operations Transformation Summit

2|4|16   |   03:52   |   (0) comments


The Digital Operations Transformation Summit on February 21, 2016 at the Crowne Plaza Barcelona Fira Centre will bring together 50 senior executives to engage in a unique debate on the opportunities and challenges presented by the transformative evolving digital landscape. RSVP now at events@lightreading.com.
LRTV Custom TV
Making the Test: ADVA Ensemble Connector vs. Open vSwitch

2|4|16   |   01:28   |   (0) comments


Light Reading, in partnership with EANTC, recently tested ADVA's Ensemble Connector, which replaces open vSwitch and offers carrier-grade capability and interoperability. The test results strengthen ADVA's credibility as a provider in the virtualization space.
LRTV Custom TV
Bridging the Gap Between PoCs & Deployment in NFV

2|4|16   |   31:50   |   (0) comments


Charlie Ashton of Wind River presents the keynote at Light Reading's 2020 Vision executive summit in Dublin.
Between the CEOs
CEO Chat With Mike Aquino

2|3|16   |   17:34   |   (0) comments


The former CEO of Overture Networks, Mike Aquino, discusses why truly open virtualization solutions provide service providers with the greatest choice.
Shades of Ray
MWC: Buckle Up for 5G & the IIoT

2|2|16   |   02:28   |   (0) comments


This year's Mobile World Congress looks set to be a 5G land grab and a chance to get down and dirty with the Industrial Internet of Things (IIoT) – but what will the 5G discussions actually be about?
LRTV Custom TV
Case Study: Building China's Next-Gen TV Networks

2|2|16   |   5:01   |   (0) comments


With over 2 billion viewers worldwide, Shenzhen Media Group is one of China's largest content producers. By partnering with Huawei and Sobey, SZMG was able to modernize media operations with the Converged News Center, a production studio that is a model for next-generation workflows.
LRTV Custom TV
Quad Channel Modulator Driver with 46 Gbaud Capability from MACOM

1|28|16   |     |   (0) comments


MACOM's MAOM-003427 is the industry's first surface-mount modulator driver with 46 Gbaud capability to support next generation 200G and 400G applications.
Upcoming Live Events
March 10, 2016, The Cable Center, Denver, CO
April 5, 2016, The Ritz Carlton, Charlotte, NC
May 23, 2016, Austin, TX
May 24-25, 2016, Austin Convention Center, Austin, TX
All Upcoming Live Events
Hot Topics
Google's 5G Radio Ambitions Are Expanding
Dan Jones, Mobile Editor, 2/5/2016
Cincinnati Bell Joins Weight Watchers Club
Alan Breznick, Cable/Video Practice Leader, 2/5/2016
Yahoo & Verizon Sitting in a Tree...
Brian Santo, Senior editor, Test & Measurement / Components, Light Reading, 2/8/2016
Vodafone: Flexible Work Policies Boost Profits
Sarah Thomas, Editorial Operations Director, 2/8/2016
It's Time to Integrate OTT Video
Alan Breznick, Cable/Video Practice Leader, 2/8/2016
Like Us on Facebook
Twitter Feed
Webinar Archive
BETWEEN THE CEOs - Executive Interviews
Light Reading CEO and founder Steve Saunders sits down with Level 3 Communications' CTO Jack Waters to discuss hot topics like virtualization, 4K and the future of telecom...
The former CEO of Overture Networks, Mike Aquino, discusses why truly open virtualization solutions provide service providers with the greatest choice.
Animals with Phones
Retro Is the Way to Be Click Here
Some animals are too cute for smartphones.
Live Digital Audio

Broadband speeds are ramping up across Europe as the continent, at its own pace, follows North America towards a gigabit society. But there are many steps to take on the road to gigabit broadband availability and a number of technology options that can meet the various requirements of Europe’s high-speed fixed broadband network operators. During this radio show we will look at some of the catalysts for broadband network investments and examine the menu of technology options on offer, including vectoring and G.fast for copper plant evolution and the various deployment possibilities for FTTH/B.