Light Reading

The 'Big Data' Challenge

Column
Column
Column
12/5/2011
50%
50%

"Big Data" was a much-discussed concept in IT management circles last year. But the effect of Big Data -- the non-stop massive growth of information -- has been a challenge for service providers' data infrastructure and technology teams for quite some time.

Tier 1 providers have experienced billions of call-detail records (CDRs) every day for years. In response, custom database technologies are now being developed and refined. Petabyte-scale database systems holding years of call-detail records and Internet protocol detail records (IPDRs) are now considered the norm at many Tier 1 providers.

The Big Data effect has proven even more complex and challenging with the advent of more stringent compliance regulations. A single global provider must proactively configure, manage and retain specific subscriber records for a designated time period. Additionally, the regulatory regimes of different nations impose different timeframes, requiring different data types in different formats. Specific response times to access that data vary across borders as well. In the past year alone, Japan and India have imposed new regulatory initiatives upon providers, regardless of size and subscriber level.

Fulfilling lawful intercept requirements demands substantial investments by providers in storage, database and search technologies -- and, of course, trained personnel with the skills necessary to deploy these solutions. Lawful intercept involves both real-time monitoring and access to data records. For example, law enforcement may obtain judicial warrants to tap a suspect's landlines, cellphones and email accounts, and procure copies of their call, SMS/MMS and other communication records. Having this data readily available in the correct format and delivered in a timely manner creates ongoing pressure to make the data center run efficiently and cost effectively. The possibility of legal fines only compounds the pressure.

Today communications service providers (CSPs) must innovate to achieve scale and better IT economics. Interoperability with other global partners, providers and suppliers is important but not enough. The CSP must be prepared to use technology, network architecture and standardization to deal with these pressures.

A recent Gartner industry report, Predicts 2011: CSPs Must Rethink Business Paradigms to Meet Market Challenges, cited falling margins of CSPs in mobile and fixed combined services sectors in development markets. There are very few exceptions to this trend. The causes of the drop include Internet Protocol (IP) substitution, lower average revenue per unit (ARPU) from new customers, increasing competition, regulators getting a better view of non-competitive price elements and the need to invest in expensive broadband networks with long payback cycles.

Improving the customer experience and service level is absolutely a necessary part of staying ahead. New partnerships are formed daily to provide new applications and mobile capabilities to drive new channels of revenue. However, all of the front-end customer-facing products and services that are rolled out will only be sustainable if back-office and data infrastructure are capable of robust management and increasing scale in the coming years.

A service provider IT issue
Let's consider the data problem from the perspective of today's IT team.

Within the life-cycle of a single customer's communication "transactions" in a given billing cycle, core data (essentially call data detail records and Internet Protocol detail records) that are captured, collated and stored are immediately historical in nature -- once the "transaction" takes place, the data does not undergo any further change. This is quite different from traditional business transactions that are captured in an online transactional database system, such as a retail purchase. This customer communication data is then processed by a series of applications that allow various IT and business functions, including network capacity planning, traffic analysis, user trending, revenue assurance, billing, digital advertising and more.

Some of these OSS and BSS applications leverage the same database system; in other cases, the data is moved around with varying degrees of latency. Data sub-sets are formed depending on application function and requirements, and data warehouses are fed with customer activity in order to perform trending analysis and determine the overall value of a customer at any point in time. Currently, many CSPs retain much of this data in a traditional database such as Oracle. For many of these functions, the data does not need to change; therefore it can be stored in a database whose primary purpose is long-term retention.

Given the growth in data volume and complexity across diverse systems and applications, organizations are forced to retain the data for as little time as possible, largely because systems have grown bloated -- to the point where the sheer size of the data harms performance. For a large provider, even the critical function of replicating data from a complex billing application in order to provide continuous uptime can cause significant technology challenges due to the volume of the data sets.

Specialized databases
However, new database technologies are emerging that promise to solve this problem. Specialized databases with the ability to significantly de-duplicate and reduce this data to a much smaller footprint are being deployed by providers. When you consider a 100-terabyte system reduced to a storage footprint of 5 terabytes, you can begin to imagine the cost savings not only in storage but also hardware capacity. Imagine a petabyte environment reduced to 100 terabytes. These specialized data-retention databases -- built for the purpose of storing data online for many years -- can also perform fast queries using standard SQL or various business intelligence and analytics solutions. With a database solution focused on Big Data retention that enables significant cost reduction, CSPs can store critical customer activity data for much longer periods -- even beyond what is required by regulators. By having access to larger historical data sets, CSPs can gain greater insights into customer behavior and even discover new opportunities to further segment the market and create new services with a wider range of price points.

Today's R&D teams now employ data scientists who dissect and mine usage data to detect behavior patterns in order to predict key market drivers such as what causes churn. Being able to predict customer behavior is much more powerful to the business than analyzing what happened and why it happened after the fact! Once churned, it costs the business as much as six times more to win that customer loyalty back. Those data scientists don't need SQL statements or business intelligence tools -- they must have continuous access to the raw data, the detailed historical xDR data, in order to determine market and behavior patterns. Giving them a dedicated Big Data retention database is the best way to achieve that. By looking inward and discovering new technologies and new approaches to managing core customer and transaction data, CSPs can discover new service offerings -- even new markets.

The ocean of Big Data faced by today's CSP does not have to reach a boiling point. Applying innovative data-retention database capabilities across the infrastructure will significantly reduce cost, and help manage scale and complexity.

— Dr. Hossein Eslambolchi is the former CTO of AT&T and chairman and CEO of 2020 Venture Partners

(0)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View
More Blogs from Column
There is an impressive array of technology that is planned for the transition to 5G that will enable many new possible services.
5G could shape up to be a stunningly expensive fiasco.
The Swedish vendor has undergone a significant transformation during the past few years, adjusting to the demands of next-generation communications companies.
Wi-Fi Alliance members have created Wi-Fi Aware technology to bring local experiences to life -- without requiring access to the Internet or GPS.
Bamboozled by the flood of abbreviations and acronyms that SDN and NFV have inspired? Here's a guide to some of the key standards bodies whose initials you will need to know.
Flash Poll
From The Founder
The Swedish vendor has undergone a significant transformation during the past few years, adjusting to the demands of next-generation communications companies.
LRTV Documentaries
The 3GPP's Road to 5G Standardization

4|17|15   |   4:43   |   (0) comments


Satoshi Nagata, chairman of the 3GPP's TSG-RAN group and a manager at NTT Docomo, explains the standardization process for 5G, as well as the biggest challenges and opportunities.
LRTV Documentaries
AlcaLu CTO Makes the Case for a New 5G Air Interface

4|16|15   |   3:54   |   (0) comments


Michael Peeters, CTO of wireless at Alcatel-Lucent, explains why 5G will require a new air interface to meet its diverse performance targets.
LRTV Documentaries
AlcaLu + Nokia: The New Uber-Vendor

4|15|15   |   2:42   |   (4) comments


Heavy Reading Senior Analyst Gabriel Brown discusses the technological and competitive opportunities and challenges if a merger between Alcatel-Lucent and Nokia comes to pass.
LRTV Huawei Video Resource Center
Huawei's Data Center Power Play

4|15|15   |   6:22   |   (0) comments


Huawei has developed industry-leading energy efficiency capabilities for its indoor and outdoor data center solutions, explains Dr. Fang Liangzhou, vice president of Huawei's Network Energy product Line.
LRTV Huawei Video Resource Center
Huawei’s Routers, Switches Get the Green Mark

4|15|15   |   2:02   |   (0) comments


TUV Rheinland's Frank Dudley explains how Huawei's routers and switches have been successfully tested by energy efficiency experts and have gained Green Mark Certification.
LRTV Documentaries
A Finn, a Frenchman & a Guy From New Jersey Walk Into a Merger...

4|15|15   |   3:17   |   (0) comments


Stop us if you've heard this one before... Light Reading CEO Founder & CEO Steve Saunders weighs in on the technical and cultural implications of a Nokia and Alcatel-Lucent merger.
LRTV Huawei Video Resource Center
Accounting for Better Solutions

4|10|15   |   02:31   |   (1) comment


Murad Yousuf, CTO at Saudi Arabia's Ministry of Finance (Dept. of Zakat & Income Tax), talks about the benefits of deploying router technology from Huawei.
LRTV Huawei Video Resource Center
What's in Store for Huawei & DataCore?

4|10|15   |   05:44   |   (0) comments


At the CeBIT trade show in Hannover, Germany, George Teixeira, CEO of software-defined storage (SDS) specialist DataCore Software, explains why he has just signed a partnership agreement with Huawei Technologies.
LRTV Huawei Video Resource Center
Du Puts Its Faith in Huawei's Routers

4|9|15   |   3:42   |   (0) comments


Adnan Masood, director of Enterprise MS Solutions Marketing at du, the United Arab Emirates (UAE) operator also known as Emirates Integrated Telecommunications Company, explains why his company chose to use Huawei's multifunctional AR routers as part of its managed enterprise services.
LRTV Huawei Video Resource Center
Huawei Gets Active in the Data Center

4|9|15   |   3:17   |   (0) comments


With enterprise users looking to maximize the use of their data center assets, Huawei’s Chief Architect & Technical Director of IT Data Center Solutions, Bruce Su, explains how the company's six-layer active-active data center solution is eradicating the need to deploy passive, redundant data center assets.
LRTV Huawei Video Resource Center
Blue Consult & Huawei for a Better Solution

4|8|15   |   4:01   |   (0) comments


Martin Rott, CEO, and Marc Metzler, head of sales virtualization, from Germany's Blue Consult discuss their collaboration with Huawei and TrendMicro to develop a secure, scalable IT platform that can meet the needs of the most demanding enterprise users.
LRTV Huawei Video Resource Center
Beach Petroleum on eLTE & Mining

4|8|15   |   3:09   |   (0) comments


Network systems integrator Jeremy Hamlyn explains how Huawei's secure packet-based trunking communications system, eLTE, can help remote communities and companies in the mining, oil and gas sectors, deploy efficient communications networks that are perfect for video and data as well as voice.
Upcoming Live Events
May 5, 2015, Hyatt McCormick Place, Chicago, IL
May 6, 2015, Georgia World Congress, Atlanta, GA
May 12, 2015, Grand Hyatt, Denver, CO
May 13-14, 2015, The Westin Peachtree, Atlanta, GA
June 8, 2015, Chicago, IL
June 9-10, 2015, Chicago, IL
June 9, 2015, Chicago, IL
June 10, 2015, Chicago, IL
September 29-30, 2015, The Westin Grand Müchen, Munich, Germany
November 11-12, 2015, The Westin Peachtree Plaza, Atlanta, GA
All Upcoming Live Events
Infographics
Network Instruments, a JDSU division, shares results from its 2015 State of the Network, a global survey on security.
Hot Topics
Verizon Scores New OTT Content Deals
Mari Silbey, Independent Technology Editor, 4/16/2015
Can WiFi Calling Find Its Voice?
Iain Morris, News Editor, 4/13/2015
Senator Proposes New 'Title X' for Net Neutrality
Carol Wilson, Editor-at-large, 4/13/2015
Nokia, Alcatel-Lucent in Merger Talks
Iain Morris, News Editor, 4/14/2015
Nokia & Alcatel-Lucent: What's Going On?
Ray Le Maistre, Editor-in-chief, 4/15/2015
Like Us on Facebook
Twitter Feed
Webinar Archive
BETWEEN THE CEOs - Executive Interviews
Data Center Interconnect, or DCI, is one of the hottest sectors in telecom currently. Since coming back to Light Reading last year, prodigal-son style, I've ...
LR CEO and Founder Steve Saunders sits down with the head of Qosmos to talk about the changing state of the art in deep packet inspection technology, including its role in SDN and NFV architectures.