& cplSiteName &

The 'Big Data' Challenge

Column
Column
Column
12/5/2011
50%
50%

"Big Data" was a much-discussed concept in IT management circles last year. But the effect of Big Data -- the non-stop massive growth of information -- has been a challenge for service providers' data infrastructure and technology teams for quite some time.

Tier 1 providers have experienced billions of call-detail records (CDRs) every day for years. In response, custom database technologies are now being developed and refined. Petabyte-scale database systems holding years of call-detail records and Internet protocol detail records (IPDRs) are now considered the norm at many Tier 1 providers.

The Big Data effect has proven even more complex and challenging with the advent of more stringent compliance regulations. A single global provider must proactively configure, manage and retain specific subscriber records for a designated time period. Additionally, the regulatory regimes of different nations impose different timeframes, requiring different data types in different formats. Specific response times to access that data vary across borders as well. In the past year alone, Japan and India have imposed new regulatory initiatives upon providers, regardless of size and subscriber level.

Fulfilling lawful intercept requirements demands substantial investments by providers in storage, database and search technologies -- and, of course, trained personnel with the skills necessary to deploy these solutions. Lawful intercept involves both real-time monitoring and access to data records. For example, law enforcement may obtain judicial warrants to tap a suspect's landlines, cellphones and email accounts, and procure copies of their call, SMS/MMS and other communication records. Having this data readily available in the correct format and delivered in a timely manner creates ongoing pressure to make the data center run efficiently and cost effectively. The possibility of legal fines only compounds the pressure.

Today communications service providers (CSPs) must innovate to achieve scale and better IT economics. Interoperability with other global partners, providers and suppliers is important but not enough. The CSP must be prepared to use technology, network architecture and standardization to deal with these pressures.

A recent Gartner industry report, Predicts 2011: CSPs Must Rethink Business Paradigms to Meet Market Challenges, cited falling margins of CSPs in mobile and fixed combined services sectors in development markets. There are very few exceptions to this trend. The causes of the drop include Internet Protocol (IP) substitution, lower average revenue per unit (ARPU) from new customers, increasing competition, regulators getting a better view of non-competitive price elements and the need to invest in expensive broadband networks with long payback cycles.

Improving the customer experience and service level is absolutely a necessary part of staying ahead. New partnerships are formed daily to provide new applications and mobile capabilities to drive new channels of revenue. However, all of the front-end customer-facing products and services that are rolled out will only be sustainable if back-office and data infrastructure are capable of robust management and increasing scale in the coming years.

A service provider IT issue
Let's consider the data problem from the perspective of today's IT team.

Within the life-cycle of a single customer's communication "transactions" in a given billing cycle, core data (essentially call data detail records and Internet Protocol detail records) that are captured, collated and stored are immediately historical in nature -- once the "transaction" takes place, the data does not undergo any further change. This is quite different from traditional business transactions that are captured in an online transactional database system, such as a retail purchase. This customer communication data is then processed by a series of applications that allow various IT and business functions, including network capacity planning, traffic analysis, user trending, revenue assurance, billing, digital advertising and more.

Some of these OSS and BSS applications leverage the same database system; in other cases, the data is moved around with varying degrees of latency. Data sub-sets are formed depending on application function and requirements, and data warehouses are fed with customer activity in order to perform trending analysis and determine the overall value of a customer at any point in time. Currently, many CSPs retain much of this data in a traditional database such as Oracle. For many of these functions, the data does not need to change; therefore it can be stored in a database whose primary purpose is long-term retention.

Given the growth in data volume and complexity across diverse systems and applications, organizations are forced to retain the data for as little time as possible, largely because systems have grown bloated -- to the point where the sheer size of the data harms performance. For a large provider, even the critical function of replicating data from a complex billing application in order to provide continuous uptime can cause significant technology challenges due to the volume of the data sets.

Specialized databases
However, new database technologies are emerging that promise to solve this problem. Specialized databases with the ability to significantly de-duplicate and reduce this data to a much smaller footprint are being deployed by providers. When you consider a 100-terabyte system reduced to a storage footprint of 5 terabytes, you can begin to imagine the cost savings not only in storage but also hardware capacity. Imagine a petabyte environment reduced to 100 terabytes. These specialized data-retention databases -- built for the purpose of storing data online for many years -- can also perform fast queries using standard SQL or various business intelligence and analytics solutions. With a database solution focused on Big Data retention that enables significant cost reduction, CSPs can store critical customer activity data for much longer periods -- even beyond what is required by regulators. By having access to larger historical data sets, CSPs can gain greater insights into customer behavior and even discover new opportunities to further segment the market and create new services with a wider range of price points.

Today's R&D teams now employ data scientists who dissect and mine usage data to detect behavior patterns in order to predict key market drivers such as what causes churn. Being able to predict customer behavior is much more powerful to the business than analyzing what happened and why it happened after the fact! Once churned, it costs the business as much as six times more to win that customer loyalty back. Those data scientists don't need SQL statements or business intelligence tools -- they must have continuous access to the raw data, the detailed historical xDR data, in order to determine market and behavior patterns. Giving them a dedicated Big Data retention database is the best way to achieve that. By looking inward and discovering new technologies and new approaches to managing core customer and transaction data, CSPs can discover new service offerings -- even new markets.

The ocean of Big Data faced by today's CSP does not have to reach a boiling point. Applying innovative data-retention database capabilities across the infrastructure will significantly reduce cost, and help manage scale and complexity.

— Dr. Hossein Eslambolchi is the former CTO of AT&T and chairman and CEO of 2020 Venture Partners

(0)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
More Blogs from Column
New and exciting methods of automation – whether virtualization, the cloud, IoT or even best practices like network segmentation – tend to emphasize innovation over visibility. As such, networks develop blind spots that mask network problems and even faulty devices.
Unlicensed spectrum will help the 3GPP's 5G specification proliferate.
Outages are inevitable, but how can we deal with them better?
The arrival of NFV and IoT is driving a greater need for Service Quality Management (SQM) capabilities, argues Sandeep Raina.
An updated architecture, new approach to software and the ability to launch new services will give telcos a successful path to 5G within the next few years.
From The Founder
Following a recent board meeting, the New IP Agency (NIA) has a new strategy to help accelerate the adoption of NFV capabilities, explains the Agency's Founder and Secretary, Steve Saunders.
Flash Poll
Live Streaming Video
Charting the CSP's Future
Six different communications service providers join to debate their visions of the future CSP, following a landmark presentation from AT&T on its massive virtualization efforts and a look back on where the telecom industry has been and where it's going from two industry veterans.
LRTV Interviews
AT&T: Creating Dynamic Networks to Meet Business Needs

5|26|17   |   4:24   |   (0) comments


As enterprises need more dynamic networks, service providers need to deliver on-demand, virtual services to meet those needs. AT&T is creating a networking fabric to mix-and-match SDN technologies for enterprise customers, says Roman Pacewicz, AT&T senior vice president for offer management and service integration, in an interview at Light Reading's
LRTV Interviews
EdgeConneX on Industry Headwinds & Tailwinds

5|26|17   |   2:41   |   (0) comments


At Light Reading's Big Communications Event 2017, EdgeConneX CTO Don MacNeil discussed the value of partnerships in the digital world.
LRTV Documentaries
4 Steps Toward a Higher Network IQ

5|26|17   |     |   (0) comments


At the Big Communications Event in Austin, Texas, EXFO CEO Philippe Morin explains how sensors and analytics can boost a network's intelligence and enable on-demand customer experiences. Find more BCE 2017 coverage here.
LRTV Interviews
BT's McRae Sheds Light on 4K Strategy

5|25|17   |   4:45   |   (0) comments


At Light Reading's Big Communications Event 2017 in Austin, Texas, BT Group's Chief Network Architect Neil McRae talks about what it took for BT to broadcast live sports in 4K. Catch up with all our BCE coverage at http://www.lightreading.com/bce.asp.
From the Founder
How the NIA Aims to Advance NFV

5|25|17   |   08:07   |   (1) comment


Following a recent board meeting, the New IP Agency (NIA) has a new strategy to help accelerate the adoption of NFV capabilities, explains the Agency's Founder and Secretary, Steve Saunders.
LRTV Custom TV
Better Solutions That Address Growing Scale

5|25|17   |     |   (0) comments


For Comcast, the X1 rollout and 17-fold increases in broadband speeds in the past 16 years are among factors driving the need for Energy 2020 solutions that reduce cost and consumption, says Mark Hess.
LRTV Custom TV
Ethernity Network Delivers Instant Offloading of Network Functions With All-Programmable Intelligent NIC

5|25|17   |     |   (0) comments


David Levi, CEO of Ethernity Networks, explains that programmability of the hardware makes the company's All-Programmable Intelligent NIC uniquely beneficial for communications service providers that need advanced data appliances with agile support of virtualization. Utilizing the company's patented network processing technology, Ethernity offers data path ...
LRTV Documentaries
BCE 2017: Vodafone Gets Obsessed With Cloud-Native

5|25|17   |     |   (0) comments


Vodafone's Matt Beal updates us on Project Ocean and explains why simple virtualization isn't enough of a goal for network transformation. Catch up with other BCE 2017 keynotes and news at http://www.lightreading.com/bce.asp.
LRTV Documentaries
BCE 2017: Intel's Take on Network Transformation

5|24|17   |     |   (0) comments


In this BCE 2017 keynote, Lynn Comp discusses Intel's vision for areas such as analytics, automation and service assurance. For more videos and BCE coverage, see http://www.lightreading.com/bce.asp.
LRTV Documentaries
Order From Chaos: The Steve Saunders BCE Keynote

5|24|17   |   17:27   |   (0) comments


Kicking off BCE 2017, Light Reading founder Steve Saunders lays blame for NFV's slow ramp-up and urges telecom to return to old-fashioned standards building and interoperability testing.
Think of this as the video sequel to the recent columns he's written about NFV and the prospect of a telecom app store. (See

LRTV Documentaries
Service Provider Panel: Partnering in the Digital Era

5|22|17   |     |   (0) comments


Coopetition has always been part of telecom, but the ecosphere now includes data centers, vendors, apps developers, cloud service providers and Internet content providers. This BCE 2017 panel explores the new attitudes among network operators as to the value and variety of ...
LRTV Interviews
Site Demo: AT&T's IoT Flow Platform

5|19|17   |   04:25   |   (0) comments


At AT&T's R&D center in Tel Aviv, Israel, project leader Eyal Segev talks about the operator's Flow platform and how it helps to prototype IoT applications.
Infographics
With the mobile ecosystem becoming increasingly vulnerable to security threats, AdaptiveMobile has laid out some of the key considerations for the wireless community.
Hot Topics
Cities Clamor for More Clout at FCC
Mari Silbey, Senior Editor, Cable/Video, 5/23/2017
What's Blocking 4K TV Today
Alan Breznick, Cable/Video Practice Leader, Light Reading, 5/22/2017
Sonus & Genband Finally Combine to Form $745M Company
Dan Jones, Mobile Editor, 5/23/2017
Apple Looking to Cook 5G Test Devices
Dan Jones, Mobile Editor, 5/24/2017
Like Us on Facebook
Twitter Feed
BETWEEN THE CEOs - Executive Interviews
One of the nice bits of my job (other than the teeny tiny salary, obviously) is that I get to pick and choose who I interview for this slot on the Light Reading home ...
TEOCO Founder and CEO Atul Jain talks to Light Reading Founder and CEO Steve Saunders about the challenges around cost control and service monetization in the mobile and IoT sectors.
Animals with Phones
What Brogrammers Look Like to the Rest of Us Click Here
Live Digital Audio

Playing it safe can only get you so far. Sometimes the biggest bets have the biggest payouts, and that is true in your career as well. For this radio show, Caroline Chan, general manager of the 5G Infrastructure Division of the Network Platform Group at Intel, will share her own personal story of how she successfully took big bets to build a successful career, as well as offer advice on how you can do the same. We’ll cover everything from how to overcome fear and manage risk, how to be prepared for where technology is going in the future and how to structure your career in a way to ensure you keep progressing. Chan, a seasoned telecom veteran and effective risk taker herself, will also leave plenty of time to answer all your questions live on the air.