& cplSiteName &

The 'Big Data' Challenge

Column
Column
Column
12/5/2011
50%
50%

"Big Data" was a much-discussed concept in IT management circles last year. But the effect of Big Data -- the non-stop massive growth of information -- has been a challenge for service providers' data infrastructure and technology teams for quite some time.

Tier 1 providers have experienced billions of call-detail records (CDRs) every day for years. In response, custom database technologies are now being developed and refined. Petabyte-scale database systems holding years of call-detail records and Internet protocol detail records (IPDRs) are now considered the norm at many Tier 1 providers.

The Big Data effect has proven even more complex and challenging with the advent of more stringent compliance regulations. A single global provider must proactively configure, manage and retain specific subscriber records for a designated time period. Additionally, the regulatory regimes of different nations impose different timeframes, requiring different data types in different formats. Specific response times to access that data vary across borders as well. In the past year alone, Japan and India have imposed new regulatory initiatives upon providers, regardless of size and subscriber level.

Fulfilling lawful intercept requirements demands substantial investments by providers in storage, database and search technologies -- and, of course, trained personnel with the skills necessary to deploy these solutions. Lawful intercept involves both real-time monitoring and access to data records. For example, law enforcement may obtain judicial warrants to tap a suspect's landlines, cellphones and email accounts, and procure copies of their call, SMS/MMS and other communication records. Having this data readily available in the correct format and delivered in a timely manner creates ongoing pressure to make the data center run efficiently and cost effectively. The possibility of legal fines only compounds the pressure.

Today communications service providers (CSPs) must innovate to achieve scale and better IT economics. Interoperability with other global partners, providers and suppliers is important but not enough. The CSP must be prepared to use technology, network architecture and standardization to deal with these pressures.

A recent Gartner industry report, Predicts 2011: CSPs Must Rethink Business Paradigms to Meet Market Challenges, cited falling margins of CSPs in mobile and fixed combined services sectors in development markets. There are very few exceptions to this trend. The causes of the drop include Internet Protocol (IP) substitution, lower average revenue per unit (ARPU) from new customers, increasing competition, regulators getting a better view of non-competitive price elements and the need to invest in expensive broadband networks with long payback cycles.

Improving the customer experience and service level is absolutely a necessary part of staying ahead. New partnerships are formed daily to provide new applications and mobile capabilities to drive new channels of revenue. However, all of the front-end customer-facing products and services that are rolled out will only be sustainable if back-office and data infrastructure are capable of robust management and increasing scale in the coming years.

A service provider IT issue
Let's consider the data problem from the perspective of today's IT team.

Within the life-cycle of a single customer's communication "transactions" in a given billing cycle, core data (essentially call data detail records and Internet Protocol detail records) that are captured, collated and stored are immediately historical in nature -- once the "transaction" takes place, the data does not undergo any further change. This is quite different from traditional business transactions that are captured in an online transactional database system, such as a retail purchase. This customer communication data is then processed by a series of applications that allow various IT and business functions, including network capacity planning, traffic analysis, user trending, revenue assurance, billing, digital advertising and more.

Some of these OSS and BSS applications leverage the same database system; in other cases, the data is moved around with varying degrees of latency. Data sub-sets are formed depending on application function and requirements, and data warehouses are fed with customer activity in order to perform trending analysis and determine the overall value of a customer at any point in time. Currently, many CSPs retain much of this data in a traditional database such as Oracle. For many of these functions, the data does not need to change; therefore it can be stored in a database whose primary purpose is long-term retention.

Given the growth in data volume and complexity across diverse systems and applications, organizations are forced to retain the data for as little time as possible, largely because systems have grown bloated -- to the point where the sheer size of the data harms performance. For a large provider, even the critical function of replicating data from a complex billing application in order to provide continuous uptime can cause significant technology challenges due to the volume of the data sets.

Specialized databases
However, new database technologies are emerging that promise to solve this problem. Specialized databases with the ability to significantly de-duplicate and reduce this data to a much smaller footprint are being deployed by providers. When you consider a 100-terabyte system reduced to a storage footprint of 5 terabytes, you can begin to imagine the cost savings not only in storage but also hardware capacity. Imagine a petabyte environment reduced to 100 terabytes. These specialized data-retention databases -- built for the purpose of storing data online for many years -- can also perform fast queries using standard SQL or various business intelligence and analytics solutions. With a database solution focused on Big Data retention that enables significant cost reduction, CSPs can store critical customer activity data for much longer periods -- even beyond what is required by regulators. By having access to larger historical data sets, CSPs can gain greater insights into customer behavior and even discover new opportunities to further segment the market and create new services with a wider range of price points.

Today's R&D teams now employ data scientists who dissect and mine usage data to detect behavior patterns in order to predict key market drivers such as what causes churn. Being able to predict customer behavior is much more powerful to the business than analyzing what happened and why it happened after the fact! Once churned, it costs the business as much as six times more to win that customer loyalty back. Those data scientists don't need SQL statements or business intelligence tools -- they must have continuous access to the raw data, the detailed historical xDR data, in order to determine market and behavior patterns. Giving them a dedicated Big Data retention database is the best way to achieve that. By looking inward and discovering new technologies and new approaches to managing core customer and transaction data, CSPs can discover new service offerings -- even new markets.

The ocean of Big Data faced by today's CSP does not have to reach a boiling point. Applying innovative data-retention database capabilities across the infrastructure will significantly reduce cost, and help manage scale and complexity.

— Dr. Hossein Eslambolchi is the former CTO of AT&T and chairman and CEO of 2020 Venture Partners

(0)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
More Blogs from Column
Spectrum sharing is becoming a bigger issue as the 5G radio specification evolves.
Addressing current and future app demands while laying the foundation for mobile's next big network transition.
Broadcasters can no longer rely on pulling audiences to the TV screen; they need to pursue their audiences on digital, wherever they are.
Why advanced data analytics are the future for streaming video services.
5G could ride the traditional wireless hype cycle, or – quite possibly – break the chain, suggests Nokia's North American CTO.
From The Founder
Light Reading today starts a new voyage as part of a larger Enterprise.
Flash Poll
Live Streaming Video
Charting the CSP's Future
Six different communications service providers join to debate their visions of the future CSP, following a landmark presentation from AT&T on its massive virtualization efforts and a look back on where the telecom industry has been and where it's going from two industry veterans.
LRTV Interviews
Cox Clears $2B in Business Revenue

12|8|16   |     |   (0) comments


Cox's Jeff Breaux discusses how the third-largest US MSO will reach the $2 billion revenue mark this year and plans to hit $3 billion by 2021
LRTV Interviews
Can Cable Climb Upmarket?

12|7|16   |     |   (0) comments


Carol Wilson and Alan Breznick assess cable's prospects for winning more enterprises in a landscape rocked by corporate M&A activity.
Women in Comms Introduction Videos
TalkTalk Exec: Find Your North Star at Work

12|7|16   |   3:38   |   (1) comment


Women need to find their purpose, a professional North Star, and create a personal board for themselves, according to Alex Tempest, director of partners at TalkTalk Business.
LRTV Interviews
Verizon: Beware Unknown Unknowns

12|7|16   |   04:58   |   (0) comments


Chris Novak, director of the Verizon Enterprise Solutions Risk Team, explains that enterprises who don't conduct a thorough audit of their assets often leave some things unprotected because they don't know they exist. Many times these unprotected assets are part of corporate M&A activity but left unshielded they can become a hacker's playground, he tells Light ...
LRTV Interviews
ETSI's CTO Talks NFV, 5G & NGP

12|5|16   |   09:45   |   (0) comments


Adrian Scrase, CTO at standards body ETSI, talks about the various initiatives and specifications developments related to NFV, 5G and NGP (next-generation protocols) that will underpin next-gen networks.
Women in Comms Introduction Videos
Korn Ferry Consultant: How to Find, Cultivate & Be the Best Talent

11|30|16   |   4:10   |   (2) comments


Erin Callaghan, a managing consultant for Korn Ferry Futurestep, shares strategies for companies to improve how they recruit and for women to ensure they don't get lost in the pipeline.
LRTV Custom TV
We Can Make the World More Sustainable

11|29|16   |     |   (0) comments


GeSI is a global e-Sustainability Initiative organization bringing together 40 big multinational companies around the world. According to GeSI's report, information and communication technology can make the world more sustainable. Luis Neves, chairman of GeSI, shared with us his opinion at Ultra-broadband Forum (UBBF2016).
LRTV Custom TV
Finding a New Way to Engage Customers & Drive Revenue

11|29|16   |     |   (0) comments


Mobile revenues are declining. Digicel, a player in the Caribbean telecommunications/entertainment space, has found a new way to engage customers and drive revenue. John Quinn, CTO of Digicel, shared with us its story at Ultra-broadband Forum (UBBF2016)
LRTV Custom TV
Do You Really Need Gigabit Infrastructure?

11|29|16   |     |   (0) comments


Altibox is the biggest fiber-to-the-home (FTTH) player and the largest provider of video and TV in Norway. They started out with zero customers in 2002. Now they have close to half a million households and companies attached to their FTTH business. Nils Arne, CEO of Altibox shared with us their story and insight on 5G at Ultra-broadband Forum (UBBF2016).
LRTV Custom TV
BTís Openreach Strategy & Its Updates in 2016

11|29|16   |     |   (0) comments


A lot of developments at Openreach this year in terms of strategy and planned investments. Peter Bell, CIO of Openreach BT, shared with us the updates of Openreach at Ultra-broadband Forum (UBBF2016).
LRTV Custom TV
ITU: The Broadband Is Our Future

11|29|16   |     |   (0) comments


At Ultra-broadband Forum, Houlin Zhao, Secretary General of ITU, discussed how important it is for countries, companies and everybody to be working together to help to build the broadband and digital economies (UBBF2016).
LRTV Custom TV
Tackling 5G in Dallas

11|28|16   |     |   (0) comments


Here are our highlights of the 5G North America show in Dallas, Texas with Light Reading's Dan Jones.
Upcoming Live Events
May 16-17, 2017, Austin Convention Center, Austin, TX
All Upcoming Live Events
Infographics
Hot Topics
Cable Nodes Becoming a Choke Point
Brian Santo, Senior editor, Test & Measurement / Components, Light Reading, 12/5/2016
WiCipedia: After-School Coding, Salary Probing & Pro-Parenthood Companies
Eryn Leavens, Special Features & Copy Editor, 12/2/2016
Consolidated Snaps Up Fairpoint for $1.5B
Iain Morris, News Editor, 12/5/2016
Like Us on Facebook
Twitter Feed
BETWEEN THE CEOs - Executive Interviews
Eyal Waldman, CEO of Mellanox Technologies, speaks to Steve Saunders, CEO of Light Reading, for an exclusive interview about the 100 GB cable challenge, cybersecurity and much more.
Join us for an in-depth interview between Steve Saunders of Light Reading and Alexis Black Bjorlin of Intel as they discuss the release of the company's Silicon Photonics platform, its performance, long-term prospects, customer expectations and much more.
Animals with Phones
A Mobile Safari Click Here
Literally.
Latest Comment
Live Digital Audio

Even when there's a strong pipeline of female talent in the comms industry, it tends to leak all the way to the top. McKinsey & Company says women experience pipeline leakage at three primary points: being unable to enter, being stuck in the middle or being locked out of the top. Each pipeline pain point presents its own challenges, but also opportunities to stop the leak. Wireless operator Sprint is making a conscious effort to improve its own pipeline from new recruits to the C-suite, and it wants the rest of the industry to do the same. In this Women in Comms radio show, WiC Board Member and Sprint Vice President of Enterprise Sales Nelly Pitocco will give us her take on the industry's pipeline challenges. Pitocco, who joined Sprint in May and has spent 20 years in the comms industry, will also offer solutions, share how Sprint is tackling the challenge within its own organization and take your questions live on air.