Light Reading

The 'Big Data' Challenge

Column
Column
Column
12/5/2011
50%
50%

"Big Data" was a much-discussed concept in IT management circles last year. But the effect of Big Data -- the non-stop massive growth of information -- has been a challenge for service providers' data infrastructure and technology teams for quite some time.

Tier 1 providers have experienced billions of call-detail records (CDRs) every day for years. In response, custom database technologies are now being developed and refined. Petabyte-scale database systems holding years of call-detail records and Internet protocol detail records (IPDRs) are now considered the norm at many Tier 1 providers.

The Big Data effect has proven even more complex and challenging with the advent of more stringent compliance regulations. A single global provider must proactively configure, manage and retain specific subscriber records for a designated time period. Additionally, the regulatory regimes of different nations impose different timeframes, requiring different data types in different formats. Specific response times to access that data vary across borders as well. In the past year alone, Japan and India have imposed new regulatory initiatives upon providers, regardless of size and subscriber level.

Fulfilling lawful intercept requirements demands substantial investments by providers in storage, database and search technologies -- and, of course, trained personnel with the skills necessary to deploy these solutions. Lawful intercept involves both real-time monitoring and access to data records. For example, law enforcement may obtain judicial warrants to tap a suspect's landlines, cellphones and email accounts, and procure copies of their call, SMS/MMS and other communication records. Having this data readily available in the correct format and delivered in a timely manner creates ongoing pressure to make the data center run efficiently and cost effectively. The possibility of legal fines only compounds the pressure.

Today communications service providers (CSPs) must innovate to achieve scale and better IT economics. Interoperability with other global partners, providers and suppliers is important but not enough. The CSP must be prepared to use technology, network architecture and standardization to deal with these pressures.

A recent Gartner industry report, Predicts 2011: CSPs Must Rethink Business Paradigms to Meet Market Challenges, cited falling margins of CSPs in mobile and fixed combined services sectors in development markets. There are very few exceptions to this trend. The causes of the drop include Internet Protocol (IP) substitution, lower average revenue per unit (ARPU) from new customers, increasing competition, regulators getting a better view of non-competitive price elements and the need to invest in expensive broadband networks with long payback cycles.

Improving the customer experience and service level is absolutely a necessary part of staying ahead. New partnerships are formed daily to provide new applications and mobile capabilities to drive new channels of revenue. However, all of the front-end customer-facing products and services that are rolled out will only be sustainable if back-office and data infrastructure are capable of robust management and increasing scale in the coming years.

A service provider IT issue
Let's consider the data problem from the perspective of today's IT team.

Within the life-cycle of a single customer's communication "transactions" in a given billing cycle, core data (essentially call data detail records and Internet Protocol detail records) that are captured, collated and stored are immediately historical in nature -- once the "transaction" takes place, the data does not undergo any further change. This is quite different from traditional business transactions that are captured in an online transactional database system, such as a retail purchase. This customer communication data is then processed by a series of applications that allow various IT and business functions, including network capacity planning, traffic analysis, user trending, revenue assurance, billing, digital advertising and more.

Some of these OSS and BSS applications leverage the same database system; in other cases, the data is moved around with varying degrees of latency. Data sub-sets are formed depending on application function and requirements, and data warehouses are fed with customer activity in order to perform trending analysis and determine the overall value of a customer at any point in time. Currently, many CSPs retain much of this data in a traditional database such as Oracle. For many of these functions, the data does not need to change; therefore it can be stored in a database whose primary purpose is long-term retention.

Given the growth in data volume and complexity across diverse systems and applications, organizations are forced to retain the data for as little time as possible, largely because systems have grown bloated -- to the point where the sheer size of the data harms performance. For a large provider, even the critical function of replicating data from a complex billing application in order to provide continuous uptime can cause significant technology challenges due to the volume of the data sets.

Specialized databases
However, new database technologies are emerging that promise to solve this problem. Specialized databases with the ability to significantly de-duplicate and reduce this data to a much smaller footprint are being deployed by providers. When you consider a 100-terabyte system reduced to a storage footprint of 5 terabytes, you can begin to imagine the cost savings not only in storage but also hardware capacity. Imagine a petabyte environment reduced to 100 terabytes. These specialized data-retention databases -- built for the purpose of storing data online for many years -- can also perform fast queries using standard SQL or various business intelligence and analytics solutions. With a database solution focused on Big Data retention that enables significant cost reduction, CSPs can store critical customer activity data for much longer periods -- even beyond what is required by regulators. By having access to larger historical data sets, CSPs can gain greater insights into customer behavior and even discover new opportunities to further segment the market and create new services with a wider range of price points.

Today's R&D teams now employ data scientists who dissect and mine usage data to detect behavior patterns in order to predict key market drivers such as what causes churn. Being able to predict customer behavior is much more powerful to the business than analyzing what happened and why it happened after the fact! Once churned, it costs the business as much as six times more to win that customer loyalty back. Those data scientists don't need SQL statements or business intelligence tools -- they must have continuous access to the raw data, the detailed historical xDR data, in order to determine market and behavior patterns. Giving them a dedicated Big Data retention database is the best way to achieve that. By looking inward and discovering new technologies and new approaches to managing core customer and transaction data, CSPs can discover new service offerings -- even new markets.

The ocean of Big Data faced by today's CSP does not have to reach a boiling point. Applying innovative data-retention database capabilities across the infrastructure will significantly reduce cost, and help manage scale and complexity.

— Dr. Hossein Eslambolchi is the former CTO of AT&T and chairman and CEO of 2020 Venture Partners

(0)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View
More Blogs from Column
Bigger. And Better. But definitely bigger.
When trying to develop innovative technologies, engineers must be willing to take risks, make mistakes and move ahead incrementally without having all the data.
Mobile network operators have the network and a business model that makes them well suited to lead the charge as the Internet of Things (IoT) takes off.
Gathering useful information for real-time management of Ethernet and IP networks is a non-trivial issue.
There are three important questions service providers need to address for in-market and out-of-market expansion.
Flash Poll
From The Founder
The New IP is actually bigger even than business. Like another hugely important tech that Light Reading is digging into right now, the New IP has the potential to change the world by fundamentally advancing what it is possible for people to achieve with communications.
LRTV Huawei Video Resource Center
Huawei Shares Its Vision of the Future of Mobile Networks Innovations

2|26|15   |   2:30   |   (0) comments


Mobile broadband is changing our lives. It's reshaping the Internet, industry, and society. It allows us to freely connect with one another anytime, anywhere. At this year's Mobile World Congress, Huawei will share its latest insights and newest ideas and technologies that will shape the future of MBB. They will showcase their end-to-end MBB solutions that will ...
LRTV Huawei Video Resource Center
Accelerate Digitizing, Boost Digital Business

2|26|15   |   6:14   |   (0) comments


A new digital revolution is leading us to a better connected world. Together with millions of digital partners, Huawei will help CSPs to build their digital service ecosystem and aggregate a wide variety of digital services. In this video, we find out how Huawei is going to help CSPs implement digital operations.
LRTV Huawei Video Resource Center
The Secret Recipe to Enabling Hyper-Growth Industries

2|26|15   |   3:38   |   (0) comments


With a number of successful cases on network capability exposure, Huawei is going to share the secret recipe to enabling hyper-growth markets with a step-by-step approach.
LRTV Documentaries
BTE 2015 Is Bigger & Even Better

2|25|15   |   03:13   |   (3) comments


This year's Big Telecom Event (BTE) in Chicago is going to provide more opportunities than ever for networking, getting to grips with key industry challenges and opportunities and, equally as important, having some fun.
LRTV Interviews
Light Reading ICT Leaders Roundtable at MWC 2015

2|12|15   |   1:07   |   (2) comments


On Sunday March 1, 2015, Light Reading will host an ICT Leaders Roundtable in partnership with Huawei. At this half-day event, CIOs, analysts and researchers will discuss key industry trends like virtualization in the cloud with a specific focus on new business models. Located at the luxurious Renaissance Hotel near the Fira Barcelona, space is limited so please ...
LRTV Documentaries
Going Green in 2015

2|12|15   |   02:04   |   (0) comments


Energy efficiency is set to be an incredibly hot topic in the telecom industry this year.
LRTV Custom TV
SDN & NFV: Where Are We Going From Here?

2|11|15   |   11:27   |   (0) comments


Vitesse Semiconductor CTO Martin Nuss gives his perspective on why SDN and NFV should be tightly interconnected and how he sees the industry moving forward.
LRTV Documentaries
Time for Gigabit Europe?

2|9|15   |   01:27   |   (4) comments


Gigabit broadband networks are springing up all around the US and they'll soon become more commonplace in Europe.
LRTV Interviews
Brocade Brings New IP Vision to 2020 Vision Executive Summit

2|3|15   |   4:23   |   (0) comments


In December 2014, Light Reading gathered telecom executives in Reykjavik, Iceland to discuss their vision for high-capacity networks through the end of the decade. The intimate, interactive meeting was set against the backdrop of Iceland's spectacular natural beauty. As one of the event's founding sponsors, Brocade's Kelly Herrell shared his company's strategy at ...
LRTV Interviews
Brocade's Kelly Herrell on the New IP

2|2|15   |   12:36   |   (0) comments


In December 2014, Steve Saunders sat down with Brocade VP of Software Networking Kelly Herrell at Light Reading's 2020 Vision executive summit in Reykjavik, Iceland. They spoke about Brocade's approach to the New IP, the future of the telecom industry, and more.
LRTV Huawei Video Resource Center
Dr. Dong Sun Talks About Carriers' Digital Transformation & Huawei’s Telco OS

1|29|15   |   6:28   |   (0) comments


Dr. Dong Sun, Chief Architect of Digital Transformation Solutions at Huawei, discusses how telecom operators can become digital ecosystem enablers and deliver optimal user experiences that are in real-time, on-demand, all-online, DIY and social (ROADS).
LRTV Huawei Video Resource Center
Huawei's Chief Network Architect Talks about Network Experience & Operators’ Strategies

1|29|15   |   3:39   |   (0) comments


In the digital age, network experience has become the primary productivity especially for telecom operators. In this video, Wenshuan Dang, Huawei’s Chief Network Architect, discusses how carriers can tackle the challenge of infrastructure complexity in order to enhance business agility and improve user experience.
Upcoming Live Events
March 17, 2015, The Cable Center, Denver, CO
April 14, 2015, The Westin Times Square, New York City, NY
May 12, 2015, Grand Hyatt, Denver, CO
All Upcoming Live Events
Infographics
Net neutrality, broadband services and the current outlook on data consumption, as presented by the New Jersey Institute of Technology.
Hot Topics
Cyber Security Expert Warns: You're Doing It Wrong
Sarah Thomas, Editorial Operations Director, 2/23/2015
10 Weirdly Useful IoT Devices
Eryn Leavens, Copy Desk Editor, 2/24/2015
Small Cells Enabling Location Services
Sarah Thomas, Editorial Operations Director, 2/25/2015
Is FCC Weighing Net Neutrality Changes?
Mari Silbey, Independent Technology Editor, 2/25/2015
MWC: Let the Madness Begin
Ray Le Maistre, Editor-in-chief, 2/23/2015
Like Us on Facebook
Twitter Feed
Webinar Archive
BETWEEN THE CEOs - Executive Interviews
Check out Light Reading's interview with Jay Samit, the newly appointed CEO of publicly traded SeaChange International Inc. With a resume that includes Sony, EMI, and Universal, Samit brings a reputation as an entrepreneur and a disruptor to his new role at the video solutions company. Hear what he had to say about the opportunities in video, as well as the outlook for cable, telco, OTT and mobile service providers.
G'day! And welcome to an entirely new feature on Light Reading -- our weekly "CEO-to-CEO" interview.