& cplSiteName &

The 'Big Data' Challenge

Column
Column
Column
12/5/2011
50%
50%

"Big Data" was a much-discussed concept in IT management circles last year. But the effect of Big Data -- the non-stop massive growth of information -- has been a challenge for service providers' data infrastructure and technology teams for quite some time.

Tier 1 providers have experienced billions of call-detail records (CDRs) every day for years. In response, custom database technologies are now being developed and refined. Petabyte-scale database systems holding years of call-detail records and Internet protocol detail records (IPDRs) are now considered the norm at many Tier 1 providers.

The Big Data effect has proven even more complex and challenging with the advent of more stringent compliance regulations. A single global provider must proactively configure, manage and retain specific subscriber records for a designated time period. Additionally, the regulatory regimes of different nations impose different timeframes, requiring different data types in different formats. Specific response times to access that data vary across borders as well. In the past year alone, Japan and India have imposed new regulatory initiatives upon providers, regardless of size and subscriber level.

Fulfilling lawful intercept requirements demands substantial investments by providers in storage, database and search technologies -- and, of course, trained personnel with the skills necessary to deploy these solutions. Lawful intercept involves both real-time monitoring and access to data records. For example, law enforcement may obtain judicial warrants to tap a suspect's landlines, cellphones and email accounts, and procure copies of their call, SMS/MMS and other communication records. Having this data readily available in the correct format and delivered in a timely manner creates ongoing pressure to make the data center run efficiently and cost effectively. The possibility of legal fines only compounds the pressure.

Today communications service providers (CSPs) must innovate to achieve scale and better IT economics. Interoperability with other global partners, providers and suppliers is important but not enough. The CSP must be prepared to use technology, network architecture and standardization to deal with these pressures.

A recent Gartner industry report, Predicts 2011: CSPs Must Rethink Business Paradigms to Meet Market Challenges, cited falling margins of CSPs in mobile and fixed combined services sectors in development markets. There are very few exceptions to this trend. The causes of the drop include Internet Protocol (IP) substitution, lower average revenue per unit (ARPU) from new customers, increasing competition, regulators getting a better view of non-competitive price elements and the need to invest in expensive broadband networks with long payback cycles.

Improving the customer experience and service level is absolutely a necessary part of staying ahead. New partnerships are formed daily to provide new applications and mobile capabilities to drive new channels of revenue. However, all of the front-end customer-facing products and services that are rolled out will only be sustainable if back-office and data infrastructure are capable of robust management and increasing scale in the coming years.

A service provider IT issue
Let's consider the data problem from the perspective of today's IT team.

Within the life-cycle of a single customer's communication "transactions" in a given billing cycle, core data (essentially call data detail records and Internet Protocol detail records) that are captured, collated and stored are immediately historical in nature -- once the "transaction" takes place, the data does not undergo any further change. This is quite different from traditional business transactions that are captured in an online transactional database system, such as a retail purchase. This customer communication data is then processed by a series of applications that allow various IT and business functions, including network capacity planning, traffic analysis, user trending, revenue assurance, billing, digital advertising and more.

Some of these OSS and BSS applications leverage the same database system; in other cases, the data is moved around with varying degrees of latency. Data sub-sets are formed depending on application function and requirements, and data warehouses are fed with customer activity in order to perform trending analysis and determine the overall value of a customer at any point in time. Currently, many CSPs retain much of this data in a traditional database such as Oracle. For many of these functions, the data does not need to change; therefore it can be stored in a database whose primary purpose is long-term retention.

Given the growth in data volume and complexity across diverse systems and applications, organizations are forced to retain the data for as little time as possible, largely because systems have grown bloated -- to the point where the sheer size of the data harms performance. For a large provider, even the critical function of replicating data from a complex billing application in order to provide continuous uptime can cause significant technology challenges due to the volume of the data sets.

Specialized databases
However, new database technologies are emerging that promise to solve this problem. Specialized databases with the ability to significantly de-duplicate and reduce this data to a much smaller footprint are being deployed by providers. When you consider a 100-terabyte system reduced to a storage footprint of 5 terabytes, you can begin to imagine the cost savings not only in storage but also hardware capacity. Imagine a petabyte environment reduced to 100 terabytes. These specialized data-retention databases -- built for the purpose of storing data online for many years -- can also perform fast queries using standard SQL or various business intelligence and analytics solutions. With a database solution focused on Big Data retention that enables significant cost reduction, CSPs can store critical customer activity data for much longer periods -- even beyond what is required by regulators. By having access to larger historical data sets, CSPs can gain greater insights into customer behavior and even discover new opportunities to further segment the market and create new services with a wider range of price points.

Today's R&D teams now employ data scientists who dissect and mine usage data to detect behavior patterns in order to predict key market drivers such as what causes churn. Being able to predict customer behavior is much more powerful to the business than analyzing what happened and why it happened after the fact! Once churned, it costs the business as much as six times more to win that customer loyalty back. Those data scientists don't need SQL statements or business intelligence tools -- they must have continuous access to the raw data, the detailed historical xDR data, in order to determine market and behavior patterns. Giving them a dedicated Big Data retention database is the best way to achieve that. By looking inward and discovering new technologies and new approaches to managing core customer and transaction data, CSPs can discover new service offerings -- even new markets.

The ocean of Big Data faced by today's CSP does not have to reach a boiling point. Applying innovative data-retention database capabilities across the infrastructure will significantly reduce cost, and help manage scale and complexity.

— Dr. Hossein Eslambolchi is the former CTO of AT&T and chairman and CEO of 2020 Venture Partners

(0)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
Educational Resources
sponsor supplied content
Educational Resources Archive
More Blogs from Column
WiFi is offering a challenge to the network-centric cellular status quo and that's something that mobile network operator CEOs recognize, believes Devicescape CEO Dave Fraser.
NFV can bring operational headaches as well as operational gains, argues Andy Huckridge.
5G is about so much more than just very high bandwidth and low latency – and SDN is going to play a key role in enabling 5G full potential, argues ONF Executive Director Dan Pitt.
To ensure the best possible customer experience, service providers must find ways to make more performance metrics available to users.
How cable operators can prepare to grasp the mobility opportunity.
From The Founder
Download our complete guide to de-risking NFV deployment in 2016, including:
  • An eight-step strategy to deploying NFV safely, based on input from the companies that have already started virtualizing their production networks.
  • Interviews with leading executives at Colt, AT&T, Deutsche Telekom, Cisco, Nokia, ZTE, Ericsson and Heavy Reading.
  • Flash Poll
    Live Streaming Video
    Prepping for the Future: Upskill U Explained
    During this short kick-off video, Doug Webster, Vice President of Service Provider Marketing, Cisco, and Light Reading’s CEO & Founder Steve Saunders give an overview of Upskill U.
    LRTV Interviews
    Demand Surges for On-Demand Ads

    5|5|16   |     |   (0) comments


    Ed Knudson, VP, Product and Strategy at Canoe Ventures, discusses the rising appeal of VoD ads and the challenges on inserting ads dynamically in live TV programming.
    LRTV Custom TV
    Exclusive: Video Interview With Sckipio CEO David Baum

    5|5|16   |     |   (0) comments


    At his headquarters in Tel Aviv, G.fast visionary David Baum, CEO of Sckipio, provided an exclusive interview to Light Reading and showed how innovations in rate and reach, vector densities, fast reconnecting times and SFP-based residential gateways are expanding the potential of G.fast.
    Telecom Innovators Video Showcase
    Atrinet's NetACE – Migration to NFV & SDN With NetOps-Driven LSO

    5|4|16   |     |   (0) comments


    At Atrinet's headquarters, Ray Le Maistre sits down with Roy Silon to get an in-depth look into the company's focus and the secret recipe for their success.
    LRTV Huawei Video Resource Center
    Amsterdam ArenA, Powered by Huawei

    5|4|16   |     |   (0) comments


    Huawei's ICT solutions power the state-of-the-art Amsterdam ArenA, turning it into a smart stadium.
    LRTV Interviews
    Testing When There's No 'There' There

    5|4|16   |     |   (1) comment


    The benefits of SDN/NFV are well known, but the transition comes with some challenges, prominent among them is: how do you test a network that has been abstracted and has the potential to be endlessly reconfigurable? Light Reading was at NFV World Congress in Santa Clara, Calif., where we bumped into Mats Nordlund, CEO and co-founder of Netrounds, a Swedish ...
    LRTV Interviews
    Ditching the Slash & the Orchestration Wars

    5|3|16   |     |   (2) comments


    SDN and NFV have been inextricably bound with each other for so long that on a conceptual level, smooshing them together into one catch-all phrase – SDNFV – is now justifiable, according to Dan Pitt, executive director of the Open Networking Foundation (ONF). Light Reading spoke to Pitt at the NFV World Congress, where he explained that the next ...
    LRTV Custom TV
    ZTE TV Connect Highlights

    5|3|16   |     |   (0) comments


    ZTE gives us a tour of its booth and new products at TV Connect in London.
    LRTV Custom TV
    Deluxe's Unified Delivery Solution

    5|3|16   |     |   (0) comments


    Join Alan Breznick of Light Reading and visit the Deluxe booth at NAB! Here you'll find Deluxe's Unified Delivery Solution, OTT video, virtual reality, HDR, 4K and much more!
    LRTV Interviews
    Verizon Puts Gray Boxes in the Shade

    5|2|16   |   04:33   |   (1) comment


    When it comes to the white box trend, "gray" boxes, which have a slight proprietary twist, don't give service providers and end users the advantages they're seeking, according to Verizon's Vice President of Product and New Business Innovation Shawn Hakl.
    LRTV Custom TV
    Dealing With a Disrupted Video Market

    5|2|16   |     |   (0) comments


    Ericsson's Simon Frost discusses how traditional pay-TV providers can cope with the big changes wrought by the rise of OTT video and IP technology.
    LRTV Custom TV
    The VNF Responsibility of Red Hat

    5|2|16   |     |   (0) comments


    At MWC, Caroline Chappell of Heavy Reading visits the Red Hat booth and sits down with Chris Wright to talk about the responsibility the VNF needs to take on in order to ensure the operators get the carrier-grade performance they expect for their network.
    LRTV Interviews
    AT&T Expert on the Key Pillars of UC

    4|29|16   |   03:58   |   (0) comments


    Vishy Gopalakrishnan, AVP of product marketing at AT&T, talks about the three developments that are making unified communications and collaboration secure and reliable for enterprise users.
    Upcoming Live Events
    May 23, 2016, Austin, TX
    May 23, 2016, Austin Convention Center
    May 24-25, 2016, Austin Convention Center, Austin, TX
    September 13-14, 2016, The Curtis Hotel, Denver, CO
    December 6-8, 2016,
    June 16-18, 2017, Austin Convention Center, Austin, TX
    All Upcoming Live Events
    Infographics
    A new survey conducted by Heavy Reading and TM Forum shows that CSPs around the world see the move to digital operations as a necessary part of their overall virtualization strategies.
    Hot Topics
    WiCipedia: Woman Cards & Bitch Switches
    Sarah Thomas, Director, Women in Comms, 4/29/2016
    Sprint CEO: Our Spectrum Is for 5G
    Dan Jones, Mobile Editor, 5/3/2016
    Amazon & Other 'Big 4' Cloud Providers Crushing Competitors
    Mitch Wagner, West Coast Bureau Chief, Light Reading, 4/29/2016
    Rovi Reels in TiVo for $1.1B
    Mari Silbey, Senior Editor, Cable/Video, 4/29/2016
    Showdown at the OpenStack Corral
    Carol Wilson, Editor-at-large, 5/3/2016
    Like Us on Facebook
    Twitter Feed
    BETWEEN THE CEOs - Executive Interviews
    In this latest installment of the CEO Chat series, Craig Labovitz, co-founder and CEO of Deepfield, sits down with Light Reading's Steve Saunders in Light Reading's New York City office to discuss how Deepfield fits in with the big data trend and more.
    Grant van Rooyen, president and CEO of Cologix, sits down with Steve Saunders, founder and CEO of Light Reading, in the vendor's New Jersey facility to offer an inside look at the company's success story and discuss the importance of security in the telecom industry.
    Animals with Phones
    Sloth Mail Click Here
    Sloth mail -- somehow even slower than snail mail.
    Live Digital Audio

    Of all the tech companies in the Valley, Intel has made the most aggressive commitment to building a diverse and inclusive workplace culture. It's doing so by taking concrete, measurable steps, making a large financial investment and through a commitment to complete transparency about its progress. In this radio show, WiC Director Sarah Thomas will be joined by Shlomit Weiss, Intel's Vice President, Data Center Group, and General Manager of Networking Engineering, who will share with us why Intel is tackling this huge challenge, how and to what effect. She will also discuss her unique experiences leading development of Client SOC development in the past and today leading development of all of the chipmaker's silicon hardware for networking IPs and discrete devices and managing a team of 600 engineers across Israel, Europe and the US.