Topology automation results in a whopping 50% resource savings. "We can run the same number of services on half the hosts once we manage policy this way," Ihde says.
LinkedIn has been building LPS for the past two years. Rain is already in production, and additional elements will roll out later this year and in 2017.
Like other cloud providers, including Facebook and Google (Nasdaq: GOOG), LinkedIn chose to build its infrastructure internally rather than buy from vendors. (See Facebook Reinvents Data Center Networking and Google: 'Great' Data Center Networks Essential.)
"LinkedIn operates at a scale that is larger than most," Ihde said. "This put us in a position where many of the available commercial and open source solutions didn't work for us, or weren't designed to solve the problems we were trying to solve. That was really the primary motivation for building LPS." LinkedIn uses a mix of open source and home-built components.
Likewise, outsourcing to an external cloud provider -- as, for example, Netflix Inc. (Nasdaq: NFLX) has done with Amazon Web Services Inc. -- is also not an option. "We can operate this ourselves more efficiently than purchasing services from a public cloud provider," Ihde says. "We want to build our data centers as quickly as we can and we feel like we're the ones best placed to do that." (See Netflix Cloud Casts Long Shadow Over Cable .)
Security was key to the project -- specifically, preventing services from interfering with each other when sharing resources. "We were looking at it from the efficiency angle, but as we went on we realized that to make the things work and operational we needed to isolate those processes from each other as much as possible," Ihde says.
LinkedIn also found it challenging to optimize CPU resource utilization. "There is a tradeoff between making maximum use of idle capacity -- if there are CPUs cycling on a machine, why not offer them to any process that wants them? – versus consistency and predictability," Ihde says. "If the idle capacity varies over time, that impacts predictability. That's been a challenging tradeoff to work through."
Using standardized APIs helped integrate components into the overall architecture.
LinkedIn believes its experience is applicable to other companies that rely on software written internally for competitive advantage, whether in the Internet industry or some other vertical. "If you have sizable operations running your own software, this is where the approach makes the most sense," he said.
The LPS transition complements bringing online a new data center and data center architecture -- Project Altair.
"Project Altair represents for LinkedIn the transition from operating as a large enterprise that has a significant data center foot print to an environment more like a mega-data-center," LinkedIn's Bachar says.
Next Page: Stock Drop