Cloudflare Bleeds Bad News – & Good

Curtis Franklin
2/27/2017
100%
0%

"For the want of a nail the shoe was lost, For the want of a shoe the horse was lost, For the want of a horse the rider was lost, For the want of a rider the battle was lost, For the want of a battle the kingdom was lost, And all for the want of a horseshoe-nail."
— Benjamin Franklin

Substitute the ">" symbol for an old-fashioned nail, and you have a synopsis of the latest privacy catastrophe, dubbed "Cloudbleed" by security researchers. How bad is Cloudbleed? The advice to users echoed around the Internet gives a hint of the breach's scope:

Change your passwords. All of them.

What happened to cause the problem and how it has been addressed each carry instructions for enterprise security and IT professionals. What happened is simple, and the sort of coding error that can be frustratingly difficult to catch absent the most rigorous testing protocols. In a single line of code, the intended ">=" operator was typed "==". As a result, certain operations were able to fill a buffer and keep right on writing, planting data across the system.

The flaw
According to Tavis Ormandy, a researcher with Google Project Zero, that planted data can include cookies, private messages, IP addresses, and passwords. The worst part? In a blog post about the event (made public only after the vulnerability had been repaired and remediated), Ormandy wrote, "PII [personally identifiable information] was actively being downloaded by crawlers and users during normal usage, they just didn't understand what they were seeing."

Cloudflare is a service provider with a client list that includes Uber, OKCupid, Fitbit, and various financial institutions. In addition to basic network performance services that include DNS and load balancing, Cloudflare provides security capabilities such as SSL and DDoS mitigation. Because Cloudflare provides the service to businesses rather than consumers, it's nearly impossible for an individual to know whether their personal information was exposed. And it's absolutely impossible to know whether any exposed information was actually grabbed by criminals -- until, of course, some form of identity theft takes place. Hence, the advice to change all your passwords. Now.

The problem was compounded by the way that Cloudflare operates. Cloudflare's data can be crawled and cached by services such as Google. This means that personal data was not just exposed, it was exposed in a persistent manner. That's a lot of bad news wrapped up in a single wayward character. Fortunately, there is some good news in the story to balance a bit of the bad.

Rapid response
When Ormandy discovered the issue, he immediately contacted Cloudflare using Twitter (in, as one commenter noted, the sort of message that should be absolutely horrifying to anyone on a corporate security team). In very short order, Cloudflare security personnel had reproduced the issue and shut down the services making use of the affected code. According to the company, they then put teams in the US and UK on 12-hour shifts, handing code off between the teams to keep efforts to find and remediate the problem going 24 hours a day.

Cloudflare reports that all services that included the problem were disabled globally within roughly three hours of notification. Figuring out precisely where in the code the problem lay, and how it could be repaired, took a bit longer. The company's blog post on the issue (referenced above) goes into great detail about where the problem was, what its effects were, and why it had not previously appeared. The level of detail in the disclosure is part of the good news surrounding the vulnerability; in speaking with several security researchers and IT professionals, all said that this disclosure is a model of what they would like to see in future industry events. All felt that this level of disclosure is likely to assuage customer fears and reassure regulators that all proper steps have been taken.

The final step in remediation was Cloudflare working with search engines to clear caches of released information. As the company wrote in its disclosure blog post, "With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains. The leaked memory has been purged with the help of the search engines."

Six days after the initial contact from Ormandy, Cloudflare disclosed the incident to the public. The vulnerability had existed for approximately four months and the number of customer records revealed to criminals is unknown. What is known is that this will likely be used as an example of the white-hat research system working as intended; independent discovery of an issue, verification by the affected company, rapid repair and remediation, with speedy, transparent disclosure to the public.

— Curtis Franklin, Security Editor, Light Reading

(12)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
Page 1 / 2   >   >>
Curtis Franklin
100%
0%
Curtis Franklin,
User Rank: Blogger
3/6/2017 | 5:09:56 PM
Re: Ah ha
KBode I think that fingerprints are the most common biometrics in current use. While I don't see them going away, recent developments make me think that voice-print and facial recognition will each play a role going forward, with smart mobile devices acting as the hardware in many deployments.

I know the arguments about the vulnerabilities in each of these, but I don't see any of them as any weaker than the current password regime and I don't think that we should let the search for perfection keep us from moving forward toward the much better.
KBode
50%
50%
KBode,
User Rank: Light Sabre
3/6/2017 | 10:32:23 AM
Re: Ah ha
Pardon my ignorance as I'm not up to date on the subject. But what's the most commonly accepted biometric marker? Retinal scan and fingerprint? You think they're both the future?
Curtis Franklin
50%
50%
Curtis Franklin,
User Rank: Blogger
3/6/2017 | 10:05:12 AM
Re: Ah ha
KBode I think we end up going with some sort of biometric marker as the primary factor with something like a PIN as the second. I like multi-factor authentication and think that biometrics are the way to go. I'm aware of spoofing mechanisms but I believe that, overall, they're still more potentially effective than passwords.
KBode
50%
50%
KBode,
User Rank: Light Sabre
3/4/2017 | 7:41:33 AM
Re: Ah ha
Password managers can deal with this somewhat, but I think you're right. What's your preferred password alternative down the line?
Curtis Franklin
50%
50%
Curtis Franklin,
User Rank: Blogger
3/3/2017 | 3:42:19 PM
Re: Ah ha
KBode, at RSA I had someone tell me that they try to have their employees refresh their passwords weekly. I wished them all the best with this. I think the problem isn't so much with how often we roll over to new passwords as with the dependence on passwords as the ultimate form of authentication. it's time we graduated to something better.
Michelle
50%
50%
Michelle,
User Rank: Light Sabre
2/28/2017 | 9:09:34 PM
Re: Ah ha
@Curtis All true! Supply chain and vendor access are indeed security sensitive relationships. Connecting to these folks can be dangerous for your own network. 
Michelle
50%
50%
Michelle,
User Rank: Light Sabre
2/28/2017 | 9:06:26 PM
Re: Ah ha
@KB I feel the same way. I wonder if my accounts would be more secure if I changed the password every day ;)
KBode
50%
50%
KBode,
User Rank: Light Sabre
2/28/2017 | 2:59:28 PM
Re: Ah ha
Good point. And at this point I just feel like I need an automated system to change all my passwords every three weeks. :)
Curtis Franklin
100%
0%
Curtis Franklin,
User Rank: Blogger
2/27/2017 | 3:13:50 PM
Re: Is "Change All Your Passwords" really a CYA for Cloudflare?
bosco_pcs, I agree that this is terrible for most customers. The problem is that there isn't a great alternative now that the vulnerability has been exposed.

All of this is one of the reasons that I have two-factor authentication enabled with every service that offers it. Two-factor is also a pain, but it's the best current solution I know of.
Curtis Franklin
50%
50%
Curtis Franklin,
User Rank: Blogger
2/27/2017 | 3:11:31 PM
Re: Ah ha
Thank you, Michelle. And you're right: This is a stark reminder that your "trust network" extends not only to your partners, but to all of their partners, and so on, down the line. 

From what I see this isn't something that will hit most enterprise internal sites, but since so many employees re-use passwords for everything they touch, it's worth sending out notices to everyone, just in case.
Page 1 / 2   >   >>
Featured Video
Flash Poll
Upcoming Live Events
September 17-19, 2019, Dallas, Texas
October 1-2, 2019, New Orleans, Louisiana
October 10, 2019, New York, New York
October 22, 2019, Los Angeles, CA
November 5, 2019, London, England
November 7, 2019, London, UK
December 3, 2019, New York, New York
December 3-5, 2019, Vienna, Austria
March 16-18, 2020, Embassy Suites, Denver, Colorado
May 18-20, 2020, Irving Convention Center, Dallas, TX
All Upcoming Live Events
Partner Perspectives - content from our sponsors
Ryan Ding From Huawei: Industries + 5G, Enabling New Growth
By Ryan Ding, Executive Director of the Board, Huawei Technologies
Adaptive MIMO in the Era of 6GHz Wi-Fi
By James C. Chen, Quantenna
All Partner Perspectives