CloudFlare, a popular internet intermediary service that provides performance and security for a host of other sites, has had a large data leak.
Dubbed "CloudBleed", it made potentially sensitive information available online, including from popular sites like OKCupid and Authy.
What happened with Cloudflare?
From the CloudFlare blog:
Last Friday, Tavis Ormandy from Google's Project Zero contacted Cloudflare to report a security problem with our edge servers. He was seeing corrupted web pages being returned by some HTTP requests run through Cloudflare.
It turned out that in some unusual circumstances, which I'll detail below, our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data. And some of that data had been cached by search engines.
For the avoidance of doubt, Cloudflare customer SSL private keys were not leaked. Cloudflare has always terminated SSL connections through an isolated instance of NGINX that was not affected by this bug.
We quickly identified the problem and turned off three minor Cloudflare features (email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites) that were all using the same HTML parser chain that was causing the leakage. At that point it was no longer possible for memory to be returned in an HTTP response.
Because of the seriousness of such a bug, a cross-functional team from software engineering, infosec and operations formed in San Francisco and London to fully understand the underlying cause, to understand the effect of the memory leakage, and to work with Google and other search engines to remove any cached HTTP responses.
Having a global team meant that, at 12 hour intervals, work was handed over between offices enabling staff to work on the problem 24 hours a day. The team has worked continuously to ensure that this bug and its consequences are fully dealt with. One of the advantages of being a service is that bugs can go from reported to fixed in minutes to hours instead of months. The industry standard time allowed to deploy a fix for a bug like this is usually three months; we were completely finished globally in under 7 hours with an initial mitigation in 47 minutes.
The bug was serious because the leaked memory could contain private information and because it had been cached by search engines. We have also not discovered any evidence of malicious exploits of the bug or other reports of its existence.
The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that's about 0.00003% of requests).
We are grateful that it was found by one of the world's top security research teams and reported to us. This blog post is rather long but, as is our tradition, we prefer to be open and technically detailed about problems that occur with our service.
Don't iMore and Mobile Nations use CloudFlare? Are we affected?
iMore and MobileNations use CloudFlare, but we don't use any of the specific services from CloudFlare that were exposed as part of the leak. This is from the email they sent us earlier today:
Your domain is not one of the domains where we have discovered exposed data in any third party caches. The bug has been patched so it is no longer leaking data. However, we continue to work with these caches to review their records and help them purge any exposed data we find. If we discover any data leaked about your domains during this search, we will reach out to you directly and provide you full details of what we have found.
This is what Marcus Adolfsson, our CEO, posted earlier:
I just spoke with Tech ops and they confirmed that the three features causing the issue with CloudFlare (Email Address, Obfuscation, Server-side Excludes, Automatic HTTPS Rewrites) has never been active on our sites.
How do you know which sites were potentially affected?
Lists are being posted to Github, though it's tough to verify them at this point and some of the sites listed, like iMore, might not be using the specific services affected.
What do you need to do right now?
Change your passwords and make sure you use a different password for every site. There's no way to tell what information got out but you can be proactive about it.
Also, get a Password Manager like 1Password or Lastpass so you can have strong, unqiue passwords for every site. Then set up Two Factor Authentication wherever possible.
Any CloudBleed questions?
If you have any CloudBleed questions, drop them in the comments below!