Update 3 on Cloud 66 Major Service Incident
Dealing with security issues is always tricky, and not only in the technical sense. There is a lot more going on.
But first, we would like to share with you what we know:
- We were the target of a malicious security exploit on our front-end application.
- Our service was disrupted on 7th of May and a big portion of our customers were affected by that.
- We found a security issue on our front-end website exposing parts of users’ cloud API to an authenticated external party.
- We don’t keep our customers payment details so they could not be compromised.
- The attackers did not gain access to our code, our databases, our customers’ code, SSH and Git keys or our systems.
- All of our systems are back up and running and the security exploit has been fixed.
Between 16:00 and 17:00 UTC of 7th of May our systems started notifying us and our customers about customer servers shutting down. These servers were across all cloud vendors we support: Amazon, Rackspace, Linode, Digital Ocean, Joyent and Telefonica. We immediately shutdown our systems and cut off our network from public access as a precaution. Digital Ocean also noticed an increase in the number of delete requests and disabled parts of their API and reset user API access codes.
At this point a majority of our customers were affected by shutdown or terminated servers. We set these priorities in the following order:
- Preventing any unaffected customers from being exposed to this issue.
- Helping our customers to get back online
- Restoring the service back to normal
Once we ruled out human error in the matter, we disabled all destructive actions (such as server deletes) throughout the system as a precautionary measure. We also started a security sweep of our production servers, firewalls and other systems.
At the same time, we asked our vendors to provide us with detailed API logs for our investigation.
Many of our users had database backups setup through Cloud 66 stored offsite which they used to restore their service as fast as they can from restored images.
After our initial security sweep, disabling destructive actions and asking our customers to reset their cloud API keys, we restored the service so the rest of our customers could bring their applications up while monitoring access closely.
Within the next 24 hours we helped 90% of the affected customers to bring their systems back up and running while we were going through every line of code and every line of tons of log files from our servers and external entities until we found what we believe to be the source: An issue in our front-end code could be exploited by an authenticated user to retrieve parts of other user’s cloud API keys.
We also found evidence of access to our website prior to the initial disruption that could be linked to the exploit in the front-end code. Given this new information, we immediately took the following actions before running another security sweep on the code and infrastructure as well as addressing the security exploit in the front-end code and increasing the security on the site.
- Pulled the front-end application off public internet.
- Informed our cloud vendors.
- Informed all of our users and asked them to change their API access keys.
This was purely a precautionary measure to ensure security of our customers and was not triggered by any evidence of a separate attack or exploit.
We believe the initial security exploit allowed the perpetrators to obtain enough information about customer cloud API keys to shut down customer’s servers.
First and foremost, we would like to sincerely apologise for any inconvenience this incident may have caused to our customers. As a gesture of goodwill we have waived all customer fees for the month of May.
As a team of developers, we started Cloud 66 to help fellow developers; and nothing hurts us more than failing that mission. We have put our heart and soul into this, and this has been an unbelievably hard couple of days. Thank you to all our customers who have lent us their support - it is humbling.
We believe that transparency is the best approach in dealing with issues like this and have tried to be as open and transparent as possible, while ensuring the safety and security of our customers during an ongoing investigation.
Reporting security issues
If you have any concerns regarding Cloud 66 security, you can share it with us at cloud66.com/security