The Beanstalk

2013-05-19

Managed Backups with Single Click Restore

Today we are announcing a major new functionality on Cloud 66: Managed Backups with Single Click Restore.

With managed backups you can backup all of your app databases backed up with on a regular basis and restore them with a single click.

The backups are stored offsite on a highly available and reliable storage managed by Cloud 66 and you can restore your database backups with a single click.

Setting up the backup takes less than 10 seconds and is very simple:

You can see the backups icon on your stack.

After this, backups run at the scheduled times automatically:

Restoring your database to a single backup point is as simple as clicking Restore.

Your app will be reconfigured to work with the restored database automatically.

Managed backups cost $0.20 per GB per month and support MySQL, PostreSQL, MongoDB and Redis for both backup and Single Click Restore. All of your app’s databases will be included in each backup automatically to create a restore point across the whole stack.

2013-05-15

Introducing Cloud 66 iOS app

We have some awesome customers. They support us, help us with our new features and even build us iOS apps!

As of today, Cloud 66 iOS app is available from AppStore for all iOS devices. With Cloud 66 App you can see your stacks, their status and detailed information about them.

You can even redeploy your stacks from your phone and get iOS notifications when the build and deployment is finished!

We are so fortunate to have customer like you. This app was built by Gabriel Rinaldi (@gabriel_rinaldi), CTO @ CoffeeTable who is an awesome mobile developer and a really nice guy! Gabriel is offering the app for free to all Cloud 66 customers!

2013-05-10

Update 3 on Cloud 66 Major Service Incident

Dealing with security issues is always tricky, and not only in the technical sense. There is a lot more going on.

But first, we would like to share with you what we know:

We were the target of a malicious security exploit on our front-end application.
Our service was disrupted on 7th of May and a big portion of our customers were affected by that.
We found a security issue on our front-end website exposing parts of users’ cloud API to an authenticated external party.
We don’t keep our customers payment details so they could not be compromised.
The attackers did not gain access to our code, our databases, our customers’ code, SSH and Git keys or our systems.
All of our systems are back up and running and the security exploit has been fixed.

What happened?

Between 16:00 and 17:00 UTC of 7th of May our systems started notifying us and our customers about customer servers shutting down. These servers were across all cloud vendors we support: Amazon, Rackspace, Linode, Digital Ocean, Joyent and Telefonica. We immediately shutdown our systems and cut off our network from public access as a precaution. Digital Ocean also noticed an increase in the number of delete requests and disabled parts of their API and reset user API access codes.

At this point a majority of our customers were affected by shutdown or terminated servers. We set these priorities in the following order:

Preventing any unaffected customers from being exposed to this issue.
Helping our customers to get back online
Restoring the service back to normal

Once we ruled out human error in the matter, we disabled all destructive actions (such as server deletes) throughout the system as a precautionary measure. We also started a security sweep of our production servers, firewalls and other systems.

At the same time, we asked our vendors to provide us with detailed API logs for our investigation.

Many of our users had database backups setup through Cloud 66 stored offsite which they used to restore their service as fast as they can from restored images.

After our initial security sweep, disabling destructive actions and asking our customers to reset their cloud API keys, we restored the service so the rest of our customers could bring their applications up while monitoring access closely.

Within the next 24 hours we helped 90% of the affected customers to bring their systems back up and running while we were going through every line of code and every line of tons of log files from our servers and external entities until we found what we believe to be the source: An issue in our front-end code could be exploited by an authenticated user to retrieve parts of other user’s cloud API keys.

We also found evidence of access to our website prior to the initial disruption that could be linked to the exploit in the front-end code. Given this new information, we immediately took the following actions before running another security sweep on the code and infrastructure as well as addressing the security exploit in the front-end code and increasing the security on the site.

Pulled the front-end application off public internet.
Informed our cloud vendors.
Informed all of our users and asked them to change their API access keys.

This was purely a precautionary measure to ensure security of our customers and was not triggered by any evidence of a separate attack or exploit.

We believe the initial security exploit allowed the perpetrators to obtain enough information about customer cloud API keys to shut down customer’s servers.

What next?

First and foremost, we would like to sincerely apologise for any inconvenience this incident may have caused to our customers. As a gesture of goodwill we have waived all customer fees for the month of May.

As a team of developers, we started Cloud 66 to help fellow developers; and nothing hurts us more than failing that mission. We have put our heart and soul into this, and this has been an unbelievably hard couple of days. Thank you to all our customers who have lent us their support - it is humbling.

We believe that transparency is the best approach in dealing with issues like this and have tried to be as open and transparent as possible, while ensuring the safety and security of our customers during an ongoing investigation.

Reporting security issues

If you have any concerns regarding Cloud 66 security, you can share it with us at cloud66.com/security

2013-05-08

Update 2 on Cloud 66 Major Service Incident

Yesterday we had a major service disruption starting around 16:00 UTC affecting many of our customers.

Since noticing the issue, we have had two main priorities:

Helping our affected customers to go live as quickly as possible.
Protecting the rest of our customers from being affected.

So far most of our affected customers have been able to recover from this incident. The most successful were the ones whom had an off-site database backup. For those we could redeploy their full stack and redirect traffic to the new servers very quickly.

Our challenge has been mostly dealing with customers that had no backup or local backups of their databases and we are working hard to restore as many instances as possible.

To protect our customers, we also shutdown the service for around 1:30 hours last night. Although this slowed the recovery for the customers but we felt it is necessary to take this measure to protect our users.

What do we know:

None of your payment details have been compromised in this incident. We don’t store them on our servers.
Our security measures, firewalls and production access rules protect us from human error on our side in cases like this.
All critical information in our databases are encrypted, including cloud API keys.
We are still investigating the issue in detail. We are working with our customers and cloud vendors to establish critical facts about the incident and will keep you updated as we progress.

What can you do?

Contact us at hello@cloud66.com if you need help with restoring your stack.
Setup off site backups of your databases.

What are we doing?

We are working around the clock to restore the remaining customer stacks.
We have temporarily disabled server deletion. This means you’d need to manually delete the server from your cloud vendor side if you need to.
We are working with our cloud vendors and other parties following our investigations into the root of yesterdays incident.
We are improving different parts of the service to ensure improved reliability and stability of the system. We will keep you updated about these soon.

If you need help or assistance with your deployment and servers let us know at hello@cloud66.com

2013-05-07

Major Service Incident

Update on the incident today.

Firstly apologies to all for the delay in getting this update out - we’ve been at battle stations since the incident focusing on capturing/recovering everything we can.

At approximately 17:27 UTC we started getting notifications of servers being down (we have an server component that pings us every minute, and alerts if it hasn’t pinged for 10 minutes).

So, approximately <10 minutes before that point a number of our users’ virtual servers received API server destroy requests.

Since that time we have been through the code many many times, testing every combination/permutation of action that could have occurred, however up until now we have not been able to identify any possible way that our code could have inadvertently caused the server API calls.

We have not had any code releases around this time and nothing has changed on our prod servers.

We’ve also been going through the logs and can find no obvious signs of any security breach.

At this point we’ve taken following actions:

- Disabled ALL server deletion API code on our servers

- Tightened our server security by setting up additional firewall rules - we did already have firewall security in place.

As a preventative measure we recommend that all users changes change their API keys for their cloud vendors (DigitalOcean users API keys have been updated already)

If your stack has been affected by this, and you’re on on DigitalOcean then they will have restored your server image. You can restore your server from that.

If your stack has been affected by this, and you’re not on DigitalOcean, then at this point it is looking like the only way to retrieve your server is if you’ve taken an image snapshot, or database backups from which you can restore.

Needless to say that we are aware of the severity of this incident and are doing everything we can to get to the bottom of it. We will put out an additional update as soon as we have more information.

Cloud66 Crew

2013-04-30

Say Hello to Rails 4!

We are please to announce support for Rails 4 and Ruby 2 on Cloud 66.

You can include the ruby version in your Gemfiles as well as targeting a specific Rails version.

The rest of the deployment, monitoring and features are exactly the same.

Happy Coding!

2013-04-29

Digital Ocean San Francisco Data Center Support

Last week we enabled supporting Digital Ocean San Francisco data center.

You can now enjoy the quick and painless deployment, monitoring and management features of Cloud 66 for your Rails apps on Digital Ocean West Coast node as well.

Enjoy!

2013-04-23

Temporary Shell Access to your Servers

We all know it: it’s not best practice to access your production servers directly through shell, but there are times we have to do it.

Today we are announcing a new feature to make those occasions safer: Temporary shell access rights to your servers.

To avoid the dangers of forgetting to close your shell ports after you’re done, you can now use access leasing from the security page.

Just click on the plug icon and the form is prefilled with your current IP address. You need to select a time period (10 mins or 20 mins) and the port will be only open for that period of time.

After that the ports will be closed and the access rights will be revoked.

As you well know, it is best not to use this on production servers, but for other environments and those emergency occasions, use this feature to stay safer.

2013-04-17

Protect your Ruby on Rails Apps from Cloud 66 Videos on Vimeo.

2013-04-15

Riding Unicorns: We now support custom Rack servers

Another week and another feature that was on top of our most request features list is now complete: Support for other Rack servers like Unicorn, Thin and Puma.

By default we serve your Rails application with Phusion Passenger, but from today, if you would like to use Unicorn or any other Rack servers you can do that.

The process is as simple as putting a line in your Procfiles that is labeled as custom_web and following a couple of conventions to ensure traffic is redirected to your Rack server and Cloud 66 can control the process. Those are simple enough to integrate into your Unicorn configuration.

Please note: This feature is only available to new stacks.

You can find more detailed information about Unicorn support in Cloud 66.