Dashboard Downtime

The issue

Today at around 4am GMT our dashboard and some API services became unresponsive for a group of our customers. This issue was limited to a group of customers at the beginning but grew to affect more customers within 4 hours. We managed to track down the issue to an external script related to cloud health status and fix it by 8:14GMT.

This issue manifested itself by not being able to start new deployments from the UI or commandline (Redeployment stayed functional).

What happened?

The root of the problem was a faulty script on our side which was trying to retrieve public cloud status information from http://status.cloud66.com/status.json. This page was available on our old status page, but not on the new one. This script was responsible for showing the "Cloud Status" on top of every dashboard page as well as some information on the API payloads.

On the 20th of July (7 days ago) we switched our status page to status.io to provide more detailed information about our product incidents and maintenance windows.

The issue with the missing endpoint was not picked up during development or tests on our test and staging environments because status.io was returning a 404 immediately which our script was dealing with normally. However, in the early morning hours of Monday, status.io started to block the repeated calls to the wrong endpoint from our production IP addresses, causing the script to timeout.

Cloud status is cached aggressively at different stages for different purposes and different customers depending on which cloud providers they use. This means many customers were getting the cached values and were not affected until their cache started to invalidate and they started to experience the issue.

The fix

We have switched the source for the cloud status to a different source and are monitoring its performance.

Prevention

We are now reviewing our dependency to outside API endpoints and working on measures to protect us from any unpredictable behaviour from the endpoint.

We are sorry for this dashboard/API outage and are working hard to make sure it doesn't happen again. We have reached out to all customers affected publicly and privately to make sure everything is in order. If you are experiencing issues with the dashboard, please let us know.

Khash Sajadi

Khash is the founder and CEO of Cloud 66, a full stack container management as a service. Follow him on @khash

London, UK and San Francisco, US
Subscribe and get updates

Have feedback? Please get in touch @cloud66 on Twitter.

Everything you need to build, manage and maintain containers in production on your own servers and any cloud

Try Cloud 66 — 14 Days Free Trial, No credit card required