After AWS and DigitalOcean, Hetzner Cloud is the third popular cloud provider with our customers. At the time of writing, around 9% of our customers deploy their applications on Hetzner.
Our customers love their simple and straight forward product and pricing as well as their affordable bare-metal options. Over the past few months, many of our Hetzner customers reached out to us with complains of deployment failures. In this post I will try to explain the root cause of these failures in the hope of improving this problem for all of our customers and the larger Hetzner customer base.
tl;dr;
Over the past few months, many of our Hetzner customers reached out to us with complains of deployment failures. In this post I will try to explain the root cause of these failures in the hope of improving this problem for all of our customers and the larger Hetzner customer base.
In simple terms, these deployment failures are caused by connection issues between our servers and some Hetzner servers, depending on their IP address.
In short, Hetzner cloud is a popular cloud provider with some entities in Iran linked to the government that are trying to subvert US and international sanctions by diverting Iran originated traffic via Hetzner's German data centers.
When discovered, the IP address of the servers used to subvert the sanctions is linked to Iran and subsequently blocked by Google (our main cloud provider) and other major providers (AWS and Azure as well as others).
The scale of the issue
While we don't have visibility over the entire Hetzner cloud operations, our anecdotal customer issues and internal tests suggests up to around 10% of German Hetzner IPs are used or have been previously used by Iranian sanction busting efforts. We assume that the Iranian entities are constantly switching their servers after being blocked, and potentially increasing the scope of the problem if new IPs are added to the block list faster than they are removed by US companies (GCP, etc) following Iranian sanction laws.
Context
As you can imagine, having been in this business for more than 12 years, we are not unfamiliar with connectivity issues across data centers and cloud providers and have developed useful tools to detect, investigate and sometimes mitigate these issues. However, the particular circumstances surrounding Hetzner where not something we expected to face when we started looking into these issues. Also international US sanction laws are not within everyday support routine of any cloud provider and therefore any new instance of this issue has to be investigated by 2nd or 3rd level support staff within each cloud provider.
After having run many tests on traffic flow between our servers and affected servers, we managed to limit the scope of the issue to specific IP address and shared our findings with the cloud providers in question. Ultimately and after many back and forth support sessions and further escalation of the issue higher in the chain, it turned out that all of the affected IPs are part of blocked IPs that have been linked to Iranian sanctions.
Over the past few years, there have been various well documented stories of Iranian cloud providers and government entities using German data centers for traffic redirection and sanction busting (see here) so once we discovered this connection, we were not very surprised. What was surprising was first the scope of the issue and second, Hetzner's seemingly lax attitude about their responsibility towards all of their customers.
How does this affect you?
The first thing to say here is that while some of our customers are affected by this issue, the scope of this is bigger than only Cloud 66 customers. If we are right in our assumption that around 10% of Hetzner's German IPs are blocked by other cloud providers due to this issue, any traffic to and from those servers will be blocked by the network providers resulting in any domain linked to any of these IPs being inaccessible from major providers. Moreover, we think that being linked to a blocked IP address might inadvertently link the new users of those IPs to block list resulting in further loss of business, reputation and legal trouble for them. We are not lawyers but while we can assume these issues are resolvable, the potential temporary harm they can cause for businesses is not negligible.
What are we doing to improve things?
We are working with Google, Hetzner and other cloud providers to find quicker ways to first identify affected IPs and secondly remove them from block list when the original malicious actor has moved on from the IP address. However, ultimately these lists are usually created and complied by businesses that are trying to protect themselves against violating US sanction laws and therefore there is no central place for these block lists or procedure to remove them from the list. An IP blocked by a provider might be unblocked by another. In this regard, the three major cloud providers (AWS, Azure and Google) are the most aggressive ones we found.
What can you do if you are blocked?
If you have connectivity issues with your servers on Hetzner, please reach out to us and to your Hetzner support representative. Feel free to cite this post if it helps with adding context to your support ticket. Based on our experience, this issue is limited to Hetzner's German data centers, so moving your servers out of German data centers or out of Hetzner can help with this issue. You can also try recycling your IPs for affected servers, although you might get another blocked IP address.