← All Articles

Using Rails with SES, SNS and SQS to avoid bounce rate

Khash SajadiKhash Sajadi
Oct 13th 23

Using Rails with SES, SNS and SQS to avoid bounce rate

Amazon Simple Email Service (SES) is a cost-effective email service provided by AWS. It is by far the cheapest option available out there. Comparing the cost of sending emails with SES and other services like SendGrid or Mailchimp, it can be 100x cheaper. However there is a catch. Using SES directly, you will not get some of the features you might need to control the bounce rate of your emails. In a perfect world, you'd want to monitor bounce rate of your emails and take them out of your email list so no further email is sent to those recipients.

Luckily, AWS provides a few services that combined together can help you with monitoring and automatic your bounce rate.

SES

There are many posts and articles that explain how to setup SES, so in this post I'm not getting into that. Here, I'm assuming you have SES setup and you are able to send emails with it.

Setup SES and SNS

The first step is to setup Simple Notification Service (SNS) to receive notifications from SES. SES can send notifications to SNS when an email is delivered, bounced or rejected.

First, create a topic in SNS

  1. Go to AWS SNS and click on the Create Topic button.
  2. Select "Standard" as the type of topic. (SES doesn't support FIFO) type.
  3. Choose a good name for the topic. For example, "ses-bounces".
  4. Under Access Policy, choose Advanced and paste the following JSON as your policy, with the described replacements.
  5. Under Delivery Policy, set Minimum delay to 20 seconds and Maximum delay to 20 seconds.
  6. Click "Create Topic" button.

The following JSON is the policy that allows SES to send notifications to SNS. You need to replace the following values in the JSON:

{
  "Version": "2012-10-17",
  "Id": "notification-policy",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ses.amazonaws.com"
      },
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:REGION:ACCOUNT:TOPIC",
      "Condition": {
        "StringEquals": {
          "AWS:SourceAccount": "ACCOUNT",
          "AWS:SourceArn": "arn:aws:ses:REGION:ACCOUNT:identity/IDENTITY"
        }
      }
    }
  ]
}
  • Replace REGION with the region you are using. For example, us-east-1.
  • Replace ACCOUNT with your AWS account number.
  • Replace TOPIC with the name of the topic you created in the previous step.
  • Replace IDENTITY with the identity you are using in SES. For example, foo@example.com. (this should be an SES verified identity)

Now, setup notifications for SES

Head to AWS SES page and select the verified identity you want to setup notifications for. Then select Notifications tab and add your new SNS topic for both Bounce and Complaint notifications.

Optional: You can select this topic for Delivery notifications as well so you can track deliveries.

Up to here, we have a verified SES identity that can send emails, and we have a SNS topic that can receive notifications from SES. Now we need to setup a way to process these notifications and take actions based on them.

Setup SQS

Simple Queue Service (SQS) is the service to use here. SQS is free for up to 1 million requests per month. So it's a good fit for many use cases without any cost.

On the AWS dashboard, go to the SQS page.

  1. Create a new queue by clicking on Create Queue button.
  2. Select "Standard" as the type of queue.
  3. Choose a good name for the queue. For example, "ses-bounces".
  4. Set the Delivery Delay to 20 seconds.
  5. Set the Message Retention Period to 4 days.
  6. Leave all the rest the same.
  7. Click on Create Queue button.

Now click on the newly created queue and click on the Subscribe to Amazon SNS topic button. Select your SNS topic and click on Save button.

At the top of your SQS details page, take note of your queue URL. It looks like this:

https://sqs.us-west-2.amazonaws.com/111122223333/ses-bounces

A quick summary before we move on:

  • A verified SES identity that can send emails.
  • A SNS topic that can receive notifications from SES.
  • A SQS queue that can receive notifications from SNS.

Now we need to head to our Rails app and write some code that fetches the delivery, bounce and complaint notifications from SQS and take actions based on them. We've set the message retention to 4 days, so we have 4 days to process the notifications. This means your Rails app worker can be down for up to 4 days and you won't lose any notifications.

Fetching notifications from SQS in Rails

Your Rails app, needs to have some sort of background worker triggered by CRON or a similar service. In this example, I'm using Sidekiq and SidekiqCron. You can use CRON, Clockwork or any other means you prefer. The important thing is to have a worker that runs every few minutes and fetches the notifications from SQS. We are going to setup our worker to run every 5 minutes.

class EmailDeliveryReportsWorker
  include Sidekiq::Worker

  QUEUE_URL = 'https://sqs.us-west-2.amazonaws.com/111122223333/ses-bounces'
  REGION = 'us-west-2'
  MAX_NUMBER_OF_MESSAGES = 10
  IDLE_TIMEOUT = 60

  def perform
    receive_queue_url = QUEUE_URL
    poller = Aws::SQS::QueuePoller.new(receive_queue_url, { client: Aws::SQS::Client.new(region: REGION) })

    poller.poll({ max_number_of_messages: MAX_NUMBER_OF_MESSAGES, idle_timeout: IDLE_TIMEOUT }) do |messages|
      messages.each do |message|
        body = JSON.parse(message.body)
        message = JSON.parse(body['Message'])

        if message['notificationType'] == 'Bounce'
          # bounce type?
          bounce = message['bounce']
          bounce_type = bounce['bounceType']

          if bounce_type == 'Permanent'
            bounceSubType = bounce['bounceSubType']

            if bounceSubType == 'General'
              # we need to find the users by email and mark them as inactive
              bounce['bouncedRecipients'].each do |recipient|
                email = recipient['emailAddress']
                puts "Bounce to #{email} is permanent"

                user = User.find_by(email: email)
                user&.update!(email_delivery_status: 1) # no further emails
              end
            end
          end
        elsif message['notificationType'] == 'Complaint'
          # complaint type?
          complaint = message['complaint']

          # we need to find the users by email and mark them as inactive
          complaint['complainedRecipients'].each do |recipient|
            email = recipient['emailAddress']

            puts "Complaint to #{email} is permanent"

            user = User.find_by(email: email)
            user&.update!(email_delivery_status: 2) # no further emails
          end
        elsif message['notificationType'] == 'Delivery'
          delivery = message['delivery']
          delivery['recipients'].each do |recipient|
            puts "Delivery to #{recipient} successful"
          end
        end
      end
    end
  end
end

Let's quickly go through the code above. First, we are using the Aws::SQS::QueuePoller to fetch the messages from SQS. We are using max_number_of_messages to fetch up to 10 messages at a time. We are also using idle_timeout to wait for 60 seconds if there are no messages in the queue. This is to avoid unnecessary API calls to SQS.

With these settings, your job will run every 5 minutes and fetches as many as messages as it can (up to 10) at a time from SQS until the queue is drained. It will then wait up to 60 seconds for new messages to arrive. If no new messages arrive, it will exit. Since SQS marks the messages visible only to 1 worker for 20 seconds, if another worker starts at the same time you will not double process messages. You can also configure your Sidekiq or other worker framework to run only one instance of this worker at a time.

In my example, I am using a column called email_delivery_status in my users table to mark the users that have bounced or complained. You can use any other method you prefer to mark the users. My assumption then in the rest of my code is that a value of 0 (default for this column) means it is ok to send an email to this user. Any other value however, means that the user has bounced or complained and no further emails should be sent to this user.

class User < ApplicationRecord
  #...
  def can_send_email?
    self.email_delivery_status == 0
  end
  #...
end

Running in production

The code above, uses AWS official Ruby SDK. See how you can add it to your Rails app here: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-install.html

Like any other AWS service client, your Rails code needs to have access to your AWS credentials with the right access rights. This document can help you with that: https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/credentials.html

Going further

The SQS messages received by this code contain a lot more information about the email sent. If you have configured your SES identity to notify SNS with delivery notifications, you can also track delivery of your emails as well.

This article has some examples of SES delivery notification payloads: https://docs.aws.amazon.com/ses/latest/dg/notification-examples.html

If you want to see all of the possible values in the payload you can check out this page: https://docs.aws.amazon.com/ses/latest/dg/notification-contents.html

However, I noticed that some SQS payloads contain bounceSubType values that are not listed in this article, if you use SES subscription headers in your emails. (that's a topic for another post!).

Conclusion

Using SES can reduce your email bills by a great margin if you are coming from another provider. However, you need to be careful about the bounce rate of your emails. If you don't take care of the bounce rate, you might end up with a high bounce rate and your SES account might get suspended. Luckily, AWS has plenty of features and services to track deliveries, opens, clicks, subscription and channels as well a bounces and deliveries to help you with that.

Related Articles

← All Articles
Try Cloud 66 for Free, No credit card required