Ultimate Guide to Server Hardening for Kamal

What is Server Hardening?

Server hardening is securing a server by reducing its surface of vulnerability. This means minimizing the number of potential entry points for attackers by disabling unnecessary services, applying patches, and enforcing strong security controls.

If you use Kamal to deploy applications but you must harden your servers yourself. This is a guide on the steps to take to have a secure server.

The Basics of Server Security: Patch often and restart often. Patching helps with fixing known vulnerabilities. Restarting gets rid of possible resident malware from memory. But before we get there, there is a lot we can do to improve our server's security posture.

Here, I am splitting security into two parts: Day 1 and Day 2. Day 1 Security is what is needed to create a secure server. Day 2 Security is the set of practices to keep it secure as your environment, requirements and threats change.

This guide is written for an Ubuntu OS server. You might need to make minor changes if your distro is different.

Let's get started:

Day 1 Security

Things you need to do to create a secure server.

SSH Access Security

Disable password-based SSH access

sudo sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config

Disable DSA algorithm for SSH

sudo sed -i 's/^#HostKeyAlgorithms .*/HostKeyAlgorithms -ssh-dss/' /etc/ssh/sshd_config sudo sed -i 's/^#PubkeyAcceptedKeyTypes .*/PubkeyAcceptedKeyTypes -ssh-dss/' /etc/ssh/sshd_config

Disable default root user login

sudo sed -i 's/^PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config

Restart SSH

sudo systemctl restart sshd

Important: Any time you make a change to SSH, make sure to open a new SSH connection to the server before restarting the deamon.

Network Security

Install and configure UFW (Uncomplicated Firewall)

UFW is a wrapper around networking IP tables to make it easier to manage them.

sudo apt update && sudo apt install ufw sudo ufw default deny incoming sudo ufw default allow outgoing sudo ufw allow OpenSSH sudo ufw enable

Install and configure fail2ban

Fail2ban works by monitoring logs for authentication failures and banning offending IPs via firewall rules. It blocks potential threats by putting them in temporary "jails".

sudo apt install fail2ban sudo systemctl enable fail2ban sudo systemctl start fail2ban

Basic Usage and Jail Configuration

1. Verify fail2ban is running:

sudo systemctl status fail2ban

2. Check active jails:

sudo fail2ban-client status

To get details of a specific jail (e.g., sshd):

sudo fail2ban-client status sshd

3. Default configuration path: Jail configurations are stored in /etc/fail2ban/jail.d/. You can override defaults or define new jails here.

To create or edit the SSH jail configuration:

sudo nano /etc/fail2ban/jail.d/sshd.local

Example configuration:

[sshd] enabled = true port = ssh filter = sshd logpath = /var/log/auth.log maxretry = 5 bantime = 3600 findtime = 600

4. Restart fail2ban after making changes:

sudo systemctl restart fail2ban

Setup Dynamic DNS to Only Allow Your IP Address for SSH

If your home or office IP address changes periodically (as with most residential ISPs), Dynamic DNS (DDNS) allows you to point a domain name to your current IP. You can then configure your server's firewall to allow SSH access only from that dynamic domain. The alternative to this is to have unlimited access to SSH from the internet (not ideal), or at least restrict access by country (which usually works unless you live in a border town or your ISP does funky things with its allocated IPs, it happnes!)

1. Set up Dynamic DNS with a Provider (e.g., DuckDNS, No-IP)

Sign up at https://duckdns.org or https://noip.com
Follow instructions to install and configure the DDNS update client on your local machine

Example for DuckDNS on your home machine:

mkdir -p ~/duckdns cd ~/duckdns nano duck.sh

Paste the following, replacing yourdomain and yourtoken:

#!/bin/bash echo url="https://www.duckdns.org/update?domains=yourdomain&token=yourtoken&ip=" | curl -k -o ~/duckdns/duck.log -K -

Make it executable and run every 5 minutes:

chmod +x duck.sh (crontab -l ; echo "*/5 * * * * ~/duckdns/duck.sh") | crontab -

2. Configure UFW on the Server to Allow SSH Only from the DDNS IP

On your server:

# Get your current public IP and allow SSH only from that IP sudo ufw allow from $(curl -s ifconfig.me) to any port 22 # Deny all other SSH access sudo ufw deny 22

NOTE: this script lacks logging and error handling to cater for all different scenarios, so please use it carefully or you might be blocked out of the server.

Component Security

Ensure local binding for databases (example: PostgreSQL)

sudo sed -i "s/^#listen_addresses = 'localhost'/listen_addresses = 'localhost'/" /etc/postgresql/*/main/postgresql.conf sudo systemctl restart postgresql

Setup strong passwords or client certificate auth for databases

Use strong passwords in your pg_hba.conf configuration.
Consider using md5 or scram-sha-256 authentication.

Application Security

There are many good articles about writing a secure Rails application, which is outside the scope of this article.

Load Balancer / Web Server Security

CORS settings

CORS security is needed for all applications, but this example outlines it for Rails.

To securely configure CORS (Cross-Origin Resource Sharing) in a Rails application (as an example), you should define which origins are allowed to access your resources and what methods and headers are permitted.

1. Add rack-cors to your Gemfile:

gem 'rack-cors', require: 'rack/cors'

2. Create or edit the CORS initializer:

nano config/initializers/cors.rb

3. Secure example configuration:

Rails.application.config.middleware.insert_before 0, Rack::Cors do allow do origins 'https://your-frontend-domain.com' resource '*', headers: :any, methods: [:get, :post, :put, :patch, :delete, :options, :head], credentials: true, max_age: 600 end end

4. Notes for Secure CORS Configuration:

Use a specific domain instead of * in origins to prevent unauthorized access.
Only allow necessary HTTP methods.
Set credentials: true only if your frontend requires cookies or HTTP authentication.
Limit max_age to a reasonable value to reduce exposure from preflight request caching.

5. Secure HTTP Headers for Cookie Security: To protect cookies from cross-site attacks, configure the following settings in your Rails app:

In config/initializers/session_store.rb:

Rails.application.config.session_store :cookie_store, key: '_your_app_session', secure: Rails.env.production?, same_site: :strict, httponly: true

secure: true ensures cookies are only sent over HTTPS.
same_site: :strict prevents cookies from being sent with cross-site requests.
httponly: true makes cookies inaccessible to JavaScript, mitigating XSS attacks.

HTTPS and and Let's Encrypt

Kamal takes care of setting up a proxy and issuing SSL certificates using Let's Encrypt! 🎉

If you use cloud load balancers, you can benefit from using SSL certificates on the load balancer and terminating SSL on the load balancer. The rest of the traffic will flow through a private network (between the LB and your servers), reducing the load on your server CPUs. Many cloud providers (including AWS, Google and Azure) can take care of SSL certificates, renewal and SSL termination on the load balancer, so you can turn this feature off on Kamal.

Day 2 Security

Now that you have a secure server hosting your application, you need to have processes in place to keep it secure in the long run.

Keep Track of Your Firewall Ports

Use UFW to list ports:

sudo ufw status numbered

Keep a manual log (e.g., a file in /etc/firewall_rules.log) noting when ports were added/removed.

Monitoring and Logging

Install and Configure rsyslog

rsyslog is a powerful system logging daemon that collects, processes, and forwards log messages. It's the default logging system on Ubuntu and many other Linux distributions. rsyslog can handle logs from various sources including system services, applications, and remote systems, and can forward them to different destinations like local files, remote servers, or databases. It's highly configurable and supports multiple protocols, making it an essential tool for system monitoring and troubleshooting.

rsyslog is usually installed and running by default on Ubuntu. You can verify it with:

sudo systemctl status rsyslog

If it's not installed:

sudo apt install rsyslog sudo systemctl enable rsyslog sudo systemctl start rsyslog

Install and Configure logrotate

logrotate helps manage log file size by rotating, compressing, and deleting old logs. Having older log files on your server is a great asset to help find possible security breaches and evidence for their scope.

sudo apt install logrotate

Check the default configuration in /etc/logrotate.conf and directory /etc/logrotate.d/ for app-specific configs. To test:

sudo logrotate --debug /etc/logrotate.conf

Install AIDE

AIDE (Advanced Intrusion Detection Environment) is a file integrity checker. It takes a snapshot of system files and directories and compares them to detect unauthorized changes.

sudo apt install aide sudo aideinit sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db

Use AIDE to Check for Intrusions

To run a manual check and compare the current system state against the database:

sudo aide --check

This will output any files that have changed since the database was initialized.

Schedule Regular AIDE Checks and Email Alerts

Create a cron job to run AIDE daily and email the results:

sudo crontab -e

Add the following line:

@daily /usr/bin/aide --check | mail -s "AIDE Integrity Check" you@example.com

Make sure mailutils is installed and configured to send mail:

sudo apt install mailutils

This ensures you're alerted when unauthorized or unexpected file changes occur.

sudo apt install aide sudo aideinit sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db

User and Access Management

Create a non-root user

sudo adduser deployer sudo usermod -aG sudo deployer

Use sudo instead of root

sudo visudo # Ensure the following line exists: # %sudo ALL=(ALL:ALL) ALL

Restrict access by IP (example with UFW)

You can use the method described above for restricting SSH access to your IP address using Dynamic DNS.

sudo ufw allow from 192.168.1.0/24 to any port 22

Staff departure process

Ideally any staff member with SSH server access should have their own Linux account which can be removed if they leave the team.

Backups and Recovery

This is not technically a server hardening requirement, but it is obviously a good idea to have backups and it can help with recovery in case of a bad actor corrupting your data.

Automate backups with cron

crontab -e # Example daily backup job 0 2 * * * /usr/bin/pg_dump yourdb | gzip > /var/backups/yourdb-$(date +\%F).sql.gz

Encrypt and move backups offsite

gpg -c /var/backups/yourdb-2025-03-30.sql.gz rsync -avz /var/backups/ user@backup-server:/path/

Intrusion Detection and Response

OSSEC and Wazuh are powerful Host-based Intrusion Detection Systems (HIDS) that monitor your system for suspicious activities, file changes, and potential security breaches. OSSEC is the original open-source project, while Wazuh is a fork that has evolved into a more comprehensive security platform with additional features like log analysis, vulnerability detection, and cloud security monitoring. Both tools can detect rootkits, malware, and unauthorized system changes, but Wazuh offers a more modern interface and additional security features.

Install OSSEC or Wazuh (complex setup, see their docs)

# For basic detection, you can install chkrootkit sudo apt install chkrootkit sudo chkrootkit

Install OWASP modules for Nginx Kamal does not use Nginx and so it lacks protections recommended by OWASP. However it is worth taking a look at OWASP website to know the kinds of threats you might need to be aware of.

Summary

As you can see, there is a lot that goes into making and keeping a server secure. Some of these are not just a tool to install, but like practices to follow (like many other things in security).

If you prefer focusing on code and let someone else take care of deploying your application on your cloud servers, you can use Cloud 66!

If you have any suggestions for update or feedback to this post, please send them my way.