What's up? DIY Monitoring with Prometheus - part 2

Last time we only covered how to install Prometheus, but we didn't cover how to make sure it is always up, nor the firewall rules. The purpose of this series is not just to install Prometheus but to showcase some parts of Linux. For instance, in this one using service definitions and adding basic firewall rules.

Defining services

Ubuntu 16.04

The service definition of your Prometheus.

File path: /etc/systemd/system/prometheus.service

Content:

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

And for Node Exporter.

File path: /etc/systemd/system/node-exporter.service

[Unit]
Description=Node Exporter
Requires=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Ubuntu 14.04

The service definition of your Prometheus.

File path: /etc/init/prometheus.conf

Content:

description "Prometheus service"

# automatically re-spawn for 10 times with 5 seconds interval and then it will not start again

respawn
respawn limit 10 5

# To start when server boots

start on startup

# Start the service with prometheus user

exec su -c ' /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries' prometheus

The service definition of Node Exporter.

File path: /etc/init/node_exporter.conf

Content:

description "node_exporter service"

# automatically respawn for 10 times with 5 seconds interval and then it will not start again

respawn
respawn limit 10 5


# To start when server boots
start on startup

# Start the service with node_exporter user
exec su -c '/usr/bin/node_exporter' node_exporter

Now that the services are defined and hopefully up, we can set the firewall rules:

If you are using Cloud 66 you only need to go to your stack page/networks & settings and then open the Prometheus port (default 9090) to your own IP.

Using ufw you need to run:

# For Prometheus
sudo ufw allow from <your IP>  to any port 9090

# For Node Exporter
sudo ufw allow from <Prometheus-server-IP>  to any port 9100

Using iptables:

# For Prometheus
sudo iptables -I INPUT 1 -s <your-IP> -p tcp --dport 9090 -j ACCEPT

# For Node Exporter

sudo iptables -I INPUT 1 -s <Prometheus-server-IP> -p tcp --dport 9100 -j ACCEPT

How to find your IP?

In your terminal run (this is really useful for using in scripts):

curl ifconfig.me

or

dig +short myip.opendns.com @resolver1.opendns.com

Or search for "what is my ip" in your browser

Next stop setting up Grafana the beautiful dashboard!



Catch up with Part 1: What's up? DIY Monitoring with Prometheus.

Try Cloud 66 for Free, No credit card required