What's up? DIY Monitoring with Prometheus - part 2

Last time we only covered how to install Prometheus, but we didn't cover how to make sure it is always up, nor the firewall rules. The purpose of this series is not just to install Prometheus but to showcase some parts of Linux. For instance, in this one using service definitions and adding basic firewall rules.

Defining services

Ubuntu 16.04

The service definition of your Prometheus.

File path: /etc/systemd/system/prometheus.service

Content:

[Unit]
Description=Prometheus  
Wants=network-online.target  
After=network-online.target

[Service]
User=prometheus  
Group=prometheus  
Type=simple  
ExecStart=/usr/local/bin/prometheus \  
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target  

And for Node Exporter.

File path: /etc/systemd/system/node-exporter.service

[Unit]
Description=Node Exporter  
Requires=network-online.target

[Service]
User=node_exporter  
Group=node_exporter  
Type=simple  
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target  

Ubuntu 14.04

The service definition of your Prometheus.

File path: /etc/init/prometheus.conf

Content:

description "Prometheus service"

# automatically re-spawn for 10 times with 5 seconds interval and then it will not start again

respawn  
respawn limit 10 5

# To start when server boots

start on startup

# Start the service with prometheus user

exec su -c ' /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries' prometheus  

The service definition of Node Exporter.

File path: /etc/init/node_exporter.conf

Content:

description "node_exporter service"

# automatically respawn for 10 times with 5 seconds interval and then it will not start again

respawn  
respawn limit 10 5


# To start when server boots
start on startup

# Start the service with node_exporter user
exec su -c '/usr/bin/node_exporter' node_exporter  

Now that the services are defined and hopefully up, we can set the firewall rules:

If you are using Cloud 66 you only need to go to your stack page/networks & settings and then open the Prometheus port (default 9090) to your own IP.

Using ufw you need to run:

# For Prometheus
sudo ufw allow from <your IP>  to any port 9090

# For Node Exporter
sudo ufw allow from <Prometheus-server-IP>  to any port 9100  

Using iptables:

# For Prometheus
sudo iptables -I INPUT 1 -s <your-IP> -p tcp --dport 9090 -j ACCEPT

# For Node Exporter

sudo iptables -I INPUT 1 -s <Prometheus-server-IP> -p tcp --dport 9100 -j ACCEPT  

How to find your IP?

In your terminal run (this is really useful for using in scripts):

curl ifconfig.me  

or

dig +short myip.opendns.com @resolver1.opendns.com  

Or search for "what is my ip" in your browser

Next stop setting up Grafana the beautiful dashboard!



Catch up with Part 1: What's up? DIY Monitoring with Prometheus.

Try Cloud 66 for Free, No credit card required