In this era of cloud computing, containers, serverless and being a cloud-native citizen there are still servers who need attention. If you are a full-stack developer, my humble opinion is part of your toolkit is know you way around Linux. This blog post is a collection of some very handy, ten to be precise, Linux commands to make your DevOps life easy.
Although most servers are Linux-centric, you can apply those commands also to your developer box if you run MacOS.
Win-Win.
NOTE: At Cloud 66 we provision your servers with Ubuntu 16.04
1. Access the server
Before you can even run a command on your server. You need to access the server. Best practice is to access your server using SSH with private key authentication and port (22) should only be accessible from your IP address.
If your are running MacOS, open your terminal and run the command:
$ ssh [username]@[ip-address-server] -i [path_to_private_key.pem]
Or if you are using cx our toolbelt:
$ cx ssh -s [stack_name] [server_type]
NOTE: Don't forget to open up your firewall is your are not using cx ssh
2. Superpowers
By default, you only got access your userspace. If you want to install something, change some system-wide settings or access restricted resources you need get those privileges.
Having elevated privileges all the time is dangerous! You can easily f$%k-up your system. That is why you should use sudo every time you run a potentially dangerous command.
For example; Killing a running process can be dangerous. Use this following line with cautious.
$ sudo kill -9 [process_id]
When your run the sudo command Linux will run your command in another process and your environment variables are not preserved. To make sure you preserve the environment variables, use sudo -E
With great sudo power comes great responsibility.
3. No life without a package manager
Some of the commands in this post are not installed by default and you should use a package manager to get those installed. The Ubuntu distro is equipped with the package manager called APT.
Before you install new packages make sure the package manager is up-to-date, run the update command first.
$ sudo apt-get update
To install a package, like htop, run the following command:
$ sudo apt-get install htop
Upgrading your installed packages.
$ sudo apt-get install upgrade
Read the manual of apt-get to known all the very nice features of APT package manager.
Your server needs attention
A server gives you the resources your need to run your application and those can be drained during the lifecycle of your application. Of course, you have monitoring systems in place to alert if the server run low on disk space, has high CPU utilization or lag because of network congestion. As a developer, it's always good to have some tools you can use for debugging and pin-point the bottleneck.
4. Where did my diskspace go?
The df (disk free) command reports the amount of available disk space being used by file systems. The du (disk usage) command reports the sizes of directory trees inclusive of all of their contents and the sizes of individual files.
The aim is to make sure you are not overshooting the 80% threshold. If you exceed the threshold it's time to scale or clean-up the mess, because running out of resources you have the change your application show some fickle behavior.
To check in a human readable format:
$ sudo df -h
But in most cases, you want to check which part of your system is consuming lots of disk space. Use the following command:
$ sudo du -h -d 1 /var/
You need to use du recursively to pin-point the directory which causing the most disk usage. After running du -h -d 1 /var/ maybe a subdirectory like /var/deploy using the most space. For inspection, run:
$ sudo du -h -d 1 /var/deploy
Or change the depth of recursing to 2, run:
$ sudo du -h -d 2 /var/
5. Who eat all my resources?
Often you want to know when your server is creating smoke due to heavy load which process consumes what? Hopefully is all due to the increase of happy customers hitting your service and you only need to scale.
A really nice tool which you can help to see how your process are consuming memory, cpu and which cores are under pressure, use htop.
Just run:
$ sudo htop
Hit the F5 key to show all process in a tree view, to debug the subprocesses.
6. PID to the rescue
Every process in Linux has a unique ID and can be seen using the command ps.
$ sudo ps aux
- a = show processes for all users
- u = display the process's user/owner
- x = also show processes not attached to a terminal
7. Show me the contents
Part of debugging your services is to check for anomalies in your log files. Sometimes you want to hit the endpoint of our service and check what is going on in your log file without scanning through the whole file. Use tail for this, it will output the last part of files.
$ tail -f /var/log/somelog.log
-f option will follow the file and when the file grows if will append to the console output.
To output a file or concatenate some files for outputting to the console, use cat
$ cat /var/log/somelog.log
8. Filter crap with grep
When showing contents (tail), scanning for anomalies (cat) or find the right process (ps aux) you need some kind of filter.
In Linux, you can combine two things to make life easier. A piece of pipe | and some grep. A | in Linux will send the output of the left-hand command to the right-hand command. You can create pipelines of commands to achieve your goal.
To show only the request with a HTTP 404 status codes:
$ sudo tail -f /var/log/somelog.log | grep 404
Getting the PID (process id) of a Ruby process:
$ sudo ps aux | grep ruby
grep is a very powerful tool and can filter/match lots of things to help you. Read the manual of grep to level-up your Linux skills.
9. Containers are using resources too.
Containers are great to run isolated processes and should also be treated like processes and not as a VM. Containers are using resources too, it's good practice to check how much CPU and memory they are using on a regular basis. This command helps you to show the status of your running containers.
$ sudo docker stats $(docker ps | awk '{if(NR>1) print $NF}')
Yes! This command shows the CPU load, memory usage and network utilization.
You can see some serious pipelining with awk. awk is a programming language designed for text processing and typically used for data extraction and reporting tool. It is a standard feature of most Unix-like operating systems.
10. Star Wars and some fortune
And after a very productive day of work your probably want to go home and watch a movie. If you don't want to leave your terminal unattended, why don't you want Star Wars from your terminal:
$ telnet towel.blinkenlights.nl
or you want some random fortune?
$ sudo apt-get install fortune
$ fortune
Happy coding and love Linux commands.