Posted: March 6, 2023 | %t min read | by Nathan Lager (Sudoer, Red Hat)

Performance is one of those things that many system administrators dread. Sometimes you don't even think about it when things are running well. Then, suddenly you get a call from an end user, or even worse, lots of end users, that the application you're responsible for "feels slow." Or maybe it's outright unavailable. Now you have to go into troubleshooting mode.

Where do you start, though? In this article, I'll walk through some basics, but first, I'll take a step back. It's important to have some baseline information about your system before trying to identify problems. Maybe your system is doing something it shouldn't be, and that is contributing to the high load. Or perhaps it's doing exactly what it always has, and it's just under some additional load. Performance troubleshooting starts before a problem occurs by keeping good documentation and historical performance data.

A recent episode of Into The Terminal examines performance troubleshooting. This article summarizes three utilities that provide great initial performance information and introduces the concepts covered by the video. If you'd rather, you can skip straight to the video.

Start at the top

The first go-to performance troubleshooting tool is top. The top utility gives you a great, constantly updated process and performance dashboard. How much memory is in use? What's the load average? What processes are using the most resources? All of this is ready at a moment's notice in top.

top - 10:55:50 up 6 min,  1 user,  load average: 0.03, 0.18, 0.10
Tasks: 188 total,   1 running, 187 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.6 us,  0.0 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3654.6 total,   1477.3 free,    700.3 used,   1752.4 buff/cache
MiB Swap:   3280.0 total,   3280.0 free,      0.0 used.   2954.3 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 netns
      6 root      20   0       0      0      0 I   0.0   0.0   0:00.01 kworker/0:0-xfs-c+
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-even+
      8 root      20   0       0      0      0 I   0.0   0.0   0:00.16 kworker/u8:0-flus+
      9 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq
     10 root      20   0       0      0      0 I   0.0   0.0   0:00.36 kworker/u8:1-even+
     11 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_kthre
     12 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_rude_
     13 root      20   0       0      0      0 I   0.0   0.0   0:00.00 rcu_tasks_trace
     14 root      20   0       0      0      0 S   0.0   0.0   0:00.03 ksoftirqd/0
     15 root      20   0       0      0      0 I   0.0   0.0   0:00.41 rcu_preempt
     16 root      rt   0       0      0      0 S   0.0   0.0   0:00.00 migration/0
     17 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker/0:1-xfs-c+
     18 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
     19 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1

Additionally, you can filter top, search for processes, or even view the utilization of each core in your system. So, your first stop should be top!

[ Learn how to manage your Linux environment for success. ]

Are you free?

The free tool displays the system's current memory consumption, and it's a great second stop in your journey to discover what's eating at your system.

               total        used        free      shared  buff/cache   available
Mem:         3742352      717408     1512544        5312     1794520     3024944
Swap:        3358716           0     3358716

Remember that the Linux kernel will take available memory and allocate it to disk buffers and cache. Linux will release those resources when the system needs them, so if free suggests that you have no available memory, be sure to check that buffers/cache column before you conclude that you're out of memory.

Check vmstat

Using vmstat is another great way to look at your memory consumption.

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 1511816   9684 1784948    0    0   233   676  256  262  3  2 95 0  0

It breaks down the buffers and cache, and you'll also get information about your swap utilization. Swapping is generally a sign that your system is exhausting its higher-performing memory, and it must resort to swapping to disk. This process can have a devastating effect on some applications.

[ Keep your most commonly used commands handy with the Linux commands cheat sheet. ]

Take a deeper dive

Check out this episode of Into The Terminal for a deeper dive into performance troubleshooting with these tools and more.

[ Want to test your sysadmin skills? Take a skills assessment today. ]