Linux server needs a RAM upgrade? Check with top, free, vmstat and sar.

Sometimes it can be a bit of a challenge knowing if and when you should upgrade the RAM (random access memory) on your Linux server. Even more so when deciding how much memory you should add, or, if you have adequate memory, how do you make best use of it? This article will walk you though performing command line checks on Linux servers using a few real-world examples. Also included, is brief advice to help clarify your path forward if you are currently facing Linux server memory management choices.

Have a look at the first screenshot below, based on the output of ‘free -m’ on a 1GB VPS and taking into consideration the amount of swap space in use (details further below), a RAM upgrade was necessary. Required was a minimum of 3GB so we upgrade to 6GB. This ensures that there’s adequate memory available for buffers/cache. After the memory upgrade (around 9:30am), here’s how load averages responded:

Here are three additional before and after memory graphs of used vs free memory:

Checking if your Linux server needs a memory upgrade

Has your Linux server been acting slower than usual lately? Or maybe you’ve noticed higher CPU usage with the same traffic levels? Sometimes server performance can change overnight. Which could be due to a bunch of reasons such as increased web traffic, change to your database size/queries, your application or network, to name a few. To check the current state of your Linux server’s memory usage, you’ll want to use one or more of the following command line tools: free, top, vmstat and sar.

free command: check server memory and swap

The free command displays the total amount of free and used physical and swap memory in the system, as well as the buffers used by the kernel. It’s a more user-friendly method of reading the output of cat /proc/meminfo.

The free command, although simple, used to be one of the most misinterpreted Linux command line tools. However, as of Kernels 3.14 (and emulated on kernels 2.6.27+) the Linux free command made a few changes. Some of which helped admins realize that they have a lot less free or available memory than they may have once thought. A vast amount of blog posts, QAs, articles and server monitoring tools advise that your Linux server’s Free Memory = Free + Buffers + Cached = WRONG!

Using: free -m (-m = display the amount of memory in megabytes). The screenshot below is from the same 1GB memory Stacklinux VPS from the screenshots above. Here’s a breakdown of whats being reported:

991 – total: Total installed memory (MemTotal and SwapTotal in /proc/meminfo)
628 – used: Used memory (calculated as total – free – buffers – cache)
61 – free: Unused memory (MemFree and SwapFree in /proc/meminfo)
180 – shared: shared Memory used (mostly) by tmpfs (Shmem in /proc/meminfo, available on kernels 2.6.32, displayed as zero if not available)
301 – buff/cache: Memory used by kernel buffers (Buffers in /proc/meminfo) / cache: Memory used by the page cache and slabs (Cached and Slab in /proc/meminfo)
35 – available: Estimation of how much memory is available for starting new applications, without swapping. Unlike the data provided by the cache or free fields, this field takes into account page cache and also that not all reclaimable memory slabs will be reclaimed due to items being in use (MemAvailable in /proc/meminfo, available on kernels 3.14, emulated on kernels 2.6.27+, otherwise the same as free).
6015 – swap: total amount of swap memory.
1522 – swap used: amount of swap memory that is in use.
4493 – free swap: amount of swap memory that is not in use.

Older versions of free look something like this:

free -m
             total   used  free  shared  buffers  cached
Mem:           x      x      x       x       x      x
-/+ buffers/cache:    x    x
Swap:          x      x    x

Notice the extra buffers/cache line. To see this buffers/cache split in the latest 3.14+ Kernel versions of free, use the ‘-w’ option. For example: free -mw, -hw, -w, etc. When you type cat /proc/meminfo the first 3 lines are: MemTotal, MemFree and MemAvailable. On older Linux installs that third line (MemAvailable) isn’t listed.

So how much free memory do I have then?

You will always have some free memory on Linux. Even with constant swap I/O you’ll notice some free, buffered and cached memory. The Linux Kernel will use swap space (if swap is enabled) to maximize as much as possible the amount of free or available memory. As per the screenshot above there’s over 300MB of ‘buff/cache’ and 61MB marked as ‘free’. But over 1.5GB of swap. If you do not have a swap enabled on your server then you’ll notice poor performance, low ratio of buffers/cache, system freezes or worst. On a server with swap space, the best indicator that memory is low will be high frequency of swap I/O. In this case there’s 1GB of RAM but 1.5GB of swap and kswap (kernel service which manages swapping) is using most of CPU time.

Bonus: Check out the command: slabtop

top: check server memory and swap usage

In the video above I’ve used top to have a look at memory + swap usage, notice kswap and the ‘wa’ spike together. These spikes also show clearly that the performance of the web server is being directly hampered because of kswap swapping to disk swap. As mentioned, kswap is a process used by Linux to manage swap on the system, while ‘wa’, is I/O wait or time that the processor/processors are waiting during which there are outstanding I/O requests. Lets look at this more closely using vmstat and sar. Unlike the free and top commands, the remaining tools are consistently documented online.

vmstat: check server memory and swap usage

vmstat reports information about processes, memory, paging, block IO, traps, and cpu activity. Lets use vmstat 1 as per the above screenshot. Notice the si (swap in), so (swap out) and wa (wait) columns. Swap I/O is not always bad, opportunistic swapping can actually help with performance. However, a high frequency of swap i/o appears on almost every line (every second). Here’s what the same command looks like on a healthy server.

Pictured in the screenshot you can see even with 1.3G of free RAM the server also has 246M of used swap. This is from opportunistic swapping. Or swap performed during idle moments where it does not impact server’s performance negatively. Usually this 246M is data that has not been used for a very long time. Continue reading for how you can increase overall cache that the Linux server holds on to without increasing the frequency and size of swap cache.

sar: check server memory and swap usage

With sar you can collect, report or save system activity information. Like vmstat, sar also can be used for a lot more than checking memory and swap. The above screenshot is also from the same 1GB VPS we started with.

Compare these results with the sar -r results on a healthy Linux server:

Kernel cache pressure and swappiness

Another method of squeezing the most from your server memory, is to tune the system’s swappiness (tendency to swap) and cache pressure (tendency to reclaim cache).

swappiness – This control is used to define how aggressive the kernel will swap memory pages. Higher values will increase aggressiveness, lower values decrease the amount of swap. (default = 60, recommended values between 1 and 60) Remove your swap for 0 value, but usually not recommended in most cases.

vfs_cache_pressure – Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. (default = 100, recommend value 50 to 200))

To edit you can add or replace these lines in /etc/sysctl.conf file. For example,if you have low memory, until you upgrade you can try something such as:

vm.swappiness=10
vm.vfs_cache_pressure=200

This will increase the cache pressure and that may seem somewhat counter productive since caching is good for performance. However, swapping also reduces your server’s overall performance significantly more. So not keeping as much cache in memory will help reduce swap/swapcache activity. Also, with vm.swappiness set to 10 or as low as 1, will reduce disk swapping.

On a healthy server with lots of available memory use the following:

vm.swappiness=10
vm.vfs_cache_pressure=50

This will decrease the cache pressure. Since caching is good for performance, we want to keep cache in memory longer. Since the cache will grow larger we still want to reduce swapping so that it does not cause increased swap I/O.

To check current values using these commands use:

sudo cat /proc/sys/vm/swappiness
sudo cat /proc/sys/vm/vfs_cache_pressure

To enable these settings temporarily without rebooting use the following commands:

sudo sysctl -w vm.swappiness=10
sudo sysctl -w vm.vfs_cache_pressure=50

Here’s cat /proc/meminfo output of both servers side by side:

Notice the healthy server on the left, the ‘SwapCached’ is low even with a low vm.vfs_cache_pressure setting of 50.

Update: Here are some screenshots of the server @ 6GB RAM, then 1GB and then finally back to 6GB:

And finally the outputs of post-upgrade free and top:

Kernel buffer on VMs – Notice on the VPS the kernel buffer is zero. This usually should not be the case. VirtIO is being used as the storage BUS driver/interface and a lot of the disk cache handling is handed off to the KVM hypervisor.  The buffers are for disk operations. So we’ll see that number increase when the disk write cache needs to be written out. You can see the I/O on a StackLinux VPS is quite fast (Also see: Your Web Host Doesn’t Want You To Read This: Benchmark Your VPS)

Originally posted: Jul 25, 2017 | Last update: July 21st, 2021