Improving OpenVPN performance and throughput

OpenVPN, Wireguard, L2TP/IPSec, SSTP, IKEv2, PPTP, or other. If you had the luxury of choosing, which of these VPN protocols would you use? Therein lies my problem. In my current use case, I must find a way to improve OpenVPN performance and throughput.

You may have been following my Linux home lab build. One of the most important decisions when building your home lab is selecting the proper router/firewall for your network. In my case, after many hours of research, online comparisons, reading reviews, and watching Youtube videos, I went with the Edgerouter 10x (ER-10x). Note: this article includes my affiliate links, however, I only link to hardware and services that I’ve paid for and tested myself.

Well, last week, I figured out that it does not support Wireguard, at least not officially, as I recently discovered (my next to-do). This isn’t a knock on the ER-10x, not at all; it’s a remarkably capable router with many business-class features and, most important rock-solid stability.

That said, my VPN service provider of choice is ovpn.com. They have many locations, excellent performance, and offer dedicated IPs with open ports at $3/month. On routers, they support Wireguard or OpenVPN.

OpenVPN 2.4 internet speed test results using Google+M-Lab and wifiman.com.

The Edgerouter 10x is built on Debian Linux. This makes it a pleasure to work with because a lot of the functionality feels familiar. Over the past year, I’ve spent more time in the command-line and less time using the GUI.

Setting up OpenVPN is one of those command-line-only features. However, after downloading the .ovpn file and setting it up on the router, I soon hit a crippling OpenVPN limitation. The CPU! The ER-10x features 880 MHz CPU cores, which is overkill in the vast majority of cases. However, in this case, OpenVPN performance is not very efficient as throughput largely depends on the CPU’s core speed.

On my first speed test, download speeds were around 15 Mbps download and 12 Mbps upload. I needed a solid 20 Mbps down for the IoT devices connected to a VLAN that uses that VPN connection.

My basic setup at home currently is: dual WAN with backup 4G LTE ISP auto-failover, VLANs for isolated Guest WiFI network, and for the IoT devices both wired and wireless. For now, I’ll run with what I have; it works!

Improving OpenVPN Performance

OpenVPN config Screenshot from my Manjaro i3 SSH session with the router.

Note: I’ve already verified results when I initially set up everything a week ago using my Ubuntu server over higher LAN throughput with iperf.  As the hardware/CPU limits are so low on the Edgerouter, the ISP tests were very much representative of those tests. If you are using OpenVPN in a hardware-restricted setup, then give the following config tweaks a try. I’ll try to make the time and revisit this article with some redone iperf test results. If you have the time you can share your test results in the comments section below or by email using the “contact” link.

My ISP download speed is just over 100 Mbps. Which, in this part of the world, is as good as it gets for under $200 per month. That said, even before this OpenVPN setup, the IoT devices on my home network were restricted to a maximum download of 20 Mbps. I am using an EdgeSwitch to limit wired connections’ bandwidth and the Unifi controller via Unifi APs to limit wireless bandwidth. These network restrictions ensure that one or more devices don’t gobble up bandwidth.

My plan? Improve OpenVPN performance as much as possible to least hit 20 Mbps download speeds, as you can see from the above before vs. after internet speed tests. (I’ve since disabled bandwidth restrictions on the OpenVPN VLAN).

After optimizing OpenVPN’s performance, the max up/down speed is just about what the previous limits were. Let’s look at how you can go from 15 Mbps to 20 Mbps internet download speed on an 880 MHz CPU core router.

OpenVPN server Location

Whether you are using NordVPN (awesome 24/7 customer support), OVPN (best dedicated IP VPN in my opinion), or another VPN service, the first step should be to select the VPN servers that are closest to you. In my case, its servers that are located in South Miami. Not much to elaborate on here… Closer is generally faster. Still, you should test locations for yourself because not all servers perform equally. Some are under more load than others. Thankfully, OVPN shows load levels for VPN server locations.

Disable compression

If CPU isn’t a bottleneck, then feel free to enable compression.  On the Edgerouter, compression will use CPU resources leading to higher CPU usage. You can disable with:

comp-lzo no ;deprecated - remove or use 'compress' without an algorithm

or recommended for OpenVPN version 2.4+:

compress

Providing just compress without an algorithm is the equivalent of comp-lzo no which disables compression but enables the packet framing for compression.

Use UDP for better OpenVPN performance.

With OpenVPN, in most cases and especially in my usage case, UDP is faster than TCP. TCP packets are heavier, adding overhead. TCP also numbers packets in a sequence while UDP doesn’t. UDP uses very minimal headers making it less resource-intensive. Here’s the config line:

proto udp

Choosing the right Cipher

By default, OpenVPN uses Blowfish, a 128-bit cipher. When it comes to the level of security you require, you will have to decide between better encryption vs. faster throughput in respect to CPU load. Again, especially with this 880 MHz CPU. If there’s no CPU bottleneck, I would recommend using AES-256-GCM. In my case, I’m using AES-128-CBC as it resulted in faster OpenVPN throughput.

cipher AES-128-CBC

Disable cipher negotiation

You can set ncp-disable (disable “negotiable crypto parameters”). This completely disables cipher negotiation and instead uses what’s specified by the cipher option discussed previously. As of OpenVPN 2.4, this is now deprecated. Also, read OpenVPN Cipher Negotiation (Quick reference).

ncp-disable

Optimize TUN/TAP/UDP I/O writes

Set fast-io to optimize TUN/TAP/UDP I/O writes by avoiding a call to poll/epoll/select before the write operation.

“The purpose of such a call would normally be to block until the device or socket is ready to accept the write. Such blocking is unnecessary on some platforms which don’t support write blocking on UDP sockets or TUN/TAP devices. In such cases, one can optimize the event loop by avoiding the poll/epoll/select call, improving CPU efficiency by 5% to 10%. This option can only be used on non-Windows systems, when proto udp is specified, and when shaper is NOT specified.” – Source.

Set send/receive buffers

You can set the UDP socket send and receive buffer sizes. On OpenVPN 2.3.9+, this defaults to the operating system’s default (usually 64K).

Add to client config (bytes):

sndbuf 512000
rcvbuf 512000

Or, if you have access, set buffers in the server config:

sndbuf 512000
rcvbuf 512000
push "sndbuf 512000"
push "rcvbuf 512000"

Read more about fine-tuning these buffers here. These make a noticeable difference when tuned correctly.

Adjust client MTUs to match the OpenVPN server

You can use the following command to grep connection logs for ‘MTU’ mismatches. Use the warnings about size mismatch to adjust tun-mtu if necessary. My router defaults to 1500, which is also OpenVPN’s default, so no need to mess with it. Also, see warnings about adjusting tun-mtu and be sure to read about mssfix.

sudo cat /path/to/openvpn.log | grep WARNING

Which will display any warning like this:

WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1500', remote='link-mtu 3000'

Set the transmit queue length

Set the TX queue length on the TUN/TAP interface. This defaults to the system OS, which in my case is 1000.

txqueuelen 2000

Default ovpn.com config (before)

client
dev tun
remote-cert-tls server
cipher aes-256-cbc
pull
nobind
reneg-sec 0
resolv-retry infinite
verb 3
persist-key
persist-tun
remote-random
proto udp
mute-replay-warnings
comp-lzo
route-delay 10

My improved OpenVPN performance config file. (After)

client
dev tun
remote-cert-tls server

compress
proto udp
cipher aes-128-cbc
ncp-disable
fast-io

sndbuf 512000
rcvbuf 512000
txqueuelen 2000

pull
nobind
reneg-sec 0
resolv-retry infinite
verb 3
persist-key
persist-tun
remote-random
mute-replay-warnings
route-delay 10

OpenVPN Performance – Conclusion

Often, even more so in work environments than at home, we are obliged to use technology, devices, software, and other tools we wouldn’t typically support. In these cases, we are still tasked with making things work, finding a workaround, and patching things up. Using OpenVPN on an Edgerouter feels a lot like that right now.

My next article should be top five home and small business routers; what do you think? I would still include the Edgerouter 12, but I do have at least three others off the top of my head that I’m looking at next. For one, I’m interested in the Firewalla Gold, but at 2x the cost of the ER-12, it probably won’t make the list. Let’s discuss this later, yea? Please send me some suggestions to look into as well.