Testing VPN speed fairly
Change one variable at a time. Test wired rather than on Wi-Fi, use the same test server for both runs, repeat at the same time of day, and record latency and jitter alongside throughput. Otherwise you are measuring your own network rather than the tunnel.
4 min read
Most published VPN speed comparisons are close to meaningless, because they compare a tunnel to one country against a baseline to another, over Wi-Fi, at different times, once each. Any of those variables swamps the effect being measured.
Doing it properly is not difficult; it is just methodical.
The rules
Use a wired connection. Wi-Fi introduces variance larger than the thing you are measuring. A 2.4 GHz link in a busy building can lose half your line speed on its own.
Use the same test server for both runs. Comparing a tunnelled test against a server near the VPN exit with a direct test against a server near you measures geography, not the tunnel.
Test at the same time of day. Evening congestion on your line and on the VPN server both matter, and both vary by hours.
Repeat at least three times and take the median. A single run catches whatever happened to be occupying the link.
Close everything else. A sync client uploading in the background is a variable you cannot see.
Record latency and jitter, not just throughput. For calls and games these matter more than the download figure, and a tunnel affects them differently.
The procedure
- Wired, nothing else running. Run the speed test three times with the VPN off. Record download, upload, ping and jitter. Take the median.
- Connect to the nearest server. Run three more against the same test server. Median again.
- Compute the percentage change for each figure.
- Repeat step 2 for any other server you care about.
That is the whole method, and it takes about fifteen minutes.
What a normal result looks like
| Measure | Expected change on a nearby server |
|---|---|
| Download | 10 to 30 percent lower |
| Upload | 10 to 30 percent lower |
| Ping | Up by the round trip to the server |
| Jitter | Similar, sometimes slightly better |
Jitter improving is not a mistake. A tunnel to a well-connected datacentre can smooth a path that was previously taking a worse route.
If your loss is far outside that band, the cause is usually distance, a congested exit or an MTU problem rather than the protocol.
Testing under load, which nobody does and everybody should
Run a latency measurement while a download is saturating the link. If ping goes from 20 ms to 300 ms, your router is over-buffering, and that number describes your video calls far better than any idle measurement.
Bufferbloat affects the connection with or without a VPN. The tunnel makes it worse in the sense that the tunnel's own packets queue behind the same buffer, and better in the sense that fixing it improves everything at once.
Why published comparisons disagree so much
- Different test servers, which is the largest factor by far.
- Different times, on shared exits.
- Different hardware. Encryption speed varies enormously between a modern desktop and a router.
- Different protocols, sometimes not stated.
- Different lines. A test on a gigabit connection exposes CPU limits that a 100 Mbps line never reaches.
A comparison that does not state all five is not reproducible, which is another way of saying it is not a measurement.
What to do with the number
Compare servers, not providers. Your own measurement of two locations on your own line is worth more than any published table, because it includes your hardware, your line and your route.
If the nearest server on the server list costs you 15 percent and one two countries away costs 45 percent, that is a real and actionable finding. Whether some other provider is 3 percent faster on someone else's connection is not.
Measuring the tunnel rather than the internet
If you want to know what the tunnel itself costs, rather than what the whole
path costs, test against the VPN server directly instead of through it. iperf3
between your machine and an endpoint you control isolates the encryption and
framing overhead from every question about routing and congestion beyond it.
Most people do not need that number. It is useful when diagnosing a router whose CPU is the bottleneck, because it separates "this hardware cannot encrypt fast enough" from "the path to the exit is congested" — two problems with completely different answers.
Why single-stream tests understate fast lines
A single TCP connection on a high-latency path is limited by the window size and the round trip, not by the link. On a gigabit line to a distant server, one stream may reach a fraction of the capacity while eight streams in parallel saturate it.
Good speed tests use multiple streams for this reason. If you are comparing two tools and getting very different answers on a fast connection, this is usually why, and the multi-stream figure is the one that reflects what a browser downloading dozens of resources will actually experience.
