Followup Measuring BGP Stacks Performance
Systematic performance measurement of Open Source BGP Stacks, adding RustyBGP and OpenBGPD
Published on Aug 9, 2021 by Justin Pietsch
After I published the post on measuring open source BGP stacks, I was embarrassed after I realized how haphazard the testing was. I was not very systematic about the way I tried different test parameters. So I hacked on bgperf, added some more reporting and created a new batch feature that iterates through parameters systematically, and automatically graphs the results. By request, I added RustyBGP and OpenBGPD. I’ve only added rudimentary support for these just to get started testing the number of prefixes and neighbors. RustyBGP isn’t yet a fully formed BGP stack; I don’t think it has much support for any kind of policy yet.
In these tests I stopped testing GoBGP, even though bgperf still supports it. As you can see from the previous post, it uses a lot more resources and is much slower. It makes the graphs much harder to interpret. I also only test the single table version of bird, because it’s the easier config and it’s faster.
Last time for FRR I (accidentally) used a dev version which took an excess amount of time establishing neighbors when there were > 30 neighbors. This time I’m using FRRouting 7.5.1 from dockerhub container and FRRouting 8.0 that bgperf compiled from github, neither of which have problems with neighbor times. I’m not going to show neighbor results here because for the most part they are uninteresting.
There are all kinds of different uses for BGP stacks. I try to cover a set of interesting scenarios to demonstrate how these perform.
The goal in all of this is to add some quantitative power to any decision making about different BGP stacks. It’s not the only aspect to consider, often it’s the least important.
Results
The first set of results we’ll look at are from my AMD 3950 32 core machine with 64 GB RAM.
10K prefixes
We’ll start with 10K prefixes. It iterates through 10, 30, 50, and then 100 neighbors. It looks like 100 neighbors with each 10K prefixes is pretty taxing on many of these stacks.
Route reception is the time from the first route received until all routes are received.
The first thing that jumps out is how much slower OpenBGPD is especially at 100, and even 50, neighbors. While harder to see, BIRD and RustyBGP at 100 neighbors are considerably slower than the two FRRoutings, 18s compared to 9-10s.
RustyBGP uses a lot more CPU resources, even though it isn’t any faster than FRR or BIRD.
OpenBGPD uses a tremendous amount of memory, especially at 100 neighbors. I’m not sure what’s going on here, but that looks troubling to me.
results table
name
target
version
peers
prefixes per peer
neighbor (s)
elapsed (s)
prefix received (s)
exabgp (s)
total time
max cpu %
max mem (GB)
flags
date
cores
Mem (GB)
bird -s
bird
v2.0.8-59-gf761be6b
10
10000
4
2
0
2
14.1
33
0.015
-s
2021-08-02
32
62.82GB
frr
frr
FRRouting 7.5.1_git (47d89c7b746a).
10
10000
0
3
0
3
12.4
38
0.09
2021-08-02
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (8bc4400841d8).
10
10000
0
3
0
3
12.29
31
0.1
2021-08-02
32
62.82GB
rustybgp
rustybgp
10
10000
5
3
0
3
17.64
252
0.032
2021-08-02
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
10
10000
0
2
0
2
11.22
104
0.193
2021-08-02
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
30
10000
4
4
0
4
28.13
100
0.173
-s
2021-08-02
32
62.82GB
frr
frr
FRRouting 7.5.1_git (ec526afa5b01).
30
10000
0
3
0
3
22.83
79
0.299
2021-08-02
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (dbf3b58d31c8).
30
10000
1
3
0
3
24.14
83
0.294
2021-08-02
32
62.82GB
rustybgp
rustybgp
30
10000
5
4
0
4
29.67
397
0.133
2021-08-02
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
30
10000
0
11
0
11
30.15
104
1.373
2021-08-02
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
50
10000
3
5
0
5
42.18
100
0.371
-s
2021-08-02
32
62.82GB
frr
frr
FRRouting 7.5.1_git (397e49823ba4).
50
10000
0
5
0
5
36.95
101
0.493
2021-08-02
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (0059e9cea26d).
50
10000
1
5
0
5
36.77
104
0.526
2021-08-02
32
62.82GB
rustybgp
rustybgp
50
10000
7
6
0
6
46.25
522
0.337
2021-08-02
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
50
10000
0
29
0
29
58.62
104
3.588
2021-08-02
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
100
10000
3
18
0
18
93.32
100
1.333
-s
2021-08-02
32
62.82GB
frr
frr
FRRouting 7.5.1_git (d242197e9f14).
100
10000
1
9
0
9
77.01
102
1.276
2021-08-02
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (35e3bbe2dfa2).
100
10000
1
10
0
10
75.0
110
1.235
2021-08-02
32
62.82GB
rustybgp
rustybgp
100
10000
6
16
0
16
81.65
587
1.248
2021-08-02
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
100
10000
0
140
0
140
195.81
102
13.935
2021-08-02
32
62.82GB
Many (many) neighbors, 10 prefixes
This iterates through neighbors from 250 to 1750. More than that runs out of memory on my 64 GB machine (because of all the ExaBGP). It uses a very minimal amount of prefixes just to see if we can understand anything regarding max prefixes.
The time to receive prefixes, all 10 per neighbor, doesn’t matter in this test.
Interestingly, RustyBGP uses the smallest amount of CPU and OpenBGPD uses the most, by a lot. FRRouting 8 uses more CPU at 1750 neighbors than 7.5.1. I don’t know that is important since it’s still not very much
OpenBGPD uses a lot more memory, several orders of magnitude more.
Because it’s so hard to see everything but OpenBGPD, going to remove it so we can look at the others.
FRRouting 8 uses a lot more memory than FRRouting 7.5.1 or BIRD. Don’t know if this matters because it’s only 10 prefixes, but it’s interesting. There looks like some kind of per neighbor memory tax in FRRouting 8. It is too much? I don’t know. Is it impactful as there are more routes? Let’s see.
results table
name
target
version
peers
prefixes per peer
neighbor (s)
elapsed (s)
prefix received (s)
exabgp (s)
total time
max cpu %
max mem (GB)
flags
date
cores
Mem (GB)
bird -s
bird
v2.0.8-59-gf761be6b
250
10
3
1
0
1
37.45
1
0.011
-s
2021-08-02
32
62.82GB
frr
frr
FRRouting 7.5.1_git (efbcfbcfae46).
250
10
1
2
0
2
35.56
6
0.03
2021-08-02
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (58d6fcaff2bd).
250
10
1
2
0
2
35.56
2
0.183
2021-08-02
32
62.82GB
rustybgp
rustybgp
250
10
4
1
0
1
37.12
1
0.02
2021-08-02
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
250
10
0
1
0
1
33.71
2
0.115
2021-08-02
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
500
10
4
1
0
1
68.57
2
0.018
-s
2021-08-02
32
62.82GB
frr
frr
FRRouting 7.5.1_git (a0cfaad48bb8).
500
10
1
2
0
2
66.44
4
0.051
2021-08-02
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (3cc686c6c1b7).
500
10
1
2
0
2
66.3
3
0.351
2021-08-02
32
62.82GB
rustybgp
rustybgp
500
10
7
1
0
1
69.38
2
0.028
2021-08-02
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
500
10
0
1
0
1
63.87
4
0.386
2021-08-02
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
750
10
3
1
0
1
100.03
4
0.024
-s
2021-08-02
32
62.82GB
frr
frr
FRRouting 7.5.1_git (e0bd4c7ed94e).
750
10
1
2
0
2
99.83
6
0.069
2021-08-02
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (3ce1d73e7577).
750
10
1
2
0
2
99.88
5
0.518
2021-08-02
32
62.82GB
rustybgp
rustybgp
750
10
3
1
0
1
94.65
3
0.039
2021-08-02
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
750
10
0
1
0
1
95.74
7
0.825
2021-08-02
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
1000
10
4
1
0
1
136.78
6
0.032
-s
2021-08-02
32
62.82GB
frr
frr
FRRouting 7.5.1_git (2cc17c82ab54).
1000
10
1
2
0
2
139.09
11
0.089
2021-08-02
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (a943c59da7ef).
1000
10
1
1
0
1
138.45
15
0.684
2021-08-02
32
62.82GB
rustybgp
rustybgp
1000
10
3
1
0
1
126.63
3
0.049
2021-08-02
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
1000
10
0
2
0
2
133.29
19
1.431
2021-08-02
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
1250
10
4
1
0
1
179.07
8
0.038
-s
2021-08-02
32
62.82GB
frr
frr
FRRouting 7.5.1_git (98d192542e54).
1250
10
1
2
0
2
184.74
9
0.109
2021-08-02
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (bd9d6e72020c).
1250
10
1
2
0
2
184.2
9
0.854
2021-08-02
32
62.82GB
rustybgp
rustybgp
1250
10
7
1
0
1
162.06
4
0.059
2021-08-02
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
1250
10
0
2
0
2
174.82
37
2.206
2021-08-02
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
1500
10
3
1
0
1
245.5
14
0.047
-s
2021-08-02
32
62.82GB
frr
frr
FRRouting 7.5.1_git (da063c606061).
1500
10
1
2
0
2
240.3
12
0.129
2021-08-02
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (9ffc847915c4).
1500
10
1
2
0
2
240.3
11
1.025
2021-08-02
32
62.82GB
rustybgp
rustybgp
1500
10
6
1
0
1
193.44
4
0.071
2021-08-02
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
1500
10
0
2
0
2
228.24
119
3.148
2021-08-02
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
1750
10
4
1
0
1
397.92
19
0.059
-s
2021-08-02
32
62.82GB
frr
frr
FRRouting 7.5.1_git (3c5848cd611a).
1750
10
1
2
0
2
348.19
15
0.15
2021-08-02
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (a94a18882cb0).
1750
10
1
2
0
2
343.93
28
1.19
2021-08-02
32
62.82GB
rustybgp
rustybgp
1750
10
5
1
0
1
228.38
6
0.08
2021-08-02
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
1750
10
0
2
0
2
346.25
55
4.259
2021-08-03
32
62.82GB
Many neighbors, 100 prefixes
This only goes from 250 to 750 neighbors with 100 prefixes. More than that runs out of memory on my 64 GB machine (because of all the ExaBGP processes.)
Again, route reception is trivial. Maybe it matters that OpenBGPD at 750 neighbors is 4 seconds, but not sure.
Similar to the 10 prefix tests, OpenBGPD uses many more CPU resources than the others.
Similar to the 10 prefix tests, OpenBGPD uses many more memory resources than the others. What’s troubling is that at 10 prefixes and 750 neighbors it’s about 1GB but at 100 prefixes, 750 neighbors it’s 6GB.
Let’s again remove OpenBGPD so we can understand the others better.
FRRouting 8 is still a lot higher than the others, but it grows to about 0.58 GB from 0.52 at 750 neighbors. That again leads credence to a per neighbor memory tax, but doesn’t seem to be in the way as more prefixes are added.
results table
name
target
version
peers
prefixes per peer
neighbor (s)
elapsed (s)
prefix received (s)
exabgp (s)
total time
max cpu %
max mem (GB)
flags
date
cores
Mem (GB)
bird -s
bird
v2.0.8-59-gf761be6b
250
100
4
1
0
1
42.5
4
0.016
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (0c1e5f60fc82).
250
100
1
2
0
2
40.71
2
0.049
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (5682965a9a3e).
250
100
1
2
0
2
40.32
2
0.198
2021-08-03
32
62.82GB
rustybgp
rustybgp
250
100
7
1
0
1
41.65
5
0.024
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
250
100
0
1
0
1
36.33
8
0.882
2021-08-03
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
500
100
3
1
0
1
84.13
11
0.033
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (301d657c0456).
500
100
1
2
0
2
87.72
5
0.089
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (ffd5a10353a7).
500
100
1
2
0
2
87.33
5
0.391
2021-08-03
32
62.82GB
rustybgp
rustybgp
500
100
3
1
0
1
67.44
10
0.04
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
500
100
0
2
0
2
78.06
104
3.431
2021-08-03
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
750
100
4
1
0
1
149.08
18
0.06
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (89f4d47d3ae2).
750
100
1
2
0
2
152.63
20
0.134
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (314f7f88d2c2).
750
100
1
2
0
2
151.09
19
0.581
2021-08-03
32
62.82GB
rustybgp
rustybgp
750
100
4
1
0
1
99.12
13
0.057
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
750
100
0
4
0
4
123.41
107
6.336
2021-08-03
32
62.82GB
Prefix iterations
This set of tests iterates from 50 to 250 neighbors and 10 prefixes to 1000.
Route Reception is trivial, except OpenBGPD at 250 neighbors, 1000 prefixes. It plain freaks out.
Zoom in without OpenBGPD again.
Not sure how to interpret this exactly. It looks like additional routes don’t really matter for memory until 1000 ish. FRRouting 8 still uses more memory but looks like a fixed amount perf neighbor not a percentage, so probably not so concerning. RustyBGP is the lowest memory user still.
results table
name
target
version
peers
prefixes per peer
neighbor (s)
elapsed (s)
prefix received (s)
exabgp (s)
total time
max cpu %
max mem (GB)
flags
date
cores
Mem (GB)
bird -s
bird
v2.0.8-59-gf761be6b
50
10
4
1
0
1
18.51
0
0.011
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (113d0e12c79d).
50
10
1
2
0
2
12.69
1
0.02
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (929f08570080).
50
10
1
2
0
2
12.79
1
0.058
2021-08-03
32
62.82GB
rustybgp
rustybgp
50
10
4
1
0
1
15.07
0
0.019
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
50
10
0
1
0
1
10.69
0
0.022
2021-08-03
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
50
100
3
1
0
1
14.12
1
0.006
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (bfb4538bb47a).
50
100
0
2
0
2
11.79
2
0.017
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (549dc45ac858).
50
100
1
2
0
2
12.99
1
0.051
2021-08-03
32
62.82GB
rustybgp
rustybgp
50
100
6
1
0
1
17.18
2
0.01
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
50
100
0
1
0
1
10.94
2
0.051
2021-08-03
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
50
1000
3
2
0
2
17.42
8
0.012
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (3f3302806935).
50
1000
1
2
0
2
15.16
9
0.058
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (bf6b61ccc50b).
50
1000
1
2
0
2
16.03
5
0.086
2021-08-03
32
62.82GB
rustybgp
rustybgp
50
1000
5
2
0
2
19.18
20
0.024
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
50
1000
0
2
0
2
14.29
26
0.375
2021-08-03
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
100
10
3
1
0
1
19.13
0
0.007
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (ee0acdb717c0).
100
10
1
2
0
2
18.34
1
0.018
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (ae7b80997a7b).
100
10
1
2
0
2
18.36
1
0.081
2021-08-03
32
62.82GB
rustybgp
rustybgp
100
10
5
1
0
1
21.51
1
0.01
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
100
10
0
1
0
1
16.35
1
0.032
2021-08-03
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
100
100
4
1
0
1
21.07
1
0.008
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (b204d0230f45).
100
100
1
2
0
2
19.05
2
0.026
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (dcda0c4d95dc).
100
100
1
2
0
2
18.94
1
0.087
2021-08-03
32
62.82GB
rustybgp
rustybgp
100
100
6
1
0
1
23.07
3
0.013
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
100
100
0
1
0
1
16.93
3
0.159
2021-08-03
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
100
1000
4
2
0
2
30.5
24
0.05
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (17f321e4cf3d).
100
1000
1
3
0
3
26.98
19
0.096
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (96afe9d83314).
100
1000
1
2
0
2
28.19
6
0.164
2021-08-03
32
62.82GB
rustybgp
rustybgp
100
1000
5
2
0
2
26.78
33
0.042
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
100
1000
0
2
0
2
24.23
109
1.278
2021-08-03
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
250
10
3
1
0
1
37.03
1
0.011
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (d66c80e8a89f).
250
10
1
2
0
2
35.44
5
0.03
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (ac4494486900).
250
10
1
2
0
2
35.5
2
0.182
2021-08-03
32
62.82GB
rustybgp
rustybgp
250
10
5
1
0
1
38.57
1
0.017
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
250
10
0
1
0
1
33.71
2
0.114
2021-08-03
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
250
100
3
1
0
1
40.27
4
0.016
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (43f298acd0af).
250
100
1
2
0
2
40.53
3
0.049
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (76c98d3e45de).
250
100
1
2
0
2
39.94
2
0.201
2021-08-03
32
62.82GB
rustybgp
rustybgp
250
100
5
1
0
1
39.2
6
0.024
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
250
100
0
2
0
2
37.19
8
0.882
2021-08-03
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
250
1000
4
2
0
2
99.93
46
0.322
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (372804f61f88).
250
1000
1
3
0
3
98.98
55
0.255
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (d38410082c83).
250
1000
1
1
0
1
99.79
16
0.409
2021-08-03
32
62.82GB
rustybgp
rustybgp
250
1000
5
2
0
2
50.75
93
0.1
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
250
1000
0
53
0
53
102.39
104
8.234
2021-08-03
32
62.82GB
1M routes.
Since the internet is getting close to 1M routes, how do these stacks do with 1M routes and multiple neighbors?
CAVEAT As I’ve been thinking about this test I’ve realized it’s not doing a good job of representing actually connecting to internet routes. It’s 1M /32s that are all unique, so in the end, if you have 5 neighbors, you have 5 million routes. It’s what I can do for now, I’ll continue to work on improving this test.
This is the first time that RustyBGP is considerably slower than all the others.
FRR 8 is somewhat slower than 7.5.1 or BIRD. OpenBGPD looks good here.
BIRD uses 1/3 the memory of FRRouting. RustyBGP also uses significantly less memory than FRRouting, but it’s slower too. OpenBGPD uses the most memory, as usual.
results table
name
target
version
peers
prefixes per peer
neighbor (s)
elapsed (s)
prefix received (s)
exabgp (s)
total time
max cpu %
max mem (GB)
flags
date
cores
Mem (GB)
bird -s
bird
v2.0.8-59-gf761be6b
1
1000000
3
73
67
6
120.59
28
0.099
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (d3abdbb78d48).
1
1000000
0
75
68
7
119.25
100
0.748
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (0614e7d8586a).
1
1000000
0
75
67
8
116.55
83
0.753
2021-08-03
32
62.82GB
rustybgp
rustybgp
1
1000000
6
86
69
17
134.92
611
0.342
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
1
1000000
0
74
68
6
117.27
116
0.615
2021-08-03
32
62.82GB
bird -s
bird
v2.0.8-59-gf761be6b
5
1000000
4
104
71
33
307.44
100
1.329
-s
2021-08-03
32
62.82GB
frr
frr
FRRouting 7.5.1_git (6f77a7b2651b).
5
1000000
0
105
72
33
310.54
105
3.757
2021-08-03
32
62.82GB
frr 8
frr_c
FRRouting 8.0-bgperf (45f6898ec515).
5
1000000
0
113
71
42
311.39
101
3.744
2021-08-03
32
62.82GB
rustybgp
rustybgp
5
1000000
5
161
72
89
366.39
1767
1.815
2021-08-03
32
62.82GB
openbgp
openbgp
OpenBGPD 7.1
5
1000000
0
118
71
47
313.93
160
6.55
2021-08-03
32
62.82GB
Results from EC2 m5
I wanted to see what would happen with more resources, specifically twice as much RAM. However, the AMD device is a consumer device with less cores but each higher speed. That might matter to some of these tests because of the single core processes.
I ran the same test suites with some little changes. I won’t go through all the graphs, because there are pretty similiar results. But where there is something interesting I’ll mention that.
10K prefixes
Route Reception has a greater difference between both FRRs and BIRD/RustyBGP. Maybe it’s that 3950 is a consumer device with higher speed cores?
results table
name
target
version
peers
prefixes per peer
neighbor (s)
elapsed (s)
prefix received (s)
exabgp (s)
total time
max cpu %
max mem (GB)
flags
date
cores
Mem (GB)
bird -s
bird
v2.0.8-59-gf761be6b
10
10000
3
2
0
2
13.89
36
0.014
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (6c29dbcbd504).
10
10000
0
3
0
3
12.8
42
0.089
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (c03326d5cd06).
10
10000
1
3
0
3
13.88
38
0.098
2021-08-03
48
184.65GB
rustybgp
rustybgp
10
10000
4
3
0
3
17.22
310
0.037
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
10
10000
0
2
0
2
11.79
107
0.191
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
30
10000
4
3
0
3
29.86
100
0.154
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (51083cfe7d92).
30
10000
0
3
0
3
24.98
87
0.293
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (12a856b29b4a).
30
10000
1
3
0
3
26.16
97
0.274
2021-08-03
48
184.65GB
rustybgp
rustybgp
30
10000
5
4
0
4
31.78
433
0.129
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
30
10000
0
12
0
12
33.61
105
1.355
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
50
10000
4
8
0
8
48.91
100
0.216
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (6586287cc347).
50
10000
1
4
0
4
40.02
102
0.574
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (9363f3ceee54).
50
10000
1
4
0
4
39.58
96
0.53
2021-08-03
48
184.65GB
rustybgp
rustybgp
50
10000
4
5
0
5
46.52
488
0.333
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
50
10000
0
37
0
37
71.4
104
3.611
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
100
10000
3
30
0
30
109.99
100
0.539
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (195e24df8349).
100
10000
1
5
0
5
78.91
102
1.243
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (9aef3b64fbb9).
100
10000
1
9
0
9
83.74
102
1.159
2021-08-03
48
184.65GB
rustybgp
rustybgp
100
10000
4
22
0
22
93.28
645
1.266
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
100
10000
0
197
0
197
260.57
103
13.93
2021-08-03
48
184.65GB
Many (many) neighbors, 10 prefixes
I didn’t find any interesting differences here.
results table
name
target
version
peers
prefixes per peer
neighbor (s)
elapsed (s)
prefix received (s)
exabgp (s)
total time
max cpu %
max mem (GB)
flags
date
cores
Mem (GB)
bird -s
bird
v2.0.8-59-gf761be6b
250
10
4
1
0
1
43.75
1
0.011
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (9fa3b6648b40).
250
10
1
2
0
2
41.6
2
0.03
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (8b0e48e5b635).
250
10
1
2
0
2
41.75
5
0.184
2021-08-03
48
184.65GB
rustybgp
rustybgp
250
10
4
1
0
1
43.34
1
0.024
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
250
10
0
1
0
1
38.93
2
0.117
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
500
10
4
1
0
1
80.5
3
0.019
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (e27e7aa7e4b8).
500
10
1
2
0
2
79.12
4
0.051
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (02113aa72b06).
500
10
1
2
0
2
79.06
5
0.353
2021-08-03
48
184.65GB
rustybgp
rustybgp
500
10
4
1
0
1
78.79
2
0.039
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
500
10
0
1
0
1
76.21
5
0.387
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
750
10
4
1
0
1
119.31
5
0.025
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (c3eeff85707a).
750
10
1
1
0
1
116.81
8
0.071
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (554dcff645f0).
750
10
1
2
0
2
118.0
8
0.52
2021-08-03
48
184.65GB
rustybgp
rustybgp
750
10
7
1
0
1
118.72
3
0.052
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
750
10
0
1
0
1
114.57
9
0.827
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
1000
10
4
1
0
1
160.2
8
0.034
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (ab80fb0d3260).
1000
10
1
2
0
2
159.58
24
0.09
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (2a28e1e9fde4).
1000
10
1
2
0
2
160.37
9
0.689
2021-08-03
48
184.65GB
rustybgp
rustybgp
1000
10
3
1
0
1
151.55
3
0.066
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
1000
10
0
1
0
1
155.44
15
1.434
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
1250
10
4
1
0
1
202.32
11
0.04
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (03c84e9d82c5).
1250
10
1
2
0
2
203.98
12
0.11
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (d576228f82c0).
1250
10
1
2
0
2
203.42
12
0.856
2021-08-03
48
184.65GB
rustybgp
rustybgp
1250
10
7
1
0
1
194.22
4
0.08
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
1250
10
0
1
0
1
196.74
33
2.209
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
1500
10
4
1
0
1
246.76
14
0.048
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (c317f3fd1562).
1500
10
1
2
0
2
249.82
22
0.13
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (9cb67b730527).
1500
10
2
2
0
2
250.99
14
1.024
2021-08-03
48
184.65GB
rustybgp
rustybgp
1500
10
6
1
0
1
232.12
4
0.093
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
1500
10
0
2
0
2
242.93
105
3.151
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
1750
10
3
1
0
1
292.23
19
0.06
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (6e117b0dfab3).
1750
10
1
2
0
2
301.22
16
0.151
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (67fb00565b11).
1750
10
2
2
0
2
301.28
14
1.19
2021-08-03
48
184.65GB
rustybgp
rustybgp
1750
10
6
1
0
1
270.69
5
0.107
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
1750
10
0
2
0
2
292.51
168
4.263
2021-08-03
48
184.65GB
Many neighbors, 100 prefixes
OpenBGPD at 750 neighbors is a lot greater route reception time than on the AMD.
results table
name
target
version
peers
prefixes per peer
neighbor (s)
elapsed (s)
prefix received (s)
exabgp (s)
total time
max cpu %
max mem (GB)
flags
date
cores
Mem (GB)
bird -s
bird
v2.0.8-59-gf761be6b
250
100
4
1
0
1
48.05
6
0.017
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (aac6c05005e7).
250
100
1
2
0
2
44.72
3
0.049
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (bafd97767753).
250
100
1
2
0
2
45.26
3
0.205
2021-08-03
48
184.65GB
rustybgp
rustybgp
250
100
5
1
0
1
45.64
5
0.03
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
250
100
0
1
0
1
41.95
10
0.884
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
500
100
4
1
0
1
89.49
13
0.038
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (c0bc9ab47a70).
500
100
1
3
0
3
91.91
9
0.097
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (2a1be110b83d).
500
100
1
2
0
2
92.26
5
0.392
2021-08-03
48
184.65GB
rustybgp
rustybgp
500
100
5
1
0
1
82.61
10
0.052
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
500
100
0
2
0
2
85.7
106
3.26
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
750
100
4
2
0
2
142.37
29
0.067
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (6bb88afd5bd1).
750
100
1
2
0
2
149.16
17
0.139
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (64114990e4ae).
750
100
1
1
0
1
148.04
9
0.588
2021-08-03
48
184.65GB
rustybgp
rustybgp
750
100
7
2
0
2
124.17
13
0.07
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
750
100
0
12
0
12
138.13
109
6.142
2021-08-03
48
184.65GB
Prefix iterations
Because of more memory, I tested to 500 neighbors.
OpenBGPD is an order of magnitude worse at 500 neighbors, 1000 prefixes, then at 250.
We see Rusytbgp jump up in CPU usage at 500 neighbors. On the AMD is never got past 100%.
OpenBGPD is using over 30GB of RAM and is slow at 500 neighbors.
results table
name
target
version
peers
prefixes per peer
neighbor (s)
elapsed (s)
prefix received (s)
exabgp (s)
total time
max cpu %
max mem (GB)
flags
date
cores
Mem (GB)
bird -s
bird
v2.0.8-59-gf761be6b
50
10
3
1
0
1
18.95
0
0.004
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (1b7bd1287074).
50
10
1
2
0
2
13.76
1
0.013
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (f23ee8ded7cd).
50
10
1
2
0
2
13.77
1
0.047
2021-08-03
48
184.65GB
rustybgp
rustybgp
50
10
7
1
0
1
18.96
1
0.01
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
50
10
0
1
0
1
11.54
0
0.017
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
50
100
4
1
0
1
16.12
1
0.005
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (e5757cde5cf5).
50
100
1
2
0
2
13.89
1
0.017
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (74ce4743a76a).
50
100
1
2
0
2
13.95
1
0.05
2021-08-03
48
184.65GB
rustybgp
rustybgp
50
100
5
1
0
1
17.04
2
0.011
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
50
100
0
1
0
1
11.8
1
0.049
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
50
1000
4
2
0
2
19.74
11
0.011
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (ee1524e5b82e).
50
1000
1
3
0
3
17.78
8
0.054
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (969567a4ed46).
50
1000
1
2
0
2
16.65
8
0.088
2021-08-03
48
184.65GB
rustybgp
rustybgp
50
1000
6
2
0
2
21.49
10
0.026
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
50
1000
0
2
0
2
15.47
32
0.373
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
100
10
4
1
0
1
22.4
0
0.006
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (4fe15269e3c2).
100
10
1
2
0
2
20.55
1
0.017
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (ac0e558251b6).
100
10
1
2
0
2
20.37
1
0.08
2021-08-03
48
184.65GB
rustybgp
rustybgp
100
10
6
1
0
1
24.75
1
0.013
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
100
10
0
1
0
1
18.31
1
0.031
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
100
100
3
1
0
1
22.19
2
0.007
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (6145e734701d).
100
100
1
2
0
2
21.33
1
0.024
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (4cae93bd405c).
100
100
1
2
0
2
21.26
4
0.088
2021-08-03
48
184.65GB
rustybgp
rustybgp
100
100
5
1
0
1
24.13
3
0.016
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
100
100
0
1
0
1
18.98
3
0.158
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
100
1000
3
2
0
2
30.45
24
0.047
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (8dd5bfdc0070).
100
1000
1
2
0
2
28.17
21
0.101
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (de7122c38a5f).
100
1000
1
2
0
2
28.68
4
0.165
2021-08-03
48
184.65GB
rustybgp
rustybgp
100
1000
7
2
0
2
32.02
35
0.05
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
100
1000
0
2
0
2
25.87
108
1.205
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
250
10
3
1
0
1
42.49
1
0.01
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (6edac209a780).
250
10
1
2
0
2
41.79
2
0.031
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (7b36871c45fd).
250
10
1
2
0
2
41.77
7
0.182
2021-08-03
48
184.65GB
rustybgp
rustybgp
250
10
6
1
0
1
45.63
1
0.023
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
250
10
0
1
0
1
39.03
2
0.118
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
250
100
3
1
0
1
45.62
5
0.017
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (19e61a045c85).
250
100
1
2
0
2
44.63
3
0.052
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (f420d5f3148e).
250
100
1
2
0
2
45.45
3
0.208
2021-08-03
48
184.65GB
rustybgp
rustybgp
250
100
4
1
0
1
44.65
5
0.03
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
250
100
0
1
0
1
42.17
11
0.882
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
250
1000
4
2
0
2
90.23
100
0.271
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (0a8e2847dfb3).
250
1000
1
2
0
2
85.26
7
0.266
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (6997fde34233).
250
1000
1
2
0
2
84.16
13
0.404
2021-08-03
48
184.65GB
rustybgp
rustybgp
250
1000
5
2
0
2
59.78
130
0.098
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
250
1000
0
77
0
77
130.97
104
8.322
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
500
10
4
1
0
1
81.1
3
0.019
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (feaf3f0aa01c).
500
10
1
2
0
2
78.94
4
0.051
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (db742eea9ece).
500
10
1
2
0
2
79.04
5
0.353
2021-08-03
48
184.65GB
rustybgp
rustybgp
500
10
5
1
0
1
80.24
2
0.038
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
500
10
0
1
0
1
76.11
5
0.387
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
500
100
4
1
0
1
89.43
13
0.036
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (f46f0020af8b).
500
100
1
1
0
1
91.63
6
0.085
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (2cde9edc1b70).
500
100
1
2
0
2
92.13
10
0.397
2021-08-03
48
184.65GB
rustybgp
rustybgp
500
100
7
1
0
1
84.86
9
0.052
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
500
100
0
2
0
2
85.36
107
3.212
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
500
1000
4
4
0
4
268.44
100
0.206
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (d920efd504bb).
500
1000
1
3
0
3
259.93
17
0.597
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (679983b9da23).
500
1000
1
3
0
3
258.02
105
0.898
2021-08-03
48
184.65GB
rustybgp
rustybgp
500
1000
6
9
0
9
118.66
472
0.951
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
500
1000
0
516
0
516
619.03
104
33.718
2021-08-03
48
184.65GB
1M routes.
RustyBGP uses twice as much CPU as the AMD, but is still slower than all the others.
results table
name
target
version
peers
prefixes per peer
neighbor (s)
elapsed (s)
prefix received (s)
exabgp (s)
total time
max cpu %
max mem (GB)
flags
date
cores
Mem (GB)
bird -s
bird
v2.0.8-59-gf761be6b
1
1000000
4
83
76
7
140.62
37
0.099
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (f08fdc332b25).
1
1000000
0
85
76
9
136.48
103
0.733
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (5effcaf57435).
1
1000000
1
85
76
9
137.54
98
0.746
2021-08-03
48
184.65GB
rustybgp
rustybgp
1
1000000
7
96
76
20
155.45
887
0.323
2021-08-03
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
1
1000000
0
84
77
7
135.26
119
0.615
2021-08-03
48
184.65GB
bird -s
bird
v2.0.8-59-gf761be6b
5
1000000
3
119
81
38
370.24
100
1.293
-s
2021-08-03
48
184.65GB
frr
frr
FRRouting 7.5.1_git (7c2318f544f4).
5
1000000
0
122
81
41
374.46
106
3.773
2021-08-03
48
184.65GB
frr 8
frr_c
FRRouting 8.0-bgperf (486402e07539).
5
1000000
0
130
81
49
375.28
102
3.714
2021-08-04
48
184.65GB
rustybgp
rustybgp
5
1000000
5
185
81
104
437.22
2918
1.304
2021-08-04
48
184.65GB
openbgp
openbgp
OpenBGPD 7.1
5
1000000
0
140
81
59
389.42
155
6.527
2021-08-04
48
184.65GB
Results from EC2 t3
I wanted to try out what would happen on a machine with more limited resources. As with the above, these are the same suite of tests, with some little limitations because of less memory. The biggest issue is that because the target stack and the tester, and the monitor, are all on the same hardware they fight for resources, especially in the many neighbors test. I’m still trying to understand the impact of bgperf on the results.
10K prefixes
It’s interesting that RustyBGP uses less CPU, because there is less, but it’s not much slower than on the other machines. What is it doing with those other CPU resources when it has them?
Many (many) neighbors, 10 prefixes
Can only test many less parameters because of memory.
While not really different, it is interesting to zoom in on just 250 and 500 neighbors.
Again, not much different, but zooming in shows the difference in memory between FRRouing 8/OpenBGPD vs the others.
Many neighbors, 100 prefixes
Nothing interestingly different here.
Prefix iterations
Nothing interestingly different here.
1M routes.
While the others are about the same, RustyBGP doubles it’s time here.
Just a lot less CPU resources for RustyBGP to use.
More extreme tests
After doing those suite of tests across all three different hardware, I realized that in some cases it’s still telling us more about how much resources the test suite uses than the limits of these stacks. These no longer test OpenBGPD (or GoBGP) because it is so much less performant than the other stacks. I can’t use my own AMD machine because these tests run out of memory, so this is all EC2 hardware.
10 Prefixes, Many Many Neighbors
Ooop, right away, something interesting. above 3000 neighbors BIRD gets sad/mad. We seem to have found a boundary in BIRD where it gets much worse
100 Prefixes, Many Neighbors
In the above tests, there was a test for 100 Prefixes up to 750 neighbors. Let’s go bigger and see what we see.
Above 1000 neighbors BIRD gets sad/mad. In the above tests we did over 1000 neighbors with BIRD, but with only 10 prefixes.
name
target
version
peers
prefixes per peer
neighbor (s)
elapsed (s)
prefix received (s)
exabgp (s)
total time
max cpu %
max mem (GB)
flags
date
cores
Mem (GB)
bird -s
bird
v2.0.8-59-gf761be6b
500
100
3
1
0
1
86.09
12
0.042
-s
2021-08-06
96
373.71GB
frr
frr
FRRouting 7.5.1_git (3f48ff72e985).
500
100
1
2
0
2
86.62
9
0.093
2021-08-06
96
373.71GB
frr 8
frr_c
FRRouting 8.0-bgperf (9f7e89faa23b).
500
100
1
2
0
2
86.62
5
0.402
2021-08-06
96
373.71GB
rustybgp
rustybgp
rustybgpd
500
100
4
1
0
1
84.71
14
0.083
2021-08-06
96
373.71GB
bird -s
bird
v2.0.8-59-gf761be6b
750
100
3
2
0
2
132.26
32
0.076
-s
2021-08-06
96
373.71GB
frr
frr
FRRouting 7.5.1_git (3a605a30d502).
750
100
1
2
0
2
132.56
13
0.133
2021-08-06
96
373.71GB
frr 8
frr_c
FRRouting 8.0-bgperf (7d71ee1eae78).
750
100
1
2
0
2
132.72
9
0.594
2021-08-06
96
373.71GB
rustybgp
rustybgp
rustybgpd
750
100
3
1
0
1
122.23
27
0.113
2021-08-06
96
373.71GB
bird -s
bird
v2.0.8-59-gf761be6b
1000
100
4
23
0
23
203.0
102
0.066
-s
2021-08-06
96
373.71GB
frr
frr
FRRouting 7.5.1_git (e6ecdb58fce7).
1000
100
1
2
0
2
183.73
13
0.166
2021-08-06
96
373.71GB
frr 8
frr_c
FRRouting 8.0-bgperf (2d3a2461279c).
1000
100
1
2
0
2
182.45
15
0.791
2021-08-06
96
373.71GB
rustybgp
rustybgp
rustybgpd
1000
100
5
2
0
2
166.61
27
0.148
2021-08-06
96
373.71GB
bird -s
bird
v2.0.8-59-gf761be6b
1250
100
3
106
0
106
333.29
102
0.08
-s
2021-08-06
96
373.71GB
frr
frr
FRRouting 7.5.1_git (8530711cc1eb).
1250
100
1
2
0
2
238.56
30
0.22
2021-08-06
96
373.71GB
frr 8
frr_c
FRRouting 8.0-bgperf (d80d45396d14).
1250
100
1
2
0
2
239.83
19
0.981
2021-08-06
96
373.71GB
rustybgp
rustybgp
rustybgpd
1250
100
6
2
0
2
208.36
35
0.177
2021-08-06
96
373.71GB
bird -s
bird
v2.0.8-59-gf761be6b
1500
100
4
265
0
265
544.04
102
0.12
-s
2021-08-06
96
373.71GB
frr
frr
FRRouting 7.5.1_git (3486fff94217).
1500
100
1
2
0
2
299.12
27
0.258
2021-08-06
96
373.71GB
frr 8
frr_c
FRRouting 8.0-bgperf (35d8bdf2376a).
1500
100
2
2
0
2
300.5
25
1.168
2021-08-06
96
373.71GB
rustybgp
rustybgp
rustybgpd
1500
100
3
2
0
2
246.67
48
0.206
2021-08-06
96
373.71GB
1000 Prefixes, Many Neighbors
Once again, BIRD is mad. RustyBGP also is > an order of magnitude slower than FRRouting.
RustyBGP is using lots of CPU here. It looks like FRRouting doesn’t get to 100% CPU until there are 750 neighbors here.
Interestingly, BIRD uses the least amount of memory. RustyBGP uses a lot of memory for RustyBGP. It is usually the lowest or near the lowest memory users in these tests, but not here.
name
target
version
peers
prefixes per peer
neighbor (s)
elapsed (s)
prefix received (s)
exabgp (s)
total time
max cpu %
max mem (GB)
flags
date
cores
Mem (GB)
bird -s
bird
v2.0.8-59-gf761be6b
500
1000
4
100
0
100
233.72
101
0.125
-s
2021-08-06
96
373.71GB
frr
frr
FRRouting 7.5.1_git (8e9dc5fee3a8).
500
1000
1
3
0
3
168.5
34
0.577
2021-08-06
96
373.71GB
frr 8
frr_c
FRRouting 8.0-bgperf (3877e90d9055).
500
1000
1
3
0
3
165.99
28
0.915
2021-08-06
96
373.71GB
rustybgp
rustybgp
rustybgpd
500
1000
5
19
0
19
128.23
916
1.67
2021-08-06
96
373.71GB
bird -s
bird
v2.0.8-59-gf761be6b
750
1000
3
233
0
233
434.58
101
0.227
-s
2021-08-06
96
373.71GB
frr
frr
FRRouting 7.5.1_git (608af43f977b).
750
1000
1
5
0
5
315.49
112
1.046
2021-08-06
96
373.71GB
frr 8
frr_c
FRRouting 8.0-bgperf (90af929fb66d).
750
1000
1
4
0
4
329.84
108
1.383
2021-08-06
96
373.71GB
rustybgp
rustybgp
rustybgpd
750
1000
6
68
0
68
232.51
1042
5.338
2021-08-06
96
373.71GB
Observations
These tests use quite a bit of hardware in CPU and memory. Most of those resources are going into ExaBGP sending the prefixes rather than in the targets. In some ways this is a test of the test system. I’m still figuring out how to best test the targets without the test system getting in the way.
RustyBGP
I’ve seen twitter claims that RustyBGP is significantly faster that FRRouting, but I can’t reproduce that here. I’m sure I’m missing something. RustyBGP is still missing a lot of necessary BGP daemon features, I’m just not testing them.
BIRD with more than 1000 neighbors and more than just a handful of prefixes becomes problematic. And with 1000 prefixes, By 500 neighbors it’s performing badly again. I don’t know what it is exactly that is the bottleneck here.
Conclusion
OpenBGP struggles in places that the other stacks do not, but it’s still faster than GoBGP. For anybody running OpenBGP, are there scenarios that make it more useful than the other stacks? All other things being equal, if performance or memory use matters, it looks like OpenBGP is the loser here.
Not sure the winner, maybe FRRouting 7.5.1? Is that enough to not use FRRouting 8? I doubt it, but I’m not testing functionality, just straight up speed on these simple tests. There are some cases in which RustyBGP is faster or uses less memory.
I’m not really going to declare a winner.
Try out bgperf yourself.
Questions
There are probably patterns in here that I missed. What analysis did I miss looking through this data?
Are the scenarios that I’m using to test useful? Are there different combinations of prefixes and neighbors that I should be testing?
Does anybody else have either BGP testing tools that they can share, or other results that they can share?
followup / todo
Still more to do to be more representative of real loads and not so artificial.
