At the end of August 2019, Whiteblock received a joint grant from the Ethereum Foundation and ConsenSys. Read our blog post for more details and find the official announcement from the EF here.

Specifically, our work was to focus on benchmarking Gossipsub, a general purpose pubsub protocol. Gossipsub works with Libp2p, the networking stack for many Web3 projects, including Eth2. These projects are maintained by a set of contributors shepherded by Protocol Labs.

The main output of our research was a detailed report, which can be accessed on our GitHub. Read on for a high level overview of the research findings. We ran tests in a variety of conditions to prove the gossipsub’s viability as a key networking component.

Initial Parameters and Peer Topology

Here are the parameters and their settings:

• Number of Gossip Nodes: 95
• Global Msg/Sec: 200
• Warm-up Time: 60s
• Test Time: 180s
• Cool-Down Time: 600s
• Msg Size: 1000 bytes
• Discovery (mDNS): Off
• Routing (kad-DHT): Off
• Security: SECIO
• Peering: Barabasi-Albert (seed=42, varying input degree-parameters) - { 2, 6, 12, 16 }

The Barábasi-Albert (B-A) model is a way to describe the connections between nodes in a distributed network. The topology works with a degree parameter, m, as input. The graph below illustrates how the power-law node connection plays out.

It is important to note that Eth2’s topology will depend on the discovery protocol implemented. That being said, it’s still useful to test a known network topology model like B-A.

Topology Test

Now let’s get into some of the results. The first test run kept all parameters as constants except the B-A input degree parameters:

For each increase in B-A degree inputs, the general shape of the message dissemination time distribution stayed the same. While there are slight differences in each histogram, the initial spike of short nano times and heights of the first lobe did not follow a particular trend. The effects of degree distributions on message dissemination time are inconclusive. The presence of lobes cannot be explained by the GossipSubHeartbeatInterval[6], which was left set at the default of 1 second. A larger set of test runs would be needed to determine the effects of degree distributions on message dissemination time.

Test Series

Following the control tests, 6 additional tests were run which introduce gossipsub to various network impairments. See the full report to explore test parameters:

• General topology baseline
• Latency
• Packet loss
• Bandwidth
• High Latency
• Stress Test

These tests present some stress to the protocol to uncover any issues of the go-libp2p implementation. Tests were run via the Whiteblock Genesis platform using a fork (with many changes) of the Phase 1 Libp2p Host implementation, and random topologies generated using Barabasi-Albert model. Generally speaking, the degree distribution of a network affected the Last Delivery Hop (LDH). The LDH is the number of hops a single gossip message has made when it reached the absolute last receiving node (i.e., to complete message dissemination). There is a correlation between the degree of connectivity and last delivery hop.

As shown in the graph above, we can see that as the degree was increased, the average delivery hop count decreased. We suggest that the Eth2 peering technology should avoid a power law degree distribution resulting in edge nodes with peer sets of 2. This will avoid the low performance seen in all Phase 2 tests with B-A input m = 2 (the first column).

Message Dissemination (“Total Nano Time”) Results

The total message dissemination times logged indicate the amount of time each message takes to reach its final destination. The network paths gossip messages take is entirely dictated by the gossipsub protocol as well as the generated network topology. The message time varies from test to test, as shown in the results, and the sparser networks (lower degree of connectivity) are found to have longer times to reach its destination than the tests that had denser network topologies.

Dissemination times for each of the test series are overlaid in the graph below, each a different color. It’s clear to see that different network impairments have a direct relationship to the time it takes for a message to traverse the network.

Next Steps — Community Solicited Research

While the first two phases of this testing effort shows that the Libp2p Gossipsub protocol performs sufficiently well for Ethereum 2.0 specifications, testing with a larger network size should be the primary focus for future work. Before testing larger network sizes, various improvements to the go-libp2p-pubsub implementation as well as the testing methodology should be completed. Below is a non-exhaustive list of todos for the community:

• Permanent fix which does not require manually increasing Golang channel queue sizes in go-libp2p-pubsub should be implemented.
• go-libp2p-pubsub-tracer should be integrated into the node implementation for tests
• Erratic message interarrival times should be investigated and fixed
• Testing methodology should port to the next version of Whiteblock Genesis for larger scale tests
• Integrate floodsub into testing methodology for comparison.
• With these additional features, a richer set of future tests can be performed. Lastly, future work should include the variation of gossipsub parameters (e.g. GossipSubD, GossipSubHeartbeatInterval, etc.) at different network sizes.

We’d like to again thank ConsenSys and the Ethereum Foundation for funding this research. Additional thanks to Protocol Labs, Raúl and Proto for consistent feedback on our strategy.

If you’d like to try out the gossipsub tests on your own, the yaml can be accessed in our sample repo:

We’d love to see some additional testing from other community members with other parameter settings. If you have any feedback to share, please join the discussion on our community forum or our telegram.