New P2P layer of the Internet Computer introduces QUIC for State Sync

Until now, state sync on the Internet Computer used the legacy P2P layer, which works over TCP. The new P2P layer is tailored for state sync and uses the QUIC transport protocol, making it simpler, more performant, secure and ready for future evolution.

Author: Yotam Harchol

Introduction

The Internet Computer blockchain relies on a peer-to-peer (P2P) protocol that distributes messages among nodes of each subnet. These messages are generated by the Internet Computer protocol’s clients. These clients each implement higher-layer protocols, such as the Internet Computer Consensus protocol, or the state-sync protocol, which this article will explain in detail later.

Essentially, each client has very different requirements and behaviors. Until now, a single P2P component was used for all intra-subnet communication, by all protocol clients. This approach led to complex code and sub-optimal performance, making it harder to introduce changes in the implementation.

The new P2P layer for state sync decouples the P2P communication of different clients, simplifies the code, and makes it possible to introduce new features and improved networking for the Internet Computer. Part of this is achieved by adopting the QUIC transport protocol, which replaces the currently used TCP. QUIC has many advantages over TCP in the setting of the Internet Computer — first and foremost, the ability to multiplex multiple streams.

At the time of this blog post, the NNS (the DAO that governs the Internet Computer) has already begun with the rollout of the new P2P layer for state sync as a response to the community’s proposal adoption of replica version cabe2ae3 for eight subnets. This means the new QUIC transport is now active on all of the Internet Computer subnets. However, state sync is using the new P2P layer only in a smaller set of subnets for the moment.

This article explains the technical changes in detail, and how the new QUIC features are being leveraged to create a new and improved P2P layer for the Internet Computer.

The peer-to-peer protocol of the Internet Computer

The peer-to-peer (P2P) layer of the Internet Computer is responsible for message delivery within subnets of nodes. Each subnet operates a separate peer-to-peer network, and nodes within each such subnet use this network to send messages to each other.

On top of this layer there are multiple components that use the P2P layer to exchange messages between peers, the most famous being the Internet Computer Consensus protocol. Other components, including the state sync protocol, also use the P2P layer to exchange messages with peers on the same subnet.

The state sync protocol of the Internet Computer

Figure 1: The state sync protocol of the Internet Computer.

At a very high level, the state sync protocol of the Internet Computer enables nodes, for example, ones that have fallen behind, to synchronize the replicated state of the subnet without having to (re-)execute all the blocks in the respective subnet blockchain. Instead, they can immediately download the required state, and verify its authenticity relative to the subnet’s chain key, using this state sync protocol.

The state sync protocol works as follows: Up-to-date nodes create a so-called checkpoint every couple of hundreds of rounds (usually 500). This involves writing the replicated state to disk, computing a hash of this state, and consensus attempting to agree on this state by including the hash in so-called catch-up packages (CUPs). The hash is computed in a manner that makes it impossible to come up with two different states that have the same hash, which means that the existence of an agreed-upon CUP for a particular height also implies that (1) the majority of nodes agree on the state at a particular height and (2) a majority of nodes is actually able to serve this state as part of state sync.

The process is depicted in Figure 1: Nodes periodically advertise all state versions they have available locally to their peers on the same subnet (refer to as an advert). This version is essentially the height of the block to which the state corresponds, and the hash of this state. If a node sees a CUP that is more recent than its local state, it can conclude that it has fallen behind and can request the state corresponding to this CUP from its peers the next time it sees an advert for this state. The protocol also ensures that unchanged pieces of the state available to a node are not re-downloaded, but can directly be used by the node. The state can be viewed as a file tree, where each file is, in turn, split into chunks. The resuming node may have many of the chunks already, but possibly not all of them. A node can simultaneously request chunks from multiple peers, similar to BitTorrent.

The resuming node starts by requesting a special chunk called the manifest from the peer who sent the advert that triggered the state synchronization. This manifest contains a list of all files and chunks that correspond to this state. There is also a hash and some metadata included for each file and each chunk. Similar to a Torrent file, the manifest does not include peer information and its content does not depend on any specific peer. The manifest’s hash is the hash that is included in the CUP. Once the hash of the manifest is verified against the hash included in the CUP, one can conclude that the file and chunk hashes in the manifest are authentic and can be used to verify files and chunks by comparing their hashes against the hashes included in the manifest. This mechanism ensures that nodes cannot trick each other into downloading and using fake state chunks.

After downloading the manifest, the resuming node knows exactly which chunks it is missing. It can then trigger multiple simultaneous download requests to other peers — all peers that advertised the same state — to fetch the missing chunks as soon as possible.

Using QUIC to simplify P2P for state sync

Figure 2: The design of the new P2P layer for state sync

So far, the state sync protocol has used the existing P2P layer of the Internet Computer. As hinted above, this P2P layer multiplexes messages from multiple clients and disseminates between peers on the same subnet. However, it is designed for the distribution of relatively small messages, which is the requirement of most clients, but not of the state sync protocol. Therefore, until now, we had a single monolithic P2P protocol that tried to solve the problems of all clients. However, this makes the code more complex than necessary as well as harder to improve it.

For these reasons, separating the P2P layer into two makes a lot of sense: one for state sync, and another for the rest of the (existing) clients. Together with our ongoing effort to simplify and enhance the P2P layer for other clients, this would form a new P2P layer that is much simpler, more performant, and more secure.

The new P2P layer for state sync, shown in Figure 2, does not change the state sync protocol but only the networking layer below it and uses a new transport component. The transport component is the one used by P2P to create and maintain connections between peers. For the new P2P layer for state sync, we created a new transport component that supports two async API functions:

• push(message, peer_id)
• rpc(message, peer_id) -> response

Because transport used TCP, with a single stream, it was impossible to relate requests to responses without keeping state and tracking requests and responses in Transport. The new transport component uses the QUIC [link to IETF draft] transport protocol. QUIC is a modern transport protocol that is implemented on top of UDP in user space. It has many advantages over TCP, the most important one in our case is the ability to multiplex multiple streams over one connection. Some other blockchains already use QUIC.

On top of this new transport component, two new P2P components will run: one for state sync, and another for other clients (see below). The new P2P component for state sync uses the new transport layer as follows: periodically, it calls push() with the current states, advertising them to all peers. When receiving such adverts, if a node notices it is behind, it uses the rpc() calls to request specific chunks from peers.

Figure 3: Multiplexing over the QUIC connection

Benefits of switching to QUIC

The QUIC stream multiplexing enables us to transform the P2P layer code to be completely asynchronous, and hence better utilize the available CPU and bandwidth resources. It also helps us avoid head-of-line blocking in request processing, and improves the reliability of the code.

Another benefit of using QUIC is that due to the multi-stream structure of the connection, every response is tied to a corresponding request, without having to maintain state for that in the application-layer.

Using the new transport for other clients

As mentioned above, we are also working on revamping the P2P layer used by other clients. Our plan is to also make it use the new transport component, and therefore eventually have messages from all clients multiplexed on the same QUIC connection, obsoleting the TCP transport. Using QUIC could help dynamically prioritize traffic of different clients. For example, since state sync may be a bandwidth-heavy operation, we may want to prioritize it over other clients, so it finishes as soon as possible. However, if many nodes on the same subnet are state-syncing, we might also want to make sure that the consensus protocol has precedence over state sync, otherwise it might not make progress fast enough.

The revamped P2P layer for consensus, which is currently being implemented, significantly improves the existing P2P layer in several aspects including scalability, performance, and reliability of the P2P and the Consensus layers. We will share more information on this topic soon.

Ways to improve the state sync protocol

With this change to the P2P layer, it will be easier to extend it to accommodate new features in the state sync protocol. One such feature is the ability of nodes to download chunks from peers that do not have the entire state. This will significantly improve the performance of the protocol especially when multiple nodes in the same subnet need to synchronize their state.

Read the paper on the Internet Computer Consensus protocol in PODC 2022.

Learn more about the Internet Computer: internetcomputer.org
Follow the tech on Twitter: @DFINITYDev

Start building on the Internet Computer:

Developer Docs | Internet Computer

---

New P2P layer of the Internet Computer introduces QUIC for State Sync was originally published in The Internet Computer Review on Medium, where people are continuing the conversation by highlighting and responding to this story.