Enforcing invariants in a complex system

By Misha Zharov

TL;DR: Coinbase is a leader in securing crypto assets. As part of our proactive approach to security, we’re introducing a new security control that will provide guarantees about our transactions.

Problem Background

What is an invariant?

In Computer Science, an invariant is a property of a system that holds true during some operations being performed by the system. Invariants are important for ensuring that a given system is behaving the way it was designed to.

At Coinbase, we have certain invariants that we enforce in order to provide guarantees about how transactions are processed. For instance, when a client submits a withdrawal request, the system which processes the transaction could have an invariant which states that the destination of the withdrawal cannot change after the client has submitted it. Such an invariant would allow us to confidently process the transaction.

But what happens when an invariant is broken? This could lead to a bug which degrades the performance of our platform, or perform an action that does not meet the customers intent. For instance, if the previously discussed invariant was broken, the customers funds could be misallocated.

Enforcing invariants through a centralized validation system

In order to maintain invariants, we have a system called “IVy” (Integrity Verification) which enforces these ground truths, and notifies the team when they are violated. Its job is to make sure that transactions are not modified in unexpected ways while other systems are processing them, and to act as a centralized validation framework. This decreased the complexity of existing systems and created an extensible framework for enforcing new invariants as needed.

Impact

A continuous check on intent

It is critical that we broadcast transactions accurately because if we do not, funds will not be allocated properly. Furthermore, the complexity of dealing with a variety of fields used by different cryptocurrencies greatly increases the number of validation steps required and we want to avoid duplicating this logic everywhere. We designed IVy in a way that allows clients to sign their payloads once and then these payloads can be validated anywhere. In general, since various currencies have a lot of differences in what is valid versus what is not (like some currencies having memo fields) we decided to centralize the validation logic for transactions in one place so that we can implement invariants once, and use them everywhere.

Design

We designed this security control as a standalone service with a simple API. The endpoints can be used by other services to feed data to our service, and query whether a transaction is valid. The Integrity Verification system would use the information provided by other systems to verify that the transaction has not been tampered with, and that it’s not malicious.

This security control records the withdrawal request of the customer on the backend as soon as it is submitted. This is shown in Fig. 2., which describes how the withdrawal transactions immutability is preserved.

Fig. 1. the original design. Requests come in from the web and hit the custody backend. Then transactions are funnelled to cold storage (which has an online, offline, and human component). The red dashed connections actually have many other components in between which may drop or change fields from the transaction. Then the transaction is broadcast.Fig. 2. the new design. Transactions are stored as early as possible in the process, and are verified again right before broadcast (see the transaction coming into the custody backend, and being simultaneously recorded and forwarded to Cold Storage).

API design

For this service to be useful and effective, clients need to be able to interact with it. As a result we designed a robust API on top of gRPC. gRPC allows us to have flexibility in the future when we want to update the API because the client middleware code can be regenerated easily. As a result, we can store a customers intent to verify later like so:

# Create the payload # and add metadata   payload = TransactionRequest.new(     unsigned_transaction: UnsignedTransaction.new(       transaction_data: CbTransactionData.new(         transaction_id: id,         currency_ticker: currency_symbol       ),     # unsigned_tx is a json string       # with the unsigned transaction       unsigned_tx: unsigned_tx           )   )   # Send the request to the# Integrity Verification service client.register_transaction(payload)

Later when a service would like to check if a signed payload matches the user intent, we can use Rosetta to parse the signed transaction and compare it with the user intent.

Verification

Before broadcasting, we verify the immutability invariant for transactions by sending the signed transactions to IVy. Since we have previously registered the user intent with IVy, we’re able to compare it with the signed copy. This ensures that the transactions have not been modified in undesirable ways.

The way we implemented the comparison/validation logic yields some usability/flexibility benefits. Each currency has its own validator which enforces the necessary rules for that currency. Furthermore, there are some general utilities which can be used by each validator to perform common checks. Also, all of the parsers conform to the same interface which allows us to generically integrate with them on the backend. As a result of these ideas, the parsers are lightweight and easy to review. Since the parsers have a very narrow scope, they are easy to write which gives us flexibility in how new currencies will work, or to make changes to a specific subset of currencies without affecting others.

Impact

As a result of our design, we were able to verify invariants related to our backend systems. This security control eliminates an entire failure mode class, and makes our platform more reliable.

Future work

As IVy evolves, additional transaction properties and policies will be developed and enforced. These properties and policies will become a core component of integrity and customer intent at Coinbase.

If you are interested in securing the cryptoeconomy, Coinbase is hiring.

This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

All images provided herein are by Coinbase.

Enforcing invariants in a complex system was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.