Trustless Upgrades in Solidity

Rob Hitchens and Ali Azam

Photo by James Hammond on Unsplash

A few weeks ago, we published Selfdestruct is a Bug, itself inspired by Upgradeability is a Bug by Steve Marx of Consensys Diligence. The gist of both pieces is that changing contracts undermines immutability. One might well ask — what is so special about blockchain if the code can be changed at the whim of a privileged user?

If You Aren’t Losing Sleep, You Might be Doing it Wrong

Ethereum Dapp Developers have to deal with an unfamiliar paradigm (blockchain) and an unfamiliar language (Solidity) to create unfamiliar solutions (governance, assets, rights) to unfamiliar problems (game theory, business process re-engineering, marketplaces, and more). It’s of little wonder that upgradability is a welcomed possibility.

Upgradable Contracts To the Rescue!

In 2016, Coloney published Writing Upgradable Contracts in Solidity. The gist of their “Eternal Storage” pattern is

… isolate your datastore from the rest of your code, and make it as flexible as possible, so that it is unlikely to need to be upgraded.

At the risk of grossly over-simplifying, you can have a data storage contractthat is so simple, so flexible and so well-solved that you can be confident that it won’t ever need revision. At least, that’s the idea. For example, you might settle on a key/value store and then stop.

Consider something simple:

contract EternalStorage is Ownable { 

  mapping(bytes32 => uint) UIntStorage;

  function setUintValue(bytes32 record, uint value) public {     UintStorage[record] = value;  }

  function getUIntValue(bytes32 record) constant returns (uint){    return UIntStorage[record];   }}  

// the end

Surely, we can QA something like that and call it a win. We might congratulate ourselves about how that much, at least, should work forever.

Another contract, the logic contract, owns the storage. Ownership is transferable to another logic contract. This structure offers the possibility of swapping the old logic contract for a new one. Et Voila! Subject to getting that data layer right, you can have an upgradable system of contracts. Have a look at RocketPool for an example of a project that pursued this approach in a system with multiple contracts.

A frequent criticism of this approach is the complexity that bubbles up to the logic level. Everything persistent needs to be described in a way the storage contract will understand. That can, and does, add complexity and diminish readability because reducing all variable names, arrays and mapped elements to unique keys for a key/value store adds extra steps in the contracts that store everything this way.

For example, you don’t get to say:

counter++

You might have to say:

counter = datastore.get(keyForCounter);datastore.set(keyForCounter, counter++);

Not horrible, but visual noise adds up.

DelegateCall, Libraries and Proxy Patterns

With the Homestead hard fork, the EVM picked up an OPCODE called DELEGATECALL. As the name suggests, a contract will “delegate” a function to another contract. The Solidity team added librarythat uses it.

A librarylooks a lot like a contract but with a possibly strange-seeming limitation. They can’t hold any data. That’s because they run in the context of the contract that decided to DELEGATECALL. So, if contract I decides to DELEGATECALL to a library function called makeBed, then it is I’s bed (mine) that gets made, not that of the library. If I want to upgrade the makeBed process, I need a new library and I need the contract to start delegating calls to the new library.

You can have a main contract that delegates everything important to one or more libraries.

pragma solidity 0.5.1;

library DoStuff {    struct DataStore {        mapping(bytes32 => uint) value;    }    function setVal(DataStore storage self, bytes32 key, uint value)        internal     {        self.value[key]= value;    }    function getVal(DataStore storage self, bytes32 key)         internal view returns(uint)     {        return self.value[key];    }}

contract Switchboard {    using DoStuff for DoStuff.DataStore;    DoStuff.DataStore data;    function set(bytes32 key, uint value) public {        data.setVal(key,value);    }    function get(bytes32 key) public view returns(uint) {        return data.getVal(key);    }}

This isn’t a complete tutorial about libraries,but here are some quick takeaways:

• The actual values are stored in the SwitchBoard, but the library defines the layout of the data types it understands.
• A storage pointer gives the library something to work with. Thelibrary writes directly to Switchboard's storage. You might say, Switchboard authorized it to do that when it passed a storage pointer.
• There is syntax sugar in play, and that’s why a) there seems to be a missing argument (count them) and b) the library functions are magically invoked as methods for data that was defined as an instance of the struct (layout only) in the library.
• The contract functions are deliberately simple. They just take the inputs and pass it through to a library that should know what it’s doing.

A nice thing about libraries is that once you catch on to the overall idea, they don’t call for too much adjustment to familiar contract coding styles. In summary, we have a library that will “do stuff” to any appropriately laid out DataStore you throw at it.

Functions Need Upgrades, Too

For brevity, let us set aside the mechanism that will allow us to swap in a new library and just accept that can be done with a little more effort. We won’t dwell on this because there is a bigger problem. What if we want to add a function that the SwitchBoard doesn’t do? What if we want to add an argument that one of the existing functions doesn’t accept?

It’s not entirely reasonable to assume that the function signatures can be worked out for all time, in advance. The future is unknowable and this is, in part, why we might be interested in upgradable contracts.

A “Proxy” contract can solve for a flexible interface by forwarding anything to an implementation contract, and returning anything that comes back. Assembly is, by its nature, a little down in the weeds for my taste but this assembly code has been in existence for a few years now. It is a very popular and well-known pattern.

function () external payable {  assembly {    let ptr := mload(0x40)    calldatacopy(ptr, 0, calldatasize)    let result := delegatecall(      gas,       implementationAddress,       ptr, calldatasize, 0, 0)    let size := returndatasize    returndatacopy(ptr, 0, size)    switch result    case 0 { revert(ptr, size) }    default { return(ptr, size) }  }}

• First, it’s a fallback function, so it runs when the Proxy contract can’t find a match. It’s payable because it should support payable implementation function, to be useful.
• It copies the incoming calldata and passes it to delegatecall which sends it on to an implementationAddress that, hopefully, knows what to do. implementationAddress is a contract address we will, for now, presume was set up in advance.
• Maybe the implementationAddress doesn’t know what to do, or maybe it reverts. delegatecall would return 0/false in that case, so this implementation checks that and reverts if the implementationAddress wants to revert
• The function returns the implementationAddress response if the operation was a success. It has no idea what the inputs or outputs mean but you can use the ABI of the implementation contract at the address of the Proxy contract. This slight of hand creates a convincing illusion of working directly with the implementation contract when, in fact, one is channeling everything through the Proxy.

One of the very nice features of this pattern is that the implementationAddress is a contract that doesn’t necessarily know it’s upgradable. It has state variables laid out in the usual way, using the usual syntax, but, in fact, everything is written to the Proxy contract's state.

This is so popular, in fact, that Zeppelin use it as the basis for ZeppelinOS’s Transparent Proxy. You will also find it in Gnosis Safe.

One of the big caution items you will discover is the very realistic possibility of overwriting important state data if a replacement implementation is defective or nefarious. It is important to establish a framework and/or practices that ensure that such an accident isn’t possible.

Even if everything works as expected, it does not seem to address an important concern that arises from upgradability itself. Let us return to the higher-order concern, that upgradability itself diminishes immutability. Let us consider what we might do to mitigate this concern.

When is Upgradability NOT a Bug?

In this overview we’ve touched on Eternal Storage, Switchboards and Transparent Proxy patterns, any one of which might be of interest if you care to dig in and research more about how.

You will still arrive at a troubling question. Why should anyone trust your contract if it’s upgradable?

After all, isn’t the whole point of this form of software about creating certainty? What happens to certainty if a privileged user can change the implementation? It disappears. You might as well write the software in your favorite language, deploy it on your favorite platform and use traditional measures to convince your users that you, yourself, won’t do anything that disadvantages them. Odds are you will save yourself a lot of trouble.

I first heard of Ali Azam’s approach to this problem from his presentation at DevCon IV. His approach is to require the consent of the users. Not all of the users, mind you. Not a democratic process that could leave a minority of users forced to accept an upgrade they do not want.

Each and every single user should decide, for themselves, if they want to continue with the contract they already signed up for, or migrate voluntarily to a new version.

The Registry

This is a more granular upgrade process with each user potentially using their favorite implementation contract. The Registry contract is tasked with recording the available implementation contracts, and each user’s preferred implementation.

This setup looks something like this:

mapping(address => address) userImplementationChoices;

The Transparent Proxy introduced earlier accommodates that by looking up the user’s preferred implementation and the carrying on with the Proxy mechanics as described earlier:

address implementationAddress = userImplementation(msg.sender);

The function userImplementation(address) works out a few details and returns an implementation contract address, for the user. It is the user that decides if they want to select a certain implementation or opt-in to push updates.

Trustless Upgrade means the user decides.

Astute readers might realize that different backing contracts could imply the need for different client-side components such as the UI. Yes, indeed, and the way to find out which ABI is in play is to ask the Registry.

function userImplementation(address user) public view returns(address) {

More Decentralized, By Design

It would probably be madness to let everyone use any contract they like, especially when they share communal storage in a system. No, we want to limit this to valid the choices the developers and QA professionals agree are compatible.

The Registry allows a privileged user to approve a new contract that users can choose, if they want to. The “privileged user” can be a governance contract wherein users deliberate about applicants that should even be admitted.

The flow can easily accommodate unambiguous proposals and approvals.

• Someone deploys a new implementation.
• Someone proposes that the contract at 0x123... should be added to Registry’s list of valid implementations.
• Decision, add to Registry with:

function addImplementation(address implementationAddress)   public   onlyOwner 

Importantly, no one can be forced to move away from the original contract, as it was, when they signed up. And, the users don’t have to trust that the upgradability feature won’t be used to their disadvantage. It is designed to facilitate the opposite of that — a transparent process where review of the code and public deliberations are the norm and there is no possibility of sidestepping the agreed governance policy.

Default Implementation and Other Administrative Concerns

With the basic principle of operation out of the way, the remaining concerns are largely administrative. Users can either:

• Opt in, and accept new versions as they appear, or
• Opt out, and accept no new versions unless they manually change their preference.
• The default choice of all new users (opt-in or opt-out) is a Registry setting that cannot be amended post-deployment, because changing the users’ understanding of the choices they have and the processes they can depend upon would be a bug.
• An Emergency Recall function allows the privileged user to deprecate a release. This will migrate affected users, only those who were using the deprecated release, to another recommended choice. The privileged user controls the suggested version, known as the default implementation. Recall is the only case where users can be forcibly migrated to something else. An alternative approach would be to simply disable implementation for the affected users until they explicitly choose another version. We have designed this in the interest of user experience, with the expectation that governance would prevent a nefarious implementation from even making it to the list.

Version Style Guide

The module includes an inheritable contract that addresses a security concern, namely that no one should invoke an implementation contract directly (i.e. without going through the Proxy). That would make no sense because the data resides in the Proxy, not the implementations. A modifier, onlyProxy blocks that inappropriate invocation.

As a general rule, subsequent versions of an implementation should inherit from previous implementations. Doing so ensures the state layout always evolves in an additive way. Overriding functions is acceptable. Adding new functions and new arguments is acceptable. Disabling functions is acceptable (override with revert("deprecated");).

The sample implementation includes a Hello World contract and a second version called HelloUniverse. The concerns described above are all addressed with minimal intrusiveness:

contract HelloWorld is Upgradable {

  function doSomething() ... onlyProxy ...{    ...

contract HelloUniverse is HelloWorld {  ...

That’s it.

Wait. There is one other thing. The registry uses an arbitrary bytes32 called “componentId”. This will be generated when a Proxy is deployed, which itself deploys a corresponding Registry (One Registry per Proxy and one Proxy per upgradable contract).

componentIdis easily inspected with Registry.componentId() and it’s value must be passed into each upgradable contract. For example, in Hello World:

constructor(bytes32 componentUid) Upgradable(componentUid) public {

This facilitates a rudimentary check to help catch deployment-time errors such as putting implementations of the wrong component into a Registry, in production. The Registry will reject implementation contracts that don’t claim to implement the expected component.

Variants

The example code approaches implementation as a user-by-user concern. The idea of a context-aware Proxy can be applied in other ways. For example, consider off-chain data and on-chain validation contracts. It would be possible to use an upgradable validation contract that gets a signal from off-chain assets — A doc that says, approximately, use validation version 3.2 to parse me.

The code

The experimental repo, https://github.com/rob-Hitchens/TrustlessUpgrades may, from time to time, be a few commits ahead of the community repo, https://github.com/ali2251/Upgradable-contracts

Ali’s Trustless Upgrades documentation and workshop handouts are available online at https://docs.upgradablecontracts.com/.

Rob Hitchens is a Canadian smart contract design consultant, co-founder of Ethereum smart contract auditor Solidified.io and a courseware co-author and mentor of Ethereum, Hyperledger Fabric, Hyperledger Sawtooth Lake, Corda, Quorum and Tezos bootcamps by B9lab.

Ali Azam is a senior Blockchain Developer at Vaultplatform, previously worked at King’s College London for over two years and has presented various workshops and keynotes at King’s College London, Devcon4, Blockercon and many other places. Ali is a senior mentor and a core member at Work on Blockchain monthly blockchain developer bootcamp and leads Smart Contract Upgradability Workshop held in London.

Trustless Upgrades in Solidity was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.