A Journey into Stellarator: Part 2

Motoko’s Enhanced Orthogonal Persistence

By Luc Bläser

In this 3-part series, we will unveil the technical achievements that will enable significant data handling improvements for applications running on the Internet Computer Protocol (ICP).

This upgrade refers to the Stellarator milestone of the ICP roadmap, which is now being rolled out across the network. Stellarator is a breakthrough in on-chain data persistence, substantially increasing scalability and handling of data storage, unlocking new opportunities for complex data-rich applications previously limited by system constraints.

This advancement allows developers to build sophisticated applications requiring large-scale data processing, bringing a new level of utility to blockchain technology.

For the second installment of this series, Luc Bläser will take us through Motoko’s Enhanced Orthogonal Persistence. If you missed the first part on data persistence, you can find it here.

Simple, Safe, and Fast: Motoko’s Enhanced Orthogonal Persistence

Motoko introduces enhanced orthogonal persistence, a distinct feature that aims to relieve programmers from dealing with stable memory by offering an upgrade mechanism that is simple, safe, and fast at the same time.

Motoko has always been able to automatically preserve program state across upgrades without any additional code to handle persistence. Unfortunately, the previous implementation of this feature did not scale to large or deeply nested data.

This feature has now been substantially enhanced in terms of safety, performance, and scalability. The key innovation is to avoid any transfer of state to stable memory by simply retaining, not erasing, the main memory.

The runtime system now ensures data consistency on upgrades in a very efficient way, independently of the memory size. Finally, we switch from a 32-bit to a 64-bit main memory to eventually scale to huge amounts of persistent data.

Background

Canister upgrades are a challenging aspect on the Internet Computer, often involving substantial complexity, overheads, and some risk of data loss. Although the main memory (also called the WebAssembly memory) of a canister is persisted across transactions, in the past it was erased on upgrades, a rather artificial step considering that the memory is already file-backed on the Internet Computer.

The reason for this behavior is because mainstream programming language implementations are not designed with persistence in mind: they rearrange memory structures in an uncontrolled manner on re-compilation or at runtime and cannot simply resume execution of a modified program with the memory of its previous version.

As a consequence, programmers had to explicitly use the stable memory API or special stable data structures to preserve their data across upgrades. This is similar to traditional computer architectures that provide a volatile main memory and persistent secondary storage at the cost of increased complexity in software development (think of object-database mappers, etc.).

Differently, in Motoko, we have full control over the compiler and runtime system and thus also over the memory layout. This allows the support of orthogonal persistence in Motoko that also survives upgrades.

As a result, Motoko programmers can conveniently develop any data structures (of first-order types) in the standard language concepts without having to use explicit stable memory, dedicated stable data structures, or other database-like abstractions. The runtime system automatically persists the necessary objects.

Previously, the Motoko runtime system implemented orthogonal persistence by serializing and deserializing the persistent object graph to and from the stable memory. This caused severe scalability and performance issues, since the serialization / deserialization process is very expensive and may even exceed the instruction limit, ultimately preventing upgrades. These pitfalls made it impractical for larger applications.This is why we redesigned the orthogonal persistence support in Motoko.

By changing the Internet Computer and Motoko runtime system, we enabled scalable upgrades without expensive serialization and deserialization to and from secondary stable memory space. Instead, we simply keep the main memory persistent across upgrades. At the same time, we extended the address space for orthogonal persistence to 64-bit to (in future) scale to the same capacity offered by 64-bit stable memory.

Enhanced Orthogonal Persistence

Our goal is to simplify software development on the Internet Computer and relieve programmers from the burden of dealing with stable memory. For this reason, Motoko has been optimized for persistence on the Internet Computer by enabling upgrades that are simple, safe, and fast at the same time:

• Simple: By way of orthogonal persistence (the stable variables in Motoko), transitively reachable structures of any first-order type are automatically persisted across upgrades. No stable memory or stable data structures are needed.
• Safe: The runtime system rigorously checks type compatibility on upgrades and supports several kinds of data changes by implicit migration. Any more complex migration can be implemented by custom code. This prevents any data corruption or misinterpretation at the memory level.
• Fast: Upgrades become super-fast because the main memory is simply retained on an upgrade. No copying to and from stable memory is needed. The main memory has been extended to 64-bit to scale as large as stable memory in the future.

Design Overview

As prerequisites for enhanced orthogonal persistence, the Internet Computer Protocol was extended to support main memory retention across upgrades and 64-bit main memory based on WebAssembly Memory64 proposal.

With a bespoke compiler and runtime system design, Motoko defines a compilation-invariant memory layout with all objects being allocated in the dynamic heap, accompanied by ample metadata that allows new program versions to safely pick up the state of previous versions.

WebAssembly passive data segments thereby prove to be useful as they permit lazy allocation of static program data (such as text literals) in the runtime system without needing to reserve static address ranges.

We even persist the incremental garbage collector (GC) state across upgrades. This means that upgrades can happen at any time, without having to await completion of a GC run.

To ensure rigorous memory and type safety, the runtime system stores the types of the current program version and uses this information to check the memory compatibility when trying to upgrade to a new program version.

Since this check only depends on the number of types, and not on the number of objects, upgrades are extremely fast and scale to arbitrary memory sizes. Certain data migrations, such as adding or dropping stable variables, promoting types, add variant option, etc. are automatically supported, while any more complex migration can be manually programmed while continuously guarding type safety.

Motoko implements automatic data migration from the old classical persistence to enhanced orthogonal persistence. To allow potentially radical memory layout changes in the future, which we anticipate to be a seldom case, Motoko also includes a secondary persistence mechanism that is based on stable memory serialization with a graph copy algorithm, and an unlimited kind of deterministic time slicing that is not bounded by any instruction limit of the Internet Computer Protocol.

You can find more details about enhanced orthogonal persistence, its design, its implementation, data migration scenarios, and performance evaluations in our recently published paper: Smarter Contract Upgrades with Orthogonal Persistence.

Production Rollout

In a first step, enhanced orthogonal persistence is available as an opt-in feature that can be activated by the compiler option '--enhanced-orthogonal-persistence'. The corresponding argument can be specified as follows in dfx.json:

{ 
  "main": "your-program.mo",
  "type" : "motoko",
  "args" : "--enhanced-orthogonal-persistence",
}

Motoko automatically migrates from old 32-bit classical persistence to enhanced orthogonal persistence. However, please note that the reverse direction, downgrading from EOP to 32-bit classical persistence, is not supported.

In the current version, the Internet Computer only exposes 4GB of main memory, despite our switch to 64-bit. This is only a conservative measure as a first step, where we follow a risk-averse launch, first starting small and then gradually increasing capacity over time.

In fact, because of the 64-bit switch and the starting limit of 4GB, net memory usage is expected to be smaller compared to 32-bit in this initial phase (due to the doubled pointer sizes). Instruction costs have also been increased for 64-bit memory accesses to conservatively cover the hardware costs. This may be reduced in future.

Most importantly though, upgrades will become extremely fast and no longer hit the Internet Computer instruction limit, even for maximum heap usage (also considering that upgrade hooks are no longer needed).

In a next step, after collecting feedback and potentially refining the system and expanding memory capacity, we plan to make enhanced orthogonal persistence the default mode.

Conclusion

Motoko’s enhanced orthogonal persistence allows Internet Computer developers to focus on their core application without worrying about stable memory.

This not only enables simple and safe persistence, but also significantly reduces the cost of upgrades and data accesses on the Internet Computer.

Enhanced orthogonal persistence is something that can only be achieved by a bespoke compiler and runtime system, like the one we now have for Motoko.

More information:

• Luc Bläser, Claudio Russo, Gabor Greif, Ryan Vandersmith, and Jason Ibrahim. Smarter Contract Upgrades with Orthogonal Persistence, VMIL 2024. https://dl.acm.org/doi/10.1145/3689490.3690401
• DFINITY Forum Post: Beta-Testing Enhanced Orthogonal Persistence. https://forum.dfinity.org/t/beta-testing-motoko-s-enhanced-orthogonal-persistence-eop/35787
• Documentation: Motokos’s enhanced orthogonal persistence. https://internetcomputer.org/docs/current/motoko/main/canister-maintenance/orthogonal-persistence/enhanced
• Documentation: Stable variables, upgrades, and data migration in Motoko:
  https://internetcomputer.org/docs/current/motoko/main/canister-maintenance/upgrades

We are happy to hear your feedback. Please share your thoughts on the DFINITY Developers X channel or on the Motoko GitHub repo. And join us tomorrow for Part 3 of A Journey into Stellarator, with Kamil Popielarz and Yvonne-Anne Pignolet on increasing ingress message throughput.

A Journey into Stellarator: Part 2 was originally published in The Internet Computer Review on Medium, where people are continuing the conversation by highlighting and responding to this story.