Porting a ZK Rollup Payments Solution from Ethereum to RSK
By Raúl Laprida, Julian Len, Shreemoy Mishra, and Diego Masini
The need for cheap and fast payments
Over the past couple of years DeFi protocols have dominated the scene compared to other use cases of public blockchains. In the case of Ethereum, the growth of DeFi has even priced out some other applications. The RSK ecosystem has also experienced its share of growth of DeFi projects. Despite the striking growth, current DeFi protocols — with leveraged trading, automated market maker pools, yield farming and so on — remain quite complex for regular people to use in their daily lives. IOV Labs is targeting the development of simpler financial products including those for cheap and fast P2P payments.
For a long time, payment channels were thought to offer the best approach for P2P payments. However, despite their technical maturity, projects such as the Lightning Network on Bitcoin or the Raiden network on Ethereum have not gained mass adoption. Similarly, on RSK, the Lumino project also struggled with adoption. The challenges are not purely technological. For instance, in a recent blog post, developers of the Breeze wallet suggest that payment failures in the lightning network frequently arise due to liquidity issues.
This is not to say that payment channels are a dead end. On the contrary, there is a lot of renewed interest in using them to support stablecoin payments. One prominent example is a new proposal called Taro from Lightning Labs. There is another approach based on the Omni protocol called Omni-Bolt. The IOV Labs R&I team is closely following these developments and working to build partnerships. However, these solutions are far from being fully developed.
So what can be done now for scalable payments on RSK? At present, the only demonstrably cheap, fast, and secure alternative for EVM blockchains is to use rollup technology. ZK rollups, in particular, have gathered a lot of attention. We reached out to Matter Labs to integrate their L2 rollup service, ZKSync (version 1), with RSK.
However, the timing wasn’t quite right as Matter Labs were preoccupied with the development of version 2 of their system. We look forward to working with Matter Labs and other innovators in this space in the future. Nevertheless, since ZKSync V1 is an open source project with a permissive license — we were able to fork it, understand how it operates, and port the solution as well as some related DApps to work on RSK. To be clear, this was a proof of concept working in local development (regtest) mode. While we did deploy the contracts on RSK testnet, the plans for a full-featured deployment are still work in progress.
The porting exercise
ZKSync uses a rollup contract on Ethereum to bridge ERC20 tokens with their L2 blockchain. Once on L2, the tokens can be transferred much faster and cheaper than on L1. Users can even pay for gas using tokens. Security is provided through Proofs of Computational Integrity, often achieved using zero-knowledge cryptography.
Like most rollup services, ZKSync V1 uses a centralized sequencer to process L2 transactions. This means users have to submit their L2 transactions to a single server through an API. In addition, ZKSync uses on-chain Data Availability — meaning that state changes associated with each L2 transaction are communicated to L1. In case of irrecoverable failure of the rollup system, data availability permits users to reconstruct the L2 state and recover locked assets from the rollup contract on L1.
Zero Knowledge cryptography (based on PLONK proof system) is used for validation proofs. The proofs guarantee the validity of all L2 transactions. Proofs are created off chain (code is open source), but are submitted to the rollup contract on L1. Thus, there is no need to trust any third parties. This is a significant advantage compared to other systems such as opportunistic rollups which rely on economic agents for fraud proofs and game-theoretic incentives. The combination of validation proofs, on-chain data availability, an open source (auditable) codebase and a trustless asset recovery system provides a very high degree of transparency and security.
The RSK blockchain aspires to offer Dapp developers and projects functional compatibility with EVM — so their applications can be deployed in either Network with minimal effort. Therefore, this exercise in porting ZKSync was also a learning experience to discover pain points. Porting a Dapp from one EVM chain project is not simply about swapping out one blockchain client (Geth) for another (RSKJ) — but it should not be too complicated a process.
For example, we discovered an inconsistency in the way the RSKJ client handled two JSON RPCs: web3_sha3 and eth_sendRawTransaction, which lead to mismatches in transaction IDs (hashes). This in turn resulted in integration test failures — as the backend server (in ZKSync) was unable to validate L2 transactions on L1. We worked with RSK core developers to fix the issue, which improved compatibility with Web3. We also modified the corresponding code in the ZKsync server itself to make it more robust on how to compute transaction hashes.
To understand how ZKSync works, we gathered information from multiple sources including blog posts, developer documentation, user documentation, in-source documentation in the code base, and their Discord server. Sometimes there were inconsistencies across these sources. Such inconsistencies are not surprising given the complexity of the system and the fact that the project has evolved over time. Indeed, many open source software projects (including RSK) suffer from some divergence and inconsistency of information across multiple channels. We reported some of the glaring inconsistencies to ZKSync representatives on Discord.
Working with the backend server code was an opportunity for us to learn to work with a codebase in Rust — a language we did not have prior experience with. It turned out to be one of many high points of working on this project. For example, we made some other changes to the server, such as adding the ability to use secure API access using HTTPS. By itself, such a change would not be remarkable. However, it turned out that introducing this simple feature revealed some of the challenges (regarding dependencies) when working with a relatively new, but still hugely popular language like Rust. We initially added HTTPS support using a library called rustls. However, after some weeks, the feature broke when we merged our code with newer commits from Matter Lab’s upstream branch.This happened because, in the meantime, the upstream branch had incorporated a different version of the same library — with a different API, and it broke our feature. To avoid working with different versions of the same library, we thought it safer to switch to a different library, one that used openssl and had a more consistent API across versions.
We ran into some other dependency issues. Sometimes, Matter Labs representatives were able to assist on Discord. In other cases, we found some solutions that we subsequently shared in their community channels. In one case, we reported a breaking change in a library (wasm-pack) to Matter labs and created a tiny PR with a temporary fix, which was subsequently merged. We think these are isolated events, but as always, one needs to be mindful when working with dependencies that are not part of a language’s standard library or those developed by individuals or very small teams.
Back on the RSKJ side, some of the tests required manual control of block timestamps and for the automine feature to be turned on (in regtest mode). In this setting, the RSKJ node mines blocks only when transactions are sent to it manually (e.g. over RPC). We also initially had to increase the number of transaction account slots in the RSKJ node (using node.conf) from the default 5 to something around 20. This was done to avoid “nonce too high” errors as some test scripts sent too many transactions, in a very short timeframe, from a single account. Later, we modified the tests to wait for some transactions to be mined before sending the rest.
One of the most vexing issues happened when we were trying to set up CI using github actions. ZKSync uses a dockerized deployment of various components — the backend server, the RSKJ node, postgres server etc. A simple docker networking issue prevented the RSKJ client from being visible to the backend server. It turned out that the name of the RSKJ docker service (from the main docker-compose file) needs to be included in the list of RPC providers hosts in the RSK node.conf file as well.
While simple, such details of configuration settings or differences with the Geth ecosystem can affect the portability of other projects. For instance, in the course of the project, some team members migrated to M1 Macs (arm64). But there were no RSKJ docker images or ppa for M1 Macs. So we created a dockerfile for our use and to share with the RSK community.
Depending on the application being ported, other changes may also be needed in RSKJ or the RSK ecosystem. For instance, ZK rollups rely on using calldata for on-chain data availability. In Ethereum, the cost of calldata was reduced explicitly with rollups in mind. The RSK community may also need to consider such a change in the future. A final example of ecosystem changes is the availability of multiple price oracles to feed token price data into the service — especially since fees are payable in tokens.
Porting related DApps
Once the porting of the backend rollup service (to RSK) was complete, it was a natural next step to port some existing applications to demonstrate utility and functionality.
Wallet dApp
The wallet dApp can be used to transfer assets on layer 2. In zkSync version 1 this L2 wallet needs to be connected to an Ethereum-compatible wallet such as Metamask. In zkSync v2 users can use an Ethereum-compatible wallet directly.
Checkout dApp
The checkout app allows a user to batch several payment transactions (up to 20, with different recipients) into a single batch transaction. A common scenario is a checkout page, where each transaction represents an item being purchased, potentially from a different vendor.
Explorer App
The explorer app is based on an old L2 explorer created by Matter Labs. The current version of zkSync’s explorer app is not public yet but Matter Labs assured us the source code will be published soon. As a workaround we ported an old, but fully-functional, version of the L2 explorer app.
Hardware Experiments
ZKSync is a very complex project, one that has been under development for a long time. It is not surprising, that in many situations, we found inconsistent and conflicting documentation across various channels: Matter Labs’ discord channels, blog posts, web site, official documentation, github repo. Naturally, there are such inconsistencies in most blockchain projects including RSK! A great thing about Matter Labs is they kept the core code open source — which made this whole porting exercise even possible. We are grateful for that.
However, there was one area in which we struggled to get even some high level input from them. Generating Zero Knowledge proofs requires high-end hardware. Despite numerous requests, the Matter Labs team were not comfortable divulging details of the hardware infrastructure they use to run the prover(s). So we had to experiment and iterate, until we were able to pass all the resource-heavy tests. After all, without that knowledge, it is impossible to estimate potential costs of running a rollup service — which will ultimately be paid by users.
In this regard, we are grateful to our Devops Team for helping us by patiently deploying and upgrading cloud infrastructure resources per our evolving needs. We used increasingly larger AWS EC2 servers to generate proofs to successfully pass the integration tests and ZK circuit tests. We determined that the minimum hardware requirements involve 128GB of RAM and about 32 CPUs (e.g. AWS EC2 m5a.8Xlarge instance or similar). It is possible to run multiple prover threads, to generate proofs in parallel. In a production scenario, depending on the load, multiple servers can be deployed as needed — but each should meet the minimum specification.
Operational Costs
Matter Labs does not pass the cost of ZK proof generation to end users. However, users collectively must pay fees to cover all other operational costs. These costs include transaction fees (gas) for relaying information to L1 about state changes, token withdrawals, block commitments, block validation (SNARK proof verification) etc. We collected transaction data from ZKSync’s explorer to perform simple statistical analysis of the costs of the rollup service. We were then able to correlate these costs with the relevant part of the backend server code which handles fee estimation — this will be useful when deploying the system (e.g. for recalibrating fee components).
Anticipated benefits of adopting this solution
For simplicity, assume 1BTC is about 30K US Dollars. We estimate that deploying ZKSync V1.x on RSK can increase transaction processing capacity by 80X (from 6 to 470 token transfers per second) and lower the cost of token transfers by 25X (from USD 0.085 to USD 0.0033 per transfer). The cost estimate is a combination of on-chain cost of 1500 gas per transfer (USD 0.003 at current gas price) and off-chain infrastructure cost of around USD 0.0003 per transfer. The average infrastructure cost is calculated (conservatively) assuming that the number of rollup transactions is at least as high as the current capacity for regular token transfers on layer 1. Future RSKIPs to lower the cost of calldata or ZK verification can improve upon both capacity and cost estimates.
Wrapup
The rollup-centric future envisioned by Vitalik Buterin for Ethereum may apply to all L1s more broadly. There are a lot of EVM compatible chains in the ecosystem now. Some non-EVM chains such as Near Protocol are integrating L2 systems that are EVM compatible. There are even suggestions for building ZK rollups on top of Bitcoin. RSK is EVM compatible by design, so it comes as no surprise that this porting exercise ultimately did not require any critical code changes. Most of our time was spent developing context and trying to understand how ZKSync works. Apart from a few speed bumps mentioned earlier, the process was largely uncomplicated, even without direct support from Matter Labs. However, more complex tasks lie ahead as we assist other teams to deploy the system into production starting with testnet.
Acknowledgments: Many thanks to Sergio Lerner, Patricio Gallardo, Gaby Genovese and other colleagues for suggestions and assistance in preparing this post.
