Ledger Sharding

Ledger sharding is the practice of splitting a financial ledger's accounts and entries across several separate database partitions, called shards, so that a payment platform can record more transactions per second than a single database could handle. It is used by digital wallets, neobanks, card issuers, payment processors and exchanges whose transaction volume outgrows one machine. The term is engineering jargon rather than a formal standard: it describes a design choice, not a rule set.

The idea is the same one behind sharding in blockchains and large databases, applied to the books of a financial business. The difficulty is specific to money: a ledger must keep debits equal to credits, and a transfer between two customers whose accounts live on different shards still has to post both sides or neither. Sharded operational ledgers usually sit beneath a company's general ledger, acting as high-volume subledgers that are summarised into it.

Picture a busy bank branch that keeps every customer's account card in one drawer, with one clerk allowed to touch it. As the queue grows, the clerk becomes the bottleneck. The branch splits the cards into four drawers by surname, A to F, G to M and so on, with a clerk at each. Four times as many customers can now be served at once.

The catch is a payment from Mr Ahmed (drawer one) to Ms Smith (drawer four). Two clerks have to update two cards, and if one clerk writes the change and the other is interrupted, the books no longer add up. Ledger sharding is mostly about how those clerks agree, so that money never appears or vanishes between drawers.

A single relational database can only commit so many writes per second, and every money movement is a write. Two pressures push platforms to split the ledger:

• Overall volume. A wallet or card programme with millions of users can exceed what one primary database can commit, especially at peaks such as payday or holiday sales.
• Hot accounts. Some accounts receive a huge share of the traffic: a merchant's settlement account, an omnibus account, a for-benefit-of (FBO) account holding customer funds, or a fee account. Every entry must lock and update the same balance, so that one account limits the whole system.
• Isolation. Splitting by region or legal entity can keep one country's data in that country and stop an outage in one shard from taking down all customers.

Platforms usually combine several approaches. Partitioning by account or customer ID (often a hash of the ID) spreads load evenly and keeps all of one customer's entries together. Partitioning by tenant, product or legal entity matches how the business reports. Time-based partitioning moves old, closed periods to cheaper storage. For hot accounts, a common technique is to split one logical account into several sub-accounts, each on its own shard or row, and add them up when the balance is needed; this pairs naturally with balance caching.

Within one shard, a transfer is a single database transaction. Across shards, platforms use one of three broad methods: a two-phase commit, in which both shards prepare and then commit together; a saga, in which each step is committed separately and a compensating entry undoes earlier steps if a later one fails; or an internal clearing account on each shard, so that every shard balances on its own. The third method is the easiest to audit. A $100 transfer from customer A on shard 1 to customer B on shard 2 posts as:

Each shard balances by itself, and the clearing accounts must net to zero across all shards once every transfer has completed:

Here $n$ is the number of shards and $\text{Clearing}_s$ is the signed balance of the clearing account on shard $s$. In the example, shard 1's clearing account holds a $100 credit and shard 2's a $100 debit, so the sum is zero. A non-zero total points to a transfer stuck halfway, which transaction reconciliation or balance reconciliation process must find and repair. Every step should also be protected by idempotency so that a retried leg does not post twice.

Sharding moves complexity rather than removing it. Reports that once ran as one query now have to gather data from every shard; a trial balance becomes a sum of per-shard trial balances. Moving accounts between shards (resharding) as the platform grows is delicate, and reads may briefly see one leg of a cross-shard transfer before the other. These are operational risks that auditors and banking partners ask about, because a regulated firm must be able to show accurate customer balances at any moment, for example when safeguarding funds or reconciling a sponsor bank account. Sharding an internal database should also not be confused with distributed ledger technology (DLT): a sharded ledger is still controlled by one operator, while a distributed ledger is shared among several parties who reach consensus.

• Sharding pattern (Microsoft Azure Architecture Center)
• Compensating Transaction pattern (Microsoft Azure Architecture Center)