RonDB training · module 02 of 25 · 18 min
Cluster anatomy: nodes, node groups, partitions and replicas
Where every row lives, how many copies there are, and which node answers a query.
Nodes · every node has a node id and speaks the NDB protocol
Three kinds of nodes: data nodes, management servers, API nodes
Data nodes
ndbmtd, multi-threaded. They hold all the data, run the transactions, and replicate and recover the data.
Management servers
ndb_mgmd. Hold the cluster configuration; every node fetches its configuration from them at start. Needed for node starts and the cluster log.
API nodes
MySQL servers (mysqld), the REST API server, and applications on the NDB API or ClusterJ. They send requests to the data nodes.
docs: rondb_arch.md
Management server · important at start, quiet afterwards
Once the nodes are up, the cluster runs without the management server
- Every node fetches its configuration from
ndb_mgmdwhen it starts. - A node can only start while a management server is reachable.
- After the start its main job is the cluster log.
- Data nodes and API nodes keep working if every management server goes offline.
One, on a third machine
One management server, on a machine without a data node. It is also the arbitrator when the network splits (module 15), and needs little CPU or memory.
docs: rondb_arch.md · rondb_spec_mysql.md · intro_relational.md (Replicated Relational Model) · rondb_resiliency_model.md (Cluster Setup Recommendations)
Topology · the recommended minimum
Recommended minimum: two machines run everything, a third the management server
Machines A and B each run a data node, a MySQL server and a REST API server. Machine C runs the one management server, which is also the arbitrator.
docs: rondb_resiliency_model.md (Cluster Setup Recommendations: at least three machines, the arbitrator on the third)
Topology · specialised MySQL servers
Specialised MySQL servers: schema changes and replication get their own
Schema changes
During ALTER TABLE the table is read-only in the MySQL server running it, while all other MySQL servers and API nodes read and write it. Run schema changes from a MySQL server that serves no queries.
Replication
The data nodes ship the change log to the MySQL servers set up for replication, not to the one that ran the update. Two of them, so one can take over (module 24).
Cheap to colocate
Both use very little CPU and memory, so they run next to the query MySQL servers on machines A and B. One MySQL server can do both jobs.
Two specialised MySQL servers on different machines are enough, also for bigger clusters: those add data nodes, MySQL and REST API servers.
docs: rondb_spec_mysql.md
PART 1
Distribution: partitions and node groups
Automatic sharding, and synchronous replicas inside every shard.
Horizontal partitioning · a row lives in exactly one partition
A table is split into partitions; each partition holds whole rows
- Each row maps to one partition, by a hash of its partition key.
- A partition holds a subset of the rows, with all their columns.
- Each partition has one primary replica and NoOfReplicas − 1 backup replicas.
| Word | Means |
|---|---|
| partition | a slice of a table, as the user sees it |
| fragment | the same slice inside the data nodes |
| fragment replica | one copy of a fragment, on one node |
| fully replicated | one partition, many fragments that copy each other |
docs: intro_relational.md (Distributed Relational Model) · rondb_node_groups.md (Partitioning) · code: ConfigInfo.cpp (NoOfReplicas)
Node groups · RonDB's shards
Nodes are grouped by NoOfReplicas; a node group holds its partitions on every node
With NoOfReplicas=2, nodes 1 and 2 form node group 0 and nodes 3 and 4 form node group 1. No row of node group 1 is stored in node group 0.
Inside a node group the nodes hold the same partitions and the same rows: one node failure per node group is survivable.
Unlike plain sharding, queries and foreign keys span node groups, so the cluster stops if all nodes of one node group fail.
docs: rondb_node_groups.md · code: DbdihMain.cpp (execCREATE_FRAGMENTATION_REQ: partitions go round-robin over the node groups)
Where a row lives · decided when the table is created
Where a row lives: from its primary key to a node, a thread and a replica
Every table has a partition key: the primary key, or a part of it.
The key is hashed. The same key always gives the same hash, on every API node.
The hash picks one of 3840 buckets in the table's hashmap.
The hashmap maps the bucket to a partition, and the partition lives in one node group.
The row is written to the primary and the backup replica, in the same ldm thread on every node of the group.
docs: intro_relational.md (Hashmaps) · rondb_route_queries.md (Choosing the ldm thread)
Partition count · set by the configuration, not by the table size
How many partitions a table gets
| Parameter | Default | Range | Effect |
|---|---|---|---|
| PartitionsPerNode | 2 | 1–32 | partitions per node and table |
| NoOfReplicas | 2 | 1–3 | nodes per node group, copies of every row |
| NodeActive=0 slots | — | — | counted in NoOfReplicas: inactive slots keep the partition count stable as nodes are added |
docs: rondb_node_groups.md (Partitioning) · code: storage/ndb/src/common/mgmcommon/ConfigInfo.cpp (PartitionsPerNode, NoOfReplicas)
Example · 2 node groups × 2 replicas × PartitionsPerNode 2
Partition placement: 8 partitions, every node primary for some, backup for others
| Table A | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 |
|---|---|---|---|---|---|---|---|---|
| node 1 · NG 0 | primary | primary | backup | backup | ||||
| node 2 · NG 0 | backup | backup | primary | primary | ||||
| node 3 · NG 1 | primary | primary | backup | backup | ||||
| node 4 · NG 1 | backup | backup | primary | primary |
Partitions per table = 2 × (2 × 2) = 8. They alternate between the node groups; in each group a node is primary for 2 partitions in a row.
Adding node groups adds partitions on the new nodes only; NoOfReplicas changes how many partitions sit on each node.
code: DbdihMain.cpp (execCREATE_FRAGMENTATION_REQ: node groups round-robin; add_nodes_to_fragment: PartitionsPerNode primaries per node) · NdbDictionaryImpl.cpp (default PartitionBalance_ForRPByLDM)
Balance · one node group, NoOfReplicas = 3, PartitionsPerNode = 2
Why the replica count is part of the partition count
2 partitions: node 3 is never primary
| Node 1 | Node 2 | Node 3 | |
|---|---|---|---|
| P1 | primary | backup | backup |
| P2 | backup | primary | backup |
6 partitions: 2 primaries on every node
| Node 1 | Node 2 | Node 3 | |
|---|---|---|---|
| P1, P2 | primary | backup | backup |
| P3, P4 | backup | primary | backup |
| P5, P6 | backup | backup | primary |
Primaries need more CPU. Counting the replicas gives every node the same number of primaries, so heavy use of one table loads all nodes evenly.
docs: rondb_node_groups.md (Partitioning)
Hashmaps · balance that linear hashing cannot give
Hashmaps: rows hash into 3840 buckets, and buckets map to partitions
Linear hashing, 6 partitions
Hashmap of 3840 buckets, 6 partitions
3840 = 2⁸ × 3 × 5: any partition count built from those primes divides evenly. With 14 nodes, the factor 7 leaves a small imbalance.
docs: intro_relational.md (Hashmaps) · 3840 / 6 = 640 buckets per partition
Online scaling · adding a node group
Adding a node group: only the buckets the new node group needs move
A table with 8 partitions in two node groups: each partition holds 480 of the 3840 buckets.
A third node group gives 12 partitions, the 4 new ones in the new node group (black). Each old partition hands 160 buckets to them and keeps 320: a third of the rows move, the fewest possible.
code: NdbDictionary.cpp (initDefaultHashMap: bucket i → partition i % count; prepareHashMap: a bucket moves only when i % new count is a new partition) · DbdihMain.cpp (new partitions go to the new node group) · module 05 shows the REORGANIZE that does it online
Limits · in RonDB 26.10
Limits: how big a cluster can get
144
data nodes, node ids 1–144
72
node groups, with 2 replicas
1–3
replicas per node group
8191
the highest node id, for API nodes
Node ids went from 255 to 2039 in 25.10, and to 8191 in 26.02.11. Ids above 2039 need every data node on 26.02.11 or later. The code also accepts 4 replicas, but 4 is not yet tested and so not supported.
code: storage/ndb/include/kernel/ndb_limits.h (MAX_DATA_NODE_ID 144, MAX_NDB_NODE_GROUPS 72, PREV_MAX_NODES 2039, MAX_NODES_ID 8191) · ConfigInfo.cpp (NoOfReplicas, up to 4) · docs: intro_relational.md · config_num_replicas.md · release_notes_25102.md (RONDB-914) · release_notes_260211.md
PART 2
Transactions across replicas
What ACID means in RonDB, and how a write reaches every replica.
ACID · with two deliberate choices
ACID in RonDB: read committed and network durability
Atomicity
A commit protocol over all replicas of all touched partitions: every node applies the transaction, or none.
Consistency
Constraints and foreign keys are enforced inside the data nodes, for every API.
Isolation
Read committed, with row locks. Read committed takes no lock, so a read's latency does not depend on other transactions.
Durability
Network durability: at commit the change is in memory on at least two computers; the REDO log reaches disk shortly after, in global checkpoints.
docs: intro_transactions.md (RonDB Transaction Model) · rondb_checkpoints_global_redo.md · rondb_concurrency_control.md
Commit protocol · no quorum needed
One two-phase commit gives both atomicity and replication
Most distributed databases
2PC for atomicity, plus a consensus protocol such as Raft or Paxos for each partition's replicas. A quorum needs an odd number of replicas, at least 3.
RonDB
The commit protocol writes every replica of every partition. A fixed node hierarchy, set when nodes join, plus an arbitrator for network splits, replace the quorum: highly available with two replicas.
docs: rondb_nonblocking_2pc.md (Atomicity vs. Replication)
A write · one row, two replicas
A write to two replicas: prepare, commit, complete
The application sends its operations to a transaction coordinator (TC) in one of the data nodes.
Prepare runs linearly within a row: primary first, then the backup, which answers the TC. Rows of one transaction run in parallel.
Commit runs in reverse: backup first, primary last. The primary then knows every replica committed and releases its locks.
Complete clears the rest. With Read Backup, the default, the application hears back after complete, so it can read any replica.
docs: rondb_nonblocking_2pc.md (RonDB's 2PC, Complete Stage, Read Backup Feature) · module 09 goes deeper
Routing · who does the work for one request
Routing a request: most choices are made for you
| Choice | How it is made |
|---|---|
| MySQL server | Any of them: round robin in the client, MySQL Router or a load balancer |
| REST API server | Any of them: round robin in the client or a load balancer |
| TC node | A hint from the first key: a node holding the row, preferring the configured neighbour, then a localhost connection |
| tc thread | Round robin inside the chosen node |
| Replica to read | Read Backup tables: the replica on the TC's node if there is one, else the primary |
| ldm for write | Fixed by the partition: the same ldm on every node of the group |
| ldm for read | A dynamic scheduler selects the thread to use (module 11) |
| Replicas to write | All of them, primary first |
docs: rondb_route_queries.md · code: TransporterCallback.cpp (V_QUERY: committed reads go round robin over the threads of a group)
Fully replicated tables · read scaling for one data set
Fully replicated tables: the same data in every node group
- Every node group holds a copy fragment of every partition.
- A write updates the main fragment; an internal trigger updates the copies in the same transaction.
- A read committed query reads any replica, usually the one on its own machine.
- Grows in steps of the replica count, up to 144 nodes, without changing the partition count.
A natural layout
One MySQL server and one data node per machine: many millions of SQL queries against the same data, with no replication lag.
docs: intro_relational.md (Fully replicated tables) · module 04 shows the table option
Recap · module 02
What to take away
- Three node types: data nodes, management servers, API nodes.
- Node groups are shards; inside a node group every node has the same rows.
- Partitions per table = PartitionsPerNode × nodes; a 3840-bucket hashmap balances them and keeps reorganisation small.
- Read committed, network durability, and one commit protocol for atomicity and replication.
- Read Backup lets any replica serve reads; fully replicated tables put all data in every node group.
RonDB training · 25 modules · you are at 02