RonDB training · module 16 of 25 · 18 min

RonDB recovery

What a data node does between "start" and "started": from its own files, from the live nodes, and while the cluster keeps serving.

Start types · four kinds

Start types: with or without its files, alone or with the whole cluster

Start type	Cluster	Files	When
Initial cluster start	starting	none	A brand new cluster
Cluster restart	starting	own	After a crash or shutdown of the whole cluster
Node restart	running	own	Rolling restarts, config changes, OS and simple software upgrades
Initial node restart · `--initial`	running	none	Corrupt files, a new machine with empty NVMe, complex upgrades or downgrades

docs: rondb_startup.md (Startup Types)

Recovery · two sources

Node recovery: rebuild from its own files, then catch up from live nodes

First the node rebuilds the state it last saved: checkpoint, UNDO for disk data, REDO up to the last complete global checkpoint, then the ordered indexes.

In a running cluster the data moved on meanwhile: the node copies the difference from a live replica. An initial node restart copies everything.

docs: rondb_startup_phases.md · rondb_checkpoints_global_redo.md (P0.sysfile)

Node start · the steps the node log reports

Node start: sixteen steps from process start to a fully started node

Yellow: local recovery from the node's own files. Black: catching up from live nodes. The node log reports each step as [NODE-START], every 15 s while one runs.

code: src/kernel/vm/NodeStartLog.hpp (enum Step NSL_INIT … NSL_BARRIER, step names) · ConfigInfo.cpp (NodeStartLogReportFrequency 15 s)

Steps 1–7 · getting into the cluster

Steps 1–7, joining the cluster: configuration, memory, heartbeats, schema

Step	What takes time
Configuration and threads	Node id, thread and memory sizes come from the management server
Memory	All memory is allocated and touched: roughly 1–3 GB per second, depending on the OS
Join	1–3 seconds; nodes join one at a time
System files	Initial starts write REDO, UNDO and tablespace files in full: usually under a minute
Metadata	The schema log is copied if another node has a newer one; then the node joins global and local checkpoints

docs: rondb_startup_phases.md (Receive Configuration, Start Virtual Machine, Node Registration Protocol, Write System Files, Copy MySQL Metadata, Join GCP & LCP Protocol)

Step 8 · restore from the local checkpoint

Step 8, restore: every thread loads checkpoint rows; 6 threads load 200 GB in about 3 minutes

1 M

rows per second per thread, for rows of a few hundred bytes

all

ldm threads and query workers take part: one per CPU

≈3 min

for a 200 GB database, with about 6 threads

Each part is read from its last ALL-ROWS checkpoint and the CHANGE-ROWS checkpoints after it (module 14). Every row inserted also goes into its hash index. 200 GB of 200-byte rows is 1 billion rows: 3 minutes takes about 6 threads at 1 M rows per second each, and disks that read over 1 GB/s. More CPUs make it faster.

docs: rondb_startup_phases.md (Restore Local Checkpoint: 1M records per second per thread, 200 GiB in 3 minutes) · code: backup/Backup.cpp (restore order) · 6 threads and 1 GB/s worked out from them

Steps 9–10 · logs

Steps 9–10, logs: UNDO runs back to the checkpoint, REDO forward to the GCI

UNDO · disk data

Read by the rep thread, routed by PGMAN to the owning ldm thread.
Applied serially, backwards, to the state at the checkpoint's start.
Proportional to disk changes since the checkpoint: usually well under a minute.

REDO · all data

Replayed up to the GCI in P0.sysfile; newer records are skipped.
The normal write path, without distributed commit.
Inserts as fast as checkpoint loading; updates slower, one read each.

The less REDO since the last checkpoint, the faster this goes: checkpoint speed is also restart speed.

docs: rondb_startup_phases.md (Apply UNDO Log, Execute REDO Log)

Step 11 · ordered indexes

Step 11: ordered indexes are rebuilt from the rows, pure CPU work

Why

No index is checkpointed (module 08). The T-trees are built once all rows are back.

How

One table at a time, its partitions in parallel. BuildIndexThreads defaults to 128.

Not hash indexes

Filled in step 8, as each checkpoint row is inserted. Unique indexes are tables, so theirs too.

docs: rondb_startup_phases.md (Rebuild Ordered Indexes) · code: ConfigInfo.cpp (BuildIndexThreads 128)

PART 1

Catching up with live nodes

Copying fragments while the cluster keeps writing.

Step 12 · copy fragment, row id by row id

Step 12, copy fragment: the live node sends every row the starting node lacks

Row ids are the same on all replicas (module 07), so the nodes compare row id by row id. Equal GCI: already there, skipped.

Newer rows are sent. An empty slot with a newer GCI is a delete; a slot reused by another key sends a delete and an insert.

The starting node joins live transactions, but never as their coordinator: rows behind the scan stay in step.

docs: rondb_startup_phases.md (Synchronise with live nodes) · rows and GCIs are an illustration

Initial node restart · TwoPassInitialNodeRestartCopy

Initial node restart: an empty node copies twice, fast without locks, then exact

Part I · no locks

Copy all rows without locking the live node, without joining transactions.
A fuzzy copy, not transaction consistent.
Then build ordered indexes in parallel: fast.

Part II · with locks

Copy again what changed: shared lock on the live row, exclusive on the starting one.
The starting node joins live transactions; for rows not yet copied it accepts and discards the change.
REDO logging starts once all fragments are in step.

On by default. Off: one copy with indexes built row by row, but the live node scans only once.

docs: rondb_startup_phases.md (Synchronise Part I and Part II) · code: ConfigInfo.cpp (TwoPassInitialNodeRestartCopy true) · NdbcntrMain.cpp

Steps 13–15 · from in-step to durable

Steps 13–15: a synchronised node must join a checkpoint to be recoverable

Wait for LCP: the data is current, but a crash now would leave nothing to recover from. A distributed local checkpoint including the node fixes that; earlier local-only checkpoints make it shorter.
Activate: join global checkpoints fully, activate indexes and foreign keys.
Hand over event subscriptions from the live replica: waits up to RestartSubscriberConnectTimeout (120 s) for subscribers such as binlog servers.

docs: rondb_startup_phases.md (Wait for Local Checkpoint) · code: NodeStartLog.hpp (wait-lcp, activate, handover) · ConfigInfo.cpp (RestartSubscriberConnectTimeout 120000)

Step 16 · new in 25.10.19

Step 16, restart barrier: wait until every restarting node has recovered

Two nodes of different node groups restart at once, as an orchestrator such as Kubernetes may do.

The first to recover parks in start phase 110. It already serves traffic, but nothing may stop another node meanwhile: a second failure could hit a node group with only one live replica.

When all restarting nodes have recovered, all are released. RestartBarrierTimeout bounds the wait.

docs: release_notes_251019.md (RONDB-1096: Restart barrier for node restarts) · code: NdbcntrMain.cpp (start phase 110) · ConfigInfo.cpp (RestartBarrierTimeout, GracefulShutdownTimeout)

Cluster restart · all nodes at once

In a cluster restart the nodes wait for each other, then recover in parallel

Sync barrier

Starting nodes wait for each other: the next steps run one node at a time.

Partial start

After StartPartialTimeout (30 s) the cluster may start without missing nodes, if it can still recover all data.

Same GCI

All nodes restore to the same global checkpoint. Nodes stopped at different times sync like a node restart.

docs: rondb_startup_phases.md (Synchronisation Barrier) · code: ConfigInfo.cpp (StartPartialTimeout 30000, StartPartitionedTimeout 0)

Monitoring · ndbinfo.restart_info

restart_info shows how long each phase of the last restart took

mysql real output · rondb_monitoring_restarts

SELECT * FROM ndbinfo.restart_info \G

node_id: 2

node_restart_status: Wait LCP to ensure durability

secs_to_allocate_node_id: 57

secs_to_include_in_heartbeat_protocol: 2

secs_to_include_node: 3

secs_restore_fragments: 0

secs_exec_redo_log: 0

secs_index_rebuild: 0

… one column per phase …

One row per restarted node, a column per phase.
The node log adds [NODE-START] progress every 15 s.
Long phases point to what to tune: disks, CPUs, checkpoint speed.

docs: rondb_monitoring_restarts.md (lines picked, alignment simplified) · code: NodeStartLog.hpp

Tuning · where restart time goes

Fast restarts need CPUs, fast disks and short REDO logs

Phase	Helped by
Restore checkpoint	More CPUs, NVMe or high-bandwidth block storage
Execute REDO	Frequent checkpoints, so less REDO to replay
Rebuild indexes	More CPUs; BuildIndexThreads
Synchronise	More ldm threads, fixed-size in-memory columns, less load on the cluster
Wait for LCP	Higher write speeds during restarts: MaxDiskWriteSpeedOwnRestart 200M, …OtherNodeRestart 50M

Local NVMe makes node restarts fast, but a new machine means an initial node restart.

docs: rondb_startup_phases.md (Optimisations, Notes on Performance) · code: ConfigInfo.cpp

Recap · module 16

What to take away

Four start types: with or without files, alone or with the cluster.
Local recovery: checkpoint, UNDO backwards, REDO forwards, indexes.
Then copy fragment, row id by row id, while transactions run.
A node is durable only after a checkpoint that includes it.
The restart barrier keeps simultaneous restarts safe; restart_info shows the timings.

Next · 17 · Backup and restore

RonDB training · 25 modules · you are at 16