RonDB training · module 14 of 25 · 18 min

Durability: REDO log and checkpoints

How a commit in memory becomes data that survives a crash of the whole cluster.

Network durability · durable in memory first

Network durability: a commit lives in several machines' memory, on disk a second later

At commit

  • Every replica holds the change in memory.
  • All later reads see it.
  • Survives node failures: another replica has it.

A second or two later

  • The REDO log reaches disk at a global checkpoint.
  • Survives a crash of the whole cluster.
  • A cluster crash loses at most the last unfinished global checkpoint.

No disk write on the commit path: that is how commits take microseconds, not milliseconds.

docs: rondb_checkpoints_global_redo.md · intro_transactions.md (RonDB transaction model)

Recovery · what is written to disk

Recovery: four mechanisms together make a cluster recoverable

REDO logevery change, logically:only changed columns Global checkpointconsistent cuts throughall REDO logs Local checkpointrows of each fragment,so the REDO log can be cut UNDO logdisk columns: pages backto the checkpoint's start Restart after a crash: load the local checkpoint → apply UNDO → replay REDO up to the last global checkpoint (module 16)

The REDO log records changes; global checkpoints make it consistent; local checkpoints keep it short; UNDO serves disk data.

A fifth, the schema log, keeps schema transactions recoverable (module 03).

docs: rondb_checkpoints_global_redo.md · rondb_checkpoints_local.md · disk_columns.md (UNDO log) · table_options.md (REDO logs only changed columns)

PART 1

Global checkpoints

Consistent points across every node's REDO log.

GCI · a high and a low number

Epochs and global checkpoints: an epoch every 100 ms, a checkpoint every 2 s

1000.11000.2… epochs every 100 ms (TimeBetweenEpochs): consistent units for replication GCI 1001.0 · on disk every 2 s (TimeBetweenGlobalCheckpoints): REDO flushed

A GCI has a high and a low part. Each epoch, a micro GCP, bumps the low part: global replication ships transactions epoch by epoch (module 24).

When the high part moves on, the REDO logs are flushed up to it: a persistent GCP, a point the whole cluster can restart from.

docs: rondb_checkpoints_global_redo.md (micro GCPs and persistent GCPs) · code: ConfigInfo.cpp (TimeBetweenEpochs 100 ms, TimeBetweenGlobalCheckpoints 2000 ms) · GCI values are an illustration

GCP phase 1 · every epoch

GCP phase 1: the master holds back commits, then all nodes move to the new GCI

master DBDIHoldest data node DBDIH · node 2 DBDIH · node 3 DBDIH · node 4 1 GCP_PREPARE: stop handing out the GCI · commits queue 2 GCP_PREPARECONF 3 GCP_COMMIT: GCI + 1, release the queued commits 4 GCP_NODEFINISH: old GCI's transactions done

DBDIH on the oldest data node, the master, runs the protocol for the whole cluster.

Every TC asks DBDIH for the GCI at commit. During PREPARE, new commits wait: the serialisation point that makes each GCI a clean cut.

After COMMIT the queued commits go out together, a batch effect that even helps throughput. NODEFINISH comes once the old GCI's transactions are complete.

docs: rondb_checkpoints_global_redo.md (First Phase of Global Checkpoint Protocol) · code: dbdih/DbdihMain.cpp (GCP_PREPARE, GCP_COMMIT, GCP_NOMORETRANS, GCP_NODEFINISH)

GCP phase 2 · every 2 seconds

GCP phase 2: flush every REDO log up to the GCI, then record it in two places

  1. GCP_SAVEREQ: each node flushes its REDO logs up to the last record of this GCI.
  2. Records of the next GCI may reach disk too: there is no GCI order within a log, and recovery skips any record newer than its GCI.
  3. GCP_SAVECONF: the GCI is now recoverable from the logs.
  4. COPY_GCIREQ: each node writes the GCI to its P0.sysfile, twice, so a restart knows where to stop without scanning the logs.

If a global checkpoint takes longer than TimeBetweenGlobalCheckpointsTimeout (120 s), the stuck node is stopped: durability must not silently fall behind.

docs: rondb_checkpoints_global_redo.md (Second phase of Global Checkpoint Protocol) · code: ConfigInfo.cpp (TimeBetweenGlobalCheckpointsTimeout 120000) · DbdihMain.cpp (GCP stop)

REDO log · per ldm thread, in parts

The REDO log: 64 GB and circular by default, cut at its tail by checkpoints

live REDO: needed for recovery tail head: new records tail moves when a local checkpoint completes a long transaction's records pin the tail

Each ldm thread writes REDO records, prepare and commit, into log parts: 4 parts × 16 files × 1 GB = 64 GB by default.

Space before the start of the last completed local checkpoint can be reused.

A transaction that runs for a long time keeps the tail in place until it ends.

docs: rondb_config.md (REDO log size) · rondb_checkpoints_global_redo.md · code: ConfigInfo.cpp (NoOfFragmentLogParts 4, NoOfFragmentLogFiles 16, FragmentLogFileSize 1G)

REDO overload · three ways it shows

When the REDO log cannot keep up, transactions wait or abort

ProblemSettingWhat happens
REDO buffer fullRedoBuffer, one per log part: 32 MB per ldm thread by defaultOperations queue (DefaultOperationRedoProblemAction=QUEUE) or abort
Disk too slowRedoOverCommitLimit 20 s, Counter 3Sustained IO lag starts aborting transactions; lag is logged every second
Log fulllog size; checkpoint speedCheckpoints are sped up first; when that fails, transactions abort

Give REDO and checkpoints a fast disk.

docs: rondb_advanced_fs_config.md (DefaultOperationRedoProblemAction, RedoOverCommitLimit and RedoOverCommitCounter) · rondb_config.md · code: vm/Configuration.cpp (get_and_set_redo_buffer)

PART 2

Local checkpoints

Partial, row-based, adaptive.

Partial LCP · since NDB 7.6

Partial LCP: full checkpoints do not work for terabytes; partial ones do

Full checkpoints

  • Write all data every time; keep two copies: at least 2× the data on disk.
  • During a big load, checkpoints never finish: the REDO log must hold everything.
  • Loading 5 TB would need almost 10 TB of REDO.

Partial checkpoints

  • Write changed rows plus a slice of all rows.
  • About 1.6× the data on disk, less when compressed.
  • TPC-C with 53 000 warehouses, 5 TB, loaded with the default 64 GB REDO log.

Designed for DataMemory up to 16 TB per node. A fragment's checkpoint is restorable as soon as it completes.

docs: rondb_checkpoints_local.md (Considerations for Partial checkpoints)

The math · 1 TB of 300-byte rows, 1 million updates per second

Checkpointing rows writes 350 MB/s; checkpointing pages would write 25 GB/s

25 GB/s

32 KB pages: 8 million pages touched per 10-second checkpoint, 256 GB each time

6 GB/s

even with 8 KB pages

350 MB/s

rows: 3000 MB of changed rows + 500 MB for 1 part of 2048, per 10 s

Within reach of cloud block storage. Disk columns, which live in pages, still use page-based checkpoints.

docs: rondb_checkpoints_local.md (Row-based vs Page-based checkpoints: the example and its numbers)

Parts · each fragment split 2048 ways

LCP parts: each checkpoint writes some parts in full, changed rows of the rest

fragment rows · 2048 parts LCP n: ALL ROWS LCP n: CHANGE ROWS only · rows with a GCI newer than LCP n−1 changes LCP n+1: ALL LCP n+1: CHANGE ROWS only

A fragment's rows are divided into 2048 parts.

Each checkpoint writes some parts with all rows, and from the others only rows changed since the previous checkpoint: their row GCI tells.

The next checkpoint moves on to other parts. To restore a part: its last ALL-ROWS write, then the changes written after it.

docs: rondb_checkpoints_local.md (Size vs checkpoint speed, Partial checkpoint scans) · code: backup/Backup.cpp (parts, restore order) · BackupFormat.hpp (2048 parts)

LCP scan · find changes fast

The LCP scan: row GCIs find changed rows; page bits skip unchanged pages

Row GCI

Each row's header holds the GCI of its last change. Newer than the last checkpoint: write it.

Change bits

Page headers mark which 4 KB pieces changed. Untouched pages are skipped: scans up to 100× faster.

Exact point in time

A row written before the scan reaches it is checkpointed first, and marked LCP_SKIP.

New pages are written whole; dropped pages become one Delete Page record instead of many row deletes.

docs: rondb_checkpoints_local.md (Partial checkpoint scans) · code: dbtup/tuppage.hpp (change map) · Dbtup.hpp (LCP_SKIP)

Tuning · disk space against write speed

RecoveryWork trades disk space for checkpoint writing

SettingDefaultEffect
RecoveryWork60 (25–100)Aim for checkpoints of 1.6× the data. Lower saves disk but writes more parts in full: cost grows like 1 / (1 − x)
InsertRecoveryWork40 (0–70)The same during heavy inserts, such as a reload: higher eases the disk, using more space for a while
CompressedLCPfalseCompress checkpoints: less disk, more CPU in io threads

Leave these alone unless you know the disks can take more writes, or must save space.

docs: rondb_checkpoints_local.md (Size vs checkpoint speed of partial checkpoints) · code: ConfigInfo.cpp (RecoveryWork 60, 25–100; InsertRecoveryWork 40, 0–70; CompressedLCP false)

Write speed · adapts to CPU and disk

Checkpoints slow down when the ldm thread is busy or REDO falls behind

Slower

  • ldm CPU at 95%: small step down; bigger at 97% and 99%.
  • Any REDO part with 2 s of IO lag: down quickly.
  • Never below MinDiskWriteSpeed.

Faster

  • CPU below 90%: step up again.
  • Higher caps during restarts: 50 MB/s when another node restarts, 200 MB/s for our own.
  • MaxDiskWriteSpeed otherwise: 4 MB/s.

Checkpoints may slow, never stop: stopping would fill the REDO log. Backups share this write budget (module 17).

docs: rondb_advanced_fs_config.md (MinDiskWriteSpeed) · code: backup/Backup.cpp (CPU 95/97/99%, REDO IO lag) · ConfigInfo.cpp (MinDiskWriteSpeed 2M, MaxDiskWriteSpeed 4M, …OtherNodeRestart 50M, …OwnRestart 200M)

On disk · the LCP directory

LCP files: control files say what each data file holds

# under FileSystemPath, per data node
LCP/0/T13F3.ctl   # control file, table 13 fragment 3
LCP/1/T13F3.ctl   # and its alternate
LCP/2/T13F3.Data  # data files of checkpoints
LCP/3/T13F3.Data
…
LCP/2063/
  • Control files live in directories 0 and 1; data files use directories 0 to 2063 in turn.
  • Writes go in large blocks, normally 256 KB, the largest IO many clouds allow.
  • Disk columns: PGMAN flushes a page only after its UNDO record is on disk.

docs: rondb_checkpoints_local.md (Partial checkpoint disk files) · code: ndbfs/Filename.cpp (LCP/<n>/T<t>F<f>) · backup/BackupFormat.hpp (NDB_MAX_LCP_FILES 2064) · storage/ndb/docs/undo-log-hole-handling.html · the listing is an illustration

Recap · module 14

What to take away

  • Network durability: committed in memory on all replicas, on disk within seconds.
  • Epochs every 100 ms; persistent global checkpoints every 2 s.
  • The 64 GB REDO log is cut at its tail as checkpoints complete.
  • Partial, row-based checkpoints: 2048 parts, about 1.6× the data on disk.
  • Checkpoint speed adapts to CPU and REDO lag, but never stops.

Next · 15 · High availability

RonDB training · 25 modules · you are at 14

Next · 15 · High availability