RonDB training · module 24 of 25 · 18 min

Global replication

Asynchronous replication between clusters: across regions, through disasters and upgrades, and with both sides writing.

Why · beyond one cluster

Why global replication: a second cluster survives what one cannot

Disasters

A region or a whole cluster lost. From 99.999% for one cluster towards 99.9999%, about 30 s a year, with failover.

Upgrades

Changes that need a copy or an initial restart, done one cluster at a time while the other serves (module 05).

Distance

Data near users in several regions; reads stay local.

Inside a cluster replication is synchronous: consistency first (CP). Between clusters it is asynchronous: availability first (AP), across links of 10–100 ms.

docs: rondb_global_intro.md · rondb_global_extreme.md (availability classes) · rondb_resiliency_model.md (Replication Models and CAP Theorem)

Architecture · from commit to the other cluster

Replication path: an event, a binlog transaction, then a replica transaction

primary cluster data nodes · SUMAevents per epoch binlog servermysqld, ndb-log-bin=1 binary log backup cluster replica applierIO thread → relay log data nodesone transaction per epoch async

Commits fire events in SUMA (module 06). A binlog server, a MySQL server set aside for it, collects them per epoch and writes its binary log.

The backup cluster's applier pulls the binlog and applies each epoch as one transaction. Not every MySQL server logs: only the binlog servers.

docs: rondb_global_internal.md · rondb_global_mysql_internals.md · notes: replication-perf-news.md (the path)

Epochs · the unit of replication

Epochs: each 100 ms epoch becomes one transaction, with its own progress marker

64 bits

A 32-bit GCI and a 32-bit micro-GCI: the epoch of module 14, every 100 ms, 20 per global checkpoint.

Consistent

An epoch holds whole transactions: applied as one, the replica is always at a consistent point.

Self-tracking

Each applied epoch also writes its number to mysql.ndb_apply_status in the same transaction.

So after any crash the replica knows exactly which epoch it applied last, and where to continue.

docs: rondb_global_stream_to_backup.md (epoch format) · rondb_global_internal.md · code: ConfigInfo.cpp (TimeBetweenEpochs 100)

SUMA · events kept until delivered

SUMA: data nodes keep each epoch's events until every subscriber has them

Fault tolerant

  • Events are kept until all subscribers acknowledge the epoch.
  • The replicas of a node group share the sending; if one fails, another resends what was not acknowledged.
  • A binlog server failing loses nothing if another one subscribes.

Bounded

  • MaxBufferedEpochs 100: a bit over 10 s of lag.
  • MaxBufferedEpochBytes 500M.
  • A subscriber further behind is disconnected; its binlog gets a GAP and appliers stop there.

code: ConfigInfo.cpp (MaxBufferedEpochs 100, MaxBufferedEpochBytes 500M) · Suma.cpp (lagging subscribers, out-of-buffer handling) · docs: rondb_global_internal.md

PART 1

Setting it up

Binlog servers, a backup, and a start position.

Binlog servers · two per cluster, for safety

A binlog server is a MySQL server started to log the cluster's changes

mysqld --log-bin=mysql-bin \
       --ndb-log-bin=1 \
       --server-id=100 \
       --log-replica-updates=ON \
       --ndb-log-apply-status=ON
  • ndb-log-bin is off by default: only these servers log.
  • Each server needs its own server id; run two, ids 100 and 101, so one can fail.
  • The last two options let a replica pass changes on, for chains and circles.
  • Start them before any writes or backups you want replicated.
  • About 4 CPUs per replication MySQL server.

docs: rondb_global_mysql_internals.md (server-id, log-bin, ndb-log-bin) · rondb_global_stream_ha.md · rondb_global_internal.md (Bottlenecks) · code: ha_ndbcluster.cc (ndb_log_bin default 0)

Bookkeeping · two system tables

ndb_binlog_index maps epochs to binlog positions; ndb_apply_status records progress

mysql.ndb_binlog_index

  • In each binlog server: one row per epoch written.
  • Binlog file and position, and the next ones.
  • Used to find where a replica must start, and when old binlogs may be purged.

mysql.ndb_apply_status

  • In the replica cluster: the last epoch applied, per source server id.
  • Written in the same transaction as the epoch's rows.
  • A restore with --restore-epoch writes a row with server_id 0.

docs: rondb_global_stream_to_backup.md · rondb_global_mysql_internals.md (ndb_apply_status) · rondb_restore.md

Starting a replica · from a backup of the primary

Starting a replica: restore a backup, find the epoch after it, start there

  1. Back up the primary; restore it on the replica with --no-binlog and --restore-epoch (module 17).
  2. SELECT MAX(epoch) FROM mysql.ndb_apply_status WHERE server_id = 0: the backup's epoch, a pseudo-epoch whose micro-GCI is 0xffffffff.
  3. In ndb_binlog_index on the binlog server: the first epoch greater than it gives file and position.
  4. CHANGE REPLICATION SOURCE TO SOURCE_HOST=…, SOURCE_LOG_FILE=…, SOURCE_LOG_POS=…, IGNORE_SERVER_IDS=(…), then START REPLICA.

The pseudo-epoch sorts after every real epoch of its GCI and before the next GCI, so "greater than" always finds the right place.

docs: rondb_global_stream_to_backup.md (restore epoch, the ndb_binlog_index query, CHANGE REPLICATION SOURCE TO; shortened)

Applier · keeping up with the primary

The applier: batch it, and apply in parallel where transactions allow

Batching

replica-allow-batching: many rows per round trip, about 10× faster. ndb_replica_batch_size 2 MB.

Parallel

Several workers apply independent transactions, using write sets from ndb_log_transaction_dependency on the source.

One channel

One replication channel per applier; no multi-source. More streams need more MySQL server pairs with filters.

Run two appliers per replica cluster but only one active; the second takes over when the first fails.

docs: rondb_global_internal.md (Bottlenecks) · code: ha_ndbcluster.cc (ndb_replica_batch_size 2 MiB) · ndb_replica.cc (only the default channel) · mysql-test/suite/ndb_rpl (parallel applier tests)

Failover · moving the application

Replication failover: when the primary fails, the replica becomes the primary

cluster Aprimary cluster Breplica cluster A · down cluster Bnew primary B applied up to its last complete epoch;applications move to B A returns: restored from B and replicating from it, until roles switch back

Applications write to A; B follows a fraction of a second behind.

A fails. B is consistent at its last applied epoch; the last moments of A may be missing. Applications move to B.

A planned switch first lets B catch up, so nothing is lost. The old primary returns as a replica.

docs: rondb_global_failover.md · rondb_global_stream_ha.md

PART 2

Active-active and conflicts

Both clusters write; conflicts are detected in the data nodes.

Topologies · from one replica to circles

Topologies: one primary with replicas, or every cluster writing

Active-passive

One primary, one or more read-only replicas, possibly in chains.

Active-active

Two clusters replicate to each other; both take writes. Conflicts must be handled.

Circular

Several clusters in a ring, each passing on the others' changes; server ids keep changes from looping.

Any of them can run on Kubernetes with the Helm chart's global replication settings (module 19).

docs: rondb_global_intro.md · rondb_global_dist.md · rondb_helm.md

Conflict detection · per table, in mysql.ndb_replication

Conflict detection: per table, the latest change wins or one cluster wins

FunctionRule
NDB$MAX · NDB$MAX_DELETE_WINHigher value of a timestamp column wins; the variant lets deletes win
NDB$MAX_INS · NDB$MAX_DEL_WIN_INSThe same, also applied to inserts
NDB$OLDApply only if the row still has the before-image's value
NDB$EPOCH · NDB$EPOCH2The primary cluster wins; EPOCH2 also reflects the primary's value back to the secondary
NDB$EPOCH_TRANS · NDB$EPOCH2_TRANSAs above, rolling back whole transactions, not single rows

Rejected changes can be logged in an exceptions table, <table>$EX, for the application to resolve.

code: storage/ndb/plugin/ndb_conflict.h, ndb_conflict.cc (the nine functions) · docs: rondb_global_dist.md

NDB$EPOCH2 · detection inside the data nodes

NDB$EPOCH2: the primary wins; colliding secondary changes are undone

  1. Each row remembers which cluster wrote it last and in which epoch: ROW_AUTHOR, ROW_GCI64.
  2. When the primary applies a secondary's change, an interpreted program checks the row in the data node: changed locally after the last epoch the secondary had seen? Conflict.
  3. The primary keeps its value, and sends a reflected operation back so the secondary returns to it.
  4. Both clusters converge; the conflict is logged and counted in Ndb_conflict_* status variables.

The cluster roles are set with ndb_applier_conflict_role: PRIMARY, SECONDARY, PASS or NONE.

code: ndb_conflict.cc (NDB$EPOCH interpreted program, ROW_GCI64, ROW_AUTHOR) · ha_ndbcluster.cc (ndb_applier_conflict_role, Ndb_conflict_* status) · docs: rondb_global_dist.md, rondb_global_extreme.md

Recap · module 24

What to take away

  • Synchronous inside a cluster, asynchronous between clusters.
  • SUMA → binlog server → applier; each 100 ms epoch is one transaction.
  • ndb_binlog_index and ndb_apply_status tell where to start and where you are.
  • Start a replica from a backup with --restore-epoch; fail over by moving applications.
  • Active-active with conflict functions: latest wins, or the primary wins.

Next · 25 · Performance and what's new

RonDB training · 25 modules · you are at 24

Next · 25 · Performance and what's new