RonDB training · module 15 of 25 · 15 min

High availability

How a cluster notices a failure within seconds, agrees on it in a millisecond, and keeps one consistent cluster through network splits.

Always on · more than replication

Always on: under 30 seconds of downtime a year takes seven ingredients

  1. Replication: synchronous, inside the cluster.
  2. Instant failover: no log replay before taking over.
  3. Global replication: survive a cluster failure (module 24).
  4. Fail-fast software: crash on doubt, never limp on.
  5. Modular software: blocks and signals (module 06).
  6. Crash analysis: every crash leaves traces to fix it.
  7. Managed software: operations done right every time.

docs: how_to_always_on.md (NDB has delivered systems with less than 30 seconds of downtime per year; the seven parts)

Failover · nothing to replay

Failover changes only who serves the data: under a millisecond once detected

Shared disk, log shipping

The standby replays logs to catch up before it can take over.

RonDB

Every transaction already wrote all replicas. Failover updates the distribution: one internal transaction, under 1 ms.

So detection is the cost

A crashed process drops its sockets: noticed in microseconds. A silent failure, such as power loss, waits for heartbeats: seconds.

docs: how_to_always_on.md (Instant Failover)

Resiliency · node groups

Resiliency: a cluster survives while every node group has one live node

node group 0 node 1 node 2 node group 1 node 3 node 4 node group 2 node 5 node 6 node group 3 node 7 node 8 node 2 ✕node 3 ✕node 6 ✕node 7 ✕ 4 nodes down, one per node group: the cluster runs node 1 ✕ both nodes of node group 0 down: the cluster stops

Each node group holds its own partitions, replicated on its nodes (module 02).

Up to (NoOfReplicas − 1) × node groups failures are survivable, if they hit different node groups: 4 here.

Losing every replica of one node group stops the cluster: NoOfReplicas failures in the wrong place. Spread-out replicas would fail after any two.

docs: rondb_resiliency_model.md (Rationale for Node Groups, Conclusion)

Setup · machines and zones

HA setup: at least three machines, two for data and one for the arbitrator

Three machines

Two data nodes and an arbitrator elsewhere. On two machines, half of all machine failures would stop the cluster.

Replicas apart

Nodes of one node group on different machines, VMs on different hosts, ideally different availability zones.

Two replicas suffice

The arbitrator replaces a quorum. Three replicas lose less throughput when one fails; four are still beta.

docs: rondb_resiliency_model.md (Cluster Setup Recommendations, Synchronous Replication inside a Cluster)

PART 1

Detecting and handling failures

Heartbeats, a master that is always known, and failures handled as transactions.

Node registration · one node at a time

Node registration: nodes join in an agreed order; the oldest is the master

Join

Only one node joins at a time; the members agree which. Each gets the next place in a dynamic order.

Master

The oldest member coordinates global checkpoints, local checkpoints, restarts and failures.

Takeover

If the master dies, the next oldest takes over: every node computes the same answer, no election needed.

docs: rondb_failure_protocols.md (Node Registration Protocol) · code: src/kernel/blocks/qmgr/QmgrMain.cpp

Heartbeats · a ring in node order

Heartbeats: each node watches the one before it; four misses mean dead

node 1 · oldest node 2 node 3 node 4 · newest sent every 2.5 s, checked every 5 s · 4 silent intervals ≈ 20 s → failed

Heartbeats follow the join order in a ring: node 4, the newest, sends to node 1, the oldest.

Any signal from the predecessor counts, so a busy send buffer never fakes a death. With HeartbeatIntervalDbDb = 5000 ms, a silent node is declared dead after about 20 s; lower it for faster detection.

docs: rondb_failure_protocols.md (Heartbeat Protocol) · code: ConfigInfo.cpp (HeartbeatIntervalDbDb 5000, "considered dead after 4 missed HBs"; HeartbeatIntervalDbApi 1500) · QmgrMain.cpp

Detection · most failures are faster than heartbeats

Failure detection: closed sockets, signals and the watchdog beat heartbeats

Signal of troubleDetected
The process crashed: its sockets closewithin microseconds
A node behaves oddly: another node sends it a shutdown signalat once
A thread makes no progress: the watchdog noticesprints after 100 ms; stops the node after 3 × TimeBetweenWatchDogCheck, about 18 s
Silent: power loss, frozen VMheartbeats, seconds

ConnectCheckIntervalDelay (0, off) adds a connectivity check among all nodes before a failure is declared.

docs: rondb_failure_protocols.md (Watchdog Handling, Node Failure Protocol) · code: ConfigInfo.cpp (TimeBetweenWatchDogCheck 6000, ConnectCheckIntervalDelay 0) · WatchDog.cpp

Node failure protocol · a failure is a transaction

Node failure protocol: the survivors first agree on exactly which nodes failed

  1. The master sends a prepare with the set of failed nodes to all survivors.
  2. Each may add nodes it also lost, never remove any. If the set grew, another round.
  3. When the set is stable, commit: every survivor now has the same view of who is alive.
  4. Then each protocol cleans up: TC takeover (module 09), checkpoint and restart masters, subscriptions.

The failure is serialised with all transactions: no transaction sees half a failure.

docs: rondb_failure_protocols.md (Node Failure Protocol)

PART 2

Network partitions

One cluster must survive, never two.

Arbitration · when neither side can be sure

Arbitration: when the network splits, the side the arbitrator answers first survives

zone A node 1 · NG 0 node 3 · NG 1 zone B node 2 · NG 0 node 4 · NG 1 zone C · mgmdthe arbitrator link A–B breaks asks first: yes asks second: no → shuts down

Each zone holds one node of every node group; a third zone holds the arbitrator, agreed on in advance.

The link between A and B fails. Each side has a full set of node groups but no majority: either could go on alone, so both must not.

Both ask the arbitrator; it says yes to the first and no to the rest. The losing side shuts down; a new arbitrator is chosen at once.

docs: rondb_resiliency_model.md (Arbitrator, Cluster Setup Recommendations) · rondb_nonblocking_2pc.md (arbitrator instead of quorum)

The rules · after a split

Arbitration is needed only when both sides could run alone

A side of the split…Outcome
misses every node of some node groupShuts down: its data is incomplete
has a majority of the data nodes and all node groupsContinues
has all node groups but no majorityAsks the arbitrator; continues only on a yes
cannot reach the arbitrator within ArbitrationTimeout (7.5 s)Shuts down

Inside a cluster RonDB chooses consistency over availability (CP); between clusters, availability (AP, module 24).

docs: rondb_resiliency_model.md (Arbitrator, Replication Models and CAP Theorem) · code: ConfigInfo.cpp (ArbitrationTimeout 7500)

Configuration · who may arbitrate

ArbitrationRank picks the arbitrator among management and API nodes

ArbitrationRank

  • Set on [ndb_mgmd] and [api] slots.
  • 0 never; 1 first choice; 2 only when no rank 1 node is up.
  • Management servers are light: ideal arbitrators in a third zone.

External arbitration

  • Arbitration=WaitExternal: data nodes log "Continuing after wait for external arbitration, node: 1,2".
  • Your own program reads the cluster log and kills one side.
  • Set ArbitrationTimeout to at least twice its polling interval.

docs: rondb_resiliency_model.md (Technical: External Arbitrators) · code: ConfigInfo.cpp (ArbitrationRank: "selects arbitrators in order 1, 2"; ArbitrationRankWait 60000) · QmgrMain.cpp

PART 3

Planned stops and restarts

Without aborting transactions.

Graceful shutdown · ndb_mgm STOP or kill -TERM

Graceful shutdown: a planned stop lets running transactions finish first

Stop new work

The node takes no new transactions; its share moves to the other replicas.

Finish old work

Transactions already running complete. Stragglers are aborted after a few seconds.

Time budget

GracefulShutdownTimeout 27 s, then an immediate stop. It may wait for another node's restart (module 16).

kill -TERM on the data node process is graceful: Kubernetes and managed RonDB stop nodes this way.

docs: rondb_failure_protocols.md (Graceful Shutdown Protocol) · code: ConfigInfo.cpp (GracefulShutdownTimeout 27000)

StopOnError · restart or stay down

StopOnError: by default a crashed data node stays down; an angel can restart it

StopOnError = true · default

  • The node stops after a crash and leaves its trace files.
  • An operator, or Kubernetes, decides when to restart.

StopOnError = false

  • The angel process restarts the node automatically.
  • Up to MaxStartFailRetries (3) failed starts in a row.

code: ConfigInfo.cpp (StopOnError default true: "If set to N, ndbd automatically restarts/recovers"; MaxStartFailRetries 3, "requires StopOnError=0") · angel.cpp

Recap · module 15

What to take away

  • Failover is instant; detection is the cost: microseconds for crashes, seconds for silence.
  • Survive up to (replicas − 1) × node groups failures; three machines minimum.
  • Heartbeats ring in join order; 4 missed intervals declare a node dead.
  • Survivors agree on the failed set, then every protocol cleans up.
  • In a split with no majority, the arbitrator's first yes decides.

Next · 16 · RonDB recovery

RonDB training · 25 modules · you are at 15

Next · 16 · Recovery