RonDB training · module 18 of 25 · 18 min

Operations

Install, start, stop and change a cluster: configuration, rolling restarts, upgrades and growth, all while it keeps serving.

Install · a binary tarball

RonDB ships as one tarball with every program

TARBALL=rondb-<version>-linux-glibc2.28-x86_64
wget https://repo.hops.works/master/$TARBALL.tar.gz
sudo tar -C /usr/local -xzf $TARBALL.tar.gz
sudo ln -sfn /usr/local/$TARBALL /usr/local/rondb
export PATH=/usr/local/rondb/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/rondb/lib:$LD_LIBRARY_PATH
ProgramIs
ndb_mgmdmanagement server
ndbmtddata node
mysqldMySQL server
rdrs2REST API server
ndb_mgmmanagement client
ndb_restore · ndb_desc · ndb_toptools

Pick the tarball for your OS's C library and CPU. On Ubuntu also install libaio, and on x86 libncurses and libnuma. A symlink makes upgrades a link swap.

docs: rondb_installation.md · module 21 covers the REST server

Starting · always in this order

Start order: management servers, then data nodes, then MySQL and REST servers

1 ndb_mgmdholds the configuration 2 ndbmtddata nodes, in parallel 3 mysqldMySQL servers 4 REST serversoptional every program finds the cluster through the connect string: --ndb-connectstring=mgm1:1186,mgm2:1186 or NDB_CONNECTSTRING

One reachable management server is enough; listing several survives one being down. Files and pid files are named by node id, or by --service-name for systemd.

docs: rondb_starting_cluster.md (Cluster Start Ordering, Concept of NDB Connect String, Naming Files and Directories)

Starting · the commands

Start commands: each program needs the connect string and, best, a fixed node id

# management server: config.ini read on first start, then kept in --configdir
ndb_mgmd --config-file=/etc/rondb/config.ini --configdir=/var/lib/rondb --ndb-nodeid=65
# data nodes: --initial only for initial starts (module 16)
ndbmtd --ndb-connectstring=mgm_host --ndb-nodeid=1
# MySQL server: first time with --initialize-insecure, then normally
mysqld --defaults-file=/etc/rondb/my.cnf --daemonize
# wait until data nodes are started
ndb_waiter --ndb-connectstring=mgm_host

A data node needs only the management server to start; the rest of its configuration comes from there.

docs: rondb_managing_mgmds.md · rondb_startup_phases.md (ndbmtd command) · rondb_managing_mysqlds.md (--initialize-insecure, --daemonize, ndb_waiter) · paths are illustrations

ndb_mgm · the management client

ndb_mgm SHOW: every node slot and who is connected

ndb_mgm real output · RonDB 24.10.0, shortened
show
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=1 @127.0.0.1 (RonDB-24.10.0, Nodegroup: 0, *)
id=2 @127.0.0.1 (RonDB-24.10.0, Nodegroup: 0)
[ndb_mgmd(MGM)] 1 node(s)
id=65 @127.0.0.1 (RonDB-24.10.0)
[mysqld(API)] 14 node(s)
id=67 @127.0.0.1 (RonDB-24.10.0)
id=192 (not connected, accepting connect from localhost)
  • * marks the master data node, the oldest.
  • Each data node shows its node group and version: handy during upgrades.
  • Free API slots show "not connected".
  • ndb_mgm -e "show" runs one command from a script.

docs: rondb_mgm_client.md (SHOW command; lines picked)

ndb_mgm · the commands that matter

ndb_mgm commands: stop, restart, report and log, per node or for ALL

CommandDoes
2 STATUS · ALL STATUSStart phase or state of a data node
2 RESTART [-n] [-i] [-a] [-f]Graceful restart; -n stop and wait for START; -i initial; -a abrupt; -f force
2 STOP · SHUTDOWNStop one data node · stop the whole cluster
2 REPORT MemoryUsage · BackupStatusQuick reports into the client
START BACKUP · ABORT BACKUPModule 17
ENTER · EXIT SINGLE USER MODEModule 05
CLUSTERLOG ON · OFF · categoriesWhat reaches the cluster log

ndb_mgm can stop and restart data nodes, not start processes that are not running: that is the job of systemd, Kubernetes or an agent.

docs: rondb_mgm_client.md · code: mgmclient/CommandInterpreter.cpp (RESTART flags)

Stopping · in reverse order

Stopping: kill -TERM stops any RonDB program gracefully

Applications first

REST servers and MySQL servers, so no new work reaches the data nodes.

Data nodes

kill -TERM: graceful, running transactions finish (module 15). Or ndb_mgm SHUTDOWN for all of them.

Management last

The management servers keep the configuration and the cluster log to the end.

docs: rondb_stop_programs.md (kill -TERM for ndb_mgmd, ndbmtd, mysqld; graceful shutdown of data nodes) · rondb_failure_protocols.md (Graceful Shutdown Protocol)

PART 1

Changing a running cluster

Configuration, machines, software, size.

Configuration · kept by the management server

config.ini is read once; after that the management server's database rules

The configuration database

  • Stored under --configdir, numbered generations.
  • With a database present, config.ini is ignored on restart.
  • --reload reads config.ini as the next generation; --initial throws the database away.

Changes need --reload when they

  • add, remove or change node slots, beyond activating;
  • add node groups;
  • change [ndbd default] or [tcp] values.

After a reload the data nodes pick up the change at their next restart: a rolling restart.

docs: rondb_managing_mgmds.md (configuration database, Configuration Changes)

No restart needed · from ndb_mgm

Runtime changes: some take effect at once, from the management client

CommandDoes
4 HOSTNAME ndbd4_host · 4 ACTIVATEMove an inactive slot to a host and activate it; then start the node there
1 DEACTIVATETake a slot out: its node stops
ALL SET MaxDiskWriteSpeed 100MCheckpoint and backup write speed
ALL SET EnableProactiveDeadlockDetection 1Module 09
ALL SET AdaptiveSendThreshold 16API nodes: favour throughput over latency
ALL SET CompiledInterpreter AUTOJIT-compile pushed programs (x86-64, ARM64)

SET updates the saved configuration too, so restarting nodes keep the new value.

docs: rondb_changing_cluster.md (hostname, activate, deactivate) · code: mgmclient/CommandInterpreter.cpp (helpTextSet: supported parameters and examples)

Machines · scale out, scale up, replace

Inactive slots let you add or move machines without a rolling restart

# scale out: activate prepared slots
ndb_mgm -e "2 hostname ndbd3_hostname"
ndb_mgm -e "2 activate"
# then start ndbmtd on ndbd3_hostname

# move node 1 to a bigger machine
ndb_mgm -e "1 deactivate"
ndb_mgm -e "1 hostname <new_host>"
ndb_mgm -e "1 activate"
# then start node 1 there, --initial
  • Slots with NodeActive=0 are prepared in config.ini (module 13).
  • Deactivating a slot stops its node.
  • A node on a new machine with empty disks starts with --initial and copies its data from the live replica (module 16).
  • New machines must hold the data the old one held.

docs: rondb_changing_cluster.md (Adding & Removing Machines)

Rolling restart · the cluster stays online

Rolling restart: one node per node group at a time, in start order

1 mgmd 65, then 66 2 data nodes NG 0 · node 1 NG 1 · node 3 NG 0 · node 2 NG 1 · node 4 round 1: one node from each groupround 2: the other ones 3 mysqld, rdrs

Every node group must keep a live node at all times, or the cluster fails. The restart barrier makes simultaneous restarts across node groups safe (module 16).

docs: rondb_changing_cluster.md (Rolling Restarts and Software Up-/Downgrades) · release_notes_251019.md (restart barrier)

Upgrades · a rolling restart with new binaries

Upgrade management servers, then data nodes, then API nodes

  1. Take a backup before moving to a new major version.
  2. Install the new tarball beside the old one; switch the symlink: ln -sfn.
  3. Restart management servers one at a time, then data nodes one per node group at a time, then MySQL and REST servers.
  4. Nodes check each other's version before using a new feature, so mixed versions work during the upgrade. Backups are refused until all data nodes match.

Complex upgrades and downgrades may need initial node restarts, or a backup and restore (module 05 showed the two-cluster way).

docs: rondb_upgrade.md (General approach to Upgrading RonDB) · rondb_startup.md (initial node restart for complex upgrades) · code: backup/Backup.cpp (refused in mixed-version clusters)

Growing · more node groups, online

Growing online: add node groups, then spread every table onto them

# 1 config.ini: new [ndbd] slots with NodeGroup=65536, mgmd --reload and a rolling
#   restart; or activate slots prepared earlier with NodeActive=0
# 2 start the new data nodes
ndbmtd --ndb-connectstring=mgm_host --ndb-nodeid=5 --initial
ndbmtd --ndb-connectstring=mgm_host --ndb-nodeid=6 --initial
# 3 make them a node group
ndb_mgm -e "CREATE NODEGROUP 5,6"
# 4 per table, in a MySQL server
ALTER TABLE t1 ALGORITHM=INPLACE, REORGANIZE PARTITION;
OPTIMIZE TABLE t1;   # reclaim space the moved rows left behind

New tables use the new node group at once; REORGANIZE moves buckets of existing tables onto it (module 05).

docs: alter_table.md (Add Node Group, Reorganise existing tables) · rondb_online_config.md (CREATE NODEGROUP 5,6; OPTIMIZE TABLE)

Replicas · from two to three, online

Going to three replicas: activate an inactive slot

  1. Configure NoOfReplicas=3 from the start, with the third node of each node group NodeActive=0 (module 13).
  2. When needed: HOSTNAME and ACTIVATE the slots in ndb_mgm.
  3. Start the new nodes with --initial: they copy their node group's data from the live replicas.
  4. Managed RonDB works this way: always 3 replicas configured, 2 active unless asked for more.

docs: rondb_changing_cluster.md · config_num_replicas.md · table_options.md (Managed RonDB configures 3 replicas, one inactive)

Recap · module 18

What to take away

  • Start mgmd → ndbmtd → mysqld → REST; stop in reverse, with kill -TERM.
  • After the first start, the management server's configuration database rules; --reload changes it.
  • HOSTNAME, ACTIVATE, DEACTIVATE and SET change a cluster without restarts.
  • Rolling restarts: one node per node group at a time; upgrades the same way.
  • Grow with CREATE NODEGROUP and REORGANIZE; add replicas by activating slots.

Next · 19 · Monitoring and deployment