RonDB training · module 18 of 25 · 18 min
Operations
Install, start, stop and change a cluster: configuration, rolling restarts, upgrades and growth, all while it keeps serving.
Install · a binary tarball
RonDB ships as one tarball with every program
TARBALL=rondb-<version>-linux-glibc2.28-x86_64 wget https://repo.hops.works/master/$TARBALL.tar.gz sudo tar -C /usr/local -xzf $TARBALL.tar.gz sudo ln -sfn /usr/local/$TARBALL /usr/local/rondb export PATH=/usr/local/rondb/bin:$PATH export LD_LIBRARY_PATH=/usr/local/rondb/lib:$LD_LIBRARY_PATH
| Program | Is |
|---|---|
| ndb_mgmd | management server |
| ndbmtd | data node |
| mysqld | MySQL server |
| rdrs2 | REST API server |
| ndb_mgm | management client |
| ndb_restore · ndb_desc · ndb_top | tools |
Pick the tarball for your OS's C library and CPU. On Ubuntu also install libaio, and on x86 libncurses and libnuma. A symlink makes upgrades a link swap.
docs: rondb_installation.md · module 21 covers the REST server
Starting · always in this order
Start order: management servers, then data nodes, then MySQL and REST servers
One reachable management server is enough; listing several survives one being down. Files and pid files are named by node id, or by --service-name for systemd.
docs: rondb_starting_cluster.md (Cluster Start Ordering, Concept of NDB Connect String, Naming Files and Directories)
Starting · the commands
Start commands: each program needs the connect string and, best, a fixed node id
# management server: config.ini read on first start, then kept in --configdir ndb_mgmd --config-file=/etc/rondb/config.ini --configdir=/var/lib/rondb --ndb-nodeid=65 # data nodes: --initial only for initial starts (module 16) ndbmtd --ndb-connectstring=mgm_host --ndb-nodeid=1 # MySQL server: first time with --initialize-insecure, then normally mysqld --defaults-file=/etc/rondb/my.cnf --daemonize # wait until data nodes are started ndb_waiter --ndb-connectstring=mgm_host
A data node needs only the management server to start; the rest of its configuration comes from there.
docs: rondb_managing_mgmds.md · rondb_startup_phases.md (ndbmtd command) · rondb_managing_mysqlds.md (--initialize-insecure, --daemonize, ndb_waiter) · paths are illustrations
ndb_mgm · the management client
ndb_mgm SHOW: every node slot and who is connected
- * marks the master data node, the oldest.
- Each data node shows its node group and version: handy during upgrades.
- Free API slots show "not connected".
ndb_mgm -e "show"runs one command from a script.
docs: rondb_mgm_client.md (SHOW command; lines picked)
ndb_mgm · the commands that matter
ndb_mgm commands: stop, restart, report and log, per node or for ALL
| Command | Does |
|---|---|
| 2 STATUS · ALL STATUS | Start phase or state of a data node |
| 2 RESTART [-n] [-i] [-a] [-f] | Graceful restart; -n stop and wait for START; -i initial; -a abrupt; -f force |
| 2 STOP · SHUTDOWN | Stop one data node · stop the whole cluster |
| 2 REPORT MemoryUsage · BackupStatus | Quick reports into the client |
| START BACKUP · ABORT BACKUP | Module 17 |
| ENTER · EXIT SINGLE USER MODE | Module 05 |
| CLUSTERLOG ON · OFF · categories | What reaches the cluster log |
ndb_mgm can stop and restart data nodes, not start processes that are not running: that is the job of systemd, Kubernetes or an agent.
docs: rondb_mgm_client.md · code: mgmclient/CommandInterpreter.cpp (RESTART flags)
Stopping · in reverse order
Stopping: kill -TERM stops any RonDB program gracefully
Applications first
REST servers and MySQL servers, so no new work reaches the data nodes.
Data nodes
kill -TERM: graceful, running transactions finish (module 15). Or ndb_mgm SHUTDOWN for all of them.
Management last
The management servers keep the configuration and the cluster log to the end.
docs: rondb_stop_programs.md (kill -TERM for ndb_mgmd, ndbmtd, mysqld; graceful shutdown of data nodes) · rondb_failure_protocols.md (Graceful Shutdown Protocol)
PART 1
Changing a running cluster
Configuration, machines, software, size.
Configuration · kept by the management server
config.ini is read once; after that the management server's database rules
The configuration database
- Stored under
--configdir, numbered generations. - With a database present, config.ini is ignored on restart.
--reloadreads config.ini as the next generation;--initialthrows the database away.
Changes need --reload when they
- add, remove or change node slots, beyond activating;
- add node groups;
- change
[ndbd default]or[tcp]values.
After a reload the data nodes pick up the change at their next restart: a rolling restart.
docs: rondb_managing_mgmds.md (configuration database, Configuration Changes)
No restart needed · from ndb_mgm
Runtime changes: some take effect at once, from the management client
| Command | Does |
|---|---|
| 4 HOSTNAME ndbd4_host · 4 ACTIVATE | Move an inactive slot to a host and activate it; then start the node there |
| 1 DEACTIVATE | Take a slot out: its node stops |
| ALL SET MaxDiskWriteSpeed 100M | Checkpoint and backup write speed |
| ALL SET EnableProactiveDeadlockDetection 1 | Module 09 |
| ALL SET AdaptiveSendThreshold 16 | API nodes: favour throughput over latency |
| ALL SET CompiledInterpreter AUTO | JIT-compile pushed programs (x86-64, ARM64) |
SET updates the saved configuration too, so restarting nodes keep the new value.
docs: rondb_changing_cluster.md (hostname, activate, deactivate) · code: mgmclient/CommandInterpreter.cpp (helpTextSet: supported parameters and examples)
Machines · scale out, scale up, replace
Inactive slots let you add or move machines without a rolling restart
# scale out: activate prepared slots ndb_mgm -e "2 hostname ndbd3_hostname" ndb_mgm -e "2 activate" # then start ndbmtd on ndbd3_hostname # move node 1 to a bigger machine ndb_mgm -e "1 deactivate" ndb_mgm -e "1 hostname <new_host>" ndb_mgm -e "1 activate" # then start node 1 there, --initial
- Slots with
NodeActive=0are prepared in config.ini (module 13). - Deactivating a slot stops its node.
- A node on a new machine with empty disks starts with
--initialand copies its data from the live replica (module 16). - New machines must hold the data the old one held.
docs: rondb_changing_cluster.md (Adding & Removing Machines)
Rolling restart · the cluster stays online
Rolling restart: one node per node group at a time, in start order
Every node group must keep a live node at all times, or the cluster fails. The restart barrier makes simultaneous restarts across node groups safe (module 16).
docs: rondb_changing_cluster.md (Rolling Restarts and Software Up-/Downgrades) · release_notes_251019.md (restart barrier)
Upgrades · a rolling restart with new binaries
Upgrade management servers, then data nodes, then API nodes
- Take a backup before moving to a new major version.
- Install the new tarball beside the old one; switch the symlink:
ln -sfn. - Restart management servers one at a time, then data nodes one per node group at a time, then MySQL and REST servers.
- Nodes check each other's version before using a new feature, so mixed versions work during the upgrade. Backups are refused until all data nodes match.
Complex upgrades and downgrades may need initial node restarts, or a backup and restore (module 05 showed the two-cluster way).
docs: rondb_upgrade.md (General approach to Upgrading RonDB) · rondb_startup.md (initial node restart for complex upgrades) · code: backup/Backup.cpp (refused in mixed-version clusters)
Growing · more node groups, online
Growing online: add node groups, then spread every table onto them
# 1 config.ini: new [ndbd] slots with NodeGroup=65536, mgmd --reload and a rolling # restart; or activate slots prepared earlier with NodeActive=0 # 2 start the new data nodes ndbmtd --ndb-connectstring=mgm_host --ndb-nodeid=5 --initial ndbmtd --ndb-connectstring=mgm_host --ndb-nodeid=6 --initial # 3 make them a node group ndb_mgm -e "CREATE NODEGROUP 5,6" # 4 per table, in a MySQL server ALTER TABLE t1 ALGORITHM=INPLACE, REORGANIZE PARTITION; OPTIMIZE TABLE t1; # reclaim space the moved rows left behind
New tables use the new node group at once; REORGANIZE moves buckets of existing tables onto it (module 05).
docs: alter_table.md (Add Node Group, Reorganise existing tables) · rondb_online_config.md (CREATE NODEGROUP 5,6; OPTIMIZE TABLE)
Replicas · from two to three, online
Going to three replicas: activate an inactive slot
- Configure
NoOfReplicas=3from the start, with the third node of each node groupNodeActive=0(module 13). - When needed:
HOSTNAMEandACTIVATEthe slots in ndb_mgm. - Start the new nodes with
--initial: they copy their node group's data from the live replicas. - Managed RonDB works this way: always 3 replicas configured, 2 active unless asked for more.
docs: rondb_changing_cluster.md · config_num_replicas.md · table_options.md (Managed RonDB configures 3 replicas, one inactive)
Recap · module 18
What to take away
- Start mgmd → ndbmtd → mysqld → REST; stop in reverse, with kill -TERM.
- After the first start, the management server's configuration database rules;
--reloadchanges it. - HOSTNAME, ACTIVATE, DEACTIVATE and SET change a cluster without restarts.
- Rolling restarts: one node per node group at a time; upgrades the same way.
- Grow with CREATE NODEGROUP and REORGANIZE; add replicas by activating slots.
RonDB training · 25 modules · you are at 18