RonDB training · module 19 of 25 · 16 min

Monitoring and deployment

Seeing inside a running cluster, finding out why something failed, and the ways to run RonDB: by hand, in containers, on Kubernetes, or managed.

Monitoring · four sources

Monitoring: SQL views for live state, logs for history, trace files for crashes

ndbinfoSQL tables of live statefrom every data node ndb_topCPU per thread, livecounts spinning right Logscluster log: all nodesnode log: one process Trace filesper thread, at a crash:code path and signals

ndb_mgm REPORT MemoryUsage and REPORT BackupStatus give quick answers too (module 18).

docs: rondb_ndbinfo.md · rondb_logs.md · rondb_troubleshoot.md · rondb_advanced_thread.md (ndb_top)

ndbinfo · a database of the cluster's state

An ndbinfo query asks every block of every thread of every data node

A special scan

Sent to all data nodes, every thread and block; blocks that know the table answer with rows.

Cannot overload

One node at a time, 10 rows per round trip from MySQL, at most 256. Big answers just take longer.

Cheap to poll

ndb_top queries it every second with barely noticeable load. Fragment tables can return millions of rows: careful.

Base tables are ndb$…; the friendly tables are views over them. MySQL and API nodes report their part through the data nodes.

docs: rondb_ndbinfo.md (How ndbinfo works) · code: src/common/debugger/NdbinfoTables.cpp · plugin/ha_ndbinfo.cc (ndbinfo_max_rows 10, at most 256)

ndbinfo · where to look

ndbinfo tables: one for every question

QuestionTable
Am I running out of memory?memoryusage · resources · table_memory_usage
How busy are the threads?cpustat · cpustat_1sec · threadstat
Is the network keeping up?transporters
Are REDO and UNDO logs filling?logspaces · logbuffers
Is the disk page cache big enough?diskpagebuffer
Who holds locks, which transactions run?cluster_locks · cluster_transactions
How long did the last restart take?restart_info
Who is master and arbitrator?membership · nodes

docs: rondb_ndbinfo.md (table sections)

Memory · the first thing to watch

Watch DataMemory: normal operation stops at 95%

SELECT node_id, resource_name, used, max
  FROM ndbinfo.resources
 WHERE resource_name = 'DATA_MEMORY';

-- used and max are in 32 KB pages
  • Inserts start failing near the limit; leave headroom for recovery (module 13).
  • Error 410 on writes means the REDO log is full: checkpoints or disks cannot keep up (module 14).
  • table_memory_usage shows which tables use it.

docs: rondb_probe_cluster.md (the query; unit is pages of 32768 bytes; error 410) · automatic_memory_config.md (95%)

CPU · per thread

CPU per thread: top lies about spinning threads; ndb_top and cpustat do not

Why top misleads

  • Spinning threads look 100% busy to the OS (module 12).
  • Several threads per CPU hide who does what.

ndb_top and cpustat

  • Per thread: time executing, sending, spinning, sleeping.
  • Over the last 50 ms, 1 s and 20 s windows.
  • Spot the bottleneck: ldm, tc, recv or send.

A recv thread above about 60% busy is a sign to add recv threads (module 11).

docs: rondb_vm_comm.md (CPU spinning makes top less useful) · rondb_ndbinfo.md (cpustat, cpustat_50ms, cpustat_1sec, cpustat_20sec) · rondb_vm_arch.md (recv threads)

Logs · cluster-wide and per process

The cluster log tells what happened; node logs tell why

Cluster log

Written by the management server: starts, stops, checkpoints, backups, node failures, from all nodes. Filtered by category with CLUSTERLOG.

Node log

One per data node and management server: configuration chosen, memory sizes, restart progress, watchdog warnings.

MySQL error log

For each MySQL server: connection to the cluster, schema distribution, its own problems.

Watchdog lines appear when a thread is stuck over 100 ms: often the OS or a VM stalling RonDB, sometimes a bug. A few during early startup are normal.

docs: rondb_logs.md (The Cluster log, Data node logs) · rondb_failure_protocols.md (Watchdog Handling)

Fail fast · crash, recover, analyse

Fail fast: a data node crashes on the first sign of inconsistency, and records why

Fail fast

Asserts everywhere: better to restart clean than run on with bad pointers. The other replica serves meanwhile.

jam()

Every thread records its last 1024 code positions, file and line: AXE's jump trace in C++, for 5–10% overhead.

Signal log

The last few thousand signals per thread: milliseconds to seconds of history, even across threads and nodes.

The error log names the crashed thread; a trace file per block thread holds jam and signals. Tar the log directory and attach it to a GitHub issue.

docs: rondb_troubleshoot.md (Data node crash output) · rondb_report.md (Preparing a crash tarball) · code: vm/Emulator.hpp (EMULATED_JAM_SIZE 1024 in release builds)

PART 1

Deployment

By hand, in containers, on Kubernetes, or managed.

Deployment · four ways

Deployment: pick the way that matches who operates the cluster

WayGood for
Tarballs, by hand or with your own toolingBare metal and VMs you control; modules 13 and 18
Docker, docker-composeLocal clusters for development and learning
Kubernetes with the Helm chartAny cloud or your own Kubernetes; scaling, backups to S3, global replication
Managed RonDB in HopsworksNo operations: choose sizes, the platform runs it

docs: rondb_docker.md · rondb_helm.md · managed_rondb.md · rondb_installation.md

Kubernetes · the RonDB Helm chart, Apache 2.0

One Helm release is a whole cluster, sized by a values file

helm upgrade -i my-rondb \
  --namespace rondb-default \
  --values ./values/minikube/small.yaml \
  .

# change the values file, run it again
helm test -n rondb-default my-rondb \
  --logs --filter name=generate-data
  • Custom sizes, multi-AZ clusters, more or fewer replicas.
  • MySQL and REST servers scale out automatically when busy, above about 70% CPU.
  • Backups to S3 on a schedule; restore from them.
  • Global replication between clusters.

docs: rondb_helm.md (supported features, helm upgrade -i, helm test, autoscaling) · github.com/logicalclocks/rondb-helm

Kubernetes · getting the automatic configuration right

Kubernetes: give data node pods whole CPUs and a memory limit; RonDB sizes itself

CPUs

  • The static CPU manager (Kubernetes 1.27+) pins a pod to whole CPUs.
  • RonDB then sees exactly those CPUs, and locks its threads to them (module 12).
  • Ask for an integer number of CPUs for data nodes.

Memory

  • Automatic memory reads the container's cgroup limit (module 13).
  • Set a smaller OsStaticOverhead: the node has the pod to itself.
  • Graceful stops via SIGTERM fit Kubernetes pod shutdown (module 15).

docs: rondb_helm.md (Static CPU Manager) · automatic_memory_config.md (Kubernetes example, OsStaticOverhead) · code: NdbHW.cpp (cgroup memory limit)

Helm · scheduled backups

Backups go to S3-compatible storage on a cron schedule

backups:
  enabled: true
  pathPrefix: "rondb_backup"
  schedule: "0 3 * * mon"
  s3:
    provider: Other
    endpoint: "https://s3.bhs.io.cloud.ovh.net"
    bucketName: "jun-25"
    region: "bhs"
    keyCredentialsSecret:
      name: "aws-credentials"
      key: "aws-access-key-id"
  • Every Monday at 03:00 in this example, from OVH's cloud.
  • Credentials come from Kubernetes secrets.
  • restoreFromBackup in the values starts a new cluster from one of them.

docs: rondb_helm.md (backups example, shortened; restoreFromBackup)

Managed RonDB · in Hopsworks

Managed RonDB in Hopsworks: choose the sizes, the platform does the rest

Data nodes

A multiple of the replicas: 1, 2 or 3 replicas; more node groups only when data outgrows one machine.

MySQL and API

MySQL servers and API nodes sized to the query load, each with its own VM type.

Online changes

Configured with 3 replicas, inactive ones held back: replicas and machines change online (module 18).

Hopsworks 4.x and 5.x run RonDB with the Helm chart, in the same way; Hopsworks 3.x had its own management.

docs: managed_rondb.md (Create a new RonDB cluster, Setting up RonDB Data nodes) · table_options.md (Managed RonDB configures 3 replicas)

Recap · module 19

What to take away

  • ndbinfo: SQL views of live state from every data node, safe to poll.
  • Watch DataMemory, thread CPU (ndb_top), transporters and log spaces.
  • Cluster log for events, node logs for detail; trace files explain crashes.
  • Fail fast: a node crashes cleanly, its replica serves, the logs tell why.
  • Run by hand, in Docker, with the Helm chart on Kubernetes, or managed.

Next · 20 · Native APIs

RonDB training · 25 modules · you are at 19

Next · 20 · Native APIs