RonDB training · module 11 of 25 · 18 min

The RonDB thread model

Which threads a data node runs, what each one owns, and the pipeline a request travels through them.

The problem · millions of tiny tasks

The threading problem: a task takes 500 nanoseconds to 10 microseconds

An operating system thread per task

A context switch costs about as much as the task itself.
Time slicing interrupts work at random points.
Terrible for both latency and throughput at this task size.

RonDB's answer

A few long-lived threads, ideally one per CPU.
Each runs blocks and its own scheduler of signals (module 06).
Tasks are state machines, split further into signals.

In the early days this alone let NDB beat time-sharing competitors by an order of magnitude.

docs: automatic_thread_config.md (task lengths of 500 ns up to around 10 microseconds; Traditional thread design for key-value stores)

History · how the model grew

Thread model history: from one thread to a pipeline, then query work everywhere

The first data node, ndbd, ran everything in one thread: efficient, but one CPU.

ndbmtd split the blocks over threads by function, and the data over ldm threads by partition.

Query threads let reads of a partition run beside its ldm thread.

Now any block thread can serve reads: there are no query threads, only query workers.

code: git d0793a5dc60 (2023-12-15, "RONDB-556: All block threads can become query workers") · src/common/mgmcommon/thr_config.cpp (no query thread type) · docs: rondb_main_programs.md

Thread types · as ThreadConfig names them

Nine thread types; only recv, io and watchdog always exist

Type	Count	Runs
ldm	0 – many	Owns partitions: DBLQH, DBACC, DBTUP, DBTUX, BACKUP, RESTORE, PGMAN
tc	0 – many	DBTC and DBSPJ: transactions, pushed queries
recv	1 – many	Reads sockets, routes signals; TRPMAN
send	0 – many	Writes sockets
main · rep	0 – 1 each	Schema, distribution, membership, files · change events, replication
io	pool	File reads and writes, compression; grows as needed
watchdog	1	Checks that every thread makes progress
idxbld	0 – 1	Offline ordered index builds during restarts

code: src/common/mgmcommon/thr_config.cpp (m_parse_entries, m_entries: min and max per type) · module 06 lists the blocks

ldm · local data manager

An ldm thread owns its partitions: every write to them runs there

Owns

Rows, hash indexes, ordered indexes, REDO logging and checkpoints of its fragments.

Efficient

IPC of 1.5 in Sysbench, above 2 in filtering scans. Under 2% branch mispredictions and L1 misses.

Cache-hungry

About 15% last-level cache misses: the row data. Keep it on its own CPU, locked.

ldm threads signal tc, main and rep threads, and other ldm threads. A second hyperthread on the same core adds only about 15%.

docs: rondb_vm_arch.md (ldm threads) · rondb_hyperthreading.md · code: src/kernel/vm/mt.cpp (may_communicate: ldm to tc, main, rep and ldm)

Query workers · reads beyond the owning thread

Query workers: reads of a partition can run in any thread of its round-robin group

Writes, and everything that changes a partition, go to the owning ldm thread.

Reads go through the virtual block V_QUERY to query-worker copies of the ldm blocks (DBQLQH, DBQTUP …) in any thread of the group. Locked reads too, by default.

code: BlockNumbers.h (DBQLQH … QRESTORE, V_QUERY) · ConfigInfo.cpp (KeyBatchSizeQueryWorker 4, UseQueryThreadLockedReads true, MaxRRGroupSize 8, 8–32) · git d0793a5dc60

Consequence · fewer partitions are enough

Query workers make reads scale without more partitions

Before

Only the owning ldm thread could touch a partition: to use more CPUs, tables needed more partitions.

Now

One partition can be read by many threads at once: millions of reads per second from a single partition.

So

PartitionsPerNode = 2 by default: cheap unpruned scans, and the CPUs still all busy.

Partitions stay invisible to the application: the API never needs to know which thread holds a row.

docs: table_options.md (Number of partitions per table, PartitionsPerNode) · code: ConfigInfo.cpp (PartitionsPerNode default 2)

tc · transaction coordination

tc threads run many short signals and do much of the sending

Coordinate

DBTC routes operations and runs the commit protocol; DBSPJ runs pushed joins.

Short and bursty

Tiny signals, lower IPC, frequent sleeps. Hyperthreading adds about 40%.

Senders

They pass through the scheduler loop often, so they assist sending most: lower response times.

Too many tc threads send many small packets to the API nodes and can lower total performance.

docs: rondb_vm_arch.md (tc threads) · rondb_hyperthreading.md

recv · send · the network

recv threads must wake fast; the other block threads help the send threads

recv

Each owns a set of connections and waits in epoll.
Keep each under about 60% CPU, so it wakes at once: add recv threads above that.
Hyperthreading adds up to 70%.

send

About one send thread per 12 threads: one for a small node, 4 for a 48-CPU node (module 12).
ldm, tc, main and rep threads help below about 75% load; recv threads do not, so they stay free to receive.
Hyperthreading adds about 40%.

docs: rondb_vm_arch.md (recv threads, send threads) · code: thrman.cpp (load levels 30, 60, 75, 85) · mt.cpp (do_send: recv threads assist only when there are no ldm threads) · thr_config.cpp ("recv threads never assist send threads"; 1 send thread at 16 CPUs, 2 at 24, 4 at 48, n/12 above)

main · rep · io · watchdog · idxbld · the quiet ones

The service threads are idle most of the time, and spike on schema changes

main · rep

Schema, distribution, heartbeats, change events. Busy during ALTER TABLE, index and foreign key builds. Idle, they help the send threads. With none, recv threads do this work.

io

File reads and writes. Little CPU for in-memory data; much more with compressed backups and checkpoints or disk columns.

watchdog · idxbld

One thread checks all others make progress (module 19). Index builds at restart can get their own threads.

docs: rondb_vm_arch.md (main threads, rep threads, io threads) · automatic_thread_config.md · code: thr_config.cpp

CPU share · a busy data node

CPU share: about half the CPU reads and writes data; the rest moves it

docs: automatic_thread_config.md (ldm 50-60%, tc 20-25%, recv 10-15%, send around 10%)

PART 2

The thread pipeline

A request passes through several threads, each running at full speed.

The pipeline · one key lookup

A key lookup's journey: recv, tc, ldm or query worker, then back to the application

A recv thread reads the request from the socket. The API never needs to know which thread holds the data.

It queues the request for a tc thread, through a lock-free job buffer.

DBTC routes it to the owning ldm thread, or for a read, to a query worker.

The reply goes back without passing tc again. A key lookup takes 2–3 microseconds of execution in all.

docs: design_thread_pipeline.md (Latency in the Thread Pipeline) · rondb_signal_arch.md

Measured · splitting the work over threads

Splitting a data node into a pipeline doubled its throughput

One thread doing everything handled about 750 000 operations per second.

A separate recv thread: 835 000. Receiving is a small part of the work.

Splitting transaction processing from data ownership: 1.5 million, without splitting the data.

Two threads of each type scaled to 3 million. Critical code reaches an IPC of 1.27; many databases report 0.25–0.5.

docs: design_thread_pipeline.md (results 750k, 835k, 1.5M, 3M operations per second; Efficiency of the Thread Pipeline)

Latency · the price of hand-offs

Hand-off latency: wake-ups cost up to 25 µs when idle, almost nothing under load

5–10 µs

to wake a sleeping thread; up to 15–25 µs added when all are asleep

2–3 µs

of execution for a key lookup; a write about 5 µs

≈50 µs

latency in a loaded data node: wake-ups add at most about 10%

Under load threads stay awake and batches grow: RonDB gets more efficient as load rises. Module 12 shows how spinning hides the idle wake-up cost.

docs: design_thread_pipeline.md (Latency in the Thread Pipeline)

Measured · dedicated ldm threads or all-in-recv

Separate ldm threads beat doing everything in recv threads

	LDM setup	Receive setup
Key lookups per second	1.34 M	1.05 M
IPC of the data thread	1.40	1.05
CPU frequency	3.96 GHz	3.468 GHz

Threads that run one kind of code keep their caches warm and the CPU at full clock. Very small nodes may still run everything in recv threads (module 12).

docs: research_thread_pipeline.md (LDM setup 1.34M vs Receive setup 1.05M lookups per second; IPC and frequency)

Hyperthreading · gain from the second CPU of a core

Hyperthreading helps the network threads most, the ldm threads least

Thread	Gain from two CPUs per core	Why
ldm	+15%	Already keeps the core's execution units busy
tc · send	+40%	Short code paths, frequent waits
recv	up to +70%	Waits on the network a lot
MySQL server threads	+60%	For comparison

Automatic configuration therefore puts two ldm threads on the two CPUs of a core, and pairs leftovers with tc threads.

docs: rondb_hyperthreading.md · code: thr_config.cpp ("We will assign 2 LDM threads, next 2 non-LDM threads"; a leftover ldm shares its core with a tc thread; RONDB-1068)

Recap · module 11

What to take away

A few long-lived threads with their own schedulers run tasks of 0.5–10 µs.
ldm threads own partitions; tc coordinates; recv and send move signals.
Every block thread is a query worker: reads scale without more partitions.
Round-robin groups keep reads within one L3 cache.
The pipeline doubled throughput; wake-up costs vanish under load.

Next · 12 · Thread configuration, CPU locking and spinning

RonDB training · 25 modules · you are at 11