Architecture · from the AXE switch (module 01)

A data node is a set of blocks that talk only through signals

A bug can only reach another block through the data it sends, so errors stay isolated and findable. The price: reading the code means following signal flows.

docs: rondb_dn_arch.md (Block and Signal Architecture, Blocks)

In the code · one method per signal

A block receives a signal in a method named after it

// storage/ndb/src/kernel/blocks/dbtc/DbtcMain.cpp
void Dbtc::execTCKEYREQ(Signal *signal) {
  // read the request from signal->theData,
  // find the partition and its nodes,
  // then send LQHKEYREQ to the LDM
  // that owns the row
}

// storage/ndb/src/kernel/blocks/dblqh/DblqhMain.cpp
void Dblqh::execLQHKEYREQ(Signal *signal) { ... }

code: dbtc/DbtcMain.cpp (Dbtc::execTCKEYREQ) · dblqh/DblqhMain.cpp (Dblqh::execLQHKEYREQ) · the method bodies are summaries · docs: rondb_dn_arch.md (Scheduling rules)

The blocks · data and transactions

Blocks in the ldm and tc threads do the work of every query

code: storage/ndb/include/kernel/BlockNumbers.h (0xF4 BACKUP … 0x111 V_QUERY) · src/kernel/vm/mt.cpp (add_thr_map, mt_add_thr_map) · module 11 covers the threads

The blocks · running the cluster

Blocks in the main, rep and recv threads run the cluster itself

code: BlockNumbers.h · mt.cpp (add_thr_map: thr_GLOBAL is the main thread, thr_LOCAL the rep thread) · 30 block numbers, 244 to 273

Multi-threaded blocks · one instance per thread, one proxy

A proxy block fans a signal out to every instance

code: mt.cpp ("For multithreaded blocks we will assign the Proxy blocks below to their thread") · src/kernel/blocks/LocalProxy.hpp

Block reference · 32 bits, like an IP address

Block references: a node, a block and an instance in one 32-bit word

code: include/kernel/RefConvert.hpp (NDBMT_REAL_BLOCK_BITS 6, FIRST_BLOCK 244) · ndb_limits.h (MAX_NODES_ID 8191, MAX_DATA_NODE_ID 144)

Routing · decided by the address alone

Signal routing: another node goes to a send buffer, this node to a job buffer

docs: rondb_dn_arch.md (Block references, Send handling in Data Node)

Signal sizes · three kinds

Short, long and fragmented signals

Inside a node a signal takes at most 34 words in a job buffer: a 9-word header and 25 data words. Sections travel by reference.

code: src/kernel/vm/mt.cpp (MAX_SIGNAL_SIZE 34) · VMSignal.hpp · ndb_limits.h (60-word segments) · TransporterDefinitions.hpp (32768-byte message) · docs: rondb_dn_arch.md (Signal data structure)

On the network · after the connection is set up

A signal on the wire: header, data, section lengths, sections

docs: rondb_dn_arch.md (RonDB signal header definition, Send handling) · code: include/kernel/signaldata/PackedSignal.hpp

Connection setup · before any signal

Connection setup: two nodes greet each other in five lines of text

code: src/common/util/SocketAuthenticator.cpp (SocketAuthTls: "ndbd TLS disabled", "Cleartext ok", "TLS ok") · transporter/TCP_Transporter.cpp (use_tls_client_auth) · transporter/Transporter.cpp (hello "%d %d %d %d")

Scheduling · the block's own discipline

A signal runs to the end, so it must be short: 1–2 microseconds, never over 10

docs: rondb_dn_arch.md (Scheduling rules) · module 10 covers how scans split their work

Signals · priority, and starting new work

Signal priorities: two levels, and parallelism that costs nanoseconds

docs: rondb_dn_arch.md (Asynchronous signals, Parallelising signal execution) · code: ConfigInfo.cpp (MaxNoOfConcurrentScans default 256, 2–500)

Job buffers · one writer, one reader

Job buffers: every pair of threads has its own queue, so no locks are needed

One writer and one reader per queue: memory barriers instead of mutexes. Queues are 32 KB pages. Above 32 block threads, writers share 32 queues behind a mutex each.

code: mt.cpp (NUM_JOB_BUFFERS_PER_THREAD 32, MAX_SIGNALS_PER_JB 75, thr_job_buffer 8188 words) · docs: rondb_dn_arch.md (Receive handling)

The scheduler loop · one per block thread

The scheduler loop: execute, flush, send, check timers, sleep or spin

1. Execute signals from the job buffers, up to 75 from each.
2. Flush signals for other threads in this node: after 2 signals in a recv thread, 5 in a tc thread, 20 in others.
3. Send: hand send buffers to the send threads, or send directly when not overloaded.
4. Timers: signals sent with a delay of some milliseconds become due.
5. Sleep when idle, after spinning a while if spinning is likely to pay off.

SchedulerResponsiveness tunes how many signals run between flushes and sends: lower latency, or more throughput.

code: mt.cpp (MAX_SIGNALS_BEFORE_FLUSH_RECEIVER 2, _TC 5, _OTHER 20) · docs: rondb_dn_arch.md (Send handling) · module 12 covers spinning

The hot path · signals between threads, calls inside one

Inside an ldm thread, a key operation is three function calls

code: dblqh/DblqhMain.cpp (c_acc->execACCKEYREQ, c_tup->execTUPKEYREQ) · docs: rondb_dn_arch.md (Synchronous signals)

Sending · who writes the sockets

Send threads send, and block threads help when they have time

recv threads wait in epoll and wake as soon as data arrives. They do not help the send threads.

docs: rondb_dn_arch.md (Send handling in Data Node, Receive handling in Data Node) · code: mt.cpp (do_send, assist_send_thread) · thrman.cpp (get_load_status)

Recap · module 06

What to take away

• Blocks own their data; signals are the only way in. Bugs stay isolated.
• A 32-bit reference names node, block and instance: 64 blocks, 1024 instances.
• Short signals carry 25 words; long ones up to 32 KB in 3 sections.
• Signals run to completion in 1–2 microseconds; long work re-queues itself.
• Lock-free job buffers between threads; direct calls inside one.
• Key lookups call LQH → ACC → TUP directly.
• Range scans call LQH → TUX → TUP, and ACC when they lock rows.

Next · 07 · Data structure layout

RonDB training · 25 modules · you are at 06

Next · 07 · Data structure layout