iClaustron training for Rust developers · module 2 of 9 · 20 min

Thread architecture

Which threads an iClaustron API node runs, what each one owns, and the path a key read takes through them.

The rule · fixed for the life of the project

The one rule: receive threads receive and route, user threads execute

The reference NDB API

  • Its receive thread decodes each signal and updates the transaction.
  • Then it wakes the user thread waiting for it.
  • With many user threads, that receive thread is the bottleneck.

iClaustron

  • A receive thread's work per signal: a table lookup and a copy into a page.
  • The user thread that owns the query decodes rows, updates the transaction, runs callbacks.
  • That work grows with the application's threads, not with a fixed pool.

One exception, by design: signals about the link itself, such as HEARTBEAT_CONF, run on the receive thread that owns the node.

iclaustron: doc/rust/architecture.html §1 (Threading architecture); doc/rust/02-architecture.md (Threading model); crates/ic_apid/src/thread_conn.rs (module note); rec_thread.rs (Receiver::execute)

The thread map · learn it once, it comes back

The thread map: who sends, who receives, who keeps the links up

user threads · a connection each… up to 4 711 connections user thread 0block 0x8000 · inbox user thread 1block 0x8001 · inbox data nodesreceive threadscoordinatorfragment ownerssend threads send bufferper data node · writer flag send pool · 1 threadwrites what came meanwhile 1 write per node receive threadsroute by block numberone page per inbox replies heartbeat1 thread connect1 per node connect, handshake executora user thread whose inboxalso takes requests application threadsapp threadapp threadapp threadno connection

Every thread that talks to RonDB holds a connection: its own block number, its own inbox, state no other thread touches.

Requests leave through one send buffer per data node: the user thread writes the socket, the send pool only what came meanwhile.

Replies arrive on the receive thread that owns the node. It reads the block number only and posts one page per inbox.

The heartbeat thread and a connect thread per data node keep the links up. Neither carries a transaction's signals.

An executor is a user thread that runs requests for application threads with no connection of their own.

iclaustron: doc/rust/architecture.html §1 (The threads today); crates/ic_apid/src/apid_global.rs (ApidGlobal::start_with_options); thread_conn.rs (ThreadTable, Router); executor.rs (Executor::start)

On the map · one primary key read

A primary key read, step by step through the threads

user threads · a connection each… up to 4 711 connections user thread 0block 0x8000 · inbox user thread 1block 0x8001 · inbox data nodesreceive threadscoordinatorfragment ownerssend threads send bufferper data node · writer flag send pool · 1 threadwrites what came meanwhile receive threadsroute by block numberone page per inbox heartbeat1 thread connect1 per node executora user thread whose inboxalso takes requests application threadsapp threadapp threadapp threadno connection KEY_OP_REQ by thread 0 KEY_OP_CONF ROW_DATA_REP to block 0x8000 inbox

User thread 0 reads one row by primary key. Watch which thread does each part of the work.

Define: read_key packs KEY_OP_REQ into the connection's staged buffer for the coordinator's node. No lock, nothing sent.

Send: flags are set in place; thread 0 appends to the node's send buffer and, with no one else writing, writes the socket.

Data node: the coordinator answers KEY_OP_CONF, the node that read the row sends ROW_DATA_REP; both to block 0x8000.

Route: the receive thread that owns the link reads only the block number, 0x8000 + 0, and puts both in thread 0's page.

Post: once a round, that page goes into thread 0's inbox under one short lock. Thread 0 is woken only if it sleeps.

Poll: thread 0 joins any fragments, finds transaction and query by each reply's first word, decodes the row, runs the callback.

iclaustron: crates/ic_apid/src/transaction.rs (read_key, define, send_queries, route_reply); apid_global.rs (NodeShared::send_words); rec_thread.rs (route_signals); thread_conn.rs (post); apid_conn.rs (poll_with_requests)

Thread types · as the global starts them

Thread types: how many of each, what each does and touches

ThreadHow manyWhat it doesWhat it touches
User threadone per connection, up to 4 711packs and sends requests, polls its inbox, decodes rows, runs callbacksits connection, with no lock; a node's send buffer, to append
Executora user thread; as many as startedruns application threads' requests, many to a sendits connection; the requests in its inbox
Receive thread1 by default, at most one per data nodereads its nodes' sockets, routes by block number; runs HEARTBEAT_CONF and the node-loss reportsits sockets; each inbox's lock once a round; its nodes' state, as only writer
Send pool1writes what came during another thread's write; ends adaptive-send waitsthe nodes' send buffers
Heartbeat1HEARTBEAT_REQ to every node that is up; asks for a silent link to be droppednode state, read only
Connectone per data nodegets the port, connects, handshakes, hands the link overthe membership word, by compare-and-swap

iclaustron: doc/rust/architecture.html §1 (thread table); crates/ic_apid/src/apid_global.rs (start_with_options); rec_thread.rs (Receiver::execute); send_pool.rs; heartbeat.rs; connect_thread.rs; executor.rs; crates/ic_port/src/consts.rs (IC_MAX_THREAD_CONNECTIONS)

Starting · from the Rust examples

Connecting starts the library's threads; a connection is one user thread's

let global = ApidGlobal::connect(
  connect_string(args),
  Some(name),
  &GlobalOptions::default(),
)?;
if global.wait_for_all_started(EX_START_WAIT_MS) == 0 {
  return Err(IcError::new(err::IC_ERROR_NO_STARTED_DATA_NODE));
}
let conn = global.create_connection()?;
Ok((global, conn))
  • ApidGlobal::connect starts the receive threads, the send pool, the heartbeat thread and a connect thread per data node, then returns. Dropping the global stops them all and waits for each.
  • receive_threads in the options: 1 by default, never more than there are data nodes.
  • Each connection is one user thread's: block 0x8000 plus its id, at most 4 711 in a process.

iclaustron: examples/rust/lib.rs (connect); crates/ic_apid/src/apid_global.rs (GlobalOptions, ApidGlobal and its Drop, ApidGlobal::connect, start_with_options, create_connection); crates/ic_util/src/threadpool.rs (ThreadPool::stop, Drop); crates/ic_signals/src/blocks.rs (api_block_of_thread)

PART 1

Sending and receiving

One write per data node out, one page per user thread back.

Send path · the thread that sends does the write

The send path: one buffer per data node, and the sender writes it

user thread 1appends, then writes send buffer · data node 2 waitingsignals appended spareempty, kept writer flag: none the node's mutex: held to append and claim,never for the write data node 2one socket writer flag: user thread 1 outside the mutex user thread 2appends, goes on send pool1 threadtakes what came meanwhile handed to the pool

User thread 1 appends its packed requests to data node 2's send buffer, under the node's mutex.

No one is writing: it sets the writer flag, swaps the waiting buffer for the spare, lets go of the mutex and writes the socket.

User thread 2 appends meanwhile, sees the flag and goes back to its own work: its signals wait for the thread that writes.

Write done, more waiting: thread 1 does not write for others. It hands the node to the send pool, which writes the rest.

iclaustron: crates/ic_apid/src/apid_global.rs (NodeShared::send_words_for_generation, write_out, take_waiting); send_pool.rs (module note, SendPool::ask, run_send_thread); doc/rust/architecture.html §1 (The thread that sends does the write)

Send path · what is held back, and for how long

Adaptive send and acknowledgements: writes that wait for company

Adaptive send

  • Without force, a node's write may wait for the next thread's signals, if no write is under way.
  • Bounded: 200 µs as built, and only as many sends as it has learned to hold, at most 8.
  • The send pool writes them when the wait ends. With force the write happens now.

Commit acknowledgements

  • The acknowledgement a commit calls for, COMMIT_MARKER_FREE_ORD, is queued in the connection.
  • It leaves with the next send, in the same write as new requests.
  • Written alone only after 1 ms, or when nothing is in flight.

A thread sending alone in lock step never sees the count rise. pipeline.rs flushes with force false, finish in lib.rs with true.

iclaustron: crates/ic_apid/src/adaptive_send.rs (AdaptiveSend::decide, adjust); apid_global.rs (send_words); apid_conn.rs (poll_with_requests); transaction.rs (send_commit_ack, flush); examples/rust/pipeline.rs; lib.rs (finish)

Receive path · pages that go round

The receive path: receive pages in, one signal page per user thread

receive thread inbox user thread receive page 128 KB: 4 largest messages its own, per link read again while full sealed while others hold it up to 64 kept free router splits signals unpacks BUNDLE sorts by block no lock held signal page per user thread, per round < 256 words: copied ≥ 256 words: by reference holds its receive page 8 emptied pages kept poll pages out, empties back fragments joined first read in place row into attr_row callbacks run emptied pages go back

A receive thread reads its nodes' sockets into receive pages of its own, reading again while each read fills the room it had.

The router splits the bytes into signals, unpacks BUNDLE and sorts each by block number, with no lock held.

Small signals are copied into the thread's page; a large ROW_DATA_REP stays in its receive page, sealed until it comes back.

poll gives the emptied pages back and takes the posted ones. Each page is allocated and freed on the thread that owns it.

iclaustron: crates/ic_apid/src/signal_reader.rs (SignalReader, IC_RECEIVE_PAGE_WORDS); rec_thread.rs (read_node, route_signals); signal_page.rs (module note); thread_conn.rs (IC_SPARE_PAGES); apid_global.rs (IC_LARGE_SIGNAL_WORDS, IC_EXTRA_READS); apid_conn.rs (receive_view)

Receive path · posting to an inbox

Inbox posts: one lock per round, a wake-up only for a sleeping thread

One lock a round

The router sorts a round's signals per user thread with no lock held, then posts each page under one short lock: a hundred signals cost one lock.

Woken only asleep

Before it sleeps, the user thread sets the inbox's waiting flag under the lock. A post to a busy thread makes no system call.

Woken after unlock

The wake-up goes after the lock is let go, so the thread never wakes into a held mutex. A wake-up for signals already taken is harmless.

Several receive threads post to one inbox, a ThreadConnection behind an Arc, each under the same lock. conn.poll gives emptied pages back and takes the posted ones in one lock.

iclaustron: crates/ic_apid/src/thread_conn.rs (module note; ThreadConnection::post, exchange_with_requests; Router::post_all); rec_thread.rs (Receiver::round); apid_conn.rs (ApidConnection, poll); doc/rust/architecture.html §1 (The rules that keep it fast)

PART 2

Node state and failure

One writer per node, and a lost link kept apart from a lost node.

Node state · no mutex, one writer

Node state has one writer: the receive thread that owns the node

the node's receive threadsees the socket closeexecutes HEARTBEAT_CONF published link stateconnected · start level · generationlast HEARTBEAT_CONF time user threadsread it before sending writes reads heartbeat threadreads it, asks for drops membership wordmay the node be connected to? any receive threadNODE_LOST_REP, NODE_LOST_DONE_REP connect threadclaims a new link link lost

A node is dealt to one receive thread at start and stays there. It sees the socket close and executes HEARTBEAT_CONF: the only writer of its PublishedNodeState, atomics others read with no lock.

The heartbeat thread reads when the last HEARTBEAT_CONF came. After 4 check intervals without one, it asks the owner to drop the link.

Membership has several writers, so every change is a compare-and-swap: the first evidence wins. A failure report outranks the rest.

iclaustron: crates/ic_apid/src/node_state.rs (module note); apid_global.rs (NodeShared, cas_membership, node_failure_reported, claim_link); heartbeat.rs (keep_time); rec_thread.rs (install, link_lost); doc/rust/02-architecture.md (Node state has one writer)

Node failure · from the threads' side

Link lost or node lost: what a waiting transaction sees

Link lost: our own evidence

  • The socket closes, a write fails, or 4 check intervals pass without HEARTBEAT_CONF.
  • The owner publishes the link down and bumps a link-event counter; a poll checks it with one load.
  • Transactions coordinated over that link end at the next poll with 7135, the outcome unknown.

Node lost: NODE_LOST_REP

  • A data node reports the node failed; the link is dropped if it still stood.
  • Its transactions wait for the takeover: TAKEOVER_CONF says committed, TAKEOVER_REF aborted (4031).
  • At NODE_LOST_DONE_REP, whatever is still open is rolled back with 4010.

A lost link says nothing about the node, which may be up and serving everyone else. Neither case is reported as the cluster failing.

iclaustron: crates/ic_apid/src/rec_thread.rs (read_node, link_lost, execute); apid_global.rs (note_link_event, node_failure_reported, takeover_reported); transaction.rs (module note, lost_transactions, fail_lost_transactions); doc/user-guide.md §10

Node failure · when a node is connected to again

Membership decides when a connect thread connects to a node again

connecteda link is up disconnectedits connect thread connects to it awaiting takeovernot connected to connect thread claims a new link link lost: connect againafter 1 s, up to 10 s NODE_LOST_REP, from any state NODE_LOST_DONE_REP: connected to at onceor no data node left connected

With no data node connected, every wait ends, cached table definitions are dropped, and a node id the management server chose is claimed again first.

iclaustron: crates/ic_apid/src/apid_global.rs (LinkStatus, retry_delay_ms, claim_link, node_failure_reported, takeover_reported, note_link_lost); connect_thread.rs (run_connect_thread); doc/rust/02-architecture.md (Node failure flow)

PART 3

Executors, locks and numbers

A user thread that works for others, the order locks are taken in, and what it all does today.

On the map · executor threads

An executor is a user thread whose inbox also takes requests

user threads · a connection each… up to 4 711 connections user thread 0block 0x8000 · inbox user thread 1block 0x8001 · inbox data nodesreceive threadscoordinatorfragment ownerssend threads send bufferper data node · writer flag send pool · 1 threadwrites what came meanwhile receive threadsroute by block numberone page per inbox heartbeat1 thread connect1 per node executora user thread whose inboxalso takes requests application threadsapp threadapp threadapp threadno connection one inbox · one wait point 1 write per node

An executor is a user thread like the others, started by the library: a connection, a block number, an inbox.

Application threads with no connection submit requests through a submitter, into a second list in the executor's inbox.

Replies arrive in the same inbox, posted by the receive threads. Scan readers leave nudges on a list of their own.

One mutex, one condition, one waiting flag: the executor sleeps in one place and wakes for whichever comes first.

What it admits in one pass goes out in one write per data node, carrying every caller's requests. Module 06 goes deeper.

iclaustron: crates/ic_apid/src/executor.rs (module note, Executor::start, Submitter); thread_conn.rs (Requests, submit, nudge, exchange_with_requests); doc/rust/architecture.html §3 (An executor with two inputs; One wait point)

Executors · one pass of the loop

The executor's loop: take, execute, deliver, admit, send

  1. Take, in one lock: posted pages, requests, scan readers' nudges. Sleep only with nothing to do.
  2. Execute the replies: they complete operations and free coordinator records.
  3. Deliver each finished request to its slot, waking its caller, or to its callback.
  4. Admit waiting requests while fewer than the in-flight limit, 400 by default, are out.
  5. Send what this pass admitted: one write per data node.

Replies before requests: room in the pipeline is freed first. At its in-flight limit the executor sleeps for replies alone.

iclaustron: crates/ic_apid/src/executor.rs (Runner::run, admit_waiting, IC_EXECUTOR_IN_FLIGHT); thread_conn.rs (exchange_with_requests); doc/rust/architecture.html §3 (The executor's loop)

Locks · ic_port::sync levels · inside the crates: module 09

Mutex levels: rising order only

LevelMutexProtects
1a table's AUTO_INCREMENT rangeits values, across a refill's round trip
2the counter connectionthe connection the AUTO_INCREMENT counters are updated on
3dictionary cachetable and index definitions, shared by every thread
4globalthe thread table; our node id and configuration
5a user thread's inboxposted pages; an executor's requests and nudges
8a data nodeits send buffer and writer flag; the link a connect thread hands over
10send poolits queue of nodes to write for
11completiona slot an executor leaves a result in; a scan reader's feed

Each IcMutex is made with its level, and a debug build aborts on a lock taken out of order. Levels 6, 7 and 9 are not taken: no node changes receive thread, receive threads keep their own pages, the heartbeat keeps no list.

iclaustron: crates/ic_port/src/sync.rs (IC_MUTEX_LEVEL_*, check_order); IcMutex::new in crates/ic_apid/src; doc/rust/02-architecture.md (mutex levels)

Measured · primary key reads with ic_bench

Measured: 2.6 M reads a second on one user thread, up to 8.9 M on eight

SetupReads a secondClient CPU per readWhat limits it
Mac, 1 user thread, depth 42.6 M545 nsthe user thread: about 350 ns of user time per read
Mac, 4 user threads, batch 3005.8 M495 nscores shared with the data nodes' threads
Mac, 8 user threads, batch 3007.0 – 8.9 M512 – 565 nsone coordinator thread of a data node at 96%
Linux, 32 CPUs, 8 user threads, 2 receive threadsup to 7.1 M—the data node's receive thread

Reads of table t9; two release data nodes on the same machine; batches of 200 or 300, two in flight per thread unless stated. The limits today are on the data node side.

iclaustron: doc/rust/architecture.html §1 (What it does, measured); doc/rust/12-performance-model.md (Calibration: the Mac is an M5 Pro)

Planned · not built yet

How the threads can evolve: three planned steps

Planned

Poll set wake-up

A receive thread waits up to 20 ms in its poll set: a new link can go that long unwatched. A self-pipe or eventfd would wake it at once.

Planned

Wake-up helpers

Helper threads would take the wake-ups when a round wakes hundreds of threads, or an executor its callers. The send pool could grow.

Planned

Placement by load

Nodes are dealt to receive threads in turn at start. Placing them by measured load needs a hand-over that keeps one writer per node.

Each keeps the one rule: receive threads only route. Executors and the 4 711-connection limit were on this list and are built.

iclaustron: doc/rust/architecture.html §1 (How it can evolve); doc/rust/02-architecture.md (Building it in steps, step 7); crates/ic_apid/src/rec_thread.rs (module note); send_pool.rs (module note); apid_global.rs (IC_RECEIVE_POLL_MS)

Recap · module 02

Thread architecture: what to take away

  • Receive threads only route; user threads decode, update and call back.
  • A user thread writes its own requests: one write per data node a send.
  • Replies come through the inbox: one page and one lock per round.
  • A node's state has one writer, the receive thread that owns the node.
  • A lost link fails with 7135 and the node is connected to again; a lost node waits for takeover.
  • An executor is a user thread whose inbox also takes requests.

Next · 03 · Object architecture

iclaustron: doc/rust/architecture.html §1 and §3; doc/rust/02-architecture.md; crates/ic_apid/src (apid_global.rs, rec_thread.rs, thread_conn.rs, send_pool.rs, executor.rs)

iClaustron training for Rust developers · 9 modules · you are at 02 · this module for C developers

Next · 03 · Object architecture