iClaustron training for C developers · module 6 of 8 · 18 min

Key operations through an executor

Synchronous and batched synchronous calls from threads with no connection of their own: submitters, shapes, requests, and one executor thread that sends for them all.

The problem · work that arrives on the application's threads

Why executors: a thread per request cannot each have a connection

A connection per thread (module 04)

  • A connection is a user thread with a block of its own: at most 4 711 in a process.
  • A thread that sends one call at a time pays a whole round's costs for each: a write, a read, a round trip.

An executor

  • A thread of the library's, with one connection, runs the requests of any number of threads.
  • Each thread holds only a submitter: no block, no connection of its own.
  • Whatever one pass admits goes in one write per data node.

For programs whose work arrives on threads of their own, such as a server with a thread per request.

iclaustron: doc/user-guide.md §5 (Through an executor), §2 (one connection per thread) · crates/ic_apid/src/executor.rs (module note) · ic_port consts.rs (IC_MAX_THREAD_CONNECTIONS)

Objects · module 03 in short

The objects: one executor, a submitter per thread, shapes and requests

application threadIC_APID_SUBMITTER · IC_REQUEST application threadIC_APID_SUBMITTER · IC_REQUEST … any number, none with a connection IC_OP_SHAPEmade once · used by any thread IC_APID_EXECUTORthread ic_executor_N, started on the global IC_APID_CONNECTIONits own block · queries kept per shape inboxreplies · requests · nudges, one mutex IC_APID_GLOBALreceive threads · send poolheartbeat · connect threads data nodescoordinator · fragments submit send post signals

The executor, its shapes and its stats serve any thread; a submitter and a request belong to one thread at a time.

iclaustron: include/iclaustron/ic_apid.h (header note "Executors"; IC_APID_EXECUTOR, IC_APID_SUBMITTER, IC_OP_SHAPE, IC_REQUEST) · crates/ic_apid/src/executor.rs (Executor::start, Submitter) · thread_conn.rs (Inbox)

Starting · one executor, a submitter per thread

Starting an executor: its options, then a submitter per thread

IC_EXECUTOR_OPTIONS options;
IC_APID_EXECUTOR *executor;
IC_APID_SUBMITTER *sub;
IC_TABLE_DEF *table;
ic_executor_options_init(&options, sizeof options);
options.in_flight = 32;              /* 400 by default */
ret = ic_apid_executor_start(global, &options, &executor);
/* in each application thread: */
ret = ic_apid_executor_submitter(executor, &sub);
ret = ic_apid_submitter_table_bind(sub, "ic_ex", "t1", &table);
/* ... requests ... */
ic_apid_submitter_free(sub);
ic_apid_executor_free(executor);     /* before the global */
sizeset by the _init; a later release adds fields at the end
in_flight400: requests out at once, each a transaction at its coordinator
queue4 000: requests waiting; beyond, execute waits for room
scans64: scans running at once (module 07)

iclaustron: ic_apid.h (IC_EXECUTOR_OPTIONS, ic_apid_executor_start, _free) · examples/c/scan_executor.c (main) · ic_capi tests/c_api.rs

Shapes · the executor's form of a query

Operation shapes: a table or unique index and two records, made once

IC_RECORD *rec = ex_t1_record(table, 0);   /* t1.h */
IC_RECORD *code_rec = ex_code_record(table);
IC_INDEX_DEF *index;
IC_OP_SHAPE *shape, *by_code;

ret = ic_op_shape_create(table, rec, rec, &shape);
ret = ic_apid_submitter_index_bind(sub, "ic_ex", "t1_code$unique",
                                   "t1", &index);
ret = ic_op_shape_create_unique(index, code_rec, rec, &by_code);
/* ... at the end, any order: a request keeps what it needs */
ic_op_shape_free(by_code);
ic_op_shape_free(shape);
  • A shape says what an operation reads or writes: a key record and an attribute record, not the rows.
  • Made once, shared by every thread and request.
  • The executor keeps idle queries per shape and copies rows in and out: nothing is allocated per request.
  • Bind once: the executor waits for the dictionary while it binds.

iclaustron: include/iclaustron/ic_apid.h (ic_op_shape_create, ic_op_shape_create_unique, ic_apid_submitter_index_bind) · examples/c/t1.h, unique.c · executor.rs (OpShape, Runner::query_for, Runner::sweep)

Synchronous through an executor · one operation, one call

Synchronous through an executor: one operation, one blocking call

IC_REQUEST *req;
uint32_t op;
size_t len;

ic_request_create(&req);
ic_request_add_read(req, shape, IC_READ_KIND_COMMITTED, &op);
memcpy(ic_request_key_row(req, op, &len), &key, sizeof key);
if ((ret = ic_apid_submitter_execute(sub, req)) != 0)
  goto error;
ret = ic_request_op_error(req, op);
if (ret == IC_NDB_ERROR_NO_SUCH_ROW)
  /* no such row */;
else if (ret != 0)
  goto error;
else
  memcpy(&row, ic_request_attr_row(req, op, &len), sizeof row);
  • A request of one operation is a transaction of its own; execute blocks until it has ended.
  • 0 if it committed. A read of a missing row fails alone: committed, and the operation says 626.
  • Keep the request: put the next key in the same row and run it again.

iclaustron: doc/user-guide.md §5 (Through an executor, the C example) · include/iclaustron/ic_apid.h (ic_apid_submitter_execute, ic_request_op_error) · executor.rs (Request::outcome, Runner::complete_op)

Results · read once execute has returned

A request's results: each operation's outcome, then the transaction's

CallGives, for the last run
ic_apid_submitter_execute0 if committed; else the transaction's error: 630 for an insert of a row already there
ic_request_op_errorthe operation's error, or 0; 626 for a read of a missing row
ic_request_op_error_datawhat the data node said beyond the code: for 893, the unique index's id
ic_request_op_result_lenbytes a read brought; 0 for a write, or a row that was not there
ic_request_stateIC_COMMIT_STATE_COMMITTED or IC_COMMIT_STATE_ROLLED_BACK
ic_request_errorwhy it did not commit, or why its reads were lost when a takeover committed it
ic_request_gcithe global checkpoint the commit went into

iclaustron: include/iclaustron/ic_apid.h (ic_apid_submitter_execute, ic_request_op_error, _op_error_data, _op_result_len, ic_request_state, _error, _gci) · executor.rs (Runner::finish, KeyOp::error)

On the threads · module 02's threads, one call

A synchronous call through the threads: submit, take, send, wake

applicationthread executorthread receivethread data nodes time → execute inboxrequest list asleep on its slot takeone lock admitdefine, commit sendone write/node KEY_OP_REQ data node KEY_OP_CONFROW_DATA_REP routeby block post executerow → request hand backslot + wake returns 0

ic_apid_submitter_execute appends the request to the inbox's request list, then sleeps on the submitter's slot.

The executor's loop takes the request list in the same lock as the reply pages. If it slept, the submit woke it, after unlocking.

It admits the request: a transaction placed by the key, the operation as a query of its shape, the commit on it. Then one write per data node.

The coordinator answers KEY_OP_CONF, the row comes in ROW_DATA_REP, and a receive thread routes both by block number.

They reach the executor's inbox. Its next pass executes them: the row is copied into the request, the transaction is closed.

The request goes into the caller's slot and the caller is woken: execute returns 0, the row at ic_request_attr_row.

iclaustron: executor.rs (Submitter::execute, Runner::run, admit, complete_op, finish, deliver; Slot) · thread_conn.rs (submit, exchange_with_requests, post) · doc/rust/05-ndb-protocol.md (Names; §6.5)

Batched synchronous through an executor · many operations, one transaction

Batched synchronous through an executor: many operations, one call

ic_request_clear(req);
for (i = 0; i < n; i++) {
  ic_request_add_write(req, shape, IC_WRITE_KIND_INSERT, &op);
  memcpy(ic_request_key_row(req, op, &len), &r[i], sizeof r[i]);
  memcpy(ic_request_attr_row(req, op, &len), &r[i], sizeof r[i]);
}
ret = ic_apid_submitter_execute(sub, req);     /* one transaction */
if (ret == IC_NDB_ERROR_ROW_EXISTS)
  /* rolled back: none of the n rows is written */;
  • Defined together, the commit on the last, sent together: one transaction.
  • Placed by the first operation's key; ic_request_set_hint names a node or a partition.
  • A failed write rolls all back, a failed read fails alone: ic_request_set_abort_option per operation says otherwise.
  • Read, decide, then write: two requests.

iclaustron: crates/ic_capi/tests/c_api.rs (two inserts in one request, run again: 630, rolled back) · ic_apid.h (ic_request_add_write, ic_request_set_hint, ic_request_set_abort_option, IC_ABORT_DEFAULT) · doc/rust/04-api-design.md (Executors)

In the objects · module 03's objects, one request of three operations

Inside the executor: each operation runs as a pooled query of its shape

IC_REQUEST op 0 · insertkey row · attribute row op 1 · insertkey row · attribute row op 2 · readkey row · row read idle queries of the shape queryrequest 7 · place 0 queryrequest 7 · place 1 queryrequest 7 · place 2 rows in transactionplaced by op 0's keycommit on op 2one write/node data nodecoordinator KEY_OP_REQ ×3 KEY_OP_CONF · ROW_DATA_REP row out done: transaction closed, request to its slot

A request of three operations, two inserts and a read. Each owns its key row and attribute row, filled before the call.

Admitted, each operation takes an idle query of its shape, its rows copied in. The query carries the request's number and the operation's place.

The three queries go on one transaction, placed by the first key, the commit on the last, sent in the pass's one write per data node.

A reply names its query, the query its request and place: the row and each error are copied out, and the query goes back to its shape.

Once every operation has completed and the transaction is done, it is closed: state, error and GCI go into the request, and it goes to its slot.

iclaustron: executor.rs (Runner::admit, start_for, define_op, query_for, complete_op, finish_ended, finish; Flight) · doc/rust/04-api-design.md (Executors) · 05-ndb-protocol.md (Names)

On the threads · three callers, one pass

Batching across callers: one pass, one write per data node

applicationthreads executorthread receivethread datanodes time → A · execute B · execute C · execute inboxrequest list: A, B, Cno wake-up: the executor is busy busylast pass asleep on their slots takeone lock admitA, B, C sendper node KEY_OP_REQ node 1 · A, C node 2 · B replies route, post hand backslot + wake ×3 C: 0B: 0A: 0

Three threads call execute while the executor is busy: the requests wait on the list, and no submit pays for a wake-up.

The next pass takes all three in one lock and admits them: three transactions, each placed by its own first key.

Everything admitted in the pass goes in one write per data node: here node 1 carries A's and C's operations, node 2 B's.

The replies come back through a receive thread, one page per round, and the next pass hands each request back to its own slot.

Each caller is woken on its own: the writes are shared, the wake-ups are not. With 64 callers a send carried 21 requests.

iclaustron: executor.rs (Runner::run, admit_waiting; module note) · thread_conn.rs (submit: wake only if waiting) · doc/rust/architecture.html §3 (The executor's loop) · 06-phases.md §8.1 (21.0 requests a send)

The loop · one pass, every pass

The executor's loop in five steps: replies before admits

  1. Take, in one lock: reply pages, scan readers' nudges, and new requests while none wait in its own queue. Sleep only with nothing to do.
  2. Execute the replies: operations complete, a read's row is copied into its request.
  3. Hand back each request whose transaction has ended: closed, then slot and wake-up.
  4. Admit waiting requests while fewer than in_flight are out: start, define, commit on the last.
  5. Send what this pass admitted: one write per data node.

Replies come first: ended transactions free their coordinator records and their place under in_flight before new ones start, and results go back at once.

iclaustron: crates/ic_apid/src/executor.rs (module note; Runner::run, take_completions, admit_waiting) · doc/rust/architecture.html §3 (The executor's loop) · apid_conn.rs (poll_with_requests)

One wait point · three lists, one lock

One wait point: replies, requests and nudges share the inbox's lock

receive threadsa page per round submittersexecute: a request scan readersbatch back, close inbox · one mutex · one condition pages of replieswake the executor if it sleeps requests · at most queuewake it only if it sleeps willing to take one nudges · module 07never refused; wake it if it sleeps flags: waiting · wants_requests executor threadtakes all three in one locksleeps on the condition take

A thread can sleep on one thing only, so all three share it. A producer appends under the lock, unlocks, then wakes the executor only if it sleeps.

iclaustron: crates/ic_apid/src/thread_conn.rs (module note "Requests", "Nudges"; Inbox, post, submit, nudge, exchange_with_requests) · doc/rust/architecture.html §3 (Why both inputs must share one wait point)

Back-pressure · two bounds

Back-pressure: a bounded request list and a limit on requests out

The inbox: queue, 4 000

  • A full list makes execute wait for room, on a second condition.
  • The executor takes the list whole, in the lock it takes replies in, then wakes those waiting for room.
  • In Rust, submit refuses instead: IC_ERROR_EXECUTOR_BUSY, 7159.

The executor: in_flight, 400

  • At most 400 requests out, each a transaction holding a record at its coordinator.
  • The rest wait in its own queue, in order; meanwhile it takes no more from the inbox.
  • At the limit it sleeps for replies alone: a request does not wake it.

The data nodes allow only so many transactions at once, across every API node: in_flight bounds one executor's share.

iclaustron: thread_conn.rs (submit, exchange_with_requests: room) · executor.rs (ExecutorOptions, Runner::run, admit_waiting, Submitter::submit) · doc/rust/architecture.html §3 (Back-pressure in two places)

Failure · nothing is lost silently

Failures: every request comes back, sent or not, with its error

What happenedThe request comes back with
The executor is stoppedIC_ERROR_EXECUTOR_STOPPED, 7160: refused, left as it was; those waiting fail unsent
Its deadline passed before admissionIC_ERROR_TIMEOUT, 7128, unsent; set with ic_request_set_timeout_ms before each run
An operation could not be definedits refusal; the transaction is rolled back before anything is sent
No data node is started7142, unsent
A node failure ends the transactionthe transaction's error, such as 4010 or 4031: temporary
No reply within the reply wait4012, after six minutes by default: the outcome is unknown

A temporary error: run the same request again, after a pause that grows. After 4012, learn whether a write committed before running it again.

iclaustron: executor.rs (module note; Runner::admit, abandon_defined, refuse_waiting, finish) · ic_apid.h (ic_apid_submitter_execute, ic_request_set_timeout_ms, ic_error_is_temporary) · doc/user-guide.md §9, §10

Measured · Mac, two release data nodes on the same machine, t9 by primary key

Measured: one executor reads 36 % more than a connection per thread

RunPer secondMedian callClient CPU a callRequests a send
64 threads, a connection each, blocking198 000 reads319 µs18.3 µs1
64 callers, one executor268 000 reads231 µs9.8 µs21.0
256 callers, two executors307 000 reads821 µs18.1 µs38.7
64 callers, one executor: updates68 500 updates918 µs29.6 µs4.0
For scale: one thread, pipelined1.99 M reads158 µs a batch0.53 µs200

Your own figure: ic_apid_executor_stats(executor, &stats, sizeof stats), then admitted over sends is requests a send.

iclaustron: doc/rust/06-phases.md §8.1 (2026-10-15; ic_bench, 100 000 keys, 10 s) · 12-performance-model.md (Calibration: M5 Pro) · user-guide.md §5 · ic_apid.h (IC_EXECUTOR_STATS)

The limit · what batching cannot share

What batching cannot share: a sleep and a wake-up for every call

≈10 µs

of CPU a blocking call: the caller's sleep and wake-up, most of it the kernel's

0.5 µs

a read pipelined on one thread, batches of 200 (module 05)

≈270 000

calls a second bound one executor: its own thread wakes every caller

More executors help while there are cores: two with 256 callers gave 15 % more, not twice. One per core for RonDB work, picked by key hash or round robin.

iclaustron: doc/rust/06-phases.md §8.1 (One executor is bound by its own thread; More executors) · architecture.html §3 (How many executors; What the hand-off costs) · user-guide.md §11

Also · and not yet

AUTO_INCREMENT through a submitter, and what executors cannot do yet

AUTO_INCREMENT

ret = ic_apid_submitter_auto_increment(sub, table, &id);

From the table's one range in the process, refilled on the library's counter connection, never through the executor. _block, _seen and _peek too.

Not through an executor yet

  • Taking over rows a scan has locked: on a connection only.
  • A request handed to a function when it ends: Rust only, Submitter::submit with Done::Callback.
  • A transaction kept open across calls: a request runs start to end.
  • Asynchronous C calls and Rust futures over executors: later.

iclaustron: ic_apid.h (ic_apid_submitter_auto_increment and _block, _seen, _peek) · doc/user-guide.md §7, §12 · 04-api-design.md (Executors; Taking scanned rows over; Kept for later) · executor.rs (Done)

Choosing · modules 04, 06 and 05 side by side

Three ways to call: a connection per thread, an executor, asynchronous

Connection per thread · 04Through an executor · 06Asynchronous · 05
Caller writesblocking callsblocking callsdefine, send, poll or callbacks
Threads on RonDBat most 4 711any number, few executorsfew user threads
In flightone per threadmany per executormany per thread
Batchingwithin one caller's transactionacross callers, by itselfchosen by the application
Latency, light loadlowesta hand-off each way morelowest
Client CPU a call18 µs, 64 threads10 µs, 64 callers0.5 µs
Reads a second198 000, 64 threads268 000, one executor2.6 million, one thread, depth 4
Fitstools, tests, low-rate servicesservers with a thread per requestloaders, servers built for it

iclaustron: doc/rust/architecture.html §4 (the table of forms A, B and asynchronous) · 06-phases.md §8.1 (measured, the Mac, 2026-10-15) · 12-performance-model.md (Calibration)

Recap · module 06

Recap: key operations through an executor

  • An executor: one library thread, one connection, any number of callers.
  • A shape is made once; a request is one transaction that owns its rows.
  • ic_apid_submitter_execute blocks until the transaction ends: 0 if committed.
  • Each pass: replies, then admits up to in_flight, then one write per data node.
  • Every request comes back with its error; run a temporary one again.
  • Writes are shared, wake-ups are not: about 10 µs a call (module 05 goes further).

Next · 07 · Scans

iClaustron training for C developers · 8 modules · you are at 06 · this module for Rust developers

Next · 07 · Scans in five models