iClaustron training for C developers · module 4 of 8 · 18 min

Key operations, synchronous

One call per row, or many queries and transactions sent together: the two blocking models on a connection the thread owns, what runs where, and what each costs.

Two models · the caller waits for its answer

Synchronous and batched synchronous, on the thread's own connection

Synchronous

  • One call is a transaction of one query, sent and waited for.
  • ic_apid_conn_read_committed, ic_apid_conn_read_unique_committed, ic_apid_conn_write.
  • The program's own struct in and out; no query or transaction to keep.

Batched synchronous

  • Queries made once, defined on one or more transactions, committed.
  • ic_apid_conn_flush until each is done: ic_apid_transaction_is_done.
  • One send per data node carries all of it.

Both block the thread owning the connection (module 03); module 05 keeps transactions in flight, module 06 uses an executor.

iclaustron: doc/user-guide.md §4 (Transactions), §5 (Synchronously); doc/rust/architecture.html §4 (A · Each thread owns a connection); include/iclaustron/ic_apid.h (ic_apid_conn_read_committed, ic_apid_conn_flush, ic_apid_transaction_is_done)

Synchronous · from examples/c/key_ops.c

Synchronous calls: one row by key, in the program's struct

ex_fill(&key, id, 0, 0, NULL);
EX_CHECK(ic_apid_conn_read_committed(conn, rec, (const uint8_t *)&key,
                                     sizeof key, rec, (uint8_t *)&row,
                                     sizeof row, &found));
if (found)
  ex_print(what, &row);
else
  printf("%s: no row %u\n", what, (unsigned)id);
/* ... */
return ic_apid_conn_write(conn, kind, rec, (const uint8_t *)row,
                          sizeof *row, rec, (const uint8_t *)row,
                          sizeof *row);
  • No row is no error: found is false.
  • A write's kind: IC_WRITE_KIND_INSERT, _UPDATE, _WRITE, _DELETE. Its error is the answer: 626 no row, 630 a row there.
  • By unique key: ic_apid_conn_read_unique_committed, given the index.
  • Read and write give up after 10 s: IC_ERROR_TIMEOUT.

iclaustron: examples/c/key_ops.c (read_and_print, write_row); include/iclaustron/ic_apid.h (ic_apid_conn_read_committed, ic_apid_conn_read_unique_committed, ic_apid_conn_write); examples/c/unique.c

Synchronous · what ic_apid_conn_read_committed does

Inside a synchronous call: the transaction is one KEY_OP_REQ

  1. Packs the key and a read of the record's fields; picks a started node holding the key's partition.
  2. Takes its coordinator record on that node: allocated by COORD_ALLOC_REQ the first time, then reused.
  3. Sends one KEY_OP_REQ now, flagged start, execute and commit: a read simple and dirty, a write aborting on error.
  4. Polls the inbox 100 ms at a time until the replies are in; after 10 s, IC_ERROR_TIMEOUT.
  5. Unpacks the row into the program's buffer, gives the record back; a refusal with 626 sets found to false.

ic_apid_conn_read_unique_committed instead runs a query and a transaction of its own, freed before it returns.

iclaustron: crates/ic_apid/src/key_op.rs (read_committed, node_for_key, run_op, wait_for_outcome, read_unique_committed); crates/ic_apid/src/apid_conn.rs (coord_record); crates/ic_capi/src/lib.rs (ic_apid_conn_read_committed)

Synchronous · the threads of module 02, one read

A synchronous read on the threads: one request out, two replies back

user threadits connection, inbox receive threadroutes by block data nodescoordinator, rows time → read_committedpacks, writes KEY_OP_REQ asleep in its poll, 100 ms at a time coordinator, row's nodereads the committed row KEY_OP_CONF ← coordinator ROW_DATA_REP ← reading node route by blockpost to its inbox wake-up poll, unpackfound, return

The call packs one KEY_OP_REQ, start, execute and commit set, and writes it to the data node's socket itself.

The coordinator, on a node that holds the row, has the row read. The user thread sleeps in its poll.

Two replies, in either order: KEY_OP_CONF from the coordinator, ROW_DATA_REP from the node that read the row.

The receive thread posts both to the thread's inbox and wakes it; the poll unpacks the row. One round trip, one wake-up.

iclaustron: crates/ic_apid/src/key_op.rs (run_op, wait_for_outcome, take_op_reply); crates/ic_apid/src/apid_global.rs (send_words_for_generation); doc/rust/architecture.html §1 (the threads, A thread is woken only when it sleeps); doc/rust/05-ndb-protocol.md §6.5

Synchronous · the objects of module 03

Objects of a synchronous call: made once, or made by every call

Made once

  • The global; the thread's connection and its inbox.
  • Table and index definitions, and records over the program's struct.
  • A coordinator record per data node: allocated by the first call there, a round trip of its own, then reused.

Made by every call

  • A transaction id, a request number, the replies it expects.
  • The request, packed and written; the row, copied out.
  • No query and no transaction object, except in ic_apid_conn_read_unique_committed, which makes both and frees them.

Little is made per call; every call pays a send, a sleep, a wake-up.

iclaustron: crates/ic_apid/src/key_op.rs (run_op, read_unique_committed); crates/ic_apid/src/apid_conn.rs (coord_record, free_coord_record); doc/user-guide.md §2 (One connection per thread), §3

Synchronous · measured, a connection per thread

Synchronous cost: one round trip per call, 1 / round trip per thread

198 000

reads a second from 64 threads, each with its own connection and one read at a time

319 µs

a call at the median: 64 calls out over 319 µs is the rate, about 3 100 a thread

18.3 µs

client CPU a read: a whole batch's costs, its write, read and wake-up, paid by every call

It grows only with threads, each with its own connection: at most 4 711 in a process.

Measured 2026-10-15 on the Mac, two release data nodes, t9 read by primary key: ic_bench, 64 threads, one committed read per transaction, one at a time.

iclaustron: doc/rust/architecture.html §4 (A · Each thread owns a connection; table: 198 000, 64 threads; 18 µs); doc/rust/12-performance-model.md (Executors: measured 198 000, 319 µs, 18.3 µs); doc/rust/06-phases.md §8.1 (the setup); doc/user-guide.md §2 (4 711)

Batched synchronous · from examples/c/retry.c

Batched synchronous in C: define, commit, flush until done

EX_CHECK(ic_apid_query_create(conn, table, rec, rec, &updater));
ex_fill(&row, 9, 91, 900, "nine, again");
ic_write_key_args_init(&write_args, sizeof write_args);
write_args.kind = IC_WRITE_KIND_UPDATE;
for (attempt = 1; attempt <= ATTEMPTS; attempt++) {
  rows = ic_apid_query_key_row(conn, updater, &len);
  memcpy(rows, &row, sizeof row);
  rows = ic_apid_query_attr_row(conn, updater, &len);
  memcpy(rows, &row, sizeof row);
  EX_CHECK(ic_apid_conn_start_transaction(conn, hint, table, &trans));
  EX_CHECK(ic_apid_conn_write_key(conn, updater, trans, &write_args));
  EX_CHECK(ic_apid_conn_commit_transaction(conn, trans));
  ret = wait_for(conn, trans, updater);
  /* ... */
}
static int32_t wait_for(IC_APID_CONNECTION *conn, IC_TRANS_ID trans,
                        IC_QUERY_ID query)
{
  /* ... */
  while (!ic_apid_transaction_is_done(conn, trans))
    EX_CHECK(ic_apid_conn_flush(conn, 100, 1, NULL));
  ret = ic_apid_transaction_error(conn, trans);
  if (ret == 0)
    ret = ic_apid_query_error(conn, query);
  /* ... */
  EX_CHECK(ic_apid_conn_close_transaction(conn, trans));
  return ret;
}

The query is made once and defined again on every transaction.

iclaustron: examples/c/retry.c (main, wait_for); examples/c/unique.c (reading the attribute row); include/iclaustron/ic_apid.h (ic_apid_conn_flush, ic_apid_transaction_is_done)

Batched synchronous · the objects of module 03

Objects in batched synchronous: queries once, a transaction per batch

Made once

  • Records, and the argument structs.
  • Queries, ic_apid_query_create: each owns a key row and an attribute row, sized by its records.

Per transaction

  • ic_apid_conn_start_transaction: an id, and a coordinator record at the hinted node, reused when one is free.
  • Local until its first send: the data node hears of it with its first query.

Per use of a query

  • Idle, defined, sent, completed; then defined again.
  • From define to completion its rows are the library's; then a read's row is in its attribute row.

Warm, nothing is allocated per transaction: coordinator records and query lists are reused.

iclaustron: doc/user-guide.md §4 (Queries, Transactions); doc/rust/architecture.html §2 (Objects and who owns them, Memory rules); crates/ic_apid/src/transaction.rs (start_transaction, release_if_done, close_transaction); include/iclaustron/ic_apid.h (IC_QUERY_*)

Batched synchronous · pack at define, patch at send

One flush sends many queries and transactions: one write per data node

transaction 1hint: node 1 transaction 2hint: node 1 transaction 3hint: node 2 read_key read_key write_key write_key read_key node 1's buffer4 requests packed node 2's buffer1 request packed start execute, commit start, execute, commit start, execute, commit node 1: one write4 KEY_OP_REQ, 2 transactions node 2: one write1 KEY_OP_REQ

ic_apid_conn_read_key and _write_key pack each KEY_OP_REQ at once, into its coordinator node's buffer. Nothing is sent yet.

Commit only asks. At the flush: start on each transaction's first request, execute and commit on its last, and one write per node.

iclaustron: doc/rust/architecture.html §2 (The send path: pack at define, patch at send); crates/ic_apid/src/transaction.rs (define, commit_transaction, send_transaction, send_query); include/iclaustron/ic_apid.h (ic_apid_conn_send)

Batched synchronous · the same threads, a whole batch

A batch on the threads: one write per data node, replies in one poll

user threadits connection, inbox receive threadroutes by block data nodescoordinator, rows time → define, commitpacked, not sent flushflags set one write per data node asleep in its poll every operation runsboth nodes at once KEY_OP_CONF per transaction ROW_DATA_REP per row read route by blockone post a round one wake-up pollcompletes them all

Defining sends nothing: each request is packed for its coordinator's node, and the commit waits to ride on the last one.

ic_apid_conn_flush sets the flags and writes once to each data node, from the user thread itself; then it sleeps in the poll.

The data nodes run the batch together: each coordinator its own transactions, each row read where it lies.

Replies come back together, a KEY_OP_CONF per transaction and a ROW_DATA_REP per row, posted to the inbox once a round.

One wake-up, and one poll completes every query that came. The loop flushes again until each transaction is done.

iclaustron: crates/ic_apid/src/transaction.rs (send_queries, flush, take_key_conf, take_row); doc/rust/architecture.html §1 (One lock per inbox per round, The thread that sends does the write), §2 (The send path); doc/rust/05-ndb-protocol.md §6.5

Kinds · IC_READ_KEY_ARGS and IC_WRITE_KEY_ARGS

Read and write kinds: the locks they take and the rows they need

IC_READ_KIND_Lock
SIMPLEdefaultshared, let go when the read is done
LOCKEDshared, held until the transaction ends
EXCLUSIVEexclusive, held until it ends
COMMITTEDnone: the last committed value
EXCLUSIVE_NO_WAITexclusive; fails rather than wait
IC_WRITE_KIND_The row
UPDATEdefaultmust be there, else 626
INSERTmust not be there, else 630
WRITEinserted, or overwritten
DELETEdeleted; 626 if not there

By unique index: simple and committed reads lock too, shared, to the end; no insert or write; no indexed column in an update (7155).

iclaustron: include/iclaustron/ic_apid.h (IC_READ_KIND_*, IC_WRITE_KIND_*, ic_read_key_args_init, ic_write_key_args_init, ic_apid_conn_write_key); crates/ic_apid/src/transaction.rs (read_key, define); doc/user-guide.md §4 (Queries)

On error · abort_option and batch_hint

Abort options: whether a failed query ends its transaction

abort_optionA read that failsA write that fails
IC_ABORT_DEFAULTfails alonerolls the transaction back
IC_ABORT_ON_ERRORrolls it back; refused on a committed readrolls it back
IC_IGNORE_ERRORfails alonefails alone; the rest go on
  • Failing alone comes back as KEY_OP_REF: ic_apid_query_error. A rollback comes as TRANS_ABORTED_REP: ic_apid_transaction_error, and every query still out fails with it.
  • batch_hint: IC_BATCH_HINT_SAFE tells the data node a write does not depend on earlier ones in its batch, IC_BATCH_HINT_UNSAFE that it does; none by default.

iclaustron: include/iclaustron/ic_apid.h (IC_ABORT_*, IC_BATCH_HINT_*); crates/ic_apid/src/transaction.rs (read_key, write_key, take_key_ref, take_rollback_rep, end_transaction); doc/rust/05-ndb-protocol.md §6.5

The hint · IC_TRANSACTION_HINT

Transaction hints: the first round trip goes where the row is

user threadstarts a transaction node 1next in turn node 2holds the row IC_HINT_PARTITION IC_HINT_ANY a hop
  • IC_HINT_PARTITION, from ic_apid_conn_hint_for_key: a started node holding the key's partition.
  • IC_HINT_NODE: that node; refused if it is not started, 7136, IC_ERROR_NODE_NOT_READY.
  • IC_HINT_ANY: the next started node in turn.
  • By unique key: ic_apid_conn_hint_for_unique_key, then ic_apid_conn_start_transaction_on_index.

The key's hint, hashed as the data nodes hash it, starts the transaction on a node holding the row: its first round trip stays there.

With IC_HINT_ANY the next node in turn forwards the operation to the row: a hop, never a wrong answer. A hint only picks the coordinator.

iclaustron: crates/ic_apid/src/transaction.rs (hint_for_key, choose_start_node); crates/ic_apid/src/hash.rs (choose_node); include/iclaustron/ic_apid.h (IC_HINT_*, ic_apid_conn_hint_for_key); doc/user-guide.md §4 (The hint); doc/rust/05-ndb-protocol.md §6.3

Outcome · IC_COMMIT_STATE_*

Transaction outcome: done once ended and every query complete

STARTED COMMIT_REQUESTED COMMITTED ROLLBACK_REQUESTED ROLLED_BACK commit, sent KEY_OP_CONF, COMMIT_CONF TRANS_ABORTED_REP rollback, sent ROLLBACK_CONF ic_apid_transaction_gci: its checkpoint ic_apid_transaction_error: why
  • Done is ended, and every query sent complete: ic_apid_transaction_is_done. Then ic_apid_conn_close_transaction lets it go.
  • A commit with no query left to carry it goes as COMMIT_REQ; one on a transaction never sent ends at once, here.

iclaustron: crates/ic_apid/src/transaction.rs (CommitState, Transaction::is_done, take_key_conf, take_commit_conf, take_rollback_rep, send_end); include/iclaustron/ic_apid.h (IC_COMMIT_STATE_*, ic_apid_transaction_gci, ic_apid_transaction_error)

Errors · from examples/c/retry.c

Temporary errors end the transaction: run all of it again

for (attempt = 1; attempt <= ATTEMPTS; attempt++) {
  /* ... */
  EX_CHECK(ic_apid_conn_start_transaction(conn, hint, table, &trans));
  EX_CHECK(ic_apid_conn_write_key(conn, updater, trans, &write_args));
  EX_CHECK(ic_apid_conn_commit_transaction(conn, trans));
  ret = wait_for(conn, trans, updater);
  if (ret == 0) {
    printf("attempt %d: committed\n", attempt);
    break;
  }
  if (!ic_error_is_temporary(ret))
    ex_fail("the update", ret);
  /* ... */
}
  • A lock wait that timed out (266), a node failure (4010), an overload: temporary, as ic_error_is_temporary says.
  • Start a new transaction and define every query again, after a pause that grows.
  • 4012: the replies stopped, and it may have committed. Read the row first, or retry only a write that does no harm twice.

iclaustron: examples/c/retry.c (main, wait_for); doc/user-guide.md §9 (Errors to know, When to try again); crates/ic_capi/src/lib.rs (ic_error_is_temporary); include/iclaustron/ic_apid.h (IC_NDB_ERROR_LOCK_TIMEOUT, IC_NDB_ERROR_NO_REPLY)

Compared · the two models side by side

Synchronous or batched synchronous: costs and when to use each

SynchronousBatched synchronous
Round tripsone per rowone per flush, for every transaction in it
Sendsone per callone write per data node per flush
Client CPU a read18.3 µs, 64 threadsabout 545 ns, one thread, batches of 200
Reads a second198 000 with 64 threads1.17 – 1.35 million on one thread
Latencyone round trip: the lowest for one rowthe batch's slowest operation
Fitstools, tests, low-rate servicesrows known together: many-row transactions, loaders

Measured on the Mac, two release data nodes, t9 read by primary key: synchronous 2026-10-15, a connection per thread, one read at a time; batched 2026-09-24, one thread, batches of 200 one-read transactions, each batch waited for.

iclaustron: doc/rust/architecture.html §1 (What it does, measured), §4 (table); doc/rust/12-performance-model.md (Cost per operation; Calibration: 1 thread, depth 1, batch 200; Executors); doc/user-guide.md §11 (Batch)

Recap · module 04

Synchronous and batched synchronous: what to take away

  • A synchronous call is a whole transaction: one KEY_OP_REQ, waited for.
  • It costs a round trip and a wake-up: throughput grows only with threads.
  • Batched synchronous: define on transactions, commit, flush until done.
  • Packed at define, flagged at send: one write per data node, one poll.
  • Hint by the key; kinds set the locks, abort options what a failure ends.
  • Retry a temporary error with the whole transaction; check after a 4012.

Next · 05 · Key operations, asynchronous

iClaustron training for C developers · 8 modules · you are at 04 · this module for Rust developers

Next · 05 · Asynchronous