iClaustron training for Rust developers · module 4 of 9 · 18 min

Key operations, synchronous

One call per row, or many queries and transactions sent together: the two blocking models on a connection the thread owns, what runs where, and what each costs.

Two models · the caller waits for its answer

Synchronous and batched synchronous, on the thread's own connection

Synchronous

  • One call is a transaction of one query, sent and waited for.
  • key_op::read_committed, key_op::read_unique_committed, key_op::write_key.
  • The program's own row slices in and out; no query or transaction to keep.

Batched synchronous

  • Queries made once, defined on one or more transactions, committed.
  • conn.flush until each is done: Transaction::is_done.
  • One send per data node carries all of it.

Both block the thread owning the connection (module 03); module 05 keeps transactions in flight, module 06 uses an executor.

iclaustron: doc/user-guide.md §4 (Transactions), §5 (Synchronously); doc/rust/architecture.html §4 (A · Each thread owns a connection); crates/ic_apid/src/key_op.rs (read_committed, read_unique_committed, write_key); crates/ic_apid/src/transaction.rs (flush, Transaction::is_done)

Synchronous · from examples/rust/key_ops.rs

Synchronous calls: one row by key, in the program's own row

let mut key = t1.row();
put_u32(&t1.rec, &mut key, t1.id, id)?;
let mut row = t1.row();
if key_op::read_committed(conn, &t1.rec, &key, &t1.rec, &mut row)? {
  println!("{}: {}", what, t1.show(&row));
} else {
  println!("{}: no row {}", what, id);
}
// ...
let mut row = t1.row();
t1.fill(&t1.rec, &mut row, values)?;
key_op::write_key(conn, operation, &t1.rec, &row, &t1.rec, &row)
  • No row is no error: the read returns Ok(false).
  • A write's kind: key_ops::IC_OP_INSERT, _UPDATE, _WRITE, _DELETE. Its error is the answer: 626 no row, 630 a row there.
  • By unique key: key_op::read_unique_committed, given the index.
  • Read and write give up after 10 s: IC_ERROR_TIMEOUT.

iclaustron: examples/rust/key_ops.rs (read_and_print, write); crates/ic_apid/src/key_op.rs (read_committed, read_unique_committed, write_key, IC_KEY_OP_WAIT_MS); examples/rust/unique.rs

Synchronous · what key_op::read_committed does

Inside a synchronous call: the transaction is one KEY_OP_REQ

  1. Packs the key and a read of the record's fields; picks a started node holding the key's partition.
  2. Takes its coordinator record on that node: allocated by COORD_ALLOC_REQ the first time, then reused.
  3. Sends one KEY_OP_REQ now, flagged start, execute and commit: a read simple and dirty, a write aborting on error.
  4. Polls the inbox 100 ms at a time until the replies are in; after 10 s, IC_ERROR_TIMEOUT.
  5. Unpacks the row into the program's buffer, gives the record back; a refusal with 626 returns Ok(false).

key_op::read_unique_committed instead runs a query and a transaction of its own, freed before it returns.

iclaustron: crates/ic_apid/src/key_op.rs (read_committed, node_for_key, run_op, wait_for_outcome, read_unique_committed, unique_read); crates/ic_apid/src/apid_conn.rs (coord_record, free_coord_record)

Synchronous · the threads of module 02, one read

A synchronous read on the threads: one request out, two replies back

user threadits connection, inbox receive threadroutes by block data nodescoordinator, rows time → read_committedpacks, writes KEY_OP_REQ asleep in its poll, 100 ms at a time coordinator, row's nodereads the committed row KEY_OP_CONF ← coordinator ROW_DATA_REP ← reading node route by blockpost to its inbox wake-up poll, unpackfound, return

The call packs one KEY_OP_REQ, start, execute and commit set, and writes it to the data node's socket itself.

The coordinator, on a node that holds the row, has the row read. The user thread sleeps in its poll.

Two replies, in either order: KEY_OP_CONF from the coordinator, ROW_DATA_REP from the node that read the row.

The receive thread posts both to the thread's inbox and wakes it; the poll unpacks the row. One round trip, one wake-up.

iclaustron: crates/ic_apid/src/key_op.rs (run_op, wait_for_outcome, take_op_reply); crates/ic_apid/src/apid_global.rs (send_words_for_generation); doc/rust/architecture.html §1 (the threads, A thread is woken only when it sleeps); doc/rust/05-ndb-protocol.md §6.5

Synchronous · the objects of module 03

Objects of a synchronous call: made once, or made by every call

Made once

  • The global; the thread's connection and its inbox.
  • Table and index definitions, and records laying out the program's rows.
  • A coordinator record per data node: allocated by the first call there, a round trip of its own, then reused.

Made by every call

  • A transaction id, a request number, the replies it expects.
  • The request, packed and written; the row, copied out.
  • No query and no transaction object, except in key_op::read_unique_committed, which makes both and frees them.

Little is made per call; every call pays a send, a sleep, a wake-up.

iclaustron: crates/ic_apid/src/key_op.rs (run_op, read_unique_committed); crates/ic_apid/src/apid_conn.rs (coord_record, free_coord_record); doc/user-guide.md §2 (One connection per thread), §3

Synchronous · measured, a connection per thread

Synchronous cost: one round trip per call, 1 / round trip per thread

198 000

reads a second from 64 threads, each with its own connection and one read at a time

319 µs

a call at the median: 64 calls out over 319 µs is the rate, about 3 100 a thread

18.3 µs

client CPU a read: a whole batch's costs, its write, read and wake-up, paid by every call

It grows only with threads, each with its own connection: at most 4 711 in a process.

Measured 2026-10-15 on the Mac, two release data nodes, t9 read by primary key: ic_bench, 64 threads, one committed read per transaction, one at a time.

iclaustron: doc/rust/architecture.html §4 (A · Each thread owns a connection; table: 198 000, 64 threads; 18 µs); doc/rust/12-performance-model.md (Executors: measured 198 000, 319 µs, 18.3 µs); doc/rust/06-phases.md §8.1 (the setup); doc/user-guide.md §2 (4 711)

Batched synchronous · from examples/rust/retry.rs and lib.rs

Batched synchronous in Rust: define, commit, flush until done

loop {
  match conn.query_mut(updater) {
    Some(query) => {
      query.key_row_mut().copy_from_slice(&row);
      query.attr_row_mut().copy_from_slice(&row);
    }
    None => return Err(no_query),
  }
  let trans = conn.start_transaction(hint, Some(&t1.table))?;
  conn.write_key(updater, trans, &update)?;
  conn.commit_transaction(trans)?;
  match committed(&mut conn, trans, updater) { /* ... */ }
  // ...
}
loop {
  let ended = match conn.transaction(trans) {
    Some(t) if t.is_done() => Some((t.commit_state(), t.error())),
    Some(_) => None,
    // ...
  };
  if let Some(ended) = ended {
    conn.close_transaction(trans)?;
    return Ok(ended);
  }
  conn.flush(100, true)?;
}

The query is made once; committed waits in finish, right.

iclaustron: examples/rust/retry.rs (run); examples/rust/lib.rs (finish, committed); crates/ic_apid/src/transaction.rs (flush, Transaction::is_done)

Batched synchronous · from examples/rust/retry.rs and unique.rs

Queries and arguments in Rust: made once, lent by the connection

let updater = conn.create_query(&t1.table, &t1.rec, &t1.rec)?;
// ...
let update = WriteKeyArgs {
  kind: WriteKind::Update,
  ..WriteKeyArgs::default()
};
match conn.query(reader) {
  Some(query) => {
    println!("read by code 700: {}", t1.show(query.attr_row()));
  }
  None => return Err(no_query),
}
  • A query lives in its connection, named by the QueryId that create_query gives; free_query lets it go once idle or completed.
  • query_mut(q) and query(q) lend it, or give None for an id that names none: a borrow of the connection, let go before defining or flushing.
  • Arguments are plain structs: set what differs, ..WriteKeyArgs::default() the rest. Once completed, attr_row() holds the row read, error() why it failed.

iclaustron: examples/rust/retry.rs (run); examples/rust/unique.rs (run); crates/ic_apid/src/transaction.rs (create_query, query, query_mut, free_query); crates/ic_apid/src/query.rs (WriteKeyArgs, ApidQuery)

Batched synchronous · the objects of module 03

Objects in batched synchronous: queries once, a transaction per batch

Made once

  • Records, and the argument structs.
  • Queries, conn.create_query: each owns a key row and an attribute row, sized by its records.

Per transaction

  • conn.start_transaction: an id, and a coordinator record at the hinted node, reused when one is free.
  • Local until its first send: the data node hears of it with its first query.

Per use of a query

  • Idle, defined, sent, completed; then defined again.
  • From define to completion its rows are the library's; then a read's row is in its attribute row.

Warm, nothing is allocated per transaction: coordinator records and query lists are reused.

iclaustron: doc/user-guide.md §4 (Queries, Transactions); doc/rust/architecture.html §2 (Objects and who owns them, Memory rules); crates/ic_apid/src/transaction.rs (start_transaction, release_if_done, close_transaction); crates/ic_apid/src/query.rs (QueryState)

Batched synchronous · pack at define, patch at send

One flush sends many queries and transactions: one write per data node

transaction 1hint: node 1 transaction 2hint: node 1 transaction 3hint: node 2 read_key read_key write_key write_key read_key node 1's buffer4 requests packed node 2's buffer1 request packed start execute, commit start, execute, commit start, execute, commit node 1: one write4 KEY_OP_REQ, 2 transactions node 2: one write1 KEY_OP_REQ

conn.read_key and write_key pack each KEY_OP_REQ at once, into its coordinator node's buffer. Nothing is sent yet.

Commit only asks. At the flush: start on each transaction's first request, execute and commit on its last, and one write per node.

iclaustron: doc/rust/architecture.html §2 (The send path: pack at define, patch at send); crates/ic_apid/src/transaction.rs (define, commit_transaction, send_queries, send_transaction, send_query)

Batched synchronous · the same threads, a whole batch

A batch on the threads: one write per data node, replies in one poll

user threadits connection, inbox receive threadroutes by block data nodescoordinator, rows time → define, commitpacked, not sent flushflags set one write per data node asleep in its poll every operation runsboth nodes at once KEY_OP_CONF per transaction ROW_DATA_REP per row read route by blockone post a round one wake-up pollcompletes them all

Defining sends nothing: each request is packed for its coordinator's node, and the commit waits to ride on the last one.

conn.flush sets the flags and writes once to each data node, from the user thread itself; then it sleeps in the poll.

The data nodes run the batch together: each coordinator its own transactions, each row read where it lies.

Replies come back together, a KEY_OP_CONF per transaction and a ROW_DATA_REP per row, posted to the inbox once a round.

One wake-up, and one poll completes every query that came. The loop flushes again until each transaction is done.

iclaustron: crates/ic_apid/src/transaction.rs (send_queries, flush, take_key_conf, take_row); doc/rust/architecture.html §1 (One lock per inbox per round, The thread that sends does the write), §2 (The send path); doc/rust/05-ndb-protocol.md §6.5

Kinds · ReadKeyArgs and WriteKeyArgs

Read and write kinds: the locks they take and the rows they need

ReadKind::Lock
Simpledefaultshared, let go when the read is done
Lockedshared, held until the transaction ends
Exclusiveexclusive, held until it ends
Committednone: the last committed value
ExclusiveNoWaitexclusive; fails rather than wait
WriteKind::The row
Updatedefaultmust be there, else 626
Insertmust not be there, else 630
Writeinserted, or overwritten
Deletedeleted; 626 if not there

By unique index: simple and committed reads lock too, shared, to the end; no insert or write; no indexed column in an update (7155).

iclaustron: crates/ic_apid/src/query.rs (ReadKind, WriteKind, ReadKeyArgs, WriteKeyArgs); crates/ic_apid/src/transaction.rs (read_key, write_key, define); doc/user-guide.md §4 (Queries)

On error · abort_option and batch_hint

Abort options: whether a failed query ends its transaction

abort_optionA read that failsA write that fails
AbortOption::Defaultfails alonerolls the transaction back
AbortOption::AbortOnErrorrolls it back; refused on a committed readrolls it back
AbortOption::IgnoreErrorfails alonefails alone; the rest go on
  • Failing alone comes back as KEY_OP_REF: ApidQuery::error. A rollback comes as TRANS_ABORTED_REP: Transaction::error, and every query still out fails with it.
  • batch_hint: BatchHint::Safe tells the data node a write does not depend on earlier ones in its batch, BatchHint::Unsafe that it does; none by default.

iclaustron: crates/ic_apid/src/query.rs (AbortOption, BatchHint); crates/ic_apid/src/transaction.rs (read_key, write_key, take_key_ref, take_rollback_rep, end_transaction); doc/rust/05-ndb-protocol.md §6.5

The hint · TransactionHint

Transaction hints: the first round trip goes where the row is

user threadstarts a transaction node 1next in turn node 2holds the row Partition(p) Any a hop
  • TransactionHint::Partition(p), from conn.hint_for_key: a started node holding the key's partition.
  • TransactionHint::Node(id): that node; refused if it is not started, 7136, IC_ERROR_NODE_NOT_READY.
  • TransactionHint::Any: the next started node in turn.
  • By unique key: conn.hint_for_unique_key, then start_transaction given Some(index.as_table()).

The key's hint, hashed as the data nodes hash it, starts the transaction on a node holding the row: its first round trip stays there.

With TransactionHint::Any the next node in turn forwards the operation to the row: a hop, never a wrong answer. A hint only picks the coordinator.

iclaustron: crates/ic_apid/src/transaction.rs (TransactionHint, hint_for_key, hint_for_unique_key, start_transaction, choose_start_node); crates/ic_apid/src/hash.rs (choose_node); doc/user-guide.md §4 (The hint); doc/rust/05-ndb-protocol.md §6.3

Outcome · CommitState

Transaction outcome: done once ended and every query complete

Started CommitRequested Committed RollbackRequested RolledBack commit, sent KEY_OP_CONF, COMMIT_CONF TRANS_ABORTED_REP rollback, sent ROLLBACK_CONF Transaction::gci: its checkpoint Transaction::error: why
  • Done is ended, and every query sent complete: Transaction::is_done. Then conn.close_transaction lets it go.
  • A commit with no query left to carry it goes as COMMIT_REQ; one on a transaction never sent ends at once, here.

iclaustron: crates/ic_apid/src/transaction.rs (CommitState, Transaction::is_done, gci, error, take_key_conf, take_commit_conf, take_rollback_rep, send_end, close_transaction)

Errors · from examples/rust/retry.rs

Temporary errors end the transaction: run all of it again

loop {
  // ...
  let trans = conn.start_transaction(hint, Some(&t1.table))?;
  conn.write_key(updater, trans, &update)?;
  conn.commit_transaction(trans)?;
  match committed(&mut conn, trans, updater) {
    Ok(()) => {
      println!("attempt {}: committed", attempt);
      break;
    }
    Err(e) if e.is_temporary() && attempt < ATTEMPTS => { /* ... */ }
    Err(e) => return Err(e),
  }
  // ...
}
  • A lock wait that timed out (266), a node failure (4010), an overload: temporary, as IcError::is_temporary says.
  • Start a new transaction and define every query again, after a pause that grows.
  • 4012: the replies stopped, and it may have committed. Read the row first, or retry only a write that does no harm twice.

iclaustron: examples/rust/retry.rs (run); doc/user-guide.md §9; crates/ic_port/src/err.rs (IcError::is_temporary), ndb_err.rs (266, 4010, 4012)

Compared · the two models side by side

Synchronous or batched synchronous: costs and when to use each

SynchronousBatched synchronous
Round tripsone per rowone per flush, for every transaction in it
Sendsone per callone write per data node per flush
Client CPU a read18.3 µs, 64 threadsabout 545 ns, one thread, batches of 200
Reads a second198 000 with 64 threads1.17 – 1.35 million on one thread
Latencyone round trip: the lowest for one rowthe batch's slowest operation
Fitstools, tests, low-rate servicesrows known together: many-row transactions, loaders

Measured on the Mac, two release data nodes, t9 read by primary key: synchronous 2026-10-15, a connection per thread, one read at a time; batched 2026-09-24, one thread, batches of 200 one-read transactions, each batch waited for.

iclaustron: doc/rust/architecture.html §1 (What it does, measured), §4 (table); doc/rust/12-performance-model.md (Cost per operation; Calibration: 1 thread, depth 1, batch 200; Executors); doc/user-guide.md §11 (Batch)

Recap · module 04

Synchronous and batched synchronous: what to take away

  • A synchronous call is a whole transaction: one KEY_OP_REQ, waited for.
  • It costs a round trip and a wake-up: throughput grows only with threads.
  • Batched synchronous: define on transactions, commit, flush until done.
  • Packed at define, flagged at send: one write per data node, one poll.
  • Hint by the key; kinds set the locks, abort options what a failure ends.
  • Retry a temporary error with the whole transaction; check after a 4012.

Next · 05 · Key operations, asynchronous

iClaustron training for Rust developers · 9 modules · you are at 04 · this module for C developers

Next · 05 · Asynchronous