ADR 0057: writes take a WriteTx from an IMMEDIATE transaction

Records the decision, why the deferred read-then-write failed with
SQLITE_BUSY_SNAPSHOT however long BUSY_TIMEOUT was, and the two alternatives
set aside. ADR 0019 is superseded in part; TODO.md keeps only what's left
(retiring the per-screen retries, the unexplained 5 s stalls, and the
backstop's reach outside the lib's own tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Greg ShuflinandClaude Opus 5.5 committed 2026-11-04 03:41:00 -08:00
1 parent eb2a65cde0
commit 751802b2d7
4 files changed
+111 -55

No files matched your search

+19 -55
View File
@@ -567,61 +567,25 @@ How it works: `rust/lib/src/stores/lock.rs` and
`Endpoint::close`" ERROR, and peers learn the connection is gone by timeout rather than
by a CONNECTION_CLOSE.
### Find out why writes wait on, or fail with, a busy database
Writes stall or fail with `database is busy` now and then, in several parts of the app, and
it doesn't reproduce on demand. Find the cause in how `stores/` uses SQLite rather than
patching each caller, and settle on one answer for every modality: gpui, TUI/CLI through the
daemon, and Android. What was found on 2026-11-03, when one case turned up:
- **Observed: two overlapping writes lose one at once.** The podcast speed dialog's Set used
to issue two writes as concurrent background tasks: the feed's speed and the custom-speed
add. On the first try in the headless gpui session, the log said `Failed to save a podcast
speed: database is busy, try again`. The store held the feed's `podcast_feed_speed` and no
`podcasts_custom_speeds` element. Android's dialog had the same race through two
coroutines. `speed::confirm_speed` now runs the two one after the other, which fixes that
case and nothing else. To reproduce the pattern, run any two `batch`es that each read and
then write from separate threads. The scratch-store and local-feed recipe in
`rust/gui-app/AGENTS.md` is how the podcast case was driven.
- **Why that happens (confirmed 2026-11-03).** `batch_on` opens a deferred transaction
(`conn.transaction()`). Its first read takes a WAL snapshot, and its first write has to
upgrade that to the write lock. If another connection has committed since the snapshot,
the upgrade fails with `SQLITE_BUSY_SNAPSHOT` (extended code 517) in about a millisecond:
SQLite skips the busy handler, since waiting can't help. So any two writers whose
transactions overlap can lose one: a speed pick against a listen-progress write, a note
save against a sync chunk (`doc_sync`'s `apply_records` reads before it writes too), and
so on. Nearly every write reads first, since `set_property` compares against the stored
value, and Notes' `update_note` merges against the stored body. Two threads each running
300 read-then-write batches lost 45-50% of them on the pool, and none at all with
`BEGIN IMMEDIATE`, at the same throughput and with no wait over a second.
- **Ruled out for the current code: nested batches.** A `batch` opened inside another's
closure that writes while the outer holds the write lock waits out `BUSY_TIMEOUT` and fails
with plain `SQLITE_BUSY` (code 5) after 5.0 s, every time. None was found in production
code. A thread-local depth check in `batch_on` and `SettingsStore::batch`, run across all
1921 workspace tests, saw only a test's own nested read. A static scan of every `.batch(`
closure in the workspace, gui-app included, for calls not handed `tx` that reach a
batch-opening function, or that block on another thread, found nothing either. So a 5 s
stall, if it recurs, means a writer that held the lock for 5 s, and `stores::contention`
now logs any write transaction held past a second with the call site that opened it, along
with every busy failure's extended code. Nesting is still worth guarding against
permanently, since a write nested in a write always fails: a debug assertion on the same
depth check would catch it in tests.
- **Ruled out as the cause: `stores::pool`.** It doesn't change the mechanism. A fresh
connection per transaction lost 13-38% in the same test. The pool's faster transactions
overlap more under saturation, but the race was there before it.
- **Fix (decided): a `WriteTx` that only an `IMMEDIATE` entry point hands out.** Making
every batch `IMMEDIATE` would serialize reads with each other and with sync chunks, which
WAL exists to avoid. Instead, `WriteTx` wraps a `Transaction` and derefs to it. Store
methods that write take `&WriteTx`, about 93 of them plus about 23 helpers in agents and
p2p. Each store's `batch` gets a `write` sibling, and the roughly 170 non-test call sites
that write move to it, along with the writing ones in tests. `SettingsStore::batch` turns
deferred like the rest, and its setters move to `write`. The type can't stop a method
declared `&Transaction` from running an `INSERT`, so keep a runtime backstop too: a
deferred `batch` that ends in `TransactionState::Write` fails a debug assertion.
- **Then retire the retries.** The workarounds today are the desktop's
`save_body_with_retry` in Notes and Android's `retryOnBusy`. The latter retries *any*
failure, twice, two seconds apart. Its doc says it can't tell a busy database from other
failures, which is stale: `SyncError::Busy` carries `Recourse::RetryNow`. Once writes are
`IMMEDIATE` they queue instead of failing at once, and a busy that still gets through has
already waited 5 s, so neither helper has anything left to do.
### Busy database: what's left after `WriteTx`
Writes used to fail with `database is busy` because a deferred transaction that read and then
wrote lost its snapshot to any commit in between (`SQLITE_BUSY_SNAPSHOT`, at once, whatever
`BUSY_TIMEOUT` says). Writes now go through `write` and a `WriteTx`, which begin `IMMEDIATE`
and queue for the lock instead (ADR 0057). `stores::contention` logs every busy failure with
its extended code and call site, and any write transaction held past a second.
- **Retire the retries.** The desktop's `save_body_with_retry` in Notes and Android's
`retryOnBusy` were workarounds for the snapshot failure, which can't happen any more. A
busy that still gets through has already waited `BUSY_TIMEOUT` (5 s), so retrying it two
seconds later is not the fix either. `retryOnBusy` also retries *any* failure, and its doc
says it can't tell a busy database from other failures, which is stale:
`SyncError::Busy` carries `Recourse::RetryNow`.
- **The ~5 s stalls are unexplained.** Nested batches were ruled out on 2026-11-03 (and are
now caught by `contention`'s nesting check), so a stall means some writer held the lock
for 5 s. If one recurs, `contention`'s "held the write lock for …" warning names it.
- **The backstop only covers tested paths.** A `batch` that writes, or a `write` nested in a
`write`, panics in the lib's own tests and is only a warning elsewhere, including in
gui-app's and the daemon's tests, where the lib isn't built with `cfg(test)`. Grep a
session's log for "wrote inside a read transaction" after exercising a screen.
### Store I/O costs
Measured by `just bench-store-io` (`lib/src/store_io_bench.rs`).
@@ -2,6 +2,10 @@
**Status**: Accepted
**Date**: 2026-08-05
**Superseded in part by**: [ADR 0057](0057-writes-take-writetx-from-an-immediate-transaction.md) —
store methods still take a transaction and `batch` is still how a read gets one,
but a method that writes takes a `WriteTx` from `write`, which begins
`IMMEDIATE`, and `SettingsStore::batch` is no longer the exception.
## Context
@@ -0,0 +1,87 @@
# 0057: Writes take a `WriteTx`, which only an `IMMEDIATE` transaction hands out
**Status:** Accepted
**Date:** 2026-11-04
## Context
Writes stalled or failed with "database is busy" now and then, on every
modality, and never on demand. ADR 0019 had made `batch` the one way to open a
transaction, and every `batch` began `DEFERRED`: it takes a WAL snapshot at its
first read and needs the write lock only at its first write. Nearly every
write reads first (`set_property` compares against the stored value, a note
save merges against the stored body, sync's `apply_records` checks tombstones),
so nearly every write transaction had to upgrade a read snapshot to the write
lock part-way through.
That upgrade can't succeed if another connection committed after the
snapshot was taken. SQLite refuses it at once with `SQLITE_BUSY_SNAPSHOT`
(extended code 517) and doesn't consult the busy handler, since waiting can't
make an old snapshot current. `BUSY_TIMEOUT` therefore protected none of these
writers. Measured on 2026-11-03, two threads each running 300 read-then-write
transactions lost 45-50% of them through the connection pool and 13-38% with
a fresh connection per transaction (the pool wasn't the cause). With `BEGIN
IMMEDIATE` they lost none, at the same throughput, and no wait exceeded a
second.
Two alternatives were set aside:
- **Make every transaction `IMMEDIATE`.** Simple, but it serializes reads
with each other and with every sync chunk, which is what WAL exists to
avoid. `SettingsStore::batch` already did this, to keep `update_settings`'
read-modify-write from reverting a concurrent setter's fields.
- **Keep one `batch` and check at runtime.** A deferred transaction that ends
in `TransactionState::Write` is a writer on the wrong path, and that is
detectable after the fact. On its own, though, it only catches the paths
tests exercise, and it leaves the choice to each call site's author.
## Decision
- **Two entry points.** `batch` stays `DEFERRED` and hands its closure a
`&Transaction`. `write` (`write_on`, `Store::write`, and each store's own
`write`) begins `IMMEDIATE` and hands a `&WriteTx`.
- **A method that writes takes `&WriteTx`.** `WriteTx` wraps the transaction,
derefs to it, and can only be constructed inside `stores`, so a writing
method called from a `batch` closure is a type error. A read method keeps
`&Transaction` and accepts a `&WriteTx` through deref coercion, so reads
inside a `write` need no change. This holds for helpers outside `stores`
that write through a transaction, and for the two `&Connection` helpers that
write (`touch_document_in_tx`, `insert_edge_in_tx`).
- **A transaction that might write is a `write`**, even if it usually doesn't
(`ensure_document`, Humans' `touching`). Reading first in a `batch` and
writing in a second transaction would reintroduce the race.
- **`SettingsStore::batch` is deferred like every other store's.** Its setters
take `&WriteTx`, which gives `update_settings` the serialization it needs.
This removes ADR 0019's one exception.
- **A runtime backstop for what the type can't see.** rusqlite lets any
`&Transaction` run an `INSERT`, so `stores::contention` checks two things
after each transaction. A `batch` that ended holding the write lock is one.
A `write` opened while the same thread already holds one is the other: it
waits out `BUSY_TIMEOUT` for a lock its own caller holds. Both panic in the
lib's tests and log a warning elsewhere. On its first run the backstop found
four writers the type change had missed, all writing through a `&Connection`
helper.
- **Busy failures are logged with their extended code and call site.** Every
`batch`/`write` wrapper is `#[track_caller]`. A write transaction held past
a second is logged the same way, since that is what makes another writer's
`BUSY_TIMEOUT` run out.
## Consequences
- Two writers that overlap now queue for the lock instead of one failing.
`SQLITE_BUSY_SNAPSHOT` can't reach a store writer any more. A busy that
still happens has waited the full `BUSY_TIMEOUT`, and the log names the
transaction that held the lock.
- A write takes the lock for its whole transaction, including the reads before
its first write. A long read-then-write blocks other writers for longer than
it used to, though under the old scheme it would have failed outright. Reads
in a `batch` never wait.
- Call sites state their intent: `batch` reads, `write` writes. A writer on
the read path fails to compile, or, if the method hides its write behind
`&Transaction`, fails the lib's tests.
- The backstop only panics in the lib's own tests. gui-app's and the daemon's
tests build the lib without `cfg(test)`, so a mistake reached only from
there shows up as a warning in the log.
- The per-screen retries that worked around the snapshot failure (desktop
Notes' `save_body_with_retry`, Android's `retryOnBusy`) have nothing left to
retry; removing them is in `TODO.md`.
+1
View File
@@ -86,3 +86,4 @@ transition alone.
| [0054](0054-comic-reimport-identity.md) | Re-importing a comic updates it in place — matched by source name, then by page set | Accepted |
| [0055](0055-android-screens-assembled-from-roles.md) | Android screens are assembled from roles, not from Material components | Accepted |
| [0056](0056-keep-synchronicity-lib-one-crate.md) | Keep `synchronicity-lib` one crate; give hot loops their own optimized crate | Accepted |
| [0057](0057-writes-take-writetx-from-an-immediate-transaction.md) | Writes take a `WriteTx`, which only an `IMMEDIATE` transaction hands out | Accepted |