ADR 0057: writes take a WriteTx from an IMMEDIATE transaction
Records the decision, why the deferred read-then-write failed with SQLITE_BUSY_SNAPSHOT however long BUSY_TIMEOUT was, and the two alternatives set aside. ADR 0019 is superseded in part; TODO.md keeps only what's left (retiring the per-screen retries, the unexplained 5 s stalls, and the backstop's reach outside the lib's own tests). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
eb2a65cde0
commit
751802b2d7
4 files changed
+111
-55
No files matched your search
@@ -567,61 +567,25 @@ How it works: `rust/lib/src/stores/lock.rs` and
|
||||
`Endpoint::close`" ERROR, and peers learn the connection is gone by timeout rather than
|
||||
by a CONNECTION_CLOSE.
|
||||
|
||||
### Find out why writes wait on, or fail with, a busy database
|
||||
Writes stall or fail with `database is busy` now and then, in several parts of the app, and
|
||||
it doesn't reproduce on demand. Find the cause in how `stores/` uses SQLite rather than
|
||||
patching each caller, and settle on one answer for every modality: gpui, TUI/CLI through the
|
||||
daemon, and Android. What was found on 2026-11-03, when one case turned up:
|
||||
- **Observed: two overlapping writes lose one at once.** The podcast speed dialog's Set used
|
||||
to issue two writes as concurrent background tasks: the feed's speed and the custom-speed
|
||||
add. On the first try in the headless gpui session, the log said `Failed to save a podcast
|
||||
speed: database is busy, try again`. The store held the feed's `podcast_feed_speed` and no
|
||||
`podcasts_custom_speeds` element. Android's dialog had the same race through two
|
||||
coroutines. `speed::confirm_speed` now runs the two one after the other, which fixes that
|
||||
case and nothing else. To reproduce the pattern, run any two `batch`es that each read and
|
||||
then write from separate threads. The scratch-store and local-feed recipe in
|
||||
`rust/gui-app/AGENTS.md` is how the podcast case was driven.
|
||||
- **Why that happens (confirmed 2026-11-03).** `batch_on` opens a deferred transaction
|
||||
(`conn.transaction()`). Its first read takes a WAL snapshot, and its first write has to
|
||||
upgrade that to the write lock. If another connection has committed since the snapshot,
|
||||
the upgrade fails with `SQLITE_BUSY_SNAPSHOT` (extended code 517) in about a millisecond:
|
||||
SQLite skips the busy handler, since waiting can't help. So any two writers whose
|
||||
transactions overlap can lose one: a speed pick against a listen-progress write, a note
|
||||
save against a sync chunk (`doc_sync`'s `apply_records` reads before it writes too), and
|
||||
so on. Nearly every write reads first, since `set_property` compares against the stored
|
||||
value, and Notes' `update_note` merges against the stored body. Two threads each running
|
||||
300 read-then-write batches lost 45-50% of them on the pool, and none at all with
|
||||
`BEGIN IMMEDIATE`, at the same throughput and with no wait over a second.
|
||||
- **Ruled out for the current code: nested batches.** A `batch` opened inside another's
|
||||
closure that writes while the outer holds the write lock waits out `BUSY_TIMEOUT` and fails
|
||||
with plain `SQLITE_BUSY` (code 5) after 5.0 s, every time. None was found in production
|
||||
code. A thread-local depth check in `batch_on` and `SettingsStore::batch`, run across all
|
||||
1921 workspace tests, saw only a test's own nested read. A static scan of every `.batch(`
|
||||
closure in the workspace, gui-app included, for calls not handed `tx` that reach a
|
||||
batch-opening function, or that block on another thread, found nothing either. So a 5 s
|
||||
stall, if it recurs, means a writer that held the lock for 5 s, and `stores::contention`
|
||||
now logs any write transaction held past a second with the call site that opened it, along
|
||||
with every busy failure's extended code. Nesting is still worth guarding against
|
||||
permanently, since a write nested in a write always fails: a debug assertion on the same
|
||||
depth check would catch it in tests.
|
||||
- **Ruled out as the cause: `stores::pool`.** It doesn't change the mechanism. A fresh
|
||||
connection per transaction lost 13-38% in the same test. The pool's faster transactions
|
||||
overlap more under saturation, but the race was there before it.
|
||||
- **Fix (decided): a `WriteTx` that only an `IMMEDIATE` entry point hands out.** Making
|
||||
every batch `IMMEDIATE` would serialize reads with each other and with sync chunks, which
|
||||
WAL exists to avoid. Instead, `WriteTx` wraps a `Transaction` and derefs to it. Store
|
||||
methods that write take `&WriteTx`, about 93 of them plus about 23 helpers in agents and
|
||||
p2p. Each store's `batch` gets a `write` sibling, and the roughly 170 non-test call sites
|
||||
that write move to it, along with the writing ones in tests. `SettingsStore::batch` turns
|
||||
deferred like the rest, and its setters move to `write`. The type can't stop a method
|
||||
declared `&Transaction` from running an `INSERT`, so keep a runtime backstop too: a
|
||||
deferred `batch` that ends in `TransactionState::Write` fails a debug assertion.
|
||||
- **Then retire the retries.** The workarounds today are the desktop's
|
||||
`save_body_with_retry` in Notes and Android's `retryOnBusy`. The latter retries *any*
|
||||
failure, twice, two seconds apart. Its doc says it can't tell a busy database from other
|
||||
failures, which is stale: `SyncError::Busy` carries `Recourse::RetryNow`. Once writes are
|
||||
`IMMEDIATE` they queue instead of failing at once, and a busy that still gets through has
|
||||
already waited 5 s, so neither helper has anything left to do.
|
||||
### Busy database: what's left after `WriteTx`
|
||||
Writes used to fail with `database is busy` because a deferred transaction that read and then
|
||||
wrote lost its snapshot to any commit in between (`SQLITE_BUSY_SNAPSHOT`, at once, whatever
|
||||
`BUSY_TIMEOUT` says). Writes now go through `write` and a `WriteTx`, which begin `IMMEDIATE`
|
||||
and queue for the lock instead (ADR 0057). `stores::contention` logs every busy failure with
|
||||
its extended code and call site, and any write transaction held past a second.
|
||||
- **Retire the retries.** The desktop's `save_body_with_retry` in Notes and Android's
|
||||
`retryOnBusy` were workarounds for the snapshot failure, which can't happen any more. A
|
||||
busy that still gets through has already waited `BUSY_TIMEOUT` (5 s), so retrying it two
|
||||
seconds later is not the fix either. `retryOnBusy` also retries *any* failure, and its doc
|
||||
says it can't tell a busy database from other failures, which is stale:
|
||||
`SyncError::Busy` carries `Recourse::RetryNow`.
|
||||
- **The ~5 s stalls are unexplained.** Nested batches were ruled out on 2026-11-03 (and are
|
||||
now caught by `contention`'s nesting check), so a stall means some writer held the lock
|
||||
for 5 s. If one recurs, `contention`'s "held the write lock for …" warning names it.
|
||||
- **The backstop only covers tested paths.** A `batch` that writes, or a `write` nested in a
|
||||
`write`, panics in the lib's own tests and is only a warning elsewhere, including in
|
||||
gui-app's and the daemon's tests, where the lib isn't built with `cfg(test)`. Grep a
|
||||
session's log for "wrote inside a read transaction" after exercising a screen.
|
||||
|
||||
### Store I/O costs
|
||||
Measured by `just bench-store-io` (`lib/src/store_io_bench.rs`).
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
|
||||
**Status**: Accepted
|
||||
**Date**: 2026-08-05
|
||||
**Superseded in part by**: [ADR 0057](0057-writes-take-writetx-from-an-immediate-transaction.md) —
|
||||
store methods still take a transaction and `batch` is still how a read gets one,
|
||||
but a method that writes takes a `WriteTx` from `write`, which begins
|
||||
`IMMEDIATE`, and `SettingsStore::batch` is no longer the exception.
|
||||
|
||||
## Context
|
||||
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
# 0057: Writes take a `WriteTx`, which only an `IMMEDIATE` transaction hands out
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-11-04
|
||||
|
||||
## Context
|
||||
|
||||
Writes stalled or failed with "database is busy" now and then, on every
|
||||
modality, and never on demand. ADR 0019 had made `batch` the one way to open a
|
||||
transaction, and every `batch` began `DEFERRED`: it takes a WAL snapshot at its
|
||||
first read and needs the write lock only at its first write. Nearly every
|
||||
write reads first (`set_property` compares against the stored value, a note
|
||||
save merges against the stored body, sync's `apply_records` checks tombstones),
|
||||
so nearly every write transaction had to upgrade a read snapshot to the write
|
||||
lock part-way through.
|
||||
|
||||
That upgrade can't succeed if another connection committed after the
|
||||
snapshot was taken. SQLite refuses it at once with `SQLITE_BUSY_SNAPSHOT`
|
||||
(extended code 517) and doesn't consult the busy handler, since waiting can't
|
||||
make an old snapshot current. `BUSY_TIMEOUT` therefore protected none of these
|
||||
writers. Measured on 2026-11-03, two threads each running 300 read-then-write
|
||||
transactions lost 45-50% of them through the connection pool and 13-38% with
|
||||
a fresh connection per transaction (the pool wasn't the cause). With `BEGIN
|
||||
IMMEDIATE` they lost none, at the same throughput, and no wait exceeded a
|
||||
second.
|
||||
|
||||
Two alternatives were set aside:
|
||||
|
||||
- **Make every transaction `IMMEDIATE`.** Simple, but it serializes reads
|
||||
with each other and with every sync chunk, which is what WAL exists to
|
||||
avoid. `SettingsStore::batch` already did this, to keep `update_settings`'
|
||||
read-modify-write from reverting a concurrent setter's fields.
|
||||
- **Keep one `batch` and check at runtime.** A deferred transaction that ends
|
||||
in `TransactionState::Write` is a writer on the wrong path, and that is
|
||||
detectable after the fact. On its own, though, it only catches the paths
|
||||
tests exercise, and it leaves the choice to each call site's author.
|
||||
|
||||
## Decision
|
||||
|
||||
- **Two entry points.** `batch` stays `DEFERRED` and hands its closure a
|
||||
`&Transaction`. `write` (`write_on`, `Store::write`, and each store's own
|
||||
`write`) begins `IMMEDIATE` and hands a `&WriteTx`.
|
||||
- **A method that writes takes `&WriteTx`.** `WriteTx` wraps the transaction,
|
||||
derefs to it, and can only be constructed inside `stores`, so a writing
|
||||
method called from a `batch` closure is a type error. A read method keeps
|
||||
`&Transaction` and accepts a `&WriteTx` through deref coercion, so reads
|
||||
inside a `write` need no change. This holds for helpers outside `stores`
|
||||
that write through a transaction, and for the two `&Connection` helpers that
|
||||
write (`touch_document_in_tx`, `insert_edge_in_tx`).
|
||||
- **A transaction that might write is a `write`**, even if it usually doesn't
|
||||
(`ensure_document`, Humans' `touching`). Reading first in a `batch` and
|
||||
writing in a second transaction would reintroduce the race.
|
||||
- **`SettingsStore::batch` is deferred like every other store's.** Its setters
|
||||
take `&WriteTx`, which gives `update_settings` the serialization it needs.
|
||||
This removes ADR 0019's one exception.
|
||||
- **A runtime backstop for what the type can't see.** rusqlite lets any
|
||||
`&Transaction` run an `INSERT`, so `stores::contention` checks two things
|
||||
after each transaction. A `batch` that ended holding the write lock is one.
|
||||
A `write` opened while the same thread already holds one is the other: it
|
||||
waits out `BUSY_TIMEOUT` for a lock its own caller holds. Both panic in the
|
||||
lib's tests and log a warning elsewhere. On its first run the backstop found
|
||||
four writers the type change had missed, all writing through a `&Connection`
|
||||
helper.
|
||||
- **Busy failures are logged with their extended code and call site.** Every
|
||||
`batch`/`write` wrapper is `#[track_caller]`. A write transaction held past
|
||||
a second is logged the same way, since that is what makes another writer's
|
||||
`BUSY_TIMEOUT` run out.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Two writers that overlap now queue for the lock instead of one failing.
|
||||
`SQLITE_BUSY_SNAPSHOT` can't reach a store writer any more. A busy that
|
||||
still happens has waited the full `BUSY_TIMEOUT`, and the log names the
|
||||
transaction that held the lock.
|
||||
- A write takes the lock for its whole transaction, including the reads before
|
||||
its first write. A long read-then-write blocks other writers for longer than
|
||||
it used to, though under the old scheme it would have failed outright. Reads
|
||||
in a `batch` never wait.
|
||||
- Call sites state their intent: `batch` reads, `write` writes. A writer on
|
||||
the read path fails to compile, or, if the method hides its write behind
|
||||
`&Transaction`, fails the lib's tests.
|
||||
- The backstop only panics in the lib's own tests. gui-app's and the daemon's
|
||||
tests build the lib without `cfg(test)`, so a mistake reached only from
|
||||
there shows up as a warning in the log.
|
||||
- The per-screen retries that worked around the snapshot failure (desktop
|
||||
Notes' `save_body_with_retry`, Android's `retryOnBusy`) have nothing left to
|
||||
retry; removing them is in `TODO.md`.
|
||||
@@ -86,3 +86,4 @@ transition alone.
|
||||
| [0054](0054-comic-reimport-identity.md) | Re-importing a comic updates it in place — matched by source name, then by page set | Accepted |
|
||||
| [0055](0055-android-screens-assembled-from-roles.md) | Android screens are assembled from roles, not from Material components | Accepted |
|
||||
| [0056](0056-keep-synchronicity-lib-one-crate.md) | Keep `synchronicity-lib` one crate; give hot loops their own optimized crate | Accepted |
|
||||
| [0057](0057-writes-take-writetx-from-an-immediate-transaction.md) | Writes take a `WriteTx`, which only an `IMMEDIATE` transaction hands out | Accepted |
|
||||
Reference in new issue
Block a user