Publish vector index files atomically, not durably - #1460

Problem

SQLiteVectorStore persists each collection's index by calling the search engine's save(), which writes directly to the final path. A crash partway through leaves a truncated file on disk.

The torn file is only half of it. _save_collection_index trims the applied _PendingOperationRows as soon as save() returns, and that log is the only other copy of those vectors — the records table has no vector column. So the trim is safe at exactly one instant: when the index at the published path holds them. Getting that instant wrong is silent, because a missing vector is indistinguishable from a vector that simply doesn't match.

Fix

A shared index_persistence helper writes the index to a sibling temp file, flushes it, and swaps it into place with os.replace; both engines adopt it and neither gains anything else. load clears a temp left by an interrupted save, so a crash leaks at most one stale file per index.

After this, a save either publishes a complete index or leaves the previous one exactly where it was. Nothing else changes: no schema change, no migration, no rename of index_path, and no new file at the base path.

What a save does not promise

It does not promise that the publication is durable. That is a decision rather than an oversight, so the reasoning is below.

A rename changes a directory entry, not file data, and the two have very different guarantees:

• POSIX closes the gap with a parent-directory fsync, but only best-effort — network and FUSE filesystems are where that bites.
• Windows has no equivalent at all. os.fsync is _commit, which is FlushFileBuffers, which is for file data; you cannot open a directory to fsync it. MOVEFILE_WRITE_THROUGH does not help either: its documented guarantee is scoped to "a move performed as a copy and delete operation", the cross-volume path, and says nothing about flushing NTFS metadata for a same-volume rename. The decisive evidence is SQLite's — its VFS threads a directory-sync flag through every commit-relevant directory operation, unixDelete honors it, and winDelete declares the same parameter /* Not used on win32 */. SQLite cares about this more than almost any software and makes no attempt on Windows.

So a power failure can roll the rename back after save returned, while the trim behind it stays committed.

Why that gap stays open

Closing it means never using a directory operation as the commit point: two index slots plus a generation record written into a file that already exists, in the spirit of SQLite's PERSIST journal mode. An earlier revision of this PR implemented exactly that, and it worked. It was still the wrong trade, for four reasons.

The window is narrow, and only one kind of failure lands in it. A clean shutdown, an unhandled exception, an OOM kill, a SIGKILL — none of these lose a rename, because the kernel still owns the page cache and writes it back. Those failures are already covered: the pending log holds everything the engine has not been checkpointed with, and startup replays it. What is left is the machine itself dying between the rename and writeback — a power cut, a host failure, a panic — plus the filesystems where a directory fsync is best-effort or unavailable anyway.

What it costs is recall, not correctness. The records table is the authority on what exists, and every query hit resolves through it, so a reverted publication cannot produce a wrong or stale result — only fewer results. Concretely: the records survive, get still returns them, query stops finding them, and the exposure is bounded by the operations applied since the previous checkpoint, i.e. at most save_threshold. Re-upserting an affected record repairs it. That is the direction this store already tolerates, and it is repairable by the same ingest path that created the record.

Detecting it is not cheap enough to be worth it. Comparing the index's size against the record count is the obvious check and it does not work: row_ids are AUTOINCREMENT and never reused, so deleting a record and upserting the same uuid again moves it to a new id. A rollback spanning that pair leaves the index holding the old id and missing the new one — one extra, one missing, identical count — and a re-embedding workload produces that shape constantly. A check that does work needs an id-set digest maintained by every engine, or a full reconcile scan at every boot, which is most expensive precisely where the index is large. Neither earns its keep against a bounded recall gap.

And the guarantee is not free to hold. It cannot be delegated to a rename, so it has to become a layout: two slots and a generation record per index, which every engine has to implement and every future engine has to be audited against. Engines that persist by writing one file — which is what both engines here do, and what most ANN libraries expose — stop being usable as they ship. Existing indexes stop loading until they are cleared and re-ingested, which is a migration this revision no longer asks anyone to perform. Fewer guarantees, less machinery to keep correct, and a wider set of engines that can plug in unmodified.

The direction that would cost — a published index that will not parse — stays closed: the atomic swap prevents it, the pre-swap fsync narrows the window where a rename outlives the data behind it, and a saved-but-unloadable index remains a loud IndexLoadError rather than a silently empty rebuild.

Nothing here detects a lost publication for the caller. A deployment that needs every record searchable after a power failure must be able to re-ingest.

Where the contract is stated

The guarantee is only useful if callers can find it, so it is written where each of them looks:

• VectorSearchEngine.save — returning means a later load reads this index or the one it replaced, never a mixture; it does not mean the publication survives a power failure.
• index_persistence — the mechanism, and why a stronger guarantee is not portable.
• sqlite_vector_store — what survives a process crash, what a power failure costs, and that re-ingest is the repair.
• _save_collection_index — why the save-then-trim order is the whole protocol.

One consumer needed more than a docstring. VectorStoreSemanticStorage.update_feature reads a feature's stored embedding back when the caller updates the feature without supplying one — the only place in the server that depends on the index still holding a vector. It reported a lost vector as Vector record not found, naming the feature, which points at the wrong thing and hides the repair; it now separates a record that is genuinely absent from one whose embedding is gone, and the second says to pass a fresh embedding.

Tests

• test_index_persistence.py: the swap publishes completely or not at all, a failed write leaves the previous index intact and removes the temp, and a stale temp is cleared on load.
• Per-engine (hnswlib + usearch): repeated saves leave no stray files, and a reload after two checkpoints sees the newer index.
• Store-level: test_a_reverted_publication_costs_search_not_records reconstructs a lost publication deterministically — restore the previous index bytes after the trim has committed — and pins the direction it fails in: the record still resolves by uuid, and only search loses it. Missing and corrupt indexes each still raise IndexLoadError once index_saved is set.

vector_store suites pass (284 passed, 140 integration deselected) and semantic_memory suites pass (345 passed, 2 skipped, 999 integration deselected); ruff and ty clean.

Changed from the earlier revision

This PR previously shipped the two-slot generation-record protocol and its breaking on-disk change. That guarantee has been dropped in favor of the atomic swap plus an explicit contract, so the breaking-change notice no longer applies: indexes written by the current code keep loading, and there is nothing to migrate.

🤖 Generated with Claude Code