Tags: nitrite/nitrite-java
Tags
Take a sorted page's order from the index instead of every document
`find(ALL, orderBy("createdAt", Descending).limit(20))` asked for 20 rows and
cost what draining the whole collection costs. `SortedDocumentStream` collects
the entire result set before `BoundedStream` gets to drop 99% of it, and the cost
is the decode, not the comparison, so it scales with document *size* as much as
count. An index on the sort field bought nothing — the index was only ever used
to filter — and page 50 cost exactly what page 1 cost, because the work finished
before the skip applied.
An index on the sort field already stores that field's value for every document.
When the query has no filter, one sort field, a limit, and a simple unique or
non-unique index on exactly that field, the keys now come from the index
(`NitriteIndex.readSortKeys`) and `IndexedStream` fetches only the rows the page
returns. On MVStore over 2000 rows each carrying a 150-element list, a `limit(20)`
page went from ~125 ms to ~1 ms.
The index replaces the documents only when it is a faithful stand-in for them: a
multi-valued field is indexed once per element, so a duplicate-id check and an
entry-count check catch it and fall back to the blocking sort. Ordering is decided
by the same comparator either way — extracted as `DocumentSorter.compareValues` so
the two cannot drift — over keys read from the index rather than from documents,
which keeps nulls first and resolves ties in document-id order exactly as before.
Where documents are small the index walk replaces a decode that was nearly free,
so a sorted page over lean rows can cost a few hundred microseconds more than it
did. The trade is deliberate: that loss is bounded and sub-millisecond, the win
grows without bound with document size.
New API is additive. `NitriteIndex.readSortKeys(long)` and
`NitriteIndexer.readSortKeys(IndexDescriptor, NitriteConfig, long)` are `default`
methods returning `null`, so an existing indexer plugin is unaffected and simply
never takes the new path.
`CollectionSortedFindTest` pins every sorted query against the same query on an
unindexed collection — ascending, descending, deep pages, ties, strings, missing
fields, unique indexes, and the multi-valued fallback — so a divergence in
ordering fails outright. `CollectionSortedFindCostTest` runs on MVStore, where
documents are actually serialized, and compares a sorted page over lean documents
against one over fat documents at the same row count: only the decode differs
between them, so it is stable under load in a way a wall-clock threshold is not
(it reports 94x without this change and ~1x with it).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
feat!: remove the no-op distinct find option BREAKING CHANGE: removed `FindOptions.withDistinct()` (both overloads), `FindOptions.distinct()` and `FindPlan.distinct`. The flag never affected the result set. A find never returns the same document twice, and the `or` sub-plan union in ReadOperations has always applied DistinctStream unconditionally - so FindOptimizer wrote the flag onto the FindPlan and nothing ever read it back. Callers passing `FindOptions.withDistinct()` can drop it with no change in results. `DistinctStream` itself is unchanged and still used for the `or` union. Also adds CollectionOrDuplicateTest, which pins the set-union behaviour of an `or` filter for the no-index, one-indexed-branch and all-indexed-branch cases. nitrite-java was already correct here; the same test found real duplicate defects in nitrite-rust and nitrite-flutter, and is added to keep the three implementations honest. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix: restore CopyOnWriteArrayList for unique and text index value lists The #1260 index rework (4.4.0) switched the per-key `List<NitriteId>` value used by unique and full-text indexes from CopyOnWriteArrayList to a plain ArrayList mutated in place. MVStore serializes dirty page values on a background thread, so mutating that list after it is written to the map races with the serializer, throwing ConcurrentModificationException (and could corrupt the id list, later surfacing as a spurious UniqueConstraintException) even under single-threaded use. Restore CopyOnWriteArrayList for these classic list-valued index paths so each mutation swaps the backing array atomically and the background serializer always sees a stable snapshot. The composite-key layout for non-unique indexes (the actual #1260 optimization) is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PreviousNext