test(goldens): fail clearly when indexing yields an empty graph - #1616
Conversation
FalkorDB Lite's storage worker subprocess has a timing-sensitive startup; under CPU contention it can fail to come up while indexing still exits 0 with an empty graph, and the export can still write a bundle. Previously the golden diff would then report every expected node as "missing", which reads exactly like a parser regression but is actually an infrastructure failure. Assert the extracted graph is non-empty right after extraction, before any golden comparison (including --update-goldens), with a message that names the real cause and includes the subprocess output.
|
@rrodriguesNutrium is attempting to deploy a commit to the shashankss1205's projects Team on Vercel. A member of the Team first needs to authorize it. |
Shashankss1205
left a comment
There was a problem hiding this comment.
Approving — this converts a confusing failure mode into a self-explaining one, and the failure message does the hard part by telling the next person not to go looking at the parser.
The reasoning for placing the guard before the --update-goldens branch is the subtle part and you got it right: bootstrapping a golden from an empty graph would bake the failure in permanently, which is strictly worse than failing loudly.
Your note about FalkorDB Lite's worker startup being timing-sensitive under CPU contention is also the best lead anyone has on #1612 — Database Parity Check intermittently has LadybugDB write 51 fewer CONTAINS edges than the other three backends, and the failing runs are consistently ~25% slower across every backend, i.e. a loaded runner. Same shape of problem: a timing-sensitive embedded backend silently under-writing rather than erroring. I've cross-referenced this there.
test_language_goldenasserts that the index/export subprocess exited 0 and that a bundle file exists — but never that the bundle contains anything.FalkorDB Lite runs its storage in a worker subprocess with a timing-sensitive startup. Under CPU contention the worker can fail to come up; indexing then completes exit 0 with an empty graph, and the export happily writes an empty bundle. The golden diff that follows reports every node as missing:
So an infrastructure failure is indistinguishable from a total parser regression.
I lost several hours to exactly this. I concluded the golden suite was broadly broken — "17 of 20 failing on main" — and reasoned at length about shared state and unix-socket path limits. It was neither: my machine was saturated by an unrelated long-running index. Idle, all 20 pass in 5m18s.
This adds a guard between extraction and comparison. If the extracted graph is empty it fails immediately, states that this indicates the indexing/database step failed rather than a parser regression, names CPU contention and the FalkorDB worker as the known cause, and includes the subprocess stdout/stderr so the real error is visible.
No retries, no sleeps, no masking — the goal is a failure that explains itself, not one that disappears.
Verified the guard fires: with a successful run, truncating the extracted
nodes.jsonl/edges.jsonlproduces the new message rather than a "Missing N expected nodes" diff. Goldens still pass normally (sample_project_java,sample_project_gochecked on this branch).One file, +33/-1, tests only.