[pull] master from cube-js:master - #748

…ne (#11629)

* fix(query-orchestrator): don't report failed pre-aggregation build jobs as done

A build job whose import failed was indistinguishable from a successful one.
The empty versioned table stayed in Cube Store as the newest version of the
partition, shadowing the previous correct one, while the job reported `done`.

`isPartitionExist` derived the job status from the existence of the target
table, which is created before the rows are imported into it and is left
behind when the import fails or the process is killed. The build outcome is
now recorded per versioned table and wins over the table being there. It is
tracked for build jobs only, and its absence still falls back to the table,
so a partition that was already up to date and never built is unaffected.

A build whose process is gone never records an outcome. Since the queue times
a build out on its own, a build still marked as running long past that timeout
is reported as a failure instead of leaving the job waiting.

A failed external upload now drops the table it was creating, unless the table
was already there before the upload started and therefore belongs to another
build.

Build jobs also went through `loadPreAggregation` ahead of the `externalRefresh`
guard, so an instance configured to never build pre-aggregations still ran them
in-process. Such an instance now refuses the job.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(cubestore): cover table visibility around external imports

Pins down the two invariants the orchestrator relies on when it decides
whether a pre-aggregation partition has been built: a table imported from a
location is published only after the import succeeds and is dropped when it
fails, while a table created without a location is queryable straight away,
before any rows are inserted into it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(query-orchestrator): let build jobs run regardless of externalRefresh

`externalRefresh` stops a query from building a pre-aggregation as a side
effect of serving it. A build job is an explicit request to build, so refusing
it on those instances breaks a supported setup: the driver test suite drives
every one of its builds through the jobs API against an instance that serves
queries only.

What made an interrupted build dangerous was that it was indistinguishable
from a finished one, and that is now handled directly, so the job runs and the
docs say what happens when the instance running it restarts.

Also, from review of the build outcome tracking:

- Record the outcome for every build, not just for the ones a job started. The
  queue de-duplicates on the query key, so a job regularly ends up waiting on
  a build some other request enqueued, and that build has to report too.
- Keep the table ahead of a stale queue error again. The recorded outcome is
  what the job status is built on now; a queue error only speaks for a
  partition that has no table.
- Rewrite the job record only when its status changed. Clients poll in a loop,
  so an unchanged status meant a write per token per poll and an expiry that
  kept moving forward.
- Drop a table left behind by a failed upload unconditionally. `CREATE TABLE`
  carries no `IF NOT EXISTS`, so leaving one behind fails every later attempt
  at that version, which is the case a "was it already there" check would have
  read as someone else's table.
- Tolerate `queueOptions` being absent, as its type allows.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(query-orchestrator): keep post-build cleanup out of the build outcome

Orphaned tables cleanup and the temp table drop run after the rows have landed
in the partition table, and they can fail on their own. Reporting the build as
failed for a partition that is complete and already serving queries is worse
than the missing failure this tracking was added for, especially since the
gateway caches a failure for the lifetime of the job.

Each strategy now reports the moment its rows land, and an error after that
point leaves the outcome alone. Landing clears the record instead of writing a
`done` into it: a complete table already answers for itself, so the record is
only ever a build in flight or a failure, and a healthy deployment leaves
nothing behind in the cache store.

Also note on the type that `startedAt` is compared against the clock of
whichever instance serves the status request, not the one that stamped it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(query-orchestrator): trim the build status comments

Both blocks had grown past what they carry. The type's comment repeated the
cross-instance `startedAt` note that `isBuildAbandoned` already makes at the
point it matters, and the one above the status chain re-derived the branches
written directly below it.

What is left is the part the code can't state on its own: the table exists
before the rows do, so a finished build is recorded by the absence of a record.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(query-orchestrator): widen the abandoned build window

`executionTimeout: 2` put the abandoned-build window at 4 seconds, which is
also the budget the test asserting a healthy in-flight build as `processing`
had to reach its assertion in. A loaded runner spending longer than that
between the import starting and the status call reads the build as abandoned
and fails on `failure: the build has not completed`.

Raising the timeout to 10 gives that assertion a 20 second window. The
abandoned test back-dates `startedAt` itself, so it moves past the new window
rather than depending on the short timeout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>