Skip to content

feat: wire discovery watermark into insights-discussions (CM-1455) - #4691

Draft
ulemons wants to merge 2 commits into
mainfrom
feat/CM-1455-discovery-watermark-wiring
Draft

ulemons wants to merge 2 commits into
mainfrom
feat/CM-1455-discovery-watermark-wiring

Conversation

@ulemons

@ulemons ulemons commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Second of two PRs for CM-1455 (watermark for discovery sources). Depends on the PR1
migration/DAL being merged first
(discoverySourceState table + DAL functions).
This PR wires the watermark into the workflow, but only for insights-discussions.
lf-criticality-score's interface is generalized (scoredAftersince) but it
still does a full fetch every run — no behavior change for that source in this PR.

Changes

  • sources/types.ts: IDiscoverySource.listAvailableDatasets takes { since? }
    instead of nothing; IDatasetDescriptor gains an optional since, stamped by
    listAvailableDatasets so it survives into fetchDatasetStream via workflow
    history (they're separate Temporal activities).
  • sources/insights-discussions/source.ts: fetchAllDiscussionRepoUrls now skips
    discussions with updatedAt older than since, client-side, after walking every
    page (no early-stop — pagination isn't ordered by updatedAt, so an early-stop
    would risk permanently missing edited discussions). Adds a skippedUnchanged
    counter to the aggregate log.
  • sources/lf-criticality-score/source.ts: renamed scoredAfter param to since
    to match the generic interface; maps sincescoredAfter internally. Behavior
    unchanged — the workflow doesn't pass it a watermark in this PR.
  • activities/activities.ts:
    • listDatasets now forwards since to the source (previously called with no
      arguments — dead code).
    • New readSourceWatermark(sourceName): reads the persisted watermark, subtracts
      a 24h overlap, and captures now() — all inside the activity, since workflow
      code can't do I/O or read the clock directly.
    • New commitSourceWatermark(sourceName, watermark, force).
    • IProcessDatasetResult gains truncated: boolean, set at all three early-exit
      paths driven by DISCOVERY_NEW_PROJECTS_LIMIT (biased toward true: a false
      negative would advance the watermark past unprocessed data).
  • workflows/discoverProjects.ts: for insights-discussions only, reads the
    watermark before listing datasets, wraps listDatasets in its own try/catch (a
    source-list failure no longer kills the whole run), and commits the new watermark
    only if the source's entire run succeeded and wasn't truncated. mode: 'full'
    skips the watermark read and forces the commit (bypasses the forward-only
    GREATEST via force: true).

Why a 24h overlap

Up to 90 minutes can elapse between listDatasets and processDataset
(startToCloseTimeout), plus retries, plus ordinary clock skew. 24h absorbs that
gap with a large margin. Re-processing already-seen discussions is harmless (the
catalog dedupes on repoUrl), so a generous overlap costs nothing.

Known first-run behavior

DISCOVERY_NEW_PROJECTS_LIMIT defaults to 20; the category currently has ~100
candidates. The first several nightly runs will be truncated and won't commit a
watermark yet — expected, and the backlog drains in a few days once new candidates
per night drop below the limit.

Out of scope

Exact watermarking from data (max(updatedAt)) — the committed value is
capturedAt, a timestamp taken inside the activity, not derived from the data.
Deemed acceptable given the 24h overlap.

Type of change

  • Enhancement (touches production call sites)

JIRA ticket

https://linuxfoundation.atlassian.net/browse/CM-1455

Depends on: PR1 (discoverySourceState table + DAL)

New table + DAL for a per-source watermark.
Nothing calls them yet; wiring follows in a separate PR.

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Reads/commits the per-source watermark around the run.
lf-criticality-score keeps its full-fetch behavior unchanged.

Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
Copilot AI balanced review requested due to automatic review settings September 21, 2026 12:51
@ulemons ulemons self-assigned this Sep 21, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

Watermarks advance only after complete successful processing, and the persistence semantics have focused coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Adds persistent discovery watermarks for incremental insights-discussions processing while preserving full-fetch behavior for LF Criticality Score.

Changes:

  • Adds watermark persistence and forward-only DAL operations.
  • Propagates since through Temporal activities and source descriptors.
  • Commits watermarks only after successful, non-truncated processing.
File Description
services/​libs/​data-access-layer/​src/​index.ts Exports discovery DAL APIs.
services/​libs/​data-access-layer/​src/​discovery/​types.ts Defines persisted source state.
services/​libs/​data-access-layer/​src/​discovery/​sourceState.ts Implements watermark reads and upserts.
services/​libs/​data-access-layer/​src/​discovery/​sourceState.test.ts Tests watermark persistence semantics.
services/​libs/​data-access-layer/​src/​discovery/​index.ts Exports discovery modules.
services/​apps/​automatic_projects_discovery_worker/​src/​workflows/​discoverProjects.ts Coordinates watermark lifecycle.
services/​apps/​automatic_projects_discovery_worker/​src/​sources/​types.ts Generalizes source filtering with since.
services/​apps/​automatic_projects_discovery_worker/​src/​sources/​lf-criticality-score/​source.ts Maps since to scoredAfter.
services/​apps/​automatic_projects_discovery_worker/​src/​sources/​insights-discussions/​source.ts Filters unchanged discussions.
services/​apps/​automatic_projects_discovery_worker/​src/​activities/​activities.ts Adds watermark activities and truncation tracking.
backend/​src/​database/​migrations/​V1789994431__discovery-source-state.sql Creates source watermark storage.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants