Skip to content

feat: Add local Iceberg materialization sink - #6747

Open
haoxu0 wants to merge 12 commits into
masterfrom
feat/iceberg-local-materialization-sink
Open

haoxu0 wants to merge 12 commits into
masterfrom
feat/iceberg-local-materialization-sink

Conversation

@haoxu0

@haoxu0 haoxu0 commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • add PyIceberg-backed catalog access and table upserts to IcebergSource
  • route local compute materialization output into Iceberg-backed derived feature views
  • validate join keys and Arrow schemas before writes, with safe contextual errors
  • add unit and component coverage using a real local PyIceberg SQL catalog
  • document configuration, upsert keys, compatibility, and failure behavior

Why

Feast could read Iceberg sources, but local materialization could not persist results back to Iceberg. This adds a local-compute sink path so derived feature data can be maintained in Iceberg with idempotent upsert semantics.

Impact

Users can configure an IcebergSource as the effective batch source of a derived feature view and have local materialization upsert rows using mapped entity join keys plus the event timestamp. Existing online and offline materialization behavior remains unchanged.

Validation

  • focused Iceberg and local compute suite: 104 passed
  • Ruff formatting and lint: passed
  • mypy for changed Python modules: passed
  • Spark correctness suite with project Python pinned: 5 passed
  • independent code review: no remaining Critical or Important findings

The broader repository unit run reached 2,469 passes. Three existing tests in unchanged on-demand transformation and SQLite paths still fail locally; the failures do not enter the Iceberg sink code. The pre-push lint-push wrapper was skipped because its uv run resolves incompatible unrelated ci and flink extras; the underlying lint and type checks were run directly and passed.

haoxu0 added 8 commits August 16, 2026 00:22
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
Signed-off-by: HaoXuAI <sduxuhao@gmail.com>
@codecov-commenter

codecov-commenter commented Aug 16, 2026

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 20.58824% with 81 lines in your changes missing coverage. Please review.
✅ Project coverage is 47.16%. Comparing base (787845a) to head (894d580).

Files with missing lines Patch % Lines
..._sources/contrib/iceberg_catalog/iceberg_source.py 10.98% 81 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6747      +/-   ##
==========================================
- Coverage   47.18%   47.16%   -0.03%     
==========================================
  Files         419      419              
  Lines       51964    52063      +99     
  Branches     7548     7564      +16     
==========================================
+ Hits        24521    24554      +33     
- Misses      25689    25757      +68     
+ Partials     1754     1752       -2     
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 48.47% <20.58%> (-0.04%) ⬇️
Files with missing lines Coverage Δ
...ast/infra/compute_engines/local/feature_builder.py 56.41% <100.00%> (+0.56%) ⬆️
.../python/feast/infra/compute_engines/local/nodes.py 85.02% <100.00%> (+1.18%) ⬆️
..._sources/contrib/iceberg_catalog/iceberg_source.py 24.33% <10.98%> (-0.67%) ⬇️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 787845a...894d580. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@haoxu0
haoxu0 marked this pull request as ready for review August 22, 2026 17:50
@haoxu0
haoxu0 requested a review from a team as a code owner August 22, 2026 17:50
@haoxu0 haoxu0 changed the title feat: add local Iceberg materialization sink feat: Add local Iceberg materialization sink Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants