Skip to content

[WIP] Adding capabilities to specify schema for RedshiftSource - #1772

Closed
blvp wants to merge 5 commits into
feast-dev:masterfrom
blvp:master
Closed

[WIP] Adding capabilities to specify schema for RedshiftSource#1772
blvp wants to merge 5 commits into
feast-dev:masterfrom
blvp:master

Conversation

@blvp

@blvp blvp commented Aug 10, 2021

Copy link
Copy Markdown

What this PR does / why we need it:
This feature is required to eliminate undefined behaviour where users can specify a table which represented in multiple schemas.
Which issue(s) this PR fixes:

Fixes #1767

Does this PR introduce a user-facing change?:

RedshiftSource optional `schema` parameter support.

@feast-ci-bot

feast-ci-bot commented Aug 10, 2021

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: blvp
To complete the pull request process, please assign jklegar after the PR has been reviewed.
You can assign the PR to them by writing /assign @jklegar in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@feast-ci-bot

feast-ci-bot commented Aug 10, 2021

Copy link
Copy Markdown
Collaborator

Hi @blvp. Thanks for your PR.

I'm waiting for a feast-dev member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@blvp

blvp commented Aug 10, 2021

Copy link
Copy Markdown
Author

Created this Draft PR to start triggering integration tests. Going to also provide additional unit tests and integration tests with that edge case.

@woop

woop commented Aug 10, 2021

Copy link
Copy Markdown
Member

This is great @blvp!

@codecov-commenter

codecov-commenter commented Aug 10, 2021 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

Merging #1772 (4fa934b) into master (8ef2053) will decrease coverage by 20.52%.
The diff coverage is 35.29%.

Impacted file tree graph

@@             Coverage Diff             @@
##           master    #1772       +/-   ##
===========================================
- Coverage   84.71%   64.19%   -20.53%     
===========================================
  Files          86       83        -3     
  Lines        6308     6261       -47     
===========================================
- Hits         5344     4019     -1325     
- Misses        964     2242     +1278
Flag Coverage Δ
integrationtests ?
unittests 64.19% <35.29%> (-0.13%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
sdk/python/feast/infra/utils/aws_utils.py 31.42% <0.00%> (-59.84%) ⬇️
sdk/python/feast/infra/offline_stores/redshift.py 38.31% <28.57%> (-51.98%) ⬇️
...thon/feast/infra/offline_stores/redshift_source.py 43.26% <36.36%> (-24.55%) ⬇️
sdk/python/feast/errors.py 66.32% <66.66%> (-4.20%) ⬇️
.../integration/online_store/test_online_retrieval.py 17.39% <0.00%> (-82.61%) ⬇️
sdk/python/tests/utils/online_read_write_test.py 18.18% <0.00%> (-81.82%) ⬇️
...gration/registration/test_feature_service_apply.py 31.25% <0.00%> (-68.75%) ⬇️
...hon/tests/integration/registration/test_cli_gcp.py 31.70% <0.00%> (-68.30%) ⬇️
...n/tests/integration/registration/test_cli_redis.py 32.55% <0.00%> (-67.45%) ⬇️
sdk/python/feast/infra/online_stores/redis.py 30.35% <0.00%> (-64.29%) ⬇️
... and 38 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ef2053...4fa934b. Read the comment docs.

achals
achals requested changes Aug 25, 2021
View reviewed changes

@achals achals left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @blvp ! I had a question on whether a field in the config class can be optional or not.

Comment thread sdk/python/feast/infra/offline_stores/redshift.py
Comment on lines +41 to +42
temp_schema_name: StrictStr
""" Redshift schema name to offload temporary tables """

@achals achals Aug 25, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this make this confguration field mandatory? Should we make it optional instead?

@blvp blvp Aug 25, 2021

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think some default should be defined explicitly, but I agree there isn't any reason to force user to specify that

@felixwang9817 felixwang9817 self-assigned this Sep 20, 2021
@felixwang9817

felixwang9817 commented Sep 20, 2021

Copy link
Copy Markdown
Collaborator

Hey @blvp, just wanted to let you know that we recently added support for schemas in Redshift in #1847. If that looks alright to you, we can probably close out this PR.

@blvp

blvp commented Sep 20, 2021

Copy link
Copy Markdown
Author

Yeah makes sense. Original issue was fixed already.

@blvp blvp closed this Sep 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@achals achals achals requested changes

Assignees

@felixwang9817 felixwang9817

Labels

do-not-merge/work-in-progress needs-kind needs-ok-to-test release-note size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add schema parameter to RedshiftSource to distinguish between database schemas

6 participants

@blvp @feast-ci-bot @woop @codecov-commenter @felixwang9817 @achals