Skip to content

Add schema parameter to RedshiftSource - #1847

Merged
feast-ci-bot merged 1 commit into
feast-dev:masterfrom
felixwang9817:redshift_schema
Sep 9, 2021
Merged

Add schema parameter to RedshiftSource#1847
feast-ci-bot merged 1 commit into
feast-dev:masterfrom
felixwang9817:redshift_schema

Conversation

@felixwang9817

@felixwang9817 felixwang9817 commented Sep 8, 2021 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Signed-off-by: Felix Wang wangfelix98@gmail.com

What this PR does / why we need it: The RedshiftSource implementation currently does not support a schema parameter. This PR adds support for a schema parameter.

Which issue(s) this PR fixes:

Fixes #1767

Does this PR introduce a user-facing change?:

schema is now an optional parameter in RedshiftSource

@felixwang9817
Add schema parameter to RedshiftSource
809ce38 
Signed-off-by: Felix Wang <wangfelix98@gmail.com>
@felixwang9817
felixwang9817 requested review from a team, achals, tsotnet and woop as code owners September 8, 2021 20:56
@codecov-commenter

codecov-commenter commented Sep 8, 2021 •
edited
Loading

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 84.46%. Comparing base (0dc13f0) to head (809ce38).
⚠️ Report is 2615 commits behind head on master.

Files with missing lines Patch % Lines
...thon/feast/infra/offline_stores/redshift_source.py 92.85% 1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #1847      +/-   ##
==========================================
+ Coverage   84.45%   84.46%   +0.01%     
==========================================
  Files          90       90              
  Lines        6773     6784      +11     
==========================================
+ Hits         5720     5730      +10     
- Misses       1053     1054       +1
Flag Coverage Δ
integrationtests 84.38% <92.85%> (+0.01%) ⬆️
unittests 63.72% <50.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@felixwang9817

felixwang9817 commented Sep 8, 2021

Copy link
Copy Markdown
Collaborator Author

/kind bug

woop
woop reviewed Sep 8, 2021
View reviewed changes
Comment thread sdk/python/feast/infra/offline_stores/redshift_source.py

self._redshift_options = RedshiftOptions(table=table, query=query)
# The default Redshift schema is named "public".
_schema = "public" if table and not schema else schema

@woop woop Sep 8, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it appropriate for us to set the default or defer to Amazon for the schema? I'm not super familiar with how AWS deals with the lack of a schema.

@felixwang9817 felixwang9817 Sep 8, 2021 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redshift comes with a default schema named public. By default, operations occur in the public schema; for example, if you reference a table without specifying a schema, Redshift will look in the public schema.

We could just defer to Redshift's default, but I think it's better to be explicit here. It will make it easier for users to understand (especially users who aren't familiar with Redshift), and easier to debug.

@woop woop Sep 9, 2021 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If Redshift uses public by default, what difference does setting the schema explicitly on our side make? Why would it be easier to understand or debug? It seems like we are moving the debugging experience from Redshift to Feast.

I am not trying to block this PR. Happy to merge. Just curious.

@woop

woop commented Sep 8, 2021

Copy link
Copy Markdown
Member

My only real question is how we know that we have solved the problem without changing any tests. Do we consider this functionality to be far enough off the happy path that changing our tests or updating our tests isn't worth it?

@felixwang9817

felixwang9817 commented Sep 8, 2021

Copy link
Copy Markdown
Collaborator Author

My only real question is how we know that we have solved the problem without changing any tests. Do we consider this functionality to be far enough off the happy path that changing our tests or updating our tests isn't worth it?

Yeah that's a fair question. Since this PR has added "public" as the default schema, the existing tests are now querying Redshift with both a table and a schema, instead of just relying on Redshift's default. However, we are not testing the case where a user might specify a different schema; I consider this far enough off the happy path that we shouldn't modify our tests.

@woop

woop commented Sep 9, 2021

Copy link
Copy Markdown
Member

/lgtm

@feast-ci-bot

feast-ci-bot commented Sep 9, 2021

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: felixwang9817, woop

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@feast-ci-bot
feast-ci-bot merged commit ac71cdf into feast-dev:master Sep 9, 2021
@felixwang9817
felixwang9817 deleted the redshift_schema branch September 9, 2021 05:09
@felixwang9817 felixwang9817 mentioned this pull request Sep 20, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@woop woop woop approved these changes
@achals achals Awaiting requested review from achals
@tsotnet tsotnet Awaiting requested review from tsotnet

Assignees

@woop woop

Labels

approved kind/bug lgtm release-note size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add schema parameter to RedshiftSource to distinguish between database schemas

4 participants

@felixwang9817 @codecov-commenter @woop @feast-ci-bot