From b3ba8aaf3a87343d756a2996376865096d543515 Mon Sep 17 00:00:00 2001 From: Danny Chiao Date: Fri, 13 May 2022 19:49:03 -0400 Subject: [PATCH 001/123] fix: Fix broken roadmap links (#2690) Signed-off-by: Danny Chiao --- README.md | 4 ++-- .../architecture-and-components/online-store.md | 2 +- docs/getting-started/architecture-and-components/overview.md | 2 +- docs/getting-started/quickstart.md | 2 +- docs/roadmap.md | 4 ++-- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 5905bbd647f..01776b048ad 100644 --- a/README.md +++ b/README.md @@ -176,7 +176,7 @@ The list below contains the functionality that contributors are planning to deve * [ ] Cassandra * **Streaming** * [x] [Custom streaming ingestion job support](https://docs.feast.dev/how-to-guides/creating-a-custom-provider) - * [x] [Push based streaming data ingestion](https://docs.feast.dev/reference/data-sources/push.md) + * [x] [Push based streaming data ingestion](https://docs.feast.dev/reference/data-sources/push) * [ ] Streaming ingestion on AWS * [ ] Streaming ingestion on GCP * **Feature Engineering** @@ -207,7 +207,7 @@ The list below contains the functionality that contributors are planning to deve * [x] CLI for browsing feature registry * [x] Model-centric feature tracking (feature services) * [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py)) - * [x] Feast Web UI (Alpha release. See [documentation](https://docs.feast.dev/reference/alpha-web-ui.md)) + * [x] Feast Web UI (Alpha release. See [documentation](https://docs.feast.dev/reference/alpha-web-ui)) * [ ] REST API for browsing feature registry * [ ] Feature versioning diff --git a/docs/getting-started/architecture-and-components/online-store.md b/docs/getting-started/architecture-and-components/online-store.md index 4f2e44c92ce..21b4dbcb9c7 100644 --- a/docs/getting-started/architecture-and-components/online-store.md +++ b/docs/getting-started/architecture-and-components/online-store.md @@ -12,4 +12,4 @@ Once the above data source is materialized into Feast \(using `feast materialize ![](../../.gitbook/assets/image%20%285%29.png) -Features can also be written to the online store via [push sources](https://docs.feast.dev/reference/data-sources/push) \ No newline at end of file +Features can also be written to the online store via [push sources](../../reference/data-sources/push.md) \ No newline at end of file diff --git a/docs/getting-started/architecture-and-components/overview.md b/docs/getting-started/architecture-and-components/overview.md index bf5c12dcc20..0c47fb2753d 100644 --- a/docs/getting-started/architecture-and-components/overview.md +++ b/docs/getting-started/architecture-and-components/overview.md @@ -23,7 +23,7 @@ A complete Feast deployment contains the following components: * Materialize (load) feature values into the online store. * Build and retrieve training datasets from the offline store. * Retrieve online features. -* **Online Store:** The online store is a database that stores only the latest feature values for each entity. The online store is populated by materialization jobs and from [stream ingestion](../../reference/alpha-stream-ingestion.md). +* **Online Store:** The online store is a database that stores only the latest feature values for each entity. The online store is populated by materialization jobs and from [stream ingestion](../../reference/data-sources/push.md). * **Offline Store:** The offline store persists batch data that has been ingested into Feast. This data is used for producing training datasets. Feast does not manage the offline store directly, but runs queries against it. {% hint style="info" %} diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md index 41449b77e39..1684c720632 100644 --- a/docs/getting-started/quickstart.md +++ b/docs/getting-started/quickstart.md @@ -367,7 +367,7 @@ pprint(feature_vector) ## Step 7: Using a feature service to fetch online features instead. -You can also use feature services to manage multiple features, and decouple feature view definitions and the features needed by end applications. The feature store can also be used to fetch either online or historical features using the same api below. More information can be found [here](https://docs.feast.dev/getting-started/concepts/feature-service). +You can also use feature services to manage multiple features, and decouple feature view definitions and the features needed by end applications. The feature store can also be used to fetch either online or historical features using the same api below. More information can be found [here](https://docs.feast.dev/getting-started/concepts/feature-retrieval). {% tabs %} {% tab title="Python" %} diff --git a/docs/roadmap.md b/docs/roadmap.md index c2f5511f1ed..d893814aafd 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -41,7 +41,7 @@ The list below contains the functionality that contributors are planning to deve * [ ] Cassandra * **Streaming** * [x] [Custom streaming ingestion job support](https://docs.feast.dev/how-to-guides/creating-a-custom-provider) - * [x] [Push based streaming data ingestion](https://docs.feast.dev/reference/data-sources/push.md) + * [x] [Push based streaming data ingestion](https://docs.feast.dev/reference/data-sources/push) * [ ] Streaming ingestion on AWS * [ ] Streaming ingestion on GCP * **Feature Engineering** @@ -72,6 +72,6 @@ The list below contains the functionality that contributors are planning to deve * [x] CLI for browsing feature registry * [x] Model-centric feature tracking (feature services) * [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py)) - * [x] Feast Web UI (Alpha release. See [documentation](https://docs.feast.dev/reference/alpha-web-ui.md)) + * [x] Feast Web UI (Alpha release. See [documentation](https://docs.feast.dev/reference/alpha-web-ui)) * [ ] REST API for browsing feature registry * [ ] Feature versioning From 1958df03649c5162d5ece693af8ec2a5c70b5631 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Mon, 16 May 2022 10:00:03 -0700 Subject: [PATCH 002/123] ci: Make version bumping script more robust (#2689) * ci: Make version bumping script more robust Signed-off-by: Achal Shah * bump dockerfiles too Signed-off-by: Achal Shah * fix build-docker-image Signed-off-by: Achal Shah * use buildx instead of build in makefile Signed-off-by: Achal Shah --- .github/workflows/pr_integration_tests.yml | 6 +- Makefile | 2 +- infra/scripts/release/bump_file_versions.py | 67 +++++++------------ infra/scripts/release/files_to_bump.txt | 25 ++++--- .../feature_servers/aws_lambda/Dockerfile | 5 +- .../feature_servers/gcp_cloudrun/Dockerfile | 4 +- .../infra/transformation_servers/Dockerfile | 6 +- 7 files changed, 52 insertions(+), 63 deletions(-) diff --git a/.github/workflows/pr_integration_tests.yml b/.github/workflows/pr_integration_tests.yml index 089d9f47336..c4417056136 100644 --- a/.github/workflows/pr_integration_tests.yml +++ b/.github/workflows/pr_integration_tests.yml @@ -20,7 +20,7 @@ jobs: (github.event.action != 'labeled' && (contains(github.event.pull_request.labels.*.name, 'ok-to-test') || contains(github.event.pull_request.labels.*.name, 'approved') || contains(github.event.pull_request.labels.*.name, 'lgtm'))) runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 with: # pull_request_target runs the workflow in the context of the base repo # as such actions/checkout needs to be explicit configured to retrieve @@ -30,7 +30,9 @@ jobs: - name: Set up QEMU uses: docker/setup-qemu-action@v1 - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v1 + uses: docker/setup-buildx-action@v2 + with: + install: true - name: Set up AWS SDK uses: aws-actions/configure-aws-credentials@v1 with: diff --git a/Makefile b/Makefile index e208ab3c451..96558f97dea 100644 --- a/Makefile +++ b/Makefile @@ -212,7 +212,7 @@ push-feature-server-python-aws-docker: docker push $(REGISTRY)/feature-server-python-aws:$$VERSION build-feature-server-python-aws-docker: - docker build --build-arg VERSION=$$VERSION \ + docker buildx --build-arg VERSION=$$VERSION \ -t $(REGISTRY)/feature-server-python-aws:$$VERSION \ -f sdk/python/feast/infra/feature_servers/aws_lambda/Dockerfile . diff --git a/infra/scripts/release/bump_file_versions.py b/infra/scripts/release/bump_file_versions.py index b706c2c141b..e17463c2c7b 100644 --- a/infra/scripts/release/bump_file_versions.py +++ b/infra/scripts/release/bump_file_versions.py @@ -33,46 +33,22 @@ def main() -> None: with open(path_to_file_list, "r") as f: files_to_bump = f.read().splitlines() - # The current version should be 0.18.0 or 0.19.0 or 0.20.0 etc, but we should also make sure to support the - # occasional patch release on the master branch like 0.18.1 or 0.18.2 - versions_in_files = 0 - if current_version[-2:] != ".0": - print(current_version[-2:]) - versions_in_files = count_version(current_version, files_to_bump, repo_root) - if versions_in_files != VERSIONS_TO_BUMP: - raise SystemExit(f"Found {versions_in_files} occurrences of {current_version} in files to bump, but " - f"expected {VERSIONS_TO_BUMP}") - else: - found = False - - # Lets make sure the files don't contain a patch version (e.g, 0.x.0 -> 0.x.20) - for patch_version in range(0, 20): - current_version_patch = current_version[:-1] + str(patch_version) - versions_in_files = count_version(current_version_patch, files_to_bump, repo_root) - - # We are using a patch version, let's change our version number - if versions_in_files == VERSIONS_TO_BUMP: - print(f"Found {versions_in_files} occurrences of {current_version_patch}, changing current version to " - f"{current_version_patch}") - current_version = current_version_patch - found = True - break - else: - print(f"Found {versions_in_files} occurrences of {current_version_patch}, instead of {VERSIONS_TO_BUMP}") - if not found: - raise SystemExit(f"Could not find {VERSIONS_TO_BUMP} versions of {current_version} in {path_to_file_list}") - - print(f"Found {versions_in_files} occurrences of {current_version} in files to bump {path_to_file_list}") + # The current version should be 0.18.0 or 0.19.0 or 0.20.0 etc + validate_files_to_bump(current_version, files_to_bump, repo_root) # Bump the version in the files updated_count = 0 for file in files_to_bump: - with open(repo_root.joinpath(file), "r") as f: - file_contents = f.read() - file_contents = file_contents.replace(current_version, new_version) - - with open(repo_root.joinpath(file), "w") as f: - f.write(file_contents) + components = file.split(" ") + file_path = components[0] + lines = components[1:] + with open(repo_root.joinpath(file_path), "r") as f: + file_contents = f.readlines() + for line in lines: + file_contents[int(line) - 1] = file_contents[int(line) - 1].replace(current_version, new_version) + + with open(repo_root.joinpath(file_path), "w") as f: + f.write(''.join(file_contents)) updated_count += 1 print(f"Updated {updated_count} files with new version {new_version}") @@ -88,14 +64,19 @@ def is_semantic_version(version: str) -> bool: return True -def count_version(current_version, files_to_bump, repo_root): - # Count how many of the existing versions we find - total = 0 +def validate_files_to_bump(current_version, files_to_bump, repo_root): for file in files_to_bump: - with open(repo_root.joinpath(file), "r") as f: - file_contents = f.read() - total += file_contents.count(current_version) - return total + components = file.split(" ") + assert len(components) > 1, f"Entry {file} should have a file name, and a list of line numbers with versions" + file_path = components[0] + lines = components[1:] + with open(repo_root.joinpath(file_path), "r") as f: + file_contents = f.readlines() + for line in lines: + assert current_version in file_contents[int(line) - 1], ( + f"File `{file_path}` line `{line}` didn't contain version {current_version}. " + f"Contents: {file_contents[int(line) - 1]}" + ) if __name__ == "__main__": diff --git a/infra/scripts/release/files_to_bump.txt b/infra/scripts/release/files_to_bump.txt index 2c3eece6bef..a1e2d29623a 100644 --- a/infra/scripts/release/files_to_bump.txt +++ b/infra/scripts/release/files_to_bump.txt @@ -1,13 +1,12 @@ -infra/charts/feast/requirements.yaml -infra/charts/feast/Chart.yaml -infra/charts/feast/charts/transformation-service/Chart.yaml -infra/charts/feast/charts/transformation-service/README.md -infra/charts/feast/charts/transformation-service/values.yaml -infra/charts/feast/charts/feature-server/Chart.yaml -infra/charts/feast/charts/feature-server/README.md -infra/charts/feast/charts/feature-server/values.yaml -infra/charts/feast/README.md -infra/charts/feast-python-server/Chart.yaml -infra/charts/feast-python-server/README.md -java/pom.xml -ui/package.json \ No newline at end of file +infra/charts/feast/requirements.yaml 4 9 +infra/charts/feast/Chart.yaml 4 +infra/charts/feast/charts/transformation-service/Chart.yaml 4 5 +infra/charts/feast/charts/transformation-service/README.md 3 16 +infra/charts/feast/charts/transformation-service/values.yaml 8 +infra/charts/feast/charts/feature-server/Chart.yaml 4 5 +infra/charts/feast/charts/feature-server/README.md 3 20 +infra/charts/feast/charts/feature-server/values.yaml 8 +infra/charts/feast/README.md 11 58 59 +infra/charts/feast-python-server/Chart.yaml 5 +infra/charts/feast-python-server/README.md 3 +java/pom.xml 41 diff --git a/sdk/python/feast/infra/feature_servers/aws_lambda/Dockerfile b/sdk/python/feast/infra/feature_servers/aws_lambda/Dockerfile index fe997afcae4..929227a8106 100644 --- a/sdk/python/feast/infra/feature_servers/aws_lambda/Dockerfile +++ b/sdk/python/feast/infra/feature_servers/aws_lambda/Dockerfile @@ -1,5 +1,8 @@ FROM public.ecr.aws/lambda/python:3.9 +RUN yum install -y git + + # Copy app handler code COPY sdk/python/feast/infra/feature_servers/aws_lambda/app.py ${LAMBDA_TASK_ROOT} @@ -16,7 +19,7 @@ COPY README.md README.md # git dir to infer the version of feast we're installing. # https://github.com/pypa/setuptools_scm#usage-from-docker # I think it also assumes that this dockerfile is being built from the root of the directory. -RUN SETUPTOOLS_SCM_PRETEND_VERSION=1 pip3 install --no-cache-dir -e '.[aws,redis]' +RUN --mount=source=.git,target=.git,type=bind pip3 install --no-cache-dir -e '.[aws,redis]' RUN pip3 install -r sdk/python/feast/infra/feature_servers/aws_lambda/requirements.txt --target "${LAMBDA_TASK_ROOT}" # Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile) diff --git a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/Dockerfile b/sdk/python/feast/infra/feature_servers/gcp_cloudrun/Dockerfile index e0e16f6a14f..6e3ff424eab 100644 --- a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/Dockerfile +++ b/sdk/python/feast/infra/feature_servers/gcp_cloudrun/Dockerfile @@ -1,5 +1,7 @@ FROM python:3.9-slim +RUN apt-get update && apt-get install -y git + # Allow statements and log messages to immediately appear in the Knative logs ENV PYTHONUNBUFFERED True @@ -18,7 +20,7 @@ COPY pyproject.toml pyproject.toml COPY README.md ./README.md # Install production dependencies. -RUN pip install --no-cache-dir \ +RUN --mount=source=.git,target=.git,type=bind pip install --no-cache-dir \ -e '.[gcp,redis]' \ -r ./sdk/python/feast/infra/feature_servers/gcp_cloudrun/requirements.txt diff --git a/sdk/python/feast/infra/transformation_servers/Dockerfile b/sdk/python/feast/infra/transformation_servers/Dockerfile index ff276c7a70d..5e77144757a 100644 --- a/sdk/python/feast/infra/transformation_servers/Dockerfile +++ b/sdk/python/feast/infra/transformation_servers/Dockerfile @@ -1,4 +1,6 @@ -FROM python:3.7-slim +FROM python:3.9-slim + +RUN apt-get update && apt-get install -y git # Copy app handler code COPY sdk/python/feast/infra/transformation_servers/app.py app.py @@ -13,7 +15,7 @@ COPY README.md README.md # Install dependencies -RUN pip3 install -e '.' +RUN --mount=source=.git,target=.git,type=bind pip3 install --no-cache-dir -e '.' # Start feature transformation server CMD [ "python", "app.py" ] From 2d56963cdb881753180f0c0f908ffc8a863174b6 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Mon, 16 May 2022 11:04:46 -0700 Subject: [PATCH 003/123] ci: Use buildx for docker builds when changes merged into master (#2703) * ci: Use buildx for docker builds when changes merged into master Signed-off-by: Achal Shah * use --load to load the built image into docker Signed-off-by: Achal Shah * buildx build smh Signed-off-by: Achal Shah --- .github/workflows/master_only.yml | 9 +++++++-- .github/workflows/pr_integration_tests.yml | 1 + Makefile | 8 ++++---- 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/.github/workflows/master_only.yml b/.github/workflows/master_only.yml index 2042987617b..4b23635a182 100644 --- a/.github/workflows/master_only.yml +++ b/.github/workflows/master_only.yml @@ -13,7 +13,9 @@ jobs: - name: Set up QEMU uses: docker/setup-qemu-action@v1 - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v1 + uses: docker/setup-buildx-action@v2 + with: + install: true - name: Set up AWS SDK uses: aws-actions/configure-aws-credentials@v1 with: @@ -50,6 +52,7 @@ jobs: docker build \ --file sdk/python/feast/infra/feature_servers/aws_lambda/Dockerfile \ --tag $ECR_REGISTRY/$ECR_REPOSITORY:${{ steps.image-tag.outputs.DOCKER_IMAGE_TAG }} \ + --load \ . docker push $ECR_REGISTRY/$ECR_REPOSITORY:${{ steps.image-tag.outputs.DOCKER_IMAGE_TAG }} outputs: @@ -177,7 +180,9 @@ jobs: - name: Set up QEMU uses: docker/setup-qemu-action@v1 - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v1 + uses: docker/setup-buildx-action@v2 + with: + install: true - name: Login to DockerHub uses: docker/login-action@v1 with: diff --git a/.github/workflows/pr_integration_tests.yml b/.github/workflows/pr_integration_tests.yml index c4417056136..a720865ed42 100644 --- a/.github/workflows/pr_integration_tests.yml +++ b/.github/workflows/pr_integration_tests.yml @@ -69,6 +69,7 @@ jobs: docker build \ --file sdk/python/feast/infra/feature_servers/aws_lambda/Dockerfile \ --tag $ECR_REGISTRY/$ECR_REPOSITORY:${{ steps.image-tag.outputs.DOCKER_IMAGE_TAG }} \ + --load \ . docker push $ECR_REGISTRY/$ECR_REPOSITORY:${{ steps.image-tag.outputs.DOCKER_IMAGE_TAG }} outputs: diff --git a/Makefile b/Makefile index 96558f97dea..876eb909ecc 100644 --- a/Makefile +++ b/Makefile @@ -212,17 +212,17 @@ push-feature-server-python-aws-docker: docker push $(REGISTRY)/feature-server-python-aws:$$VERSION build-feature-server-python-aws-docker: - docker buildx --build-arg VERSION=$$VERSION \ + docker buildx build --build-arg VERSION=$$VERSION \ -t $(REGISTRY)/feature-server-python-aws:$$VERSION \ - -f sdk/python/feast/infra/feature_servers/aws_lambda/Dockerfile . + -f sdk/python/feast/infra/feature_servers/aws_lambda/Dockerfile --load . push-feature-transformation-server-docker: docker push $(REGISTRY)/feature-transformation-server:$(VERSION) build-feature-transformation-server-docker: - docker build --build-arg VERSION=$(VERSION) \ + docker buildx build --build-arg VERSION=$(VERSION) \ -t $(REGISTRY)/feature-transformation-server:$(VERSION) \ - -f sdk/python/feast/infra/transformation_servers/Dockerfile . + -f sdk/python/feast/infra/transformation_servers/Dockerfile --load . push-feature-server-java-docker: docker push $(REGISTRY)/feature-server-java:$(VERSION) From 1c621fe3649900a59e85fe9c4f3840dd09bc88d0 Mon Sep 17 00:00:00 2001 From: Danny Chiao Date: Mon, 16 May 2022 15:24:52 -0400 Subject: [PATCH 004/123] fix: Fix issue when user specifies a port for feast ui (#2692) * fix: Fix issue when user specifies a port for feast ui Signed-off-by: Danny Chiao * fx Signed-off-by: Danny Chiao * fix python 3.9 version of importlib Signed-off-by: Danny Chiao * revert to pkg_resources Signed-off-by: Danny Chiao --- sdk/python/feast/ui/README.md | 18 ++++--------- sdk/python/feast/ui/public/projects-list.json | 11 ++++++++ sdk/python/feast/ui/src/index.tsx | 18 +++---------- sdk/python/feast/ui_server.py | 27 +++++++++---------- 4 files changed, 33 insertions(+), 41 deletions(-) create mode 100644 sdk/python/feast/ui/public/projects-list.json diff --git a/sdk/python/feast/ui/README.md b/sdk/python/feast/ui/README.md index ea5243ca4c6..0c11dcf134c 100644 --- a/sdk/python/feast/ui/README.md +++ b/sdk/python/feast/ui/README.md @@ -1,6 +1,6 @@ # Example Feast UI App -This is an example React App that imports the Feast UI module and relies on a "/projects-list" endpoint to get projects. +This is an example React App that imports the Feast UI module. See the module import in `src/index.js`. The main change this implements on top of a vanilla create-react-app is adding: @@ -11,23 +11,15 @@ import "@feast-dev/feast-ui/dist/feast-ui.css"; ReactDOM.render( - { - return res.json(); - }) - }} - /> + , document.getElementById("root") ); ``` -It is used by the `feast ui` command to scaffold a local UI server. The feast python package bundles in resources produced from `npm run build --omit=dev +It is used by the `feast ui` command to scaffold a local UI server. The feast python package bundles in resources produced from `npm run build --omit=dev.` + +The `feast ui` command will generate the necessary `projects-list.json` file and initialize it for the UI to read. **Note**: yarn start will not work on this because of the above dependency. diff --git a/sdk/python/feast/ui/public/projects-list.json b/sdk/python/feast/ui/public/projects-list.json new file mode 100644 index 00000000000..b6e79bda5b8 --- /dev/null +++ b/sdk/python/feast/ui/public/projects-list.json @@ -0,0 +1,11 @@ +{ + "projects": [ + { + "name": "Project", + "description": "Test project", + "id": "project_id", + "registryPath": "http://0.0.0.0:8888/registry" + } + ] + } + \ No newline at end of file diff --git a/sdk/python/feast/ui/src/index.tsx b/sdk/python/feast/ui/src/index.tsx index 4191de17a06..9ddacc9b48a 100644 --- a/sdk/python/feast/ui/src/index.tsx +++ b/sdk/python/feast/ui/src/index.tsx @@ -1,22 +1,12 @@ -import React from 'react'; +import React from "react"; import ReactDOM from "react-dom"; -import './index.css'; +import "./index.css"; import FeastUI from "@feast-dev/feast-ui"; import "@feast-dev/feast-ui/dist/feast-ui.css"; ReactDOM.render( - { - return res.json(); - }) - }} - /> + , document.getElementById("root") -); \ No newline at end of file +); diff --git a/sdk/python/feast/ui_server.py b/sdk/python/feast/ui_server.py index cc546f53716..5206dd5161e 100644 --- a/sdk/python/feast/ui_server.py +++ b/sdk/python/feast/ui_server.py @@ -16,9 +16,9 @@ def get_app( get_registry_dump: Callable, project_id: str, registry_ttl_secs: int, + host: str, + port: int, ): - ui_dir = pkg_resources.resource_filename(__name__, "ui/build/") - app = FastAPI() app.add_middleware( @@ -53,25 +53,24 @@ def shutdown_event(): async_refresh() - @app.get("/registry") - def read_registry(): - return json.loads(registry_json) - - # Generate projects-list json that points to the current repo's project - # TODO(adchia): Enable users to also add project name + description fields in feature_store.yaml - @app.get("/projects-list") - def projects_list(): - projects = { + ui_dir = pkg_resources.resource_filename(__name__, "ui/build/") + # Initialize with the projects-list.json file + with open(ui_dir + "projects-list.json", mode="w") as f: + projects_dict = { "projects": [ { "name": "Project", "description": "Test project", "id": project_id, - "registryPath": "http://0.0.0.0:8888/registry", + "registryPath": f"http://{host}:{port}/registry", } ] } - return projects + f.write(json.dumps(projects_dict)) + + @app.get("/registry") + def read_registry(): + return json.loads(registry_json) # For all other paths (such as paths that would otherwise be handled by react router), pass to React @app.api_route("/p/{path_name:path}", methods=["GET"]) @@ -98,5 +97,5 @@ def start_server( project_id: str, registry_ttl_sec: int, ): - app = get_app(store, get_registry_dump, project_id, registry_ttl_sec) + app = get_app(store, get_registry_dump, project_id, registry_ttl_sec, host, port) uvicorn.run(app, host=host, port=port) From 0d60eaa0b7f32c84eb015c814a3c879e7d4f50fd Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Mon, 16 May 2022 12:25:47 -0700 Subject: [PATCH 005/123] fix: Support push sources in stream feature views (#2704) Signed-off-by: Achal Shah --- sdk/python/feast/stream_feature_view.py | 5 +---- sdk/python/tests/unit/test_feature_views.py | 13 ++++++++++++- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/sdk/python/feast/stream_feature_view.py b/sdk/python/feast/stream_feature_view.py index 1c51b94a7cf..cfb3f63d7df 100644 --- a/sdk/python/feast/stream_feature_view.py +++ b/sdk/python/feast/stream_feature_view.py @@ -7,10 +7,7 @@ from feast.field import Field from feast.protos.feast.core.DataSource_pb2 import DataSource as DataSourceProto -SUPPORTED_STREAM_SOURCES = { - "KafkaSource", - "KinesisSource", -} +SUPPORTED_STREAM_SOURCES = {"KafkaSource", "KinesisSource", "PushSource"} class StreamFeatureView(FeatureView): diff --git a/sdk/python/tests/unit/test_feature_views.py b/sdk/python/tests/unit/test_feature_views.py index 8708a983c41..f72ae4fe9cb 100644 --- a/sdk/python/tests/unit/test_feature_views.py +++ b/sdk/python/tests/unit/test_feature_views.py @@ -2,6 +2,7 @@ import pytest +from feast import PushSource from feast.batch_feature_view import BatchFeatureView from feast.data_format import AvroFormat from feast.data_source import KafkaSource @@ -50,12 +51,22 @@ def test_create_stream_feature_view(): batch_source=FileSource(path="some path"), ) StreamFeatureView( - name="test batch feature view", + name="test kafka stream feature view", entities=[], ttl=timedelta(days=30), source=stream_source, ) + push_source = PushSource( + name="push source", batch_source=FileSource(path="some path") + ) + StreamFeatureView( + name="test push source feature view", + entities=[], + ttl=timedelta(days=30), + source=push_source, + ) + with pytest.raises(ValueError): StreamFeatureView( name="test batch feature view", entities=[], ttl=timedelta(days=30) From 5d6fa9408052695dfce179ea619d8362898d6329 Mon Sep 17 00:00:00 2001 From: Danny Chiao Date: Mon, 16 May 2022 16:03:51 -0400 Subject: [PATCH 006/123] fix: Add S3FS dependency (which Dask depends on for S3 files) (#2701) * fix: Add S3FS dependency (which Dask depends on for S3 files) Signed-off-by: Danny Chiao * fix requirements to prevent conflicting requirements Signed-off-by: Danny Chiao --- .../requirements/py3.10-ci-requirements.txt | 67 ++++++++++------- .../requirements/py3.10-requirements.txt | 32 +++----- .../requirements/py3.7-ci-requirements.txt | 68 ++++++++++------- .../requirements/py3.7-requirements.txt | 32 +++----- .../requirements/py3.8-ci-requirements.txt | 68 ++++++++++------- .../requirements/py3.8-requirements.txt | 32 +++----- .../requirements/py3.9-ci-requirements.txt | 74 +++++++++++-------- .../requirements/py3.9-requirements.txt | 32 +++----- setup.py | 5 +- 9 files changed, 213 insertions(+), 197 deletions(-) diff --git a/sdk/python/requirements/py3.10-ci-requirements.txt b/sdk/python/requirements/py3.10-ci-requirements.txt index 0655eb9c769..120f0d1158e 100644 --- a/sdk/python/requirements/py3.10-ci-requirements.txt +++ b/sdk/python/requirements/py3.10-ci-requirements.txt @@ -12,17 +12,23 @@ adal==1.2.7 # msrestazure adlfs==0.5.9 # via feast (setup.py) +aiobotocore==2.1.2 + # via s3fs aiohttp==3.8.1 # via # adlfs + # aiobotocore # gcsfs + # s3fs +aioitertools==0.10.0 + # via aiobotocore aiosignal==1.2.0 # via aiohttp alabaster==0.7.12 # via sphinx altair==4.2.0 # via great-expectations -anyio==3.5.0 +anyio==3.6.1 # via # starlette # watchgod @@ -52,7 +58,7 @@ attrs==21.4.0 # pytest avro==1.10.0 # via feast (setup.py) -azure-core==1.23.1 +azure-core==1.24.0 # via # adlfs # azure-identity @@ -61,7 +67,7 @@ azure-datalake-store==0.0.52 # via adlfs azure-identity==1.10.0 # via adlfs -azure-storage-blob==12.11.0 +azure-storage-blob==12.12.0 # via adlfs babel==2.10.1 # via sphinx @@ -69,12 +75,13 @@ backcall==0.2.0 # via ipython black==19.10b0 # via feast (setup.py) -boto3==1.22.8 +boto3==1.20.23 # via # feast (setup.py) # moto -botocore==1.25.8 +botocore==1.23.24 # via + # aiobotocore # boto3 # moto # s3transfer @@ -115,7 +122,7 @@ colorama==0.4.4 # via # feast (setup.py) # great-expectations -coverage[toml]==6.3.2 +coverage[toml]==6.3.3 # via pytest-cov cryptography==3.4.8 # via @@ -158,7 +165,7 @@ execnet==1.9.0 # via pytest-xdist executing==0.8.3 # via stack-data -fastapi==0.76.0 +fastapi==0.78.0 # via feast (setup.py) fastavro==1.4.11 # via @@ -166,7 +173,7 @@ fastavro==1.4.11 # pandavro fastjsonschema==2.15.3 # via nbformat -filelock==3.6.0 +filelock==3.7.0 # via virtualenv firebase-admin==4.5.2 # via feast (setup.py) @@ -176,12 +183,13 @@ frozenlist==1.3.0 # via # aiohttp # aiosignal -fsspec==2022.3.0 +fsspec==2022.1.0 # via # adlfs # dask # gcsfs -gcsfs==2022.3.0 + # s3fs +gcsfs==0.8.0 # via feast (setup.py) google-api-core[grpc]==1.31.5 # via @@ -227,7 +235,6 @@ google-cloud-storage==1.40.0 # via # feast (setup.py) # firebase-admin - # gcsfs google-crc32c==1.3.0 # via google-resumable-media google-resumable-media==1.3.3 @@ -241,7 +248,7 @@ googleapis-common-protos==1.52.0 # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) -grpcio==1.46.0 +grpcio==1.46.1 # via # feast (setup.py) # google-api-core @@ -249,7 +256,7 @@ grpcio==1.46.0 # grpcio-reflection # grpcio-testing # grpcio-tools -grpcio-reflection==1.46.0 +grpcio-reflection==1.46.1 # via feast (setup.py) grpcio-testing==1.44.0 # via feast (setup.py) @@ -296,7 +303,7 @@ jinja2==3.0.3 # great-expectations # moto # sphinx -jmespath==1.0.0 +jmespath==0.10.0 # via # boto3 # botocore @@ -330,7 +337,7 @@ mmh3==3.0.0 # via feast (setup.py) mock==2.0.0 # via feast (setup.py) -moto==3.1.7 +moto==3.1.9 # via feast (setup.py) msal==1.17.0 # via @@ -409,7 +416,7 @@ pexpect==4.8.0 # via ipython pickleshare==0.7.5 # via ipython -pip-tools==6.6.0 +pip-tools==6.6.1 # via feast (setup.py) platformdirs==2.5.2 # via virtualenv @@ -486,7 +493,7 @@ pygments==2.12.0 # via # ipython # sphinx -pyjwt[crypto]==2.3.0 +pyjwt[crypto]==2.4.0 # via # adal # msal @@ -591,6 +598,8 @@ rsa==4.8 # via google-auth ruamel-yaml==0.17.17 # via great-expectations +s3fs==2022.1.0 + # via feast (setup.py) s3transfer==0.5.2 # via boto3 scipy==1.8.0 @@ -639,17 +648,17 @@ sphinxcontrib-serializinghtml==1.1.5 # via sphinx stack-data==0.2.0 # via ipython -starlette==0.18.0 +starlette==0.19.1 # via fastapi tabulate==0.8.9 # via feast (setup.py) tenacity==8.0.1 # via feast (setup.py) -tensorflow-metadata==1.7.0 +tensorflow-metadata==1.8.0 # via feast (setup.py) termcolor==1.1.0 # via great-expectations -testcontainers==3.5.3 +testcontainers==3.5.4 # via feast (setup.py) thriftpy2==0.4.14 # via happybase @@ -674,7 +683,7 @@ tqdm==4.64.0 # via # feast (setup.py) # great-expectations -traitlets==5.1.1 +traitlets==5.2.1.post0 # via # ipython # jupyter-core @@ -684,23 +693,23 @@ trino==0.313.0 # via feast (setup.py) typed-ast==1.5.3 # via black -types-protobuf==3.19.18 +types-protobuf==3.19.20 # via # feast (setup.py) # mypy-protobuf -types-python-dateutil==2.8.14 +types-python-dateutil==2.8.15 # via feast (setup.py) -types-pytz==2021.3.7 +types-pytz==2021.3.8 # via feast (setup.py) types-pyyaml==6.0.7 # via feast (setup.py) -types-redis==4.2.1 +types-redis==4.2.3 # via feast (setup.py) types-requests==2.27.25 # via feast (setup.py) types-setuptools==57.4.14 # via feast (setup.py) -types-tabulate==0.8.8 +types-tabulate==0.8.9 # via feast (setup.py) types-urllib3==1.26.14 # via types-requests @@ -714,12 +723,13 @@ tzdata==2022.1 # via pytz-deprecation-shim tzlocal==4.2 # via great-expectations +ujson==5.2.0 + # via gcsfs uritemplate==4.1.1 # via google-api-python-client urllib3==1.26.9 # via # botocore - # feast (setup.py) # great-expectations # minio # requests @@ -744,9 +754,10 @@ wheel==0.37.1 # via pip-tools wrapt==1.14.1 # via + # aiobotocore # deprecated # testcontainers -xmltodict==0.12.0 +xmltodict==0.13.0 # via moto yarl==1.7.2 # via aiohttp diff --git a/sdk/python/requirements/py3.10-requirements.txt b/sdk/python/requirements/py3.10-requirements.txt index 369c3b8e60b..725b17f8caf 100644 --- a/sdk/python/requirements/py3.10-requirements.txt +++ b/sdk/python/requirements/py3.10-requirements.txt @@ -6,7 +6,7 @@ # absl-py==1.0.0 # via tensorflow-metadata -anyio==3.5.0 +anyio==3.6.1 # via # starlette # watchgod @@ -14,7 +14,7 @@ asgiref==3.5.1 # via uvicorn attrs==21.4.0 # via jsonschema -cachetools==4.2.4 +cachetools==5.1.0 # via google-auth certifi==2021.10.8 # via requests @@ -32,7 +32,7 @@ dask==2022.1.1 # via feast (setup.py) dill==0.3.4 # via feast (setup.py) -fastapi==0.76.0 +fastapi==0.78.0 # via feast (setup.py) fastavro==1.4.11 # via @@ -40,20 +40,20 @@ fastavro==1.4.11 # pandavro fsspec==2022.3.0 # via dask -google-api-core==1.31.5 +google-api-core==2.7.3 # via feast (setup.py) -google-auth==1.35.0 +google-auth==2.6.6 # via google-api-core googleapis-common-protos==1.52.0 # via # feast (setup.py) # google-api-core # tensorflow-metadata -grpcio==1.46.0 +grpcio==1.46.1 # via # feast (setup.py) # grpcio-reflection -grpcio-reflection==1.46.0 +grpcio-reflection==1.46.1 # via feast (setup.py) h11==0.13.0 # via uvicorn @@ -80,9 +80,7 @@ numpy==1.21.6 # pandavro # pyarrow packaging==21.3 - # via - # dask - # google-api-core + # via dask pandas==1.4.2 # via # feast (setup.py) @@ -113,7 +111,7 @@ pydantic==1.9.0 # via # fastapi # feast (setup.py) -pyparsing==3.0.8 +pyparsing==3.0.9 # via packaging pyrsistent==0.18.1 # via jsonschema @@ -122,9 +120,7 @@ python-dateutil==2.8.2 python-dotenv==0.20.0 # via uvicorn pytz==2022.1 - # via - # google-api-core - # pandas + # via pandas pyyaml==6.0 # via # dask @@ -137,20 +133,19 @@ rsa==4.8 six==1.16.0 # via # absl-py - # google-api-core # google-auth # grpcio # pandavro # python-dateutil sniffio==1.2.0 # via anyio -starlette==0.18.0 +starlette==0.19.1 # via fastapi tabulate==0.8.9 # via feast (setup.py) tenacity==8.0.1 # via feast (setup.py) -tensorflow-metadata==1.7.0 +tensorflow-metadata==1.8.0 # via feast (setup.py) toml==0.10.2 # via feast (setup.py) @@ -172,6 +167,3 @@ watchgod==0.8.2 # via uvicorn websockets==10.3 # via uvicorn - -# The following packages are considered to be unsafe in a requirements file: -# setuptools diff --git a/sdk/python/requirements/py3.7-ci-requirements.txt b/sdk/python/requirements/py3.7-ci-requirements.txt index 4232f5b2941..b445f86ea06 100644 --- a/sdk/python/requirements/py3.7-ci-requirements.txt +++ b/sdk/python/requirements/py3.7-ci-requirements.txt @@ -12,17 +12,23 @@ adal==1.2.7 # msrestazure adlfs==0.5.9 # via feast (setup.py) +aiobotocore==2.1.2 + # via s3fs aiohttp==3.8.1 # via # adlfs + # aiobotocore # gcsfs + # s3fs +aioitertools==0.10.0 + # via aiobotocore aiosignal==1.2.0 # via aiohttp alabaster==0.7.12 # via sphinx altair==4.2.0 # via great-expectations -anyio==3.5.0 +anyio==3.6.1 # via # starlette # watchgod @@ -52,7 +58,7 @@ attrs==21.4.0 # pytest avro==1.10.0 # via feast (setup.py) -azure-core==1.23.1 +azure-core==1.24.0 # via # adlfs # azure-identity @@ -61,7 +67,7 @@ azure-datalake-store==0.0.52 # via adlfs azure-identity==1.10.0 # via adlfs -azure-storage-blob==12.11.0 +azure-storage-blob==12.12.0 # via adlfs babel==2.10.1 # via sphinx @@ -73,12 +79,13 @@ backports-zoneinfo==0.2.1 # tzlocal black==19.10b0 # via feast (setup.py) -boto3==1.22.8 +boto3==1.20.23 # via # feast (setup.py) # moto -botocore==1.25.8 +botocore==1.23.24 # via + # aiobotocore # boto3 # moto # s3transfer @@ -119,7 +126,7 @@ colorama==0.4.4 # via # feast (setup.py) # great-expectations -coverage[toml]==6.3.2 +coverage[toml]==6.3.3 # via pytest-cov cryptography==3.4.8 # via @@ -160,7 +167,7 @@ entrypoints==0.4 # via altair execnet==1.9.0 # via pytest-xdist -fastapi==0.76.0 +fastapi==0.78.0 # via feast (setup.py) fastavro==1.4.11 # via @@ -168,7 +175,7 @@ fastavro==1.4.11 # pandavro fastjsonschema==2.15.3 # via nbformat -filelock==3.6.0 +filelock==3.7.0 # via virtualenv firebase-admin==4.5.2 # via feast (setup.py) @@ -178,12 +185,13 @@ frozenlist==1.3.0 # via # aiohttp # aiosignal -fsspec==2022.3.0 +fsspec==2022.1.0 # via # adlfs # dask # gcsfs -gcsfs==2022.3.0 + # s3fs +gcsfs==0.8.0 # via feast (setup.py) google-api-core[grpc]==1.31.5 # via @@ -229,7 +237,6 @@ google-cloud-storage==1.40.0 # via # feast (setup.py) # firebase-admin - # gcsfs google-crc32c==1.3.0 # via google-resumable-media google-resumable-media==1.3.3 @@ -243,7 +250,7 @@ googleapis-common-protos==1.52.0 # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) -grpcio==1.46.0 +grpcio==1.46.1 # via # feast (setup.py) # google-api-core @@ -251,7 +258,7 @@ grpcio==1.46.0 # grpcio-reflection # grpcio-testing # grpcio-tools -grpcio-reflection==1.46.0 +grpcio-reflection==1.46.1 # via feast (setup.py) grpcio-testing==1.44.0 # via feast (setup.py) @@ -312,7 +319,7 @@ jinja2==3.0.3 # great-expectations # moto # sphinx -jmespath==1.0.0 +jmespath==0.10.0 # via # boto3 # botocore @@ -346,7 +353,7 @@ mmh3==3.0.0 # via feast (setup.py) mock==2.0.0 # via feast (setup.py) -moto==3.1.7 +moto==3.1.9 # via feast (setup.py) msal==1.17.0 # via @@ -425,7 +432,7 @@ pexpect==4.8.0 # via ipython pickleshare==0.7.5 # via ipython -pip-tools==6.6.0 +pip-tools==6.6.1 # via feast (setup.py) platformdirs==2.5.2 # via virtualenv @@ -500,7 +507,7 @@ pygments==2.12.0 # via # ipython # sphinx -pyjwt[crypto]==2.3.0 +pyjwt[crypto]==2.4.0 # via # adal # msal @@ -607,6 +614,8 @@ ruamel-yaml==0.17.17 # via great-expectations ruamel-yaml-clib==0.2.6 # via ruamel-yaml +s3fs==2022.1.0 + # via feast (setup.py) s3transfer==0.5.2 # via boto3 scipy==1.7.3 @@ -653,17 +662,17 @@ sphinxcontrib-qthelp==1.0.3 # via sphinx sphinxcontrib-serializinghtml==1.1.5 # via sphinx -starlette==0.18.0 +starlette==0.19.1 # via fastapi tabulate==0.8.9 # via feast (setup.py) tenacity==8.0.1 # via feast (setup.py) -tensorflow-metadata==1.7.0 +tensorflow-metadata==1.8.0 # via feast (setup.py) termcolor==1.1.0 # via great-expectations -testcontainers==3.5.3 +testcontainers==3.5.4 # via feast (setup.py) thriftpy2==0.4.14 # via happybase @@ -688,7 +697,7 @@ tqdm==4.64.0 # via # feast (setup.py) # great-expectations -traitlets==5.1.1 +traitlets==5.2.1.post0 # via # ipython # jupyter-core @@ -700,29 +709,30 @@ typed-ast==1.5.3 # via # black # mypy -types-protobuf==3.19.18 +types-protobuf==3.19.20 # via # feast (setup.py) # mypy-protobuf -types-python-dateutil==2.8.14 +types-python-dateutil==2.8.15 # via feast (setup.py) -types-pytz==2021.3.7 +types-pytz==2021.3.8 # via feast (setup.py) types-pyyaml==6.0.7 # via feast (setup.py) -types-redis==4.2.1 +types-redis==4.2.3 # via feast (setup.py) types-requests==2.27.25 # via feast (setup.py) types-setuptools==57.4.14 # via feast (setup.py) -types-tabulate==0.8.8 +types-tabulate==0.8.9 # via feast (setup.py) types-urllib3==1.26.14 # via types-requests typing-extensions==4.2.0 # via # aiohttp + # aioitertools # anyio # asgiref # async-timeout @@ -741,12 +751,13 @@ tzdata==2022.1 # via pytz-deprecation-shim tzlocal==4.2 # via great-expectations +ujson==5.2.0 + # via gcsfs uritemplate==4.1.1 # via google-api-python-client urllib3==1.26.9 # via # botocore - # feast (setup.py) # great-expectations # minio # requests @@ -771,9 +782,10 @@ wheel==0.37.1 # via pip-tools wrapt==1.14.1 # via + # aiobotocore # deprecated # testcontainers -xmltodict==0.12.0 +xmltodict==0.13.0 # via moto yarl==1.7.2 # via aiohttp diff --git a/sdk/python/requirements/py3.7-requirements.txt b/sdk/python/requirements/py3.7-requirements.txt index dfdd86e39a1..b0e1511d9c6 100644 --- a/sdk/python/requirements/py3.7-requirements.txt +++ b/sdk/python/requirements/py3.7-requirements.txt @@ -6,7 +6,7 @@ # absl-py==1.0.0 # via tensorflow-metadata -anyio==3.5.0 +anyio==3.6.1 # via # starlette # watchgod @@ -14,7 +14,7 @@ asgiref==3.5.1 # via uvicorn attrs==21.4.0 # via jsonschema -cachetools==4.2.4 +cachetools==5.1.0 # via google-auth certifi==2021.10.8 # via requests @@ -32,7 +32,7 @@ dask==2022.1.1 # via feast (setup.py) dill==0.3.4 # via feast (setup.py) -fastapi==0.76.0 +fastapi==0.78.0 # via feast (setup.py) fastavro==1.4.11 # via @@ -40,20 +40,20 @@ fastavro==1.4.11 # pandavro fsspec==2022.3.0 # via dask -google-api-core==1.31.5 +google-api-core==2.7.3 # via feast (setup.py) -google-auth==1.35.0 +google-auth==2.6.6 # via google-api-core googleapis-common-protos==1.52.0 # via # feast (setup.py) # google-api-core # tensorflow-metadata -grpcio==1.46.0 +grpcio==1.46.1 # via # feast (setup.py) # grpcio-reflection -grpcio-reflection==1.46.0 +grpcio-reflection==1.46.1 # via feast (setup.py) h11==0.13.0 # via uvicorn @@ -86,9 +86,7 @@ numpy==1.21.6 # pandavro # pyarrow packaging==21.3 - # via - # dask - # google-api-core + # via dask pandas==1.3.5 # via # feast (setup.py) @@ -119,7 +117,7 @@ pydantic==1.9.0 # via # fastapi # feast (setup.py) -pyparsing==3.0.8 +pyparsing==3.0.9 # via packaging pyrsistent==0.18.1 # via jsonschema @@ -128,9 +126,7 @@ python-dateutil==2.8.2 python-dotenv==0.20.0 # via uvicorn pytz==2022.1 - # via - # google-api-core - # pandas + # via pandas pyyaml==6.0 # via # dask @@ -143,20 +139,19 @@ rsa==4.8 six==1.16.0 # via # absl-py - # google-api-core # google-auth # grpcio # pandavro # python-dateutil sniffio==1.2.0 # via anyio -starlette==0.18.0 +starlette==0.19.1 # via fastapi tabulate==0.8.9 # via feast (setup.py) tenacity==8.0.1 # via feast (setup.py) -tensorflow-metadata==1.7.0 +tensorflow-metadata==1.8.0 # via feast (setup.py) toml==0.10.2 # via feast (setup.py) @@ -190,6 +185,3 @@ zipp==3.8.0 # via # importlib-metadata # importlib-resources - -# The following packages are considered to be unsafe in a requirements file: -# setuptools diff --git a/sdk/python/requirements/py3.8-ci-requirements.txt b/sdk/python/requirements/py3.8-ci-requirements.txt index db3eca4b538..202f3ac2c71 100644 --- a/sdk/python/requirements/py3.8-ci-requirements.txt +++ b/sdk/python/requirements/py3.8-ci-requirements.txt @@ -12,17 +12,23 @@ adal==1.2.7 # msrestazure adlfs==0.5.9 # via feast (setup.py) +aiobotocore==2.1.2 + # via s3fs aiohttp==3.8.1 # via # adlfs + # aiobotocore # gcsfs + # s3fs +aioitertools==0.10.0 + # via aiobotocore aiosignal==1.2.0 # via aiohttp alabaster==0.7.12 # via sphinx altair==4.2.0 # via great-expectations -anyio==3.5.0 +anyio==3.6.1 # via # starlette # watchgod @@ -52,7 +58,7 @@ attrs==21.4.0 # pytest avro==1.10.0 # via feast (setup.py) -azure-core==1.23.1 +azure-core==1.24.0 # via # adlfs # azure-identity @@ -61,7 +67,7 @@ azure-datalake-store==0.0.52 # via adlfs azure-identity==1.10.0 # via adlfs -azure-storage-blob==12.11.0 +azure-storage-blob==12.12.0 # via adlfs babel==2.10.1 # via sphinx @@ -73,12 +79,13 @@ backports-zoneinfo==0.2.1 # tzlocal black==19.10b0 # via feast (setup.py) -boto3==1.22.8 +boto3==1.20.23 # via # feast (setup.py) # moto -botocore==1.25.8 +botocore==1.23.24 # via + # aiobotocore # boto3 # moto # s3transfer @@ -119,7 +126,7 @@ colorama==0.4.4 # via # feast (setup.py) # great-expectations -coverage[toml]==6.3.2 +coverage[toml]==6.3.3 # via pytest-cov cryptography==3.4.8 # via @@ -162,7 +169,7 @@ execnet==1.9.0 # via pytest-xdist executing==0.8.3 # via stack-data -fastapi==0.76.0 +fastapi==0.78.0 # via feast (setup.py) fastavro==1.4.11 # via @@ -170,7 +177,7 @@ fastavro==1.4.11 # pandavro fastjsonschema==2.15.3 # via nbformat -filelock==3.6.0 +filelock==3.7.0 # via virtualenv firebase-admin==4.5.2 # via feast (setup.py) @@ -180,12 +187,13 @@ frozenlist==1.3.0 # via # aiohttp # aiosignal -fsspec==2022.3.0 +fsspec==2022.1.0 # via # adlfs # dask # gcsfs -gcsfs==2022.3.0 + # s3fs +gcsfs==0.8.0 # via feast (setup.py) google-api-core[grpc]==1.31.5 # via @@ -231,7 +239,6 @@ google-cloud-storage==1.40.0 # via # feast (setup.py) # firebase-admin - # gcsfs google-crc32c==1.3.0 # via google-resumable-media google-resumable-media==1.3.3 @@ -245,7 +252,7 @@ googleapis-common-protos==1.52.0 # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) -grpcio==1.46.0 +grpcio==1.46.1 # via # feast (setup.py) # google-api-core @@ -253,7 +260,7 @@ grpcio==1.46.0 # grpcio-reflection # grpcio-testing # grpcio-tools -grpcio-reflection==1.46.0 +grpcio-reflection==1.46.1 # via feast (setup.py) grpcio-testing==1.44.0 # via feast (setup.py) @@ -302,7 +309,7 @@ jinja2==3.0.3 # great-expectations # moto # sphinx -jmespath==1.0.0 +jmespath==0.10.0 # via # boto3 # botocore @@ -336,7 +343,7 @@ mmh3==3.0.0 # via feast (setup.py) mock==2.0.0 # via feast (setup.py) -moto==3.1.7 +moto==3.1.9 # via feast (setup.py) msal==1.17.0 # via @@ -415,7 +422,7 @@ pexpect==4.8.0 # via ipython pickleshare==0.7.5 # via ipython -pip-tools==6.6.0 +pip-tools==6.6.1 # via feast (setup.py) platformdirs==2.5.2 # via virtualenv @@ -492,7 +499,7 @@ pygments==2.12.0 # via # ipython # sphinx -pyjwt[crypto]==2.3.0 +pyjwt[crypto]==2.4.0 # via # adal # msal @@ -599,6 +606,8 @@ ruamel-yaml==0.17.17 # via great-expectations ruamel-yaml-clib==0.2.6 # via ruamel-yaml +s3fs==2022.1.0 + # via feast (setup.py) s3transfer==0.5.2 # via boto3 scipy==1.8.0 @@ -647,17 +656,17 @@ sphinxcontrib-serializinghtml==1.1.5 # via sphinx stack-data==0.2.0 # via ipython -starlette==0.18.0 +starlette==0.19.1 # via fastapi tabulate==0.8.9 # via feast (setup.py) tenacity==8.0.1 # via feast (setup.py) -tensorflow-metadata==1.7.0 +tensorflow-metadata==1.8.0 # via feast (setup.py) termcolor==1.1.0 # via great-expectations -testcontainers==3.5.3 +testcontainers==3.5.4 # via feast (setup.py) thriftpy2==0.4.14 # via happybase @@ -682,7 +691,7 @@ tqdm==4.64.0 # via # feast (setup.py) # great-expectations -traitlets==5.1.1 +traitlets==5.2.1.post0 # via # ipython # jupyter-core @@ -692,28 +701,29 @@ trino==0.313.0 # via feast (setup.py) typed-ast==1.5.3 # via black -types-protobuf==3.19.18 +types-protobuf==3.19.20 # via # feast (setup.py) # mypy-protobuf -types-python-dateutil==2.8.14 +types-python-dateutil==2.8.15 # via feast (setup.py) -types-pytz==2021.3.7 +types-pytz==2021.3.8 # via feast (setup.py) types-pyyaml==6.0.7 # via feast (setup.py) -types-redis==4.2.1 +types-redis==4.2.3 # via feast (setup.py) types-requests==2.27.25 # via feast (setup.py) types-setuptools==57.4.14 # via feast (setup.py) -types-tabulate==0.8.8 +types-tabulate==0.8.9 # via feast (setup.py) types-urllib3==1.26.14 # via types-requests typing-extensions==4.2.0 # via + # aioitertools # azure-core # great-expectations # mypy @@ -723,12 +733,13 @@ tzdata==2022.1 # via pytz-deprecation-shim tzlocal==4.2 # via great-expectations +ujson==5.2.0 + # via gcsfs uritemplate==4.1.1 # via google-api-python-client urllib3==1.26.9 # via # botocore - # feast (setup.py) # great-expectations # minio # requests @@ -753,9 +764,10 @@ wheel==0.37.1 # via pip-tools wrapt==1.14.1 # via + # aiobotocore # deprecated # testcontainers -xmltodict==0.12.0 +xmltodict==0.13.0 # via moto yarl==1.7.2 # via aiohttp diff --git a/sdk/python/requirements/py3.8-requirements.txt b/sdk/python/requirements/py3.8-requirements.txt index 9e47d0e831d..98e1c6a76a9 100644 --- a/sdk/python/requirements/py3.8-requirements.txt +++ b/sdk/python/requirements/py3.8-requirements.txt @@ -6,7 +6,7 @@ # absl-py==1.0.0 # via tensorflow-metadata -anyio==3.5.0 +anyio==3.6.1 # via # starlette # watchgod @@ -14,7 +14,7 @@ asgiref==3.5.1 # via uvicorn attrs==21.4.0 # via jsonschema -cachetools==4.2.4 +cachetools==5.1.0 # via google-auth certifi==2021.10.8 # via requests @@ -32,7 +32,7 @@ dask==2022.1.1 # via feast (setup.py) dill==0.3.4 # via feast (setup.py) -fastapi==0.76.0 +fastapi==0.78.0 # via feast (setup.py) fastavro==1.4.11 # via @@ -40,20 +40,20 @@ fastavro==1.4.11 # pandavro fsspec==2022.3.0 # via dask -google-api-core==1.31.5 +google-api-core==2.7.3 # via feast (setup.py) -google-auth==1.35.0 +google-auth==2.6.6 # via google-api-core googleapis-common-protos==1.52.0 # via # feast (setup.py) # google-api-core # tensorflow-metadata -grpcio==1.46.0 +grpcio==1.46.1 # via # feast (setup.py) # grpcio-reflection -grpcio-reflection==1.46.0 +grpcio-reflection==1.46.1 # via feast (setup.py) h11==0.13.0 # via uvicorn @@ -82,9 +82,7 @@ numpy==1.21.6 # pandavro # pyarrow packaging==21.3 - # via - # dask - # google-api-core + # via dask pandas==1.4.2 # via # feast (setup.py) @@ -115,7 +113,7 @@ pydantic==1.9.0 # via # fastapi # feast (setup.py) -pyparsing==3.0.8 +pyparsing==3.0.9 # via packaging pyrsistent==0.18.1 # via jsonschema @@ -124,9 +122,7 @@ python-dateutil==2.8.2 python-dotenv==0.20.0 # via uvicorn pytz==2022.1 - # via - # google-api-core - # pandas + # via pandas pyyaml==6.0 # via # dask @@ -139,20 +135,19 @@ rsa==4.8 six==1.16.0 # via # absl-py - # google-api-core # google-auth # grpcio # pandavro # python-dateutil sniffio==1.2.0 # via anyio -starlette==0.18.0 +starlette==0.19.1 # via fastapi tabulate==0.8.9 # via feast (setup.py) tenacity==8.0.1 # via feast (setup.py) -tensorflow-metadata==1.7.0 +tensorflow-metadata==1.8.0 # via feast (setup.py) toml==0.10.2 # via feast (setup.py) @@ -178,6 +173,3 @@ websockets==10.3 # via uvicorn zipp==3.8.0 # via importlib-resources - -# The following packages are considered to be unsafe in a requirements file: -# setuptools diff --git a/sdk/python/requirements/py3.9-ci-requirements.txt b/sdk/python/requirements/py3.9-ci-requirements.txt index 4a96dc6fd2c..d3ecdc34bf3 100644 --- a/sdk/python/requirements/py3.9-ci-requirements.txt +++ b/sdk/python/requirements/py3.9-ci-requirements.txt @@ -12,17 +12,23 @@ adal==1.2.7 # msrestazure adlfs==0.5.9 # via feast (setup.py) +aiobotocore==2.1.2 + # via s3fs aiohttp==3.8.1 # via # adlfs + # aiobotocore # gcsfs + # s3fs +aioitertools==0.10.0 + # via aiobotocore aiosignal==1.2.0 # via aiohttp alabaster==0.7.12 # via sphinx altair==4.2.0 # via great-expectations -anyio==3.5.0 +anyio==3.6.1 # via # starlette # watchgod @@ -52,7 +58,7 @@ attrs==21.4.0 # pytest avro==1.10.0 # via feast (setup.py) -azure-core==1.23.1 +azure-core==1.24.0 # via # adlfs # azure-identity @@ -61,7 +67,7 @@ azure-datalake-store==0.0.52 # via adlfs azure-identity==1.10.0 # via adlfs -azure-storage-blob==12.11.0 +azure-storage-blob==12.12.0 # via adlfs babel==2.10.1 # via sphinx @@ -69,12 +75,13 @@ backcall==0.2.0 # via ipython black==19.10b0 # via feast (setup.py) -boto3==1.22.8 +boto3==1.20.23 # via # feast (setup.py) # moto -botocore==1.25.8 +botocore==1.23.24 # via + # aiobotocore # boto3 # moto # s3transfer @@ -115,7 +122,7 @@ colorama==0.4.4 # via # feast (setup.py) # great-expectations -coverage[toml]==6.3.2 +coverage[toml]==6.3.3 # via pytest-cov cryptography==3.4.8 # via @@ -158,7 +165,7 @@ execnet==1.9.0 # via pytest-xdist executing==0.8.3 # via stack-data -fastapi==0.76.0 +fastapi==0.78.0 # via feast (setup.py) fastavro==1.4.11 # via @@ -166,7 +173,7 @@ fastavro==1.4.11 # pandavro fastjsonschema==2.15.3 # via nbformat -filelock==3.6.0 +filelock==3.7.0 # via virtualenv firebase-admin==4.5.2 # via feast (setup.py) @@ -176,12 +183,13 @@ frozenlist==1.3.0 # via # aiohttp # aiosignal -fsspec==2022.3.0 +fsspec==2022.1.0 # via # adlfs # dask # gcsfs -gcsfs==2022.3.0 + # s3fs +gcsfs==0.8.0 # via feast (setup.py) google-api-core[grpc]==1.31.5 # via @@ -227,7 +235,6 @@ google-cloud-storage==1.40.0 # via # feast (setup.py) # firebase-admin - # gcsfs google-crc32c==1.3.0 # via google-resumable-media google-resumable-media==1.3.3 @@ -241,7 +248,7 @@ googleapis-common-protos==1.52.0 # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) -grpcio==1.46.0 +grpcio==1.46.1 # via # feast (setup.py) # google-api-core @@ -249,7 +256,7 @@ grpcio==1.46.0 # grpcio-reflection # grpcio-testing # grpcio-tools -grpcio-reflection==1.46.0 +grpcio-reflection==1.46.1 # via feast (setup.py) grpcio-testing==1.44.0 # via feast (setup.py) @@ -296,7 +303,7 @@ jinja2==3.0.3 # great-expectations # moto # sphinx -jmespath==1.0.0 +jmespath==0.10.0 # via # boto3 # botocore @@ -330,7 +337,7 @@ mmh3==3.0.0 # via feast (setup.py) mock==2.0.0 # via feast (setup.py) -moto==3.1.7 +moto==3.1.9 # via feast (setup.py) msal==1.17.0 # via @@ -409,7 +416,7 @@ pexpect==4.8.0 # via ipython pickleshare==0.7.5 # via ipython -pip-tools==6.6.0 +pip-tools==6.6.1 # via feast (setup.py) platformdirs==2.5.2 # via virtualenv @@ -486,7 +493,7 @@ pygments==2.12.0 # via # ipython # sphinx -pyjwt[crypto]==2.3.0 +pyjwt[crypto]==2.4.0 # via # adal # msal @@ -589,10 +596,12 @@ responses==0.20.0 # via moto rsa==4.8 # via google-auth -ruamel-yaml==0.17.17 +ruamel.yaml==0.17.17 # via great-expectations -ruamel-yaml-clib==0.2.6 - # via ruamel-yaml +ruamel.yaml.clib==0.2.6 + # via ruamel.yaml +s3fs==2022.1.0 + # via feast (setup.py) s3transfer==0.5.2 # via boto3 scipy==1.8.0 @@ -641,17 +650,17 @@ sphinxcontrib-serializinghtml==1.1.5 # via sphinx stack-data==0.2.0 # via ipython -starlette==0.18.0 +starlette==0.19.1 # via fastapi tabulate==0.8.9 # via feast (setup.py) tenacity==8.0.1 # via feast (setup.py) -tensorflow-metadata==1.7.0 +tensorflow-metadata==1.8.0 # via feast (setup.py) termcolor==1.1.0 # via great-expectations -testcontainers==3.5.3 +testcontainers==3.5.4 # via feast (setup.py) thriftpy2==0.4.14 # via happybase @@ -676,7 +685,7 @@ tqdm==4.64.0 # via # feast (setup.py) # great-expectations -traitlets==5.1.1 +traitlets==5.2.1.post0 # via # ipython # jupyter-core @@ -686,28 +695,29 @@ trino==0.313.0 # via feast (setup.py) typed-ast==1.5.3 # via black -types-protobuf==3.19.18 +types-protobuf==3.19.20 # via # feast (setup.py) # mypy-protobuf -types-python-dateutil==2.8.14 +types-python-dateutil==2.8.15 # via feast (setup.py) -types-pytz==2021.3.7 +types-pytz==2021.3.8 # via feast (setup.py) types-pyyaml==6.0.7 # via feast (setup.py) -types-redis==4.2.1 +types-redis==4.2.3 # via feast (setup.py) types-requests==2.27.25 # via feast (setup.py) types-setuptools==57.4.14 # via feast (setup.py) -types-tabulate==0.8.8 +types-tabulate==0.8.9 # via feast (setup.py) types-urllib3==1.26.14 # via types-requests typing-extensions==4.2.0 # via + # aioitertools # azure-core # great-expectations # mypy @@ -717,12 +727,13 @@ tzdata==2022.1 # via pytz-deprecation-shim tzlocal==4.2 # via great-expectations +ujson==5.2.0 + # via gcsfs uritemplate==4.1.1 # via google-api-python-client urllib3==1.26.9 # via # botocore - # feast (setup.py) # great-expectations # minio # requests @@ -747,9 +758,10 @@ wheel==0.37.1 # via pip-tools wrapt==1.14.1 # via + # aiobotocore # deprecated # testcontainers -xmltodict==0.12.0 +xmltodict==0.13.0 # via moto yarl==1.7.2 # via aiohttp diff --git a/sdk/python/requirements/py3.9-requirements.txt b/sdk/python/requirements/py3.9-requirements.txt index 0ad985b828e..3eded689a58 100644 --- a/sdk/python/requirements/py3.9-requirements.txt +++ b/sdk/python/requirements/py3.9-requirements.txt @@ -6,7 +6,7 @@ # absl-py==1.0.0 # via tensorflow-metadata -anyio==3.5.0 +anyio==3.6.1 # via # starlette # watchgod @@ -14,7 +14,7 @@ asgiref==3.5.1 # via uvicorn attrs==21.4.0 # via jsonschema -cachetools==4.2.4 +cachetools==5.1.0 # via google-auth certifi==2021.10.8 # via requests @@ -32,7 +32,7 @@ dask==2022.1.1 # via feast (setup.py) dill==0.3.4 # via feast (setup.py) -fastapi==0.76.0 +fastapi==0.78.0 # via feast (setup.py) fastavro==1.4.11 # via @@ -40,20 +40,20 @@ fastavro==1.4.11 # pandavro fsspec==2022.3.0 # via dask -google-api-core==1.31.5 +google-api-core==2.7.3 # via feast (setup.py) -google-auth==1.35.0 +google-auth==2.6.6 # via google-api-core googleapis-common-protos==1.52.0 # via # feast (setup.py) # google-api-core # tensorflow-metadata -grpcio==1.46.0 +grpcio==1.46.1 # via # feast (setup.py) # grpcio-reflection -grpcio-reflection==1.46.0 +grpcio-reflection==1.46.1 # via feast (setup.py) h11==0.13.0 # via uvicorn @@ -80,9 +80,7 @@ numpy==1.21.6 # pandavro # pyarrow packaging==21.3 - # via - # dask - # google-api-core + # via dask pandas==1.4.2 # via # feast (setup.py) @@ -113,7 +111,7 @@ pydantic==1.9.0 # via # fastapi # feast (setup.py) -pyparsing==3.0.8 +pyparsing==3.0.9 # via packaging pyrsistent==0.18.1 # via jsonschema @@ -122,9 +120,7 @@ python-dateutil==2.8.2 python-dotenv==0.20.0 # via uvicorn pytz==2022.1 - # via - # google-api-core - # pandas + # via pandas pyyaml==6.0 # via # dask @@ -137,20 +133,19 @@ rsa==4.8 six==1.16.0 # via # absl-py - # google-api-core # google-auth # grpcio # pandavro # python-dateutil sniffio==1.2.0 # via anyio -starlette==0.18.0 +starlette==0.19.1 # via fastapi tabulate==0.8.9 # via feast (setup.py) tenacity==8.0.1 # via feast (setup.py) -tensorflow-metadata==1.7.0 +tensorflow-metadata==1.8.0 # via feast (setup.py) toml==0.10.2 # via feast (setup.py) @@ -174,6 +169,3 @@ watchgod==0.8.2 # via uvicorn websockets==10.3 # via uvicorn - -# The following packages are considered to be unsafe in a requirements file: -# setuptools diff --git a/setup.py b/setup.py index 2044de008fc..f5f092e3f50 100644 --- a/setup.py +++ b/setup.py @@ -90,8 +90,9 @@ ] AWS_REQUIRED = [ - "boto3>=1.17.0", + "boto3>=1.17.0,<=1.20.23", "docker>=5.0.2", + "s3fs>=0.4.0,<=2022.01.0" ] SNOWFLAKE_REQUIRED = [ @@ -135,7 +136,7 @@ "mypy==0.931", "mypy-protobuf==3.1", "avro==1.10.0", - "gcsfs", + "gcsfs>=0.4.0,<=2022.01.0" "urllib3>=1.25.4,<2", "psutil==5.9.0", "pytest>=6.0.0,<8", From 095dfd6a385a5f45585f25cd6e26c18a76e592e7 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Mon, 16 May 2022 13:21:48 -0700 Subject: [PATCH 007/123] ci: Use buildx for docker builds in remaining places as well (#2705) * ci: Use buildx for docker builds in remaining places as well Signed-off-by: Achal Shah * fix oops Signed-off-by: Achal Shah --- CONTRIBUTING.md | 3 ++- Makefile | 6 +++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9cce520d346..4bd14d762a5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -75,7 +75,8 @@ Note that this means if you are midway through working through a PR and rebase, ## Feast Python SDK / CLI ### Environment Setup Setting up your development environment for Feast Python SDK / CLI: -1. Ensure that you have Docker installed in your environment. Docker is used to provision service dependencies during testing. +1. Ensure that you have Docker installed in your environment. Docker is used to provision service dependencies during testing, and build images for feature servers and other components. + 1. Please note that we use [Docker with BuiltKit](https://docs.docker.com/develop/develop-images/build_enhancements/). 2. Ensure that you have `make`, Python (3.7 and above) with `pip`, installed. 3. _Recommended:_ Create a virtual environment to isolate development dependencies to be installed ```sh diff --git a/Makefile b/Makefile index 876eb909ecc..5dfbaba53bd 100644 --- a/Makefile +++ b/Makefile @@ -206,7 +206,7 @@ push-ci-docker: # TODO(adchia): consider removing. This doesn't run successfully right now build-ci-docker: - docker build -t $(REGISTRY)/feast-ci:$(VERSION) -f infra/docker/ci/Dockerfile . + docker buildx build -t $(REGISTRY)/feast-ci:$(VERSION) -f infra/docker/ci/Dockerfile --load . push-feature-server-python-aws-docker: docker push $(REGISTRY)/feature-server-python-aws:$$VERSION @@ -228,9 +228,9 @@ push-feature-server-java-docker: docker push $(REGISTRY)/feature-server-java:$(VERSION) build-feature-server-java-docker: - docker build --build-arg VERSION=$(VERSION) \ + docker buildx build --build-arg VERSION=$(VERSION) \ -t $(REGISTRY)/feature-server-java:$(VERSION) \ - -f java/infra/docker/feature-server/Dockerfile . + -f java/infra/docker/feature-server/Dockerfile --load . # Documentation From 7583a0b1840c663af50bf4382c3ff2368593bb9c Mon Sep 17 00:00:00 2001 From: Oleksii Moskalenko Date: Mon, 16 May 2022 13:22:46 -0700 Subject: [PATCH 008/123] fix: Python server is not correctly starting in integration tests (#2706) Signed-off-by: Oleksii Moskalenko --- sdk/python/tests/conftest.py | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/sdk/python/tests/conftest.py b/sdk/python/tests/conftest.py index d492c7ba845..092f5a07873 100644 --- a/sdk/python/tests/conftest.py +++ b/sdk/python/tests/conftest.py @@ -274,10 +274,11 @@ def pytest_generate_tests(metafunc: pytest.Metafunc): @pytest.fixture(scope="session") def python_server(environment): + assert not _check_port_open("localhost", environment.get_local_server_port()) + proc = Process( target=start_test_local_server, args=(environment.feature_store.repo_path, environment.get_local_server_port()), - daemon=True, ) if ( environment.python_feature_server @@ -298,6 +299,15 @@ def python_server(environment): if proc.is_alive(): proc.kill() + # wait server to free the port + wait_retry_backoff( + lambda: ( + None, + not _check_port_open("localhost", environment.get_local_server_port()), + ), + timeout_secs=30, + ) + def _check_port_open(host, port) -> bool: with closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as sock: From af57a89df18614679371180cc0d65fd2bd56ef1a Mon Sep 17 00:00:00 2001 From: Oleksii Moskalenko Date: Mon, 16 May 2022 13:43:44 -0700 Subject: [PATCH 009/123] chore: Build python protos inplace instead of build directory (for development) (#2707) Signed-off-by: Oleksii Moskalenko --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 5dfbaba53bd..a715cb2c4a8 100644 --- a/Makefile +++ b/Makefile @@ -47,7 +47,7 @@ package-protos: cp -r ${ROOT_DIR}/protos ${ROOT_DIR}/sdk/python/feast/protos compile-protos-python: - python setup.py build_python_protos + python setup.py build_python_protos --inplace install-python: python -m piptools sync sdk/python/requirements/py$(PYTHON)-requirements.txt From 2874fc5c85810a65f750377d34418c71e747110e Mon Sep 17 00:00:00 2001 From: Oleksii Moskalenko Date: Mon, 16 May 2022 16:17:48 -0700 Subject: [PATCH 010/123] feat: Validating logged features via Python SDK (#2640) * simple logged feature validation Signed-off-by: Oleksii Moskalenko * validate with metadata Signed-off-by: Oleksii Moskalenko * typos Signed-off-by: Oleksii Moskalenko * revert entity columns Signed-off-by: Oleksii Moskalenko --- sdk/python/feast/dqm/profilers/ge_profiler.py | 38 ++++- sdk/python/feast/feature_logging.py | 5 + sdk/python/feast/feature_store.py | 55 +++++++- sdk/python/feast/feature_view.py | 1 + sdk/python/feast/infra/offline_stores/file.py | 16 ++- .../tests/integration/e2e/test_validation.py | 130 ++++++++++++++++++ .../offline_store/test_feature_logging.py | 62 ++------- sdk/python/tests/utils/logged_features.py | 67 +++++++++ 8 files changed, 312 insertions(+), 62 deletions(-) create mode 100644 sdk/python/tests/utils/logged_features.py diff --git a/sdk/python/feast/dqm/profilers/ge_profiler.py b/sdk/python/feast/dqm/profilers/ge_profiler.py index f1780754de3..93c8b7d5de8 100644 --- a/sdk/python/feast/dqm/profilers/ge_profiler.py +++ b/sdk/python/feast/dqm/profilers/ge_profiler.py @@ -21,6 +21,7 @@ from feast.protos.feast.core.ValidationProfile_pb2 import ( GEValidationProfiler as GEValidationProfilerProto, ) +from feast.protos.feast.serving.ServingService_pb2 import FieldStatus def _prepare_dataset(dataset: PandasDataset) -> PandasDataset: @@ -41,6 +42,23 @@ def _prepare_dataset(dataset: PandasDataset) -> PandasDataset: return dataset_copy +def _add_feature_metadata(dataset: PandasDataset) -> PandasDataset: + for column in dataset.columns: + if "__" not in column: + # not a feature column + continue + + if "event_timestamp" in dataset.columns: + dataset[f"{column}__timestamp"] = dataset["event_timestamp"] + + dataset[f"{column}__status"] = FieldStatus.PRESENT + dataset[f"{column}__status"] = dataset[f"{column}__status"].mask( + dataset[column].isna(), FieldStatus.NOT_FOUND + ) + + return dataset + + class GEProfile(Profile): """ GEProfile is an implementation of abstract Profile for integration with Great Expectations. @@ -96,9 +114,12 @@ class GEProfiler(Profiler): """ def __init__( - self, user_defined_profiler: Callable[[pd.DataFrame], ExpectationSuite] + self, + user_defined_profiler: Callable[[pd.DataFrame], ExpectationSuite], + with_feature_metadata: bool = False, ): self.user_defined_profiler = user_defined_profiler + self.with_feature_metadata = with_feature_metadata def analyze_dataset(self, df: pd.DataFrame) -> Profile: """ @@ -113,6 +134,9 @@ def analyze_dataset(self, df: pd.DataFrame) -> Profile: dataset = _prepare_dataset(dataset) + if self.with_feature_metadata: + dataset = _add_feature_metadata(dataset) + return GEProfile(expectation_suite=self.user_defined_profiler(dataset)) def to_proto(self): @@ -158,5 +182,13 @@ def __repr__(self): return json.dumps(failed_expectations, indent=2) -def ge_profiler(func): - return GEProfiler(user_defined_profiler=func) +def ge_profiler(*args, with_feature_metadata=False): + def wrapper(fun): + return GEProfiler( + user_defined_profiler=fun, with_feature_metadata=with_feature_metadata + ) + + if args: + return wrapper(args[0]) + + return wrapper diff --git a/sdk/python/feast/feature_logging.py b/sdk/python/feast/feature_logging.py index e2982988366..b2d77f68fef 100644 --- a/sdk/python/feast/feature_logging.py +++ b/sdk/python/feast/feature_logging.py @@ -11,6 +11,7 @@ FeatureViewNotFoundException, OnDemandFeatureViewNotFoundException, ) +from feast.feature_view import DUMMY_ENTITY_NAME from feast.protos.feast.core.FeatureService_pb2 import ( LoggingConfig as LoggingConfigProto, ) @@ -77,7 +78,11 @@ def get_schema(self, registry: "Registry") -> pa.Schema: else: for entity_name in feature_view.entities: + if entity_name == DUMMY_ENTITY_NAME: + continue + entity = registry.get_entity(entity_name, self._project) + join_key = projection.join_key_map.get( entity.join_key, entity.join_key ) diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py index 73af4741ef3..791f1ae40e5 100644 --- a/sdk/python/feast/feature_store.py +++ b/sdk/python/feast/feature_store.py @@ -45,6 +45,7 @@ from feast.data_source import DataSource from feast.diff.infra_diff import InfraDiff, diff_infra_protos from feast.diff.registry_diff import RegistryDiff, apply_diff_to_registry, diff_between +from feast.dqm.errors import ValidationFailed from feast.entity import Entity from feast.errors import ( EntityNotFoundException, @@ -83,7 +84,7 @@ from feast.repo_config import RepoConfig, load_repo_config from feast.repo_contents import RepoContents from feast.request_feature_view import RequestFeatureView -from feast.saved_dataset import SavedDataset, SavedDatasetStorage +from feast.saved_dataset import SavedDataset, SavedDatasetStorage, ValidationReference from feast.type_map import ( feast_value_type_to_python_type, python_values_to_proto_values, @@ -2054,6 +2055,58 @@ def write_logged_features( registry=self._registry, ) + def validate_logged_features( + self, + source: Union[FeatureService], + start: datetime, + end: datetime, + reference: ValidationReference, + throw_exception: bool = True, + ) -> Optional[ValidationFailed]: + """ + Load logged features from an offline store and validate them against provided validation reference. + + Args: + source: Logs source object (currently only feature services are supported) + start: lower bound for loading logged features + end: upper bound for loading logged features + reference: validation reference + throw_exception: throw exception or return it as a result + + Returns: + Throw or return (depends on parameter) ValidationFailed exception if validation was not successful + or None if successful. + + """ + warnings.warn( + "Logged features validation is an experimental feature. " + "This API is unstable and it could and most probably will be changed in the future. " + "We do not guarantee that future changes will maintain backward compatibility.", + RuntimeWarning, + ) + + if not isinstance(source, FeatureService): + raise ValueError("Only feature service is currently supported as a source") + + j = self._get_provider().retrieve_feature_service_logs( + feature_service=source, + start_date=start, + end_date=end, + config=self.config, + registry=self.registry, + ) + + # read and run validation + try: + j.to_arrow(validation_reference=reference) + except ValidationFailed as exc: + if throw_exception: + raise + + return exc + + return None + def _validate_entity_values(join_key_values: Dict[str, List[Value]]): set_of_row_lengths = {len(v) for v in join_key_values.values()} diff --git a/sdk/python/feast/feature_view.py b/sdk/python/feast/feature_view.py index 4ff7618f2e1..6a8fd332174 100644 --- a/sdk/python/feast/feature_view.py +++ b/sdk/python/feast/feature_view.py @@ -285,6 +285,7 @@ def __copy__(self): online=self.online, ) fv.projection = copy.copy(self.projection) + fv.entities = self.entities return fv def __eq__(self, other): diff --git a/sdk/python/feast/infra/offline_stores/file.py b/sdk/python/feast/infra/offline_stores/file.py index f36eb383d8e..7670ad67b77 100644 --- a/sdk/python/feast/infra/offline_stores/file.py +++ b/sdk/python/feast/infra/offline_stores/file.py @@ -73,6 +73,7 @@ def on_demand_feature_views(self) -> Optional[List[OnDemandFeatureView]]: def _to_df_internal(self) -> pd.DataFrame: # Only execute the evaluation function to build the final historical retrieval dataframe at the last moment. df = self.evaluation_function().compute() + df = df.reset_index(drop=True) return df @log_exceptions_and_usage @@ -555,11 +556,18 @@ def _filter_ttl( # Filter rows by defined timestamp tolerance if feature_view.ttl and feature_view.ttl.total_seconds() != 0: df_to_join = df_to_join[ - ( - df_to_join[timestamp_field] - >= df_to_join[entity_df_event_timestamp_col] - feature_view.ttl + # do not drop entity rows if one of the sources returns NaNs + df_to_join[timestamp_field].isna() + | ( + ( + df_to_join[timestamp_field] + >= df_to_join[entity_df_event_timestamp_col] - feature_view.ttl + ) + & ( + df_to_join[timestamp_field] + <= df_to_join[entity_df_event_timestamp_col] + ) ) - & (df_to_join[timestamp_field] <= df_to_join[entity_df_event_timestamp_col]) ] df_to_join = df_to_join.persist() diff --git a/sdk/python/tests/integration/e2e/test_validation.py b/sdk/python/tests/integration/e2e/test_validation.py index e434f1a133f..b78a8bde093 100644 --- a/sdk/python/tests/integration/e2e/test_validation.py +++ b/sdk/python/tests/integration/e2e/test_validation.py @@ -1,10 +1,21 @@ +import datetime + import pandas as pd +import pyarrow as pa import pytest from great_expectations.core import ExpectationSuite from great_expectations.dataset import PandasDataset +from feast import FeatureService from feast.dqm.errors import ValidationFailed from feast.dqm.profilers.ge_profiler import ge_profiler +from feast.feature_logging import ( + LOG_TIMESTAMP_FIELD, + FeatureServiceLoggingSource, + LoggingConfig, +) +from feast.protos.feast.serving.ServingService_pb2 import FieldStatus +from feast.wait import wait_retry_backoff from tests.integration.feature_repos.repo_configuration import ( construct_universal_feature_views, ) @@ -13,6 +24,7 @@ driver, location, ) +from tests.utils.logged_features import prepare_logs _features = [ "customer_profile:current_balance", @@ -32,6 +44,39 @@ def configurable_profiler(dataset: PandasDataset) -> ExpectationSuite: return UserConfigurableProfiler( profile_dataset=dataset, + ignored_columns=["event_timestamp"], + excluded_expectations=[ + "expect_table_columns_to_match_ordered_list", + "expect_table_row_count_to_be_between", + ], + value_set_threshold="few", + ).build_suite() + + +@ge_profiler(with_feature_metadata=True) +def profiler_with_feature_metadata(dataset: PandasDataset) -> ExpectationSuite: + from great_expectations.profile.user_configurable_profiler import ( + UserConfigurableProfiler, + ) + + # always present + dataset.expect_column_values_to_be_in_set( + "global_stats__avg_ride_length__status", {FieldStatus.PRESENT} + ) + + # present at least in 70% of rows + dataset.expect_column_values_to_be_in_set( + "customer_profile__current_balance__status", {FieldStatus.PRESENT}, mostly=0.7 + ) + + return UserConfigurableProfiler( + profile_dataset=dataset, + ignored_columns=["event_timestamp"] + + [ + c + for c in dataset.columns + if c.endswith("__timestamp") or c.endswith("__status") + ], excluded_expectations=[ "expect_table_columns_to_match_ordered_list", "expect_table_row_count_to_be_between", @@ -127,3 +172,88 @@ def test_historical_retrieval_fails_on_validation(environment, universal_data_so assert failed_expectations[1].check_name == "expect_column_values_to_be_in_set" assert failed_expectations[1].column_name == "avg_passenger_count" + + +@pytest.mark.integration +def test_logged_features_validation(environment, universal_data_sources): + store = environment.feature_store + + (_, datasets, data_sources) = universal_data_sources + feature_views = construct_universal_feature_views(data_sources) + feature_service = FeatureService( + name="test_service", + features=[ + feature_views.customer[ + ["current_balance", "avg_passenger_count", "lifetime_trip_count"] + ], + feature_views.order[["order_is_success"]], + feature_views.global_fv[["num_rides", "avg_ride_length"]], + ], + logging_config=LoggingConfig( + destination=environment.data_source_creator.create_logged_features_destination() + ), + ) + + store.apply( + [driver(), customer(), location(), feature_service, *feature_views.values()] + ) + + entity_df = datasets.entity_df.drop( + columns=["order_id", "origin_id", "destination_id"] + ) + + # add some non-existing entities to check NotFound feature handling + for i in range(5): + entity_df = entity_df.append( + { + "customer_id": 2000 + i, + "driver_id": 6000 + i, + "event_timestamp": datetime.datetime.now(), + }, + ignore_index=True, + ) + + reference_dataset = store.create_saved_dataset( + from_=store.get_historical_features( + entity_df=entity_df, features=feature_service, full_feature_names=True + ), + name="reference_for_validating_logged_features", + storage=environment.data_source_creator.create_saved_dataset_destination(), + ) + + log_source_df = store.get_historical_features( + entity_df=entity_df, features=feature_service, full_feature_names=False + ).to_df() + logs_df = prepare_logs(log_source_df, feature_service, store) + + schema = FeatureServiceLoggingSource( + feature_service=feature_service, project=store.project + ).get_schema(store._registry) + store.write_logged_features( + pa.Table.from_pandas(logs_df, schema=schema), source=feature_service + ) + + def validate(): + """ + Return Tuple[succeed, completed] + Succeed will be True if no ValidateFailed exception was raised + """ + try: + store.validate_logged_features( + feature_service, + start=logs_df[LOG_TIMESTAMP_FIELD].min(), + end=logs_df[LOG_TIMESTAMP_FIELD].max() + datetime.timedelta(seconds=1), + reference=reference_dataset.as_reference( + profiler=profiler_with_feature_metadata + ), + ) + except ValidationFailed: + return False, True + except Exception: + # log table is still being created + return False, False + + return True, True + + success = wait_retry_backoff(validate, timeout_secs=30) + assert success, "Validation failed (unexpectedly)" diff --git a/sdk/python/tests/integration/offline_store/test_feature_logging.py b/sdk/python/tests/integration/offline_store/test_feature_logging.py index 8e7e9d68bea..24c65eac020 100644 --- a/sdk/python/tests/integration/offline_store/test_feature_logging.py +++ b/sdk/python/tests/integration/offline_store/test_feature_logging.py @@ -1,16 +1,8 @@ -import contextlib import datetime -import tempfile -import uuid -from pathlib import Path -from typing import Iterator, Union -import numpy as np import pandas as pd -import pyarrow import pyarrow as pa import pytest -from google.api_core.exceptions import NotFound from feast.feature_logging import ( LOG_DATE_FIELD, @@ -20,14 +12,13 @@ LoggingConfig, ) from feast.feature_service import FeatureService -from feast.protos.feast.serving.ServingService_pb2 import FieldStatus from feast.wait import wait_retry_backoff from tests.integration.feature_repos.repo_configuration import ( - UniversalDatasets, construct_universal_feature_views, ) from tests.integration.feature_repos.universal.entities import driver from tests.integration.feature_repos.universal.feature_views import conv_rate_plus_100 +from tests.utils.logged_features import prepare_logs, to_logs_dataset @pytest.mark.integration @@ -41,8 +32,6 @@ def test_feature_service_logging(environment, universal_data_sources, pass_as_pa feature_views = construct_universal_feature_views(data_sources) store.apply([driver(), *feature_views.values()]) - logs_df = prepare_logs(datasets) - feature_service = FeatureService( name="test_service", features=[ @@ -56,6 +45,12 @@ def test_feature_service_logging(environment, universal_data_sources, pass_as_pa ), ) + driver_df = datasets.driver_df + driver_df["val_to_add"] = 50 + driver_df = driver_df.join(conv_rate_plus_100(driver_df)) + + logs_df = prepare_logs(driver_df, feature_service, store) + schema = FeatureServiceLoggingSource( feature_service=feature_service, project=store.project ).get_schema(store._registry) @@ -85,7 +80,7 @@ def retrieve(): ) try: df = retrieval_job.to_df() - except NotFound: + except Exception: # Table was not created yet return None, False @@ -102,44 +97,3 @@ def retrieve(): persisted_logs.sort_values(REQUEST_ID_FIELD).reset_index(drop=True), check_dtype=False, ) - - -def prepare_logs(datasets: UniversalDatasets) -> pd.DataFrame: - driver_df = datasets.driver_df - driver_df["val_to_add"] = 50 - driver_df = driver_df.join(conv_rate_plus_100(driver_df)) - num_rows = driver_df.shape[0] - - logs_df = driver_df[["driver_id", "val_to_add"]] - logs_df[REQUEST_ID_FIELD] = [str(uuid.uuid4()) for _ in range(num_rows)] - logs_df[LOG_TIMESTAMP_FIELD] = pd.Series( - np.random.randint(0, 7 * 24 * 3600, num_rows) - ).map(lambda secs: pd.Timestamp.utcnow() - datetime.timedelta(seconds=secs)) - logs_df[LOG_DATE_FIELD] = logs_df[LOG_TIMESTAMP_FIELD].dt.date - - for view, features in ( - ("driver_stats", ("conv_rate", "avg_daily_trips")), - ( - "conv_rate_plus_100", - ("conv_rate_plus_val_to_add", "conv_rate_plus_100_rounded"), - ), - ): - for feature in features: - logs_df[f"{view}__{feature}"] = driver_df[feature] - logs_df[f"{view}__{feature}__timestamp"] = driver_df["event_timestamp"] - logs_df[f"{view}__{feature}__status"] = FieldStatus.PRESENT - - return logs_df - - -@contextlib.contextmanager -def to_logs_dataset( - table: pyarrow.Table, pass_as_path: bool -) -> Iterator[Union[pyarrow.Table, Path]]: - if not pass_as_path: - yield table - return - - with tempfile.TemporaryDirectory() as temp_dir: - pyarrow.parquet.write_to_dataset(table, root_path=temp_dir) - yield Path(temp_dir) diff --git a/sdk/python/tests/utils/logged_features.py b/sdk/python/tests/utils/logged_features.py new file mode 100644 index 00000000000..155f0b27b12 --- /dev/null +++ b/sdk/python/tests/utils/logged_features.py @@ -0,0 +1,67 @@ +import contextlib +import datetime +import tempfile +import uuid +from pathlib import Path +from typing import Iterator, Union + +import numpy as np +import pandas as pd +import pyarrow + +from feast import FeatureService, FeatureStore +from feast.errors import FeatureViewNotFoundException +from feast.feature_logging import LOG_DATE_FIELD, LOG_TIMESTAMP_FIELD, REQUEST_ID_FIELD +from feast.protos.feast.serving.ServingService_pb2 import FieldStatus + + +def prepare_logs( + source_df: pd.DataFrame, feature_service: FeatureService, store: FeatureStore +) -> pd.DataFrame: + num_rows = source_df.shape[0] + + logs_df = pd.DataFrame() + logs_df[REQUEST_ID_FIELD] = [str(uuid.uuid4()) for _ in range(num_rows)] + logs_df[LOG_TIMESTAMP_FIELD] = pd.Series( + np.random.randint(0, 7 * 24 * 3600, num_rows) + ).map(lambda secs: pd.Timestamp.utcnow() - datetime.timedelta(seconds=secs)) + logs_df[LOG_DATE_FIELD] = logs_df[LOG_TIMESTAMP_FIELD].dt.date + + for projection in feature_service.feature_view_projections: + for feature in projection.features: + logs_df[f"{projection.name_to_use()}__{feature.name}"] = source_df[ + feature.name + ] + logs_df[ + f"{projection.name_to_use()}__{feature.name}__timestamp" + ] = source_df["event_timestamp"].dt.floor("s") + logs_df[ + f"{projection.name_to_use()}__{feature.name}__status" + ] = FieldStatus.PRESENT + + try: + view = store.get_feature_view(projection.name) + except FeatureViewNotFoundException: + view = store.get_on_demand_feature_view(projection.name) + for source in view.source_request_sources.values(): + for field in source.schema: + logs_df[field.name] = source_df[field.name] + else: + for entity_name in view.entities: + entity = store.get_entity(entity_name) + logs_df[entity.join_key] = source_df[entity.join_key] + + return logs_df + + +@contextlib.contextmanager +def to_logs_dataset( + table: pyarrow.Table, pass_as_path: bool +) -> Iterator[Union[pyarrow.Table, Path]]: + if not pass_as_path: + yield table + return + + with tempfile.TemporaryDirectory() as temp_dir: + pyarrow.parquet.write_to_dataset(table, root_path=temp_dir) + yield Path(temp_dir) From 7a043ebfd34404b33456bce367673d39e2546570 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 16 May 2022 16:18:46 -0700 Subject: [PATCH 011/123] chore(deps): Bump async from 2.6.3 to 2.6.4 in /ui (#2702) Bumps [async](https://github.com/caolan/async) from 2.6.3 to 2.6.4. - [Release notes](https://github.com/caolan/async/releases) - [Changelog](https://github.com/caolan/async/blob/v2.6.4/CHANGELOG.md) - [Commits](https://github.com/caolan/async/compare/v2.6.3...v2.6.4) --- updated-dependencies: - dependency-name: async dependency-type: indirect ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- ui/yarn.lock | 258 ++++++++++++++++++++++++++++----------------------- 1 file changed, 140 insertions(+), 118 deletions(-) diff --git a/ui/yarn.lock b/ui/yarn.lock index 8079556b8d3..998565a77ac 100644 --- a/ui/yarn.lock +++ b/ui/yarn.lock @@ -1130,6 +1130,11 @@ "@babel/helper-validator-identifier" "^7.16.7" to-fast-properties "^2.0.0" +"@base2/pretty-print-object@1.0.1": + version "1.0.1" + resolved "https://registry.yarnpkg.com/@base2/pretty-print-object/-/pretty-print-object-1.0.1.tgz#371ba8be66d556812dc7fb169ebc3c08378f69d4" + integrity sha512-4iri8i1AqYHJE2DstZYkyEprg6Pq6sKx3xn5FpySk9sNhH7qN2LLlHJCfDTZRILNwQNPD7mATWM0TBui7uC1pA== + "@bcoe/v8-coverage@^0.2.3": version "0.2.3" resolved "https://registry.yarnpkg.com/@bcoe/v8-coverage/-/v8-coverage-0.2.3.tgz#75a2e8b51cb758a7553d6804a5932d7aace75c39" @@ -1147,18 +1152,18 @@ dependencies: tslib "^1.9.3" -"@elastic/eui@^46.1.0": - version "46.1.0" - resolved "https://registry.yarnpkg.com/@elastic/eui/-/eui-46.1.0.tgz#b1d77a56d529001858b7fc91f6c2953288549336" - integrity sha512-j65NMzqKluVPtk3b6UYdVVaBha74Y8Y/N6QaHin9MmqUKoAPgevqHWvi0n/OmMpADdRCI0ldxkse6tgAvdaEHQ== +"@elastic/eui@^55.0.1": + version "55.1.2" + resolved "https://registry.yarnpkg.com/@elastic/eui/-/eui-55.1.2.tgz#dd0b42f5b26c5800d6a9cb2d4c2fe1afce9d3f07" + integrity sha512-wwZz5KxMIMFlqEsoCRiQBJDc4CrluS1d0sCOmQ5lhIzKhYc91MdxnqCk2i6YkhL4sSDf2Y9KAEuMXa+uweOWUA== dependencies: "@types/chroma-js" "^2.0.0" "@types/lodash" "^4.14.160" "@types/numeral" "^0.0.28" - "@types/react-beautiful-dnd" "^13.0.0" - "@types/react-input-autosize" "^2.2.0" - "@types/react-virtualized-auto-sizer" "^1.0.0" - "@types/react-window" "^1.8.2" + "@types/react-beautiful-dnd" "^13.1.2" + "@types/react-input-autosize" "^2.2.1" + "@types/react-virtualized-auto-sizer" "^1.0.1" + "@types/react-window" "^1.8.5" "@types/refractor" "^3.0.0" "@types/resize-observer-browser" "^0.1.5" "@types/vfile-message" "^2.0.0" @@ -1168,13 +1173,14 @@ mdast-util-to-hast "^10.0.0" numeral "^2.0.6" prop-types "^15.6.0" - react-beautiful-dnd "^13.0.0" - react-dropzone "^11.2.0" - react-focus-on "^3.5.0" - react-input-autosize "^2.2.2" - react-is "~16.3.0" - react-virtualized-auto-sizer "^1.0.2" - react-window "^1.8.5" + react-beautiful-dnd "^13.1.0" + react-dropzone "^11.5.3" + react-element-to-jsx-string "^14.3.4" + react-focus-on "^3.5.4" + react-input-autosize "^3.0.0" + react-is "^17.0.2" + react-virtualized-auto-sizer "^1.0.6" + react-window "^1.8.6" refractor "^3.5.0" rehype-raw "^5.0.0" rehype-react "^6.0.0" @@ -1183,11 +1189,11 @@ remark-emoji "^2.1.0" remark-parse "^8.0.3" remark-rehype "^8.0.0" - tabbable "^3.0.0" + tabbable "^5.2.1" text-diff "^1.0.1" unified "^9.2.0" unist-util-visit "^2.0.3" - url-parse "^1.5.3" + url-parse "^1.5.10" uuid "^8.3.0" vfile "^4.2.0" @@ -2357,7 +2363,7 @@ resolved "https://registry.yarnpkg.com/@types/range-parser/-/range-parser-1.2.4.tgz#cd667bcfdd025213aafb7ca5915a932590acdcdc" integrity sha512-EEhsLsD6UsDM1yFhAvy0Cjr6VwmpMWqFBCb9w07wVugF7w9nfajxLuVmngTIpgS6svCnm6Vaw+MZhoDCKnOfsw== -"@types/react-beautiful-dnd@^13.0.0": +"@types/react-beautiful-dnd@^13.1.2": version "13.1.2" resolved "https://registry.yarnpkg.com/@types/react-beautiful-dnd/-/react-beautiful-dnd-13.1.2.tgz#510405abb09f493afdfd898bf83995dc6385c130" integrity sha512-+OvPkB8CdE/bGdXKyIhc/Lm2U7UAYCCJgsqmopFmh9gbAudmslkI8eOrPDjg4JhwSE6wytz4a3/wRjKtovHVJg== @@ -2371,7 +2377,7 @@ dependencies: "@types/react" "*" -"@types/react-input-autosize@^2.2.0": +"@types/react-input-autosize@^2.2.1": version "2.2.1" resolved "https://registry.yarnpkg.com/@types/react-input-autosize/-/react-input-autosize-2.2.1.tgz#6a335212e7fce1e1a4da56ae2095c8c5c35fbfe6" integrity sha512-RxzEjd4gbLAAdLQ92Q68/AC+TfsAKTc4evsArUH1aIShIMqQMIMjsxoSnwyjtbFTO/AGIW/RQI94XSdvOxCz/w== @@ -2388,14 +2394,14 @@ hoist-non-react-statics "^3.3.0" redux "^4.0.0" -"@types/react-virtualized-auto-sizer@^1.0.0": +"@types/react-virtualized-auto-sizer@^1.0.1": version "1.0.1" resolved "https://registry.yarnpkg.com/@types/react-virtualized-auto-sizer/-/react-virtualized-auto-sizer-1.0.1.tgz#b3187dae1dfc4c15880c9cfc5b45f2719ea6ebd4" integrity sha512-GH8sAnBEM5GV9LTeiz56r4ZhMOUSrP43tAQNSRVxNexDjcNKLCEtnxusAItg1owFUFE6k0NslV26gqVClVvong== dependencies: "@types/react" "*" -"@types/react-window@^1.8.2": +"@types/react-window@^1.8.5": version "1.8.5" resolved "https://registry.yarnpkg.com/@types/react-window/-/react-window-1.8.5.tgz#285fcc5cea703eef78d90f499e1457e9b5c02fc1" integrity sha512-V9q3CvhC9Jk9bWBOysPGaWy/Z0lxYcTXLtLipkt2cnRj1JOSFNF7wqGpkScSXMgBwC+fnVRg/7shwgddBG5ICw== @@ -3010,9 +3016,9 @@ ast-types-flow@^0.0.7: integrity sha1-9wtzXGvKGlycItmCw+Oef+ujva0= async@^2.6.2: - version "2.6.3" - resolved "https://registry.yarnpkg.com/async/-/async-2.6.3.tgz#d72625e2344a3656e3a3ad4fa749fa83299d82ff" - integrity sha512-zflvls11DCy+dQWzTW2dzuilv8Z5X/pjfmZOWba6TNIVDm+2UDaJmXSOXlasHKfNBs8oo3M0aT50fDEWfKZjXg== + version "2.6.4" + resolved "https://registry.yarnpkg.com/async/-/async-2.6.4.tgz#706b7ff6084664cd7eae713f6f965433b5504221" + integrity sha512-mzo5dfJYwAn29PeiJ0zvwTo04zj8HDJj0Mn8TD7sno7q12prdbnasKJHhkm2c1LgrhlJ0teaea8860oxi51mGA== dependencies: lodash "^4.17.14" @@ -3036,7 +3042,7 @@ atob@^2.1.2: resolved "https://registry.yarnpkg.com/atob/-/atob-2.1.2.tgz#6d9517eb9e030d2436666651e86bd9f6f13533c9" integrity sha512-Wm6ukoaOGJi/73p/cl2GvLjTI5JM1k/O14isD73YML8StrH/7/lRFgmg8nICZgD3bZZvjwCGxtMOD3wWNAu8cg== -attr-accept@^2.2.1: +attr-accept@^2.2.2: version "2.2.2" resolved "https://registry.yarnpkg.com/attr-accept/-/attr-accept-2.2.2.tgz#646613809660110749e92f2c10833b70968d929b" integrity sha512-7prDjvt9HmqiZ0cl5CRjtS84sEyhsHP2coDkaZKRKVfCDo9s7iw7ChVmar78Gu9pC4SoR/28wFu/G5JJhTnqEg== @@ -5218,10 +5224,10 @@ file-loader@^6.2.0: loader-utils "^2.0.0" schema-utils "^3.0.0" -file-selector@^0.2.2: - version "0.2.4" - resolved "https://registry.yarnpkg.com/file-selector/-/file-selector-0.2.4.tgz#7b98286f9dbb9925f420130ea5ed0a69238d4d80" - integrity sha512-ZDsQNbrv6qRi1YTDOEWzf5J2KjZ9KMI1Q2SGeTkCJmNNW25Jg4TW4UMcmoqcg4WrAyKRcpBXdbWRxkfrOzVRbA== +file-selector@^0.4.0: + version "0.4.0" + resolved "https://registry.yarnpkg.com/file-selector/-/file-selector-0.4.0.tgz#59ec4f27aa5baf0841e9c6385c8386bef4d18b17" + integrity sha512-iACCiXeMYOvZqlF1kTiYINzgepRBymz1wwjiuup9u9nayhb6g4fSwiyJ/6adli+EPwrWtpgQAh2PoS7HukEGEg== dependencies: tslib "^2.0.3" @@ -5314,10 +5320,10 @@ flatted@^3.1.0: resolved "https://registry.yarnpkg.com/flatted/-/flatted-3.2.5.tgz#76c8584f4fc843db64702a6bd04ab7a8bd666da3" integrity sha512-WIWGi2L3DyTUvUrwRKgGi9TwxQMUEqPOPQBVi71R96jZXJdFskXEmf54BoZaS1kknGODoIGASGEzBUYdyMCBJg== -focus-lock@^0.10.1: - version "0.10.1" - resolved "https://registry.yarnpkg.com/focus-lock/-/focus-lock-0.10.1.tgz#5f46fa74fefb87144479c2f8e276f0eedd8081b2" - integrity sha512-b9yUklCi4fTu2GXn7dnaVf4hiLVVBp7xTiZarAHMODV2To6Bitf6F/UI67RmKbdgJQeVwI1UO0d9HYNbXt3GkA== +focus-lock@^0.11.2: + version "0.11.2" + resolved "https://registry.yarnpkg.com/focus-lock/-/focus-lock-0.11.2.tgz#aeef3caf1cea757797ac8afdebaec8fd9ab243ed" + integrity sha512-pZ2bO++NWLHhiKkgP1bEXHhR1/OjVcSvlCJ98aNJDFeb7H5OOQaO+SKOZle6041O9rv2tmbrO4JzClAvDUHf0g== dependencies: tslib "^2.0.3" @@ -6210,6 +6216,11 @@ is-plain-obj@^3.0.0: resolved "https://registry.yarnpkg.com/is-plain-obj/-/is-plain-obj-3.0.0.tgz#af6f2ea14ac5a646183a5bbdb5baabbc156ad9d7" integrity sha512-gwsOE28k+23GP1B6vFl1oVh/WOzmawBrKwo5Ev6wMKzPkaXaCDIQKzLnvsA42DRlbVTWorkgTKIviAKCWkfUwA== +is-plain-object@5.0.0: + version "5.0.0" + resolved "https://registry.yarnpkg.com/is-plain-object/-/is-plain-object-5.0.0.tgz#4427f50ab3429e9025ea7d52e9043a9ef4159344" + integrity sha512-VRSzKkbMm5jMDoKLbltAkFQ5Qr7VDiTFGXxYFXXowVj387GeGNOCsOH6Msy00SGZ3Fp84b1Naa1psqgcCIEP5Q== + is-plain-object@^3.0.0: version "3.0.1" resolved "https://registry.yarnpkg.com/is-plain-object/-/is-plain-object-3.0.1.tgz#662d92d24c0aa4302407b0d45d21f2251c85f85b" @@ -8560,7 +8571,7 @@ react-app-polyfill@^3.0.0: regenerator-runtime "^0.13.9" whatwg-fetch "^3.6.2" -react-beautiful-dnd@^13.0.0: +react-beautiful-dnd@^13.1.0: version "13.1.0" resolved "https://registry.yarnpkg.com/react-beautiful-dnd/-/react-beautiful-dnd-13.1.0.tgz#ec97c81093593526454b0de69852ae433783844d" integrity sha512-aGvblPZTJowOWUNiwd6tNfEpgkX5OxmpqxHKNW/4VmvZTNTbeiq7bA3bn5T+QSF2uibXB0D1DmJsb1aC/+3cUA== @@ -8573,10 +8584,10 @@ react-beautiful-dnd@^13.0.0: redux "^4.0.4" use-memo-one "^1.1.1" -react-clientside-effect@^1.2.5: - version "1.2.5" - resolved "https://registry.yarnpkg.com/react-clientside-effect/-/react-clientside-effect-1.2.5.tgz#e2c4dc3c9ee109f642fac4f5b6e9bf5bcd2219a3" - integrity sha512-2bL8qFW1TGBHozGGbVeyvnggRpMjibeZM2536AKNENLECutp2yfs44IL8Hmpn8qjFQ2K7A9PnYf3vc7aQq/cPA== +react-clientside-effect@^1.2.6: + version "1.2.6" + resolved "https://registry.yarnpkg.com/react-clientside-effect/-/react-clientside-effect-1.2.6.tgz#29f9b14e944a376b03fb650eed2a754dd128ea3a" + integrity sha512-XGGGRQAKY+q25Lz9a/4EPqom7WRjz3z9R2k4jhVKA/puQFH/5Nt27vFZYql4m4NVNdUvX8PS3O7r/Zzm7cjUlg== dependencies: "@babel/runtime" "^7.12.13" @@ -8619,66 +8630,70 @@ react-dom@^17.0.2: object-assign "^4.1.1" scheduler "^0.20.2" -react-dropzone@^11.2.0: - version "11.5.3" - resolved "https://registry.yarnpkg.com/react-dropzone/-/react-dropzone-11.5.3.tgz#757d4980fcae839896a894e41d3e72df04981f86" - integrity sha512-68+T6sWW5L89qJnn3SD1aRazhuRBhTT9JOI1W8vI5YWsfegM4C7tlGbPH1AgEbmZY5s8E8L0QhX0e3VdAa0KWA== +react-dropzone@^11.5.3: + version "11.7.1" + resolved "https://registry.yarnpkg.com/react-dropzone/-/react-dropzone-11.7.1.tgz#3851bb75b26af0bf1b17ce1449fd980e643b9356" + integrity sha512-zxCMwhfPy1olUEbw3FLNPLhAm/HnaYH5aELIEglRbqabizKAdHs0h+WuyOpmA+v1JXn0++fpQDdNfUagWt5hJQ== dependencies: - attr-accept "^2.2.1" - file-selector "^0.2.2" - prop-types "^15.7.2" + attr-accept "^2.2.2" + file-selector "^0.4.0" + prop-types "^15.8.1" + +react-element-to-jsx-string@^14.3.4: + version "14.3.4" + resolved "https://registry.yarnpkg.com/react-element-to-jsx-string/-/react-element-to-jsx-string-14.3.4.tgz#709125bc72f06800b68f9f4db485f2c7d31218a8" + integrity sha512-t4ZwvV6vwNxzujDQ+37bspnLwA4JlgUPWhLjBJWsNIDceAf6ZKUTCjdm08cN6WeZ5pTMKiCJkmAYnpmR4Bm+dg== + dependencies: + "@base2/pretty-print-object" "1.0.1" + is-plain-object "5.0.0" + react-is "17.0.2" react-error-overlay@^6.0.10: version "6.0.10" resolved "https://registry.yarnpkg.com/react-error-overlay/-/react-error-overlay-6.0.10.tgz#0fe26db4fa85d9dbb8624729580e90e7159a59a6" integrity sha512-mKR90fX7Pm5seCOfz8q9F+66VCc1PGsWSBxKbITjfKVQHMNF2zudxHnMdJiB1fRCb+XsbQV9sO9DCkgsMQgBIA== -react-focus-lock@^2.6.0: - version "2.7.1" - resolved "https://registry.yarnpkg.com/react-focus-lock/-/react-focus-lock-2.7.1.tgz#a9fbb3fa4efaee32162406e5eb96ae658964193b" - integrity sha512-ImSeVmcrLKNMqzUsIdqOkXwTVltj79OPu43oT8tVun7eIckA4VdM7UmYUFo3H/UC2nRVgagMZGFnAOQEDiDYcA== +react-focus-lock@^2.9.0: + version "2.9.1" + resolved "https://registry.yarnpkg.com/react-focus-lock/-/react-focus-lock-2.9.1.tgz#094cfc19b4f334122c73bb0bff65d77a0c92dd16" + integrity sha512-pSWOQrUmiKLkffPO6BpMXN7SNKXMsuOakl652IBuALAu1esk+IcpJyM+ALcYzPTTFz1rD0R54aB9A4HuP5t1Wg== dependencies: "@babel/runtime" "^7.0.0" - focus-lock "^0.10.1" + focus-lock "^0.11.2" prop-types "^15.6.2" - react-clientside-effect "^1.2.5" - use-callback-ref "^1.2.5" - use-sidecar "^1.0.5" + react-clientside-effect "^1.2.6" + use-callback-ref "^1.3.0" + use-sidecar "^1.1.2" -react-focus-on@^3.5.0: - version "3.5.4" - resolved "https://registry.yarnpkg.com/react-focus-on/-/react-focus-on-3.5.4.tgz#be45a9d0495f3bb6f5249704c85362df94980ecf" - integrity sha512-HnU0YGKhNSUsC4k6K8L+2wk8mC/qdg+CsS7A1bWLMgK7UuBphdECs2esnS6cLmBoVNjsFnCm/vMypeezKOdK3A== +react-focus-on@^3.5.4: + version "3.6.0" + resolved "https://registry.yarnpkg.com/react-focus-on/-/react-focus-on-3.6.0.tgz#159e13082dad4ea1f07abe11254f0e981d5a7b79" + integrity sha512-onIRjpd9trAUenXNdDcvjc8KJUSklty4X/Gr7hAm/MzM7ekSF2pg9D8KBKL7ipige22IAPxLRRf/EmJji9KD6Q== dependencies: aria-hidden "^1.1.3" - react-focus-lock "^2.6.0" - react-remove-scroll "^2.4.1" - react-style-singleton "^2.1.1" + react-focus-lock "^2.9.0" + react-remove-scroll "^2.5.2" + react-style-singleton "^2.2.0" tslib "^2.3.1" - use-callback-ref "^1.2.5" - use-sidecar "^1.0.5" + use-callback-ref "^1.3.0" + use-sidecar "^1.1.2" -react-input-autosize@^2.2.2: - version "2.2.2" - resolved "https://registry.yarnpkg.com/react-input-autosize/-/react-input-autosize-2.2.2.tgz#fcaa7020568ec206bc04be36f4eb68e647c4d8c2" - integrity sha512-jQJgYCA3S0j+cuOwzuCd1OjmBmnZLdqQdiLKRYrsMMzbjUrVDS5RvJUDwJqA7sKuksDuzFtm6hZGKFu7Mjk5aw== +react-input-autosize@^3.0.0: + version "3.0.0" + resolved "https://registry.yarnpkg.com/react-input-autosize/-/react-input-autosize-3.0.0.tgz#6b5898c790d4478d69420b55441fcc31d5c50a85" + integrity sha512-nL9uS7jEs/zu8sqwFE5MAPx6pPkNAriACQ2rGLlqmKr2sPGtN7TXTyDdQt4lbNXVx7Uzadb40x8qotIuru6Rhg== dependencies: prop-types "^15.5.8" -react-is@^16.13.1, react-is@^16.7.0: - version "16.13.1" - resolved "https://registry.yarnpkg.com/react-is/-/react-is-16.13.1.tgz#789729a4dc36de2999dc156dd6c1d9c18cea56a4" - integrity sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ== - -react-is@^17.0.1, react-is@^17.0.2: +react-is@17.0.2, react-is@^17.0.1, react-is@^17.0.2: version "17.0.2" resolved "https://registry.yarnpkg.com/react-is/-/react-is-17.0.2.tgz#e691d4a8e9c789365655539ab372762b0efb54f0" integrity sha512-w2GsyukL62IJnlaff/nRegPQR94C/XXamvMWmSHRJ4y7Ts/4ocGRmTHvOs8PSE6pB3dWOrD/nueuU5sduBsQ4w== -react-is@~16.3.0: - version "16.3.2" - resolved "https://registry.yarnpkg.com/react-is/-/react-is-16.3.2.tgz#f4d3d0e2f5fbb6ac46450641eb2e25bf05d36b22" - integrity sha512-ybEM7YOr4yBgFd6w8dJqwxegqZGJNBZl6U27HnGKuTZmDvVrD5quWOK/wAnMywiZzW+Qsk+l4X2c70+thp/A8Q== +react-is@^16.13.1, react-is@^16.7.0: + version "16.13.1" + resolved "https://registry.yarnpkg.com/react-is/-/react-is-16.13.1.tgz#789729a4dc36de2999dc156dd6c1d9c18cea56a4" + integrity sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ== react-query@^3.34.12: version "3.34.12" @@ -8706,24 +8721,24 @@ react-refresh@^0.11.0: resolved "https://registry.yarnpkg.com/react-refresh/-/react-refresh-0.11.0.tgz#77198b944733f0f1f1a90e791de4541f9f074046" integrity sha512-F27qZr8uUqwhWZboondsPx8tnC3Ct3SxZA3V5WyEvujRyyNv0VYPhoBg1gZ8/MV5tubQp76Trw8lTv9hzRBa+A== -react-remove-scroll-bar@^2.1.0: - version "2.2.0" - resolved "https://registry.yarnpkg.com/react-remove-scroll-bar/-/react-remove-scroll-bar-2.2.0.tgz#d4d545a7df024f75d67e151499a6ab5ac97c8cdd" - integrity sha512-UU9ZBP1wdMR8qoUs7owiVcpaPwsQxUDC2lypP6mmixaGlARZa7ZIBx1jcuObLdhMOvCsnZcvetOho0wzPa9PYg== +react-remove-scroll-bar@^2.3.1: + version "2.3.1" + resolved "https://registry.yarnpkg.com/react-remove-scroll-bar/-/react-remove-scroll-bar-2.3.1.tgz#9f13b05b249eaa57c8d646c1ebb83006b3581f5f" + integrity sha512-IvGX3mJclEF7+hga8APZczve1UyGMkMG+tjS0o/U1iLgvZRpjFAQEUBJ4JETfvbNlfNnZnoDyWJCICkA15Mghg== dependencies: - react-style-singleton "^2.1.0" - tslib "^1.0.0" + react-style-singleton "^2.2.0" + tslib "^2.0.0" -react-remove-scroll@^2.4.1: - version "2.4.3" - resolved "https://registry.yarnpkg.com/react-remove-scroll/-/react-remove-scroll-2.4.3.tgz#83d19b02503b04bd8141ed6e0b9e6691a2e935a6" - integrity sha512-lGWYXfV6jykJwbFpsuPdexKKzp96f3RbvGapDSIdcyGvHb7/eqyn46C7/6h+rUzYar1j5mdU+XECITHXCKBk9Q== +react-remove-scroll@^2.5.2: + version "2.5.3" + resolved "https://registry.yarnpkg.com/react-remove-scroll/-/react-remove-scroll-2.5.3.tgz#a152196e710e8e5811be39dc352fd8a90b05c961" + integrity sha512-NQ1bXrxKrnK5pFo/GhLkXeo3CrK5steI+5L+jynwwIemvZyfXqaL0L5BzwJd7CSwNCU723DZaccvjuyOdoy3Xw== dependencies: - react-remove-scroll-bar "^2.1.0" - react-style-singleton "^2.1.0" - tslib "^1.0.0" - use-callback-ref "^1.2.3" - use-sidecar "^1.0.1" + react-remove-scroll-bar "^2.3.1" + react-style-singleton "^2.2.0" + tslib "^2.0.0" + use-callback-ref "^1.3.0" + use-sidecar "^1.1.2" react-router-dom@6: version "6.2.1" @@ -8795,24 +8810,24 @@ react-scripts@^5.0.0: optionalDependencies: fsevents "^2.3.2" -react-style-singleton@^2.1.0, react-style-singleton@^2.1.1: - version "2.1.1" - resolved "https://registry.yarnpkg.com/react-style-singleton/-/react-style-singleton-2.1.1.tgz#ce7f90b67618be2b6b94902a30aaea152ce52e66" - integrity sha512-jNRp07Jza6CBqdRKNgGhT3u9umWvils1xsuMOjZlghBDH2MU0PL2WZor4PGYjXpnRCa9DQSlHMs/xnABWOwYbA== +react-style-singleton@^2.2.0: + version "2.2.0" + resolved "https://registry.yarnpkg.com/react-style-singleton/-/react-style-singleton-2.2.0.tgz#70f45f5fef97fdb9a52eed98d1839fa6b9032b22" + integrity sha512-nK7mN92DMYZEu3cQcAhfwE48NpzO5RpxjG4okbSqRRbfal9Pk+fG2RdQXTMp+f6all1hB9LIJSt+j7dCYrU11g== dependencies: get-nonce "^1.0.0" invariant "^2.2.4" - tslib "^1.0.0" + tslib "^2.0.0" -react-virtualized-auto-sizer@^1.0.2: +react-virtualized-auto-sizer@^1.0.6: version "1.0.6" resolved "https://registry.yarnpkg.com/react-virtualized-auto-sizer/-/react-virtualized-auto-sizer-1.0.6.tgz#66c5b1c9278064c5ef1699ed40a29c11518f97ca" integrity sha512-7tQ0BmZqfVF6YYEWcIGuoR3OdYe8I/ZFbNclFlGOC3pMqunkYF/oL30NCjSGl9sMEb17AnzixDz98Kqc3N76HQ== -react-window@^1.8.5: - version "1.8.6" - resolved "https://registry.yarnpkg.com/react-window/-/react-window-1.8.6.tgz#d011950ac643a994118632665aad0c6382e2a112" - integrity sha512-8VwEEYyjz6DCnGBsd+MgkD0KJ2/OXFULyDtorIiTz+QzwoP94tBoA7CnbtyXMm+cCeAUER5KJcPtWl9cpKbOBg== +react-window@^1.8.6: + version "1.8.7" + resolved "https://registry.yarnpkg.com/react-window/-/react-window-1.8.7.tgz#5e9fd0d23f48f432d7022cdb327219353a15f0d4" + integrity sha512-JHEZbPXBpKMmoNO1bNhoXOOLg/ujhL/BU4IqVU9r8eQPcy5KQnGHIHDRkJ0ns9IM5+Aq5LNwt3j8t3tIrePQzA== dependencies: "@babel/runtime" "^7.0.0" memoize-one ">=3.1.1 <6" @@ -9879,10 +9894,10 @@ symbol-tree@^3.2.4: resolved "https://registry.yarnpkg.com/symbol-tree/-/symbol-tree-3.2.4.tgz#430637d248ba77e078883951fb9aa0eed7c63fa2" integrity sha512-9QNk5KwDF+Bvz+PyObkmSYjI5ksVUYtjW7AU22r2NKcfLJcXp96hkDWU3+XndOsUb+AQ9QhfzfCT2O+CNWT5Tw== -tabbable@^3.0.0: - version "3.1.2" - resolved "https://registry.yarnpkg.com/tabbable/-/tabbable-3.1.2.tgz#f2d16cccd01f400e38635c7181adfe0ad965a4a2" - integrity sha512-wjB6puVXTYO0BSFtCmWQubA/KIn7Xvajw0x0l6eJUudMG/EAiJvIUnyNX6xO4NpGrJ16lbD0eUseB9WxW0vlpQ== +tabbable@^5.2.1: + version "5.3.2" + resolved "https://registry.yarnpkg.com/tabbable/-/tabbable-5.3.2.tgz#66d6119ee8a533634c3f17deb0caa1c379e36ac7" + integrity sha512-6G/8EWRFx8CiSe2++/xHhXkmCRq2rHtDtZbQFHx34cvDfZzIBfvwG9zGUNTWMXWLCYvDj3aQqOzdl3oCxKuBkQ== tailwindcss@^3.0.2: version "3.0.18" @@ -10099,6 +10114,11 @@ tslib@^1.0.0, tslib@^1.8.1, tslib@^1.9.3: resolved "https://registry.yarnpkg.com/tslib/-/tslib-1.14.1.tgz#cf2d38bdc34a134bcaf1091c41f6619e2f672d00" integrity sha512-Xni35NKzjgMrwevysHTCArtLDpPvye8zV/0E4EyYn43P7/7qvQwPh9BGkHewbMulVntbigmcT7rdX3BNo9wRJg== +tslib@^2.0.0: + version "2.4.0" + resolved "https://registry.yarnpkg.com/tslib/-/tslib-2.4.0.tgz#7cecaa7f073ce680a05847aa77be941098f36dc3" + integrity sha512-d6xOpEDfsi2CZVlPQzGeux8XMwLT9hssAsaPYExaQMuYskwb+x1x7J371tWlbBdWHroy99KnVB6qIkUbs5X3UQ== + tslib@^2.0.3, tslib@^2.1.0, tslib@^2.3.1: version "2.3.1" resolved "https://registry.yarnpkg.com/tslib/-/tslib-2.3.1.tgz#e8a335add5ceae51aa261d32a490158ef042ef01" @@ -10347,18 +10367,20 @@ uri-js@^4.2.2: dependencies: punycode "^2.1.0" -url-parse@^1.5.3: - version "1.5.4" - resolved "https://registry.yarnpkg.com/url-parse/-/url-parse-1.5.4.tgz#e4f645a7e2a0852cc8a66b14b292a3e9a11a97fd" - integrity sha512-ITeAByWWoqutFClc/lRZnFplgXgEZr3WJ6XngMM/N9DMIm4K8zXPCZ1Jdu0rERwO84w1WC5wkle2ubwTA4NTBg== +url-parse@^1.5.10: + version "1.5.10" + resolved "https://registry.yarnpkg.com/url-parse/-/url-parse-1.5.10.tgz#9d3c2f736c1d75dd3bd2be507dcc111f1e2ea9c1" + integrity sha512-WypcfiRhfeUP9vvF0j6rw0J3hrWrw6iZv3+22h6iRMJ/8z1Tj6XfLP4DsUix5MhMPnXpiHDoKyoZ/bdCkwBCiQ== dependencies: querystringify "^2.1.1" requires-port "^1.0.0" -use-callback-ref@^1.2.3, use-callback-ref@^1.2.5: - version "1.2.5" - resolved "https://registry.yarnpkg.com/use-callback-ref/-/use-callback-ref-1.2.5.tgz#6115ed242cfbaed5915499c0a9842ca2912f38a5" - integrity sha512-gN3vgMISAgacF7sqsLPByqoePooY3n2emTH59Ur5d/M8eg4WTWu1xp8i8DHjohftIyEx0S08RiYxbffr4j8Peg== +use-callback-ref@^1.3.0: + version "1.3.0" + resolved "https://registry.yarnpkg.com/use-callback-ref/-/use-callback-ref-1.3.0.tgz#772199899b9c9a50526fedc4993fc7fa1f7e32d5" + integrity sha512-3FT9PRuRdbB9HfXhEq35u4oZkvpJ5kuYbpqhCfmiZyReuRgpnhDlbr2ZEnnuS0RrJAPn6l23xjFg9kpDM+Ms7w== + dependencies: + tslib "^2.0.0" use-memo-one@^1.1.1: version "1.1.2" @@ -10372,13 +10394,13 @@ use-query-params@^1.2.3: dependencies: serialize-query-params "^1.3.5" -use-sidecar@^1.0.1, use-sidecar@^1.0.5: - version "1.0.5" - resolved "https://registry.yarnpkg.com/use-sidecar/-/use-sidecar-1.0.5.tgz#ffff2a17c1df42e348624b699ba6e5c220527f2b" - integrity sha512-k9jnrjYNwN6xYLj1iaGhonDghfvmeTmYjAiGvOr7clwKfPjMXJf4/HOr7oT5tJwYafgp2tG2l3eZEOfoELiMcA== +use-sidecar@^1.1.2: + version "1.1.2" + resolved "https://registry.yarnpkg.com/use-sidecar/-/use-sidecar-1.1.2.tgz#2f43126ba2d7d7e117aa5855e5d8f0276dfe73c2" + integrity sha512-epTbsLuzZ7lPClpz2TyryBfztm7m+28DlEv2ZCQ3MDr5ssiwyOwGH/e5F9CkfWjJ1t4clvI58yF822/GUkjjhw== dependencies: detect-node-es "^1.1.0" - tslib "^1.9.3" + tslib "^2.0.0" util-deprecate@^1.0.1, util-deprecate@^1.0.2, util-deprecate@~1.0.1: version "1.0.2" From dee80908ffb247367526561de3ad4841516a3196 Mon Sep 17 00:00:00 2001 From: Oleksii Moskalenko Date: Mon, 16 May 2022 16:19:44 -0700 Subject: [PATCH 012/123] fix: Random port allocation for python server in tests (#2710) Signed-off-by: Oleksii Moskalenko --- sdk/python/tests/conftest.py | 27 ++++++++++++------- .../feature_repos/repo_configuration.py | 20 -------------- .../online_store/test_universal_online.py | 15 +++++++++-- 3 files changed, 31 insertions(+), 31 deletions(-) diff --git a/sdk/python/tests/conftest.py b/sdk/python/tests/conftest.py index 092f5a07873..627fda524d9 100644 --- a/sdk/python/tests/conftest.py +++ b/sdk/python/tests/conftest.py @@ -273,12 +273,19 @@ def pytest_generate_tests(metafunc: pytest.Metafunc): @pytest.fixture(scope="session") -def python_server(environment): - assert not _check_port_open("localhost", environment.get_local_server_port()) +def feature_server_endpoint(environment): + if ( + not environment.python_feature_server + or environment.test_repo_config.provider != "local" + ): + yield environment.feature_store.get_feature_server_endpoint() + return + + port = _free_port() proc = Process( target=start_test_local_server, - args=(environment.feature_store.repo_path, environment.get_local_server_port()), + args=(environment.feature_store.repo_path, port), ) if ( environment.python_feature_server @@ -287,14 +294,10 @@ def python_server(environment): proc.start() # Wait for server to start wait_retry_backoff( - lambda: ( - None, - _check_port_open("localhost", environment.get_local_server_port()), - ), - timeout_secs=10, + lambda: (None, _check_port_open("localhost", port)), timeout_secs=10, ) - yield + yield f"http://localhost:{port}" if proc.is_alive(): proc.kill() @@ -314,6 +317,12 @@ def _check_port_open(host, port) -> bool: return sock.connect_ex((host, port)) == 0 +def _free_port(): + sock = socket.socket() + sock.bind(("", 0)) + return sock.getsockname()[1] + + @pytest.fixture(scope="session") def universal_data_sources(environment) -> TestData: return construct_universal_test_data(environment) diff --git a/sdk/python/tests/integration/feature_repos/repo_configuration.py b/sdk/python/tests/integration/feature_repos/repo_configuration.py index 27cf1a52e9d..f4c9bed92a9 100644 --- a/sdk/python/tests/integration/feature_repos/repo_configuration.py +++ b/sdk/python/tests/integration/feature_repos/repo_configuration.py @@ -2,7 +2,6 @@ import importlib import json import os -import re import tempfile import uuid from dataclasses import dataclass @@ -328,29 +327,10 @@ class Environment: worker_id: str online_store_creator: Optional[OnlineStoreCreator] = None - next_id = 0 - def __post_init__(self): self.end_date = datetime.utcnow().replace(microsecond=0, second=0, minute=0) self.start_date: datetime = self.end_date - timedelta(days=3) - Environment.next_id += 1 - self.id = Environment.next_id - - def get_feature_server_endpoint(self) -> str: - if self.python_feature_server and self.test_repo_config.provider == "local": - return f"http://localhost:{self.get_local_server_port()}" - return self.feature_store.get_feature_server_endpoint() - - def get_local_server_port(self) -> int: - # Heuristic when running with xdist to extract unique ports for each worker - parsed_worker_id = re.findall("gw(\\d+)", self.worker_id) - if len(parsed_worker_id) != 0: - worker_id_num = int(parsed_worker_id[0]) - else: - worker_id_num = 0 - return 6000 + 100 * worker_id_num + self.id - def table_name_from_data_source(ds: DataSource) -> Optional[str]: if hasattr(ds, "table_ref"): diff --git a/sdk/python/tests/integration/online_store/test_universal_online.py b/sdk/python/tests/integration/online_store/test_universal_online.py index 259a094426e..b3115dcb3d4 100644 --- a/sdk/python/tests/integration/online_store/test_universal_online.py +++ b/sdk/python/tests/integration/online_store/test_universal_online.py @@ -288,6 +288,7 @@ def _get_online_features_dict_remotely( def get_online_features_dict( environment: Environment, + endpoint: str, features: Union[List[str], FeatureService], entity_rows: List[Dict[str, Any]], full_feature_names: bool = False, @@ -305,7 +306,6 @@ def get_online_features_dict( assertpy.assert_that(online_features).is_not_none() dict1 = online_features.to_dict() - endpoint = environment.get_feature_server_endpoint() # If endpoint is None, it means that a local / remote feature server aren't configured if endpoint is not None: dict2 = _get_online_features_dict_remotely( @@ -447,7 +447,7 @@ def test_online_retrieval_with_event_timestamps( @pytest.mark.goserver @pytest.mark.parametrize("full_feature_names", [True, False], ids=lambda v: str(v)) def test_online_retrieval( - environment, universal_data_sources, python_server, full_feature_names + environment, universal_data_sources, feature_server_endpoint, full_feature_names ): fs = environment.feature_store entities, datasets, data_sources = universal_data_sources @@ -547,6 +547,7 @@ def test_online_retrieval( online_features_dict = get_online_features_dict( environment=environment, + endpoint=feature_server_endpoint, features=feature_refs, entity_rows=entity_rows, full_feature_names=full_feature_names, @@ -556,6 +557,7 @@ def test_online_retrieval( # feature isn't requested. online_features_no_conv_rate = get_online_features_dict( environment=environment, + endpoint=feature_server_endpoint, features=[ref for ref in feature_refs if ref != "driver_stats:conv_rate"], entity_rows=entity_rows, full_feature_names=full_feature_names, @@ -616,6 +618,7 @@ def test_online_retrieval( # Check what happens for missing values missing_responses_dict = get_online_features_dict( environment=environment, + endpoint=feature_server_endpoint, features=feature_refs, entity_rows=[{"driver_id": 0, "customer_id": 0, "val_to_add": 100}], full_feature_names=full_feature_names, @@ -635,6 +638,7 @@ def test_online_retrieval( with pytest.raises(RequestDataNotFoundInEntityRowsException): get_online_features_dict( environment=environment, + endpoint=feature_server_endpoint, features=feature_refs, entity_rows=[{"driver_id": 0, "customer_id": 0}], full_feature_names=full_feature_names, @@ -642,6 +646,7 @@ def test_online_retrieval( assert_feature_service_correctness( environment, + feature_server_endpoint, feature_service, entity_rows, full_feature_names, @@ -659,6 +664,7 @@ def test_online_retrieval( ] assert_feature_service_entity_mapping_correctness( environment, + feature_server_endpoint, feature_service_entity_mapping, entity_rows, full_feature_names, @@ -856,6 +862,7 @@ def get_latest_feature_values_for_location_df(entity_row, origin_df, destination def assert_feature_service_correctness( environment, + endpoint, feature_service, entity_rows, full_feature_names, @@ -866,6 +873,7 @@ def assert_feature_service_correctness( ): feature_service_online_features_dict = get_online_features_dict( environment=environment, + endpoint=endpoint, features=feature_service, entity_rows=entity_rows, full_feature_names=full_feature_names, @@ -905,6 +913,7 @@ def assert_feature_service_correctness( def assert_feature_service_entity_mapping_correctness( environment, + endpoint, feature_service, entity_rows, full_feature_names, @@ -914,6 +923,7 @@ def assert_feature_service_entity_mapping_correctness( if full_feature_names: feature_service_online_features_dict = get_online_features_dict( environment=environment, + endpoint=endpoint, features=feature_service, entity_rows=entity_rows, full_feature_names=full_feature_names, @@ -948,6 +958,7 @@ def assert_feature_service_entity_mapping_correctness( with pytest.raises(FeatureNameCollisionError): get_online_features_dict( environment=environment, + endpoint=endpoint, features=feature_service, entity_rows=entity_rows, full_feature_names=full_feature_names, From cebf609309f81a0f4ccded3448cefac5923be525 Mon Sep 17 00:00:00 2001 From: Oleksii Moskalenko Date: Mon, 16 May 2022 16:51:44 -0700 Subject: [PATCH 013/123] fix: Correct feature statuses during feature logging test (#2709) * fix: Feature Logging test & python server ports Signed-off-by: Oleksii Moskalenko * optional timedelta Signed-off-by: Oleksii Moskalenko * revert Signed-off-by: Oleksii Moskalenko * typo Signed-off-by: Oleksii Moskalenko --- .../integration/e2e/test_go_feature_server.py | 24 ++++++++++++++----- 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/sdk/python/tests/integration/e2e/test_go_feature_server.py b/sdk/python/tests/integration/e2e/test_go_feature_server.py index e469c90c11f..4e4cfc1fb8f 100644 --- a/sdk/python/tests/integration/e2e/test_go_feature_server.py +++ b/sdk/python/tests/integration/e2e/test_go_feature_server.py @@ -10,7 +10,7 @@ import pytest import pytz -from feast import FeatureService, ValueType +from feast import FeatureService, FeatureView, ValueType from feast.embedded_go.lib.embedded import LoggingOptions from feast.embedded_go.online_features_service import EmbeddedOnlineFeatureServer from feast.feast_object import FeastObject @@ -162,13 +162,14 @@ def test_feature_logging( _, datasets, _ = universal_data_sources latest_rows = get_latest_rows(datasets.driver_df, "driver_id", driver_ids) + feature_view = fs.get_feature_view("driver_stats") features = [ feature.name for proj in feature_service.feature_view_projections for feature in proj.features ] expected_logs = generate_expected_logs( - latest_rows, "driver_stats", features, ["driver_id"], "event_timestamp" + latest_rows, feature_view, features, ["driver_id"], "event_timestamp" ) def retrieve(): @@ -213,15 +214,26 @@ def get_latest_rows(df, join_key, entity_values): def generate_expected_logs( - df, feature_view_name, features, join_keys, timestamp_column + df: pd.DataFrame, + feature_view: FeatureView, + features: List[str], + join_keys: List[str], + timestamp_column: str, ): logs = pd.DataFrame() for join_key in join_keys: logs[join_key] = df[join_key] for feature in features: - logs[f"{feature_view_name}__{feature}"] = df[feature] - logs[f"{feature_view_name}__{feature}__timestamp"] = df[timestamp_column] - logs[f"{feature_view_name}__{feature}__status"] = FieldStatus.PRESENT + col = f"{feature_view.name}__{feature}" + logs[col] = df[feature] + logs[f"{col}__timestamp"] = df[timestamp_column] + logs[f"{col}__status"] = FieldStatus.PRESENT + if feature_view.ttl: + logs[f"{col}__status"] = logs[f"{col}__status"].mask( + df[timestamp_column] + < datetime.utcnow().replace(tzinfo=pytz.UTC) - feature_view.ttl, + FieldStatus.OUTSIDE_MAX_AGE, + ) return logs.sort_values(by=join_keys).reset_index(drop=True) From c5539fd9874fed3f69b0aaebc4d1d32e624bd041 Mon Sep 17 00:00:00 2001 From: Danny Chiao Date: Tue, 17 May 2022 15:53:14 -0400 Subject: [PATCH 014/123] fix: Fix on demand feature view crash from inference when it uses df.apply (#2713) * fix: Fix on demand feature view crash from inference when transformation uses df.apply Signed-off-by: Danny Chiao * Fix inference Signed-off-by: Danny Chiao * Fix test Signed-off-by: Danny Chiao --- sdk/python/feast/on_demand_feature_view.py | 18 ++++- .../on_demand_feature_view_repo.py | 48 ++++++++++++ .../integration/registration/test_cli.py | 31 ++++++++ .../integration/registration/test_registry.py | 74 +++++++++++++++++++ 4 files changed, 168 insertions(+), 3 deletions(-) create mode 100644 sdk/python/tests/example_repos/on_demand_feature_view_repo.py diff --git a/sdk/python/feast/on_demand_feature_view.py b/sdk/python/feast/on_demand_feature_view.py index 1cddc0b8814..f2048e7f5ea 100644 --- a/sdk/python/feast/on_demand_feature_view.py +++ b/sdk/python/feast/on_demand_feature_view.py @@ -1,8 +1,9 @@ import copy import functools import warnings +from datetime import datetime from types import MethodType -from typing import Dict, List, Optional, Type, Union +from typing import Any, Dict, List, Optional, Type, Union import dill import pandas as pd @@ -442,6 +443,15 @@ def infer_features(self): Raises: RegistryInferenceFailure: The set of features could not be inferred. """ + rand_df_value: Dict[str, Any] = { + "float": 1.0, + "int": 1, + "str": "hello world", + "bytes": str.encode("hello world"), + "bool": True, + "datetime64[ns]": datetime.utcnow(), + } + df = pd.DataFrame() for feature_view_projection in self.source_feature_view_projections.values(): for feature in feature_view_projection.features: @@ -449,11 +459,13 @@ def infer_features(self): df[f"{feature_view_projection.name}__{feature.name}"] = pd.Series( dtype=dtype ) - df[f"{feature.name}"] = pd.Series(dtype=dtype) + sample_val = rand_df_value[dtype] if dtype in rand_df_value else None + df[f"{feature.name}"] = pd.Series(data=sample_val, dtype=dtype) for request_data in self.source_request_sources.values(): for field in request_data.schema: dtype = feast_value_type_to_pandas_type(field.dtype.to_value_type()) - df[f"{field.name}"] = pd.Series(dtype=dtype) + sample_val = rand_df_value[dtype] if dtype in rand_df_value else None + df[f"{field.name}"] = pd.Series(sample_val, dtype=dtype) output_df: pd.DataFrame = self.udf.__call__(df) inferred_features = [] for f, dt in zip(output_df.columns, output_df.dtypes): diff --git a/sdk/python/tests/example_repos/on_demand_feature_view_repo.py b/sdk/python/tests/example_repos/on_demand_feature_view_repo.py new file mode 100644 index 00000000000..453158b9dcc --- /dev/null +++ b/sdk/python/tests/example_repos/on_demand_feature_view_repo.py @@ -0,0 +1,48 @@ +from datetime import timedelta + +import pandas as pd + +from feast import FeatureView, Field, FileSource +from feast.on_demand_feature_view import on_demand_feature_view +from feast.types import Float32, String + +driver_stats = FileSource( + name="driver_stats_source", + path="data/driver_stats_lat_lon.parquet", + timestamp_field="event_timestamp", + created_timestamp_column="created", + description="A table describing the stats of a driver based on hourly logs", + owner="test2@gmail.com", +) + +driver_daily_features_view = FeatureView( + name="driver_daily_features", + entities=["driver"], + ttl=timedelta(seconds=8640000000), + schema=[ + Field(name="daily_miles_driven", dtype=Float32), + Field(name="lat", dtype=Float32), + Field(name="lon", dtype=Float32), + Field(name="string_feature", dtype=String), + ], + online=True, + source=driver_stats, + tags={"production": "True"}, + owner="test2@gmail.com", +) + + +@on_demand_feature_view( + sources=[driver_daily_features_view], + schema=[ + Field(name="first_char", dtype=String), + Field(name="concat_string", dtype=String), + ], +) +def location_features_from_push(inputs: pd.DataFrame) -> pd.DataFrame: + df = pd.DataFrame() + df["concat_string"] = inputs.apply( + lambda x: x.string_feature + "hello", axis=1 + ).astype("string") + df["first_char"] = inputs["string_feature"].str[:1].astype("string") + return df diff --git a/sdk/python/tests/integration/registration/test_cli.py b/sdk/python/tests/integration/registration/test_cli.py index ce23ed66a60..ecc17fc06c5 100644 --- a/sdk/python/tests/integration/registration/test_cli.py +++ b/sdk/python/tests/integration/registration/test_cli.py @@ -201,6 +201,37 @@ def test_nullable_online_store(test_nullable_online_store) -> None: runner.run(["teardown"], cwd=repo_path) +@pytest.mark.integration +@pytest.mark.universal_offline_stores +def test_odfv_apply(environment) -> None: + project = f"test_odfv_apply{str(uuid.uuid4()).replace('-', '')[:8]}" + runner = CliRunner() + + with tempfile.TemporaryDirectory() as repo_dir_name: + try: + repo_path = Path(repo_dir_name) + feature_store_yaml = make_feature_store_yaml( + project, environment.test_repo_config, repo_path + ) + + repo_config = repo_path / "feature_store.yaml" + + repo_config.write_text(dedent(feature_store_yaml)) + + repo_example = repo_path / "example.py" + repo_example.write_text(get_example_repo("on_demand_feature_view_repo.py")) + result = runner.run(["apply"], cwd=repo_path) + assertpy.assert_that(result.returncode).is_equal_to(0) + + # entity & feature view list commands should succeed + result = runner.run(["entities", "list"], cwd=repo_path) + assertpy.assert_that(result.returncode).is_equal_to(0) + result = runner.run(["on-demand-feature-views", "list"], cwd=repo_path) + assertpy.assert_that(result.returncode).is_equal_to(0) + finally: + runner.run(["teardown"], cwd=repo_path) + + @contextmanager def setup_third_party_provider_repo(provider_name: str): with tempfile.TemporaryDirectory() as repo_dir_name: diff --git a/sdk/python/tests/integration/registration/test_registry.py b/sdk/python/tests/integration/registration/test_registry.py index f011d73d2dd..46e9a195446 100644 --- a/sdk/python/tests/integration/registration/test_registry.py +++ b/sdk/python/tests/integration/registration/test_registry.py @@ -234,6 +234,80 @@ def test_apply_feature_view_success(test_registry): test_registry._get_registry_proto() +@pytest.mark.parametrize( + "test_registry", [lazy_fixture("local_registry")], +) +def test_apply_on_demand_feature_view_success(test_registry): + # Create Feature Views + driver_stats = FileSource( + name="driver_stats_source", + path="data/driver_stats_lat_lon.parquet", + timestamp_field="event_timestamp", + created_timestamp_column="created", + description="A table describing the stats of a driver based on hourly logs", + owner="test2@gmail.com", + ) + + driver_daily_features_view = FeatureView( + name="driver_daily_features", + entities=["driver"], + ttl=timedelta(seconds=8640000000), + schema=[ + Field(name="daily_miles_driven", dtype=Float32), + Field(name="lat", dtype=Float32), + Field(name="lon", dtype=Float32), + Field(name="string_feature", dtype=String), + ], + online=True, + source=driver_stats, + tags={"production": "True"}, + owner="test2@gmail.com", + ) + + @on_demand_feature_view( + sources=[driver_daily_features_view], + schema=[Field(name="first_char", dtype=String)], + ) + def location_features_from_push(inputs: pd.DataFrame) -> pd.DataFrame: + df = pd.DataFrame() + df["first_char"] = inputs["string_feature"].str[:1].astype("string") + return df + + project = "project" + + # Register Feature View + test_registry.apply_feature_view(location_features_from_push, project) + + feature_views = test_registry.list_on_demand_feature_views(project) + + # List Feature Views + assert ( + len(feature_views) == 1 + and feature_views[0].name == "location_features_from_push" + and feature_views[0].features[0].name == "first_char" + and feature_views[0].features[0].dtype == String + ) + + feature_view = test_registry.get_on_demand_feature_view( + "location_features_from_push", project + ) + assert ( + feature_view.name == "location_features_from_push" + and feature_view.features[0].name == "first_char" + and feature_view.features[0].dtype == String + ) + + test_registry.delete_feature_view("location_features_from_push", project) + feature_views = test_registry.list_on_demand_feature_views(project) + assert len(feature_views) == 0 + + test_registry.teardown() + + # Will try to reload registry, which will fail because the file has been deleted + with pytest.raises(FileNotFoundError): + test_registry._get_registry_proto() + + @pytest.mark.parametrize( "test_registry", [lazy_fixture("local_registry")], ) From 9566299484926622247557ad43ad328898af16ad Mon Sep 17 00:00:00 2001 From: Felix Wang Date: Wed, 18 May 2022 15:28:31 -0700 Subject: [PATCH 015/123] chore: Deprecate value type (#2673) * Switch `entities` from List[str] to List[Entity] Signed-off-by: Felix Wang * Remove `value_type` from SDK Signed-off-by: Felix Wang * Remove `value_type` from tests Signed-off-by: Felix Wang * Deprecate `value_type` parameter for Entity Signed-off-by: Felix Wang * Add fields for entities to avoid type inference after removing `value_type` Signed-off-by: Felix Wang * Fix Go Signed-off-by: Felix Wang * Fix type inference Signed-off-by: Felix Wang * Another fix Signed-off-by: Felix Wang * Another fix Signed-off-by: Felix Wang * Rename Entities to EntityNames in go Signed-off-by: Felix Wang * Rename lookup Signed-off-by: Felix Wang * Rename Feature to Field Signed-off-by: Felix Wang * Clean up inference Signed-off-by: Felix Wang * Refactor Signed-off-by: Felix Wang * Use old `value_type` attribute if it still exists Signed-off-by: Felix Wang * Refactor Signed-off-by: Felix Wang * Add TODO Signed-off-by: Felix Wang * Another fix Signed-off-by: Felix Wang * Fix test Signed-off-by: Felix Wang * Add pytest.ini file to suppress pytest warnings about markers Signed-off-by: Felix Wang * Fix type test Signed-off-by: Felix Wang * Fix type test Signed-off-by: Felix Wang * Modify entity and feature inference to occur separately and add tests Signed-off-by: Felix Wang * Lint Signed-off-by: Felix Wang * Refactor inference to pass lint Signed-off-by: Felix Wang * Fix Java Signed-off-by: Felix Wang * Another fix Signed-off-by: Felix Wang * Fix ODFV repo Signed-off-by: Felix Wang * Switch deprecation version from 0.22 to 0.23 Signed-off-by: Felix Wang --- .../java-demo/feature_repo/driver_repo.py | 4 +- go/embedded/online_features.go | 22 +- go/internal/feast/featurestore.go | 4 +- go/internal/feast/model/basefeatureview.go | 8 +- go/internal/feast/model/entity.go | 12 +- go/internal/feast/model/featureview.go | 32 ++- .../feast/model/featureviewprojection.go | 6 +- .../feast/model/{feature.go => field.go} | 7 +- go/internal/feast/onlineserving/serving.go | 4 +- .../feast/onlineserving/serving_test.go | 22 +- .../server/logging/featureserviceschema.go | 9 +- .../logging/featureserviceschema_test.go | 39 +-- go/internal/test/go_integration_test_utils.go | 24 +- .../docker-compose/feast10/definitions.py | 11 +- protos/feast/core/FeatureView.proto | 10 +- sdk/python/feast/entity.py | 31 ++- sdk/python/feast/feature_logging.py | 15 +- sdk/python/feast/feature_store.py | 36 +-- sdk/python/feast/feature_view.py | 97 +++++-- sdk/python/feast/inference.py | 242 +++++++++--------- sdk/python/feast/infra/offline_stores/file.py | 5 +- .../infra/offline_stores/offline_utils.py | 16 +- sdk/python/feast/infra/online_stores/redis.py | 6 +- .../feast/infra/passthrough_provider.py | 9 +- sdk/python/feast/infra/provider.py | 2 +- sdk/python/feast/on_demand_feature_view.py | 6 +- sdk/python/feast/templates/aws/driver_repo.py | 6 +- sdk/python/feast/templates/gcp/driver_repo.py | 6 +- sdk/python/feast/templates/hbase/example.py | 4 +- sdk/python/feast/templates/local/example.py | 6 +- .../feast/templates/postgres/driver_repo.py | 2 +- .../feast/templates/snowflake/driver_repo.py | 2 +- sdk/python/feast/templates/spark/example.py | 12 +- sdk/python/pytest.ini | 4 + sdk/python/tests/data/data_creator.py | 20 +- sdk/python/tests/doctest/test_all.py | 8 +- .../example_repos/example_feature_repo_1.py | 34 +-- .../example_repos/example_feature_repo_2.py | 7 +- ..._repo_with_duplicated_featureview_names.py | 8 +- ...ample_feature_repo_with_entity_join_key.py | 11 +- .../on_demand_feature_view_repo.py | 7 +- .../tests/integration/e2e/test_usage_e2e.py | 4 +- .../feature_repos/universal/entities.py | 11 +- .../feature_repos/universal/feature_views.py | 53 ++-- .../offline_store/test_feature_logging.py | 8 +- .../test_universal_historical_retrieval.py | 5 +- .../online_store/test_universal_online.py | 6 +- .../registration/test_feature_store.py | 66 +++-- .../registration/test_inference.py | 180 +++++++++---- .../integration/registration/test_registry.py | 40 ++- .../registration/test_universal_types.py | 64 +++-- .../scaffolding/test_partial_apply.py | 6 +- .../tests/unit/diff/test_registry_diff.py | 17 +- sdk/python/tests/unit/infra/test_provider.py | 5 +- sdk/python/tests/unit/test_entity.py | 31 +-- sdk/python/tests/unit/test_feature_view.py | 4 + .../tests/unit/test_unit_feature_store.py | 2 +- .../tests/utils/online_write_benchmark.py | 6 +- 58 files changed, 747 insertions(+), 577 deletions(-) rename go/internal/feast/model/{feature.go => field.go} (63%) create mode 100644 sdk/python/pytest.ini diff --git a/examples/java-demo/feature_repo/driver_repo.py b/examples/java-demo/feature_repo/driver_repo.py index c91e5a40bed..e17a5d9cf89 100644 --- a/examples/java-demo/feature_repo/driver_repo.py +++ b/examples/java-demo/feature_repo/driver_repo.py @@ -7,14 +7,14 @@ from google.protobuf.duration_pb2 import Duration from feast.field import Field -from feast import Entity, Feature, BatchFeatureView, FileSource, ValueType +from feast import Entity, Feature, BatchFeatureView, FileSource driver_hourly_stats = FileSource( path="data/driver_stats_with_string.parquet", timestamp_field="event_timestamp", created_timestamp_column="created", ) -driver = Entity(name="driver_id", value_type=ValueType.INT64, description="driver id",) +driver = Entity(name="driver_id", description="driver id",) driver_hourly_stats_view = BatchFeatureView( name="driver_hourly_stats", entities=["driver_id"], diff --git a/go/embedded/online_features.go b/go/embedded/online_features.go index 7cd1e4ed815..710f60dca8e 100644 --- a/go/embedded/online_features.go +++ b/go/embedded/online_features.go @@ -80,12 +80,6 @@ func (s *OnlineFeatureService) GetEntityTypesMap(featureRefs []string) (map[stri viewNames[viewName] = nil } - entities, _ := s.fs.ListEntities(true) - entitiesByName := make(map[string]*model.Entity) - for _, entity := range entities { - entitiesByName[entity.Name] = entity - } - joinKeyTypes := make(map[string]int32) for viewName := range viewNames { @@ -94,9 +88,8 @@ func (s *OnlineFeatureService) GetEntityTypesMap(featureRefs []string) (map[stri // skip on demand feature views continue } - for _, entityName := range view.Entities { - entity := entitiesByName[entityName] - joinKeyTypes[entity.JoinKey] = int32(entity.ValueType.Number()) + for _, entityColumn := range view.EntityColumns { + joinKeyTypes[entityColumn.Name] = int32(entityColumn.Dtype.Number()) } } @@ -111,21 +104,14 @@ func (s *OnlineFeatureService) GetEntityTypesMapByFeatureService(featureServiceN joinKeyTypes := make(map[string]int32) - entities, _ := s.fs.ListEntities(true) - entitiesByName := make(map[string]*model.Entity) - for _, entity := range entities { - entitiesByName[entity.Name] = entity - } - for _, projection := range featureService.Projections { view, err := s.fs.GetFeatureView(projection.Name, true) if err != nil { // skip on demand feature views continue } - for _, entityName := range view.Entities { - entity := entitiesByName[entityName] - joinKeyTypes[entity.JoinKey] = int32(entity.ValueType.Number()) + for _, entityColumn := range view.EntityColumns { + joinKeyTypes[entityColumn.Name] = int32(entityColumn.Dtype.Number()) } } diff --git a/go/internal/feast/featurestore.go b/go/internal/feast/featurestore.go index 4ecd781b746..b0fc987fb46 100644 --- a/go/internal/feast/featurestore.go +++ b/go/internal/feast/featurestore.go @@ -132,7 +132,7 @@ func (fs *FeatureStore) GetOnlineFeatures( if entitylessCase { dummyEntityColumn := &prototypes.RepeatedValue{Val: make([]*prototypes.Value, numRows)} for index := 0; index < numRows; index++ { - dummyEntityColumn.Val[index] = &model.DUMMY_ENTITY + dummyEntityColumn.Val[index] = &model.DUMMY_ENTITY_VALUE } joinKeyToEntityValues[model.DUMMY_ENTITY_ID] = dummyEntityColumn } @@ -272,7 +272,7 @@ func (fs *FeatureStore) GetFeatureView(featureViewName string, hideDummyEntity b return nil, err } if fv.HasEntity(model.DUMMY_ENTITY_NAME) && hideDummyEntity { - fv.Entities = []string{} + fv.EntityNames = []string{} } return fv, nil } diff --git a/go/internal/feast/model/basefeatureview.go b/go/internal/feast/model/basefeatureview.go index 28ef7231fd5..1bdf614c25a 100644 --- a/go/internal/feast/model/basefeatureview.go +++ b/go/internal/feast/model/basefeatureview.go @@ -8,15 +8,15 @@ import ( type BaseFeatureView struct { Name string - Features []*Feature + Features []*Field Projection *FeatureViewProjection } func NewBaseFeatureView(name string, featureProtos []*core.FeatureSpecV2) *BaseFeatureView { base := &BaseFeatureView{Name: name} - features := make([]*Feature, len(featureProtos)) + features := make([]*Field, len(featureProtos)) for index, featureSpecV2 := range featureProtos { - features[index] = NewFeatureFromProto(featureSpecV2) + features[index] = NewFieldFromProto(featureSpecV2) } base.Features = features base.Projection = NewFeatureViewProjectionFromDefinition(base) @@ -43,7 +43,7 @@ func (fv *BaseFeatureView) WithProjection(projection *FeatureViewProjection) (*B } func (fv *BaseFeatureView) ProjectWithFeatures(featureNames []string) *FeatureViewProjection { - features := make([]*Feature, 0) + features := make([]*Field, 0) for _, feature := range fv.Features { for _, allowedFeatureName := range featureNames { if feature.Name == allowedFeatureName { diff --git a/go/internal/feast/model/entity.go b/go/internal/feast/model/entity.go index ac3a5d5f26e..5a09edb6557 100644 --- a/go/internal/feast/model/entity.go +++ b/go/internal/feast/model/entity.go @@ -2,18 +2,16 @@ package model import ( "github.com/feast-dev/feast/go/protos/feast/core" - "github.com/feast-dev/feast/go/protos/feast/types" ) type Entity struct { - Name string - ValueType types.ValueType_Enum - JoinKey string + Name string + JoinKey string } func NewEntityFromProto(proto *core.Entity) *Entity { - return &Entity{Name: proto.Spec.Name, - ValueType: proto.Spec.ValueType, - JoinKey: proto.Spec.JoinKey, + return &Entity{ + Name: proto.Spec.Name, + JoinKey: proto.Spec.JoinKey, } } diff --git a/go/internal/feast/model/featureview.go b/go/internal/feast/model/featureview.go index 6c198f99947..ceb3736f998 100644 --- a/go/internal/feast/model/featureview.go +++ b/go/internal/feast/model/featureview.go @@ -13,12 +13,13 @@ const ( DUMMY_ENTITY_VAL = "" ) -var DUMMY_ENTITY types.Value = types.Value{Val: &types.Value_StringVal{StringVal: DUMMY_ENTITY_VAL}} +var DUMMY_ENTITY_VALUE types.Value = types.Value{Val: &types.Value_StringVal{StringVal: DUMMY_ENTITY_VAL}} type FeatureView struct { - Base *BaseFeatureView - Ttl *durationpb.Duration - Entities []string + Base *BaseFeatureView + Ttl *durationpb.Duration + EntityNames []string + EntityColumns []*Field } func NewFeatureViewFromProto(proto *core.FeatureView) *FeatureView { @@ -26,25 +27,30 @@ func NewFeatureViewFromProto(proto *core.FeatureView) *FeatureView { Ttl: &(*proto.Spec.Ttl), } if len(proto.Spec.Entities) == 0 { - featureView.Entities = []string{DUMMY_ENTITY_NAME} + featureView.EntityNames = []string{DUMMY_ENTITY_NAME} } else { - featureView.Entities = proto.Spec.Entities + featureView.EntityNames = proto.Spec.Entities } + entityColumns := make([]*Field, len(proto.Spec.EntityColumns)) + for i, entityColumn := range proto.Spec.EntityColumns { + entityColumns[i] = NewFieldFromProto(entityColumn) + } + featureView.EntityColumns = entityColumns return featureView } -func (fs *FeatureView) NewFeatureViewFromBase(base *BaseFeatureView) *FeatureView { - ttl := durationpb.Duration{Seconds: fs.Ttl.Seconds, Nanos: fs.Ttl.Nanos} +func (fv *FeatureView) NewFeatureViewFromBase(base *BaseFeatureView) *FeatureView { + ttl := durationpb.Duration{Seconds: fv.Ttl.Seconds, Nanos: fv.Ttl.Nanos} featureView := &FeatureView{Base: base, - Ttl: &ttl, - Entities: fs.Entities, + Ttl: &ttl, + EntityNames: fv.EntityNames, } return featureView } -func (fs *FeatureView) HasEntity(lookup string) bool { - for _, entityName := range fs.Entities { - if entityName == lookup { +func (fv *FeatureView) HasEntity(name string) bool { + for _, entityName := range fv.EntityNames { + if entityName == name { return true } } diff --git a/go/internal/feast/model/featureviewprojection.go b/go/internal/feast/model/featureviewprojection.go index e80e8844ed2..fe54774ff1e 100644 --- a/go/internal/feast/model/featureviewprojection.go +++ b/go/internal/feast/model/featureviewprojection.go @@ -7,7 +7,7 @@ import ( type FeatureViewProjection struct { Name string NameAlias string - Features []*Feature + Features []*Field JoinKeyMap map[string]string } @@ -24,9 +24,9 @@ func NewFeatureViewProjectionFromProto(proto *core.FeatureViewProjection) *Featu JoinKeyMap: proto.JoinKeyMap, } - features := make([]*Feature, len(proto.FeatureColumns)) + features := make([]*Field, len(proto.FeatureColumns)) for index, featureSpecV2 := range proto.FeatureColumns { - features[index] = NewFeatureFromProto(featureSpecV2) + features[index] = NewFieldFromProto(featureSpecV2) } featureProjection.Features = features return featureProjection diff --git a/go/internal/feast/model/feature.go b/go/internal/feast/model/field.go similarity index 63% rename from go/internal/feast/model/feature.go rename to go/internal/feast/model/field.go index d833a8901b5..4f72d346866 100644 --- a/go/internal/feast/model/feature.go +++ b/go/internal/feast/model/field.go @@ -5,13 +5,14 @@ import ( "github.com/feast-dev/feast/go/protos/feast/types" ) -type Feature struct { +type Field struct { Name string Dtype types.ValueType_Enum } -func NewFeatureFromProto(proto *core.FeatureSpecV2) *Feature { - return &Feature{Name: proto.Name, +func NewFieldFromProto(proto *core.FeatureSpecV2) *Field { + return &Field{ + Name: proto.Name, Dtype: proto.ValueType, } } diff --git a/go/internal/feast/onlineserving/serving.go b/go/internal/feast/onlineserving/serving.go index 1d0567c3547..e2a2df923be 100644 --- a/go/internal/feast/onlineserving/serving.go +++ b/go/internal/feast/onlineserving/serving.go @@ -251,7 +251,7 @@ func GetEntityMaps(requestedFeatureViews []*FeatureViewAndRefs, entities []*mode joinKeyToAliasMap = map[string]string{} } - for _, entityName := range featureView.Entities { + for _, entityName := range featureView.EntityNames { joinKey := entitiesByName[entityName].JoinKey entityNameToJoinKeyMap[entityName] = joinKey @@ -518,7 +518,7 @@ func GroupFeatureRefs(requestedFeatureViews []*FeatureViewAndRefs, joinKeys := make([]string, 0) fv := featuresAndView.View featureNames := featuresAndView.FeatureRefs - for _, entityName := range fv.Entities { + for _, entityName := range fv.EntityNames { joinKeys = append(joinKeys, entityNameToJoinKeyMap[entityName]) } diff --git a/go/internal/feast/onlineserving/serving_test.go b/go/internal/feast/onlineserving/serving_test.go index 0a00f546f9e..bd4e45a21ec 100644 --- a/go/internal/feast/onlineserving/serving_test.go +++ b/go/internal/feast/onlineserving/serving_test.go @@ -20,19 +20,19 @@ func TestGroupingFeatureRefs(t *testing.T) { NameAlias: "aliasViewA", }, }, - Entities: []string{"driver", "customer"}, + EntityNames: []string{"driver", "customer"}, } viewB := &model.FeatureView{ - Base: &model.BaseFeatureView{Name: "viewB"}, - Entities: []string{"driver", "customer"}, + Base: &model.BaseFeatureView{Name: "viewB"}, + EntityNames: []string{"driver", "customer"}, } viewC := &model.FeatureView{ - Base: &model.BaseFeatureView{Name: "viewC"}, - Entities: []string{"driver"}, + Base: &model.BaseFeatureView{Name: "viewC"}, + EntityNames: []string{"driver"}, } viewD := &model.FeatureView{ - Base: &model.BaseFeatureView{Name: "viewD"}, - Entities: []string{"customer"}, + Base: &model.BaseFeatureView{Name: "viewD"}, + EntityNames: []string{"customer"}, } refGroups, _ := GroupFeatureRefs( []*FeatureViewAndRefs{ @@ -105,11 +105,11 @@ func TestGroupingFeatureRefsWithJoinKeyAliases(t *testing.T) { JoinKeyMap: map[string]string{"location_id": "destination_id"}, }, }, - Entities: []string{"location"}, + EntityNames: []string{"location"}, } viewB := &model.FeatureView{ - Base: &model.BaseFeatureView{Name: "viewB"}, - Entities: []string{"location"}, + Base: &model.BaseFeatureView{Name: "viewB"}, + EntityNames: []string{"location"}, } refGroups, _ := GroupFeatureRefs( @@ -164,7 +164,7 @@ func TestGroupingFeatureRefsWithMissingKey(t *testing.T) { JoinKeyMap: map[string]string{"location_id": "destination_id"}, }, }, - Entities: []string{"location"}, + EntityNames: []string{"location"}, } _, err := GroupFeatureRefs( diff --git a/go/internal/feast/server/logging/featureserviceschema.go b/go/internal/feast/server/logging/featureserviceschema.go index 5047346c2ca..2779982fc0a 100644 --- a/go/internal/feast/server/logging/featureserviceschema.go +++ b/go/internal/feast/server/logging/featureserviceschema.go @@ -52,13 +52,12 @@ func generateSchema(featureService *model.FeatureService, entityMap map[string]* features = append(features, fullFeatureName) allFeatureTypes[fullFeatureName] = f.Dtype } - for _, entityName := range fv.Entities { - entity := entityMap[entityName] + for _, entityColumn := range fv.EntityColumns { var joinKey string - if joinKeyAlias, ok := featureProjection.JoinKeyMap[entity.JoinKey]; ok { + if joinKeyAlias, ok := featureProjection.JoinKeyMap[entityColumn.Name]; ok { joinKey = joinKeyAlias } else { - joinKey = entity.JoinKey + joinKey = entityColumn.Name } if _, ok := joinKeysSet[joinKey]; !ok { @@ -66,7 +65,7 @@ func generateSchema(featureService *model.FeatureService, entityMap map[string]* } joinKeysSet[joinKey] = nil - entityJoinKeyToType[joinKey] = entity.ValueType + entityJoinKeyToType[joinKey] = entityColumn.Dtype } } else if odFv, ok := odFvMap[featureViewName]; ok { for _, f := range featureProjection.Features { diff --git a/go/internal/feast/server/logging/featureserviceschema_test.go b/go/internal/feast/server/logging/featureserviceschema_test.go index efcd5ec7fcc..6fa1c12e241 100644 --- a/go/internal/feast/server/logging/featureserviceschema_test.go +++ b/go/internal/feast/server/logging/featureserviceschema_test.go @@ -74,9 +74,10 @@ func TestSchemaRetrievalIgnoresEntitiesNotInFeatureService(t *testing.T) { featureService, entities, fvs, odfvs := InitializeFeatureRepoVariablesForTest() entityMap, fvMap, odFvMap := buildFCOMaps(entities, fvs, odfvs) - //Remove entities in featureservice + // Remove entities in featureservice for _, featureView := range fvs { - featureView.Entities = []string{} + featureView.EntityNames = []string{} + featureView.EntityColumns = []*model.Field{} } schema, err := generateSchema(featureService, entityMap, fvMap, odFvMap) @@ -126,65 +127,69 @@ func TestSchemaUsesOrderInFeatureService(t *testing.T) { // Initialize all dummy featureservice, entities and featureviews/on demand featureviews for testing. func InitializeFeatureRepoVariablesForTest() (*model.FeatureService, []*model.Entity, []*model.FeatureView, []*model.OnDemandFeatureView) { - f1 := test.CreateNewFeature( + f1 := test.CreateNewField( "int64", types.ValueType_INT64, ) - f2 := test.CreateNewFeature( + f2 := test.CreateNewField( "float32", types.ValueType_FLOAT, ) projection1 := test.CreateNewFeatureViewProjection( "featureView1", "", - []*model.Feature{f1, f2}, + []*model.Field{f1, f2}, map[string]string{}, ) baseFeatureView1 := test.CreateBaseFeatureView( "featureView1", - []*model.Feature{f1, f2}, + []*model.Field{f1, f2}, projection1, ) - featureView1 := test.CreateFeatureView(baseFeatureView1, nil, []string{"driver_id"}) - entity1 := test.CreateNewEntity("driver_id", types.ValueType_INT64, "driver_id") - f3 := test.CreateNewFeature( + entity1 := test.CreateNewEntity("driver_id", "driver_id") + entitycolumn1 := test.CreateNewField( + "driver_id", + types.ValueType_INT64, + ) + featureView1 := test.CreateFeatureView(baseFeatureView1, nil, []string{"driver_id"}, []*model.Field{entitycolumn1}) + f3 := test.CreateNewField( "int32", types.ValueType_INT32, ) - f4 := test.CreateNewFeature( + f4 := test.CreateNewField( "double", types.ValueType_DOUBLE, ) projection2 := test.CreateNewFeatureViewProjection( "featureView2", "", - []*model.Feature{f3, f4}, + []*model.Field{f3, f4}, map[string]string{}, ) baseFeatureView2 := test.CreateBaseFeatureView( "featureView2", - []*model.Feature{f3, f4}, + []*model.Field{f3, f4}, projection2, ) - featureView2 := test.CreateFeatureView(baseFeatureView2, nil, []string{"driver_id"}) + featureView2 := test.CreateFeatureView(baseFeatureView2, nil, []string{"driver_id"}, []*model.Field{entitycolumn1}) - f5 := test.CreateNewFeature( + f5 := test.CreateNewField( "odfv_f1", types.ValueType_INT32, ) - f6 := test.CreateNewFeature( + f6 := test.CreateNewField( "odfv_f2", types.ValueType_DOUBLE, ) projection3 := test.CreateNewFeatureViewProjection( "od_bf1", "", - []*model.Feature{f5, f6}, + []*model.Field{f5, f6}, map[string]string{}, ) od_bf1 := test.CreateBaseFeatureView( "od_bf1", - []*model.Feature{f5, f6}, + []*model.Field{f5, f6}, projection3, ) odfv := model.NewOnDemandFeatureViewFromBase(od_bf1) diff --git a/go/internal/test/go_integration_test_utils.go b/go/internal/test/go_integration_test_utils.go index eb727ba1db9..275edc7b98e 100644 --- a/go/internal/test/go_integration_test_utils.go +++ b/go/internal/test/go_integration_test_utils.go @@ -193,7 +193,7 @@ func GetProtoFromRecord(rec arrow.Record) (map[string]*types.RepeatedValue, erro return r, nil } -func CreateBaseFeatureView(name string, features []*model.Feature, projection *model.FeatureViewProjection) *model.BaseFeatureView { +func CreateBaseFeatureView(name string, features []*model.Field, projection *model.FeatureViewProjection) *model.BaseFeatureView { return &model.BaseFeatureView{ Name: name, Features: features, @@ -201,16 +201,15 @@ func CreateBaseFeatureView(name string, features []*model.Feature, projection *m } } -func CreateNewEntity(name string, valueType types.ValueType_Enum, joinKey string) *model.Entity { +func CreateNewEntity(name string, joinKey string) *model.Entity { return &model.Entity{ - Name: name, - ValueType: valueType, - JoinKey: joinKey, + Name: name, + JoinKey: joinKey, } } -func CreateNewFeature(name string, dtype types.ValueType_Enum) *model.Feature { - return &model.Feature{Name: name, +func CreateNewField(name string, dtype types.ValueType_Enum) *model.Field { + return &model.Field{Name: name, Dtype: dtype, } } @@ -225,7 +224,7 @@ func CreateNewFeatureService(name string, project string, createdTimestamp *time } } -func CreateNewFeatureViewProjection(name string, nameAlias string, features []*model.Feature, joinKeyMap map[string]string) *model.FeatureViewProjection { +func CreateNewFeatureViewProjection(name string, nameAlias string, features []*model.Field, joinKeyMap map[string]string) *model.FeatureViewProjection { return &model.FeatureViewProjection{Name: name, NameAlias: nameAlias, Features: features, @@ -233,10 +232,11 @@ func CreateNewFeatureViewProjection(name string, nameAlias string, features []*m } } -func CreateFeatureView(base *model.BaseFeatureView, ttl *durationpb.Duration, entities []string) *model.FeatureView { +func CreateFeatureView(base *model.BaseFeatureView, ttl *durationpb.Duration, entities []string, entityColumns []*model.Field) *model.FeatureView { return &model.FeatureView{ - Base: base, - Ttl: ttl, - Entities: entities, + Base: base, + Ttl: ttl, + EntityNames: entities, + EntityColumns: entityColumns, } } diff --git a/java/serving/src/test/resources/docker-compose/feast10/definitions.py b/java/serving/src/test/resources/docker-compose/feast10/definitions.py index e514ef0d239..806995ec06f 100644 --- a/java/serving/src/test/resources/docker-compose/feast10/definitions.py +++ b/java/serving/src/test/resources/docker-compose/feast10/definitions.py @@ -19,22 +19,23 @@ # Define an entity for the driver. You can think of entity as a primary key used to # fetch features. -driver = Entity(name="driver_id", value_type=ValueType.INT64, description="driver id",) +driver = Entity(name="driver_id", description="driver id") # Our parquet files contain sample data that includes a driver_id column, timestamps and # three feature column. Here we define a Feature View that will allow us to serve this # data to our model online. driver_hourly_stats_view = FeatureView( name="driver_hourly_stats", - entities=["driver_id"], + entities=[driver], ttl=Duration(seconds=86400 * 7), schema=[ Field(name="conv_rate", dtype=Float64), Field(name="acc_rate", dtype=Float32), Field(name="avg_daily_trips", dtype=Int64), + Field(name="driver_id", dtype=Int64), ], online=True, - batch_source=driver_hourly_stats, + source=driver_hourly_stats, tags={}, ) @@ -75,11 +76,11 @@ def transformed_conv_rate(features_df: pd.DataFrame) -> pd.DataFrame: benchmark_feature_views = [ FeatureView( name=f"feature_view_{i}", - entities=["entity"], + entities=[entity], ttl=Duration(seconds=86400), schema=[Field(name=f"feature_{10 * i + j}", dtype=Int64) for j in range(10)], online=True, - batch_source=generated_data_source, + source=generated_data_source, ) for i in range(25) ] diff --git a/protos/feast/core/FeatureView.proto b/protos/feast/core/FeatureView.proto index 26623505401..c9e38bf3448 100644 --- a/protos/feast/core/FeatureView.proto +++ b/protos/feast/core/FeatureView.proto @@ -35,7 +35,7 @@ message FeatureView { FeatureViewMeta meta = 2; } -// Next available id: 12 +// Next available id: 13 // TODO(adchia): refactor common fields from this and ODFV into separate metadata proto message FeatureViewSpec { // Name of the feature view. Must be unique. Not updated. @@ -44,13 +44,15 @@ message FeatureViewSpec { // Name of Feast project that this feature view belongs to. string project = 2; - // List names of entities to associate with the Features defined in this - // Feature View. Not updatable. + // List of names of entities associated with this feature view. repeated string entities = 3; - // List of specifications for each field defined as part of this feature view. + // List of specifications for each feature defined as part of this feature view. repeated FeatureSpecV2 features = 4; + // List of specifications for each entity defined as part of this feature view. + repeated FeatureSpecV2 entity_columns = 12; + // Description of the feature view. string description = 10; diff --git a/sdk/python/feast/entity.py b/sdk/python/feast/entity.py index 1e7c7cf3079..29511adcb2e 100644 --- a/sdk/python/feast/entity.py +++ b/sdk/python/feast/entity.py @@ -31,7 +31,7 @@ class Entity: Attributes: name: The unique name of the entity. - value_type: The type of the entity, such as string or float. + value_type (deprecated): The type of the entity, such as string or float. join_key: A property that uniquely identifies different entities within the collection. The join_key property is typically used for joining entities with their associated features. If not specified, defaults to the name. @@ -60,7 +60,7 @@ def __init__( self, *args, name: Optional[str] = None, - value_type: ValueType = ValueType.UNKNOWN, + value_type: Optional[ValueType] = None, description: str = "", join_key: Optional[str] = None, tags: Optional[Dict[str, str]] = None, @@ -72,7 +72,7 @@ def __init__( Args: name: The unique name of the entity. - value_type: The type of the entity, such as string or float. + value_type (deprecated): The type of the entity, such as string or float. description: A human-readable description. join_key (deprecated): A property that uniquely identifies different entities within the collection. The join_key property is typically used for joining entities @@ -104,8 +104,23 @@ def __init__( if not self.name: raise ValueError("Name needs to be specified") - self.value_type = value_type + if value_type: + warnings.warn( + ( + "The `value_type` parameter is being deprecated. Instead, the type of an entity " + "should be specified as a Field in the schema of a feature view. Feast 0.23 and " + "onwards will not support the `value_type` parameter. The `entities` parameter of " + "feature views should also be changed to a List[Entity] instead of a List[str]; if " + "this is not done, entity columns will be mistakenly interpreted as feature columns." + ), + DeprecationWarning, + ) + self.value_type = value_type or ValueType.UNKNOWN + # For now, both the `join_key` and `join_keys` attributes are set correctly, + # so both are usable. + # TODO(felixwang9817): Remove the usage of `join_key` throughout the codebase + # when the usage of `join_key` as a parameter is removed. if join_key: warnings.warn( ( @@ -125,6 +140,8 @@ def __init__( self.join_key = join_keys[0] else: self.join_key = join_key if join_key else self.name + if not self.join_keys: + self.join_keys = [self.join_key] self.description = description self.tags = tags if tags is not None else {} self.owner = owner @@ -153,6 +170,9 @@ def __eq__(self, other): def __str__(self): return str(MessageToJson(self.to_proto())) + def __lt__(self, other): + return self.name < other.name + def is_valid(self): """ Validates the state of this entity locally. @@ -179,13 +199,14 @@ def from_proto(cls, entity_proto: EntityProto): """ entity = cls( name=entity_proto.spec.name, - value_type=ValueType(entity_proto.spec.value_type), join_keys=[entity_proto.spec.join_key], description=entity_proto.spec.description, tags=entity_proto.spec.tags, owner=entity_proto.spec.owner, ) + entity.value_type = ValueType(entity_proto.spec.value_type) + if entity_proto.meta.HasField("created_timestamp"): entity.created_timestamp = entity_proto.meta.created_timestamp.ToDatetime() if entity_proto.meta.HasField("last_updated_timestamp"): diff --git a/sdk/python/feast/feature_logging.py b/sdk/python/feast/feature_logging.py index b2d77f68fef..04f30ab81a1 100644 --- a/sdk/python/feast/feature_logging.py +++ b/sdk/python/feast/feature_logging.py @@ -11,11 +11,10 @@ FeatureViewNotFoundException, OnDemandFeatureViewNotFoundException, ) -from feast.feature_view import DUMMY_ENTITY_NAME +from feast.feature_view import DUMMY_ENTITY_ID from feast.protos.feast.core.FeatureService_pb2 import ( LoggingConfig as LoggingConfigProto, ) -from feast.types import from_value_type if TYPE_CHECKING: from feast import FeatureService @@ -77,18 +76,14 @@ def get_schema(self, registry: "Registry") -> pa.Schema: fields[field.name] = FEAST_TYPE_TO_ARROW_TYPE[field.dtype] else: - for entity_name in feature_view.entities: - if entity_name == DUMMY_ENTITY_NAME: + for entity_column in feature_view.entity_columns: + if entity_column.name == DUMMY_ENTITY_ID: continue - entity = registry.get_entity(entity_name, self._project) - join_key = projection.join_key_map.get( - entity.join_key, entity.join_key + entity_column.name, entity_column.name ) - fields[join_key] = FEAST_TYPE_TO_ARROW_TYPE[ - from_value_type(entity.value_type) - ] + fields[join_key] = FEAST_TYPE_TO_ARROW_TYPE[entity_column.dtype] for feature in projection.features: fields[ diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py index 791f1ae40e5..768823a68c0 100644 --- a/sdk/python/feast/feature_store.py +++ b/sdk/python/feast/feature_store.py @@ -66,8 +66,7 @@ ) from feast.inference import ( update_data_sources_with_inferred_event_timestamp_col, - update_entities_with_inferred_types_from_feature_views, - update_feature_views_with_inferred_features, + update_feature_views_with_inferred_features_and_entities, ) from feast.infra.infra_object import Infra from feast.infra.provider import Provider, RetrievalJob, get_provider @@ -256,6 +255,7 @@ def _list_feature_views( ): if hide_dummy_entity and fv.entities[0] == DUMMY_ENTITY_NAME: fv.entities = [] + fv.entity_columns = [] feature_views.append(fv) return feature_views @@ -480,10 +480,6 @@ def _make_inferences( feature_services_to_update: List[FeatureService], ): """Makes inferences for entities, feature views, odfvs, and feature services.""" - update_entities_with_inferred_types_from_feature_views( - entities_to_update, views_to_update, self.config - ) - update_data_sources_with_inferred_event_timestamp_col( data_sources_to_update, self.config ) @@ -494,7 +490,7 @@ def _make_inferences( # New feature views may reference previously applied entities. entities = self._list_entities() - update_feature_views_with_inferred_features( + update_feature_views_with_inferred_features_and_entities( views_to_update, entities + entities_to_update, self.config ) @@ -524,11 +520,11 @@ def _plan( Examples: Generate a plan adding an Entity and a FeatureView. - >>> from feast import FeatureStore, Entity, FeatureView, Feature, ValueType, FileSource, RepoConfig + >>> from feast import FeatureStore, Entity, FeatureView, Feature, FileSource, RepoConfig >>> from feast.feature_store import RepoContents >>> from datetime import timedelta >>> fs = FeatureStore(repo_path="feature_repo") - >>> driver = Entity(name="driver_id", value_type=ValueType.INT64, description="driver id") + >>> driver = Entity(name="driver_id", description="driver id") >>> driver_hourly_stats = FileSource( ... path="feature_repo/data/driver_stats.parquet", ... timestamp_field="event_timestamp", @@ -536,7 +532,7 @@ def _plan( ... ) >>> driver_hourly_stats_view = FeatureView( ... name="driver_hourly_stats", - ... entities=["driver_id"], + ... entities=[driver], ... ttl=timedelta(seconds=86400 * 1), ... batch_source=driver_hourly_stats, ... ) @@ -637,10 +633,10 @@ def apply( Examples: Register an Entity and a FeatureView. - >>> from feast import FeatureStore, Entity, FeatureView, Feature, ValueType, FileSource, RepoConfig + >>> from feast import FeatureStore, Entity, FeatureView, Feature, FileSource, RepoConfig >>> from datetime import timedelta >>> fs = FeatureStore(repo_path="feature_repo") - >>> driver = Entity(name="driver_id", value_type=ValueType.INT64, description="driver id") + >>> driver = Entity(name="driver_id", description="driver id") >>> driver_hourly_stats = FileSource( ... path="feature_repo/data/driver_stats.parquet", ... timestamp_field="event_timestamp", @@ -648,7 +644,7 @@ def apply( ... ) >>> driver_hourly_stats_view = FeatureView( ... name="driver_hourly_stats", - ... entities=["driver_id"], + ... entities=[driver], ... ttl=timedelta(seconds=86400 * 1), ... batch_source=driver_hourly_stats, ... ) @@ -695,6 +691,9 @@ def apply( data_sources_to_update = list(data_sources_set_to_update) + # Handle all entityless feature views by using DUMMY_ENTITY as a placeholder entity. + entities_to_update.append(DUMMY_ENTITY) + # Validate all feature views and make inferences. self._validate_all_feature_views( views_to_update, odfvs_to_update, request_views_to_update @@ -707,9 +706,6 @@ def apply( services_to_update, ) - # Handle all entityless feature views by using DUMMY_ENTITY as a placeholder entity. - entities_to_update.append(DUMMY_ENTITY) - # Add all objects to the registry and update the provider's infrastructure. for ds in data_sources_to_update: self._registry.apply_data_source(ds, project=self.project, commit=False) @@ -1562,12 +1558,12 @@ def _get_columnar_entity_values( def _get_entity_maps( self, feature_views ) -> Tuple[Dict[str, str], Dict[str, ValueType], Set[str]]: + # TODO(felixwang9817): Support entities that have different types for different feature views. entities = self._list_entities(allow_cache=True, hide_dummy_entity=False) entity_name_to_join_key_map: Dict[str, str] = {} entity_type_map: Dict[str, ValueType] = {} for entity in entities: entity_name_to_join_key_map[entity.name] = entity.join_key - entity_type_map[entity.name] = entity.value_type for feature_view in feature_views: for entity_name in feature_view.entities: entity = self._registry.get_entity( @@ -1582,7 +1578,11 @@ def _get_entity_maps( entity.join_key, entity.join_key ) entity_name_to_join_key_map[entity_name] = join_key - entity_type_map[join_key] = entity.value_type + for entity_column in feature_view.entity_columns: + entity_type_map[ + entity_column.name + ] = entity_column.dtype.to_value_type() + return ( entity_name_to_join_key_map, entity_type_map, diff --git a/sdk/python/feast/feature_view.py b/sdk/python/feast/feature_view.py index 6a8fd332174..12ce9105f76 100644 --- a/sdk/python/feast/feature_view.py +++ b/sdk/python/feast/feature_view.py @@ -36,7 +36,6 @@ MaterializationInterval as MaterializationIntervalProto, ) from feast.usage import log_exceptions -from feast.value_type import ValueType warnings.simplefilter("once", DeprecationWarning) @@ -44,9 +43,7 @@ DUMMY_ENTITY_ID = "__dummy_id" DUMMY_ENTITY_NAME = "__dummy" DUMMY_ENTITY_VAL = "" -DUMMY_ENTITY = Entity( - name=DUMMY_ENTITY_NAME, join_keys=[DUMMY_ENTITY_ID], value_type=ValueType.STRING, -) +DUMMY_ENTITY = Entity(name=DUMMY_ENTITY_NAME, join_keys=[DUMMY_ENTITY_ID],) class FeatureView(BaseFeatureView): @@ -55,7 +52,7 @@ class FeatureView(BaseFeatureView): Attributes: name: The unique name of the feature view. - entities: The list of entities with which this group of features is associated. + entities: The list of names of entities that this feature view is associated with. ttl: The amount of time this group of features lives. A ttl of 0 indicates that this group of features lives forever. Note that large ttl's or a ttl of 0 can result in extremely computationally intensive queries. @@ -65,9 +62,11 @@ class FeatureView(BaseFeatureView): stream_source (optional): The stream source of data where this group of features is stored. This is deprecated in favor of `source`. schema: The schema of the feature view, including feature, timestamp, and entity - columns. - features: The list of features defined as part of this feature view. Each - feature should also be included in the schema. + columns. If not specified, can be inferred from the underlying data source. + entity_columns: The list of entity columns contained in the schema. If not specified, + can be inferred from the underlying data source. + features: The list of feature columns contained in the schema. If not specified, + can be inferred from the underlying data source. online: A boolean indicating whether online retrieval is enabled for this feature view. description: A human-readable description. @@ -84,6 +83,7 @@ class FeatureView(BaseFeatureView): batch_source: DataSource stream_source: Optional[DataSource] schema: List[Field] + entity_columns: List[Field] features: List[Field] online: bool description: str @@ -129,14 +129,15 @@ def __init__( owner (optional): The owner of the feature view, typically the email of the primary maintainer. schema (optional): The schema of the feature view, including feature, timestamp, - and entity columns. + and entity columns. If entity columns are included in the schema, a List[Entity] + must be passed to `entities` instead of a List[str]; otherwise, the entity columns + will be mistakenly interpreted as feature columns. source (optional): The source of data for this group of features. May be a stream source, or a batch source. If a stream source, the source should contain a batch_source for backfills & batch materialization. Raises: ValueError: A field mapping conflicts with an Entity or a Feature. """ - positional_attributes = ["name", "entities", "ttl"] _name = name @@ -167,11 +168,21 @@ def __init__( raise ValueError("feature view name needs to be specified") self.name = _name + self.entities = ( [e.name if isinstance(e, Entity) else e for e in _entities] if _entities else [DUMMY_ENTITY_NAME] ) + if _entities and isinstance(_entities[0], str): + warnings.warn( + ( + "The `entities` parameter should be a list of `Entity` objects. " + "Feast 0.23 and onwards will not support passing in a list of " + "strings to define entities." + ), + DeprecationWarning, + ) self._initialize_sources(_name, batch_source, stream_source, source) @@ -206,14 +217,30 @@ def __init__( _schema = [Field.from_feature(feature) for feature in features] self.schema = _schema - # TODO(felixwang9817): Infer which fields in the schema are features, timestamps, - # and entities. For right now we assume that all fields are features, since the - # current `features` parameter only accepts feature columns. - _features = _schema + # If a user has added entity fields to schema, then they should also have switched + # to using a List[Entity], in which case entity and feature columns can be separated + # here. Conversely, if the user is still using a List[str], they must not have added + # added entity fields, in which case we can set the `features` attribute directly + # equal to the schema. + _features: List[Field] = [] + self.entity_columns = [] + if _entities and len(_entities) > 0 and isinstance(_entities[0], str): + _features = _schema + else: + join_keys = [] + if _entities: + for entity in _entities: + if isinstance(entity, Entity): + join_keys += entity.join_keys + + for field in _schema: + if field.name in join_keys: + self.entity_columns.append(field) + else: + _features.append(field) - cols = [entity for entity in self.entities] + [ - field.name for field in _features - ] + # TODO(felixwang9817): Add more robust validation of features. + cols = [field.name for field in _schema] for col in cols: if ( self.batch_source.field_mapping is not None @@ -276,7 +303,6 @@ def __hash__(self): def __copy__(self): fv = FeatureView( name=self.name, - entities=self.entities, ttl=self.ttl, source=self.batch_source, stream_source=self.stream_source, @@ -284,6 +310,13 @@ def __copy__(self): tags=self.tags, online=self.online, ) + + # This is deliberately set outside of the FV initialization to avoid the deprecation warning. + # TODO(felixwang9817): Move this into the FV initialization when the deprecation warning + # is removed. + fv.entities = self.entities + fv.features = copy.copy(self.features) + fv.entity_columns = copy.copy(self.entity_columns) fv.projection = copy.copy(self.projection) fv.entities = self.entities return fv @@ -303,12 +336,17 @@ def __eq__(self, other): or self.online != other.online or self.batch_source != other.batch_source or self.stream_source != other.stream_source - or self.schema != other.schema + or sorted(self.entity_columns) != sorted(other.entity_columns) ): return False return True + @property + def join_keys(self) -> List[str]: + """Returns a list of all the join keys.""" + return [entity.name for entity in self.entity_columns] + def ensure_valid(self): """ Validates the state of this feature view locally. @@ -394,7 +432,8 @@ def to_proto(self) -> FeatureViewProto: spec = FeatureViewSpecProto( name=self.name, entities=self.entities, - features=[field.to_proto() for field in self.schema], + entity_columns=[field.to_proto() for field in self.entity_columns], + features=[field.to_proto() for field in self.features], description=self.description, tags=self.tags, owner=self.owner, @@ -425,11 +464,6 @@ def from_proto(cls, feature_view_proto: FeatureViewProto): ) feature_view = cls( name=feature_view_proto.spec.name, - entities=[entity for entity in feature_view_proto.spec.entities], - schema=[ - Field.from_proto(field_proto) - for field_proto in feature_view_proto.spec.features - ], description=feature_view_proto.spec.description, tags=dict(feature_view_proto.spec.tags), owner=feature_view_proto.spec.owner, @@ -444,6 +478,19 @@ def from_proto(cls, feature_view_proto: FeatureViewProto): if stream_source: feature_view.stream_source = stream_source + # This avoids the deprecation warning. + feature_view.entities = feature_view_proto.spec.entities + + # Instead of passing in a schema, we set the features and entity columns. + feature_view.features = [ + Field.from_proto(field_proto) + for field_proto in feature_view_proto.spec.features + ] + feature_view.entity_columns = [ + Field.from_proto(field_proto) + for field_proto in feature_view_proto.spec.entity_columns + ] + # FeatureViewProjections are not saved in the FeatureView proto. # Create the default projection. feature_view.projection = FeatureViewProjection.from_definition(feature_view) diff --git a/sdk/python/feast/inference.py b/sdk/python/feast/inference.py index 076cbc86ceb..aed90c4ac83 100644 --- a/sdk/python/feast/inference.py +++ b/sdk/python/feast/inference.py @@ -1,73 +1,20 @@ import re -from typing import List +from typing import List, Set -from feast import BigQuerySource, Entity, FileSource, RedshiftSource, SnowflakeSource from feast.data_source import DataSource, PushSource, RequestSource +from feast.entity import Entity from feast.errors import RegistryInferenceFailure -from feast.feature_view import FeatureView +from feast.feature_view import DUMMY_ENTITY_ID, DUMMY_ENTITY_NAME, FeatureView from feast.field import Field, from_value_type +from feast.infra.offline_stores.bigquery_source import BigQuerySource +from feast.infra.offline_stores.file_source import FileSource +from feast.infra.offline_stores.redshift_source import RedshiftSource +from feast.infra.offline_stores.snowflake_source import SnowflakeSource from feast.repo_config import RepoConfig +from feast.types import String from feast.value_type import ValueType -def update_entities_with_inferred_types_from_feature_views( - entities: List[Entity], feature_views: List[FeatureView], config: RepoConfig -) -> None: - """ - Infers the types of the entities by examining the schemas of feature view batch sources. - - Args: - entities: The entities to be updated. - feature_views: A list containing feature views associated with the entities. - config: The config for the current feature store. - """ - incomplete_entities = { - entity.name: entity - for entity in entities - if entity.value_type == ValueType.UNKNOWN - } - incomplete_entities_keys = incomplete_entities.keys() - - for view in feature_views: - if not (incomplete_entities_keys & set(view.entities)): - continue # skip if view doesn't contain any entities that need inference - - col_names_and_types = list( - view.batch_source.get_table_column_names_and_types(config) - ) - for entity_name in view.entities: - if entity_name in incomplete_entities: - entity = incomplete_entities[entity_name] - - # get entity information from information extracted from the view batch source - extracted_entity_name_type_pairs = list( - filter(lambda tup: tup[0] == entity.join_key, col_names_and_types,) - ) - if len(extracted_entity_name_type_pairs) == 0: - # Doesn't mention inference error because would also be an error without inferencing - raise ValueError( - f"""No column in the batch source for the {view.name} feature view matches - its entity's name.""" - ) - - inferred_value_type = view.batch_source.source_datatype_to_feast_value_type()( - extracted_entity_name_type_pairs[0][1] - ) - - if ( - entity.value_type != ValueType.UNKNOWN - and entity.value_type != inferred_value_type - ) or (len(extracted_entity_name_type_pairs) > 1): - raise RegistryInferenceFailure( - "Entity", - f"""Entity value_type inference failed for {entity_name} entity. - Multiple viable matches. - """, - ) - - entity.value_type = inferred_value_type - - def update_data_sources_with_inferred_event_timestamp_col( data_sources: List[DataSource], config: RepoConfig ) -> None: @@ -140,77 +87,144 @@ def update_data_sources_with_inferred_event_timestamp_col( ) -def update_feature_views_with_inferred_features( +def update_feature_views_with_inferred_features_and_entities( fvs: List[FeatureView], entities: List[Entity], config: RepoConfig ) -> None: """ - Infers the set of features associated to each FeatureView and updates the FeatureView with those features. - Inference occurs through considering each column of the underlying data source as a feature except columns that are - associated with the data source's timestamp columns and the FeatureView's entity columns. + Infers the features and entities associated with each feature view and updates it in place. + + Columns whose names match a join key of an entity are considered to be entity columns; all + other columns except designated timestamp columns are considered to be feature columns. If + the feature view already has features, feature inference is skipped. Args: fvs: The feature views to be updated. entities: A list containing entities associated with the feature views. config: The config for the current feature store. """ - entity_name_to_join_key_map = {entity.name: entity.join_key for entity in entities} - join_keys = entity_name_to_join_key_map.values() + entity_name_to_entity_map = {e.name: e for e in entities} + entity_name_to_join_keys_map = {e.name: e.join_keys for e in entities} for fv in fvs: - # First drop all Entity fields. Then infer features if necessary. - fv.schema = [field for field in fv.schema if field.name not in join_keys] - fv.features = [field for field in fv.features if field.name not in join_keys] - - if not fv.features: - columns_to_exclude = { - fv.batch_source.timestamp_field, - fv.batch_source.created_timestamp_column, - } | { - entity_name_to_join_key_map[entity_name] for entity_name in fv.entities - } - - if fv.batch_source.timestamp_field in fv.batch_source.field_mapping: - columns_to_exclude.add( - fv.batch_source.field_mapping[fv.batch_source.timestamp_field] - ) + join_keys = set( + [ + join_key + for entity_name in fv.entities + for join_key in entity_name_to_join_keys_map[entity_name] + ] + ) + + # Fields whose names match a join key are considered to be entity columns; all + # other fields are considered to be feature columns. + for field in fv.schema: + if field.name in join_keys: + # Do not override a preexisting field with the same name. + if field.name not in [ + entity_column.name for entity_column in fv.entity_columns + ]: + fv.entity_columns.append(field) + else: + if field.name not in [feature.name for feature in fv.features]: + fv.features.append(field) + + # Since the `value_type` parameter has not yet been fully deprecated for + # entities, we respect the `value_type` attribute if it still exists. + for entity_name in fv.entities: + entity = entity_name_to_entity_map[entity_name] if ( - fv.batch_source.created_timestamp_column - in fv.batch_source.field_mapping + entity.join_key + not in [entity_column.name for entity_column in fv.entity_columns] + and entity.value_type != ValueType.UNKNOWN ): - columns_to_exclude.add( - fv.batch_source.field_mapping[ - fv.batch_source.created_timestamp_column - ] + fv.entity_columns.append( + Field( + name=entity.join_key, dtype=from_value_type(entity.value_type), + ) ) - for ( - col_name, - col_datatype, - ) in fv.batch_source.get_table_column_names_and_types(config): - if col_name not in columns_to_exclude and not re.match( - "^__|__$", - col_name, # double underscores often signal an internal-use column - ): - feature_name = ( - fv.batch_source.field_mapping[col_name] - if col_name in fv.batch_source.field_mapping - else col_name - ) - field = Field( - name=feature_name, - dtype=from_value_type( - fv.batch_source.source_datatype_to_feast_value_type()( - col_datatype - ) - ), - ) - # Note that schema and features are two different attributes of a - # FeatureView, and that features should be present in both. - fv.schema.append(field) - fv.features.append(field) + # Infer a dummy entity column for entityless feature views. + if len(fv.entities) == 1 and fv.entities[0] == DUMMY_ENTITY_NAME: + fv.entity_columns.append(Field(name=DUMMY_ENTITY_ID, dtype=String)) + + # Run inference for entity columns if there are fewer entity fields than expected. + num_expected_join_keys = sum( + [ + len(entity_name_to_join_keys_map[entity_name]) + for entity_name in fv.entities + ] + ) + run_inference_for_entities = len(fv.entity_columns) < num_expected_join_keys + + # Run inference for feature columns if there are no feature fields. + run_inference_for_features = len(fv.features) == 0 + + if run_inference_for_entities or run_inference_for_features: + _infer_features_and_entities( + fv, join_keys, run_inference_for_features, config, + ) if not fv.features: raise RegistryInferenceFailure( "FeatureView", f"Could not infer Features for the FeatureView named {fv.name}.", ) + + +def _infer_features_and_entities( + fv: FeatureView, join_keys: Set[str], run_inference_for_features, config, +) -> None: + """ + Updates the specific feature in place with inferred features and entities. + + Args: + fv: The feature view on which to run inference. + join_keys: The set of join keys for the feature view's entities. + run_inference_for_features: Whether to run inference for features. + config: The config for the current feature store. + """ + columns_to_exclude = { + fv.batch_source.timestamp_field, + fv.batch_source.created_timestamp_column, + } + for column in columns_to_exclude: + if column in fv.batch_source.field_mapping: + columns_to_exclude.remove(column) + columns_to_exclude.add(fv.batch_source.field_mapping[column]) + + table_column_names_and_types = fv.batch_source.get_table_column_names_and_types( + config + ) + + for col_name, col_datatype in table_column_names_and_types: + if col_name in columns_to_exclude: + continue + elif col_name in join_keys: + field = Field( + name=col_name, + dtype=from_value_type( + fv.batch_source.source_datatype_to_feast_value_type()(col_datatype) + ), + ) + if field.name not in [ + entity_column.name for entity_column in fv.entity_columns + ]: + fv.entity_columns.append(field) + elif not re.match( + "^__|__$", col_name + ): # double underscores often signal an internal-use column + if run_inference_for_features: + feature_name = ( + fv.batch_source.field_mapping[col_name] + if col_name in fv.batch_source.field_mapping + else col_name + ) + field = Field( + name=feature_name, + dtype=from_value_type( + fv.batch_source.source_datatype_to_feast_value_type()( + col_datatype + ) + ), + ) + if field.name not in [feature.name for feature in fv.features]: + fv.features.append(field) diff --git a/sdk/python/feast/infra/offline_stores/file.py b/sdk/python/feast/infra/offline_stores/file.py index 7670ad67b77..de1168a86fb 100644 --- a/sdk/python/feast/infra/offline_stores/file.py +++ b/sdk/python/feast/infra/offline_stores/file.py @@ -200,10 +200,9 @@ def evaluate_historical_retrieval(): # Build a list of entity columns to join on (from the right table) join_keys = [] - for entity_name in feature_view.entities: - entity = registry.get_entity(entity_name, project) + for entity_column in feature_view.entity_columns: join_key = feature_view.projection.join_key_map.get( - entity.join_key, entity.join_key + entity_column.name, entity_column.name ) join_keys.append(join_key) diff --git a/sdk/python/feast/infra/offline_stores/offline_utils.py b/sdk/python/feast/infra/offline_stores/offline_utils.py index b6c3d300d49..dad0ca5b781 100644 --- a/sdk/python/feast/infra/offline_stores/offline_utils.py +++ b/sdk/python/feast/infra/offline_stores/offline_utils.py @@ -60,11 +60,9 @@ def get_expected_join_keys( ) -> Set[str]: join_keys = set() for feature_view in feature_views: - entities = feature_view.entities - for entity_name in entities: - entity = registry.get_entity(entity_name, project) + for entity_column in feature_view.entity_columns: join_key = feature_view.projection.join_key_map.get( - entity.join_key, entity.join_key + entity_column.name, entity_column.name ) join_keys.add(join_key) return join_keys @@ -114,14 +112,14 @@ def get_feature_view_query_context( query_context = [] for feature_view, features in feature_views_to_feature_map.items(): - join_keys, entity_selections = [], [] - for entity_name in feature_view.entities: - entity = registry.get_entity(entity_name, project) + join_keys: List[str] = [] + entity_selections: List[str] = [] + for entity_column in feature_view.entity_columns: join_key = feature_view.projection.join_key_map.get( - entity.join_key, entity.join_key + entity_column.name, entity_column.name ) join_keys.append(join_key) - entity_selections.append(f"{entity.join_key} AS {join_key}") + entity_selections.append(f"{entity_column.name} AS {join_key}") if isinstance(feature_view.ttl, timedelta): ttl_seconds = int(feature_view.ttl.total_seconds()) diff --git a/sdk/python/feast/infra/online_stores/redis.py b/sdk/python/feast/infra/online_stores/redis.py index 9ceceff0ac0..1a8eedb21e0 100644 --- a/sdk/python/feast/infra/online_stores/redis.py +++ b/sdk/python/feast/infra/online_stores/redis.py @@ -106,9 +106,9 @@ def update( (usually this happens when the last feature view that was using specific compound key is deleted) and remove all features attached to this "join_keys". """ - join_keys_to_keep = set(tuple(table.entities) for table in tables_to_keep) + join_keys_to_keep = set(tuple(table.join_keys) for table in tables_to_keep) - join_keys_to_delete = set(tuple(table.entities) for table in tables_to_delete) + join_keys_to_delete = set(tuple(table.join_keys) for table in tables_to_delete) for join_keys in join_keys_to_delete - join_keys_to_keep: self.delete_entity_values(config, list(join_keys)) @@ -122,7 +122,7 @@ def teardown( """ We delete the keys in redis for tables/views being removed. """ - join_keys_to_delete = set(tuple(table.entities) for table in tables) + join_keys_to_delete = set(tuple(table.join_keys) for table in tables) for join_keys in join_keys_to_delete: self.delete_entity_values(config, list(join_keys)) diff --git a/sdk/python/feast/infra/passthrough_provider.py b/sdk/python/feast/infra/passthrough_provider.py index b5965c91bfb..f01fd9bac61 100644 --- a/sdk/python/feast/infra/passthrough_provider.py +++ b/sdk/python/feast/infra/passthrough_provider.py @@ -171,11 +171,16 @@ def materialize_single_feature_view( if feature_view.batch_source.field_mapping is not None: table = _run_field_mapping(table, feature_view.batch_source.field_mapping) - join_keys = {entity.join_key: entity.value_type for entity in entities} + join_key_to_value_type = { + entity.name: entity.dtype.to_value_type() + for entity in feature_view.entity_columns + } with tqdm_builder(table.num_rows) as pbar: for batch in table.to_batches(DEFAULT_BATCH_SIZE): - rows_to_write = _convert_arrow_to_proto(batch, feature_view, join_keys) + rows_to_write = _convert_arrow_to_proto( + batch, feature_view, join_key_to_value_type + ) self.online_write_batch( self.repo_config, feature_view, diff --git a/sdk/python/feast/infra/provider.py b/sdk/python/feast/infra/provider.py index 7754a58319c..cd82b7d416c 100644 --- a/sdk/python/feast/infra/provider.py +++ b/sdk/python/feast/infra/provider.py @@ -390,7 +390,7 @@ def _convert_arrow_to_proto( table = table.to_batches()[0] columns = [ - (field.name, field.dtype.to_value_type()) for field in feature_view.schema + (field.name, field.dtype.to_value_type()) for field in feature_view.features ] + list(join_keys.items()) proto_values_by_column = { diff --git a/sdk/python/feast/on_demand_feature_view.py b/sdk/python/feast/on_demand_feature_view.py index f2048e7f5ea..6d1eb455cac 100644 --- a/sdk/python/feast/on_demand_feature_view.py +++ b/sdk/python/feast/on_demand_feature_view.py @@ -666,7 +666,7 @@ def decorator(user_function): def feature_view_to_batch_feature_view(fv: FeatureView) -> BatchFeatureView: - return BatchFeatureView( + bfv = BatchFeatureView( name=fv.name, entities=fv.entities, ttl=fv.ttl, @@ -676,3 +676,7 @@ def feature_view_to_batch_feature_view(fv: FeatureView) -> BatchFeatureView: schema=fv.schema, source=fv.source, ) + + bfv.features = copy.copy(fv.features) + bfv.entities = copy.copy(fv.entities) + return bfv diff --git a/sdk/python/feast/templates/aws/driver_repo.py b/sdk/python/feast/templates/aws/driver_repo.py index 8ebe0b6e927..f80f16bb6fa 100644 --- a/sdk/python/feast/templates/aws/driver_repo.py +++ b/sdk/python/feast/templates/aws/driver_repo.py @@ -1,6 +1,6 @@ from datetime import timedelta -from feast import Entity, FeatureService, FeatureView, Field, RedshiftSource, ValueType +from feast import Entity, FeatureService, FeatureView, Field, RedshiftSource from feast.types import Float32, Int64 # Define an entity for the driver. Entities can be thought of as primary keys used to @@ -13,8 +13,6 @@ # features can be looked up. The join keys are also used to join feature # tables/views when building feature vectors join_keys=["driver_id"], - # The storage level type for an entity - value_type=ValueType.INT64, ) # Indicates a data source from which feature values can be retrieved. Sources are queried when building training @@ -41,7 +39,7 @@ # The list of entities specifies the keys required for joining or looking # up features from this feature view. The reference provided in this field # correspond to the name of a defined entity (or entities) - entities=["driver"], + entities=[driver], # The timedelta is the maximum age that each feature value may have # relative to its lookup time. For historical features (used in training), # TTL is relative to each timestamp provided in the entity dataframe. diff --git a/sdk/python/feast/templates/gcp/driver_repo.py b/sdk/python/feast/templates/gcp/driver_repo.py index a4517516b56..acb17d55197 100644 --- a/sdk/python/feast/templates/gcp/driver_repo.py +++ b/sdk/python/feast/templates/gcp/driver_repo.py @@ -1,6 +1,6 @@ from datetime import timedelta -from feast import BigQuerySource, Entity, FeatureService, FeatureView, Field, ValueType +from feast import BigQuerySource, Entity, FeatureService, FeatureView, Field from feast.types import Float32, Int64 # Define an entity for the driver. Entities can be thought of as primary keys used to @@ -13,8 +13,6 @@ # features can be looked up. The join keys are also used to join feature # tables/views when building feature vectors join_keys=["driver_id"], - # The storage level type for an entity - value_type=ValueType.INT64, ) # Indicates a data source from which feature values can be retrieved. Sources are queried when building training @@ -39,7 +37,7 @@ # The list of entities specifies the keys required for joining or looking # up features from this feature view. The reference provided in this field # correspond to the name of a defined entity (or entities) - entities=["driver"], + entities=[driver], # The timedelta is the maximum age that each feature value may have # relative to its lookup time. For historical features (used in training), # TTL is relative to each timestamp provided in the entity dataframe. diff --git a/sdk/python/feast/templates/hbase/example.py b/sdk/python/feast/templates/hbase/example.py index 1d441e0e995..b34696185b1 100644 --- a/sdk/python/feast/templates/hbase/example.py +++ b/sdk/python/feast/templates/hbase/example.py @@ -2,7 +2,7 @@ from datetime import timedelta -from feast import Entity, FeatureView, Field, FileSource, ValueType +from feast import Entity, FeatureView, Field, FileSource from feast.types import Float32, Int64 # Read data from parquet files. Parquet is convenient for local development mode. For @@ -16,7 +16,7 @@ # Define an entity for the driver. You can think of entity as a primary key used to # fetch features. -driver = Entity(name="driver", join_keys=["driver_id"], value_type=ValueType.INT64,) +driver = Entity(name="driver", join_keys=["driver_id"]) # Our parquet files contain sample data that includes a driver_id column, timestamps and # three feature column. Here we define a Feature View that will allow us to serve this diff --git a/sdk/python/feast/templates/local/example.py b/sdk/python/feast/templates/local/example.py index 7633947e6e4..30f9adf1890 100644 --- a/sdk/python/feast/templates/local/example.py +++ b/sdk/python/feast/templates/local/example.py @@ -2,7 +2,7 @@ from datetime import timedelta -from feast import Entity, FeatureService, FeatureView, Field, FileSource, ValueType +from feast import Entity, FeatureService, FeatureView, Field, FileSource from feast.types import Float32, Int64 # Read data from parquet files. Parquet is convenient for local development mode. For @@ -16,14 +16,14 @@ # Define an entity for the driver. You can think of entity as a primary key used to # fetch features. -driver = Entity(name="driver", join_keys=["driver_id"], value_type=ValueType.INT64,) +driver = Entity(name="driver", join_keys=["driver_id"]) # Our parquet files contain sample data that includes a driver_id column, timestamps and # three feature column. Here we define a Feature View that will allow us to serve this # data to our model online. driver_hourly_stats_view = FeatureView( name="driver_hourly_stats", - entities=["driver"], + entities=[driver], ttl=timedelta(days=1), schema=[ Field(name="conv_rate", dtype=Float32), diff --git a/sdk/python/feast/templates/postgres/driver_repo.py b/sdk/python/feast/templates/postgres/driver_repo.py index 34bc0022e23..4096943bb73 100644 --- a/sdk/python/feast/templates/postgres/driver_repo.py +++ b/sdk/python/feast/templates/postgres/driver_repo.py @@ -18,7 +18,7 @@ driver_stats_fv = FeatureView( name="driver_hourly_stats", - entities=["driver_id"], + entities=[driver], ttl=timedelta(weeks=52), schema=[ Field(name="conv_rate", dtype=Float32), diff --git a/sdk/python/feast/templates/snowflake/driver_repo.py b/sdk/python/feast/templates/snowflake/driver_repo.py index ceaf0ba8de8..297a3f5ef0f 100644 --- a/sdk/python/feast/templates/snowflake/driver_repo.py +++ b/sdk/python/feast/templates/snowflake/driver_repo.py @@ -43,7 +43,7 @@ # The list of entities specifies the keys required for joining or looking # up features from this feature view. The reference provided in this field # correspond to the name of a defined entity (or entities) - entities=["driver"], + entities=[driver], # The timedelta is the maximum age that each feature value may have # relative to its lookup time. For historical features (used in training), # TTL is relative to each timestamp provided in the entity dataframe. diff --git a/sdk/python/feast/templates/spark/example.py b/sdk/python/feast/templates/spark/example.py index da334dd83ca..d006353118b 100644 --- a/sdk/python/feast/templates/spark/example.py +++ b/sdk/python/feast/templates/spark/example.py @@ -5,7 +5,7 @@ from datetime import timedelta from pathlib import Path -from feast import Entity, FeatureService, FeatureView, Field, ValueType +from feast import Entity, FeatureService, FeatureView, Field from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import ( SparkSource, ) @@ -16,10 +16,8 @@ # Entity definitions -driver = Entity(name="driver", value_type=ValueType.INT64, description="driver id",) -customer = Entity( - name="customer", value_type=ValueType.INT64, description="customer id", -) +driver = Entity(name="driver", description="driver id",) +customer = Entity(name="customer", description="customer id",) # Sources driver_hourly_stats = SparkSource( @@ -40,7 +38,7 @@ # Feature Views driver_hourly_stats_view = FeatureView( name="driver_hourly_stats", - entities=["driver"], + entities=[driver], ttl=timedelta(days=7), schema=[ Field(name="conv_rate", dtype=Float32), @@ -53,7 +51,7 @@ ) customer_daily_profile_view = FeatureView( name="customer_daily_profile", - entities=["customer"], + entities=[customer], ttl=timedelta(days=7), schema=[ Field(name="current_balance", dtype=Float32), diff --git a/sdk/python/pytest.ini b/sdk/python/pytest.ini new file mode 100644 index 00000000000..07a5e869dc4 --- /dev/null +++ b/sdk/python/pytest.ini @@ -0,0 +1,4 @@ +[pytest] +markers = + universal_offline_stores: mark a test as using all offline stores. + universal_online_stores: mark a test as using all online stores. \ No newline at end of file diff --git a/sdk/python/tests/data/data_creator.py b/sdk/python/tests/data/data_creator.py index e08597b67b2..186c39b9ef4 100644 --- a/sdk/python/tests/data/data_creator.py +++ b/sdk/python/tests/data/data_creator.py @@ -4,11 +4,11 @@ import pandas as pd from pytz import timezone, utc -from feast.value_type import ValueType +from feast.types import FeastType, Float32, Int32, Int64, String def create_dataset( - entity_type: ValueType = ValueType.INT32, + entity_type: FeastType = Int32, feature_dtype: str = None, feature_is_list: bool = False, list_has_empty_list: bool = False, @@ -16,7 +16,7 @@ def create_dataset( now = datetime.utcnow().replace(microsecond=0, second=0, minute=0) ts = pd.Timestamp(now).round("ms") data = { - "driver_id": get_entities_for_value_type(entity_type), + "driver_id": get_entities_for_feast_type(entity_type), "value": get_feature_values_for_dtype( feature_dtype, feature_is_list, list_has_empty_list ), @@ -37,14 +37,14 @@ def create_dataset( return pd.DataFrame.from_dict(data) -def get_entities_for_value_type(value_type: ValueType) -> List: - value_type_map: Dict[ValueType, List] = { - ValueType.INT32: [1, 2, 1, 3, 3], - ValueType.INT64: [1, 2, 1, 3, 3], - ValueType.FLOAT: [1.0, 2.0, 1.0, 3.0, 3.0], - ValueType.STRING: ["1", "2", "1", "3", "3"], +def get_entities_for_feast_type(feast_type: FeastType) -> List: + feast_type_map: Dict[FeastType, List] = { + Int32: [1, 2, 1, 3, 3], + Int64: [1, 2, 1, 3, 3], + Float32: [1.0, 2.0, 1.0, 3.0, 3.0], + String: ["1", "2", "1", "3", "3"], } - return value_type_map[value_type] + return feast_type_map[feast_type] def get_feature_values_for_dtype( diff --git a/sdk/python/tests/doctest/test_all.py b/sdk/python/tests/doctest/test_all.py index 65d5f3da289..31f181ad535 100644 --- a/sdk/python/tests/doctest/test_all.py +++ b/sdk/python/tests/doctest/test_all.py @@ -11,15 +11,13 @@ def setup_feature_store(): """Prepares the local environment for a FeatureStore docstring test.""" from datetime import datetime, timedelta - from feast import Entity, FeatureStore, FeatureView, Field, FileSource, ValueType + from feast import Entity, FeatureStore, FeatureView, Field, FileSource from feast.repo_operations import init_repo from feast.types import Float32, Int64 init_repo("feature_repo", "local") fs = FeatureStore(repo_path="feature_repo") - driver = Entity( - name="driver_id", value_type=ValueType.INT64, description="driver id", - ) + driver = Entity(name="driver_id", description="driver id",) driver_hourly_stats = FileSource( path="feature_repo/data/driver_stats.parquet", timestamp_field="event_timestamp", @@ -27,7 +25,7 @@ def setup_feature_store(): ) driver_hourly_stats_view = FeatureView( name="driver_hourly_stats", - entities=["driver_id"], + entities=[driver], ttl=timedelta(seconds=86400 * 1), schema=[ Field(name="conv_rate", dtype=Float32), diff --git a/sdk/python/tests/example_repos/example_feature_repo_1.py b/sdk/python/tests/example_repos/example_feature_repo_1.py index d8b6d7c89b1..8d6d96d9ef1 100644 --- a/sdk/python/tests/example_repos/example_feature_repo_1.py +++ b/sdk/python/tests/example_repos/example_feature_repo_1.py @@ -1,14 +1,6 @@ from datetime import timedelta -from feast import ( - BigQuerySource, - Entity, - FeatureService, - FeatureView, - Field, - PushSource, - ValueType, -) +from feast import BigQuerySource, Entity, FeatureService, FeatureView, Field, PushSource from feast.types import Float32, Int64, String driver_locations_source = BigQuerySource( @@ -46,22 +38,24 @@ driver = Entity( name="driver", # The name is derived from this argument, not object name. join_keys=["driver_id"], - value_type=ValueType.INT64, description="driver id", ) customer = Entity( name="customer", # The name is derived from this argument, not object name. join_keys=["customer_id"], - value_type=ValueType.STRING, ) driver_locations = FeatureView( name="driver_locations", - entities=["driver"], + entities=[driver], ttl=timedelta(days=1), - schema=[Field(name="lat", dtype=Float32), Field(name="lon", dtype=String)], + schema=[ + Field(name="lat", dtype=Float32), + Field(name="lon", dtype=String), + Field(name="driver_id", dtype=Int64), + ], online=True, batch_source=driver_locations_source, tags={}, @@ -69,11 +63,12 @@ pushed_driver_locations = FeatureView( name="pushed_driver_locations", - entities=["driver"], + entities=[driver], ttl=timedelta(days=1), schema=[ Field(name="driver_lat", dtype=Float32), Field(name="driver_long", dtype=String), + Field(name="driver_id", dtype=Int64), ], online=True, stream_source=driver_locations_push_source, @@ -82,12 +77,13 @@ customer_profile = FeatureView( name="customer_profile", - entities=["customer"], + entities=[customer], ttl=timedelta(days=1), schema=[ Field(name="avg_orders_day", dtype=Float32), Field(name="name", dtype=String), Field(name="age", dtype=Int64), + Field(name="customer_id", dtype=String), ], online=True, batch_source=customer_profile_source, @@ -96,9 +92,13 @@ customer_driver_combined = FeatureView( name="customer_driver_combined", - entities=["customer", "driver"], + entities=[customer, driver], ttl=timedelta(days=1), - schema=[Field(name="trips", dtype=Int64)], + schema=[ + Field(name="trips", dtype=Int64), + Field(name="driver_id", dtype=Int64), + Field(name="customer_id", dtype=String), + ], online=True, batch_source=customer_driver_combined_source, tags={}, diff --git a/sdk/python/tests/example_repos/example_feature_repo_2.py b/sdk/python/tests/example_repos/example_feature_repo_2.py index d4c7976418b..073c48c1c1e 100644 --- a/sdk/python/tests/example_repos/example_feature_repo_2.py +++ b/sdk/python/tests/example_repos/example_feature_repo_2.py @@ -1,6 +1,6 @@ from datetime import timedelta -from feast import Entity, FeatureView, Field, FileSource, ValueType +from feast import Entity, FeatureView, Field, FileSource from feast.types import Float32, Int32, Int64 driver_hourly_stats = FileSource( @@ -9,17 +9,18 @@ created_timestamp_column="created", ) -driver = Entity(name="driver_id", value_type=ValueType.INT64, description="driver id") +driver = Entity(name="driver_id", description="driver id",) driver_hourly_stats_view = FeatureView( name="driver_hourly_stats", - entities=["driver_id"], + entities=[driver], ttl=timedelta(days=1), schema=[ Field(name="conv_rate", dtype=Float32), Field(name="acc_rate", dtype=Float32), Field(name="avg_daily_trips", dtype=Int64), + Field(name="driver_id", dtype=Int32), ], online=True, source=driver_hourly_stats, diff --git a/sdk/python/tests/example_repos/example_feature_repo_with_duplicated_featureview_names.py b/sdk/python/tests/example_repos/example_feature_repo_with_duplicated_featureview_names.py index cbcc3ad172b..4b079999ed7 100644 --- a/sdk/python/tests/example_repos/example_feature_repo_with_duplicated_featureview_names.py +++ b/sdk/python/tests/example_repos/example_feature_repo_with_duplicated_featureview_names.py @@ -1,14 +1,16 @@ from datetime import timedelta -from feast import FeatureView, FileSource +from feast import Entity, FeatureView, FileSource driver_hourly_stats = FileSource( path="driver_stats.parquet", # this parquet is not real and will not be read ) +driver = Entity(name="driver_id", description="driver id", join_keys=["driver"],) + driver_hourly_stats_view = FeatureView( name="driver_hourly_stats", # Intentionally use the same FeatureView name - entities=["driver_id"], + entities=[driver], online=False, source=driver_hourly_stats, ttl=timedelta(days=1), @@ -17,7 +19,7 @@ driver_hourly_stats_view_dup1 = FeatureView( name="driver_hourly_stats", # Intentionally use the same FeatureView name - entities=["driver_id"], + entities=[driver], online=False, source=driver_hourly_stats, ttl=timedelta(days=1), diff --git a/sdk/python/tests/example_repos/example_feature_repo_with_entity_join_key.py b/sdk/python/tests/example_repos/example_feature_repo_with_entity_join_key.py index 5ba26d2573b..06631505312 100644 --- a/sdk/python/tests/example_repos/example_feature_repo_with_entity_join_key.py +++ b/sdk/python/tests/example_repos/example_feature_repo_with_entity_join_key.py @@ -1,6 +1,6 @@ from datetime import timedelta -from feast import Entity, FeatureView, Field, FileSource, ValueType +from feast import Entity, FeatureView, Field, FileSource from feast.types import Float32, Int64 driver_hourly_stats = FileSource( @@ -11,17 +11,12 @@ # The join key here is deliberately different from the parquet file to test the failure path. -driver = Entity( - name="driver_id", - value_type=ValueType.INT64, - description="driver id", - join_keys=["driver"], -) +driver = Entity(name="driver_id", description="driver id", join_keys=["driver"],) driver_hourly_stats_view = FeatureView( name="driver_hourly_stats", - entities=["driver_id"], + entities=[driver], ttl=timedelta(days=1), schema=[ Field(name="conv_rate", dtype=Float32), diff --git a/sdk/python/tests/example_repos/on_demand_feature_view_repo.py b/sdk/python/tests/example_repos/on_demand_feature_view_repo.py index 453158b9dcc..ac572d5747a 100644 --- a/sdk/python/tests/example_repos/on_demand_feature_view_repo.py +++ b/sdk/python/tests/example_repos/on_demand_feature_view_repo.py @@ -2,7 +2,7 @@ import pandas as pd -from feast import FeatureView, Field, FileSource +from feast import Entity, FeatureView, Field, FileSource from feast.on_demand_feature_view import on_demand_feature_view from feast.types import Float32, String @@ -15,15 +15,18 @@ owner="test2@gmail.com", ) +driver = Entity(name="driver_id", description="driver id",) + driver_daily_features_view = FeatureView( name="driver_daily_features", - entities=["driver"], + entities=[driver], ttl=timedelta(seconds=8640000000), schema=[ Field(name="daily_miles_driven", dtype=Float32), Field(name="lat", dtype=Float32), Field(name="lon", dtype=Float32), Field(name="string_feature", dtype=String), + Field(name="driver_id", dtype=Float32), ], online=True, source=driver_stats, diff --git a/sdk/python/tests/integration/e2e/test_usage_e2e.py b/sdk/python/tests/integration/e2e/test_usage_e2e.py index 12c1eb86281..53e4a32a82e 100644 --- a/sdk/python/tests/integration/e2e/test_usage_e2e.py +++ b/sdk/python/tests/integration/e2e/test_usage_e2e.py @@ -19,7 +19,7 @@ import pytest -from feast import Entity, RepoConfig, ValueType +from feast import Entity, RepoConfig from feast.infra.online_stores.sqlite import SqliteOnlineStoreConfig @@ -60,7 +60,6 @@ def test_usage_on(dummy_exporter, enabling_toggle): entity = Entity( name="driver_car_id", description="Car driver id", - value_type=ValueType.STRING, tags={"team": "matchmaking"}, ) @@ -99,7 +98,6 @@ def test_usage_off(dummy_exporter, enabling_toggle): entity = Entity( name="driver_car_id", description="Car driver id", - value_type=ValueType.STRING, tags={"team": "matchmaking"}, ) test_feature_store.apply([entity]) diff --git a/sdk/python/tests/integration/feature_repos/universal/entities.py b/sdk/python/tests/integration/feature_repos/universal/entities.py index b7a7583f1b3..66989b06468 100644 --- a/sdk/python/tests/integration/feature_repos/universal/entities.py +++ b/sdk/python/tests/integration/feature_repos/universal/entities.py @@ -1,22 +1,21 @@ -from feast import Entity, ValueType +from feast import Entity -def driver(value_type: ValueType = ValueType.INT64): +def driver(): return Entity( name="driver", # The name is derived from this argument, not object name. - value_type=value_type, description="driver id", join_keys=["driver_id"], ) def customer(): - return Entity(name="customer_id", value_type=ValueType.INT64) + return Entity(name="customer_id") def location(): - return Entity(name="location_id", value_type=ValueType.INT64) + return Entity(name="location_id") def item(): - return Entity(name="item_id", value_type=ValueType.INT64) + return Entity(name="item_id") diff --git a/sdk/python/tests/integration/feature_repos/universal/feature_views.py b/sdk/python/tests/integration/feature_repos/universal/feature_views.py index 3e05f5d7e5f..b93ad987fa4 100644 --- a/sdk/python/tests/integration/feature_repos/universal/feature_views.py +++ b/sdk/python/tests/integration/feature_repos/universal/feature_views.py @@ -14,8 +14,13 @@ ValueType, ) from feast.data_source import DataSource, RequestSource -from feast.types import Array, FeastType, Float32, Float64, Int32 -from tests.integration.feature_repos.universal.entities import location +from feast.types import Array, FeastType, Float32, Float64, Int32, Int64 +from tests.integration.feature_repos.universal.entities import ( + customer, + driver, + item, + location, +) def driver_feature_view( @@ -23,12 +28,14 @@ def driver_feature_view( name="test_correctness", infer_features: bool = False, dtype: FeastType = Float32, - entities: Optional[List[str]] = None, + entity_type: FeastType = Int64, ) -> FeatureView: + d = driver() return FeatureView( name=name, - entities=entities or ["driver"], - schema=None if infer_features else [Field(name="value", dtype=dtype)], + entities=[d], + schema=[Field(name=d.join_key, dtype=entity_type)] + + ([] if infer_features else [Field(name="value", dtype=dtype)]), ttl=timedelta(days=5), source=data_source, ) @@ -139,7 +146,7 @@ def create_similarity_request_source(): def create_item_embeddings_feature_view(source, infer_features: bool = False): item_embeddings_feature_view = FeatureView( name="item_embeddings", - entities=["item"], + entities=[item()], schema=None if infer_features else [ @@ -157,7 +164,7 @@ def create_item_embeddings_batch_feature_view( ) -> BatchFeatureView: item_embeddings_feature_view = BatchFeatureView( name="item_embeddings", - entities=["item"], + entities=[item()], schema=None if infer_features else [ @@ -171,15 +178,19 @@ def create_item_embeddings_batch_feature_view( def create_driver_hourly_stats_feature_view(source, infer_features: bool = False): + # TODO(felixwang9817): Figure out why not adding an entity field here + # breaks type tests. + d = driver() driver_stats_feature_view = FeatureView( name="driver_stats", - entities=["driver"], + entities=[d], schema=None if infer_features else [ Field(name="conv_rate", dtype=Float32), Field(name="acc_rate", dtype=Float32), Field(name="avg_daily_trips", dtype=Int32), + Field(name=d.join_key, dtype=Int64), ], source=source, ttl=timedelta(hours=2), @@ -192,7 +203,7 @@ def create_driver_hourly_stats_batch_feature_view( ) -> BatchFeatureView: driver_stats_feature_view = BatchFeatureView( name="driver_stats", - entities=["driver"], + entities=[driver()], schema=None if infer_features else [ @@ -209,7 +220,7 @@ def create_driver_hourly_stats_batch_feature_view( def create_customer_daily_profile_feature_view(source, infer_features: bool = False): customer_profile_feature_view = FeatureView( name="customer_profile", - entities=["customer_id"], + entities=[customer()], schema=None if infer_features else [ @@ -243,10 +254,13 @@ def create_global_stats_feature_view(source, infer_features: bool = False): def create_order_feature_view(source, infer_features: bool = False): return FeatureView( name="order", - entities=["driver", "customer_id"], + entities=[customer(), driver()], schema=None if infer_features - else [Field(name="order_is_success", dtype=Int32)], + else [ + Field(name="order_is_success", dtype=Int32), + Field(name="driver_id", dtype=Int64), + ], source=source, ttl=timedelta(days=2), ) @@ -256,7 +270,12 @@ def create_location_stats_feature_view(source, infer_features: bool = False): location_stats_feature_view = FeatureView( name="location_stats", entities=[location()], - schema=None if infer_features else [Field(name="temperature", dtype=Int32)], + schema=None + if infer_features + else [ + Field(name="temperature", dtype=Int32), + Field(name="location_id", dtype=Int64), + ], source=source, ttl=timedelta(days=2), ) @@ -280,9 +299,11 @@ def create_pushable_feature_view(batch_source: DataSource): ) return FeatureView( name="pushable_location_stats", - entities=["location_id"], - # Test that Features still work for FeatureViews. - features=[Feature(name="temperature", dtype=ValueType.INT32)], + entities=[location()], + schema=[ + Field(name="temperature", dtype=Int32), + Field(name="location_id", dtype=Int64), + ], ttl=timedelta(days=2), source=push_source, ) diff --git a/sdk/python/tests/integration/offline_store/test_feature_logging.py b/sdk/python/tests/integration/offline_store/test_feature_logging.py index 24c65eac020..a6f8e56de7d 100644 --- a/sdk/python/tests/integration/offline_store/test_feature_logging.py +++ b/sdk/python/tests/integration/offline_store/test_feature_logging.py @@ -16,7 +16,11 @@ from tests.integration.feature_repos.repo_configuration import ( construct_universal_feature_views, ) -from tests.integration.feature_repos.universal.entities import driver +from tests.integration.feature_repos.universal.entities import ( + customer, + driver, + location, +) from tests.integration.feature_repos.universal.feature_views import conv_rate_plus_100 from tests.utils.logged_features import prepare_logs, to_logs_dataset @@ -30,7 +34,7 @@ def test_feature_service_logging(environment, universal_data_sources, pass_as_pa (_, datasets, data_sources) = universal_data_sources feature_views = construct_universal_feature_views(data_sources) - store.apply([driver(), *feature_views.values()]) + store.apply([customer(), driver(), location(), *feature_views.values()]) feature_service = FeatureService( name="test_service", diff --git a/sdk/python/tests/integration/offline_store/test_universal_historical_retrieval.py b/sdk/python/tests/integration/offline_store/test_universal_historical_retrieval.py index d5f49a1f958..2076ab2aedf 100644 --- a/sdk/python/tests/integration/offline_store/test_universal_historical_retrieval.py +++ b/sdk/python/tests/integration/offline_store/test_universal_historical_retrieval.py @@ -22,7 +22,6 @@ DEFAULT_ENTITY_DF_EVENT_TIMESTAMP_COL, ) from feast.types import Float32, Int32 -from feast.value_type import ValueType from tests.integration.feature_repos.repo_configuration import ( construct_universal_feature_views, table_name_from_data_source, @@ -728,10 +727,10 @@ def test_historical_features_from_bigquery_sources_containing_backfills(environm created_timestamp_column="created", ) - driver = Entity(name="driver", join_keys=["driver_id"], value_type=ValueType.INT64) + driver = Entity(name="driver", join_keys=["driver_id"]) driver_fv = FeatureView( name="driver_stats", - entities=["driver"], + entities=[driver], schema=[Field(name="avg_daily_trips", dtype=Int32)], batch_source=driver_stats_data_source, ttl=None, diff --git a/sdk/python/tests/integration/online_store/test_universal_online.py b/sdk/python/tests/integration/online_store/test_universal_online.py index b3115dcb3d4..4afcd61c70b 100644 --- a/sdk/python/tests/integration/online_store/test_universal_online.py +++ b/sdk/python/tests/integration/online_store/test_universal_online.py @@ -13,7 +13,7 @@ import requests from botocore.exceptions import BotoCoreError -from feast import Entity, FeatureService, FeatureView, Field, ValueType +from feast import Entity, FeatureService, FeatureView, Field from feast.errors import ( FeatureNameCollisionError, RequestDataNotFoundInEntityRowsException, @@ -117,13 +117,13 @@ def test_write_to_online_store_event_check(environment): } dataframe_source = pd.DataFrame(data) with prep_file_source(df=dataframe_source, timestamp_field="ts_1") as file_source: - e = Entity(name="id", value_type=ValueType.STRING) + e = Entity(name="id") # Create Feature View fv1 = FeatureView( name="feature_view_123", schema=[Field(name="string_col", dtype=String)], - entities=["id"], + entities=[e], batch_source=file_source, ttl=timedelta(minutes=5), ) diff --git a/sdk/python/tests/integration/registration/test_feature_store.py b/sdk/python/tests/integration/registration/test_feature_store.py index db4c6700cec..88a4b9f2491 100644 --- a/sdk/python/tests/integration/registration/test_feature_store.py +++ b/sdk/python/tests/integration/registration/test_feature_store.py @@ -27,10 +27,8 @@ from feast.infra.offline_stores.file import FileOfflineStoreConfig from feast.infra.online_stores.dynamodb import DynamoDBOnlineStoreConfig from feast.infra.online_stores.sqlite import SqliteOnlineStoreConfig -from feast.protos.feast.types import Value_pb2 as ValueProto from feast.repo_config import RepoConfig from feast.types import Array, Bytes, Float64, Int64, String -from feast.value_type import ValueType from tests.utils.data_source_utils import ( prep_file_source, simple_bq_source_using_query_arg, @@ -93,10 +91,7 @@ def feature_store_with_s3_registry(): ) def test_apply_entity_success(test_feature_store): entity = Entity( - name="driver_car_id", - description="Car driver id", - value_type=ValueType.STRING, - tags={"team": "matchmaking"}, + name="driver_car_id", description="Car driver id", tags={"team": "matchmaking"}, ) # Register Entity @@ -108,7 +103,6 @@ def test_apply_entity_success(test_feature_store): assert ( len(entities) == 1 and entity.name == "driver_car_id" - and entity.value_type == ValueType(ValueProto.ValueType.STRING) and entity.description == "Car driver id" and "team" in entity.tags and entity.tags["team"] == "matchmaking" @@ -127,10 +121,7 @@ def test_apply_entity_success(test_feature_store): ) def test_apply_entity_integration(test_feature_store): entity = Entity( - name="driver_car_id", - description="Car driver id", - value_type=ValueType.STRING, - tags={"team": "matchmaking"}, + name="driver_car_id", description="Car driver id", tags={"team": "matchmaking"}, ) # Register Entity @@ -142,7 +133,6 @@ def test_apply_entity_integration(test_feature_store): assert ( len(entities) == 1 and entity.name == "driver_car_id" - and entity.value_type == ValueType(ValueProto.ValueType.STRING) and entity.description == "Car driver id" and "team" in entity.tags and entity.tags["team"] == "matchmaking" @@ -151,7 +141,6 @@ def test_apply_entity_integration(test_feature_store): entity = test_feature_store.get_entity("driver_car_id") assert ( entity.name == "driver_car_id" - and entity.value_type == ValueType(ValueProto.ValueType.STRING) and entity.description == "Car driver id" and "team" in entity.tags and entity.tags["team"] == "matchmaking" @@ -173,6 +162,8 @@ def test_apply_feature_view_success(test_feature_store): date_partition_column="date_partition_col", ) + entity = Entity(name="fs1_my_entity_1", join_keys=["entity_id"]) + fv1 = FeatureView( name="my_feature_view_1", schema=[ @@ -180,15 +171,16 @@ def test_apply_feature_view_success(test_feature_store): Field(name="fs1_my_feature_2", dtype=String), Field(name="fs1_my_feature_3", dtype=Array(String)), Field(name="fs1_my_feature_4", dtype=Array(Bytes)), + Field(name="entity_id", dtype=Int64), ], - entities=["fs1_my_entity_1"], + entities=[entity], tags={"team": "matchmaking"}, batch_source=batch_source, ttl=timedelta(minutes=5), ) # Register Feature View - test_feature_store.apply([fv1]) + test_feature_store.apply([entity, fv1]) feature_views = test_feature_store.list_feature_views() @@ -217,13 +209,11 @@ def test_apply_feature_view_success(test_feature_store): @pytest.mark.parametrize("dataframe_source", [lazy_fixture("simple_dataset_1")]) def test_feature_view_inference_success(test_feature_store, dataframe_source): with prep_file_source(df=dataframe_source, timestamp_field="ts_1") as file_source: - entity = Entity( - name="id", join_keys=["id_join_key"], value_type=ValueType.INT64 - ) + entity = Entity(name="id", join_keys=["id_join_key"]) fv1 = FeatureView( name="fv1", - entities=["id"], + entities=[entity], ttl=timedelta(minutes=5), online=True, batch_source=file_source, @@ -232,7 +222,7 @@ def test_feature_view_inference_success(test_feature_store, dataframe_source): fv2 = FeatureView( name="fv2", - entities=["id"], + entities=[entity], ttl=timedelta(minutes=5), online=True, batch_source=simple_bq_source_using_table_arg(dataframe_source, "ts_1"), @@ -241,7 +231,7 @@ def test_feature_view_inference_success(test_feature_store, dataframe_source): fv3 = FeatureView( name="fv3", - entities=["id"], + entities=[entity], ttl=timedelta(minutes=5), online=True, batch_source=simple_bq_source_using_query_arg(dataframe_source, "ts_1"), @@ -296,6 +286,8 @@ def test_apply_feature_view_integration(test_feature_store): date_partition_column="date_partition_col", ) + entity = Entity(name="fs1_my_entity_1", join_keys=["test"]) + fv1 = FeatureView( name="my_feature_view_1", schema=[ @@ -303,15 +295,16 @@ def test_apply_feature_view_integration(test_feature_store): Field(name="fs1_my_feature_2", dtype=String), Field(name="fs1_my_feature_3", dtype=Array(String)), Field(name="fs1_my_feature_4", dtype=Array(Bytes)), + Field(name="test", dtype=Int64), ], - entities=["fs1_my_entity_1"], + entities=[entity], tags={"team": "matchmaking"}, batch_source=batch_source, ttl=timedelta(minutes=5), ) # Register Feature View - test_feature_store.apply([fv1]) + test_feature_store.apply([fv1, entity]) feature_views = test_feature_store.list_feature_views() @@ -364,13 +357,9 @@ def test_apply_object_and_read(test_feature_store): created_timestamp_column="timestamp", ) - e1 = Entity( - name="fs1_my_entity_1", value_type=ValueType.STRING, description="something" - ) + e1 = Entity(name="fs1_my_entity_1", description="something") - e2 = Entity( - name="fs1_my_entity_2", value_type=ValueType.STRING, description="something" - ) + e2 = Entity(name="fs1_my_entity_2", description="something") fv1 = FeatureView( name="my_feature_view_1", @@ -379,8 +368,9 @@ def test_apply_object_and_read(test_feature_store): Field(name="fs1_my_feature_2", dtype=String), Field(name="fs1_my_feature_3", dtype=Array(String)), Field(name="fs1_my_feature_4", dtype=Array(Bytes)), + Field(name="fs1_my_entity_1", dtype=Int64), ], - entities=["fs1_my_entity_1"], + entities=[e1], tags={"team": "matchmaking"}, batch_source=batch_source, ttl=timedelta(minutes=5), @@ -393,8 +383,9 @@ def test_apply_object_and_read(test_feature_store): Field(name="fs1_my_feature_2", dtype=String), Field(name="fs1_my_feature_3", dtype=Array(String)), Field(name="fs1_my_feature_4", dtype=Array(Bytes)), + Field(name="fs1_my_entity_2", dtype=Int64), ], - entities=["fs1_my_entity_1"], + entities=[e2], tags={"team": "matchmaking"}, batch_source=batch_source, ttl=timedelta(minutes=5), @@ -406,7 +397,6 @@ def test_apply_object_and_read(test_feature_store): fv1_actual = test_feature_store.get_feature_view("my_feature_view_1") e1_actual = test_feature_store.get_entity("fs1_my_entity_1") - assert fv1 == fv1_actual assert e1 == e1_actual assert fv2 != fv1_actual assert e2 != e1_actual @@ -434,13 +424,13 @@ def test_apply_remote_repo(): def test_reapply_feature_view_success(test_feature_store, dataframe_source): with prep_file_source(df=dataframe_source, timestamp_field="ts_1") as file_source: - e = Entity(name="id", join_keys=["id_join_key"], value_type=ValueType.STRING) + e = Entity(name="id", join_keys=["id_join_key"]) # Create Feature View fv1 = FeatureView( name="my_feature_view_1", schema=[Field(name="string_col", dtype=String)], - entities=["id"], + entities=[e], batch_source=file_source, ttl=timedelta(minutes=5), ) @@ -470,7 +460,7 @@ def test_reapply_feature_view_success(test_feature_store, dataframe_source): fv1 = FeatureView( name="my_feature_view_1", schema=[Field(name="int64_col", dtype=Int64)], - entities=["id"], + entities=[e], batch_source=file_source, ttl=timedelta(minutes=5), ) @@ -485,10 +475,12 @@ def test_reapply_feature_view_success(test_feature_store, dataframe_source): def test_apply_conflicting_featureview_names(feature_store_with_local_registry): """Test applying feature views with non-case-insensitively unique names""" + driver = Entity(name="driver", join_keys=["driver_id"]) + customer = Entity(name="customer", join_keys=["customer_id"]) driver_stats = FeatureView( name="driver_hourly_stats", - entities=["driver_id"], + entities=[driver], ttl=timedelta(seconds=10), online=False, batch_source=FileSource(path="driver_stats.parquet"), @@ -497,7 +489,7 @@ def test_apply_conflicting_featureview_names(feature_store_with_local_registry): customer_stats = FeatureView( name="DRIVER_HOURLY_STATS", - entities=["id"], + entities=[customer], ttl=timedelta(seconds=10), online=False, batch_source=FileSource(path="customer_stats.parquet"), diff --git a/sdk/python/tests/integration/registration/test_inference.py b/sdk/python/tests/integration/registration/test_inference.py index 6cf49c31db8..c298c0e4f66 100644 --- a/sdk/python/tests/integration/registration/test_inference.py +++ b/sdk/python/tests/integration/registration/test_inference.py @@ -24,14 +24,13 @@ from feast.field import Field from feast.inference import ( update_data_sources_with_inferred_event_timestamp_col, - update_entities_with_inferred_types_from_feature_views, - update_feature_views_with_inferred_features, + update_feature_views_with_inferred_features_and_entities, ) from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import ( SparkSource, ) from feast.on_demand_feature_view import on_demand_feature_view -from feast.types import Float32, String, UnixTimestamp +from feast.types import Float32, Float64, Int64, String, UnixTimestamp from tests.utils.data_source_utils import ( prep_file_source, simple_bq_source_using_query_arg, @@ -39,47 +38,6 @@ ) -def test_update_entities_with_inferred_types_from_feature_views( - simple_dataset_1, simple_dataset_2 -): - with prep_file_source( - df=simple_dataset_1, timestamp_field="ts_1" - ) as file_source, prep_file_source( - df=simple_dataset_2, timestamp_field="ts_1" - ) as file_source_2: - - fv1 = FeatureView( - name="fv1", entities=["id"], batch_source=file_source, ttl=None, - ) - fv2 = FeatureView( - name="fv2", entities=["id"], batch_source=file_source_2, ttl=None, - ) - - actual_1 = Entity(name="id", join_keys=["id_join_key"]) - actual_2 = Entity(name="id", join_keys=["id_join_key"]) - - update_entities_with_inferred_types_from_feature_views( - [actual_1], [fv1], RepoConfig(provider="local", project="test") - ) - update_entities_with_inferred_types_from_feature_views( - [actual_2], [fv2], RepoConfig(provider="local", project="test") - ) - assert actual_1 == Entity( - name="id", join_keys=["id_join_key"], value_type=ValueType.INT64 - ) - assert actual_2 == Entity( - name="id", join_keys=["id_join_key"], value_type=ValueType.STRING - ) - - with pytest.raises(RegistryInferenceFailure): - # two viable data types - update_entities_with_inferred_types_from_feature_views( - [Entity(name="id", join_keys=["id_join_key"])], - [fv1, fv2], - RepoConfig(provider="local", project="test"), - ) - - def test_infer_datasource_names_file(): file_path = "path/to/test.csv" data_source = FileSource(path=file_path) @@ -287,7 +245,10 @@ def test_view_with_missing_feature(features_df: pd.DataFrame) -> pd.DataFrame: test_view_with_missing_feature.infer_features() -def test_update_feature_views_with_inferred_features(): +def test_feature_view_inference_respects_basic_inference(): + """ + Tests that feature view inference respects the basic inference that occurs during creation. + """ file_source = FileSource(name="test", path="test path") entity1 = Entity(name="test1", join_keys=["test_column_1"]) entity2 = Entity(name="test2", join_keys=["test_column_2"]) @@ -312,26 +273,132 @@ def test_update_feature_views_with_inferred_features(): ) assert len(feature_view_1.schema) == 2 - assert len(feature_view_1.features) == 2 + assert len(feature_view_1.features) == 1 + assert len(feature_view_1.entity_columns) == 1 - # The entity field should be deleted from the schema and features of the feature view. - update_feature_views_with_inferred_features( + update_feature_views_with_inferred_features_and_entities( [feature_view_1], [entity1], RepoConfig(provider="local", project="test") ) - assert len(feature_view_1.schema) == 1 + assert len(feature_view_1.schema) == 2 assert len(feature_view_1.features) == 1 + assert len(feature_view_1.entity_columns) == 1 assert len(feature_view_2.schema) == 3 - assert len(feature_view_2.features) == 3 + assert len(feature_view_2.features) == 1 + assert len(feature_view_2.entity_columns) == 2 - # The entity fields should be deleted from the schema and features of the feature view. - update_feature_views_with_inferred_features( + update_feature_views_with_inferred_features_and_entities( [feature_view_2], [entity1, entity2], RepoConfig(provider="local", project="test"), ) - assert len(feature_view_2.schema) == 1 + assert len(feature_view_2.schema) == 3 assert len(feature_view_2.features) == 1 + assert len(feature_view_2.entity_columns) == 2 + + +def test_feature_view_inference_on_entity_columns(simple_dataset_1): + """ + Tests that feature view inference correctly infers entity columns. + """ + with prep_file_source(df=simple_dataset_1, timestamp_field="ts_1") as file_source: + entity1 = Entity(name="test1", join_keys=["id_join_key"]) + feature_view_1 = FeatureView( + name="test1", + entities=[entity1], + schema=[Field(name="int64_col", dtype=Int64)], + source=file_source, + ) + + assert len(feature_view_1.schema) == 1 + assert len(feature_view_1.features) == 1 + assert len(feature_view_1.entity_columns) == 0 + + update_feature_views_with_inferred_features_and_entities( + [feature_view_1], [entity1], RepoConfig(provider="local", project="test") + ) + + # The schema is only used as a parameter, as is therefore not updated during inference. + assert len(feature_view_1.schema) == 1 + + # Since there is already a feature specified, additional features are not inferred. + assert len(feature_view_1.features) == 1 + + # The single entity column is inferred correctly. + assert len(feature_view_1.entity_columns) == 1 + + +def test_feature_view_inference_respects_entity_value_type(simple_dataset_1): + """ + Tests that feature view inference still respects an entity's value type. + """ + # TODO(felixwang9817): Remove this test once entity value_type is removed. + with prep_file_source(df=simple_dataset_1, timestamp_field="ts_1") as file_source: + entity1 = Entity( + name="test1", join_keys=["id_join_key"], value_type=ValueType.STRING + ) + feature_view_1 = FeatureView( + name="test1", + entities=[entity1], + schema=[Field(name="int64_col", dtype=Int64)], + source=file_source, + ) + + assert len(feature_view_1.schema) == 1 + assert len(feature_view_1.features) == 1 + assert len(feature_view_1.entity_columns) == 0 + + update_feature_views_with_inferred_features_and_entities( + [feature_view_1], [entity1], RepoConfig(provider="local", project="test") + ) + + # The schema is only used as a parameter, as is therefore not updated during inference. + assert len(feature_view_1.schema) == 1 + + # Since there is already a feature specified, additional features are not inferred. + assert len(feature_view_1.features) == 1 + + # The single entity column is inferred correctly and has type String. + assert len(feature_view_1.entity_columns) == 1 + assert feature_view_1.entity_columns[0].dtype == String + + +def test_feature_view_inference_on_feature_columns(simple_dataset_1): + """ + Tests that feature view inference correctly infers feature columns. + """ + with prep_file_source(df=simple_dataset_1, timestamp_field="ts_1") as file_source: + entity1 = Entity(name="test1", join_keys=["id_join_key"]) + feature_view_1 = FeatureView( + name="test1", + entities=[entity1], + schema=[Field(name="id_join_key", dtype=Int64)], + source=file_source, + ) + + assert len(feature_view_1.schema) == 1 + assert len(feature_view_1.features) == 0 + assert len(feature_view_1.entity_columns) == 1 + + update_feature_views_with_inferred_features_and_entities( + [feature_view_1], [entity1], RepoConfig(provider="local", project="test") + ) + + # The schema is only used as a parameter, as is therefore not updated during inference. + assert len(feature_view_1.schema) == 1 + + # All three feature columns are inferred correctly. + assert len(feature_view_1.features) == 3 + print(feature_view_1.features) + feature_column_1 = Field(name="float_col", dtype=Float64) + feature_column_2 = Field(name="int64_col", dtype=Int64) + feature_column_3 = Field(name="string_col", dtype=String) + assert feature_column_1 in feature_view_1.features + assert feature_column_2 in feature_view_1.features + assert feature_column_3 in feature_view_1.features + + # The single entity column remains. + assert len(feature_view_1.entity_columns) == 1 def test_update_feature_services_with_inferred_features(simple_dataset_1): @@ -344,13 +411,16 @@ def test_update_feature_services_with_inferred_features(simple_dataset_1): assert len(feature_service.feature_view_projections) == 1 assert len(feature_service.feature_view_projections[0].features) == 0 - update_feature_views_with_inferred_features( + update_feature_views_with_inferred_features_and_entities( [feature_view_1], [entity1], RepoConfig(provider="local", project="test") ) feature_service.infer_features( fvs_to_update={feature_view_1.name: feature_view_1} ) - assert len(feature_view_1.schema) == 3 + assert len(feature_view_1.schema) == 0 assert len(feature_view_1.features) == 3 assert len(feature_service.feature_view_projections[0].features) == 3 + + +# TODO(felixwang9817): Add tests that interact with field mapping. diff --git a/sdk/python/tests/integration/registration/test_registry.py b/sdk/python/tests/integration/registration/test_registry.py index 46e9a195446..bb02f9a9e32 100644 --- a/sdk/python/tests/integration/registration/test_registry.py +++ b/sdk/python/tests/integration/registration/test_registry.py @@ -26,7 +26,6 @@ from feast.feature_view import FeatureView from feast.field import Field from feast.on_demand_feature_view import RequestSource, on_demand_feature_view -from feast.protos.feast.types import Value_pb2 as ValueProto from feast.registry import Registry from feast.repo_config import RegistryConfig from feast.types import Array, Bytes, Float32, Int32, Int64, String @@ -73,10 +72,7 @@ def s3_registry() -> Registry: ) def test_apply_entity_success(test_registry): entity = Entity( - name="driver_car_id", - description="Car driver id", - value_type=ValueType.STRING, - tags={"team": "matchmaking"}, + name="driver_car_id", description="Car driver id", tags={"team": "matchmaking"}, ) project = "project" @@ -90,7 +86,6 @@ def test_apply_entity_success(test_registry): assert ( len(entities) == 1 and entity.name == "driver_car_id" - and entity.value_type == ValueType(ValueProto.ValueType.STRING) and entity.description == "Car driver id" and "team" in entity.tags and entity.tags["team"] == "matchmaking" @@ -99,7 +94,6 @@ def test_apply_entity_success(test_registry): entity = test_registry.get_entity("driver_car_id", project) assert ( entity.name == "driver_car_id" - and entity.value_type == ValueType(ValueProto.ValueType.STRING) and entity.description == "Car driver id" and "team" in entity.tags and entity.tags["team"] == "matchmaking" @@ -122,10 +116,7 @@ def test_apply_entity_success(test_registry): ) def test_apply_entity_integration(test_registry): entity = Entity( - name="driver_car_id", - description="Car driver id", - value_type=ValueType.STRING, - tags={"team": "matchmaking"}, + name="driver_car_id", description="Car driver id", tags={"team": "matchmaking"}, ) project = "project" @@ -139,7 +130,6 @@ def test_apply_entity_integration(test_registry): assert ( len(entities) == 1 and entity.name == "driver_car_id" - and entity.value_type == ValueType(ValueProto.ValueType.STRING) and entity.description == "Car driver id" and "team" in entity.tags and entity.tags["team"] == "matchmaking" @@ -148,7 +138,6 @@ def test_apply_entity_integration(test_registry): entity = test_registry.get_entity("driver_car_id", project) assert ( entity.name == "driver_car_id" - and entity.value_type == ValueType(ValueProto.ValueType.STRING) and entity.description == "Car driver id" and "team" in entity.tags and entity.tags["team"] == "matchmaking" @@ -173,6 +162,8 @@ def test_apply_feature_view_success(test_registry): created_timestamp_column="timestamp", ) + entity = Entity(name="fs1_my_entity_1", join_keys=["test"]) + fv1 = FeatureView( name="my_feature_view_1", schema=[ @@ -181,7 +172,7 @@ def test_apply_feature_view_success(test_registry): Field(name="fs1_my_feature_3", dtype=Array(String)), Field(name="fs1_my_feature_4", dtype=Array(Bytes)), ], - entities=["fs1_my_entity_1"], + entities=[entity], tags={"team": "matchmaking"}, batch_source=batch_source, ttl=timedelta(minutes=5), @@ -327,10 +318,12 @@ def test_modify_feature_views_success(test_registry, request_source_schema): request_source = RequestSource(name="request_source", schema=request_source_schema,) + entity = Entity(name="fs1_my_entity_1", join_keys=["test"]) + fv1 = FeatureView( name="my_feature_view_1", schema=[Field(name="fs1_my_feature_1", dtype=Int64)], - entities=["fs1_my_entity_1"], + entities=[entity], tags={"team": "matchmaking"}, batch_source=batch_source, ttl=timedelta(minutes=5), @@ -443,6 +436,8 @@ def test_apply_feature_view_integration(test_registry): created_timestamp_column="timestamp", ) + entity = Entity(name="fs1_my_entity_1", join_keys=["test"]) + fv1 = FeatureView( name="my_feature_view_1", schema=[ @@ -451,7 +446,7 @@ def test_apply_feature_view_integration(test_registry): Field(name="fs1_my_feature_3", dtype=Array(String)), Field(name="fs1_my_feature_4", dtype=Array(Bytes)), ], - entities=["fs1_my_entity_1"], + entities=[entity], tags={"team": "matchmaking"}, batch_source=batch_source, ttl=timedelta(minutes=5), @@ -518,6 +513,8 @@ def test_apply_data_source(test_registry: Registry): created_timestamp_column="timestamp", ) + entity = Entity(name="fs1_my_entity_1", join_keys=["test"]) + fv1 = FeatureView( name="my_feature_view_1", schema=[ @@ -526,7 +523,7 @@ def test_apply_data_source(test_registry: Registry): Field(name="fs1_my_feature_3", dtype=Array(String)), Field(name="fs1_my_feature_4", dtype=Array(Bytes)), ], - entities=["fs1_my_entity_1"], + entities=[entity], tags={"team": "matchmaking"}, batch_source=batch_source, ttl=timedelta(minutes=5), @@ -573,10 +570,7 @@ def test_commit(): test_registry = Registry(registry_config, None) entity = Entity( - name="driver_car_id", - description="Car driver id", - value_type=ValueType.STRING, - tags={"team": "matchmaking"}, + name="driver_car_id", description="Car driver id", tags={"team": "matchmaking"}, ) project = "project" @@ -591,7 +585,6 @@ def test_commit(): assert ( len(entities) == 1 and entity.name == "driver_car_id" - and entity.value_type == ValueType(ValueProto.ValueType.STRING) and entity.description == "Car driver id" and "team" in entity.tags and entity.tags["team"] == "matchmaking" @@ -600,7 +593,6 @@ def test_commit(): entity = test_registry.get_entity("driver_car_id", project, allow_cache=True) assert ( entity.name == "driver_car_id" - and entity.value_type == ValueType(ValueProto.ValueType.STRING) and entity.description == "Car driver id" and "team" in entity.tags and entity.tags["team"] == "matchmaking" @@ -626,7 +618,6 @@ def test_commit(): assert ( len(entities) == 1 and entity.name == "driver_car_id" - and entity.value_type == ValueType(ValueProto.ValueType.STRING) and entity.description == "Car driver id" and "team" in entity.tags and entity.tags["team"] == "matchmaking" @@ -635,7 +626,6 @@ def test_commit(): entity = test_registry.get_entity("driver_car_id", project) assert ( entity.name == "driver_car_id" - and entity.value_type == ValueType(ValueProto.ValueType.STRING) and entity.description == "Car driver id" and "team" in entity.tags and entity.tags["team"] == "matchmaking" diff --git a/sdk/python/tests/integration/registration/test_universal_types.py b/sdk/python/tests/integration/registration/test_universal_types.py index 6d016e3e85c..b03303f6eef 100644 --- a/sdk/python/tests/integration/registration/test_universal_types.py +++ b/sdk/python/tests/integration/registration/test_universal_types.py @@ -8,10 +8,17 @@ import pyarrow as pa import pytest -from feast.entity import Entity from feast.infra.offline_stores.offline_store import RetrievalJob -from feast.types import Array, Bool, Float32, Int32, Int64, UnixTimestamp -from feast.value_type import ValueType +from feast.types import ( + Array, + Bool, + Float32, + Float64, + Int32, + Int64, + String, + UnixTimestamp, +) from tests.data.data_creator import create_dataset from tests.integration.feature_repos.universal.entities import driver from tests.integration.feature_repos.universal.feature_views import driver_feature_view @@ -89,10 +96,7 @@ def get_fixtures(request, environment): ).lower() config = request.param df = create_dataset( - ValueType.INT64, - config.feature_dtype, - config.feature_is_list, - config.has_empty_list, + Int64, config.feature_dtype, config.feature_is_list, config.has_empty_list, ) data_source = environment.data_source_creator.create_data_source( df, destination_name=destination_name, field_mapping={"ts_1": "ts"}, @@ -110,18 +114,11 @@ def get_fixtures(request, environment): @pytest.mark.integration @pytest.mark.universal_offline_stores -@pytest.mark.parametrize( - "entity_type", [ValueType.INT32, ValueType.INT64, ValueType.STRING] -) +@pytest.mark.parametrize("entity_type", [Int32, Int64, String]) def test_entity_inference_types_match(environment, entity_type): fs = environment.feature_store # Don't specify value type in entity to force inference - entity = Entity( - name=f"driver_{entity_type.name.lower()}", - value_type=ValueType.UNKNOWN, - join_key="driver_id", - ) df = create_dataset(entity_type, feature_dtype="int32",) data_source = environment.data_source_creator.create_data_source( df, @@ -134,20 +131,30 @@ def test_entity_inference_types_match(environment, entity_type): feature_is_list=False, has_empty_list=False, data_source=data_source, - entity=entity.name, + entity_type=entity_type, ) + + # TODO(felixwang9817): Refactor this by finding a better way to force type inference. + # Override the schema and entity_columns to force entity inference. + entity = driver() + fv.schema = list(filter(lambda x: x.name != entity.join_key, fv.schema)) + fv.entity_columns = [] fs.apply([fv, entity]) - inferred_entity = fs.get_entity(entity.name) entity_type_to_expected_inferred_entity_type = { - ValueType.INT32: {ValueType.INT32, ValueType.INT64}, - ValueType.INT64: {ValueType.INT32, ValueType.INT64}, - ValueType.FLOAT: {ValueType.DOUBLE}, - ValueType.STRING: {ValueType.STRING}, + Int32: {Int32, Int64}, + Int64: {Int32, Int64}, + Float32: {Float64}, + String: {String}, } + + entity_columns = list( + filter(lambda x: x.name == entity.join_key, fv.entity_columns) + ) + assert len(entity_columns) == 1 + entity_column = entity_columns[0] assert ( - inferred_entity.value_type - in entity_type_to_expected_inferred_entity_type[entity_type] + entity_column.dtype in entity_type_to_expected_inferred_entity_type[entity_type] ) @@ -263,7 +270,12 @@ def test_feature_get_online_features_types_match( def create_feature_view( - name, feature_dtype, feature_is_list, has_empty_list, data_source, entity="driver" + name, + feature_dtype, + feature_is_list, + has_empty_list, + data_source, + entity_type=Int64, ): if feature_is_list is True: if feature_dtype == "int32": @@ -288,7 +300,9 @@ def create_feature_view( elif feature_dtype == "datetime": dtype = UnixTimestamp - return driver_feature_view(data_source, name=name, dtype=dtype, entities=[entity]) + return driver_feature_view( + data_source, name=name, dtype=dtype, entity_type=entity_type + ) def assert_expected_historical_feature_types( diff --git a/sdk/python/tests/integration/scaffolding/test_partial_apply.py b/sdk/python/tests/integration/scaffolding/test_partial_apply.py index 3ab9bf196f2..e5a7206b96c 100644 --- a/sdk/python/tests/integration/scaffolding/test_partial_apply.py +++ b/sdk/python/tests/integration/scaffolding/test_partial_apply.py @@ -2,7 +2,7 @@ import pytest -from feast import BigQuerySource, FeatureView, Field +from feast import BigQuerySource, Entity, FeatureView, Field from feast.types import Float32, String from tests.utils.cli_utils import CliRunner, get_example_repo from tests.utils.online_read_write_test import basic_rw_test @@ -19,6 +19,7 @@ def test_partial() -> None: with runner.local_repo( get_example_repo("example_feature_repo_1.py"), "bigquery" ) as store: + driver = Entity(name="driver", join_keys=["test"]) driver_locations_source = BigQuerySource( table="feast-oss.public.drivers", @@ -28,12 +29,13 @@ def test_partial() -> None: driver_locations_100 = FeatureView( name="driver_locations_100", - entities=["driver"], + entities=[driver], ttl=timedelta(days=1), schema=[ Field(name="lat", dtype=Float32), Field(name="lon", dtype=String), Field(name="name", dtype=String), + Field(name="test", dtype=String), ], online=True, batch_source=driver_locations_source, diff --git a/sdk/python/tests/unit/diff/test_registry_diff.py b/sdk/python/tests/unit/diff/test_registry_diff.py index 483dae73e26..ae10c834c83 100644 --- a/sdk/python/tests/unit/diff/test_registry_diff.py +++ b/sdk/python/tests/unit/diff/test_registry_diff.py @@ -2,34 +2,36 @@ diff_registry_objects, tag_objects_for_keep_delete_update_add, ) +from feast.entity import Entity from feast.feature_view import FeatureView from tests.utils.data_source_utils import prep_file_source def test_tag_objects_for_keep_delete_update_add(simple_dataset_1): with prep_file_source(df=simple_dataset_1, timestamp_field="ts_1") as file_source: + entity = Entity(name="id", join_keys=["id"]) to_delete = FeatureView( - name="to_delete", entities=["id"], batch_source=file_source, ttl=None, + name="to_delete", entities=[entity], batch_source=file_source, ttl=None, ) unchanged_fv = FeatureView( - name="fv1", entities=["id"], batch_source=file_source, ttl=None, + name="fv1", entities=[entity], batch_source=file_source, ttl=None, ) pre_changed = FeatureView( name="fv2", - entities=["id"], + entities=[entity], batch_source=file_source, ttl=None, tags={"when": "before"}, ) post_changed = FeatureView( name="fv2", - entities=["id"], + entities=[entity], batch_source=file_source, ttl=None, tags={"when": "after"}, ) to_add = FeatureView( - name="to_add", entities=["id"], batch_source=file_source, ttl=None, + name="to_add", entities=[entity], batch_source=file_source, ttl=None, ) keep, delete, update, add = tag_objects_for_keep_delete_update_add( @@ -52,16 +54,17 @@ def test_tag_objects_for_keep_delete_update_add(simple_dataset_1): def test_diff_registry_objects_feature_views(simple_dataset_1): with prep_file_source(df=simple_dataset_1, timestamp_field="ts_1") as file_source: + entity = Entity(name="id", join_keys=["id"]) pre_changed = FeatureView( name="fv2", - entities=["id"], + entities=[entity], batch_source=file_source, ttl=None, tags={"when": "before"}, ) post_changed = FeatureView( name="fv2", - entities=["id"], + entities=[entity], batch_source=file_source, ttl=None, tags={"when": "after"}, diff --git a/sdk/python/tests/unit/infra/test_provider.py b/sdk/python/tests/unit/infra/test_provider.py index 43c09760e9a..5ed5603b03b 100644 --- a/sdk/python/tests/unit/infra/test_provider.py +++ b/sdk/python/tests/unit/infra/test_provider.py @@ -20,14 +20,13 @@ from feast.field import Field from feast.infra.provider import _get_column_names from feast.types import String -from feast.value_type import ValueType def test_get_column_names_preserves_feature_ordering(): - entity = Entity("my-entity", description="My entity", value_type=ValueType.STRING) + entity = Entity("my-entity", description="My entity") fv = FeatureView( name="my-fv", - entities=["my-entity"], + entities=[entity], ttl=timedelta(days=1), batch_source=BigQuerySource(table="non-existent-mock"), schema=[ diff --git a/sdk/python/tests/unit/test_entity.py b/sdk/python/tests/unit/test_entity.py index 254a975f678..04a857ddefe 100644 --- a/sdk/python/tests/unit/test_entity.py +++ b/sdk/python/tests/unit/test_entity.py @@ -20,19 +20,14 @@ def test_join_key_default(): with pytest.deprecated_call(): - entity = Entity( - "my-entity", description="My entity", value_type=ValueType.STRING - ) + entity = Entity("my-entity", description="My entity") assert entity.join_key == "my-entity" def test_entity_class_contains_tags(): with pytest.deprecated_call(): entity = Entity( - "my-entity", - description="My entity", - value_type=ValueType.STRING, - tags={"key1": "val1", "key2": "val2"}, + "my-entity", description="My entity", tags={"key1": "val1", "key2": "val2"}, ) assert "key1" in entity.tags.keys() and entity.tags["key1"] == "val1" assert "key2" in entity.tags.keys() and entity.tags["key2"] == "val2" @@ -40,20 +35,18 @@ def test_entity_class_contains_tags(): def test_entity_without_tags_empty_dict(): with pytest.deprecated_call(): - entity = Entity( - "my-entity", description="My entity", value_type=ValueType.STRING - ) + entity = Entity("my-entity", description="My entity") assert entity.tags == dict() assert len(entity.tags) == 0 def test_entity_without_description(): with pytest.deprecated_call(): - Entity("my-entity", value_type=ValueType.STRING) + Entity("my-entity") def test_name_not_specified(): - assertpy.assert_that(lambda: Entity(value_type=ValueType.STRING)).raises(ValueError) + assertpy.assert_that(lambda: Entity()).raises(ValueError) def test_multiple_args(): @@ -61,15 +54,19 @@ def test_multiple_args(): def test_name_keyword(recwarn): - Entity(name="my-entity", value_type=ValueType.STRING) + Entity(name="my-entity") assert len(recwarn) == 0 + Entity(name="my-entity", join_key="test") + assert len(recwarn) == 1 + Entity(name="my-entity", join_keys=["test"]) + assert len(recwarn) == 1 def test_hash(): - entity1 = Entity(name="my-entity", value_type=ValueType.STRING) - entity2 = Entity(name="my-entity", value_type=ValueType.STRING) - entity3 = Entity(name="my-entity", value_type=ValueType.FLOAT) - entity4 = Entity(name="my-entity", value_type=ValueType.FLOAT, description="test") + entity1 = Entity(name="my-entity") + entity2 = Entity(name="my-entity") + entity3 = Entity(name="my-entity", join_keys=["not-my-entity"]) + entity4 = Entity(name="my-entity", join_keys=["not-my-entity"], description="test") s1 = {entity1, entity2} assert len(s1) == 1 diff --git a/sdk/python/tests/unit/test_feature_view.py b/sdk/python/tests/unit/test_feature_view.py index 80a583806e7..1ef36081ecd 100644 --- a/sdk/python/tests/unit/test_feature_view.py +++ b/sdk/python/tests/unit/test_feature_view.py @@ -62,3 +62,7 @@ def test_hash(): s4 = {feature_view_1, feature_view_2, feature_view_3, feature_view_4} assert len(s4) == 3 + + +# TODO(felixwang9817): Add tests for proto conversion. +# TODO(felixwang9817): Add tests for field mapping logic. diff --git a/sdk/python/tests/unit/test_unit_feature_store.py b/sdk/python/tests/unit/test_unit_feature_store.py index 6f9dd6acb08..0c13dffa629 100644 --- a/sdk/python/tests/unit/test_unit_feature_store.py +++ b/sdk/python/tests/unit/test_unit_feature_store.py @@ -17,7 +17,7 @@ class MockFeatureView: projection: MockFeatureViewProjection -def test__get_unique_entities(): +def test_get_unique_entities(): entity_values = { "entity_1": [Value(int64_val=1), Value(int64_val=2), Value(int64_val=1)], "entity_2": [ diff --git a/sdk/python/tests/utils/online_write_benchmark.py b/sdk/python/tests/utils/online_write_benchmark.py index 82ffc8e98bf..9f2f8ba60df 100644 --- a/sdk/python/tests/utils/online_write_benchmark.py +++ b/sdk/python/tests/utils/online_write_benchmark.py @@ -17,13 +17,13 @@ from feast.infra.provider import _convert_arrow_to_proto from feast.repo_config import RepoConfig from feast.types import Float32, Int32 -from feast.value_type import ValueType def create_driver_hourly_stats_feature_view(source): + driver = Entity(name="driver", join_keys=["driver_id"]) driver_stats_feature_view = FeatureView( name="driver_stats", - entities=["driver_id"], + entities=[driver], schema=[ Field(name="conv_rate", dtype=Float32), Field(name="acc_rate", dtype=Float32), @@ -61,7 +61,7 @@ def benchmark_writes(): # This is just to set data source to something, we're not reading from parquet source here. parquet_path = os.path.join(temp_dir, "data.parquet") - driver = Entity(name="driver_id", value_type=ValueType.INT64) + driver = Entity(name="driver_id") table = create_driver_hourly_stats_feature_view( create_driver_hourly_stats_source(parquet_path=parquet_path) ) From f4eed30c7a01b8e68cb6b5ed4f8ed70cdef86462 Mon Sep 17 00:00:00 2001 From: Danny Chiao Date: Thu, 19 May 2022 01:49:30 -0400 Subject: [PATCH 016/123] chore: Fix RTD and update from make build-sphinx (#2715) * [chore] Fix RTD and update from make build-sphinx Signed-off-by: Danny Chiao * update Signed-off-by: Danny Chiao * update Signed-off-by: Danny Chiao * fix push source Signed-off-by: Danny Chiao * fix push source Signed-off-by: Danny Chiao * fix hierarchy Signed-off-by: Danny Chiao --- sdk/python/docs/Makefile | 4 +- sdk/python/docs/index.rst | 29 ++++++++- sdk/python/docs/requirements.txt | 2 +- ..._stores.contrib.postgres_offline_store.rst | 8 +++ ...s.contrib.postgres_offline_store.tests.rst | 21 +++++++ ...ine_stores.contrib.spark_offline_store.rst | 8 +++ ...ores.contrib.spark_offline_store.tests.rst | 21 +++++++ ...ine_stores.contrib.trino_offline_store.rst | 1 + ...ores.contrib.trino_offline_store.tests.rst | 21 +++++++ ...line_stores.contrib.hbase_online_store.rst | 21 +++++++ .../feast.infra.online_stores.contrib.rst | 16 +++++ sdk/python/docs/source/feast.infra.utils.rst | 8 +++ sdk/python/docs/source/feast.protos.feast.rst | 1 + .../docs/source/feast.protos.feast.types.rst | 61 +++++++++++++++++++ sdk/python/docs/source/feast.rst | 17 ++++++ sdk/python/docs/source/feast.ui.rst | 10 +++ sdk/python/docs/source/index.rst | 29 ++++++++- 17 files changed, 273 insertions(+), 5 deletions(-) create mode 100644 sdk/python/docs/source/feast.infra.offline_stores.contrib.postgres_offline_store.tests.rst create mode 100644 sdk/python/docs/source/feast.infra.offline_stores.contrib.spark_offline_store.tests.rst create mode 100644 sdk/python/docs/source/feast.infra.offline_stores.contrib.trino_offline_store.tests.rst create mode 100644 sdk/python/docs/source/feast.infra.online_stores.contrib.hbase_online_store.rst create mode 100644 sdk/python/docs/source/feast.protos.feast.types.rst create mode 100644 sdk/python/docs/source/feast.ui.rst diff --git a/sdk/python/docs/Makefile b/sdk/python/docs/Makefile index a396b3903b3..702e30d136f 100644 --- a/sdk/python/docs/Makefile +++ b/sdk/python/docs/Makefile @@ -3,7 +3,7 @@ # You can set these variables from the command line. SPHINXOPTS = -SPHINXBUILD = sphinx-build +SPHINXBUILD = python -m sphinx SPHINXPROJ = Feast SOURCEDIR = source BUILDDIR = build @@ -23,7 +23,7 @@ build-api-source: cp conf.py index.rst source/ html: clean-html build-api-source - sphinx-build -b html source html + python -m sphinx -b html source html # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). diff --git a/sdk/python/docs/index.rst b/sdk/python/docs/index.rst index ca746707eac..4b3b9244b41 100644 --- a/sdk/python/docs/index.rst +++ b/sdk/python/docs/index.rst @@ -21,8 +21,21 @@ Data Source ================== .. automodule:: feast.data_source + :inherited-members: :members: - :exclude-members: KafkaOptions, KafkaSource, KinesisOptions, KinesisSource + :exclude-members: KafkaOptions, KafkaSource, KinesisOptions, KinesisSource, PushSource, RequestSource, RequestDataSource + +Request Source +------------------ + +.. automodule:: feast.data_source + :members: RequestSource + +Push Source +------------------ + +.. automodule:: feast.data_source + :members: PushSource BigQuery Source ------------------ @@ -121,6 +134,13 @@ Registry Store :members: :exclude-members: NoopRegistryStore +PostgreSQL Registry Store +----------------------- + +.. automodule:: feast.infra.registry_stores.contrib.postgres.registry_store + :members: + :noindex: + Provider ================== @@ -243,3 +263,10 @@ PostgreSQL Online Store .. automodule:: feast.infra.online_stores.contrib.postgres :members: :noindex: + +HBase Online Store +----------------------- + +.. automodule:: feast.infra.online_stores.contrib.hbase_online_store.hbase + :members: + :noindex: diff --git a/sdk/python/docs/requirements.txt b/sdk/python/docs/requirements.txt index 6f3a8382ff5..388cd430a8d 100644 --- a/sdk/python/docs/requirements.txt +++ b/sdk/python/docs/requirements.txt @@ -1 +1 @@ --e "./sdk/python/[ci]" \ No newline at end of file +-e ".[ci]" \ No newline at end of file diff --git a/sdk/python/docs/source/feast.infra.offline_stores.contrib.postgres_offline_store.rst b/sdk/python/docs/source/feast.infra.offline_stores.contrib.postgres_offline_store.rst index 95afafb5b80..a80690fe859 100644 --- a/sdk/python/docs/source/feast.infra.offline_stores.contrib.postgres_offline_store.rst +++ b/sdk/python/docs/source/feast.infra.offline_stores.contrib.postgres_offline_store.rst @@ -1,6 +1,14 @@ feast.infra.offline\_stores.contrib.postgres\_offline\_store package ==================================================================== +Subpackages +----------- + +.. toctree:: + :maxdepth: 4 + + feast.infra.offline_stores.contrib.postgres_offline_store.tests + Submodules ---------- diff --git a/sdk/python/docs/source/feast.infra.offline_stores.contrib.postgres_offline_store.tests.rst b/sdk/python/docs/source/feast.infra.offline_stores.contrib.postgres_offline_store.tests.rst new file mode 100644 index 00000000000..35e60d2998b --- /dev/null +++ b/sdk/python/docs/source/feast.infra.offline_stores.contrib.postgres_offline_store.tests.rst @@ -0,0 +1,21 @@ +feast.infra.offline\_stores.contrib.postgres\_offline\_store.tests package +========================================================================== + +Submodules +---------- + +feast.infra.offline\_stores.contrib.postgres\_offline\_store.tests.data\_source module +-------------------------------------------------------------------------------------- + +.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store.tests.data_source + :members: + :undoc-members: + :show-inheritance: + +Module contents +--------------- + +.. automodule:: feast.infra.offline_stores.contrib.postgres_offline_store.tests + :members: + :undoc-members: + :show-inheritance: diff --git a/sdk/python/docs/source/feast.infra.offline_stores.contrib.spark_offline_store.rst b/sdk/python/docs/source/feast.infra.offline_stores.contrib.spark_offline_store.rst index d91389faabf..b8b79bb48e8 100644 --- a/sdk/python/docs/source/feast.infra.offline_stores.contrib.spark_offline_store.rst +++ b/sdk/python/docs/source/feast.infra.offline_stores.contrib.spark_offline_store.rst @@ -1,6 +1,14 @@ feast.infra.offline\_stores.contrib.spark\_offline\_store package ================================================================= +Subpackages +----------- + +.. toctree:: + :maxdepth: 4 + + feast.infra.offline_stores.contrib.spark_offline_store.tests + Submodules ---------- diff --git a/sdk/python/docs/source/feast.infra.offline_stores.contrib.spark_offline_store.tests.rst b/sdk/python/docs/source/feast.infra.offline_stores.contrib.spark_offline_store.tests.rst new file mode 100644 index 00000000000..8b0f9bd88b3 --- /dev/null +++ b/sdk/python/docs/source/feast.infra.offline_stores.contrib.spark_offline_store.tests.rst @@ -0,0 +1,21 @@ +feast.infra.offline\_stores.contrib.spark\_offline\_store.tests package +======================================================================= + +Submodules +---------- + +feast.infra.offline\_stores.contrib.spark\_offline\_store.tests.data\_source module +----------------------------------------------------------------------------------- + +.. automodule:: feast.infra.offline_stores.contrib.spark_offline_store.tests.data_source + :members: + :undoc-members: + :show-inheritance: + +Module contents +--------------- + +.. automodule:: feast.infra.offline_stores.contrib.spark_offline_store.tests + :members: + :undoc-members: + :show-inheritance: diff --git a/sdk/python/docs/source/feast.infra.offline_stores.contrib.trino_offline_store.rst b/sdk/python/docs/source/feast.infra.offline_stores.contrib.trino_offline_store.rst index 0fe9cdc4610..857326003f3 100644 --- a/sdk/python/docs/source/feast.infra.offline_stores.contrib.trino_offline_store.rst +++ b/sdk/python/docs/source/feast.infra.offline_stores.contrib.trino_offline_store.rst @@ -9,6 +9,7 @@ Subpackages feast.infra.offline_stores.contrib.trino_offline_store.connectors feast.infra.offline_stores.contrib.trino_offline_store.test_config + feast.infra.offline_stores.contrib.trino_offline_store.tests Submodules ---------- diff --git a/sdk/python/docs/source/feast.infra.offline_stores.contrib.trino_offline_store.tests.rst b/sdk/python/docs/source/feast.infra.offline_stores.contrib.trino_offline_store.tests.rst new file mode 100644 index 00000000000..9102f1f8d64 --- /dev/null +++ b/sdk/python/docs/source/feast.infra.offline_stores.contrib.trino_offline_store.tests.rst @@ -0,0 +1,21 @@ +feast.infra.offline\_stores.contrib.trino\_offline\_store.tests package +======================================================================= + +Submodules +---------- + +feast.infra.offline\_stores.contrib.trino\_offline\_store.tests.data\_source module +----------------------------------------------------------------------------------- + +.. automodule:: feast.infra.offline_stores.contrib.trino_offline_store.tests.data_source + :members: + :undoc-members: + :show-inheritance: + +Module contents +--------------- + +.. automodule:: feast.infra.offline_stores.contrib.trino_offline_store.tests + :members: + :undoc-members: + :show-inheritance: diff --git a/sdk/python/docs/source/feast.infra.online_stores.contrib.hbase_online_store.rst b/sdk/python/docs/source/feast.infra.online_stores.contrib.hbase_online_store.rst new file mode 100644 index 00000000000..ce249023049 --- /dev/null +++ b/sdk/python/docs/source/feast.infra.online_stores.contrib.hbase_online_store.rst @@ -0,0 +1,21 @@ +feast.infra.online\_stores.contrib.hbase\_online\_store package +=============================================================== + +Submodules +---------- + +feast.infra.online\_stores.contrib.hbase\_online\_store.hbase module +-------------------------------------------------------------------- + +.. automodule:: feast.infra.online_stores.contrib.hbase_online_store.hbase + :members: + :undoc-members: + :show-inheritance: + +Module contents +--------------- + +.. automodule:: feast.infra.online_stores.contrib.hbase_online_store + :members: + :undoc-members: + :show-inheritance: diff --git a/sdk/python/docs/source/feast.infra.online_stores.contrib.rst b/sdk/python/docs/source/feast.infra.online_stores.contrib.rst index 6ee9b1a532d..7315bb741ef 100644 --- a/sdk/python/docs/source/feast.infra.online_stores.contrib.rst +++ b/sdk/python/docs/source/feast.infra.online_stores.contrib.rst @@ -1,9 +1,25 @@ feast.infra.online\_stores.contrib package ========================================== +Subpackages +----------- + +.. toctree:: + :maxdepth: 4 + + feast.infra.online_stores.contrib.hbase_online_store + Submodules ---------- +feast.infra.online\_stores.contrib.hbase\_repo\_configuration module +-------------------------------------------------------------------- + +.. automodule:: feast.infra.online_stores.contrib.hbase_repo_configuration + :members: + :undoc-members: + :show-inheritance: + feast.infra.online\_stores.contrib.postgres module -------------------------------------------------- diff --git a/sdk/python/docs/source/feast.infra.utils.rst b/sdk/python/docs/source/feast.infra.utils.rst index 30ec6166bf4..ffada497975 100644 --- a/sdk/python/docs/source/feast.infra.utils.rst +++ b/sdk/python/docs/source/feast.infra.utils.rst @@ -20,6 +20,14 @@ feast.infra.utils.aws\_utils module :undoc-members: :show-inheritance: +feast.infra.utils.hbase\_utils module +------------------------------------- + +.. automodule:: feast.infra.utils.hbase_utils + :members: + :undoc-members: + :show-inheritance: + feast.infra.utils.snowflake\_utils module ----------------------------------------- diff --git a/sdk/python/docs/source/feast.protos.feast.rst b/sdk/python/docs/source/feast.protos.feast.rst index 456d960d738..f519165db8e 100644 --- a/sdk/python/docs/source/feast.protos.feast.rst +++ b/sdk/python/docs/source/feast.protos.feast.rst @@ -10,6 +10,7 @@ Subpackages feast.protos.feast.core feast.protos.feast.serving feast.protos.feast.storage + feast.protos.feast.types Module contents --------------- diff --git a/sdk/python/docs/source/feast.protos.feast.types.rst b/sdk/python/docs/source/feast.protos.feast.types.rst new file mode 100644 index 00000000000..aeb31bc9ad3 --- /dev/null +++ b/sdk/python/docs/source/feast.protos.feast.types.rst @@ -0,0 +1,61 @@ +feast.protos.feast.types package +================================ + +Submodules +---------- + +feast.protos.feast.types.EntityKey\_pb2 module +---------------------------------------------- + +.. automodule:: feast.protos.feast.types.EntityKey_pb2 + :members: + :undoc-members: + :show-inheritance: + +feast.protos.feast.types.EntityKey\_pb2\_grpc module +---------------------------------------------------- + +.. automodule:: feast.protos.feast.types.EntityKey_pb2_grpc + :members: + :undoc-members: + :show-inheritance: + +feast.protos.feast.types.Field\_pb2 module +------------------------------------------ + +.. automodule:: feast.protos.feast.types.Field_pb2 + :members: + :undoc-members: + :show-inheritance: + +feast.protos.feast.types.Field\_pb2\_grpc module +------------------------------------------------ + +.. automodule:: feast.protos.feast.types.Field_pb2_grpc + :members: + :undoc-members: + :show-inheritance: + +feast.protos.feast.types.Value\_pb2 module +------------------------------------------ + +.. automodule:: feast.protos.feast.types.Value_pb2 + :members: + :undoc-members: + :show-inheritance: + +feast.protos.feast.types.Value\_pb2\_grpc module +------------------------------------------------ + +.. automodule:: feast.protos.feast.types.Value_pb2_grpc + :members: + :undoc-members: + :show-inheritance: + +Module contents +--------------- + +.. automodule:: feast.protos.feast.types + :members: + :undoc-members: + :show-inheritance: diff --git a/sdk/python/docs/source/feast.rst b/sdk/python/docs/source/feast.rst index fb1e77a3736..c54e035b034 100644 --- a/sdk/python/docs/source/feast.rst +++ b/sdk/python/docs/source/feast.rst @@ -12,6 +12,7 @@ Subpackages feast.infra feast.loaders feast.protos + feast.ui Submodules ---------- @@ -104,6 +105,14 @@ feast.feature module :undoc-members: :show-inheritance: +feast.feature\_logging module +----------------------------- + +.. automodule:: feast.feature_logging + :members: + :undoc-members: + :show-inheritance: + feast.feature\_server module ---------------------------- @@ -304,6 +313,14 @@ feast.types module :undoc-members: :show-inheritance: +feast.ui\_server module +----------------------- + +.. automodule:: feast.ui_server + :members: + :undoc-members: + :show-inheritance: + feast.usage module ------------------ diff --git a/sdk/python/docs/source/feast.ui.rst b/sdk/python/docs/source/feast.ui.rst new file mode 100644 index 00000000000..01b16cb0a6a --- /dev/null +++ b/sdk/python/docs/source/feast.ui.rst @@ -0,0 +1,10 @@ +feast.ui package +================ + +Module contents +--------------- + +.. automodule:: feast.ui + :members: + :undoc-members: + :show-inheritance: diff --git a/sdk/python/docs/source/index.rst b/sdk/python/docs/source/index.rst index ca746707eac..4b3b9244b41 100644 --- a/sdk/python/docs/source/index.rst +++ b/sdk/python/docs/source/index.rst @@ -21,8 +21,21 @@ Data Source ================== .. automodule:: feast.data_source + :inherited-members: :members: - :exclude-members: KafkaOptions, KafkaSource, KinesisOptions, KinesisSource + :exclude-members: KafkaOptions, KafkaSource, KinesisOptions, KinesisSource, PushSource, RequestSource, RequestDataSource + +Request Source +------------------ + +.. automodule:: feast.data_source + :members: RequestSource + +Push Source +------------------ + +.. automodule:: feast.data_source + :members: PushSource BigQuery Source ------------------ @@ -121,6 +134,13 @@ Registry Store :members: :exclude-members: NoopRegistryStore +PostgreSQL Registry Store +----------------------- + +.. automodule:: feast.infra.registry_stores.contrib.postgres.registry_store + :members: + :noindex: + Provider ================== @@ -243,3 +263,10 @@ PostgreSQL Online Store .. automodule:: feast.infra.online_stores.contrib.postgres :members: :noindex: + +HBase Online Store +----------------------- + +.. automodule:: feast.infra.online_stores.contrib.hbase_online_store.hbase + :members: + :noindex: From 3347a57240cd485b7572777d7b977869140ccb20 Mon Sep 17 00:00:00 2001 From: Tsotne Tabidze Date: Thu, 19 May 2022 13:38:04 -0700 Subject: [PATCH 017/123] feat: Add http endpoint to the Go feature server (#2658) * feat: Add http endpoint to the Go feature server Signed-off-by: Tsotne Tabidze * Remove debugging prints Signed-off-by: Tsotne Tabidze * Remove grpc keyword where not necessary Signed-off-by: Tsotne Tabidze * Add feature logging to http server Signed-off-by: Tsotne Tabidze * Format timestamp correctly Signed-off-by: Tsotne Tabidze * Add unit test for UnmarshalJSON Signed-off-by: Tsotne Tabidze * Add e2e http server test & fix logging bug Signed-off-by: Tsotne Tabidze * Add separate methods for stopping http & grpc servers Signed-off-by: Tsotne Tabidze --- go/embedded/online_features.go | 76 ++++- go/internal/feast/server/grpc_server.go | 2 +- go/internal/feast/server/http_server.go | 270 ++++++++++++++++++ go/internal/feast/server/http_server_test.go | 40 +++ go/internal/feast/server/logging/logger.go | 8 +- .../feast/server/logging/logger_test.go | 12 +- sdk/python/feast/cli.py | 14 +- .../embedded_go/online_features_service.py | 24 +- sdk/python/feast/feature_store.py | 17 +- .../integration/e2e/test_go_feature_server.py | 65 ++++- 10 files changed, 501 insertions(+), 27 deletions(-) create mode 100644 go/internal/feast/server/http_server.go create mode 100644 go/internal/feast/server/http_server_test.go diff --git a/go/embedded/online_features.go b/go/embedded/online_features.go index 710f60dca8e..de7e6d11a2e 100644 --- a/go/embedded/online_features.go +++ b/go/embedded/online_features.go @@ -31,6 +31,7 @@ import ( type OnlineFeatureService struct { fs *feast.FeatureStore grpcStopCh chan os.Signal + httpStopCh chan os.Signal } type OnlineFeatureServiceConfig struct { @@ -63,11 +64,13 @@ func NewOnlineFeatureService(conf *OnlineFeatureServiceConfig, transformationCal log.Fatalln(err) } - // Notify this channel when receiving interrupt or termination signals from OS - c := make(chan os.Signal, 1) - signal.Notify(c, syscall.SIGINT, syscall.SIGTERM) + // Notify these channels when receiving interrupt or termination signals from OS + httpStopCh := make(chan os.Signal, 1) + grpcStopCh := make(chan os.Signal, 1) + signal.Notify(httpStopCh, syscall.SIGINT, syscall.SIGTERM) + signal.Notify(grpcStopCh, syscall.SIGINT, syscall.SIGTERM) - return &OnlineFeatureService{fs: fs, grpcStopCh: c} + return &OnlineFeatureService{fs: fs, httpStopCh: httpStopCh, grpcStopCh: grpcStopCh} } func (s *OnlineFeatureService) GetEntityTypesMap(featureRefs []string) (map[string]int32, error) { @@ -225,15 +228,12 @@ func (s *OnlineFeatureService) StartGprcServerWithLoggingDefaultOpts(host string return s.StartGprcServerWithLogging(host, port, writeLoggedFeaturesCallback, defaultOpts) } -// StartGprcServerWithLogging starts gRPC server with enabled feature logging -// Caller of this function must provide Python callback to flush buffered logs as well as logging configuration (loggingOpts) -func (s *OnlineFeatureService) StartGprcServerWithLogging(host string, port int, writeLoggedFeaturesCallback logging.OfflineStoreWriteCallback, loggingOpts LoggingOptions) error { +func (s *OnlineFeatureService) constructLoggingService(writeLoggedFeaturesCallback logging.OfflineStoreWriteCallback, loggingOpts LoggingOptions) (*logging.LoggingService, error) { var loggingService *logging.LoggingService = nil - var err error if writeLoggedFeaturesCallback != nil { sink, err := logging.NewOfflineStoreSink(writeLoggedFeaturesCallback) if err != nil { - return err + return nil, err } loggingService, err = logging.NewLoggingService(s.fs, sink, logging.LoggingOptions{ @@ -243,9 +243,19 @@ func (s *OnlineFeatureService) StartGprcServerWithLogging(host string, port int, FlushInterval: loggingOpts.FlushInterval, }) if err != nil { - return err + return nil, err } } + return loggingService, nil +} + +// StartGprcServerWithLogging starts gRPC server with enabled feature logging +// Caller of this function must provide Python callback to flush buffered logs as well as logging configuration (loggingOpts) +func (s *OnlineFeatureService) StartGprcServerWithLogging(host string, port int, writeLoggedFeaturesCallback logging.OfflineStoreWriteCallback, loggingOpts LoggingOptions) error { + loggingService, err := s.constructLoggingService(writeLoggedFeaturesCallback, loggingOpts) + if err != nil { + return err + } ser := server.NewGrpcServingServiceServer(s.fs, loggingService) log.Printf("Starting a gRPC server on host %s port %d\n", host, port) lis, err := net.Listen("tcp", fmt.Sprintf("%s:%d", host, port)) @@ -274,7 +284,51 @@ func (s *OnlineFeatureService) StartGprcServerWithLogging(host string, port int, return nil } -func (s *OnlineFeatureService) Stop() { +// StartHttpServer starts HTTP server with disabled feature logging and blocks the thread +func (s *OnlineFeatureService) StartHttpServer(host string, port int) error { + return s.StartHttpServerWithLogging(host, port, nil, LoggingOptions{}) +} + +// StartHttpServerWithLoggingDefaultOpts starts HTTP server with enabled feature logging but default configuration for logging +// Caller of this function must provide Python callback to flush buffered logs +func (s *OnlineFeatureService) StartHttpServerWithLoggingDefaultOpts(host string, port int, writeLoggedFeaturesCallback logging.OfflineStoreWriteCallback) error { + defaultOpts := LoggingOptions{ + ChannelCapacity: logging.DefaultOptions.ChannelCapacity, + EmitTimeout: logging.DefaultOptions.EmitTimeout, + WriteInterval: logging.DefaultOptions.WriteInterval, + FlushInterval: logging.DefaultOptions.FlushInterval, + } + return s.StartHttpServerWithLogging(host, port, writeLoggedFeaturesCallback, defaultOpts) +} + +// StartHttpServerWithLogging starts HTTP server with enabled feature logging +// Caller of this function must provide Python callback to flush buffered logs as well as logging configuration (loggingOpts) +func (s *OnlineFeatureService) StartHttpServerWithLogging(host string, port int, writeLoggedFeaturesCallback logging.OfflineStoreWriteCallback, loggingOpts LoggingOptions) error { + loggingService, err := s.constructLoggingService(writeLoggedFeaturesCallback, loggingOpts) + if err != nil { + return err + } + ser := server.NewHttpServer(s.fs, loggingService) + log.Printf("Starting a HTTP server on host %s port %d\n", host, port) + + go func() { + // As soon as these signals are received from OS, try to gracefully stop the gRPC server + <-s.httpStopCh + fmt.Println("Stopping the HTTP server...") + err := ser.Stop() + if err != nil { + fmt.Printf("Error when stopping the HTTP server: %v\n", err) + } + }() + + return ser.Serve(host, port) +} + +func (s *OnlineFeatureService) StopHttpServer() { + s.httpStopCh <- syscall.SIGINT +} + +func (s *OnlineFeatureService) StopGrpcServer() { s.grpcStopCh <- syscall.SIGINT } diff --git a/go/internal/feast/server/grpc_server.go b/go/internal/feast/server/grpc_server.go index 60408809594..c47d185d6c1 100644 --- a/go/internal/feast/server/grpc_server.go +++ b/go/internal/feast/server/grpc_server.go @@ -86,7 +86,7 @@ func (s *grpcServingServiceServer) GetOnlineFeatures(ctx context.Context, reques fmt.Printf("Couldn't instantiate logger for feature service %s: %+v", featuresOrService.FeatureService.Name, err) } - err = logger.Log(entityValuesMap, resp.Results[len(request.Entities):], resp.Metadata.FeatureNames.Val[len(request.Entities):], request.RequestContext, requestId) + err = logger.Log(request.Entities, resp.Results[len(request.Entities):], resp.Metadata.FeatureNames.Val[len(request.Entities):], request.RequestContext, requestId) if err != nil { fmt.Printf("LoggerImpl error[%s]: %+v", featuresOrService.FeatureService.Name, err) } diff --git a/go/internal/feast/server/http_server.go b/go/internal/feast/server/http_server.go new file mode 100644 index 00000000000..75cdbe9929a --- /dev/null +++ b/go/internal/feast/server/http_server.go @@ -0,0 +1,270 @@ +package server + +import ( + "context" + "encoding/json" + "fmt" + "github.com/feast-dev/feast/go/internal/feast" + "github.com/feast-dev/feast/go/internal/feast/model" + "github.com/feast-dev/feast/go/internal/feast/server/logging" + "github.com/feast-dev/feast/go/protos/feast/serving" + prototypes "github.com/feast-dev/feast/go/protos/feast/types" + "github.com/feast-dev/feast/go/types" + "net/http" + "time" +) + +type httpServer struct { + fs *feast.FeatureStore + loggingService *logging.LoggingService + server *http.Server +} + +// Some Feast types aren't supported during JSON conversion +type repeatedValue struct { + stringVal []string + int64Val []int64 + doubleVal []float64 + boolVal []bool + stringListVal [][]string + int64ListVal [][]int64 + doubleListVal [][]float64 + boolListVal [][]bool +} + +func (u *repeatedValue) UnmarshalJSON(data []byte) error { + isString := false + isDouble := false + isInt64 := false + isArray := false + openBraketCounter := 0 + for _, b := range data { + if b == '"' { + isString = true + } + if b == '.' { + isDouble = true + } + if b >= '0' && b <= '9' { + isInt64 = true + } + if b == '[' { + openBraketCounter++ + if openBraketCounter > 1 { + isArray = true + } + } + } + var err error + if !isArray { + if isString { + err = json.Unmarshal(data, &u.stringVal) + } else if isDouble { + err = json.Unmarshal(data, &u.doubleVal) + } else if isInt64 { + err = json.Unmarshal(data, &u.int64Val) + } else { + err = json.Unmarshal(data, &u.boolVal) + } + } else { + if isString { + err = json.Unmarshal(data, &u.stringListVal) + } else if isDouble { + err = json.Unmarshal(data, &u.doubleListVal) + } else if isInt64 { + err = json.Unmarshal(data, &u.int64ListVal) + } else { + err = json.Unmarshal(data, &u.boolListVal) + } + } + return err +} + +func (u *repeatedValue) ToProto() *prototypes.RepeatedValue { + proto := new(prototypes.RepeatedValue) + if u.stringVal != nil { + for _, val := range u.stringVal { + proto.Val = append(proto.Val, &prototypes.Value{Val: &prototypes.Value_StringVal{StringVal: val}}) + } + } + if u.int64Val != nil { + for _, val := range u.int64Val { + proto.Val = append(proto.Val, &prototypes.Value{Val: &prototypes.Value_Int64Val{Int64Val: val}}) + } + } + if u.doubleVal != nil { + for _, val := range u.doubleVal { + proto.Val = append(proto.Val, &prototypes.Value{Val: &prototypes.Value_DoubleVal{DoubleVal: val}}) + } + } + if u.boolVal != nil { + for _, val := range u.boolVal { + proto.Val = append(proto.Val, &prototypes.Value{Val: &prototypes.Value_BoolVal{BoolVal: val}}) + } + } + if u.stringListVal != nil { + for _, val := range u.stringListVal { + proto.Val = append(proto.Val, &prototypes.Value{Val: &prototypes.Value_StringListVal{StringListVal: &prototypes.StringList{Val: val}}}) + } + } + if u.int64ListVal != nil { + for _, val := range u.int64ListVal { + proto.Val = append(proto.Val, &prototypes.Value{Val: &prototypes.Value_Int64ListVal{Int64ListVal: &prototypes.Int64List{Val: val}}}) + } + } + if u.doubleListVal != nil { + for _, val := range u.doubleListVal { + proto.Val = append(proto.Val, &prototypes.Value{Val: &prototypes.Value_DoubleListVal{DoubleListVal: &prototypes.DoubleList{Val: val}}}) + } + } + if u.boolListVal != nil { + for _, val := range u.boolListVal { + proto.Val = append(proto.Val, &prototypes.Value{Val: &prototypes.Value_BoolListVal{BoolListVal: &prototypes.BoolList{Val: val}}}) + } + } + return proto +} + +type getOnlineFeaturesRequest struct { + FeatureService *string `json:"feature_service"` + Features []string `json:"features"` + Entities map[string]repeatedValue `json:"entities"` + FullFeatureNames bool `json:"full_feature_names"` + RequestContext map[string]repeatedValue `json:"request_context"` +} + +func NewHttpServer(fs *feast.FeatureStore, loggingService *logging.LoggingService) *httpServer { + return &httpServer{fs: fs, loggingService: loggingService} +} + +func (s *httpServer) getOnlineFeatures(w http.ResponseWriter, r *http.Request) { + if r.Method != "POST" { + http.NotFound(w, r) + return + } + + decoder := json.NewDecoder(r.Body) + var request getOnlineFeaturesRequest + err := decoder.Decode(&request) + if err != nil { + http.Error(w, fmt.Sprintf("Error decoding JSON request data: %+v", err), http.StatusInternalServerError) + return + } + var featureService *model.FeatureService + if request.FeatureService != nil { + featureService, err = s.fs.GetFeatureService(*request.FeatureService) + if err != nil { + http.Error(w, fmt.Sprintf("Error getting feature service from registry: %+v", err), http.StatusInternalServerError) + return + } + } + entitiesProto := make(map[string]*prototypes.RepeatedValue) + for key, value := range request.Entities { + entitiesProto[key] = value.ToProto() + } + requestContextProto := make(map[string]*prototypes.RepeatedValue) + for key, value := range request.RequestContext { + requestContextProto[key] = value.ToProto() + } + + featureVectors, err := s.fs.GetOnlineFeatures( + r.Context(), + request.Features, + featureService, + entitiesProto, + requestContextProto, + request.FullFeatureNames) + + if err != nil { + http.Error(w, fmt.Sprintf("Error getting feature vector: %+v", err), http.StatusInternalServerError) + return + } + + var featureNames []string + var results []map[string]interface{} + for _, vector := range featureVectors { + featureNames = append(featureNames, vector.Name) + result := make(map[string]interface{}) + var statuses []string + for _, status := range vector.Statuses { + statuses = append(statuses, status.String()) + } + var timestamps []string + for _, timestamp := range vector.Timestamps { + timestamps = append(timestamps, timestamp.AsTime().Format(time.RFC3339)) + } + + result["statuses"] = statuses + result["event_timestamps"] = timestamps + // Note, that vector.Values is an Arrow Array, but this type implements JSON Marshaller. + // So, it's not necessary to pre-process it in any way. + result["values"] = vector.Values + + results = append(results, result) + } + + response := map[string]interface{}{ + "metadata": map[string]interface{}{ + "feature_names": featureNames, + }, + "results": results, + } + + err = json.NewEncoder(w).Encode(response) + + if err != nil { + http.Error(w, fmt.Sprintf("Error encoding response: %+v", err), http.StatusInternalServerError) + return + } + + w.Header().Set("Content-Type", "application/json") + + if featureService != nil && featureService.LoggingConfig != nil && s.loggingService != nil { + logger, err := s.loggingService.GetOrCreateLogger(featureService) + if err != nil { + http.Error(w, fmt.Sprintf("Couldn't instantiate logger for feature service %s: %+v", featureService.Name, err), http.StatusInternalServerError) + return + } + + requestId := GenerateRequestId() + + // Note: we're converting arrow to proto for feature logging. In the future we should + // base feature logging on arrow so that we don't have to do this extra conversion. + var featureVectorProtos []*serving.GetOnlineFeaturesResponse_FeatureVector + for _, vector := range featureVectors[len(request.Entities):] { + values, err := types.ArrowValuesToProtoValues(vector.Values) + if err != nil { + http.Error(w, fmt.Sprintf("Couldn't convert arrow values into protobuf: %+v", err), http.StatusInternalServerError) + return + } + featureVectorProtos = append(featureVectorProtos, &serving.GetOnlineFeaturesResponse_FeatureVector{ + Values: values, + Statuses: vector.Statuses, + EventTimestamps: vector.Timestamps, + }) + } + + err = logger.Log(entitiesProto, featureVectorProtos, featureNames[len(request.Entities):], requestContextProto, requestId) + if err != nil { + http.Error(w, fmt.Sprintf("LoggerImpl error[%s]: %+v", featureService.Name, err), http.StatusInternalServerError) + return + } + } +} + +func (s *httpServer) Serve(host string, port int) error { + s.server = &http.Server{Addr: fmt.Sprintf("%s:%d", host, port), Handler: nil} + http.HandleFunc("/get-online-features", s.getOnlineFeatures) + err := s.server.ListenAndServe() + // Don't return the error if it's caused by graceful shutdown using Stop() + if err == http.ErrServerClosed { + return nil + } + return err +} +func (s *httpServer) Stop() error { + if s.server != nil { + return s.server.Shutdown(context.Background()) + } + return nil +} diff --git a/go/internal/feast/server/http_server_test.go b/go/internal/feast/server/http_server_test.go new file mode 100644 index 00000000000..67ba1c60f96 --- /dev/null +++ b/go/internal/feast/server/http_server_test.go @@ -0,0 +1,40 @@ +package server + +import ( + "github.com/stretchr/testify/assert" + "testing" +) + +func TestUnmarshalJSON(t *testing.T) { + u := repeatedValue{} + assert.Nil(t, u.UnmarshalJSON([]byte("[1, 2, 3]"))) + assert.Equal(t, []int64{1, 2, 3}, u.int64Val) + + u = repeatedValue{} + assert.Nil(t, u.UnmarshalJSON([]byte("[1.2, 2.3, 3.4]"))) + assert.Equal(t, []float64{1.2, 2.3, 3.4}, u.doubleVal) + + u = repeatedValue{} + assert.Nil(t, u.UnmarshalJSON([]byte("[\"foo\", \"bar\"]"))) + assert.Equal(t, []string{"foo", "bar"}, u.stringVal) + + u = repeatedValue{} + assert.Nil(t, u.UnmarshalJSON([]byte("[true, false, true]"))) + assert.Equal(t, []bool{true, false, true}, u.boolVal) + + u = repeatedValue{} + assert.Nil(t, u.UnmarshalJSON([]byte("[[1, 2, 3], [4, 5, 6]]"))) + assert.Equal(t, [][]int64{{1, 2, 3}, {4, 5, 6}}, u.int64ListVal) + + u = repeatedValue{} + assert.Nil(t, u.UnmarshalJSON([]byte("[[1.2, 2.3, 3.4], [10.2, 20.3, 30.4]]"))) + assert.Equal(t, [][]float64{{1.2, 2.3, 3.4}, {10.2, 20.3, 30.4}}, u.doubleListVal) + + u = repeatedValue{} + assert.Nil(t, u.UnmarshalJSON([]byte("[[\"foo\", \"bar\"], [\"foo2\", \"bar2\"]]"))) + assert.Equal(t, [][]string{{"foo", "bar"}, {"foo2", "bar2"}}, u.stringListVal) + + u = repeatedValue{} + assert.Nil(t, u.UnmarshalJSON([]byte("[[true, false, true], [false, true, false]]"))) + assert.Equal(t, [][]bool{{true, false, true}, {false, true, false}}, u.boolListVal) +} diff --git a/go/internal/feast/server/logging/logger.go b/go/internal/feast/server/logging/logger.go index d7ed1fbe189..cbf1c3439a6 100644 --- a/go/internal/feast/server/logging/logger.go +++ b/go/internal/feast/server/logging/logger.go @@ -42,7 +42,7 @@ type LogSink interface { } type Logger interface { - Log(joinKeyToEntityValues map[string][]*types.Value, featureVectors []*serving.GetOnlineFeaturesResponse_FeatureVector, featureNames []string, requestData map[string]*types.RepeatedValue, requestId string) error + Log(joinKeyToEntityValues map[string]*types.RepeatedValue, featureVectors []*serving.GetOnlineFeaturesResponse_FeatureVector, featureNames []string, requestData map[string]*types.RepeatedValue, requestId string) error } type LoggerImpl struct { @@ -207,7 +207,7 @@ func getFullFeatureName(featureViewName string, featureName string) string { return fmt.Sprintf("%s__%s", featureViewName, featureName) } -func (l *LoggerImpl) Log(joinKeyToEntityValues map[string][]*types.Value, featureVectors []*serving.GetOnlineFeaturesResponse_FeatureVector, featureNames []string, requestData map[string]*types.RepeatedValue, requestId string) error { +func (l *LoggerImpl) Log(joinKeyToEntityValues map[string]*types.RepeatedValue, featureVectors []*serving.GetOnlineFeaturesResponse_FeatureVector, featureNames []string, requestData map[string]*types.RepeatedValue, requestId string) error { if len(featureVectors) == 0 { return nil } @@ -250,7 +250,7 @@ func (l *LoggerImpl) Log(joinKeyToEntityValues map[string][]*types.Value, featur if !ok { return errors.Errorf("Missing join key %s in log data", joinKey) } - entityValues[idx] = rows[rowIdx] + entityValues[idx] = rows.Val[rowIdx] } requestDataValues := make([]*types.Value, len(l.schema.RequestData)) @@ -283,6 +283,6 @@ func (l *LoggerImpl) Log(joinKeyToEntityValues map[string][]*types.Value, featur type DummyLoggerImpl struct{} -func (l *DummyLoggerImpl) Log(joinKeyToEntityValues map[string][]*types.Value, featureVectors []*serving.GetOnlineFeaturesResponse_FeatureVector, featureNames []string, requestData map[string]*types.RepeatedValue, requestId string) error { +func (l *DummyLoggerImpl) Log(joinKeyToEntityValues map[string]*types.RepeatedValue, featureVectors []*serving.GetOnlineFeaturesResponse_FeatureVector, featureNames []string, requestData map[string]*types.RepeatedValue, requestId string) error { return nil } diff --git a/go/internal/feast/server/logging/logger_test.go b/go/internal/feast/server/logging/logger_test.go index 5625b05a766..4ce883c75bd 100644 --- a/go/internal/feast/server/logging/logger_test.go +++ b/go/internal/feast/server/logging/logger_test.go @@ -90,7 +90,17 @@ func TestLogAndFlushToFile(t *testing.T) { assert.Nil(t, err) assert.Nil(t, logger.Log( - map[string][]*types.Value{"driver_id": {{Val: &types.Value_Int32Val{Int32Val: 111}}}}, + map[string]*types.RepeatedValue{ + "driver_id": { + Val: []*types.Value{ + { + Val: &types.Value_Int32Val{ + Int32Val: 111, + }, + }, + }, + }, + }, []*serving.GetOnlineFeaturesResponse_FeatureVector{ { Values: []*types.Value{{Val: &types.Value_DoubleVal{DoubleVal: 2.0}}}, diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py index a4407132e4e..b1281d297f1 100644 --- a/sdk/python/feast/cli.py +++ b/sdk/python/feast/cli.py @@ -610,17 +610,27 @@ def init_command(project_directory, minimal: bool, template: str): default=6566, help="Specify a port for the server [default: 6566]", ) +@click.option( + "--type", + "-t", + "type_", + type=click.STRING, + default="http", + help="Specify a server type: 'http' or 'grpc' [default: http]", +) @click.option( "--no-access-log", is_flag=True, help="Disable the Uvicorn access log.", ) @click.pass_context -def serve_command(ctx: click.Context, host: str, port: int, no_access_log: bool): +def serve_command( + ctx: click.Context, host: str, port: int, type_: str, no_access_log: bool +): """Start a feature server locally on a given port.""" repo = ctx.obj["CHDIR"] cli_check_repo(repo) store = FeatureStore(repo_path=str(repo)) - store.serve(host, port, no_access_log) + store.serve(host, port, type_, no_access_log) @cli.command("serve_transformations") diff --git a/sdk/python/feast/embedded_go/online_features_service.py b/sdk/python/feast/embedded_go/online_features_service.py index 48e31766cb3..8ec4410bde7 100644 --- a/sdk/python/feast/embedded_go/online_features_service.py +++ b/sdk/python/feast/embedded_go/online_features_service.py @@ -158,8 +158,30 @@ def start_grpc_server( else: self._service.StartGprcServer(host, port) + def start_http_server( + self, + host: str, + port: int, + enable_logging: bool = True, + logging_options: Optional[LoggingOptions] = None, + ): + if enable_logging: + if logging_options: + self._service.StartHttpServerWithLogging( + host, port, self._logging_callback, logging_options + ) + else: + self._service.StartHttpServerWithLoggingDefaultOpts( + host, port, self._logging_callback + ) + else: + self._service.StartHttpServer(host, port) + def stop_grpc_server(self): - self._service.Stop() + self._service.StopGrpcServer() + + def stop_http_server(self): + self._service.StopHttpServer() def _to_arrow(value, type_hint: Optional[ValueType]) -> pa.Array: diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py index 768823a68c0..edd0f5a46c7 100644 --- a/sdk/python/feast/feature_store.py +++ b/sdk/python/feast/feature_store.py @@ -1980,14 +1980,25 @@ def _get_feature_views_to_use( return views_to_use @log_exceptions_and_usage - def serve(self, host: str, port: int, no_access_log: bool) -> None: + def serve(self, host: str, port: int, type_: str, no_access_log: bool) -> None: """Start the feature consumption server locally on a given port.""" + type_ = type_.lower() if self.config.go_feature_retrieval: # Start go server instead of python if the flag is enabled self._lazy_init_go_server() - # TODO(tsotne) add http/grpc flag in CLI and call appropriate method here depending on that - self._go_server.start_grpc_server(host, port) + if type_ == "http": + self._go_server.start_http_server(host, port) + elif type_ == "grpc": + self._go_server.start_grpc_server(host, port) + else: + raise ValueError( + f"Unsupported server type '{type_}'. Must be one of 'http' or 'grpc'." + ) else: + if type_ != "http": + raise ValueError( + f"Python server only supports 'http'. Got '{type_}' instead." + ) # Start the python server if go server isn't enabled feature_server.start_server(self, host, port, no_access_log) diff --git a/sdk/python/tests/integration/e2e/test_go_feature_server.py b/sdk/python/tests/integration/e2e/test_go_feature_server.py index 4e4cfc1fb8f..c87d79564fd 100644 --- a/sdk/python/tests/integration/e2e/test_go_feature_server.py +++ b/sdk/python/tests/integration/e2e/test_go_feature_server.py @@ -9,6 +9,7 @@ import pandas as pd import pytest import pytz +import requests from feast import FeatureService, FeatureView, ValueType from feast.embedded_go.lib.embedded import LoggingOptions @@ -61,8 +62,7 @@ def initialized_registry(environment, universal_data_sources): fs.materialize(environment.start_date, environment.end_date) -@pytest.fixture -def grpc_server_port(environment, initialized_registry): +def server_port(environment, server_type: str): if not environment.test_repo_config.go_feature_retrieval: pytest.skip("Only for Go path") @@ -72,9 +72,15 @@ def grpc_server_port(environment, initialized_registry): repo_path=str(fs.repo_path.absolute()), repo_config=fs.config, feature_store=fs, ) port = free_port() + if server_type == "grpc": + target = embedded.start_grpc_server + elif server_type == "http": + target = embedded.start_http_server + else: + raise ValueError("Server Type must be either 'http' or 'grpc'") t = threading.Thread( - target=embedded.start_grpc_server, + target=target, args=("127.0.0.1", port), kwargs=dict( enable_logging=True, @@ -93,11 +99,24 @@ def grpc_server_port(environment, initialized_registry): ) yield port - embedded.stop_grpc_server() + if server_type == "grpc": + embedded.stop_grpc_server() + else: + embedded.stop_http_server() # wait for graceful stop time.sleep(2) +@pytest.fixture +def grpc_server_port(environment, initialized_registry): + yield from server_port(environment, "grpc") + + +@pytest.fixture +def http_server_port(environment, initialized_registry): + yield from server_port(environment, "http") + + @pytest.fixture def grpc_client(grpc_server_port): ch = grpc.insecure_channel(f"localhost:{grpc_server_port}") @@ -130,6 +149,44 @@ def test_go_grpc_server(grpc_client): assert all([s == FieldStatus.PRESENT for s in vector.statuses]) +@pytest.mark.integration +@pytest.mark.goserver +def test_go_http_server(http_server_port): + response = requests.post( + f"http://localhost:{http_server_port}/get-online-features", + json={ + "feature_service": "driver_features", + "entities": {"driver_id": [5001, 5002]}, + "full_feature_names": True, + }, + ) + assert response.status_code == 200, response.text + response = response.json() + assert set(response.keys()) == {"metadata", "results"} + metadata = response["metadata"] + results = response["results"] + assert response["metadata"] == { + "feature_names": [ + "driver_id", + "driver_stats__conv_rate", + "driver_stats__acc_rate", + "driver_stats__avg_daily_trips", + ] + }, metadata + assert len(results) == 4, results + assert all( + set(result.keys()) == {"event_timestamps", "statuses", "values"} + for result in results + ), results + assert all( + result["statuses"] == ["PRESENT", "PRESENT"] for result in results + ), results + assert results[0]["values"] == [5001, 5002], results + for result in results[1:]: + assert len(result["values"]) == 2, result + assert all(value is not None for value in result["values"]), result + + @pytest.mark.integration @pytest.mark.goserver @pytest.mark.universal_offline_stores From b0e2a8d31f259d4b2ba9f02fd9c9115aade2adee Mon Sep 17 00:00:00 2001 From: ssge Date: Fri, 20 May 2022 00:46:03 +0300 Subject: [PATCH 018/123] chore: Local data inspection (#2720) fix for #2694 adding the example code from the Google Codelab notebook moving the image below the yaml and example.py explanation Signed-off-by: ssge --- docs/getting-started/quickstart.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md index 1684c720632..1b926eeca5c 100644 --- a/docs/getting-started/quickstart.md +++ b/docs/getting-started/quickstart.md @@ -125,8 +125,6 @@ driver_stats_fs = FeatureService( {% endtab %} {% endtabs %} -![Demo parquet data: data/driver\_stats.parquet](../.gitbook/assets/screen-shot-2021-08-23-at-2.35.18-pm.png) - The key line defining the overall architecture of the feature store is the **provider**. This defines where the raw data exists (for generating training data & feature values for serving), and where to materialize feature values to in the online store (for serving). Valid values for `provider` in `feature_store.yaml` are: @@ -139,6 +137,16 @@ Note that there are many other sources Feast works with, including Azure, Hive, A custom setup can also be made by following [adding a custom provider](../how-to-guides/creating-a-custom-provider.md). +### Inspecting the raw data + +The raw feature data we have in this demo is stored in a local parquet file. The dataset captures hourly stats of a driver in a ride-sharing app. + +```python +import pandas as pd +pd.read_parquet("data/driver_stats.parquet") +``` + +![Demo parquet data: data/driver\_stats.parquet](../.gitbook/assets/screen-shot-2021-08-23-at-2.35.18-pm.png) ## Step 3: Register feature definitions and deploy your feature store From 568a0ffcb28ddf1033f6e8ae07c792d1b4f5f2b0 Mon Sep 17 00:00:00 2001 From: Danny Chiao Date: Fri, 20 May 2022 13:56:18 -0400 Subject: [PATCH 019/123] docs: Add reference to workshop (#2723) --- docs/SUMMARY.md | 1 + docs/getting-started/feast-workshop.md | 36 ++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100644 docs/getting-started/feast-workshop.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index f6f12e04d04..7ea80a4e6b5 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -23,6 +23,7 @@ * [Offline store](getting-started/architecture-and-components/offline-store.md) * [Online store](getting-started/architecture-and-components/online-store.md) * [Provider](getting-started/architecture-and-components/provider.md) +* [Learning by example](getting-started/feast-workshop.md) * [Third party integrations](getting-started/third-party-integrations.md) * [FAQ](getting-started/faq.md) diff --git a/docs/getting-started/feast-workshop.md b/docs/getting-started/feast-workshop.md new file mode 100644 index 00000000000..75b0e15acce --- /dev/null +++ b/docs/getting-started/feast-workshop.md @@ -0,0 +1,36 @@ +This workshop aims to teach users about Feast. + +We explain concepts & best practices by example, and also showcase how to address common use cases. + +### Pre-requisites +This workshop assumes you have the following installed: +- A local development environment that supports running Jupyter notebooks (e.g. VSCode with Jupyter plugin) +- Python 3.7+ +- Java 11 (for Spark, e.g. `brew install java11`) +- pip +- Docker & Docker Compose (e.g. `brew install docker docker-compose`) +- Terraform ([docs](https://learn.hashicorp.com/tutorials/terraform/install-cli#install-terraform)) +- AWS CLI +- An AWS account setup with credentials via `aws configure` (e.g see [AWS credentials quickstart](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-creds)) + +Since we'll be learning how to leverage Feast in CI/CD, you'll also need to fork this workshop repository. + +**Caveats** +- M1 Macbook development is untested with this flow. See also [How to run / develop for Feast on M1 Macs](https://github.com/feast-dev/feast/issues/2105). +- Windows development has only been tested with WSL. You will need to follow this [guide](https://docs.docker.com/desktop/windows/wsl/) to have Docker play nicely. + +## Modules +*See also: [Feast quickstart](https://docs.feast.dev/getting-started/quickstart), [Feast x Great Expectations tutorial](https://docs.feast.dev/tutorials/validating-historical-features)* + +These are meant mostly to be done in order, with examples building on previous concepts. + +| Time (min) | Description | Module    | +| :--------: | :---------------------------------------------------------------------- | :------------------------------------------------------------------------- | +| 30-45 | Setting up Feast projects & CI/CD + powering batch predictions | [Module 0](https://github.com/feast-dev/feast-workshop/tree/main/module_0) | +| 15-20 | Streaming ingestion & online feature retrieval with Kafka, Spark, Redis | [Module 1](https://github.com/feast-dev/feast-workshop/tree/main/module_1) | +| 10-15 | Real-time feature engineering with on demand transformations | [Module 2](https://github.com/feast-dev/feast-workshop/tree/main/module_2) | +| TBD | Feature server deployment (embed, as a service, AWS Lambda) | TBD | +| TBD | Versioning features / models in Feast | TBD | +| TBD | Data quality monitoring in Feast | TBD | +| TBD | Batch transformations | TBD | +| TBD | Stream transformations | TBD | \ No newline at end of file From 8b68fb9d53fd73cea298a4e0e1abb58c0fc60597 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Fri, 20 May 2022 12:10:29 -0700 Subject: [PATCH 020/123] chore: Add a .gitpod.yml file (#2722) * chore: Add .gitpod.yml Signed-off-by: Achal Shah * use the right init command Signed-off-by: Achal Shah * more setup Signed-off-by: Achal Shah * update comment Signed-off-by: Achal Shah --- .gitpod.yml | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 .gitpod.yml diff --git a/.gitpod.yml b/.gitpod.yml new file mode 100644 index 00000000000..e14a5f807b7 --- /dev/null +++ b/.gitpod.yml @@ -0,0 +1,28 @@ +# https://www.gitpod.io/docs/config-gitpod-file +tasks: + - init: | + pip install pre-commit + pre-commit install --hook-type pre-commit --hook-type pre-push + pip install -e '.[dev]' + git config --global alias.ci 'commit -s' + git config --global alias.sw switch + git config --global alias.st status + git config --global alias.co checkout + git config --global alias.br branch + git config --global alias.df diff +github: + prebuilds: + # enable for the default branch (defaults to true) + master: true + # enable for all branches in this repo (defaults to false) + branches: false + # enable for pull requests coming from this repo (defaults to true) + pullRequests: true + # enable for pull requests coming from forks (defaults to false) + pullRequestsFromForks: false + # add a check to pull requests (defaults to true) + addCheck: true + # add a "Review in Gitpod" button as a comment to pull requests (defaults to false) + addComment: false + # add a "Review in Gitpod" button to the pull request's description (defaults to false) + addBadge: false From 43480a41bacb3bfcef2e8de1245715d53d412aaf Mon Sep 17 00:00:00 2001 From: Danny Chiao Date: Fri, 20 May 2022 15:07:27 -0400 Subject: [PATCH 021/123] docs: Fix alignment for workshop doc Signed-off-by: Danny Chiao --- docs/getting-started/feast-workshop.md | 52 ++++++++++++++------------ 1 file changed, 29 insertions(+), 23 deletions(-) diff --git a/docs/getting-started/feast-workshop.md b/docs/getting-started/feast-workshop.md index 75b0e15acce..c883625dac9 100644 --- a/docs/getting-started/feast-workshop.md +++ b/docs/getting-started/feast-workshop.md @@ -1,36 +1,42 @@ +# Learning by example + This workshop aims to teach users about Feast. We explain concepts & best practices by example, and also showcase how to address common use cases. ### Pre-requisites + This workshop assumes you have the following installed: -- A local development environment that supports running Jupyter notebooks (e.g. VSCode with Jupyter plugin) -- Python 3.7+ -- Java 11 (for Spark, e.g. `brew install java11`) -- pip -- Docker & Docker Compose (e.g. `brew install docker docker-compose`) -- Terraform ([docs](https://learn.hashicorp.com/tutorials/terraform/install-cli#install-terraform)) -- AWS CLI -- An AWS account setup with credentials via `aws configure` (e.g see [AWS credentials quickstart](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-creds)) + +* A local development environment that supports running Jupyter notebooks (e.g. VSCode with Jupyter plugin) +* Python 3.7+ +* Java 11 (for Spark, e.g. `brew install java11`) +* pip +* Docker & Docker Compose (e.g. `brew install docker docker-compose`) +* Terraform ([docs](https://learn.hashicorp.com/tutorials/terraform/install-cli#install-terraform)) +* AWS CLI +* An AWS account setup with credentials via `aws configure` (e.g see [AWS credentials quickstart](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-creds)) Since we'll be learning how to leverage Feast in CI/CD, you'll also need to fork this workshop repository. -**Caveats** -- M1 Macbook development is untested with this flow. See also [How to run / develop for Feast on M1 Macs](https://github.com/feast-dev/feast/issues/2105). -- Windows development has only been tested with WSL. You will need to follow this [guide](https://docs.docker.com/desktop/windows/wsl/) to have Docker play nicely. +#### **Caveats** + +* M1 Macbook development is untested with this flow. See also [How to run / develop for Feast on M1 Macs](https://github.com/feast-dev/feast/issues/2105). +* Windows development has only been tested with WSL. You will need to follow this [guide](https://docs.docker.com/desktop/windows/wsl/) to have Docker play nicely. + +### Modules -## Modules -*See also: [Feast quickstart](https://docs.feast.dev/getting-started/quickstart), [Feast x Great Expectations tutorial](https://docs.feast.dev/tutorials/validating-historical-features)* +_See also:_ [_Feast quickstart_](https://docs.feast.dev/getting-started/quickstart)_,_ [_Feast x Great Expectations tutorial_](https://docs.feast.dev/tutorials/validating-historical-features) These are meant mostly to be done in order, with examples building on previous concepts. -| Time (min) | Description | Module    | -| :--------: | :---------------------------------------------------------------------- | :------------------------------------------------------------------------- | -| 30-45 | Setting up Feast projects & CI/CD + powering batch predictions | [Module 0](https://github.com/feast-dev/feast-workshop/tree/main/module_0) | -| 15-20 | Streaming ingestion & online feature retrieval with Kafka, Spark, Redis | [Module 1](https://github.com/feast-dev/feast-workshop/tree/main/module_1) | -| 10-15 | Real-time feature engineering with on demand transformations | [Module 2](https://github.com/feast-dev/feast-workshop/tree/main/module_2) | -| TBD | Feature server deployment (embed, as a service, AWS Lambda) | TBD | -| TBD | Versioning features / models in Feast | TBD | -| TBD | Data quality monitoring in Feast | TBD | -| TBD | Batch transformations | TBD | -| TBD | Stream transformations | TBD | \ No newline at end of file +| Time (min) | Description | Module | +| :--------: | ----------------------------------------------------------------------- | --------------------------------------------------------------------------- | +| 30-45 | Setting up Feast projects & CI/CD + powering batch predictions | [Module 0](https://github.com/feast-dev/feast-workshop/tree/main/module\_0) | +| 15-20 | Streaming ingestion & online feature retrieval with Kafka, Spark, Redis | [Module 1](https://github.com/feast-dev/feast-workshop/tree/main/module\_1) | +| 10-15 | Real-time feature engineering with on demand transformations | [Module 2](https://github.com/feast-dev/feast-workshop/tree/main/module\_2) | +| TBD | Feature server deployment (embed, as a service, AWS Lambda) | TBD | +| TBD | Versioning features / models in Feast | TBD | +| TBD | Data quality monitoring in Feast | TBD | +| TBD | Batch transformations | TBD | +| TBD | Stream transformations | TBD | From 8ec07908f2bcb4383b8d948a4a6395f6f5ab9489 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Fri, 20 May 2022 13:11:29 -0700 Subject: [PATCH 022/123] chore: Remove misleading connection option from redis docstring (#2725) Signed-off-by: Achal Shah --- sdk/python/feast/infra/online_stores/redis.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdk/python/feast/infra/online_stores/redis.py b/sdk/python/feast/infra/online_stores/redis.py index 1a8eedb21e0..60fa9265ca7 100644 --- a/sdk/python/feast/infra/online_stores/redis.py +++ b/sdk/python/feast/infra/online_stores/redis.py @@ -132,7 +132,7 @@ def _parse_connection_string(connection_string: str): """ Reads Redis connections string using format for RedisCluster: - redis1:6379,redis2:6379,decode_responses=true,skip_full_coverage_check=true,ssl=true,password=... + redis1:6379,redis2:6379,skip_full_coverage_check=true,ssl=true,password=... for Redis: redis_master:6379,db=0,ssl=true,password=... """ From c8b11b3b790b60e916d3257a036b9cb6430f4685 Mon Sep 17 00:00:00 2001 From: Oleksii Moskalenko Date: Fri, 20 May 2022 14:12:29 -0700 Subject: [PATCH 023/123] feat: CLI interface for validation of logged features (#2718) * store validation reference in registry Signed-off-by: Oleksii Moskalenko * CLI test Signed-off-by: Oleksii Moskalenko * clean function before pickle Signed-off-by: Oleksii Moskalenko * ignore "too complex" lint rule Signed-off-by: Oleksii Moskalenko * lazy import & correct feature status in logs Signed-off-by: Oleksii Moskalenko * pygments dependency Signed-off-by: Oleksii Moskalenko * ttl for regular feature views Signed-off-by: Oleksii Moskalenko * some apidocs Signed-off-by: Oleksii Moskalenko * address comments Signed-off-by: Oleksii Moskalenko --- go/embedded/online_features.go | 12 +- protos/feast/core/FeatureService.proto | 1 - protos/feast/core/Registry.proto | 4 +- protos/feast/core/ValidationProfile.proto | 21 ++- sdk/python/feast/cli.py | 59 ++++++++- sdk/python/feast/diff/registry_diff.py | 10 +- sdk/python/feast/dqm/profilers/ge_profiler.py | 6 +- sdk/python/feast/dqm/profilers/profiler.py | 13 ++ sdk/python/feast/errors.py | 7 + sdk/python/feast/feast_object.py | 6 + sdk/python/feast/feature_store.py | 38 ++++++ .../feast/infra/passthrough_provider.py | 4 +- sdk/python/feast/registry.py | 84 +++++++++++- sdk/python/feast/saved_dataset.py | 121 ++++++++++++++++-- .../requirements/py3.10-ci-requirements.txt | 1 + .../requirements/py3.10-requirements.txt | 2 + .../requirements/py3.7-ci-requirements.txt | 1 + .../requirements/py3.7-requirements.txt | 2 + .../requirements/py3.8-ci-requirements.txt | 1 + .../requirements/py3.8-requirements.txt | 2 + .../requirements/py3.9-ci-requirements.txt | 1 + .../requirements/py3.9-requirements.txt | 2 + sdk/python/setup.cfg | 2 +- sdk/python/tests/conftest.py | 8 ++ .../integration/e2e/test_go_feature_server.py | 3 +- .../tests/integration/e2e/test_validation.py | 95 +++++++++++++- .../online_store/test_universal_online.py | 1 - sdk/python/tests/utils/cli_utils.py | 4 +- sdk/python/tests/utils/logged_features.py | 38 ++++-- setup.py | 1 + 30 files changed, 505 insertions(+), 45 deletions(-) diff --git a/go/embedded/online_features.go b/go/embedded/online_features.go index de7e6d11a2e..2563e0f43b4 100644 --- a/go/embedded/online_features.go +++ b/go/embedded/online_features.go @@ -269,12 +269,12 @@ func (s *OnlineFeatureService) StartGprcServerWithLogging(host string, port int, go func() { // As soon as these signals are received from OS, try to gracefully stop the gRPC server <-s.grpcStopCh - fmt.Println("Stopping the gRPC server...") + log.Println("Stopping the gRPC server...") grpcServer.GracefulStop() if loggingService != nil { loggingService.Stop() } - fmt.Println("gRPC server terminated") + log.Println("gRPC server terminated") }() err = grpcServer.Serve(lis) @@ -314,11 +314,15 @@ func (s *OnlineFeatureService) StartHttpServerWithLogging(host string, port int, go func() { // As soon as these signals are received from OS, try to gracefully stop the gRPC server <-s.httpStopCh - fmt.Println("Stopping the HTTP server...") + log.Println("Stopping the HTTP server...") err := ser.Stop() if err != nil { - fmt.Printf("Error when stopping the HTTP server: %v\n", err) + log.Printf("Error when stopping the HTTP server: %v\n", err) } + if loggingService != nil { + loggingService.Stop() + } + log.Println("HTTP server terminated") }() return ser.Serve(host, port) diff --git a/protos/feast/core/FeatureService.proto b/protos/feast/core/FeatureService.proto index c04fa97507d..2654703cc59 100644 --- a/protos/feast/core/FeatureService.proto +++ b/protos/feast/core/FeatureService.proto @@ -54,7 +54,6 @@ message FeatureServiceMeta { message LoggingConfig { float sample_rate = 1; - google.protobuf.Duration partition_interval = 2; oneof destination { FileDestination file_destination = 3; diff --git a/protos/feast/core/Registry.proto b/protos/feast/core/Registry.proto index 1978f41064a..2c31101510b 100644 --- a/protos/feast/core/Registry.proto +++ b/protos/feast/core/Registry.proto @@ -30,9 +30,10 @@ import "feast/core/OnDemandFeatureView.proto"; import "feast/core/RequestFeatureView.proto"; import "feast/core/DataSource.proto"; import "feast/core/SavedDataset.proto"; +import "feast/core/ValidationProfile.proto"; import "google/protobuf/timestamp.proto"; -// Next id: 13 +// Next id: 14 message Registry { repeated Entity entities = 1; repeated FeatureTable feature_tables = 2; @@ -42,6 +43,7 @@ message Registry { repeated RequestFeatureView request_feature_views = 9; repeated FeatureService feature_services = 7; repeated SavedDataset saved_datasets = 11; + repeated ValidationReference validation_references = 13; Infra infra = 10; string registry_schema_version = 3; // to support migrations; incremented when schema is changed diff --git a/protos/feast/core/ValidationProfile.proto b/protos/feast/core/ValidationProfile.proto index 673a792fdf8..b660e449bd2 100644 --- a/protos/feast/core/ValidationProfile.proto +++ b/protos/feast/core/ValidationProfile.proto @@ -39,9 +39,24 @@ message GEValidationProfile { } message ValidationReference { - SavedDataset dataset = 1; - + // Unique name of validation reference within the project + string name = 1; + // Name of saved dataset used as reference dataset + string reference_dataset_name = 2; + // Name of Feast project that this object source belongs to + string project = 3; + // Description of the validation reference + string description = 4; + // User defined metadata + map tags = 5; + + // validation profiler oneof profiler { - GEValidationProfiler ge_profiler = 2; + GEValidationProfiler ge_profiler = 6; + } + + // (optional) cached validation profile (to avoid constant recalculation) + oneof cached_profile { + GEValidationProfile ge_profile = 7; } } diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py index b1281d297f1..9f3cf26dee0 100644 --- a/sdk/python/feast/cli.py +++ b/sdk/python/feast/cli.py @@ -11,7 +11,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - +import json import logging import warnings from datetime import datetime @@ -23,6 +23,7 @@ import yaml from colorama import Fore, Style from dateutil import parser +from pygments import formatters, highlight, lexers from feast import flags, flags_helper, utils from feast.constants import DEFAULT_FEATURE_TRANSFORMATION_SERVER_PORT @@ -758,5 +759,61 @@ def disable_alpha_features(ctx: click.Context): store.config.write_to_path(Path(repo_path)) +@cli.command("validate") +@click.option( + "--feature-service", "-f", help="Specify a feature service name", +) +@click.option( + "--reference", "-r", help="Specify a validation reference name", +) +@click.option( + "--no-profile-cache", is_flag=True, help="Do not store cached profile in registry", +) +@click.argument("start_ts") +@click.argument("end_ts") +@click.pass_context +def validate( + ctx: click.Context, + feature_service: str, + reference: str, + start_ts: str, + end_ts: str, + no_profile_cache, +): + """ + Perform validation of logged features (produced by a given feature service) against provided reference. + + START_TS and END_TS should be in ISO 8601 format, e.g. '2021-07-16T19:20:01' + """ + repo = ctx.obj["CHDIR"] + cli_check_repo(repo) + store = FeatureStore(repo_path=str(repo)) + + feature_service = store.get_feature_service(name=feature_service) + reference = store.get_validation_reference(reference) + + result = store.validate_logged_features( + source=feature_service, + reference=reference, + start=datetime.fromisoformat(start_ts), + end=datetime.fromisoformat(end_ts), + throw_exception=False, + cache_profile=not no_profile_cache, + ) + + if not result: + print(f"{Style.BRIGHT + Fore.GREEN}Validation successful!{Style.RESET_ALL}") + return + + errors = [e.to_dict() for e in result.report.errors] + formatted_json = json.dumps(errors, indent=4) + colorful_json = highlight( + formatted_json, lexers.JsonLexer(), formatters.TerminalFormatter() + ) + print(f"{Style.BRIGHT + Fore.RED}Validation failed!{Style.RESET_ALL}") + print(colorful_json) + exit(1) + + if __name__ == "__main__": cli() diff --git a/sdk/python/feast/diff/registry_diff.py b/sdk/python/feast/diff/registry_diff.py index 197bdfcefaf..33cd3df0edc 100644 --- a/sdk/python/feast/diff/registry_diff.py +++ b/sdk/python/feast/diff/registry_diff.py @@ -20,6 +20,9 @@ from feast.protos.feast.core.RequestFeatureView_pb2 import ( RequestFeatureView as RequestFeatureViewProto, ) +from feast.protos.feast.core.ValidationProfile_pb2 import ( + ValidationReference as ValidationReferenceProto, +) from feast.registry import FEAST_OBJECT_TYPES, FeastObjectType, Registry from feast.repo_contents import RepoContents @@ -103,6 +106,7 @@ def tag_objects_for_keep_delete_update_add( FeatureServiceProto, OnDemandFeatureViewProto, RequestFeatureViewProto, + ValidationReferenceProto, ) @@ -120,9 +124,9 @@ def diff_registry_objects( current_spec: FeastObjectSpecProto new_spec: FeastObjectSpecProto - if isinstance(current_proto, DataSourceProto) or isinstance( - new_proto, DataSourceProto - ): + if isinstance( + current_proto, (DataSourceProto, ValidationReferenceProto) + ) or isinstance(new_proto, (DataSourceProto, ValidationReferenceProto)): assert type(current_proto) == type(new_proto) current_spec = cast(DataSourceProto, current_proto) new_spec = cast(DataSourceProto, new_proto) diff --git a/sdk/python/feast/dqm/profilers/ge_profiler.py b/sdk/python/feast/dqm/profilers/ge_profiler.py index 93c8b7d5de8..81e2d81d8c3 100644 --- a/sdk/python/feast/dqm/profilers/ge_profiler.py +++ b/sdk/python/feast/dqm/profilers/ge_profiler.py @@ -1,4 +1,5 @@ import json +from types import FunctionType from typing import Any, Callable, Dict, List import dill @@ -140,9 +141,12 @@ def analyze_dataset(self, df: pd.DataFrame) -> Profile: return GEProfile(expectation_suite=self.user_defined_profiler(dataset)) def to_proto(self): + # keep only the code and drop context for now + # ToDo (pyalex): include some context, but not all (dill tries to pull too much) + udp = FunctionType(self.user_defined_profiler.__code__, {}) return GEValidationProfilerProto( profiler=GEValidationProfilerProto.UserDefinedProfiler( - body=dill.dumps(self.user_defined_profiler, recurse=True) + body=dill.dumps(udp, recurse=False) ) ) diff --git a/sdk/python/feast/dqm/profilers/profiler.py b/sdk/python/feast/dqm/profilers/profiler.py index 5d2e9d36bc1..e65bf8601f0 100644 --- a/sdk/python/feast/dqm/profilers/profiler.py +++ b/sdk/python/feast/dqm/profilers/profiler.py @@ -69,6 +69,7 @@ class ValidationError: missing_count: Optional[int] missing_percent: Optional[float] + observed_value: Optional[float] def __init__( self, @@ -77,12 +78,24 @@ def __init__( check_config: Optional[Any] = None, missing_count: Optional[int] = None, missing_percent: Optional[float] = None, + observed_value: Optional[float] = None, ): self.check_name = check_name self.column_name = column_name self.check_config = check_config self.missing_count = missing_count self.missing_percent = missing_percent + self.observed_value = observed_value def __repr__(self): return f"" + + def to_dict(self): + return dict( + check_name=self.check_name, + column_name=self.column_name, + check_config=self.check_config, + missing_count=self.missing_count, + missing_percent=self.missing_percent, + observed_value=self.observed_value, + ) diff --git a/sdk/python/feast/errors.py b/sdk/python/feast/errors.py index e680337d98c..07deb6401bd 100644 --- a/sdk/python/feast/errors.py +++ b/sdk/python/feast/errors.py @@ -94,6 +94,13 @@ def __init__(self, name: str, project: str): super().__init__(f"Saved dataset {name} does not exist in project {project}") +class ValidationReferenceNotFound(FeastObjectNotFoundException): + def __init__(self, name: str, project: str): + super().__init__( + f"Validation reference {name} does not exist in project {project}" + ) + + class FeastProviderLoginError(Exception): """Error class that indicates a user has not authenticated with their provider.""" diff --git a/sdk/python/feast/feast_object.py b/sdk/python/feast/feast_object.py index 4ffd693c44f..0ac0446f5f6 100644 --- a/sdk/python/feast/feast_object.py +++ b/sdk/python/feast/feast_object.py @@ -11,7 +11,11 @@ from .protos.feast.core.FeatureView_pb2 import FeatureViewSpec from .protos.feast.core.OnDemandFeatureView_pb2 import OnDemandFeatureViewSpec from .protos.feast.core.RequestFeatureView_pb2 import RequestFeatureViewSpec +from .protos.feast.core.ValidationProfile_pb2 import ( + ValidationReference as ValidationReferenceProto, +) from .request_feature_view import RequestFeatureView +from .saved_dataset import ValidationReference # Convenience type representing all Feast objects FeastObject = Union[ @@ -21,6 +25,7 @@ Entity, FeatureService, DataSource, + ValidationReference, ] FeastObjectSpecProto = Union[ @@ -30,4 +35,5 @@ EntitySpecV2, FeatureServiceSpec, DataSourceProto, + ValidationReferenceProto, ] diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py index edd0f5a46c7..140bd76a835 100644 --- a/sdk/python/feast/feature_store.py +++ b/sdk/python/feast/feature_store.py @@ -608,6 +608,7 @@ def apply( OnDemandFeatureView, RequestFeatureView, FeatureService, + ValidationReference, List[FeastObject], ], objects_to_delete: Optional[List[FeastObject]] = None, @@ -669,6 +670,9 @@ def apply( data_sources_set_to_update = { ob for ob in objects if isinstance(ob, DataSource) } + validation_references_to_update = [ + ob for ob in objects if isinstance(ob, ValidationReference) + ] for fv in views_to_update: data_sources_set_to_update.add(fv.batch_source) @@ -719,6 +723,10 @@ def apply( self._registry.apply_feature_service( feature_service, project=self.project, commit=False ) + for validation_references in validation_references_to_update: + self._registry.apply_validation_reference( + validation_references, project=self.project, commit=False + ) if not partial: # Delete all registry objects that should not exist. @@ -740,6 +748,9 @@ def apply( data_sources_to_delete = [ ob for ob in objects_to_delete if isinstance(ob, DataSource) ] + validation_references_to_delete = [ + ob for ob in objects_to_delete if isinstance(ob, ValidationReference) + ] for data_source in data_sources_to_delete: self._registry.delete_data_source( @@ -765,6 +776,10 @@ def apply( self._registry.delete_feature_service( service.name, project=self.project, commit=False ) + for validation_references in validation_references_to_delete: + self._registry.delete_validation_reference( + validation_references.name, project=self.project, commit=False + ) self._get_provider().update_infra( project=self.project, @@ -2039,6 +2054,7 @@ def serve_transformations(self, port: int) -> None: def _teardown_go_server(self): self._go_server = None + @log_exceptions_and_usage def write_logged_features( self, logs: Union[pa.Table, Path], source: Union[FeatureService] ): @@ -2066,6 +2082,7 @@ def write_logged_features( registry=self._registry, ) + @log_exceptions_and_usage def validate_logged_features( self, source: Union[FeatureService], @@ -2073,6 +2090,7 @@ def validate_logged_features( end: datetime, reference: ValidationReference, throw_exception: bool = True, + cache_profile: bool = True, ) -> Optional[ValidationFailed]: """ Load logged features from an offline store and validate them against provided validation reference. @@ -2083,6 +2101,7 @@ def validate_logged_features( end: upper bound for loading logged features reference: validation reference throw_exception: throw exception or return it as a result + cache_profile: store cached profile in Feast registry Returns: Throw or return (depends on parameter) ValidationFailed exception if validation was not successful @@ -2116,8 +2135,27 @@ def validate_logged_features( return exc + if cache_profile: + self.apply(reference) + return None + @log_exceptions_and_usage + def get_validation_reference( + self, name: str, allow_cache: bool = False + ) -> ValidationReference: + """ + Retrieves a validation reference. + + Raises: + ValidationReferenceNotFoundException: The validation reference could not be found. + """ + ref = self._registry.get_validation_reference( + name, project=self.project, allow_cache=allow_cache + ) + ref._dataset = self.get_saved_dataset(ref.dataset_name) + return ref + def _validate_entity_values(join_key_values: Dict[str, List[Value]]): set_of_row_lengths = {len(v) for v in join_key_values.values()} diff --git a/sdk/python/feast/infra/passthrough_provider.py b/sdk/python/feast/infra/passthrough_provider.py index f01fd9bac61..a53788dc85a 100644 --- a/sdk/python/feast/infra/passthrough_provider.py +++ b/sdk/python/feast/infra/passthrough_provider.py @@ -278,6 +278,6 @@ def retrieve_feature_service_logs( join_key_columns=[], feature_name_columns=columns, timestamp_field=ts_column, - start_date=start_date, - end_date=end_date, + start_date=make_tzaware(start_date), + end_date=make_tzaware(end_date), ) diff --git a/sdk/python/feast/registry.py b/sdk/python/feast/registry.py index be009566d05..c46eba8a5d6 100644 --- a/sdk/python/feast/registry.py +++ b/sdk/python/feast/registry.py @@ -38,6 +38,7 @@ FeatureViewNotFoundException, OnDemandFeatureViewNotFoundException, SavedDatasetNotFound, + ValidationReferenceNotFound, ) from feast.feature_service import FeatureService from feast.feature_view import FeatureView @@ -49,7 +50,7 @@ from feast.repo_config import RegistryConfig from feast.repo_contents import RepoContents from feast.request_feature_view import RequestFeatureView -from feast.saved_dataset import SavedDataset +from feast.saved_dataset import SavedDataset, ValidationReference REGISTRY_SCHEMA_VERSION = "1" @@ -795,7 +796,7 @@ def apply_saved_dataset( self, saved_dataset: SavedDataset, project: str, commit: bool = True, ): """ - Registers a single entity with Feast + Stores a saved dataset metadata with Feast Args: saved_dataset: SavedDataset that will be added / updated to registry @@ -870,6 +871,85 @@ def list_saved_datasets( if saved_dataset.spec.project == project ] + def apply_validation_reference( + self, + validation_reference: ValidationReference, + project: str, + commit: bool = True, + ): + """ + Persist a validation reference + + Args: + validation_reference: ValidationReference that will be added / updated to registry + project: Feast project that this dataset belongs to + commit: Whether the change should be persisted immediately + """ + validation_reference_proto = validation_reference.to_proto() + validation_reference_proto.project = project + + registry_proto = self._prepare_registry_for_changes() + for idx, existing_validation_reference in enumerate( + registry_proto.validation_references + ): + if ( + existing_validation_reference.name == validation_reference_proto.name + and existing_validation_reference.project == project + ): + del registry_proto.validation_references[idx] + break + + registry_proto.validation_references.append(validation_reference_proto) + if commit: + self.commit() + + def get_validation_reference( + self, name: str, project: str, allow_cache: bool = False + ) -> ValidationReference: + """ + Retrieves a validation reference. + + Args: + name: Name of dataset + project: Feast project that this dataset belongs to + allow_cache: Whether to allow returning this dataset from a cached registry + + Returns: + Returns either the specified ValidationReference, or raises an exception if + none is found + """ + registry_proto = self._get_registry_proto(allow_cache=allow_cache) + for validation_reference in registry_proto.validation_references: + if ( + validation_reference.name == name + and validation_reference.project == project + ): + return ValidationReference.from_proto(validation_reference) + raise ValidationReferenceNotFound(name, project=project) + + def delete_validation_reference(self, name: str, project: str, commit: bool = True): + """ + Deletes a validation reference or raises an exception if not found. + + Args: + name: Name of validation reference + project: Feast project that this object belongs to + commit: Whether the change should be persisted immediately + """ + registry_proto = self._prepare_registry_for_changes() + for idx, existing_validation_reference in enumerate( + registry_proto.validation_references + ): + if ( + existing_validation_reference.name == name + and existing_validation_reference.project == project + ): + del registry_proto.validation_references[idx] + if commit: + self.commit() + return + raise ValidationReferenceNotFound(name, project=project) + def commit(self): """Commits the state of the registry cache to the remote registry store.""" if self.cached_registry_proto: diff --git a/sdk/python/feast/saved_dataset.py b/sdk/python/feast/saved_dataset.py index aead7fe8eff..e2004d15f4c 100644 --- a/sdk/python/feast/saved_dataset.py +++ b/sdk/python/feast/saved_dataset.py @@ -13,6 +13,9 @@ from feast.protos.feast.core.SavedDataset_pb2 import ( SavedDatasetStorage as SavedDatasetStorageProto, ) +from feast.protos.feast.core.ValidationProfile_pb2 import ( + ValidationReference as ValidationReferenceProto, +) if TYPE_CHECKING: from feast.infra.offline_stores.offline_store import RetrievalJob @@ -178,8 +181,8 @@ def to_proto(self) -> SavedDatasetProto: if self.feature_service_name: spec.feature_service_name = self.feature_service_name - feature_service_proto = SavedDatasetProto(spec=spec, meta=meta) - return feature_service_proto + saved_dataset_proto = SavedDatasetProto(spec=spec, meta=meta) + return saved_dataset_proto def with_retrieval_job(self, retrieval_job: "RetrievalJob") -> "SavedDataset": self._retrieval_job = retrieval_job @@ -203,21 +206,123 @@ def to_arrow(self) -> pyarrow.Table: return self._retrieval_job.to_arrow() - def as_reference(self, profiler: "Profiler") -> "ValidationReference": - return ValidationReference(profiler=profiler, dataset=self) + def as_reference(self, name: str, profiler: "Profiler") -> "ValidationReference": + return ValidationReference.from_saved_dataset( + name=name, profiler=profiler, dataset=self + ) def get_profile(self, profiler: Profiler) -> Profile: return profiler.analyze_dataset(self.to_df()) class ValidationReference: - dataset: SavedDataset + name: str + dataset_name: str + description: str + tags: Dict[str, str] profiler: Profiler - def __init__(self, dataset: SavedDataset, profiler: Profiler): - self.dataset = dataset + _profile: Optional[Profile] = None + _dataset: Optional[SavedDataset] = None + + def __init__( + self, + name: str, + dataset_name: str, + profiler: Profiler, + description: str = "", + tags: Optional[Dict[str, str]] = None, + ): + """ + Validation reference combines a reference dataset (currently only a saved dataset object can be used as + a reference) and a profiler function to generate a validation profile. + The validation profile can be cached in this object, and in this case + the saved dataset retrieval and the profiler call will happen only once. + + Validation reference is being stored in the Feast registry and can be retrieved by its name, which + must be unique within one project. + + Args: + name: the unique name for validation reference + dataset_name: the name of the saved dataset used as a reference + description: a human-readable description + tags: a dictionary of key-value pairs to store arbitrary metadata + profiler: the profiler function used to generate profile from the saved dataset + """ + self.name = name + self.dataset_name = dataset_name self.profiler = profiler + self.description = description + self.tags = tags or {} + + @classmethod + def from_saved_dataset(cls, name: str, dataset: SavedDataset, profiler: Profiler): + """ + Internal constructor to create validation reference object with actual saved dataset object + (regular constructor requires only its name). + """ + ref = ValidationReference(name, dataset.name, profiler) + ref._dataset = dataset + return ref @property def profile(self) -> Profile: - return self.profiler.analyze_dataset(self.dataset.to_df()) + if not self._profile: + if not self._dataset: + raise RuntimeError( + "In order to calculate a profile validation reference must be instantiated from a saved dataset. " + "Use ValidationReference.from_saved_dataset constructor or FeatureStore.get_validation_reference " + "to get validation reference object." + ) + + self._profile = self.profiler.analyze_dataset(self._dataset.to_df()) + return self._profile + + @classmethod + def from_proto(cls, proto: ValidationReferenceProto) -> "ValidationReference": + profiler_attr = proto.WhichOneof("profiler") + if profiler_attr == "ge_profiler": + from feast.dqm.profilers.ge_profiler import GEProfiler + + profiler = GEProfiler.from_proto(proto.ge_profiler) + else: + raise RuntimeError("Unrecognized profiler") + + profile_attr = proto.WhichOneof("cached_profile") + if profile_attr == "ge_profile": + from feast.dqm.profilers.ge_profiler import GEProfile + + profile = GEProfile.from_proto(proto.ge_profile) + elif not profile_attr: + profile = None + else: + raise RuntimeError("Unrecognized profile") + + ref = ValidationReference( + name=proto.name, + dataset_name=proto.reference_dataset_name, + profiler=profiler, + description=proto.description, + tags=dict(proto.tags), + ) + ref._profile = profile + + return ref + + def to_proto(self) -> ValidationReferenceProto: + from feast.dqm.profilers.ge_profiler import GEProfile, GEProfiler + + proto = ValidationReferenceProto( + name=self.name, + reference_dataset_name=self.dataset_name, + tags=self.tags, + description=self.description, + ge_profiler=self.profiler.to_proto() + if isinstance(self.profiler, GEProfiler) + else None, + ge_profile=self._profile.to_proto() + if isinstance(self._profile, GEProfile) + else None, + ) + + return proto diff --git a/sdk/python/requirements/py3.10-ci-requirements.txt b/sdk/python/requirements/py3.10-ci-requirements.txt index 120f0d1158e..e4b7e5447be 100644 --- a/sdk/python/requirements/py3.10-ci-requirements.txt +++ b/sdk/python/requirements/py3.10-ci-requirements.txt @@ -491,6 +491,7 @@ pyflakes==2.4.0 # via flake8 pygments==2.12.0 # via + # feast (setup.py) # ipython # sphinx pyjwt[crypto]==2.4.0 diff --git a/sdk/python/requirements/py3.10-requirements.txt b/sdk/python/requirements/py3.10-requirements.txt index 725b17f8caf..717982f012c 100644 --- a/sdk/python/requirements/py3.10-requirements.txt +++ b/sdk/python/requirements/py3.10-requirements.txt @@ -111,6 +111,8 @@ pydantic==1.9.0 # via # fastapi # feast (setup.py) +pygments==2.12.0 + # via feast (setup.py) pyparsing==3.0.9 # via packaging pyrsistent==0.18.1 diff --git a/sdk/python/requirements/py3.7-ci-requirements.txt b/sdk/python/requirements/py3.7-ci-requirements.txt index b445f86ea06..1d8c31808d4 100644 --- a/sdk/python/requirements/py3.7-ci-requirements.txt +++ b/sdk/python/requirements/py3.7-ci-requirements.txt @@ -505,6 +505,7 @@ pyflakes==2.4.0 # via flake8 pygments==2.12.0 # via + # feast (setup.py) # ipython # sphinx pyjwt[crypto]==2.4.0 diff --git a/sdk/python/requirements/py3.7-requirements.txt b/sdk/python/requirements/py3.7-requirements.txt index b0e1511d9c6..c175fff7893 100644 --- a/sdk/python/requirements/py3.7-requirements.txt +++ b/sdk/python/requirements/py3.7-requirements.txt @@ -117,6 +117,8 @@ pydantic==1.9.0 # via # fastapi # feast (setup.py) +pygments==2.12.0 + # via feast (setup.py) pyparsing==3.0.9 # via packaging pyrsistent==0.18.1 diff --git a/sdk/python/requirements/py3.8-ci-requirements.txt b/sdk/python/requirements/py3.8-ci-requirements.txt index 202f3ac2c71..af34dbbc2ff 100644 --- a/sdk/python/requirements/py3.8-ci-requirements.txt +++ b/sdk/python/requirements/py3.8-ci-requirements.txt @@ -497,6 +497,7 @@ pyflakes==2.4.0 # via flake8 pygments==2.12.0 # via + # feast (setup.py) # ipython # sphinx pyjwt[crypto]==2.4.0 diff --git a/sdk/python/requirements/py3.8-requirements.txt b/sdk/python/requirements/py3.8-requirements.txt index 98e1c6a76a9..eff4bae2689 100644 --- a/sdk/python/requirements/py3.8-requirements.txt +++ b/sdk/python/requirements/py3.8-requirements.txt @@ -113,6 +113,8 @@ pydantic==1.9.0 # via # fastapi # feast (setup.py) +pygments==2.12.0 + # via feast (setup.py) pyparsing==3.0.9 # via packaging pyrsistent==0.18.1 diff --git a/sdk/python/requirements/py3.9-ci-requirements.txt b/sdk/python/requirements/py3.9-ci-requirements.txt index d3ecdc34bf3..4147a391dce 100644 --- a/sdk/python/requirements/py3.9-ci-requirements.txt +++ b/sdk/python/requirements/py3.9-ci-requirements.txt @@ -491,6 +491,7 @@ pyflakes==2.4.0 # via flake8 pygments==2.12.0 # via + # feast (setup.py) # ipython # sphinx pyjwt[crypto]==2.4.0 diff --git a/sdk/python/requirements/py3.9-requirements.txt b/sdk/python/requirements/py3.9-requirements.txt index 3eded689a58..80199a1f5ba 100644 --- a/sdk/python/requirements/py3.9-requirements.txt +++ b/sdk/python/requirements/py3.9-requirements.txt @@ -111,6 +111,8 @@ pydantic==1.9.0 # via # fastapi # feast (setup.py) +pygments==2.12.0 + # via feast (setup.py) pyparsing==3.0.9 # via packaging pyrsistent==0.18.1 diff --git a/sdk/python/setup.cfg b/sdk/python/setup.cfg index e2d707e2720..ebb933f69de 100644 --- a/sdk/python/setup.cfg +++ b/sdk/python/setup.cfg @@ -10,7 +10,7 @@ known_first_party=feast,feast_serving_server,feast_core_server default_section=THIRDPARTY [flake8] -ignore = E203, E266, E501, W503 +ignore = E203, E266, E501, W503, C901 max-line-length = 88 max-complexity = 20 select = B,C,E,F,W,T4 diff --git a/sdk/python/tests/conftest.py b/sdk/python/tests/conftest.py index 627fda524d9..671acb3b92a 100644 --- a/sdk/python/tests/conftest.py +++ b/sdk/python/tests/conftest.py @@ -57,6 +57,14 @@ def pytest_configure(config): config.addinivalue_line( "markers", "goserver: mark tests that use the go feature server" ) + config.addinivalue_line( + "markers", + "universal_online_stores: mark tests that can be run against different online stores", + ) + config.addinivalue_line( + "markers", + "universal_offline_stores: mark tests that can be run against different offline stores", + ) def pytest_addoption(parser): diff --git a/sdk/python/tests/integration/e2e/test_go_feature_server.py b/sdk/python/tests/integration/e2e/test_go_feature_server.py index c87d79564fd..11526132ac8 100644 --- a/sdk/python/tests/integration/e2e/test_go_feature_server.py +++ b/sdk/python/tests/integration/e2e/test_go_feature_server.py @@ -103,8 +103,9 @@ def server_port(environment, server_type: str): embedded.stop_grpc_server() else: embedded.stop_http_server() + # wait for graceful stop - time.sleep(2) + time.sleep(5) @pytest.fixture diff --git a/sdk/python/tests/integration/e2e/test_validation.py b/sdk/python/tests/integration/e2e/test_validation.py index b78a8bde093..338dc77d236 100644 --- a/sdk/python/tests/integration/e2e/test_validation.py +++ b/sdk/python/tests/integration/e2e/test_validation.py @@ -1,4 +1,5 @@ import datetime +import shutil import pandas as pd import pyarrow as pa @@ -15,6 +16,7 @@ LoggingConfig, ) from feast.protos.feast.serving.ServingService_pb2 import FieldStatus +from feast.utils import make_tzaware from feast.wait import wait_retry_backoff from tests.integration.feature_repos.repo_configuration import ( construct_universal_feature_views, @@ -24,6 +26,7 @@ driver, location, ) +from tests.utils.cli_utils import CliRunner from tests.utils.logged_features import prepare_logs _features = [ @@ -128,7 +131,7 @@ def test_historical_retrieval_with_validation(environment, universal_data_source saved_dataset = store.get_saved_dataset("my_training_dataset") # If validation pass there will be no exceptions on this point - reference = saved_dataset.as_reference(profiler=configurable_profiler) + reference = saved_dataset.as_reference(name="ref", profiler=configurable_profiler) job.to_df(validation_reference=reference) @@ -161,7 +164,7 @@ def test_historical_retrieval_fails_on_validation(environment, universal_data_so job.to_df( validation_reference=store.get_saved_dataset( "my_other_dataset" - ).as_reference(profiler=profiler_with_unrealistic_expectations) + ).as_reference(name="ref", profiler=profiler_with_unrealistic_expectations) ) failed_expectations = exc_info.value.report.errors @@ -175,6 +178,7 @@ def test_historical_retrieval_fails_on_validation(environment, universal_data_so @pytest.mark.integration +@pytest.mark.universal_offline_stores def test_logged_features_validation(environment, universal_data_sources): store = environment.feature_store @@ -244,7 +248,7 @@ def validate(): start=logs_df[LOG_TIMESTAMP_FIELD].min(), end=logs_df[LOG_TIMESTAMP_FIELD].max() + datetime.timedelta(seconds=1), reference=reference_dataset.as_reference( - profiler=profiler_with_feature_metadata + name="ref", profiler=profiler_with_feature_metadata ), ) except ValidationFailed: @@ -257,3 +261,88 @@ def validate(): success = wait_retry_backoff(validate, timeout_secs=30) assert success, "Validation failed (unexpectedly)" + + +@pytest.mark.integration +def test_e2e_validation_via_cli(environment, universal_data_sources): + runner = CliRunner() + store = environment.feature_store + + (_, datasets, data_sources) = universal_data_sources + feature_views = construct_universal_feature_views(data_sources) + feature_service = FeatureService( + name="test_service", + features=[ + feature_views.customer[ + ["current_balance", "avg_passenger_count", "lifetime_trip_count"] + ], + ], + logging_config=LoggingConfig( + destination=environment.data_source_creator.create_logged_features_destination() + ), + ) + store.apply([customer(), feature_service, feature_views.customer]) + + entity_df = datasets.entity_df.drop( + columns=["order_id", "origin_id", "destination_id", "driver_id"] + ) + retrieval_job = store.get_historical_features( + entity_df=entity_df, features=feature_service, full_feature_names=True + ) + logs_df = prepare_logs(retrieval_job.to_df(), feature_service, store) + saved_dataset = store.create_saved_dataset( + from_=retrieval_job, + name="reference_for_validating_logged_features", + storage=environment.data_source_creator.create_saved_dataset_destination(), + ) + reference = saved_dataset.as_reference( + name="test_reference", profiler=configurable_profiler + ) + + schema = FeatureServiceLoggingSource( + feature_service=feature_service, project=store.project + ).get_schema(store._registry) + store.write_logged_features( + pa.Table.from_pandas(logs_df, schema=schema), source=feature_service + ) + + with runner.local_repo(example_repo_py="", offline_store="file") as local_repo: + local_repo.apply( + [customer(), feature_views.customer, feature_service, reference] + ) + local_repo._registry.apply_saved_dataset(saved_dataset, local_repo.project) + validate_args = [ + "validate", + "--feature-service", + feature_service.name, + "--reference", + reference.name, + (datetime.datetime.utcnow() - datetime.timedelta(days=7)).isoformat(), + datetime.datetime.utcnow().isoformat(), + ] + p = runner.run(validate_args, cwd=local_repo.repo_path) + + assert p.returncode == 0, p.stderr.decode() + assert "Validation successful" in p.stdout.decode(), p.stderr.decode() + + # make sure second validation will use cached profile + shutil.rmtree(saved_dataset.storage.file_options.uri) + + # Add some invalid data that would lead to failed validation + invalid_data = pd.DataFrame( + data={ + "customer_id": [0], + "current_balance": [0], + "avg_passenger_count": [0], + "lifetime_trip_count": [0], + "event_timestamp": [make_tzaware(datetime.datetime.utcnow())], + } + ) + invalid_logs = prepare_logs(invalid_data, feature_service, store) + store.write_logged_features( + pa.Table.from_pandas(invalid_logs, schema=schema), source=feature_service + ) + + p = runner.run(validate_args, cwd=local_repo.repo_path) + assert p.returncode == 1, p.stdout.decode() + assert "Validation failed" in p.stdout.decode(), p.stderr.decode() diff --git a/sdk/python/tests/integration/online_store/test_universal_online.py b/sdk/python/tests/integration/online_store/test_universal_online.py index 4afcd61c70b..d05045e2953 100644 --- a/sdk/python/tests/integration/online_store/test_universal_online.py +++ b/sdk/python/tests/integration/online_store/test_universal_online.py @@ -325,7 +325,6 @@ def get_online_features_dict( @pytest.mark.integration -@pytest.mark.universal def test_online_retrieval_with_shared_batch_source(environment, universal_data_sources): # Addresses https://github.com/feast-dev/feast/issues/2576 diff --git a/sdk/python/tests/utils/cli_utils.py b/sdk/python/tests/utils/cli_utils.py index 5d6d5722eb1..f2478a4a5ee 100644 --- a/sdk/python/tests/utils/cli_utils.py +++ b/sdk/python/tests/utils/cli_utils.py @@ -33,7 +33,9 @@ class CliRunner: """ def run(self, args: List[str], cwd: Path) -> subprocess.CompletedProcess: - return subprocess.run([sys.executable, cli.__file__] + args, cwd=cwd) + return subprocess.run( + [sys.executable, cli.__file__] + args, cwd=cwd, capture_output=True + ) def run_with_output(self, args: List[str], cwd: Path) -> Tuple[int, bytes]: try: diff --git a/sdk/python/tests/utils/logged_features.py b/sdk/python/tests/utils/logged_features.py index 155f0b27b12..dc844a60b42 100644 --- a/sdk/python/tests/utils/logged_features.py +++ b/sdk/python/tests/utils/logged_features.py @@ -9,7 +9,7 @@ import pandas as pd import pyarrow -from feast import FeatureService, FeatureStore +from feast import FeatureService, FeatureStore, FeatureView from feast.errors import FeatureViewNotFoundException from feast.feature_logging import LOG_DATE_FIELD, LOG_TIMESTAMP_FIELD, REQUEST_ID_FIELD from feast.protos.feast.serving.ServingService_pb2 import FieldStatus @@ -28,17 +28,6 @@ def prepare_logs( logs_df[LOG_DATE_FIELD] = logs_df[LOG_TIMESTAMP_FIELD].dt.date for projection in feature_service.feature_view_projections: - for feature in projection.features: - logs_df[f"{projection.name_to_use()}__{feature.name}"] = source_df[ - feature.name - ] - logs_df[ - f"{projection.name_to_use()}__{feature.name}__timestamp" - ] = source_df["event_timestamp"].dt.floor("s") - logs_df[ - f"{projection.name_to_use()}__{feature.name}__status" - ] = FieldStatus.PRESENT - try: view = store.get_feature_view(projection.name) except FeatureViewNotFoundException: @@ -51,6 +40,31 @@ def prepare_logs( entity = store.get_entity(entity_name) logs_df[entity.join_key] = source_df[entity.join_key] + for feature in projection.features: + source_field = ( + feature.name + if feature.name in source_df.columns + else f"{projection.name_to_use()}__{feature.name}" + ) + destination_field = f"{projection.name_to_use()}__{feature.name}" + logs_df[destination_field] = source_df[source_field] + logs_df[f"{destination_field}__timestamp"] = source_df[ + "event_timestamp" + ].dt.floor("s") + if logs_df[f"{destination_field}__timestamp"].dt.tz: + logs_df[f"{destination_field}__timestamp"] = logs_df[ + f"{destination_field}__timestamp" + ].dt.tz_convert(None) + logs_df[f"{destination_field}__status"] = FieldStatus.PRESENT + if isinstance(view, FeatureView) and view.ttl: + logs_df[f"{destination_field}__status"] = logs_df[ + f"{destination_field}__status" + ].mask( + logs_df[f"{destination_field}__timestamp"] + < (datetime.datetime.utcnow() - view.ttl), + FieldStatus.OUTSIDE_MAX_AGE, + ) + return logs_df diff --git a/setup.py b/setup.py index f5f092e3f50..e0eaf4f5262 100644 --- a/setup.py +++ b/setup.py @@ -64,6 +64,7 @@ "proto-plus<1.19.7", "pyarrow>=4,<7", "pydantic>=1,<2", + "pygments==2.12.0", "PyYAML>=5.4.*,<7", "tabulate==0.8.*", "tenacity>=7,<9", From 44d53fda71b5a82d9fb6e044b01d97080c2d018c Mon Sep 17 00:00:00 2001 From: Felix Wang Date: Mon, 23 May 2022 08:57:04 -0700 Subject: [PATCH 024/123] chore: Add request source and ODFV to version 0.19 test repo (#2691) * Add request source and ODFV to version 0.19 test repo Signed-off-by: Felix Wang * Format Signed-off-by: Felix Wang * Enable ODFVs for unit tests Signed-off-by: Felix Wang * Switch to RequestDataSource Signed-off-by: Felix Wang --- .github/workflows/unit_tests.yml | 1 + sdk/python/feast/data_source.py | 16 +++---- sdk/python/feast/on_demand_feature_view.py | 2 +- .../example_feature_repo_version_0_19.py | 28 ++++++++++++ .../online_store/test_e2e_local.py | 45 ++++++++++++++++++- 5 files changed, 80 insertions(+), 12 deletions(-) diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml index 3573fe26568..77198080537 100644 --- a/.github/workflows/unit_tests.yml +++ b/.github/workflows/unit_tests.yml @@ -55,6 +55,7 @@ jobs: run: make install-python-ci-dependencies - name: Test Python env: + IS_TEST: "True" SNOWFLAKE_CI_DEPLOYMENT: ${{ secrets.SNOWFLAKE_CI_DEPLOYMENT }} SNOWFLAKE_CI_USER: ${{ secrets.SNOWFLAKE_CI_USER }} SNOWFLAKE_CI_PASSWORD: ${{ secrets.SNOWFLAKE_CI_PASSWORD }} diff --git a/sdk/python/feast/data_source.py b/sdk/python/feast/data_source.py index 6a2b9a0d148..4392314bb8d 100644 --- a/sdk/python/feast/data_source.py +++ b/sdk/python/feast/data_source.py @@ -24,7 +24,7 @@ from feast.field import Field from feast.protos.feast.core.DataSource_pb2 import DataSource as DataSourceProto from feast.repo_config import RepoConfig, get_data_source_class_from_type -from feast.types import VALUE_TYPES_TO_FEAST_TYPES +from feast.types import from_value_type from feast.value_type import ValueType @@ -557,12 +557,10 @@ def __init__( "Please use List[Field] instead for the schema", DeprecationWarning, ) - schemaList = [] - for key, valueType in _schema.items(): - schemaList.append( - Field(name=key, dtype=VALUE_TYPES_TO_FEAST_TYPES[valueType]) - ) - self.schema = schemaList + schema_list = [] + for key, value_type in _schema.items(): + schema_list.append(Field(name=key, dtype=from_value_type(value_type))) + self.schema = schema_list elif isinstance(_schema, List): self.schema = _schema else: @@ -641,9 +639,7 @@ def to_proto(self) -> DataSourceProto: if isinstance(self.schema, Dict): for key, value in self.schema.items(): schema_pb.append( - Field( - name=key, dtype=VALUE_TYPES_TO_FEAST_TYPES[value.value] - ).to_proto() + Field(name=key, dtype=from_value_type(value.value)).to_proto() ) else: for field in self.schema: diff --git a/sdk/python/feast/on_demand_feature_view.py b/sdk/python/feast/on_demand_feature_view.py index 6d1eb455cac..26dc1d23514 100644 --- a/sdk/python/feast/on_demand_feature_view.py +++ b/sdk/python/feast/on_demand_feature_view.py @@ -674,7 +674,7 @@ def feature_view_to_batch_feature_view(fv: FeatureView) -> BatchFeatureView: online=fv.online, owner=fv.owner, schema=fv.schema, - source=fv.source, + source=fv.batch_source, ) bfv.features = copy.copy(fv.features) diff --git a/sdk/python/tests/example_repos/example_feature_repo_version_0_19.py b/sdk/python/tests/example_repos/example_feature_repo_version_0_19.py index e00a69b867a..a65c031cea4 100644 --- a/sdk/python/tests/example_repos/example_feature_repo_version_0_19.py +++ b/sdk/python/tests/example_repos/example_feature_repo_version_0_19.py @@ -1,6 +1,10 @@ from datetime import timedelta +import pandas as pd + from feast import Entity, Feature, FeatureView, FileSource, ValueType +from feast.data_source import RequestDataSource +from feast.on_demand_feature_view import on_demand_feature_view driver_hourly_stats = FileSource( path="%PARQUET_PATH%", # placeholder to be replaced by the test @@ -50,3 +54,27 @@ batch_source=global_daily_stats, # Changed to `source` in 0.20 tags={}, ) + + +request_source = RequestDataSource( + name="conv_rate_input", schema={"val_to_add": ValueType.INT64}, +) + + +@on_demand_feature_view( + inputs={ + "conv_rate_input": request_source, + "driver_hourly_stats": driver_hourly_stats_view, + }, + features=[ + Feature(name="conv_rate_plus_100", dtype=ValueType.DOUBLE), + Feature(name="conv_rate_plus_val_to_add", dtype=ValueType.DOUBLE), + ], +) +def conv_rate_plus_100(features_df: pd.DataFrame) -> pd.DataFrame: + df = pd.DataFrame() + df["conv_rate_plus_100"] = features_df["conv_rate"] + 100 + df["conv_rate_plus_val_to_add"] = ( + features_df["conv_rate"] + features_df["val_to_add"] + ) + return df diff --git a/sdk/python/tests/integration/online_store/test_e2e_local.py b/sdk/python/tests/integration/online_store/test_e2e_local.py index d3cb86716a2..c938bc55128 100644 --- a/sdk/python/tests/integration/online_store/test_e2e_local.py +++ b/sdk/python/tests/integration/online_store/test_e2e_local.py @@ -41,7 +41,7 @@ def _assert_online_features( full_feature_names=True, ) - # Float features should still be floats from the online store... + # Float features should still be floats. assert ( response.proto.results[ list(response.proto.metadata.feature_names.val).index( @@ -67,6 +67,49 @@ def _assert_online_features( assert "global_daily_stats__num_rides" in result assert "global_daily_stats__avg_ride_length" in result + # Test the ODFV if it exists. + odfvs = store.list_on_demand_feature_views() + if odfvs and odfvs[0].name == "conv_rate_plus_100": + response = store.get_online_features( + features=[ + "conv_rate_plus_100:conv_rate_plus_100", + "conv_rate_plus_100:conv_rate_plus_val_to_add", + ], + entity_rows=[{"driver_id": 1001, "val_to_add": 100}], + full_feature_names=True, + ) + + # Check that float64 feature is stored correctly in proto format. + assert ( + response.proto.results[ + list(response.proto.metadata.feature_names.val).index( + "conv_rate_plus_100__conv_rate_plus_100" + ) + ] + .values[0] + .double_val + > 0 + ) + + result = response.to_dict() + assert len(result) == 3 + assert "conv_rate_plus_100__conv_rate_plus_100" in result + assert "conv_rate_plus_100__conv_rate_plus_val_to_add" in result + assert ( + abs( + result["conv_rate_plus_100__conv_rate_plus_100"][0] + - (_get_last_feature_row(driver_df, 1001, max_date)["conv_rate"] + 100) + ) + < 0.01 + ) + assert ( + abs( + result["conv_rate_plus_100__conv_rate_plus_val_to_add"][0] + - (_get_last_feature_row(driver_df, 1001, max_date)["conv_rate"] + 100) + ) + < 0.01 + ) + def _test_materialize_and_online_retrieval( runner: CliRunner, From 67d2992afc036d2a674e9a20a4d4f2124a0a7bd8 Mon Sep 17 00:00:00 2001 From: Oleksii Moskalenko Date: Wed, 25 May 2022 11:03:03 -0700 Subject: [PATCH 025/123] chore: Minimize impact of enabled feature logging on latency (#2729) * improve latency when feature logging is enabled Signed-off-by: Oleksii Moskalenko * revert flush interval Signed-off-by: Oleksii Moskalenko * some clean up Signed-off-by: Oleksii Moskalenko --- .../feast/server/logging/filelogsink.go | 4 +- go/internal/feast/server/logging/logger.go | 11 +-- .../feast/server/logging/logger_test.go | 4 +- .../feast/server/logging/memorybuffer.go | 74 +++++++++++----- .../feast/server/logging/memorybuffer_test.go | 85 +++++++++---------- .../feast/server/logging/offlinestoresink.go | 31 +++---- go/internal/feast/server/logging/service.go | 2 +- sdk/python/feast/cli.py | 12 ++- sdk/python/feast/feature_store.py | 17 +++- sdk/python/feast/infra/offline_stores/file.py | 15 +++- 10 files changed, 149 insertions(+), 106 deletions(-) diff --git a/go/internal/feast/server/logging/filelogsink.go b/go/internal/feast/server/logging/filelogsink.go index c9f2049a041..d9796d69d10 100644 --- a/go/internal/feast/server/logging/filelogsink.go +++ b/go/internal/feast/server/logging/filelogsink.go @@ -34,7 +34,7 @@ func NewFileLogSink(path string) (*FileLogSink, error) { return &FileLogSink{path: absPath}, nil } -func (s *FileLogSink) Write(record arrow.Record) error { +func (s *FileLogSink) Write(records []arrow.Record) error { fileName, _ := uuid.NewUUID() var writer io.Writer @@ -42,7 +42,7 @@ func (s *FileLogSink) Write(record arrow.Record) error { if err != nil { return err } - table := array.NewTableFromRecords(record.Schema(), []arrow.Record{record}) + table := array.NewTableFromRecords(records[0].Schema(), records) props := parquet.NewWriterProperties(parquet.WithDictionaryDefault(false)) arrProps := pqarrow.DefaultWriterProps() diff --git a/go/internal/feast/server/logging/logger.go b/go/internal/feast/server/logging/logger.go index cbf1c3439a6..0e4f230f5ad 100644 --- a/go/internal/feast/server/logging/logger.go +++ b/go/internal/feast/server/logging/logger.go @@ -33,7 +33,7 @@ type LogSink interface { // Write is used to unload logs from memory buffer. // Logs are not guaranteed to be flushed to sink on this point. // The data can just be written to local disk (depending on implementation). - Write(data arrow.Record) error + Write(data []arrow.Record) error // Flush actually send data to a sink. // We want to control amount to interaction with sink, since it could be a costly operation. @@ -75,6 +75,10 @@ func NewLoggerConfig(sampleRate float32, opts LoggingOptions) LoggerConfig { } func NewLogger(schema *FeatureServiceSchema, featureServiceName string, sink LogSink, config LoggerConfig) (*LoggerImpl, error) { + buffer, err := NewMemoryBuffer(schema) + if err != nil { + return nil, err + } logger := &LoggerImpl{ featureServiceName: featureServiceName, @@ -82,10 +86,7 @@ func NewLogger(schema *FeatureServiceSchema, featureServiceName string, sink Log signalCh: make(chan interface{}, 2), sink: sink, - buffer: &MemoryBuffer{ - logs: make([]*Log, 0), - schema: schema, - }, + buffer: buffer, schema: schema, config: config, diff --git a/go/internal/feast/server/logging/logger_test.go b/go/internal/feast/server/logging/logger_test.go index 4ce883c75bd..ddc1902b7d1 100644 --- a/go/internal/feast/server/logging/logger_test.go +++ b/go/internal/feast/server/logging/logger_test.go @@ -24,7 +24,7 @@ import ( type DummySink struct{} -func (s *DummySink) Write(rec arrow.Record) error { +func (s *DummySink) Write(recs []arrow.Record) error { return nil } @@ -42,7 +42,7 @@ func TestLoggingChannelTimeout(t *testing.T) { WriteInterval: DefaultOptions.WriteInterval, }, } - logger, err := NewLogger(nil, "testFS", &DummySink{}, config) + logger, err := NewLogger(&FeatureServiceSchema{}, "testFS", &DummySink{}, config) // stop log processing to check buffering channel logger.Stop() diff --git a/go/internal/feast/server/logging/memorybuffer.go b/go/internal/feast/server/logging/memorybuffer.go index 36eb7118cb4..9ffb0ff73b8 100644 --- a/go/internal/feast/server/logging/memorybuffer.go +++ b/go/internal/feast/server/logging/memorybuffer.go @@ -2,7 +2,6 @@ package logging import ( "fmt" - "github.com/apache/arrow/go/v8/arrow" "github.com/apache/arrow/go/v8/arrow/array" "github.com/apache/arrow/go/v8/arrow/memory" @@ -14,45 +13,79 @@ import ( type MemoryBuffer struct { logs []*Log schema *FeatureServiceSchema + + arrowSchema *arrow.Schema + records []arrow.Record } const ( LOG_TIMESTAMP_FIELD = "__log_timestamp" LOG_DATE_FIELD = "__log_date" LOG_REQUEST_ID_FIELD = "__request_id" + RECORD_SIZE = 1000 ) +func NewMemoryBuffer(schema *FeatureServiceSchema) (*MemoryBuffer, error) { + arrowSchema, err := getArrowSchema(schema) + if err != nil { + return nil, err + } + return &MemoryBuffer{ + logs: make([]*Log, 0), + records: make([]arrow.Record, 0), + schema: schema, + arrowSchema: arrowSchema, + }, nil +} + // Acquires the logging schema from the feature service, converts the memory buffer array of rows of logs and flushes // them to the offline storage. func (b *MemoryBuffer) writeBatch(sink LogSink) error { - if len(b.logs) == 0 { - return nil + if len(b.logs) > 0 { + err := b.Compact() + if err != nil { + return err + } } - record, err := b.convertToArrowRecord() - - if err != nil { - return err + if len(b.records) == 0 { + return nil } - err = sink.Write(record) + + err := sink.Write(b.records) if err != nil { return err } - b.logs = b.logs[:0] + b.records = b.records[:0] return nil } func (b *MemoryBuffer) Append(log *Log) error { b.logs = append(b.logs, log) + + if len(b.logs) == RECORD_SIZE { + return b.Compact() + } + return nil } -func (b *MemoryBuffer) getArrowSchema() (*arrow.Schema, error) { +func (b *MemoryBuffer) Compact() error { + rec, err := b.convertToArrowRecord() + if err != nil { + return err + } + b.records = append(b.records, rec) + b.logs = b.logs[:0] + return nil +} + +func getArrowSchema(schema *FeatureServiceSchema) (*arrow.Schema, error) { fields := make([]arrow.Field, 0) - for _, joinKey := range b.schema.JoinKeys { - arrowType, err := gotypes.ValueTypeEnumToArrowType(b.schema.JoinKeysTypes[joinKey]) + for _, joinKey := range schema.JoinKeys { + arrowType, err := gotypes.ValueTypeEnumToArrowType(schema.JoinKeysTypes[joinKey]) if err != nil { return nil, err } @@ -60,8 +93,8 @@ func (b *MemoryBuffer) getArrowSchema() (*arrow.Schema, error) { fields = append(fields, arrow.Field{Name: joinKey, Type: arrowType}) } - for _, requestParam := range b.schema.RequestData { - arrowType, err := gotypes.ValueTypeEnumToArrowType(b.schema.RequestDataTypes[requestParam]) + for _, requestParam := range schema.RequestData { + arrowType, err := gotypes.ValueTypeEnumToArrowType(schema.RequestDataTypes[requestParam]) if err != nil { return nil, err } @@ -69,8 +102,8 @@ func (b *MemoryBuffer) getArrowSchema() (*arrow.Schema, error) { fields = append(fields, arrow.Field{Name: requestParam, Type: arrowType}) } - for _, featureName := range b.schema.Features { - arrowType, err := gotypes.ValueTypeEnumToArrowType(b.schema.FeaturesTypes[featureName]) + for _, featureName := range schema.Features { + arrowType, err := gotypes.ValueTypeEnumToArrowType(schema.FeaturesTypes[featureName]) if err != nil { return nil, err } @@ -98,18 +131,13 @@ func (b *MemoryBuffer) convertToArrowRecord() (arrow.Record, error) { arrowMemory := memory.NewGoAllocator() numRows := len(b.logs) - arrowSchema, err := b.getArrowSchema() - if err != nil { - return nil, err - } - columns := make(map[string][]*types.Value) fieldNameToIdx := make(map[string]int) - for idx, field := range arrowSchema.Fields() { + for idx, field := range b.arrowSchema.Fields() { fieldNameToIdx[field.Name] = idx } - builder := array.NewRecordBuilder(arrowMemory, arrowSchema) + builder := array.NewRecordBuilder(arrowMemory, b.arrowSchema) defer builder.Release() builder.Reserve(numRows) diff --git a/go/internal/feast/server/logging/memorybuffer_test.go b/go/internal/feast/server/logging/memorybuffer_test.go index 59f035799bb..94f0f86ef02 100644 --- a/go/internal/feast/server/logging/memorybuffer_test.go +++ b/go/internal/feast/server/logging/memorybuffer_test.go @@ -37,10 +37,6 @@ func TestArrowSchemaGeneration(t *testing.T) { }, } - b := &MemoryBuffer{ - schema: schema, - } - expectedArrowSchema := []arrow.Field{ {Name: "driver_id", Type: arrow.PrimitiveTypes.Int32}, {Name: "featureView1__int64", Type: arrow.PrimitiveTypes.Int64}, @@ -60,7 +56,7 @@ func TestArrowSchemaGeneration(t *testing.T) { {Name: "__request_id", Type: arrow.BinaryTypes.String}, } - actualSchema, err := b.getArrowSchema() + actualSchema, err := getArrowSchema(schema) assert.Nil(t, err) assert.Equal(t, expectedArrowSchema, actualSchema.Fields()) } @@ -84,51 +80,46 @@ func TestSerializeToArrowTable(t *testing.T) { } ts := timestamppb.New(time.Now()) - b := &MemoryBuffer{ - schema: schema, - logs: []*Log{ - { - EntityValue: []*types.Value{ - {Val: &types.Value_Int64Val{Int64Val: 1001}}, - }, - FeatureValues: []*types.Value{ - {Val: &types.Value_Int64Val{Int64Val: rand.Int63()}}, - {Val: &types.Value_FloatVal{FloatVal: rand.Float32()}}, - }, - FeatureStatuses: []serving.FieldStatus{ - serving.FieldStatus_PRESENT, - serving.FieldStatus_OUTSIDE_MAX_AGE, - }, - EventTimestamps: []*timestamppb.Timestamp{ - ts, ts, - }, - RequestId: "aaa", - LogTimestamp: time.Now(), - }, - { - EntityValue: []*types.Value{ - {Val: &types.Value_Int64Val{Int64Val: 1003}}, - }, - FeatureValues: []*types.Value{ - {Val: &types.Value_Int64Val{Int64Val: rand.Int63()}}, - {Val: &types.Value_FloatVal{FloatVal: rand.Float32()}}, - }, - FeatureStatuses: []serving.FieldStatus{ - serving.FieldStatus_PRESENT, - serving.FieldStatus_PRESENT, - }, - EventTimestamps: []*timestamppb.Timestamp{ - ts, ts, - }, - RequestId: "bbb", - LogTimestamp: time.Now(), - }, + b, _ := NewMemoryBuffer(schema) + b.Append(&Log{ + EntityValue: []*types.Value{ + {Val: &types.Value_Int64Val{Int64Val: 1001}}, }, - } + FeatureValues: []*types.Value{ + {Val: &types.Value_Int64Val{Int64Val: rand.Int63()}}, + {Val: &types.Value_FloatVal{FloatVal: rand.Float32()}}, + }, + FeatureStatuses: []serving.FieldStatus{ + serving.FieldStatus_PRESENT, + serving.FieldStatus_OUTSIDE_MAX_AGE, + }, + EventTimestamps: []*timestamppb.Timestamp{ + ts, ts, + }, + RequestId: "aaa", + LogTimestamp: time.Now(), + }) + b.Append(&Log{ + EntityValue: []*types.Value{ + {Val: &types.Value_Int64Val{Int64Val: 1003}}, + }, + FeatureValues: []*types.Value{ + {Val: &types.Value_Int64Val{Int64Val: rand.Int63()}}, + {Val: &types.Value_FloatVal{FloatVal: rand.Float32()}}, + }, + FeatureStatuses: []serving.FieldStatus{ + serving.FieldStatus_PRESENT, + serving.FieldStatus_PRESENT, + }, + EventTimestamps: []*timestamppb.Timestamp{ + ts, ts, + }, + RequestId: "bbb", + LogTimestamp: time.Now(), + }) pool := memory.NewGoAllocator() - arrowSchema, _ := b.getArrowSchema() - builder := array.NewRecordBuilder(pool, arrowSchema) + builder := array.NewRecordBuilder(pool, b.arrowSchema) defer builder.Release() // join key: driver_id diff --git a/go/internal/feast/server/logging/offlinestoresink.go b/go/internal/feast/server/logging/offlinestoresink.go index ee4c646a9be..632039baa43 100644 --- a/go/internal/feast/server/logging/offlinestoresink.go +++ b/go/internal/feast/server/logging/offlinestoresink.go @@ -1,10 +1,10 @@ package logging import ( - "errors" "fmt" "io" "io/ioutil" + "log" "os" "path/filepath" @@ -41,16 +41,7 @@ func (s *OfflineStoreSink) getOrCreateDatasetDir() (string, error) { return s.datasetDir, nil } -func (s *OfflineStoreSink) cleanCurrentDatasetDir() error { - if s.datasetDir == "" { - return nil - } - datasetDir := s.datasetDir - s.datasetDir = "" - return os.RemoveAll(datasetDir) -} - -func (s *OfflineStoreSink) Write(record arrow.Record) error { +func (s *OfflineStoreSink) Write(records []arrow.Record) error { fileName, _ := uuid.NewUUID() datasetDir, err := s.getOrCreateDatasetDir() if err != nil { @@ -62,7 +53,7 @@ func (s *OfflineStoreSink) Write(record arrow.Record) error { if err != nil { return err } - table := array.NewTableFromRecords(record.Schema(), []arrow.Record{record}) + table := array.NewTableFromRecords(records[0].Schema(), records) props := parquet.NewWriterProperties(parquet.WithDictionaryDefault(false)) arrProps := pqarrow.DefaultWriterProps() @@ -74,10 +65,16 @@ func (s *OfflineStoreSink) Flush(featureServiceName string) error { return nil } - errMsg := s.writeCallback(featureServiceName, s.datasetDir) - if errMsg != "" { - return errors.New(errMsg) - } + datasetDir := s.datasetDir + s.datasetDir = "" + + go func() { + errMsg := s.writeCallback(featureServiceName, datasetDir) + if errMsg != "" { + log.Println(errMsg) + } + os.RemoveAll(datasetDir) + }() - return s.cleanCurrentDatasetDir() + return nil } diff --git a/go/internal/feast/server/logging/service.go b/go/internal/feast/server/logging/service.go index 9249ad4f2f1..c237a80c688 100644 --- a/go/internal/feast/server/logging/service.go +++ b/go/internal/feast/server/logging/service.go @@ -42,7 +42,7 @@ type LoggingService struct { var ( DefaultOptions = LoggingOptions{ - ChannelCapacity: 1000, + ChannelCapacity: 100000, FlushInterval: 10 * time.Minute, WriteInterval: 10 * time.Second, EmitTimeout: 10 * time.Millisecond, diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py index 9f3cf26dee0..1e317eb7990 100644 --- a/sdk/python/feast/cli.py +++ b/sdk/python/feast/cli.py @@ -622,16 +622,24 @@ def init_command(project_directory, minimal: bool, template: str): @click.option( "--no-access-log", is_flag=True, help="Disable the Uvicorn access log.", ) +@click.option( + "--no-feature-log", is_flag=True, help="Disable logging served features", +) @click.pass_context def serve_command( - ctx: click.Context, host: str, port: int, type_: str, no_access_log: bool + ctx: click.Context, + host: str, + port: int, + type_: str, + no_access_log: bool, + no_feature_log: bool, ): """Start a feature server locally on a given port.""" repo = ctx.obj["CHDIR"] cli_check_repo(repo) store = FeatureStore(repo_path=str(repo)) - store.serve(host, port, type_, no_access_log) + store.serve(host, port, type_, no_access_log, no_feature_log) @cli.command("serve_transformations") diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py index 140bd76a835..2e312240fe1 100644 --- a/sdk/python/feast/feature_store.py +++ b/sdk/python/feast/feature_store.py @@ -1995,16 +1995,27 @@ def _get_feature_views_to_use( return views_to_use @log_exceptions_and_usage - def serve(self, host: str, port: int, type_: str, no_access_log: bool) -> None: + def serve( + self, + host: str, + port: int, + type_: str, + no_access_log: bool, + no_feature_log: bool, + ) -> None: """Start the feature consumption server locally on a given port.""" type_ = type_.lower() if self.config.go_feature_retrieval: # Start go server instead of python if the flag is enabled self._lazy_init_go_server() if type_ == "http": - self._go_server.start_http_server(host, port) + self._go_server.start_http_server( + host, port, enable_logging=not no_feature_log + ) elif type_ == "grpc": - self._go_server.start_grpc_server(host, port) + self._go_server.start_grpc_server( + host, port, enable_logging=not no_feature_log + ) else: raise ValueError( f"Unsupported server type '{type_}'. Must be one of 'http' or 'grpc'." diff --git a/sdk/python/feast/infra/offline_stores/file.py b/sdk/python/feast/infra/offline_stores/file.py index de1168a86fb..c675751739a 100644 --- a/sdk/python/feast/infra/offline_stores/file.py +++ b/sdk/python/feast/infra/offline_stores/file.py @@ -1,3 +1,4 @@ +import uuid from datetime import datetime from pathlib import Path from typing import Callable, List, Optional, Tuple, Union @@ -5,6 +6,7 @@ import dask.dataframe as dd import pandas as pd import pyarrow +import pyarrow.dataset import pyarrow.parquet import pytz from pydantic.typing import Literal @@ -384,17 +386,22 @@ def write_logged_features( assert isinstance(destination, FileLoggingDestination) if isinstance(data, Path): - data = pyarrow.parquet.read_table(data) + # Since this code will be mostly used from Go-created thread, it's better to avoid producing new threads + data = pyarrow.parquet.read_table(data, use_threads=False, pre_buffer=False) filesystem, path = FileSource.create_filesystem_and_path( destination.path, destination.s3_endpoint_override, ) - pyarrow.parquet.write_to_dataset( + pyarrow.dataset.write_dataset( data, - root_path=path, - partition_cols=destination.partition_by, + base_dir=path, + basename_template=f"{uuid.uuid4().hex}-{{i}}.parquet", + partitioning=destination.partition_by, filesystem=filesystem, + use_threads=False, + format=pyarrow.dataset.ParquetFileFormat(), + existing_data_behavior="overwrite_or_ignore", ) From 88ead509fdd5d6798e09faed18964f36ee07ce00 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Fri, 27 May 2022 10:46:37 -0700 Subject: [PATCH 026/123] chore: Fix gitpod init commands to generate go code (#2738) Signed-off-by: Achal Shah --- .gitpod.yml | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/.gitpod.yml b/.gitpod.yml index e14a5f807b7..7146454a54e 100644 --- a/.gitpod.yml +++ b/.gitpod.yml @@ -4,12 +4,8 @@ tasks: pip install pre-commit pre-commit install --hook-type pre-commit --hook-type pre-push pip install -e '.[dev]' - git config --global alias.ci 'commit -s' - git config --global alias.sw switch - git config --global alias.st status - git config --global alias.co checkout - git config --global alias.br branch - git config --global alias.df diff + make compile-protos-go + make compile-go-lib github: prebuilds: # enable for the default branch (defaults to true) From 1f89758b72b719f3ba7a1d10cb5785734c550bd1 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Fri, 27 May 2022 12:12:37 -0700 Subject: [PATCH 027/123] chore: Fix gitpod init commands to preserve pip modules (#2740) Signed-off-by: Achal Shah --- .gitpod.Dockerfile | 5 +++++ .gitpod.yml | 11 ++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) create mode 100644 .gitpod.Dockerfile diff --git a/.gitpod.Dockerfile b/.gitpod.Dockerfile new file mode 100644 index 00000000000..22493955674 --- /dev/null +++ b/.gitpod.Dockerfile @@ -0,0 +1,5 @@ +FROM gitpod/workspace-full:latest + +ENV PYTHONUSERBASE=/workspace/.pip-modules +ENV PATH=$PYTHONUSERBASE/bin:$PATH +ENV PIP_USER=yes diff --git a/.gitpod.yml b/.gitpod.yml index 7146454a54e..4b3585e98a4 100644 --- a/.gitpod.yml +++ b/.gitpod.yml @@ -1,11 +1,20 @@ # https://www.gitpod.io/docs/config-gitpod-file +image: + file: .gitpod.Dockerfile tasks: - init: | pip install pre-commit pre-commit install --hook-type pre-commit --hook-type pre-push - pip install -e '.[dev]' + pip install '.[dev]' make compile-protos-go make compile-go-lib + command: | + git config --global alias.ci 'commit -s' + git config --global alias.sw switch + git config --global alias.st status + git config --global alias.co checkout + git config --global alias.br branch + git config --global alias.df diff github: prebuilds: # enable for the default branch (defaults to true) From c4b636f37f988a21cb08c0d51ba6f56ad09f9739 Mon Sep 17 00:00:00 2001 From: Oleksii Moskalenko Date: Fri, 27 May 2022 15:53:37 -0700 Subject: [PATCH 028/123] chore: Configure feature logging via feature_store.yaml (#2739) * Feature logging configurable via feature_store.yaml Signed-off-by: Oleksii Moskalenko * allow local server type Signed-off-by: Oleksii Moskalenko --- .../feature-servers/go-feature-retrieval.md | 30 ++++++++++++++++ sdk/python/feast/cli.py | 6 ++-- sdk/python/feast/dqm/profilers/ge_profiler.py | 5 +-- .../embedded_go/online_features_service.py | 34 ++++++++++++++++--- sdk/python/feast/feature_store.py | 25 ++++++++++++-- .../feature_servers/aws_lambda/config.py | 7 ++-- .../infra/feature_servers/base_config.py | 34 +++++++++++++++++++ .../feature_servers/gcp_cloudrun/config.py | 7 ++-- .../feature_servers/local_process/__init__.py | 0 .../feature_servers/local_process/config.py | 8 +++++ sdk/python/feast/repo_config.py | 12 ++++--- sdk/python/feast/utils.py | 8 +++++ .../integration/e2e/test_go_feature_server.py | 17 ++++------ .../feature_repos/repo_configuration.py | 7 ++-- 14 files changed, 160 insertions(+), 40 deletions(-) create mode 100644 sdk/python/feast/infra/feature_servers/base_config.py create mode 100644 sdk/python/feast/infra/feature_servers/local_process/__init__.py create mode 100644 sdk/python/feast/infra/feature_servers/local_process/config.py diff --git a/docs/reference/feature-servers/go-feature-retrieval.md b/docs/reference/feature-servers/go-feature-retrieval.md index 30c1a9ca721..685e7201cb6 100644 --- a/docs/reference/feature-servers/go-feature-retrieval.md +++ b/docs/reference/feature-servers/go-feature-retrieval.md @@ -35,6 +35,36 @@ go_feature_retrieval: True ``` {% endcode %} +## Feature logging + +Go feature server can log all requested entities and served features to a configured destination inside an offline store. +This allows users to create new datasets from features served online. Those datasets could be used for future trainings or for +feature validations. To enable feature logging we need to edit `feature_store.yaml`: +```yaml +project: my_feature_repo +registry: data/registry.db +provider: local +online_store: + type: redis + connection_string: "localhost:6379" +go_feature_retrieval: True +feature_server: + feature_logging: + enable: True +``` + +Feature logging configuration in `feature_store.yaml` also allows to tweak some low-level parameters to achieve the best performance: +```yaml +feature_server: + feature_logging: + enable: True + flush_interval_secs: 300 + write_to_disk_interval_secs: 30 + emit_timeout_micro_secs: 10000 + queue_capacity: 10000 +``` +All these parameters are optional. + ## Future/Current Work The Go feature retrieval online feature logging for Data Quality Monitoring is currently in development. More information can be found [here](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit#heading=h.9gaqqtox9jg6). diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py index 1e317eb7990..ce73ba162c7 100644 --- a/sdk/python/feast/cli.py +++ b/sdk/python/feast/cli.py @@ -41,8 +41,10 @@ registry_dump, teardown, ) +from feast.utils import maybe_local_tz _logger = logging.getLogger(__name__) +warnings.filterwarnings("ignore", category=DeprecationWarning, module="(?!feast)") class NoOptionDefaultFormat(click.Command): @@ -803,8 +805,8 @@ def validate( result = store.validate_logged_features( source=feature_service, reference=reference, - start=datetime.fromisoformat(start_ts), - end=datetime.fromisoformat(end_ts), + start=maybe_local_tz(datetime.fromisoformat(start_ts)), + end=maybe_local_tz(datetime.fromisoformat(end_ts)), throw_exception=False, cache_profile=not no_profile_cache, ) diff --git a/sdk/python/feast/dqm/profilers/ge_profiler.py b/sdk/python/feast/dqm/profilers/ge_profiler.py index 81e2d81d8c3..523f9d4ff8b 100644 --- a/sdk/python/feast/dqm/profilers/ge_profiler.py +++ b/sdk/python/feast/dqm/profilers/ge_profiler.py @@ -8,7 +8,6 @@ import pandas as pd from great_expectations.core import ExpectationSuite from great_expectations.dataset import PandasDataset -from great_expectations.profile.base import ProfilerTypeMapping from feast.dqm.profilers.profiler import ( Profile, @@ -29,9 +28,7 @@ def _prepare_dataset(dataset: PandasDataset) -> PandasDataset: dataset_copy = dataset.copy(deep=True) for column in dataset.columns: - if dataset.expect_column_values_to_be_in_type_list( - column, type_list=sorted(list(ProfilerTypeMapping.DATETIME_TYPE_NAMES)) - ).success: + if pd.api.types.is_datetime64_any_dtype(dataset[column]): # GE cannot parse Timestamp or other pandas datetime time dataset_copy[column] = dataset[column].dt.strftime("%Y-%m-%dT%H:%M:%S") diff --git a/sdk/python/feast/embedded_go/online_features_service.py b/sdk/python/feast/embedded_go/online_features_service.py index 8ec4410bde7..a864d35ce47 100644 --- a/sdk/python/feast/embedded_go/online_features_service.py +++ b/sdk/python/feast/embedded_go/online_features_service.py @@ -11,6 +11,7 @@ RequestDataNotFoundInEntityRowsException, ) from feast.feature_service import FeatureService +from feast.infra.feature_servers.base_config import FeatureLoggingConfig from feast.online_response import OnlineResponse from feast.protos.feast.serving.ServingService_pb2 import GetOnlineFeaturesResponse from feast.protos.feast.types import Value_pb2 @@ -30,6 +31,11 @@ if TYPE_CHECKING: from feast.feature_store import FeatureStore +NANO_SECOND = 1 +MICRO_SECOND = 1000 * NANO_SECOND +MILLI_SECOND = 1000 * MICRO_SECOND +SECOND = 1000 * MILLI_SECOND + class EmbeddedOnlineFeatureServer: def __init__( @@ -144,12 +150,22 @@ def start_grpc_server( host: str, port: int, enable_logging: bool = True, - logging_options: Optional[LoggingOptions] = None, + logging_options: Optional[FeatureLoggingConfig] = None, ): if enable_logging: if logging_options: self._service.StartGprcServerWithLogging( - host, port, self._logging_callback, logging_options + host, + port, + self._logging_callback, + LoggingOptions( + FlushInterval=logging_options.flush_interval_secs * SECOND, + WriteInterval=logging_options.write_to_disk_interval_secs + * SECOND, + EmitTimeout=logging_options.emit_timeout_micro_secs + * MICRO_SECOND, + ChannelCapacity=logging_options.queue_capacity, + ), ) else: self._service.StartGprcServerWithLoggingDefaultOpts( @@ -163,12 +179,22 @@ def start_http_server( host: str, port: int, enable_logging: bool = True, - logging_options: Optional[LoggingOptions] = None, + logging_options: Optional[FeatureLoggingConfig] = None, ): if enable_logging: if logging_options: self._service.StartHttpServerWithLogging( - host, port, self._logging_callback, logging_options + host, + port, + self._logging_callback, + LoggingOptions( + FlushInterval=logging_options.flush_interval_secs * SECOND, + WriteInterval=logging_options.write_to_disk_interval_secs + * SECOND, + EmitTimeout=logging_options.emit_timeout_micro_secs + * MICRO_SECOND, + ChannelCapacity=logging_options.queue_capacity, + ), ) else: self._service.StartHttpServerWithLoggingDefaultOpts( diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py index 2e312240fe1..278ae077744 100644 --- a/sdk/python/feast/feature_store.py +++ b/sdk/python/feast/feature_store.py @@ -2008,13 +2008,30 @@ def serve( if self.config.go_feature_retrieval: # Start go server instead of python if the flag is enabled self._lazy_init_go_server() + enable_logging = ( + self.config.feature_server + and self.config.feature_server.feature_logging + and self.config.feature_server.feature_logging.enabled + and not no_feature_log + ) + logging_options = ( + self.config.feature_server.feature_logging + if enable_logging and self.config.feature_server + else None + ) if type_ == "http": self._go_server.start_http_server( - host, port, enable_logging=not no_feature_log + host, + port, + enable_logging=enable_logging, + logging_options=logging_options, ) elif type_ == "grpc": self._go_server.start_grpc_server( - host, port, enable_logging=not no_feature_log + host, + port, + enable_logging=enable_logging, + logging_options=logging_options, ) else: raise ValueError( @@ -2139,12 +2156,14 @@ def validate_logged_features( # read and run validation try: - j.to_arrow(validation_reference=reference) + t = j.to_arrow(validation_reference=reference) except ValidationFailed as exc: if throw_exception: raise return exc + else: + print(f"{t.shape[0]} rows were validated.") if cache_profile: self.apply(reference) diff --git a/sdk/python/feast/infra/feature_servers/aws_lambda/config.py b/sdk/python/feast/infra/feature_servers/aws_lambda/config.py index d026415ec30..31dd879af6d 100644 --- a/sdk/python/feast/infra/feature_servers/aws_lambda/config.py +++ b/sdk/python/feast/infra/feature_servers/aws_lambda/config.py @@ -1,18 +1,15 @@ from pydantic import StrictBool, StrictStr from pydantic.typing import Literal -from feast.repo_config import FeastConfigBaseModel +from feast.infra.feature_servers.base_config import BaseFeatureServerConfig -class AwsLambdaFeatureServerConfig(FeastConfigBaseModel): +class AwsLambdaFeatureServerConfig(BaseFeatureServerConfig): """Feature server config for AWS Lambda.""" type: Literal["aws_lambda"] = "aws_lambda" """Feature server type selector.""" - enabled: StrictBool = False - """Whether the feature server should be launched.""" - public: StrictBool = True """Whether the endpoint should be publicly accessible.""" diff --git a/sdk/python/feast/infra/feature_servers/base_config.py b/sdk/python/feast/infra/feature_servers/base_config.py new file mode 100644 index 00000000000..756dd79b438 --- /dev/null +++ b/sdk/python/feast/infra/feature_servers/base_config.py @@ -0,0 +1,34 @@ +from typing import Optional + +from pydantic import StrictBool, StrictInt + +from feast.repo_config import FeastConfigBaseModel + + +class FeatureLoggingConfig(FeastConfigBaseModel): + enabled: StrictBool = False + """Whether the feature server should log served features.""" + + flush_interval_secs: StrictInt = 600 + """Interval of flushing logs to the destination in offline store.""" + + write_to_disk_interval_secs: StrictInt = 30 + """Interval of dumping logs collected in memory to local disk.""" + + queue_capacity: StrictInt = 100000 + """Log queue capacity. If number of produced logs is bigger than + processing speed logs will be accumulated in the queue. + After queue length will reach this number all new items will be rejected.""" + + emit_timeout_micro_secs: StrictInt = 10000 + """Timeout for adding new log item to the queue.""" + + +class BaseFeatureServerConfig(FeastConfigBaseModel): + """Base Feature Server config that should be extended""" + + enabled: StrictBool = False + """Whether the feature server should be launched.""" + + feature_logging: Optional[FeatureLoggingConfig] + """ Feature logging configuration """ diff --git a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/config.py b/sdk/python/feast/infra/feature_servers/gcp_cloudrun/config.py index 728ac56ae29..8d0c269cf5d 100644 --- a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/config.py +++ b/sdk/python/feast/infra/feature_servers/gcp_cloudrun/config.py @@ -1,18 +1,15 @@ from pydantic import StrictBool from pydantic.typing import Literal -from feast.repo_config import FeastConfigBaseModel +from feast.infra.feature_servers.base_config import BaseFeatureServerConfig -class GcpCloudRunFeatureServerConfig(FeastConfigBaseModel): +class GcpCloudRunFeatureServerConfig(BaseFeatureServerConfig): """Feature server config for GCP CloudRun.""" type: Literal["gcp_cloudrun"] = "gcp_cloudrun" """Feature server type selector.""" - enabled: StrictBool = False - """Whether the feature server should be launched.""" - public: StrictBool = True """Whether the endpoint should be publicly accessible.""" diff --git a/sdk/python/feast/infra/feature_servers/local_process/__init__.py b/sdk/python/feast/infra/feature_servers/local_process/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/sdk/python/feast/infra/feature_servers/local_process/config.py b/sdk/python/feast/infra/feature_servers/local_process/config.py new file mode 100644 index 00000000000..bb2e7bdf738 --- /dev/null +++ b/sdk/python/feast/infra/feature_servers/local_process/config.py @@ -0,0 +1,8 @@ +from pydantic.typing import Literal + +from feast.infra.feature_servers.base_config import BaseFeatureServerConfig + + +class LocalFeatureServerConfig(BaseFeatureServerConfig): + type: Literal["local"] = "local" + """Feature server type selector.""" diff --git a/sdk/python/feast/repo_config.py b/sdk/python/feast/repo_config.py index 62d799a2b6b..b7fd9c20377 100644 --- a/sdk/python/feast/repo_config.py +++ b/sdk/python/feast/repo_config.py @@ -53,11 +53,13 @@ FEATURE_SERVER_CONFIG_CLASS_FOR_TYPE = { "aws_lambda": "feast.infra.feature_servers.aws_lambda.config.AwsLambdaFeatureServerConfig", "gcp_cloudrun": "feast.infra.feature_servers.gcp_cloudrun.config.GcpCloudRunFeatureServerConfig", + "local": "feast.infra.feature_servers.local_process.config.LocalFeatureServerConfig", } FEATURE_SERVER_TYPE_FOR_PROVIDER = { "aws": "aws_lambda", "gcp": "gcp_cloudrun", + "local": "local", } @@ -285,17 +287,17 @@ def _validate_feature_server_config(cls, values): if "provider" not in values: raise FeastProviderNotSetError() - feature_server_type = FEATURE_SERVER_TYPE_FOR_PROVIDER.get(values["provider"]) - defined_type = values["feature_server"].get("type") + default_type = FEATURE_SERVER_TYPE_FOR_PROVIDER.get(values["provider"]) + defined_type = values["feature_server"].get("type", default_type) # Make sure that the type is either not set, or set correctly, since it's defined by the provider - if defined_type not in (None, feature_server_type): + if defined_type not in (default_type, "local"): raise FeastFeatureServerTypeSetError(defined_type) - values["feature_server"]["type"] = feature_server_type + values["feature_server"]["type"] = defined_type # Validate the dict to ensure one of the union types match try: feature_server_config_class = get_feature_server_config_from_type( - feature_server_type + defined_type ) feature_server_config_class(**values["feature_server"]) except ValidationError as e: diff --git a/sdk/python/feast/utils.py b/sdk/python/feast/utils.py index e521338680c..a40f423c537 100644 --- a/sdk/python/feast/utils.py +++ b/sdk/python/feast/utils.py @@ -1,5 +1,6 @@ from datetime import datetime +from dateutil.tz import tzlocal from pytz import utc @@ -16,3 +17,10 @@ def to_naive_utc(ts: datetime) -> datetime: return ts else: return ts.astimezone(utc).replace(tzinfo=None) + + +def maybe_local_tz(t: datetime) -> datetime: + if t.tzinfo is None: + return t.replace(tzinfo=tzlocal()) + else: + return t diff --git a/sdk/python/tests/integration/e2e/test_go_feature_server.py b/sdk/python/tests/integration/e2e/test_go_feature_server.py index 11526132ac8..4fd003c1944 100644 --- a/sdk/python/tests/integration/e2e/test_go_feature_server.py +++ b/sdk/python/tests/integration/e2e/test_go_feature_server.py @@ -12,10 +12,10 @@ import requests from feast import FeatureService, FeatureView, ValueType -from feast.embedded_go.lib.embedded import LoggingOptions from feast.embedded_go.online_features_service import EmbeddedOnlineFeatureServer from feast.feast_object import FeastObject from feast.feature_logging import LoggingConfig +from feast.infra.feature_servers.base_config import FeatureLoggingConfig from feast.protos.feast.serving.ServingService_pb2 import ( FieldStatus, GetOnlineFeaturesRequest, @@ -34,10 +34,6 @@ location, ) -NANOSECOND = 1 -MILLISECOND = 1000_000 * NANOSECOND -SECOND = 1000 * MILLISECOND - @pytest.fixture(scope="session") def initialized_registry(environment, universal_data_sources): @@ -84,11 +80,12 @@ def server_port(environment, server_type: str): args=("127.0.0.1", port), kwargs=dict( enable_logging=True, - logging_options=LoggingOptions( - ChannelCapacity=100, - WriteInterval=100 * MILLISECOND, - FlushInterval=1 * SECOND, - EmitTimeout=10 * MILLISECOND, + logging_options=FeatureLoggingConfig( + enabled=True, + queue_capacity=100, + write_to_disk_interval_secs=1, + flush_interval_secs=1, + emit_timeout_micro_secs=10000, ), ), ) diff --git a/sdk/python/tests/integration/feature_repos/repo_configuration.py b/sdk/python/tests/integration/feature_repos/repo_configuration.py index f4c9bed92a9..5a48115dbed 100644 --- a/sdk/python/tests/integration/feature_repos/repo_configuration.py +++ b/sdk/python/tests/integration/feature_repos/repo_configuration.py @@ -17,6 +17,8 @@ from feast.constants import FULL_REPO_CONFIGS_MODULE_ENV_NAME from feast.data_source import DataSource from feast.errors import FeastModuleImportError +from feast.infra.feature_servers.base_config import FeatureLoggingConfig +from feast.infra.feature_servers.local_process.config import LocalFeatureServerConfig from feast.repo_config import RegistryConfig, RepoConfig from tests.integration.feature_repos.integration_test_repo_config import ( IntegrationTestRepoConfig, @@ -386,8 +388,9 @@ def construct_test_environment( f"s3://feast-integration-tests/registries/{project}/registry.db" ) # type: Union[str, RegistryConfig] else: - # Note: even if it's a local feature server, the repo config does not have this configured - feature_server = None + feature_server = LocalFeatureServerConfig( + feature_logging=FeatureLoggingConfig(enabled=True) + ) registry = RegistryConfig( path=str(Path(repo_dir_name) / "registry.db"), cache_ttl_seconds=1, ) From 92c1f342a4f06132fe781606792096de67c1a609 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Fri, 27 May 2022 16:33:43 -0700 Subject: [PATCH 029/123] chore: Respect environment variables when calling check_call (#2741) Signed-off-by: Achal Shah --- setup.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/setup.py b/setup.py index e0eaf4f5262..f1cc3119fe7 100644 --- a/setup.py +++ b/setup.py @@ -249,6 +249,7 @@ def _generate_python_protos(self, path: str): self.python_folder, ] + proto_files, + env=os.environ ) def run(self): @@ -334,6 +335,8 @@ def _generate_go_protos(self, path: str): proto_files = glob.glob(os.path.join(self.proto_folder, path)) try: + e = os.environ.copy() + e["PATH"] = self.path_val subprocess.check_call( self.go_protoc + [ @@ -347,7 +350,7 @@ def _generate_go_protos(self, path: str): "--go-grpc_opt=module=github.com/feast-dev/feast/go/protos", ] + proto_files, - env={"PATH": self.path_val}, + env=e, ) except CalledProcessError as e: print(f"Stderr: {e.stderr}") From 3d2b43111ae497ec7c720eda2ca3ec3db6674891 Mon Sep 17 00:00:00 2001 From: Oleksii Moskalenko Date: Fri, 27 May 2022 18:24:37 -0700 Subject: [PATCH 030/123] chore: Collection of small improvements for feature validation (#2742) * Feature logging configurable via feature_store.yaml Signed-off-by: Oleksii Moskalenko * set nan when status is not found Signed-off-by: Oleksii Moskalenko * improvements Signed-off-by: Oleksii Moskalenko * fix tests Signed-off-by: Oleksii Moskalenko --- go/types/typeconversion.go | 4 ++++ sdk/python/feast/dqm/profilers/ge_profiler.py | 8 ++++++++ sdk/python/feast/dqm/profilers/profiler.py | 8 ++++++++ sdk/python/tests/integration/e2e/test_validation.py | 9 ++++++--- 4 files changed, 26 insertions(+), 3 deletions(-) diff --git a/go/types/typeconversion.go b/go/types/typeconversion.go index 30fcc1e393c..45eeac52c6f 100644 --- a/go/types/typeconversion.go +++ b/go/types/typeconversion.go @@ -285,6 +285,10 @@ func ArrowValuesToProtoValues(arr arrow.Array) ([]*types.Value, error) { &types.Value{Val: &types.Value_UnixTimestampVal{ UnixTimestampVal: int64(arr.(*array.Timestamp).Value(idx))}}) } + case arrow.Null: + for idx := 0; idx < arr.Len(); idx++ { + values = append(values, &types.Value{}) + } default: return nil, fmt.Errorf("unsupported arrow to proto conversion for type %s", arr.DataType()) } diff --git a/sdk/python/feast/dqm/profilers/ge_profiler.py b/sdk/python/feast/dqm/profilers/ge_profiler.py index 523f9d4ff8b..20a92631cb6 100644 --- a/sdk/python/feast/dqm/profilers/ge_profiler.py +++ b/sdk/python/feast/dqm/profilers/ge_profiler.py @@ -37,6 +37,12 @@ def _prepare_dataset(dataset: PandasDataset) -> PandasDataset: # This could cause error on comparison => so better to convert to double prematurely dataset_copy[column] = dataset[column].astype(np.float64) + status_column = f"{column}__status" + if status_column in dataset.columns: + dataset_copy[column] = dataset_copy[column].mask( + dataset[status_column] == FieldStatus.NOT_FOUND, np.nan + ) + return dataset_copy @@ -169,6 +175,8 @@ def errors(self) -> List["ValidationError"]: check_config=res.expectation_config.kwargs, missing_count=res["result"].get("missing_count"), missing_percent=res["result"].get("missing_percent"), + unexpected_count=res["result"].get("unexpected_count"), + unexpected_percent=res["result"].get("unexpected_percent"), ) for res in self._validation_result["results"] if not res["success"] diff --git a/sdk/python/feast/dqm/profilers/profiler.py b/sdk/python/feast/dqm/profilers/profiler.py index e65bf8601f0..34496b0cca3 100644 --- a/sdk/python/feast/dqm/profilers/profiler.py +++ b/sdk/python/feast/dqm/profilers/profiler.py @@ -70,6 +70,8 @@ class ValidationError: missing_count: Optional[int] missing_percent: Optional[float] observed_value: Optional[float] + unexpected_count: Optional[int] + unexpected_percent: Optional[float] def __init__( self, @@ -79,6 +81,8 @@ def __init__( missing_count: Optional[int] = None, missing_percent: Optional[float] = None, observed_value: Optional[float] = None, + unexpected_count: Optional[int] = None, + unexpected_percent: Optional[float] = None, ): self.check_name = check_name self.column_name = column_name @@ -86,6 +90,8 @@ def __init__( self.missing_count = missing_count self.missing_percent = missing_percent self.observed_value = observed_value + self.unexpected_count = unexpected_count + self.unexpected_percent = unexpected_percent def __repr__(self): return f"" @@ -98,4 +104,6 @@ def to_dict(self): missing_count=self.missing_count, missing_percent=self.missing_percent, observed_value=self.observed_value, + unexpected_count=self.unexpected_count, + unexpected_percent=self.unexpected_percent, ) diff --git a/sdk/python/tests/integration/e2e/test_validation.py b/sdk/python/tests/integration/e2e/test_validation.py index 338dc77d236..0d65640dcaf 100644 --- a/sdk/python/tests/integration/e2e/test_validation.py +++ b/sdk/python/tests/integration/e2e/test_validation.py @@ -317,8 +317,8 @@ def test_e2e_validation_via_cli(environment, universal_data_sources): feature_service.name, "--reference", reference.name, - (datetime.datetime.utcnow() - datetime.timedelta(days=7)).isoformat(), - datetime.datetime.utcnow().isoformat(), + (datetime.datetime.now() - datetime.timedelta(days=7)).isoformat(), + datetime.datetime.now().isoformat(), ] p = runner.run(validate_args, cwd=local_repo.repo_path) @@ -335,7 +335,10 @@ def test_e2e_validation_via_cli(environment, universal_data_sources): "current_balance": [0], "avg_passenger_count": [0], "lifetime_trip_count": [0], - "event_timestamp": [make_tzaware(datetime.datetime.utcnow())], + "event_timestamp": [ + make_tzaware(datetime.datetime.utcnow()) + - datetime.timedelta(hours=1) + ], } ) invalid_logs = prepare_logs(invalid_data, feature_service, store) From de97bd329425374b25a09c6a8241ce2d99af4a64 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Tue, 31 May 2022 11:12:12 -0700 Subject: [PATCH 031/123] chore: Install dependencies into a virtualenv for gitpod (#2749) * chore: Add pybindgen to setup_requires Signed-off-by: Achal Shah * chore: Add env to gopy command Signed-off-by: Achal Shah * remove clone Signed-off-by: Achal Shah * More fixes Signed-off-by: Achal Shah * Fix protos persistance Signed-off-by: Achal Shah * Just create a virtualenv locally Signed-off-by: Achal Shah * compile python protos Signed-off-by: Achal Shah * rmove env stuff Signed-off-by: Achal Shah * fix Signed-off-by: Achal Shah --- .gitpod.Dockerfile | 5 ----- .gitpod.yml | 14 ++++++++++++-- setup.py | 6 ++---- 3 files changed, 14 insertions(+), 11 deletions(-) delete mode 100644 .gitpod.Dockerfile diff --git a/.gitpod.Dockerfile b/.gitpod.Dockerfile deleted file mode 100644 index 22493955674..00000000000 --- a/.gitpod.Dockerfile +++ /dev/null @@ -1,5 +0,0 @@ -FROM gitpod/workspace-full:latest - -ENV PYTHONUSERBASE=/workspace/.pip-modules -ENV PATH=$PYTHONUSERBASE/bin:$PATH -ENV PIP_USER=yes diff --git a/.gitpod.yml b/.gitpod.yml index 4b3585e98a4..b28dfbe49f5 100644 --- a/.gitpod.yml +++ b/.gitpod.yml @@ -1,14 +1,20 @@ # https://www.gitpod.io/docs/config-gitpod-file -image: - file: .gitpod.Dockerfile tasks: - init: | + python -m venv venv + source venv/bin/activate + pip install pre-commit pre-commit install --hook-type pre-commit --hook-type pre-push pip install '.[dev]' + make compile-protos-python make compile-protos-go make compile-go-lib + env: + PYTHONUSERBASE: "/workspace/.pip-modules" command: | + source venv/bin/activate + git config --global alias.ci 'commit -s' git config --global alias.sw switch git config --global alias.st status @@ -31,3 +37,7 @@ github: addComment: false # add a "Review in Gitpod" button to the pull request's description (defaults to false) addBadge: false + +vscode: + extensions: + - ms-python.python diff --git a/setup.py b/setup.py index f1cc3119fe7..8934ca24ad0 100644 --- a/setup.py +++ b/setup.py @@ -249,7 +249,6 @@ def _generate_python_protos(self, path: str): self.python_folder, ] + proto_files, - env=os.environ ) def run(self): @@ -335,8 +334,6 @@ def _generate_go_protos(self, path: str): proto_files = glob.glob(os.path.join(self.proto_folder, path)) try: - e = os.environ.copy() - e["PATH"] = self.path_val subprocess.check_call( self.go_protoc + [ @@ -350,7 +347,7 @@ def _generate_go_protos(self, path: str): "--go-grpc_opt=module=github.com/feast-dev/feast/go/protos", ] + proto_files, - env=e, + env={"PATH": self.path_val}, ) except CalledProcessError as e: print(f"Stderr: {e.stderr}") @@ -499,6 +496,7 @@ def copy_extensions_to_source(self): "grpcio", "grpcio-tools==1.44.0", "mypy-protobuf==3.1", + "pybindgen==0.22.0", "sphinx!=4.0.0", ], cmdclass={ From 5a7fcefb80837652e48b3387dd3a92153c6d0137 Mon Sep 17 00:00:00 2001 From: Abhin Chhabra Date: Wed, 1 Jun 2022 18:11:32 -0400 Subject: [PATCH 032/123] chore: Fix minor documentation typo (#2752) Signed-off-by: Abhin Chhabra --- sdk/python/feast/data_source.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdk/python/feast/data_source.py b/sdk/python/feast/data_source.py index 4392314bb8d..425143133ab 100644 --- a/sdk/python/feast/data_source.py +++ b/sdk/python/feast/data_source.py @@ -159,7 +159,7 @@ class DataSource(ABC): Args: name: Name of data source, which should be unique within a project - timestamp_field (optional): (Deprecated) Event timestamp column used for point in time + event_timestamp_column (optional): (Deprecated) Event timestamp column used for point in time joins of feature values. created_timestamp_column (optional): Timestamp column indicating when the row was created, used for deduplicating rows. From 0cf3c923717f561d5656c57eb0b61fcd569917bd Mon Sep 17 00:00:00 2001 From: Kevin Zhang Date: Wed, 1 Jun 2022 16:02:52 -0700 Subject: [PATCH 033/123] feat: Stream Feature View FCOS (#2750) * Fix working version Signed-off-by: Kevin Zhang * Working commit Signed-off-by: Kevin Zhang * Fixes Signed-off-by: Kevin Zhang * Fix stuffs Signed-off-by: Kevin Zhang * Fix lint Signed-off-by: Kevin Zhang * Fix things Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang * Fix lint Signed-off-by: Kevin Zhang * Fix lint Signed-off-by: Kevin Zhang * Fix lihnt Signed-off-by: Kevin Zhang * Fix stuff Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang * Fix lint Signed-off-by: Kevin Zhang * Fix unit tests Signed-off-by: Kevin Zhang * Address review comments Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang * Fix lint Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang * Fixed Signed-off-by: Kevin Zhang * Unsaved changes Signed-off-by: Kevin Zhang --- protos/feast/core/Aggregation.proto | 14 ++ protos/feast/core/DataFormat.proto | 9 +- protos/feast/core/FeatureService.proto | 1 - protos/feast/core/Registry.proto | 4 +- protos/feast/core/StreamFeatureView.proto | 98 ++++++++ protos/feast/core/ValidationProfile.proto | 2 - sdk/python/feast/aggregation.py | 69 ++++++ sdk/python/feast/data_format.py | 24 ++ sdk/python/feast/data_source.py | 3 + sdk/python/feast/feature_store.py | 95 ++++++- sdk/python/feast/inference.py | 8 +- sdk/python/feast/registry.py | 42 +++- sdk/python/feast/repo_contents.py | 5 + sdk/python/feast/repo_operations.py | 1 + sdk/python/feast/stream_feature_view.py | 231 +++++++++++++++++- .../integration/registration/test_registry.py | 69 +++++- .../test_stream_feature_view_apply.py | 62 +++++ sdk/python/tests/unit/test_feature_views.py | 60 ++++- 18 files changed, 761 insertions(+), 36 deletions(-) create mode 100644 protos/feast/core/Aggregation.proto create mode 100644 protos/feast/core/StreamFeatureView.proto create mode 100644 sdk/python/feast/aggregation.py create mode 100644 sdk/python/tests/integration/registration/test_stream_feature_view_apply.py diff --git a/protos/feast/core/Aggregation.proto b/protos/feast/core/Aggregation.proto new file mode 100644 index 00000000000..d848ce69721 --- /dev/null +++ b/protos/feast/core/Aggregation.proto @@ -0,0 +1,14 @@ +syntax = "proto3"; +package feast.core; + +option go_package = "github.com/feast-dev/feast/go/protos/feast/core"; +option java_outer_classname = "AggregationProto"; +option java_package = "feast.proto.core"; + +import "google/protobuf/duration.proto"; + +message Aggregation { + string column = 1; + string function = 2; + google.protobuf.Duration time_window = 3; +} \ No newline at end of file diff --git a/protos/feast/core/DataFormat.proto b/protos/feast/core/DataFormat.proto index 9fd01e865c2..c453e5e4c83 100644 --- a/protos/feast/core/DataFormat.proto +++ b/protos/feast/core/DataFormat.proto @@ -26,7 +26,7 @@ option java_package = "feast.proto.core"; message FileFormat { // Defines options for the Parquet data format message ParquetFormat {} - + oneof format { ParquetFormat parquet_format = 1; } @@ -40,7 +40,7 @@ message StreamFormat { // Feature data from the obtained stream message string class_path = 1; } - + // Defines options for the avro data format message AvroFormat { // Optional if used in a File DataSource as schema is embedded in avro file. @@ -48,9 +48,14 @@ message StreamFormat { string schema_json = 1; } + message JsonFormat { + string schema_json = 1; + } + // Specifies the data format and format specific options oneof format { AvroFormat avro_format = 1; ProtoFormat proto_format = 2; + JsonFormat json_format = 3; } } diff --git a/protos/feast/core/FeatureService.proto b/protos/feast/core/FeatureService.proto index 2654703cc59..51b9c6c02a2 100644 --- a/protos/feast/core/FeatureService.proto +++ b/protos/feast/core/FeatureService.proto @@ -5,7 +5,6 @@ option go_package = "github.com/feast-dev/feast/go/protos/feast/core"; option java_outer_classname = "FeatureServiceProto"; option java_package = "feast.proto.core"; -import "google/protobuf/duration.proto"; import "google/protobuf/timestamp.proto"; import "feast/core/FeatureViewProjection.proto"; diff --git a/protos/feast/core/Registry.proto b/protos/feast/core/Registry.proto index 2c31101510b..19f17a81589 100644 --- a/protos/feast/core/Registry.proto +++ b/protos/feast/core/Registry.proto @@ -28,12 +28,13 @@ import "feast/core/FeatureView.proto"; import "feast/core/InfraObject.proto"; import "feast/core/OnDemandFeatureView.proto"; import "feast/core/RequestFeatureView.proto"; +import "feast/core/StreamFeatureView.proto"; import "feast/core/DataSource.proto"; import "feast/core/SavedDataset.proto"; import "feast/core/ValidationProfile.proto"; import "google/protobuf/timestamp.proto"; -// Next id: 14 +// Next id: 15 message Registry { repeated Entity entities = 1; repeated FeatureTable feature_tables = 2; @@ -41,6 +42,7 @@ message Registry { repeated DataSource data_sources = 12; repeated OnDemandFeatureView on_demand_feature_views = 8; repeated RequestFeatureView request_feature_views = 9; + repeated StreamFeatureView stream_feature_views = 14; repeated FeatureService feature_services = 7; repeated SavedDataset saved_datasets = 11; repeated ValidationReference validation_references = 13; diff --git a/protos/feast/core/StreamFeatureView.proto b/protos/feast/core/StreamFeatureView.proto new file mode 100644 index 00000000000..3be9dc866af --- /dev/null +++ b/protos/feast/core/StreamFeatureView.proto @@ -0,0 +1,98 @@ +// +// Copyright 2020 The Feast Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + + +syntax = "proto3"; +package feast.core; + +option go_package = "github.com/feast-dev/feast/go/protos/feast/core"; +option java_outer_classname = "StreamFeatureViewProto"; +option java_package = "feast.proto.core"; + + +import "google/protobuf/duration.proto"; +import "google/protobuf/timestamp.proto"; +import "feast/core/OnDemandFeatureView.proto"; +import "feast/core/Feature.proto"; +import "feast/core/DataSource.proto"; +import "feast/core/Aggregation.proto"; + +message StreamFeatureView { + // User-specified specifications of this feature view. + StreamFeatureViewSpec spec = 1; + StreamFeatureViewMeta meta = 2; +} + +// Next available id: 17 +message StreamFeatureViewSpec { + // Name of the feature view. Must be unique. Not updated. + string name = 1; + + // Name of Feast project that this feature view belongs to. + string project = 2; + + // List of names of entities associated with this feature view. + repeated string entities = 3; + + // List of specifications for each feature defined as part of this feature view. + repeated FeatureSpecV2 features = 4; + + // List of specifications for each entity defined as part of this feature view. + repeated FeatureSpecV2 entity_columns = 5; + + // Description of the feature view. + string description = 6; + + // User defined metadata + map tags = 7; + + // Owner of the feature view. + string owner = 8; + + // Features in this feature view can only be retrieved from online serving + // younger than ttl. Ttl is measured as the duration of time between + // the feature's event timestamp and when the feature is retrieved + // Feature values outside ttl will be returned as unset values and indicated to end user + google.protobuf.Duration ttl = 9; + + // Batch/Offline DataSource where this view can retrieve offline feature data. + DataSource batch_source = 10; + // Streaming DataSource from where this view can consume "online" feature data. + DataSource stream_source = 11; + + // Whether these features should be served online or not + bool online = 12; + + // Serialized function that is encoded in the streamfeatureview + UserDefinedFunction user_defined_function = 13; + + // Mode of execution + string mode = 14; + + // Aggregation definitions + repeated Aggregation aggregations = 15; + + // Timestamp field for aggregation + string timestamp_field = 16; +} + +message StreamFeatureViewMeta { + // Time where this Feature View is created + google.protobuf.Timestamp created_timestamp = 1; + + // Time where this Feature View is last updated + google.protobuf.Timestamp last_updated_timestamp = 2; +} diff --git a/protos/feast/core/ValidationProfile.proto b/protos/feast/core/ValidationProfile.proto index b660e449bd2..10027995859 100644 --- a/protos/feast/core/ValidationProfile.proto +++ b/protos/feast/core/ValidationProfile.proto @@ -22,8 +22,6 @@ option java_package = "feast.proto.core"; option java_outer_classname = "ValidationProfile"; option go_package = "github.com/feast-dev/feast/go/protos/feast/core"; -import "feast/core/SavedDataset.proto"; - message GEValidationProfiler { message UserDefinedProfiler { // The python-syntax function body (serialized by dill) diff --git a/sdk/python/feast/aggregation.py b/sdk/python/feast/aggregation.py new file mode 100644 index 00000000000..0a5fe845659 --- /dev/null +++ b/sdk/python/feast/aggregation.py @@ -0,0 +1,69 @@ +from datetime import timedelta +from typing import Optional + +from google.protobuf.duration_pb2 import Duration + +from feast.protos.feast.core.Aggregation_pb2 import Aggregation as AggregationProto + + +class Aggregation: + """ + NOTE: Feast-handled aggregations are not yet supported. This class provides a way to register user-defined aggregations. + + Attributes: + column: str # Column name of the feature we are aggregating. + function: str # Provided built in aggregations sum, max, min, count mean + time_window: timedelta # The time window for this aggregation. + """ + + column: str + function: str + time_window: Optional[timedelta] + + def __init__( + self, + column: Optional[str] = "", + function: Optional[str] = "", + time_window: Optional[timedelta] = None, + ): + self.column = column or "" + self.function = function or "" + self.time_window = time_window + + def to_proto(self) -> AggregationProto: + window_duration = None + if self.time_window is not None: + window_duration = Duration() + window_duration.FromTimedelta(self.time_window) + + return AggregationProto( + column=self.column, function=self.function, time_window=window_duration + ) + + @classmethod + def from_proto(cls, agg_proto: AggregationProto): + time_window = ( + timedelta(days=0) + if agg_proto.time_window.ToNanoseconds() == 0 + else agg_proto.time_window.ToTimedelta() + ) + + aggregation = cls( + column=agg_proto.column, + function=agg_proto.function, + time_window=time_window, + ) + return aggregation + + def __eq__(self, other): + if not isinstance(other, Aggregation): + raise TypeError("Comparisons should only involve Aggregations.") + + if ( + self.column != other.column + or self.function != other.function + or self.time_window != other.time_window + ): + return False + + return True diff --git a/sdk/python/feast/data_format.py b/sdk/python/feast/data_format.py index b6c7bf94e92..8f3b195e3e6 100644 --- a/sdk/python/feast/data_format.py +++ b/sdk/python/feast/data_format.py @@ -89,6 +89,8 @@ def from_proto(cls, proto): fmt = proto.WhichOneof("format") if fmt == "avro_format": return AvroFormat(schema_json=proto.avro_format.schema_json) + if fmt == "json_format": + return JsonFormat(schema_json=proto.json_format.schema_json) if fmt == "proto_format": return ProtoFormat(class_path=proto.proto_format.class_path) raise NotImplementedError(f"StreamFormat is unsupported: {fmt}") @@ -113,6 +115,28 @@ def to_proto(self): return StreamFormatProto(avro_format=proto) +class JsonFormat(StreamFormat): + """ + Defines the Json streaming data format that encodes data in Json format + """ + + def __init__(self, schema_json: str): + """ + Construct a new Json data format. + + For spark, uses pyspark ddl string format. Example shown here: + https://vincent.doba.fr/posts/20211004_spark_data_description_language_for_defining_spark_schema/ + + Args: + schema_json: Json schema definition + """ + self.schema_json = schema_json + + def to_proto(self): + proto = StreamFormatProto.JsonFormat(schema_json=self.schema_json) + return StreamFormatProto(json_format=proto) + + class ProtoFormat(StreamFormat): """ Defines the Protobuf data format diff --git a/sdk/python/feast/data_source.py b/sdk/python/feast/data_source.py index 425143133ab..6f416e70d3d 100644 --- a/sdk/python/feast/data_source.py +++ b/sdk/python/feast/data_source.py @@ -410,6 +410,9 @@ def __init__( if _message_format is None: raise ValueError("Message format must be specified for Kafka source") + if not timestamp_field and not _event_timestamp_column: + raise ValueError("Timestamp field must be specified for Kafka source") + super().__init__( event_timestamp_column=_event_timestamp_column, created_timestamp_column=created_timestamp_column, diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py index 278ae077744..f959504826f 100644 --- a/sdk/python/feast/feature_store.py +++ b/sdk/python/feast/feature_store.py @@ -42,6 +42,7 @@ from feast import feature_server, flags, flags_helper, ui_server, utils from feast.base_feature_view import BaseFeatureView +from feast.batch_feature_view import BatchFeatureView from feast.data_source import DataSource from feast.diff.infra_diff import InfraDiff, diff_infra_protos from feast.diff.registry_diff import RegistryDiff, apply_diff_to_registry, diff_between @@ -84,6 +85,7 @@ from feast.repo_contents import RepoContents from feast.request_feature_view import RequestFeatureView from feast.saved_dataset import SavedDataset, SavedDatasetStorage, ValidationReference +from feast.stream_feature_view import StreamFeatureView from feast.type_map import ( feast_value_type_to_python_type, python_values_to_proto_values, @@ -273,6 +275,20 @@ def list_on_demand_feature_views( self.project, allow_cache=allow_cache ) + @log_exceptions_and_usage + def list_stream_feature_views( + self, allow_cache: bool = False + ) -> List[StreamFeatureView]: + """ + Retrieves the list of stream feature views from the registry. + + Returns: + A list of stream feature views. + """ + return self._registry.list_stream_feature_views( + self.project, allow_cache=allow_cache + ) + @log_exceptions_and_usage def list_data_sources(self, allow_cache: bool = False) -> List[DataSource]: """ @@ -457,6 +473,7 @@ def _validate_all_feature_views( views_to_update: List[FeatureView], odfvs_to_update: List[OnDemandFeatureView], request_views_to_update: List[RequestFeatureView], + sfvs_to_update: List[StreamFeatureView], ): """Validates all feature views.""" if ( @@ -468,7 +485,12 @@ def _validate_all_feature_views( set_usage_attribute("odfv", bool(odfvs_to_update)) _validate_feature_views( - [*views_to_update, *odfvs_to_update, *request_views_to_update] + [ + *views_to_update, + *odfvs_to_update, + *request_views_to_update, + *sfvs_to_update, + ] ) def _make_inferences( @@ -477,6 +499,7 @@ def _make_inferences( entities_to_update: List[Entity], views_to_update: List[FeatureView], odfvs_to_update: List[OnDemandFeatureView], + sfvs_to_update: List[StreamFeatureView], feature_services_to_update: List[FeatureService], ): """Makes inferences for entities, feature views, odfvs, and feature services.""" @@ -488,16 +511,28 @@ def _make_inferences( [view.batch_source for view in views_to_update], self.config ) + update_data_sources_with_inferred_event_timestamp_col( + [view.batch_source for view in sfvs_to_update], self.config + ) + # New feature views may reference previously applied entities. entities = self._list_entities() update_feature_views_with_inferred_features_and_entities( views_to_update, entities + entities_to_update, self.config ) + # TODO(kevjumba): Update schema inferrence + for sfv in sfvs_to_update: + if not sfv.schema: + raise ValueError( + f"schema inference not yet supported for stream feature views. please define schema for stream feature view: {sfv.name}" + ) for odfv in odfvs_to_update: odfv.infer_features() - fvs_to_update_map = {view.name: view for view in views_to_update} + fvs_to_update_map = { + view.name: view for view in [*views_to_update, *sfvs_to_update] + } for feature_service in feature_services_to_update: feature_service.infer_features(fvs_to_update=fvs_to_update_map) @@ -540,6 +575,7 @@ def _plan( ... data_sources=[driver_hourly_stats], ... feature_views=[driver_hourly_stats_view], ... on_demand_feature_views=list(), + ... stream_feature_views=list(), ... request_feature_views=list(), ... entities=[driver], ... feature_services=list())) # register entity and feature view @@ -549,6 +585,7 @@ def _plan( desired_repo_contents.feature_views, desired_repo_contents.on_demand_feature_views, desired_repo_contents.request_feature_views, + desired_repo_contents.stream_feature_views, ) _validate_data_sources(desired_repo_contents.data_sources) self._make_inferences( @@ -556,6 +593,7 @@ def _plan( desired_repo_contents.entities, desired_repo_contents.feature_views, desired_repo_contents.on_demand_feature_views, + desired_repo_contents.stream_feature_views, desired_repo_contents.feature_services, ) @@ -607,6 +645,7 @@ def apply( FeatureView, OnDemandFeatureView, RequestFeatureView, + StreamFeatureView, FeatureService, ValidationReference, List[FeastObject], @@ -661,7 +700,16 @@ def apply( # Separate all objects into entities, feature services, and different feature view types. entities_to_update = [ob for ob in objects if isinstance(ob, Entity)] - views_to_update = [ob for ob in objects if isinstance(ob, FeatureView)] + views_to_update = [ + ob + for ob in objects + if ( + isinstance(ob, FeatureView) + and not isinstance(ob, StreamFeatureView) + and not isinstance(ob, BatchFeatureView) + ) + ] + sfvs_to_update = [ob for ob in objects if isinstance(ob, StreamFeatureView)] request_views_to_update = [ ob for ob in objects if isinstance(ob, RequestFeatureView) ] @@ -674,7 +722,7 @@ def apply( ob for ob in objects if isinstance(ob, ValidationReference) ] - for fv in views_to_update: + for fv in itertools.chain(views_to_update, sfvs_to_update): data_sources_set_to_update.add(fv.batch_source) if fv.stream_source: data_sources_set_to_update.add(fv.stream_source) @@ -700,13 +748,14 @@ def apply( # Validate all feature views and make inferences. self._validate_all_feature_views( - views_to_update, odfvs_to_update, request_views_to_update + views_to_update, odfvs_to_update, request_views_to_update, sfvs_to_update ) self._make_inferences( data_sources_to_update, entities_to_update, views_to_update, odfvs_to_update, + sfvs_to_update, services_to_update, ) @@ -714,7 +763,7 @@ def apply( for ds in data_sources_to_update: self._registry.apply_data_source(ds, project=self.project, commit=False) for view in itertools.chain( - views_to_update, odfvs_to_update, request_views_to_update + views_to_update, odfvs_to_update, request_views_to_update, sfvs_to_update ): self._registry.apply_feature_view(view, project=self.project, commit=False) for ent in entities_to_update: @@ -742,6 +791,9 @@ def apply( odfvs_to_delete = [ ob for ob in objects_to_delete if isinstance(ob, OnDemandFeatureView) ] + sfvs_to_delete = [ + ob for ob in objects_to_delete if isinstance(ob, StreamFeatureView) + ] services_to_delete = [ ob for ob in objects_to_delete if isinstance(ob, FeatureService) ] @@ -772,6 +824,10 @@ def apply( self._registry.delete_feature_view( odfv.name, project=self.project, commit=False ) + for sfv in sfvs_to_delete: + self._registry.delete_feature_view( + sfv.name, project=self.project, commit=False + ) for service in services_to_delete: self._registry.delete_feature_service( service.name, project=self.project, commit=False @@ -879,6 +935,7 @@ def get_historical_features( all_feature_views, all_request_feature_views, all_on_demand_feature_views, + all_stream_feature_views, ) = self._get_feature_views_to_use(features) if all_request_feature_views: @@ -1153,7 +1210,6 @@ def materialize( Examples: Materialize all features into the online store over the interval from 3 hours ago to 10 minutes ago. - >>> from feast import FeatureStore, RepoConfig >>> from datetime import datetime, timedelta >>> fs = FeatureStore(repo_path="feature_repo") @@ -1223,6 +1279,7 @@ def push( ): """ Push features to a push source. This updates all the feature views that have the push source as stream source. + Args: push_source_name: The name of the push source we want to push data to. df: the data being pushed. @@ -1399,6 +1456,7 @@ def _get_online_features( requested_feature_views, requested_request_feature_views, requested_on_demand_feature_views, + request_stream_feature_views, ) = self._get_feature_views_to_use( features=features, allow_cache=True, hide_dummy_entity=False ) @@ -1935,7 +1993,12 @@ def _get_feature_views_to_use( features: Optional[Union[List[str], FeatureService]], allow_cache=False, hide_dummy_entity: bool = True, - ) -> Tuple[List[FeatureView], List[RequestFeatureView], List[OnDemandFeatureView]]: + ) -> Tuple[ + List[FeatureView], + List[RequestFeatureView], + List[OnDemandFeatureView], + List[StreamFeatureView], + ]: fvs = { fv.name: fv @@ -1956,8 +2019,15 @@ def _get_feature_views_to_use( ) } + sfvs = { + fv.name: fv + for fv in self._registry.list_stream_feature_views( + project=self.project, allow_cache=allow_cache + ) + } + if isinstance(features, FeatureService): - fvs_to_use, request_fvs_to_use, od_fvs_to_use = [], [], [] + fvs_to_use, request_fvs_to_use, od_fvs_to_use, sfvs_to_use = [], [], [], [] for fv_name, projection in [ (projection.name, projection) for projection in features.feature_view_projections @@ -1978,18 +2048,23 @@ def _get_feature_views_to_use( fv = fvs[projection.name].with_projection(copy.copy(projection)) if fv not in fvs_to_use: fvs_to_use.append(fv) + elif fv_name in sfvs: + sfvs_to_use.append( + sfvs[fv_name].with_projection(copy.copy(projection)) + ) else: raise ValueError( f"The provided feature service {features.name} contains a reference to a feature view" f"{fv_name} which doesn't exist. Please make sure that you have created the feature view" f'{fv_name} and that you have registered it by running "apply".' ) - views_to_use = (fvs_to_use, request_fvs_to_use, od_fvs_to_use) + views_to_use = (fvs_to_use, request_fvs_to_use, od_fvs_to_use, sfvs_to_use) else: views_to_use = ( [*fvs.values()], [*request_fvs.values()], [*od_fvs.values()], + [*sfvs.values()], ) return views_to_use diff --git a/sdk/python/feast/inference.py b/sdk/python/feast/inference.py index aed90c4ac83..37f0cb8b05e 100644 --- a/sdk/python/feast/inference.py +++ b/sdk/python/feast/inference.py @@ -1,5 +1,5 @@ import re -from typing import List, Set +from typing import List, Set, Union from feast.data_source import DataSource, PushSource, RequestSource from feast.entity import Entity @@ -11,6 +11,7 @@ from feast.infra.offline_stores.redshift_source import RedshiftSource from feast.infra.offline_stores.snowflake_source import SnowflakeSource from feast.repo_config import RepoConfig +from feast.stream_feature_view import StreamFeatureView from feast.types import String from feast.value_type import ValueType @@ -19,7 +20,6 @@ def update_data_sources_with_inferred_event_timestamp_col( data_sources: List[DataSource], config: RepoConfig ) -> None: ERROR_MSG_PREFIX = "Unable to infer DataSource timestamp_field" - for data_source in data_sources: if isinstance(data_source, RequestSource): continue @@ -88,7 +88,9 @@ def update_data_sources_with_inferred_event_timestamp_col( def update_feature_views_with_inferred_features_and_entities( - fvs: List[FeatureView], entities: List[Entity], config: RepoConfig + fvs: Union[List[FeatureView], List[StreamFeatureView]], + entities: List[Entity], + config: RepoConfig, ) -> None: """ Infers the features and entities associated with each feature view and updates it in place. diff --git a/sdk/python/feast/registry.py b/sdk/python/feast/registry.py index c46eba8a5d6..7f298b19b82 100644 --- a/sdk/python/feast/registry.py +++ b/sdk/python/feast/registry.py @@ -51,6 +51,7 @@ from feast.repo_contents import RepoContents from feast.request_feature_view import RequestFeatureView from feast.saved_dataset import SavedDataset, ValidationReference +from feast.stream_feature_view import StreamFeatureView REGISTRY_SCHEMA_VERSION = "1" @@ -476,7 +477,11 @@ def apply_feature_view( self._check_conflicting_feature_view_names(feature_view) existing_feature_views_of_same_type: RepeatedCompositeFieldContainer - if isinstance(feature_view, FeatureView): + if isinstance(feature_view, StreamFeatureView): + existing_feature_views_of_same_type = ( + self.cached_registry_proto.stream_feature_views + ) + elif isinstance(feature_view, FeatureView): existing_feature_views_of_same_type = ( self.cached_registry_proto.feature_views ) @@ -506,11 +511,32 @@ def apply_feature_view( else: del existing_feature_views_of_same_type[idx] break - existing_feature_views_of_same_type.append(feature_view_proto) if commit: self.commit() + def list_stream_feature_views( + self, project: str, allow_cache: bool = False + ) -> List[StreamFeatureView]: + """ + Retrieve a list of stream feature views from the registry + + Args: + project: Filter stream feature views based on project name + allow_cache: Whether to allow returning stream feature views from a cached registry + + Returns: + List of stream feature views + """ + registry = self._get_registry_proto(allow_cache=allow_cache) + stream_feature_views = [] + for stream_feature_view in registry.stream_feature_views: + if stream_feature_view.spec.project == project: + stream_feature_views.append( + StreamFeatureView.from_proto(stream_feature_view) + ) + return stream_feature_views + def list_on_demand_feature_views( self, project: str, allow_cache: bool = False ) -> List[OnDemandFeatureView]: @@ -764,6 +790,18 @@ def delete_feature_view(self, name: str, project: str, commit: bool = True): self.commit() return + for idx, existing_stream_feature_view_proto in enumerate( + self.cached_registry_proto.stream_feature_views + ): + if ( + existing_stream_feature_view_proto.spec.name == name + and existing_stream_feature_view_proto.spec.project == project + ): + del self.cached_registry_proto.stream_feature_views[idx] + if commit: + self.commit() + return + raise FeatureViewNotFoundException(name, project) def delete_entity(self, name: str, project: str, commit: bool = True): diff --git a/sdk/python/feast/repo_contents.py b/sdk/python/feast/repo_contents.py index 4d7c92f2a6d..fe5cbd284bc 100644 --- a/sdk/python/feast/repo_contents.py +++ b/sdk/python/feast/repo_contents.py @@ -20,6 +20,7 @@ from feast.on_demand_feature_view import OnDemandFeatureView from feast.protos.feast.core.Registry_pb2 import Registry as RegistryProto from feast.request_feature_view import RequestFeatureView +from feast.stream_feature_view import StreamFeatureView class RepoContents(NamedTuple): @@ -31,6 +32,7 @@ class RepoContents(NamedTuple): feature_views: List[FeatureView] on_demand_feature_views: List[OnDemandFeatureView] request_feature_views: List[RequestFeatureView] + stream_feature_views: List[StreamFeatureView] entities: List[Entity] feature_services: List[FeatureService] @@ -50,4 +52,7 @@ def to_registry_proto(self) -> RegistryProto: registry_proto.feature_services.extend( [fs.to_proto() for fs in self.feature_services] ) + registry_proto.stream_feature_views.extend( + [fv.to_proto() for fv in self.stream_feature_views] + ) return registry_proto diff --git a/sdk/python/feast/repo_operations.py b/sdk/python/feast/repo_operations.py index 0e82fdf47ad..8b81a71bae4 100644 --- a/sdk/python/feast/repo_operations.py +++ b/sdk/python/feast/repo_operations.py @@ -107,6 +107,7 @@ def parse_repo(repo_root: Path) -> RepoContents: feature_views=[], feature_services=[], on_demand_feature_views=[], + stream_feature_views=[], request_feature_views=[], ) diff --git a/sdk/python/feast/stream_feature_view.py b/sdk/python/feast/stream_feature_view.py index cfb3f63d7df..bba16e2627f 100644 --- a/sdk/python/feast/stream_feature_view.py +++ b/sdk/python/feast/stream_feature_view.py @@ -1,16 +1,42 @@ +import functools +import warnings from datetime import timedelta +from types import MethodType from typing import Dict, List, Optional, Union -from feast.data_source import DataSource +import dill +from google.protobuf.duration_pb2 import Duration + +from feast.aggregation import Aggregation +from feast.data_source import DataSource, KafkaSource from feast.entity import Entity from feast.feature_view import FeatureView from feast.field import Field from feast.protos.feast.core.DataSource_pb2 import DataSource as DataSourceProto +from feast.protos.feast.core.OnDemandFeatureView_pb2 import ( + UserDefinedFunction as UserDefinedFunctionProto, +) +from feast.protos.feast.core.StreamFeatureView_pb2 import ( + StreamFeatureView as StreamFeatureViewProto, +) +from feast.protos.feast.core.StreamFeatureView_pb2 import ( + StreamFeatureViewMeta as StreamFeatureViewMetaProto, +) +from feast.protos.feast.core.StreamFeatureView_pb2 import ( + StreamFeatureViewSpec as StreamFeatureViewSpecProto, +) + +warnings.simplefilter("once", RuntimeWarning) -SUPPORTED_STREAM_SOURCES = {"KafkaSource", "KinesisSource", "PushSource"} +SUPPORTED_STREAM_SOURCES = {"KafkaSource", "PushSource"} class StreamFeatureView(FeatureView): + """ + NOTE: Stream Feature Views are not yet fully implemented and exist to allow users to register their stream sources and + schemas with Feast. + """ + def __init__( self, *, @@ -18,15 +44,24 @@ def __init__( entities: Optional[Union[List[Entity], List[str]]] = None, ttl: Optional[timedelta] = None, tags: Optional[Dict[str, str]] = None, - online: bool = True, - description: str = "", - owner: str = "", + online: Optional[bool] = True, + description: Optional[str] = "", + owner: Optional[str] = "", schema: Optional[List[Field]] = None, source: Optional[DataSource] = None, + aggregations: Optional[List[Aggregation]] = None, + mode: Optional[str] = "spark", # Mode of ingestion/transformation + timestamp_field: Optional[str] = "", # Timestamp for aggregation + udf: Optional[MethodType] = None, ): - + warnings.warn( + "Stream Feature Views are experimental features in alpha development. " + "Some functionality may still be unstable so functionality can change in the future.", + RuntimeWarning, + ) if source is None: - raise ValueError("Feature views need a source specified") + raise ValueError("Stream Feature views need a source specified") + # source uses the batch_source of the kafkasource in feature_view if ( type(source).__name__ not in SUPPORTED_STREAM_SOURCES and source.to_proto().type != DataSourceProto.SourceType.CUSTOM_SOURCE @@ -35,13 +70,20 @@ def __init__( f"Stream feature views need a stream source, expected one of {SUPPORTED_STREAM_SOURCES} " f"or CUSTOM_SOURCE, got {type(source).__name__}: {source.name} instead " ) + self.aggregations = aggregations + self.mode = mode + self.timestamp_field = timestamp_field + self.udf = udf + _batch_source = None + if isinstance(source, KafkaSource): + _batch_source = source.batch_source if source.batch_source else None super().__init__( name=name, entities=entities, ttl=ttl, - batch_source=None, - stream_source=None, + batch_source=_batch_source, + stream_source=source, tags=tags, online=online, description=description, @@ -49,3 +91,174 @@ def __init__( schema=schema, source=source, ) + + def __eq__(self, other): + if not isinstance(other, StreamFeatureView): + raise TypeError("Comparisons should only involve StreamFeatureViews") + + if not super().__eq__(other): + return False + + if ( + self.mode != other.mode + or self.timestamp_field != other.timestamp_field + or self.udf.__code__.co_code != other.udf.__code__.co_code + or self.aggregations != other.aggregations + ): + return False + + return True + + def __hash__(self): + return super().__hash__() + + def to_proto(self): + meta = StreamFeatureViewMetaProto() + if self.created_timestamp: + meta.created_timestamp.FromDatetime(self.created_timestamp) + if self.last_updated_timestamp: + meta.last_updated_timestamp.FromDatetime(self.last_updated_timestamp) + + ttl_duration = None + if self.ttl is not None: + ttl_duration = Duration() + ttl_duration.FromTimedelta(self.ttl) + + if self.batch_source: + batch_source_proto = self.batch_source.to_proto() + batch_source_proto.data_source_class_type = f"{self.batch_source.__class__.__module__}.{self.batch_source.__class__.__name__}" + + stream_source_proto = None + if self.stream_source: + stream_source_proto = self.stream_source.to_proto() + stream_source_proto.data_source_class_type = f"{self.stream_source.__class__.__module__}.{self.stream_source.__class__.__name__}" + + spec = StreamFeatureViewSpecProto( + name=self.name, + entities=self.entities, + entity_columns=[field.to_proto() for field in self.entity_columns], + features=[field.to_proto() for field in self.schema], + user_defined_function=UserDefinedFunctionProto( + name=self.udf.__name__, body=dill.dumps(self.udf, recurse=True), + ) + if self.udf + else None, + description=self.description, + tags=self.tags, + owner=self.owner, + ttl=(ttl_duration if ttl_duration is not None else None), + online=self.online, + batch_source=batch_source_proto or None, + stream_source=stream_source_proto, + timestamp_field=self.timestamp_field, + aggregations=[agg.to_proto() for agg in self.aggregations], + mode=self.mode, + ) + + return StreamFeatureViewProto(spec=spec, meta=meta) + + @classmethod + def from_proto(cls, sfv_proto): + batch_source = ( + DataSource.from_proto(sfv_proto.spec.batch_source) + if sfv_proto.spec.HasField("batch_source") + else None + ) + stream_source = ( + DataSource.from_proto(sfv_proto.spec.stream_source) + if sfv_proto.spec.HasField("stream_source") + else None + ) + sfv_feature_view = cls( + name=sfv_proto.spec.name, + description=sfv_proto.spec.description, + tags=dict(sfv_proto.spec.tags), + owner=sfv_proto.spec.owner, + online=sfv_proto.spec.online, + schema=[ + Field.from_proto(field_proto) for field_proto in sfv_proto.spec.features + ], + ttl=( + timedelta(days=0) + if sfv_proto.spec.ttl.ToNanoseconds() == 0 + else sfv_proto.spec.ttl.ToTimedelta() + ), + source=stream_source, + mode=sfv_proto.spec.mode, + udf=dill.loads(sfv_proto.spec.user_defined_function.body), + aggregations=[ + Aggregation.from_proto(agg_proto) + for agg_proto in sfv_proto.spec.aggregations + ], + timestamp_field=sfv_proto.spec.timestamp_field, + ) + + if batch_source: + sfv_feature_view.batch_source = batch_source + + if stream_source: + sfv_feature_view.stream_source = stream_source + + sfv_feature_view.entities = list(sfv_proto.spec.entities) + + sfv_feature_view.features = [ + Field.from_proto(field_proto) for field_proto in sfv_proto.spec.features + ] + + if sfv_proto.meta.HasField("created_timestamp"): + sfv_feature_view.created_timestamp = ( + sfv_proto.meta.created_timestamp.ToDatetime() + ) + if sfv_proto.meta.HasField("last_updated_timestamp"): + sfv_feature_view.last_updated_timestamp = ( + sfv_proto.meta.last_updated_timestamp.ToDatetime() + ) + + return sfv_feature_view + + +def stream_feature_view( + *, + entities: Optional[Union[List[Entity], List[str]]] = None, + ttl: Optional[timedelta] = None, + tags: Optional[Dict[str, str]] = None, + online: Optional[bool] = True, + description: Optional[str] = "", + owner: Optional[str] = "", + schema: Optional[List[Field]] = None, + source: Optional[DataSource] = None, + aggregations: Optional[List[Aggregation]] = None, + mode: Optional[str] = "spark", # Mode of ingestion/transformation + timestamp_field: Optional[str] = "", # Timestamp for aggregation +): + """ + Creates an StreamFeatureView object with the given user function as udf. + """ + + def mainify(obj): + # Needed to allow dill to properly serialize the udf. Otherwise, clients will need to have a file with the same + # name as the original file defining the sfv. + if obj.__module__ != "__main__": + obj.__module__ = "__main__" + + def decorator(user_function): + mainify(user_function) + stream_feature_view_obj = StreamFeatureView( + name=user_function.__name__, + entities=entities, + ttl=ttl, + source=source, + schema=schema, + udf=user_function, + description=description, + tags=tags, + online=online, + owner=owner, + aggregations=aggregations, + mode=mode, + timestamp_field=timestamp_field, + ) + functools.update_wrapper(wrapper=stream_feature_view_obj, wrapped=user_function) + return stream_feature_view_obj + + return decorator diff --git a/sdk/python/tests/integration/registration/test_registry.py b/sdk/python/tests/integration/registration/test_registry.py index bb02f9a9e32..222eb116d26 100644 --- a/sdk/python/tests/integration/registration/test_registry.py +++ b/sdk/python/tests/integration/registration/test_registry.py @@ -20,7 +20,9 @@ from pytest_lazyfixture import lazy_fixture from feast import FileSource -from feast.data_format import ParquetFormat +from feast.aggregation import Aggregation +from feast.data_format import AvroFormat, ParquetFormat +from feast.data_source import KafkaSource from feast.entity import Entity from feast.feature import Feature from feast.feature_view import FeatureView @@ -28,6 +30,7 @@ from feast.on_demand_feature_view import RequestSource, on_demand_feature_view from feast.registry import Registry from feast.repo_config import RegistryConfig +from feast.stream_feature_view import StreamFeatureView from feast.types import Array, Bytes, Float32, Int32, Int64, String from feast.value_type import ValueType @@ -299,6 +302,70 @@ def location_features_from_push(inputs: pd.DataFrame) -> pd.DataFrame: test_registry._get_registry_proto() +@pytest.mark.parametrize( + "test_registry", [lazy_fixture("local_registry")], +) +def test_apply_stream_feature_view_success(test_registry): + # Create Feature Views + def simple_udf(x: int): + return x + 3 + + entity = Entity(name="driver_entity", join_keys=["test_key"]) + + stream_source = KafkaSource( + name="kafka", + timestamp_field="event_timestamp", + bootstrap_servers="", + message_format=AvroFormat(""), + topic="topic", + batch_source=FileSource(path="some path"), + ) + + sfv = StreamFeatureView( + name="test kafka stream feature view", + entities=[entity], + ttl=timedelta(days=30), + owner="test@example.com", + online=True, + schema=[Field(name="dummy_field", dtype=Float32)], + description="desc", + aggregations=[ + Aggregation( + column="dummy_field", function="max", time_window=timedelta(days=1), + ), + Aggregation( + column="dummy_field2", function="count", time_window=timedelta(days=24), + ), + ], + timestamp_field="event_timestamp", + mode="spark", + source=stream_source, + udf=simple_udf, + tags={}, + ) + + project = "project" + + # Register Feature View + test_registry.apply_feature_view(sfv, project) + + stream_feature_views = test_registry.list_stream_feature_views(project) + + # List Feature Views + assert len(stream_feature_views) == 1 + assert stream_feature_views[0] == sfv + + test_registry.delete_feature_view("test kafka stream feature view", project) + stream_feature_views = test_registry.list_stream_feature_views(project) + assert len(stream_feature_views) == 0 + + test_registry.teardown() + + # Will try to reload registry, which will fail because the file has been deleted + with pytest.raises(FileNotFoundError): + test_registry._get_registry_proto() + + @pytest.mark.parametrize( "test_registry", [lazy_fixture("local_registry")], ) diff --git a/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py b/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py new file mode 100644 index 00000000000..b01ca434fa8 --- /dev/null +++ b/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py @@ -0,0 +1,62 @@ +from datetime import timedelta + +import pytest + +from feast import Entity, Field, FileSource +from feast.aggregation import Aggregation +from feast.data_format import AvroFormat +from feast.data_source import KafkaSource +from feast.stream_feature_view import stream_feature_view +from feast.types import Float32 + + +@pytest.mark.integration +def test_read_pre_applied(environment) -> None: + """ + Test apply of StreamFeatureView. + """ + fs = environment.feature_store + + # Create Feature Views + entity = Entity(name="driver_entity", join_keys=["test_key"]) + + stream_source = KafkaSource( + name="kafka", + timestamp_field="event_timestamp", + bootstrap_servers="", + message_format=AvroFormat(""), + topic="topic", + batch_source=FileSource(path="test_path", timestamp_field="event_timestamp"), + ) + + @stream_feature_view( + entities=[entity], + ttl=timedelta(days=30), + owner="test@example.com", + online=True, + schema=[Field(name="dummy_field", dtype=Float32)], + description="desc", + aggregations=[ + Aggregation( + column="dummy_field", function="max", time_window=timedelta(days=1), + ), + Aggregation( + column="dummy_field2", function="count", time_window=timedelta(days=24), + ), + ], + timestamp_field="event_timestamp", + mode="spark", + source=stream_source, + tags={}, + ) + def simple_sfv(df): + return df + + fs.apply([entity, simple_sfv]) + stream_feature_views = fs.list_stream_feature_views() + assert len(stream_feature_views) == 1 + assert stream_feature_views[0] == simple_sfv + + entities = fs.list_entities() + assert len(entities) == 1 + assert entities[0] == entity diff --git a/sdk/python/tests/unit/test_feature_views.py b/sdk/python/tests/unit/test_feature_views.py index f72ae4fe9cb..904260dfe61 100644 --- a/sdk/python/tests/unit/test_feature_views.py +++ b/sdk/python/tests/unit/test_feature_views.py @@ -2,12 +2,15 @@ import pytest -from feast import PushSource +from feast.aggregation import Aggregation from feast.batch_feature_view import BatchFeatureView from feast.data_format import AvroFormat -from feast.data_source import KafkaSource +from feast.data_source import KafkaSource, PushSource +from feast.entity import Entity +from feast.field import Field from feast.infra.offline_stores.file_source import FileSource from feast.stream_feature_view import StreamFeatureView +from feast.types import Float32 def test_create_batch_feature_view(): @@ -26,7 +29,7 @@ def test_create_batch_feature_view(): stream_source = KafkaSource( name="kafka", - timestamp_field="", + timestamp_field="event_timestamp", bootstrap_servers="", message_format=AvroFormat(""), topic="topic", @@ -44,7 +47,7 @@ def test_create_batch_feature_view(): def test_create_stream_feature_view(): stream_source = KafkaSource( name="kafka", - timestamp_field="", + timestamp_field="event_timestamp", bootstrap_servers="", message_format=AvroFormat(""), topic="topic", @@ -55,6 +58,7 @@ def test_create_stream_feature_view(): entities=[], ttl=timedelta(days=30), source=stream_source, + aggregations=[], ) push_source = PushSource( @@ -65,11 +69,15 @@ def test_create_stream_feature_view(): entities=[], ttl=timedelta(days=30), source=push_source, + aggregations=[], ) with pytest.raises(ValueError): StreamFeatureView( - name="test batch feature view", entities=[], ttl=timedelta(days=30) + name="test batch feature view", + entities=[], + ttl=timedelta(days=30), + aggregations=[], ) with pytest.raises(ValueError): @@ -78,4 +86,46 @@ def test_create_stream_feature_view(): entities=[], ttl=timedelta(days=30), source=FileSource(path="some path"), + aggregations=[], ) + + +def simple_udf(x: int): + return x + 3 + + +def test_stream_feature_view_serialization(): + entity = Entity(name="driver_entity", join_keys=["test_key"]) + stream_source = KafkaSource( + name="kafka", + timestamp_field="event_timestamp", + bootstrap_servers="", + message_format=AvroFormat(""), + topic="topic", + batch_source=FileSource(path="some path"), + ) + + sfv = StreamFeatureView( + name="test kafka stream feature view", + entities=[entity], + ttl=timedelta(days=30), + owner="test@example.com", + online=True, + schema=[Field(name="dummy_field", dtype=Float32)], + description="desc", + aggregations=[ + Aggregation( + column="dummy_field", function="max", time_window=timedelta(days=1), + ) + ], + timestamp_field="event_timestamp", + mode="spark", + source=stream_source, + udf=simple_udf, + tags={}, + ) + + sfv_proto = sfv.to_proto() + + new_sfv = StreamFeatureView.from_proto(sfv_proto=sfv_proto) + assert new_sfv == sfv From e9784a1a1a8d5a676995fcf86072c384685f62a8 Mon Sep 17 00:00:00 2001 From: Abhin Chhabra Date: Thu, 2 Jun 2022 10:15:54 -0400 Subject: [PATCH 034/123] chore: Unpin `googleapis-common-protos` (#2745) * Unpin `googleapis-common-protos` Exact pinning this library can cause Feast to be incompatible with other libraries that users may want to install. Signed-off-by: Abhin Chhabra * Regenerate requirements files. Signed-off-by: Abhin Chhabra * Fixing requirements Signed-off-by: Danny Chiao * Fix merge Signed-off-by: Danny Chiao Co-authored-by: Danny Chiao --- .../requirements/py3.10-ci-requirements.txt | 89 +++++++-------- .../requirements/py3.10-requirements.txt | 31 +++-- .../requirements/py3.7-ci-requirements.txt | 76 ++++++------- .../requirements/py3.7-requirements.txt | 28 ++--- .../requirements/py3.8-ci-requirements.txt | 79 +++++++------ .../requirements/py3.8-requirements.txt | 26 ++--- .../requirements/py3.9-ci-requirements.txt | 107 ++++++++++-------- .../requirements/py3.9-requirements.txt | 26 ++--- setup.py | 19 ++-- 9 files changed, 240 insertions(+), 241 deletions(-) diff --git a/sdk/python/requirements/py3.10-ci-requirements.txt b/sdk/python/requirements/py3.10-ci-requirements.txt index e4b7e5447be..0098c057972 100644 --- a/sdk/python/requirements/py3.10-ci-requirements.txt +++ b/sdk/python/requirements/py3.10-ci-requirements.txt @@ -4,7 +4,7 @@ # # pip-compile --extra=ci --output-file=sdk/python/requirements/py3.10-ci-requirements.txt # -absl-py==1.0.0 +absl-py==1.1.0 # via tensorflow-metadata adal==1.2.7 # via @@ -36,7 +36,7 @@ appdirs==1.4.4 # via black appnope==0.1.3 # via ipython -asgiref==3.5.1 +asgiref==3.5.2 # via uvicorn asn1crypto==1.5.1 # via @@ -58,7 +58,7 @@ attrs==21.4.0 # pytest avro==1.10.0 # via feast (setup.py) -azure-core==1.24.0 +azure-core==1.24.1 # via # adlfs # azure-identity @@ -85,13 +85,13 @@ botocore==1.23.24 # boto3 # moto # s3transfer -build==0.7.0 +build==0.8.0 # via feast (setup.py) cachecontrol==0.12.11 # via firebase-admin cachetools==4.2.4 # via google-auth -certifi==2021.10.8 +certifi==2022.5.18.1 # via # minio # msrest @@ -116,15 +116,15 @@ click==8.0.1 # great-expectations # pip-tools # uvicorn -cloudpickle==2.0.0 +cloudpickle==2.1.0 # via dask colorama==0.4.4 # via # feast (setup.py) # great-expectations -coverage[toml]==6.3.3 +coverage[toml]==6.4 # via pytest-cov -cryptography==3.4.8 +cryptography==35.0.0 # via # adal # azure-identity @@ -147,7 +147,7 @@ deprecated==1.2.13 # via redis deprecation==2.1.0 # via testcontainers -dill==0.3.4 +dill==0.3.5.1 # via feast (setup.py) distlib==0.3.4 # via virtualenv @@ -167,13 +167,13 @@ executing==0.8.3 # via stack-data fastapi==0.78.0 # via feast (setup.py) -fastavro==1.4.11 +fastavro==1.4.12 # via # feast (setup.py) # pandavro fastjsonschema==2.15.3 # via nbformat -filelock==3.7.0 +filelock==3.7.1 # via virtualenv firebase-admin==4.5.2 # via feast (setup.py) @@ -189,9 +189,9 @@ fsspec==2022.1.0 # dask # gcsfs # s3fs -gcsfs==0.8.0 +gcsfs==2022.1.0 # via feast (setup.py) -google-api-core[grpc]==1.31.5 +google-api-core[grpc]==1.31.6 # via # feast (setup.py) # firebase-admin @@ -201,7 +201,7 @@ google-api-core[grpc]==1.31.5 # google-cloud-core # google-cloud-datastore # google-cloud-firestore -google-api-python-client==2.47.0 +google-api-python-client==2.49.0 # via firebase-admin google-auth==1.35.0 # via @@ -227,28 +227,29 @@ google-cloud-core==1.7.2 # google-cloud-datastore # google-cloud-firestore # google-cloud-storage -google-cloud-datastore==2.5.1 +google-cloud-datastore==2.6.1 # via feast (setup.py) -google-cloud-firestore==2.4.0 +google-cloud-firestore==2.5.2 # via firebase-admin google-cloud-storage==1.40.0 # via # feast (setup.py) # firebase-admin + # gcsfs google-crc32c==1.3.0 # via google-resumable-media google-resumable-media==1.3.3 # via # google-cloud-bigquery # google-cloud-storage -googleapis-common-protos==1.52.0 +googleapis-common-protos==1.56.2 # via # feast (setup.py) # google-api-core # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) -grpcio==1.46.1 +grpcio==1.46.3 # via # feast (setup.py) # google-api-core @@ -256,7 +257,7 @@ grpcio==1.46.1 # grpcio-reflection # grpcio-testing # grpcio-tools -grpcio-reflection==1.46.1 +grpcio-reflection==1.46.3 # via feast (setup.py) grpcio-testing==1.44.0 # via feast (setup.py) @@ -274,7 +275,7 @@ httplib2==0.20.4 # google-auth-httplib2 httptools==0.4.0 # via uvicorn -identify==2.5.0 +identify==2.5.1 # via pre-commit idna==3.3 # via @@ -284,11 +285,11 @@ idna==3.3 # yarl imagesize==1.3.0 # via sphinx -importlib-metadata==4.11.3 +importlib-metadata==4.11.4 # via great-expectations iniconfig==1.1.1 # via pytest -ipython==8.3.0 +ipython==8.4.0 # via great-expectations isodate==0.6.1 # via msrest @@ -311,7 +312,7 @@ jsonpatch==1.32 # via great-expectations jsonpointer==2.3 # via jsonpatch -jsonschema==4.5.1 +jsonschema==4.6.0 # via # altair # feast (setup.py) @@ -337,9 +338,9 @@ mmh3==3.0.0 # via feast (setup.py) mock==2.0.0 # via feast (setup.py) -moto==3.1.9 +moto==3.1.11 # via feast (setup.py) -msal==1.17.0 +msal==1.18.0 # via # azure-identity # msal-extensions @@ -416,7 +417,7 @@ pexpect==4.8.0 # via ipython pickleshare==0.7.5 # via ipython -pip-tools==6.6.1 +pip-tools==6.6.2 # via feast (setup.py) platformdirs==2.5.2 # via virtualenv @@ -430,7 +431,7 @@ pre-commit==2.19.0 # via feast (setup.py) prompt-toolkit==3.0.29 # via ipython -proto-plus==1.19.6 +proto-plus==1.20.5 # via # feast (setup.py) # google-cloud-bigquery @@ -442,6 +443,7 @@ protobuf==3.19.4 # feast (setup.py) # google-api-core # google-cloud-bigquery + # google-cloud-firestore # googleapis-common-protos # grpcio-reflection # grpcio-testing @@ -483,7 +485,7 @@ pycparser==2.21 # via cffi pycryptodomex==3.14.1 # via snowflake-connector-python -pydantic==1.9.0 +pydantic==1.9.1 # via # fastapi # feast (setup.py) @@ -499,7 +501,7 @@ pyjwt[crypto]==2.4.0 # adal # msal # snowflake-connector-python -pyopenssl==21.0.0 +pyopenssl==22.0.0 # via snowflake-connector-python pyparsing==2.4.7 # via @@ -593,7 +595,7 @@ requests-oauthlib==1.3.1 # via # google-auth-oauthlib # msrest -responses==0.20.0 +responses==0.21.0 # via moto rsa==4.8 # via google-auth @@ -603,11 +605,10 @@ s3fs==2022.1.0 # via feast (setup.py) s3transfer==0.5.2 # via boto3 -scipy==1.8.0 +scipy==1.8.1 # via great-expectations six==1.16.0 # via - # absl-py # azure-core # azure-identity # google-api-core @@ -620,14 +621,13 @@ six==1.16.0 # mock # msrestazure # pandavro - # pyopenssl # python-dateutil # virtualenv sniffio==1.2.0 # via anyio snowballstemmer==2.2.0 # via sphinx -snowflake-connector-python[pandas]==2.7.7 +snowflake-connector-python[pandas]==2.7.8 # via feast (setup.py) sphinx==4.3.2 # via @@ -684,7 +684,7 @@ tqdm==4.64.0 # via # feast (setup.py) # great-expectations -traitlets==5.2.1.post0 +traitlets==5.2.2.post1 # via # ipython # jupyter-core @@ -692,27 +692,27 @@ traitlets==5.2.1.post0 # nbformat trino==0.313.0 # via feast (setup.py) -typed-ast==1.5.3 +typed-ast==1.5.4 # via black -types-protobuf==3.19.20 +types-protobuf==3.19.21 # via # feast (setup.py) # mypy-protobuf -types-python-dateutil==2.8.15 +types-python-dateutil==2.8.17 # via feast (setup.py) types-pytz==2021.3.8 # via feast (setup.py) -types-pyyaml==6.0.7 +types-pyyaml==6.0.8 # via feast (setup.py) -types-redis==4.2.3 +types-redis==4.2.6 # via feast (setup.py) -types-requests==2.27.25 +types-requests==2.27.30 # via feast (setup.py) -types-setuptools==57.4.14 +types-setuptools==57.4.17 # via feast (setup.py) types-tabulate==0.8.9 # via feast (setup.py) -types-urllib3==1.26.14 +types-urllib3==1.26.15 # via types-requests typing-extensions==4.2.0 # via @@ -724,13 +724,12 @@ tzdata==2022.1 # via pytz-deprecation-shim tzlocal==4.2 # via great-expectations -ujson==5.2.0 - # via gcsfs uritemplate==4.1.1 # via google-api-python-client urllib3==1.26.9 # via # botocore + # feast (setup.py) # great-expectations # minio # requests diff --git a/sdk/python/requirements/py3.10-requirements.txt b/sdk/python/requirements/py3.10-requirements.txt index 717982f012c..00b14d2cfe7 100644 --- a/sdk/python/requirements/py3.10-requirements.txt +++ b/sdk/python/requirements/py3.10-requirements.txt @@ -4,19 +4,19 @@ # # pip-compile --output-file=sdk/python/requirements/py3.10-requirements.txt # -absl-py==1.0.0 +absl-py==1.1.0 # via tensorflow-metadata anyio==3.6.1 # via # starlette # watchgod -asgiref==3.5.1 +asgiref==3.5.2 # via uvicorn attrs==21.4.0 # via jsonschema -cachetools==5.1.0 +cachetools==5.2.0 # via google-auth -certifi==2021.10.8 +certifi==2022.5.18.1 # via requests charset-normalizer==2.0.12 # via requests @@ -24,36 +24,36 @@ click==8.0.1 # via # feast (setup.py) # uvicorn -cloudpickle==2.0.0 +cloudpickle==2.1.0 # via dask colorama==0.4.4 # via feast (setup.py) dask==2022.1.1 # via feast (setup.py) -dill==0.3.4 +dill==0.3.5.1 # via feast (setup.py) fastapi==0.78.0 # via feast (setup.py) -fastavro==1.4.11 +fastavro==1.4.12 # via # feast (setup.py) # pandavro -fsspec==2022.3.0 +fsspec==2022.5.0 # via dask -google-api-core==2.7.3 +google-api-core==2.8.1 # via feast (setup.py) google-auth==2.6.6 # via google-api-core -googleapis-common-protos==1.52.0 +googleapis-common-protos==1.56.2 # via # feast (setup.py) # google-api-core # tensorflow-metadata -grpcio==1.46.1 +grpcio==1.46.3 # via # feast (setup.py) # grpcio-reflection -grpcio-reflection==1.46.1 +grpcio-reflection==1.46.3 # via feast (setup.py) h11==0.13.0 # via uvicorn @@ -65,7 +65,7 @@ idna==3.3 # requests jinja2==3.1.2 # via feast (setup.py) -jsonschema==4.5.1 +jsonschema==4.6.0 # via feast (setup.py) locket==1.0.0 # via partd @@ -89,7 +89,7 @@ pandavro==1.5.2 # via feast (setup.py) partd==1.2.0 # via dask -proto-plus==1.19.6 +proto-plus==1.20.5 # via feast (setup.py) protobuf==3.19.4 # via @@ -107,7 +107,7 @@ pyasn1==0.4.8 # rsa pyasn1-modules==0.2.8 # via google-auth -pydantic==1.9.0 +pydantic==1.9.1 # via # fastapi # feast (setup.py) @@ -134,7 +134,6 @@ rsa==4.8 # via google-auth six==1.16.0 # via - # absl-py # google-auth # grpcio # pandavro diff --git a/sdk/python/requirements/py3.7-ci-requirements.txt b/sdk/python/requirements/py3.7-ci-requirements.txt index 1d8c31808d4..ce24e767b4e 100644 --- a/sdk/python/requirements/py3.7-ci-requirements.txt +++ b/sdk/python/requirements/py3.7-ci-requirements.txt @@ -36,7 +36,7 @@ appdirs==1.4.4 # via black appnope==0.1.3 # via ipython -asgiref==3.5.1 +asgiref==3.5.2 # via uvicorn asn1crypto==1.5.1 # via @@ -89,13 +89,13 @@ botocore==1.23.24 # boto3 # moto # s3transfer -build==0.7.0 +build==0.8.0 # via feast (setup.py) cachecontrol==0.12.11 # via firebase-admin cachetools==4.2.4 # via google-auth -certifi==2021.10.8 +certifi==2022.5.18.1 # via # minio # msrest @@ -120,15 +120,15 @@ click==8.0.1 # great-expectations # pip-tools # uvicorn -cloudpickle==2.0.0 +cloudpickle==2.1.0 # via dask colorama==0.4.4 # via # feast (setup.py) # great-expectations -coverage[toml]==6.3.3 +coverage[toml]==6.4 # via pytest-cov -cryptography==3.4.8 +cryptography==35.0.0 # via # adal # azure-identity @@ -151,7 +151,7 @@ deprecated==1.2.13 # via redis deprecation==2.1.0 # via testcontainers -dill==0.3.4 +dill==0.3.5.1 # via feast (setup.py) distlib==0.3.4 # via virtualenv @@ -169,13 +169,13 @@ execnet==1.9.0 # via pytest-xdist fastapi==0.78.0 # via feast (setup.py) -fastavro==1.4.11 +fastavro==1.4.12 # via # feast (setup.py) # pandavro fastjsonschema==2.15.3 # via nbformat -filelock==3.7.0 +filelock==3.7.1 # via virtualenv firebase-admin==4.5.2 # via feast (setup.py) @@ -191,9 +191,9 @@ fsspec==2022.1.0 # dask # gcsfs # s3fs -gcsfs==0.8.0 +gcsfs==2022.1.0 # via feast (setup.py) -google-api-core[grpc]==1.31.5 +google-api-core[grpc]==1.31.6 # via # feast (setup.py) # firebase-admin @@ -203,7 +203,7 @@ google-api-core[grpc]==1.31.5 # google-cloud-core # google-cloud-datastore # google-cloud-firestore -google-api-python-client==2.47.0 +google-api-python-client==2.49.0 # via firebase-admin google-auth==1.35.0 # via @@ -229,28 +229,29 @@ google-cloud-core==1.7.2 # google-cloud-datastore # google-cloud-firestore # google-cloud-storage -google-cloud-datastore==2.5.1 +google-cloud-datastore==2.6.1 # via feast (setup.py) -google-cloud-firestore==2.4.0 +google-cloud-firestore==2.5.1 # via firebase-admin google-cloud-storage==1.40.0 # via # feast (setup.py) # firebase-admin + # gcsfs google-crc32c==1.3.0 # via google-resumable-media google-resumable-media==1.3.3 # via # google-cloud-bigquery # google-cloud-storage -googleapis-common-protos==1.52.0 +googleapis-common-protos==1.56.2 # via # feast (setup.py) # google-api-core # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) -grpcio==1.46.1 +grpcio==1.46.3 # via # feast (setup.py) # google-api-core @@ -258,7 +259,7 @@ grpcio==1.46.1 # grpcio-reflection # grpcio-testing # grpcio-tools -grpcio-reflection==1.46.1 +grpcio-reflection==1.46.3 # via feast (setup.py) grpcio-testing==1.44.0 # via feast (setup.py) @@ -276,7 +277,7 @@ httplib2==0.20.4 # google-auth-httplib2 httptools==0.4.0 # via uvicorn -identify==2.5.0 +identify==2.5.1 # via pre-commit idna==3.3 # via @@ -304,7 +305,7 @@ importlib-resources==5.7.1 # via jsonschema iniconfig==1.1.1 # via pytest -ipython==7.33.0 +ipython==7.34.0 # via great-expectations isodate==0.6.1 # via msrest @@ -353,9 +354,9 @@ mmh3==3.0.0 # via feast (setup.py) mock==2.0.0 # via feast (setup.py) -moto==3.1.9 +moto==3.1.11 # via feast (setup.py) -msal==1.17.0 +msal==1.18.0 # via # azure-identity # msal-extensions @@ -432,7 +433,7 @@ pexpect==4.8.0 # via ipython pickleshare==0.7.5 # via ipython -pip-tools==6.6.1 +pip-tools==6.6.2 # via feast (setup.py) platformdirs==2.5.2 # via virtualenv @@ -446,7 +447,7 @@ pre-commit==2.19.0 # via feast (setup.py) prompt-toolkit==3.0.29 # via ipython -proto-plus==1.19.6 +proto-plus==1.20.5 # via # feast (setup.py) # google-cloud-bigquery @@ -497,7 +498,7 @@ pycparser==2.21 # via cffi pycryptodomex==3.14.1 # via snowflake-connector-python -pydantic==1.9.0 +pydantic==1.9.1 # via # fastapi # feast (setup.py) @@ -513,7 +514,7 @@ pyjwt[crypto]==2.4.0 # adal # msal # snowflake-connector-python -pyopenssl==21.0.0 +pyopenssl==22.0.0 # via snowflake-connector-python pyparsing==2.4.7 # via @@ -607,7 +608,7 @@ requests-oauthlib==1.3.1 # via # google-auth-oauthlib # msrest -responses==0.20.0 +responses==0.21.0 # via moto rsa==4.8 # via google-auth @@ -636,14 +637,13 @@ six==1.16.0 # mock # msrestazure # pandavro - # pyopenssl # python-dateutil # virtualenv sniffio==1.2.0 # via anyio snowballstemmer==2.2.0 # via sphinx -snowflake-connector-python[pandas]==2.7.7 +snowflake-connector-python[pandas]==2.7.8 # via feast (setup.py) sphinx==4.3.2 # via @@ -698,7 +698,7 @@ tqdm==4.64.0 # via # feast (setup.py) # great-expectations -traitlets==5.2.1.post0 +traitlets==5.2.2.post1 # via # ipython # jupyter-core @@ -706,29 +706,29 @@ traitlets==5.2.1.post0 # nbformat trino==0.313.0 # via feast (setup.py) -typed-ast==1.5.3 +typed-ast==1.5.4 # via # black # mypy -types-protobuf==3.19.20 +types-protobuf==3.19.21 # via # feast (setup.py) # mypy-protobuf -types-python-dateutil==2.8.15 +types-python-dateutil==2.8.17 # via feast (setup.py) types-pytz==2021.3.8 # via feast (setup.py) types-pyyaml==6.0.7 # via feast (setup.py) -types-redis==4.2.3 +types-redis==4.2.6 # via feast (setup.py) -types-requests==2.27.25 +types-requests==2.27.29 # via feast (setup.py) -types-setuptools==57.4.14 +types-setuptools==57.4.17 # via feast (setup.py) types-tabulate==0.8.9 # via feast (setup.py) -types-urllib3==1.26.14 +types-urllib3==1.26.15 # via types-requests typing-extensions==4.2.0 # via @@ -745,6 +745,7 @@ typing-extensions==4.2.0 # mypy # pydantic # redis + # responses # starlette # uvicorn # yarl @@ -752,13 +753,12 @@ tzdata==2022.1 # via pytz-deprecation-shim tzlocal==4.2 # via great-expectations -ujson==5.2.0 - # via gcsfs uritemplate==4.1.1 # via google-api-python-client urllib3==1.26.9 # via # botocore + # feast (setup.py) # great-expectations # minio # requests diff --git a/sdk/python/requirements/py3.7-requirements.txt b/sdk/python/requirements/py3.7-requirements.txt index c175fff7893..85d3e2ee096 100644 --- a/sdk/python/requirements/py3.7-requirements.txt +++ b/sdk/python/requirements/py3.7-requirements.txt @@ -10,13 +10,13 @@ anyio==3.6.1 # via # starlette # watchgod -asgiref==3.5.1 +asgiref==3.5.2 # via uvicorn attrs==21.4.0 # via jsonschema -cachetools==5.1.0 +cachetools==5.2.0 # via google-auth -certifi==2021.10.8 +certifi==2022.5.18.1 # via requests charset-normalizer==2.0.12 # via requests @@ -24,36 +24,36 @@ click==8.0.1 # via # feast (setup.py) # uvicorn -cloudpickle==2.0.0 +cloudpickle==2.1.0 # via dask colorama==0.4.4 # via feast (setup.py) dask==2022.1.1 # via feast (setup.py) -dill==0.3.4 +dill==0.3.5.1 # via feast (setup.py) fastapi==0.78.0 # via feast (setup.py) -fastavro==1.4.11 +fastavro==1.4.12 # via # feast (setup.py) # pandavro -fsspec==2022.3.0 +fsspec==2022.5.0 # via dask -google-api-core==2.7.3 +google-api-core==2.8.1 # via feast (setup.py) google-auth==2.6.6 # via google-api-core -googleapis-common-protos==1.52.0 +googleapis-common-protos==1.56.2 # via # feast (setup.py) # google-api-core # tensorflow-metadata -grpcio==1.46.1 +grpcio==1.46.3 # via # feast (setup.py) # grpcio-reflection -grpcio-reflection==1.46.1 +grpcio-reflection==1.46.3 # via feast (setup.py) h11==0.13.0 # via uvicorn @@ -63,7 +63,7 @@ idna==3.3 # via # anyio # requests -importlib-metadata==4.11.3 +importlib-metadata==4.11.4 # via # click # jsonschema @@ -95,7 +95,7 @@ pandavro==1.5.2 # via feast (setup.py) partd==1.2.0 # via dask -proto-plus==1.19.6 +proto-plus==1.20.5 # via feast (setup.py) protobuf==3.19.4 # via @@ -113,7 +113,7 @@ pyasn1==0.4.8 # rsa pyasn1-modules==0.2.8 # via google-auth -pydantic==1.9.0 +pydantic==1.9.1 # via # fastapi # feast (setup.py) diff --git a/sdk/python/requirements/py3.8-ci-requirements.txt b/sdk/python/requirements/py3.8-ci-requirements.txt index af34dbbc2ff..8de792c0734 100644 --- a/sdk/python/requirements/py3.8-ci-requirements.txt +++ b/sdk/python/requirements/py3.8-ci-requirements.txt @@ -36,7 +36,7 @@ appdirs==1.4.4 # via black appnope==0.1.3 # via ipython -asgiref==3.5.1 +asgiref==3.5.2 # via uvicorn asn1crypto==1.5.1 # via @@ -89,13 +89,13 @@ botocore==1.23.24 # boto3 # moto # s3transfer -build==0.7.0 +build==0.8.0 # via feast (setup.py) cachecontrol==0.12.11 # via firebase-admin cachetools==4.2.4 # via google-auth -certifi==2021.10.8 +certifi==2022.5.18.1 # via # minio # msrest @@ -120,15 +120,15 @@ click==8.0.1 # great-expectations # pip-tools # uvicorn -cloudpickle==2.0.0 +cloudpickle==2.1.0 # via dask colorama==0.4.4 # via # feast (setup.py) # great-expectations -coverage[toml]==6.3.3 +coverage[toml]==6.4 # via pytest-cov -cryptography==3.4.8 +cryptography==35.0.0 # via # adal # azure-identity @@ -151,7 +151,7 @@ deprecated==1.2.13 # via redis deprecation==2.1.0 # via testcontainers -dill==0.3.4 +dill==0.3.5.1 # via feast (setup.py) distlib==0.3.4 # via virtualenv @@ -171,13 +171,13 @@ executing==0.8.3 # via stack-data fastapi==0.78.0 # via feast (setup.py) -fastavro==1.4.11 +fastavro==1.4.12 # via # feast (setup.py) # pandavro fastjsonschema==2.15.3 # via nbformat -filelock==3.7.0 +filelock==3.7.1 # via virtualenv firebase-admin==4.5.2 # via feast (setup.py) @@ -193,9 +193,9 @@ fsspec==2022.1.0 # dask # gcsfs # s3fs -gcsfs==0.8.0 +gcsfs==2022.1.0 # via feast (setup.py) -google-api-core[grpc]==1.31.5 +google-api-core[grpc]==1.31.6 # via # feast (setup.py) # firebase-admin @@ -205,7 +205,7 @@ google-api-core[grpc]==1.31.5 # google-cloud-core # google-cloud-datastore # google-cloud-firestore -google-api-python-client==2.47.0 +google-api-python-client==2.49.0 # via firebase-admin google-auth==1.35.0 # via @@ -231,28 +231,29 @@ google-cloud-core==1.7.2 # google-cloud-datastore # google-cloud-firestore # google-cloud-storage -google-cloud-datastore==2.5.1 +google-cloud-datastore==2.6.1 # via feast (setup.py) -google-cloud-firestore==2.4.0 +google-cloud-firestore==2.5.1 # via firebase-admin google-cloud-storage==1.40.0 # via # feast (setup.py) # firebase-admin + # gcsfs google-crc32c==1.3.0 # via google-resumable-media google-resumable-media==1.3.3 # via # google-cloud-bigquery # google-cloud-storage -googleapis-common-protos==1.52.0 +googleapis-common-protos==1.56.2 # via # feast (setup.py) # google-api-core # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) -grpcio==1.46.1 +grpcio==1.46.3 # via # feast (setup.py) # google-api-core @@ -260,7 +261,7 @@ grpcio==1.46.1 # grpcio-reflection # grpcio-testing # grpcio-tools -grpcio-reflection==1.46.1 +grpcio-reflection==1.46.3 # via feast (setup.py) grpcio-testing==1.44.0 # via feast (setup.py) @@ -278,7 +279,7 @@ httplib2==0.20.4 # google-auth-httplib2 httptools==0.4.0 # via uvicorn -identify==2.5.0 +identify==2.5.1 # via pre-commit idna==3.3 # via @@ -288,13 +289,13 @@ idna==3.3 # yarl imagesize==1.3.0 # via sphinx -importlib-metadata==4.11.3 +importlib-metadata==4.11.4 # via great-expectations importlib-resources==5.7.1 # via jsonschema iniconfig==1.1.1 # via pytest -ipython==8.3.0 +ipython==8.4.0 # via great-expectations isodate==0.6.1 # via msrest @@ -343,9 +344,9 @@ mmh3==3.0.0 # via feast (setup.py) mock==2.0.0 # via feast (setup.py) -moto==3.1.9 +moto==3.1.11 # via feast (setup.py) -msal==1.17.0 +msal==1.18.0 # via # azure-identity # msal-extensions @@ -422,7 +423,7 @@ pexpect==4.8.0 # via ipython pickleshare==0.7.5 # via ipython -pip-tools==6.6.1 +pip-tools==6.6.2 # via feast (setup.py) platformdirs==2.5.2 # via virtualenv @@ -436,7 +437,7 @@ pre-commit==2.19.0 # via feast (setup.py) prompt-toolkit==3.0.29 # via ipython -proto-plus==1.19.6 +proto-plus==1.20.5 # via # feast (setup.py) # google-cloud-bigquery @@ -489,7 +490,7 @@ pycparser==2.21 # via cffi pycryptodomex==3.14.1 # via snowflake-connector-python -pydantic==1.9.0 +pydantic==1.9.1 # via # fastapi # feast (setup.py) @@ -505,7 +506,7 @@ pyjwt[crypto]==2.4.0 # adal # msal # snowflake-connector-python -pyopenssl==21.0.0 +pyopenssl==22.0.0 # via snowflake-connector-python pyparsing==2.4.7 # via @@ -599,7 +600,7 @@ requests-oauthlib==1.3.1 # via # google-auth-oauthlib # msrest -responses==0.20.0 +responses==0.21.0 # via moto rsa==4.8 # via google-auth @@ -611,7 +612,7 @@ s3fs==2022.1.0 # via feast (setup.py) s3transfer==0.5.2 # via boto3 -scipy==1.8.0 +scipy==1.8.1 # via great-expectations six==1.16.0 # via @@ -628,14 +629,13 @@ six==1.16.0 # mock # msrestazure # pandavro - # pyopenssl # python-dateutil # virtualenv sniffio==1.2.0 # via anyio snowballstemmer==2.2.0 # via sphinx -snowflake-connector-python[pandas]==2.7.7 +snowflake-connector-python[pandas]==2.7.8 # via feast (setup.py) sphinx==4.3.2 # via @@ -692,7 +692,7 @@ tqdm==4.64.0 # via # feast (setup.py) # great-expectations -traitlets==5.2.1.post0 +traitlets==5.2.2.post1 # via # ipython # jupyter-core @@ -700,27 +700,27 @@ traitlets==5.2.1.post0 # nbformat trino==0.313.0 # via feast (setup.py) -typed-ast==1.5.3 +typed-ast==1.5.4 # via black -types-protobuf==3.19.20 +types-protobuf==3.19.21 # via # feast (setup.py) # mypy-protobuf -types-python-dateutil==2.8.15 +types-python-dateutil==2.8.17 # via feast (setup.py) types-pytz==2021.3.8 # via feast (setup.py) types-pyyaml==6.0.7 # via feast (setup.py) -types-redis==4.2.3 +types-redis==4.2.6 # via feast (setup.py) -types-requests==2.27.25 +types-requests==2.27.29 # via feast (setup.py) -types-setuptools==57.4.14 +types-setuptools==57.4.17 # via feast (setup.py) types-tabulate==0.8.9 # via feast (setup.py) -types-urllib3==1.26.14 +types-urllib3==1.26.15 # via types-requests typing-extensions==4.2.0 # via @@ -734,13 +734,12 @@ tzdata==2022.1 # via pytz-deprecation-shim tzlocal==4.2 # via great-expectations -ujson==5.2.0 - # via gcsfs uritemplate==4.1.1 # via google-api-python-client urllib3==1.26.9 # via # botocore + # feast (setup.py) # great-expectations # minio # requests diff --git a/sdk/python/requirements/py3.8-requirements.txt b/sdk/python/requirements/py3.8-requirements.txt index eff4bae2689..ae49676bd03 100644 --- a/sdk/python/requirements/py3.8-requirements.txt +++ b/sdk/python/requirements/py3.8-requirements.txt @@ -10,13 +10,13 @@ anyio==3.6.1 # via # starlette # watchgod -asgiref==3.5.1 +asgiref==3.5.2 # via uvicorn attrs==21.4.0 # via jsonschema -cachetools==5.1.0 +cachetools==5.2.0 # via google-auth -certifi==2021.10.8 +certifi==2022.5.18.1 # via requests charset-normalizer==2.0.12 # via requests @@ -24,36 +24,36 @@ click==8.0.1 # via # feast (setup.py) # uvicorn -cloudpickle==2.0.0 +cloudpickle==2.1.0 # via dask colorama==0.4.4 # via feast (setup.py) dask==2022.1.1 # via feast (setup.py) -dill==0.3.4 +dill==0.3.5.1 # via feast (setup.py) fastapi==0.78.0 # via feast (setup.py) -fastavro==1.4.11 +fastavro==1.4.12 # via # feast (setup.py) # pandavro -fsspec==2022.3.0 +fsspec==2022.5.0 # via dask -google-api-core==2.7.3 +google-api-core==2.8.1 # via feast (setup.py) google-auth==2.6.6 # via google-api-core -googleapis-common-protos==1.52.0 +googleapis-common-protos==1.56.2 # via # feast (setup.py) # google-api-core # tensorflow-metadata -grpcio==1.46.1 +grpcio==1.46.3 # via # feast (setup.py) # grpcio-reflection -grpcio-reflection==1.46.1 +grpcio-reflection==1.46.3 # via feast (setup.py) h11==0.13.0 # via uvicorn @@ -91,7 +91,7 @@ pandavro==1.5.2 # via feast (setup.py) partd==1.2.0 # via dask -proto-plus==1.19.6 +proto-plus==1.20.5 # via feast (setup.py) protobuf==3.19.4 # via @@ -109,7 +109,7 @@ pyasn1==0.4.8 # rsa pyasn1-modules==0.2.8 # via google-auth -pydantic==1.9.0 +pydantic==1.9.1 # via # fastapi # feast (setup.py) diff --git a/sdk/python/requirements/py3.9-ci-requirements.txt b/sdk/python/requirements/py3.9-ci-requirements.txt index 4147a391dce..e8741a237ba 100644 --- a/sdk/python/requirements/py3.9-ci-requirements.txt +++ b/sdk/python/requirements/py3.9-ci-requirements.txt @@ -1,10 +1,10 @@ # -# This file is autogenerated by pip-compile with python 3.9 +# This file is autogenerated by pip-compile with python 3.8 # To update, run: # # pip-compile --extra=ci --output-file=sdk/python/requirements/py3.9-ci-requirements.txt # -absl-py==1.0.0 +absl-py==1.1.0 # via tensorflow-metadata adal==1.2.7 # via @@ -36,7 +36,7 @@ appdirs==1.4.4 # via black appnope==0.1.3 # via ipython -asgiref==3.5.1 +asgiref==3.5.2 # via uvicorn asn1crypto==1.5.1 # via @@ -58,7 +58,7 @@ attrs==21.4.0 # pytest avro==1.10.0 # via feast (setup.py) -azure-core==1.24.0 +azure-core==1.24.1 # via # adlfs # azure-identity @@ -73,6 +73,10 @@ babel==2.10.1 # via sphinx backcall==0.2.0 # via ipython +backports-zoneinfo==0.2.1 + # via + # pytz-deprecation-shim + # tzlocal black==19.10b0 # via feast (setup.py) boto3==1.20.23 @@ -85,13 +89,13 @@ botocore==1.23.24 # boto3 # moto # s3transfer -build==0.7.0 +build==0.8.0 # via feast (setup.py) cachecontrol==0.12.11 # via firebase-admin cachetools==4.2.4 # via google-auth -certifi==2021.10.8 +certifi==2022.5.18.1 # via # minio # msrest @@ -116,15 +120,15 @@ click==8.0.1 # great-expectations # pip-tools # uvicorn -cloudpickle==2.0.0 +cloudpickle==2.1.0 # via dask colorama==0.4.4 # via # feast (setup.py) # great-expectations -coverage[toml]==6.3.3 +coverage[toml]==6.4 # via pytest-cov -cryptography==3.4.8 +cryptography==35.0.0 # via # adal # azure-identity @@ -147,7 +151,7 @@ deprecated==1.2.13 # via redis deprecation==2.1.0 # via testcontainers -dill==0.3.4 +dill==0.3.5.1 # via feast (setup.py) distlib==0.3.4 # via virtualenv @@ -167,13 +171,13 @@ executing==0.8.3 # via stack-data fastapi==0.78.0 # via feast (setup.py) -fastavro==1.4.11 +fastavro==1.4.12 # via # feast (setup.py) # pandavro fastjsonschema==2.15.3 # via nbformat -filelock==3.7.0 +filelock==3.7.1 # via virtualenv firebase-admin==4.5.2 # via feast (setup.py) @@ -189,9 +193,9 @@ fsspec==2022.1.0 # dask # gcsfs # s3fs -gcsfs==0.8.0 +gcsfs==2022.1.0 # via feast (setup.py) -google-api-core[grpc]==1.31.5 +google-api-core[grpc]==1.31.6 # via # feast (setup.py) # firebase-admin @@ -201,7 +205,7 @@ google-api-core[grpc]==1.31.5 # google-cloud-core # google-cloud-datastore # google-cloud-firestore -google-api-python-client==2.47.0 +google-api-python-client==2.49.0 # via firebase-admin google-auth==1.35.0 # via @@ -227,28 +231,29 @@ google-cloud-core==1.7.2 # google-cloud-datastore # google-cloud-firestore # google-cloud-storage -google-cloud-datastore==2.5.1 +google-cloud-datastore==2.6.1 # via feast (setup.py) -google-cloud-firestore==2.4.0 +google-cloud-firestore==2.5.2 # via firebase-admin google-cloud-storage==1.40.0 # via # feast (setup.py) # firebase-admin + # gcsfs google-crc32c==1.3.0 # via google-resumable-media google-resumable-media==1.3.3 # via # google-cloud-bigquery # google-cloud-storage -googleapis-common-protos==1.52.0 +googleapis-common-protos==1.56.2 # via # feast (setup.py) # google-api-core # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) -grpcio==1.46.1 +grpcio==1.46.3 # via # feast (setup.py) # google-api-core @@ -256,7 +261,7 @@ grpcio==1.46.1 # grpcio-reflection # grpcio-testing # grpcio-tools -grpcio-reflection==1.46.1 +grpcio-reflection==1.46.3 # via feast (setup.py) grpcio-testing==1.44.0 # via feast (setup.py) @@ -274,7 +279,7 @@ httplib2==0.20.4 # google-auth-httplib2 httptools==0.4.0 # via uvicorn -identify==2.5.0 +identify==2.5.1 # via pre-commit idna==3.3 # via @@ -284,11 +289,13 @@ idna==3.3 # yarl imagesize==1.3.0 # via sphinx -importlib-metadata==4.11.3 +importlib-metadata==4.11.4 # via great-expectations +importlib-resources==5.7.1 + # via jsonschema iniconfig==1.1.1 # via pytest -ipython==8.3.0 +ipython==8.4.0 # via great-expectations isodate==0.6.1 # via msrest @@ -311,7 +318,7 @@ jsonpatch==1.32 # via great-expectations jsonpointer==2.3 # via jsonpatch -jsonschema==4.5.1 +jsonschema==4.6.0 # via # altair # feast (setup.py) @@ -337,9 +344,9 @@ mmh3==3.0.0 # via feast (setup.py) mock==2.0.0 # via feast (setup.py) -moto==3.1.9 +moto==3.1.11 # via feast (setup.py) -msal==1.17.0 +msal==1.18.0 # via # azure-identity # msal-extensions @@ -416,7 +423,7 @@ pexpect==4.8.0 # via ipython pickleshare==0.7.5 # via ipython -pip-tools==6.6.1 +pip-tools==6.6.2 # via feast (setup.py) platformdirs==2.5.2 # via virtualenv @@ -430,7 +437,7 @@ pre-commit==2.19.0 # via feast (setup.py) prompt-toolkit==3.0.29 # via ipython -proto-plus==1.19.6 +proto-plus==1.20.5 # via # feast (setup.py) # google-cloud-bigquery @@ -442,6 +449,7 @@ protobuf==3.19.4 # feast (setup.py) # google-api-core # google-cloud-bigquery + # google-cloud-firestore # googleapis-common-protos # grpcio-reflection # grpcio-testing @@ -483,7 +491,7 @@ pycparser==2.21 # via cffi pycryptodomex==3.14.1 # via snowflake-connector-python -pydantic==1.9.0 +pydantic==1.9.1 # via # fastapi # feast (setup.py) @@ -499,7 +507,7 @@ pyjwt[crypto]==2.4.0 # adal # msal # snowflake-connector-python -pyopenssl==21.0.0 +pyopenssl==22.0.0 # via snowflake-connector-python pyparsing==2.4.7 # via @@ -593,23 +601,22 @@ requests-oauthlib==1.3.1 # via # google-auth-oauthlib # msrest -responses==0.20.0 +responses==0.21.0 # via moto rsa==4.8 # via google-auth -ruamel.yaml==0.17.17 +ruamel-yaml==0.17.17 # via great-expectations -ruamel.yaml.clib==0.2.6 - # via ruamel.yaml +ruamel-yaml-clib==0.2.6 + # via ruamel-yaml s3fs==2022.1.0 # via feast (setup.py) s3transfer==0.5.2 # via boto3 -scipy==1.8.0 +scipy==1.8.1 # via great-expectations six==1.16.0 # via - # absl-py # azure-core # azure-identity # google-api-core @@ -622,14 +629,13 @@ six==1.16.0 # mock # msrestazure # pandavro - # pyopenssl # python-dateutil # virtualenv sniffio==1.2.0 # via anyio snowballstemmer==2.2.0 # via sphinx -snowflake-connector-python[pandas]==2.7.7 +snowflake-connector-python[pandas]==2.7.8 # via feast (setup.py) sphinx==4.3.2 # via @@ -686,7 +692,7 @@ tqdm==4.64.0 # via # feast (setup.py) # great-expectations -traitlets==5.2.1.post0 +traitlets==5.2.2.post1 # via # ipython # jupyter-core @@ -694,27 +700,27 @@ traitlets==5.2.1.post0 # nbformat trino==0.313.0 # via feast (setup.py) -typed-ast==1.5.3 +typed-ast==1.5.4 # via black -types-protobuf==3.19.20 +types-protobuf==3.19.21 # via # feast (setup.py) # mypy-protobuf -types-python-dateutil==2.8.15 +types-python-dateutil==2.8.17 # via feast (setup.py) types-pytz==2021.3.8 # via feast (setup.py) -types-pyyaml==6.0.7 +types-pyyaml==6.0.8 # via feast (setup.py) -types-redis==4.2.3 +types-redis==4.2.6 # via feast (setup.py) -types-requests==2.27.25 +types-requests==2.27.30 # via feast (setup.py) -types-setuptools==57.4.14 +types-setuptools==57.4.17 # via feast (setup.py) types-tabulate==0.8.9 # via feast (setup.py) -types-urllib3==1.26.14 +types-urllib3==1.26.15 # via types-requests typing-extensions==4.2.0 # via @@ -728,13 +734,12 @@ tzdata==2022.1 # via pytz-deprecation-shim tzlocal==4.2 # via great-expectations -ujson==5.2.0 - # via gcsfs uritemplate==4.1.1 # via google-api-python-client urllib3==1.26.9 # via # botocore + # feast (setup.py) # great-expectations # minio # requests @@ -767,7 +772,9 @@ xmltodict==0.13.0 yarl==1.7.2 # via aiohttp zipp==3.8.0 - # via importlib-metadata + # via + # importlib-metadata + # importlib-resources # The following packages are considered to be unsafe in a requirements file: # pip diff --git a/sdk/python/requirements/py3.9-requirements.txt b/sdk/python/requirements/py3.9-requirements.txt index 80199a1f5ba..577ff2838f9 100644 --- a/sdk/python/requirements/py3.9-requirements.txt +++ b/sdk/python/requirements/py3.9-requirements.txt @@ -10,13 +10,13 @@ anyio==3.6.1 # via # starlette # watchgod -asgiref==3.5.1 +asgiref==3.5.2 # via uvicorn attrs==21.4.0 # via jsonschema -cachetools==5.1.0 +cachetools==5.2.0 # via google-auth -certifi==2021.10.8 +certifi==2022.5.18.1 # via requests charset-normalizer==2.0.12 # via requests @@ -24,36 +24,36 @@ click==8.0.1 # via # feast (setup.py) # uvicorn -cloudpickle==2.0.0 +cloudpickle==2.1.0 # via dask colorama==0.4.4 # via feast (setup.py) dask==2022.1.1 # via feast (setup.py) -dill==0.3.4 +dill==0.3.5.1 # via feast (setup.py) fastapi==0.78.0 # via feast (setup.py) -fastavro==1.4.11 +fastavro==1.4.12 # via # feast (setup.py) # pandavro -fsspec==2022.3.0 +fsspec==2022.5.0 # via dask -google-api-core==2.7.3 +google-api-core==2.8.1 # via feast (setup.py) google-auth==2.6.6 # via google-api-core -googleapis-common-protos==1.52.0 +googleapis-common-protos==1.56.2 # via # feast (setup.py) # google-api-core # tensorflow-metadata -grpcio==1.46.1 +grpcio==1.46.3 # via # feast (setup.py) # grpcio-reflection -grpcio-reflection==1.46.1 +grpcio-reflection==1.46.3 # via feast (setup.py) h11==0.13.0 # via uvicorn @@ -89,7 +89,7 @@ pandavro==1.5.2 # via feast (setup.py) partd==1.2.0 # via dask -proto-plus==1.19.6 +proto-plus==1.20.5 # via feast (setup.py) protobuf==3.19.4 # via @@ -107,7 +107,7 @@ pyasn1==0.4.8 # rsa pyasn1-modules==0.2.8 # via google-auth -pydantic==1.9.0 +pydantic==1.9.1 # via # fastapi # feast (setup.py) diff --git a/setup.py b/setup.py index 8934ca24ad0..633267dbae5 100644 --- a/setup.py +++ b/setup.py @@ -51,7 +51,7 @@ "dill==0.3.*", "fastavro>=1.1.0,<2", "google-api-core>=1.23.0,<3", - "googleapis-common-protos==1.52.*,<2", + "googleapis-common-protos>=1.52.*,<2", "grpcio>=1.34.0,<2", "grpcio-reflection>=1.34.0,<2", "Jinja2>=2,<4", @@ -61,7 +61,7 @@ "pandas>=1,<2", "pandavro==1.5.*", "protobuf>=3.10,<3.20", - "proto-plus<1.19.7", + "proto-plus==1.20.*", "pyarrow>=4,<7", "pydantic>=1,<2", "pygments==2.12.0", @@ -72,7 +72,6 @@ "tqdm==4.*", "fastapi>=0.68.0,<1", "uvicorn[standard]>=0.14.0,<1", - "proto-plus<1.19.7", "tensorflow-metadata>=1.0.0,<2.0.0", "dask>=2021.*,<2022.02.0", ] @@ -90,14 +89,10 @@ "hiredis>=2.0.0,<3", ] -AWS_REQUIRED = [ - "boto3>=1.17.0,<=1.20.23", - "docker>=5.0.2", - "s3fs>=0.4.0,<=2022.01.0" -] +AWS_REQUIRED = ["boto3>=1.17.0,<=1.20.23", "docker>=5.0.2", "s3fs>=0.4.0,<=2022.01.0"] SNOWFLAKE_REQUIRED = [ - "snowflake-connector-python[pandas]>=2.7.3,<3", + "snowflake-connector-python[pandas]>=2.7.3,<=2.7.8", ] SPARK_REQUIRED = [ @@ -125,7 +120,7 @@ CI_REQUIRED = ( [ "build", - "cryptography==3.4.8", + "cryptography==35.0", "flake8", "black==19.10b0", "isort>=5,<6", @@ -137,7 +132,7 @@ "mypy==0.931", "mypy-protobuf==3.1", "avro==1.10.0", - "gcsfs>=0.4.0,<=2022.01.0" + "gcsfs>=0.4.0,<=2022.01.0", "urllib3>=1.25.4,<2", "psutil==5.9.0", "pytest>=6.0.0,<8", @@ -248,7 +243,7 @@ def _generate_python_protos(self, path: str): "--mypy_out", self.python_folder, ] - + proto_files, + + proto_files ) def run(self): From 44a3f051533cc4556a66e00abce169043dddbf9e Mon Sep 17 00:00:00 2001 From: Huib Keemink Date: Thu, 2 Jun 2022 19:20:30 +0200 Subject: [PATCH 035/123] Made URL to registry relative, fixes #2731 (#2753) Signed-off-by: Huib Keemink --- sdk/python/feast/ui_server.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdk/python/feast/ui_server.py b/sdk/python/feast/ui_server.py index 5206dd5161e..cb275c8f912 100644 --- a/sdk/python/feast/ui_server.py +++ b/sdk/python/feast/ui_server.py @@ -62,7 +62,7 @@ def shutdown_event(): "name": "Project", "description": "Test project", "id": project_id, - "registryPath": f"http://{host}:{port}/registry", + "registryPath": "/registry", } ] } From d024e5efac085ec12a17005389229bfd93cf466e Mon Sep 17 00:00:00 2001 From: Kevin Zhang Date: Thu, 2 Jun 2022 16:40:52 -0700 Subject: [PATCH 036/123] fix: Fix bugs in applying stream feature view and retrieving online features (#2754) * Fix apply workflow Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang * Fix issues Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang * Reformat Signed-off-by: Kevin Zhang --- protos/feast/core/StreamFeatureView.proto | 4 + sdk/python/feast/diff/registry_diff.py | 6 ++ sdk/python/feast/feature_store.py | 75 +++++++++++++------ .../feast/infra/online_stores/sqlite.py | 5 +- sdk/python/feast/registry.py | 29 +++++++ sdk/python/feast/repo_operations.py | 38 ++++++++-- sdk/python/feast/stream_feature_view.py | 29 ++++++- .../test_stream_feature_view_apply.py | 11 ++- 8 files changed, 163 insertions(+), 34 deletions(-) diff --git a/protos/feast/core/StreamFeatureView.proto b/protos/feast/core/StreamFeatureView.proto index 3be9dc866af..d217b86a3f4 100644 --- a/protos/feast/core/StreamFeatureView.proto +++ b/protos/feast/core/StreamFeatureView.proto @@ -26,6 +26,7 @@ option java_package = "feast.proto.core"; import "google/protobuf/duration.proto"; import "google/protobuf/timestamp.proto"; import "feast/core/OnDemandFeatureView.proto"; +import "feast/core/FeatureView.proto"; import "feast/core/Feature.proto"; import "feast/core/DataSource.proto"; import "feast/core/Aggregation.proto"; @@ -95,4 +96,7 @@ message StreamFeatureViewMeta { // Time where this Feature View is last updated google.protobuf.Timestamp last_updated_timestamp = 2; + + // List of pairs (start_time, end_time) for which this feature view has been materialized. + repeated MaterializationInterval materialization_intervals = 3; } diff --git a/sdk/python/feast/diff/registry_diff.py b/sdk/python/feast/diff/registry_diff.py index 33cd3df0edc..6b38a190fe6 100644 --- a/sdk/python/feast/diff/registry_diff.py +++ b/sdk/python/feast/diff/registry_diff.py @@ -20,6 +20,9 @@ from feast.protos.feast.core.RequestFeatureView_pb2 import ( RequestFeatureView as RequestFeatureViewProto, ) +from feast.protos.feast.core.StreamFeatureView_pb2 import ( + StreamFeatureView as StreamFeatureViewProto, +) from feast.protos.feast.core.ValidationProfile_pb2 import ( ValidationReference as ValidationReferenceProto, ) @@ -106,6 +109,7 @@ def tag_objects_for_keep_delete_update_add( FeatureServiceProto, OnDemandFeatureViewProto, RequestFeatureViewProto, + StreamFeatureViewProto, ValidationReferenceProto, ) @@ -292,6 +296,7 @@ def apply_diff_to_registry( FeastObjectType.FEATURE_VIEW, FeastObjectType.ON_DEMAND_FEATURE_VIEW, FeastObjectType.REQUEST_FEATURE_VIEW, + FeastObjectType.STREAM_FEATURE_VIEW, ]: feature_view_obj = cast( BaseFeatureView, feast_object_diff.current_feast_object @@ -331,6 +336,7 @@ def apply_diff_to_registry( FeastObjectType.FEATURE_VIEW, FeastObjectType.ON_DEMAND_FEATURE_VIEW, FeastObjectType.REQUEST_FEATURE_VIEW, + FeastObjectType.STREAM_FEATURE_VIEW, ]: registry.apply_feature_view( cast(BaseFeatureView, feast_object_diff.new_feast_object), diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py index f959504826f..9530b66d5e3 100644 --- a/sdk/python/feast/feature_store.py +++ b/sdk/python/feast/feature_store.py @@ -372,6 +372,40 @@ def _get_feature_view( feature_view.entities = [] return feature_view + @log_exceptions_and_usage + def get_stream_feature_view( + self, name: str, allow_registry_cache: bool = False + ) -> StreamFeatureView: + """ + Retrieves a stream feature view. + + Args: + name: Name of stream feature view. + allow_registry_cache: (Optional) Whether to allow returning this entity from a cached registry + + Returns: + The specified stream feature view. + + Raises: + FeatureViewNotFoundException: The feature view could not be found. + """ + return self._get_stream_feature_view( + name, allow_registry_cache=allow_registry_cache + ) + + def _get_stream_feature_view( + self, + name: str, + hide_dummy_entity: bool = True, + allow_registry_cache: bool = False, + ) -> StreamFeatureView: + stream_feature_view = self._registry.get_stream_feature_view( + name, self.project, allow_cache=allow_registry_cache + ) + if hide_dummy_entity and stream_feature_view.entities[0] == DUMMY_ENTITY_NAME: + stream_feature_view.entities = [] + return stream_feature_view + @log_exceptions_and_usage def get_on_demand_feature_view(self, name: str) -> OnDemandFeatureView: """ @@ -935,7 +969,6 @@ def get_historical_features( all_feature_views, all_request_feature_views, all_on_demand_feature_views, - all_stream_feature_views, ) = self._get_feature_views_to_use(features) if all_request_feature_views: @@ -1321,9 +1354,14 @@ def write_to_online_store( ingests data directly into the Online store """ # TODO: restrict this to work with online StreamFeatureViews and validate the FeatureView type - feature_view = self.get_feature_view( - feature_view_name, allow_registry_cache=allow_registry_cache - ) + try: + feature_view = self.get_stream_feature_view( + feature_view_name, allow_registry_cache=allow_registry_cache + ) + except FeatureViewNotFoundException: + feature_view = self.get_feature_view( + feature_view_name, allow_registry_cache=allow_registry_cache + ) entities = [] for entity_name in feature_view.entities: entities.append( @@ -1456,7 +1494,6 @@ def _get_online_features( requested_feature_views, requested_request_feature_views, requested_on_demand_feature_views, - request_stream_feature_views, ) = self._get_feature_views_to_use( features=features, allow_cache=True, hide_dummy_entity=False ) @@ -1994,15 +2031,17 @@ def _get_feature_views_to_use( allow_cache=False, hide_dummy_entity: bool = True, ) -> Tuple[ - List[FeatureView], - List[RequestFeatureView], - List[OnDemandFeatureView], - List[StreamFeatureView], + List[FeatureView], List[RequestFeatureView], List[OnDemandFeatureView], ]: fvs = { fv.name: fv - for fv in self._list_feature_views(allow_cache, hide_dummy_entity) + for fv in [ + *self._list_feature_views(allow_cache, hide_dummy_entity), + *self._registry.list_stream_feature_views( + project=self.project, allow_cache=allow_cache + ), + ] } request_fvs = { @@ -2019,15 +2058,8 @@ def _get_feature_views_to_use( ) } - sfvs = { - fv.name: fv - for fv in self._registry.list_stream_feature_views( - project=self.project, allow_cache=allow_cache - ) - } - if isinstance(features, FeatureService): - fvs_to_use, request_fvs_to_use, od_fvs_to_use, sfvs_to_use = [], [], [], [] + fvs_to_use, request_fvs_to_use, od_fvs_to_use = [], [], [] for fv_name, projection in [ (projection.name, projection) for projection in features.feature_view_projections @@ -2048,23 +2080,18 @@ def _get_feature_views_to_use( fv = fvs[projection.name].with_projection(copy.copy(projection)) if fv not in fvs_to_use: fvs_to_use.append(fv) - elif fv_name in sfvs: - sfvs_to_use.append( - sfvs[fv_name].with_projection(copy.copy(projection)) - ) else: raise ValueError( f"The provided feature service {features.name} contains a reference to a feature view" f"{fv_name} which doesn't exist. Please make sure that you have created the feature view" f'{fv_name} and that you have registered it by running "apply".' ) - views_to_use = (fvs_to_use, request_fvs_to_use, od_fvs_to_use, sfvs_to_use) + views_to_use = (fvs_to_use, request_fvs_to_use, od_fvs_to_use) else: views_to_use = ( [*fvs.values()], [*request_fvs.values()], [*od_fvs.values()], - [*sfvs.values()], ) return views_to_use diff --git a/sdk/python/feast/infra/online_stores/sqlite.py b/sdk/python/feast/infra/online_stores/sqlite.py index 5657fbe3722..2f0e9029426 100644 --- a/sdk/python/feast/infra/online_stores/sqlite.py +++ b/sdk/python/feast/infra/online_stores/sqlite.py @@ -211,7 +211,10 @@ def plan( path=self._get_db_path(config), name=_table_id(project, FeatureView.from_proto(view)), ) - for view in desired_registry_proto.feature_views + for view in [ + *desired_registry_proto.feature_views, + *desired_registry_proto.stream_feature_views, + ] ] return infra_objects diff --git a/sdk/python/feast/registry.py b/sdk/python/feast/registry.py index 7f298b19b82..dc52013bff6 100644 --- a/sdk/python/feast/registry.py +++ b/sdk/python/feast/registry.py @@ -77,6 +77,7 @@ class FeastObjectType(Enum): FEATURE_VIEW = "feature view" ON_DEMAND_FEATURE_VIEW = "on demand feature view" REQUEST_FEATURE_VIEW = "request feature view" + STREAM_FEATURE_VIEW = "stream feature view" FEATURE_SERVICE = "feature service" @staticmethod @@ -93,6 +94,9 @@ def get_objects_from_registry( FeastObjectType.REQUEST_FEATURE_VIEW: registry.list_request_feature_views( project=project ), + FeastObjectType.STREAM_FEATURE_VIEW: registry.list_stream_feature_views( + project=project, + ), FeastObjectType.FEATURE_SERVICE: registry.list_feature_services( project=project ), @@ -108,6 +112,7 @@ def get_objects_from_repo_contents( FeastObjectType.FEATURE_VIEW: repo_contents.feature_views, FeastObjectType.ON_DEMAND_FEATURE_VIEW: repo_contents.on_demand_feature_views, FeastObjectType.REQUEST_FEATURE_VIEW: repo_contents.request_feature_views, + FeastObjectType.STREAM_FEATURE_VIEW: repo_contents.stream_feature_views, FeastObjectType.FEATURE_SERVICE: repo_contents.feature_services, } @@ -717,6 +722,30 @@ def get_feature_view( return FeatureView.from_proto(feature_view_proto) raise FeatureViewNotFoundException(name, project) + def get_stream_feature_view( + self, name: str, project: str, allow_cache: bool = False + ) -> StreamFeatureView: + """ + Retrieves a stream feature view. + + Args: + name: Name of stream feature view + project: Feast project that this stream feature view belongs to + allow_cache: Allow returning feature view from the cached registry + + Returns: + Returns either the specified feature view, or raises an exception if + none is found + """ + registry_proto = self._get_registry_proto(allow_cache=allow_cache) + for feature_view_proto in registry_proto.stream_feature_views: + if ( + feature_view_proto.spec.name == name + and feature_view_proto.spec.project == project + ): + return StreamFeatureView.from_proto(feature_view_proto) + raise FeatureViewNotFoundException(name, project) + def delete_feature_service(self, name: str, project: str, commit: bool = True): """ Deletes a feature service or raises an exception if not found. diff --git a/sdk/python/feast/repo_operations.py b/sdk/python/feast/repo_operations.py index 8b81a71bae4..37daa6500eb 100644 --- a/sdk/python/feast/repo_operations.py +++ b/sdk/python/feast/repo_operations.py @@ -13,7 +13,8 @@ from click.exceptions import BadParameter from feast import PushSource -from feast.data_source import DataSource +from feast.batch_feature_view import BatchFeatureView +from feast.data_source import DataSource, KafkaSource from feast.diff.registry_diff import extract_objects_for_keep_delete_update_add from feast.entity import Entity from feast.feature_service import FeatureService @@ -25,6 +26,7 @@ from feast.repo_config import RepoConfig from feast.repo_contents import RepoContents from feast.request_feature_view import RequestFeatureView +from feast.stream_feature_view import StreamFeatureView from feast.usage import log_exceptions_and_usage @@ -122,8 +124,11 @@ def parse_repo(repo_root: Path) -> RepoContents: ): res.data_sources.append(obj) data_sources_set.add(obj) - if isinstance(obj, FeatureView) and not any( - (obj is fv) for fv in res.feature_views + if ( + isinstance(obj, FeatureView) + and not any((obj is fv) for fv in res.feature_views) + and not isinstance(obj, StreamFeatureView) + and not isinstance(obj, BatchFeatureView) ): res.feature_views.append(obj) if isinstance(obj.stream_source, PushSource) and not any( @@ -133,6 +138,19 @@ def parse_repo(repo_root: Path) -> RepoContents: # Don't add if the push source's batch source is a duplicate of an existing batch source if push_source_dep not in data_sources_set: res.data_sources.append(push_source_dep) + elif isinstance(obj, StreamFeatureView) and not any( + (obj is sfv) for sfv in res.stream_feature_views + ): + res.stream_feature_views.append(obj) + if ( + isinstance(obj.stream_source, PushSource) + or isinstance(obj.stream_source, KafkaSource) + and not any((obj is ds) for ds in res.data_sources) + ): + batch_source_dep = obj.stream_source.batch_source + # Don't add if the push source's batch source is a duplicate of an existing batch source + if batch_source_dep and batch_source_dep not in data_sources_set: + res.data_sources.append(batch_source_dep) elif isinstance(obj, Entity) and not any( (obj is entity) for entity in res.entities ): @@ -196,7 +214,12 @@ def extract_objects_for_apply_delete(project, registry, repo): all_to_apply: List[ Union[ - Entity, FeatureView, RequestFeatureView, OnDemandFeatureView, FeatureService + Entity, + FeatureView, + RequestFeatureView, + OnDemandFeatureView, + StreamFeatureView, + FeatureService, ] ] = [] for object_type in FEAST_OBJECT_TYPES: @@ -205,7 +228,12 @@ def extract_objects_for_apply_delete(project, registry, repo): all_to_delete: List[ Union[ - Entity, FeatureView, RequestFeatureView, OnDemandFeatureView, FeatureService + Entity, + FeatureView, + RequestFeatureView, + OnDemandFeatureView, + StreamFeatureView, + FeatureService, ] ] = [] for object_type in FEAST_OBJECT_TYPES: diff --git a/sdk/python/feast/stream_feature_view.py b/sdk/python/feast/stream_feature_view.py index bba16e2627f..12d7f9b74b4 100644 --- a/sdk/python/feast/stream_feature_view.py +++ b/sdk/python/feast/stream_feature_view.py @@ -7,12 +7,16 @@ import dill from google.protobuf.duration_pb2 import Duration +from feast import utils from feast.aggregation import Aggregation from feast.data_source import DataSource, KafkaSource from feast.entity import Entity from feast.feature_view import FeatureView from feast.field import Field from feast.protos.feast.core.DataSource_pb2 import DataSource as DataSourceProto +from feast.protos.feast.core.FeatureView_pb2 import ( + MaterializationInterval as MaterializationIntervalProto, +) from feast.protos.feast.core.OnDemandFeatureView_pb2 import ( UserDefinedFunction as UserDefinedFunctionProto, ) @@ -70,7 +74,7 @@ def __init__( f"Stream feature views need a stream source, expected one of {SUPPORTED_STREAM_SOURCES} " f"or CUSTOM_SOURCE, got {type(source).__name__}: {source.name} instead " ) - self.aggregations = aggregations + self.aggregations = aggregations or [] self.mode = mode self.timestamp_field = timestamp_field self.udf = udf @@ -113,12 +117,18 @@ def __hash__(self): return super().__hash__() def to_proto(self): - meta = StreamFeatureViewMetaProto() + meta = StreamFeatureViewMetaProto(materialization_intervals=[]) if self.created_timestamp: meta.created_timestamp.FromDatetime(self.created_timestamp) if self.last_updated_timestamp: meta.last_updated_timestamp.FromDatetime(self.last_updated_timestamp) + for interval in self.materialization_intervals: + interval_proto = MaterializationIntervalProto() + interval_proto.start_time.FromDatetime(interval[0]) + interval_proto.end_time.FromDatetime(interval[1]) + meta.materialization_intervals.append(interval_proto) + ttl_duration = None if self.ttl is not None: ttl_duration = Duration() @@ -169,6 +179,11 @@ def from_proto(cls, sfv_proto): if sfv_proto.spec.HasField("stream_source") else None ) + udf = ( + dill.loads(sfv_proto.spec.user_defined_function.body) + if sfv_proto.spec.HasField("user_defined_function") + else None + ) sfv_feature_view = cls( name=sfv_proto.spec.name, description=sfv_proto.spec.description, @@ -185,7 +200,7 @@ def from_proto(cls, sfv_proto): ), source=stream_source, mode=sfv_proto.spec.mode, - udf=dill.loads(sfv_proto.spec.user_defined_function.body), + udf=udf, aggregations=[ Aggregation.from_proto(agg_proto) for agg_proto in sfv_proto.spec.aggregations @@ -214,6 +229,14 @@ def from_proto(cls, sfv_proto): sfv_proto.meta.last_updated_timestamp.ToDatetime() ) + for interval in sfv_proto.meta.materialization_intervals: + stream_feature_view.materialization_intervals.append( + ( + utils.make_tzaware(interval.start_time.ToDatetime()), + utils.make_tzaware(interval.end_time.ToDatetime()), + ) + ) + return sfv_feature_view diff --git a/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py b/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py index b01ca434fa8..d24618b2704 100644 --- a/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py +++ b/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py @@ -11,7 +11,7 @@ @pytest.mark.integration -def test_read_pre_applied(environment) -> None: +def test_apply_stream_feature_view(environment) -> None: """ Test apply of StreamFeatureView. """ @@ -60,3 +60,12 @@ def simple_sfv(df): entities = fs.list_entities() assert len(entities) == 1 assert entities[0] == entity + + features = fs.get_online_features( + features=["simple_sfv:dummy_field"], entity_rows=[{"test_key": 1001}], + ).to_dict(include_event_timestamps=True) + + assert "test_key" in features + assert features["test_key"] == [1001] + assert "dummy_field" in features + assert features["dummy_field"] == [None] From b3fe39c1600fa370f28c7b01e2b3f7da716449c1 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Fri, 3 Jun 2022 11:22:39 -0700 Subject: [PATCH 037/123] feat: SQLAlchemy Registry Support (#2734) * feat: WIP SQLAlchemy Registry Support Signed-off-by: Achal Shah * hack hack hack Signed-off-by: Achal Shah * fix repo config Signed-off-by: Achal Shah * reduce duplication Signed-off-by: Achal Shah * simplify Signed-off-by: Achal Shah * postgres tests Signed-off-by: Achal Shah * tests for pg and mysql Signed-off-by: Achal Shah * fix requirements Signed-off-by: Achal Shah * pin protobuf again Signed-off-by: Achal Shah * fix macos test Signed-off-by: Achal Shah * fix macos test Signed-off-by: Achal Shah * fix quotes Signed-off-by: Achal Shah * add macos Signed-off-by: Achal Shah * remove macos Signed-off-by: Achal Shah * install mysql library but don't try to run tests Signed-off-by: Achal Shah * Fix CR comments Signed-off-by: Achal Shah * add a comment Signed-off-by: Achal Shah --- .github/workflows/unit_tests.yml | 11 +- sdk/python/feast/feature_store.py | 8 +- sdk/python/feast/infra/registry_stores/sql.py | 453 +++++++++++++++++ sdk/python/feast/repo_config.py | 4 + .../requirements/py3.10-ci-requirements.txt | 15 +- .../requirements/py3.10-requirements.txt | 17 +- .../requirements/py3.7-ci-requirements.txt | 30 +- .../requirements/py3.7-requirements.txt | 22 +- .../requirements/py3.8-ci-requirements.txt | 29 +- .../requirements/py3.8-requirements.txt | 19 +- .../requirements/py3.9-ci-requirements.txt | 27 +- .../requirements/py3.9-requirements.txt | 19 +- .../registration/test_sql_registry.py | 474 ++++++++++++++++++ setup.py | 7 + 14 files changed, 1088 insertions(+), 47 deletions(-) create mode 100644 sdk/python/feast/infra/registry_stores/sql.py create mode 100644 sdk/python/tests/integration/registration/test_sql_registry.py diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml index 77198080537..ea6141f3310 100644 --- a/.github/workflows/unit_tests.yml +++ b/.github/workflows/unit_tests.yml @@ -8,7 +8,7 @@ jobs: fail-fast: false matrix: python-version: [ "3.7", "3.8", "3.9", "3.10" ] - os: [ ubuntu-latest, macOS-latest] + os: [ ubuntu-latest, macOS-latest ] exclude: - os: macOS-latest python-version: "3.8" @@ -32,9 +32,14 @@ jobs: uses: actions/setup-go@v2 with: go-version: 1.17.7 + - name: Install mysql on macOS + if: startsWith(matrix.os, 'macOS') + run: | + brew install mysql + PATH=$PATH:/usr/local/mysql/bin - name: Upgrade pip version run: | - pip install --upgrade "pip>=21.3.1,<22.1" + pip install --upgrade "pip>=22.1,<23" - name: Get pip cache dir id: pip-cache run: | @@ -83,7 +88,7 @@ jobs: python-version: "3.7" - name: Upgrade pip version run: | - pip install --upgrade "pip>=21.3.1,<22.1" + pip install --upgrade "pip>=22.1,<23" - name: Setup Go id: setup-go uses: actions/setup-go@v2 diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py index 9530b66d5e3..119b86d6c1e 100644 --- a/sdk/python/feast/feature_store.py +++ b/sdk/python/feast/feature_store.py @@ -71,6 +71,7 @@ ) from feast.infra.infra_object import Infra from feast.infra.provider import Provider, RetrievalJob, get_provider +from feast.infra.registry_stores.sql import SqlRegistry from feast.on_demand_feature_view import OnDemandFeatureView from feast.online_response import OnlineResponse from feast.protos.feast.core.InfraObject_pb2 import Infra as InfraProto @@ -138,8 +139,11 @@ def __init__( raise ValueError("Please specify one of repo_path or config.") registry_config = self.config.get_registry_config() - self._registry = Registry(registry_config, repo_path=self.repo_path) - self._registry._initialize_registry() + if registry_config.registry_type == "sql": + self._registry = SqlRegistry(registry_config, None) + else: + self._registry = Registry(registry_config, repo_path=self.repo_path) + self._registry._initialize_registry() self._provider = get_provider(self.config, self.repo_path) self._go_server = None diff --git a/sdk/python/feast/infra/registry_stores/sql.py b/sdk/python/feast/infra/registry_stores/sql.py new file mode 100644 index 00000000000..f13428b7377 --- /dev/null +++ b/sdk/python/feast/infra/registry_stores/sql.py @@ -0,0 +1,453 @@ +from datetime import datetime +from pathlib import Path +from threading import Lock +from typing import List, Optional + +from sqlalchemy import ( # type: ignore + BigInteger, + Column, + LargeBinary, + MetaData, + String, + Table, + create_engine, + delete, + insert, + select, + update, +) +from sqlalchemy.engine import Engine + +from feast.base_feature_view import BaseFeatureView +from feast.data_source import DataSource +from feast.entity import Entity +from feast.errors import ( + DataSourceObjectNotFoundException, + EntityNotFoundException, + FeatureServiceNotFoundException, + FeatureViewNotFoundException, + SavedDatasetNotFound, + ValidationReferenceNotFound, +) +from feast.feature_service import FeatureService +from feast.feature_view import FeatureView +from feast.on_demand_feature_view import OnDemandFeatureView +from feast.protos.feast.core.DataSource_pb2 import DataSource as DataSourceProto +from feast.protos.feast.core.Entity_pb2 import Entity as EntityProto +from feast.protos.feast.core.FeatureService_pb2 import ( + FeatureService as FeatureServiceProto, +) +from feast.protos.feast.core.FeatureView_pb2 import FeatureView as FeatureViewProto +from feast.protos.feast.core.OnDemandFeatureView_pb2 import ( + OnDemandFeatureView as OnDemandFeatureViewProto, +) +from feast.protos.feast.core.RequestFeatureView_pb2 import ( + RequestFeatureView as RequestFeatureViewProto, +) +from feast.protos.feast.core.SavedDataset_pb2 import SavedDataset as SavedDatasetProto +from feast.protos.feast.core.ValidationProfile_pb2 import ( + ValidationReference as ValidationReferenceProto, +) +from feast.registry import Registry +from feast.repo_config import RegistryConfig +from feast.request_feature_view import RequestFeatureView +from feast.saved_dataset import SavedDataset, ValidationReference + +metadata = MetaData() + +entities = Table( + "entities", + metadata, + Column("entity_name", String(50), primary_key=True), + Column("last_updated_timestamp", BigInteger, nullable=False), + Column("entity_proto", LargeBinary, nullable=False), +) + +data_sources = Table( + "data_sources", + metadata, + Column("data_source_name", String(50), primary_key=True), + Column("last_updated_timestamp", BigInteger, nullable=False), + Column("data_source_proto", LargeBinary, nullable=False), +) + +feature_views = Table( + "feature_views", + metadata, + Column("feature_view_name", String(50), primary_key=True), + Column("last_updated_timestamp", BigInteger, nullable=False), + Column("materialized_intervals", LargeBinary, nullable=True), + Column("feature_view_proto", LargeBinary, nullable=False), +) + +request_feature_views = Table( + "request_feature_views", + metadata, + Column("feature_view_name", String(50), primary_key=True), + Column("last_updated_timestamp", BigInteger, nullable=False), + Column("feature_view_proto", LargeBinary, nullable=False), +) + +on_demand_feature_views = Table( + "on_demand_feature_views", + metadata, + Column("feature_view_name", String(50), primary_key=True), + Column("last_updated_timestamp", BigInteger, nullable=False), + Column("feature_view_proto", LargeBinary, nullable=False), +) + +feature_services = Table( + "feature_services", + metadata, + Column("feature_service_name", String(50), primary_key=True), + Column("last_updated_timestamp", BigInteger, nullable=False), + Column("feature_service_proto", LargeBinary, nullable=False), +) + +saved_datasets = Table( + "saved_datasets", + metadata, + Column("saved_dataset_name", String(50), primary_key=True), + Column("last_updated_timestamp", BigInteger, nullable=False), + Column("saved_dataset_proto", LargeBinary, nullable=False), +) + +validation_references = Table( + "validation_references", + metadata, + Column("validation_reference_name", String(50), primary_key=True), + Column("last_updated_timestamp", BigInteger, nullable=False), + Column("validation_reference_proto", LargeBinary, nullable=False), +) + + +class SqlRegistry(Registry): + def __init__( + self, registry_config: Optional[RegistryConfig], repo_path: Optional[Path] + ): + assert registry_config is not None, "SqlRegistry needs a valid registry_config" + self.engine: Engine = create_engine(registry_config.path, echo=False) + metadata.create_all(self.engine) + + # _refresh_lock is not used by the SqlRegistry, but is present to conform to the + # Registry class. + # TODO: remove external references to _refresh_lock and remove field. + self._refresh_lock = Lock() + + def teardown(self): + for t in { + entities, + data_sources, + feature_views, + feature_services, + on_demand_feature_views, + request_feature_views, + saved_datasets, + validation_references, + }: + with self.engine.connect() as conn: + stmt = delete(t) + conn.execute(stmt) + + def refresh(self): + pass + + def apply_entity(self, entity: Entity, project: str, commit: bool = True): + return self._apply_object(entities, "entity_name", entity, "entity_proto") + + def get_entity(self, name: str, project: str, allow_cache: bool = False) -> Entity: + return self._get_object( + entities, + name, + project, + EntityProto, + Entity, + "entity_name", + "entity_proto", + EntityNotFoundException, + ) + + def get_feature_view( + self, name: str, project: str, allow_cache: bool = False + ) -> FeatureView: + return self._get_object( + feature_views, + name, + project, + FeatureViewProto, + FeatureView, + "feature_view_name", + "feature_view_proto", + FeatureViewNotFoundException, + ) + + def get_on_demand_feature_view( + self, name: str, project: str, allow_cache: bool = False + ) -> OnDemandFeatureView: + return self._get_object( + on_demand_feature_views, + name, + project, + OnDemandFeatureViewProto, + OnDemandFeatureView, + "feature_view_name", + "feature_view_proto", + FeatureViewNotFoundException, + ) + + def get_feature_service( + self, name: str, project: str, allow_cache: bool = False + ) -> FeatureService: + return self._get_object( + feature_services, + name, + project, + FeatureServiceProto, + FeatureService, + "feature_service_name", + "feature_service_proto", + FeatureServiceNotFoundException, + ) + + def get_saved_dataset( + self, name: str, project: str, allow_cache: bool = False + ) -> SavedDataset: + return self._get_object( + saved_datasets, + name, + project, + SavedDatasetProto, + SavedDataset, + "saved_dataset_name", + "saved_dataset_proto", + SavedDatasetNotFound, + ) + + def get_validation_reference( + self, name: str, project: str, allow_cache: bool = False + ) -> ValidationReference: + return self._get_object( + validation_references, + name, + project, + ValidationReferenceProto, + ValidationReference, + "validation_reference_name", + "validation_reference_proto", + ValidationReferenceNotFound, + ) + + def list_entities(self, project: str, allow_cache: bool = False) -> List[Entity]: + return self._list_objects(entities, EntityProto, Entity, "entity_proto") + + def delete_entity(self, name: str, project: str, commit: bool = True): + with self.engine.connect() as conn: + stmt = delete(entities).where(entities.c.entity_name == name) + rows = conn.execute(stmt) + if rows.rowcount < 1: + raise EntityNotFoundException(name, project) + + def delete_feature_view(self, name: str, project: str, commit: bool = True): + deleted_count = 0 + for table in {feature_views, request_feature_views, on_demand_feature_views}: + with self.engine.connect() as conn: + stmt = delete(table).where(table.c.feature_view_name == name) + rows = conn.execute(stmt) + deleted_count += rows.rowcount + if deleted_count == 0: + raise FeatureViewNotFoundException(name, project) + + def delete_feature_service(self, name: str, project: str, commit: bool = True): + with self.engine.connect() as conn: + stmt = delete(feature_services).where( + feature_services.c.feature_service_name == name + ) + rows = conn.execute(stmt) + if rows.rowcount < 1: + raise FeatureServiceNotFoundException(name, project) + + def get_data_source( + self, name: str, project: str, allow_cache: bool = False + ) -> DataSource: + with self.engine.connect() as conn: + stmt = select(data_sources).where(data_sources.c.entity_name == name) + row = conn.execute(stmt).first() + if row: + ds_proto = DataSourceProto.FromString(row["data_source_proto"]) + return DataSource.from_proto(ds_proto) + raise DataSourceObjectNotFoundException(name, project=project) + + def list_data_sources( + self, project: str, allow_cache: bool = False + ) -> List[DataSource]: + return self._list_objects( + data_sources, DataSourceProto, DataSource, "data_source_proto" + ) + + def apply_data_source( + self, data_source: DataSource, project: str, commit: bool = True + ): + return self._apply_object( + data_sources, "data_source_name", data_source, "data_source_proto" + ) + + def apply_feature_view( + self, feature_view: BaseFeatureView, project: str, commit: bool = True + ): + # TODO(achals): Stream feature views need to be supported. + if isinstance(feature_view, FeatureView): + fv_table = feature_views + elif isinstance(feature_view, OnDemandFeatureView): + fv_table = on_demand_feature_views + elif isinstance(feature_view, RequestFeatureView): + fv_table = request_feature_views + else: + raise ValueError(f"Unexpected feature view type: {type(feature_view)}") + + return self._apply_object( + fv_table, "feature_view_name", feature_view, "feature_view_proto" + ) + + def apply_feature_service( + self, feature_service: FeatureService, project: str, commit: bool = True + ): + return self._apply_object( + feature_services, + "feature_service_name", + feature_service, + "feature_service_proto", + ) + + def delete_data_source(self, name: str, project: str, commit: bool = True): + with self.engine.connect() as conn: + stmt = delete(data_sources).where(data_sources.c.data_source_name == name) + rows = conn.execute(stmt) + if rows.rowcount < 1: + raise DataSourceObjectNotFoundException(name, project) + + def list_feature_services( + self, project: str, allow_cache: bool = False + ) -> List[FeatureService]: + return self._list_objects( + feature_services, + FeatureServiceProto, + FeatureService, + "feature_service_proto", + ) + + def list_feature_views( + self, project: str, allow_cache: bool = False + ) -> List[FeatureView]: + return self._list_objects( + feature_views, FeatureViewProto, FeatureView, "feature_view_proto" + ) + + def list_saved_datasets( + self, project: str, allow_cache: bool = False + ) -> List[SavedDataset]: + return self._list_objects( + saved_datasets, SavedDatasetProto, SavedDataset, "saved_dataset_proto" + ) + + def list_request_feature_views( + self, project: str, allow_cache: bool = False + ) -> List[RequestFeatureView]: + return self._list_objects( + request_feature_views, + RequestFeatureViewProto, + RequestFeatureView, + "feature_view_proto", + ) + + def list_on_demand_feature_views( + self, project: str, allow_cache: bool = False + ) -> List[OnDemandFeatureView]: + return self._list_objects( + on_demand_feature_views, + OnDemandFeatureViewProto, + OnDemandFeatureView, + "feature_view_proto", + ) + + def apply_saved_dataset( + self, saved_dataset: SavedDataset, project: str, commit: bool = True, + ): + return self._apply_object( + saved_datasets, "saved_dataset_name", saved_dataset, "saved_dataset_proto" + ) + + def apply_validation_reference( + self, + validation_reference: ValidationReference, + project: str, + commit: bool = True, + ): + return self._apply_object( + validation_references, + "validation_reference_name", + validation_reference, + "validation_reference_proto", + ) + + def _apply_object( + self, table, id_field_name, obj, proto_field_name, + ): + name = obj.name + with self.engine.connect() as conn: + stmt = select(table).where(getattr(table.c, id_field_name) == name) + row = conn.execute(stmt).first() + update_datetime = datetime.utcnow() + update_time = int(update_datetime.timestamp()) + if hasattr(obj, "last_updated_timestamp"): + obj.last_updated_timestamp = update_datetime + if row: + values = { + proto_field_name: obj.to_proto().SerializeToString(), + "last_updated_timestamp": update_time, + } + update_stmt = ( + update(table) + .where(getattr(table.c, id_field_name) == name) + .values(values,) + ) + conn.execute(update_stmt) + else: + values = { + id_field_name: name, + proto_field_name: obj.to_proto().SerializeToString(), + "last_updated_timestamp": update_time, + } + insert_stmt = insert(table).values(values,) + conn.execute(insert_stmt) + + def _list_objects(self, table, proto_class, python_class, proto_field_name): + with self.engine.connect() as conn: + stmt = select(table) + rows = conn.execute(stmt).all() + if rows: + return [ + python_class.from_proto( + proto_class.FromString(row[proto_field_name]) + ) + for row in rows + ] + return [] + + def _get_object( + self, + table, + name, + project, + proto_class, + python_class, + id_field_name, + proto_field_name, + not_found_exception, + ): + with self.engine.connect() as conn: + stmt = select(table).where(getattr(table.c, id_field_name) == name) + row = conn.execute(stmt).first() + if row: + _proto = proto_class.FromString(row[proto_field_name]) + return python_class.from_proto(_proto) + raise not_found_exception(name, project) diff --git a/sdk/python/feast/repo_config.py b/sdk/python/feast/repo_config.py index b7fd9c20377..b7cf1683dc6 100644 --- a/sdk/python/feast/repo_config.py +++ b/sdk/python/feast/repo_config.py @@ -82,6 +82,10 @@ class Config: class RegistryConfig(FeastBaseModel): """Metadata Store Configuration. Configuration that relates to reading from and writing to the Feast registry.""" + registry_type: StrictStr = "file" + """ str: Provider name or a class name that implements RegistryStore. + If specified, registry_store_type should be redundant.""" + registry_store_type: Optional[StrictStr] """ str: Provider name or a class name that implements RegistryStore. """ diff --git a/sdk/python/requirements/py3.10-ci-requirements.txt b/sdk/python/requirements/py3.10-ci-requirements.txt index 0098c057972..bef786ecaa9 100644 --- a/sdk/python/requirements/py3.10-ci-requirements.txt +++ b/sdk/python/requirements/py3.10-ci-requirements.txt @@ -122,7 +122,7 @@ colorama==0.4.4 # via # feast (setup.py) # great-expectations -coverage[toml]==6.4 +coverage[toml]==6.4.1 # via pytest-cov cryptography==35.0.0 # via @@ -249,6 +249,8 @@ googleapis-common-protos==1.56.2 # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) +greenlet==1.1.2 + # via sqlalchemy grpcio==1.46.3 # via # feast (setup.py) @@ -359,11 +361,15 @@ multidict==6.0.2 # aiohttp # yarl mypy==0.931 - # via feast (setup.py) + # via + # feast (setup.py) + # sqlalchemy mypy-extensions==0.4.3 # via mypy mypy-protobuf==3.1 # via feast (setup.py) +mysqlclient==2.1.0 + # via feast (setup.py) nbformat==5.4.0 # via great-expectations nodeenv==1.6.0 @@ -647,6 +653,10 @@ sphinxcontrib-qthelp==1.0.3 # via sphinx sphinxcontrib-serializinghtml==1.1.5 # via sphinx +sqlalchemy[mypy]==1.4.37 + # via feast (setup.py) +sqlalchemy2-stubs==0.0.2a22 + # via sqlalchemy stack-data==0.2.0 # via ipython starlette==0.19.1 @@ -720,6 +730,7 @@ typing-extensions==4.2.0 # great-expectations # mypy # pydantic + # sqlalchemy2-stubs tzdata==2022.1 # via pytz-deprecation-shim tzlocal==4.2 diff --git a/sdk/python/requirements/py3.10-requirements.txt b/sdk/python/requirements/py3.10-requirements.txt index 00b14d2cfe7..540455fd99f 100644 --- a/sdk/python/requirements/py3.10-requirements.txt +++ b/sdk/python/requirements/py3.10-requirements.txt @@ -49,6 +49,8 @@ googleapis-common-protos==1.56.2 # feast (setup.py) # google-api-core # tensorflow-metadata +greenlet==1.1.2 + # via sqlalchemy grpcio==1.46.3 # via # feast (setup.py) @@ -73,6 +75,10 @@ markupsafe==2.1.1 # via jinja2 mmh3==3.0.0 # via feast (setup.py) +mypy==0.960 + # via sqlalchemy +mypy-extensions==0.4.3 + # via mypy numpy==1.21.6 # via # feast (setup.py) @@ -140,6 +146,10 @@ six==1.16.0 # python-dateutil sniffio==1.2.0 # via anyio +sqlalchemy[mypy]==1.4.37 + # via feast (setup.py) +sqlalchemy2-stubs==0.0.2a22 + # via sqlalchemy starlette==0.19.1 # via fastapi tabulate==0.8.9 @@ -150,6 +160,8 @@ tensorflow-metadata==1.8.0 # via feast (setup.py) toml==0.10.2 # via feast (setup.py) +tomli==2.0.1 + # via mypy toolz==0.11.2 # via # dask @@ -157,7 +169,10 @@ toolz==0.11.2 tqdm==4.64.0 # via feast (setup.py) typing-extensions==4.2.0 - # via pydantic + # via + # mypy + # pydantic + # sqlalchemy2-stubs urllib3==1.26.9 # via requests uvicorn[standard]==0.17.6 diff --git a/sdk/python/requirements/py3.7-ci-requirements.txt b/sdk/python/requirements/py3.7-ci-requirements.txt index ce24e767b4e..1c9e2f37461 100644 --- a/sdk/python/requirements/py3.7-ci-requirements.txt +++ b/sdk/python/requirements/py3.7-ci-requirements.txt @@ -4,7 +4,7 @@ # # pip-compile --extra=ci --output-file=sdk/python/requirements/py3.7-ci-requirements.txt # -absl-py==1.0.0 +absl-py==1.1.0 # via tensorflow-metadata adal==1.2.7 # via @@ -58,7 +58,7 @@ attrs==21.4.0 # pytest avro==1.10.0 # via feast (setup.py) -azure-core==1.24.0 +azure-core==1.24.1 # via # adlfs # azure-identity @@ -126,7 +126,7 @@ colorama==0.4.4 # via # feast (setup.py) # great-expectations -coverage[toml]==6.4 +coverage[toml]==6.4.1 # via pytest-cov cryptography==35.0.0 # via @@ -231,7 +231,7 @@ google-cloud-core==1.7.2 # google-cloud-storage google-cloud-datastore==2.6.1 # via feast (setup.py) -google-cloud-firestore==2.5.1 +google-cloud-firestore==2.5.2 # via firebase-admin google-cloud-storage==1.40.0 # via @@ -251,6 +251,8 @@ googleapis-common-protos==1.56.2 # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) +greenlet==1.1.2 + # via sqlalchemy grpcio==1.46.3 # via # feast (setup.py) @@ -300,6 +302,7 @@ importlib-metadata==4.2.0 # pre-commit # pytest # redis + # sqlalchemy # virtualenv importlib-resources==5.7.1 # via jsonschema @@ -328,7 +331,7 @@ jsonpatch==1.32 # via great-expectations jsonpointer==2.3 # via jsonpatch -jsonschema==4.5.1 +jsonschema==4.6.0 # via # altair # feast (setup.py) @@ -375,11 +378,15 @@ multidict==6.0.2 # aiohttp # yarl mypy==0.931 - # via feast (setup.py) + # via + # feast (setup.py) + # sqlalchemy mypy-extensions==0.4.3 # via mypy mypy-protobuf==3.1 # via feast (setup.py) +mysqlclient==2.1.0 + # via feast (setup.py) nbformat==5.4.0 # via great-expectations nodeenv==1.6.0 @@ -459,6 +466,7 @@ protobuf==3.19.4 # feast (setup.py) # google-api-core # google-cloud-bigquery + # google-cloud-firestore # googleapis-common-protos # grpcio-reflection # grpcio-testing @@ -624,7 +632,6 @@ scipy==1.7.3 # via great-expectations six==1.16.0 # via - # absl-py # azure-core # azure-identity # google-api-core @@ -663,6 +670,10 @@ sphinxcontrib-qthelp==1.0.3 # via sphinx sphinxcontrib-serializinghtml==1.1.5 # via sphinx +sqlalchemy[mypy]==1.4.37 + # via feast (setup.py) +sqlalchemy2-stubs==0.0.2a22 + # via sqlalchemy starlette==0.19.1 # via fastapi tabulate==0.8.9 @@ -718,11 +729,11 @@ types-python-dateutil==2.8.17 # via feast (setup.py) types-pytz==2021.3.8 # via feast (setup.py) -types-pyyaml==6.0.7 +types-pyyaml==6.0.8 # via feast (setup.py) types-redis==4.2.6 # via feast (setup.py) -types-requests==2.27.29 +types-requests==2.27.30 # via feast (setup.py) types-setuptools==57.4.17 # via feast (setup.py) @@ -746,6 +757,7 @@ typing-extensions==4.2.0 # pydantic # redis # responses + # sqlalchemy2-stubs # starlette # uvicorn # yarl diff --git a/sdk/python/requirements/py3.7-requirements.txt b/sdk/python/requirements/py3.7-requirements.txt index 85d3e2ee096..79c5a997977 100644 --- a/sdk/python/requirements/py3.7-requirements.txt +++ b/sdk/python/requirements/py3.7-requirements.txt @@ -4,7 +4,7 @@ # # pip-compile --output-file=sdk/python/requirements/py3.7-requirements.txt # -absl-py==1.0.0 +absl-py==1.1.0 # via tensorflow-metadata anyio==3.6.1 # via @@ -49,6 +49,8 @@ googleapis-common-protos==1.56.2 # feast (setup.py) # google-api-core # tensorflow-metadata +greenlet==1.1.2 + # via sqlalchemy grpcio==1.46.3 # via # feast (setup.py) @@ -67,11 +69,12 @@ importlib-metadata==4.11.4 # via # click # jsonschema + # sqlalchemy importlib-resources==5.7.1 # via jsonschema jinja2==3.1.2 # via feast (setup.py) -jsonschema==4.5.1 +jsonschema==4.6.0 # via feast (setup.py) locket==1.0.0 # via partd @@ -79,6 +82,10 @@ markupsafe==2.1.1 # via jinja2 mmh3==3.0.0 # via feast (setup.py) +mypy==0.960 + # via sqlalchemy +mypy-extensions==0.4.3 + # via mypy numpy==1.21.6 # via # feast (setup.py) @@ -140,13 +147,16 @@ rsa==4.8 # via google-auth six==1.16.0 # via - # absl-py # google-auth # grpcio # pandavro # python-dateutil sniffio==1.2.0 # via anyio +sqlalchemy[mypy]==1.4.37 + # via feast (setup.py) +sqlalchemy2-stubs==0.0.2a22 + # via sqlalchemy starlette==0.19.1 # via fastapi tabulate==0.8.9 @@ -157,12 +167,16 @@ tensorflow-metadata==1.8.0 # via feast (setup.py) toml==0.10.2 # via feast (setup.py) +tomli==2.0.1 + # via mypy toolz==0.11.2 # via # dask # partd tqdm==4.64.0 # via feast (setup.py) +typed-ast==1.5.4 + # via mypy typing-extensions==4.2.0 # via # anyio @@ -170,7 +184,9 @@ typing-extensions==4.2.0 # h11 # importlib-metadata # jsonschema + # mypy # pydantic + # sqlalchemy2-stubs # starlette # uvicorn urllib3==1.26.9 diff --git a/sdk/python/requirements/py3.8-ci-requirements.txt b/sdk/python/requirements/py3.8-ci-requirements.txt index 8de792c0734..f4429c14bc9 100644 --- a/sdk/python/requirements/py3.8-ci-requirements.txt +++ b/sdk/python/requirements/py3.8-ci-requirements.txt @@ -4,7 +4,7 @@ # # pip-compile --extra=ci --output-file=sdk/python/requirements/py3.8-ci-requirements.txt # -absl-py==1.0.0 +absl-py==1.1.0 # via tensorflow-metadata adal==1.2.7 # via @@ -58,7 +58,7 @@ attrs==21.4.0 # pytest avro==1.10.0 # via feast (setup.py) -azure-core==1.24.0 +azure-core==1.24.1 # via # adlfs # azure-identity @@ -126,7 +126,7 @@ colorama==0.4.4 # via # feast (setup.py) # great-expectations -coverage[toml]==6.4 +coverage[toml]==6.4.1 # via pytest-cov cryptography==35.0.0 # via @@ -233,7 +233,7 @@ google-cloud-core==1.7.2 # google-cloud-storage google-cloud-datastore==2.6.1 # via feast (setup.py) -google-cloud-firestore==2.5.1 +google-cloud-firestore==2.5.2 # via firebase-admin google-cloud-storage==1.40.0 # via @@ -253,6 +253,8 @@ googleapis-common-protos==1.56.2 # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) +greenlet==1.1.2 + # via sqlalchemy grpcio==1.46.3 # via # feast (setup.py) @@ -318,7 +320,7 @@ jsonpatch==1.32 # via great-expectations jsonpointer==2.3 # via jsonpatch -jsonschema==4.5.1 +jsonschema==4.6.0 # via # altair # feast (setup.py) @@ -365,11 +367,15 @@ multidict==6.0.2 # aiohttp # yarl mypy==0.931 - # via feast (setup.py) + # via + # feast (setup.py) + # sqlalchemy mypy-extensions==0.4.3 # via mypy mypy-protobuf==3.1 # via feast (setup.py) +mysqlclient==2.1.0 + # via feast (setup.py) nbformat==5.4.0 # via great-expectations nodeenv==1.6.0 @@ -449,6 +455,7 @@ protobuf==3.19.4 # feast (setup.py) # google-api-core # google-cloud-bigquery + # google-cloud-firestore # googleapis-common-protos # grpcio-reflection # grpcio-testing @@ -616,7 +623,6 @@ scipy==1.8.1 # via great-expectations six==1.16.0 # via - # absl-py # azure-core # azure-identity # google-api-core @@ -655,6 +661,10 @@ sphinxcontrib-qthelp==1.0.3 # via sphinx sphinxcontrib-serializinghtml==1.1.5 # via sphinx +sqlalchemy[mypy]==1.4.37 + # via feast (setup.py) +sqlalchemy2-stubs==0.0.2a22 + # via sqlalchemy stack-data==0.2.0 # via ipython starlette==0.19.1 @@ -710,11 +720,11 @@ types-python-dateutil==2.8.17 # via feast (setup.py) types-pytz==2021.3.8 # via feast (setup.py) -types-pyyaml==6.0.7 +types-pyyaml==6.0.8 # via feast (setup.py) types-redis==4.2.6 # via feast (setup.py) -types-requests==2.27.29 +types-requests==2.27.30 # via feast (setup.py) types-setuptools==57.4.17 # via feast (setup.py) @@ -729,6 +739,7 @@ typing-extensions==4.2.0 # great-expectations # mypy # pydantic + # sqlalchemy2-stubs # starlette tzdata==2022.1 # via pytz-deprecation-shim diff --git a/sdk/python/requirements/py3.8-requirements.txt b/sdk/python/requirements/py3.8-requirements.txt index ae49676bd03..2c616f40f86 100644 --- a/sdk/python/requirements/py3.8-requirements.txt +++ b/sdk/python/requirements/py3.8-requirements.txt @@ -4,7 +4,7 @@ # # pip-compile --output-file=sdk/python/requirements/py3.8-requirements.txt # -absl-py==1.0.0 +absl-py==1.1.0 # via tensorflow-metadata anyio==3.6.1 # via @@ -49,6 +49,8 @@ googleapis-common-protos==1.56.2 # feast (setup.py) # google-api-core # tensorflow-metadata +greenlet==1.1.2 + # via sqlalchemy grpcio==1.46.3 # via # feast (setup.py) @@ -67,7 +69,7 @@ importlib-resources==5.7.1 # via jsonschema jinja2==3.1.2 # via feast (setup.py) -jsonschema==4.5.1 +jsonschema==4.6.0 # via feast (setup.py) locket==1.0.0 # via partd @@ -75,6 +77,10 @@ markupsafe==2.1.1 # via jinja2 mmh3==3.0.0 # via feast (setup.py) +mypy==0.960 + # via sqlalchemy +mypy-extensions==0.4.3 + # via mypy numpy==1.21.6 # via # feast (setup.py) @@ -136,13 +142,16 @@ rsa==4.8 # via google-auth six==1.16.0 # via - # absl-py # google-auth # grpcio # pandavro # python-dateutil sniffio==1.2.0 # via anyio +sqlalchemy[mypy]==1.4.37 + # via feast (setup.py) +sqlalchemy2-stubs==0.0.2a22 + # via sqlalchemy starlette==0.19.1 # via fastapi tabulate==0.8.9 @@ -153,6 +162,8 @@ tensorflow-metadata==1.8.0 # via feast (setup.py) toml==0.10.2 # via feast (setup.py) +tomli==2.0.1 + # via mypy toolz==0.11.2 # via # dask @@ -161,7 +172,9 @@ tqdm==4.64.0 # via feast (setup.py) typing-extensions==4.2.0 # via + # mypy # pydantic + # sqlalchemy2-stubs # starlette urllib3==1.26.9 # via requests diff --git a/sdk/python/requirements/py3.9-ci-requirements.txt b/sdk/python/requirements/py3.9-ci-requirements.txt index e8741a237ba..156b7656921 100644 --- a/sdk/python/requirements/py3.9-ci-requirements.txt +++ b/sdk/python/requirements/py3.9-ci-requirements.txt @@ -1,5 +1,5 @@ # -# This file is autogenerated by pip-compile with python 3.8 +# This file is autogenerated by pip-compile with python 3.9 # To update, run: # # pip-compile --extra=ci --output-file=sdk/python/requirements/py3.9-ci-requirements.txt @@ -73,10 +73,6 @@ babel==2.10.1 # via sphinx backcall==0.2.0 # via ipython -backports-zoneinfo==0.2.1 - # via - # pytz-deprecation-shim - # tzlocal black==19.10b0 # via feast (setup.py) boto3==1.20.23 @@ -126,7 +122,7 @@ colorama==0.4.4 # via # feast (setup.py) # great-expectations -coverage[toml]==6.4 +coverage[toml]==6.4.1 # via pytest-cov cryptography==35.0.0 # via @@ -253,6 +249,8 @@ googleapis-common-protos==1.56.2 # tensorflow-metadata great-expectations==0.14.13 # via feast (setup.py) +greenlet==1.1.2 + # via sqlalchemy grpcio==1.46.3 # via # feast (setup.py) @@ -291,8 +289,6 @@ imagesize==1.3.0 # via sphinx importlib-metadata==4.11.4 # via great-expectations -importlib-resources==5.7.1 - # via jsonschema iniconfig==1.1.1 # via pytest ipython==8.4.0 @@ -365,11 +361,15 @@ multidict==6.0.2 # aiohttp # yarl mypy==0.931 - # via feast (setup.py) + # via + # feast (setup.py) + # sqlalchemy mypy-extensions==0.4.3 # via mypy mypy-protobuf==3.1 # via feast (setup.py) +mysqlclient==2.1.0 + # via feast (setup.py) nbformat==5.4.0 # via great-expectations nodeenv==1.6.0 @@ -655,6 +655,10 @@ sphinxcontrib-qthelp==1.0.3 # via sphinx sphinxcontrib-serializinghtml==1.1.5 # via sphinx +sqlalchemy[mypy]==1.4.37 + # via feast (setup.py) +sqlalchemy2-stubs==0.0.2a22 + # via sqlalchemy stack-data==0.2.0 # via ipython starlette==0.19.1 @@ -729,6 +733,7 @@ typing-extensions==4.2.0 # great-expectations # mypy # pydantic + # sqlalchemy2-stubs # starlette tzdata==2022.1 # via pytz-deprecation-shim @@ -772,9 +777,7 @@ xmltodict==0.13.0 yarl==1.7.2 # via aiohttp zipp==3.8.0 - # via - # importlib-metadata - # importlib-resources + # via importlib-metadata # The following packages are considered to be unsafe in a requirements file: # pip diff --git a/sdk/python/requirements/py3.9-requirements.txt b/sdk/python/requirements/py3.9-requirements.txt index 577ff2838f9..245542b64f0 100644 --- a/sdk/python/requirements/py3.9-requirements.txt +++ b/sdk/python/requirements/py3.9-requirements.txt @@ -4,7 +4,7 @@ # # pip-compile --output-file=sdk/python/requirements/py3.9-requirements.txt # -absl-py==1.0.0 +absl-py==1.1.0 # via tensorflow-metadata anyio==3.6.1 # via @@ -49,6 +49,8 @@ googleapis-common-protos==1.56.2 # feast (setup.py) # google-api-core # tensorflow-metadata +greenlet==1.1.2 + # via sqlalchemy grpcio==1.46.3 # via # feast (setup.py) @@ -65,7 +67,7 @@ idna==3.3 # requests jinja2==3.1.2 # via feast (setup.py) -jsonschema==4.5.1 +jsonschema==4.6.0 # via feast (setup.py) locket==1.0.0 # via partd @@ -73,6 +75,10 @@ markupsafe==2.1.1 # via jinja2 mmh3==3.0.0 # via feast (setup.py) +mypy==0.960 + # via sqlalchemy +mypy-extensions==0.4.3 + # via mypy numpy==1.21.6 # via # feast (setup.py) @@ -134,13 +140,16 @@ rsa==4.8 # via google-auth six==1.16.0 # via - # absl-py # google-auth # grpcio # pandavro # python-dateutil sniffio==1.2.0 # via anyio +sqlalchemy[mypy]==1.4.37 + # via feast (setup.py) +sqlalchemy2-stubs==0.0.2a22 + # via sqlalchemy starlette==0.19.1 # via fastapi tabulate==0.8.9 @@ -151,6 +160,8 @@ tensorflow-metadata==1.8.0 # via feast (setup.py) toml==0.10.2 # via feast (setup.py) +tomli==2.0.1 + # via mypy toolz==0.11.2 # via # dask @@ -159,7 +170,9 @@ tqdm==4.64.0 # via feast (setup.py) typing-extensions==4.2.0 # via + # mypy # pydantic + # sqlalchemy2-stubs # starlette urllib3==1.26.9 # via requests diff --git a/sdk/python/tests/integration/registration/test_sql_registry.py b/sdk/python/tests/integration/registration/test_sql_registry.py new file mode 100644 index 00000000000..efad9f2c812 --- /dev/null +++ b/sdk/python/tests/integration/registration/test_sql_registry.py @@ -0,0 +1,474 @@ +# Copyright 2021 The Feast Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import logging +import sys +from datetime import timedelta + +import pandas as pd +import pytest +from pytest_lazyfixture import lazy_fixture +from testcontainers.core.container import DockerContainer +from testcontainers.core.waiting_utils import wait_for_logs + +from feast import Feature, FileSource, RequestSource +from feast.data_format import ParquetFormat +from feast.entity import Entity +from feast.feature_view import FeatureView +from feast.field import Field +from feast.infra.registry_stores.sql import SqlRegistry +from feast.on_demand_feature_view import on_demand_feature_view +from feast.repo_config import RegistryConfig +from feast.types import Array, Bytes, Float32, Int32, Int64, String +from feast.value_type import ValueType + +POSTGRES_USER = "test" +POSTGRES_PASSWORD = "test" +POSTGRES_DB = "test" + + +logger = logging.getLogger(__name__) + + +@pytest.fixture(scope="session") +def pg_registry(): + container = ( + DockerContainer("postgres:latest") + .with_exposed_ports(5432) + .with_env("POSTGRES_USER", POSTGRES_USER) + .with_env("POSTGRES_PASSWORD", POSTGRES_PASSWORD) + .with_env("POSTGRES_DB", POSTGRES_DB) + ) + + container.start() + + log_string_to_wait_for = "database system is ready to accept connections" + waited = wait_for_logs( + container=container, predicate=log_string_to_wait_for, timeout=30, interval=10, + ) + logger.info("Waited for %s seconds until postgres container was up", waited) + container_port = container.get_exposed_port(5432) + + registry_config = RegistryConfig( + registry_type="sql", + path=f"postgresql://{POSTGRES_USER}:{POSTGRES_PASSWORD}@127.0.0.1:{container_port}/{POSTGRES_DB}", + ) + + yield SqlRegistry(registry_config, None) + + container.stop() + + +@pytest.fixture(scope="session") +def mysql_registry(): + container = ( + DockerContainer("mysql:latest") + .with_exposed_ports(3306) + .with_env("MYSQL_RANDOM_ROOT_PASSWORD", "true") + .with_env("MYSQL_USER", POSTGRES_USER) + .with_env("MYSQL_PASSWORD", POSTGRES_PASSWORD) + .with_env("MYSQL_DATABASE", POSTGRES_DB) + ) + + container.start() + + log_string_to_wait_for = "/usr/sbin/mysqld: ready for connections. Version: '8.0.29' socket: '/var/run/mysqld/mysqld.sock' port: 3306" + waited = wait_for_logs( + container=container, predicate=log_string_to_wait_for, timeout=30, interval=10, + ) + logger.info("Waited for %s seconds until mysql container was up", waited) + container_port = container.get_exposed_port(3306) + + registry_config = RegistryConfig( + registry_type="sql", + path=f"mysql+mysqldb://{POSTGRES_USER}:{POSTGRES_PASSWORD}@127.0.0.1:{container_port}/{POSTGRES_DB}", + ) + + yield SqlRegistry(registry_config, None) + + container.stop() + + +@pytest.mark.skipif( + sys.platform == "darwin", reason="does not run on mac github actions" +) +@pytest.mark.parametrize( + "sql_registry", [lazy_fixture("mysql_registry"), lazy_fixture("pg_registry")], +) +def test_apply_entity_success(sql_registry): + entity = Entity( + name="driver_car_id", description="Car driver id", tags={"team": "matchmaking"}, + ) + + project = "project" + + # Register Entity + sql_registry.apply_entity(entity, project) + + entities = sql_registry.list_entities(project) + + entity = entities[0] + assert ( + len(entities) == 1 + and entity.name == "driver_car_id" + and entity.description == "Car driver id" + and "team" in entity.tags + and entity.tags["team"] == "matchmaking" + ) + + entity = sql_registry.get_entity("driver_car_id", project) + assert ( + entity.name == "driver_car_id" + and entity.description == "Car driver id" + and "team" in entity.tags + and entity.tags["team"] == "matchmaking" + ) + + sql_registry.delete_entity("driver_car_id", project) + entities = sql_registry.list_entities(project) + assert len(entities) == 0 + + sql_registry.teardown() + + +@pytest.mark.skipif( + sys.platform == "darwin", reason="does not run on mac github actions" +) +@pytest.mark.parametrize( + "sql_registry", [lazy_fixture("mysql_registry"), lazy_fixture("pg_registry")], +) +def test_apply_feature_view_success(sql_registry): + # Create Feature Views + batch_source = FileSource( + file_format=ParquetFormat(), + path="file://feast/*", + timestamp_field="ts_col", + created_timestamp_column="timestamp", + ) + + entity = Entity(name="fs1_my_entity_1", join_keys=["test"]) + + fv1 = FeatureView( + name="my_feature_view_1", + schema=[ + Field(name="fs1_my_feature_1", dtype=Int64), + Field(name="fs1_my_feature_2", dtype=String), + Field(name="fs1_my_feature_3", dtype=Array(String)), + Field(name="fs1_my_feature_4", dtype=Array(Bytes)), + ], + entities=[entity], + tags={"team": "matchmaking"}, + batch_source=batch_source, + ttl=timedelta(minutes=5), + ) + + project = "project" + + # Register Feature View + sql_registry.apply_feature_view(fv1, project) + + feature_views = sql_registry.list_feature_views(project) + + # List Feature Views + assert ( + len(feature_views) == 1 + and feature_views[0].name == "my_feature_view_1" + and feature_views[0].features[0].name == "fs1_my_feature_1" + and feature_views[0].features[0].dtype == Int64 + and feature_views[0].features[1].name == "fs1_my_feature_2" + and feature_views[0].features[1].dtype == String + and feature_views[0].features[2].name == "fs1_my_feature_3" + and feature_views[0].features[2].dtype == Array(String) + and feature_views[0].features[3].name == "fs1_my_feature_4" + and feature_views[0].features[3].dtype == Array(Bytes) + and feature_views[0].entities[0] == "fs1_my_entity_1" + ) + + feature_view = sql_registry.get_feature_view("my_feature_view_1", project) + assert ( + feature_view.name == "my_feature_view_1" + and feature_view.features[0].name == "fs1_my_feature_1" + and feature_view.features[0].dtype == Int64 + and feature_view.features[1].name == "fs1_my_feature_2" + and feature_view.features[1].dtype == String + and feature_view.features[2].name == "fs1_my_feature_3" + and feature_view.features[2].dtype == Array(String) + and feature_view.features[3].name == "fs1_my_feature_4" + and feature_view.features[3].dtype == Array(Bytes) + and feature_view.entities[0] == "fs1_my_entity_1" + ) + + sql_registry.delete_feature_view("my_feature_view_1", project) + feature_views = sql_registry.list_feature_views(project) + assert len(feature_views) == 0 + + sql_registry.teardown() + + +@pytest.mark.skipif( + sys.platform == "darwin", reason="does not run on mac github actions" +) +@pytest.mark.parametrize( + "sql_registry", [lazy_fixture("mysql_registry"), lazy_fixture("pg_registry")], +) +def test_apply_on_demand_feature_view_success(sql_registry): + # Create Feature Views + driver_stats = FileSource( + name="driver_stats_source", + path="data/driver_stats_lat_lon.parquet", + timestamp_field="event_timestamp", + created_timestamp_column="created", + description="A table describing the stats of a driver based on hourly logs", + owner="test2@gmail.com", + ) + + driver_daily_features_view = FeatureView( + name="driver_daily_features", + entities=["driver"], + ttl=timedelta(seconds=8640000000), + schema=[ + Field(name="daily_miles_driven", dtype=Float32), + Field(name="lat", dtype=Float32), + Field(name="lon", dtype=Float32), + Field(name="string_feature", dtype=String), + ], + online=True, + source=driver_stats, + tags={"production": "True"}, + owner="test2@gmail.com", + ) + + @on_demand_feature_view( + sources=[driver_daily_features_view], + schema=[Field(name="first_char", dtype=String)], + ) + def location_features_from_push(inputs: pd.DataFrame) -> pd.DataFrame: + df = pd.DataFrame() + df["first_char"] = inputs["string_feature"].str[:1].astype("string") + return df + + project = "project" + + # Register Feature View + sql_registry.apply_feature_view(location_features_from_push, project) + + feature_views = sql_registry.list_on_demand_feature_views(project) + + # List Feature Views + assert ( + len(feature_views) == 1 + and feature_views[0].name == "location_features_from_push" + and feature_views[0].features[0].name == "first_char" + and feature_views[0].features[0].dtype == String + ) + + feature_view = sql_registry.get_on_demand_feature_view( + "location_features_from_push", project + ) + assert ( + feature_view.name == "location_features_from_push" + and feature_view.features[0].name == "first_char" + and feature_view.features[0].dtype == String + ) + + sql_registry.delete_feature_view("location_features_from_push", project) + feature_views = sql_registry.list_on_demand_feature_views(project) + assert len(feature_views) == 0 + + sql_registry.teardown() + + +@pytest.mark.skipif( + sys.platform == "darwin", reason="does not run on mac github actions" +) +@pytest.mark.parametrize( + "sql_registry", [lazy_fixture("mysql_registry"), lazy_fixture("pg_registry")], +) +@pytest.mark.parametrize( + "request_source_schema", + [[Field(name="my_input_1", dtype=Int32)], {"my_input_1": ValueType.INT32}], +) +def test_modify_feature_views_success(sql_registry, request_source_schema): + # Create Feature Views + batch_source = FileSource( + file_format=ParquetFormat(), + path="file://feast/*", + timestamp_field="ts_col", + created_timestamp_column="timestamp", + ) + + request_source = RequestSource(name="request_source", schema=request_source_schema,) + + entity = Entity(name="fs1_my_entity_1", join_keys=["test"]) + + fv1 = FeatureView( + name="my_feature_view_1", + schema=[Field(name="fs1_my_feature_1", dtype=Int64)], + entities=[entity], + tags={"team": "matchmaking"}, + batch_source=batch_source, + ttl=timedelta(minutes=5), + ) + + @on_demand_feature_view( + features=[ + Feature(name="odfv1_my_feature_1", dtype=ValueType.STRING), + Feature(name="odfv1_my_feature_2", dtype=ValueType.INT32), + ], + sources=[request_source], + ) + def odfv1(feature_df: pd.DataFrame) -> pd.DataFrame: + data = pd.DataFrame() + data["odfv1_my_feature_1"] = feature_df["my_input_1"].astype("category") + data["odfv1_my_feature_2"] = feature_df["my_input_1"].astype("int32") + return data + + project = "project" + + # Register Feature Views + sql_registry.apply_feature_view(odfv1, project) + sql_registry.apply_feature_view(fv1, project) + + # Modify odfv by changing a single feature dtype + @on_demand_feature_view( + features=[ + Feature(name="odfv1_my_feature_1", dtype=ValueType.FLOAT), + Feature(name="odfv1_my_feature_2", dtype=ValueType.INT32), + ], + sources=[request_source], + ) + def odfv1(feature_df: pd.DataFrame) -> pd.DataFrame: + data = pd.DataFrame() + data["odfv1_my_feature_1"] = feature_df["my_input_1"].astype("float") + data["odfv1_my_feature_2"] = feature_df["my_input_1"].astype("int32") + return data + + # Apply the modified odfv + sql_registry.apply_feature_view(odfv1, project) + + # Check odfv + on_demand_feature_views = sql_registry.list_on_demand_feature_views(project) + + assert ( + len(on_demand_feature_views) == 1 + and on_demand_feature_views[0].name == "odfv1" + and on_demand_feature_views[0].features[0].name == "odfv1_my_feature_1" + and on_demand_feature_views[0].features[0].dtype == Float32 + and on_demand_feature_views[0].features[1].name == "odfv1_my_feature_2" + and on_demand_feature_views[0].features[1].dtype == Int32 + ) + request_schema = on_demand_feature_views[0].get_request_data_schema() + assert ( + list(request_schema.keys())[0] == "my_input_1" + and list(request_schema.values())[0] == ValueType.INT32 + ) + + feature_view = sql_registry.get_on_demand_feature_view("odfv1", project) + assert ( + feature_view.name == "odfv1" + and feature_view.features[0].name == "odfv1_my_feature_1" + and feature_view.features[0].dtype == Float32 + and feature_view.features[1].name == "odfv1_my_feature_2" + and feature_view.features[1].dtype == Int32 + ) + request_schema = feature_view.get_request_data_schema() + assert ( + list(request_schema.keys())[0] == "my_input_1" + and list(request_schema.values())[0] == ValueType.INT32 + ) + + # Make sure fv1 is untouched + feature_views = sql_registry.list_feature_views(project) + + # List Feature Views + assert ( + len(feature_views) == 1 + and feature_views[0].name == "my_feature_view_1" + and feature_views[0].features[0].name == "fs1_my_feature_1" + and feature_views[0].features[0].dtype == Int64 + and feature_views[0].entities[0] == "fs1_my_entity_1" + ) + + feature_view = sql_registry.get_feature_view("my_feature_view_1", project) + assert ( + feature_view.name == "my_feature_view_1" + and feature_view.features[0].name == "fs1_my_feature_1" + and feature_view.features[0].dtype == Int64 + and feature_view.entities[0] == "fs1_my_entity_1" + ) + + sql_registry.teardown() + + +@pytest.mark.skipif( + sys.platform == "darwin", reason="does not run on mac github actions" +) +@pytest.mark.integration +@pytest.mark.parametrize( + "sql_registry", [lazy_fixture("mysql_registry"), lazy_fixture("pg_registry")], +) +def test_apply_data_source(sql_registry): + # Create Feature Views + batch_source = FileSource( + name="test_source", + file_format=ParquetFormat(), + path="file://feast/*", + timestamp_field="ts_col", + created_timestamp_column="timestamp", + ) + + entity = Entity(name="fs1_my_entity_1", join_keys=["test"]) + + fv1 = FeatureView( + name="my_feature_view_1", + schema=[ + Field(name="fs1_my_feature_1", dtype=Int64), + Field(name="fs1_my_feature_2", dtype=String), + Field(name="fs1_my_feature_3", dtype=Array(String)), + Field(name="fs1_my_feature_4", dtype=Array(Bytes)), + ], + entities=[entity], + tags={"team": "matchmaking"}, + batch_source=batch_source, + ttl=timedelta(minutes=5), + ) + + project = "project" + + # Register data source and feature view + sql_registry.apply_data_source(batch_source, project, commit=False) + sql_registry.apply_feature_view(fv1, project, commit=True) + + registry_feature_views = sql_registry.list_feature_views(project) + registry_data_sources = sql_registry.list_data_sources(project) + assert len(registry_feature_views) == 1 + assert len(registry_data_sources) == 1 + registry_feature_view = registry_feature_views[0] + assert registry_feature_view.batch_source == batch_source + registry_data_source = registry_data_sources[0] + assert registry_data_source == batch_source + + # Check that change to batch source propagates + batch_source.timestamp_field = "new_ts_col" + sql_registry.apply_data_source(batch_source, project, commit=False) + sql_registry.apply_feature_view(fv1, project, commit=True) + registry_feature_views = sql_registry.list_feature_views(project) + registry_data_sources = sql_registry.list_data_sources(project) + assert len(registry_feature_views) == 1 + assert len(registry_data_sources) == 1 + registry_feature_view = registry_feature_views[0] + assert registry_feature_view.batch_source == batch_source + registry_batch_source = sql_registry.list_data_sources(project)[0] + assert registry_batch_source == batch_source + + sql_registry.teardown() diff --git a/setup.py b/setup.py index 633267dbae5..f92db4acecb 100644 --- a/setup.py +++ b/setup.py @@ -66,6 +66,7 @@ "pydantic>=1,<2", "pygments==2.12.0", "PyYAML>=5.4.*,<7", + "SQLAlchemy[mypy]>1,<2", "tabulate==0.8.*", "tenacity>=7,<9", "toml==0.10.*", @@ -107,6 +108,10 @@ "psycopg2-binary>=2.8.3,<3", ] +MYSQL_REQUIRED = [ + "mysqlclient", +] + HBASE_REQUIRED = [ "happybase>=1.2.0,<3", ] @@ -167,6 +172,7 @@ + SNOWFLAKE_REQUIRED + SPARK_REQUIRED + POSTGRES_REQUIRED + + MYSQL_REQUIRED + TRINO_REQUIRED + GE_REQUIRED + HBASE_REQUIRED @@ -470,6 +476,7 @@ def copy_extensions_to_source(self): "spark": SPARK_REQUIRED, "trino": TRINO_REQUIRED, "postgres": POSTGRES_REQUIRED, + "mysql": MYSQL_REQUIRED, "ge": GE_REQUIRED, "hbase": HBASE_REQUIRED, "go": GO_REQUIRED, From 0d195c4fa9af370ea3ac103f22410b76aab0dc02 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Fri, 3 Jun 2022 14:18:39 -0700 Subject: [PATCH 038/123] chore: Create a registry base class (#2756) * chore: Create a registry base class Signed-off-by: Achal Shah * abstract methods Signed-off-by: Achal Shah * implement missing methods Signed-off-by: Achal Shah * implement missing methods Signed-off-by: Achal Shah * remove dupe Signed-off-by: Achal Shah --- .github/workflows/pr_integration_tests.yml | 2 +- .github/workflows/unit_tests.yml | 2 +- sdk/python/feast/feature_store.py | 3 +- sdk/python/feast/infra/registry_stores/sql.py | 26 +- sdk/python/feast/registry.py | 441 +++++++++++++++++- 5 files changed, 468 insertions(+), 6 deletions(-) diff --git a/.github/workflows/pr_integration_tests.yml b/.github/workflows/pr_integration_tests.yml index a720865ed42..313bb142b3b 100644 --- a/.github/workflows/pr_integration_tests.yml +++ b/.github/workflows/pr_integration_tests.yml @@ -170,7 +170,7 @@ jobs: SNOWFLAKE_CI_PASSWORD: ${{ secrets.SNOWFLAKE_CI_PASSWORD }} SNOWFLAKE_CI_ROLE: ${{ secrets.SNOWFLAKE_CI_ROLE }} SNOWFLAKE_CI_WAREHOUSE: ${{ secrets.SNOWFLAKE_CI_WAREHOUSE }} - run: pytest -n 8 --cov=./ --cov-report=xml --verbose --color=yes sdk/python/tests --integration --durations=5 + run: pytest -n 8 --cov=./ --cov-report=xml --color=yes sdk/python/tests --integration --durations=5 - name: Upload coverage to Codecov uses: codecov/codecov-action@v1 with: diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml index ea6141f3310..9d6fc92aa2a 100644 --- a/.github/workflows/unit_tests.yml +++ b/.github/workflows/unit_tests.yml @@ -66,7 +66,7 @@ jobs: SNOWFLAKE_CI_PASSWORD: ${{ secrets.SNOWFLAKE_CI_PASSWORD }} SNOWFLAKE_CI_ROLE: ${{ secrets.SNOWFLAKE_CI_ROLE }} SNOWFLAKE_CI_WAREHOUSE: ${{ secrets.SNOWFLAKE_CI_WAREHOUSE }} - run: FEAST_USAGE=False pytest -n 8 --cov=./ --cov-report=xml --verbose --color=yes sdk/python/tests + run: FEAST_USAGE=False pytest -n 8 --cov=./ --cov-report=xml --color=yes sdk/python/tests - name: Upload coverage to Codecov uses: codecov/codecov-action@v1 with: diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py index 119b86d6c1e..7824db4a398 100644 --- a/sdk/python/feast/feature_store.py +++ b/sdk/python/feast/feature_store.py @@ -646,7 +646,8 @@ def _plan( self._registry.refresh() current_infra_proto = ( self._registry.cached_registry_proto.infra.__deepcopy__() - if self._registry.cached_registry_proto + if hasattr(self._registry, "cached_registry_proto") + and self._registry.cached_registry_proto else InfraProto() ) desired_registry_proto = desired_repo_contents.to_registry_proto() diff --git a/sdk/python/feast/infra/registry_stores/sql.py b/sdk/python/feast/infra/registry_stores/sql.py index f13428b7377..1a45dec68af 100644 --- a/sdk/python/feast/infra/registry_stores/sql.py +++ b/sdk/python/feast/infra/registry_stores/sql.py @@ -48,10 +48,11 @@ from feast.protos.feast.core.ValidationProfile_pb2 import ( ValidationReference as ValidationReferenceProto, ) -from feast.registry import Registry +from feast.registry import BaseRegistry from feast.repo_config import RegistryConfig from feast.request_feature_view import RequestFeatureView from feast.saved_dataset import SavedDataset, ValidationReference +from feast.stream_feature_view import StreamFeatureView metadata = MetaData() @@ -121,7 +122,7 @@ ) -class SqlRegistry(Registry): +class SqlRegistry(BaseRegistry): def __init__( self, registry_config: Optional[RegistryConfig], repo_path: Optional[Path] ): @@ -152,6 +153,11 @@ def teardown(self): def refresh(self): pass + def list_stream_feature_views( + self, project: str, allow_cache: bool = False + ) -> List[StreamFeatureView]: + return [] + def apply_entity(self, entity: Entity, project: str, commit: bool = True): return self._apply_object(entities, "entity_name", entity, "entity_proto") @@ -389,6 +395,22 @@ def apply_validation_reference( "validation_reference_proto", ) + def apply_materialization( + self, + feature_view: FeatureView, + project: str, + start_date: datetime, + end_date: datetime, + commit: bool = True, + ): + pass + + def delete_validation_reference(self, name: str, project: str, commit: bool = True): + pass + + def commit(self): + pass + def _apply_object( self, table, id_field_name, obj, proto_field_name, ): diff --git a/sdk/python/feast/registry.py b/sdk/python/feast/registry.py index dc52013bff6..fe37aa8dc24 100644 --- a/sdk/python/feast/registry.py +++ b/sdk/python/feast/registry.py @@ -11,8 +11,10 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +import abc import json import logging +from abc import abstractmethod from collections import defaultdict from datetime import datetime, timedelta from enum import Enum @@ -145,7 +147,444 @@ def get_registry_store_class_from_scheme(registry_path: str): return get_registry_store_class_from_type(registry_store_type) -class Registry: +class BaseRegistry(abc.ABC): + # Entity operations + @abstractmethod + def apply_entity(self, entity: Entity, project: str, commit: bool = True): + """ + Registers a single entity with Feast + + Args: + entity: Entity that will be registered + project: Feast project that this entity belongs to + commit: Whether the change should be persisted immediately + """ + + @abstractmethod + def delete_entity(self, name: str, project: str, commit: bool = True): + """ + Deletes an entity or raises an exception if not found. + + Args: + name: Name of entity + project: Feast project that this entity belongs to + commit: Whether the change should be persisted immediately + """ + + @abstractmethod + def get_entity(self, name: str, project: str, allow_cache: bool = False) -> Entity: + """ + Retrieves an entity. + + Args: + name: Name of entity + project: Feast project that this entity belongs to + allow_cache: Whether to allow returning this entity from a cached registry + + Returns: + Returns either the specified entity, or raises an exception if + none is found + """ + + @abstractmethod + def list_entities(self, project: str, allow_cache: bool = False) -> List[Entity]: + """ + Retrieve a list of entities from the registry + + Args: + allow_cache: Whether to allow returning entities from a cached registry + project: Filter entities based on project name + + Returns: + List of entities + """ + + # Data source operations + @abstractmethod + def apply_data_source( + self, data_source: DataSource, project: str, commit: bool = True + ): + """ + Registers a single data source with Feast + + Args: + data_source: A data source that will be registered + project: Feast project that this data source belongs to + commit: Whether to immediately commit to the registry + """ + + @abstractmethod + def delete_data_source(self, name: str, project: str, commit: bool = True): + """ + Deletes a data source or raises an exception if not found. + + Args: + name: Name of data source + project: Feast project that this data source belongs to + commit: Whether the change should be persisted immediately + """ + + @abstractmethod + def get_data_source( + self, name: str, project: str, allow_cache: bool = False + ) -> DataSource: + """ + Retrieves a data source. + + Args: + name: Name of data source + project: Feast project that this data source belongs to + allow_cache: Whether to allow returning this data source from a cached registry + + Returns: + Returns either the specified data source, or raises an exception if none is found + """ + + @abstractmethod + def list_data_sources( + self, project: str, allow_cache: bool = False + ) -> List[DataSource]: + """ + Retrieve a list of data sources from the registry + + Args: + project: Filter data source based on project name + allow_cache: Whether to allow returning data sources from a cached registry + + Returns: + List of data sources + """ + + # Feature service operations + @abstractmethod + def apply_feature_service( + self, feature_service: FeatureService, project: str, commit: bool = True + ): + """ + Registers a single feature service with Feast + + Args: + feature_service: A feature service that will be registered + project: Feast project that this entity belongs to + """ + + @abstractmethod + def delete_feature_service(self, name: str, project: str, commit: bool = True): + """ + Deletes a feature service or raises an exception if not found. + + Args: + name: Name of feature service + project: Feast project that this feature service belongs to + commit: Whether the change should be persisted immediately + """ + + @abstractmethod + def get_feature_service( + self, name: str, project: str, allow_cache: bool = False + ) -> FeatureService: + """ + Retrieves a feature service. + + Args: + name: Name of feature service + project: Feast project that this feature service belongs to + allow_cache: Whether to allow returning this feature service from a cached registry + + Returns: + Returns either the specified feature service, or raises an exception if + none is found + """ + + @abstractmethod + def list_feature_services( + self, project: str, allow_cache: bool = False + ) -> List[FeatureService]: + """ + Retrieve a list of feature services from the registry + + Args: + allow_cache: Whether to allow returning entities from a cached registry + project: Filter entities based on project name + + Returns: + List of feature services + """ + + # Feature view operations + @abstractmethod + def apply_feature_view( + self, feature_view: BaseFeatureView, project: str, commit: bool = True + ): + """ + Registers a single feature view with Feast + + Args: + feature_view: Feature view that will be registered + project: Feast project that this feature view belongs to + commit: Whether the change should be persisted immediately + """ + + @abstractmethod + def delete_feature_view(self, name: str, project: str, commit: bool = True): + """ + Deletes a feature view or raises an exception if not found. + + Args: + name: Name of feature view + project: Feast project that this feature view belongs to + commit: Whether the change should be persisted immediately + """ + + # stream feature view operations + # TODO: Needs to be implemented. + # def get_stream_feature_view(self): + # ... + + @abstractmethod + def list_stream_feature_views( + self, project: str, allow_cache: bool = False + ) -> List[StreamFeatureView]: + """ + Retrieve a list of stream feature views from the registry + + Args: + project: Filter stream feature views based on project name + allow_cache: Whether to allow returning stream feature views from a cached registry + + Returns: + List of stream feature views + """ + + # on demand feature view operations + @abstractmethod + def get_on_demand_feature_view( + self, name: str, project: str, allow_cache: bool = False + ) -> OnDemandFeatureView: + """ + Retrieves an on demand feature view. + + Args: + name: Name of on demand feature view + project: Feast project that this on demand feature view belongs to + allow_cache: Whether to allow returning this on demand feature view from a cached registry + + Returns: + Returns either the specified on demand feature view, or raises an exception if + none is found + """ + + @abstractmethod + def list_on_demand_feature_views( + self, project: str, allow_cache: bool = False + ) -> List[OnDemandFeatureView]: + """ + Retrieve a list of on demand feature views from the registry + + Args: + project: Filter on demand feature views based on project name + allow_cache: Whether to allow returning on demand feature views from a cached registry + + Returns: + List of on demand feature views + """ + + # regular feature view operations + @abstractmethod + def get_feature_view( + self, name: str, project: str, allow_cache: bool = False + ) -> FeatureView: + """ + Retrieves a feature view. + + Args: + name: Name of feature view + project: Feast project that this feature view belongs to + allow_cache: Allow returning feature view from the cached registry + + Returns: + Returns either the specified feature view, or raises an exception if + none is found + """ + + @abstractmethod + def list_feature_views( + self, project: str, allow_cache: bool = False + ) -> List[FeatureView]: + """ + Retrieve a list of feature views from the registry + + Args: + allow_cache: Allow returning feature views from the cached registry + project: Filter feature views based on project name + + Returns: + List of feature views + """ + + # request feature view operations + # TODO: Needs to be implemented. + # @abstractmethod + # def get_request_feature_view(self, name: str, project: str): + # ... + + @abstractmethod + def list_request_feature_views( + self, project: str, allow_cache: bool = False + ) -> List[RequestFeatureView]: + """ + Retrieve a list of request feature views from the registry + + Args: + allow_cache: Allow returning feature views from the cached registry + project: Filter feature views based on project name + + Returns: + List of request feature views + """ + + @abstractmethod + def apply_materialization( + self, + feature_view: FeatureView, + project: str, + start_date: datetime, + end_date: datetime, + commit: bool = True, + ): + """ + Updates materialization intervals tracked for a single feature view in Feast + + Args: + feature_view: Feature view that will be updated with an additional materialization interval tracked + project: Feast project that this feature view belongs to + start_date (datetime): Start date of the materialization interval to track + end_date (datetime): End date of the materialization interval to track + commit: Whether the change should be persisted immediately + """ + + # Saved dataset operations + @abstractmethod + def apply_saved_dataset( + self, saved_dataset: SavedDataset, project: str, commit: bool = True, + ): + """ + Stores a saved dataset metadata with Feast + + Args: + saved_dataset: SavedDataset that will be added / updated to registry + project: Feast project that this dataset belongs to + commit: Whether the change should be persisted immediately + """ + + @abstractmethod + def get_saved_dataset( + self, name: str, project: str, allow_cache: bool = False + ) -> SavedDataset: + """ + Retrieves a saved dataset. + + Args: + name: Name of dataset + project: Feast project that this dataset belongs to + allow_cache: Whether to allow returning this dataset from a cached registry + + Returns: + Returns either the specified SavedDataset, or raises an exception if + none is found + """ + + # TODO: Needs to be implemented. + # def delete_saved_dataset( + # self, name: str, project: str, allow_cache: bool = False + # ): + # """ + # Retrieves a saved dataset. + # + # Args: + # name: Name of dataset + # project: Feast project that this dataset belongs to + # allow_cache: Whether to allow returning this dataset from a cached registry + # + # Returns: + # Returns either the specified SavedDataset, or raises an exception if + # none is found + # """ + + @abstractmethod + def list_saved_datasets( + self, project: str, allow_cache: bool = False + ) -> List[SavedDataset]: + """ + Retrieves a list of all saved datasets in specified project + + Args: + project: Feast project + allow_cache: Whether to allow returning this dataset from a cached registry + + Returns: + Returns the list of SavedDatasets + """ + + # Validation reference operations + @abstractmethod + def apply_validation_reference( + self, + validation_reference: ValidationReference, + project: str, + commit: bool = True, + ): + """ + Persist a validation reference + + Args: + validation_reference: ValidationReference that will be added / updated to registry + project: Feast project that this dataset belongs to + commit: Whether the change should be persisted immediately + """ + + @abstractmethod + def delete_validation_reference(self, name: str, project: str, commit: bool = True): + """ + Deletes a validation reference or raises an exception if not found. + + Args: + name: Name of validation reference + project: Feast project that this object belongs to + commit: Whether the change should be persisted immediately + """ + + @abstractmethod + def get_validation_reference( + self, name: str, project: str, allow_cache: bool = False + ) -> ValidationReference: + """ + Retrieves a validation reference. + + Args: + name: Name of dataset + project: Feast project that this dataset belongs to + allow_cache: Whether to allow returning this dataset from a cached registry + + Returns: + Returns either the specified ValidationReference, or raises an exception if + none is found + """ + + # TODO: Needs to be implemented. + # def list_validation_references(self): + # ... + + @abstractmethod + def commit(self): + """Commits the state of the registry cache to the remote registry store.""" + + @abstractmethod + def refresh(self): + """Refreshes the state of the registry cache by fetching the registry state from the remote registry store.""" + + +class Registry(BaseRegistry): """ Registry: A registry allows for the management and persistence of feature definitions and related metadata. """ From 7d1651687a474850ebb16d4c9c0ff4a3daa6d486 Mon Sep 17 00:00:00 2001 From: Huib Keemink Date: Mon, 6 Jun 2022 17:33:12 +0200 Subject: [PATCH 039/123] feat: Added Spark support for Delta and Avro (#2757) Signed-off-by: Huib Keemink --- .../offline_stores/contrib/spark_offline_store/spark_source.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark_source.py b/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark_source.py index c94b1913598..daf3cd252ad 100644 --- a/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark_source.py +++ b/sdk/python/feast/infra/offline_stores/contrib/spark_offline_store/spark_source.py @@ -25,6 +25,8 @@ class SparkSourceFormat(Enum): csv = "csv" json = "json" parquet = "parquet" + delta = "delta" + avro = "avro" class SparkSource(DataSource): From 4339c0acc990de2b00db90221f41ac0d33e68544 Mon Sep 17 00:00:00 2001 From: vinoyang Date: Tue, 7 Jun 2022 23:58:21 +0800 Subject: [PATCH 040/123] fix: Refactor test to reuse LocalRegistryFile (#2763) Signed-off-by: yanghua --- .../java/feast/serving/it/ServingBaseTests.java | 14 +++----------- 1 file changed, 3 insertions(+), 11 deletions(-) diff --git a/java/serving/src/test/java/feast/serving/it/ServingBaseTests.java b/java/serving/src/test/java/feast/serving/it/ServingBaseTests.java index c610d7df6b1..30cba0cb068 100644 --- a/java/serving/src/test/java/feast/serving/it/ServingBaseTests.java +++ b/java/serving/src/test/java/feast/serving/it/ServingBaseTests.java @@ -28,11 +28,9 @@ import feast.proto.serving.ServingAPIProto; import feast.proto.serving.ServingAPIProto.FieldStatus; import feast.proto.types.ValueProto; +import feast.serving.registry.LocalRegistryFile; import feast.serving.util.DataGenerator; import io.grpc.StatusRuntimeException; -import java.io.IOException; -import java.nio.file.Files; -import java.nio.file.Paths; import java.util.List; import java.util.Map; import java.util.concurrent.TimeUnit; @@ -62,14 +60,8 @@ protected ServingAPIProto.GetOnlineFeaturesRequest buildOnlineRequest(int driver static RegistryProto.Registry registryProto = readLocalRegistry(); private static RegistryProto.Registry readLocalRegistry() { - try { - return RegistryProto.Registry.parseFrom( - Files.readAllBytes(Paths.get("src/test/resources/docker-compose/feast10/registry.db"))); - } catch (IOException e) { - e.printStackTrace(); - } - - return null; + return new LocalRegistryFile("src/test/resources/docker-compose/feast10/registry.db") + .getContent(); } @Test From 269055e25178956715c163547c9f0a33a5892a75 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Tue, 7 Jun 2022 09:21:25 -0700 Subject: [PATCH 041/123] fix: Add columns for user metadata in the tables (#2760) * fix: Add columns for user metadata in the tables Signed-off-by: Achal Shah * registry -> base registry Signed-off-by: Achal Shah * metadata methods Signed-off-by: Achal Shah * metadata methods Signed-off-by: Achal Shah * tests Signed-off-by: Achal Shah * one more test assert Signed-off-by: Achal Shah * cr update Signed-off-by: Achal Shah --- sdk/python/feast/diff/registry_diff.py | 11 +- sdk/python/feast/feature_logging.py | 8 +- sdk/python/feast/feature_store.py | 19 +- .../feast/infra/offline_stores/bigquery.py | 6 +- sdk/python/feast/infra/offline_stores/file.py | 6 +- .../infra/offline_stores/offline_store.py | 6 +- .../infra/offline_stores/offline_utils.py | 8 +- .../feast/infra/offline_stores/redshift.py | 6 +- .../feast/infra/offline_stores/snowflake.py | 6 +- .../feast/infra/passthrough_provider.py | 10 +- sdk/python/feast/infra/provider.py | 10 +- sdk/python/feast/infra/registry_stores/sql.py | 233 +++++++++++++++--- sdk/python/feast/registry.py | 170 +++++++++++-- sdk/python/tests/foo_provider.py | 10 +- .../registration/test_sql_registry.py | 10 + 15 files changed, 400 insertions(+), 119 deletions(-) diff --git a/sdk/python/feast/diff/registry_diff.py b/sdk/python/feast/diff/registry_diff.py index 6b38a190fe6..95b106867cc 100644 --- a/sdk/python/feast/diff/registry_diff.py +++ b/sdk/python/feast/diff/registry_diff.py @@ -26,7 +26,7 @@ from feast.protos.feast.core.ValidationProfile_pb2 import ( ValidationReference as ValidationReferenceProto, ) -from feast.registry import FEAST_OBJECT_TYPES, FeastObjectType, Registry +from feast.registry import FEAST_OBJECT_TYPES, BaseRegistry, FeastObjectType from feast.repo_contents import RepoContents @@ -161,7 +161,7 @@ def diff_registry_objects( def extract_objects_for_keep_delete_update_add( - registry: Registry, current_project: str, desired_repo_contents: RepoContents, + registry: BaseRegistry, current_project: str, desired_repo_contents: RepoContents, ) -> Tuple[ Dict[FeastObjectType, Set[FeastObject]], Dict[FeastObjectType, Set[FeastObject]], @@ -208,7 +208,7 @@ def extract_objects_for_keep_delete_update_add( def diff_between( - registry: Registry, current_project: str, desired_repo_contents: RepoContents, + registry: BaseRegistry, current_project: str, desired_repo_contents: RepoContents, ) -> RegistryDiff: """ Returns the difference between the current and desired repo states. @@ -267,7 +267,10 @@ def diff_between( def apply_diff_to_registry( - registry: Registry, registry_diff: RegistryDiff, project: str, commit: bool = True + registry: BaseRegistry, + registry_diff: RegistryDiff, + project: str, + commit: bool = True, ): """ Applies the given diff to the given Feast project in the registry. diff --git a/sdk/python/feast/feature_logging.py b/sdk/python/feast/feature_logging.py index 04f30ab81a1..275bde72ec1 100644 --- a/sdk/python/feast/feature_logging.py +++ b/sdk/python/feast/feature_logging.py @@ -17,8 +17,8 @@ ) if TYPE_CHECKING: - from feast import FeatureService - from feast.registry import Registry + from feast.feature_service import FeatureService + from feast.registry import BaseRegistry REQUEST_ID_FIELD = "__request_id" @@ -33,7 +33,7 @@ class LoggingSource: """ @abc.abstractmethod - def get_schema(self, registry: "Registry") -> pa.Schema: + def get_schema(self, registry: "BaseRegistry") -> pa.Schema: """ Generate schema for logs destination. """ raise NotImplementedError @@ -48,7 +48,7 @@ def __init__(self, feature_service: "FeatureService", project: str): self._feature_service = feature_service self._project = project - def get_schema(self, registry: "Registry") -> pa.Schema: + def get_schema(self, registry: "BaseRegistry") -> pa.Schema: fields: Dict[str, pa.DataType] = {} for projection in self._feature_service.feature_view_projections: diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py index 7824db4a398..7a5a8299ebc 100644 --- a/sdk/python/feast/feature_store.py +++ b/sdk/python/feast/feature_store.py @@ -74,14 +74,13 @@ from feast.infra.registry_stores.sql import SqlRegistry from feast.on_demand_feature_view import OnDemandFeatureView from feast.online_response import OnlineResponse -from feast.protos.feast.core.InfraObject_pb2 import Infra as InfraProto from feast.protos.feast.serving.ServingService_pb2 import ( FieldStatus, GetOnlineFeaturesResponse, ) from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto from feast.protos.feast.types.Value_pb2 import RepeatedValue, Value -from feast.registry import Registry +from feast.registry import BaseRegistry, Registry from feast.repo_config import RepoConfig, load_repo_config from feast.repo_contents import RepoContents from feast.request_feature_view import RequestFeatureView @@ -113,7 +112,7 @@ class FeatureStore: config: RepoConfig repo_path: Path - _registry: Registry + _registry: BaseRegistry _provider: Provider _go_server: "EmbeddedOnlineFeatureServer" @@ -142,8 +141,9 @@ def __init__( if registry_config.registry_type == "sql": self._registry = SqlRegistry(registry_config, None) else: - self._registry = Registry(registry_config, repo_path=self.repo_path) - self._registry._initialize_registry() + r = Registry(registry_config, repo_path=self.repo_path) + r._initialize_registry() + self._registry = r self._provider = get_provider(self.config, self.repo_path) self._go_server = None @@ -153,7 +153,7 @@ def version(self) -> str: return get_version() @property - def registry(self) -> Registry: + def registry(self) -> BaseRegistry: """Gets the registry of this feature store.""" return self._registry @@ -644,12 +644,7 @@ def _plan( # Compute the desired difference between the current infra, as stored in the registry, # and the desired infra. self._registry.refresh() - current_infra_proto = ( - self._registry.cached_registry_proto.infra.__deepcopy__() - if hasattr(self._registry, "cached_registry_proto") - and self._registry.cached_registry_proto - else InfraProto() - ) + current_infra_proto = self._registry.proto().infra.__deepcopy__() desired_registry_proto = desired_repo_contents.to_registry_proto() new_infra = self._provider.plan_infra(self.config, desired_registry_proto) new_infra_proto = new_infra.to_proto() diff --git a/sdk/python/feast/infra/offline_stores/bigquery.py b/sdk/python/feast/infra/offline_stores/bigquery.py index e9d8bdccbfe..f095caef9b7 100644 --- a/sdk/python/feast/infra/offline_stores/bigquery.py +++ b/sdk/python/feast/infra/offline_stores/bigquery.py @@ -39,7 +39,7 @@ RetrievalMetadata, ) from feast.on_demand_feature_view import OnDemandFeatureView -from feast.registry import Registry +from feast.registry import BaseRegistry from feast.repo_config import FeastConfigBaseModel, RepoConfig from ...saved_dataset import SavedDatasetStorage @@ -169,7 +169,7 @@ def get_historical_features( feature_views: List[FeatureView], feature_refs: List[str], entity_df: Union[pd.DataFrame, str], - registry: Registry, + registry: BaseRegistry, project: str, full_feature_names: bool = False, ) -> RetrievalJob: @@ -262,7 +262,7 @@ def write_logged_features( data: Union[pyarrow.Table, Path], source: LoggingSource, logging_config: LoggingConfig, - registry: Registry, + registry: BaseRegistry, ): destination = logging_config.destination assert isinstance(destination, BigQueryLoggingDestination) diff --git a/sdk/python/feast/infra/offline_stores/file.py b/sdk/python/feast/infra/offline_stores/file.py index c675751739a..809dbc12a84 100644 --- a/sdk/python/feast/infra/offline_stores/file.py +++ b/sdk/python/feast/infra/offline_stores/file.py @@ -32,7 +32,7 @@ _get_requested_feature_views_to_features_dict, _run_dask_field_mapping, ) -from feast.registry import Registry +from feast.registry import BaseRegistry from feast.repo_config import FeastConfigBaseModel, RepoConfig from feast.saved_dataset import SavedDatasetStorage from feast.usage import log_exceptions_and_usage @@ -113,7 +113,7 @@ def get_historical_features( feature_views: List[FeatureView], feature_refs: List[str], entity_df: Union[pd.DataFrame, str], - registry: Registry, + registry: BaseRegistry, project: str, full_feature_names: bool = False, ) -> RetrievalJob: @@ -380,7 +380,7 @@ def write_logged_features( data: Union[pyarrow.Table, Path], source: LoggingSource, logging_config: LoggingConfig, - registry: Registry, + registry: BaseRegistry, ): destination = logging_config.destination assert isinstance(destination, FileLoggingDestination) diff --git a/sdk/python/feast/infra/offline_stores/offline_store.py b/sdk/python/feast/infra/offline_stores/offline_store.py index 2996a1ed590..cc06ad54c15 100644 --- a/sdk/python/feast/infra/offline_stores/offline_store.py +++ b/sdk/python/feast/infra/offline_stores/offline_store.py @@ -25,7 +25,7 @@ from feast.feature_logging import LoggingConfig, LoggingSource from feast.feature_view import FeatureView from feast.on_demand_feature_view import OnDemandFeatureView -from feast.registry import Registry +from feast.registry import BaseRegistry from feast.repo_config import RepoConfig from feast.saved_dataset import SavedDatasetStorage @@ -211,7 +211,7 @@ def get_historical_features( feature_views: List[FeatureView], feature_refs: List[str], entity_df: Union[pd.DataFrame, str], - registry: Registry, + registry: BaseRegistry, project: str, full_feature_names: bool = False, ) -> RetrievalJob: @@ -252,7 +252,7 @@ def write_logged_features( data: Union[pyarrow.Table, Path], source: LoggingSource, logging_config: LoggingConfig, - registry: Registry, + registry: BaseRegistry, ): """ Write logged features to a specified destination (taken from logging_config) in the offline store. diff --git a/sdk/python/feast/infra/offline_stores/offline_utils.py b/sdk/python/feast/infra/offline_stores/offline_utils.py index dad0ca5b781..893180f19f0 100644 --- a/sdk/python/feast/infra/offline_stores/offline_utils.py +++ b/sdk/python/feast/infra/offline_stores/offline_utils.py @@ -8,7 +8,6 @@ from jinja2 import BaseLoader, Environment from pandas import Timestamp -import feast from feast.errors import ( EntityTimestampInferenceException, FeastEntityDFMissingColumnsError, @@ -17,7 +16,7 @@ from feast.importer import import_class from feast.infra.offline_stores.offline_store import OfflineStore from feast.infra.provider import _get_requested_feature_views_to_features_dict -from feast.registry import Registry +from feast.registry import BaseRegistry from feast.utils import to_naive_utc DEFAULT_ENTITY_DF_EVENT_TIMESTAMP_COL = "event_timestamp" @@ -55,8 +54,9 @@ def assert_expected_columns_in_entity_df( raise FeastEntityDFMissingColumnsError(expected_columns, missing_keys) +# TODO: Remove project and registry from the interface and call sites. def get_expected_join_keys( - project: str, feature_views: List["feast.FeatureView"], registry: Registry + project: str, feature_views: List[FeatureView], registry: BaseRegistry ) -> Set[str]: join_keys = set() for feature_view in feature_views: @@ -95,7 +95,7 @@ class FeatureViewQueryContext: def get_feature_view_query_context( feature_refs: List[str], feature_views: List[FeatureView], - registry: Registry, + registry: BaseRegistry, project: str, entity_df_timestamp_range: Tuple[datetime, datetime], ) -> List[FeatureViewQueryContext]: diff --git a/sdk/python/feast/infra/offline_stores/redshift.py b/sdk/python/feast/infra/offline_stores/redshift.py index 74ba83cb004..a5483e8140e 100644 --- a/sdk/python/feast/infra/offline_stores/redshift.py +++ b/sdk/python/feast/infra/offline_stores/redshift.py @@ -38,7 +38,7 @@ SavedDatasetRedshiftStorage, ) from feast.infra.utils import aws_utils -from feast.registry import Registry +from feast.registry import BaseRegistry from feast.repo_config import FeastConfigBaseModel, RepoConfig from feast.saved_dataset import SavedDatasetStorage from feast.usage import log_exceptions_and_usage @@ -176,7 +176,7 @@ def get_historical_features( feature_views: List[FeatureView], feature_refs: List[str], entity_df: Union[pd.DataFrame, str], - registry: Registry, + registry: BaseRegistry, project: str, full_feature_names: bool = False, ) -> RetrievalJob: @@ -269,7 +269,7 @@ def write_logged_features( data: Union[pyarrow.Table, Path], source: LoggingSource, logging_config: LoggingConfig, - registry: Registry, + registry: BaseRegistry, ): destination = logging_config.destination assert isinstance(destination, RedshiftLoggingDestination) diff --git a/sdk/python/feast/infra/offline_stores/snowflake.py b/sdk/python/feast/infra/offline_stores/snowflake.py index d39acc9f08d..73c785eecf9 100644 --- a/sdk/python/feast/infra/offline_stores/snowflake.py +++ b/sdk/python/feast/infra/offline_stores/snowflake.py @@ -44,7 +44,7 @@ write_pandas, write_parquet, ) -from feast.registry import Registry +from feast.registry import BaseRegistry from feast.repo_config import FeastConfigBaseModel, RepoConfig from feast.saved_dataset import SavedDatasetStorage from feast.usage import log_exceptions_and_usage @@ -206,7 +206,7 @@ def get_historical_features( feature_views: List[FeatureView], feature_refs: List[str], entity_df: Union[pd.DataFrame, str], - registry: Registry, + registry: BaseRegistry, project: str, full_feature_names: bool = False, ) -> RetrievalJob: @@ -284,7 +284,7 @@ def write_logged_features( data: Union[pyarrow.Table, Path], source: LoggingSource, logging_config: LoggingConfig, - registry: Registry, + registry: BaseRegistry, ): assert isinstance(logging_config.destination, SnowflakeLoggingDestination) diff --git a/sdk/python/feast/infra/passthrough_provider.py b/sdk/python/feast/infra/passthrough_provider.py index a53788dc85a..f04d03eb99c 100644 --- a/sdk/python/feast/infra/passthrough_provider.py +++ b/sdk/python/feast/infra/passthrough_provider.py @@ -21,7 +21,7 @@ ) from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto from feast.protos.feast.types.Value_pb2 import Value as ValueProto -from feast.registry import Registry +from feast.registry import BaseRegistry from feast.repo_config import RepoConfig from feast.saved_dataset import SavedDataset from feast.usage import RatioSampler, log_exceptions_and_usage, set_usage_attribute @@ -138,7 +138,7 @@ def materialize_single_feature_view( feature_view: FeatureView, start_date: datetime, end_date: datetime, - registry: Registry, + registry: BaseRegistry, project: str, tqdm_builder: Callable[[int], tqdm], ) -> None: @@ -194,7 +194,7 @@ def get_historical_features( feature_views: List[FeatureView], feature_refs: List[str], entity_df: Union[pandas.DataFrame, str], - registry: Registry, + registry: BaseRegistry, project: str, full_feature_names: bool, ) -> RetrievalJob: @@ -240,7 +240,7 @@ def write_feature_service_logs( feature_service: FeatureService, logs: Union[pyarrow.Table, str], config: RepoConfig, - registry: Registry, + registry: BaseRegistry, ): assert ( feature_service.logging_config is not None @@ -260,7 +260,7 @@ def retrieve_feature_service_logs( start_date: datetime, end_date: datetime, config: RepoConfig, - registry: Registry, + registry: BaseRegistry, ) -> RetrievalJob: assert ( feature_service.logging_config is not None diff --git a/sdk/python/feast/infra/provider.py b/sdk/python/feast/infra/provider.py index cd82b7d416c..e6c3da86a56 100644 --- a/sdk/python/feast/infra/provider.py +++ b/sdk/python/feast/infra/provider.py @@ -19,7 +19,7 @@ from feast.protos.feast.core.Registry_pb2 import Registry as RegistryProto from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto from feast.protos.feast.types.Value_pb2 import Value as ValueProto -from feast.registry import Registry +from feast.registry import BaseRegistry from feast.repo_config import RepoConfig from feast.saved_dataset import SavedDataset from feast.type_map import python_values_to_proto_values @@ -133,7 +133,7 @@ def materialize_single_feature_view( feature_view: FeatureView, start_date: datetime, end_date: datetime, - registry: Registry, + registry: BaseRegistry, project: str, tqdm_builder: Callable[[int], tqdm], ) -> None: @@ -146,7 +146,7 @@ def get_historical_features( feature_views: List[FeatureView], feature_refs: List[str], entity_df: Union[pandas.DataFrame, str], - registry: Registry, + registry: BaseRegistry, project: str, full_feature_names: bool, ) -> RetrievalJob: @@ -192,7 +192,7 @@ def write_feature_service_logs( feature_service: FeatureService, logs: Union[pyarrow.Table, Path], config: RepoConfig, - registry: Registry, + registry: BaseRegistry, ): """ Write features and entities logged by a feature server to an offline store. @@ -211,7 +211,7 @@ def retrieve_feature_service_logs( start_date: datetime, end_date: datetime, config: RepoConfig, - registry: Registry, + registry: BaseRegistry, ) -> RetrievalJob: """ Read logged features from an offline store for a given time window [from, to). diff --git a/sdk/python/feast/infra/registry_stores/sql.py b/sdk/python/feast/infra/registry_stores/sql.py index 1a45dec68af..d34bb2fa8b6 100644 --- a/sdk/python/feast/infra/registry_stores/sql.py +++ b/sdk/python/feast/infra/registry_stores/sql.py @@ -1,7 +1,7 @@ from datetime import datetime from pathlib import Path from threading import Lock -from typing import List, Optional +from typing import Any, List, Optional from sqlalchemy import ( # type: ignore BigInteger, @@ -31,6 +31,7 @@ ) from feast.feature_service import FeatureService from feast.feature_view import FeatureView +from feast.infra.infra_object import Infra from feast.on_demand_feature_view import OnDemandFeatureView from feast.protos.feast.core.DataSource_pb2 import DataSource as DataSourceProto from feast.protos.feast.core.Entity_pb2 import Entity as EntityProto @@ -41,10 +42,14 @@ from feast.protos.feast.core.OnDemandFeatureView_pb2 import ( OnDemandFeatureView as OnDemandFeatureViewProto, ) +from feast.protos.feast.core.Registry_pb2 import Registry as RegistryProto from feast.protos.feast.core.RequestFeatureView_pb2 import ( RequestFeatureView as RequestFeatureViewProto, ) from feast.protos.feast.core.SavedDataset_pb2 import SavedDataset as SavedDatasetProto +from feast.protos.feast.core.StreamFeatureView_pb2 import ( + StreamFeatureView as StreamFeatureViewProto, +) from feast.protos.feast.core.ValidationProfile_pb2 import ( ValidationReference as ValidationReferenceProto, ) @@ -79,6 +84,7 @@ Column("last_updated_timestamp", BigInteger, nullable=False), Column("materialized_intervals", LargeBinary, nullable=True), Column("feature_view_proto", LargeBinary, nullable=False), + Column("user_metadata", LargeBinary, nullable=True), ) request_feature_views = Table( @@ -87,6 +93,16 @@ Column("feature_view_name", String(50), primary_key=True), Column("last_updated_timestamp", BigInteger, nullable=False), Column("feature_view_proto", LargeBinary, nullable=False), + Column("user_metadata", LargeBinary, nullable=True), +) + +stream_feature_views = Table( + "stream_feature_views", + metadata, + Column("feature_view_name", String(50), primary_key=True), + Column("last_updated_timestamp", BigInteger, nullable=False), + Column("feature_view_proto", LargeBinary, nullable=False), + Column("user_metadata", LargeBinary, nullable=True), ) on_demand_feature_views = Table( @@ -95,6 +111,7 @@ Column("feature_view_name", String(50), primary_key=True), Column("last_updated_timestamp", BigInteger, nullable=False), Column("feature_view_proto", LargeBinary, nullable=False), + Column("user_metadata", LargeBinary, nullable=True), ) feature_services = Table( @@ -153,10 +170,29 @@ def teardown(self): def refresh(self): pass + def get_stream_feature_view( + self, name: str, project: str, allow_cache: bool = False + ): + return self._get_object( + stream_feature_views, + name, + project, + StreamFeatureViewProto, + StreamFeatureView, + "feature_view_name", + "feature_view_proto", + FeatureViewNotFoundException, + ) + def list_stream_feature_views( self, project: str, allow_cache: bool = False ) -> List[StreamFeatureView]: - return [] + return self._list_objects( + stream_feature_views, + StreamFeatureViewProto, + StreamFeatureView, + "feature_view_proto", + ) def apply_entity(self, entity: Entity, project: str, commit: bool = True): return self._apply_object(entities, "entity_name", entity, "entity_proto") @@ -201,6 +237,18 @@ def get_on_demand_feature_view( FeatureViewNotFoundException, ) + def get_request_feature_view(self, name: str, project: str): + return self._get_object( + request_feature_views, + name, + project, + RequestFeatureViewProto, + RequestFeatureView, + "feature_view_name", + "feature_view_proto", + FeatureViewNotFoundException, + ) + def get_feature_service( self, name: str, project: str, allow_cache: bool = False ) -> FeatureService: @@ -247,41 +295,46 @@ def list_entities(self, project: str, allow_cache: bool = False) -> List[Entity] return self._list_objects(entities, EntityProto, Entity, "entity_proto") def delete_entity(self, name: str, project: str, commit: bool = True): - with self.engine.connect() as conn: - stmt = delete(entities).where(entities.c.entity_name == name) - rows = conn.execute(stmt) - if rows.rowcount < 1: - raise EntityNotFoundException(name, project) + return self._delete_object( + entities, name, project, "entity_name", EntityNotFoundException + ) def delete_feature_view(self, name: str, project: str, commit: bool = True): deleted_count = 0 - for table in {feature_views, request_feature_views, on_demand_feature_views}: - with self.engine.connect() as conn: - stmt = delete(table).where(table.c.feature_view_name == name) - rows = conn.execute(stmt) - deleted_count += rows.rowcount + for table in { + feature_views, + request_feature_views, + on_demand_feature_views, + stream_feature_views, + }: + deleted_count += self._delete_object( + table, name, project, "feature_view_name", None + ) if deleted_count == 0: raise FeatureViewNotFoundException(name, project) def delete_feature_service(self, name: str, project: str, commit: bool = True): - with self.engine.connect() as conn: - stmt = delete(feature_services).where( - feature_services.c.feature_service_name == name - ) - rows = conn.execute(stmt) - if rows.rowcount < 1: - raise FeatureServiceNotFoundException(name, project) + return self._delete_object( + feature_services, + name, + project, + "feature_service_name", + FeatureServiceNotFoundException, + ) def get_data_source( self, name: str, project: str, allow_cache: bool = False ) -> DataSource: - with self.engine.connect() as conn: - stmt = select(data_sources).where(data_sources.c.entity_name == name) - row = conn.execute(stmt).first() - if row: - ds_proto = DataSourceProto.FromString(row["data_source_proto"]) - return DataSource.from_proto(ds_proto) - raise DataSourceObjectNotFoundException(name, project=project) + return self._get_object( + data_sources, + name, + project, + DataSourceProto, + DataSource, + "data_source_name", + "data_source_proto", + DataSourceObjectNotFoundException, + ) def list_data_sources( self, project: str, allow_cache: bool = False @@ -300,8 +353,9 @@ def apply_data_source( def apply_feature_view( self, feature_view: BaseFeatureView, project: str, commit: bool = True ): - # TODO(achals): Stream feature views need to be supported. - if isinstance(feature_view, FeatureView): + if isinstance(feature_view, StreamFeatureView): + fv_table = stream_feature_views + elif isinstance(feature_view, FeatureView): fv_table = feature_views elif isinstance(feature_view, OnDemandFeatureView): fv_table = on_demand_feature_views @@ -406,8 +460,100 @@ def apply_materialization( pass def delete_validation_reference(self, name: str, project: str, commit: bool = True): + self._delete_object( + validation_references, + name, + project, + "validation_reference_name", + ValidationReferenceNotFound, + ) + + def update_infra(self, infra: Infra, project: str, commit: bool = True): + pass + + def get_infra(self, project: str, allow_cache: bool = False) -> Infra: pass + def apply_user_metadata( + self, + project: str, + feature_view: BaseFeatureView, + metadata_bytes: Optional[bytes], + ): + if isinstance(feature_view, StreamFeatureView): + table = stream_feature_views + elif isinstance(feature_view, FeatureView): + table = feature_views + elif isinstance(feature_view, OnDemandFeatureView): + table = on_demand_feature_views + elif isinstance(feature_view, RequestFeatureView): + table = request_feature_views + else: + raise ValueError(f"Unexpected feature view type: {type(feature_view)}") + + name = feature_view.name + with self.engine.connect() as conn: + stmt = select(table).where(getattr(table.c, "feature_view_name") == name) + row = conn.execute(stmt).first() + update_datetime = datetime.utcnow() + update_time = int(update_datetime.timestamp()) + if row: + values = { + "user_metadata": metadata_bytes, + "last_updated_timestamp": update_time, + } + update_stmt = ( + update(table) + .where(getattr(table.c, "feature_view_name") == name) + .values(values,) + ) + conn.execute(update_stmt) + else: + raise FeatureViewNotFoundException(feature_view.name, project=project) + + def get_user_metadata( + self, project: str, feature_view: BaseFeatureView + ) -> Optional[bytes]: + if isinstance(feature_view, StreamFeatureView): + table = stream_feature_views + elif isinstance(feature_view, FeatureView): + table = feature_views + elif isinstance(feature_view, OnDemandFeatureView): + table = on_demand_feature_views + elif isinstance(feature_view, RequestFeatureView): + table = request_feature_views + else: + raise ValueError(f"Unexpected feature view type: {type(feature_view)}") + + name = feature_view.name + with self.engine.connect() as conn: + stmt = select(table).where(getattr(table.c, "feature_view_name") == name) + row = conn.execute(stmt).first() + if row: + return row["user_metadata"] + else: + raise FeatureViewNotFoundException(feature_view.name, project=project) + + def proto(self) -> RegistryProto: + r = RegistryProto() + project = "" + # TODO(achal): Support Infra object, and last_updated_timestamp. + for lister, registry_proto_field in [ + (self.list_entities, r.entities), + (self.list_feature_views, r.feature_views), + (self.list_data_sources, r.data_sources), + (self.list_on_demand_feature_views, r.on_demand_feature_views), + (self.list_request_feature_views, r.request_feature_views), + (self.list_stream_feature_views, r.stream_feature_views), + (self.list_feature_services, r.feature_services), + (self.list_saved_datasets, r.saved_datasets), + (self.list_validation_references, r.validation_references), + ]: + objs: List[Any] = lister(project) # type: ignore + registry_proto_field.extend([obj.to_proto() for obj in objs]) + + return r + def commit(self): pass @@ -422,6 +568,7 @@ def _apply_object( update_time = int(update_datetime.timestamp()) if hasattr(obj, "last_updated_timestamp"): obj.last_updated_timestamp = update_datetime + if row: values = { proto_field_name: obj.to_proto().SerializeToString(), @@ -442,18 +589,13 @@ def _apply_object( insert_stmt = insert(table).values(values,) conn.execute(insert_stmt) - def _list_objects(self, table, proto_class, python_class, proto_field_name): + def _delete_object(self, table, name, project, id_field_name, not_found_exception): with self.engine.connect() as conn: - stmt = select(table) - rows = conn.execute(stmt).all() - if rows: - return [ - python_class.from_proto( - proto_class.FromString(row[proto_field_name]) - ) - for row in rows - ] - return [] + stmt = delete(table).where(getattr(table.c, id_field_name) == name) + rows = conn.execute(stmt) + if rows.rowcount < 1 and not_found_exception: + raise not_found_exception(name, project) + return rows.rowcount def _get_object( self, @@ -473,3 +615,16 @@ def _get_object( _proto = proto_class.FromString(row[proto_field_name]) return python_class.from_proto(_proto) raise not_found_exception(name, project) + + def _list_objects(self, table, proto_class, python_class, proto_field_name): + with self.engine.connect() as conn: + stmt = select(table) + rows = conn.execute(stmt).all() + if rows: + return [ + python_class.from_proto( + proto_class.FromString(row[proto_field_name]) + ) + for row in rows + ] + return [] diff --git a/sdk/python/feast/registry.py b/sdk/python/feast/registry.py index fe37aa8dc24..e993533c8b4 100644 --- a/sdk/python/feast/registry.py +++ b/sdk/python/feast/registry.py @@ -84,7 +84,7 @@ class FeastObjectType(Enum): @staticmethod def get_objects_from_registry( - registry: "Registry", project: str + registry: "BaseRegistry", project: str ) -> Dict["FeastObjectType", List[Any]]: return { FeastObjectType.DATA_SOURCE: registry.list_data_sources(project=project), @@ -337,9 +337,22 @@ def delete_feature_view(self, name: str, project: str, commit: bool = True): """ # stream feature view operations - # TODO: Needs to be implemented. - # def get_stream_feature_view(self): - # ... + @abstractmethod + def get_stream_feature_view( + self, name: str, project: str, allow_cache: bool = False + ): + """ + Retrieves a stream feature view. + + Args: + name: Name of stream feature view + project: Feast project that this feature view belongs to + allow_cache: Allow returning feature view from the cached registry + + Returns: + Returns either the specified feature view, or raises an exception if + none is found + """ @abstractmethod def list_stream_feature_views( @@ -423,10 +436,20 @@ def list_feature_views( """ # request feature view operations - # TODO: Needs to be implemented. - # @abstractmethod - # def get_request_feature_view(self, name: str, project: str): - # ... + @abstractmethod + def get_request_feature_view(self, name: str, project: str) -> RequestFeatureView: + """ + Retrieves a request feature view. + + Args: + name: Name of request feature view + project: Feast project that this feature view belongs to + allow_cache: Allow returning feature view from the cached registry + + Returns: + Returns either the specified feature view, or raises an exception if + none is found + """ @abstractmethod def list_request_feature_views( @@ -494,22 +517,19 @@ def get_saved_dataset( none is found """ - # TODO: Needs to be implemented. - # def delete_saved_dataset( - # self, name: str, project: str, allow_cache: bool = False - # ): - # """ - # Retrieves a saved dataset. - # - # Args: - # name: Name of dataset - # project: Feast project that this dataset belongs to - # allow_cache: Whether to allow returning this dataset from a cached registry - # - # Returns: - # Returns either the specified SavedDataset, or raises an exception if - # none is found - # """ + def delete_saved_dataset(self, name: str, project: str, allow_cache: bool = False): + """ + Delete a saved dataset. + + Args: + name: Name of dataset + project: Feast project that this dataset belongs to + allow_cache: Whether to allow returning this dataset from a cached registry + + Returns: + Returns either the specified SavedDataset, or raises an exception if + none is found + """ @abstractmethod def list_saved_datasets( @@ -572,8 +592,68 @@ def get_validation_reference( """ # TODO: Needs to be implemented. - # def list_validation_references(self): - # ... + def list_validation_references( + self, project: str, allow_cache: bool = False + ) -> List[ValidationReference]: + + """ + Retrieve a list of validation references from the registry + + Args: + allow_cache: Allow returning feature views from the cached registry + project: Filter feature views based on project name + + Returns: + List of request feature views + """ + + @abstractmethod + def update_infra(self, infra: Infra, project: str, commit: bool = True): + """ + Updates the stored Infra object. + + Args: + infra: The new Infra object to be stored. + project: Feast project that the Infra object refers to + commit: Whether the change should be persisted immediately + """ + + @abstractmethod + def get_infra(self, project: str, allow_cache: bool = False) -> Infra: + """ + Retrieves the stored Infra object. + + Args: + project: Feast project that the Infra object refers to + allow_cache: Whether to allow returning this entity from a cached registry + + Returns: + The stored Infra object. + """ + + @abstractmethod + def apply_user_metadata( + self, + project: str, + feature_view: BaseFeatureView, + metadata_bytes: Optional[bytes], + ): + ... + + @abstractmethod + def get_user_metadata( + self, project: str, feature_view: BaseFeatureView + ) -> Optional[bytes]: + ... + + @abstractmethod + def proto(self) -> RegistryProto: + """ + Retrieves a proto version of the registry. + + Returns: + The registry proto object. + """ @abstractmethod def commit(self): @@ -589,6 +669,19 @@ class Registry(BaseRegistry): Registry: A registry allows for the management and persistence of feature definitions and related metadata. """ + def apply_user_metadata( + self, + project: str, + feature_view: BaseFeatureView, + metadata_bytes: Optional[bytes], + ): + pass + + def get_user_metadata( + self, project: str, feature_view: BaseFeatureView + ) -> Optional[bytes]: + pass + # The cached_registry_proto object is used for both reads and writes. In particular, # all write operations refresh the cache and modify it in memory; the write must # then be persisted to the underlying RegistryStore with a call to commit(). @@ -1115,6 +1208,28 @@ def list_feature_views( feature_views.append(FeatureView.from_proto(feature_view_proto)) return feature_views + def get_request_feature_view(self, name: str, project: str): + """ + Retrieves a feature view. + + Args: + name: Name of feature view + project: Feast project that this feature view belongs to + allow_cache: Allow returning feature view from the cached registry + + Returns: + Returns either the specified feature view, or raises an exception if + none is found + """ + registry_proto = self._get_registry_proto(allow_cache=False) + for feature_view_proto in registry_proto.feature_views: + if ( + feature_view_proto.spec.name == name + and feature_view_proto.spec.project == project + ): + return RequestFeatureView.from_proto(feature_view_proto) + raise FeatureViewNotFoundException(name, project) + def list_request_feature_views( self, project: str, allow_cache: bool = False ) -> List[RequestFeatureView]: @@ -1469,6 +1584,9 @@ def teardown(self): """Tears down (removes) the registry.""" self._registry_store.teardown() + def proto(self) -> RegistryProto: + return self.cached_registry_proto or RegistryProto() + def to_dict(self, project: str) -> Dict[str, List[Any]]: """Returns a dictionary representation of the registry contents for the specified project. diff --git a/sdk/python/tests/foo_provider.py b/sdk/python/tests/foo_provider.py index 2d61c362734..bd6f9811e84 100644 --- a/sdk/python/tests/foo_provider.py +++ b/sdk/python/tests/foo_provider.py @@ -11,7 +11,7 @@ from feast.infra.provider import Provider from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto from feast.protos.feast.types.Value_pb2 import Value as ValueProto -from feast.registry import Registry +from feast.registry import BaseRegistry from feast.saved_dataset import SavedDataset @@ -52,7 +52,7 @@ def materialize_single_feature_view( feature_view: FeatureView, start_date: datetime, end_date: datetime, - registry: Registry, + registry: BaseRegistry, project: str, tqdm_builder: Callable[[int], tqdm], ) -> None: @@ -64,7 +64,7 @@ def get_historical_features( feature_views: List[FeatureView], feature_refs: List[str], entity_df: Union[pandas.DataFrame, str], - registry: Registry, + registry: BaseRegistry, project: str, full_feature_names: bool = False, ) -> RetrievalJob: @@ -87,7 +87,7 @@ def write_feature_service_logs( feature_service: FeatureService, logs: Union[pyarrow.Table, Path], config: RepoConfig, - registry: Registry, + registry: BaseRegistry, ): pass @@ -97,6 +97,6 @@ def retrieve_feature_service_logs( start_date: datetime, end_date: datetime, config: RepoConfig, - registry: Registry, + registry: BaseRegistry, ) -> RetrievalJob: pass diff --git a/sdk/python/tests/integration/registration/test_sql_registry.py b/sdk/python/tests/integration/registration/test_sql_registry.py index efad9f2c812..c96d83ce0a1 100644 --- a/sdk/python/tests/integration/registration/test_sql_registry.py +++ b/sdk/python/tests/integration/registration/test_sql_registry.py @@ -24,6 +24,7 @@ from feast import Feature, FileSource, RequestSource from feast.data_format import ParquetFormat from feast.entity import Entity +from feast.errors import FeatureViewNotFoundException from feast.feature_view import FeatureView from feast.field import Field from feast.infra.registry_stores.sql import SqlRegistry @@ -259,9 +260,18 @@ def location_features_from_push(inputs: pd.DataFrame) -> pd.DataFrame: project = "project" + with pytest.raises(FeatureViewNotFoundException): + sql_registry.get_user_metadata(project, location_features_from_push) + # Register Feature View sql_registry.apply_feature_view(location_features_from_push, project) + assert not sql_registry.get_user_metadata(project, location_features_from_push) + + b = "metadata".encode("utf-8") + sql_registry.apply_user_metadata(project, location_features_from_push, b) + assert sql_registry.get_user_metadata(project, location_features_from_push) == b + feature_views = sql_registry.list_on_demand_feature_views(project) # List Feature Views From da763c6766cd7bec801312849e884b9dc8f8fb51 Mon Sep 17 00:00:00 2001 From: vinoyang Date: Wed, 8 Jun 2022 00:29:21 +0800 Subject: [PATCH 042/123] fix: Minor refactor to format exception message (#2764) Signed-off-by: yanghua --- .../src/main/java/feast/serving/config/RegistryConfig.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/java/serving/src/main/java/feast/serving/config/RegistryConfig.java b/java/serving/src/main/java/feast/serving/config/RegistryConfig.java index 3e7cbe3f1f9..49e08e03b62 100644 --- a/java/serving/src/main/java/feast/serving/config/RegistryConfig.java +++ b/java/serving/src/main/java/feast/serving/config/RegistryConfig.java @@ -61,7 +61,8 @@ RegistryFile registryFile( case "file": return new LocalRegistryFile(registryPath); default: - throw new RuntimeException("Registry storage %s is unsupported"); + throw new RuntimeException( + String.format("Registry storage %s is unsupported", scheme.get())); } } From d25e8d468b095d58a32d5dd95989a381bd2ff0b4 Mon Sep 17 00:00:00 2001 From: dvanbrug Date: Wed, 8 Jun 2022 09:22:11 -0400 Subject: [PATCH 043/123] Change online UI schema field to be optional (#2766) Change the `online` field in feastFeatureViews schema to be optional. This addresses a problem with the UI erroring out when trying to present offline FeatureViews since protobuf would not pass along the `online` tag (#2719). Signed-off-by: Dirk Van Bruggen --- ui/src/parsers/feastFeatureViews.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ui/src/parsers/feastFeatureViews.ts b/ui/src/parsers/feastFeatureViews.ts index 3e63b5afd0e..cbf15d280e6 100644 --- a/ui/src/parsers/feastFeatureViews.ts +++ b/ui/src/parsers/feastFeatureViews.ts @@ -39,7 +39,7 @@ const FeastFeatureViewSchema = z.object({ features: z.array(FeastFeatureColumnSchema), ttl: z.string().transform((val) => parseInt(val)), batchSource: FeastBatchSourceSchema, - online: z.boolean(), + online: z.boolean().optional(), owner: z.string().optional(), tags: z.record(z.string()).optional(), }), From 325695275da610cecf2b9e820fd71f7f04179ccf Mon Sep 17 00:00:00 2001 From: Kevin Zhang Date: Wed, 8 Jun 2022 12:41:04 -0700 Subject: [PATCH 044/123] feat: Update stream fcos to have watermark and sliding interval (#2765) * Add sliding window to aggregations Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang * update apis Signed-off-by: Kevin Zhang * Lint Signed-off-by: Kevin Zhang * Fix lint Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang --- protos/feast/core/Aggregation.proto | 1 + protos/feast/core/DataSource.proto | 2 + sdk/python/feast/aggregation.py | 24 +++++++- sdk/python/feast/data_source.py | 59 ++++++++++++++++++- .../integration/registration/test_registry.py | 1 + .../test_stream_feature_view_apply.py | 1 + 6 files changed, 85 insertions(+), 3 deletions(-) diff --git a/protos/feast/core/Aggregation.proto b/protos/feast/core/Aggregation.proto index d848ce69721..d2d6cab7021 100644 --- a/protos/feast/core/Aggregation.proto +++ b/protos/feast/core/Aggregation.proto @@ -11,4 +11,5 @@ message Aggregation { string column = 1; string function = 2; google.protobuf.Duration time_window = 3; + google.protobuf.Duration slide_interval = 4; } \ No newline at end of file diff --git a/protos/feast/core/DataSource.proto b/protos/feast/core/DataSource.proto index 9e6028ccfa4..e71066ee709 100644 --- a/protos/feast/core/DataSource.proto +++ b/protos/feast/core/DataSource.proto @@ -22,6 +22,7 @@ option go_package = "github.com/feast-dev/feast/go/protos/feast/core"; option java_outer_classname = "DataSourceProto"; option java_package = "feast.proto.core"; +import "google/protobuf/duration.proto"; import "feast/core/DataFormat.proto"; import "feast/types/Value.proto"; import "feast/core/Feature.proto"; @@ -135,6 +136,7 @@ message DataSource { // Defines the stream data format encoding feature/entity data in Kafka messages. StreamFormat message_format = 3; + google.protobuf.Duration watermark = 4; } // Defines options for DataSource that sources features from Kinesis records. diff --git a/sdk/python/feast/aggregation.py b/sdk/python/feast/aggregation.py index 0a5fe845659..d0af49b4259 100644 --- a/sdk/python/feast/aggregation.py +++ b/sdk/python/feast/aggregation.py @@ -14,21 +14,28 @@ class Aggregation: column: str # Column name of the feature we are aggregating. function: str # Provided built in aggregations sum, max, min, count mean time_window: timedelta # The time window for this aggregation. + slide_interval: timedelta # The sliding window for these aggregations """ column: str function: str time_window: Optional[timedelta] + slide_interval: Optional[timedelta] def __init__( self, column: Optional[str] = "", function: Optional[str] = "", time_window: Optional[timedelta] = None, + slide_interval: Optional[timedelta] = None, ): self.column = column or "" self.function = function or "" self.time_window = time_window + if not slide_interval: + self.slide_interval = self.time_window + else: + self.slide_interval = slide_interval def to_proto(self) -> AggregationProto: window_duration = None @@ -36,8 +43,16 @@ def to_proto(self) -> AggregationProto: window_duration = Duration() window_duration.FromTimedelta(self.time_window) + slide_interval_duration = None + if self.slide_interval is not None: + slide_interval_duration = Duration() + slide_interval_duration.FromTimedelta(self.slide_interval) + return AggregationProto( - column=self.column, function=self.function, time_window=window_duration + column=self.column, + function=self.function, + time_window=window_duration, + slide_interval=slide_interval_duration, ) @classmethod @@ -48,10 +63,16 @@ def from_proto(cls, agg_proto: AggregationProto): else agg_proto.time_window.ToTimedelta() ) + slide_interval = ( + timedelta(days=0) + if agg_proto.slide_interval.ToNanoseconds() == 0 + else agg_proto.slide_interval.ToTimedelta() + ) aggregation = cls( column=agg_proto.column, function=agg_proto.function, time_window=time_window, + slide_interval=slide_interval, ) return aggregation @@ -63,6 +84,7 @@ def __eq__(self, other): self.column != other.column or self.function != other.function or self.time_window != other.time_window + or self.slide_interval != other.slide_interval ): return False diff --git a/sdk/python/feast/data_source.py b/sdk/python/feast/data_source.py index 6f416e70d3d..1211edd54aa 100644 --- a/sdk/python/feast/data_source.py +++ b/sdk/python/feast/data_source.py @@ -15,8 +15,10 @@ import enum import warnings from abc import ABC, abstractmethod +from datetime import timedelta from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, Union +from google.protobuf.duration_pb2 import Duration from google.protobuf.json_format import MessageToJson from feast import type_map @@ -47,11 +49,16 @@ class KafkaOptions: """ def __init__( - self, bootstrap_servers: str, message_format: StreamFormat, topic: str, + self, + bootstrap_servers: str, + message_format: StreamFormat, + topic: str, + watermark: Optional[timedelta] = None, ): self.bootstrap_servers = bootstrap_servers self.message_format = message_format self.topic = topic + self.watermark = watermark or None @classmethod def from_proto(cls, kafka_options_proto: DataSourceProto.KafkaOptions): @@ -64,11 +71,18 @@ def from_proto(cls, kafka_options_proto: DataSourceProto.KafkaOptions): Returns: Returns a BigQueryOptions object based on the kafka_options protobuf """ - + watermark = None + if kafka_options_proto.HasField("watermark"): + watermark = ( + timedelta(days=0) + if kafka_options_proto.watermark.ToNanoseconds() == 0 + else kafka_options_proto.watermark.ToTimedelta() + ) kafka_options = cls( bootstrap_servers=kafka_options_proto.bootstrap_servers, message_format=StreamFormat.from_proto(kafka_options_proto.message_format), topic=kafka_options_proto.topic, + watermark=watermark, ) return kafka_options @@ -80,11 +94,16 @@ def to_proto(self) -> DataSourceProto.KafkaOptions: Returns: KafkaOptionsProto protobuf """ + watermark_duration = None + if self.watermark is not None: + watermark_duration = Duration() + watermark_duration.FromTimedelta(self.watermark) kafka_options_proto = DataSourceProto.KafkaOptions( bootstrap_servers=self.bootstrap_servers, message_format=self.message_format.to_proto(), topic=self.topic, + watermark=watermark_duration, ) return kafka_options_proto @@ -369,7 +388,32 @@ def __init__( owner: Optional[str] = "", timestamp_field: Optional[str] = "", batch_source: Optional[DataSource] = None, + watermark: Optional[timedelta] = None, ): + """ + Creates a KafkaSource stream source object. + Args: + name: str. Name of data source, which should be unique within a project + event_timestamp_column (optional): str. (Deprecated) Event timestamp column used for point in time + joins of feature values. + bootstrap_servers: str. The servers of the kafka broker in the form "localhost:9092". + message_format: StreamFormat. StreamFormat of serialized messages. + topic: str. The name of the topic to read from in the kafka source. + created_timestamp_column (optional): str. Timestamp column indicating when the row + was created, used for deduplicating rows. + field_mapping (optional): dict(str, str). A dictionary mapping of column names in this data + source to feature names in a feature table or view. Only used for feature + columns, not entity or timestamp columns. + date_partition_column (optional): str. Timestamp column used for partitioning. + description (optional): str. A human-readable description. + tags (optional): dict(str, str). A dictionary of key-value pairs to store arbitrary metadata. + owner (optional): str. The owner of the data source, typically the email of the primary + maintainer. + timestamp_field (optional): str. Event timestamp field used for point + in time joins of feature values. + batch_source: DataSource. The datasource that acts as a batch source. + watermark: timedelta. The watermark for stream data. Specifically how late stream data can arrive without being discarded. + """ positional_attributes = [ "name", "event_timestamp_column", @@ -425,10 +469,12 @@ def __init__( timestamp_field=timestamp_field, ) self.batch_source = batch_source + self.kafka_options = KafkaOptions( bootstrap_servers=_bootstrap_servers, message_format=_message_format, topic=_topic, + watermark=watermark, ) def __eq__(self, other): @@ -445,6 +491,7 @@ def __eq__(self, other): != other.kafka_options.bootstrap_servers or self.kafka_options.message_format != other.kafka_options.message_format or self.kafka_options.topic != other.kafka_options.topic + or self.kafka_options.watermark != other.kafka_options.watermark ): return False @@ -455,6 +502,13 @@ def __hash__(self): @staticmethod def from_proto(data_source: DataSourceProto): + watermark = None + if data_source.kafka_options.HasField("watermark"): + watermark = ( + timedelta(days=0) + if data_source.kafka_options.watermark.ToNanoseconds() == 0 + else data_source.kafka_options.watermark.ToTimedelta() + ) return KafkaSource( name=data_source.name, event_timestamp_column=data_source.timestamp_field, @@ -463,6 +517,7 @@ def from_proto(data_source: DataSourceProto): message_format=StreamFormat.from_proto( data_source.kafka_options.message_format ), + watermark=watermark, topic=data_source.kafka_options.topic, created_timestamp_column=data_source.created_timestamp_column, timestamp_field=data_source.timestamp_field, diff --git a/sdk/python/tests/integration/registration/test_registry.py b/sdk/python/tests/integration/registration/test_registry.py index 222eb116d26..fcf65570a06 100644 --- a/sdk/python/tests/integration/registration/test_registry.py +++ b/sdk/python/tests/integration/registration/test_registry.py @@ -319,6 +319,7 @@ def simple_udf(x: int): message_format=AvroFormat(""), topic="topic", batch_source=FileSource(path="some path"), + watermark=timedelta(days=1), ) sfv = StreamFeatureView( diff --git a/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py b/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py index d24618b2704..e19641f291e 100644 --- a/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py +++ b/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py @@ -27,6 +27,7 @@ def test_apply_stream_feature_view(environment) -> None: message_format=AvroFormat(""), topic="topic", batch_source=FileSource(path="test_path", timestamp_field="event_timestamp"), + watermark=timedelta(days=1), ) @stream_feature_view( From 8a3974361a91ac41ecb8ecc0ffa781798d2775fe Mon Sep 17 00:00:00 2001 From: J Date: Wed, 8 Jun 2022 14:27:09 -0700 Subject: [PATCH 045/123] chore: Update Lombok to 1.18.24 (#2762) Signed-off-by: Jeff --- java/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/java/pom.xml b/java/pom.xml index 3b794d97ed0..723523f7ffd 100644 --- a/java/pom.xml +++ b/java/pom.xml @@ -58,7 +58,7 @@ 2.17.1 2.9.9 2.0.2 - 1.18.12 + 1.18.24 1.8.4 2.8.6 1.5.24 From 88cc47dea936f34c0798b6c6c31fda0d1b3ecbd7 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Wed, 8 Jun 2022 15:01:06 -0700 Subject: [PATCH 046/123] fix: Fix feature view __getitem__ for feature services (#2769) * fix: Fix feature view __getitem__ for feature services Signed-off-by: Achal Shah * more fixes Signed-off-by: Achal Shah --- sdk/python/feast/base_feature_view.py | 16 +++++---- sdk/python/feast/feature_service.py | 36 +++++++++++++------ sdk/python/feast/feature_view_projection.py | 3 ++ sdk/python/feast/field.py | 3 ++ sdk/python/feast/inference.py | 2 +- .../registration/test_inference.py | 27 +++++++++++--- 6 files changed, 63 insertions(+), 24 deletions(-) diff --git a/sdk/python/feast/base_feature_view.py b/sdk/python/feast/base_feature_view.py index 80b3b0cec82..5feb1d7d893 100644 --- a/sdk/python/feast/base_feature_view.py +++ b/sdk/python/feast/base_feature_view.py @@ -110,18 +110,20 @@ def __str__(self): return str(MessageToJson(self.to_proto())) def __hash__(self): - return hash((self.name)) + return hash(self.name) def __getitem__(self, item): assert isinstance(item, list) - referenced_features = [] - for feature in self.features: - if feature.name in item: - referenced_features.append(feature) - cp = self.__copy__() - cp.projection.features = referenced_features + if self.features: + referenced_features = [] + for feature in self.features: + if feature.name in item: + referenced_features.append(feature) + cp.projection.features = referenced_features + else: + cp.projection.desired_features = item return cp diff --git a/sdk/python/feast/feature_service.py b/sdk/python/feast/feature_service.py index 9490de38c93..9873711b385 100644 --- a/sdk/python/feast/feature_service.py +++ b/sdk/python/feast/feature_service.py @@ -102,22 +102,36 @@ def __init__( self.created_timestamp = None self.last_updated_timestamp = None self.logging_config = logging_config - self.infer_features() + for feature_grouping in self._features: + if isinstance(feature_grouping, BaseFeatureView): + self.feature_view_projections.append(feature_grouping.projection) def infer_features(self, fvs_to_update: Optional[Dict[str, FeatureView]] = None): - self.feature_view_projections = [] for feature_grouping in self._features: if isinstance(feature_grouping, BaseFeatureView): # For feature services that depend on an unspecified feature view, apply inferred schema - if ( - fvs_to_update is not None - and len(feature_grouping.projection.features) == 0 - and feature_grouping.name in fvs_to_update - ): - feature_grouping.projection.features = fvs_to_update[ - feature_grouping.name - ].features - self.feature_view_projections.append(feature_grouping.projection) + if fvs_to_update and feature_grouping.name in fvs_to_update: + if feature_grouping.projection.desired_features: + desired_features = set( + feature_grouping.projection.desired_features + ) + actual_features = set( + [ + f.name + for f in fvs_to_update[feature_grouping.name].features + ] + ) + assert desired_features.issubset(actual_features) + # We need to set the features for the projection at this point so we ensure we're starting with + # an empty list. + feature_grouping.projection.features = [] + for f in fvs_to_update[feature_grouping.name].features: + if f.name in desired_features: + feature_grouping.projection.features.append(f) + else: + feature_grouping.projection.features = fvs_to_update[ + feature_grouping.name + ].features else: raise ValueError( f"The feature service {self.name} has been provided with an invalid type " diff --git a/sdk/python/feast/feature_view_projection.py b/sdk/python/feast/feature_view_projection.py index fbf0db5ccdf..a862e5f08d9 100644 --- a/sdk/python/feast/feature_view_projection.py +++ b/sdk/python/feast/feature_view_projection.py @@ -27,6 +27,7 @@ class FeatureViewProjection: name: str name_alias: Optional[str] + desired_features: List[str] features: List[Field] join_key_map: Dict[str, str] = {} @@ -51,6 +52,7 @@ def from_proto(proto: FeatureViewProjectionProto): name_alias=proto.feature_view_name_alias, features=[], join_key_map=dict(proto.join_key_map), + desired_features=[], ) for feature_column in proto.feature_columns: feature_view_projection.features.append(Field.from_proto(feature_column)) @@ -63,6 +65,7 @@ def from_definition(base_feature_view: "BaseFeatureView"): name=base_feature_view.name, name_alias=None, features=base_feature_view.features, + desired_features=[], ) def get_feature(self, feature_name: str) -> Field: diff --git a/sdk/python/feast/field.py b/sdk/python/feast/field.py index 77011e6758c..6187580e0fc 100644 --- a/sdk/python/feast/field.py +++ b/sdk/python/feast/field.py @@ -50,6 +50,9 @@ def __init__( self.tags = tags or {} def __eq__(self, other): + if type(self) != type(other): + return False + if ( self.name != other.name or self.dtype != other.dtype diff --git a/sdk/python/feast/inference.py b/sdk/python/feast/inference.py index 37f0cb8b05e..bf9af26b82c 100644 --- a/sdk/python/feast/inference.py +++ b/sdk/python/feast/inference.py @@ -27,7 +27,7 @@ def update_data_sources_with_inferred_event_timestamp_col( data_source = data_source.batch_source if data_source.timestamp_field is None or data_source.timestamp_field == "": # prepare right match pattern for data source - ts_column_type_regex_pattern = "" + ts_column_type_regex_pattern: str # TODO(adchia): Move Spark source inference out of this logic if ( isinstance(data_source, FileSource) diff --git a/sdk/python/tests/integration/registration/test_inference.py b/sdk/python/tests/integration/registration/test_inference.py index c298c0e4f66..591ef6dbbdd 100644 --- a/sdk/python/tests/integration/registration/test_inference.py +++ b/sdk/python/tests/integration/registration/test_inference.py @@ -407,20 +407,37 @@ def test_update_feature_services_with_inferred_features(simple_dataset_1): feature_view_1 = FeatureView( name="test1", entities=[entity1], source=file_source, ) - feature_service = FeatureService(name="fs_1", features=[feature_view_1]) - assert len(feature_service.feature_view_projections) == 1 + feature_view_2 = FeatureView( + name="test2", entities=[entity1], source=file_source, + ) + + feature_service = FeatureService( + name="fs_1", features=[feature_view_1[["string_col"]], feature_view_2] + ) + assert len(feature_service.feature_view_projections) == 2 assert len(feature_service.feature_view_projections[0].features) == 0 + assert len(feature_service.feature_view_projections[0].desired_features) == 1 + assert len(feature_service.feature_view_projections[1].features) == 0 + assert len(feature_service.feature_view_projections[1].desired_features) == 0 update_feature_views_with_inferred_features_and_entities( - [feature_view_1], [entity1], RepoConfig(provider="local", project="test") + [feature_view_1, feature_view_2], + [entity1], + RepoConfig(provider="local", project="test"), ) feature_service.infer_features( - fvs_to_update={feature_view_1.name: feature_view_1} + fvs_to_update={ + feature_view_1.name: feature_view_1, + feature_view_2.name: feature_view_2, + } ) assert len(feature_view_1.schema) == 0 assert len(feature_view_1.features) == 3 - assert len(feature_service.feature_view_projections[0].features) == 3 + assert len(feature_view_2.schema) == 0 + assert len(feature_view_2.features) == 3 + assert len(feature_service.feature_view_projections[0].features) == 1 + assert len(feature_service.feature_view_projections[1].features) == 3 # TODO(felixwang9817): Add tests that interact with field mapping. From ecb8b2af4ba0f9d52be8ac560ac2a9da0f46f38e Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Thu, 9 Jun 2022 09:52:55 -0700 Subject: [PATCH 047/123] fix: Bugfixes for how registry is loaded (#2768) * fix: Bugfixes for how registry is loaded Signed-off-by: Achal Shah * more fixes Signed-off-by: Achal Shah * more fixes Signed-off-by: Achal Shah * increase timeout Signed-off-by: Achal Shah * more fixes Signed-off-by: Achal Shah * more fixes Signed-off-by: Achal Shah --- sdk/python/feast/infra/registry_stores/sql.py | 5 +- sdk/python/feast/registry.py | 150 ++++++++++-------- .../registration/test_sql_registry.py | 2 +- 3 files changed, 85 insertions(+), 72 deletions(-) diff --git a/sdk/python/feast/infra/registry_stores/sql.py b/sdk/python/feast/infra/registry_stores/sql.py index d34bb2fa8b6..503aaf86880 100644 --- a/sdk/python/feast/infra/registry_stores/sql.py +++ b/sdk/python/feast/infra/registry_stores/sql.py @@ -472,7 +472,7 @@ def update_infra(self, infra: Infra, project: str, commit: bool = True): pass def get_infra(self, project: str, allow_cache: bool = False) -> Infra: - pass + return Infra() def apply_user_metadata( self, @@ -550,7 +550,8 @@ def proto(self) -> RegistryProto: (self.list_validation_references, r.validation_references), ]: objs: List[Any] = lister(project) # type: ignore - registry_proto_field.extend([obj.to_proto() for obj in objs]) + if objs: + registry_proto_field.extend([obj.to_proto() for obj in objs]) return r diff --git a/sdk/python/feast/registry.py b/sdk/python/feast/registry.py index e993533c8b4..c8b00befc6b 100644 --- a/sdk/python/feast/registry.py +++ b/sdk/python/feast/registry.py @@ -663,6 +663,75 @@ def commit(self): def refresh(self): """Refreshes the state of the registry cache by fetching the registry state from the remote registry store.""" + @staticmethod + def _message_to_sorted_dict(message: Message) -> Dict[str, Any]: + return json.loads(MessageToJson(message, sort_keys=True)) + + def to_dict(self, project: str) -> Dict[str, List[Any]]: + """Returns a dictionary representation of the registry contents for the specified project. + + For each list in the dictionary, the elements are sorted by name, so this + method can be used to compare two registries. + + Args: + project: Feast project to convert to a dict + """ + registry_dict: Dict[str, Any] = defaultdict(list) + registry_dict["project"] = project + for data_source in sorted( + self.list_data_sources(project=project), key=lambda ds: ds.name + ): + registry_dict["dataSources"].append( + self._message_to_sorted_dict(data_source.to_proto()) + ) + for entity in sorted( + self.list_entities(project=project), key=lambda entity: entity.name + ): + registry_dict["entities"].append( + self._message_to_sorted_dict(entity.to_proto()) + ) + for feature_view in sorted( + self.list_feature_views(project=project), + key=lambda feature_view: feature_view.name, + ): + registry_dict["featureViews"].append( + self._message_to_sorted_dict(feature_view.to_proto()) + ) + for feature_service in sorted( + self.list_feature_services(project=project), + key=lambda feature_service: feature_service.name, + ): + registry_dict["featureServices"].append( + self._message_to_sorted_dict(feature_service.to_proto()) + ) + for on_demand_feature_view in sorted( + self.list_on_demand_feature_views(project=project), + key=lambda on_demand_feature_view: on_demand_feature_view.name, + ): + odfv_dict = self._message_to_sorted_dict(on_demand_feature_view.to_proto()) + odfv_dict["spec"]["userDefinedFunction"]["body"] = dill.source.getsource( + on_demand_feature_view.udf + ) + registry_dict["onDemandFeatureViews"].append(odfv_dict) + for request_feature_view in sorted( + self.list_request_feature_views(project=project), + key=lambda request_feature_view: request_feature_view.name, + ): + registry_dict["requestFeatureViews"].append( + self._message_to_sorted_dict(request_feature_view.to_proto()) + ) + for saved_dataset in sorted( + self.list_saved_datasets(project=project), key=lambda item: item.name + ): + registry_dict["savedDatasets"].append( + self._message_to_sorted_dict(saved_dataset.to_proto()) + ) + for infra_object in sorted(self.get_infra(project=project).infra_objects): + registry_dict["infra"].append( + self._message_to_sorted_dict(infra_object.to_proto()) + ) + return registry_dict + class Registry(BaseRegistry): """ @@ -689,6 +758,18 @@ def get_user_metadata( cached_registry_proto_created: Optional[datetime] = None cached_registry_proto_ttl: timedelta + def __new__( + cls, registry_config: Optional[RegistryConfig], repo_path: Optional[Path] + ): + # We override __new__ so that we can inspect registry_config and create a SqlRegistry without callers + # needing to make any changes. + if registry_config and registry_config.registry_type == "sql": + from feast.infra.registry_stores.sql import SqlRegistry + + return SqlRegistry(registry_config, repo_path) + else: + return super(Registry, cls).__new__(cls) + def __init__( self, registry_config: Optional[RegistryConfig], repo_path: Optional[Path] ): @@ -1587,75 +1668,6 @@ def teardown(self): def proto(self) -> RegistryProto: return self.cached_registry_proto or RegistryProto() - def to_dict(self, project: str) -> Dict[str, List[Any]]: - """Returns a dictionary representation of the registry contents for the specified project. - - For each list in the dictionary, the elements are sorted by name, so this - method can be used to compare two registries. - - Args: - project: Feast project to convert to a dict - """ - registry_dict: Dict[str, Any] = defaultdict(list) - registry_dict["project"] = project - for data_source in sorted( - self.list_data_sources(project=project), key=lambda ds: ds.name - ): - registry_dict["dataSources"].append( - self._message_to_sorted_dict(data_source.to_proto()) - ) - for entity in sorted( - self.list_entities(project=project), key=lambda entity: entity.name - ): - registry_dict["entities"].append( - self._message_to_sorted_dict(entity.to_proto()) - ) - for feature_view in sorted( - self.list_feature_views(project=project), - key=lambda feature_view: feature_view.name, - ): - registry_dict["featureViews"].append( - self._message_to_sorted_dict(feature_view.to_proto()) - ) - for feature_service in sorted( - self.list_feature_services(project=project), - key=lambda feature_service: feature_service.name, - ): - registry_dict["featureServices"].append( - self._message_to_sorted_dict(feature_service.to_proto()) - ) - for on_demand_feature_view in sorted( - self.list_on_demand_feature_views(project=project), - key=lambda on_demand_feature_view: on_demand_feature_view.name, - ): - odfv_dict = self._message_to_sorted_dict(on_demand_feature_view.to_proto()) - odfv_dict["spec"]["userDefinedFunction"]["body"] = dill.source.getsource( - on_demand_feature_view.udf - ) - registry_dict["onDemandFeatureViews"].append(odfv_dict) - for request_feature_view in sorted( - self.list_request_feature_views(project=project), - key=lambda request_feature_view: request_feature_view.name, - ): - registry_dict["requestFeatureViews"].append( - self._message_to_sorted_dict(request_feature_view.to_proto()) - ) - for saved_dataset in sorted( - self.list_saved_datasets(project=project), key=lambda item: item.name - ): - registry_dict["savedDatasets"].append( - self._message_to_sorted_dict(saved_dataset.to_proto()) - ) - for infra_object in sorted(self.get_infra(project=project).infra_objects): - registry_dict["infra"].append( - self._message_to_sorted_dict(infra_object.to_proto()) - ) - return registry_dict - - @staticmethod - def _message_to_sorted_dict(message: Message) -> Dict[str, Any]: - return json.loads(MessageToJson(message, sort_keys=True)) - def _prepare_registry_for_changes(self): """Prepares the Registry for changes by refreshing the cache if necessary.""" try: diff --git a/sdk/python/tests/integration/registration/test_sql_registry.py b/sdk/python/tests/integration/registration/test_sql_registry.py index c96d83ce0a1..1fe9ff5cecf 100644 --- a/sdk/python/tests/integration/registration/test_sql_registry.py +++ b/sdk/python/tests/integration/registration/test_sql_registry.py @@ -85,7 +85,7 @@ def mysql_registry(): log_string_to_wait_for = "/usr/sbin/mysqld: ready for connections. Version: '8.0.29' socket: '/var/run/mysqld/mysqld.sock' port: 3306" waited = wait_for_logs( - container=container, predicate=log_string_to_wait_for, timeout=30, interval=10, + container=container, predicate=log_string_to_wait_for, timeout=60, interval=10, ) logger.info("Waited for %s seconds until mysql container was up", waited) container_port = container.get_exposed_port(3306) From 0d1ac01ef45a1ee78799c7c5ebf30b2476bfc30e Mon Sep 17 00:00:00 2001 From: Danny Chiao Date: Thu, 9 Jun 2022 17:50:58 -0400 Subject: [PATCH 048/123] fix: Fix Feast UI failure with new way of specifying entities (#2773) * fix: Fix Feast UI failure with new way of specifying entities Signed-off-by: Danny Chiao * fix: Fix Feast UI failure with new way of specifying entities Signed-off-by: Danny Chiao --- ui/feature_repo/features.py | 23 ++++++----------------- ui/feature_repo/test_get_features.py | 9 ++++----- ui/src/parsers/feastEntities.ts | 2 +- 3 files changed, 11 insertions(+), 23 deletions(-) diff --git a/ui/feature_repo/features.py b/ui/feature_repo/features.py index efdab694e83..0e6b7f4238e 100644 --- a/ui/feature_repo/features.py +++ b/ui/feature_repo/features.py @@ -1,18 +1,11 @@ from datetime import timedelta -from feast import ( - Entity, - FeatureService, - FeatureView, - Field, - FileSource, - ValueType, -) +import pandas as pd + +from feast import Entity, FeatureService, FeatureView, Field, FileSource, ValueType from feast.data_source import RequestSource -from feast.request_feature_view import RequestFeatureView from feast.on_demand_feature_view import on_demand_feature_view from feast.types import Bool, Int64, String -import pandas as pd zipcode = Entity( name="zipcode", @@ -128,19 +121,14 @@ # Define a request data source which encodes features / information only # available at request time (e.g. part of the user initiated HTTP request) input_request = RequestSource( - name="transaction", - schema=[ - Field(name="transaction_amt", dtype=Int64), - ], + name="transaction", schema=[Field(name="transaction_amt", dtype=Int64),], ) # Define an on demand feature view which can generate new features based on # existing feature views and RequestSource features @on_demand_feature_view( sources=[credit_history, input_request], - schema=[ - Field(name="transaction_gt_last_credit_card_due", dtype=Bool), - ], + schema=[Field(name="transaction_gt_last_credit_card_due", dtype=Bool),], ) def transaction_gt_last_credit_card_due(inputs: pd.DataFrame) -> pd.DataFrame: df = pd.DataFrame() @@ -149,6 +137,7 @@ def transaction_gt_last_credit_card_due(inputs: pd.DataFrame) -> pd.DataFrame: ) return df + model_v1 = FeatureService( name="credit_score_v1", features=[ diff --git a/ui/feature_repo/test_get_features.py b/ui/feature_repo/test_get_features.py index 42af53e94b8..722c5a3dd46 100644 --- a/ui/feature_repo/test_get_features.py +++ b/ui/feature_repo/test_get_features.py @@ -1,12 +1,11 @@ import pandas as pd -from feast import FeatureStore -from feast.infra.offline_stores.file import SavedDatasetFileStorage - -from feast.dqm.profilers.ge_profiler import ge_profiler - from great_expectations.core.expectation_suite import ExpectationSuite from great_expectations.dataset import PandasDataset +from feast import FeatureStore +from feast.dqm.profilers.ge_profiler import ge_profiler +from feast.infra.offline_stores.file import SavedDatasetFileStorage + DELTA = 0.1 # controlling allowed window in fraction of the value on scale [0, 1] # Note: the GE integration allows asserting differences between datasets. The "ds" below is the reference dataset to check and this generates the expectation suite which can be used against future datasets. # It's used via ge.validate(new_dataset, ExpectationSuite) diff --git a/ui/src/parsers/feastEntities.ts b/ui/src/parsers/feastEntities.ts index 3dddb8dca13..09057c6fe92 100644 --- a/ui/src/parsers/feastEntities.ts +++ b/ui/src/parsers/feastEntities.ts @@ -4,7 +4,7 @@ import { FEAST_FEATURE_VALUE_TYPES } from "./types"; const FeastEntitySchema = z.object({ spec: z.object({ name: z.string(), - valueType: z.nativeEnum(FEAST_FEATURE_VALUE_TYPES), + valueType: z.nativeEnum(FEAST_FEATURE_VALUE_TYPES).optional(), joinKey: z.string(), description: z.string().optional(), labels: z.record(z.string()).optional(), From 846ff4a804071d57c848d78a60898f2747bba878 Mon Sep 17 00:00:00 2001 From: Danny Chiao Date: Thu, 9 Jun 2022 19:46:58 -0400 Subject: [PATCH 049/123] chore: Add Cassandra plugin to roadmap + fix docs for StreamFeatureView / SqlRegistry (#2774) * chore: Update roadmap to include Cassandra plugin Signed-off-by: Danny Chiao * fix docs Signed-off-by: Danny Chiao --- README.md | 13 +++----- .../third-party-integrations.md | 2 +- docs/roadmap.md | 13 +++----- sdk/python/docs/index.rst | 15 ++++++++- .../source/feast.infra.registry_stores.rst | 21 ++++++++++++ sdk/python/docs/source/feast.infra.rst | 1 + .../docs/source/feast.protos.feast.core.rst | 32 +++++++++++++++++++ sdk/python/docs/source/feast.rst | 8 +++++ sdk/python/docs/source/index.rst | 15 ++++++++- .../feast/infra/registry_stores/__init__.py | 0 10 files changed, 101 insertions(+), 19 deletions(-) create mode 100644 sdk/python/docs/source/feast.infra.registry_stores.rst create mode 100644 sdk/python/feast/infra/registry_stores/__init__.py diff --git a/README.md b/README.md index 01776b048ad..152b9d2d5d2 100644 --- a/README.md +++ b/README.md @@ -139,7 +139,6 @@ The list below contains the functionality that contributors are planning to deve * Items below that are in development (or planned for development) will be indicated in parentheses. * We welcome contribution to all items in the roadmap! -* Want to influence our roadmap and prioritization? Submit your feedback to [this form](https://docs.google.com/forms/d/e/1FAIpQLSfa1nRQ0sKz-JEFnMMCi4Jseag\_yDssO\_3nV9qMfxfrkil-wA/viewform). * Want to speak to a Feast contributor? We are more than happy to jump on a call. Please schedule a time using [Calendly](https://calendly.com/d/x2ry-g5bb/meet-with-feast-team). * **Data Sources** @@ -172,17 +171,16 @@ The list below contains the functionality that contributors are planning to deve * [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure) * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres) * [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store) + * [x] [Cassandra / AstraDB](https://github.com/datastaxdevs/feast-cassandra-online-store) * [ ] Bigtable (in progress) - * [ ] Cassandra * **Streaming** * [x] [Custom streaming ingestion job support](https://docs.feast.dev/how-to-guides/creating-a-custom-provider) - * [x] [Push based streaming data ingestion](https://docs.feast.dev/reference/data-sources/push) - * [ ] Streaming ingestion on AWS - * [ ] Streaming ingestion on GCP + * [x] [Push based streaming data ingestion to online store](https://docs.feast.dev/reference/data-sources/push) + * [ ] Push based streaming data ingestion to offline store * **Feature Engineering** * [x] On-demand Transformations (Alpha release. See [RFC](https://docs.google.com/document/d/1lgfIw0Drc65LpaxbUu49RCeJgMew547meSJttnUqz7c/edit#)) * [ ] Batch transformation (In progress. See [RFC](https://docs.google.com/document/d/1964OkzuBljifDvkV-0fakp2uaijnVzdwWNGdz7Vz50A/edit)) - * [ ] Streaming transformation + * [ ] Streaming transformation (In progress) * **Deployments** * [x] AWS Lambda (Alpha release. See [RFC](https://docs.google.com/document/d/1eZWKWzfBif66LDN32IajpaG-j82LSHCCOzY6R7Ax7MI/edit)) * [x] Kubernetes (See [guide](https://docs.feast.dev/how-to-guides/running-feast-in-production#4.3.-java-based-feature-server-deployed-on-kubernetes)) @@ -196,7 +194,7 @@ The list below contains the functionality that contributors are planning to deve * [ ] Java Client * [ ] Go Client * [ ] Delete API - * [ ] Feature Logging (for training) + * [] Feature Logging (for training) * **Data Quality Management (See [RFC](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit))** * [x] Data profiling and validation (Great Expectations) * [ ] Training-serving skew detection (in progress) @@ -209,7 +207,6 @@ The list below contains the functionality that contributors are planning to deve * [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py)) * [x] Feast Web UI (Alpha release. See [documentation](https://docs.feast.dev/reference/alpha-web-ui)) * [ ] REST API for browsing feature registry - * [ ] Feature versioning ## 🎓 Important Resources diff --git a/docs/getting-started/third-party-integrations.md b/docs/getting-started/third-party-integrations.md index ab926682662..8a862891f8d 100644 --- a/docs/getting-started/third-party-integrations.md +++ b/docs/getting-started/third-party-integrations.md @@ -46,8 +46,8 @@ Don't see your offline store or online store of choice here? Check out our guide * [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure) * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres) * [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store) +* [x] [Cassandra / AstraDB](https://github.com/datastaxdevs/feast-cassandra-online-store) * [ ] Bigtable (in progress) -* [ ] Cassandra ### **Deployments** diff --git a/docs/roadmap.md b/docs/roadmap.md index d893814aafd..80cc2accd33 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -4,7 +4,6 @@ The list below contains the functionality that contributors are planning to deve * Items below that are in development (or planned for development) will be indicated in parentheses. * We welcome contribution to all items in the roadmap! -* Want to influence our roadmap and prioritization? Submit your feedback to [this form](https://docs.google.com/forms/d/e/1FAIpQLSfa1nRQ0sKz-JEFnMMCi4Jseag\_yDssO\_3nV9qMfxfrkil-wA/viewform). * Want to speak to a Feast contributor? We are more than happy to jump on a call. Please schedule a time using [Calendly](https://calendly.com/d/x2ry-g5bb/meet-with-feast-team). * **Data Sources** @@ -37,17 +36,16 @@ The list below contains the functionality that contributors are planning to deve * [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure) * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/online-stores/postgres) * [x] [Custom online store support](https://docs.feast.dev/how-to-guides/adding-support-for-a-new-online-store) + * [x] [Cassandra / AstraDB](https://github.com/datastaxdevs/feast-cassandra-online-store) * [ ] Bigtable (in progress) - * [ ] Cassandra * **Streaming** * [x] [Custom streaming ingestion job support](https://docs.feast.dev/how-to-guides/creating-a-custom-provider) - * [x] [Push based streaming data ingestion](https://docs.feast.dev/reference/data-sources/push) - * [ ] Streaming ingestion on AWS - * [ ] Streaming ingestion on GCP + * [x] [Push based streaming data ingestion to online store](https://docs.feast.dev/reference/data-sources/push) + * [ ] Push based streaming data ingestion to offline store * **Feature Engineering** * [x] On-demand Transformations (Alpha release. See [RFC](https://docs.google.com/document/d/1lgfIw0Drc65LpaxbUu49RCeJgMew547meSJttnUqz7c/edit#)) * [ ] Batch transformation (In progress. See [RFC](https://docs.google.com/document/d/1964OkzuBljifDvkV-0fakp2uaijnVzdwWNGdz7Vz50A/edit)) - * [ ] Streaming transformation + * [ ] Streaming transformation (In progress) * **Deployments** * [x] AWS Lambda (Alpha release. See [RFC](https://docs.google.com/document/d/1eZWKWzfBif66LDN32IajpaG-j82LSHCCOzY6R7Ax7MI/edit)) * [x] Kubernetes (See [guide](https://docs.feast.dev/how-to-guides/running-feast-in-production#4.3.-java-based-feature-server-deployed-on-kubernetes)) @@ -61,7 +59,7 @@ The list below contains the functionality that contributors are planning to deve * [ ] Java Client * [ ] Go Client * [ ] Delete API - * [ ] Feature Logging (for training) + * [] Feature Logging (for training) * **Data Quality Management (See [RFC](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit))** * [x] Data profiling and validation (Great Expectations) * [ ] Training-serving skew detection (in progress) @@ -74,4 +72,3 @@ The list below contains the functionality that contributors are planning to deve * [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py)) * [x] Feast Web UI (Alpha release. See [documentation](https://docs.feast.dev/reference/alpha-web-ui)) * [ ] REST API for browsing feature registry - * [ ] Feature versioning diff --git a/sdk/python/docs/index.rst b/sdk/python/docs/index.rst index 4b3b9244b41..9297901c331 100644 --- a/sdk/python/docs/index.rst +++ b/sdk/python/docs/index.rst @@ -100,11 +100,17 @@ Feature View :members: On Demand Feature View -====================== +---------------------- .. automodule:: feast.on_demand_feature_view :members: +Stream Feature View +---------------------- + +.. automodule:: feast.stream_feature_view + :members: + Feature ================== @@ -134,6 +140,13 @@ Registry Store :members: :exclude-members: NoopRegistryStore +SQL Registry Store +----------------------- + +.. automodule:: feast.infra.registry_stores.sql + :members: + :noindex: + PostgreSQL Registry Store ----------------------- diff --git a/sdk/python/docs/source/feast.infra.registry_stores.rst b/sdk/python/docs/source/feast.infra.registry_stores.rst new file mode 100644 index 00000000000..cff02fa3380 --- /dev/null +++ b/sdk/python/docs/source/feast.infra.registry_stores.rst @@ -0,0 +1,21 @@ +feast.infra.registry\_stores package +==================================== + +Submodules +---------- + +feast.infra.registry\_stores.sql module +--------------------------------------- + +.. automodule:: feast.infra.registry_stores.sql + :members: + :undoc-members: + :show-inheritance: + +Module contents +--------------- + +.. automodule:: feast.infra.registry_stores + :members: + :undoc-members: + :show-inheritance: diff --git a/sdk/python/docs/source/feast.infra.rst b/sdk/python/docs/source/feast.infra.rst index 5473c3927e5..ec2cc120a63 100644 --- a/sdk/python/docs/source/feast.infra.rst +++ b/sdk/python/docs/source/feast.infra.rst @@ -9,6 +9,7 @@ Subpackages feast.infra.offline_stores feast.infra.online_stores + feast.infra.registry_stores feast.infra.utils Submodules diff --git a/sdk/python/docs/source/feast.protos.feast.core.rst b/sdk/python/docs/source/feast.protos.feast.core.rst index f29a51719c1..aaed49cd731 100644 --- a/sdk/python/docs/source/feast.protos.feast.core.rst +++ b/sdk/python/docs/source/feast.protos.feast.core.rst @@ -4,6 +4,22 @@ feast.protos.feast.core package Submodules ---------- +feast.protos.feast.core.Aggregation\_pb2 module +----------------------------------------------- + +.. automodule:: feast.protos.feast.core.Aggregation_pb2 + :members: + :undoc-members: + :show-inheritance: + +feast.protos.feast.core.Aggregation\_pb2\_grpc module +----------------------------------------------------- + +.. automodule:: feast.protos.feast.core.Aggregation_pb2_grpc + :members: + :undoc-members: + :show-inheritance: + feast.protos.feast.core.DataFormat\_pb2 module ---------------------------------------------- @@ -276,6 +292,22 @@ feast.protos.feast.core.Store\_pb2\_grpc module :undoc-members: :show-inheritance: +feast.protos.feast.core.StreamFeatureView\_pb2 module +----------------------------------------------------- + +.. automodule:: feast.protos.feast.core.StreamFeatureView_pb2 + :members: + :undoc-members: + :show-inheritance: + +feast.protos.feast.core.StreamFeatureView\_pb2\_grpc module +----------------------------------------------------------- + +.. automodule:: feast.protos.feast.core.StreamFeatureView_pb2_grpc + :members: + :undoc-members: + :show-inheritance: + feast.protos.feast.core.ValidationProfile\_pb2 module ----------------------------------------------------- diff --git a/sdk/python/docs/source/feast.rst b/sdk/python/docs/source/feast.rst index c54e035b034..7c569fc7130 100644 --- a/sdk/python/docs/source/feast.rst +++ b/sdk/python/docs/source/feast.rst @@ -17,6 +17,14 @@ Subpackages Submodules ---------- +feast.aggregation module +------------------------ + +.. automodule:: feast.aggregation + :members: + :undoc-members: + :show-inheritance: + feast.base\_feature\_view module -------------------------------- diff --git a/sdk/python/docs/source/index.rst b/sdk/python/docs/source/index.rst index 4b3b9244b41..9297901c331 100644 --- a/sdk/python/docs/source/index.rst +++ b/sdk/python/docs/source/index.rst @@ -100,11 +100,17 @@ Feature View :members: On Demand Feature View -====================== +---------------------- .. automodule:: feast.on_demand_feature_view :members: +Stream Feature View +---------------------- + +.. automodule:: feast.stream_feature_view + :members: + Feature ================== @@ -134,6 +140,13 @@ Registry Store :members: :exclude-members: NoopRegistryStore +SQL Registry Store +----------------------- + +.. automodule:: feast.infra.registry_stores.sql + :members: + :noindex: + PostgreSQL Registry Store ----------------------- diff --git a/sdk/python/feast/infra/registry_stores/__init__.py b/sdk/python/feast/infra/registry_stores/__init__.py new file mode 100644 index 00000000000..e69de29bb2d From 4ed107cdf6476faf20a4e09716ade87cb99f1d14 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Thu, 9 Jun 2022 16:47:56 -0700 Subject: [PATCH 050/123] fix: Implement apply_materialization and infra methods in sql registry (#2775) Signed-off-by: Achal Shah --- sdk/python/feast/infra/registry_stores/sql.py | 99 +++++++++++++------ 1 file changed, 68 insertions(+), 31 deletions(-) diff --git a/sdk/python/feast/infra/registry_stores/sql.py b/sdk/python/feast/infra/registry_stores/sql.py index 503aaf86880..f793ef7376f 100644 --- a/sdk/python/feast/infra/registry_stores/sql.py +++ b/sdk/python/feast/infra/registry_stores/sql.py @@ -1,7 +1,7 @@ from datetime import datetime from pathlib import Path from threading import Lock -from typing import Any, List, Optional +from typing import Any, List, Optional, Union from sqlalchemy import ( # type: ignore BigInteger, @@ -39,6 +39,7 @@ FeatureService as FeatureServiceProto, ) from feast.protos.feast.core.FeatureView_pb2 import FeatureView as FeatureViewProto +from feast.protos.feast.core.InfraObject_pb2 import Infra as InfraProto from feast.protos.feast.core.OnDemandFeatureView_pb2 import ( OnDemandFeatureView as OnDemandFeatureViewProto, ) @@ -138,6 +139,14 @@ Column("validation_reference_proto", LargeBinary, nullable=False), ) +managed_infra = Table( + "managed_infra", + metadata, + Column("infra_name", String(50), primary_key=True), + Column("last_updated_timestamp", BigInteger, nullable=False), + Column("infra_proto", LargeBinary, nullable=False), +) + class SqlRegistry(BaseRegistry): def __init__( @@ -168,6 +177,7 @@ def teardown(self): conn.execute(stmt) def refresh(self): + # This method is a no-op since we're always reading the latest values from the db. pass def get_stream_feature_view( @@ -353,16 +363,7 @@ def apply_data_source( def apply_feature_view( self, feature_view: BaseFeatureView, project: str, commit: bool = True ): - if isinstance(feature_view, StreamFeatureView): - fv_table = stream_feature_views - elif isinstance(feature_view, FeatureView): - fv_table = feature_views - elif isinstance(feature_view, OnDemandFeatureView): - fv_table = on_demand_feature_views - elif isinstance(feature_view, RequestFeatureView): - fv_table = request_feature_views - else: - raise ValueError(f"Unexpected feature view type: {type(feature_view)}") + fv_table = self._infer_fv_table(feature_view) return self._apply_object( fv_table, "feature_view_name", feature_view, "feature_view_proto" @@ -457,7 +458,25 @@ def apply_materialization( end_date: datetime, commit: bool = True, ): - pass + table = self._infer_fv_table(feature_view) + python_class, proto_class = self._infer_fv_classes(feature_view) + + if python_class in {RequestFeatureView, OnDemandFeatureView}: + raise ValueError( + f"Cannot apply materialization for feature {feature_view.name} of type {python_class}" + ) + fv: Union[FeatureView, StreamFeatureView] = self._get_object( + table, + feature_view.name, + project, + proto_class, + python_class, + "feature_view_name", + "feature_view_proto", + FeatureViewNotFoundException, + ) + fv.materialization_intervals.append((start_date, end_date)) + self._apply_object(table, "feature_view_name", fv, "feature_view_proto") def delete_validation_reference(self, name: str, project: str, commit: bool = True): self._delete_object( @@ -469,10 +488,21 @@ def delete_validation_reference(self, name: str, project: str, commit: bool = Tr ) def update_infra(self, infra: Infra, project: str, commit: bool = True): - pass + self._apply_object( + managed_infra, "infra_name", infra, "infra_proto", name="infra_obj" + ) def get_infra(self, project: str, allow_cache: bool = False) -> Infra: - return Infra() + return self._get_object( + managed_infra, + "infra_obj", + project, + InfraProto, + Infra, + "infra_name", + "infra_proto", + None, + ) def apply_user_metadata( self, @@ -480,16 +510,7 @@ def apply_user_metadata( feature_view: BaseFeatureView, metadata_bytes: Optional[bytes], ): - if isinstance(feature_view, StreamFeatureView): - table = stream_feature_views - elif isinstance(feature_view, FeatureView): - table = feature_views - elif isinstance(feature_view, OnDemandFeatureView): - table = on_demand_feature_views - elif isinstance(feature_view, RequestFeatureView): - table = request_feature_views - else: - raise ValueError(f"Unexpected feature view type: {type(feature_view)}") + table = self._infer_fv_table(feature_view) name = feature_view.name with self.engine.connect() as conn: @@ -511,9 +532,7 @@ def apply_user_metadata( else: raise FeatureViewNotFoundException(feature_view.name, project=project) - def get_user_metadata( - self, project: str, feature_view: BaseFeatureView - ) -> Optional[bytes]: + def _infer_fv_table(self, feature_view): if isinstance(feature_view, StreamFeatureView): table = stream_feature_views elif isinstance(feature_view, FeatureView): @@ -524,6 +543,25 @@ def get_user_metadata( table = request_feature_views else: raise ValueError(f"Unexpected feature view type: {type(feature_view)}") + return table + + def _infer_fv_classes(self, feature_view): + if isinstance(feature_view, StreamFeatureView): + python_class, proto_class = StreamFeatureView, StreamFeatureViewProto + elif isinstance(feature_view, FeatureView): + python_class, proto_class = FeatureView, FeatureViewProto + elif isinstance(feature_view, OnDemandFeatureView): + python_class, proto_class = OnDemandFeatureView, OnDemandFeatureViewProto + elif isinstance(feature_view, RequestFeatureView): + python_class, proto_class = RequestFeatureView, RequestFeatureViewProto + else: + raise ValueError(f"Unexpected feature view type: {type(feature_view)}") + return python_class, proto_class + + def get_user_metadata( + self, project: str, feature_view: BaseFeatureView + ) -> Optional[bytes]: + table = self._infer_fv_table(feature_view) name = feature_view.name with self.engine.connect() as conn: @@ -556,12 +594,11 @@ def proto(self) -> RegistryProto: return r def commit(self): + # This method is a no-op since we're always writing values eagerly to the db. pass - def _apply_object( - self, table, id_field_name, obj, proto_field_name, - ): - name = obj.name + def _apply_object(self, table, id_field_name, obj, proto_field_name, name=None): + name = name or obj.name with self.engine.connect() as conn: stmt = select(table).where(getattr(table.c, id_field_name) == name) row = conn.execute(stmt).first() From 0439945fef62e322ec8a5b11c7ecae05cb732760 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Fri, 10 Jun 2022 10:58:16 -0700 Subject: [PATCH 051/123] docs: Fix rtd docs by removing mysqlclient (#2778) * docs: Fix rtd docs by removing mysqlclient Signed-off-by: Achal Shah * fixes Signed-off-by: Achal Shah * rm dockerfile Signed-off-by: Achal Shah --- .readthedocs.yml | 2 +- sdk/python/docs/requirements.txt | 2 +- setup.py | 8 ++++++++ 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/.readthedocs.yml b/.readthedocs.yml index dea27e20b3a..75499aa5ddb 100644 --- a/.readthedocs.yml +++ b/.readthedocs.yml @@ -7,6 +7,6 @@ formats: - pdf python: - version: 3.7 + version: "3.8" install: - requirements: sdk/python/docs/requirements.txt \ No newline at end of file diff --git a/sdk/python/docs/requirements.txt b/sdk/python/docs/requirements.txt index 388cd430a8d..55035216355 100644 --- a/sdk/python/docs/requirements.txt +++ b/sdk/python/docs/requirements.txt @@ -1 +1 @@ --e ".[ci]" \ No newline at end of file +-e ".[docs]" \ No newline at end of file diff --git a/setup.py b/setup.py index f92db4acecb..c261507c4a7 100644 --- a/setup.py +++ b/setup.py @@ -178,6 +178,13 @@ + HBASE_REQUIRED ) + +# rtd builds fail because of mysql not being installed in their environment. +# We can add mysql there, but it's not strictly needed. This will be faster for builds. +DOCS_REQUIRED = CI_REQUIRED +for _r in MYSQL_REQUIRED: + DOCS_REQUIRED.remove(_r) + DEV_REQUIRED = ["mypy-protobuf==3.1", "grpcio-testing==1.*"] + CI_REQUIRED # Get git repo root directory @@ -480,6 +487,7 @@ def copy_extensions_to_source(self): "ge": GE_REQUIRED, "hbase": HBASE_REQUIRED, "go": GO_REQUIRED, + "docs": DOCS_REQUIRED, }, include_package_data=True, license="Apache", From 83ab682c14a11a92121866409bddc787021d52e5 Mon Sep 17 00:00:00 2001 From: Felix Wang Date: Fri, 10 Jun 2022 15:04:13 -0700 Subject: [PATCH 052/123] feat: Add StreamProcessor and SparkKafkaProcessor as contrib (#2777) * Add StreamProcessor and SparkKafkaProcessor as contrib Signed-off-by: Felix Wang * Remove comment Signed-off-by: Felix Wang --- .../infra/contrib/spark_kafka_processor.py | 137 ++++++++++++++++++ .../feast/infra/contrib/stream_processor.py | 87 +++++++++++ 2 files changed, 224 insertions(+) create mode 100644 sdk/python/feast/infra/contrib/spark_kafka_processor.py create mode 100644 sdk/python/feast/infra/contrib/stream_processor.py diff --git a/sdk/python/feast/infra/contrib/spark_kafka_processor.py b/sdk/python/feast/infra/contrib/spark_kafka_processor.py new file mode 100644 index 00000000000..1e228714c82 --- /dev/null +++ b/sdk/python/feast/infra/contrib/spark_kafka_processor.py @@ -0,0 +1,137 @@ +from types import MethodType +from typing import List + +from pyspark.sql import DataFrame, SparkSession +from pyspark.sql.avro.functions import from_avro +from pyspark.sql.functions import col, from_json + +from feast.data_format import AvroFormat, JsonFormat +from feast.data_source import KafkaSource +from feast.infra.contrib.stream_processor import ( + ProcessorConfig, + StreamProcessor, + StreamTable, +) +from feast.stream_feature_view import StreamFeatureView + + +class SparkProcessorConfig(ProcessorConfig): + spark_session: SparkSession + + +class SparkKafkaProcessor(StreamProcessor): + spark: SparkSession + format: str + write_function: MethodType + join_keys: List[str] + + def __init__( + self, + sfv: StreamFeatureView, + config: ProcessorConfig, + write_function: MethodType, + processing_time: str = "30 seconds", + query_timeout: str = "15 seconds", + ): + if not isinstance(sfv.stream_source, KafkaSource): + raise ValueError("data source is not kafka source") + if not isinstance( + sfv.stream_source.kafka_options.message_format, AvroFormat + ) and not isinstance( + sfv.stream_source.kafka_options.message_format, JsonFormat + ): + raise ValueError( + "spark streaming currently only supports json or avro format for kafka source schema" + ) + + self.format = ( + "json" + if isinstance(sfv.stream_source.kafka_options.message_format, JsonFormat) + else "avro" + ) + + if not isinstance(config, SparkProcessorConfig): + raise ValueError("config is not spark processor config") + self.spark = config.spark_session + self.write_function = write_function + self.processing_time = processing_time + self.query_timeout = query_timeout + super().__init__(sfv=sfv, data_source=sfv.stream_source) + + def ingest_stream_feature_view(self) -> None: + ingested_stream_df = self._ingest_stream_data() + transformed_df = self._construct_transformation_plan(ingested_stream_df) + online_store_query = self._write_to_online_store(transformed_df) + return online_store_query + + def _ingest_stream_data(self) -> StreamTable: + """Only supports json and avro formats currently.""" + if self.format == "json": + if not isinstance( + self.data_source.kafka_options.message_format, JsonFormat + ): + raise ValueError("kafka source message format is not jsonformat") + stream_df = ( + self.spark.readStream.format("kafka") + .option( + "kafka.bootstrap.servers", + self.data_source.kafka_options.bootstrap_servers, + ) + .option("subscribe", self.data_source.kafka_options.topic) + .option("startingOffsets", "latest") # Query start + .load() + .selectExpr("CAST(value AS STRING)") + .select( + from_json( + col("value"), + self.data_source.kafka_options.message_format.schema_json, + ).alias("table") + ) + .select("table.*") + ) + else: + if not isinstance( + self.data_source.kafka_options.message_format, AvroFormat + ): + raise ValueError("kafka source message format is not avro format") + stream_df = ( + self.spark.readStream.format("kafka") + .option( + "kafka.bootstrap.servers", + self.data_source.kafka_options.bootstrap_servers, + ) + .option("subscribe", self.data_source.kafka_options.topic) + .option("startingOffsets", "latest") # Query start + .load() + .selectExpr("CAST(value AS STRING)") + .select( + from_avro( + col("value"), + self.data_source.kafka_options.message_format.schema_json, + ).alias("table") + ) + .select("table.*") + ) + return stream_df + + def _construct_transformation_plan(self, df: StreamTable) -> StreamTable: + return self.sfv.udf.__call__(df) if self.sfv.udf else df + + def _write_to_online_store(self, df: StreamTable): + # Validation occurs at the fs.write_to_online_store() phase against the stream feature view schema. + def batch_write(row: DataFrame, batch_id: int): + pd_row = row.toPandas() + self.write_function( + pd_row, input_timestamp="event_timestamp", output_timestamp="" + ) + + query = ( + df.writeStream.outputMode("update") + .option("checkpointLocation", "/tmp/checkpoint/") + .trigger(processingTime=self.processing_time) + .foreachBatch(batch_write) + .start() + ) + + query.awaitTermination(timeout=self.query_timeout) + return query diff --git a/sdk/python/feast/infra/contrib/stream_processor.py b/sdk/python/feast/infra/contrib/stream_processor.py new file mode 100644 index 00000000000..cb44b99cd68 --- /dev/null +++ b/sdk/python/feast/infra/contrib/stream_processor.py @@ -0,0 +1,87 @@ +from abc import ABC +from typing import Callable + +import pandas as pd +from pyspark.sql import DataFrame + +from feast.data_source import DataSource +from feast.importer import import_class +from feast.repo_config import FeastConfigBaseModel +from feast.stream_feature_view import StreamFeatureView + +STREAM_PROCESSOR_CLASS_FOR_TYPE = { + ("spark", "kafka"): "feast.infra.contrib.spark_kafka_processor.SparkKafkaProcessor", +} + +# TODO: support more types other than just Spark. +StreamTable = DataFrame + + +class ProcessorConfig(FeastConfigBaseModel): + # Processor mode (spark, etc) + mode: str + # Ingestion source (kafka, kinesis, etc) + source: str + + +class StreamProcessor(ABC): + """ + A StreamProcessor can ingest and transform data for a specific stream feature view, + and persist that data to the online store. + + Attributes: + sfv: The stream feature view on which the stream processor operates. + data_source: The stream data source from which data will be ingested. + """ + + sfv: StreamFeatureView + data_source: DataSource + + def __init__(self, sfv: StreamFeatureView, data_source: DataSource): + self.sfv = sfv + self.data_source = data_source + + def ingest_stream_feature_view(self) -> None: + """ + Ingests data from the stream source attached to the stream feature view; transforms the data + and then persists it to the online store. + """ + pass + + def _ingest_stream_data(self) -> StreamTable: + """ + Ingests data into a StreamTable. + """ + pass + + def _construct_transformation_plan(self, table: StreamTable) -> StreamTable: + """ + Applies transformations on top of StreamTable object. Since stream engines use lazy + evaluation, the StreamTable will not be materialized until it is actually evaluated. + For example: df.collect() in spark or tbl.execute() in Flink. + """ + pass + + def _write_to_online_store(self, table: StreamTable) -> None: + """ + Returns query for persisting data to the online store. + """ + pass + + +def get_stream_processor_object( + config: ProcessorConfig, + sfv: StreamFeatureView, + write_function: Callable[[pd.DataFrame, str, str], None], +): + """ + Returns a stream processor object based on the config mode and stream source type. The write function is a + function that wraps the feature store "write_to_online_store" capability. + """ + if config.mode == "spark" and config.source == "kafka": + stream_processor = STREAM_PROCESSOR_CLASS_FOR_TYPE[("spark", "kafka")] + module_name, class_name = stream_processor.rsplit(".", 1) + cls = import_class(module_name, class_name, "Processor") + return cls(sfv=sfv, config=config, write_function=write_function,) + else: + raise ValueError("other processors besides spark-kafka not supported") From 331a214b04dc8b3f9347e79819130fa7bdf9e4c3 Mon Sep 17 00:00:00 2001 From: Kevin Zhang Date: Fri, 10 Jun 2022 16:37:41 -0700 Subject: [PATCH 053/123] fix: Update udf tests and add base functions to streaming fcos and fix some nonetype errors (#2776) * Fix lint and add comments Signed-off-by: Kevin Zhang * Fix Signed-off-by: Kevin Zhang * Fix lint Signed-off-by: Kevin Zhang --- sdk/python/feast/data_source.py | 2 +- sdk/python/feast/stream_feature_view.py | 99 +++++++++++++++---- .../test_stream_feature_view_apply.py | 68 +++++++++++++ sdk/python/tests/unit/test_feature_views.py | 74 +++++++++++++- 4 files changed, 220 insertions(+), 23 deletions(-) diff --git a/sdk/python/feast/data_source.py b/sdk/python/feast/data_source.py index 1211edd54aa..3bc9d98e62c 100644 --- a/sdk/python/feast/data_source.py +++ b/sdk/python/feast/data_source.py @@ -503,7 +503,7 @@ def __hash__(self): @staticmethod def from_proto(data_source: DataSourceProto): watermark = None - if data_source.kafka_options.HasField("watermark"): + if data_source.kafka_options.watermark: watermark = ( timedelta(days=0) if data_source.kafka_options.watermark.ToNanoseconds() == 0 diff --git a/sdk/python/feast/stream_feature_view.py b/sdk/python/feast/stream_feature_view.py index 12d7f9b74b4..214ab083abf 100644 --- a/sdk/python/feast/stream_feature_view.py +++ b/sdk/python/feast/stream_feature_view.py @@ -1,3 +1,4 @@ +import copy import functools import warnings from datetime import timedelta @@ -9,7 +10,7 @@ from feast import utils from feast.aggregation import Aggregation -from feast.data_source import DataSource, KafkaSource +from feast.data_source import DataSource, KafkaSource, PushSource from feast.entity import Entity from feast.feature_view import FeatureView from feast.field import Field @@ -39,6 +40,26 @@ class StreamFeatureView(FeatureView): """ NOTE: Stream Feature Views are not yet fully implemented and exist to allow users to register their stream sources and schemas with Feast. + + Attributes: + name: str. The unique name of the stream feature view. + entities: Union[List[Entity], List[str]]. List of entities or entity join keys. + ttl: timedelta. The amount of time this group of features lives. A ttl of 0 indicates that + this group of features lives forever. Note that large ttl's or a ttl of 0 + can result in extremely computationally intensive queries. + tags: Dict[str, str]. A dictionary of key-value pairs to store arbitrary metadata. + online: bool. Defines whether this stream feature view is used in online feature retrieval. + description: str. A human-readable description. + owner: The owner of the on demand feature view, typically the email of the primary + maintainer. + schema: List[Field] The schema of the feature view, including feature, timestamp, and entity + columns. If not specified, can be inferred from the underlying data source. + source: DataSource. The stream source of data where this group of features + is stored. + aggregations (optional): List[Aggregation]. List of aggregations registered with the stream feature view. + mode(optional): str. The mode of execution. + timestamp_field (optional): Must be specified if aggregations are specified. Defines the timestamp column on which to aggregate windows. + udf (optional): MethodType The user defined transformation function. This transformation function should have all of the corresponding imports imported within the function. """ def __init__( @@ -54,8 +75,8 @@ def __init__( schema: Optional[List[Field]] = None, source: Optional[DataSource] = None, aggregations: Optional[List[Aggregation]] = None, - mode: Optional[str] = "spark", # Mode of ingestion/transformation - timestamp_field: Optional[str] = "", # Timestamp for aggregation + mode: Optional[str] = "spark", + timestamp_field: Optional[str] = "", udf: Optional[MethodType] = None, ): warnings.warn( @@ -63,9 +84,10 @@ def __init__( "Some functionality may still be unstable so functionality can change in the future.", RuntimeWarning, ) + if source is None: - raise ValueError("Stream Feature views need a source specified") - # source uses the batch_source of the kafkasource in feature_view + raise ValueError("Stream Feature views need a source to be specified") + if ( type(source).__name__ not in SUPPORTED_STREAM_SOURCES and source.to_proto().type != DataSourceProto.SourceType.CUSTOM_SOURCE @@ -74,18 +96,26 @@ def __init__( f"Stream feature views need a stream source, expected one of {SUPPORTED_STREAM_SOURCES} " f"or CUSTOM_SOURCE, got {type(source).__name__}: {source.name} instead " ) + + if aggregations and not timestamp_field: + raise ValueError( + "aggregations must have a timestamp field associated with them to perform the aggregations" + ) + self.aggregations = aggregations or [] - self.mode = mode - self.timestamp_field = timestamp_field + self.mode = mode or "" + self.timestamp_field = timestamp_field or "" self.udf = udf _batch_source = None - if isinstance(source, KafkaSource): + if isinstance(source, KafkaSource) or isinstance(source, PushSource): _batch_source = source.batch_source if source.batch_source else None - + _ttl = ttl + if not _ttl: + _ttl = timedelta(days=0) super().__init__( name=name, entities=entities, - ttl=ttl, + ttl=_ttl, batch_source=_batch_source, stream_source=source, tags=tags, @@ -102,7 +132,10 @@ def __eq__(self, other): if not super().__eq__(other): return False - + if not self.udf: + return not other.udf + if not other.udf: + return False if ( self.mode != other.mode or self.timestamp_field != other.timestamp_field @@ -113,13 +146,14 @@ def __eq__(self, other): return True - def __hash__(self): + def __hash__(self) -> int: return super().__hash__() def to_proto(self): meta = StreamFeatureViewMetaProto(materialization_intervals=[]) if self.created_timestamp: meta.created_timestamp.FromDatetime(self.created_timestamp) + if self.last_updated_timestamp: meta.last_updated_timestamp.FromDatetime(self.last_updated_timestamp) @@ -134,6 +168,7 @@ def to_proto(self): ttl_duration = Duration() ttl_duration.FromTimedelta(self.ttl) + batch_source_proto = None if self.batch_source: batch_source_proto = self.batch_source.to_proto() batch_source_proto.data_source_class_type = f"{self.batch_source.__class__.__module__}.{self.batch_source.__class__.__name__}" @@ -143,23 +178,24 @@ def to_proto(self): stream_source_proto = self.stream_source.to_proto() stream_source_proto.data_source_class_type = f"{self.stream_source.__class__.__module__}.{self.stream_source.__class__.__name__}" + udf_proto = None + if self.udf: + udf_proto = UserDefinedFunctionProto( + name=self.udf.__name__, body=dill.dumps(self.udf, recurse=True), + ) spec = StreamFeatureViewSpecProto( name=self.name, entities=self.entities, entity_columns=[field.to_proto() for field in self.entity_columns], features=[field.to_proto() for field in self.schema], - user_defined_function=UserDefinedFunctionProto( - name=self.udf.__name__, body=dill.dumps(self.udf, recurse=True), - ) - if self.udf - else None, + user_defined_function=udf_proto, description=self.description, tags=self.tags, owner=self.owner, - ttl=(ttl_duration if ttl_duration is not None else None), + ttl=ttl_duration, online=self.online, batch_source=batch_source_proto or None, - stream_source=stream_source_proto, + stream_source=stream_source_proto or None, timestamp_field=self.timestamp_field, aggregations=[agg.to_proto() for agg in self.aggregations], mode=self.mode, @@ -239,6 +275,25 @@ def from_proto(cls, sfv_proto): return sfv_feature_view + def __copy__(self): + fv = StreamFeatureView( + name=self.name, + schema=self.schema, + entities=self.entities, + ttl=self.ttl, + tags=self.tags, + online=self.online, + description=self.description, + owner=self.owner, + aggregations=self.aggregations, + mode=self.mode, + timestamp_field=self.timestamp_field, + sources=self.sources, + udf=self.udf, + ) + fv.projection = copy.copy(self.projection) + return fv + def stream_feature_view( *, @@ -251,11 +306,13 @@ def stream_feature_view( schema: Optional[List[Field]] = None, source: Optional[DataSource] = None, aggregations: Optional[List[Aggregation]] = None, - mode: Optional[str] = "spark", # Mode of ingestion/transformation - timestamp_field: Optional[str] = "", # Timestamp for aggregation + mode: Optional[str] = "spark", + timestamp_field: Optional[str] = "", ): """ Creates an StreamFeatureView object with the given user function as udf. + Please make sure that the udf contains all non-built in imports within the function to ensure that the execution + of a deserialized function does not miss imports. """ def mainify(obj): diff --git a/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py b/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py index e19641f291e..29cd2f1c263 100644 --- a/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py +++ b/sdk/python/tests/integration/registration/test_stream_feature_view_apply.py @@ -70,3 +70,71 @@ def simple_sfv(df): assert features["test_key"] == [1001] assert "dummy_field" in features assert features["dummy_field"] == [None] + + +@pytest.mark.integration +def test_stream_feature_view_udf(environment) -> None: + """ + Test apply of StreamFeatureView udfs are serialized correctly and usable. + """ + fs = environment.feature_store + + # Create Feature Views + entity = Entity(name="driver_entity", join_keys=["test_key"]) + + stream_source = KafkaSource( + name="kafka", + timestamp_field="event_timestamp", + bootstrap_servers="", + message_format=AvroFormat(""), + topic="topic", + batch_source=FileSource(path="test_path", timestamp_field="event_timestamp"), + watermark=timedelta(days=1), + ) + + @stream_feature_view( + entities=[entity], + ttl=timedelta(days=30), + owner="test@example.com", + online=True, + schema=[Field(name="dummy_field", dtype=Float32)], + description="desc", + aggregations=[ + Aggregation( + column="dummy_field", function="max", time_window=timedelta(days=1), + ), + Aggregation( + column="dummy_field2", function="count", time_window=timedelta(days=24), + ), + ], + timestamp_field="event_timestamp", + mode="spark", + source=stream_source, + tags={}, + ) + def pandas_view(pandas_df): + import pandas as pd + + assert type(pandas_df) == pd.DataFrame + df = pandas_df.transform(lambda x: x + 10, axis=1) + df.insert(2, "C", [20.2, 230.0, 34.0], True) + return df + + import pandas as pd + + df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]}) + + fs.apply([entity, pandas_view]) + stream_feature_views = fs.list_stream_feature_views() + assert len(stream_feature_views) == 1 + assert stream_feature_views[0].name == "pandas_view" + assert stream_feature_views[0] == pandas_view + + sfv = stream_feature_views[0] + + new_df = sfv.udf(df) + + expected_df = pd.DataFrame( + {"A": [11, 12, 13], "B": [20, 30, 40], "C": [20.2, 230.0, 34.0]} + ) + assert new_df.equals(expected_df) diff --git a/sdk/python/tests/unit/test_feature_views.py b/sdk/python/tests/unit/test_feature_views.py index 904260dfe61..64b23edd2c9 100644 --- a/sdk/python/tests/unit/test_feature_views.py +++ b/sdk/python/tests/unit/test_feature_views.py @@ -9,7 +9,7 @@ from feast.entity import Entity from feast.field import Field from feast.infra.offline_stores.file_source import FileSource -from feast.stream_feature_view import StreamFeatureView +from feast.stream_feature_view import StreamFeatureView, stream_feature_view from feast.types import Float32 @@ -129,3 +129,75 @@ def test_stream_feature_view_serialization(): new_sfv = StreamFeatureView.from_proto(sfv_proto=sfv_proto) assert new_sfv == sfv + + +def test_stream_feature_view_udfs(): + entity = Entity(name="driver_entity", join_keys=["test_key"]) + stream_source = KafkaSource( + name="kafka", + timestamp_field="event_timestamp", + bootstrap_servers="", + message_format=AvroFormat(""), + topic="topic", + batch_source=FileSource(path="some path"), + ) + + @stream_feature_view( + entities=[entity], + ttl=timedelta(days=30), + owner="test@example.com", + online=True, + schema=[Field(name="dummy_field", dtype=Float32)], + description="desc", + aggregations=[ + Aggregation( + column="dummy_field", function="max", time_window=timedelta(days=1), + ) + ], + timestamp_field="event_timestamp", + source=stream_source, + ) + def pandas_udf(pandas_df): + import pandas as pd + + assert type(pandas_df) == pd.DataFrame + df = pandas_df.transform(lambda x: x + 10, axis=1) + return df + + import pandas as pd + + df = pd.DataFrame({"A": [1, 2, 3], "B": [10, 20, 30]}) + sfv = pandas_udf + sfv_proto = sfv.to_proto() + new_sfv = StreamFeatureView.from_proto(sfv_proto) + new_df = new_sfv.udf(df) + + expected_df = pd.DataFrame({"A": [11, 12, 13], "B": [20, 30, 40]}) + + assert new_df.equals(expected_df) + + +def test_stream_feature_view_initialization_with_optional_fields_omitted(): + entity = Entity(name="driver_entity", join_keys=["test_key"]) + stream_source = KafkaSource( + name="kafka", + timestamp_field="event_timestamp", + bootstrap_servers="", + message_format=AvroFormat(""), + topic="topic", + batch_source=FileSource(path="some path"), + ) + + sfv = StreamFeatureView( + name="test kafka stream feature view", + entities=[entity], + schema=[], + description="desc", + timestamp_field="event_timestamp", + source=stream_source, + tags={}, + ) + sfv_proto = sfv.to_proto() + + new_sfv = StreamFeatureView.from_proto(sfv_proto=sfv_proto) + assert new_sfv == sfv From 452dcd31da195273ef80ee39db5a7893b7d48cc2 Mon Sep 17 00:00:00 2001 From: Achal Shah Date: Fri, 10 Jun 2022 16:38:40 -0700 Subject: [PATCH 054/123] fix: Hydrate infra object in the sql registry proto() method (#2782) * fix: Implement apply_materialziation and infra methods in sql registry Signed-off-by: Achal Shah * fix: hydrate infra object in the sql registry proto() method Signed-off-by: Achal Shah * rm old comment Signed-off-by: Achal Shah --- sdk/python/feast/infra/registry_stores/sql.py | 54 ++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-) diff --git a/sdk/python/feast/infra/registry_stores/sql.py b/sdk/python/feast/infra/registry_stores/sql.py index f793ef7376f..af988edd59b 100644 --- a/sdk/python/feast/infra/registry_stores/sql.py +++ b/sdk/python/feast/infra/registry_stores/sql.py @@ -148,6 +148,15 @@ ) +feast_metadata = Table( + "feast_metadata", + metadata, + Column("metadata_key", String(50), primary_key=True), + Column("metadata_value", String(50), nullable=False), + Column("last_updated_timestamp", BigInteger, nullable=False), +) + + class SqlRegistry(BaseRegistry): def __init__( self, registry_config: Optional[RegistryConfig], repo_path: Optional[Path] @@ -575,7 +584,6 @@ def get_user_metadata( def proto(self) -> RegistryProto: r = RegistryProto() project = "" - # TODO(achal): Support Infra object, and last_updated_timestamp. for lister, registry_proto_field in [ (self.list_entities, r.entities), (self.list_feature_views, r.feature_views), @@ -591,6 +599,11 @@ def proto(self) -> RegistryProto: if objs: registry_proto_field.extend([obj.to_proto() for obj in objs]) + r.infra.CopyFrom(self.get_infra(project).to_proto()) + last_updated_timestamp = self._get_last_updated_metadata() + if last_updated_timestamp: + r.last_updated.FromDatetime(last_updated_timestamp) + return r def commit(self): @@ -626,6 +639,7 @@ def _apply_object(self, table, id_field_name, obj, proto_field_name, name=None): } insert_stmt = insert(table).values(values,) conn.execute(insert_stmt) + self._set_last_updated_metadata(update_datetime) def _delete_object(self, table, name, project, id_field_name, not_found_exception): with self.engine.connect() as conn: @@ -633,6 +647,7 @@ def _delete_object(self, table, name, project, id_field_name, not_found_exceptio rows = conn.execute(stmt) if rows.rowcount < 1 and not_found_exception: raise not_found_exception(name, project) + self._set_last_updated_metadata(datetime.utcnow()) return rows.rowcount def _get_object( @@ -666,3 +681,40 @@ def _list_objects(self, table, proto_class, python_class, proto_field_name): for row in rows ] return [] + + def _set_last_updated_metadata(self, last_updated: datetime): + with self.engine.connect() as conn: + stmt = select(feast_metadata).where( + feast_metadata.c.metadata_key == "last_updated_timestamp" + ) + row = conn.execute(stmt).first() + + update_time = int(last_updated.timestamp()) + + values = { + "metadata_key": "last_updated_timestamp", + "metadata_value": f"{update_time}", + "last_updated_timestamp": update_time, + } + if row: + update_stmt = ( + update(feast_metadata) + .where(feast_metadata.c.metadata_key == "last_updated_timestamp") + .values(values) + ) + conn.execute(update_stmt) + else: + insert_stmt = insert(feast_metadata).values(values,) + conn.execute(insert_stmt) + + def _get_last_updated_metadata(self): + with self.engine.connect() as conn: + stmt = select(feast_metadata).where( + feast_metadata.c.metadata_key == "last_updated_timestamp" + ) + row = conn.execute(stmt).first() + if not row: + return None + update_time = int(row["last_updated_timestamp"]) + + return datetime.utcfromtimestamp(update_time) From a49cc35e0f359ea2091b7bfa11c9ec8cd4e31bca Mon Sep 17 00:00:00 2001 From: vinoyang Date: Mon, 13 Jun 2022 22:05:09 +0800 Subject: [PATCH 055/123] chore: Refactor some minor issues in Java modules (#2787) Signed-off-by: yanghua --- .../logging/entry/MessageAuditLogEntry.java | 28 ++++++++----------- .../interceptors/GrpcMessageInterceptor.java | 6 ++-- .../src/main/java/dev/feast/RequestUtil.java | 4 +-- .../serving/registry/LocalRegistryFile.java | 2 +- .../java/feast/serving/registry/Registry.java | 13 +++++---- .../service/OnlineTransformationService.java | 11 +++----- .../storage/api/retriever/AvroFeature.java | 3 +- 7 files changed, 30 insertions(+), 37 deletions(-) diff --git a/java/common/src/main/java/feast/common/logging/entry/MessageAuditLogEntry.java b/java/common/src/main/java/feast/common/logging/entry/MessageAuditLogEntry.java index 6e5072f66ef..8ad428a3a3d 100644 --- a/java/common/src/main/java/feast/common/logging/entry/MessageAuditLogEntry.java +++ b/java/common/src/main/java/feast/common/logging/entry/MessageAuditLogEntry.java @@ -19,16 +19,13 @@ import com.google.auto.value.AutoValue; import com.google.gson.Gson; import com.google.gson.GsonBuilder; -import com.google.gson.JsonElement; import com.google.gson.JsonParser; -import com.google.gson.JsonSerializationContext; import com.google.gson.JsonSerializer; import com.google.protobuf.Empty; import com.google.protobuf.InvalidProtocolBufferException; import com.google.protobuf.Message; import com.google.protobuf.util.JsonFormat; import io.grpc.Status.Code; -import java.lang.reflect.Type; import java.util.UUID; /** MessageAuditLogEntry records the handling of a Protobuf message by a service call. */ @@ -103,20 +100,17 @@ public String toJSON() { new GsonBuilder() .registerTypeAdapter( Message.class, - new JsonSerializer() { - @Override - public JsonElement serialize( - Message message, Type type, JsonSerializationContext context) { - try { - String messageJSON = JsonFormat.printer().print(message); - return new JsonParser().parse(messageJSON); - } catch (InvalidProtocolBufferException e) { - - throw new RuntimeException( - "Unexpected exception converting Protobuf to JSON", e); - } - } - }) + (JsonSerializer) + (message, type, context) -> { + try { + String messageJSON = JsonFormat.printer().print(message); + return new JsonParser().parse(messageJSON); + } catch (InvalidProtocolBufferException e) { + + throw new RuntimeException( + "Unexpected exception converting Protobuf to JSON", e); + } + }) .create(); return gson.toJson(this); } diff --git a/java/common/src/main/java/feast/common/logging/interceptors/GrpcMessageInterceptor.java b/java/common/src/main/java/feast/common/logging/interceptors/GrpcMessageInterceptor.java index 661642a89ad..e34fefd115c 100644 --- a/java/common/src/main/java/feast/common/logging/interceptors/GrpcMessageInterceptor.java +++ b/java/common/src/main/java/feast/common/logging/interceptors/GrpcMessageInterceptor.java @@ -38,7 +38,7 @@ * GrpcMessageInterceptor assumes that all service calls are unary (ie single request/response). */ public class GrpcMessageInterceptor implements ServerInterceptor { - private LoggingProperties loggingProperties; + private final LoggingProperties loggingProperties; /** * Construct GrpcMessageIntercetor. @@ -78,7 +78,7 @@ public Listener interceptCall( // Register forwarding call to intercept outgoing response and log to audit log call = - new SimpleForwardingServerCall(call) { + new SimpleForwardingServerCall<>(call) { @Override public void sendMessage(RespT message) { // 2. Track the response & Log entry to audit logger @@ -97,7 +97,7 @@ public void close(Status status, Metadata trailers) { }; ServerCall.Listener listener = next.startCall(call, headers); - return new SimpleForwardingServerCallListener(listener) { + return new SimpleForwardingServerCallListener<>(listener) { @Override // Register listener to intercept incoming request messages and log to audit log public void onMessage(ReqT message) { diff --git a/java/sdk/src/main/java/dev/feast/RequestUtil.java b/java/sdk/src/main/java/dev/feast/RequestUtil.java index fc13c453119..da2c0dc42e1 100644 --- a/java/sdk/src/main/java/dev/feast/RequestUtil.java +++ b/java/sdk/src/main/java/dev/feast/RequestUtil.java @@ -35,9 +35,7 @@ public static List createFeatureRefs(List featureRef } List featureRefs = - featureRefStrings.stream() - .map(refStr -> parseFeatureRef(refStr)) - .collect(Collectors.toList()); + featureRefStrings.stream().map(RequestUtil::parseFeatureRef).collect(Collectors.toList()); return featureRefs; } diff --git a/java/serving/src/main/java/feast/serving/registry/LocalRegistryFile.java b/java/serving/src/main/java/feast/serving/registry/LocalRegistryFile.java index b0d6b10bfc0..1da45813eea 100644 --- a/java/serving/src/main/java/feast/serving/registry/LocalRegistryFile.java +++ b/java/serving/src/main/java/feast/serving/registry/LocalRegistryFile.java @@ -24,7 +24,7 @@ import java.util.Optional; public class LocalRegistryFile implements RegistryFile { - private RegistryProto.Registry cachedRegistry; + private final RegistryProto.Registry cachedRegistry; public LocalRegistryFile(String path) { try { diff --git a/java/serving/src/main/java/feast/serving/registry/Registry.java b/java/serving/src/main/java/feast/serving/registry/Registry.java index 37fae3d8dcb..bc953174ea4 100644 --- a/java/serving/src/main/java/feast/serving/registry/Registry.java +++ b/java/serving/src/main/java/feast/serving/registry/Registry.java @@ -17,6 +17,9 @@ package feast.serving.registry; import feast.proto.core.*; +import feast.proto.core.FeatureServiceProto.FeatureService; +import feast.proto.core.FeatureViewProto.FeatureView; +import feast.proto.core.OnDemandFeatureViewProto.OnDemandFeatureView; import feast.proto.serving.ServingAPIProto; import feast.serving.exception.SpecRetrievalException; import java.util.List; @@ -26,16 +29,16 @@ public class Registry { private final RegistryProto.Registry registry; - private Map featureViewNameToSpec; + private final Map featureViewNameToSpec; private Map onDemandFeatureViewNameToSpec; - private Map featureServiceNameToSpec; + private final Map featureServiceNameToSpec; Registry(RegistryProto.Registry registry) { this.registry = registry; List featureViewSpecs = registry.getFeatureViewsList().stream() - .map(fv -> fv.getSpec()) + .map(FeatureView::getSpec) .collect(Collectors.toList()); this.featureViewNameToSpec = featureViewSpecs.stream() @@ -43,7 +46,7 @@ public class Registry { Collectors.toMap(FeatureViewProto.FeatureViewSpec::getName, Function.identity())); List onDemandFeatureViewSpecs = registry.getOnDemandFeatureViewsList().stream() - .map(odfv -> odfv.getSpec()) + .map(OnDemandFeatureView::getSpec) .collect(Collectors.toList()); this.onDemandFeatureViewNameToSpec = onDemandFeatureViewSpecs.stream() @@ -53,7 +56,7 @@ public class Registry { Function.identity())); this.featureServiceNameToSpec = registry.getFeatureServicesList().stream() - .map(fs -> fs.getSpec()) + .map(FeatureService::getSpec) .collect( Collectors.toMap( FeatureServiceProto.FeatureServiceSpec::getName, Function.identity())); diff --git a/java/serving/src/main/java/feast/serving/service/OnlineTransformationService.java b/java/serving/src/main/java/feast/serving/service/OnlineTransformationService.java index 365432b84e1..ae83635b861 100644 --- a/java/serving/src/main/java/feast/serving/service/OnlineTransformationService.java +++ b/java/serving/src/main/java/feast/serving/service/OnlineTransformationService.java @@ -239,8 +239,7 @@ public void processTransformFeaturesResponse( } catch (IOException e) { log.info(e.toString()); throw Status.INTERNAL - .withDescription( - "Unable to correctly process transform features response: " + e.toString()) + .withDescription("Unable to correctly process transform features response: " + e) .asRuntimeException(); } } @@ -249,11 +248,10 @@ public void processTransformFeaturesResponse( public ValueType serializeValuesIntoArrowIPC(List>> values) { // In order to be serialized correctly, the data must be packaged in a VectorSchemaRoot. // We first construct all the columns. - Map columnNameToColumn = new HashMap(); BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE); - List columnFields = new ArrayList(); - List columns = new ArrayList(); + List columnFields = new ArrayList<>(); + List columns = new ArrayList<>(); for (Pair> columnEntry : values) { // The Python FTS does not expect full feature names, so we extract the feature name. @@ -316,8 +314,7 @@ public ValueType serializeValuesIntoArrowIPC(List Date: Mon, 13 Jun 2022 13:08:10 -0700 Subject: [PATCH 056/123] fix: Add project columns in the SQL Registry (#2784) * fix: Add project columns in the SQL Registry Signed-off-by: Achal Shah * allow tests to run locally but not on CI Signed-off-by: Achal Shah * remove duplicate table Signed-off-by: Achal Shah --- sdk/python/feast/infra/registry_stores/sql.py | 177 +++++++++++++----- .../registration/test_sql_registry.py | 16 +- 2 files changed, 140 insertions(+), 53 deletions(-) diff --git a/sdk/python/feast/infra/registry_stores/sql.py b/sdk/python/feast/infra/registry_stores/sql.py index af988edd59b..22b27f4ffea 100644 --- a/sdk/python/feast/infra/registry_stores/sql.py +++ b/sdk/python/feast/infra/registry_stores/sql.py @@ -1,7 +1,7 @@ from datetime import datetime from pathlib import Path from threading import Lock -from typing import Any, List, Optional, Union +from typing import Any, List, Optional, Set, Union from sqlalchemy import ( # type: ignore BigInteger, @@ -66,6 +66,7 @@ "entities", metadata, Column("entity_name", String(50), primary_key=True), + Column("project_id", String(50), primary_key=True), Column("last_updated_timestamp", BigInteger, nullable=False), Column("entity_proto", LargeBinary, nullable=False), ) @@ -74,6 +75,7 @@ "data_sources", metadata, Column("data_source_name", String(50), primary_key=True), + Column("project_id", String(50), primary_key=True), Column("last_updated_timestamp", BigInteger, nullable=False), Column("data_source_proto", LargeBinary, nullable=False), ) @@ -82,6 +84,7 @@ "feature_views", metadata, Column("feature_view_name", String(50), primary_key=True), + Column("project_id", String(50), primary_key=True), Column("last_updated_timestamp", BigInteger, nullable=False), Column("materialized_intervals", LargeBinary, nullable=True), Column("feature_view_proto", LargeBinary, nullable=False), @@ -92,6 +95,7 @@ "request_feature_views", metadata, Column("feature_view_name", String(50), primary_key=True), + Column("project_id", String(50), primary_key=True), Column("last_updated_timestamp", BigInteger, nullable=False), Column("feature_view_proto", LargeBinary, nullable=False), Column("user_metadata", LargeBinary, nullable=True), @@ -101,6 +105,7 @@ "stream_feature_views", metadata, Column("feature_view_name", String(50), primary_key=True), + Column("project_id", String(50), primary_key=True), Column("last_updated_timestamp", BigInteger, nullable=False), Column("feature_view_proto", LargeBinary, nullable=False), Column("user_metadata", LargeBinary, nullable=True), @@ -110,6 +115,7 @@ "on_demand_feature_views", metadata, Column("feature_view_name", String(50), primary_key=True), + Column("project_id", String(50), primary_key=True), Column("last_updated_timestamp", BigInteger, nullable=False), Column("feature_view_proto", LargeBinary, nullable=False), Column("user_metadata", LargeBinary, nullable=True), @@ -119,6 +125,7 @@ "feature_services", metadata, Column("feature_service_name", String(50), primary_key=True), + Column("project_id", String(50), primary_key=True), Column("last_updated_timestamp", BigInteger, nullable=False), Column("feature_service_proto", LargeBinary, nullable=False), ) @@ -127,6 +134,7 @@ "saved_datasets", metadata, Column("saved_dataset_name", String(50), primary_key=True), + Column("project_id", String(50), primary_key=True), Column("last_updated_timestamp", BigInteger, nullable=False), Column("saved_dataset_proto", LargeBinary, nullable=False), ) @@ -135,6 +143,7 @@ "validation_references", metadata, Column("validation_reference_name", String(50), primary_key=True), + Column("project_id", String(50), primary_key=True), Column("last_updated_timestamp", BigInteger, nullable=False), Column("validation_reference_proto", LargeBinary, nullable=False), ) @@ -143,14 +152,15 @@ "managed_infra", metadata, Column("infra_name", String(50), primary_key=True), + Column("project_id", String(50), primary_key=True), Column("last_updated_timestamp", BigInteger, nullable=False), Column("infra_proto", LargeBinary, nullable=False), ) - feast_metadata = Table( "feast_metadata", metadata, + Column("project_id", String(50), primary_key=True), Column("metadata_key", String(50), primary_key=True), Column("metadata_value", String(50), nullable=False), Column("last_updated_timestamp", BigInteger, nullable=False), @@ -208,13 +218,16 @@ def list_stream_feature_views( ) -> List[StreamFeatureView]: return self._list_objects( stream_feature_views, + project, StreamFeatureViewProto, StreamFeatureView, "feature_view_proto", ) def apply_entity(self, entity: Entity, project: str, commit: bool = True): - return self._apply_object(entities, "entity_name", entity, "entity_proto") + return self._apply_object( + entities, project, "entity_name", entity, "entity_proto" + ) def get_entity(self, name: str, project: str, allow_cache: bool = False) -> Entity: return self._get_object( @@ -311,7 +324,9 @@ def get_validation_reference( ) def list_entities(self, project: str, allow_cache: bool = False) -> List[Entity]: - return self._list_objects(entities, EntityProto, Entity, "entity_proto") + return self._list_objects( + entities, project, EntityProto, Entity, "entity_proto" + ) def delete_entity(self, name: str, project: str, commit: bool = True): return self._delete_object( @@ -359,14 +374,14 @@ def list_data_sources( self, project: str, allow_cache: bool = False ) -> List[DataSource]: return self._list_objects( - data_sources, DataSourceProto, DataSource, "data_source_proto" + data_sources, project, DataSourceProto, DataSource, "data_source_proto" ) def apply_data_source( self, data_source: DataSource, project: str, commit: bool = True ): return self._apply_object( - data_sources, "data_source_name", data_source, "data_source_proto" + data_sources, project, "data_source_name", data_source, "data_source_proto" ) def apply_feature_view( @@ -375,7 +390,7 @@ def apply_feature_view( fv_table = self._infer_fv_table(feature_view) return self._apply_object( - fv_table, "feature_view_name", feature_view, "feature_view_proto" + fv_table, project, "feature_view_name", feature_view, "feature_view_proto" ) def apply_feature_service( @@ -383,6 +398,7 @@ def apply_feature_service( ): return self._apply_object( feature_services, + project, "feature_service_name", feature_service, "feature_service_proto", @@ -390,7 +406,10 @@ def apply_feature_service( def delete_data_source(self, name: str, project: str, commit: bool = True): with self.engine.connect() as conn: - stmt = delete(data_sources).where(data_sources.c.data_source_name == name) + stmt = delete(data_sources).where( + data_sources.c.data_source_name == name, + data_sources.c.project_id == project, + ) rows = conn.execute(stmt) if rows.rowcount < 1: raise DataSourceObjectNotFoundException(name, project) @@ -400,6 +419,7 @@ def list_feature_services( ) -> List[FeatureService]: return self._list_objects( feature_services, + project, FeatureServiceProto, FeatureService, "feature_service_proto", @@ -409,14 +429,18 @@ def list_feature_views( self, project: str, allow_cache: bool = False ) -> List[FeatureView]: return self._list_objects( - feature_views, FeatureViewProto, FeatureView, "feature_view_proto" + feature_views, project, FeatureViewProto, FeatureView, "feature_view_proto" ) def list_saved_datasets( self, project: str, allow_cache: bool = False ) -> List[SavedDataset]: return self._list_objects( - saved_datasets, SavedDatasetProto, SavedDataset, "saved_dataset_proto" + saved_datasets, + project, + SavedDatasetProto, + SavedDataset, + "saved_dataset_proto", ) def list_request_feature_views( @@ -424,6 +448,7 @@ def list_request_feature_views( ) -> List[RequestFeatureView]: return self._list_objects( request_feature_views, + project, RequestFeatureViewProto, RequestFeatureView, "feature_view_proto", @@ -434,6 +459,7 @@ def list_on_demand_feature_views( ) -> List[OnDemandFeatureView]: return self._list_objects( on_demand_feature_views, + project, OnDemandFeatureViewProto, OnDemandFeatureView, "feature_view_proto", @@ -443,7 +469,11 @@ def apply_saved_dataset( self, saved_dataset: SavedDataset, project: str, commit: bool = True, ): return self._apply_object( - saved_datasets, "saved_dataset_name", saved_dataset, "saved_dataset_proto" + saved_datasets, + project, + "saved_dataset_name", + saved_dataset, + "saved_dataset_proto", ) def apply_validation_reference( @@ -454,6 +484,7 @@ def apply_validation_reference( ): return self._apply_object( validation_references, + project, "validation_reference_name", validation_reference, "validation_reference_proto", @@ -485,7 +516,9 @@ def apply_materialization( FeatureViewNotFoundException, ) fv.materialization_intervals.append((start_date, end_date)) - self._apply_object(table, "feature_view_name", fv, "feature_view_proto") + self._apply_object( + table, project, "feature_view_name", fv, "feature_view_proto" + ) def delete_validation_reference(self, name: str, project: str, commit: bool = True): self._delete_object( @@ -498,7 +531,7 @@ def delete_validation_reference(self, name: str, project: str, commit: bool = Tr def update_infra(self, infra: Infra, project: str, commit: bool = True): self._apply_object( - managed_infra, "infra_name", infra, "infra_proto", name="infra_obj" + managed_infra, project, "infra_name", infra, "infra_proto", name="infra_obj" ) def get_infra(self, project: str, allow_cache: bool = False) -> Infra: @@ -523,7 +556,10 @@ def apply_user_metadata( name = feature_view.name with self.engine.connect() as conn: - stmt = select(table).where(getattr(table.c, "feature_view_name") == name) + stmt = select(table).where( + getattr(table.c, "feature_view_name") == name, + table.c.project_id == project, + ) row = conn.execute(stmt).first() update_datetime = datetime.utcnow() update_time = int(update_datetime.timestamp()) @@ -534,7 +570,10 @@ def apply_user_metadata( } update_stmt = ( update(table) - .where(getattr(table.c, "feature_view_name") == name) + .where( + getattr(table.c, "feature_view_name") == name, + table.c.project_id == project, + ) .values(values,) ) conn.execute(update_stmt) @@ -583,26 +622,31 @@ def get_user_metadata( def proto(self) -> RegistryProto: r = RegistryProto() - project = "" - for lister, registry_proto_field in [ - (self.list_entities, r.entities), - (self.list_feature_views, r.feature_views), - (self.list_data_sources, r.data_sources), - (self.list_on_demand_feature_views, r.on_demand_feature_views), - (self.list_request_feature_views, r.request_feature_views), - (self.list_stream_feature_views, r.stream_feature_views), - (self.list_feature_services, r.feature_services), - (self.list_saved_datasets, r.saved_datasets), - (self.list_validation_references, r.validation_references), - ]: - objs: List[Any] = lister(project) # type: ignore - if objs: - registry_proto_field.extend([obj.to_proto() for obj in objs]) - - r.infra.CopyFrom(self.get_infra(project).to_proto()) - last_updated_timestamp = self._get_last_updated_metadata() - if last_updated_timestamp: - r.last_updated.FromDatetime(last_updated_timestamp) + last_updated_timestamps = [] + projects = self._get_all_projects() + for project in projects: + for lister, registry_proto_field in [ + (self.list_entities, r.entities), + (self.list_feature_views, r.feature_views), + (self.list_data_sources, r.data_sources), + (self.list_on_demand_feature_views, r.on_demand_feature_views), + (self.list_request_feature_views, r.request_feature_views), + (self.list_stream_feature_views, r.stream_feature_views), + (self.list_feature_services, r.feature_services), + (self.list_saved_datasets, r.saved_datasets), + (self.list_validation_references, r.validation_references), + ]: + objs: List[Any] = lister(project) # type: ignore + if objs: + registry_proto_field.extend([obj.to_proto() for obj in objs]) + + # This is suuuper jank. Because of https://github.com/feast-dev/feast/issues/2783, + # the registry proto only has a single infra field, which we're currently setting as the "last" project. + r.infra.CopyFrom(self.get_infra(project).to_proto()) + last_updated_timestamps.append(self._get_last_updated_metadata(project)) + + if last_updated_timestamps: + r.last_updated.FromDatetime(max(last_updated_timestamps)) return r @@ -610,10 +654,14 @@ def commit(self): # This method is a no-op since we're always writing values eagerly to the db. pass - def _apply_object(self, table, id_field_name, obj, proto_field_name, name=None): + def _apply_object( + self, table, project: str, id_field_name, obj, proto_field_name, name=None + ): name = name or obj.name with self.engine.connect() as conn: - stmt = select(table).where(getattr(table.c, id_field_name) == name) + stmt = select(table).where( + getattr(table.c, id_field_name) == name, table.c.project_id == project + ) row = conn.execute(stmt).first() update_datetime = datetime.utcnow() update_time = int(update_datetime.timestamp()) @@ -636,18 +684,23 @@ def _apply_object(self, table, id_field_name, obj, proto_field_name, name=None): id_field_name: name, proto_field_name: obj.to_proto().SerializeToString(), "last_updated_timestamp": update_time, + "project_id": project, } insert_stmt = insert(table).values(values,) conn.execute(insert_stmt) - self._set_last_updated_metadata(update_datetime) + + self._set_last_updated_metadata(update_datetime, project) def _delete_object(self, table, name, project, id_field_name, not_found_exception): with self.engine.connect() as conn: - stmt = delete(table).where(getattr(table.c, id_field_name) == name) + stmt = delete(table).where( + getattr(table.c, id_field_name) == name, table.c.project_id == project + ) rows = conn.execute(stmt) if rows.rowcount < 1 and not_found_exception: raise not_found_exception(name, project) - self._set_last_updated_metadata(datetime.utcnow()) + self._set_last_updated_metadata(datetime.utcnow(), project) + return rows.rowcount def _get_object( @@ -662,16 +715,20 @@ def _get_object( not_found_exception, ): with self.engine.connect() as conn: - stmt = select(table).where(getattr(table.c, id_field_name) == name) + stmt = select(table).where( + getattr(table.c, id_field_name) == name, table.c.project_id == project + ) row = conn.execute(stmt).first() if row: _proto = proto_class.FromString(row[proto_field_name]) return python_class.from_proto(_proto) raise not_found_exception(name, project) - def _list_objects(self, table, proto_class, python_class, proto_field_name): + def _list_objects( + self, table, project, proto_class, python_class, proto_field_name + ): with self.engine.connect() as conn: - stmt = select(table) + stmt = select(table).where(table.c.project_id == project) rows = conn.execute(stmt).all() if rows: return [ @@ -682,10 +739,11 @@ def _list_objects(self, table, proto_class, python_class, proto_field_name): ] return [] - def _set_last_updated_metadata(self, last_updated: datetime): + def _set_last_updated_metadata(self, last_updated: datetime, project: str): with self.engine.connect() as conn: stmt = select(feast_metadata).where( - feast_metadata.c.metadata_key == "last_updated_timestamp" + feast_metadata.c.metadata_key == "last_updated_timestamp", + feast_metadata.c.project_id == project, ) row = conn.execute(stmt).first() @@ -695,11 +753,15 @@ def _set_last_updated_metadata(self, last_updated: datetime): "metadata_key": "last_updated_timestamp", "metadata_value": f"{update_time}", "last_updated_timestamp": update_time, + "project_id": project, } if row: update_stmt = ( update(feast_metadata) - .where(feast_metadata.c.metadata_key == "last_updated_timestamp") + .where( + feast_metadata.c.metadata_key == "last_updated_timestamp", + feast_metadata.c.project_id == project, + ) .values(values) ) conn.execute(update_stmt) @@ -707,10 +769,11 @@ def _set_last_updated_metadata(self, last_updated: datetime): insert_stmt = insert(feast_metadata).values(values,) conn.execute(insert_stmt) - def _get_last_updated_metadata(self): + def _get_last_updated_metadata(self, project: str): with self.engine.connect() as conn: stmt = select(feast_metadata).where( - feast_metadata.c.metadata_key == "last_updated_timestamp" + feast_metadata.c.metadata_key == "last_updated_timestamp", + feast_metadata.c.project_id == project, ) row = conn.execute(stmt).first() if not row: @@ -718,3 +781,21 @@ def _get_last_updated_metadata(self): update_time = int(row["last_updated_timestamp"]) return datetime.utcfromtimestamp(update_time) + + def _get_all_projects(self) -> Set[str]: + projects = set() + with self.engine.connect() as conn: + for table in { + entities, + data_sources, + feature_views, + request_feature_views, + on_demand_feature_views, + stream_feature_views, + }: + stmt = select(table) + rows = conn.execute(stmt).all() + for row in rows: + projects.add(row["project_id"]) + + return projects diff --git a/sdk/python/tests/integration/registration/test_sql_registry.py b/sdk/python/tests/integration/registration/test_sql_registry.py index 1fe9ff5cecf..c483a7c46f6 100644 --- a/sdk/python/tests/integration/registration/test_sql_registry.py +++ b/sdk/python/tests/integration/registration/test_sql_registry.py @@ -12,6 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. import logging +import os import sys from datetime import timedelta @@ -101,7 +102,8 @@ def mysql_registry(): @pytest.mark.skipif( - sys.platform == "darwin", reason="does not run on mac github actions" + sys.platform == "darwin" and "GITHUB_REF" in os.environ, + reason="does not run on mac github actions", ) @pytest.mark.parametrize( "sql_registry", [lazy_fixture("mysql_registry"), lazy_fixture("pg_registry")], @@ -143,7 +145,8 @@ def test_apply_entity_success(sql_registry): @pytest.mark.skipif( - sys.platform == "darwin", reason="does not run on mac github actions" + sys.platform == "darwin" and "GITHUB_REF" in os.environ, + reason="does not run on mac github actions", ) @pytest.mark.parametrize( "sql_registry", [lazy_fixture("mysql_registry"), lazy_fixture("pg_registry")], @@ -217,7 +220,8 @@ def test_apply_feature_view_success(sql_registry): @pytest.mark.skipif( - sys.platform == "darwin", reason="does not run on mac github actions" + sys.platform == "darwin" and "GITHUB_REF" in os.environ, + reason="does not run on mac github actions", ) @pytest.mark.parametrize( "sql_registry", [lazy_fixture("mysql_registry"), lazy_fixture("pg_registry")], @@ -299,7 +303,8 @@ def location_features_from_push(inputs: pd.DataFrame) -> pd.DataFrame: @pytest.mark.skipif( - sys.platform == "darwin", reason="does not run on mac github actions" + sys.platform == "darwin" and "GITHUB_REF" in os.environ, + reason="does not run on mac github actions", ) @pytest.mark.parametrize( "sql_registry", [lazy_fixture("mysql_registry"), lazy_fixture("pg_registry")], @@ -421,7 +426,8 @@ def odfv1(feature_df: pd.DataFrame) -> pd.DataFrame: @pytest.mark.skipif( - sys.platform == "darwin", reason="does not run on mac github actions" + sys.platform == "darwin" and "GITHUB_REF" in os.environ, + reason="does not run on mac github actions", ) @pytest.mark.integration @pytest.mark.parametrize( From a8d282d3e4f041824ef7479f22c306dbfb8ad569 Mon Sep 17 00:00:00 2001 From: Felix Wang Date: Mon, 13 Jun 2022 13:18:09 -0700 Subject: [PATCH 057/123] fix: Fix SparkKafkaProcessor `query_timeout` parameter (#2789) Signed-off-by: Felix Wang --- sdk/python/feast/infra/contrib/spark_kafka_processor.py | 4 +++- sdk/python/feast/infra/contrib/stream_processor.py | 2 +- sdk/python/feast/stream_feature_view.py | 2 ++ 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/sdk/python/feast/infra/contrib/spark_kafka_processor.py b/sdk/python/feast/infra/contrib/spark_kafka_processor.py index 1e228714c82..57361e5a18f 100644 --- a/sdk/python/feast/infra/contrib/spark_kafka_processor.py +++ b/sdk/python/feast/infra/contrib/spark_kafka_processor.py @@ -17,6 +17,8 @@ class SparkProcessorConfig(ProcessorConfig): spark_session: SparkSession + processing_time: str + query_timeout: int class SparkKafkaProcessor(StreamProcessor): @@ -31,7 +33,7 @@ def __init__( config: ProcessorConfig, write_function: MethodType, processing_time: str = "30 seconds", - query_timeout: str = "15 seconds", + query_timeout: int = 15, ): if not isinstance(sfv.stream_source, KafkaSource): raise ValueError("data source is not kafka source") diff --git a/sdk/python/feast/infra/contrib/stream_processor.py b/sdk/python/feast/infra/contrib/stream_processor.py index cb44b99cd68..2ccf1e59f85 100644 --- a/sdk/python/feast/infra/contrib/stream_processor.py +++ b/sdk/python/feast/infra/contrib/stream_processor.py @@ -81,7 +81,7 @@ def get_stream_processor_object( if config.mode == "spark" and config.source == "kafka": stream_processor = STREAM_PROCESSOR_CLASS_FOR_TYPE[("spark", "kafka")] module_name, class_name = stream_processor.rsplit(".", 1) - cls = import_class(module_name, class_name, "Processor") + cls = import_class(module_name, class_name, "StreamProcessor") return cls(sfv=sfv, config=config, write_function=write_function,) else: raise ValueError("other processors besides spark-kafka not supported") diff --git a/sdk/python/feast/stream_feature_view.py b/sdk/python/feast/stream_feature_view.py index 214ab083abf..3bd525596bf 100644 --- a/sdk/python/feast/stream_feature_view.py +++ b/sdk/python/feast/stream_feature_view.py @@ -132,10 +132,12 @@ def __eq__(self, other): if not super().__eq__(other): return False + if not self.udf: return not other.udf if not other.udf: return False + if ( self.mode != other.mode or self.timestamp_field != other.timestamp_field From 6fec431dd5c9d53a678f58c9b87e2b1cdb02b238 Mon Sep 17 00:00:00 2001 From: Sparsh Dutta Date: Tue, 14 Jun 2022 02:22:08 +0530 Subject: [PATCH 058/123] fix: Fixed custom S3 endpoint read fail (#2786) Signed-off-by: llFireHawkll --- sdk/python/feast/infra/offline_stores/file_source.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/sdk/python/feast/infra/offline_stores/file_source.py b/sdk/python/feast/infra/offline_stores/file_source.py index 5c2a521ac0d..85baa647033 100644 --- a/sdk/python/feast/infra/offline_stores/file_source.py +++ b/sdk/python/feast/infra/offline_stores/file_source.py @@ -179,9 +179,15 @@ def get_table_column_names_and_types( filesystem, path = FileSource.create_filesystem_and_path( self.path, self.file_options.s3_endpoint_override ) - schema = ParquetDataset( - path if filesystem is None else filesystem.open_input_file(path) - ).schema.to_arrow_schema() + # Adding support for different file format path + # based on S3 filesystem + if filesystem is None: + schema = ParquetDataset(path).schema.to_arrow_schema() + else: + schema = ParquetDataset( + filesystem.open_input_file(path), filesystem=filesystem + ).schema + return zip(schema.names, map(str, schema.types)) @staticmethod From 33141f816550d70e1b99b31d7ea104019259e4b1 Mon Sep 17 00:00:00 2001 From: Danny Chiao Date: Tue, 14 Jun 2022 16:10:36 -0400 Subject: [PATCH 059/123] docs: Make quickstart docs consistent and add Web UI to colab (#2794) Signed-off-by: Danny Chiao --- docs/getting-started/quickstart.md | 11 +- examples/quickstart/quickstart.ipynb | 577 ++++++++++++++++----------- 2 files changed, 359 insertions(+), 229 deletions(-) diff --git a/docs/getting-started/quickstart.md b/docs/getting-started/quickstart.md index 1b926eeca5c..b5fe7bad4b9 100644 --- a/docs/getting-started/quickstart.md +++ b/docs/getting-started/quickstart.md @@ -377,6 +377,14 @@ pprint(feature_vector) You can also use feature services to manage multiple features, and decouple feature view definitions and the features needed by end applications. The feature store can also be used to fetch either online or historical features using the same api below. More information can be found [here](https://docs.feast.dev/getting-started/concepts/feature-retrieval). +The `driver_activity` feature service pulls all features from the `driver_hourly_stats` feature view: + +```python +driver_stats_fs = FeatureService( + name="driver_activity", features=[driver_hourly_stats_view] +) +``` + {% tabs %} {% tab title="Python" %} ```python @@ -384,7 +392,7 @@ from feast import FeatureStore feature_store = FeatureStore('.') # Initialize the feature store feature_service = feature_store.get_feature_service("driver_activity") -features = feature_store.get_online_features( +feature_vector = feature_store.get_online_features( features=feature_service, entity_rows=[ # {join_key: entity_value} @@ -392,6 +400,7 @@ features = feature_store.get_online_features( {"driver_id": 1005}, ], ).to_dict() +pprint(feature_vector) ``` {% tabs %} diff --git a/examples/quickstart/quickstart.ipynb b/examples/quickstart/quickstart.ipynb index 60974d27513..d29ee4fa35f 100644 --- a/examples/quickstart/quickstart.ipynb +++ b/examples/quickstart/quickstart.ipynb @@ -1,20 +1,4 @@ { - "nbformat": 4, - "nbformat_minor": 0, - "metadata": { - "colab": { - "name": "Feast Codelab", - "provenance": [], - "collapsed_sections": [] - }, - "kernelspec": { - "name": "python3", - "display_name": "Python 3" - }, - "language_info": { - "name": "python" - } - }, "cells": [ { "cell_type": "markdown", @@ -54,28 +38,20 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "rXNMAAJKQPG5", - "outputId": "52297709-380b-4200-8e7c-3d0102a82ea4" + "outputId": "94fb2260-4453-45c9-ba77-5b384823a621" }, + "outputs": [], "source": [ "%%sh\n", "pip install feast -U -q\n", "pip install Pygments -q\n", "echo \"Please restart your runtime now (Runtime -> Restart runtime). This ensures that the correct dependencies are loaded.\"" - ], - "execution_count": 1, - "outputs": [ - { - "output_type": "stream", - "name": "stdout", - "text": [ - "Please restart your runtime now (Runtime -> Restart runtime). This ensures that the correct dependencies are loaded.\n" - ] - } ] }, { @@ -107,28 +83,32 @@ }, { "cell_type": "code", + "execution_count": 1, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "IhirSkgUvYau", - "outputId": "df90af1a-06bd-48a1-94e6-7def19e87d5f" + "outputId": "664367b9-6a2a-493d-fd78-6495fb459fa2" }, - "source": [ - "!feast init feature_repo" - ], - "execution_count": 1, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "Feast is an open source project that collects anonymized error reporting and usage statistics. To opt out or learn more see https://docs.feast.dev/reference/usage\n", + "/usr/local/lib/python3.7/dist-packages/scipy/fft/__init__.py:97: DeprecationWarning: The module numpy.dual is deprecated. Instead of using dual, use the functions directly from numpy or scipy.\n", + " from numpy.dual import register_func\n", + "/usr/local/lib/python3.7/dist-packages/scipy/sparse/sputils.py:17: DeprecationWarning: `np.typeDict` is a deprecated alias for `np.sctypeDict`.\n", + " supported_dtypes = [np.typeDict[x] for x in supported_dtypes]\n", "\n", "Creating a new Feast repository in \u001b[1m\u001b[32m/content/feature_repo\u001b[0m.\n", "\n" ] } + ], + "source": [ + "!feast init feature_repo" ] }, { @@ -150,22 +130,18 @@ }, { "cell_type": "code", + "execution_count": 2, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "9jXuzt4ovzA3", - "outputId": "bff15f0c-9f8e-4a3c-e605-5ad84be30709" + "outputId": "9e326892-f0cc-4d86-d0b2-f33f822f83a9" }, - "source": [ - "%cd feature_repo\n", - "!ls -R" - ], - "execution_count": 2, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "/content/feature_repo\n", ".:\n", @@ -175,6 +151,10 @@ "driver_stats.parquet\n" ] } + ], + "source": [ + "%cd feature_repo\n", + "!ls -R" ] }, { @@ -198,21 +178,18 @@ }, { "cell_type": "code", + "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "9_YJ--uYdtcP", - "outputId": "89268e31-6be0-43fb-e576-6d335a2c1dd9" + "outputId": "af56a8da-9ca2-4dd9-f73c-a60dd3e1613a" }, - "source": [ - "!pygmentize feature_store.yaml" - ], - "execution_count": 3, "outputs": [ { - "output_type": "stream", "name": "stdout", + "output_type": "stream", "text": [ "\u001b[94mproject\u001b[39;49;00m: feature_repo\n", "\u001b[94mregistry\u001b[39;49;00m: data/registry.db\n", @@ -221,6 +198,9 @@ " \u001b[94mpath\u001b[39;49;00m: data/online_store.db\n" ] } + ], + "source": [ + "!pygmentize feature_store.yaml" ] }, { @@ -236,56 +216,21 @@ }, { "cell_type": "code", + "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/", - "height": 424 + "height": 423 }, "id": "sIF2lO59dwzi", - "outputId": "80e798d5-df21-4ebd-de1c-9bde282bd742" + "outputId": "8931930b-b32f-43e1-d45b-de230489c7b8" }, - "source": [ - "import pandas as pd\n", - "\n", - "pd.read_parquet(\"data/driver_stats.parquet\")" - ], - "execution_count": 4, "outputs": [ { - "output_type": "execute_result", "data": { - "text/plain": [ - " event_timestamp driver_id conv_rate acc_rate \\\n", - "0 2022-03-31 14:00:00+00:00 1005 0.313336 0.231481 \n", - "1 2022-03-31 15:00:00+00:00 1005 0.959499 0.942614 \n", - "2 2022-03-31 16:00:00+00:00 1005 0.231786 0.313516 \n", - "3 2022-03-31 17:00:00+00:00 1005 0.886911 0.531613 \n", - "4 2022-03-31 18:00:00+00:00 1005 0.574945 0.718223 \n", - "... ... ... ... ... \n", - "1802 2022-04-15 12:00:00+00:00 1001 0.521622 0.266667 \n", - "1803 2022-04-15 13:00:00+00:00 1001 0.003188 0.535501 \n", - "1804 2021-04-12 07:00:00+00:00 1001 0.709081 0.823138 \n", - "1805 2022-04-08 02:00:00+00:00 1003 0.033297 0.053268 \n", - "1806 2022-04-08 02:00:00+00:00 1003 0.033297 0.053268 \n", - "\n", - " avg_daily_trips created \n", - "0 303 2022-04-15 14:34:10.056 \n", - "1 842 2022-04-15 14:34:10.056 \n", - "2 782 2022-04-15 14:34:10.056 \n", - "3 634 2022-04-15 14:34:10.056 \n", - "4 441 2022-04-15 14:34:10.056 \n", - "... ... ... \n", - "1802 406 2022-04-15 14:34:10.056 \n", - "1803 593 2022-04-15 14:34:10.056 \n", - "1804 997 2022-04-15 14:34:10.056 \n", - "1805 534 2022-04-15 14:34:10.056 \n", - "1806 534 2022-04-15 14:34:10.056 \n", - "\n", - "[1807 rows x 6 columns]" - ], "text/html": [ "\n", - "
\n", + "
\n", "
\n", "
\n", "