Skip to content

Commit c8628eb

Browse files
VperiodtVperiodt
authored
feat(cli): Updated feast init demo by adding rag template (#5946)
feat(cli): add RAG template as opt-in option for feast init Add a new RAG (Retrieval-Augmented Generation) template that can be selected via `feast init -t rag`. The template provides a City Q&A demo using Feast for feature management and Milvus for vector search. The default `feast init` behavior is unchanged — it continues to use the local template, preserving operator compatibility and avoiding heavy dependencies (pymilvus, torch, transformers) in the default getting-started flow. Fixes #5264 Signed-off-by: Vanshika Vanshika <vvanshik@redhat.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED
1 parent ef307c6 commit c8628eb

9 files changed

Lines changed: 503 additions & 0 deletions

File tree

‎sdk/python/feast/cli/cli.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -477,6 +477,7 @@ def materialize_incremental_command(
477477
"milvus",
478478
"ray",
479479
"ray_rag",
480+
"rag",
480481
"pytorch_nlp",
481482
],
482483
case_sensitive=False,

‎sdk/python/feast/templates/rag/README.md‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# City Information Q&A — RAG Demo with Feast
2+
3+
A complete Retrieval-Augmented Generation (RAG) demo using Feast for feature management and Milvus for vector search.
4+
5+
6+
## Project Structure
7+
8+
```
9+
rag/
10+
├── feature_repo/
11+
│ ├── data/
12+
│ │ └── city_wikipedia_summaries_with_embeddings.parquet # Sample data (US cities)
13+
│ ├── example_repo.py # Entity, Feature Views, Feature Service definitions
14+
│ ├── feature_store.yaml # Feast config (Milvus online store, file offline store)
15+
│ └── test_workflow.py # End-to-end demo: apply → materialize → search
16+
└── README.md
17+
```
18+
19+
## Quick Start
20+
21+
### 1. Initialize the template
22+
23+
```bash
24+
feast init -t rag my_city_qa
25+
cd my_city_qa/feature_repo
26+
```
27+
28+
### 2. Install dependencies
29+
30+
```bash
31+
pip install feast torch transformers pymilvus
32+
```
33+
34+
### 3. Apply feature definitions
35+
36+
```bash
37+
feast apply
38+
```
39+
40+
### 4. Explore in the Feast UI
41+
42+
```bash
43+
feast ui
44+
```
45+
46+
### 5. Run the demo workflow
47+
48+
```bash
49+
python test_workflow.py
50+
```
51+
52+
53+
## Key Commands
54+
55+
| Command | Description |
56+
|---------|-------------|
57+
| `feast apply` | Register entities, feature views, and feature services |
58+
| `feast materialize --disable-event-timestamp` | Load parquet data into the online store (Milvus) for vector search. Optionally add `-v city_summary_embeddings -v city_metadata` to materialize only those views. |
59+
| `feast feature-views list` | List registered feature views |
60+
| `feast entities list` | List registered entities |
61+
| `feast feature-services list` | List registered feature services |
62+
| `feast ui` | Start the Feast UI at http://localhost:8888 |
63+
64+
65+
## Architecture
66+
67+
```
68+
┌─────────────────────────────────────────────────────────────────┐
69+
│ City Q&A Pipeline │
70+
├─────────────────────────────────────────────────────────────────┤
71+
│ │
72+
│ User Question │
73+
│ │ │
74+
│ ▼ │
75+
│ ┌─────────────┐ │
76+
│ │ Embed Query │ (MiniLM 384-dim) │
77+
│ └─────────────┘ │
78+
│ │ │
79+
│ ▼ │
80+
│ ┌─────────────────────────────────────┐ │
81+
│ │ city_summary_embeddings (Milvus) │ ← Vector Search │
82+
│ │ - vector (COSINE similarity) │ │
83+
│ │ - sentence_chunks │ │
84+
│ └─────────────────────────────────────┘ │
85+
│ │ │
86+
│ ▼ (top-k city_ids) │
87+
│ ┌─────────────────────────────────────┐ │
88+
│ │ city_metadata (Feast Online Store) │ ← Metadata Lookup │
89+
│ │ - state │ │
90+
│ │ - wiki_summary │ │
91+
│ └─────────────────────────────────────┘ │
92+
│ │ │
93+
│ ▼ │
94+
│ ┌─────────────┐ │
95+
│ │ LLM Answer │ (optional: GPT/Claude) │
96+
│ └─────────────┘ │
97+
│ │
98+
└─────────────────────────────────────────────────────────────────┘
99+
```

‎sdk/python/feast/templates/rag/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎sdk/python/feast/templates/rag/bootstrap.py‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
def bootstrap():
2+
# Bootstrap() is called from init_repo() during `feast init`
3+
import pathlib
4+
from datetime import datetime
5+
6+
import numpy as np
7+
import pandas as pd
8+
9+
repo_path = pathlib.Path(__file__).parent.absolute() / "feature_repo"
10+
data_path = repo_path / "data"
11+
data_path.mkdir(exist_ok=True)
12+
13+
# Minimal city data with embeddings (384-d to match feature_store embedding_dim)
14+
embedding_dim = 384
15+
now = datetime.now().replace(microsecond=0, tzinfo=None)
16+
cities = [
17+
(
18+
1,
19+
"New York",
20+
"New York",
21+
"New York City is the most populous city in the United States.",
22+
),
23+
(
24+
2,
25+
"Los Angeles",
26+
"California",
27+
"Los Angeles is the second most populous city in the United States.",
28+
),
29+
(
30+
3,
31+
"Chicago",
32+
"Illinois",
33+
"Chicago is the third most populous city in the United States.",
34+
),
35+
]
36+
rows = []
37+
for city_id, city_name, state, wiki_summary in cities:
38+
vec = np.random.randn(embedding_dim).astype(np.float32)
39+
vec = (vec / np.linalg.norm(vec)).tolist()
40+
rows.append(
41+
{
42+
"city_id": city_id,
43+
"event_timestamp": pd.Timestamp(now),
44+
"vector": vec,
45+
"sentence_chunks": wiki_summary[:200],
46+
"state": f"{city_name}, {state}",
47+
"wiki_summary": wiki_summary,
48+
}
49+
)
50+
df = pd.DataFrame(rows)
51+
parquet_path = data_path / "city_wikipedia_summaries_with_embeddings.parquet"
52+
df.to_parquet(path=str(parquet_path), index=False)
53+
54+
55+
if __name__ == "__main__":
56+
bootstrap()

‎sdk/python/feast/templates/rag/feature_repo/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎sdk/python/feast/templates/rag/feature_repo/example_repo.py‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
from datetime import timedelta
2+
3+
from feast import (
4+
Entity,
5+
FeatureService,
6+
FeatureView,
7+
Field,
8+
FileSource,
9+
PushSource,
10+
)
11+
from feast.data_format import ParquetFormat
12+
from feast.types import Array, Float32, String
13+
from feast.value_type import ValueType
14+
15+
# Entity: Identifies each city document/chunk in the knowledge base
16+
city = Entity(
17+
name="city_id",
18+
value_type=ValueType.INT64,
19+
description="Unique identifier for each city Wikipedia summary (document chunk ID).",
20+
join_keys=["city_id"],
21+
)
22+
23+
# Data Source: Parquet file containing city summaries with pre-computed embeddings
24+
city_summaries_source = FileSource(
25+
name="city_summaries_source",
26+
file_format=ParquetFormat(),
27+
path="./data/city_wikipedia_summaries_with_embeddings.parquet",
28+
timestamp_field="event_timestamp",
29+
description="Wikipedia summaries of US cities (batch).",
30+
)
31+
32+
# Push Source: same schema as batch; allows near real-time ingestion of new/updated docs
33+
city_summaries_push_source = PushSource(
34+
name="city_summaries_push_source",
35+
batch_source=city_summaries_source,
36+
description="Push source for real-time updates to city summaries/embeddings.",
37+
)
38+
39+
# Feature View 1: City embeddings for semantic/vector search (RAG retrieval)
40+
city_summary_embeddings = FeatureView(
41+
name="city_summary_embeddings",
42+
description="City Wikipedia summaries with embeddings for semantic search. ",
43+
entities=[city],
44+
schema=[
45+
Field(
46+
name="vector",
47+
dtype=Array(Float32),
48+
description="384-dimensional sentence embedding for semantic similarity search (MiniLM).",
49+
vector_index=True,
50+
vector_search_metric="COSINE",
51+
),
52+
Field(
53+
name="sentence_chunks",
54+
dtype=String,
55+
description="Chunked sentences from the Wikipedia summary.",
56+
),
57+
],
58+
source=city_summaries_source,
59+
ttl=timedelta(days=1),
60+
online=True,
61+
tags={"team": "ml-platform", "use_case": "city_qa", "type": "vector"},
62+
)
63+
64+
# Feature View 2: City metadata for scalar lookups (no vector search)
65+
city_metadata = FeatureView(
66+
name="city_metadata",
67+
description="City metadata including state and full Wikipedia summary. ",
68+
entities=[city],
69+
schema=[
70+
Field(
71+
name="state",
72+
dtype=String,
73+
description="US state where the city is located (e.g., 'New York, New York').",
74+
),
75+
Field(
76+
name="wiki_summary",
77+
dtype=String,
78+
description="Full Wikipedia summary of the city.",
79+
),
80+
],
81+
source=city_summaries_source,
82+
ttl=timedelta(hours=2),
83+
online=True,
84+
tags={"team": "ml-platform", "use_case": "city_qa", "type": "metadata"},
85+
)
86+
87+
# Feature View 3: Fresh embeddings (PushSource) for near real-time doc updates
88+
city_summary_embeddings_realtime = FeatureView(
89+
name="city_summary_embeddings_realtime",
90+
description="Same as city_summary_embeddings but with real-time ingestion (PushSource).",
91+
entities=[city],
92+
schema=[
93+
Field(
94+
name="vector",
95+
dtype=Array(Float32),
96+
description="384-dimensional sentence embedding for semantic similarity search.",
97+
vector_index=True,
98+
vector_search_metric="COSINE",
99+
),
100+
Field(
101+
name="sentence_chunks",
102+
dtype=String,
103+
description="Chunked sentences from the Wikipedia summary.",
104+
),
105+
],
106+
source=city_summaries_push_source,
107+
ttl=timedelta(hours=2),
108+
online=True,
109+
tags={
110+
"team": "ml-platform",
111+
"use_case": "city_qa",
112+
"type": "vector",
113+
"ingestion": "push",
114+
},
115+
)
116+
117+
# Feature Service: Bundles features for the City Q&A retrieval endpoint
118+
city_qa_v1 = FeatureService(
119+
name="city_qa_v1",
120+
features=[
121+
city_summary_embeddings,
122+
city_metadata,
123+
],
124+
description="Feature service for City Information Q&A. ",
125+
tags={"team": "ml-platform", "version": "v1"},
126+
)
127+
128+
# Feature service that includes push-backed and request-time features
129+
city_qa_v2 = FeatureService(
130+
name="city_qa_v2",
131+
features=[
132+
city_summary_embeddings_realtime,
133+
city_metadata,
134+
],
135+
description="City Q&A with push ingestion and request-time context (query_text, user_id).",
136+
tags={"team": "ml-platform", "version": "v2"},
137+
)

‎sdk/python/feast/templates/rag/feature_repo/feature_store.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
project: my_project
2+
3+
project_description: |
4+
This project is a smart City Q&A assistant that gives you accurate, detailed answers about any city instantly.
5+
provider: local
6+
registry: data/registry.db
7+
online_store:
8+
type: milvus
9+
path: data/online_store.db
10+
vector_enabled: true
11+
embedding_dim: 384
12+
index_type: "FLAT"
13+
metric_type: "COSINE"
14+
15+
offline_store:
16+
type: file
17+
entity_key_serialization_version: 3
18+
# By default, no_auth for authentication and authorization
19+
auth:
20+
type: no_auth

0 commit comments

Comments
 (0)