Overview

Relevant source files

• CONTRIBUTING.md
• README.md
• docs/SUMMARY.md
• docs/getting-started/architecture/write-patterns.md
• docs/getting-started/quickstart.md
• docs/getting-started/third-party-integrations.md
• docs/how-to-guides/feast-on-kubernetes.md
• docs/how-to-guides/running-feast-in-production.md
• docs/how-to-guides/scaling-feast.md
• docs/how-to-guides/structuring-repos.md
• docs/project/maintainers.md
• docs/reference/codebase-structure.md
• docs/reference/feature-servers/offline-feature-server.md
• docs/reference/registries/README.md
• docs/reference/registries/hdfs.md
• docs/reference/registries/metadata.md
• docs/roadmap.md
• examples/quickstart/quickstart.ipynb
• infra/feast-operator/README.md
• infra/feast-operator/config/samples/kustomization.yaml
• infra/templates/README.md.jinja2
• protos/feast/core/Registry.proto
• sdk/python/feast/driver_test_data.py
• sdk/python/feast/errors.py
• sdk/python/feast/feature_store.py
• sdk/python/feast/infra/offline_stores/bigquery.py
• sdk/python/feast/infra/offline_stores/offline_store.py
• sdk/python/feast/infra/offline_stores/offline_utils.py
• sdk/python/feast/infra/offline_stores/redshift.py
• sdk/python/feast/infra/online_stores/online_store.py
• sdk/python/feast/infra/online_stores/sqlite.py
• sdk/python/feast/infra/passthrough_provider.py
• sdk/python/feast/infra/provider.py
• sdk/python/feast/infra/registry/contrib/hdfs/__init__.py
• sdk/python/feast/infra/registry/contrib/hdfs/hdfs_registry_store.py
• sdk/python/feast/infra/utils/aws_utils.py
• sdk/python/feast/repo_config.py
• sdk/python/feast/repo_operations.py
• sdk/python/tests/foo_provider.py
• sdk/python/tests/unit/infra/registry/test_sql_registry.py

This document provides a high-level introduction to Feast (Feature Store), explaining its purpose as a machine learning feature platform and its core architecture. For detailed information about specific components, see:

• System Architecture details: System Architecture
• Getting Started guide: Getting Started and CLI
• Core concepts and abstractions: Core Concepts
• Data integration specifics: Data Integration

What is Feast?

Feast (Feature Store) is an open-source machine learning feature store that manages the infrastructure for productionizing analytic data for model training and online inference. It provides a unified data access layer that abstracts feature storage from feature retrieval.

Key Capabilities:

• Training-Serving Consistency: Manages both an offline store (historical data for batch scoring/training) and an online store (low-latency real-time prediction)
• Point-in-Time Correctness: Prevents data leakage by ensuring future feature values don't leak to models during training through point-in-time correct joins
• Infrastructure Decoupling: Abstracts underlying data infrastructure, allowing models to remain portable across different data systems

Primary Use Cases:

1. Building training datasets with point-in-time correct feature joins
2. Materializing features from offline to online stores
3. Serving features for real-time model inference
4. Managing feature definitions and metadata across teams

Sources: README.md26-38 docs/getting-started/quickstart.md1-16 sdk/python/feast/feature_store.py104-107

Central Orchestration: The FeatureStore Class

FeatureStore Class Architecture

The FeatureStore class at sdk/python/feast/feature_store.py104-192 is the primary entry point for all Feast operations. It orchestrates interactions between the registry, provider, offline stores, and online stores.

Core Components:

• config (RepoConfig): Configuration loaded from feature_store.yaml containing project name, registry settings, offline/online store configs
• _registry (BaseRegistry): Metadata store that persists feature definitions (entities, feature views, feature services)
• _provider (Provider): Abstraction layer that delegates operations to offline/online stores
• repo_path (Path): File system path to the feature repository

Initialization Flow:

1. Loads configuration from feature_store.yaml or accepts a RepoConfig object sdk/python/feast/feature_store.py120-157
2. Instantiates the appropriate registry type (File, SQL, Snowflake, or Remote) based on registry_type sdk/python/feast/feature_store.py159-180
3. Gets the provider implementation via get_provider() sdk/python/feast/feature_store.py182

Sources: sdk/python/feast/feature_store.py104-207

Configuration Management

Configuration System Architecture

The RepoConfig class at sdk/python/feast/repo_config.py194-558 manages all Feast configuration, loaded from feature_store.yaml. It uses Pydantic for validation and supports multiple configuration patterns.

Key Configuration Properties:

Property	Type	Purpose
project	str	Unique project identifier for multi-tenant deployments
provider	str	Provider type: local, gcp, aws, azure
registry	RegistryConfig	Registry backend configuration
offline_store	OfflineStoreConfig	Offline store type and connection details
online_store	OnlineStoreConfig	Online store type and connection details
batch_engine	ComputeEngineConfig	Materialization engine configuration

Registry Configuration Types sdk/python/feast/repo_config.py136-184:

• File Registry: Path-based (local filesystem, S3, GCS, Azure Blob) with cache_ttl_seconds for caching
• SQL Registry: Database connection string (PostgreSQL, MySQL, SQLite) for scalable transactional registry
• Snowflake Registry: Native Snowflake table storage
• Remote Registry: gRPC/HTTP endpoint for centralized multi-client access

Type Resolution sdk/python/feast/repo_config.py39-107:

• REGISTRY_CLASS_FOR_TYPE: Maps string types to registry implementation classes
• OFFLINE_STORE_CLASS_FOR_TYPE: Maps offline store types (e.g., "bigquery" → "feast.infra.offline_stores.bigquery.BigQueryOfflineStore")
• ONLINE_STORE_CLASS_FOR_TYPE: Maps online store types (e.g., "redis" → "feast.infra.online_stores.redis.RedisOnlineStore")

Sources: sdk/python/feast/repo_config.py194-558 sdk/python/feast/repo_config.py39-122

Feature Lifecycle and Data Flow

Feature Data Flow Through Feast

Phase 1: Development sdk/python/feast/repo_operations.py114-220

• Data scientists define feature views, entities, and data sources in Python files
• The parse_repo() function discovers these definitions by importing Python modules

Phase 2: Registration sdk/python/feast/feature_store.py862-1070

• feast apply invokes FeatureStore.apply() which:
  • Validates feature definitions sdk/python/feast/feature_store.py578-597
  • Performs schema inference via update_feature_views_with_inferred_features_and_entities() sdk/python/feast/feature_store.py624-635
  • Commits metadata to the registry using apply_diff_to_registry() sdk/python/feast/feature_store.py845-846
  • Updates infrastructure (online store tables) via Provider.update_infra() sdk/python/feast/infra/provider.py69-92

Phase 3: Training Data Generation sdk/python/feast/feature_store.py1440-1599

• get_historical_features() accepts an entity DataFrame with timestamps
• Delegates to Provider.get_historical_features() which calls the offline store implementation
• Performs point-in-time correct joins to prevent data leakage

Phase 4: Materialization sdk/python/feast/feature_store.py2041-2135

• materialize() or materialize_incremental() loads features from offline to online stores
• Calls Provider.materialize_single_feature_view() for each feature view
• Uses OfflineStore.pull_latest_from_table_or_query() to read latest feature values sdk/python/feast/infra/offline_stores/bigquery.py127-183
• Writes to online store via Provider.online_write_batch() sdk/python/feast/infra/provider.py124-147

Phase 5: Online Serving sdk/python/feast/feature_store.py1615-1799

• get_online_features() retrieves features for real-time inference
• Constructs entity keys and calls Provider.get_online_features() sdk/python/feast/infra/provider.py308-320
• Returns OnlineResponse with feature values and metadata

Phase 6: Streaming Ingestion sdk/python/feast/feature_store.py1944-2014

• write_to_online_store() accepts streaming data (Kafka/Kinesis) or direct pushes
• Converts data to online store format and writes via Provider.online_write_batch()

Sources: sdk/python/feast/feature_store.py862-1070 sdk/python/feast/feature_store.py1440-1599 sdk/python/feast/feature_store.py2041-2135 sdk/python/feast/feature_store.py1615-1799 sdk/python/feast/repo_operations.py114-220

Storage Architecture

Dual Storage System

Feast maintains two storage layers:

Offline Stores (Historical Feature Values)

• Purpose: Store complete historical feature data for training and batch scoring
• Access Pattern: Large-scale batch reads with point-in-time joins
• Implementation: sdk/python/feast/infra/offline_stores/offline_store.py73-338

Supported Offline Stores sdk/python/feast/repo_config.py91-107:

Store Type	Class	Primary Method
BigQuery	BigQueryOfflineStore	bigquery.py125-341
Snowflake	SnowflakeOfflineStore	snowflake.py
Redshift	RedshiftOfflineStore	redshift.py95-183
File/Dask	DaskOfflineStore	Local parquet files
Spark	SparkOfflineStore	Distributed processing

Key Methods:

• pull_latest_from_table_or_query(): Retrieves latest feature values per entity sdk/python/feast/infra/offline_stores/bigquery.py127-183
• get_historical_features(): Performs point-in-time joins for training data sdk/python/feast/infra/offline_stores/bigquery.py235-340

Online Stores (Real-Time Feature Values)

• Purpose: Serve feature values for online inference with low latency (typically <10ms)
• Access Pattern: Key-value reads by entity key
• Implementation: sdk/python/feast/infra/online_stores/online_store.py29-380

Supported Online Stores sdk/python/feast/repo_config.py68-89:

Store Type	Class	Infrastructure Object
SQLite	SqliteOnlineStore	sqlite.py93-567
Redis	RedisOnlineStore	In-memory cache
DynamoDB	DynamoDBOnlineStore	AWS managed NoSQL
Snowflake	SnowflakeOnlineStore	Hybrid tables

Key Methods:

• online_write_batch(): Writes feature values to online store sdk/python/feast/infra/online_stores/sqlite.py253-312
• online_read(): Retrieves feature values by entity key sdk/python/feast/infra/online_stores/sqlite.py314-357
• update(): Creates/updates online store tables sdk/python/feast/infra/online_stores/sqlite.py159-227

Provider Abstraction sdk/python/feast/infra/passthrough_provider.py58-495

• The PassthroughProvider delegates all storage operations to the configured offline and online stores
• Implements the Provider interface defined in sdk/python/feast/infra/provider.py49-474
• Properties: offline_store, online_store, batch_engine sdk/python/feast/infra/passthrough_provider.py69-129

Sources: sdk/python/feast/infra/offline_stores/offline_store.py73-338 sdk/python/feast/infra/online_stores/online_store.py29-380 sdk/python/feast/infra/offline_stores/bigquery.py125-341 sdk/python/feast/infra/online_stores/sqlite.py93-567 sdk/python/feast/infra/passthrough_provider.py58-129 sdk/python/feast/repo_config.py68-107

Registry: Metadata Management

Registry Architecture

The registry is Feast's metadata store, persisting all feature definitions. The BaseRegistry abstract class defines the interface at sdk/python/feast/infra/registry/base_registry.py

Registry Contents:

• Entities: Primary keys for feature joins
• Feature Views: Collections of features and their data sources
• On-Demand Feature Views: Computed features
• Stream Feature Views: Real-time streaming features
• Feature Services: Named groups of features for serving
• Data Sources: Source table/file configurations
• Permissions: Access control policies
• Infrastructure State: Materialization history, online store metadata

Registry Implementations:

1. File Registry sdk/python/feast/infra/registry/registry.py

   • Serializes registry to a single file (protocol buffer format)
   • Supports multiple backends via RegistryStore:
     • Local filesystem: path: data/registry.db
     • S3: path: s3://bucket/registry.db
     • GCS: path: gs://bucket/registry.db
     • Azure Blob: Supported via pluggable store
   • Configuration: sdk/python/feast/repo_config.py136-184
   • Cache TTL: cache_ttl_seconds (default 600s)

2. SQL Registry sdk/python/feast/infra/registry/sql.py

   • Uses SQLAlchemy for transactional metadata storage
   • Supports: PostgreSQL, MySQL, SQLite
   • Configuration: path: postgresql+psycopg://user:pass@host/db
   • Type mapping: sdk/python/feast/repo_config.py39-44

3. Snowflake Registry sdk/python/feast/infra/registry/snowflake.py

   • Stores registry in native Snowflake tables
   • Configuration: registry_type: snowflake.registry

4. Remote Registry sdk/python/feast/infra/registry/remote.py

   • gRPC/HTTP client connecting to a centralized registry server
   • Enables multi-client, multi-environment deployments
   • Configuration: registry_type: remote

Caching Strategy:

• Registries implement a TTL-based cache to reduce I/O
• cache_ttl_seconds: Time before cache refresh sdk/python/feast/repo_config.py150-154
• cache_mode: sync (immediate refresh) or thread (background refresh) sdk/python/feast/repo_config.py156-160
• Manual refresh: FeatureStore.refresh_registry() sdk/python/feast/feature_store.py208-223

Registry Selection in FeatureStore sdk/python/feast/feature_store.py159-180:

Sources: sdk/python/feast/infra/registry/base_registry.py sdk/python/feast/infra/registry/registry.py sdk/python/feast/infra/registry/sql.py sdk/python/feast/feature_store.py159-180 sdk/python/feast/repo_config.py136-184

Deployment Patterns

Local Development:

• SQLite online store: type: sqlite, path: data/online_store.db sdk/python/feast/infra/online_stores/sqlite.py93-567
• File registry: registry: data/registry.db docs/getting-started/quickstart.md106-118
• Dask offline store: Local parquet files
• Use case: Prototyping, testing, small datasets

Cloud Deployment:

• Managed services for scalability:
  • Offline: BigQuery, Snowflake, Redshift sdk/python/feast/repo_config.py91-107
  • Online: Redis, DynamoDB sdk/python/feast/repo_config.py68-89
  • Registry: SQL database or S3/GCS sdk/python/feast/repo_config.py39-44
• Configuration example from docs/getting-started/quickstart.md106-118:

Kubernetes Deployment docs/how-to-guides/running-feast-in-production.md1-10:

• Feast Operator manages custom resources infra/feast-operator/README.md1-7
• Components deployed as Kubernetes resources:
  • Feature servers (Python/Go/Java)
  • Registry server (optional, for remote registry)
  • Web UI
  • Materialization CronJobs
• Helm charts available for deployment infra/feast-operator/README.md

Multi-Environment Setup docs/how-to-guides/running-feast-in-production.md44-49:

• Staging environment for testing feature definitions
• Production environment with production data
• CI/CD pipeline running feast plan and feast apply docs/how-to-guides/running-feast-in-production.md39-42

Scaling Considerations docs/how-to-guides/scaling-feast.md1-5:

• Registry: Use SQL registry for multi-client scenarios
• Materialization: Use compute engines (Snowflake, Spark, Bytewax) for large-scale ETL
• Online serving: Deploy feature servers (Python/Go) for high-throughput inference

Sources: docs/getting-started/quickstart.md106-118 docs/how-to-guides/running-feast-in-production.md1-86 docs/how-to-guides/scaling-feast.md1-5 infra/feast-operator/README.md1-7 sdk/python/feast/infra/online_stores/sqlite.py93-567 sdk/python/feast/repo_config.py39-107

Command Line Interface

The Feast CLI provides commands for managing the feature store lifecycle. Key commands are invoked through sdk/python/feast/repo_operations.py:

Core Commands:

Command	Function	Implementation
feast init	Bootstrap new feature repository	repo_operations.py491-579
feast apply	Register features and update infrastructure	repo_operations.py432-458
feast plan	Preview changes (dry-run)	repo_operations.py223-246
feast materialize	Load features to online store	feature_store.py2041-2135
feast materialize-incremental	Incremental materialization	feature_store.py2137-2229
feast serve	Start feature server	feature_server.py
feast ui	Start web UI	ui_server.py

Apply Workflow sdk/python/feast/repo_operations.py432-458:

1. Parse feature repository using parse_repo() repo_operations.py114-220
2. Validate data sources repo_operations.py345-351
3. Calculate registry diff using diff_between() repo_operations.py354-359
4. Update infrastructure via store.apply() repo_operations.py384-390
5. Commit changes to registry

Sources: sdk/python/feast/repo_operations.py114-579 sdk/python/feast/feature_store.py2041-2229

Summary

Feast provides a complete feature store platform with:

1. Centralized Orchestration: FeatureStore class managing all operations
2. Flexible Configuration: RepoConfig supporting multiple storage backends
3. Dual Storage Architecture: Offline stores (training) + Online stores (serving)
4. Metadata Management: Registry implementations for various deployment scales
5. Point-in-Time Correctness: Preventing data leakage in training pipelines
6. Multi-Language Support: Python SDK (primary), Go/Java feature servers
7. Production-Ready: CI/CD integration, Kubernetes deployment, scalable compute engines

For specific implementation details, see the subsequent pages in this wiki covering individual components and subsystems.

Sources: README.md26-244 sdk/python/feast/feature_store.py104-207 sdk/python/feast/repo_config.py194-558 docs/getting-started/quickstart.md1-206