Skip to content
Merged
16 changes: 8 additions & 8 deletions .secrets.baseline
Original file line number Diff line number Diff line change
Expand Up @@ -957,7 +957,7 @@
"filename": "infra/feast-operator/api/v1/featurestore_types.go",
"hashed_secret": "44e17306b837162269a410204daaa5ecee4ec22c",
"is_verified": false,
"line_number": 970
"line_number": 1029
}
],
"infra/feast-operator/api/v1/zz_generated.deepcopy.go": [
Expand All @@ -966,21 +966,21 @@
"filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go",
"hashed_secret": "f914fc9324de1bec1ad13dec94a8ea2ddb41fc87",
"is_verified": false,
"line_number": 842
"line_number": 909
},
{
"type": "Secret Keyword",
"filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go",
"hashed_secret": "44e17306b837162269a410204daaa5ecee4ec22c",
"is_verified": false,
"line_number": 903
"line_number": 970
},
{
"type": "Secret Keyword",
"filename": "infra/feast-operator/api/v1/zz_generated.deepcopy.go",
"hashed_secret": "c2028031c154bbe86fd69bef740855c74b927dcf",
"is_verified": false,
"line_number": 1605
"line_number": 1672
}
],
"infra/feast-operator/api/v1alpha1/featurestore_types.go": [
Expand Down Expand Up @@ -1165,14 +1165,14 @@
"filename": "infra/feast-operator/internal/controller/services/repo_config.go",
"hashed_secret": "44e17306b837162269a410204daaa5ecee4ec22c",
"is_verified": false,
"line_number": 133
"line_number": 137
},
{
"type": "Secret Keyword",
"filename": "infra/feast-operator/internal/controller/services/repo_config.go",
"hashed_secret": "e2fb052132fd6a07a56af2013e0b62a1f510572c",
"is_verified": false,
"line_number": 241
"line_number": 245
}
],
"infra/feast-operator/internal/controller/services/services.go": [
Expand All @@ -1181,7 +1181,7 @@
"filename": "infra/feast-operator/internal/controller/services/services.go",
"hashed_secret": "36dc326eb15c7bdd8d91a6b87905bcea20b637d1",
"is_verified": false,
"line_number": 184
"line_number": 233
}
],
"infra/feast-operator/internal/controller/services/tls_test.go": [
Expand Down Expand Up @@ -1564,5 +1564,5 @@
}
]
},
"generated_at": "2026-08-18T09:56:16Z"
"generated_at": "2026-08-18T12:17:21Z"
}
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@
* [5 — Security](how-to-guides/feast-operator/05-security.md)
* [6 — Batch & Jobs](how-to-guides/feast-operator/06-batch-and-jobs.md)
* [7 — OpenLineage & Materialization](how-to-guides/feast-operator/07-openlineage-and-materialization.md)
* [8 — MLflow Integration](how-to-guides/feast-operator/08-mlflow-integration.md)
* [Feast Production Deployment Topologies](how-to-guides/production-deployment-topologies.md)
* [Online Server Performance Tuning](how-to-guides/online-server-performance-tuning.md)
* [Customizing Feast](how-to-guides/customizing-feast/README.md)
Expand Down
193 changes: 193 additions & 0 deletions docs/how-to-guides/feast-operator/08-mlflow-integration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
# Guide 8 — MLflow Integration

The operator auto-discovers MLflow on RHOAI/ODH clusters and enables experiment tracking
for every FeatureStore deployment. When the MLflow operator is present and healthy, Feast
pods receive MLflow configuration automatically — no manual YAML editing required.

---

## Auto-discovery

The operator lists all `MLflow` CRs (`mlflow.opendatahub.io/v1`) in the cluster and uses
the first one with an `Available=True` or `Ready=True` condition. When found, it populates
`tracking_uri` from `status.address.url` and `ui_url` from `status.url`.

If the MLflow CR does not report conditions (older operator versions), auto-discovery will
not activate. Set `trackingUri` explicitly in that case.

> **No MLflow?** The FeatureStore stays Ready. Non-MLflow FeatureViews and all other Feast
> services are completely unaffected.

---

## FeatureStore CR configuration

### Auto-enabled (default when MLflow is present)

No `spec.mlflow` needed. The operator auto-enables when an Available MLflow CR is detected:

```yaml
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
name: my-store
spec:
feastProject: my_project
services:
onlineStore: {}
registry: {}
ui: {}
```

### Explicit configuration

Override defaults or enable additional features:

```yaml
apiVersion: feast.dev/v1
kind: FeatureStore
metadata:
name: my-store
spec:
feastProject: my_project
services:
onlineStore: {}
registry: {}
ui: {}
mlflow:
enabled: true
trackingUri: "https://custom-mlflow.example.com:8443"
uiUrl: "https://dashboard.example.com/mlflow"
trackingAuth: "kubernetes-namespaced"
autoLog: true
autoLogEntityDf: true
entityDfMaxRows: 50000
logOperations: true
opsExperimentSuffix: "-feast-ops"
```

### Opt-out

Disable MLflow even when the MLflow operator is present:

```yaml
spec:
mlflow:
enabled: false
```

---

## Field reference

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `enabled` | bool | auto-detected | Master switch for MLflow integration |
| `trackingUri` | string | auto-discovered | MLflow tracking server URI (in-cluster, from `status.address.url`) |
| `uiUrl` | string | auto-discovered | Browser-reachable MLflow URL for Feast UI lineage links (from `status.url`) |
| `trackingAuth` | *string | `"kubernetes-namespaced"` | Auth method for Feast pods calling MLflow |
| `autoLog` | *bool | `true` | Auto-log feature metadata on every retrieval |
| `autoLogEntityDf` | *bool | `false` | Save entity DataFrame as artifact |
| `entityDfMaxRows` | *int32 | `100000` | Skip artifact for large DataFrames |
| `logOperations` | *bool | `false` | Log `feast apply` / `materialize` to ops experiment |
| `opsExperimentSuffix` | *string | `"-feast-ops"` | Ops experiment name suffix |
| `extraConfig` | map[string]string | — | Additional YAML fields (coerced to native types) |

---

## Authentication

The operator injects `MLFLOW_TRACKING_AUTH` into all Feast pod containers. The MLflow
Python client's auth plugin system uses this env var to attach credentials to tracking
server requests.

| `trackingAuth` value | Behavior |
|---------------------|----------|
| `"kubernetes-namespaced"` (default) | SA token + `X-MLFLOW-WORKSPACE: <namespace>` header. Multi-tenant on RHOAI. |
| `"kubernetes"` | SA token only. Single-tenant setups. |
| `"basic"` | HTTP Basic auth via `MLFLOW_TRACKING_USERNAME` / `MLFLOW_TRACKING_PASSWORD` env vars. |
| `"bearer"` | Static bearer token from `MLFLOW_TRACKING_TOKEN` env var. |
| `""` (empty string) | No auth header. Local dev or unprotected MLflow. |

No Kubernetes RoleBinding is needed for MLflow tracking API access. The MLflow server
validates the SA token directly via TokenReview.

---

## Tracking URI resolution order

1. Explicit `trackingUri` in the FeatureStore CR
2. Auto-discovered from MLflow CR `status.address.url` (first Available/Ready CR)
3. `MLFLOW_TRACKING_URI` environment variable (on workbench pods, injected by the MLflow operator)
4. MLflow default (`./mlruns`)

---

## UI URL resolution order

Used for browser hyperlinks in Feast UI lineage panels:

1. Explicit `uiUrl` in the FeatureStore CR
2. `MLFLOW_UI_URL` environment variable
3. Auto-discovered from MLflow CR `status.url` (external gateway route)
4. Falls back to `trackingUri` (works for local dev)

---

## Graceful degradation

| Scenario | Behavior |
|----------|----------|
| MLflow operator not installed | No `mlflow` block in YAML; FeatureStore stays Ready |
| MLflow CR exists but not Ready | Discovery returns empty; MLflow stays off |
| Tracking URI becomes unreachable | SDK logs a warning; feature retrieval is not blocked |
| `spec.mlflow.enabled: false` | MLflow integration explicitly disabled |

---

## Workbench usage

In a RHOAI workbench notebook connected to the FeatureStore:

```python
from feast import FeatureStore

store = FeatureStore(...) # from mounted client config

with store.mlflow.start_run(run_name="training"):
df = store.get_historical_features(
entity_df=entity_df,
features=["driver_stats:conv_rate", "driver_stats:acc_rate"],
).to_df()
model = train(df)
store.mlflow.log_model(model, "model")
```

> **Dependency**: the Feast image must include `feast[mlflow]` (`mlflow` or `mlflow-skinny`).

---

## RBAC permissions

The operator needs `get`, `list`, `watch` on `mlflows` in the `mlflow.opendatahub.io` API
group. This is included in the default operator ClusterRole.

```yaml
- apiGroups:
- mlflow.opendatahub.io
resources:
- mlflows
verbs:
- get
- list
- watch
```

---

## See also

- [API field reference — `MlflowConfig`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md)
- [MLflow DataSource reference](../../reference/mlflow.md)
- [Guide 5 — Security](05-security.md) (RBAC / OIDC auth)
- [Guide 7 — OpenLineage & Materialization](07-openlineage-and-materialization.md)
2 changes: 2 additions & 0 deletions docs/how-to-guides/feast-operator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ look for store-specific YAML options in the Feast SDK docs.
| 5 | [Security](05-security.md) | Kubernetes RBAC roles vs OIDC auth; TLS for all servers |
| 6 | [Batch Jobs](06-batch-and-jobs.md) | `batchEngine` ConfigMap contract, `cronJob` for scheduled materialization |
| 7 | [OpenLineage & Materialization](07-openlineage-and-materialization.md) | Lineage transports, API key Secret, materialization batch size |
| 8 | [MLflow Integration](08-mlflow-integration.md) | Auto-discovery, experiment tracking, auth, Feast UI lineage |

---

Expand All @@ -42,6 +43,7 @@ look for store-specific YAML options in the Feast SDK docs.
- **"How do I enable Kubernetes RBAC or OIDC auth?"** → [Guide 5](05-security.md)
- **"How do I schedule nightly materialization?"** → [Guide 6](06-batch-and-jobs.md)
- **"How do I send lineage events to Marquez?"** → [Guide 7](07-openlineage-and-materialization.md)
- **"How do I connect Feast to MLflow for experiment tracking?"** → [Guide 8](08-mlflow-integration.md)
- **"What are all valid fields on `ServingConfig`?"** → [API ref](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#servingconfig)

---
Expand Down
59 changes: 59 additions & 0 deletions infra/feast-operator/api/v1/featurestore_types.go
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,59 @@ type OpenLineageConsumerConfig struct {
NamespaceMapping map[string]string `json:"namespaceMapping,omitempty"`
}

// MlflowConfig enables MLflow experiment tracking integration for Feast.
// When enabled, feature retrieval metadata is automatically logged to MLflow runs
// and the Feast UI displays lineage from feature views to registered models.
// +kubebuilder:validation:XValidation:rule="!has(self.extraConfig) || !('enabled' in self.extraConfig) && !('tracking_uri' in self.extraConfig) && !('ui_url' in self.extraConfig) && !('tracking_auth' in self.extraConfig) && !('auto_log' in self.extraConfig) && !('auto_log_entity_df' in self.extraConfig) && !('entity_df_max_rows' in self.extraConfig) && !('log_operations' in self.extraConfig) && !('ops_experiment_suffix' in self.extraConfig)",message="extraConfig must not contain keys that duplicate typed fields (enabled, tracking_uri, ui_url, tracking_auth, auto_log, auto_log_entity_df, entity_df_max_rows, log_operations, ops_experiment_suffix); use the corresponding spec fields instead."
type MlflowConfig struct {
// Enable MLflow integration.
Enabled bool `json:"enabled"`
// MLflow tracking server URI. When omitted, the operator auto-discovers
// from the cluster MLflow CR (status.address.url). Falls back to
// MLFLOW_TRACKING_URI env var on pods.
// +optional
TrackingUri *string `json:"trackingUri,omitempty"`
// Browser-reachable MLflow UI URL used for hyperlinks in Feast UI lineage.
// When omitted, the operator auto-discovers from the MLflow CR status.url
// (the external gateway route). Falls back to MLFLOW_UI_URL env var, then
// to trackingUri. Only needed when the tracking URI is cluster-internal.
// +optional
UiUrl *string `json:"uiUrl,omitempty"`
// Automatically log feature metadata on every retrieval inside an active MLflow run.
// Defaults to true when enabled.
// +optional
AutoLog *bool `json:"autoLog,omitempty"`
// Save entity DataFrame as MLflow artifact on historical retrieval.
// Defaults to false.
// +optional
AutoLogEntityDf *bool `json:"autoLogEntityDf,omitempty"`
// Maximum number of entity DataFrame rows to save as an MLflow artifact.
// DataFrames exceeding this limit are skipped. Defaults to 100000.
// +kubebuilder:validation:Minimum=1
// +optional
EntityDfMaxRows *int32 `json:"entityDfMaxRows,omitempty"`
// Log feast apply and materialize operations to a separate MLflow experiment.
// Defaults to false.
// +optional
LogOperations *bool `json:"logOperations,omitempty"`
// Suffix appended to the project name for the operations experiment.
// Defaults to "-feast-ops".
// +optional
OpsExperimentSuffix *string `json:"opsExperimentSuffix,omitempty"`
// Authentication method used by Feast pods when calling the MLflow tracking
// server. Common values: "kubernetes-namespaced" (token-based, default on
// OpenShift AI), "basic", "bearer", or "" (no auth for local/dev).
// Defaults to "kubernetes-namespaced".
// +optional
TrackingAuth *string `json:"trackingAuth,omitempty"`
// ExtraConfig holds additional MLflow key-value settings written inline into
// the mlflow block of feature_store.yaml. Boolean and integer string values
// are coerced to native YAML types. Keys must be valid Feast MlflowConfig
// YAML field names.
// +optional
ExtraConfig map[string]string `json:"extraConfig,omitempty"`
}

// FeatureStoreSpec defines the desired state of FeatureStore
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 || !has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)",message="replicas > 1 and services.scaling.autoscaling are mutually exclusive."
// +kubebuilder:validation:XValidation:rule="self.replicas <= 1 && (!has(self.services) || !has(self.services.scaling) || !has(self.services.scaling.autoscaling)) || (has(self.services) && has(self.services.onlineStore) && has(self.services.onlineStore.persistence) && has(self.services.onlineStore.persistence.store))",message="Scaling requires DB-backed persistence for the online store. Configure services.onlineStore.persistence.store when using replicas > 1 or autoscaling."
Expand Down Expand Up @@ -172,6 +225,12 @@ type FeatureStoreSpec struct {
// Written into feature_store.yaml for all service pods.
// +optional
OpenLineage *OpenLineageConfig `json:"openlineage,omitempty"`
// Mlflow enables MLflow experiment tracking integration for Feast.
// Written into feature_store.yaml for all service pods and the client ConfigMap.
// When omitted and a cluster MLflow instance is detected, defaults to enabled
// with the discovered tracking URI.
// +optional
Mlflow *MlflowConfig `json:"mlflow,omitempty"`
}

// FeastProjectDir defines how to create the feast project directory.
Expand Down
Loading
Loading