# Introduction ## What is Feast? Feast (**Fea**ture **St**ore) is an [open-source](https://github.com/feast-dev/feast) feature store that helps teams operate production ML systems at scale by allowing them to define, manage, validate, and serve features for production AI/ML. Feast's feature store is composed of two foundational components: (1) an [offline store](/master/getting-started/components/offline-store) for historical feature extraction used in model training and (2) an [online store](/master/getting-started/components/online-store) for serving features at low-latency in production systems and applications. Feast is a configurable operational data system that re-uses existing infrastructure to manage and serve machine learning features to real-time models. For more details, please review our [architecture](/master/getting-started/architecture/overview). Concretely, Feast provides: * A Python SDK for programmatically defining features, entities, sources, and (optionally) transformations * A Python SDK for reading and writing features to configured offline and online data stores * An [optional feature server](/master/reference/feature-servers) for reading and writing features (useful for non-python languages) * A [UI](/master/reference/alpha-web-ui) for viewing and exploring information about features defined in the project * A [CLI tool](/master/reference/feast-cli-commands) for viewing and updating feature information Feast allows ML platform teams to: * **Make features consistently available for training and low-latency serving** by managing an *offline store* (to process historical data for scale-out batch scoring or model training), a low-latency *online store* (to power real-time prediction)*,* and a battle-tested *feature server* (to serve pre-computed features online). * **Avoid data leakage** by generating point-in-time correct feature sets so data scientists can focus on feature engineering rather than debugging error-prone dataset joining logic. This ensures that future feature values do not leak to models during training. * **Decouple ML from data infrastructure** by providing a single data access layer that abstracts feature storage from feature retrieval, ensuring models remain portable as you move from training models to serving models, from batch models to real-time models, and from one data infra system to another. {% hint style="info" %} **Note:** Feast today primarily addresses *timestamped structured data*. {% endhint %} ![](/files/pz51PMnsQviEBGWU7rW1) {% hint style="info" %} **Note:** Feast uses a push model for online serving. This means that the feature store pushes feature values to the online store, which reduces the latency of feature retrieval. This is more efficient than a pull model, where the model serving system must make a request to the feature store to retrieve feature values. See [this document](/master/getting-started/architecture/push-vs-pull-model) for a more detailed discussion. {% endhint %} ## Who is Feast for? Feast helps ML platform/MLOps teams with DevOps experience productionize real-time models. Feast also helps these teams build a feature platform that improves collaboration between data engineers, software engineers, machine learning engineers, and data scientists. * *For Data Scientists*: Feast is a tool where you can easily define, store, and retrieve your features for both model development and model deployment. By using Feast, you can focus on what you do best: build features that power your AI/ML models and maximize the value of your data. * *For MLOps Engineers*: Feast is a library that allows you to connect your existing infrastructure (e.g., online database, application server, microservice, analytical database, and orchestration tooling) that enables your Data Scientists to ship features for their models to production using a friendly SDK without having to be concerned with software engineering challenges that occur from serving real-time production systems. By using Feast, you can focus on maintaining a resilient system, instead of implementing features for Data Scientists. * *For Data Engineers*: Feast provides a centralized catalog for storing feature definitions, allowing one to maintain a single source of truth for feature data. It provides the abstraction for reading and writing to many different types of offline and online data stores. Using either the provided Python SDK or the feature server service, users can write data to the online and/or offline stores and then read that data out again in either low-latency online scenarios for model inference, or in batch scenarios for model training. * *For AI Engineers*: Feast provides a platform designed to scale your AI applications by enabling seamless integration of richer data and facilitating fine-tuning. With Feast, you can optimize the performance of your AI models while ensuring a scalable and efficient data pipeline. ![](/files/wYIdVnRNtK9kJsjFLLpD) ## What Feast is not? ### Feast is not * **An** [**ETL**](https://en.wikipedia.org/wiki/Extract,_transform,_load) / [**ELT**](https://en.wikipedia.org/wiki/Extract,_load,_transform) **system.** Feast is not a general purpose data pipelining system. Users often leverage tools like [dbt](https://www.getdbt.com) to manage upstream data transformations. Feast does support some [transformations](/master/getting-started/architecture/feature-transformation). * **A data orchestration tool:** Feast does not manage or orchestrate complex workflow DAGs. It relies on upstream data pipelines to produce feature values and integrations with tools like [Airflow](https://airflow.apache.org) to make features consistently available. * **A data warehouse:** Feast is not a replacement for your data warehouse or the source of truth for all transformed data in your organization. Rather, Feast is a lightweight downstream layer that can serve data from an existing data warehouse (or other data sources) to models in production. * **A database:** Feast is not a database, but helps manage data stored in other systems (e.g. BigQuery, Snowflake, DynamoDB, Redis) to make features consistently available at training / serving time ### Feast does not *fully* solve * **reproducible model training / model backtesting / experiment management**: Feast captures feature and model metadata, but does not version-control datasets / labels or manage train / test splits. Other tools like [DVC](https://dvc.org/), [MLflow](https://www.mlflow.org/), and [Kubeflow](https://www.kubeflow.org/) are better suited for this. * **batch feature engineering**: Feast supports on-demand and streaming transformations. Feast is also investing in supporting batch transformations. * **native streaming feature integration:** Feast enables users to push streaming features, but does not pull from streaming sources or manage streaming pipelines. * **lineage:** Feast helps tie feature values to model versions, but is not a complete solution for capturing end-to-end lineage from raw data sources to model versions. Feast also has community contributed plugins with [DataHub](https://datahubproject.io/docs/generated/ingestion/sources/feast/) and [Amundsen](https://github.com/amundsen-io/amundsen/blob/4a9d60176767c4d68d1cad5b093320ea22e26a49/databuilder/databuilder/extractor/feast_extractor.py). * **data quality / drift detection**: Feast includes built-in [Feature Quality Monitoring](/master/how-to-guides/feature-monitoring) that computes statistical metrics (null rates, distributions, percentiles), detects drift across batch data and serving logs, and provides a monitoring UI dashboard. ## Example use cases Many companies have used Feast to power real-world ML use cases such as: * Personalizing online recommendations by leveraging pre-computed historical user or item features. * Online fraud detection, using features that compare against (pre-computed) historical transaction patterns * Churn prediction (an offline model), generating feature values for all users at a fixed cadence in batch * Credit scoring, using pre-computed historical features to compute the probability of default ## How can I get started? {% hint style="info" %} The best way to learn Feast is to use it. Head over to our [Quickstart](/master/getting-started/quickstart) and try it out! {% endhint %} Explore the following resources to get started with Feast: * [Quickstart](/master/getting-started/quickstart) is the fastest way to get started with Feast * [Concepts](/master/getting-started/concepts) describes all important Feast API concepts * [Architecture](/master/getting-started/architecture) describes Feast's overall architecture. * [Tutorials](/master/tutorials/tutorials-overview) show full examples of using Feast in machine learning applications. * [Running Feast with Snowflake/GCP/AWS](/master/how-to-guides/feast-snowflake-gcp-aws) provides a more in-depth guide to using Feast. * [Reference](/master/reference/feast-cli-commands) contains detailed API and design documents. * [Contributing](/master/project/contributing) contains resources for anyone who wants to contribute to Feast. # Blog Welcome to the Feast blog! Here you'll find articles about feature store development, new features, and community updates. ## Featured Posts {% content-ref url="" %} {% endcontent-ref %} {% content-ref url="" %} {% endcontent-ref %} {% content-ref url="" %} {% endcontent-ref %} {% content-ref url="" %} {% endcontent-ref %} # Community & getting help ## Links & Resources * [Come say hi on Slack!](https://slack.feast.dev/) * As a part of the Linux Foundation, we ask community members to adhere to the [Linux Foundation Code of Conduct](https://events.linuxfoundation.org/about/code-of-conduct/) * [GitHub Repository](https://github.com/feast-dev/feast/): Find the complete Feast codebase on GitHub. * [Community Governance Doc](https://github.com/feast-dev/feast/blob/master/community): See the governance model of Feast, including who the maintainers are and how decisions are made. * [Google Folder](https://drive.google.com/drive/u/0/folders/1jgMHOPDT2DvBlJeO9LCM79DP4lm4eOrR): This folder is used as a central repository for all Feast resources. For example: * Design proposals in the form of Request for Comments (RFC). * User surveys and meeting minutes. * Slide decks of conferences our contributors have spoken at. * [Feast Linux Foundation Wiki](https://wiki.lfaidata.foundation/display/FEAST/Feast+Home): Our LFAI wiki page contains links to resources for contributors and maintainers. ## How can I get help? * **GitHub Issues:** Found a bug or need a feature? [Create an issue on GitHub](https://github.com/feast-dev/feast/issues/new). # Roadmap The list below contains the functionality that contributors are planning to develop for Feast. * We welcome contribution to all items in the roadmap! * **Natural Language Processing** * [x] Vector Search (Alpha release. See [RFC](https://docs.google.com/document/d/18IWzLEA9i2lDWnbfbwXnMCg3StlqaLVI-uRpQjr_Vos/edit#heading=h.9gaqqtox9jg6)) * [ ] [Enhanced Feature Server and SDK for native support for NLP](https://github.com/feast-dev/feast/issues/4964) * **Data Sources** * [x] [Snowflake source](https://docs.feast.dev/reference/data-sources/snowflake) * [x] [Redshift source](https://docs.feast.dev/reference/data-sources/redshift) * [x] [BigQuery source](https://docs.feast.dev/reference/data-sources/bigquery) * [x] [Parquet file source](https://docs.feast.dev/reference/data-sources/file) * [x] [Azure Synapse + Azure SQL source (contrib plugin)](https://docs.feast.dev/reference/data-sources/mssql) * [x] [Hive (community plugin)](https://github.com/baineng/feast-hive) * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/data-sources/postgres) * [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/data-sources/spark) * [x] [Couchbase (contrib plugin)](https://docs.feast.dev/reference/data-sources/couchbase) * [x] [Athena (contrib plugin)](https://docs.feast.dev/reference/data-sources/athena) * [x] [Clickhouse (contrib plugin)](https://docs.feast.dev/reference/data-sources/clickhouse) * [x] [Oracle (contrib plugin)](https://docs.feast.dev/reference/data-sources/oracle) * [x] [MongoDB (contrib plugin)](https://docs.feast.dev/reference/data-sources/mongodb) * [x] [Ray source (contrib plugin)](https://docs.feast.dev/reference/data-sources/ray) * [x] Kafka / Kinesis sources (via [push support into the online store](https://docs.feast.dev/reference/data-sources/push)) * **Offline Stores** * [x] [Snowflake](https://docs.feast.dev/reference/offline-stores/snowflake) * [x] [Redshift](https://docs.feast.dev/reference/offline-stores/redshift) * [x] [BigQuery](https://docs.feast.dev/reference/offline-stores/bigquery) * [x] [DuckDB](https://docs.feast.dev/reference/offline-stores/duckdb) * [x] [Dask](https://docs.feast.dev/reference/offline-stores/dask) * [x] [Remote](https://docs.feast.dev/reference/offline-stores/remote-offline-store) * [x] [Azure Synapse + Azure SQL (contrib plugin)](https://docs.feast.dev/reference/offline-stores/mssql) * [x] [Hive (community plugin)](https://github.com/baineng/feast-hive) * [x] [Postgres (contrib plugin)](https://docs.feast.dev/reference/offline-stores/postgres) * [x] [Trino (contrib plugin)](https://docs.feast.dev/reference/offline-stores/trino) * [x] [Spark (contrib plugin)](https://docs.feast.dev/reference/offline-stores/spark) * [x] [Couchbase (contrib plugin)](https://docs.feast.dev/reference/offline-stores/couchbase) * [x] [Athena (contrib plugin)](https://docs.feast.dev/reference/offline-stores/athena) * [x] [Clickhouse (contrib plugin)](https://docs.feast.dev/reference/offline-stores/clickhouse) * [x] [Ray (contrib plugin)](https://docs.feast.dev/reference/offline-stores/ray) * [x] [Oracle (contrib plugin)](https://docs.feast.dev/reference/offline-stores/oracle) * [x] [MongoDB (contrib plugin)](https://docs.feast.dev/reference/offline-stores/mongodb) * [x] [Hybrid](https://docs.feast.dev/reference/offline-stores/hybrid) * [x] [Custom offline store support](https://docs.feast.dev/how-to-guides/customizing-feast/adding-a-new-offline-store) * **Online Stores** * [x] [Snowflake](https://docs.feast.dev/reference/online-stores/snowflake) * [x] [DynamoDB](https://docs.feast.dev/reference/online-stores/dynamodb) * [x] [Redis](https://docs.feast.dev/reference/online-stores/redis) * [x] [Dragonfly](https://docs.feast.dev/reference/online-stores/dragonfly) * [x] [Datastore](https://docs.feast.dev/reference/online-stores/datastore) * [x] [Bigtable](https://docs.feast.dev/reference/online-stores/bigtable) * [x] [SQLite](https://docs.feast.dev/reference/online-stores/sqlite) * [x] [Remote](https://docs.feast.dev/reference/online-stores/remote) * [x] [Postgres](https://docs.feast.dev/reference/online-stores/postgres) * [x] [HBase](https://docs.feast.dev/reference/online-stores/hbase) * [x] [Cassandra / AstraDB](https://docs.feast.dev/reference/online-stores/cassandra) * [x] [ScyllaDB](https://docs.feast.dev/reference/online-stores/scylladb) * [x] [MySQL](https://docs.feast.dev/reference/online-stores/mysql) * [x] [Hazelcast](https://docs.feast.dev/reference/online-stores/hazelcast) * [x] [Elasticsearch](https://docs.feast.dev/reference/online-stores/elasticsearch) * [x] [SingleStore](https://docs.feast.dev/reference/online-stores/singlestore) * [x] [Couchbase](https://docs.feast.dev/reference/online-stores/couchbase) * [x] [MongoDB](https://docs.feast.dev/reference/online-stores/mongodb) * [x] [Aerospike](https://docs.feast.dev/reference/online-stores/aerospike) * [x] [Qdrant (vector store)](https://docs.feast.dev/reference/online-stores/qdrant) * [x] [Milvus (vector store)](https://docs.feast.dev/reference/online-stores/milvus) * [x] [Faiss (vector store)](https://docs.feast.dev/reference/online-stores/faiss) * [x] [Hybrid](https://docs.feast.dev/reference/online-stores/hybrid) * [x] [Azure Cache for Redis (community plugin)](https://github.com/Azure/feast-azure) * [x] [Custom online store support](https://docs.feast.dev/how-to-guides/customizing-feast/adding-support-for-a-new-online-store) * **Feature Engineering** * [x] On-demand Transformations (On Read) (Beta release. See [RFC](https://docs.google.com/document/d/1lgfIw0Drc65LpaxbUu49RCeJgMew547meSJttnUqz7c/edit#)) * [x] Streaming Transformations (Alpha release. See [RFC](https://docs.google.com/document/d/1UzEyETHUaGpn0ap4G82DHluiCj7zEbrQLkJJkKSv4e8/edit)) * [ ] Batch transformation (In progress. See [RFC](https://docs.google.com/document/d/1964OkzuBljifDvkV-0fakp2uaijnVzdwWNGdz7Vz50A/edit)) * [x] On-demand Transformations (On Write) (Beta release. See [GitHub Issue](https://github.com/feast-dev/feast/issues/4376)) * **Streaming** * [x] [Custom streaming ingestion job support](https://docs.feast.dev/how-to-guides/customizing-feast/creating-a-custom-provider) * [x] [Push based streaming data ingestion to online store](https://docs.feast.dev/reference/data-sources/push) * [x] [Push based streaming data ingestion to offline store](https://docs.feast.dev/reference/data-sources/push) * **Deployments** * [x] AWS Lambda (Alpha release. See [RFC](https://docs.google.com/document/d/1eZWKWzfBif66LDN32IajpaG-j82LSHCCOzY6R7Ax7MI/edit)) * [x] Kubernetes (See [guide](https://docs.feast.dev/how-to-guides/running-feast-in-production)) * **Feature Serving** * [x] Python Client * [x] [Python feature server](https://docs.feast.dev/reference/feature-servers/python-feature-server) * [x] [Feast Operator (alpha)](https://github.com/feast-dev/feast/blob/master/infra/feast-operator/README.md) * [x] [Java feature server (alpha)](https://github.com/feast-dev/feast/blob/master/infra/charts/feast/README.md) * [x] [Go feature server (alpha)](https://docs.feast.dev/reference/feature-servers/go-feature-server) * [x] [Offline Feature Server (alpha)](https://docs.feast.dev/reference/feature-servers/offline-feature-server) * [x] [Registry server (alpha)](https://github.com/feast-dev/feast/blob/master/docs/reference/feature-servers/registry-server.md) * **Data Quality Management (See** [**RFC**](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit)**)** * [x] [Feature Quality Monitoring](https://docs.feast.dev/how-to-guides/feature-monitoring) — built-in metrics, drift detection, serving log monitoring, and UI dashboard * **Feature Discovery and Governance** * [x] Python SDK for browsing feature registry * [x] CLI for browsing feature registry * [x] Model-centric feature tracking (feature services) * [x] Amundsen integration (see [Feast extractor](https://github.com/amundsen-io/amundsen/blob/main/databuilder/databuilder/extractor/feast_extractor.py)) * [x] DataHub integration (see [DataHub Feast docs](https://datahubproject.io/docs/generated/ingestion/sources/feast/)) * [x] Feast Web UI (Beta release. See [docs](https://docs.feast.dev/reference/alpha-web-ui)) * [ ] Feast Lineage Explorer # Quickstart ## What is Feast? Feast (Feature Store) is an open-source feature store designed to facilitate the management and serving of machine learning features in a way that supports both batch and real-time applications. * *For Data Scientists*: Feast is a tool where you can easily define, store, and retrieve your features for both model development and model deployment. By using Feast, you can focus on what you do best: build features that power your AI/ML models and maximize the value of your data. * *For MLOps Engineers*: Feast is a library that allows you to connect your existing infrastructure (e.g., online database, application server, microservice, analytical database, and orchestration tooling) that enables your Data Scientists to ship features for their models to production using a friendly SDK without having to be concerned with software engineering challenges that occur from serving real-time production systems. By using Feast, you can focus on maintaining a resilient system, instead of implementing features for Data Scientists. * *For Data Engineers*: Feast provides a centralized catalog for storing feature definitions allowing one to maintain a single source of truth for feature data. It provides the abstraction for reading and writing to many different types of offline and online data stores. Using either the provided python SDK or the feature server service, users can write data to the online and/or offline stores and then read that data out again in either low-latency online scenarios for model inference, or in batch scenarios for model training. * *For AI Engineers*: Feast provides a platform designed to scale your AI applications by enabling seamless integration of richer data and facilitating fine-tuning. With Feast, you can optimize the performance of your AI models while ensuring a scalable and efficient data pipeline. For more info refer to [Introduction to feast](/master) ## Prerequisites * Ensure that you have Python (3.9 or above) installed. * It is recommended to create and work in a virtual environment: ```sh # create & activate a virtual environment python -m venv venv/ source venv/bin/activate ``` ## Overview In this tutorial we will: 1. Deploy a local feature store with a **Parquet file offline store** and **Sqlite online store**. 2. Build a training dataset using our time series features from our **Parquet files**. 3. Ingest batch features ("materialization") and streaming features (via a Push API) into the online store. 4. Read the latest features from the offline store for batch scoring 5. Read the latest features from the online store for real-time inference. 6. Explore the (experimental) Feast UI ***Note*** - Feast provides a python SDK as well as an optional [hosted service](/master/reference/feature-servers/python-feature-server) for reading and writing feature data to the online and offline data stores. The latter might be useful when non-python languages are required. For this tutorial, we will be using the python SDK. In this tutorial, we'll use Feast to generate training data and power online model inference for a ride-sharing driver satisfaction prediction model. Feast solves several common issues in this flow: 1. **Training-serving skew and complex data joins:** Feature values often exist across multiple tables. Joining these datasets can be complicated, slow, and error-prone. * Feast joins these tables with battle-tested logic that ensures *point-in-time* correctness so future feature values do not leak to models. 2. **Online feature availability:** At inference time, models often need access to features that aren't readily available and need to be precomputed from other data sources. * Feast manages deployment to a variety of online stores (e.g. DynamoDB, Redis, Google Cloud Datastore) and ensures necessary features are consistently *available* and *freshly computed* at inference time. 3. **Feature and model versioning:** Different teams within an organization are often unable to reuse features across projects, resulting in duplicate feature creation logic. Models have data dependencies that need to be versioned, for example when running A/B tests on model versions. * Feast enables discovery of and collaboration on previously used features and enables versioning of sets of features (via *feature services*). * *(Experimental)* Feast enables light-weight feature transformations so users can re-use transformation logic across online / offline use cases and across models. ## Step 1: Install Feast Install the Feast SDK and CLI using pip: * In this tutorial, we focus on a local deployment. For a more in-depth guide on how to use Feast with Snowflake / GCP / AWS deployments, see [Running Feast with Snowflake/GCP/AWS](/master/how-to-guides/feast-snowflake-gcp-aws) {% tabs %} {% tab title="Bash" %} ```bash pip install feast ``` {% endtab %} {% endtabs %} ## Step 2: Create a feature repository Bootstrap a new feature repository using `feast init` from the command line. {% tabs %} {% tab title="Bash" %} ```bash feast init my_project cd my_project/feature_repo ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Output" %} ``` Creating a new Feast repository in /home/Jovyan/my_project. ``` {% endtab %} {% endtabs %} Let's take a look at the resulting demo repo itself. It breaks down into * `data/` contains raw demo parquet data * `feature_definitions.py` contains demo feature definitions * `feature_store.yaml` contains a demo setup configuring where data sources are * `test_workflow.py` showcases how to run all key Feast commands, including defining, retrieving, and pushing features. You can run this with `python test_workflow.py`. {% tabs %} {% tab title="feature\_store.yaml" %} ```yaml project: my_project # By default, the registry is a file (but can be turned into a more scalable SQL-backed registry) registry: data/registry.db # The provider primarily specifies default offline / online stores & storing the registry in a given cloud provider: local online_store: type: sqlite path: data/online_store.db entity_key_serialization_version: 3 ``` {% endtab %} {% tab title="feature\_definitions.py" %} ```python # This is an example feature definition file from datetime import timedelta import pandas as pd from feast import ( Entity, FeatureService, FeatureView, Field, FileSource, Project, PushSource, RequestSource, ) from feast.on_demand_feature_view import on_demand_feature_view from feast.types import Float32, Float64, Int64 # Define a project for the feature repo project = Project(name="my_project", description="A project for driver statistics") # Define an entity for the driver. You can think of an entity as a primary key used to # fetch features. driver = Entity(name="driver", join_keys=["driver_id"]) # Read data from parquet files. Parquet is convenient for local development mode. For # production, you can use your favorite DWH, such as BigQuery. See Feast documentation # for more info. driver_stats_source = FileSource( name="driver_hourly_stats_source", path="%PARQUET_PATH%", timestamp_field="event_timestamp", created_timestamp_column="created", ) # Our parquet files contain sample data that includes a driver_id column, timestamps and # three feature column. Here we define a Feature View that will allow us to serve this # data to our model online. driver_stats_fv = FeatureView( # The unique name of this feature view. Two feature views in a single # project cannot have the same name, and names must be unique across # all feature view types (regular, stream, on-demand) to avoid conflicts # during `feast apply`. name="driver_hourly_stats", entities=[driver], ttl=timedelta(days=1), # The list of features defined below act as a schema to both define features # for both materialization of features into a store, and are used as references # during retrieval for building a training dataset or serving features schema=[ Field(name="conv_rate", dtype=Float32), Field(name="acc_rate", dtype=Float32), Field(name="avg_daily_trips", dtype=Int64, description="Average daily trips"), ], online=True, source=driver_stats_source, # Tags are user defined key/value pairs that are attached to each # feature view tags={"team": "driver_performance"}, ) # Define a request data source which encodes features / information only # available at request time (e.g. part of the user initiated HTTP request) input_request = RequestSource( name="vals_to_add", schema=[ Field(name="val_to_add", dtype=Int64), Field(name="val_to_add_2", dtype=Int64), ], ) # Define an on demand feature view which can generate new features based on # existing feature views and RequestSource features @on_demand_feature_view( sources=[driver_stats_fv, input_request], schema=[ Field(name="conv_rate_plus_val1", dtype=Float64), Field(name="conv_rate_plus_val2", dtype=Float64), ], ) def transformed_conv_rate(inputs: pd.DataFrame) -> pd.DataFrame: df = pd.DataFrame() df["conv_rate_plus_val1"] = inputs["conv_rate"] + inputs["val_to_add"] df["conv_rate_plus_val2"] = inputs["conv_rate"] + inputs["val_to_add_2"] return df # This groups features into a model version driver_activity_v1 = FeatureService( name="driver_activity_v1", features=[ driver_stats_fv[["conv_rate"]], # Sub-selects a feature from a feature view transformed_conv_rate, # Selects all features from the feature view ], ) driver_activity_v2 = FeatureService( name="driver_activity_v2", features=[driver_stats_fv, transformed_conv_rate] ) # Defines a way to push data (to be available offline, online or both) into Feast. driver_stats_push_source = PushSource( name="driver_stats_push_source", batch_source=driver_stats_source, ) # Defines a slightly modified version of the feature view from above, where the source # has been changed to the push source. This allows fresh features to be directly pushed # to the online store for this feature view. driver_stats_fresh_fv = FeatureView( name="driver_hourly_stats_fresh", entities=[driver], ttl=timedelta(days=1), schema=[ Field(name="conv_rate", dtype=Float32), Field(name="acc_rate", dtype=Float32), Field(name="avg_daily_trips", dtype=Int64), ], online=True, source=driver_stats_push_source, # Changed from above tags={"team": "driver_performance"}, ) # Define an on demand feature view which can generate new features based on # existing feature views and RequestSource features @on_demand_feature_view( sources=[driver_stats_fresh_fv, input_request], # relies on fresh version of FV schema=[ Field(name="conv_rate_plus_val1", dtype=Float64), Field(name="conv_rate_plus_val2", dtype=Float64), ], ) def transformed_conv_rate_fresh(inputs: pd.DataFrame) -> pd.DataFrame: df = pd.DataFrame() df["conv_rate_plus_val1"] = inputs["conv_rate"] + inputs["val_to_add"] df["conv_rate_plus_val2"] = inputs["conv_rate"] + inputs["val_to_add_2"] return df driver_activity_v3 = FeatureService( name="driver_activity_v3", features=[driver_stats_fresh_fv, transformed_conv_rate_fresh], ) ``` {% endtab %} {% endtabs %} The `feature_store.yaml` file configures the key overall architecture of the feature store. The provider value sets default offline and online stores. * The offline store provides the compute layer to process historical data (for generating training data & feature values for serving). * The online store is a low latency store of the latest feature values (for powering real-time inference). Valid values for `provider` in `feature_store.yaml` are: * local: use a SQL registry or local file registry. By default, use a file / Dask based offline store + SQLite online store * gcp: use a SQL registry or GCS file registry. By default, use BigQuery (offline store) + Google Cloud Datastore (online store) * aws: use a SQL registry or S3 file registry. By default, use Redshift (offline store) + DynamoDB (online store) Note that there are many other offline / online stores Feast works with, including Spark, Azure, Hive, Trino, and PostgreSQL via community plugins. See [Third party integrations](/master/getting-started/third-party-integrations) for all supported data sources. A custom setup can also be made by following [Customizing Feast](/master/how-to-guides/customizing-feast). ### Inspecting the raw data The raw feature data we have in this demo is stored in a local parquet file. The dataset captures hourly stats of a driver in a ride-sharing app. ```python import pandas as pd pd.read_parquet("data/driver_stats.parquet") ``` ![Demo parquet data: data/driver\_stats.parquet](/files/-MhoI1np_5cAwx99AqKj) ## Step 3: Run sample workflow There's an included `test_workflow.py` file which runs through a full sample workflow: 1. Register feature definitions through `feast apply` 2. Generate a training dataset (using `get_historical_features`) 3. Generate features for batch scoring (using `get_historical_features`) 4. Ingest batch features into an online store (using `materialize_incremental`) 5. Fetch online features to power real time inference (using `get_online_features`) 6. Ingest streaming features into offline / online stores (using `push`) 7. Verify online features are updated / fresher We'll walk through some snippets of code below and explain ### Step 4: Register feature definitions and deploy your feature store The `apply` command scans python files in the current directory for feature view/entity definitions, registers the objects, and deploys infrastructure. In this example, it reads `feature_definitions.py` and sets up SQLite online store tables. Note that we had specified SQLite as the default online store by configuring `online_store` in `feature_store.yaml`. {% tabs %} {% tab title="Bash" %} ```bash feast apply ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Output" %} ``` Created entity driver Created feature view driver_hourly_stats Created feature view driver_hourly_stats_fresh Created on demand feature view transformed_conv_rate Created on demand feature view transformed_conv_rate_fresh Created feature service driver_activity_v3 Created feature service driver_activity_v1 Created feature service driver_activity_v2 Created sqlite table my_project_driver_hourly_stats_fresh Created sqlite table my_project_driver_hourly_stats ``` {% endtab %} {% endtabs %} ### Step 5: Generating training data or powering batch scoring models To train a model, we need features and labels. Often, this label data is stored separately (e.g. you have one table storing user survey results and another set of tables with feature values). Feast can help generate the features that map to these labels. Feast needs a list of **entities** (e.g. driver ids) and **timestamps**. Feast will intelligently join relevant tables to create the relevant feature vectors. There are two ways to generate this list: 1. The user can query that table of labels with timestamps and pass that into Feast as an *entity dataframe* for training data generation. 2. The user can also query that table with a *SQL query* which pulls entities. See the documentation on [feature retrieval](https://docs.feast.dev/getting-started/concepts/feature-retrieval) for details * Note that we include timestamps because we want the features for the same driver at various timestamps to be used in a model. #### Generating training data {% tabs %} {% tab title="Python" %} ```python from datetime import datetime import pandas as pd from feast import FeatureStore # Note: see https://docs.feast.dev/getting-started/concepts/feature-retrieval for # more details on how to retrieve for all entities in the offline store instead entity_df = pd.DataFrame.from_dict( { # entity's join key -> entity values "driver_id": [1001, 1002, 1003], # "event_timestamp" (reserved key) -> timestamps # Each timestamp acts as the upper bound for the point-in-time join: # Feast retrieves the latest feature values at or before this time, # preventing data leakage from future events. "event_timestamp": [ datetime(2021, 4, 12, 10, 59, 42), datetime(2021, 4, 12, 8, 12, 10), datetime(2021, 4, 12, 16, 40, 26), ], # (optional) label name -> label values. Feast does not process these "label_driver_reported_satisfaction": [1, 5, 3], # values we're using for an on-demand transformation "val_to_add": [1, 2, 3], "val_to_add_2": [10, 20, 30], } ) store = FeatureStore(repo_path=".") training_df = store.get_historical_features( entity_df=entity_df, features=[ "driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate", "driver_hourly_stats:avg_daily_trips", "transformed_conv_rate:conv_rate_plus_val1", "transformed_conv_rate:conv_rate_plus_val2", ], ).to_df() print("----- Feature schema -----\n") print(training_df.info()) print() print("----- Example features -----\n") print(training_df.head()) ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Output" %} ```bash ----- Feature schema ----- RangeIndex: 3 entries, 0 to 2 Data columns (total 10 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 driver_id 3 non-null int64 1 event_timestamp 3 non-null datetime64[ns, UTC] 2 label_driver_reported_satisfaction 3 non-null int64 3 val_to_add 3 non-null int64 4 val_to_add_2 3 non-null int64 5 conv_rate 3 non-null float32 6 acc_rate 3 non-null float32 7 avg_daily_trips 3 non-null int32 8 conv_rate_plus_val1 3 non-null float64 9 conv_rate_plus_val2 3 non-null float64 dtypes: datetime64[ns, UTC](1), float32(2), float64(2), int32(1), int64(4) memory usage: 336.0 bytes None ----- Example features ----- driver_id event_timestamp label_driver_reported_satisfaction \ 0 1001 2021-04-12 10:59:42+00:00 1 1 1002 2021-04-12 08:12:10+00:00 5 2 1003 2021-04-12 16:40:26+00:00 3 val_to_add val_to_add_2 conv_rate acc_rate avg_daily_trips \ 0 1 10 0.800648 0.265174 643 1 2 20 0.644141 0.996602 765 2 3 30 0.855432 0.546345 954 conv_rate_plus_val1 conv_rate_plus_val2 0 1.800648 10.800648 1 2.644141 20.644141 2 3.855432 30.855432 ``` {% endtab %} {% endtabs %} #### Run offline inference (batch scoring) To power a batch model, we primarily need to generate features with the `get_historical_features` call, but using the current timestamp {% tabs %} {% tab title="Python" %} ```python entity_df["event_timestamp"] = pd.to_datetime("now", utc=True) training_df = store.get_historical_features( entity_df=entity_df, features=[ "driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate", "driver_hourly_stats:avg_daily_trips", "transformed_conv_rate:conv_rate_plus_val1", "transformed_conv_rate:conv_rate_plus_val2", ], ).to_df() print("\n----- Example features -----\n") print(training_df.head()) ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Output" %} ``` ----- Example features ----- driver_id event_timestamp \ 0 1001 2024-04-19 14:58:16.452895+00:00 1 1002 2024-04-19 14:58:16.452895+00:00 2 1003 2024-04-19 14:58:16.452895+00:00 label_driver_reported_satisfaction val_to_add val_to_add_2 conv_rate \ 0 1 1 10 0.535773 1 5 2 20 0.171976 2 3 3 30 0.275669 acc_rate avg_daily_trips conv_rate_plus_val1 conv_rate_plus_val2 0 0.689705 428 1.535773 10.535773 1 0.737113 369 2.171976 20.171976 2 0.156630 116 3.275669 30.275669 ``` {% endtab %} {% endtabs %} ### Step 6: Ingest batch features into your online store We now serialize the latest values of features since the beginning of time to prepare for serving. Note, `materialize_incremental` serializes all new features since the last `materialize` call, or since the time provided minus the `ttl` timedelta. In this case, this will be `CURRENT_TIME - 1 day` (`ttl` was set on the `FeatureView` instances in `feature_definitions.py`). {% tabs %} {% tab title="Bash (with timestamp)" %} ```bash CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S") feast materialize-incremental $CURRENT_TIME ``` {% endtab %} {% tab title="Bash (simple)" %} ```bash # Alternative: Materialize all data using current timestamp (for data without event timestamps) feast materialize --disable-event-timestamp ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Output" %} ```bash Materializing 2 feature views to 2024-04-19 10:59:58-04:00 into the sqlite online store. driver_hourly_stats from 2024-04-18 15:00:46-04:00 to 2024-04-19 10:59:58-04:00: 100%|████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 370.32it/s] driver_hourly_stats_fresh from 2024-04-18 15:00:46-04:00 to 2024-04-19 10:59:58-04:00: 100%|███████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1046.64it/s] Materializing 2 feature views to 2024-04-19 10:59:58-04:00 into the sqlite online store. ``` {% endtab %} {% endtabs %} ### Step 7: Fetching feature vectors for inference At inference time, we need to quickly read the latest feature values for different drivers (which otherwise might have existed only in batch sources) from the online feature store using `get_online_features()`. These feature vectors can then be fed to the model. {% tabs %} {% tab title="Python" %} ```python from pprint import pprint from feast import FeatureStore store = FeatureStore(repo_path=".") feature_vector = store.get_online_features( features=[ "driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate", "driver_hourly_stats:avg_daily_trips", ], entity_rows=[ # {join_key: entity_value} {"driver_id": 1004}, {"driver_id": 1005}, ], ).to_dict() pprint(feature_vector) ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Output" %} ```bash { 'acc_rate': [0.25351759791374207, 0.8949751853942871], 'avg_daily_trips': [712, 791], 'conv_rate': [0.5038306713104248, 0.9839504361152649], 'driver_id': [1004, 1005] } ``` {% endtab %} {% endtabs %} ### Step 8: Using a feature service to fetch online features instead. You can also use feature services to manage multiple features, and decouple feature view definitions and the features needed by end applications. The feature store can also be used to fetch either online or historical features using the same API below. More information can be found [here](https://docs.feast.dev/getting-started/concepts/feature-retrieval). The `driver_activity_v1` feature service pulls all features from the `driver_hourly_stats` feature view: ```python from feast import FeatureService driver_stats_fs = FeatureService( name="driver_activity_v1", features=[driver_stats_fv] ) ``` {% tabs %} {% tab title="Python" %} ```python from pprint import pprint from feast import FeatureStore feature_store = FeatureStore('.') # Initialize the feature store feature_service = feature_store.get_feature_service("driver_activity_v1") feature_vector = feature_store.get_online_features( features=feature_service, entity_rows=[ # {join_key: entity_value} {"driver_id": 1004}, {"driver_id": 1005}, ], ).to_dict() pprint(feature_vector) ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Output" %} ```bash { 'acc_rate': [0.5732735991477966, 0.7828438878059387], 'avg_daily_trips': [33, 984], 'conv_rate': [0.15498852729797363, 0.6263588070869446], 'driver_id': [1004, 1005] } ``` {% endtab %} {% endtabs %} ## Step 9: Browse your features with the Web UI (experimental) View all registered features, data sources, entities, and feature services with the Web UI. One of the ways to view this is with the `feast ui` command. {% tabs %} {% tab title="Bash" %} ```bash feast ui ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Output" %} ```bash INFO: Started server process [66664] 08/17/2022 01:25:49 PM uvicorn.error INFO: Started server process [66664] INFO: Waiting for application startup. 08/17/2022 01:25:49 PM uvicorn.error INFO: Waiting for application startup. INFO: Application startup complete. 08/17/2022 01:25:49 PM uvicorn.error INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8888 (Press CTRL+C to quit) 08/17/2022 01:25:49 PM uvicorn.error INFO: Uvicorn running on http://0.0.0.0:8888 (Press CTRL+C to quit) ``` {% endtab %} {% endtabs %} ![](/files/qbp2vaWC0vrhBwJBfoQP) ## Step 10: Re-examine `test_workflow.py` Take a look at `test_workflow.py` again. It showcases many sample flows on how to interact with Feast. You'll see these show up in the upcoming concepts + architecture + tutorial pages as well. ## Next steps * Run `feast demo-notebooks` to generate tailored Jupyter notebooks for your project. See [Demo Notebooks](/master/tutorials/demo-notebooks). * Read the [Concepts](/master/getting-started/concepts) page to understand the Feast data model. * Read the [Architecture](/master/getting-started/architecture) page. * Check out our [Tutorials](/master/tutorials/tutorials-overview) section for more examples on how to use Feast. * Follow our [Running Feast with Snowflake/GCP/AWS](/master/how-to-guides/feast-snowflake-gcp-aws) guide for a more in-depth tutorial on using Feast. # GenAI ## Overview Feast provides robust support for Generative AI applications, enabling teams to build, deploy, and manage feature infrastructure for Large Language Models (LLMs) and other Generative AI (GenAI) applications. With Feast's vector database integrations and feature management capabilities, teams can implement production-ready Retrieval Augmented Generation (RAG) systems and other GenAI applications with the same reliability and operational excellence as traditional ML systems. ## Key Capabilities for GenAI ### Vector Database Support Feast integrates with popular vector databases to store and retrieve embedding vectors efficiently: * **Milvus**: Full support for vector similarity search with the `retrieve_online_documents_v2` method * **SQLite**: Local vector storage and retrieval for development and testing * **Elasticsearch**: Scalable vector search capabilities * **Postgres with PGVector**: SQL-based vector operations * **Qdrant**: Purpose-built vector database integration * **ScyllaDB**: Native `vector` type with HNSW ANN index, full `retrieve_online_documents_v2` support These integrations allow you to: * Store embeddings as features * Perform vector similarity search to find relevant context * Retrieve both vector embeddings and traditional features in a single API call ### Retrieval Augmented Generation (RAG) Feast simplifies building RAG applications by providing: 1. **Embedding storage**: Store and version embeddings alongside your other features 2. **Vector similarity search**: Find the most relevant data/documents for a given query 3. **Feature retrieval**: Combine embeddings with structured features for richer context 4. **Versioning and governance**: Track changes to your document repository over time The typical RAG workflow with Feast involves: ``` ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ Document │ │ Document │ │ Feast │ │ LLM │ │ Processing │────▶│ Embedding │────▶│ Feature │────▶│ Context │ │ │ │ │ │ Store │ │ Generation │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ ``` ### Transforming Unstructured Data to Structured Data Feast provides powerful capabilities for transforming unstructured data (like PDFs, text documents, and images) into structured embeddings that can be used for RAG applications: * **Document Processing Pipelines**: Integrate with document processing tools like Docling to extract text from PDFs and other document formats * **Chunking and Embedding Generation**: Process documents into smaller chunks and generate embeddings using models like Sentence Transformers * **On-Demand Transformations**: Use `@on_demand_feature_view` decorator to transform raw documents into embeddings in real-time * **Batch Processing with Spark**: Scale document processing for large datasets using Spark integration The transformation workflow typically involves: 1. **Raw Data Ingestion**: Load documents or other data from various sources (file systems, databases, etc.) 2. **Text Extraction**: Extract text content from unstructured documents 3. **Chunking**: Split documents into smaller, semantically meaningful chunks 4. **Embedding Generation**: Convert text chunks into vector embeddings 5. **Storage**: Store embeddings and metadata in Feast's feature store ### DocEmbedder: End-to-End Document Ingestion Pipeline The `DocEmbedder` class provides an end-to-end pipeline for ingesting documents into Feast's online vector store. It handles chunking, embedding generation, and writing results -- all in a single step. #### Key Components * **`DocEmbedder`**: High-level orchestrator that runs the full pipeline: chunk → embed → schema transform → write to online store * **`BaseChunker` / `TextChunker`**: Pluggable chunking layer. `TextChunker` splits text by word count with configurable `chunk_size`, `chunk_overlap`, `min_chunk_size`, and `max_chunk_chars` * **`BaseEmbedder` / `MultiModalEmbedder`**: Pluggable embedding layer with modality routing. `MultiModalEmbedder` supports text (via sentence-transformers) and image (via CLIP) with lazy model loading * **`SchemaTransformFn`**: A user-defined function that transforms the chunked + embedded DataFrame into the format expected by the FeatureView schema #### Quick Example ```python from feast import DocEmbedder import pandas as pd # Prepare your documents df = pd.DataFrame({ "id": ["doc1", "doc2"], "text": ["First document content...", "Second document content..."], }) # Create DocEmbedder -- automatically generates a FeatureView and applies the repo embedder = DocEmbedder( repo_path="feature_repo/", feature_view_name="text_feature_view", ) # Embed and ingest documents in one step result = embedder.embed_documents( documents=df, id_column="id", source_column="text", column_mapping=("text", "text_embedding"), ) ``` #### Features * **Auto-generates FeatureView**: Creates a Python file with Entity and FeatureView definitions compatible with `feast apply` * **Auto-applies repo**: Registers the generated FeatureView in the registry automatically * **Custom schema transform**: Provide your own `SchemaTransformFn` to control how chunked + embedded data maps to your FeatureView schema * **Extensible**: Subclass `BaseChunker` or `BaseEmbedder` to plug in your own chunking or embedding strategies For a complete walkthrough, see the [DocEmbedder tutorial notebook](https://github.com/feast-dev/feast/tree/master/examples/rag-retriever/rag_feast_docembedder.ipynb). ### Feature Transformation for LLMs Feast supports transformations that can be used to: * Process raw text into embeddings * Chunk documents for more effective retrieval * Normalize and preprocess features before serving to LLMs * Apply custom transformations to adapt features for specific LLM requirements ## Use Cases ### Document Question-Answering Build document Q\&A systems by: 1. Storing document chunks and their embeddings in Feast 2. Converting user questions to embeddings 3. Retrieving relevant document chunks 4. Providing these chunks as context to an LLM ### Knowledge Base Augmentation Enhance your LLM's knowledge by: 1. Storing company-specific information as embeddings 2. Retrieving relevant information based on user queries 3. Injecting this information into the LLM's context ### Semantic Search Implement semantic search by: 1. Storing document embeddings in Feast 2. Converting search queries to embeddings 3. Finding semantically similar documents using vector search ### AI Agents with Context and Memory Feast can serve as both the **context provider** and **persistent memory layer** for AI agents. Unlike stateless RAG pipelines, agents make autonomous decisions about which tools to call and can write state back to the feature store: 1. **Structured context**: Retrieve customer profiles, account data, and other entity-keyed features 2. **Knowledge retrieval**: Search vector embeddings for relevant documents 3. **Persistent memory**: Store and recall per-entity interaction history (last topic, resolution, preferences) using `write_to_online_store` 4. **Governed access**: All reads and writes are subject to the same RBAC, TTL, and audit policies as any other feature With MCP enabled, agents built with any framework (LangChain, LlamaIndex, CrewAI, AutoGen, or custom) can discover and call Feast tools dynamically. See the [Feast-Powered AI Agent example](/master/tutorials/agent_feature_store) and the blog post [Building AI Agents with Feast](https://feast.dev/blog/feast-agents-mcp/) for a complete walkthrough. ### Scaling with Spark Integration Feast integrates with Apache Spark to enable large-scale processing of unstructured data for GenAI applications: * **Spark Data Source**: Load data from Spark tables, files, or SQL queries for feature generation * **Spark Offline Store**: Process large document collections and generate embeddings at scale * **Spark Batch Materialization**: Efficiently materialize features from offline to online stores * **Distributed Processing**: Handle gigabytes of documents and millions of embeddings This integration enables: * Processing large document collections in parallel * Generating embeddings for millions of text chunks * Efficiently materializing features to vector databases * Scaling RAG applications to enterprise-level document repositories ### Scaling with Ray Integration Feast integrates with Ray to enable distributed processing for RAG applications: * **Ray Compute Engine**: Distributed feature computation using Ray's task and actor model * **Ray Offline Store**: Process large document collections and generate embeddings at scale * **Ray Batch Materialization**: Efficiently materialize features from offline to online stores * **Distributed Embedding Generation**: Scale embedding generation across multiple nodes This integration enables: * Distributed processing of large document collections * Parallel embedding generation for millions of text chunks * Kubernetes-native scaling for RAG applications * Efficient resource utilization across multiple nodes * Production-ready distributed RAG pipelines For detailed information on building distributed RAG applications with Feast and Ray, see [Feast + Ray: Distributed Processing for RAG Applications](https://feast.dev/blog/feast-ray-distributed-processing/). ## Model Context Protocol (MCP) Support Feast supports the Model Context Protocol (MCP), which enables AI agents and applications to interact with your feature store through standardized MCP interfaces. This allows seamless integration with LLMs and AI agents for GenAI applications. ### Key Benefits of MCP Support * **Standardized AI Integration**: Enable AI agents to discover and use features dynamically without hardcoded definitions * **Easy Setup**: Add MCP support with a simple configuration change and `pip install feast[mcp]` * **Agent-Friendly APIs**: Expose feature store capabilities through MCP tools that AI agents can understand and use * **Production Ready**: Built on top of Feast's proven feature serving infrastructure ### Getting Started with MCP 1. **Install MCP support**: ```bash pip install feast[mcp] ``` 2. **Configure your feature store** to use MCP: ```yaml feature_server: type: mcp enabled: true mcp_enabled: true mcp_transport: http mcp_server_name: "feast-feature-store" mcp_server_version: "1.0.0" ``` By default, Feast uses the SSE-based MCP transport (`mcp_transport: sse`). Streamable HTTP (`mcp_transport: http`) is recommended for improved compatibility with some MCP clients. ### How It Works The MCP integration uses the `fastapi_mcp` library to automatically transform your Feast feature server's FastAPI endpoints into MCP-compatible tools. When you enable MCP support: 1. **Automatic Discovery**: The integration scans your FastAPI application and discovers all available endpoints 2. **Tool Generation**: Each endpoint becomes an MCP tool with auto-generated schemas and descriptions 3. **Dynamic Access**: AI agents can discover and call these tools dynamically without hardcoded definitions 4. **Standard Protocol**: Uses the Model Context Protocol for standardized AI-to-API communication ### Available MCP Tools The fastapi\_mcp integration automatically exposes your Feast feature server's FastAPI endpoints as MCP tools. This means AI assistants can: * **Call `/get-online-features`** to retrieve features from the feature store * **Call `/search`** to perform vector similarity search (`/retrieve-online-documents` is a deprecated alias) * **Call `/v1/vector_stores/{feature_view}/search`** for OpenAI-compatible text search with server-side embedding * **Call `/write-to-online-store`** to persist agent state (memory, notes, interaction history) * **Use `/health`** to check server status For a basic MCP example, see the [MCP Feature Store Example](/master/tutorials/mcp_feature_store). For a full agent with persistent memory, see the [Feast-Powered AI Agent Example](/master/tutorials/agent_feature_store). ## Learn More For more detailed information and examples: * [Vector Database Reference](/master/reference/alpha-vector-database) * [RAG Tutorial with Docling](/master/tutorials/rag-with-docling) * [DocEmbedder Tutorial Notebook](https://github.com/feast-dev/feast/tree/master/examples/rag-retriever/rag_feast_docembedder.ipynb) * [RAG Fine Tuning with Feast and Milvus](/master/tutorials/rag-retriever) * [Milvus Quickstart Example](https://github.com/feast-dev/feast/tree/master/examples/rag/milvus-quickstart.ipynb) * [Feast + Ray: Distributed Processing for RAG Applications](https://feast.dev/blog/feast-ray-distributed-processing/) * [MCP Feature Store Example](/master/tutorials/mcp_feature_store) * [Feast-Powered AI Agent Example (with Memory)](/master/tutorials/agent_feature_store) * [Blog: Building AI Agents with Feast](https://feast.dev/blog/feast-agents-mcp/) * [MCP Feature Server Reference](/master/reference/feature-servers/mcp-feature-server) * [Spark Data Source](/master/reference/data-sources/spark) * [Spark Offline Store](/master/reference/offline-stores/spark) * [Spark Compute Engine](/master/reference/compute-engine/spark) # Architecture {% content-ref url="/pages/S6hBrzLOV5frNoeEwunq" %} [Overview](/master/getting-started/architecture/overview) {% endcontent-ref %} {% content-ref url="/pages/0I8MFKtAu57RGqxeDwH5" %} [Language](/master/getting-started/architecture/language) {% endcontent-ref %} {% content-ref url="/pages/0apigNRqqKgqBDuGmVaf" %} [Push vs Pull Model](/master/getting-started/architecture/push-vs-pull-model) {% endcontent-ref %} {% content-ref url="/pages/5uuGXMzduM5MjQ40R19Z" %} [Write Patterns](/master/getting-started/architecture/write-patterns) {% endcontent-ref %} {% content-ref url="/pages/K4M7pMPa6Q4ownl8vZyi" %} [Feature Transformation](/master/getting-started/architecture/feature-transformation) {% endcontent-ref %} {% content-ref url="/pages/FsTvnawwmPh5WYtP0mBZ" %} [Feature Serving and Model Inference](/master/getting-started/architecture/model-inference) {% endcontent-ref %} {% content-ref url="/pages/LkqCqP066wfr3BCVO9v5" %} [Role-Based Access Control (RBAC)](/master/getting-started/architecture/rbac) {% endcontent-ref %} # Overview ![Feast Architecture Diagram](/files/pz51PMnsQviEBGWU7rW1) Feast's architecture is designed to be flexible and scalable. It is composed of several components that work together to provide a feature store that can be used to serve features for training and inference. * Feast uses a [Push Model](/master/getting-started/architecture/push-vs-pull-model) to ingest data from different sources and store feature values in the online store. This allows Feast to serve features in real-time with low latency. * Feast supports [feature transformation](/master/getting-started/architecture/feature-transformation) for On Demand and Streaming data sources and will support Batch transformations in the future. For Streaming and Batch data sources, Feast requires a separate [Feature Transformation Engine](/master/getting-started/architecture/feature-transformation#feature-transformation-engines) (in the batch case, this is typically your Offline Store). We are exploring adding a default streaming engine to Feast. * Domain expertise is recommended when integrating a data source with Feast understand the [tradeoffs from different write patterns](/master/getting-started/architecture/write-patterns) to your application * We recommend [using Python](/master/getting-started/architecture/language) for your Feature Store microservice. As mentioned in the document, precomputing features is the recommended optimal path to ensure low latency performance. Reducing feature serving to a lightweight database lookup is the ideal pattern, which means the marginal overhead of Python should be tolerable. Because of this we believe the pros of Python outweigh the costs, as reimplementing feature logic is undesirable. Java and Go Clients are also available for online feature retrieval. * [Role-Based Access Control (RBAC)](/master/getting-started/architecture/rbac) is a security mechanism that restricts access to resources based on the roles/groups/namespaces of individual users within an organization. In the context of the Feast, RBAC ensures that only authorized users or groups can access or modify specific resources, thereby maintaining data security and operational integrity. # Language Use Python to serve your features. ## Why should you use Python to Serve features for Machine Learning? Python has emerged as the primary language for machine learning, and this extends to feature serving and there are five main reasons Feast recommends using a microservice written in Python. ## 1. Python is the language of Machine Learning You should meet your users where they are. Python’s popularity in the machine learning community is undeniable. Its simplicity and readability make it an ideal language for writing and understanding complex algorithms. Python boasts a rich ecosystem of libraries such as TensorFlow, PyTorch, XGBoost, and scikit-learn, which provide robust support for developing and deploying machine learning models and we want Feast in this ecosystem. ## 2. Precomputation is The Way Precomputing features is the recommended optimal path to ensure low latency performance. Reducing feature serving to a lightweight database lookup is the ideal pattern, which means the marginal overhead of Python should be tolerable. Precomputation ensures product experiences for downstream services are also fast. Slow user experiences are bad user experiences. Precompute and persist data as much as you can. ## 3. Serving features in another language can lead to skew Ensuring that features used during model training (offline serving) and online serving are available in production to make real-time predictions is critical. When features are initially developed, they are typically written in Python. This is due to the convenience and efficiency provided by Python's data manipulation libraries. However, in a production environment, there is often interest or pressure to rewrite these features in a different language, like Java, Go, or C++, for performance reasons. This reimplementation introduces a significant risk: training and serving skew. Note that there will always be some minor exceptions (e.g., any *Time Since Last Event* types of features) but this should not be the rule. Training and serving skew occurs when there are discrepancies between the features used during model training and those used during prediction. This can lead to degraded model performance, unreliable predictions, and reduced velocity in releasing new features and new models. The process of rewriting features in another language is prone to errors and inconsistencies, which exacerbate this issue. ## 4. Reimplementation is Excessive Rewriting features in another language is not only risky but also resource-intensive. It requires significant time and effort from engineers to ensure that the features are correctly translated. This process can introduce bugs and inconsistencies, further increasing the risk of training and serving skew. Additionally, maintaining two versions of the same feature codebase adds unnecessary complexity and overhead. More importantly, the opportunity cost of this work is high and requires twice the amount of resourcing. Reimplementing code should only be done when the performance gains are worth the investment. Features should largely be precomputed so the latency performance gains should not be the highest impact work that your team can accomplish. ## 5. Use existing Python Optimizations Rather than switching languages, it is more efficient to optimize the performance of your feature store while keeping Python as the primary language. Optimization is a two step process. ### Step 1: Quantify latency bottlenecks in your feature calculations Use tools like [CProfile](https://docs.python.org/3/library/profile.html) to understand latency bottlenecks in your code. This will help you prioritize the biggest inefficiencies first. When we initially launched Python native transformations in Python, [profiling the code](https://github.com/feast-dev/feast/issues/4207#issuecomment-2155754504) helped us identify that Pandas resulted in a 10x overhead due to type conversion. ### Step 2: Optimize your feature calculations As mentioned, precomputation is the recommended path. In some cases, you may want fully synchronous writes from your data producer to your online feature store, in which case you will want your feature computations and writes to be very fast. In this case, we recommend optimizing the feature calculation code first. You should optimize your code using libraries, tools, and caching. For example, identify whether your feature calculations can be optimized through vectorized calculations in NumPy; explore tools like Numba for faster execution; and cache frequently accessed data using tools like an lru\_cache. Lastly, Feast will continue to optimize serving in Python and making the overall infrastructure more performant. This will better serve the community. So we recommend focusing on optimizing your feature-specific code, reporting latency bottlenecks to the maintainers, and contributing to help the infrastructure be more performant. By keeping features in Python and optimizing performance, you can ensure consistency between training and serving, reduce the risk of errors, and focus on launching more product experiences for your customers. Embrace Python for feature serving, and leverage its strengths to build robust and reliable machine learning systems. # Push vs Pull Model ## Push vs Pull Model Feast uses a [Push Model](https://en.wikipedia.org/wiki/Push_technology), i.e., Data Producers push data to the feature store and Feast stores the feature values in the online store, to serve features in real-time. In a [Pull Model](https://en.wikipedia.org/wiki/Pull_technology), Feast would pull data from the data producers at request time and store the feature values in the online store before serving them (storing them would actually be unnecessary). This approach would incur additional network latency as Feast would need to orchestrate a request to each data producer, which would mean the latency would be at least as long as your slowest call. So, in order to serve features as fast as possible, we push data to Feast and store the feature values in the online store. The trade-off with the Push Model is that strong consistency is not guaranteed out of the box. Instead, strong consistency has to be explicitly designed for in orchestrating the updates to Feast and the client usage. The significant advantage with this approach is that Feast is read-optimized for low-latency feature retrieval. ## How to Push Implicit in the Push model are decisions about *how* and *when* to push feature values to the online store. From a developer's perspective, there are three ways to push feature values to the online store with different tradeoffs. They are discussed further in the [Write Patterns](/master/getting-started/architecture/write-patterns) section. # Write Patterns Feast uses a [Push Model](/master/getting-started/architecture/push-vs-pull-model) to push features to the online store. This has two important consequences: (1) communication patterns between the Data Producer (i.e., the client) and Feast (i.e,. the server) and (2) feature computation and *feature value* write patterns to Feast's online store. Data Producers (i.e., services that generate data) send data to Feast so that Feast can write feature values to the online store. That data can be either raw data where Feast computes and stores the feature values or precomputed feature values. ## Communication Patterns There are two ways a client (or Data Producer) can *send* data to the online store: 1. Synchronously * Using a synchronous API call for a small number of entities or a single entity (e.g., using the [`push` or `write_to_online_store` methods](/master/reference/data-sources/push#pushing-data)) or the Feature Server's [`push` endpoint](/master/reference/feature-servers/python-feature-server#pushing-features-to-the-online-and-offline-stores)) 2. Asynchronously * Using an asynchronous API call for a small number of entities or a single entity (e.g., using the [`push` or `write_to_online_store` methods](/master/reference/data-sources/push#pushing-data)) or the Feature Server's [`push` endpoint](/master/reference/feature-servers/python-feature-server#pushing-features-to-the-online-and-offline-stores)) * Using a "batch job" for a large number of entities (e.g., using a [compute engine](/master/getting-started/components/compute-engine)) Note, in some contexts, developers may "batch" a group of entities together and write them to the online store in a single API call. This is a common pattern when writing data to the online store to reduce write loads but we would not qualify this as a batch job. ## Feature Value Write Patterns Writing feature values to the online store (i.e., the server) can be done in two ways: Precomputing the transformations client-side or Computing the transformations On Demand server-side. ### Combining Approaches In some scenarios, a combination of Precomputed and On Demand transformations may be optimal. When selecting feature value write patterns, one must consider the specific requirements of your application, the acceptable correctness of the data, the latency tolerance, and the computational resources available. Making deliberate choices can help the performance and reliability of your service. There are two ways the client can write *feature values* to the online store: 1. Precomputing transformations 2. Computing transformations On Demand 3. Hybrid (Precomputed + On Demand) ### 1. Precomputing Transformations Precomputed transformations can happen outside of Feast (e.g., via some batch job or streaming application) or inside of the Feast feature server when writing to the online store via the `push` or `write-to-online-store` api. ### 2. Computing Transformations On Demand On Demand transformations can only happen inside of Feast at either (1) the time of the client's request or (2) when the data producer writes to the online store. With the `transform_on_write` parameter, you can control whether transformations are applied during write operations, allowing you to skip transformations for pre-processed data while still enabling transformations during API calls. ### 3. Hybrid (Precomputed + On Demand) The hybrid approach allows for precomputed transformations to happen inside or outside of Feast and have the On Demand transformations happen at client request time. This is particularly convenient for "Time Since Last" types of features (e.g., time since purchase). ## Tradeoffs When deciding between synchronous and asynchronous data writes, several tradeoffs should be considered: * **Data Consistency**: Asynchronous writes allow Data Producers to send data without waiting for the write operation to complete, which can lead to situations where the data in the online store is stale. This might be acceptable in scenarios where absolute freshness is not critical. However, for critical operations, such as calculating loan amounts in financial applications, stale data can lead to incorrect decisions, making synchronous writes essential. * **Correctness**: The risk of data being out-of-date must be weighed against the operational requirements. For instance, in a lending application, having up-to-date feature data can be crucial for correctness (depending upon the features and raw data), thus favoring synchronous writes. In less sensitive contexts, the eventual consistency offered by asynchronous writes might be sufficient. * **Service Coupling**: Synchronous writes result in tighter coupling between services. If a write operation fails, it can cause the dependent service operation to fail as well, which might be a significant drawback in systems requiring high reliability and independence between services. * **Application Latency**: Asynchronous writes typically reduce the perceived latency from the client's perspective because the client does not wait for the write operation to complete. This can enhance the user experience and efficiency in environments where operations are not critically dependent on immediate data freshness. The table below can help guide the most appropriate data write and feature computation strategies based on specific application needs and data sensitivity. | Data Write Type | Feature Computation | Scenario | Recommended Approach | | --------------- | -------------------------------- | ------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | Asynchronous | On Demand | Data-intensive applications tolerant to staleness | Opt for asynchronous writes with on-demand computation to balance load and manage resource usage efficiently. | | Asynchronous | Precomputed | High volume, non-critical data processing | Use asynchronous batch jobs with precomputed transformations for efficiency and scalability. | | Synchronous | On Demand | High-stakes decision making | Use synchronous writes with on-demand feature computation to ensure data freshness and correctness. | | Synchronous | Precomputed | User-facing applications requiring quick feedback | Use synchronous writes with precomputed features to reduce latency and improve user experience. | | Synchronous | Hybrid (Precomputed + On Demand) | High-stakes decision making that want to optimize for latency under constraints | Use synchronous writes with precomputed features where possible and a select set of on demand computations to reduce latency and improve user experience. | # Feature Transformation A *feature transformation* is a function that takes some set of input data and returns some set of output data. Feature transformations can happen on either raw data or derived data. ## Feature Transformation Engines Feature transformations can be executed by three types of "transformation engines": 1. The Feast Feature Server 2. An Offline Store (e.g., Snowflake, BigQuery, DuckDB, Spark, etc.) 3. [A Compute Engine](/master/reference/compute-engine) The three transformation engines are coupled with the [communication pattern used for writes](/master/getting-started/architecture/write-patterns). Importantly, this implies that different feature transformation code may be used under different transformation engines, so understanding the tradeoffs of when to use which transformation engine/communication pattern is extremely critical to the success of your implementation. In general, we recommend transformation engines and network calls to be chosen by aligning it with what is most appropriate for the data producer, feature/model usage, and overall product. ## API ### feature\_transformation `feature_transformation` or `udf` are the core APIs for defining feature transformations in Feast. They allow you to specify custom logic that can be applied to the data during materialization or retrieval. Examples include: ```python def remove_extra_spaces(df: DataFrame) -> DataFrame: df['name'] = df['name'].str.replace('\s+', ' ') return df spark_transformation = SparkTransformation( mode=TransformationMode.SPARK, udf=remove_extra_spaces, udf_string="remove extra spaces", ) feature_view = FeatureView( feature_transformation=spark_transformation, ... ) ``` OR ```python spark_transformation = Transformation( mode=TransformationMode.SPARK_SQL, udf=remove_extra_spaces_sql, udf_string="remove extra spaces sql", ) feature_view = FeatureView( feature_transformation=spark_transformation, ... ) ``` OR ```python @transformation(mode=TransformationMode.SPARK) def remove_extra_spaces_udf(df: pd.DataFrame) -> pd.DataFrame: return df.assign(name=df['name'].str.replace('\s+', ' ')) feature_view = FeatureView( feature_transformation=remove_extra_spaces_udf, ... ) ``` ### Aggregation Aggregation is builtin API for defining batch or streamable aggregations on data. It allows you to specify how to aggregate data over a time window, such as calculating the average or sum of a feature over a specified period. Examples include: ```python from feast import Aggregation feature_view = FeatureView( aggregations=[ Aggregation( column="amount", function="sum" ) Aggregation( column="amount", function="avg", time_window="1h" ), ] ... ) ``` ### Filter ttl: They amount of time that the features will be available for materialization or retrieval. The entity rows' timestamp higher that the current time minus the ttl will be used to filter the features. This is useful for ensuring that only recent data is used in feature calculations. Examples include: ```python feature_view = FeatureView( ttl="1d", # Features will be available for 1 day ... ) ``` ### Join Feast can join multiple feature views together to create a composite feature view. This allows you to combine features from different sources or views into a single view. Examples include: ```python feature_view = FeatureView( name="composite_feature_view", entities=["entity_id"], source=[ FeatureView( name="feature_view_1", features=["feature_1", "feature_2"], ... ), FeatureView( name="feature_view_2", features=["feature_3", "feature_4"], ... ) ] ... ) ``` The underlying implementation of the join is an inner join by default, and join key is the entity id. # Feature Serving and Model Inference ![](/files/22mvBDCTtoPJNOtiSX6X) {% hint style="info" %} **Note:** this ML Infrastructure diagram highlights an orchestration pattern that is driven by a client application. This is not the only approach that can be taken and different patterns will result in different trade-offs. {% endhint %} Production machine learning systems can choose from four approaches to serving machine learning predictions (the output of model inference): 1. Online model inference with online features 2. Offline mode inference without online features 3. Online model inference with online features and cached predictions 4. Online model inference without features *Note: online features can be sourced from batch, streaming, or request data sources.* These four approaches have different tradeoffs but, in general, have significant implementation differences. ## 1. Online Model Inference with Online Features Online model inference with online features is a powerful approach to serving data-driven machine learning applications. This requires a feature store to serve online features and a model server to serve model predictions (e.g., KServe). This approach is particularly useful for applications where request-time data is required to run inference. ```python features = store.get_online_features( feature_refs=[ "user_data:click_through_rate", "user_data:number_of_clicks", "user_data:average_page_duration", ], entity_rows=[{"user_id": 1}], ) model_predictions = model_server.predict(features) ``` ## 2. Offline Model Inference without Online Features Typically, Machine Learning teams find serving precomputed model predictions to be the most straightforward to implement. This approach simply treats the model predictions as a feature and serves them from the feature store using the standard Feast sdk. These model predictions are typically generated through some batch process where the model scores are precomputed. As a concrete example, the batch process can be as simple as a script that runs model inference locally for a set of users that can output a CSV. This output file could be used for materialization so that the model could be served online as shown in the code below. ```python model_predictions = store.get_online_features( feature_refs=[ "user_data:model_predictions", ], entity_rows=[{"user_id": 1}], ) ``` Notice that the model server is not involved in this approach. Instead, the model predictions are precomputed and materialized to the online store. While this approach can lead to quick impact for different business use cases, it suffers from stale data as well as only serving users/entities that were available at the time of the batch computation. In some cases, this tradeoff may be tolerable. ## 3. Online Model Inference with Online Features and Cached Predictions This approach is the most sophisticated where inference is optimized for low-latency by caching predictions and running model inference when data producers write features to the online store. This approach is particularly useful for applications where features are coming from multiple data sources, the model is computationally expensive to run, or latency is a significant constraint. ```python # Client Reads features = store.get_online_features( feature_refs=[ "user_data:click_through_rate", "user_data:number_of_clicks", "user_data:average_page_duration", "user_data:model_predictions", ], entity_rows=[{"user_id": 1}], ) if features.to_dict().get('user_data:model_predictions') is None: model_predictions = model_server.predict(features) store.write_to_online_store(feature_view_name="user_data", df=pd.DataFrame(model_predictions)) ``` Note that in this case a separate call to `write_to_online_store` is required when the underlying data changes and predictions change along with it. ```python # Client Writes from the Data Producer user_data = request.POST.get('user_data') model_predictions = model_server.predict(user_data) # assume this includes `user_data` in the Data Frame store.write_to_online_store(feature_view_name="user_data", df=pd.DataFrame(model_predictions)) ``` While this requires additional writes for every data producer, this approach will result in the lowest latency for model inference. ## 4. Online Model Inference without Features This approach does not require Feast. The model server can directly serve predictions without any features. This approach is common in Large Language Models (LLMs) and other models that do not require features to make predictions. Note that generative models using Retrieval Augmented Generation (RAG) do require features where the [document embeddings](/master/reference/alpha-vector-database) are treated as features, which Feast supports (this would fall under "Online Model Inference with Online Features"). ### Client Orchestration Implicit in the code examples above is a design choice about how clients orchestrate calls to get features and run model inference. The examples had a Feast-centric pattern because they are inputs to the model, so the sequencing is fairly obvious. An alternative approach can be Inference-centric where a client would call an inference endpoint and the inference service would be responsible for orchestration. # Role-Based Access Control (RBAC) ## Introduction Role-Based Access Control (RBAC) is a security mechanism that restricts access to resources based on the roles/groups/namespaces of individual users within an organization. In the context of the Feast, RBAC ensures that only authorized users or groups/namespaces can access or modify specific resources, thereby maintaining data security and operational integrity. ## Functional Requirements The RBAC implementation in Feast is designed to: * **Assign Permissions**: Allow administrators to assign permissions for various operations and resources to users or groups/namespaces. * **Seamless Integration**: Integrate smoothly with existing business code without requiring significant modifications. * **Backward Compatibility**: Maintain support for non-authorized models as the default to ensure backward compatibility. ## Business Goals The primary business goals of implementing RBAC in the Feast are: 1. **Feature Sharing**: Enable multiple teams to share the feature store while ensuring controlled access. This allows for collaborative work without compromising data security. 2. **Access Control Management**: Prevent unauthorized access to team-specific resources and spaces, governing the operations that each user or group can perform. ## Reference Architecture Feast operates as a collection of connected services, each enforcing authorization permissions. The architecture is designed as a distributed microservices system with the following key components: * **Service Endpoints**: These enforce authorization permissions, ensuring that only authorized requests are processed. * **Client Integration**: Clients authenticate with feature servers by attaching authorization token to each request. * **Service-to-Service Communication**: This is always granted. ![rbac.jpg](/files/EvIy8vA1h4yiD7zIQeJs) ## Permission Model The RBAC system in Feast uses a permission model that defines the following concepts: * **Resource**: An object within Feast that needs to be secured against unauthorized access. * **Action**: A logical operation performed on a resource, such as Create, Describe, Update, Delete, Read, or write operations. * **Policy**: A set of rules that enforce authorization decisions on resources. The polices are based on user roles or groups or namespaces or combined. ## Authorization Architecture The authorization architecture in Feast is built with the following components: * **Token Extractor**: Extracts the authorization token from the request header. * **Token Parser**: Parses the token to retrieve user details. * **Policy Enforcer**: Validates the secured endpoint against the retrieved user details. * **Token Injector**: Adds the authorization token to each secured request header. # Concepts {% content-ref url="/pages/-MjpO74pNw9rnG6QrPjI" %} [Overview](/master/getting-started/concepts/overview) {% endcontent-ref %} {% content-ref url="/pages/BsVP42BJXmjLcdqXN9S8" %} [Project](/master/getting-started/concepts/project) {% endcontent-ref %} {% content-ref url="/pages/ziKx4clwaCpHmAY8NhPi" %} [Data ingestion](/master/getting-started/concepts/data-ingestion) {% endcontent-ref %} {% content-ref url="/pages/-MhU2UQ75Mndf\_wZdh1V" %} [Entity](/master/getting-started/concepts/entity) {% endcontent-ref %} {% content-ref url="/pages/lRvTMmy8d0nqbTUxBgu4" %} [Data types](/master/getting-started/concepts/feast-types) {% endcontent-ref %} {% content-ref url="/pages/7neKQVoqL89JMR6jUYu8" %} [Feature repository](/master/getting-started/concepts/feature-repo) {% endcontent-ref %} {% content-ref url="/pages/-Mca6IfuaJ3huv2R\_CTk" %} [Feature view](/master/getting-started/concepts/feature-view) {% endcontent-ref %} {% content-ref url="/pages/SlWn2n7rZgyGOiWnyk5E" %} [Batch feature view](/master/getting-started/concepts/batch-feature-view) {% endcontent-ref %} {% content-ref url="/pages/G6iQTn5Vqf1Itc3mJY7R" %} [Stream feature view](/master/getting-started/concepts/stream-feature-view) {% endcontent-ref %} {% content-ref url="/pages/WCI1vA867daD0qwEz8B3" %} [Tiling with Intermediate Representations](/master/getting-started/concepts/tiling) {% endcontent-ref %} {% content-ref url="/pages/-MhU1OETjxILD0uz\_jCi" %} [Feature retrieval](/master/getting-started/concepts/feature-retrieval) {% endcontent-ref %} {% content-ref url="/pages/-MjpVqlHk1CRVgOboWKy" %} [Point-in-time joins](/master/getting-started/concepts/point-in-time-joins) {% endcontent-ref %} {% content-ref url="/pages/w4Vl78V1aswbRvui4sBJ" %} [\[Alpha\] Saved dataset](/master/getting-started/concepts/dataset) {% endcontent-ref %} {% content-ref url="/pages/tUVVW5HTARJbyK8IrQbK" %} [Label view](/master/getting-started/concepts/label-view) {% endcontent-ref %} {% content-ref url="/pages/vrPLclWPENyZzIQR6Ad6" %} [Permission](/master/getting-started/concepts/permission) {% endcontent-ref %} {% content-ref url="/pages/Ba69pzwUUAPPBiwr1qKw" %} [Tags](/master/getting-started/concepts/tags) {% endcontent-ref %} # Overview ### Feast project structure The top-level namespace within Feast is a **project**. Users define one or more [feature views](/master/getting-started/concepts/feature-view) within a project. Each feature view contains one or more [features](/master/getting-started/concepts/feature-view#feature). These features typically relate to one or more [entities](/master/getting-started/concepts/entity). A feature view must always have a [data source](/master/getting-started/concepts/data-ingestion), which in turn is used during the generation of training [datasets](/master/getting-started/concepts/feature-retrieval#dataset) and when materializing feature values into the online store. You can read more about Feast projects in the [project page](/master/getting-started/concepts/project). ### Data ingestion For *offline use cases* that only rely on batch data, Feast does not need to ingest data and can query your existing data (leveraging a compute engine, whether it be a data warehouse or (experimental) Spark / Trino). Feast can help manage **pushing** streaming features to a batch source to make features available for training. For *online use cases*, Feast supports **ingesting** features from batch sources to make them available online (through a process called **materialization**), and **pushing** streaming features to make them available both offline / online. We explore this more in the next concept page ([Data ingestion](/master/getting-started/concepts/data-ingestion)) ### Feature registration and retrieval Features are *registered* as code in a version controlled repository, and tie to data sources + model versions via the concepts of **entities, feature views,** and **feature services.** We explore these concepts more in the upcoming concept pages. These features are then *stored* in a **registry**, which can be accessed across users and services. The features can then be *retrieved* via SDK API methods or via a deployed **feature server** which exposes endpoints to query for online features (to power real time models). Feast supports several patterns of feature retrieval. | Use case | Example | API | | :------------------------------------------------------: | :----------------------------------------------------------------------------------------------------: | :-----------------------: | | Training data generation | Fetching user and item features for (user, item) pairs when training a production recommendation model | `get_historical_features` | | Offline feature retrieval for batch predictions | Predicting user churn for all users on a daily basis | `get_historical_features` | | Online feature retrieval for real-time model predictions | Fetching pre-computed features to predict whether a real-time credit card transaction is fraudulent | `get_online_features` | # Project Projects provide complete isolation of feature stores at the infrastructure level. This is accomplished through resource namespacing, e.g., prefixing table names with the associated project. Each project should be considered a completely separate universe of entities and features. It is not possible to retrieve features from multiple projects in a single request. We recommend having a single feature store and a single project per environment (`dev`, `staging`, `prod`). ![](/files/-MaKTnYQG9kFhl8yPNTM) Users define one or more [feature views](/master/getting-started/concepts/feature-view) within a project. Each feature view contains one or more [features](/master/getting-started/concepts/feature-view#field). These features typically relate to one or more [entities](/master/getting-started/concepts/entity). A feature view must always have a [data source](/master/getting-started/concepts/data-ingestion), which in turn is used during the generation of training [datasets](/master/getting-started/concepts/feature-retrieval#dataset) and when materializing feature values into the online store. The concept of a "project" provide the following benefits: **Logical Grouping**: Projects group related features together, making it easier to manage and track them. **Feature Definitions**: Within a project, you can define features, including their metadata, types, and sources. This helps standardize how features are created and consumed. **Isolation**: Projects provide a way to isolate different environments, such as development, testing, and production, ensuring that changes in one project do not affect others. **Collaboration**: By organizing features within projects, teams can collaborate more effectively, with clear boundaries around the features they are responsible for. **Access Control**: Projects can implement permissions, allowing different users or teams to access only the features relevant to their work. # Data ingestion ## Data source A data source in Feast refers to raw underlying data that users own (e.g. in a table in BigQuery). Feast does not manage any of the raw underlying data but instead, is in charge of loading this data and performing different operations on the data to retrieve or serve features. Feast uses a time-series data model to represent data. This data model is used to interpret feature data in data sources in order to build training datasets or materialize features into an online store. Below is an example data source with a single entity column (`driver`) and two feature columns (`trips_today`, and `rating`). ![Ride-hailing data source](/files/-MaKSgD8bNlCMB-D9YZ2) Feast supports primarily **time-stamped** tabular data as data sources. There are many kinds of possible data sources: * **Batch data sources:** ideally, these live in data warehouses (BigQuery, Snowflake, Redshift), but can be in data lakes (S3, GCS, etc). Feast supports ingesting and querying data across both. * **Stream data sources**: Feast does **not** have native streaming integrations. It does however facilitate making streaming features available in different environments. There are two kinds of sources: * **Push sources** allow users to push features into Feast, and make it available for training / batch scoring ("offline"), for realtime feature serving ("online") or both. * **\[Alpha] Stream sources** allow users to register metadata from Kafka or Kinesis sources. The onus is on the user to ingest from these sources, though Feast provides some limited helper methods to ingest directly from Kafka / Kinesis topics. * **(Experimental) Request data sources:** This is data that is only available at request time (e.g. from a user action that needs an immediate model prediction response). This is primarily relevant as an input into [**on-demand feature views**](/master/reference/beta-on-demand-feature-view), which allow light-weight feature engineering and combining features across sources. ## Batch data ingestion Ingesting from batch sources is only necessary to power real-time models. This is done through **materialization**. Under the hood, Feast manages an *offline store* (to scalably generate training data from batch sources) and an *online store* (to provide low-latency access to features for real-time models). A key command to use in Feast is the `materialize_incremental` command, which fetches the *latest* values for all entities in the batch source and ingests these values into the online store. When working with On Demand Feature Views with `write_to_online_store=True`, you can also control whether transformations are applied during ingestion by using the `transform_on_write` parameter. Setting `transform_on_write=False` allows you to materialize pre-transformed features without reapplying transformations, which is particularly useful for large batch datasets that have already been processed. Materialization can be called programmatically or through the CLI:
Code example: programmatic scheduled materialization This snippet creates a feature store object which points to the registry (which knows of all defined features) and the online store (DynamoDB in this case), and ```python # Define Python callable def materialize(): repo_config = RepoConfig( registry=RegistryConfig(path="s3://[YOUR BUCKET]/registry.pb"), project="feast_demo_aws", provider="aws", offline_store="file", online_store=DynamoDBOnlineStoreConfig(region="us-west-2") ) store = FeatureStore(config=repo_config) store.materialize_incremental(datetime.datetime.now()) # (In production) Use Airflow PythonOperator materialize_python = PythonOperator( task_id='materialize_python', python_callable=materialize, ) ```
Code example: CLI based materialization **How to run this in the CLI** **With timestamps:** ```bash CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S") feast materialize-incremental $CURRENT_TIME ``` **Simple materialization (for data without event timestamps):** ```bash feast materialize --disable-event-timestamp ``` **How to run this on Airflow** ```python # Use BashOperator materialize_bash = BashOperator( task_id='materialize', bash_command=f'feast materialize-incremental {datetime.datetime.now().replace(microsecond=0).isoformat()}', ) ```
### Batch data schema inference If the `schema` parameter is not specified when defining a data source, Feast attempts to infer the schema of the data source during `feast apply`. The way it does this depends on the implementation of the offline store. For the offline stores that ship with Feast out of the box this inference is performed by inspecting the schema of the table in the cloud data warehouse, or if a query is provided to the source, by running the query with a `LIMIT` clause and inspecting the result. ## Stream data ingestion Ingesting from stream sources happens either via a Push API or via a contrib processor that leverages an existing Spark context. * To **push data into the offline or online stores**: see [push sources](/master/reference/data-sources/push) for details. * (experimental) To **use a contrib Spark processor** to ingest from a topic, see [Tutorial: Building streaming features](/master/tutorials/building-streaming-features) # Entity An entity is a collection of semantically related features. Users define entities to map to the domain of their use case. For example, a ride-hailing service could have customers and drivers as their entities, which group related features that correspond to these customers and drivers. ```python driver = Entity(name='driver', join_keys=['driver_id']) ``` The *entity name* is used to uniquely identify the entity (for example to show in the experimental Web UI). The *join key* is used to identify the physical primary key on which feature values should be joined together to be retrieved during feature retrieval. Entities are used by Feast in many contexts, as we explore below: ### Use case #1: Defining and storing features Feast's primary object for defining features is a *feature view,* which is a collection of features. Feature views map to 0 or more entities, since a feature can be associated with: * zero entities (e.g. a global feature like *num\_daily\_global\_transactions*) * one entity (e.g. a user feature like *user\_age* or *last\_5\_bought\_items*) * multiple entities, aka a composite key (e.g. a user + merchant category feature like *num\_user\_purchases\_in\_merchant\_category)* Feast refers to this collection of entities for a feature view as an **entity key**. ![](/files/-MaKdRWc81UMJLidewa6) Entities should be reused across feature views. This helps with discovery of features, since it enables data scientists understand how other teams build features for the entity they are most interested in. Feast will use the feature view concept to then define the schema of groups of features in a low-latency online store. ### Use case #2: Retrieving features At *training time*, users control what entities they want to look up, for example corresponding to train / test / validation splits. A user specifies a list of *entity keys + timestamps* they want to fetch [point-in-time](/master/getting-started/concepts/point-in-time-joins) correct features for to generate a training dataset. At *serving time*, users specify *entity key(s)* to fetch the latest feature values which can power real-time model prediction (e.g. a fraud detection model that needs to fetch the latest transaction user's features to make a prediction). {% hint style="info" %} **Q: Can I retrieve features for all entities?** Kind of. In practice, this is most relevant for *batch scoring models* (e.g. predict user churn for all existing users) that are offline only. For these use cases, Feast supports generating features for a SQL-backed list of entities. There is an [open GitHub issue](https://github.com/feast-dev/feast/issues/1611) that welcomes contribution to make this a more intuitive API. For *real-time feature retrieval*, there is no out of the box support for this because it would promote expensive and slow scan operations which can affect the performance of other operations on your data sources. Users can still pass in a large list of entities for retrieval, but this does not scale well. {% endhint %} # Data types Feast frequently has to mediate data across platforms and systems, each with its own unique type system. To make this possible, Feast itself has a type system for all the types it is able to handle natively. Feast's type system is built on top of [protobuf](https://github.com/protocolbuffers/protobuf). The messages that make up the type system can be found [here](https://github.com/feast-dev/feast/blob/master/protos/feast/types/Value.proto), and the corresponding python classes that wrap them can be found [here](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/types.py). Feast supports the following categories of data types: * **Primitive types**: numerical values (`Int32`, `Int64`, `Float32`, `Float64`), `String`, `Bytes`, `Bool`, and `UnixTimestamp`. * **Zoned timestamp type**: `ZonedTimestamp` stores a timezone-aware datetime as both the UTC instant and its originating zone, so the original wall-clock zone round-trips losslessly. This differs from `UnixTimestamp`, which is always decoded as UTC and discards the source zone. Use `ZonedTimestamp` when local time-of-day or the offset/zone itself is meaningful. It must be explicitly declared in schema (it is not inferred by any backend), and is not supported as an entity key. * **Domain-specific primitives**: `PdfBytes` (PDF binary data for RAG/document pipelines) and `ImageBytes` (image binary data for multimodal pipelines). These are semantic aliases over `Bytes` and must be explicitly declared in schema — no backend infers them. * **UUID types**: `Uuid` and `TimeUuid` for universally unique identifiers. Stored as strings at the proto level but deserialized to `uuid.UUID` objects in Python. * **Array types**: ordered lists of any primitive type, e.g. `Array(Int64)`, `Array(String)`, `Array(Uuid)`. * **Set types**: unordered collections of unique values for any primitive type, e.g. `Set(String)`, `Set(Int64)`. Set types are not inferred by any backend and must be explicitly declared. They are best suited for online serving use cases. * **Map types**: dictionary-like structures. `Map` has string keys and values that can be any supported Feast type (including nested maps), e.g. `Map`, `Array(Map)`. `ScalarMap` has non-string scalar keys (int, float, bool, UUID, Decimal, bytes, datetime) — Feast infers `ScalarMap` automatically when the first key is not a string. `ScalarMap` must be explicitly declared in schema and is not inferred by any backend. * **JSON type**: opaque JSON data stored as a string at the proto level but semantically distinct from `String` — backends use native JSON types (`jsonb`, `VARIANT`, etc.), e.g. `Json`, `Array(Json)`. * **Struct type**: schema-aware structured type with named, typed fields. Unlike `Map` (which is schema-free), a `Struct` declares its field names and their types, enabling schema validation, e.g. `Struct({"name": String, "age": Int32})`. For a complete reference with examples, see [Type System](/master/reference/type-system). Each feature or schema field in Feast is associated with a data type, which is stored in Feast's [registry](https://github.com/feast-dev/feast/blob/master/docs/getting-started/concepts/registry.md). These types are also used to ensure that Feast operates on values correctly (e.g. making sure that timestamp columns used for [point-in-time correct joins](/master/getting-started/concepts/point-in-time-joins) actually have the timestamp type). As a result, each system that Feast interacts with needs a way to translate data types from the native platform into a Feast type. E.g., Snowflake SQL types are converted to Feast types [here](https://rtd.feast.dev/en/master/feast.html#feast.type_map.snowflake_python_type_to_feast_value_type). The onus is therefore on authors of offline or online store connectors to make sure that this type mapping happens correctly. ### Backend Type Mapping for Complex Types Map, JSON, and Struct types are supported across all major Feast backends: | Backend | Native Type | Feast Type | | ---------- | --------------------------- | ------------------------------------ | | PostgreSQL | `jsonb` | `Map`, `Json`, `Struct` | | PostgreSQL | `jsonb[]` | `Array(Map)` | | Snowflake | `VARIANT`, `OBJECT` | `Map` | | Snowflake | `JSON` | `Json` | | Redshift | `SUPER` | `Map` | | Redshift | `json` | `Json` | | BigQuery | `JSON` | `Json` | | BigQuery | `STRUCT`, `RECORD` | `Struct` | | Spark | `map` | `Map` | | Spark | `array>` | `Array(Map)` | | Spark | `struct<...>` | `Struct` | | Spark | `array>` | `Array(Struct(...))` | | MSSQL | `nvarchar(max)` | `Map`, `Json`, `Struct` | | DynamoDB | Proto bytes | `Map`, `Json`, `Struct`, `ScalarMap` | | Redis | Proto bytes | `Map`, `Json`, `Struct`, `ScalarMap` | | Milvus | `VARCHAR` (serialized) | `Map`, `Json`, `Struct` | **Note**: When the backend native type is ambiguous (e.g., `jsonb` could be `Map`, `Json`, or `Struct`), the **schema-declared Feast type takes precedence**. The backend-to-Feast type mappings above are only used for schema inference when no explicit type is provided. **Note**: Feast currently does *not* support a null type in its type system. # Feature repository ## Feature repository Feast users use Feast to manage two important sets of configuration: * Configuration about how to run Feast on your infrastructure * Feature definitions With Feast, the above configuration can be written declaratively and stored as code in a central location. This central location is called a feature repository. The feature repository is the declarative source of truth for what the desired state of a feature store should be. A feature repository is the collection of python files that define entities, feature views and data sources. Feature Repos also have a `feature_store.yaml` file at their root. Users can collaborate by making and reviewing changes to Feast object definitions (feature views, entities, etc.) in the feature repo. But, these objects must be applied, either through API, or the CLI, for them to be available by downstream Feast actions (such as materialization, or retrieving online features). Internally, Feast only looks at the registry when performing these actions, and not at the feature repo directly. ## Declarative Feature Definitions When using the CLI to apply changes (via `feast apply`), the CLI determines the state of the feature repo from the source files and updates the registry state to reflect the definitions in the feature repo files. This means that new feature views are added to the registry, existing feature views are updated as necessary, and Feast objects removed from the source files are deleted from the registry. For more details, see the [Feature repository](/master/reference/feature-repository) reference. # Feature view ## Feature views {% hint style="warning" %} **Note**: Feature views do not work with non-timestamped data. A workaround is to insert dummy timestamps. {% endhint %} A **feature view** is defined as a *collection of features*. * In the online settings, this is a *stateful* collection of features that are read when the `get_online_features` method is called. * In the offline setting, this is a *stateless* collection of features that are created when the `get_historical_features` method is called. A feature view is an object representing a logical group of time-series feature data as it is found in a [data source](/master/getting-started/concepts/data-ingestion). Depending on the kind of feature view, it may contain some lightweight (experimental) feature transformations (see [\[Beta\] On demand feature views](/master/reference/beta-on-demand-feature-view)). Feature views consist of: * a [data source](/master/getting-started/concepts/data-ingestion) * zero or more [entities](/master/getting-started/concepts/entity) * If the features are not related to a specific object, the feature view might not have entities; see [feature views without entities](#feature-views-without-entities) below. * a name to uniquely identify this feature view in the project. * (optional, but recommended) a schema specifying one or more [features](#field) (without this, Feast will infer the schema by reading from the data source) * (optional, but recommended) metadata (for example, description, or other free-form metadata via `tags`) * (optional) `owner`: the email of the primary maintainer * (optional) `org`: the organizational unit that owns the feature view (e.g. `"ads"`, `"search"`); useful for grouping feature views by team or product area * (optional) a TTL, which limits how far back Feast will look when generating historical datasets * (optional) `enable_validation=True`, which enables schema validation during materialization (see [Schema Validation](#schema-validation) below) Feature views allow Feast to model your existing feature data in a consistent way in both an offline (training) and online (serving) environment. Feature views generally contain features that are properties of a specific object, in which case that object is defined as an entity and included in the feature view. {% tabs %} {% tab title="driver\_trips\_feature\_view\.py" %} ```python from feast import BigQuerySource, Entity, FeatureView, Field from feast.types import Float32, Int64 driver = Entity(name="driver", join_keys=["driver_id"]) driver_stats_fv = FeatureView( name="driver_activity", entities=[driver], schema=[ Field(name="trips_today", dtype=Int64), Field(name="rating", dtype=Float32), ], source=BigQuerySource( table="feast-oss.demo_data.driver_activity" ) ) ``` {% endtab %} {% endtabs %} Feature views are used during * The generation of training datasets by querying the data source of feature views in order to find historical feature values. A single training dataset may consist of features from multiple feature views. * Loading of feature values into an online store. Feature views determine the storage schema in the online store. Feature values can be loaded from batch sources or from [stream sources](/master/reference/data-sources/push). * Retrieval of features from the online store. Feature views provide the schema definition to Feast in order to look up features from the online store. ## Feature views without entities If a feature view contains features that are not related to a specific entity, the feature view can be defined without entities (only timestamps are needed for this feature view). {% tabs %} {% tab title="global\_stats.py" %} ```python from feast import BigQuerySource, FeatureView, Field from feast.types import Int64 global_stats_fv = FeatureView( name="global_stats", entities=[], schema=[ Field(name="total_trips_today_by_all_drivers", dtype=Int64), ], source=BigQuerySource( table="feast-oss.demo_data.global_stats" ) ) ``` {% endtab %} {% endtabs %} ## Feature inferencing If the `schema` parameter is not specified in the creation of the feature view, Feast will infer the features during `feast apply` by creating a `Field` for each column in the underlying data source except the columns corresponding to the entities of the feature view or the columns corresponding to the timestamp columns of the feature view's data source. The names and value types of the inferred features will use the names and data types of the columns from which the features were inferred. ## Entity aliasing "Entity aliases" can be specified to join `entity_dataframe` columns that do not match the column names in the source table of a FeatureView. This could be used if a user has no control over these column names or if multiple entities are subclasses of a more general entity. For example, "spammer" and "reporter" could be aliases of a "user" entity, and "origin" and "destination" could be aliases of a "location" entity as shown below. It is suggested that you dynamically specify the new FeatureView name using `.with_name` and `join_key_map` override using `.with_join_key_map` instead of needing to register each new copy. {% tabs %} {% tab title="location\_stats\_feature\_view\.py" %} ```python from feast import BigQuerySource, Entity, FeatureView, Field from feast.types import Int32, Int64 location = Entity(name="location", join_keys=["location_id"]) location_stats_fv= FeatureView( name="location_stats", entities=[location], schema=[ Field(name="temperature", dtype=Int32), Field(name="location_id", dtype=Int64), ], source=BigQuerySource( table="feast-oss.demo_data.location_stats" ), ) ``` {% endtab %} {% tab title="temperatures\_feature\_service.py" %} ```python from location_stats_feature_view import location_stats_fv temperatures_fs = FeatureService( name="temperatures", features=[ location_stats_fv .with_name("origin_stats") .with_join_key_map( {"location_id": "origin_id"} ), location_stats_fv .with_name("destination_stats") .with_join_key_map( {"location_id": "destination_id"} ), ], ) ``` {% endtab %} {% endtabs %} ## Field A field or feature is an individual measurable property. It is typically a property observed on a specific entity, but does not have to be associated with an entity. For example, a feature of a `customer` entity could be the number of transactions they have made on an average month, while a feature that is not observed on a specific entity could be the total number of posts made by all users in the last month. Supported types for fields in Feast can be found in `sdk/python/feast/types.py`. Fields are defined as part of feature views. Since Feast does not transform data, a field is essentially a schema that only contains a name and a type: ```python from feast import Field from feast.types import Float32 trips_today = Field( name="trips_today", dtype=Float32 ) ``` Together with [data sources](/master/getting-started/concepts/data-ingestion), they indicate to Feast where to find your feature values, e.g., in a specific parquet file or BigQuery table. Feature definitions are also used when reading features from the feature store, using [feature references](/master/getting-started/concepts/feature-retrieval#feature-references). Feature names must be unique within a [feature view](#feature-view). Each field can have additional metadata associated with it, specified as key-value [tags](https://rtd.feast.dev/en/master/feast.html#feast.field.Field). ## \[Alpha] Versioning Feature views support automatic version tracking. Every time `feast apply` detects a schema or UDF change, a versioned snapshot is saved to the registry. This enables auditing what changed, reverting to a prior version, querying specific versions via `@v` syntax, and staging new versions without promoting them. Version history tracking is **always active** with no configuration needed. The `version` parameter is fully optional — omitting it preserves existing behavior. ```python # Pin to a specific version (reverts the active definition to v2's snapshot) driver_stats = FeatureView( name="driver_stats", entities=[driver], schema=[...], source=my_source, version="v2", ) ``` For full details on version pinning, version-qualified reads, staged publishing (`--no-promote`), online store support, and known limitations, see the [**\[Alpha\] Feature View Versioning**](/master/reference/alpha-feature-view-versioning) reference page. ## Schema Validation Feature views support an optional `enable_validation` parameter that enables schema validation during materialization and historical feature retrieval. When enabled, Feast verifies that: * All declared feature columns are present in the input data. * Column data types match the expected Feast types (mismatches are logged as warnings). This is useful for catching data quality issues early in the pipeline. To enable it: ```python from feast import FeatureView, Field from feast.types import Int32, Int64, Float32, Json, Map, String, Struct validated_fv = FeatureView( name="validated_features", entities=[driver], schema=[ Field(name="trips_today", dtype=Int64), Field(name="rating", dtype=Float32), Field(name="preferences", dtype=Map), Field(name="config", dtype=Json), # opaque JSON data Field(name="address", dtype=Struct({"street": String, "city": String, "zip": Int32})), # typed struct ], source=my_source, enable_validation=True, # enables schema checks ) ``` **JSON vs Map vs Struct**: These three complex types serve different purposes: * **`Map`**: Schema-free dictionary with string keys. Use when the keys and values are dynamic. * **`Json`**: Opaque JSON data stored as a string. Backends use native JSON types (`jsonb`, `VARIANT`). Use for configuration blobs or API responses where you don't need field-level typing. * **`Struct`**: Schema-aware structured type with named, typed fields. Persisted through the registry via Field tags. Use when you know the exact structure and want type safety. Validation is supported in all compute engines (Local, Spark, and Ray). When a required column is missing, a `ValueError` is raised. Type mismatches are logged as warnings but do not block execution, allowing for safe gradual adoption. The `enable_validation` parameter is also available on `BatchFeatureView` and `StreamFeatureView`, as well as their respective decorators (`@batch_feature_view` and `@stream_feature_view`). ## \[Alpha] On demand feature views On demand feature views allows data scientists to use existing features and request time data (features only available at request time) to transform and create new features. Users define python transformation logic which is executed in both the historical retrieval and online retrieval paths. Currently, these transformations are executed locally. This is fine for online serving, but does not scale well to offline retrieval. ### Why use on demand feature views? This enables data scientists to easily impact the online feature retrieval path. For example, a data scientist could 1. Call `get_historical_features` to generate a training dataframe 2. Iterate in notebook on feature engineering in Pandas 3. Copy transformation logic into on demand feature views and commit to a dev branch of the feature repository 4. Verify with `get_historical_features` (on a small dataset) that the transformation gives expected output over historical data 5. Verify with `get_online_features` on dev branch that the transformation correctly outputs online features 6. Submit a pull request to the staging / prod branches which impact production traffic ```python from feast import Field, RequestSource from feast.on_demand_feature_view import on_demand_feature_view from feast.types import Float64 # Define a request data source which encodes features / information only # available at request time (e.g. part of the user initiated HTTP request) input_request = RequestSource( name="vals_to_add", schema=[ Field(name="val_to_add", dtype=PrimitiveFeastType.INT64), Field(name="val_to_add_2": dtype=PrimitiveFeastType.INT64), ] ) # Use the input data and feature view features to create new features @on_demand_feature_view( sources=[ driver_hourly_stats_view, input_request ], schema=[ Field(name='conv_rate_plus_val1', dtype=Float64), Field(name='conv_rate_plus_val2', dtype=Float64) ] ) def transformed_conv_rate(features_df: pd.DataFrame) -> pd.DataFrame: df = pd.DataFrame() df['conv_rate_plus_val1'] = (features_df['conv_rate'] + features_df['val_to_add']) df['conv_rate_plus_val2'] = (features_df['conv_rate'] + features_df['val_to_add_2']) return df ``` ## \[Alpha] Stream feature views A stream feature view is an extension of a normal feature view. The primary difference is that stream feature views have both stream and batch data sources, whereas a normal feature view only has a batch data source. Stream feature views should be used instead of normal feature views when there are stream data sources (e.g. Kafka and Kinesis) available to provide fresh features in an online setting. Like regular feature views, stream feature views support the optional `org` field for grouping by organizational unit. Here is an example definition of a stream feature view with an attached transformation: ```python from datetime import timedelta from feast import Field, FileSource, KafkaSource, stream_feature_view from feast.data_format import JsonFormat from feast.types import Float32 driver_stats_batch_source = FileSource( name="driver_stats_source", path="data/driver_stats.parquet", timestamp_field="event_timestamp", ) driver_stats_stream_source = KafkaSource( name="driver_stats_stream", kafka_bootstrap_servers="localhost:9092", topic="drivers", timestamp_field="event_timestamp", batch_source=driver_stats_batch_source, message_format=JsonFormat( schema_json="driver_id integer, event_timestamp timestamp, conv_rate double, acc_rate double, created timestamp" ), watermark_delay_threshold=timedelta(minutes=5), ) @stream_feature_view( entities=[driver], ttl=timedelta(seconds=8640000000), mode="spark", schema=[ Field(name="conv_percentage", dtype=Float32), Field(name="acc_percentage", dtype=Float32), ], timestamp_field="event_timestamp", online=True, source=driver_stats_stream_source, org="ads", # optional ) def driver_hourly_stats_stream(df: DataFrame): from pyspark.sql.functions import col return ( df.withColumn("conv_percentage", col("conv_rate") * 100.0) .withColumn("acc_percentage", col("acc_rate") * 100.0) .drop("conv_rate", "acc_rate") ) ``` See [here](https://github.com/feast-dev/streaming-tutorial) for an example of how to use stream feature views to register your own streaming data pipelines in Feast. # Batch feature view `BatchFeatureView` is a flexible abstraction in Feast that allows users to define features derived from batch data sources or even other `FeatureView`s, enabling composable and reusable feature pipelines. It is an extension of the `FeatureView` class, with support for user-defined transformations, aggregations, and recursive chaining of feature logic. ## Supported Compute Engines * [x] LocalComputeEngine * [x] SparkComputeEngine * [ ] LambdaComputeEngine * [ ] KubernetesComputeEngine *** ## ✅ Key Capabilities * **Composable DAG of FeatureViews**: Supports defining a `BatchFeatureView` on top of one or more other `FeatureView`s. * **Transformations**: Apply [transformation](/master/getting-started/architecture/feature-transformation) logic (`feature_transformation` or `udf`) to raw data source, can also be used to deal with multiple data sources. * **Aggregations**: Define time-windowed aggregations (e.g. `sum`, `avg`) over event-timestamped data. * **Feature resolution & execution**: Automatically resolves and executes DAGs of dependent views during materialization or retrieval. More details in the [Compute engine documentation](/master/reference/compute-engine). * **Materialization Sink Customization**: Specify a custom `sink_source` to define where derived feature data should be persisted. *** ## 📐 Class Signature ```python class BatchFeatureView(FeatureView): def __init__( *, name: str, source: Optional[Union[DataSource, FeatureView, List[FeatureView]]] = None, sink_source: Optional[DataSource] = None, schema: Optional[List[Field]] = None, entities: Optional[List[Entity]] = None, aggregations: Optional[List[Aggregation]] = None, udf: Optional[Callable[[DataFrame], DataFrame]] = None, udf_string: Optional[str] = None, ttl: Optional[timedelta] = timedelta(days=0), online: bool = True, offline: bool = False, description: str = "", tags: Optional[Dict[str, str]] = None, owner: str = "", ) ``` *** ## 🧠 Usage ### 1. Simple Feature View from Data Source ```python from feast import BatchFeatureView, Field from feast.types import Float32, Int32 from feast import FileSource from feast.aggregation import Aggregation from datetime import timedelta source = FileSource( path="s3://bucket/path/data.parquet", timestamp_field="event_timestamp", created_timestamp_column="created", ) driver_fv = BatchFeatureView( name="driver_hourly_stats", entities=["driver_id"], schema=[ Field(name="driver_id", dtype=Int32), Field(name="conv_rate", dtype=Float32), ], aggregations=[ Aggregation(column="conv_rate", function="sum", time_window=timedelta(days=1), name="total_conv_rate_1d"), ], source=source, ) ``` *** ### 2. Derived Feature View from Another View You can build feature views on top of other features by deriving a feature view from another view. Let's take a look at an example. ```python from feast import BatchFeatureView, Field from pyspark.sql import DataFrame from feast.types import Float32, Int32 from feast import FileSource def transform(df: DataFrame) -> DataFrame: return df.withColumn("conv_rate", df["conv_rate"] * 2) daily_driver_stats = BatchFeatureView( name="daily_driver_stats", entities=["driver_id"], schema=[ Field(name="driver_id", dtype=Int32), Field(name="conv_rate", dtype=Float32), ], udf=transform, source=driver_fv, sink_source=FileSource( # Required to specify where to sink the derived view name="daily_driver_stats_sink", path="s3://bucket/daily_stats/", file_format="parquet", timestamp_field="event_timestamp", created_timestamp_column="created", ), ) ``` *** ## 🔄 Execution Flow Feast automatically resolves the DAG of `BatchFeatureView` dependencies during: * `materialize()`: recursively resolves and executes the feature view graph. * `get_historical_features()`: builds the execution plan for retrieving point-in-time correct features. * `apply()`: registers the feature view DAG structure to the registry. Each transformation and aggregation is turned into a DAG node (e.g., `SparkTransformationNode`, `SparkAggregationNode`) executed by the compute engine (e.g., `SparkComputeEngine`). *** ## ⚙️ How Materialization Works * If the `BatchFeatureView` is backed by a base source (`FileSource`, `BigQuerySource`, `SparkSource` etc), the `batch_source` is used directly. * If the source is another feature view (i.e., chained views), the `sink_source` must be provided to define the materialization target data source. * During DAG planning, `SparkWriteNode` uses the `sink_source` as the batch sink. *** ## 🧪 Example Tests See: * `test_spark_dag_materialize_recursive_view()`: Validates chaining of two feature views and output validation. * `test_spark_compute_engine_materialize()`: Validates transformation and write of features into offline and online stores. *** ## 🛑 Gotchas * `sink_source` is **required** when chaining views (i.e., `source` is another FeatureView or list of them). * `source` is optional; if omitted (`None`), the feature view has no associated batch data source. * Schema fields must be consistent with `sink_source`, `batch_source.field_mapping` if field mappings exist. * Aggregation logic must reference columns present in the raw source or transformed inputs. * The output feature name for an aggregation defaults to `{function}_{column}` (e.g., `sum_conv_rate`). Use the `name` parameter to override it (e.g., `name="total_conv_rate_1d"`). *** ## 🔮 Future Directions * Support additional offline stores (e.g., Snowflake, Redshift) with auto-generated sink sources. * Enable fully declarative transform logic (SQL + UDF mix). * Introduce optimization passes for DAG pruning and fusion. # Stream feature view `StreamFeatureView` is a type of feature view in Feast that allows you to define features that are continuously updated from a streaming source. It is designed to handle real-time data ingestion and feature generation, making it suitable for use cases where features need to be updated frequently as new data arrives. ### Supported Compute Engines * [x] LocalComputeEngine * [x] SparkComputeEngine * [ ] FlinkComputeEngine ### Key Capabilities * **Real-time Feature Generation**: Supports defining features that are continuously updated from a streaming source. * **Transformations**: Apply transformation logic (e.g., `feature_transformation` or `udf`) to raw data source. * **Aggregations**: Define time-windowed aggregations (e.g., `sum`, `avg`) over event-timestamped data. * **⚡ Tiling with Intermediate Representations**: Enable efficient pre-aggregation with correct merging semantics for holistic aggregations like `avg` and `std`. This provides faster queries while maintaining mathematical accuracy. [Learn more about tiling](/master/getting-started/concepts/tiling) * **Feature resolution & execution**: Automatically resolves and executes dependent views during materialization or retrieval. # Tiling with Intermediate Representations ## Overview **Tiling** is an optimization technique for **streaming time-windowed aggregations** that enables massively efficient feature computation by pre-aggregating data into smaller time intervals (tiles) and storing **Intermediate Representations (IRs)** for correct merging. **Primary Use Case: Streaming** Tiling provides **speedup** for streaming scenarios where features are updated frequently (every few minutes) from sources like Kafka, Kinesis, or PushSource. **Key Benefits (Streaming):** * **Faster**: Reuse 90%+ of tiles between updates instead of recomputing from scratch * **Correct results**: IRs ensure mathematically accurate merging for all aggregation types * **Memory efficient**: Only process new events, reuse previous tiles in memory * **Real-time capable**: Handle high-throughput streaming with low latency * **Incremental updates**: Compute 1 new tile instead of rescanning entire window *** ## The Problem: Why Intermediate Representations? Traditional approaches to time-windowed aggregations either: 1. **Recompute from raw data** every time → Slow, expensive 2. **Store final aggregated values** per tile → Fast but often **incorrect** when merging ### The Merging Problem You **cannot correctly merge** many common aggregations: ``` WRONG: avg(tile1, tile2) ≠ (avg_tile1 + avg_tile2) / 2 Example: tile1: [10, 20, 30] → avg = 20 tile2: [100] → avg = 100 Correct merged avg: (10+20+30+100) / 4 = 40 Wrong merged avg: (20 + 100) / 2 = 60 ``` **The same problem exists for:** * Standard deviation (`std`) * Variance (`var`) * Median and percentiles * Any "holistic" aggregation that requires knowledge of all values *** ## The Solution: Intermediate Representations (IRs) Instead of storing **final aggregated values**, store **intermediate data** that preserves the mathematical properties needed for correct merging. ### Example: Average **Traditional (Incorrect)**: ``` Tile 1: avg = 20 Tile 2: avg = 20 Merged avg = (20 + 20) / 2 = 20 - WRONG ``` **With IRs (Correct)**: ``` Tile 1: sum = 60, count = 3 Tile 2: sum = 100, count = 1 Merged: sum = 160, count = 4 Merged avg = 160 / 4 = 40 - CORRECT ``` *** ## Aggregation Categories ### Algebraic Aggregations These can be merged by applying the same aggregation function to tiles: | Aggregation | Stored Value | Merge Strategy | Storage | | ----------- | ------------ | ------------------ | -------- | | `sum` | sum | `sum(tile_sums)` | 1 column | | `count` | count | `sum(tile_counts)` | 1 column | | `max` | max | `max(tile_maxes)` | 1 column | | `min` | min | `min(tile_mins)` | 1 column | **No IRs needed** - the final value is the IR! *** ### Holistic Aggregations These require storing multiple intermediate values: #### Average (`avg`, `mean`) **Stored IRs**: `sum`, `count`\ **Final computation**: `avg = sum / count`\ **Merge strategy**: Sum the sums and counts, then divide **Storage**: 3 columns (final + 2 IRs) *** #### Standard Deviation (`std`, `stddev`) **Stored IRs**: `count`, `sum`, `sum_of_squares`\ **Final computation**: ```python variance = (sum_sq - sum²/count) / (count - δ) std = sqrt(variance) # δ = 1 for sample, 0 for population ``` **Merge strategy**: Sum all three IRs, then apply formula **Storage**: 4 columns (final + 3 IRs) *** #### Variance (`var`, `variance`) **Stored IRs**: `count`, `sum`, `sum_of_squares`\ **Final computation**: Same as std but without `sqrt()` **Storage**: 4 columns (final + 3 IRs) *** ## How Tiling Works Tiling is optimized for **streaming scenarios** with frequent updates (e.g., every few minutes). ### 1. Continuous Tile Updates ``` Stream Events → Partition by Hop Intervals → Compute IRs → Store Windowed Aggregations | | | | | | | └─> Online Store (Redis, etc.) | | └─> avg_sum, avg_count, std_sum_sq, etc. | └─> 5-min hops: [00:00-00:05], [00:05-00:10], ... └─> customer_id=1: [txn1, txn2, txn3, ...] Every 5 minutes: - New events arrive - Only 1 new tile computed (5 min of data) - 11 previous tiles reused (in memory during streaming session) - Final aggregation = merge 12 tiles (1 new + 11 reused) ``` **Why It's Fast:** * **Without tiling:** Scan entire 1-hour window (1000+ events) every 5 minutes * **With tiling:** Only process 5 minutes of new events, reuse previous tiles * **Speedup:** Faster for streaming updates! *** ### 2. Streaming Update Efficiency | Update | Without Tiling | With Tiling | Tile Reuse | | ------- | -------------------------- | ------------------------- | ------------------ | | T=00:00 | Compute 1hr | Compute 12 tiles | 0% reuse (initial) | | T=00:05 | Compute 1hr (1000+ events) | Compute 1 tile + reuse 11 | 92% reuse | | T=00:10 | Compute 1hr (1000+ events) | Compute 1 tile + reuse 11 | 92% reuse | | T=00:15 | Compute 1hr (1000+ events) | Compute 1 tile + reuse 11 | 92% reuse | **Key Benefit:** Tiles stay in memory during the streaming session, enabling massive reuse. *** ## Tiling Algorithm ### Sawtooth Window Tiling 1. **Partition events** into hop-sized intervals (e.g., 5 minutes) 2. **Compute cumulative tail aggregations** for each hop from the start of the materialization window 3. **Subtract tiles** to form windowed aggregations (current\_tile - previous\_tile) 4. **Store IRs** for correct merging of holistic aggregations 5. **At materialization**, store windowed aggregations in online store **Benefits**: * Efficient query-time performance (pre-computed windows) * Minimal storage overhead (only hop-sized tiles) * Mathematically correct for all aggregation types *** ## Configuration ### Recommended: StreamFeatureView (Streaming Scenarios) Tiling provides **maximum benefit for streaming scenarios** with frequent updates: ```python from feast import StreamFeatureView, Aggregation from feast.data_source import PushSource, KafkaSource from datetime import timedelta # Example with Kafka streaming source customer_features = StreamFeatureView( name="customer_transaction_features", entities=[customer], source=KafkaSource( name="transactions_stream", kafka_bootstrap_servers="localhost:9092", topic="transactions", timestamp_field="event_timestamp", batch_source=file_source, # For historical data ), aggregations=[ Aggregation(column="amount", function="sum", time_window=timedelta(hours=1), name="sum_amount_1h"), Aggregation(column="amount", function="avg", time_window=timedelta(hours=1), name="avg_amount_1h"), Aggregation(column="amount", function="std", time_window=timedelta(hours=1), name="std_amount_1h"), ], timestamp_field="event_timestamp", online=True, # Tiling configuration enable_tiling=True, # speedup for streaming! tiling_hop_size=timedelta(minutes=5), # Update frequency ) ``` **When to Enable:** * Streaming data sources (Kafka, Kinesis, PushSource) * Frequent updates (every few minutes) * Real-time feature serving * High-throughput event processing *** ### Key Parameters * `aggregations`: List of time-windowed aggregations to compute. Each `Aggregation` accepts: * `column`: source column to aggregate * `function`: aggregation function (`sum`, `avg`, `mean`, `min`, `max`, `count`, `std`) * `time_window`: duration of the aggregation window * `slide_interval`: hop/slide size (defaults to `time_window`) * `name` *(optional)*: output feature name. Defaults to `{function}_{column}` (e.g., `sum_amount`). Set this to use a custom name (e.g., `name="sum_amount_1h"`). * `timestamp_field`: Column name for timestamps (required when aggregations are specified) * `enable_tiling`: Enable tiling optimization (default: `False`) * Set to `True` for **streaming scenarios** * `tiling_hop_size`: Time interval between tiles (default: 5 minutes) * Smaller = more granular tiles, potentially higher memory during processing window * Larger = less granular tiles, potentially lower memory during processing window ### Compute Engine Requirements * **Spark Compute Engine**: Fully supported for streaming and batch * **Ray Compute Engine**: Fully supported for streaming and batch * **Local Compute Engine**: Does NOT support time-windowed aggregations *** ## Architecture Tiling in Feast uses a **simple, pure pandas architecture** that works with any compute engine: ### How It Works ``` ┌─────────────────┐ │ Engine DataFrame│ (Spark/Ray/etc) └────────┬────────┘ │ .toPandas() / .to_pandas() ▼ ┌─────────────────┐ │ Pandas DataFrame│ └────────┬────────┘ │ orchestrator.apply_sawtooth_window_tiling() ▼ ┌─────────────────┐ │ Cumulative │ (pandas with _tile_start, _tile_end, IRs) │ Tiles │ └────────┬────────┘ │ tile_subtraction.convert_cumulative_to_windowed() ▼ ┌─────────────────┐ │ Windowed │ (pandas with final aggregations) │ Aggregations │ └────────┬────────┘ │ spark.createDataFrame() / ray.from_pandas() ▼ ┌─────────────────┐ │ Engine DataFrame│ └─────────────────┘ ``` ## Summary Tiling with Intermediate Representations provides a powerful optimization for **streaming time-windowed aggregations** in Feast. # Label view ## What is a Label View? A **label view** is a Feast primitive for storing **judgments about entities** — reward signals, safety scores, human reviews, ground-truth answers — separately from the **immutable observations** stored in [feature views](/master/getting-started/concepts/feature-view). | | Feature View | Label View | | --------------------- | ------------------------- | ------------------------------------------ | | **Stores** | What was observed | What was judged | | **Example** | Agent prompt and response | `response_quality: "poor"`, `is_safe: 0` | | **Writers** | Usually one source | Multiple labelers (human, LLM judge, code) | | **Changes over time** | Append-only | Updated by new label writes | Use a label view when you need **governed, training-ready labels** that multiple sources can write, disagree on, and resolve — not when you need append-only feature data. ## Prerequisites Before using label views, you need: * Feast with the LabelView primitive * A feature repo with at least one `Entity` * A `PushSource` (or batch `DataSource`) for label ingestion * `feast apply` run after defining label views ## Why use Label Views? **Separate features from judgments.** Mixing reward labels into feature views blurs two different lifecycles — observations are append-only; labels are overwritten, corrected, and debated. **Support multiple labelers.** Human reviewers, safety scanners, and LLM judges can all label the same entity. Feast tracks who wrote what via `labeler_field` and resolves conflicts via `ConflictPolicy`. **Generate training datasets.** Compose label views with feature views in a `FeatureService` and retrieve features + labels together with point-in-time correctness. **Data label in the UI.** Configure data labeling profiles (annotation profiles) so data scientists can label data directly in the Feast UI — entity forms, document spans, bulk review, or active learning. ## Feedback vs Expectations Not all labels serve the same purpose. Feast distinguishes two common types — both stored in a label view, modeled with field names and tags: | | **Feedback** | **Expectation** | | --------------------- | ----------------------------------------------------- | ------------------------------------------- | | **Question** | How good was the actual output? | What is the correct answer? | | **Example** | `response_quality: "poor"`, `relevance: "irrelevant"` | `ground_truth: "relevant"`, `is_default: 1` | | **Typical writers** | Human, LLM judge, automated code | Human experts (gold standard) | | **Training use** | Reward signal, quality filter, active-learning queue | Supervised target column | | **Conflict handling** | Common — use `ConflictPolicy` | Rare — usually one authoritative source | Feast does not require separate primitives for feedback and expectations. Model them with **field names** and **tags**: ```python tags={ "feast.io/field-role:response_quality": "feedback", # judgment about output "feast.io/field-role:ground_truth": "expectation", # correct answer } ``` **Practical pattern:** * **Feedback label view** — multi-labeler, `ConflictPolicy`, fields like `response_quality`, `safety_score`, `relevance` * **Expectation fields or view** — stable ground truth, fields like `ground_truth`, `expected_answer`, `is_default` * **Mixed view** — one label view with both (e.g. RAG: `relevance` = feedback, `ground_truth` = expectation) ## Sources of Labels Label views accept labels from any source. Track the writer in `labeler_field`: | Source | `labeler` example | Typical role | | --------------------- | ---------------------------- | --------------------------------- | | **Human reviewer** | `human-reviewer@company.com` | Feedback or expectation | | **LLM judge** | `gpt-4-evaluator` | Feedback (quality scores) | | **Automated scanner** | `nemo-guardrails` | Feedback (safety signals) | | **Batch import** | `risk-ops-team` | Expectation (historical outcomes) | ```python labels_df = pd.DataFrame({ "user_id": ["user-001"], "response_quality": ["poor"], "is_safe": [0], "labeler": ["human-reviewer@company.com"], "event_timestamp": [pd.Timestamp.now()], }) store.push("agent_feedback_labels_push_source", labels_df) ``` ## When to use Label Views | Use a **FeatureView** when… | Use a **LabelView** when… | | ------------------------------------- | -------------------------------------------------------- | | Data is observational and append-only | Data is a judgment or data label (annotation) | | One source writes the data | Multiple labelers may disagree | | No conflict resolution needed | You need governed conflict resolution | | No labeling UI needed | You want structured data labeling workflows (annotation) | ## How Label Views Work ### Step 1: Define a label view ```python from datetime import timedelta from feast import Entity, FeatureService, Field, FileSource, PushSource from feast.labeling import ConflictPolicy, LabelView from feast.types import Float32, Int64, String user = Entity(name="user_id", join_keys=["user_id"]) agent_feedback_labels = LabelView( name="agent_feedback_labels", entities=[user], schema=[ Field(name="response_quality", dtype=String), Field(name="is_safe", dtype=Int64), Field(name="reviewer_notes", dtype=String), Field(name="labeler", dtype=String), ], source=PushSource( name="agent_feedback_labels_push_source", batch_source=FileSource( name="agent_feedback_labels_batch", path="data/agent_feedback.parquet", timestamp_field="event_timestamp", ), ), labeler_field="labeler", conflict_policy=ConflictPolicy.LAST_WRITE_WINS, reference_feature_view="user_profile", description="Human and automated feedback on agent responses.", tags={ "feast.io/labeling-method": "entity-form", "feast.io/field-role:response_quality": "feedback", "feast.io/field-role:is_safe": "feedback", "feast.io/field-role:reviewer_notes": "metadata", "feast.io/label-values:response_quality": "excellent,good,acceptable,poor,harmful", "feast.io/label-values:is_safe": "1,0", "feast.io/label-widget:response_quality": "enum", "feast.io/label-widget:is_safe": "binary", "feast.io/label-widget:reviewer_notes": "text", }, ) ``` Run `feast apply` to register the label view. ### Step 2: Push labels Labels are written with `FeatureStore.push()`: ```python import pandas as pd from feast import FeatureStore store = FeatureStore(repo_path="feature_repo/") labels_df = pd.DataFrame({ "user_id": ["user-001", "user-002"], "response_quality": ["good", "harmful"], "is_safe": [1, 0], "reviewer_notes": ["Accurate summary", "Unsafe medical advice"], "labeler": ["human-reviewer", "nemo-guardrails"], "event_timestamp": pd.to_datetime(["2025-01-15", "2025-01-15"]), }) store.push("agent_feedback_labels_push_source", labels_df) ``` Each push appends to the offline store (full history retained) and updates the online store (latest value per key). ### Step 3: Data label in the Feast UI Open the label view in the Feast UI **Data Labeling** tab (Annotate tab). The UI reads data labeling tags (annotation tags) and shows the right workflow: 1. Open **Label Views** in the sidebar 2. Select a label view (check the **Data Labeling** badge on the list page) 3. Go to the **Data Labeling** tab (Annotate tab) 4. Choose a data labeling method (annotation method): Entity Form, Document Span, Review & Edit, or Active Learning 5. Submit labels — they are pushed to the label view's `PushSource` ### Step 4: Join labels with features for training ```python training_service = FeatureService( name="agent_training_service", features=[ user_profile_fv, # immutable features agent_feedback_labels, # mutable labels ], ) training_df = store.get_historical_features( entity_df=entities_df, features=training_service, ).to_df() ``` Training pipelines get features and resolved labels in one retrieval call. ## Data Labeling Profiles (Annotation Profiles) Data labeling profiles (annotation profiles) configure **how** labels are created in the UI. Set them via `tags` — no schema changes required. ### Supported profiles | Profile | Best for | UI experience | | ----------------- | ---------------------------------------------- | ----------------------------------- | | `entity-form` | RLHF, safety review, per-entity feedback | Form — one entity at a time | | `document-span` | RAG chunk labeling, span labeling (annotation) | Load document, label chunks | | `table` | Bulk review, correcting existing labels | Editable table with dropdowns | | `active-learning` | Label high-value unlabeled entities | Queue from a reference feature view | ### Choosing a profile Answer one question: 1. **"I need to review agent responses one at a time"** → `entity-form` 2. **"I need to label document chunks for RAG"** → `document-span` 3. **"I need to correct labels in bulk"** → `table` 4. **"I want to label only the most valuable unlabeled items"** → `active-learning` (requires `reference_feature_view`) The **Data Labeling** tab (Annotate tab) shows only relevant data labeling methods (annotation methods) per profile: | Profile | Methods shown | | ----------------- | ------------------------------------------- | | `document-span` | Document Span, Review & Edit | | `entity-form` | Entity Form, Review & Edit, Active Learning | | `table` | Review & Edit, Active Learning, Entity Form | | `active-learning` | Active Learning, Entity Form, Review & Edit | ### Tag reference | Tag | Purpose | Example values | | ------------------------------- | ------------------------------------------------ | ----------------------------------------------------------------------------------- | | `feast.io/labeling-method` | Primary data labeling method (annotation method) | `entity-form`, `document-span`, `table` | | `feast.io/field-role:` | Semantic role of a field | `feedback`, `expectation`, `label`, `metadata`, `content`, `span_start`, `span_end` | | `feast.io/label-values:` | Allowed label values | `relevant,irrelevant` | | `feast.io/label-widget:` | Input widget type | `enum`, `binary`, `text`, `number` | ## Examples by use case ### Agent feedback (RLHF / safety) Feedback from human reviewers and automated safety layers on agent responses. ```python agent_feedback_labels = LabelView( name="agent_feedback_labels", entities=[user], schema=[ Field(name="response_quality", dtype=String), Field(name="is_safe", dtype=Int64), Field(name="reviewer_notes", dtype=String), Field(name="labeler", dtype=String), ], source=feedback_push_source, labeler_field="labeler", conflict_policy=ConflictPolicy.LAST_WRITE_WINS, reference_feature_view="user_profile", tags={"feast.io/labeling-method": "entity-form", ...}, ) ``` ### RAG chunk labeling (feedback + expectation) One view can hold both a retrieval judgment and ground truth: ```python rag_chunk_labels = LabelView( name="rag_chunk_labels", entities=[chunk], schema=[ Field(name="source_document", dtype=String), Field(name="chunk_text", dtype=String), Field(name="relevance", dtype=String), # feedback Field(name="ground_truth", dtype=String), # expectation Field(name="chunk_start", dtype=Int64), Field(name="chunk_end", dtype=Int64), Field(name="labeler", dtype=String), ], source=rag_push_source, labeler_field="labeler", conflict_policy=ConflictPolicy.MAJORITY_VOTE, tags={ "feast.io/labeling-method": "document-span", "feast.io/field-role:relevance": "feedback", "feast.io/field-role:ground_truth": "expectation", "feast.io/field-role:chunk_text": "content", "feast.io/label-values:relevance": "relevant,irrelevant", }, ) ``` ### Historical ground truth (batch labels) Pre-existing label tables loaded via `feast materialize`: ```python loan_default_labels = LabelView( name="loan_default_labels", entities=[loan], schema=[ Field(name="is_default", dtype=Int64), Field(name="delinquency_days", dtype=Int64), Field(name="loss_severity", dtype=Float32), Field(name="labeler", dtype=String), ], source=PushSource( name="loan_default_labels_push_source", batch_source=FileSource( path="data/loan_defaults.parquet", timestamp_field="event_timestamp", ), ), labeler_field="labeler", conflict_policy=ConflictPolicy.LABELER_PRIORITY, tags={ "feast.io/labeling-method": "table", "feast.io/field-role:is_default": "expectation", }, ) ``` ## Conflict policies When multiple labelers write different values for the same entity, `ConflictPolicy` picks one value for **offline store reads** (training, UI browse): | Policy | When to use | | ------------------ | ------------------------------------------------------------- | | `LAST_WRITE_WINS` | Default. Most recent write wins. | | `LABELER_PRIORITY` | Trusted labelers override others (e.g. human over LLM judge). | | `MAJORITY_VOTE` | Consensus labeling (e.g. multiple labelers on RAG chunks). | {% hint style="info" %} Conflict policies apply to the **offline store** (training). The **online store** always uses last-write-wins. Full label history is always retained in the offline store. {% endhint %} ## Best practices **Name fields by intent.** Use `response_quality` (feedback) and `ground_truth` (expectation) — not generic `score` or `label`. **Tag field roles.** Set `feast.io/field-role:` to `feedback` or `expectation` so your team and UI know what each field means. **Match conflict policy to label type.** Use `LABELER_PRIORITY` when humans correct automated judges. Use `MAJORITY_VOTE` for multi-labeler consensus. Use `LAST_WRITE_WINS` for simple feedback streams. **Link to features.** Set `reference_feature_view` so the UI and documentation show which feature view the labels apply to. **Separate noisy feedback from stable ground truth.** When possible, put expectations in dedicated fields or views with stricter writer conventions (human-only). ## Limitations * Conflict policies are enforced on offline reads only; online store is always last-write-wins. * `LABELER_PRIORITY` requires explicit labeler ordering configuration. * Data labeling profiles (annotation profiles) are UI configuration via tags — not enforced at the SDK write path. ## Next steps * [Feature view](/master/getting-started/concepts/feature-view) — immutable features that label views apply labels to * [Feature retrieval](/master/getting-started/concepts/feature-retrieval) — point-in-time joins for training * [ADR-0012: LabelView](/master/project/adr/adr-0012-label-view) — full design rationale # Feature retrieval ## Overview Generally, Feast supports several patterns of feature retrieval: 1. Training data generation (via `feature_store.get_historical_features(...)`) 2. Offline feature retrieval for batch scoring (via `feature_store.get_historical_features(...)`) 3. Online feature retrieval for real-time model predictions * via the SDK: `feature_store.get_online_features(...)` * via deployed feature server endpoints: `requests.post('http://localhost:6566/get-online-features', data=json.dumps(online_request))` Each of these retrieval mechanisms accept: * some way of specifying entities (to fetch features for) * some way to specify the features to fetch (either via [feature services](#feature-services), which group features needed for a model version, or [feature references](#feature-references)) Before beginning, you need to instantiate a local `FeatureStore` object that knows how to parse the registry (see [more details](https://docs.feast.dev/getting-started/concepts/registry)) For code examples of how the below work, inspect the generated repository from `feast init -t [YOUR TEMPLATE]` (`gcp`, `snowflake`, and `aws` are the most fully fleshed). ## Concepts Before diving into how to retrieve features, we need to understand some high level concepts in Feast. ### Feature Services A feature service is an object that represents a logical group of features from one or more [feature views](/master/getting-started/concepts/feature-view#feature-view). Feature Services allows features from within a feature view to be used as needed by an ML model. Users can expect to create one feature service per model version, allowing for tracking of the features used by models. {% tabs %} {% tab title="driver\_trips\_feature\_service.py" %} ```python from driver_ratings_feature_view import driver_ratings_fv from driver_trips_feature_view import driver_stats_fv driver_stats_fs = FeatureService( name="driver_activity", features=[driver_stats_fv, driver_ratings_fv[["lifetime_rating"]]] ) ``` {% endtab %} {% endtabs %} Feature services are used during * The generation of training datasets when querying feature views in order to find historical feature values. A single training dataset may consist of features from multiple feature views. * Retrieval of features for batch scoring from the offline store (e.g. with an entity dataframe where all timestamps are `now()`) * Retrieval of features from the online store for online inference (with smaller batch sizes). The features retrieved from the online store may also belong to multiple feature views. {% hint style="info" %} Applying a feature service does not result in an actual service being deployed. {% endhint %} Feature services enable referencing all or some features from a feature view. #### Pre-computed feature vectors (`precompute_online`) For latency-critical online serving, you can enable **pre-computed feature vectors** on a feature service. When `precompute_online=True`, Feast stores all of the service's features for each entity as a single serialized blob in the online store. At read time, this reduces the number of store reads from O(N feature views) to O(1), regardless of how many feature views the service spans. ```python # A feature service with pre-computed vectors enabled low_latency_service = FeatureService( name="low_latency_inference", features=[driver_stats_fv, vehicle_stats_fv, route_features_fv], precompute_online=True, ) ``` After running `feast apply`, the pre-computed vectors are automatically built and refreshed whenever you run `feast materialize` or `feast materialize-incremental`. Feast detects which feature services have `precompute_online=True` and rebuilds their vectors for every affected entity after the per-feature-view writes complete. Vectors are also refreshed automatically on `feast push`. {% hint style="info" %} `precompute_online` is **opt-in** — it defaults to `False`. When enabled, the pre-computed path is used exclusively for that service; there is no silent fallback to per-feature-view reads. If vectors are missing or stale, the server raises an error, making problems visible immediately. {% endhint %} {% hint style="warning" %} `precompute_online` is not compatible with on-demand feature views (ODFVs) that have `write_to_online_store=False`. ODFVs with `write_to_online_store=True` are supported since their values are materialized. {% endhint %} See the [performance tuning guide](/master/how-to-guides/online-server-performance-tuning#pre-computed-feature-vectors) for benchmarks and detailed configuration. Retrieving from the online store with a feature service ```python from feast import FeatureStore feature_store = FeatureStore('.') # Initialize the feature store feature_service = feature_store.get_feature_service("driver_activity") features = feature_store.get_online_features( features=feature_service, entity_rows=[entity_dict] ) ``` Retrieving from the offline store with a feature service ```python from feast import FeatureStore feature_store = FeatureStore('.') # Initialize the feature store feature_service = feature_store.get_feature_service("driver_activity") feature_store.get_historical_features(features=feature_service, entity_df=entity_df) ``` ### Feature References This mechanism of retrieving features is only recommended as you're experimenting. Once you want to launch experiments or serve models, feature services are recommended. Feature references uniquely identify feature values in Feast. The structure of a feature reference in string form is as follows: `[@version]:` The `@version` part is optional. When omitted, the latest (active) version is used. You can specify a version like `@v2` to read from a specific historical version snapshot. Feature references are used for the retrieval of features from Feast: ```python online_features = fs.get_online_features( features=[ 'driver_locations:lon', # latest version (default) 'drivers_activity:trips_today', # latest version (default) 'drivers_activity@v2:trips_today', # specific version 'drivers_activity@latest:trips_today', # explicit latest ], entity_rows=[ # {join_key: entity_value} {'driver': 'driver_1001'} ] ) ``` {% hint style="info" %} Version-qualified reads (`@v`) require `enable_online_feature_view_versioning: true` in your registry config and are currently supported only on the SQLite online store. See the [feature view versioning docs](/master/getting-started/concepts/feature-view#version-qualified-feature-references) for details. {% endhint %} It is possible to retrieve features from multiple feature views with a single request, and Feast is able to join features from multiple tables in order to build a training dataset. However, it is not possible to reference (or retrieve) features from multiple projects at the same time. {% hint style="info" %} Note, if you're using [Feature views without entities](/master/getting-started/concepts/feature-view#feature-views-without-entities), then those features can be added here without additional entity values in the `entity_rows` parameter. {% endhint %} ### Event timestamp The timestamp on which an event occurred, as found in a feature view's data source. The event timestamp describes the event time at which a feature was observed or generated. Event timestamps are used during point-in-time joins to ensure that the latest feature values are joined from feature views onto entity rows. Event timestamps are also used to ensure that old feature values aren't served to models during online serving. #### Why `event_timestamp` is required in the entity dataframe When calling `get_historical_features()`, the `entity_df` must include an `event_timestamp` column. This timestamp acts as the **upper bound (inclusive)** for which feature values are allowed to be retrieved for each entity row. Feast performs a point-in-time join (also called a "last known good value" temporal join): for each entity row, it retrieves the latest feature values with a timestamp **at or before** the entity row's `event_timestamp`. This ensures **point-in-time correctness**, which is critical to prevent **data leakage** during model training. Without this constraint, features generated *after* the prediction time could leak into training data—effectively letting the model "see the future"—leading to inflated offline metrics that do not translate to real-world performance. For example, if you want to predict whether a driver will be rated well on April 12 at 10:00 AM, the entity dataframe row should have `event_timestamp = datetime(2021, 4, 12, 10, 0, 0)`. Feast will then only join feature values observed on or before that time, excluding any data generated after 10:00 AM. #### Retrieving features without an entity dataframe While the entity dataframe is the standard way to retrieve historical features, Feast also supports **entity-less historical feature retrieval** by datetime range. This is useful when: * You are training **time-series or population-level models** and don't have a pre-defined list of entity IDs. * You want **all features in a time window** for exploratory analysis or batch training on full history. * Constructing an entity dataframe upfront is unnecessarily complex or expensive. Instead of passing `entity_df`, you specify a time window with `start_date` and/or `end_date`: ```python from datetime import datetime training_df = store.get_historical_features( features=[ "driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate", "driver_hourly_stats:avg_daily_trips", ], start_date=datetime(2025, 7, 1), end_date=datetime(2025, 7, 2), ).to_df() ``` If `start_date` is omitted, it defaults to `end_date` minus the feature view TTL. If `end_date` is omitted, it defaults to the current time. Point-in-time correctness is still preserved. {% hint style="warning" %} Entity-less retrieval is currently supported for the **Postgres**, **Dask**, **Spark**, and **Ray** offline stores. You cannot mix `entity_df` with `start_date`/`end_date` in the same call. {% endhint %} For more details, see the [FAQ](/master/getting-started/faq#how-do-i-run-get_historical_features-without-providing-an-entity-dataframe) and [this blog post on entity-less historical feature retrieval](https://feast.dev/blog/entity-less-historical-features-retrieval/). ### Dataset A dataset is a collection of rows that is produced by a historical retrieval from Feast in order to train a model. A dataset is produced by a join from one or more feature views onto an entity dataframe. Therefore, a dataset may consist of features from multiple feature views. **Dataset vs Feature View:** Feature views contain the schema of data and a reference to where data can be found (through its data source). Datasets are the actual data manifestation of querying those data sources. **Dataset vs Data Source:** Datasets are the output of historical retrieval, whereas data sources are the inputs. One or more data sources can be used in the creation of a dataset. ## Retrieving historical features (for training data or batch scoring) Feast abstracts away point-in-time join complexities with the `get_historical_features` API. We go through the major steps, and also show example code. Note that the quickstart templates generally have end-to-end working examples for all these cases.
Full example: generate training data ```python entity_df = pd.DataFrame.from_dict( { "driver_id": [1001, 1002, 1003, 1004, 1001], "event_timestamp": [ datetime(2021, 4, 12, 10, 59, 42), datetime(2021, 4, 12, 8, 12, 10), datetime(2021, 4, 12, 16, 40, 26), datetime(2021, 4, 12, 15, 1, 12), datetime.now() ] } ) training_df = store.get_historical_features( entity_df=entity_df, features=store.get_feature_service("model_v1"), ).to_df() print(training_df.head()) ```
Full example: retrieve offline features for batch scoring The main difference here compared to training data generation is how to handle timestamps in the entity dataframe. You want to pass in the **current time** to get the latest feature values for all your entities. ```python from feast import FeatureStore store = FeatureStore(repo_path=".") # Get the latest feature values for unique entities entity_sql = f""" SELECT driver_id, CURRENT_TIMESTAMP() as event_timestamp FROM {store.get_data_source("driver_hourly_stats_source").get_table_query_string()} WHERE event_timestamp BETWEEN '2021-01-01' and '2021-12-31' GROUP BY driver_id """ batch_scoring_features = store.get_historical_features( entity_df=entity_sql, features=store.get_feature_service("model_v2"), ).to_df() # predictions = model.predict(batch_scoring_features) ```
### Step 1: Specifying Features Feast accepts either: * [feature services](#feature-services), which group features needed for a model version * [feature references](#feature-references) #### Example: querying a feature service (recommended) ```python training_df = store.get_historical_features( entity_df=entity_df, features=store.get_feature_service("model_v1"), ).to_df() ``` #### Example: querying a list of feature references ```python training_df = store.get_historical_features( entity_df=entity_df, features=[ "driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate", "driver_daily_features:daily_miles_driven" ], ).to_df() ``` ### Step 2: Specifying Entities Feast accepts either a **Pandas dataframe** as the entity dataframe (including entity keys and timestamps) or a **SQL query** to generate the entities. Both approaches must specify the full **entity key** needed as well as the **timestamps**. Feast then joins features onto this dataframe. #### Example: entity dataframe for generating training data ```python entity_df = pd.DataFrame.from_dict( { "driver_id": [1001, 1002, 1003, 1004, 1001], "event_timestamp": [ datetime(2021, 4, 12, 10, 59, 42), datetime(2021, 4, 12, 8, 12, 10), datetime(2021, 4, 12, 16, 40, 26), datetime(2021, 4, 12, 15, 1, 12), datetime.now() ] } ) training_df = store.get_historical_features( entity_df=entity_df, features=[ "driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate", "driver_daily_features:daily_miles_driven" ], ).to_df() ``` #### Example: entity SQL query for generating training data You can also pass a SQL string to generate the above dataframe. This is useful for getting all entities in a timeframe from some data source. ```python entity_sql = f""" SELECT driver_id, event_timestamp FROM {store.get_data_source("driver_hourly_stats_source").get_table_query_string()} WHERE event_timestamp BETWEEN '2021-01-01' and '2021-12-31' """ training_df = store.get_historical_features( entity_df=entity_sql, features=[ "driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate", "driver_daily_features:daily_miles_driven" ], ).to_df() ``` ### Step 3: Choosing an output format `get_historical_features()` returns a `RetrievalJob` object. You can convert it to the format that suits your downstream pipeline: **Data conversion methods** | Method | Returns | When to use | | -------------------------- | ------------------------- | ------------------------------------------------------------------------------- | | `.to_df()` | `pandas.DataFrame` | General-purpose; scikit-learn, XGBoost, statsmodels | | `.to_feast_df()` | `FeastDataFrame` | Feast-native wrapper with engine metadata; preferred for Feast-internal tooling | | `.to_arrow()` | `pyarrow.Table` | Arrow-native pipelines, Polars, DuckDB, zero-copy interchange | | `.to_tensor(kind="torch")` | `Dict[str, torch.Tensor]` | Direct PyTorch training loops; numeric columns become tensors | | `.to_ray_dataset()` | `ray.data.Dataset` | Ray Train, Ray Serve, distributed ML workloads | **Persistence methods** | Method | Effect | When to use | | ---------------------- | ----------------------------------------- | --------------------------------------------------- | | `.persist(storage)` | Writes result to offline storage | Save a training dataset for later reuse or auditing | | `.to_remote_storage()` | Exports result to S3/GCS as Parquet files | Hand off to external systems or data pipelines | #### Retrieving as a Ray Dataset `to_ray_dataset()` is a **first-class method** on every `RetrievalJob`. When the underlying offline store is a `RayOfflineStore`, the dataset is returned directly without a copy through Arrow. For all other offline stores, a zero-copy Arrow → Ray Dataset conversion is used as a fallback. ```python from feast import FeatureStore store = FeatureStore(".") # to_ray_dataset() is a first-class method on the RetrievalJob — chain it # directly after get_historical_features(). ray_ds = store.get_historical_features( entity_df=entity_df, features=["driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate"], ).to_ray_dataset() # Use with Ray Train import ray.train trainer = ray.train.torch.TorchTrainer( train_loop_per_worker=..., datasets={"train": ray_ds}, ) ``` > **Note:** `to_ray_dataset()` requires `feast[ray]` to be installed. *** ## Retrieving online features (for model inference) Feast will ensure the latest feature values for registered features are available. At retrieval time, you need to supply a list of **entities** and the corresponding **features** to be retrieved. Similar to `get_historical_features`, we recommend using feature services as a mechanism for grouping features in a model version. *Note: unlike `get_historical_features`, the `entity_rows`* ***do not need timestamps** since you only want one feature value per entity key.* There are several options for retrieving online features: through the SDK, or through a feature server
Full example: retrieve online features for real-time model inference (Python SDK) ```python from feast import RepoConfig, FeatureStore from feast.repo_config import RegistryConfig repo_config = RepoConfig( registry=RegistryConfig(path="gs://feast-test-gcs-bucket/registry.pb"), project="feast_demo_gcp", provider="gcp", ) store = FeatureStore(config=repo_config) features = store.get_online_features( features=[ "driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate", "driver_daily_features:daily_miles_driven", ], entity_rows=[ { "driver_id": 1001, } ], ).to_dict() ```
Full example: retrieve online features for real-time model inference (Feature Server) This approach requires you to deploy a feature server (see [Python feature server](https://github.com/feast-dev/feast/blob/master/docs/reference/feature-servers/python-feature-server/README.md)). ```python import requests import json online_request = { "features": [ "driver_hourly_stats:conv_rate", ], "entities": {"driver_id": [1001, 1002]}, } r = requests.post('http://localhost:6566/get-online-features', data=json.dumps(online_request)) print(json.dumps(r.json(), indent=4, sort_keys=True)) ```
# Point-in-time joins Feature values in Feast are modeled as time-series records. Below is an example of a driver feature view with two feature columns (`trips_today`, and `earnings_today`): ![](/files/-Mjpv6KpglenWs--FcV_) The above table can be registered with Feast through the following feature view: ```python from feast import Entity, FeatureView, Field, FileSource from feast.types import Float32, Int64 from datetime import timedelta driver = Entity(name="driver", join_keys=["driver_id"]) driver_stats_fv = FeatureView( name="driver_hourly_stats", entities=[driver], schema=[ Field(name="trips_today", dtype=Int64), Field(name="earnings_today", dtype=Float32), ], ttl=timedelta(hours=2), source=FileSource( path="driver_hourly_stats.parquet" ) ) ``` Feast is able to join features from one or more feature views onto an entity dataframe in a point-in-time correct way. This means Feast is able to reproduce the state of features at a specific point in the past. Given the following entity dataframe, imagine a user would like to join the above `driver_hourly_stats` feature view onto it, while preserving the `trip_success` column: ![Entity dataframe containing timestamps, driver ids, and the target variable](/files/-Mjq7HmbXIPkaiRq3AOB) The timestamps within the entity dataframe above are the events at which we want to reproduce the state of the world (i.e., what the feature values were at those specific points in time). In order to do a point-in-time join, a user would load the entity dataframe and run historical retrieval: ```python # Read in entity dataframe entity_df = pd.read_csv("entity_df.csv") training_df = store.get_historical_features( entity_df=entity_df, features = [ 'driver_hourly_stats:trips_today', 'driver_hourly_stats:earnings_today' ], ) ``` For each row within the entity dataframe, Feast will query and join the selected features from the appropriate feature view data source. Feast will scan backward in time from the entity dataframe timestamp up to a maximum of the TTL time specified. ![](/files/-MjqF1Le7DbSBibiUaD-) {% hint style="info" %} Please note that the TTL time is relative to each timestamp within the entity dataframe. TTL is not relative to the current point in time (when you run the query). {% endhint %} Below is the resulting joined training dataframe. It contains both the original entity rows and joined feature values: ![](/files/-MjqyMRdkUdk5681zX_3) Three feature rows were successfully joined to the entity dataframe rows. The first row in the entity dataframe was older than the earliest feature rows in the feature view and could not be joined. The last row in the entity dataframe was outside of the TTL window (the event happened 11 hours after the feature row) and also couldn't be joined. ## Retrieving features as of the event time By default, point-in-time joins only constrain the feature's event timestamp. If a data source also has a `created_timestamp_column`, it is used to deduplicate rows that share an event timestamp (the row with the highest created timestamp wins), but it is not otherwise filtered. This means a value that was backfilled or corrected *after* an entity dataframe timestamp can still be returned for it. To restrict retrieval to feature values that were already available at each entity row's timestamp, pass `filter_by_created_timestamp=True`: ```python training_df = store.get_historical_features( entity_df=entity_df, features = [ 'driver_hourly_stats:trips_today', 'driver_hourly_stats:earnings_today' ], filter_by_created_timestamp=True, ) ``` This adds a `created_timestamp <= entity_timestamp` condition to the join, so each entity dataframe row only sees feature values whose created timestamp is at or before its own timestamp. This is useful to keep backfilled values from leaking into training data, and to reproduce what the online store would have served at each event time (assuming the created timestamp reflects when the value became available online). {% hint style="info" %} Rows with a NULL created timestamp are excluded when the flag is enabled, so the column should be non-null. Not all offline stores support this flag yet; unsupported stores raise an error rather than silently ignoring it. {% endhint %} # \[Alpha] Saved dataset Feast datasets allow for conveniently saving dataframes that include both features and entities to be subsequently used for data analysis and model training. Data Quality Monitoring was the original motivation for creating the dataset concept. Dataset's metadata is stored in the Feast registry and raw data (features, entities, additional input keys and timestamp) is stored in the [offline store](/master/getting-started/components/offline-store). Dataset can be created from: 1. Results of historical retrieval 2. \[planned] Logging request (including input for [on demand transformation](/master/reference/beta-on-demand-feature-view)) and response during feature serving 3. \[planned] Logging features during writing to online store (from batch source or stream) ### Creating a saved dataset from historical retrieval To create a saved dataset from historical features for later retrieval or analysis, a user needs to call `get_historical_features` method first and then pass the returned retrieval job to `create_saved_dataset` method. `create_saved_dataset` will trigger the provided retrieval job (by calling `.persist()` on it) to store the data using the specified `storage` behind the scenes. Storage type must be the same as the globally configured offline store (e.g it's impossible to persist data to a different offline source). `create_saved_dataset` will also create a `SavedDataset` object with all of the related metadata and will write this object to the registry. ```python from feast import FeatureStore from feast.infra.offline_stores.bigquery_source import SavedDatasetBigQueryStorage store = FeatureStore() historical_job = store.get_historical_features( features=["driver:avg_trip"], entity_df=..., ) dataset = store.create_saved_dataset( from_=historical_job, name='my_training_dataset', storage=SavedDatasetBigQueryStorage(table_ref='..my_training_dataset'), tags={'author': 'oleksii'} ) dataset.to_df() ``` Saved dataset can be retrieved later using the `get_saved_dataset` method in the feature store: ```python dataset = store.get_saved_dataset('my_training_dataset') dataset.to_df() ``` *** Check out our [tutorial on validating historical features](https://github.com/feast-dev/feast/tree/master/docs/tutorials/validating-historical-features.md) to see how this concept can be applied in a real-world use case. # Permission ## Overview The Feast permissions model allows to configure granular permission policies to all the resources defined in a feature store. The configured permissions are stored in the Feast registry and accessible through the CLI and the registry APIs. The permission authorization enforcement is performed when requests are executed through one of the Feast (Python) servers * The online feature server (REST) * The offline feature server (Arrow Flight) * The registry server (gRPC) Note that there is no permission enforcement when accessing the Feast API with a local provider. ## Concepts The permission model is based on the following components: * A `resource` is a Feast object that we want to secure against unauthorized access. * We assume that the resource has a `name` attribute and optional dictionary of associated key-value `tags`. * An `action` is a logical operation executed on the secured resource, like: * `create`: Create an instance. * `describe`: Access the instance state. * `update`: Update the instance state. * `delete`: Delete an instance. * `read`: Read both online and offline stores. * `read_online`: Read the online store. * `read_offline`: Read the offline store. * `write`: Write on any store. * `write_online`: Write to the online store. * `write_offline`: Write to the offline store. * A `policy` identifies the rule for enforcing authorization decisions on secured resources, based on the current user. * A default implementation is provided for role-based policies, using the user roles to grant or deny access to the requested actions on the secured resources. The `Permission` class identifies a single permission configured on the feature store and is identified by these attributes: * `name`: The permission name. * `types`: The list of protected resource types. Defaults to all managed types, e.g. the `ALL_RESOURCE_TYPES` alias. All sub-classes are included in the resource match. * `name_patterns`: A list of regex patterns to match resource names. If any regex matches, the `Permission` policy is applied. Defaults to `[]`, meaning no name filtering is applied. * `required_tags`: Dictionary of key-value pairs that must match the resource tags. Defaults to `None`, meaning that no tags filtering is applied. * `actions`: The actions authorized by this permission. Defaults to `ALL_VALUES`, an alias defined in the `action` module. * `policy`: The policy to be applied to validate a client request. To simplify configuration, several constants are defined to streamline the permissions setup: * In module `feast.feast_object`: * `ALL_RESOURCE_TYPES` is the list of all the `FeastObject` types. * `ALL_FEATURE_VIEW_TYPES` is the list of all the feature view types, including those not inheriting from `FeatureView` type like `OnDemandFeatureView`. * In module `feast.permissions.action`: * `ALL_ACTIONS` is the list of all managed actions. * `READ` includes all the read actions for online and offline store. * `WRITE` includes all the write actions for online and offline store. * `CRUD` includes all the state management actions to create, describe, update or delete a Feast resource. Given the above definitions, the feature store can be configured with granular control over each resource, enabling partitioned access by teams to meet organizational requirements for service and data sharing, and protection of sensitive information. The `feast` CLI includes a new `permissions` command to list the registered permissions, with options to identify the matching resources for each configured permission and the existing resources that are not covered by any permission. {% hint style="info" %} **Note**: Feast resources that do not match any of the configured permissions are not secured by any authorization policy, meaning any user can execute any action on such resources. {% endhint %} ## Definition examples This permission definition grants access to the resource state and the ability to read all of the stores for any feature view or feature service to all users with the role `super-reader`: ```py Permission( name="feature-reader", types=[FeatureView, FeatureService], policy=RoleBasedPolicy(roles=["super-reader"]), actions=[AuthzedAction.DESCRIBE, *READ], ) ``` This example grants permission to write on all the data sources with `risk_level` tag set to `high` only to users with role `admin` or `data_team`: ```py Permission( name="ds-writer", types=[DataSource], required_tags={"risk_level": "high"}, policy=RoleBasedPolicy(roles=["admin", "data_team"]), actions=[AuthzedAction.WRITE], ) ``` {% hint style="info" %} **Note**: When using multiple roles in a role-based policy, the user must be granted at least one of the specified roles. {% endhint %} The following permission grants authorization to read the offline store of all the feature views including `risky` in the name, to users with role `trusted`: ```py Permission( name="reader", types=[FeatureView], name_patterns=".*risky.*", # Accepts both `str` or `list[str]` types policy=RoleBasedPolicy(roles=["trusted"]), actions=[AuthzedAction.READ_OFFLINE], ) ``` ## Permission granting order When mixing and matching policies in permissions script, the permission granting order is as follows: 1. The first matching policy wins in the list of policies and the permission is granted based on the matching policy rules and rest policies are ignored. 2. If any policy matches from the list of policies, the permission is granted based on the matching policy rules and rest policies are ignored 3. If no policy matches, the permission is denied ## Authorization configuration In order to leverage the permission functionality, the `auth` section is needed in the `feature_store.yaml` configuration. Currently, Feast supports OIDC and Kubernetes RBAC authorization protocols. The default configuration, if you don't specify the `auth` configuration section, is `no_auth`, indicating that no permission enforcement is applied. The `auth` section includes a `type` field specifying the actual authorization protocol, and protocol-specific fields that are specified in [Authorization Manager](/master/getting-started/components/authz_manager). # Tags ## Overview Tags in Feast allow for efficient filtering of Feast objects when listing them in the UI, CLI, or querying the registry directly. The way to define tags on the feast objects is through the definition file or directly in the object that will be applied to the feature store. ## Examples In this example we define a Feature View in a definition file that has a tag: ```python driver_stats_fv = FeatureView( name="driver_hourly_stats", entities=[driver], ttl=timedelta(days=1), schema=[ Field(name="conv_rate", dtype=Float32), Field(name="acc_rate", dtype=Float32), Field(name="avg_daily_trips", dtype=Int64, description="Average daily trips"), ], online=True, source=driver_stats_source, # Tags are user defined key/value pairs that are attached to each # feature view tags={"team": "driver_performance"}, ) ``` In this example we define a Stream Feature View that has a tag, in the code: ```python sfv = StreamFeatureView( name="test kafka stream feature view", entities=[entity], schema=[], description="desc", timestamp_field="event_timestamp", source=stream_source, tags={"team": "driver_performance"}, ``` An example of filtering feature-views with the tag `team:driver_performance`: ```commandline $ feast feature-views list --tags team:driver_performance NAME ENTITIES TYPE driver_hourly_stats {'driver'} FeatureView driver_hourly_stats_fresh {'driver'} FeatureView ``` The same example of listing feature-views without tag filtering: ```commandline $ feast feature-views list NAME ENTITIES TYPE driver_hourly_stats {'driver'} FeatureView driver_hourly_stats_fresh {'driver'} FeatureView transformed_conv_rate_fresh {'driver'} OnDemandFeatureView transformed_conv_rate {'driver'} OnDemandFeatureView ``` # Use Cases This page covers common use cases for Feast and how a feature store can benefit your AI/ML workflows. ## Recommendation Engines Recommendation engines require personalized feature data related to users, items, and their interactions. Feast can help by: * **Managing feature data**: Store and serve user preferences, item characteristics, and interaction history * **Low-latency serving**: Provide real-time features for dynamic recommendations * **Point-in-time correctness**: Ensure training and serving data are consistent to avoid data leakage * **Feature reuse**: Allow different recommendation models to share the same feature definitions ### Example: User-Item Recommendations A typical recommendation engine might need features such as: * User features: demographics, preferences, historical behavior * Item features: categories, attributes, popularity scores * Interaction features: past user-item interactions, ratings Feast allows you to define these features once and reuse them across different recommendation models, ensuring consistency between training and serving environments. {% content-ref url="/pages/F7HjJ0aZCcjIwOvVYPYk" %} [Driver ranking](/master/tutorials/tutorials-overview/driver-ranking-with-feast) {% endcontent-ref %} ## Risk Scorecards Risk scorecards (such as credit risk, fraud risk, and marketing propensity models) require a comprehensive view of entity data with historical contexts. Feast helps by: * **Feature consistency**: Ensure all models use the same feature definitions * **Historical feature retrieval**: Generate training datasets with correct point-in-time feature values * **Feature monitoring**: Track feature distributions to detect data drift * **Governance**: Maintain an audit trail of features used in regulated environments ### Example: Credit Risk Scoring Credit risk models might use features like: * Transaction history patterns * Account age and status * Payment history features * External credit bureau data * Employment and income verification Feast enables you to combine these features from disparate sources while maintaining data consistency and freshness. {% content-ref url="/pages/x5xrWxUU8BSXC4IZIy0P" %} [Real-time credit scoring on AWS](/master/tutorials/tutorials-overview/real-time-credit-scoring-on-aws) {% endcontent-ref %} {% content-ref url="/pages/NDGOfA9wrV27tKF0iazj" %} [Fraud detection on GCP](/master/tutorials/tutorials-overview/fraud-detection) {% endcontent-ref %} ## NLP / RAG / Information Retrieval Natural Language Processing (NLP) and Retrieval Augmented Generation (RAG) applications require efficient storage and retrieval of text embeddings. Feast supports these use cases by: * **Vector storage**: Store and index embedding vectors for efficient similarity search * **Document metadata**: Associate embeddings with metadata for contextualized retrieval * **Scaling retrieval**: Serve vectors with low latency for real-time applications * **Versioning**: Track changes to embedding models and document collections ### Example: Retrieval Augmented Generation RAG systems can leverage Feast to: * Store document embeddings and chunks in a vector database * Retrieve contextually relevant documents for user queries * Combine document retrieval with entity-specific features * Scale to large document collections Feast makes it remarkably easy to make data available for retrieval by providing a simple API for both storing and querying vector embeddings. {% content-ref url="/pages/46Xekrzpns4mDIx4ZZyA" %} [Retrieval Augmented Generation (RAG) with Feast](/master/tutorials/rag-with-docling) {% endcontent-ref %} ## Time Series Forecasting Time series forecasting for demand planning, inventory management, and anomaly detection benefits from Feast through: * **Temporal feature management**: Store and retrieve time-bound features * **Feature engineering**: Create time-based aggregations and transformations * **Consistent feature retrieval**: Ensure training and inference use the same feature definitions * **Backfilling capabilities**: Generate historical features for model training ### Example: Demand Forecasting Demand forecasting applications typically use features such as: * Historical sales data with temporal patterns * Seasonal indicators and holiday flags * Weather data * Price changes and promotions * External economic indicators Feast allows you to combine these diverse data sources and make them available for both batch training and online inference. ## Image and Multi-Modal Processing While Feast was initially built for structured data, it can also support multi-modal applications by: * **Storing feature metadata**: Keep track of image paths, embeddings, and metadata * **Vector embeddings**: Store image embeddings for similarity search * **Feature fusion**: Combine image features with structured data features ## Why Feast Is Impactful Across all these use cases, Feast provides several core benefits: 1. **Consistency between training and serving**: Eliminate training-serving skew by using the same feature definitions 2. **Feature reuse**: Define features once and use them across multiple models 3. **Scalable feature serving**: Serve features at low latency for production applications 4. **Feature governance**: Maintain a central registry of feature definitions with metadata 5. **Data freshness**: Keep online features up-to-date with batch and streaming ingestion 6. **Reduced operational complexity**: Standardize feature access patterns across models By implementing a feature store with Feast, teams can focus on model development rather than data engineering challenges, accelerating the delivery of ML applications to production. # Components {% content-ref url="/pages/-MhvbE3x2Qt82NZ0V1S7" %} [Registry](/master/getting-started/components/registry) {% endcontent-ref %} {% content-ref url="/pages/-Mca7yQLP8ZkAg69Li9\_" %} [Offline store](/master/getting-started/components/offline-store) {% endcontent-ref %} {% content-ref url="/pages/-Mca7vtNzNdXSwg5Jd4t" %} [Online store](/master/getting-started/components/online-store) {% endcontent-ref %} {% content-ref url="/pages/fP4XRhVB2XcEpR6z0ebH" %} [Feature server](/master/getting-started/components/feature-server) {% endcontent-ref %} {% content-ref url="/pages/dWMHsUK2Ub4eHr5vbGnZ" %} [Compute Engine](/master/getting-started/components/compute-engine) {% endcontent-ref %} {% content-ref url="/pages/qPcIV7c6C1jJt3uoSDzN" %} [Stream Processor](/master/getting-started/components/stream-processor) {% endcontent-ref %} {% content-ref url="/pages/-Mca7zze95F6fhgoiIeJ" %} [Provider](/master/getting-started/components/provider) {% endcontent-ref %} {% content-ref url="/pages/WBCaYZPGv5RELd4j8kJh" %} [Authorization Manager](/master/getting-started/components/authz_manager) {% endcontent-ref %} {% content-ref url="/pages/fUbN3Do2FBTf5Twc2TJE" %} [OpenTelemetry Integration](/master/getting-started/components/open-telemetry) {% endcontent-ref %} # Overview ![Feast Architecture Diagram](/files/-MYIg-9V1lQ5bpkxeGIB) ## Functionality * **Create Batch Features:** ELT/ETL systems like Spark and SQL are used to transform data in the batch store. * **Create Stream Features:** Stream features are created from streaming services such as Kafka or Kinesis, and can be pushed directly into Feast via the [Push API](/master/reference/data-sources/push). * **Feast Apply:** The user (or CI) publishes versioned controlled feature definitions using `feast apply`. This CLI command updates infrastructure and persists definitions in the object store registry. * **Feast Materialize:** The user (or scheduler) executes `feast materialize` (with timestamps or `--disable-event-timestamp` to materialize all data with current timestamps) which loads features from the offline store into the online store. * **Model Training:** A model training pipeline is launched. It uses the Feast Python SDK to retrieve a training dataset that can be used for training models. * **Get Historical Features:** Feast exports a point-in-time correct training dataset based on the list of features and entity dataframe provided by the model training pipeline. * **Deploy Model:** The trained model binary (and list of features) are deployed into a model serving system. This step is not executed by Feast. * **Prediction:** A backend system makes a request for a prediction from the model serving service. * **Get Online Features:** The model serving service makes a request to the Feast Online Serving service for online features using a Feast SDK. * **Feature Retrieval:** The online serving service retrieves the latest feature values from the online store and returns them to the model serving service. ## Components A complete Feast deployment contains the following components: * **Feast Registry**: An object store (GCS, S3) based registry used to persist feature definitions that are registered with the feature store. Systems can discover feature data by interacting with the registry through the Feast SDK. * **Feast Python SDK/CLI:** The primary user facing SDK. Used to: * Manage version controlled feature definitions. * Materialize (load) feature values into the online store. * Build and retrieve training datasets from the offline store. * Retrieve online features. * **Feature Server:** The Feature Server is a REST API server that serves feature values for a given entity key and feature reference. The Feature Server is designed to be horizontally scalable and can be deployed in a distributed manner. * **Stream Processor:** The Stream Processor can be used to ingest feature data from streams and write it into the online or offline stores. Currently, there's an experimental Spark processor that's able to consume data from Kafka. * **Compute Engine:** The [Compute Engine](/master/getting-started/components/compute-engine) component launches a process which loads data into the online store from the offline store. By default, Feast uses a local in-process engine implementation to materialize data. However, additional infrastructure can be used for a more scalable materialization process. * **Online Store:** The online store is a database that stores only the latest feature values for each entity. The online store is either populated through materialization jobs or through [stream ingestion](/master/reference/data-sources/push). * **Offline Store:** The offline store persists batch data that has been ingested into Feast. This data is used for producing training datasets. For feature retrieval and materialization, Feast does not manage the offline store directly, but runs queries against it. However, offline stores can be configured to support writes if Feast configures logging functionality of served features. * **Authorization Manager**: The authorization manager detects authentication tokens from client requests to Feast servers and uses this information to enforce permission policies on the requested services. # Registry The Feast feature registry is a central catalog of all feature definitions and their related metadata. Feast uses the registry to store all applied Feast objects (e.g. Feature views, entities, etc). It allows data scientists to search, discover, and collaborate on new features. The registry exposes methods to apply, list, retrieve and delete these objects, and is an abstraction with multiple implementations. Feast comes with built-in file-based and sql-based registry implementations. By default, Feast uses a file-based registry, which stores the protobuf representation of the registry as a serialized file in the local file system. For more details on which registries are supported, please see [Registries](/master/reference/registries). ## Updating the registry We recommend users store their Feast feature definitions in a version controlled repository, which then via CI/CD automatically stays synced with the registry. Users will often also want multiple registries to correspond to different environments (e.g. dev vs staging vs prod), with staging and production registries with locked down write access since they can impact real user traffic. See [Running Feast in Production](https://docs.feast.dev/master/getting-started/components/pages/-Mf00Obp_QcDmpvQPAf2#1.-automatically-deploying-changes-to-your-feature-definitions) for details on how to set this up. ## Deleting objects from the registry {% hint style="warning" %} Simply removing a feature definition from your code and running `feast apply` or `FeatureStore.apply()` **does not** delete the object from the registry. You must explicitly delete objects using the dedicated delete methods or CLI commands. {% endhint %} ### Using the CLI The simplest way to delete objects is using the `feast delete` command: ```bash # Delete any Feast object by name feast delete my_feature_view feast delete my_entity feast delete my_feature_service ``` See the [CLI documentation](/master/reference/feast-cli-commands#delete) for more details. ### Using the Python SDK To delete objects programmatically, use the explicit delete methods provided by the `FeatureStore` class: #### Deleting feature views ```python from feast import FeatureStore store = FeatureStore(repo_path=".") store.delete_feature_view("my_feature_view") ``` #### Deleting feature services ```python store.delete_feature_service("my_feature_service") ``` #### Deleting entities, data sources, and other objects For entities, data sources, and other registry objects, you can use the registry methods directly: ```python # Delete an entity store._registry.delete_entity("my_entity", project=store.project) # Delete a data source store._registry.delete_data_source("my_data_source", project=store.project) # Delete a saved dataset store._registry.delete_saved_dataset("my_saved_dataset", project=store.project) # Delete a validation reference store._registry.delete_validation_reference("my_validation_reference", project=store.project) ``` {% hint style="info" %} When using `feast apply` via the CLI, you can also use the `objects_to_delete` parameter with `partial=False` to delete objects as part of the apply operation. However, this is less common and typically used in automated deployment scenarios. {% endhint %} ## Accessing the registry from clients Users can specify the registry through a `feature_store.yaml` config file, or programmatically. We often see teams preferring the programmatic approach because it makes notebook driven development very easy: ### Option 1: programmatically specifying the registry ```python repo_config = RepoConfig( registry=RegistryConfig(path="gs://feast-test-gcs-bucket/registry.pb"), project="feast_demo_gcp", provider="gcp", offline_store="file", # Could also be the OfflineStoreConfig e.g. FileOfflineStoreConfig online_store="null", # Could also be the OnlineStoreConfig e.g. RedisOnlineStoreConfig ) store = FeatureStore(config=repo_config) ``` ### Option 2: specifying the registry in the project's `feature_store.yaml` file ```yaml project: feast_demo_aws provider: aws registry: s3://feast-test-s3-bucket/registry.pb online_store: null offline_store: type: file ``` Instantiating a `FeatureStore` object can then point to this: ```python store = FeatureStore(repo_path=".") ``` {% hint style="info" %} The file-based feature registry is a [Protobuf representation](https://github.com/feast-dev/feast/blob/master/protos/feast/core/Registry.proto) of Feast metadata. This Protobuf file can be read programmatically from other programming languages, but no compatibility guarantees are made on the internal structure of the registry. {% endhint %} # Offline store An offline store is an interface for working with historical time-series feature values that are stored in [data sources](/master/getting-started/concepts/data-ingestion). The `OfflineStore` interface has several different implementations, such as the `BigQueryOfflineStore`, each of which is backed by a different storage and compute engine. For more details on which offline stores are supported, please see [Offline Stores](/master/reference/offline-stores). Offline stores are primarily used for two reasons: 1. Building training datasets from time-series features. 2. Materializing (loading) features into an online store to serve those features at low-latency in a production setting. Offline stores are configured through the [feature\_store.yaml](/master/reference/feature-repository/feature-store-yaml). When building training datasets or materializing features into an online store, Feast will use the configured offline store with your configured data sources to execute the necessary data operations. Only a single offline store can be used at a time. Moreover, offline stores are not compatible with all data sources; for example, the `BigQuery` offline store cannot be used to query a file-based data source. Please see [Push Source](/master/reference/data-sources/push) for more details on how to push features directly to the offline store in your feature store. # Online store Feast uses online stores to serve features at low latency. Feature values are loaded from data sources into the online store through *materialization*, which can be triggered through the `materialize` command (either with specific timestamps or using `--disable-event-timestamp` to materialize all data with current timestamps). The storage schema of features within the online store mirrors that of the original data source. One key difference is that for each [entity key](/master/getting-started/concepts/entity), only the latest feature values are stored. No historical values are stored. Here is an example batch data source: ![](/files/-MaKw4Bm7IddTPx8VX55) Once the above data source is materialized into Feast (using `feast materialize` with timestamps or `feast materialize --disable-event-timestamp`), the feature values will be stored as follows: ![](/files/-MaKwD6t0z51_Mr6N9M7) Features can also be written directly to the online store via [push sources](/master/reference/data-sources/push) . # Feature server The Feature Server is a core architectural component in Feast, designed to provide low-latency feature retrieval and updates for machine learning applications. It is a REST API server built using [FastAPI](https://fastapi.tiangolo.com/) and exposes a limited set of endpoints to serve features, push data, and support materialization operations. The server is scalable, flexible, and designed to work seamlessly with various deployment environments, including local setups and cloud-based systems. ## Motivation In machine learning workflows, real-time access to feature values is critical for enabling low-latency predictions. The Feature Server simplifies this requirement by: 1. **Serving Features:** Allowing clients to retrieve feature values for specific entities in real-time, reducing the complexity of direct interactions with the online store. 2. **Data Integration:** Providing endpoints to push feature data directly into the online or offline store, ensuring data freshness and consistency. 3. **Scalability:** Supporting horizontal scaling to handle high request volumes efficiently. 4. **Standardized API:** Exposing HTTP/JSON endpoints that integrate seamlessly with various programming languages and ML pipelines. 5. **Secure Communication:** Supporting TLS (SSL) for secure data transmission in production environments. ## Architecture The Feature Server operates as a stateless service backed by two key components: * [**Online Store**](/master/getting-started/components/online-store)**:** The primary data store used for low-latency feature retrieval. * [**Registry**](/master/getting-started/components/registry)**:** The metadata store that defines feature sets, feature views, and their relationships to entities. ## Key Features 1. **RESTful API:** Provides standardized endpoints for feature retrieval and data pushing. 2. **CLI Integration:** Easily managed through the Feast CLI with commands like `feast serve`. 3. **Flexible Deployment:** Can be deployed locally, via Docker, or on Kubernetes using Helm charts. 4. **Scalability:** Designed for distributed deployments to handle large-scale workloads. 5. **TLS Support:** Ensures secure communication in production setups. ## Endpoints Overview | Endpoint | Description | | ------------------------------- | ----------------------------------------------------------------------- | | `/get-online-features` | Retrieves feature values for specified entities and feature references. | | `/push` | Pushes feature data to the online and/or offline store. | | `/materialize` | Materializes features within a specific time range to the online store. | | `/materialize-incremental` | Incrementally materializes features up to the current timestamp. | | `/search` | Vector similarity search for RAG (Alpha endpoint) | | `/v1/vector_stores/{id}/search` | OpenAI-compatible vector search with server-side embedding | | `/retrieve-online-documents` | **Deprecated.** Use `/search` instead. | | `/docs` | API Contract for available endpoints | # Compute Engine Note: The materialization is now constructed via unified compute engine interface. A Compute Engine in Feast is a component that handles materialization and historical retrieval tasks. It is responsible for executing the logic defined in feature views, such as aggregations, transformations, and custom user-defined functions (UDFs). A materialization task abstracts over specific technologies or frameworks that are used to materialize data. It allows users to use a pure local serialized approach (which is the default LocalComputeEngine), or delegates the materialization to separate components (e.g. AWS Lambda, as implemented by the LambdaComputeEngine). If the built-in engines are not sufficient, you can create your own custom materialization engine. Please see [this guide](/master/how-to-guides/customizing-feast/creating-a-custom-compute-engine) for more details. Please see [feature\_store.yaml](/master/reference/feature-repository/feature-store-yaml#overview) for configuring engines. ### Supported Compute Engines ```markdown | Compute Engine | Description | Supported | Link | |-------------------------|-------------------------------------------------------------------------------------------------|------------|------| | LocalComputeEngine | Runs on Arrow + Pandas/Polars/Dask etc., designed for light weight transformation. | ✅ | | | SparkComputeEngine | Runs on Apache Spark, designed for large-scale distributed feature generation. | ✅ | | | SnowflakeComputeEngine | Runs on Snowflake, designed for scalable feature generation using Snowflake SQL. | ✅ | | | LambdaComputeEngine | Runs on AWS Lambda, designed for serverless feature generation. | ✅ | | | FlinkComputeEngine | Runs on Apache Flink, designed for distributed feature generation through PyFlink Table API. | ✅ | | | RayComputeEngine | Runs on Ray, designed for distributed feature generation and machine learning workloads. | ✅ | | ``` ### Batch Engine Batch Engine Config can be configured in the `feature_store.yaml` file, and it serves as the default configuration for all materialization and historical retrieval tasks. The `batch_engine` config in BatchFeatureView. E.g ```yaml batch_engine: type: spark.engine config: spark_master: "local[*]" spark_app_name: "Feast Batch Engine" spark_conf: spark.sql.shuffle.partitions: 100 spark.executor.memory: "4g" ``` in BatchFeatureView. ```python from feast import BatchFeatureView fv = BatchFeatureView( batch_engine={ "spark_conf": { "spark.sql.shuffle.partitions": 200, "spark.executor.memory": "8g" }, } ) ``` Then, when you materialize the feature view, it will use the batch\_engine configuration specified in the feature view, which has shuffle partitions set to 200 and executor memory set to 8g. ### Stream Engine Stream Engine Config can be configured in the `feature_store.yaml` file, and it serves as the default configuration for all stream materialization and historical retrieval tasks. The `stream_engine` config in FeatureView. E.g ```yaml stream_engine: type: spark.engine config: spark_master: "local[*]" spark_app_name: "Feast Stream Engine" spark_conf: spark.sql.shuffle.partitions: 100 spark.executor.memory: "4g" ``` ```python from feast import StreamFeatureView fv = StreamFeatureView( stream_engine={ "spark_conf": { "spark.sql.shuffle.partitions": 200, "spark.executor.memory": "8g" }, } ) ``` Then, when you materialize the feature view, it will use the stream\_engine configuration specified in the feature view, which has shuffle partitions set to 200 and executor memory set to 8g. ### API The compute engine builds the execution plan in a DAG format named FeatureBuilder. It derives feature generation from Feature View definitions including: ``` 1. Transformation (via Transformation API) 2. Aggregation (via Aggregation API) 3. Join (join with entity datasets, customized JOIN or join with another Feature View) 4. Filter (Point in time filter, ttl filter, filter by custom expression) ... ``` ### Components The compute engine is responsible for executing the materialization and retrieval tasks defined in the feature views. It builds a directed acyclic graph (DAG) of operations that need to be performed to generate the features. The Core components of the compute engine are: #### Feature Builder The Feature builder is responsible for resolving the features from the feature views and executing the operations defined in the DAG. It handles the execution of transformations, aggregations, joins, and filters. #### Feature Resolver The Feature resolver is the core component of the compute engine that constructs the execution plan for feature generation. It takes the definitions from feature views and builds a directed acyclic graph (DAG) of operations that need to be performed to generate the features. #### DAG The DAG represents the directed acyclic graph of operations that need to be performed to generate the features. It contains nodes for each operation, such as transformations, aggregations, joins, and filters. The DAG is built by the Feature Resolver and executed by the Feature Builder. DAG nodes are defined as follows: ``` +---------------------+ | SourceReadNode | <- Read data from offline store (e.g. Snowflake, BigQuery, etc. or custom source) +---------------------+ | v +--------------------------------------+ | TransformationNode / JoinNode (*) | <- Merge data sources, custom transformations by user, or default join +--------------------------------------+ | v +---------------------+ | FilterNode | <- used for point-in-time filtering +---------------------+ | v +---------------------+ | AggregationNode (*) | <- only if aggregations are defined +---------------------+ | v +---------------------+ | DeduplicationNode | <- used if no aggregation and for history +---------------------+ retrieval | v +---------------------+ | ValidationNode (*) | <- optional validation checks +---------------------+ | v +----------+ | Output | +----------+ / \ v v +----------------+ +----------------+ | OnlineStoreWrite| OfflineStoreWrite| +----------------+ +----------------+ ``` # Stream Processor A Stream Processor is responsible for consuming data from stream sources (such as Kafka, Kinesis, etc.) and loading it directly into the online (and optionally the offline store). A Stream Processor abstracts over specific technologies or frameworks that are used to materialize data. An experimental Spark Processor for Kafka is available in Feast. If the built-in processor is not sufficient, you can create your own custom processor. Please see [this tutorial](/master/tutorials/building-streaming-features) for more details. # Provider A provider is an implementation of a feature store using specific feature store components (e.g. offline store, online store) targeting a specific environment (e.g. GCP stack). Providers orchestrate various components (offline store, online store, infrastructure, compute) inside an environment. For example, the `gcp` provider supports [BigQuery](https://cloud.google.com/bigquery) as an offline store and [Datastore](https://cloud.google.com/datastore) as an online store, ensuring that these components can work together seamlessly. Feast has three built-in providers (`local`, `gcp`, and `aws`) with default configurations that make it easy for users to start a feature store in a specific environment. These default configurations can be overridden easily. For instance, you can use the `gcp` provider but use Redis as the online store instead of Datastore. If the built-in providers are not sufficient, you can create your own custom provider. Please see [this guide](/master/how-to-guides/customizing-feast/creating-a-custom-provider) for more details. Please see [feature\_store.yaml](/master/reference/feature-repository/feature-store-yaml#overview) for configuring providers. # Authorization Manager An Authorization Manager is an instance of the `AuthManager` class that is plugged into one of the Feast servers to extract user details from the current request and inject them into the [permission](/master/getting-started/concepts/permission) framework. {% hint style="info" %} **Note**: Feast does not provide authentication capabilities; it is the client's responsibility to manage the authentication token and pass it to the Feast server, which then validates the token and extracts user details from the configured authentication server. {% endhint %} Two authorization managers are supported out-of-the-box: * One using a configurable OIDC server to extract the user details. * One using the Kubernetes RBAC resources to extract the user details. These instances are created when the Feast servers are initialized, according to the authorization configuration defined in their own `feature_store.yaml`. Feast servers and clients must have consistent authorization configuration, so that the client proxies can automatically inject the authorization tokens that the server can properly identify and use to enforce permission validations. ## Design notes The server-side implementation of the authorization functionality is defined [here](https://github.com/feast-dev/feast/tree/master/sdk/python/feast/permissions/server/README.md). Few of the key models, classes to understand the authorization implementation on the client side can be found [here](https://github.com/feast-dev/feast/tree/master/sdk/python/feast/permissions/client/README.md). ## Configuring Authorization The authorization is configured using a dedicated `auth` section in the `feature_store.yaml` configuration. **Note**: As a consequence, when deploying the Feast servers with the Helm [charts](https://github.com/feast-dev/feast/tree/master/infra/charts/feast-feature-server/README.md), the `feature_store_yaml_base64` value must include the `auth` section to specify the authorization configuration. ### No Authorization This configuration applies the default `no_auth` authorization: ```yaml project: my-project auth: type: no_auth ... ``` ### OIDC Authorization With OIDC authorization, the Feast client proxies retrieve the JWT token from an OIDC server (or [Identity Provider](https://openid.net/developers/how-connect-works/)) and append it in every request to a Feast server, using an [Authorization Bearer Token](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication#bearer). The server, in turn, uses the same OIDC server to validate the token and extract user details — including username, roles, and groups — from the token itself. Some assumptions are made in the OIDC server configuration: * The OIDC token refers to a client with roles matching the RBAC roles of the configured `Permission`s (\*) * The roles are exposed in the access token under `resource_access..roles` (Keycloak) or in the top-level `roles` claim (Entra ID app roles). Roles found in both are merged. * The JWT token is expected to have a verified signature and not be expired. The Feast OIDC token parser logic validates for `verify_signature` and `verify_exp` so make sure that the given OIDC provider is configured to meet these requirements. The token's audience and issuer claims are **not** verified by default; both checks can be enabled with the `audience` and `issuer` options (see [Server-Side Configuration](#server-side-configuration)). * The username is read from the first of `preferred_username`, `upn`, `azp`, `appid`, `sub` present in the token. Entra ID client-credentials (app-only) tokens carry no user claim, so they authenticate as the calling application. * For `GroupBasedPolicy` support, the `groups` claim should be present in the access token (requires a "Group Membership" protocol mapper in Keycloak). * **Entra ID limitation**: Group claims use object IDs (GUIDs) instead of names, and are omitted entirely when a user exceeds the group overage threshold. GroupBasedPolicy must reference GUIDs and cannot be used for principals with large group memberships. (\*) Please note that **the role match is case-sensitive**, e.g. the name of the role in the OIDC server and in the `Permission` configuration must be exactly the same. For example, the access token for a client `app` of a user with `reader` role and membership in the `data-team` group should have the following claims: ```json { "preferred_username": "alice", "resource_access": { "app": { "roles": [ "reader" ] } }, "groups": [ "data-team" ] } ``` A Microsoft Entra ID (Azure AD) client-credentials (app-only) token has no user claim; the application authenticates as itself, and its app roles arrive in the top-level `roles` claim: ```json { "azp": "11111111-2222-3333-4444-555555555555", "roles": [ "reader" ] } ``` #### Server-Side Configuration The server requires `auth_discovery_url` and `client_id` to validate incoming JWT tokens via JWKS: ```yaml project: my-project auth: type: oidc client_id: _CLIENT_ID_ auth_discovery_url: _OIDC_SERVER_URL_/realms/master/.well-known/openid-configuration ... ``` When the OIDC provider uses a self-signed or untrusted TLS certificate (e.g. internal Keycloak on OpenShift), set `verify_ssl` to `false` to disable certificate verification: ```yaml auth: type: oidc client_id: _CLIENT_ID_ auth_discovery_url: https://keycloak.internal/realms/master/.well-known/openid-configuration verify_ssl: false ``` {% hint style="warning" %} Setting `verify_ssl: false` disables TLS certificate verification for all OIDC provider communication (discovery, JWKS, token endpoint). Only use this in development or internal environments where you accept the security risk. {% endhint %} By default the server verifies only the token's signature and expiry: any validly-signed, unexpired token from the configured provider is accepted regardless of the audience it was minted for, and authorization (role matching) is the only remaining gate. For defense in depth, set `audience` and/or `issuer` to additionally require a matching `aud` / `iss` claim: ```yaml auth: type: oidc client_id: _CLIENT_ID_ auth_discovery_url: https://login.example.com/.well-known/openid-configuration audience: api://feast-feature-server issuer: https://login.example.com/realms/master ``` A token whose `aud` (or `iss`) claim does not match is rejected at authentication. The two options are independent; leave one unset to skip that check. {% hint style="warning" %} Set these to the values your IdP puts **in the token itself**, which are not always the ones in the discovery document. For example, Microsoft Entra ID commonly issues v1.0 tokens (`iss: https://sts.windows.net//`, `aud: api://`) even when `auth_discovery_url` points at the v2.0 endpoint. That setup keeps working with these options unset, or set to the v1.0 values — but copying the v2.0 issuer from the discovery document would reject every v1.0 token. {% endhint %} To validate token signatures the server fetches the provider's JWKS document and caches it, refetching when the cache expires or when a token presents an unknown key id. Two options tune that behavior: ```yaml auth: type: oidc client_id: _CLIENT_ID_ auth_discovery_url: https://login.example.com/.well-known/openid-configuration jwks_cache_lifespan_seconds: 300 # default; how long the fetched key set is reused jwks_request_timeout_seconds: 10 # default; network timeout for the JWKS fetch ``` `jwks_cache_lifespan_seconds` also bounds how long a key the provider has **revoked** continues to validate tokens, so lower it if your provider rotates or revokes aggressively; each reduction costs proportionally more JWKS fetches. Key rotations that introduce a new key id are picked up immediately regardless of this setting, because an unknown key id triggers a refetch. `jwks_request_timeout_seconds` bounds how long an unresponsive provider can block request serving. Both must be greater than zero. #### Client-Side Configuration The client supports multiple token source modes. The SDK resolves tokens in the following priority order: 1. **Intra-communication token** — internal server-to-server calls (via `INTRA_COMMUNICATION_BASE64` env var) 2. **`token`** — a static JWT string provided directly in the configuration 3. **`token_env_var`** — the name of an environment variable containing the JWT 4. **`client_secret`** — fetches a token from the OIDC provider using client credentials or ROPC flow (requires `auth_discovery_url` and `client_id`) 5. **`FEAST_OIDC_TOKEN`** — default fallback environment variable 6. **Kubernetes service account token** — read from `/var/run/secrets/kubernetes.io/serviceaccount/token` when running inside a pod **Token passthrough** (for use with external token providers like [kube-authkit](https://github.com/opendatahub-io/kube-authkit)): ```yaml project: my-project auth: type: oidc token_env_var: FEAST_OIDC_TOKEN ``` Or with a bare `type: oidc` (no other fields) — the SDK falls back to the `FEAST_OIDC_TOKEN` environment variable or a mounted Kubernetes service account token: ```yaml project: my-project auth: type: oidc ``` **Client credentials / ROPC flow** (existing behavior, unchanged): ```yaml project: my-project auth: type: oidc client_id: test_client_id client_secret: test_client_secret username: test_user_name password: test_password auth_discovery_url: http://localhost:8080/realms/master/.well-known/openid-configuration ``` When using client credentials or ROPC flows, the `verify_ssl` setting also applies to the discovery and token endpoint requests. #### Multi-Token Support (OIDC + Kubernetes Service Account) When the Feast server is configured with OIDC auth and deployed on Kubernetes, the `OidcTokenParser` can handle both Keycloak JWT tokens and Kubernetes service account tokens. Incoming tokens that contain a `kubernetes.io` claim are validated via the Kubernetes Token Access Review API and the namespace is extracted from the authenticated identity — no RBAC queries are performed, so the server service account only needs `tokenreviews/create` permission. All other tokens follow the standard OIDC/Keycloak JWKS validation path. This enables `NamespaceBasedPolicy` enforcement for service account tokens while using `GroupBasedPolicy` and `RoleBasedPolicy` for OIDC user tokens. ### Kubernetes RBAC Authorization With Kubernetes RBAC Authorization, the client uses the service account token as the authorizarion bearer token, and the server fetches the associated roles from the Kubernetes RBAC resources. Feast supports advanced authorization by extracting user groups and namespaces from Kubernetes tokens, enabling fine-grained access control beyond simple role matching. This is achieved by leveraging Kubernetes Token Access Review, which allows Feast to determine the groups and namespaces associated with a user or service account. An example of Kubernetes RBAC authorization configuration is the following: {% hint style="info" %} **NOTE**: This configuration will only work if you deploy feast on Openshift or a Kubernetes platform. {% endhint %} \`\`\`yaml project: my-project auth: type: kubernetes user\_token: #Optional, else service account token Or env var is used for getting the token ... \`\`\` In case the client cannot run on the same cluster as the servers, the client token can be injected using the `LOCAL_K8S_TOKEN` environment variable on the client side. The value must refer to the token of a service account created on the servers cluster and linked to the desired RBAC roles/groups/namespaces. More details can be found in [Setting up kubernetes doc](/master/reference/kubernetes_auth_setup) # OpenTelemetry Integration The OpenTelemetry integration in Feast provides comprehensive monitoring and observability capabilities for your feature serving infrastructure. This component enables you to track key metrics, traces, and logs from your Feast deployment. ## Motivation Monitoring and observability are critical for production machine learning systems. The OpenTelemetry integration addresses these needs by: 1. **Performance Monitoring:** Track CPU and memory usage of feature servers 2. **Operational Insights:** Collect metrics to understand system behavior and performance 3. **Troubleshooting:** Enable effective debugging through distributed tracing 4. **Resource Optimization:** Monitor resource utilization to optimize deployments 5. **Production Readiness:** Provide enterprise-grade observability capabilities ## Architecture The OpenTelemetry integration in Feast consists of several components working together: * **OpenTelemetry Collector:** Receives, processes, and exports telemetry data * **Prometheus Integration:** Enables metrics collection and monitoring * **Instrumentation:** Automatic Python instrumentation for tracking metrics * **Exporters:** Components that send telemetry data to monitoring systems ## Key Features 1. **Automated Instrumentation:** Python auto-instrumentation for comprehensive metric collection 2. **Metric Collection:** Track key performance indicators including: * Memory usage * CPU utilization * Request latencies * Feature retrieval statistics 3. **Flexible Configuration:** Customizable metric collection and export settings 4. **Kubernetes Integration:** Native support for Kubernetes deployments 5. **Prometheus Compatibility:** Integration with Prometheus for metrics visualization ## Setup and Configuration To add monitoring to the Feast Feature Server, follow these steps: ### 1. Deploy Prometheus Operator Follow the [Prometheus Operator documentation](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/getting-started.md) to install the operator. ### 2. Deploy OpenTelemetry Operator Before installing the OpenTelemetry Operator: 1. Install `cert-manager` 2. Validate that the `pods` are running 3. Apply the OpenTelemetry operator: ```bash kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml ``` For additional installation steps, refer to the [OpenTelemetry Operator documentation](https://github.com/open-telemetry/opentelemetry-operator). ### 3. Configure OpenTelemetry Collector Add the OpenTelemetry Collector configuration under the metrics section in your values.yaml file: ```yaml metrics: enabled: true otelCollector: endpoint: "otel-collector.default.svc.cluster.local:4317" # sample headers: api-key: "your-api-key" ``` ### 4. Add Instrumentation Configuration Add the following annotations and environment variables to your deployment.yaml: ```yaml template: metadata: annotations: instrumentation.opentelemetry.io/inject-python: "true" ``` ```yaml - name: OTEL_EXPORTER_OTLP_ENDPOINT value: http://{{ .Values.service.name }}-collector.{{ .Release.namespace }}.svc.cluster.local:{{ .Values.metrics.endpoint.port}} - name: OTEL_EXPORTER_OTLP_INSECURE value: "true" ``` ### 5. Add Metric Checks Add metric checks to all manifests and deployment files: ```yaml {{ if .Values.metrics.enabled }} apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: feast-instrumentation spec: exporter: endpoint: http://{{ .Values.service.name }}-collector.{{ .Release.Namespace }}.svc.cluster.local:4318 env: propagators: - tracecontext - baggage python: env: - name: OTEL_METRICS_EXPORTER value: console,otlp_proto_http - name: OTEL_LOGS_EXPORTER value: otlp_proto_http - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED value: "true" {{end}} ``` ### 6. Add Required Manifests Add the following components to your chart: * Instrumentation * OpenTelemetryCollector * ServiceMonitors * Prometheus Instance * RBAC rules ### 7. Deploy Feast Deploy Feast with metrics enabled: ```bash helm install feast-release infra/charts/feast-feature-server --set metric=true --set feature_store_yaml_base64="" ``` ## Usage To enable OpenTelemetry monitoring in your Feast deployment: 1. Set `metrics.enabled=true` in your Helm values 2. Configure the OpenTelemetry Collector endpoint 3. Deploy with proper annotations and environment variables Example configuration: ```yaml metrics: enabled: true otelCollector: endpoint: "otel-collector.default.svc.cluster.local:4317" ``` ## Monitoring Once configured, you can monitor various metrics including: * `feast_feature_server_memory_usage`: Memory utilization of the feature server * `feast_feature_server_cpu_usage`: CPU usage statistics * `feast_feature_server_request_latency_seconds`: Request latency with feature count dimensions * `feast_feature_server_online_store_read_duration_seconds`: Online store read phase duration * `feast_feature_server_transformation_duration_seconds`: ODFV read-path transformation duration (per ODFV, requires `track_metrics=True`) * `feast_feature_server_write_transformation_duration_seconds`: ODFV write-path transformation duration (per ODFV, requires `track_metrics=True`) * Additional custom metrics based on your configuration For the full list of metrics, see the [Python Feature Server reference](/master/reference/feature-servers/python-feature-server#available-metrics). These metrics can be visualized using Prometheus and other compatible monitoring tools. # Third party integrations We integrate with a wide set of tools and technologies so you can make Feast work in your existing stack. Many of these integrations are maintained as plugins to the main Feast repo. {% hint style="info" %} Don't see your offline store or online store of choice here? Check out our guides to make a custom one! * [Adding a new offline store](/master/how-to-guides/customizing-feast/adding-a-new-offline-store) * [Adding a new online store](/master/how-to-guides/customizing-feast/adding-support-for-a-new-online-store) {% endhint %} ## Integrations See [Functionality and Roadmap](/master/roadmap) ## Standards In order for a plugin integration to be highlighted, it must meet the following requirements: 1. The plugin must have tests. Ideally it would use the Feast universal tests (see this [guide](/master/how-to-guides/adding-or-reusing-tests) for an example), but custom tests are fine. 2. The plugin must have some basic documentation on how it should be used. 3. The author must work with a maintainer to pass a basic code review (e.g. to ensure that the implementation roughly matches the core Feast implementations). In order for a plugin integration to be merged into the main Feast repo, it must meet the following requirements: 1. The PR must pass all integration tests. The universal tests (tests specifically designed for custom integrations) must be updated to test the integration. 2. There is documentation and a tutorial on how to use the integration. 3. The author (or someone else) agrees to take ownership of all the files, and maintain those files going forward. 4. If the plugin is being contributed by an organization, and not an individual, the organization should provide the infrastructure (or credits) for integration tests. # FAQ {% hint style="info" %} **Don't see your question?** We encourage you to ask questions on [GitHub](https://github.com/feast-dev/feast). Even better, once you get an answer, add the answer to this FAQ via a [pull request](/master/project/development-guide)! {% endhint %} ## Getting started ### Which programming language should I use to run Feast in a microservice architecture? [We recommend Python](https://github.com/feast-dev/feast/blob/master/docs/getting-started/language.md). ### Do you have any examples of how Feast should be used? The [quickstart](/master/getting-started/quickstart) is the easiest way to learn about Feast. For more detailed tutorials, please check out the [tutorials](/master/tutorials/tutorials-overview) page. ## Concepts ### Do feature views have to include entities? No, there are [feature views without entities](/master/getting-started/concepts/feature-view#feature-views-without-entities). ### How does Feast handle model or feature versioning? Feast expects that each version of a model corresponds to a different feature service. Feature views once they are used by a feature service are intended to be immutable and not deleted (until a feature service is removed). In the future, `feast plan` and `feast apply` will throw errors if it sees this kind of behavior. ### What is the difference between data sources and the offline store? The data source itself defines the underlying data warehouse table in which the features are stored. The offline store interface defines the APIs required to make an arbitrary compute layer work for Feast (e.g. pulling features given a set of feature views from their sources, exporting the data set results to different formats). Please see [data sources](/master/getting-started/concepts/data-ingestion) and [offline store](/master/getting-started/components/offline-store) for more details. ### Is it possible to have offline and online stores from different providers? Yes, this is possible. For example, you can use BigQuery as an offline store and Redis as an online store. ## Functionality ### How do I run `get_historical_features` without providing an entity dataframe? Feast does support fetching historical features without passing an entity dataframe with the request. * **Supported offline stores:** Entity-less (entity dataframe–less) retrieval is supported for the **Postgres**, **Dask**, **Spark**, and **Ray** offline stores. Postgres was the first to support it; Dask, Spark, and Ray have followed. Other offline stores may be added based on priority and community demand. * **Date range:** Retrieval is controlled by the `start_date` and `end_date` parameters. Supported combinations: * Both params given → data in the given start-to-end time range. * Only `start_date` given → data from the start date to now. * Only `end_date` given → data from (end\_date minus feature view TTL) to end\_date. * Neither given → data from (TTL window) to now. * **Multiple feature views:** When requesting features from multiple feature views in entity-less mode, the feature views must share entity keys so that joins can be performed correctly. We welcome contributions to add or improve entity-less retrieval. See [GitHub issue #1611](https://github.com/feast-dev/feast/issues/1611). ### Does Feast provide security or access control? Feast currently does not support any access control other than the access control required for the Provider's environment (for example, GCP and AWS permissions). It is a good idea though to lock down the registry file so only the CI/CD pipeline can modify it. That way data scientists and other users cannot accidentally modify the registry and lose other team's data. ### Does Feast support streaming sources? Yes. In earlier versions of Feast, we used Feast Spark to manage ingestion from stream sources. In the current version of Feast, we support [push based ingestion](/master/reference/data-sources/push). Feast also defines a [stream processor](/master/tutorials/building-streaming-features) that allows a deeper integration with stream sources. ### Does Feast support feature transformation? There are several kinds of transformations: * On demand transformations (See [docs](/master/reference/beta-on-demand-feature-view)) * These transformations are Pandas transformations run on batch data when you call `get_historical_features` and at online serving time when you call \`get\_online\_features. * Note that if you use push sources to ingest streaming features, these transformations will execute on the fly as well * Batch transformations (WIP, see [RFC](https://docs.google.com/document/d/1964OkzuBljifDvkV-0fakp2uaijnVzdwWNGdz7Vz50A/edit)) * These will include SQL + PySpark based transformations on batch data sources. * Streaming transformations (RFC in progress) ### Does Feast have a Web UI? Yes. See [documentation](/master/reference/alpha-web-ui). ### Does Feast support composite keys? A feature view can be defined with multiple entities. Since each entity has a unique join\_key, using multiple entities will achieve the effect of a composite key. ### What are the performance/latency characteristics of Feast? Feast is designed to work at scale and support low latency online serving. See our [benchmark blog post](https://feast.dev/blog/feast-benchmarks/) for details. ### Does Feast support embeddings and list features? Yes. Specifically: * Simple lists / dense embeddings: * BigQuery supports list types natively * Redshift does not support list types, so you'll need to serialize these features into strings (e.g. json or protocol buffers) * Feast's implementation of online stores serializes features into Feast protocol buffers and supports list types (see [reference](https://github.com/feast-dev/feast/blob/master/docs/specs/online_store_format.md#appendix-a-value-proto-format)) * Sparse embeddings (e.g. one hot encodings) * One way to do this efficiently is to have a protobuf or string representation of ### Does Feast support X storage engine? The list of supported offline and online stores can be found [here](/master/reference/offline-stores) and [here](/master/reference/online-stores), respectively. The [roadmap](/master/roadmap) indicates the stores for which we are planning to add support. Finally, our Provider abstraction is built to be extensible, so you can plug in your own implementations of offline and online stores. Please see more details about customizing Feast [here](/master/how-to-guides/customizing-feast). ### Does Feast support using different clouds for offline vs online stores? Yes. Using a GCP or AWS provider in `feature_store.yaml` primarily sets default offline / online stores and configures where the remote registry file can live. You can override the offline and online stores to be in different clouds if you wish. ### What is the difference between a data source and an offline store? The data source and the offline store are closely tied, but separate concepts. The offline store controls how feast talks to a data store for historical feature retrieval, and the data source points to specific table (or query) within a data store. Offline stores are infrastructure-level connectors to data stores like Snowflake. Additional differences: * Data sources may be specific to a project (e.g. feed ranking), but offline stores are agnostic and used across projects. * A feast project may define several data sources that power different feature views, but a feast project has a single offline store. * Feast users typically need to define data sources when using feast, but only need to use/configure existing offline stores without creating new ones. ### How can I add a custom online store? Please follow the instructions [here](/master/how-to-guides/customizing-feast/adding-support-for-a-new-online-store). ### Can the same storage engine be used for both the offline and online store? Yes. For example, the Postgres connector can be used as both an offline and online store (as well as the registry). ### Does Feast support S3 as a data source? Yes. There are two ways to use S3 in Feast: * Using Redshift as a data source via Spectrum ([AWS tutorial](https://docs.aws.amazon.com/redshift/latest/dg/tutorial-nested-data-create-table.html)), and then continuing with the [Running Feast with Snowflake/GCP/AWS](/master/how-to-guides/feast-snowflake-gcp-aws) guide. See a [presentation](https://youtu.be/pMFbRJ7AnBk?t=9463) we did on this at our apply() meetup. * Using the `s3_endpoint_override` in a `FileSource` data source. This endpoint is more suitable for quick proof of concepts that won't necessarily scale for production use cases. ### Is Feast planning on supporting X functionality? Please see the [roadmap](/master/roadmap). ## Project ### How do I contribute to Feast? For more details on contributing to the Feast community, see [here](/master/community) and this [here](/master/project/contributing). ## Feast 0.9 (legacy) ### What is the difference between Feast 0.9 and Feast 0.10+? Feast 0.10+ is much lighter weight and more extensible than Feast 0.9. It is designed to be simple to install and use. Please see this [document](https://docs.google.com/document/d/1AOsr_baczuARjCpmZgVd8mCqTF4AZ49OEyU4Cn-uTT0) for more details. ### How do I migrate from Feast 0.9 to Feast 0.10+? Please see this [document](https://docs.google.com/document/d/1AOsr_baczuARjCpmZgVd8mCqTF4AZ49OEyU4Cn-uTT0). If you have any questions or suggestions, feel free to leave a comment on the document! ### What are the plans for Feast Core, Feast Serving, and Feast Spark? Feast Core and Feast Serving were both part of Feast Java. We plan to support Feast Serving. We will not support Feast Core; instead we will support our object store based registry. We will not support Feast Spark. For more details on what we plan on supporting, please see the [roadmap](/master/roadmap). # Sample use-case tutorials These Feast tutorials showcase how to use Feast to simplify end to end model training / serving. {% content-ref url="/pages/F7HjJ0aZCcjIwOvVYPYk" %} [Driver ranking](/master/tutorials/tutorials-overview/driver-ranking-with-feast) {% endcontent-ref %} {% content-ref url="/pages/NDGOfA9wrV27tKF0iazj" %} [Fraud detection on GCP](/master/tutorials/tutorials-overview/fraud-detection) {% endcontent-ref %} {% content-ref url="/pages/x5xrWxUU8BSXC4IZIy0P" %} [Real-time credit scoring on AWS](/master/tutorials/tutorials-overview/real-time-credit-scoring-on-aws) {% endcontent-ref %} {% content-ref url="/pages/ubeKF3aLkUCc1g2UYqmc" %} [Driver stats on Snowflake](/master/tutorials/tutorials-overview/driver-stats-on-snowflake) {% endcontent-ref %} # Driver ranking Making a prediction using a linear regression model is a common use case in ML. This model predicts if a driver will complete a trip based on features ingested into Feast. In this example, you'll learn how to use some of the key functionality in Feast. The tutorial runs in both local mode and on the Google Cloud Platform (GCP). For GCP, you must have access to a GCP project already, including read and write permissions to BigQuery. ## [Driver Ranking Example](https://github.com/feast-dev/feast-driver-ranking-tutorial) This tutorial guides you on how to use Feast with [Scikit-learn](https://scikit-learn.org/stable/). You will learn how to: * Train a model locally (on your laptop) using data from [BigQuery](https://cloud.google.com/bigquery/) * Test the model for online inference using [SQLite](https://www.sqlite.org/index.html) (for fast iteration) * Test the model for online inference using [Firestore](https://firebase.google.com/products/firestore) (for production use) Try it and let us know what you think! | ![](/files/-MfZcvLnb8TycZ9lpI4x)[ Run in Google Colab](https://colab.research.google.com/github/feast-dev/feast-driver-ranking-tutorial/blob/master/notebooks/Driver_Ranking_Tutorial.ipynb) | ![](/files/-MfZd9RB0PHVz5E171mk)[ View Source in Github](https://github.com/feast-dev/feast-driver-ranking-tutorial/blob/master/notebooks/Driver_Ranking_Tutorial.ipynb) | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | # Fraud detection on GCP A common use case in machine learning, this tutorial is an end-to-end, production-ready fraud prediction system. It predicts in real-time whether a transaction made by a user is fraudulent. Throughout this tutorial, we’ll walk through the creation of a production-ready fraud prediction system. A prediction is made in real-time as the user makes the transaction, so we need to be able to generate a prediction at low latency. ## [Fraud Detection Example](https://github.com/feast-dev/feast-fraud-tutorial) Our end-to-end example will perform the following workflows: * Computing and backfilling feature data from raw data * Building point-in-time correct training datasets from feature data and training a model * Making online predictions from feature data Here's a high-level picture of our system architecture on Google Cloud Platform (GCP): ![](/files/-Mj5_6HARkvzXxeiCyPG) | ![](/files/-MfZcvLnb8TycZ9lpI4x) [Run in Google Colab](https://colab.research.google.com/github/feast-dev/feast-fraud-tutorial/blob/master/notebooks/Fraud_Detection_Tutorial.ipynb) | ![](/files/-MfZd9RB0PHVz5E171mk)[ View Source on Github](https://github.com/feast-dev/feast-fraud-tutorial/blob/main/notebooks/Fraud_Detection_Tutorial.ipynb) | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | # Real-time credit scoring on AWS Credit scoring models are used to approve or reject loan applications. In this tutorial we will build a real-time credit scoring system on AWS. When individuals apply for loans from banks and other credit providers, the decision to approve a loan application is often made through a statistical model. This model uses information about a customer to determine the likelihood that they will repay or default on a loan, in a process called credit scoring. In this example, we will demonstrate how a real-time credit scoring system can be built using Feast and Scikit-Learn on AWS, using feature data from S3. This real-time system accepts a loan request from a customer and responds within 100ms with a decision on whether their loan has been approved or rejected. ## [Real-time Credit Scoring Example](https://github.com/feast-dev/real-time-credit-scoring-on-aws-tutorial) This end-to-end tutorial will take you through the following steps: * Deploying S3 with Parquet as your primary data source, containing both [loan features](https://github.com/feast-dev/real-time-credit-scoring-on-aws-tutorial/blob/22fc6c7272ef033e7ba0afc64ffaa6f6f8fc0277/data/loan_table_sample.csv) and [zip code features](https://github.com/feast-dev/real-time-credit-scoring-on-aws-tutorial/blob/22fc6c7272ef033e7ba0afc64ffaa6f6f8fc0277/data/zipcode_table_sample.csv) * Deploying Redshift as the interface Feast uses to build training datasets * Registering your features with Feast and configuring DynamoDB for online serving * Building a training dataset with Feast to train your credit scoring model * Loading feature values from S3 into DynamoDB * Making online predictions with your credit scoring model using features from DynamoDB | ![](/files/-MfZd9RB0PHVz5E171mk)[ View Source on Github](https://github.com/feast-dev/real-time-credit-scoring-on-aws-tutorial) | | ------------------------------------------------------------------------------------------------------------------------------- | # Driver stats on Snowflake Initial demonstration of Snowflake as an offline+online store with Feast, using the Snowflake demo template. In the steps below, we will set up a sample Feast project that leverages Snowflake as an offline store + materialization engine + online store. Starting with data in a Snowflake table, we will register that table to the feature store and define features associated with the columns in that table. From there, we will generate historical training data based on those feature definitions and then materialize the latest feature values into the online store. Lastly, we will retrieve the materialized feature values. Our template will generate new data containing driver statistics. From there, we will show you code snippets that will call to the offline store for generating training datasets, and then the code for calling the online store to serve you the latest feature values to serve models in production. ## Snowflake Offline Store Example #### Install feast-snowflake ```shell pip install 'feast[snowflake]' ``` #### Get a Snowflake Trial Account (Optional) [Snowflake Trial Account](http://trial.snowflake.com) #### Create a feature repository ```shell feast init -t snowflake {feature_repo_name} Snowflake Deployment URL (exclude .snowflakecomputing.com): Snowflake User Name:: Snowflake Password:: Snowflake Role Name (Case Sensitive):: Snowflake Warehouse Name (Case Sensitive):: Snowflake Database Name (Case Sensitive):: Should I upload example data to Snowflake (overwrite table)? [Y/n]: Y cd {feature_repo_name} ``` The following files will automatically be created in your project folder: * feature\_store.yaml -- This is your main configuration file * driver\_repo.py -- This is your main feature definition file * test.py -- This is a file to test your feature store configuration #### Inspect `feature_store.yaml` Here you will see the information that you entered. This template will use Snowflake as the offline store, materialization engine, and the online store. The main thing to remember is by default, Snowflake objects have ALL CAPS names unless lower case was specified. {% code title="feature\_store.yaml" %} ```yaml project: ... registry: ... provider: local offline_store: type: snowflake.offline account: SNOWFLAKE_DEPLOYMENT_URL #drop .snowflakecomputing.com user: USERNAME password: PASSWORD role: ROLE_NAME #case sensitive warehouse: WAREHOUSE_NAME #case sensitive database: DATABASE_NAME #case cap sensitive batch_engine: type: snowflake.engine account: SNOWFLAKE_DEPLOYMENT_URL #drop .snowflakecomputing.com user: USERNAME password: PASSWORD role: ROLE_NAME #case sensitive warehouse: WAREHOUSE_NAME #case sensitive database: DATABASE_NAME #case cap sensitive online_store: type: snowflake.online account: SNOWFLAKE_DEPLOYMENT_URL #drop .snowflakecomputing.com user: USERNAME password: PASSWORD role: ROLE_NAME #case sensitive warehouse: WAREHOUSE_NAME #case sensitive database: DATABASE_NAME #case cap sensitive ``` {% endcode %} #### Run our test python script `test.py` ```shell python test.py ``` ## What we did in `test.py` #### Initialize our Feature Store {% code title="test.py" %} ```python from datetime import datetime, timedelta import pandas as pd from driver_repo import driver, driver_stats_fv from feast import FeatureStore fs = FeatureStore(repo_path=".") fs.apply([driver, driver_stats_fv]) ``` {% endcode %} #### Create a dummy training dataframe, then call our offline store to add additional columns {% code title="test.py" %} ```python entity_df = pd.DataFrame( { "event_timestamp": [ pd.Timestamp(dt, unit="ms", tz="UTC").round("ms") for dt in pd.date_range( start=datetime.now() - timedelta(days=3), end=datetime.now(), periods=3, ) ], "driver_id": [1001, 1002, 1003], } ) features = ["driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate"] training_df = fs.get_historical_features( features=features, entity_df=entity_df ).to_df() ``` {% endcode %} #### Materialize the latest feature values into our online store {% code title="test.py" %} ```python fs.materialize_incremental(end_date=datetime.now()) ``` {% endcode %} #### Retrieve the latest values from our online store based on our entity key {% code title="test.py" %} ```python online_features = fs.get_online_features( features=features, entity_rows=[ # {join_key: entity_value} {"driver_id": 1001}, {"driver_id": 1002} ], ).to_dict() ``` {% endcode %} # Building streaming features Feast supports registering streaming feature views and Kafka and Kinesis streaming sources. It also provides an interface for stream processing called the `Stream Processor`. An example Kafka/Spark StreamProcessor is implemented in the contrib folder. For more details, please see the [RFC](https://docs.google.com/document/d/1UzEyETHUaGpn0ap4G82DHluiCj7zEbrQLkJJkKSv4e8/edit?usp=sharing) for more details. Please see [here](https://github.com/feast-dev/streaming-tutorial) for a tutorial on how to build a versioned streaming pipeline that registers your transformations, features, and data sources in Feast. # Retrieval Augmented Generation (RAG) with Feast This tutorial demonstrates how to use Feast with [Docling](https://github.com/doclingjs/docling) and [Milvus](https://milvus.io/) to build a Retrieval Augmented Generation (RAG) application. You'll learn how to store document embeddings in Feast and retrieve the most relevant documents for a given query. ## Overview > \[!NOTE] This tutorial is available [on our GitHub here](https://github.com/feast-dev/feast/tree/master/examples/rag-docling) RAG is a technique that combines generative models (e.g., LLMs) with retrieval systems to generate contextually relevant output for a particular goal (e.g., question and answering). Feast makes it easy to store and retrieve document embeddings for RAG applications by providing integrations with vector databases like Milvus. The typical RAG process involves: 1. Sourcing text data relevant for your application 2. Transforming each text document into smaller chunks of text 3. Transforming those chunks of text into embeddings 4. Inserting those chunks of text along with some identifier for the chunk and document in a database 5. Retrieving those chunks of text along with the identifiers at run-time to inject that text into the LLM's context 6. Calling some API to run inference with your LLM to generate contextually relevant output 7. Returning the output to some end user ## Prerequisites * Python 3.10 or later * Feast installed with Milvus support: `pip install feast[milvus, nlp]` * A basic understanding of feature stores and vector embeddings ## Step 0: Download, Compute, and Export the Docling Sample Dataset ```python import os import io import pypdf import logging import hashlib from datetime import datetime import requests import pandas as pd from transformers import AutoTokenizer from sentence_transformers import SentenceTransformer from docling.datamodel.base_models import ConversionStatus, InputFormat from docling.datamodel.pipeline_options import PdfPipelineOptions from docling.document_converter import DocumentConverter, PdfFormatOption from docling.chunking import HybridChunker logging.basicConfig(level=logging.INFO) _log = logging.getLogger(__name__) # Base URL for PDFs BASE_URL = 'https://raw.githubusercontent.com/DS4SD/docling/refs/heads/main/tests/data/pdf/' PDF_FILES = [ '2203.01017v2.pdf', '2305.03393v1-pg9.pdf', '2305.03393v1.pdf', 'amt_handbook_sample.pdf', 'code_and_formula.pdf', 'picture_classification.pdf', 'redp5110_sampled.pdf', 'right_to_left_01.pdf', 'right_to_left_02.pdf', 'right_to_left_03.pdf' ] INPUT_DOC_PATHS = [os.path.join(BASE_URL, pdf_file) for pdf_file in PDF_FILES] # Configure PDF processing pipeline_options = PdfPipelineOptions() pipeline_options.generate_page_images = True doc_converter = DocumentConverter( format_options={InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)} ) # Load tokenizer and embedding model EMBED_MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2" MAX_TOKENS = 64 # Small token limit for demonstration tokenizer = AutoTokenizer.from_pretrained(EMBED_MODEL_ID) embedding_model = SentenceTransformer(EMBED_MODEL_ID) chunker = HybridChunker(tokenizer=tokenizer, max_tokens=MAX_TOKENS, merge_peers=True) def embed_text(text: str) -> list[float]: """Generate an embedding for a given text.""" return embedding_model.encode([text], normalize_embeddings=True).tolist()[0] def generate_document_rows(conv_results): """ Generator that yields one row per chunk from each successfully converted document. Each yielded dict contains: - file_name: Name of the source file. - raw_markdown: Serialized text for the chunk. - chunk_embedding: The embedding vector for that chunk. """ processed_docs = 0 for conv_res in conv_results: if conv_res.status != ConversionStatus.SUCCESS: continue processed_docs += 1 file_name = conv_res.input.file.stem # FIX: Use `.file.stem` instead of `.path` # Extract the document object (which contains iterate_items) document = conv_res.document try: document_markdown = document.export_to_markdown() except: document_markdown = "" if document is None: _log.warning(f"Document conversion failed for {file_name}") continue # Process each chunk from the document for chunk in chunker.chunk(dl_doc=document): # Use `document` here! raw_chunk = chunker.serialize(chunk=chunk) embedding = embed_text(raw_chunk) yield { "file_name": file_name, "full_document_markdown": document_markdown, "raw_chunk_markdown": raw_chunk, "chunk_embedding": embedding, } _log.info(f"Processed {processed_docs} documents successfully.") def generate_chunk_id(file_name: str, raw_chunk_markdown: str) -> str: """Generate a unique chunk ID based on file_name and raw_chunk_markdown.""" unique_string = f"{file_name}-{raw_chunk_markdown}" return hashlib.sha256(unique_string.encode()).hexdigest() conv_results = doc_converter.convert_all(INPUT_DOC_PATHS, raises_on_error=False) # Build a DataFrame where each row is a unique chunk record rows = list(generate_document_rows(conv_results)) df = pd.DataFrame.from_records(rows) output_dict = {} for file_name in PDF_FILES: try: r = requests.get(BASE_URL + file_name) pdf_bytes = io.BytesIO(r.content) output_dict[file_name] = pdf_bytes.getvalue() except Exception as e: print(f"error with {file_name} \n{e}") odf = pd.DataFrame.from_dict(output_dict, orient='index', columns=['bytes']).reset_index() odf.rename({"index": "file_name"}, axis=1, inplace=True) odf['file_name'] = odf['file_name'].str.replace('.pdf', '') finaldf = df.merge(odf, on='file_name', how='left') finaldf["chunk_id"] = finaldf.apply(lambda row: generate_chunk_id(row["file_name"], row["raw_chunk_markdown"]), axis=1) finaldf['created'] = datetime.now() pdf_example = pypdf.PdfReader(io.BytesIO(finaldf['bytes'].values[0])) finaldf.drop(['full_document_markdown', 'bytes'], axis=1).to_parquet('feature_repo/data/docling_samples.parquet', index=False) odf.to_parquet('feature_repo/data/metadata_samples.parquet', index=False) ``` ## Step 1: Configure Milvus in Feast Create a `feature_store.yaml` file with the following configuration: ```yaml project: docling-rag provider: local registry: data/registry.db online_store: type: milvus path: data/online_store.db vector_enabled: true embedding_dim: 384 index_type: "IVF_FLAT" offline_store: type: file entity_key_serialization_version: 3 auth: type: no_auth ``` ## Step 2: Define your Data Sources and Views Create a `feature_repo.py` file to define your entities, data sources, and feature views: ```python from datetime import timedelta import pandas as pd from feast import ( FeatureView, Field, FileSource, Entity, RequestSource, ) from feast.data_format import ParquetFormat from feast.types import Float64, Array, String, ValueType, PdfBytes from feast.on_demand_feature_view import on_demand_feature_view from sentence_transformers import SentenceTransformer from typing import Dict, Any, List import hashlib from docling.datamodel.base_models import DocumentStream import io from docling.document_converter import DocumentConverter from transformers import AutoTokenizer from sentence_transformers import SentenceTransformer from docling.chunking import HybridChunker # Load tokenizer and embedding model EMBED_MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2" MAX_TOKENS = 64 # Small token limit for demonstration tokenizer = AutoTokenizer.from_pretrained(EMBED_MODEL_ID) embedding_model = SentenceTransformer(EMBED_MODEL_ID) chunker = HybridChunker(tokenizer=tokenizer, max_tokens=MAX_TOKENS, merge_peers=True) def embed_text(text: str) -> list[float]: """Generate an embedding for a given text.""" return embedding_model.encode([text], normalize_embeddings=True).tolist()[0] def generate_chunk_id(file_name: str, raw_chunk_markdown: str="") -> str: """Generate a unique chunk ID based on file_name and raw_chunk_markdown.""" unique_string = f"{file_name}-{raw_chunk_markdown}" if raw_chunk_markdown != "" else f"{file_name}" return hashlib.sha256(unique_string.encode()).hexdigest() # Define entities chunk = Entity( name="chunk_id", description="Chunk ID", value_type=ValueType.STRING, join_keys=["chunk_id"], ) document = Entity( name="document_id", description="Document ID", value_type=ValueType.STRING, join_keys=["document_id"], ) source = FileSource( file_format=ParquetFormat(), path="./data/docling_samples.parquet", timestamp_field="created", ) input_request_pdf = RequestSource( name="pdf_request_source", schema=[ Field(name="document_id", dtype=String), Field(name="pdf_bytes", dtype=PdfBytes), Field(name="file_name", dtype=String), ], ) # Define the view for retrieval docling_example_feature_view = FeatureView( name="docling_feature_view", entities=[chunk], schema=[ Field(name="file_name", dtype=String), Field(name="raw_chunk_markdown", dtype=String), Field( name="vector", dtype=Array(Float64), vector_index=True, vector_search_metric="COSINE", ), Field(name="chunk_id", dtype=String), ], source=source, ttl=timedelta(hours=2), ) @on_demand_feature_view( entities=[chunk, document], sources=[input_request_pdf], schema=[ Field(name="document_id", dtype=String), Field(name="chunk_id", dtype=String), Field(name="chunk_text", dtype=String), Field( name="vector", dtype=Array(Float64), vector_index=True, vector_search_metric="L2", ), ], mode="python", write_to_online_store=True, singleton=True, ) def docling_transform_docs(inputs: dict[str, Any]): document_ids, chunks, embeddings, chunk_ids = [], [], [], [] buf = io.BytesIO( inputs["pdf_bytes"], ) doc_source = DocumentStream(name=inputs["file_name"], stream=buf) converter = DocumentConverter() result = converter.convert(doc_source) for i, chunk in enumerate(chunker.chunk(dl_doc=result.document)): raw_chunk = chunker.serialize(chunk=chunk) embedding = embed_text(raw_chunk) chunk_id = f"chunk-{i}" document_ids.append(inputs["document_id"]) chunks.append(raw_chunk) chunk_ids.append(chunk_id) embeddings.append(embedding) return { "document_id": document_ids, "chunk_id": chunk_ids, "vector": embeddings, "chunk_text": chunks, } ``` ## Step 3: Update your Registry Apply the feature view definitions to the registry: ```bash feast apply ``` ## Step 4: Ingest your Data Process your documents, generate embeddings, and ingest them into the Feast online store: ```python import pandas as pd from feast import FeatureStore store = FeatureStore(repo_path=".") df = pd.read_parquet("./data/docling_samples.parquet") mdf = pd.read_parquet("./data/metadata_samples.parquet") df['chunk_embedding'] = df['vector'].apply(lambda x: x.tolist()) embedding_length = len(df['vector'][0]) print(f'embedding length = {embedding_length}') df['created'] = pd.Timestamp.now() mdf['created'] = pd.Timestamp.now() # Ingesting transformed data to the feature view that has no associated transformation store.write_to_online_store(feature_view_name='docling_feature_view', df=df) # Turning off transformation on writes is as simple as changing the default behavior store.write_to_online_store( feature_view_name='docling_transform_docs', df=df[df['document_id']!='doc-1'], transform_on_write=False, ) # Now we can transform a raw PDF on the fly store.write_to_online_store( feature_view_name='docling_transform_docs', df=mdf[mdf['document_id']=='doc-1'], transform_on_write=True, # this is the default ) ``` ## Step 5: Retrieve Relevant Documents Now you can retrieve the most relevant documents for a given query: ```python from feast import FeatureStore # Initialize FeatureStore store = FeatureStore(".") # Generate query embedding question = 'Who are the authors of the paper?' query_embedding = embed_text(question) # Retrieve similar documents context_data = store.retrieve_online_documents_v2( features=[ "docling_feature_view:vector", "docling_feature_view:file_name", "docling_feature_view:raw_chunk_markdown", "docling_feature_view:chunk_id", ], query=query_embedding, top_k=3, distance_metric='COSINE', ).to_df() print(context_data) ``` ## Step 6: Use Retrieved Documents for Generation Finally, you can use the retrieved documents as context for an LLM: ```python from openai import OpenAI import os client = OpenAI( api_key=os.environ.get("OPENAI_API_KEY"), ) # Format documents for context def format_documents(context_data, base_prompt): documents = "\n".join([f"Document {i+1}: {row['embedded_documents__sentence_chunks']}" for i, row in context_data.iterrows()]) return f"{base_prompt}\n\nContext documents:\n{documents}" BASE_PROMPT = """You are a helpful assistant that answers questions based on the provided context.""" FULL_PROMPT = format_documents(context_data, BASE_PROMPT) # Generate response response = client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": FULL_PROMPT}, {"role": "user", "content": query_embedding} ], ) print('\n'.join([c.message.content for c in response.choices])) ``` ## Alternative: Using DocEmbedder for Simplified Ingestion Instead of manually chunking, embedding, and writing documents as shown above, you can use Feast's `DocEmbedder` class to handle the entire pipeline in a single step. `DocEmbedder` automates chunking, embedding generation, FeatureView creation, and writing to the online store. ### Install Dependencies ```bash pip install feast[milvus,rag] ``` ### Set Up and Ingest with DocEmbedder ```python from feast import DocEmbedder import pandas as pd # Prepare your documents as a DataFrame df = pd.DataFrame({ "id": ["doc1", "doc2", "doc3"], "text": [ "Aaron is a prophet, high priest, and the brother of Moses...", "God at Sinai granted Aaron the priesthood for himself...", "His rod turned into a snake. Then he stretched out...", ], }) # DocEmbedder handles everything: generates FeatureView, applies repo, # chunks text, generates embeddings, and writes to the online store embedder = DocEmbedder( repo_path="feature_repo/", feature_view_name="text_feature_view", ) result = embedder.embed_documents( documents=df, id_column="id", source_column="text", column_mapping=("text", "text_embedding"), ) ``` ### Retrieve and Query Once documents are ingested, you can retrieve them the same way as shown in Step 5 above: ```python from feast import FeatureStore store = FeatureStore("feature_repo/") query_embedding = embed_text("Who are the authors of the paper?") context_data = store.retrieve_online_documents_v2( features=[ "text_feature_view:embedding", "text_feature_view:text", "text_feature_view:source_id", ], query=query_embedding, top_k=3, distance_metric="COSINE", ).to_df() ``` ### Customizing the Pipeline `DocEmbedder` is extensible at every stage. Below are examples of how to create custom components and wire them together. #### Custom Chunker Subclass `BaseChunker` to implement your own chunking strategy. The `load_parse_and_chunk` method receives each document and must return a list of chunk dictionaries. ```python from feast.chunker import BaseChunker, ChunkingConfig from typing import Any, Optional class SentenceChunker(BaseChunker): """Chunks text by sentences instead of word count.""" def load_parse_and_chunk( self, source: Any, source_id: str, source_column: str, source_type: Optional[str] = None, ) -> list[dict]: import re text = str(source) # Split on sentence boundaries sentences = re.split(r'(?<=[.!?])\s+', text) chunks = [] current_chunk = [] chunk_index = 0 for sentence in sentences: current_chunk.append(sentence) combined = " ".join(current_chunk) if len(combined.split()) >= self.config.chunk_size: chunks.append({ "chunk_id": f"{source_id}_{chunk_index}", "original_id": source_id, source_column: combined, "chunk_index": chunk_index, }) # Keep overlap by retaining the last sentence current_chunk = [sentence] chunk_index += 1 # Don't forget the last chunk if current_chunk and len(" ".join(current_chunk).split()) >= self.config.min_chunk_size: chunks.append({ "chunk_id": f"{source_id}_{chunk_index}", "original_id": source_id, source_column: " ".join(current_chunk), "chunk_index": chunk_index, }) return chunks ``` Or simply configure the built-in `TextChunker`: ```python from feast import TextChunker, ChunkingConfig chunker = TextChunker(config=ChunkingConfig( chunk_size=200, chunk_overlap=50, min_chunk_size=30, max_chunk_chars=1000, )) ``` #### Custom Embedder Subclass `BaseEmbedder` to use a different embedding model. Register modality handlers in `_register_default_modalities` and implement the `embed` method. ```python from feast.embedder import BaseEmbedder, EmbeddingConfig from typing import Any, List, Optional import numpy as np class OpenAIEmbedder(BaseEmbedder): """Embedder that uses the OpenAI API for text embeddings.""" def __init__(self, model: str = "text-embedding-3-small", config: Optional[EmbeddingConfig] = None): self.model = model self._client = None super().__init__(config) def _register_default_modalities(self) -> None: self.register_modality("text", self._embed_text) @property def client(self): if self._client is None: from openai import OpenAI self._client = OpenAI() return self._client def get_embedding_dim(self, modality: str) -> Optional[int]: # text-embedding-3-small produces 1536-dim vectors if modality == "text": return 1536 return None def embed(self, inputs: List[Any], modality: str) -> np.ndarray: if modality not in self._modality_handlers: raise ValueError(f"Unsupported modality: '{modality}'") return self._modality_handlers[modality](inputs) def _embed_text(self, inputs: List[str]) -> np.ndarray: response = self.client.embeddings.create(input=inputs, model=self.model) return np.array([item.embedding for item in response.data]) ``` #### Custom Logical Layer Function The schema transform function transforms the chunked + embedded DataFrame into the exact schema your FeatureView expects. It must accept a `pd.DataFrame` and return a `pd.DataFrame`. ```python import pandas as pd from datetime import datetime, timezone def my_schema_transform_fn(df: pd.DataFrame) -> pd.DataFrame: """Map chunked + embedded columns to the FeatureView schema.""" return pd.DataFrame({ "passage_id": df["chunk_id"], "text": df["text"], "embedding": df["text_embedding"], "event_timestamp": [datetime.now(timezone.utc)] * len(df), "source_id": df["original_id"], # Add any extra columns your FeatureView expects "chunk_index": df["chunk_index"], }) ``` #### Putting It All Together Pass your custom components to `DocEmbedder`: ```python from feast import DocEmbedder embedder = DocEmbedder( repo_path="feature_repo/", feature_view_name="text_feature_view", chunker=SentenceChunker(config=ChunkingConfig(chunk_size=150, min_chunk_size=20)), embedder=OpenAIEmbedder(model="text-embedding-3-small"), schema_transform_fn=my_schema_transform_fn, vector_length=1536, # Match the OpenAI embedding dimension ) # Embed and ingest result = embedder.embed_documents( documents=df, id_column="id", source_column="text", column_mapping=("text", "text_embedding"), ) ``` > **Note:** When using a custom `schema_transform_fn`, ensure the returned DataFrame columns match your FeatureView schema. When using a custom embedder with a different output dimension, set `vector_length` accordingly (or let it auto-detect via `get_embedding_dim`). For a complete end-to-end example, see the [DocEmbedder notebook](https://github.com/feast-dev/feast/tree/master/examples/rag-retriever/rag_feast_docembedder.ipynb). ## Why Feast for RAG? Feast makes it remarkably easy to set up and manage a RAG system by: 1. Simplifying vector database configuration and management 2. Providing a consistent API for both writing and reading embeddings 3. Supporting both batch and real-time data ingestion 4. Enabling versioning and governance of your document repository 5. Offering seamless integration with multiple vector database backends 6. Providing a unified API for managing both feature data and document embeddings For more details on using vector databases with Feast, see the [Vector Database documentation](/master/reference/alpha-vector-database). The complete demo code is available in the [GitHub repository](https://github.com/feast-dev/feast/tree/master/examples/rag-docling). # RAG Fine Tuning with Feast and Milvus ## Introduction This example notebook provides a step-by-step demonstration of building and using a RAG system with Feast and the custom FeastRagRetriever. The notebook walks through: 1. Data Preparation * Loads a subset of the [Wikipedia DPR dataset](https://huggingface.co/datasets/facebook/wiki_dpr) (1% of training data) * Implements text chunking with configurable chunk size and overlap * Processes text into manageable passages with unique IDs 2. Embedding Generation * Uses `all-MiniLM-L6-v2` sentence transformer model * Generates 384-dimensional embeddings for text passages * Demonstrates batch processing with GPU support 3. Feature Store Setup * Creates a Parquet file as the historical data source * Configures Feast with the feature repository * Demonstrates writing embeddings from data source to Milvus online store which can be used for model training later 4. RAG System Implementation * **Embedding Model**: `all-MiniLM-L6-v2` (configurable) * **Generator Model**: `granite-3.2-2b-instruct` (configurable) * **Vector Store**: Custom implementation with Feast integration * **Retriever**: Custom implementation extending HuggingFace's RagRetriever 5. Query Demonstration * Perform inference with retrieved context ## Requirements * A Kubernetes cluster with: * GPU nodes available (for model inference) * At least 200GB of storage * A standalone Milvus deployment. See example [here](https://github.com/milvus-io/milvus-helm/tree/master/charts/milvus). ## Running the example Clone this repository: Navigate to the examples/rag-retriever directory. Here you will find the following files: * **feature\_repo/feature\_store.yaml** This is the core configuration file for the RAG project's feature store, configuring a Milvus online store on a local provider. * In order to configure Milvus you should: * Update `feature_store.yaml` with your Milvus connection details: * host * port (default: 19530) * credentials (if required) * **feature\_repo/ragproject\_repo.py** This is the Feast feature repository configuration that defines the schema and data source for Wikipedia passage embeddings. * **rag\_feast.ipynb** This is a notebook demonstrating the implementation of a RAG system using Feast. The notebook provides: * A complete end-to-end example of building a RAG system with: * Data preparation using the Wiki DPR dataset * Text chunking and preprocessing * Vector embedding generation using sentence-transformers * Integration with Milvus vector store * Inference utilising a custom RagRetriever: FeastRagRetriever * Uses `all-MiniLM-L6-v2` for generating embeddings * Implements `granite-3.2-2b-instruct` as the generator model Open `rag_feast.ipynb` and follow the steps in the notebook to run the example. ## Using DocEmbedder for Simplified Ingestion As an alternative to the manual data preparation steps in the notebook above, Feast provides the `DocEmbedder` class that automates the entire document-to-embeddings pipeline: chunking, embedding generation, FeatureView creation, and writing to the online store. ### Install Dependencies ```bash pip install feast[milvus,rag] ``` ### Quick Start ```python from feast import DocEmbedder from datasets import load_dataset # Load your dataset dataset = load_dataset("facebook/wiki_dpr", "psgs_w100.nq.exact", split="train[:1%]", with_index=False, trust_remote_code=True) df = dataset.select(range(100)).to_pandas() # DocEmbedder handles everything in one step embedder = DocEmbedder( repo_path="feature_repo_docembedder/", feature_view_name="text_feature_view", ) result = embedder.embed_documents( documents=df, id_column="id", source_column="text", column_mapping=("text", "text_embedding"), ) ``` ### What DocEmbedder Does 1. **Generates a FeatureView**: Automatically creates a Python file with Entity and FeatureView definitions compatible with `feast apply` 2. **Applies the repo**: Registers the FeatureView in the Feast registry and deploys infrastructure (e.g., Milvus collection) 3. **Chunks documents**: Splits text into smaller passages using `TextChunker` (configurable chunk size, overlap, etc.) 4. **Generates embeddings**: Produces vector embeddings using `MultiModalEmbedder` (defaults to `all-MiniLM-L6-v2`) 5. **Writes to online store**: Stores the processed data in your configured online store (e.g., Milvus) ### Customization * **Custom Chunker**: Subclass `BaseChunker` for your own chunking strategy * **Custom Embedder**: Subclass `BaseEmbedder` to use a different embedding model * **Logical Layer Function**: Provide a `SchemaTransformFn` to control how the output maps to your FeatureView schema ### Example Notebook See **`rag_feast_docembedder.ipynb`** for a complete end-to-end example that uses DocEmbedder with the Wiki DPR dataset and then queries the results using `FeastRAGRetriever`. ## FeastRagRetriver Low Level Design ![Low level design for feast rag retriever](/files/OEiWDCCIepTsAYlUpF88) ## Helpful Information * Ensure your Milvus instance is properly configured and running * Vector dimensions and similarity metrics can be adjusted in the feature store configuration * The example uses Wikipedia data, but the system can be adapted for other datasets # MCP - AI Agent Example This example demonstrates how to enable MCP (Model Context Protocol) support in Feast, allowing AI agents and applications to interact with your features through standardized MCP interfaces. ## Prerequisites 1. Python 3.8+ 2. Feast installed 3. FastAPI MCP library ## Installation 1. Install Feast with MCP support: ```bash pip install feast[mcp] ``` Alternatively, you can install the dependencies separately: ```bash pip install feast pip install fastapi_mcp ``` ## Setup 1. Navigate to this example directory within your cloned Feast repository: ```bash cd examples/mcp_feature_store ``` 2. Initialize a Feast repository in this directory. We'll use the existing `feature_store.yaml` that's already configured for MCP: ```bash feast init . ``` This will create a `data` subdirectory and a `feature_repo` subdirectory if they don't exist, and will use the `feature_store.yaml` present in the current directory (`examples/mcp_feature_store`). 3. Apply the feature store configuration: ```bash cd feature_repo feast apply cd .. # Go back to examples/mcp_feature_store for the next steps ``` ## Starting the MCP-Enabled Feature Server Start the Feast feature server with MCP support: ```bash feast serve --host 0.0.0.0 --port 6566 ``` If MCP is properly configured, you should see a log message indicating that MCP support has been enabled: ``` INFO:feast.feature_server:MCP support has been enabled for the Feast feature server ``` ## Available MCP Tools The fastapi\_mcp integration automatically exposes your Feast feature server's FastAPI endpoints as MCP tools. This means AI assistants can: * **Call `/get-online-features`** to retrieve features from the feature store * **Use `/health`** to check server status ## Configuration Details The key configuration that enables MCP support: ```yaml feature_server: type: mcp # Use MCP feature server type enabled: true # Enable feature server mcp_enabled: true # Enable MCP protocol support mcp_server_name: "feast-feature-store" mcp_server_version: "1.0.0" ``` # Feast-Powered AI Agent This example demonstrates an **AI agent with persistent memory** that uses **Feast as both a feature store and a context memory layer** through the **Model Context Protocol (MCP)**. This demo uses **Milvus** as the vector-capable online store, but Feast supports multiple vector backends -- including **Milvus, Elasticsearch, Qdrant, PGVector, and FAISS** -- swappable via configuration. ## Why Feast for Agents? Agents need more than just access to data -- they need to **remember** what happened in prior interactions. Feast's online store is entity-keyed, low-latency, governed, and supports both reads and writes, making it a natural fit for agent context and memory. | Capability | How Feast Provides It | | ---------------------- | ------------------------------------------------------------------------------------------------ | | **Structured context** | Entity-keyed feature retrieval (customer profiles, account data) | | **Document search** | Vector similarity search via pluggable backends (Milvus, Elasticsearch, Qdrant, PGVector, FAISS) | | **Persistent memory** | Auto-checkpointed after each turn via `write_to_online_store` | | **Governance** | RBAC, audit trails, and feature-level permissions | | **TTL management** | Declarative expiration on feature views (memory auto-expires) | | **Offline analysis** | Memory is queryable offline like any other feature | ## Architecture ```mermaid %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#E3F2FD', 'primaryBorderColor': '#1565C0', 'primaryTextColor': '#0D47A1', 'lineColor': '#546E7A', 'secondaryColor': '#F3E5F5', 'tertiaryColor': '#E8F5E9'}}}%% flowchart LR User((("🧑 User"))):::userClass subgraph AgentLoop ["🤖 Agent Loop"] LLM["LLM Engine\ntool-calling · reasoning"]:::agentClass end subgraph Feast ["🏗️ Feast MCP Server"] direction TB ReadAPI["get_online_features\nretrieve_online_documents"]:::feastClass WriteAPI["write_to_online_store"]:::feastClass end subgraph VectorStore ["🗄️ Online Store"] direction TB Profiles[("👤 customer_profile\nplan · spend · tickets")]:::storageClass Articles[("📚 knowledge_base\nvector embeddings")]:::storageClass Memory[("🧠 agent_memory\ntopic · resolution · prefs")]:::memoryClass end User -->|"query"| LLM LLM -->|"MCP: recall_memory\nlookup_customer\nsearch_kb"| ReadAPI LLM -->|"MCP: auto-checkpoint"| WriteAPI ReadAPI --> Profiles ReadAPI --> Articles ReadAPI --> Memory WriteAPI --> Memory ReadAPI -.->|"results"| LLM LLM -->|"answer"| User classDef userClass fill:#E8EAF6,stroke:#283593,color:#1A237E classDef agentClass fill:#E3F2FD,stroke:#1565C0,color:#0D47A1 classDef feastClass fill:#FFF3E0,stroke:#E65100,color:#BF360C classDef storageClass fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20 classDef memoryClass fill:#F3E5F5,stroke:#6A1B9A,color:#4A148C ``` ## Tools (backed by Feast) The agent has four tools. Feast is both the **read path** (context) and the **write path** (memory): | Tool | Direction | What it does | When the LLM calls it | | ----------------------- | --------- | --------------------------------------------------------------------- | -------------------------------------- | | `lookup_customer` | READ | Fetches customer profile features (plan, spend, tickets) | Questions about the customer's account | | `search_knowledge_base` | READ | Retrieves support articles from the vector store | Questions needing product docs | | `recall_memory` | READ | Reads past interaction context (last topic, open issues, preferences) | Start of every conversation | Memory is **auto-saved after each agent turn** (not as an LLM tool call). This follows the same pattern used by production frameworks -- see [Memory as Infrastructure](#memory-as-infrastructure) below. ### Feast as Context Memory The `agent_memory` feature view stores per-customer interaction state: ```python agent_memory = FeatureView( name="agent_memory", entities=[customer], schema=[ Field(name="last_topic", dtype=String), Field(name="last_resolution", dtype=String), Field(name="interaction_count", dtype=Int64), Field(name="preferences", dtype=String), Field(name="open_issue", dtype=String), ], ttl=timedelta(days=30), ) ``` This gives agents **persistent, governed, entity-keyed memory** that survives across sessions, is versioned, and lives under the same RBAC as every other feature -- unlike an ad-hoc Redis cache or an in-process dict. ### Memory as Infrastructure Production agent frameworks treat memory as **infrastructure, not an LLM decision**. The framework auto-saves state after each step - the LLM never needs to "decide" to persist: | Framework | Memory mechanism | How it works | | --------------------- | ---------------------------------------------- | --------------------------------------------------------------------- | | **LangGraph** | Checkpointers (`MemorySaver`, `PostgresSaver`) | Every graph step is checkpointed automatically by `thread_id` | | **CrewAI** | Built-in memory (`memory=True`) | Short-term, long-term, and entity memory auto-persist after each task | | **AutoGen** | Teachable agents | Post-conversation hooks extract and store learnings in a vector DB | | **OpenAI Agents SDK** | Application-level | Serialize `RunResult` between turns; framework manages state | This demo follows the same pattern: the agent's three read tools (`recall_memory`, `lookup_customer`, `search_knowledge_base`) are exposed to the LLM for reasoning, while **memory persistence is handled by the framework after each turn** via `_auto_save_memory`. This ensures consistent, reliable memory regardless of LLM behaviour - no risk of the LLM forgetting to save, double-saving, or writing inconsistent state. Feast is a natural fit for this checkpoint layer because it already provides: * **Entity-keyed storage**: memory is keyed by customer ID (or any entity) * **TTL management**: memory auto-expires via declarative feature view TTL * **Schema enforcement**: typed fields prevent corrupt memory writes * **RBAC and audit trails**: memory reads/writes are governed like any other feature * **Offline queryability**: agent memory can be analysed in batch pipelines ## Prerequisites * Python 3.10+ * Feast with MCP and Milvus support * OpenAI API key (for live tool-calling; demo mode works without it) ## Quickstart ### One command ```bash cd examples/agent_feature_store ./run_demo.sh # Or with live LLM tool-calling: OPENAI_API_KEY=sk-... ./run_demo.sh ``` The script installs dependencies, generates sample data, starts the Feast server, runs the agent, and cleans up on exit. ### Step by step ### 1. Install dependencies ```bash pip install "feast[mcp,milvus]" ``` ### 2. Generate sample data and apply the registry ```bash cd examples/agent_feature_store python setup_data.py ``` This creates: * **3 customer profiles** with attributes like plan tier, spend, and satisfaction score * **6 knowledge-base articles** with 384-dimensional vector embeddings * **Empty agent memory scaffold** (populated as the agent runs) ### 3. Start the Feast MCP Feature Server ```bash cd feature_repo feast serve --host 0.0.0.0 --port 6566 --workers 1 ``` ### 4. Run the agent In a new terminal: ```bash # Without API key: runs in demo mode (simulated tool selection) python agent.py ``` To run with a real LLM, set the API key and (optionally) the base URL: ```bash # OpenAI export OPENAI_API_KEY="sk-..." #pragma: allowlist secret python agent.py # Ollama (free, local -- no API key needed) ollama pull llama3.1:8b export OPENAI_API_KEY="ollama" #pragma: allowlist secret export OPENAI_BASE_URL="http://localhost:11434/v1" export LLM_MODEL="llama3.1:8b" python agent.py # Any OpenAI-compatible provider (Azure, vLLM, LiteLLM, etc.) export OPENAI_API_KEY="your-key" #pragma: allowlist secret export OPENAI_BASE_URL="https://your-endpoint/v1" export LLM_MODEL="your-model" python agent.py ``` ### Demo mode output Without an API key, the agent simulates the decision-making process with memory: ``` ================================================================= Scene 1: Enterprise customer (C1001) asks about SSO Customer: C1001 | Query: "How do I set up SSO for my team?" ================================================================= [Demo mode] Simulating agent reasoning Round 1 | recall_memory(customer_id=C1001) -> No prior interactions found Round 1 | lookup_customer(customer_id=C1001) -> Alice Johnson | enterprise plan | $24,500 spend | 1 open tickets Round 1 | search_knowledge_base(query="How do I set up SSO for my team?...") -> Best match: "Configuring single sign-on (SSO)" Round 2 | Generating personalised response... ───────────────────────────────────────────────────────────── Agent Response: ───────────────────────────────────────────────────────────── Hi Alice! Since you're on our Enterprise plan, SSO is available for your team. Go to Settings > Security > SSO and enter your Identity Provider metadata URL. We support SAML 2.0 and OIDC... [Checkpoint] Memory saved: topic="SSO setup" ================================================================= Scene 4: C1001 returns -- does the agent remember Scene 1? Customer: C1001 | Query: "I'm back about my SSO question from earlier." ================================================================= [Demo mode] Simulating agent reasoning Round 1 | recall_memory(customer_id=C1001) -> Previous topic: SSO setup -> Open issue: none -> Interaction count: 1 Round 1 | lookup_customer(customer_id=C1001) -> Alice Johnson | enterprise plan | $24,500 spend | 1 open tickets Round 2 | Generating personalised response... ───────────────────────────────────────────────────────────── Agent Response: ───────────────────────────────────────────────────────────── Welcome back, Alice! I can see from our records that we last discussed "SSO setup". How can I help you today? [Checkpoint] Memory saved: topic="SSO setup" ``` Scene 4 demonstrates memory continuity -- the agent recalls the SSO conversation from Scene 1 without the customer re-explaining. ### Live mode output (with API key) With an API key, the LLM autonomously decides which tools to use: ``` ================================================================= Scene 1: Enterprise customer (C1001) asks about SSO Customer: C1001 | Query: "How do I set up SSO for my team?" ================================================================= [Round 1] Tool call: recall_memory({'customer_id': 'C1001'}) [Round 1] Tool call: lookup_customer({'customer_id': 'C1001'}) [Round 1] Tool call: search_knowledge_base({'query': 'SSO setup'}) Agent finished after 2 round(s) ───────────────────────────────────────────────────────────── Agent Response: ───────────────────────────────────────────────────────────── Hi Alice! Since you're on our Enterprise plan, SSO is available for your team. Go to Settings > Security > SSO and enter your Identity Provider metadata URL. We support SAML 2.0 and OIDC... [Checkpoint] Memory saved: topic="SSO setup" ``` ## How It Works > **Why a raw loop?** This example builds the agent from scratch using the OpenAI tool-calling API and the MCP Python SDK to keep dependencies minimal and make every Feast call visible. All Feast interactions go through the MCP protocol -- the agent connects to Feast's MCP endpoint, discovers tools dynamically, and invokes them via `session.call_tool()`. In production, you would use a framework like LangChain/LangGraph, LlamaIndex, CrewAI, or AutoGen -- Feast's MCP endpoint lets any of them auto-discover the tools with zero custom wiring (see [MCP Integration](#mcp-integration) below). ### The Agent Loop (`agent.py`) ```python from mcp import ClientSession from mcp.client.streamable_http import streamablehttp_client async with streamablehttp_client("http://localhost:6566/mcp") as (r, w, _): async with ClientSession(r, w) as session: await session.initialize() tools = await session.list_tools() # discover Feast tools for round in range(MAX_ROUNDS): # 1. Send messages + read tools to LLM response = call_llm(messages, tools=[...]) # 2. If LLM says "stop" -> return the answer if response.finish_reason == "stop": break # 3. Execute tool calls via MCP for tool_call in response.tool_calls: result = await session.call_tool(name, args) messages.append(tool_result(result)) # 4. Framework-style checkpoint: auto-save via MCP await session.call_tool("write_to_online_store", {...}) ``` The LLM sees the tool definitions (JSON Schema) and decides: * **Which tools to call** (can call zero, one, or multiple per round) * **What arguments to pass** (e.g., which customer ID to look up) * **When to stop** (once it has enough information to answer) All Feast calls go through **MCP** (`session.call_tool()`), not direct REST. Memory is saved **automatically after each turn** by the framework, not by the LLM. This mirrors how production frameworks handle persistence (see [Memory as Infrastructure](#memory-as-infrastructure)). ### Feature Definitions (`feature_repo/features.py`) * **`customer_profile`**: Structured data (name, plan, spend, tickets, satisfaction) * **`knowledge_base`**: Support articles with 384-dim vector embeddings (Milvus in this demo; swappable to Elasticsearch, Qdrant, PGVector, or FAISS) * **`agent_memory`**: Per-customer interaction history (last topic, resolution, preferences, open issues) ### MCP Integration The Feast Feature Server exposes all endpoints as MCP tools at `http://localhost:6566/mcp`. Any MCP-compatible framework can connect: ```python # LangChain / LangGraph from langchain_mcp_adapters.client import MultiServerMCPClient from langgraph.prebuilt import create_react_agent async with MultiServerMCPClient( {"feast": {"url": "http://localhost:6566/mcp", "transport": "streamable_http"}} ) as client: tools = client.get_tools() agent = create_react_agent(llm, tools) result = await agent.ainvoke({"messages": "How do I set up SSO?"}) ``` ```python # LlamaIndex from llama_index.tools.mcp import aget_tools_from_mcp_url from llama_index.core.agent.function_calling import FunctionCallingAgent from llama_index.llms.openai import OpenAI tools = await aget_tools_from_mcp_url("http://localhost:6566/mcp") agent = FunctionCallingAgent.from_tools(tools, llm=OpenAI(model="gpt-4o-mini")) response = await agent.achat("How do I set up SSO?") ``` ```json // Claude Desktop / Cursor { "mcpServers": { "feast": { "url": "http://localhost:6566/mcp", "transport": "streamable_http" } } } ``` > **Building the same agent with a framework:** The examples above show the Feast-specific part -- connecting to the MCP endpoint and getting the tools. Once you have the tools, building the agent follows each framework's standard patterns. The key difference from this demo's raw loop: frameworks handle the tool-calling loop, message threading, and (with LangGraph checkpointers or CrewAI `memory=True`) automatic state persistence natively. Feast's MCP endpoint means zero custom integration code -- the tools are discovered and callable immediately. **Adapting to your use case:** The demo's system prompt, tool wrappers (`lookup_customer`, `recall_memory`), and feature views are all specific to customer support. For your own agent, you define your feature views in Feast (e.g., `product_catalog`, `order_history`, `fraud_signals`), run `feast apply`, and start the server. The same three generic MCP tools -- `get_online_features`, `retrieve_online_documents`, and `write_to_online_store` -- serve any domain. With a framework like LangChain or LlamaIndex, you don't even need custom tool wrappers -- the LLM calls the generic Feast tools directly with your feature view names and entities. ## Production Deployment For production, Feast fits into a layered platform architecture: ```mermaid %%{init: {'theme': 'base', 'themeVariables': {'primaryColor': '#E3F2FD', 'primaryBorderColor': '#1565C0', 'lineColor': '#546E7A'}}}%% flowchart TB Agent(("🤖 AI Agent")):::agentClass subgraph Platform ["🛡️ Production Platform"] direction TB Gateway["🔐 MCP Gateway\nJWT auth · tool filtering"]:::platformClass Sandbox["📦 Sandboxed Container\nkernel-level isolation"]:::platformClass Guardrails["🛑 Guardrails Orchestrator\ninput/output screening"]:::platformClass end subgraph Observability ["📊 Observability + Lifecycle"] direction LR OTel["OpenTelemetry + MLflow\ntraces · metrics · audit"]:::obsClass Kagenti["Kagenti Operator\nAgentCard CRDs · discovery"]:::lifecycleClass end subgraph FeastSvc ["🏗️ Feast MCP Server"] FS["/mcp · /get-online-features · /write-to-online-store"]:::feastClass end subgraph Store ["🗄️ Online Store"] direction LR P[("👤 Profiles")]:::storageClass K[("📚 Knowledge Base")]:::storageClass M[("🧠 Agent Memory")]:::memoryClass end Agent --> Gateway Gateway --> Sandbox Sandbox --> Guardrails Guardrails --> FS FS --> P FS --> K FS <--> M OTel -.->|"traces"| FS Kagenti -.->|"discover"| FS classDef agentClass fill:#E3F2FD,stroke:#1565C0,color:#0D47A1 classDef platformClass fill:#FFEBEE,stroke:#C62828,color:#B71C1C classDef obsClass fill:#FFF8E1,stroke:#F57F17,color:#E65100 classDef lifecycleClass fill:#E0F2F1,stroke:#00695C,color:#004D40 classDef feastClass fill:#FFF3E0,stroke:#E65100,color:#BF360C classDef storageClass fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20 classDef memoryClass fill:#F3E5F5,stroke:#6A1B9A,color:#4A148C ``` This demo uses Milvus Lite (embedded). For production, swap to any supported vector-capable backend by updating `feature_store.yaml`: * **Milvus cluster**: Deploy via the [Milvus Operator](https://milvus.io/docs/install_cluster-milvusoperator.md) and set `host`/`port` instead of `path`. * **Elasticsearch**: Set `online_store: type: elasticsearch` with your cluster URL. * **Qdrant**: Set `online_store: type: qdrant` with your Qdrant endpoint. * **PGVector**: Set `online_store: type: postgres` with `pgvector_enabled: true`. * **FAISS**: Set `online_store: type: faiss` for in-process vector search. # Demo Notebooks Feast can generate tailored Jupyter notebooks for any Feast project. The notebooks adapt to your `feature_store.yaml` configuration and provide a hands-on walkthrough of core Feast functionality. ## What you get For each project discovered, Feast creates a directory with notebooks covering: | Notebook | Description | | ------------------------------------- | ----------------------------------------------------------------------------------------------------- | | **01 — Feature Store Overview** | Explore registered entities, feature views, feature services, and data sources. | | **02 — Historical Feature Retrieval** | Build a training dataset with point-in-time correct joins using `get_historical_features`. | | **03 — Online Feature Serving** | Materialize features to the online store and retrieve them at low latency with `get_online_features`. | The content adapts automatically based on: * **Online / offline store types** — descriptions reflect the actual backends configured. * **Registry type** — local registries include `feast apply`; remote registries use `refresh_registry()`. * **Authentication** — auth details from `feature_store.yaml` are surfaced when configured. * **Vector search** — a vector/RAG retrieval section is included when embeddings are detected. ## Prerequisites * Python 3.9+ * Feast installed (`pip install feast`) * A feature repository with a valid `feature_store.yaml` ## Using the CLI Run the command from (or pointing to) a directory containing `feature_store.yaml`: ```bash feast demo-notebooks ``` This searches for `feature_store.yaml` in the current directory and every file inside the `feast-config/` directory. Each file in `feast-config/` is treated as a separate project config. For each project found, notebooks are written to `./feast-demo-notebooks//`. ### Options | Option | Default | Description | | ------------------ | ------------------------ | ------------------------------------------------ | | `-o, --output-dir` | `./feast-demo-notebooks` | Root directory for generated notebooks | | `--overwrite` | `false` | Overwrite if the output directory already exists | ```bash # Write to a custom directory feast demo-notebooks -o ./my-notebooks # Overwrite existing notebooks feast demo-notebooks --overwrite # Use --chdir to point at a different feature repo feast -c /path/to/feature_repo demo-notebooks ``` ## Using the Python SDK ```python from feast import copy_demo_notebooks copy_demo_notebooks() ``` ### Parameters | Parameter | Type | Default | Description | | ------------ | ------ | -------------------------- | -------------------------------------------------- | | `output_dir` | `str` | `"./feast-demo-notebooks"` | Root directory for generated notebooks | | `repo_path` | `str` | `"."` | Directory to search for `feature_store.yaml` files | | `overwrite` | `bool` | `False` | Overwrite existing output directories | ### Examples ```python from feast import copy_demo_notebooks # Default — searches current directory, writes to ./feast-demo-notebooks/ copy_demo_notebooks() # Custom paths copy_demo_notebooks( output_dir="/home/user/notebooks", repo_path="/home/user/feast-projects/my-repo/feature_repo", overwrite=True, ) ``` ## Multi-project repositories If your `feast-config/` directory contains multiple files, each is treated as a separate project and a dedicated notebook directory is created: ``` feast-demo-notebooks/ ├── project_alpha/ │ ├── 01_feature_store_overview.ipynb │ ├── 02_historical_features_training.ipynb │ └── 03_online_features_serving.ipynb └── project_beta/ ├── 01_feature_store_overview.ipynb ├── 02_historical_features_training.ipynb └── 03_online_features_serving.ipynb ``` ## Running the notebooks Open any generated notebook in Jupyter, JupyterLab, or VS Code and run cells from top to bottom. Each notebook: 1. Configures the path to your `feature_store.yaml` automatically (no manual editing needed). 2. Connects to the feature store using the Feast Python SDK. 3. Walks through relevant operations with real data from your project. {% hint style="info" %} The first notebook (**01 — Overview**) includes a prerequisites check and `feast apply` / registry sync step. Subsequent notebooks assume these have already been completed. {% endhint %} # Feature Quality Monitoring Quickstart # Running Feast with Snowflake/GCP/AWS {% content-ref url="/pages/erz7dwkDdaq5SC7yvDij" %} [Install Feast](/master/how-to-guides/feast-snowflake-gcp-aws/install-feast) {% endcontent-ref %} {% content-ref url="/pages/CnuSJLi4Th1r1UcHCeJi" %} [Create a feature repository](/master/how-to-guides/feast-snowflake-gcp-aws/create-a-feature-repository) {% endcontent-ref %} {% content-ref url="/pages/H2HRmgAqoUzCCWwqKo6h" %} [Deploy a feature store](/master/how-to-guides/feast-snowflake-gcp-aws/deploy-a-feature-store) {% endcontent-ref %} {% content-ref url="/pages/SeQEIvQuEQwbqCocOoh1" %} [Build a training dataset](/master/how-to-guides/feast-snowflake-gcp-aws/build-a-training-dataset) {% endcontent-ref %} {% content-ref url="/pages/O7BuACOUSmntOtavVnVu" %} [Load data into the online store](/master/how-to-guides/feast-snowflake-gcp-aws/load-data-into-the-online-store) {% endcontent-ref %} {% content-ref url="/pages/nRHzHuSEqqzIskrRdydt" %} [Read features from the online store](/master/how-to-guides/feast-snowflake-gcp-aws/read-features-from-the-online-store) {% endcontent-ref %} {% content-ref url="/pages/XKgENzsRIPOmtKvEHZB1" %} [Scaling Feast](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast) {% endcontent-ref %} {% content-ref url="/pages/GEE1TVQCbquh1ud7Jgc6" %} [Structuring Feature Repos](/master/how-to-guides/feast-snowflake-gcp-aws/structuring-repos) {% endcontent-ref %} # Install Feast Install Feast using [pip](https://pip.pypa.io): ``` pip install feast ``` Install Feast with Snowflake dependencies (required when using Snowflake): ``` pip install 'feast[snowflake]' ``` Install Feast with GCP dependencies (required when using BigQuery or Firestore): ``` pip install 'feast[gcp]' ``` Install Feast with AWS dependencies (required when using Redshift or DynamoDB): ``` pip install 'feast[aws]' ``` Install Feast with Redis dependencies (required when using Redis, either through AWS Elasticache or independently): ``` pip install 'feast[redis]' ``` # Create a feature repository A feature repository is a directory that contains the configuration of the feature store and individual features. This configuration is written as code (Python/YAML) and it's highly recommended that teams track it centrally using git. See [Feature Repository](/master/reference/feature-repository) for a detailed explanation of feature repositories. The easiest way to create a new feature repository to use `feast init` command: ```bash feast init Creating a new Feast repository in /<...>/tiny_pika. ``` {% tabs %} {% tab title="Snowflake template" %} ```bash feast init -t snowflake Snowflake Deployment URL: ... Snowflake User Name: ... Snowflake Password: ... Snowflake Role Name: ... Snowflake Warehouse Name: ... Snowflake Database Name: ... Creating a new Feast repository in /<...>/tiny_pika. ``` {% endtab %} {% tab title="GCP template" %} ``` feast init -t gcp Creating a new Feast repository in /<...>/tiny_pika. ``` {% endtab %} {% tab title="AWS template" %} ``` feast init -t aws AWS Region (e.g. us-west-2): ... Redshift Cluster ID: ... Redshift Database Name: ... Redshift User Name: ... Redshift S3 Staging Location (s3://*): ... Redshift IAM Role for S3 (arn:aws:iam::*:role/*): ... Should I upload example data to Redshift (overwriting 'feast_driver_hourly_stats' table)? (Y/n): Creating a new Feast repository in /<...>/tiny_pika. ``` {% endtab %} {% endtabs %} The `init` command creates a Python file with feature definitions, sample data, and a Feast configuration file for local development: ```bash $ tree . └── tiny_pika ├── data │ └── driver_stats.parquet ├── example.py └── feature_store.yaml 1 directory, 3 files ``` Enter the directory: ``` # Replace "tiny_pika" with your auto-generated dir name cd tiny_pika ``` You can now use this feature repository for development. You can try the following: * Run `feast apply` to apply these definitions to Feast. * Edit the example feature definitions in `example.py` and run `feast apply` again to change feature definitions. * Initialize a git repository in the same directory and checking the feature repository into version control. # Deploy a feature store The Feast CLI can be used to deploy a feature store to your infrastructure, spinning up any necessary persistent resources like buckets or tables in data stores. The deployment target and effects depend on the `provider` that has been configured in your [feature\_store.yaml](/master/reference/feature-repository/feature-store-yaml) file, as well as the feature definitions found in your feature repository. {% hint style="info" %} Here we'll be using the example repository we created in the previous guide, [Create a feature store](/master/how-to-guides/feast-snowflake-gcp-aws/create-a-feature-repository). You can re-create it by running `feast init` in a new directory. {% endhint %} ## Deploying To have Feast deploy your infrastructure, run `feast apply` from your command line while inside a feature repository: ```bash feast apply # Processing example.py as example # Done! ``` Depending on whether the feature repository is configured to use a `local` provider or one of the cloud providers like `GCP` or `AWS`, it may take from a couple of seconds to a minute to run to completion. {% hint style="warning" %} At this point, no data has been materialized to your online store. Feast apply simply registers the feature definitions with Feast and spins up any necessary infrastructure such as tables. To load data into the online store, run `feast materialize`. See [Load data into the online store](/master/how-to-guides/feast-snowflake-gcp-aws/load-data-into-the-online-store) for more details. {% endhint %} ## Cleaning up If you need to clean up the infrastructure created by `feast apply`, use the `teardown` command. {% hint style="danger" %} Warning: `teardown` is an irreversible command and will remove all feature store infrastructure. Proceed with caution! {% endhint %} ``` feast teardown ``` \*\*\*\* # Build a training dataset Feast allows users to build a training dataset from time-series feature data that already exists in an offline store. Users are expected to provide a list of features to retrieve (which may span multiple feature views), and a dataframe to join the resulting features onto. Feast will then execute a point-in-time join of multiple feature views onto the provided dataframe, and return the full resulting dataframe. ## Retrieving historical features ### 1. Register your feature views Please ensure that you have created a feature repository and that you have registered (applied) your feature views with Feast. {% content-ref url="/pages/H2HRmgAqoUzCCWwqKo6h" %} [Deploy a feature store](/master/how-to-guides/feast-snowflake-gcp-aws/deploy-a-feature-store) {% endcontent-ref %} ### 2. Define feature references Start by defining the feature references (e.g., `driver_trips:average_daily_rides`) for the features that you would like to retrieve from the offline store. These features can come from multiple feature tables. The only requirement is that the feature tables that make up the feature references have the same entity (or composite entity), and that they aren't located in the same offline store. ```python feature_refs = [ "driver_trips:average_daily_rides", "driver_trips:maximum_daily_rides", "driver_trips:rating", "driver_trips:rating:trip_completed", # Optionally, reference a specific version: "driver_trips@v2:average_daily_rides" ] ``` **3. Create an entity dataframe** An entity dataframe is the target dataframe on which you would like to join feature values. The entity dataframe must contain a timestamp column called `event_timestamp` and all entities (primary keys) necessary to join feature tables onto. All entities found in feature views that are being joined onto the entity dataframe must be found as column on the entity dataframe. It is possible to provide entity dataframes as either a Pandas dataframe or a SQL query. **Pandas:** In the example below we create a Pandas based entity dataframe that has a single row with an `event_timestamp` column and a `driver_id` entity column. Pandas based entity dataframes may need to be uploaded into an offline store, which may result in longer wait times compared to a SQL based entity dataframe. ```python import pandas as pd from datetime import datetime entity_df = pd.DataFrame( { "event_timestamp": [pd.Timestamp(datetime.now(), tz="UTC")], "driver_id": [1001] } ) ``` **SQL (Alternative):** Below is an example of an entity dataframe built from a BigQuery SQL query. It is only possible to use this query when all feature views being queried are available in the same offline store (BigQuery). ```python entity_df = "SELECT event_timestamp, driver_id FROM my_gcp_project.table" ``` **4. Launch historical retrieval** ```python from feast import FeatureStore fs = FeatureStore(repo_path="path/to/your/feature/repo") training_df = fs.get_historical_features( features=[ "driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate" ], entity_df=entity_df ).to_df() ``` Once the feature references and an entity dataframe are defined, it is possible to call `get_historical_features()`. This method launches a job that executes a point-in-time join of features from the offline store onto the entity dataframe. Once completed, a job reference will be returned. This job reference can then be converted to a Pandas dataframe by calling `to_df()`. # Load data into the online store Feast allows users to load their feature data into an online store in order to serve the latest features to models for online prediction. ## Materializing features ### 1. Register feature views Before proceeding, please ensure that you have applied (registered) the feature views that should be materialized. {% content-ref url="/pages/H2HRmgAqoUzCCWwqKo6h" %} [Deploy a feature store](/master/how-to-guides/feast-snowflake-gcp-aws/deploy-a-feature-store) {% endcontent-ref %} ### 2.a Materialize The materialize command allows users to materialize features over a specific historical time range into the online store. ```bash feast materialize 2021-04-07T00:00:00 2021-04-08T00:00:00 ``` The above command will query the batch sources for all feature views over the provided time range, and load the latest feature values into the configured online store. It is also possible to materialize for specific feature views by using the `-v / --views` argument. ``` feast materialize 2021-04-07T00:00:00 2021-04-08T00:00:00 \ --views driver_hourly_stats ``` The materialize command is completely stateless. It requires the user to provide the time ranges that will be loaded into the online store. This command is best used from a scheduler that tracks state, like Airflow. ### 2.b Materialize Incremental (Alternative) For simplicity, Feast also provides a materialize command that will only ingest new data that has arrived in the offline store. Unlike `materialize`, `materialize-incremental` will track the state of previous ingestion runs inside of the feature registry. The example command below will load only new data that has arrived for each feature view up to the end date and time (`2021-04-08T00:00:00`). ``` feast materialize-incremental 2021-04-08T00:00:00 ``` The `materialize-incremental` command functions similarly to `materialize` in that it loads data over a specific time range for all feature views (or the selected feature views) into the online store. Unlike `materialize`, `materialize-incremental` automatically determines the start time from which to load features from batch sources of each feature view. The first time `materialize-incremental` is executed it will set the start time to the oldest timestamp of each data source, and the end time as the one provided by the user. For each run of `materialize-incremental`, the end timestamp will be tracked. Subsequent runs of `materialize-incremental` will then set the start time to the end time of the previous run, thus only loading new data that has arrived into the online store. Note that the end time that is tracked for each run is at the feature view level, not globally for all feature views, i.e, different feature views may have different periods that have been materialized into the online store. # Read features from the online store The Feast Python SDK allows users to retrieve feature values from an online store. This API is used to look up feature values at low latency during model serving in order to make online predictions. {% hint style="info" %} Online stores only maintain the current state of features, i.e latest feature values. No historical data is stored or served. {% endhint %} ## Retrieving online features ### 1. Ensure that feature values have been loaded into the online store Please ensure that you have materialized (loaded) your feature values into the online store before starting {% content-ref url="/pages/O7BuACOUSmntOtavVnVu" %} [Load data into the online store](/master/how-to-guides/feast-snowflake-gcp-aws/load-data-into-the-online-store) {% endcontent-ref %} ### 2. Define feature references Create a list of features that you would like to retrieve. This list typically comes from the model training step and should accompany the model binary. ```python features = [ "driver_hourly_stats:conv_rate", "driver_hourly_stats:acc_rate", # Optionally, reference a specific version (requires enable_online_feature_view_versioning): # "driver_hourly_stats@v2:conv_rate" ] ``` ### 3. Read online features Next, we will create a feature store object and call `get_online_features()` which reads the relevant feature values directly from the online store. ```python fs = FeatureStore(repo_path="path/to/feature/repo") online_features = fs.get_online_features( features=features, entity_rows=[ # {join_key: entity_value, ...} {"driver_id": 1001}, {"driver_id": 1002}] ).to_dict() ``` ```javascript { "driver_hourly_stats__acc_rate":[ 0.2897740304470062, 0.6447265148162842 ], "driver_hourly_stats__conv_rate":[ 0.6508077383041382, 0.14802511036396027 ], "driver_id":[ 1001, 1002 ] } ``` # Scaling Feast ## Overview Feast is designed to be easy to use and understand out of the box, with as few infrastructure dependencies as possible. However, there are components used by default that may not scale well. Since Feast is designed to be modular, it's possible to swap such components with more performant components, at the cost of Feast depending on additional infrastructure. ### Scaling Feast Registry The default Feast [registry](https://github.com/feast-dev/feast/blob/master/docs/getting-started/concepts/registry.md) is a file-based registry. Any changes to the feature repo, or materializing data into the online store, results in a mutation to the registry. However, there are inherent limitations with a file-based registry, since changing a single field in the registry requires re-writing the whole registry file. With multiple concurrent writers, this presents a risk of data loss, or bottlenecks writes to the registry since all changes have to be serialized (e.g. when running materialization for multiple feature views or time ranges concurrently). The recommended solution in this case is to use the [SQL based registry](https://github.com/feast-dev/feast/blob/master/docs/tutorials/using-scalable-registry.md), which allows concurrent, transactional, and fine-grained updates to the registry. This registry implementation requires access to an existing database (such as MySQL, Postgres, etc). ### Scaling Materialization The default Feast materialization process is an in-memory process, which pulls data from the offline store before writing it to the online store. However, this process does not scale for large data sets, since it's executed on a single-process. Feast supports pluggable [Compute Engines](/master/getting-started/components/compute-engine), that allow the materialization process to be scaled up. Aside from the local process, Feast supports a [Lambda-based materialization engine](https://rtd.feast.dev/en/master/#alpha-lambda-based-engine), and a [Bytewax-based materialization engine](https://rtd.feast.dev/en/master/#bytewax-engine). Users may also be able to build an engine to scale up materialization using existing infrastructure in their organizations. ### Horizontal Scaling with the Feast Operator When running Feast on Kubernetes with the [Feast Operator](/master/how-to-guides/feast-on-kubernetes), you can horizontally scale the FeatureStore deployment using `spec.replicas` or HPA autoscaling. The FeatureStore CRD implements the Kubernetes [scale sub-resource](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource), so you can also use `kubectl scale`: ```bash kubectl scale featurestore/my-feast --replicas=3 ``` **Prerequisites:** Horizontal scaling requires **DB-backed persistence** for all enabled services (online store, offline store, and registry). File-based persistence (SQLite, DuckDB, `registry.db`) is incompatible with multiple replicas because these backends do not support concurrent access from multiple pods. #### Static Replicas Set a fixed number of replicas via `spec.replicas`: ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: sample-scaling spec: feastProject: my_project replicas: 3 services: onlineStore: persistence: store: type: postgres secretRef: name: feast-data-stores registry: local: persistence: store: type: sql secretRef: name: feast-data-stores ``` #### Autoscaling with HPA Configure a HorizontalPodAutoscaler to dynamically scale based on metrics. HPA autoscaling is configured under `services.scaling.autoscaling` and is mutually exclusive with `spec.replicas > 1`: ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: sample-autoscaling spec: feastProject: my_project services: scaling: autoscaling: minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 podDisruptionBudgets: maxUnavailable: 1 onlineStore: persistence: store: type: postgres secretRef: name: feast-data-stores server: resources: requests: cpu: 200m memory: 256Mi registry: local: persistence: store: type: sql secretRef: name: feast-data-stores ``` {% hint style="info" %} When autoscaling is configured, the operator automatically sets the deployment strategy to `RollingUpdate` (instead of the default `Recreate`) to ensure zero-downtime scaling, and auto-injects soft pod anti-affinity and zone topology spread constraints. You can override any of these by explicitly setting `deploymentStrategy`, `affinity`, or `topologySpreadConstraints` in the CR. {% endhint %} #### Validation Rules The operator enforces the following rules: * `spec.replicas > 1` and `services.scaling.autoscaling` are **mutually exclusive** -- you cannot set both. * Scaling with `replicas > 1` or any `autoscaling` config is **rejected** if any enabled service uses file-based persistence. * S3 (`s3://`) and GCS (`gs://`) backed registry file persistence is allowed with scaling, since these object stores support concurrent readers. #### High Availability When scaling is enabled (`replicas > 1` or `autoscaling`), the operator provides HA features to improve resilience: **Pod Anti-Affinity** — The operator automatically injects a soft (`preferredDuringSchedulingIgnoredDuringExecution`) pod anti-affinity rule that prefers spreading pods across different nodes. This prevents multiple replicas from being co-located on the same node, improving resilience to node failures. You can override this by providing your own `affinity` configuration: ```yaml spec: replicas: 3 services: # Override with custom affinity (e.g. strict anti-affinity) affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - topologyKey: kubernetes.io/hostname labelSelector: matchLabels: feast.dev/name: my-feast # ... ``` **Topology Spread Constraints** — The operator automatically injects a soft zone-spread constraint (`whenUnsatisfiable: ScheduleAnyway`) that distributes pods across availability zones. This is a best-effort spread — if zones are unavailable, pods will still be scheduled. You can override this with explicit constraints or disable it with an empty array: ```yaml spec: replicas: 3 services: # Override with custom topology spread (e.g. strict zone spreading) topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: feast.dev/name: my-feast # ... ``` To disable the auto-injected topology spread: ```yaml spec: replicas: 3 services: topologySpreadConstraints: [] # ... ``` **PodDisruptionBudget** — You can configure a PDB to limit voluntary disruptions (e.g. during node drains or cluster upgrades). The PDB is only created when scaling is enabled. Exactly one of `minAvailable` or `maxUnavailable` must be set: ```yaml spec: replicas: 3 services: podDisruptionBudgets: maxUnavailable: 1 # at most 1 pod unavailable during disruptions # -- OR -- # podDisruptionBudgets: # minAvailable: "50%" # at least 50% of pods must remain available # ... ``` {% hint style="info" %} The PDB is not auto-injected — you must explicitly configure it. This is intentional because a misconfigured PDB (e.g. `minAvailable` equal to the replica count) can block node drains and cluster upgrades. {% endhint %} #### Using KEDA (Kubernetes Event-Driven Autoscaling) [KEDA](https://keda.sh) is also supported as an external autoscaler. KEDA should target the FeatureStore's scale sub-resource directly (since it implements the Kubernetes scale API). This is the recommended approach because the operator manages the Deployment's replica count from `spec.replicas` — targeting the Deployment directly would conflict with the operator's reconciliation. When using KEDA, do **not** set `scaling.autoscaling` or `spec.replicas > 1` -- KEDA manages the replica count through the scale sub-resource. 1. **Ensure DB-backed persistence** -- The CRD's CEL validation rules automatically enforce DB-backed persistence when KEDA scales `spec.replicas` above 1 via the scale sub-resource. The operator also automatically switches the deployment strategy to `RollingUpdate` when `replicas > 1`. 2. **Configure the FeatureStore** with DB-backed persistence: ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: sample-keda spec: feastProject: my_project services: onlineStore: persistence: store: type: postgres secretRef: name: feast-data-stores registry: local: persistence: store: type: sql secretRef: name: feast-data-stores ``` 3. **Create a KEDA `ScaledObject`** targeting the FeatureStore resource: ```yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: feast-scaledobject spec: scaleTargetRef: apiVersion: feast.dev/v1 kind: FeatureStore name: sample-keda minReplicaCount: 1 maxReplicaCount: 10 triggers: - type: prometheus metadata: serverAddress: http://prometheus.monitoring.svc:9090 metricName: http_requests_total query: sum(rate(http_requests_total{service="feast"}[2m])) threshold: "100" ``` {% hint style="warning" %} KEDA-created HPAs are not owned by the Feast operator. The operator will not interfere with them, but it also will not clean them up if the FeatureStore CR is deleted. You must manage the KEDA `ScaledObject` lifecycle independently. {% endhint %} For the full API reference, see the [FeatureStore CRD reference](https://github.com/feast-dev/feast/blob/master/infra/feast-operator/docs/api/markdown/ref.md). # Structuring Feature Repos A common scenario when using Feast in production is to want to test changes to Feast object definitions. For this, we recommend setting up a *staging* environment for your offline and online stores, which mirrors *production* (with potentially a smaller data set). Having this separate environment allows users to test changes by first applying them to staging, and then promoting the changes to production after verifying the changes on staging. ## Setting up multiple environments There are three common ways teams approach having separate environments 1. Have separate git branches for each environment 2. Have separate `feature_store.yaml` files and separate Feast object definitions that correspond to each environment 3. Have separate `feature_store.yaml` files per environment, but share the Feast object definitions ### Different version control branches To keep a clear separation of the feature repos, teams may choose to have multiple long-lived branches in their version control system, one for each environment. In this approach, with CI/CD setup, changes would first be made to the staging branch, and then copied over manually to the production branch once verified in the staging environment. ### Separate `feature_store.yaml` files and separate Feast object definitions For this approach, we have created an example repository ([Feast Repository Example](https://github.com/feast-dev/feast-ci-repo-example)) which contains two Feast projects, one per environment. The contents of this repository are shown below: ``` ├── .github │ └── workflows │ ├── production.yml │ └── staging.yml │ ├── staging │ ├── driver_repo.py │ └── feature_store.yaml │ └── production ├── driver_repo.py └── feature_store.yaml ``` The repository contains three sub-folders: * `staging/`: This folder contains the staging `feature_store.yaml` and Feast objects. Users that want to make changes to the Feast deployment in the staging environment will commit changes to this directory. * `production/`: This folder contains the production `feature_store.yaml` and Feast objects. Typically users would first test changes in staging before copying the feature definitions into the production folder, before committing the changes. * `.github`: This folder is an example of a CI system that applies the changes in either the `staging` or `production` repositories using `feast apply`. This operation saves your feature definitions to a shared registry (for example, on GCS) and configures your infrastructure for serving features. The `feature_store.yaml` contains the following: ``` project: staging registry: gs://feast-ci-demo-registry/staging/registry.db provider: gcp ``` Notice how the registry has been configured to use a Google Cloud Storage bucket. All changes made to infrastructure using `feast apply` are tracked in the `registry.db`. This registry will be accessed later by the Feast SDK in your training pipelines or model serving services in order to read features. {% hint style="success" %} It is important to note that the CI system above must have access to create, modify, or remove infrastructure in your production environment. This is unlike clients of the feature store, who will only have read access. {% endhint %} If your organization consists of many independent data science teams or a single group is working on several projects that could benefit from sharing features, entities, sources, and transformations, then we encourage you to utilize Python packages inside each environment: ``` └── production ├── common │ ├── __init__.py │ ├── sources.py │ └── entities.py ├── ranking │ ├── __init__.py │ ├── views.py │ └── transformations.py ├── segmentation │ ├── __init__.py │ ├── views.py │ └── transformations.py └── feature_store.yaml ``` ### Shared Feast Object definitions with separate `feature_store.yaml` files This approach is very similar to the previous approach, but instead of having feast objects duplicated and having to copy over changes, it may be possible to share the same Feast object definitions and have different `feature_store.yaml` configuration. An example of how such a repository would be structured is as follows: ``` ├── .github │ └── workflows │ ├── production.yml │ └── staging.yml ├── staging │ └── feature_store.yaml ├── production │ └── feature_store.yaml └── driver_repo.py ``` Users can then apply the applying them to each environment in this way: ```shell feast -f staging/feature_store.yaml apply ``` This setup has the advantage that you can share the feature definitions entirely, which may prevent issues with copy-pasting code. ## Summary In summary, once you have set up a Git based repository with CI that runs `feast apply` on changes, your infrastructure (offline store, online store, and cloud environment) will automatically be updated to support the loading of data into the feature store or retrieval of data. # Multi-Team Feature Store Setup A multi-team feature store architecture (sometimes called a "federated" feature store) allows multiple teams to collaborate on a shared Feast registry while maintaining clear ownership boundaries. This pattern is particularly useful for organizations with multiple teams or projects that need to share features while preserving autonomy. ## Overview In a multi-team setup, you typically have: * **Platform Repository (Central)**: A single repository managed by the platform team that acts as the source of truth for core infrastructure objects like entities, data sources, and batch/stream feature views. * **Team Repositories (Distributed)**: Team-owned repositories for training pipelines and inference services. Teams define their own FeatureServices and On-Demand Feature Views (ODFVs) and apply them safely without affecting objects they don't own. ## Key Concept: Understanding `partial=True` vs `partial=False` The `partial` parameter in `FeatureStore.apply()` controls how Feast handles object deletions: * **`partial=True` (default)**: Only adds or updates the specified objects. Does NOT delete any existing objects in the registry. This is safe for teams to use when they only want to manage a subset of objects. * **`partial=False`**: Performs a full synchronization - adds, updates, AND deletes objects. Objects not in the provided list (but tracked by Feast) will be removed from the registry. This should only be used by the platform team with full control. ```python # Team repository - safe partial apply # Only adds/updates the specified FeatureService # Does NOT delete any other objects store.apply([my_feature_service], partial=True) # Platform repository - full sync with deletion capability # Syncs all objects and can remove objects via objects_to_delete store.apply(all_objects, objects_to_delete=objects_to_remove, partial=False) ``` ## Architecture Diagram ``` ┌─────────────────────────────────────────────────────────────────┐ │ Platform Repository │ │ (Central, platform team) │ │ │ │ ├── feature_store.yaml (shared registry config) │ │ ├── entities.py (Entity definitions) │ │ ├── data_sources.py (DataSource definitions) │ │ └── feature_views.py (FeatureView, StreamFeatureView) │ │ │ │ Applies with partial=False (full control) │ └─────────────────────────────────────────────────────────────────┘ │ │ Shared Registry (e.g., S3, GCS, SQL) │ ┌───────────────┼───────────────┐ │ │ │ ┌─────────▼────────┐ ┌───▼────────┐ ┌────▼──────────┐ │ Team A Repo │ │ Team B │ │ Team C │ │ (Distributed) │ │ Repo │ │ Repo │ │ │ │ │ │ │ │ ├── fs.yaml │ │ ├── ... │ │ ├── ... │ │ ├── odfvs.py │ │ └── ... │ │ └── ... │ │ └── services.py │ │ │ │ │ │ │ │ │ │ │ │ partial=True │ │ partial= │ │ partial=True │ │ (safe add) │ │ True │ │ │ └──────────────────┘ └────────────┘ └───────────────┘ ``` ## Setting Up the Platform Repository The platform repository maintains full control over core Feast objects. ### 1. Platform `feature_store.yaml` ```yaml project: my_feast_project registry: s3://my-bucket/feast-registry/registry.db provider: aws online_store: type: dynamodb region: us-west-2 offline_store: type: snowflake account: my_account database: FEAST_DB warehouse: FEAST_WH ``` ### 2. Platform Repository Structure ``` platform_repo/ ├── feature_store.yaml ├── entities.py ├── data_sources.py ├── feature_views.py └── apply.py ``` ### 3. Platform Object Definitions **entities.py** ```python from feast import Entity # Core entities managed by platform team user = Entity( name="user_id", description="User entity", join_keys=["user_id"] ) driver = Entity( name="driver_id", description="Driver entity", join_keys=["driver_id"] ) ``` **data\_sources.py** ```python from feast import SnowflakeSource # Core data sources managed by platform team user_features_source = SnowflakeSource( database="FEAST_DB", schema="FEATURES", table="USER_FEATURES", timestamp_field="event_timestamp", ) driver_stats_source = SnowflakeSource( database="FEAST_DB", schema="FEATURES", table="DRIVER_STATS", timestamp_field="event_timestamp", ) ``` **feature\_views.py** ```python from datetime import timedelta from feast import FeatureView, Field from feast.types import Float32, Int64 from entities import user, driver from data_sources import user_features_source, driver_stats_source # Core feature views managed by platform team user_features = FeatureView( name="user_features", entities=[user], ttl=timedelta(days=1), schema=[ Field(name="age", dtype=Int64), Field(name="account_balance", dtype=Float32), ], source=user_features_source, ) driver_stats = FeatureView( name="driver_stats", entities=[driver], ttl=timedelta(days=1), schema=[ Field(name="conv_rate", dtype=Float32), Field(name="acc_rate", dtype=Float32), ], source=driver_stats_source, ) ``` ### 4. Platform Apply Script **apply.py** ```python from feast import FeatureStore from entities import user, driver from data_sources import user_features_source, driver_stats_source from feature_views import user_features, driver_stats # Initialize the feature store store = FeatureStore(repo_path=".") # Collect all objects managed by platform all_objects = [ user, driver, user_features_source, driver_stats_source, user_features, driver_stats, ] # Apply with partial=False to maintain full control # This allows the platform team to manage deletions store.apply(all_objects, partial=False) print("Platform objects applied successfully!") ``` ## Setting Up Team Repositories Team repositories add their own FeatureServices and ODFVs without interfering with platform-managed objects. ### 1. Team `feature_store.yaml` Teams use the **same registry configuration** as the platform: ```yaml project: my_feast_project # Same project as platform registry: s3://my-bucket/feast-registry/registry.db # Same registry provider: aws online_store: type: dynamodb region: us-west-2 offline_store: type: snowflake account: my_account database: FEAST_DB warehouse: FEAST_WH ``` ### 2. Team Repository Structure ``` team_a_repo/ ├── feature_store.yaml ├── on_demand_views.py ├── feature_services.py └── apply.py ``` ### 3. Team Object Definitions **on\_demand\_views.py** ```python from feast import FeatureStore, Field, OnDemandFeatureView, RequestSource from feast.types import Float32 # Team A's custom on-demand feature view # Uses platform-managed feature views as inputs @OnDemandFeatureView( sources=[ "user_features:age", # References platform feature view RequestSource(schema=[Field(name="current_year", dtype=Float32)]), ], schema=[Field(name="years_until_retirement", dtype=Float32)], ) def user_age_odfv(inputs): """Calculate years until retirement.""" return {"years_until_retirement": 65 - inputs["age"]} ``` **feature\_services.py** ```python from feast import FeatureService, FeatureStore # Initialize feature store to get references store = FeatureStore(repo_path=".") # Team A's feature service for their ML model user_model_features = FeatureService( name="user_model_v1", features=[ "user_features", # Platform feature view "user_age_odfv", # Team's ODFV ], ) ``` ### 4. Team Apply Script **apply.py** ```python from feast import FeatureStore from on_demand_views import user_age_odfv from feature_services import user_model_features # Initialize the feature store store = FeatureStore(repo_path=".") # Collect team's objects team_objects = [ user_age_odfv, user_model_features, ] # Apply with partial=True (safe for teams) # Only adds/updates team's objects, does NOT delete anything store.apply(team_objects, partial=True) print("Team objects applied successfully!") ``` ## Ownership Boundaries | Object Type | Platform Repository | Team Repositories | | ----------------------- | ------------------- | ----------------- | | Entities | ✅ Yes | ❌ No | | Data Sources | ✅ Yes | ❌ No | | Feature Views | ✅ Yes | ❌ No | | Stream Feature Views | ✅ Yes | ❌ No | | Feature Services | Optional | ✅ Yes | | On-Demand Feature Views | Optional | ✅ Yes | ## Strategies for Avoiding Registry Drift ### 1. Naming Conventions Establish clear naming conventions to prevent collisions: ```python # Platform objects: simple names user_features = FeatureView(name="user_features", ...) # Team objects: prefixed with team name team_a_user_service = FeatureService(name="team_a_user_model", ...) team_b_driver_service = FeatureService(name="team_b_driver_model", ...) ``` ### 2. CI/CD Integration Use CI/CD pipelines to automate applies and catch errors early: ```yaml # .github/workflows/apply-platform.yml name: Apply Platform Features on: push: branches: [main] paths: - 'platform_repo/**' jobs: apply: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Apply platform features run: | cd platform_repo pip install feast[aws,snowflake] python apply.py ``` ```yaml # .github/workflows/apply-team-a.yml name: Apply Team A Features on: push: branches: [main] paths: - 'team_a_repo/**' jobs: apply: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Apply team A features run: | cd team_a_repo pip install feast[aws,snowflake] python apply.py ``` ### 3. Registry Validation Implement validation to check for ownership violations: ```python def validate_team_objects(team_name, objects): """Validate that team only creates allowed object types.""" allowed_types = (FeatureService, OnDemandFeatureView) for obj in objects: if not isinstance(obj, allowed_types): raise ValueError( f"Team {team_name} cannot create {type(obj).__name__}. " f"Only FeatureServices and ODFVs are allowed." ) # Check naming convention if not obj.name.startswith(f"{team_name}_"): raise ValueError( f"Team {team_name} objects must be prefixed with '{team_name}_'. " f"Got: {obj.name}" ) # In team apply script validate_team_objects("team_a", team_objects) store.apply(team_objects, partial=True) ``` ### 4. Read-Only Access for Teams Configure infrastructure permissions so teams have: * **Read access** to the registry (to get platform objects) * **Write access** only for their specific object types * **No delete permissions** on registry objects ## Multi-Tenant Configuration with Schema Isolation For stronger isolation, use separate database schemas per team: ### Platform Configuration ```yaml # platform_repo/feature_store.yaml project: my_feast_project registry: s3://my-bucket/feast-registry/registry.db provider: aws offline_store: type: snowflake account: my_account database: FEAST_DB schema: PLATFORM # Platform schema warehouse: FEAST_WH ``` ### Team Configuration with Custom Schema ```yaml # team_a_repo/feature_store.yaml project: my_feast_project registry: s3://my-bucket/feast-registry/registry.db provider: aws offline_store: type: snowflake account: my_account database: FEAST_DB schema: TEAM_A # Team-specific schema warehouse: FEAST_WH ``` This approach allows teams to materialize features to their own schemas while sharing the central registry. ## Complete Working Example Here's a complete example showing both platform and team repositories in action: ### Platform Repository ```python # platform_repo/apply.py from feast import Entity, FeatureStore, FeatureView, Field, SnowflakeSource from feast.types import Float32, Int64 from datetime import timedelta # Initialize feature store store = FeatureStore(repo_path=".") # Define entities customer = Entity(name="customer_id", join_keys=["customer_id"]) # Define data source customer_source = SnowflakeSource( database="FEAST_DB", schema="RAW", table="CUSTOMER_FEATURES", timestamp_field="event_timestamp", ) # Define feature view customer_features = FeatureView( name="customer_features", entities=[customer], ttl=timedelta(days=7), schema=[ Field(name="total_purchases", dtype=Int64), Field(name="avg_order_value", dtype=Float32), ], source=customer_source, ) # Apply all platform objects with full control platform_objects = [customer, customer_source, customer_features] store.apply(platform_objects, partial=False) print("✅ Platform objects applied") ``` ### Team Repository ```python # team_marketing_repo/apply.py from feast import FeatureStore, FeatureService, Field, OnDemandFeatureView, RequestSource from feast.types import Float32, Int64 # Initialize feature store (same registry as platform) store = FeatureStore(repo_path=".") # Define on-demand feature view @OnDemandFeatureView( sources=[ "customer_features:total_purchases", "customer_features:avg_order_value", RequestSource(schema=[Field(name="current_cart_value", dtype=Float32)]), ], schema=[ Field(name="purchase_propensity_score", dtype=Float32), ], ) def marketing_propensity(inputs): """Calculate purchase propensity based on history and current cart.""" purchases = inputs["total_purchases"] avg_value = inputs["avg_order_value"] cart_value = inputs["current_cart_value"] # Simple propensity calculation propensity = (purchases * 0.3 + (cart_value / max(avg_value, 1)) * 0.7) return {"purchase_propensity_score": min(propensity, 100.0)} # Define feature service for team's ML model marketing_campaign_service = FeatureService( name="team_marketing_campaign_model", features=[ "customer_features", # Platform feature view "marketing_propensity", # Team's ODFV ], ) # Apply team objects safely with partial=True team_objects = [marketing_propensity, marketing_campaign_service] store.apply(team_objects, partial=True) print("✅ Team marketing objects applied") ``` ### Using the Features ```python # In team's training or inference code from feast import FeatureStore store = FeatureStore(repo_path=".") # Get online features using team's feature service features = store.get_online_features( features="team_marketing_campaign_model", entity_rows=[ {"customer_id": "C123", "current_cart_value": 150.0} ], ).to_dict() print(features) # Output includes both platform features and team's ODFV: # { # "customer_id": ["C123"], # "total_purchases": [42], # "avg_order_value": [125.50], # "purchase_propensity_score": [42.3] # } ``` ## Best Practices 1. **Platform Team Responsibilities**: * Maintain core entities and data sources * Define and manage batch/stream feature views * Use `partial=False` to maintain full control * Document available features for teams to use 2. **Team Responsibilities**: * Always use `partial=True` when applying objects * Follow naming conventions (prefix with team name) * Only create FeatureServices and ODFVs * Reference platform feature views by name, don't redefine them 3. **Registry Management**: * Use a centralized remote registry (S3, GCS, SQL) * Implement access controls at the infrastructure level * Set up monitoring for registry changes * Regular backups of registry state 4. **Testing Strategy**: * Use staging registries for testing changes * Validate team applies don't modify platform objects * Test feature retrieval end-to-end before production deployment 5. **Documentation**: * Maintain a catalog of available platform features * Document team ownership of FeatureServices and ODFVs * Keep architecture diagrams up to date ## Troubleshooting ### Problem: Team accidentally deleted platform objects **Cause**: Team used `partial=False` instead of `partial=True`. **Solution**: * Restore from registry backup * Add validation in team CI/CD to prevent `partial=False` * Set registry permissions to prevent teams from deleting objects ### Problem: Feature service references unknown feature view **Cause**: Feature view not yet applied by platform team. **Solution**: * Ensure platform applies are run before team applies * Use CI/CD dependencies to enforce ordering * Add validation to check that referenced feature views exist ### Problem: Registry conflicts between teams **Cause**: Teams using same object names. **Solution**: * Enforce naming conventions with prefixes * Add validation to check for name collisions * Consider using separate projects per team (advanced) ## Related Resources * [Structuring Feature Repos](/master/how-to-guides/feast-snowflake-gcp-aws/structuring-repos) - Multi-environment setup patterns * [Remote Registry](/master/reference/registries/remote) - Remote registry configuration * [Registry Server](/master/reference/feature-servers/registry-server) - Registry server setup * [Feature Store API Reference](http://rtd.feast.dev/en/stable/index.html#feast.FeatureStore.apply) - `apply()` method documentation ## Summary A multi-team feature store architecture enables scalable collaboration across multiple teams: * The **platform team** maintains core objects (entities, sources, views) with `partial=False` * **Individual teams** safely add their FeatureServices and ODFVs with `partial=True` * Clear ownership boundaries prevent accidental deletions and conflicts * Proper CI/CD, naming conventions, and validation ensure smooth operation This pattern (sometimes referred to as "federated" feature stores) allows organizations to scale Feast usage across many teams while maintaining a consistent, well-governed feature store. # Running Feast in production (e.g. on Kubernetes) ## Overview After learning about Feast concepts and playing with Feast locally, you're now ready to use Feast in production. This guide aims to help with the transition from a sandbox project to production-grade deployment in the cloud or on-premise (e.g. on Kubernetes). A typical production architecture looks like: ![Overview](/files/kBojPHK0DcjLJcDscOlt) {% hint style="success" %} **Important note:** Feast is highly customizable and modular. Most Feast blocks are loosely connected and can be used independently. Hence, you are free to build your own production configuration. For example, you might not have a stream source and, thus, no need to write features in real-time to an online store. Or you might not need to retrieve online features. Feast also often provides multiple options to achieve the same goal. We discuss tradeoffs below. Additionally, please check the how-to guide for some specific recommendations on [how to scale Feast](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast). {% endhint %} {% hint style="info" %} **Looking for production deployment patterns?** See the [Feast Production Deployment Topologies](/master/how-to-guides/production-deployment-topologies) guide for three Kubernetes-ready topologies (Minimal, Standard, Enterprise), sample FeatureStore CRs, RBAC policies, infrastructure recommendations, and scaling best practices. {% endhint %} In this guide we will show you how to: 1. Deploy your feature store and keep your infrastructure in sync with your feature repository 2. Keep the data in your online store up to date (from batch and stream sources) 3. Use Feast for model training and serving ## 1. Automatically deploying changes to your feature definitions ### 1.1 Setting up a feature repository The first step to setting up a deployment of Feast is to create a Git repository that contains your feature definitions. The recommended way to version and track your feature definitions is by committing them to a repository and tracking changes through commits. If you recall, running `feast apply` commits feature definitions to a **registry**, which users can then read elsewhere. ### 1.2 Setting up a database-backed registry Out of the box, Feast serializes all of its state into a file-based registry. When running Feast in production, we recommend using the more scalable SQL-based registry that is backed by a database. Details are available [here](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast#scaling-feast-registry). > **Note:** A SQL-based registry primarily works with a Python feature server. The Java feature server does not understand this registry type yet. ### 1.3 Setting up CI/CD to automatically update the registry We recommend typically setting up CI/CD to automatically run `feast plan` and `feast apply` when pull requests are opened / merged. ### 1.4 Setting up multiple environments A common scenario when using Feast in production is to want to test changes to Feast object definitions. For this, we recommend setting up a *staging* environment for your offline and online stores, which mirrors *production* (with potentially a smaller data set). Having this separate environment allows users to test changes by first applying them to staging, and then promoting the changes to production after verifying the changes on staging. Different options are presented in the [how-to guide](/master/how-to-guides/feast-snowflake-gcp-aws/structuring-repos). ## 2. How to load data into your online store and keep it up to date To keep your online store up to date, you need to run a job that loads feature data from your feature view sources into your online store. In Feast, this loading operation is called materialization. ### 2.1 Scalable Materialization Out of the box, Feast's materialization process uses an in-process materialization engine. This engine loads all the data being materialized into memory from the offline store, and writes it into the online store. This approach may not scale to large amounts of data, which users of Feast may be dealing with in production. In this case, we recommend using one of the more [scalable compute engines](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast#scaling-materialization), such as [Snowflake Compute Engine](/master/reference/compute-engine/snowflake). Users may also need to [write a custom compute engine](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/customizing-feast/creating-a-custom-compute-engine.md) to work on their existing infrastructure. ### 2.2 Scheduled materialization with Airflow > See also [data ingestion](/master/getting-started/concepts/data-ingestion#batch-data-ingestion) for code snippets It is up to you to orchestrate and schedule runs of materialization. Feast keeps the history of materialization in its registry so that the choice could be as simple as a [unix cron util](https://en.wikipedia.org/wiki/Cron). Cron util should be sufficient when you have just a few materialization jobs (it's usually one materialization job per feature view) triggered infrequently. However, the amount of work can quickly outgrow the resources of a single machine. That happens because the materialization job needs to repackage all rows before writing them to an online store. That leads to high utilization of CPU and memory. In this case, you might want to use a job orchestrator to run multiple jobs in parallel using several workers. Kubernetes Jobs or Airflow are good choices for more comprehensive job orchestration. For large datasets, you can also reduce peak memory on the materialization worker by setting `online_write_batch_size` in `feature_store.yaml`. This breaks the proto conversion and write into chunks instead of loading the entire dataset into memory at once: ```yaml materialization: online_write_batch_size: 10000 # rows per write batch; reduces peak memory proportionally ``` See the [Materialization write performance](/master/how-to-guides/online-server-performance-tuning#materialization-write-performance) guide for sizing recommendations and the full option reference in [feature\_store.yaml](/master/reference/feature-repository/feature-store-yaml#online_write_batch_size). If you are using Airflow as a scheduler, Feast can be invoked through a [PythonOperator](https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html) after the [Python SDK](https://pypi.org/project/feast/) has been installed into a virtual environment and your feature repo has been synced: ```python from airflow.decorators import task from feast import RepoConfig, FeatureStore from feast.infra.online_stores.dynamodb import DynamoDBOnlineStoreConfig from feast.repo_config import RegistryConfig # Define Python callable @task() def materialize(data_interval_start=None, data_interval_end=None): repo_config = RepoConfig( registry=RegistryConfig(path="s3://[YOUR BUCKET]/registry.pb"), project="feast_demo_aws", provider="aws", offline_store="file", online_store=DynamoDBOnlineStoreConfig(region="us-west-2"), entity_key_serialization_version=3 ) store = FeatureStore(config=repo_config) # Option 1: materialize just one feature view # store.materialize_incremental(datetime.datetime.now(), feature_views=["my_fv_name"]) # Option 2: materialize all feature views incrementally # store.materialize_incremental(datetime.datetime.now()) # Option 3: Let Airflow manage materialization state # Add 1 hr overlap to account for late data store.materialize(data_interval_start.subtract(hours=1), data_interval_end) ``` You can see more in an example at [Feast Workshop - Module 1](https://github.com/feast-dev/feast-workshop/blob/main/module_1/README.md#step-7-scaling-up-and-scheduling-materialization). {% hint style="success" %} Important note: Airflow worker must have read and write permissions to the registry file on GCS / S3 since it pulls configuration and updates materialization history. {% endhint %} ### 2.3 Stream feature ingestion See more details at [data ingestion](/master/getting-started/concepts/data-ingestion), which shows how to ingest streaming features or 3rd party feature data via a push API. This supports pushing feature values into Feast to both online or offline stores. ### 2.4 Scheduled batch transformations with Airflow + dbt Feast does not orchestrate batch transformation DAGs. For this, you can rely on tools like Airflow + dbt. See [Feast Workshop - Module 3](https://github.com/feast-dev/feast-workshop/blob/main/module_3/) for an example and some tips. ## 3. How to use Feast for model training ### 3.1. Generating training data > For more details, see [feature retrieval](/master/getting-started/concepts/feature-retrieval#retrieving-historical-features-for-training-data-or-batch-scoring) After we've defined our features and data sources in the repository, we can generate training datasets. We highly recommend you use a `FeatureService` to version the features that go into a specific model version. 1. The first thing we need to do in our training code is to create a `FeatureStore` object with a path to the registry. * One way to ensure your production clients have access to the feature store is to provide a copy of the `feature_store.yaml` to those pipelines. This `feature_store.yaml` file will have a reference to the feature store registry, which allows clients to retrieve features from offline or online stores. ```python from feast import FeatureStore fs = FeatureStore(repo_path="production/") ``` 2. Then, you need to generate an **entity dataframe**. You have two options * Create an entity dataframe manually and pass it in * Use a SQL query to dynamically generate lists of entities (e.g. all entities within a time range) and timestamps to pass into Feast 3. Then, training data can be retrieved as follows: ```python training_retrieval_job = fs.get_historical_features( entity_df=entity_df_or_sql_string, features=fs.get_feature_service("driver_activity_v1"), ) # Option 1: In memory model training model = ml.fit(training_retrieval_job.to_df()) # Option 2: Unloading to blob storage. Further post-processing can occur before kicking off distributed training. training_retrieval_job.to_remote_storage() ``` ### 3.2 Versioning features that power ML models The most common way to productionize ML models is by storing and versioning models in a "model store", and then deploying these models into production. When using Feast, it is recommended that the feature service name and the model versions have some established convention. For example, in MLflow: ```python import mlflow.pyfunc # Load model from MLflow model_name = "my-model" model_version = 1 model = mlflow.pyfunc.load_model( model_uri=f"models:/{model_name}/{model_version}" ) fs = FeatureStore(repo_path="production/") # Read online features using the same model name and model version feature_vector = fs.get_online_features( features=fs.get_feature_service(f"{model_name}_v{model_version}"), entity_rows=[{"driver_id": 1001}] ).to_dict() # Make a prediction prediction = model.predict(feature_vector) ``` {% hint style="success" %} It is important to note that both the training pipeline and model serving service need only read access to the feature registry and associated infrastructure. This prevents clients from accidentally making changes to the feature store. {% endhint %} ## 4. Retrieving online features for prediction Once you have successfully loaded data from batch / streaming sources into the online store, you can start consuming features for model inference. ### 4.1. Use the Python SDK within an existing Python service This approach is the most convenient to keep your infrastructure as minimalistic as possible and avoid deploying extra services. The Feast Python SDK will connect directly to the online store (Redis, Datastore, etc), pull the feature data, and run transformations locally (if required). The obvious drawback is that your service must be written in Python to use the Feast Python SDK. A benefit of using a Python stack is that you can enjoy production-grade services with integrations with many existing data science tools. To integrate online retrieval into your service use the following code: ```python from feast import FeatureStore with open('feature_refs.json', 'r') as f: feature_refs = json.loads(f) fs = FeatureStore(repo_path="production/") # Read online features feature_vector = fs.get_online_features( features=feature_refs, entity_rows=[{"driver_id": 1001}] ).to_dict() ``` ### 4.2. Deploy Feast feature servers on Kubernetes See [Feast on Kubernetes](/master/how-to-guides/feast-on-kubernetes). ## 5. Using environment variables in your yaml configuration You might want to dynamically set parts of your configuration from your environment. For instance to deploy Feast to production and development with the same configuration, but a different server. Or to inject secrets without exposing them in your git repo. To do this, it is possible to use the `${ENV_VAR}` syntax in your `feature_store.yaml` file. For instance: ```yaml project: my_project registry: data/registry.db provider: local online_store: type: redis connection_string: ${REDIS_CONNECTION_STRING} ``` *** ## Summary In summary, the overall architecture in production may look like: * Feast SDK is being triggered by CI (eg, Github Actions). It applies the latest changes from the feature repo to the Feast database-backed registry * Data ingestion * **Batch data**: Airflow manages batch transformation jobs + materialization jobs to ingest batch data from DWH to the online store periodically. When working with large datasets to materialize, we recommend using a batch materialization engine * If your offline and online workloads are in Snowflake, the Snowflake materialization engine is likely the best option. * If your offline and online workloads are not using Snowflake, but using Kubernetes is an option, the Bytewax materialization engine is likely the best option. * If none of these engines suite your needs, you may continue using the in-process engine, or write a custom engine (e.g with Spark or Ray). * **Stream data**: The Feast Push API is used within existing Spark / Beam pipelines to push feature values to offline / online stores * Online features are served via the Python feature server over HTTP, or consumed using the Feast Python SDK. * Feast Python SDK is called locally to generate a training dataset ![From Repository to Production: Feast Production Architecture](/files/p0iudI9QXdrF7g5w1Wrf) # Feast on Kubernetes This page covers deploying Feast on Kubernetes, including the Feast Operator and feature servers. ## Overview Kubernetes is a common target environment for running Feast in production. You can use Kubernetes to: 1. Run Feast feature servers for online feature retrieval. 2. Run scheduled and ad-hoc jobs (e.g. materialization jobs) as Kubernetes Jobs. 3. Operate Feast components using Kubernetes-native primitives. {% hint style="info" %} **Planning a production deployment?** See the [Feast Production Deployment Topologies](/master/how-to-guides/production-deployment-topologies) guide for architecture diagrams, sample FeatureStore CRs, RBAC policies, infrastructure recommendations, and scaling best practices across Minimal, Standard, and Enterprise topologies. {% endhint %} ## Feast Operator To deploy Feast components on Kubernetes, use the included [feast-operator](https://github.com/feast-dev/feast/blob/master/infra/feast-operator/README.md). For first-time Operator users, it may be a good exercise to try the [Feast Operator Quickstart](https://github.com/feast-dev/feast/blob/master/examples/operator-quickstart/README.md). The quickstart demonstrates some of the Operator's built-in features, e.g. git repos, `feast apply` jobs, etc. ## Deploy Feast feature servers on Kubernetes {% embed url="" %} **Basic steps** 1. Install [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) 2. Install the Operator Install the latest release: ```sh kubectl apply -f https://raw.githubusercontent.com/feast-dev/feast/refs/heads/stable/infra/feast-operator/dist/install.yaml ``` OR, install a specific version: ```sh kubectl apply -f https://raw.githubusercontent.com/feast-dev/feast/refs/tags//infra/feast-operator/dist/install.yaml ``` 3. Deploy a Feature Store ```sh kubectl apply -f https://raw.githubusercontent.com/feast-dev/feast/refs/heads/stable/infra/feast-operator/config/samples/v1_featurestore.yaml ``` Verify the status: ```sh $ kubectl get feast NAME STATUS AGE sample Ready 2m21s ``` The above will install a simple [FeatureStore CR](https://github.com/feast-dev/feast/blob/master/infra/feast-operator/docs/api/markdown/ref.md) like the following. By default, it will run the [Online Store feature server](/master/reference/feature-servers/python-feature-server): ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: sample spec: feastProject: my_project ``` > *More advanced FeatureStore CR examples can be found in the feast-operator* [*samples directory*](https://github.com/feast-dev/feast/blob/master/infra/feast-operator/config/samples/README.md)*.* {% hint style="info" %} **Advanced configuration:** To configure persistence, metrics, MCP, OpenLineage, security, batch jobs, and more via the operator, see the [Operator Configuration Guides](/master/how-to-guides/feast-on-kubernetes/feast-operator). {% endhint %} ## Upgrading the Operator ### OLM-managed installations If the operator was installed via OLM, upgrades are handled automatically. No manual steps are required — OLM recreates the operator Deployment during the upgrade process. ### kubectl-managed installations For most upgrades, re-running the install command is sufficient: ```sh kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/feast-dev/feast/refs/heads/stable/infra/feast-operator/dist/install.yaml ``` #### One-time step: upgrading from versions before 0.61.0 Version 0.61.0 updated the operator Deployment's `spec.selector` to include the `app.kubernetes.io/name: feast-operator` label, fixing a bug where the metrics service could accidentally target pods from other operators in shared namespaces. Because Kubernetes treats `spec.selector` as an immutable field, upgrading directly from a pre-0.61.0 version with `kubectl apply` will fail with: ``` The Deployment "feast-operator-controller-manager" is invalid: spec.selector: Invalid value: ... field is immutable ``` To resolve this, delete the existing operator Deployment before applying the new manifest: ```sh kubectl delete deployment feast-operator-controller-manager -n feast-operator-system --ignore-not-found=true kubectl apply --server-side --force-conflicts -f https://raw.githubusercontent.com/feast-dev/feast/refs/heads/stable/infra/feast-operator/dist/install.yaml ``` This is only required once. Existing FeatureStore CRs and their managed workloads (feature servers, registry, etc.) are not affected — the new operator pod will reconcile them automatically on startup. Future upgrades from 0.61.0 onward will not require this step. {% hint style="success" %} **Scaling & High Availability:** The Feast Operator supports horizontal scaling via static replicas, HPA autoscaling, or external autoscalers like [KEDA](https://keda.sh). Scaling requires DB-backed persistence for all enabled services. When scaling is enabled, the operator auto-injects soft pod anti-affinity and zone topology spread constraints for resilience. You can also configure a PodDisruptionBudget to protect against voluntary disruptions. See the [Horizontal Scaling with the Feast Operator](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast#horizontal-scaling-with-the-feast-operator) guide for configuration details, including [HA options](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast#high-availability), or check the general recommendations on [how to scale Feast](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast). {% endhint %} > *Sample scaling CRs are available at* [*`v1_featurestore_scaling_static.yaml`*](https://github.com/feast-dev/feast/blob/master/infra/feast-operator/config/samples/v1_featurestore_scaling_static.yaml) *and* [*`v1_featurestore_scaling_hpa.yaml`*](https://github.com/feast-dev/feast/blob/master/infra/feast-operator/config/samples/v1_featurestore_scaling_hpa.yaml)*.* # Operator Configuration Guides These guides cover the `FeatureStore` Custom Resource (CR) from an **operator perspective** — what to put in the CR, how the operator translates it into Kubernetes objects, and where to look for store-specific YAML options in the Feast SDK docs. ## How the docs are organised | Layer | Source | What it covers | | --------------------------------- | --------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | **CR field reference** | [`api/markdown/ref.md`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md) | Every field, type, validation constraint — auto-generated from Go types | | **Operator guides** (this folder) | Narrative how-tos | CR→K8s behavior, Secret formats, PVC patterns, operator-specific trade-offs | | **Feast SDK reference** | [`feast/docs/reference/`](https://github.com/feast-dev/feast/tree/master/docs/how-to-guides/reference/README.md) | Store-specific `feature_store.yaml` options (online/offline store drivers, registry drivers, etc.) | | **Sample CRs** | [`config/samples/`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/) | Copy-paste starting points for common configurations | > **Rule of thumb**: if you need to know *what to put in the CR and why*, read the operator guides. If you need to know *which keys a particular store driver accepts*, read the Feast SDK docs for that store. *** ## Guide index | # | Guide | Topic | | - | ---------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | 1 | [Project Provisioning](/master/how-to-guides/feast-on-kubernetes/feast-operator/01-project-provisioning) | `feastProjectDir`: git clone, `feast init`, or a repository packaged in an image | | 2 | [Persistence](/master/how-to-guides/feast-on-kubernetes/feast-operator/02-persistence) | File (path + PVC) vs DB store for offline/online/registry; Secret format | | 3 | [Serving & Observability](/master/how-to-guides/feast-on-kubernetes/feast-operator/03-serving-and-observability) | Feature server workers, log level, Prometheus metrics, offline push batching, MCP | | 4 | [Registry Topology](/master/how-to-guides/feast-on-kubernetes/feast-operator/04-registry-topology) | Local vs remote registry, cross-namespace `feastRef`, remote TLS | | 5 | [Security](/master/how-to-guides/feast-on-kubernetes/feast-operator/05-security) | Kubernetes RBAC roles vs OIDC auth; TLS for all servers | | 6 | [Batch Jobs](/master/how-to-guides/feast-on-kubernetes/feast-operator/06-batch-and-jobs) | `batchEngine` ConfigMap contract, `cronJob` for scheduled materialization | | 7 | [OpenLineage & Materialization](/master/how-to-guides/feast-on-kubernetes/feast-operator/07-openlineage-and-materialization) | Lineage transports, API key Secret, materialization batch size | *** ## Quick-start: which guide do I need? * **"How do I point the operator at my existing git feature repo?"** → [Guide 1](/master/how-to-guides/feast-on-kubernetes/feast-operator/01-project-provisioning) * **"How do I wire Postgres/Redis/DuckDB as my store?"** → [Guide 2](/master/how-to-guides/feast-on-kubernetes/feast-operator/02-persistence) * **"How do I enable Prometheus scraping for the feature server?"** → [Guide 3](/master/how-to-guides/feast-on-kubernetes/feast-operator/03-serving-and-observability) * **"How do I make all services share a remote registry?"** → [Guide 4](/master/how-to-guides/feast-on-kubernetes/feast-operator/04-registry-topology) * **"How do I enable Kubernetes RBAC or OIDC auth?"** → [Guide 5](/master/how-to-guides/feast-on-kubernetes/feast-operator/05-security) * **"How do I schedule nightly materialization?"** → [Guide 6](/master/how-to-guides/feast-on-kubernetes/feast-operator/06-batch-and-jobs) * **"How do I send lineage events to Marquez?"** → [Guide 7](/master/how-to-guides/feast-on-kubernetes/feast-operator/07-openlineage-and-materialization) * **"What are all valid fields on `ServingConfig`?"** → [API ref](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#servingconfig) *** ## Scaling & HA Horizontal scaling, HPA, PodDisruptionBudget, affinity, and topology spread constraints are covered in the main Feast docs: → [Horizontal Scaling with the Feast Operator](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast#horizontal-scaling-with-the-feast-operator) # 1 — Project Provisioning The operator needs a Feast feature repository (a directory containing `feature_store.yaml` and Python feature-view definitions) to work from. `spec.feastProjectDir` controls how that directory is created inside the pods. When `feastProjectDir` is specified, exactly one of `git`, `init`, or `packaged` must be set. *** ## Option A — Clone from a Git repository (`feastProjectDir.git`) The operator runs an init container that clones the repository before the Feast processes start. Use this for production: your feature definitions live in version control and the operator tracks a specific commit or branch. ### Minimal example ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: my-feature-store spec: feastProject: credit_scoring feastProjectDir: git: url: https://github.com/my-org/feast-feature-repo ref: main # branch, tag, or commit SHA ``` ### Pinning to a specific commit (recommended for production) ```yaml feastProjectDir: git: url: https://github.com/my-org/feast-feature-repo ref: 598a270 # immutable SHA — no surprise changes on pod restart ``` ### Monorepo: feature repo in a subdirectory When the Feast feature repository lives inside a larger monorepo, use `featureRepoPath` to point at the subdirectory (relative path, no leading `/`): ```yaml feastProjectDir: git: url: https://github.com/my-org/data-platform ref: e959053 featureRepoPath: ml/feast/feature_repo # relative to repo root ``` ### Private repositories — token authentication Create a Kubernetes Secret containing the token: ```yaml apiVersion: v1 kind: Secret metadata: name: git-token stringData: TOKEN: ``` Reference the Secret from `envFrom` and rewrite the remote URL via `configs`: ```yaml feastProjectDir: git: url: https://github.com/my-org/private-repo configs: # Replaces the HTTPS URL with one that includes the token 'url."https://api:${TOKEN}@github.com/".insteadOf': 'https://github.com/' envFrom: - secretRef: name: git-token ``` ### Disabling TLS verification (not recommended for production) ```yaml feastProjectDir: git: url: https://internal-git.corp/feast-repo configs: http.sslVerify: 'false' ``` ### Full `git` field reference | Field | Type | Description | | ----------------- | ------------------ | ------------------------------------------------------------------------------------------ | | `url` | string | Repository URL (HTTPS or SSH) | | `ref` | string | Branch, tag, or commit SHA. Defaults to the remote HEAD | | `featureRepoPath` | string | Relative path within the repo to the feature repository directory. Default: `feature_repo` | | `configs` | map\[string]string | Key-value pairs passed to `git -c` before clone | | `env` | EnvVar\[] | Environment variables for the git init container | | `envFrom` | EnvFromSource\[] | Sources (Secrets, ConfigMaps) for init container environment | *** ## Option B — Scaffold a new project (`feastProjectDir.init`) The operator runs `feast init` on first startup to create a minimal feature repository. Use this for development, demos, and CI environments where you do not yet have a feature repo to point at. ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: feast-dev spec: feastProject: sample_project feastProjectDir: init: {} # defaults: template=local, minimal=false ``` ### Templates `feast init` supports store-specific templates. Set `template` to generate a scaffold that matches your chosen online/offline store: ```yaml feastProjectDir: init: template: spark # scaffolds Spark-compatible feature_store.yaml ``` Available templates (validated by the CRD): `local` · `gcp` · `aws` · `snowflake` · `spark` · `postgres` · `hbase` · `cassandra` · `hazelcast` · `couchbase` · `clickhouse` ### Minimal scaffold `minimal: true` skips example feature-view files and creates only the bare `feature_store.yaml`: ```yaml feastProjectDir: init: minimal: true ``` ### Full `init` field reference | Field | Type | Default | Description | | ---------- | ------ | ------- | ----------------------------------------- | | `template` | string | `local` | Template name for `feast init --template` | | `minimal` | bool | `false` | Pass `--minimal` to `feast init` | *** ## Option C — Use a repository packaged in an image (`feastProjectDir.packaged`) Use `packaged` when the feature repository is built into a feature-server image. This is useful in air-gapped environments and in release workflows where feature definitions and their Python dependencies are promoted together as an immutable image. ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: packaged-feature-store spec: feastProject: credit_scoring feastProjectDir: packaged: image: registry.example.com/feature-server@sha256:0123456789abcdef featureRepoPath: /opt/feast/feature_repo ``` The repository can be added to a Feast feature-server image with a Dockerfile such as: ```dockerfile FROM quay.io/feastdev/feature-server:latest COPY feature_repo/ /opt/feast/feature_repo/ ``` `featureRepoPath` must be a canonical absolute, non-root path: do not use `.`, `..`, repeated separators, or a trailing separator. Put it outside operator-mounted locations such as `/feast-data`; a volume mounted there would hide files baked into the image. When init containers are enabled, the packaged path also must not equal, contain, or be contained by the staged repository path. With init containers enabled (the default), each Pod starts in this order: 1. `feast-init` replaces the operator-managed staged repository with a fresh copy of the repository from `packaged.featureRepoPath`. With the default storage configuration, for example, it copies `/opt/feast/feature_repo` from the image to `/feast-data//feature_repo`. 2. In the staged copy only, `feast-init` replaces `feature_store.yaml` (if exists in the baked image) with the configuration generated from the FeatureStore resource. The file baked into the image is not modified. The Python feature definitions come from the packaged repository, while the FeatureStore resource remains authoritative for runtime configuration. 3. When `services.runFeastApplyOnInit` is omitted or `true` (the default), `feast-apply` runs `feast apply` from the staged repository using the packaged image. Setting it to `false` skips only this step; repository staging still occurs. 4. The Feast service containers start with the staged repository as their working directory. The repository baked into the image is therefore the source artifact, while the staged repository is the runtime copy used by `feast apply` and the Feast services. For a baked repository whose own `feature_store.yaml` must remain authoritative, disable init containers: ```yaml services: disableInitContainers: true ``` In that mode, Feast service containers use `featureRepoPath` directly and neither staging nor `feast apply` runs during pod initialization. The Operator does not update the registry, so `feast apply` must be handled separately—for example, by CI/CD or a separately managed Kubernetes Job or CronJob—whenever the packaged feature definitions change. The packaged `image` is optional. When set, it is the default for repository initialization, `feast apply`, and Feast services. `services.initImage` takes precedence for the `feast-init` and `feast-apply` init containers, while an explicit image on an individual service takes precedence for that service. When the packaged image is omitted, the operator uses `RELATED_IMAGE_FEATURE_SERVER` or its built-in feature-server image fallback. ### Full `packaged` field reference | Field | Type | Required | Description | | ----------------- | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------ | | `featureRepoPath` | string | yes | Canonical absolute, non-root path to the feature repository in the image; it must not overlap the staged repository path | | `image` | string | no | Image containing the repository; defaults to the operator feature-server image | *** ## `feast apply` on startup By default, when repository initialization completes (git clone, `feast init`, or packaged repository staging), the operator runs `feast apply` before starting the servers. This registers all feature definitions with the registry. To **skip** `feast apply` on pod start (e.g. you manage registry updates separately): ```yaml services: disableInitContainers: true # skip both clone/init AND feast apply ``` Or to keep the init container but skip the apply step: ```yaml services: runFeastApplyOnInit: false ``` *** ## When `feastProjectDir` is omitted If `feastProjectDir` is not set, the operator defaults to `feastProjectDir.init: {}` and creates a local template repository. *** ## See also * [API reference — `FeastProjectDir`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#feastprojectdir) * [Sample: public git repo](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_git.yaml) * [Sample: private git repo with token](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_git_token.yaml) * [Sample: monorepo with featureRepoPath](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_git_repopath.yaml) * [Sample: feast init](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_init.yaml) * [Sample: packaged feature repository](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_packaged.yaml) # 2 — Persistence The offline store, online store, and registry each need a place to store data. The operator supports two persistence patterns for each: * **File persistence** — a path on a volume (ephemeral or PVC-backed) * **DB/store persistence** — an external database or managed store, wired via a Kubernetes Secret The pattern is the same for all three services; only the store types differ. *** ## Persistence patterns at a glance | Pattern | Best for | Data survives pod restart? | | ----------------------- | --------------------------- | -------------------------- | | **File — emptyDir** | Dev / CI | No | | **File — PVC (ref)** | Single-node prod or testing | Yes (if PVC is retained) | | **File — PVC (create)** | Operator-managed storage | Yes | | **DB store** | Production, HA, multi-pod | Yes | *** ## File persistence ### Ephemeral (emptyDir) The default when no `persistence` block is set. Data lives on the pod's local disk and is lost on restart. Suitable for development only. ```yaml services: onlineStore: server: {} # no persistence block → emptyDir ``` ### PVC — reference an existing PVC When you already have a PVC provisioned (e.g. by your storage team): ```yaml services: onlineStore: persistence: file: path: online_store.db # filename relative to mountPath pvc: ref: name: online-pvc # name of the existing PVC mountPath: /data/online # where the PVC is mounted in the pod ``` ### PVC — let the operator create one ```yaml services: offlineStore: persistence: file: type: duckdb # offline file type: file | dask | duckdb pvc: create: storageClassName: standard resources: requests: storage: 20Gi mountPath: /data/offline ``` Omitting `storageClassName` uses the cluster default StorageClass. Omitting `create` entirely creates a PVC with the operator's built-in defaults (1 Gi, default StorageClass). ```yaml registry: local: persistence: file: path: registry.db pvc: create: {} # operator defaults: 1 Gi, default StorageClass mountPath: /data/registry ``` *** ## DB / store persistence For production, point the operator at an external database. The operator reads connection details from a Kubernetes Secret and writes them into `feature_store.yaml`. ### Secret format The Secret must contain one key per store component. The **key name** is the store type (e.g. `postgres`, `sql`, `redis`). The **value** is a YAML snippet identical to what you would write under the corresponding section in `feature_store.yaml`, **minus the `type:` key** (the operator inserts it from `persistence.store.type`). ```yaml apiVersion: v1 kind: Secret metadata: name: feast-data-stores stringData: # Key for online store type "postgres" postgres: | host: postgres.feast.svc.cluster.local port: 5432 database: feast db_schema: public user: feast password: feast # Key for registry type "sql" (SQLAlchemy URL) sql: | path: postgresql+psycopg://feast:feast@postgres.feast.svc.cluster.local:5432/feast #pragma: allowlist secret cache_ttl_seconds: 60 sqlalchemy_config_kwargs: echo: false pool_pre_ping: true ``` Reference the Secret from the CR: ```yaml services: onlineStore: persistence: store: type: postgres secretRef: name: feast-data-stores # key "postgres" is read automatically registry: local: persistence: store: type: sql secretRef: name: feast-data-stores # key "sql" is read automatically ``` > **Key lookup rule**: the operator looks up the Secret key that matches `persistence.store.type` (e.g. `type: postgres` → key `postgres`). Keep all stores in one Secret or split across multiple — both work. ### Injecting DB credentials into the server pod The Secret key values in the example above hard-code passwords. For production, keep credentials in a separate Secret and inject them as environment variables using `envFrom`, then reference them with `${VAR}` substitution in the data-stores Secret value: ```yaml apiVersion: v1 kind: Secret metadata: name: postgres-creds stringData: POSTGRES_USER: feast POSTGRES_PASSWORD: s3cr3t --- apiVersion: v1 kind: Secret metadata: name: feast-data-stores stringData: postgres: | host: postgres.feast.svc.cluster.local port: 5432 database: feast user: ${POSTGRES_USER} password: ${POSTGRES_PASSWORD} ``` ```yaml services: onlineStore: server: envFrom: - secretRef: name: postgres-creds # expands ${POSTGRES_USER} / ${POSTGRES_PASSWORD} persistence: store: type: postgres secretRef: name: feast-data-stores ``` *** ## Store types by component ### Online store | `type` | Secret key | SDK docs | | ----------- | ----------- | ------------------------------------------------------------------------------------------------------------------- | | `sqlite` | `sqlite` | [SQLite](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/sqlite.md) | | `redis` | `redis` | [Redis](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/redis.md) | | `postgres` | `postgres` | [Postgres](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/postgres.md) | | `cassandra` | `cassandra` | [Cassandra](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/cassandra.md) | | `hazelcast` | `hazelcast` | [Hazelcast](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/hazelcast.md) | | `hbase` | `hbase` | [HBase](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/hbase.md) | | `datastore` | `datastore` | [Datastore](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/datastore.md) | | `dynamodb` | `dynamodb` | [DynamoDB](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/dynamodb.md) | | `bigtable` | `bigtable` | [Bigtable](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/bigtable.md) | > For all store-specific YAML keys (connection options, pool sizes, etc.) see the linked SDK docs — the Secret value accepts the same keys. ### Offline store | `type` (file) | Notes | | ------------- | ------------------------------------------ | | `file` | Default pandas-based parquet offline store | | `dask` | Dask-based parallel parquet | | `duckdb` | DuckDB in-process analytical engine | For external DB-backed offline stores (BigQuery, Snowflake, Spark, Trino, etc.), use `persistence.store.type` and a Secret with the matching key. See [Offline Stores](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/README.md) in the SDK docs. ### Registry | `type` | Secret key | Notes | | -------------------- | ------------------------------- | --------------------------------------------------- | | `file` | *(file persistence, no Secret)* | SQLite-backed file registry | | `sql` | `sql` | SQLAlchemy URL — supports PostgreSQL, MySQL, SQLite | | `snowflake.registry` | `snowflake.registry` | Snowflake-backed registry | For `sql`, the Secret value is a `path:` (SQLAlchemy URL) plus optional `cache_ttl_seconds` and `sqlalchemy_config_kwargs`: ```yaml # feast-data-stores Secret, key "sql" sql: | path: postgresql+psycopg://user:pass@host:5432/feast #pragma: allowlist secret cache_ttl_seconds: 60 ``` *** ## Common patterns ### Redis online store ```yaml apiVersion: v1 kind: Secret metadata: name: feast-data-stores stringData: redis: | connection_string: redis.feast.svc.cluster.local:6379 --- apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: sample-redis spec: feastProject: my_project services: onlineStore: server: {} persistence: store: type: redis secretRef: name: feast-data-stores ``` ### Postgres for both online store and registry ```yaml services: onlineStore: persistence: store: type: postgres secretRef: name: feast-data-stores # reads key "postgres" offlineStore: persistence: file: type: duckdb pvc: create: {} mountPath: /data/offline registry: local: persistence: store: type: sql secretRef: name: feast-data-stores # reads key "sql" ``` *** ## Overriding the Secret key name By default the operator looks up the Secret key that matches `persistence.store.type` (e.g. `type: postgres` → key `postgres`). To use a different key, set `secretKeyName`: ```yaml services: onlineStore: persistence: store: type: postgres secretRef: name: feast-data-stores secretKeyName: my_custom_key # reads key "my_custom_key" instead of "postgres" ``` This is useful when a single Secret holds configuration for multiple stores of the same type, or when you want a more descriptive key name. *** ## Validation rules The operator enforces these rules on Secret values at reconciliation time: 1. The Secret key value must be **valid YAML** that deserializes to a map. 2. If the YAML contains a `type` field, its value **must match** the CR's `persistence.store.type`. Otherwise the operator rejects it with an error. Best practice: omit `type` from the Secret. 3. If the YAML contains a `registry_type` field (for registry stores), the same matching rule applies. 4. The Secret must exist in the **same namespace** as the FeatureStore CR. 5. Only **one** of `file` or `store` may be set under each persistence block (enforced by CRD validation). *** ## Complete Secret examples by store type Below are copy-paste-ready Secret YAML snippets for every operator-supported store type. Each snippet shows the Secret data key and the YAML value the operator expects. > **Note**: omit the `type` field from Secret values — the operator injects it from `persistence.store.type`. Including a matching `type` value is tolerated but not recommended. ### Online store Secrets #### Redis ```yaml apiVersion: v1 kind: Secret metadata: name: feast-online-store stringData: redis: | connection_string: redis.feast.svc.cluster.local:6379 ``` Redis Cluster with SSL: ```yaml apiVersion: v1 kind: Secret metadata: name: feast-online-store stringData: redis: | redis_type: redis_cluster connection_string: "redis1:6379,redis2:6379,ssl=true,password=my_password" ``` CR snippet: ```yaml services: onlineStore: persistence: store: type: redis secretRef: name: feast-online-store ``` SDK reference: [Redis](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/redis.md) *** #### Postgres ```yaml apiVersion: v1 kind: Secret metadata: name: feast-online-store stringData: postgres: | host: postgres.feast.svc.cluster.local port: 5432 database: feast db_schema: public user: feast password: feast ``` With SSL: ```yaml apiVersion: v1 kind: Secret metadata: name: feast-online-store stringData: postgres: | host: postgres.feast.svc.cluster.local port: 5432 database: feast db_schema: public user: feast password: feast sslmode: verify-ca sslrootcert_path: /path/to/server-ca.pem ``` CR snippet: ```yaml services: onlineStore: persistence: store: type: postgres secretRef: name: feast-online-store ``` SDK reference: [Postgres](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/postgres.md) *** #### Cassandra ```yaml apiVersion: v1 kind: Secret metadata: name: feast-online-store stringData: cassandra: | hosts: - 192.168.1.1 - 192.168.1.2 - 192.168.1.3 keyspace: KeyspaceName port: 9042 username: user password: secret protocol_version: 5 load_balancing: local_dc: datacenter1 load_balancing_policy: TokenAwarePolicy(DCAwareRoundRobinPolicy) read_concurrency: 100 write_concurrency: 100 ``` CR snippet: ```yaml services: onlineStore: persistence: store: type: cassandra secretRef: name: feast-online-store ``` SDK reference: [Cassandra](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/cassandra.md) *** #### Snowflake (online) ```yaml apiVersion: v1 kind: Secret metadata: name: feast-online-store stringData: snowflake.online: | account: snowflake_deployment.us-east-1 user: user_login password: user_password role: SYSADMIN warehouse: COMPUTE_WH database: FEAST ``` CR snippet: ```yaml services: onlineStore: persistence: store: type: snowflake.online secretRef: name: feast-online-store ``` SDK reference: [Snowflake](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/snowflake.md) *** #### DynamoDB ```yaml apiVersion: v1 kind: Secret metadata: name: feast-online-store stringData: dynamodb: | region: us-west-2 batch_size: 100 ``` CR snippet: ```yaml services: onlineStore: persistence: store: type: dynamodb secretRef: name: feast-online-store ``` SDK reference: [DynamoDB](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/dynamodb.md) *** #### Bigtable ```yaml apiVersion: v1 kind: Secret metadata: name: feast-online-store stringData: bigtable: | project_id: my_gcp_project instance: my_bigtable_instance ``` CR snippet: ```yaml services: onlineStore: persistence: store: type: bigtable secretRef: name: feast-online-store ``` SDK reference: [Bigtable](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/bigtable.md) *** #### Datastore ```yaml apiVersion: v1 kind: Secret metadata: name: feast-online-store stringData: datastore: | project_id: my_gcp_project namespace: my_datastore_namespace ``` CR snippet: ```yaml services: onlineStore: persistence: store: type: datastore secretRef: name: feast-online-store ``` SDK reference: [Datastore](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/datastore.md) *** #### MySQL ```yaml apiVersion: v1 kind: Secret metadata: name: feast-online-store stringData: mysql: | host: mysql.feast.svc.cluster.local port: 3306 database: feast user: feast password: feast ``` CR snippet: ```yaml services: onlineStore: persistence: store: type: mysql secretRef: name: feast-online-store ``` SDK reference: [MySQL](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/mysql.md) *** #### Hazelcast ```yaml apiVersion: v1 kind: Secret metadata: name: feast-online-store stringData: hazelcast: | cluster_name: dev cluster_members: - "localhost:5701" key_ttl_seconds: 36000 ``` CR snippet: ```yaml services: onlineStore: persistence: store: type: hazelcast secretRef: name: feast-online-store ``` SDK reference: [Hazelcast](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/hazelcast.md) *** #### HBase ```yaml apiVersion: v1 kind: Secret metadata: name: feast-online-store stringData: hbase: | host: hbase-thrift.feast.svc.cluster.local port: "9090" connection_pool_size: 4 ``` CR snippet: ```yaml services: onlineStore: persistence: store: type: hbase secretRef: name: feast-online-store ``` SDK reference: [HBase](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/hbase.md) *** #### Other supported online store types The following types also use the same pattern (`persistence.store.type` + `secretRef`). Place the driver-specific YAML keys from the SDK docs under the matching Secret key: | `type` | Secret key | SDK docs | | ------------------ | ------------------ | --------------------------------------------------------------------------------------------------------------------------- | | `sqlite` | `sqlite` | [SQLite](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/sqlite.md) | | `singlestore` | `singlestore` | [SingleStore](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/singlestore.md) | | `elasticsearch` | `elasticsearch` | [Elasticsearch](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/elasticsearch.md) | | `qdrant` | `qdrant` | [Qdrant](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/qdrant.md) | | `couchbase.online` | `couchbase.online` | [Couchbase](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/couchbase.md) | | `milvus` | `milvus` | [Milvus](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/milvus.md) | | `mongodb` | `mongodb` | [MongoDB](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/mongodb.md) | | `hybrid` | `hybrid` | [Hybrid](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/hybrid.md) | *** ### Offline store Secrets Offline DB stores follow the same pattern. The `type` field tells the operator which store driver to use; the Secret value holds the connection parameters. #### Snowflake (offline) ```yaml apiVersion: v1 kind: Secret metadata: name: feast-offline-store stringData: snowflake.offline: | account: snowflake_deployment.us-east-1 user: user_login password: user_password role: SYSADMIN warehouse: COMPUTE_WH database: FEAST schema: PUBLIC ``` CR snippet: ```yaml services: offlineStore: persistence: store: type: snowflake.offline secretRef: name: feast-offline-store ``` SDK reference: [Snowflake](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/snowflake.md) *** #### BigQuery ```yaml apiVersion: v1 kind: Secret metadata: name: feast-offline-store stringData: bigquery: | dataset: feast_bq_dataset project_id: my_gcp_project ``` CR snippet: ```yaml services: offlineStore: persistence: store: type: bigquery secretRef: name: feast-offline-store ``` SDK reference: [BigQuery](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/bigquery.md) *** #### Postgres (offline) ```yaml apiVersion: v1 kind: Secret metadata: name: feast-offline-store stringData: postgres: | host: postgres.feast.svc.cluster.local port: 5432 database: feast db_schema: public user: feast password: feast ``` CR snippet: ```yaml services: offlineStore: persistence: store: type: postgres secretRef: name: feast-offline-store ``` SDK reference: [Postgres](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/postgres.md) *** #### Other supported offline store types | `type` | Secret key | SDK docs | | ------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `redshift` | `redshift` | [Redshift](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/redshift.md) | | `spark` | `spark` | [Spark](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/spark.md) | | `trino` | `trino` | [Trino](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/trino.md) | | `athena` | `athena` | [Athena](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/athena.md) | | `mssql` | `mssql` | [MSSQL](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/mssql.md) | | `couchbase.offline` | `couchbase.offline` | [Couchbase](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/couchbase.md) | | `clickhouse` | `clickhouse` | [ClickHouse](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/clickhouse.md) | | `ray` | `ray` | [Ray](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/ray.md) | | `oracle` | `oracle` | [Oracle](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/oracle.md) | *** ### Registry Secrets #### SQL (SQLAlchemy) registry The most common production registry. Uses a SQLAlchemy URL to connect to PostgreSQL, MySQL, or SQLite: ```yaml apiVersion: v1 kind: Secret metadata: name: feast-registry-store stringData: sql: | path: postgresql+psycopg://feast:feast@postgres.feast.svc.cluster.local:5432/feast #pragma: allowlist secret cache_ttl_seconds: 60 sqlalchemy_config_kwargs: echo: false pool_pre_ping: true ``` CR snippet: ```yaml services: registry: local: persistence: store: type: sql secretRef: name: feast-registry-store ``` SDK reference: [SQL Registry](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/registries/sql.md) *** #### Snowflake registry ```yaml apiVersion: v1 kind: Secret metadata: name: feast-registry-store stringData: snowflake.registry: | account: snowflake_deployment.us-east-1 user: user_login password: user_password role: SYSADMIN warehouse: COMPUTE_WH database: FEAST schema: PUBLIC cache_ttl_seconds: 60 ``` CR snippet: ```yaml services: registry: local: persistence: store: type: snowflake.registry secretRef: name: feast-registry-store ``` SDK reference: [Snowflake Registry](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/registries/snowflake.md) *** ## Multi-store Secret (single Secret for all components) You can combine all store configurations into a single Secret: ```yaml apiVersion: v1 kind: Secret metadata: name: feast-data-stores stringData: redis: | connection_string: redis.feast.svc.cluster.local:6379 snowflake.offline: | account: snowflake_deployment.us-east-1 user: user_login password: user_password role: SYSADMIN warehouse: COMPUTE_WH database: FEAST schema: PUBLIC sql: | path: postgresql+psycopg://feast:feast@postgres.feast.svc.cluster.local:5432/feast #pragma: allowlist secret cache_ttl_seconds: 60 --- apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: production-store spec: feastProject: my_project services: onlineStore: persistence: store: type: redis secretRef: name: feast-data-stores offlineStore: persistence: store: type: snowflake.offline secretRef: name: feast-data-stores registry: local: persistence: store: type: sql secretRef: name: feast-data-stores ``` *** ## ConfigMap usage (batch engine) The `batchEngine` is the only operator component that uses a **ConfigMap** rather than a Secret for its configuration. The ConfigMap must contain a YAML value under key `config` (default) or the key specified in `configMapKey`. Unlike store Secrets, the batch engine ConfigMap value **must include the `type` field**: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: feast-batch-engine data: config: | type: spark master: local spark_conf: spark.executor.memory: 4g --- apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: sample spec: feastProject: my_project batchEngine: configMapRef: name: feast-batch-engine # configMapKey: config # optional, defaults to "config" ``` See [Guide 6 — Batch & Jobs](/master/how-to-guides/feast-on-kubernetes/feast-operator/06-batch-and-jobs) for full details. *** ## Troubleshooting | Symptom | Likely cause | Fix | | ---------------------------------------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | `secret key X doesn't exist in secret Y` | The Secret key name doesn't match the store `type` | Either rename the Secret key to match `type`, or set `secretKeyName` in the CR | | `secret X contains invalid value` | The Secret value is not valid YAML | Check indentation and quoting in the `stringData` value | | `contains tag named type with value X` | The Secret includes a `type` field that doesn't match the CR's `persistence.store.type` | Remove `type` from the Secret value, or correct it to match | | `invalid secret X for offline store` | The referenced Secret doesn't exist | Create the Secret in the same namespace as the FeatureStore CR | | `One selection required between file or store` | Both `file` and `store` are set under a persistence block | Keep only one — choose either `file` persistence or `store` (DB) persistence | *** ## See also * [API reference — `OnlineStorePersistence`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#onlinestorepersistence) * [API reference — `OfflineStorePersistence`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#offlinepersistence) * [API reference — `RegistryPersistence`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#registrypersistence) * [Sample: DB persistence (Postgres)](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_db_persistence.yaml) * [Sample: PVC persistence](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_pvc_persistence.yaml) * [Sample: object store (GCS)](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_gcs.yaml) * [Feast SDK — Online Stores](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/online-stores/README.md) * [Feast SDK — Offline Stores](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/offline-stores/README.md) * [Feast SDK — Registries](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/registries/README.md) # 3 — Serving & Observability This guide covers the `spec.services.onlineStore.serving` block and general server configuration (`server` block) for all Feast services: per-worker tuning, log levels, Prometheus metrics, offline push batching, and MCP (Model Context Protocol). *** ## Server configuration (`server` block) Every deployable Feast service (online store, offline store, registry, UI) accepts a `server` block under `spec.services.`. The online and offline stores use `ServerConfigs`; the registry uses `RegistryServerConfigs` (adds `restAPI` / `grpc` toggles). ### Enable a server An empty `server: {}` is enough to deploy a service. Without it the component is not deployed (e.g. the offline store runs as a local process, not as a network server): ```yaml services: onlineStore: server: {} # deploys the online feature server on port 6566 offlineStore: server: {} # deploys the offline feature server on port 8815 registry: local: server: {} # deploys the registry server on port 6570 ui: {} # deploys the Feast UI ``` ### Log level ```yaml services: onlineStore: server: logLevel: debug # debug | info | warning | error | critical offlineStore: server: logLevel: info registry: local: server: logLevel: warning ``` ### Container image and resources The operator resolves the feature server image through the following priority chain: 1. **`server.image` in the CR** — per-service override, highest priority 2. **`RELATED_IMAGE_FEATURE_SERVER` env var on the operator pod** — cluster-wide default set by OLM/platform 3. **Built-in default** — `quay.io/feastdev/feature-server:` The same `RELATED_IMAGE_FEATURE_SERVER` image is used for **all server containers** (online, offline, registry, UI) and for the **init containers** (git clone, `feast apply`). Setting it overrides all of them at once without touching any CR. The **cronJob** container uses a separate env var: `RELATED_IMAGE_CRON_JOB` (default: `quay.io/openshift/origin-cli:4.17`). **Cluster-wide image override (operator env var)** — set this on the operator `Deployment` to redirect all pods cluster-wide to a different registry (e.g. a private mirror or a pinned digest): ```sh kubectl set env deployment/feast-operator-controller-manager \ RELATED_IMAGE_FEATURE_SERVER=my-registry.example.com/feast/feature-server:custom \ RELATED_IMAGE_CRON_JOB=my-registry.example.com/tools/cli:latest \ -n feast-operator-system ``` **Per-service CR override** — for a single `FeatureStore` or to pin one service to a different image than the cluster default: ```yaml services: onlineStore: server: image: quay.io/feastdev/feature-server:0.62.0 resources: requests: cpu: "500m" memory: "512Mi" limits: cpu: "2" memory: "2Gi" ``` ### Worker configuration (gunicorn) The online and offline servers run on gunicorn. Tune worker count, connections, and request limits: ```yaml services: onlineStore: server: workerConfigs: workers: -1 # -1 = auto (2 × CPU cores + 1) workerConnections: 1000 # simultaneous clients per worker maxRequests: 1000 # requests before worker restart (memory leak prevention) maxRequestsJitter: 50 # jitter to avoid thundering herd keepAliveTimeout: 30 # keep-alive timeout in seconds registryTtlSec: 60 # registry refresh interval in seconds ``` > **Production recommendation**: set `workers: -1` and `registryTtlSec: 60` or higher. See [Online Server Performance Tuning](/master/how-to-guides/online-server-performance-tuning) for detailed guidance. ### Environment variables and secrets Inject environment variables from Secrets or ConfigMaps into any server: ```yaml services: onlineStore: server: envFrom: - secretRef: name: my-db-credentials env: - name: FEAST_USAGE value: "false" ``` ### Volume mounts Mount additional volumes (ConfigMaps, Secrets, PVCs) into the server containers: ```yaml services: volumes: - name: ca-cert configMap: name: cluster-ca onlineStore: server: volumeMounts: - name: ca-cert mountPath: /etc/ssl/certs/cluster-ca.crt subPath: ca.crt ``` ### Dynamic Resource Allocation (DRA) Kubernetes [Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/) lets pods request hardware resources (GPUs, FPGAs, etc.) through a claim-based model. The FeatureStore CR exposes a pod-level `resourceClaims` field that defines which `ResourceClaim` objects must be allocated before the pod starts. Individual service containers then reference those claims by name to consume the allocated resources. **Step 1 — Create a `ResourceClaim`** (or use an existing one): ```yaml apiVersion: resource.k8s.io/v1 kind: ResourceClaim metadata: name: my-gpu-claim spec: devices: requests: - name: gpu exactly: deviceClassName: gpu.example.com ``` **Step 2 — Reference the claim in the FeatureStore CR:** ```yaml services: # Pod-level: define claims available to any container in the pod resourceClaims: - name: gpu resourceClaimName: my-gpu-claim # Container-level: the online store consumes the GPU claim onlineStore: server: resources: claims: - name: gpu ``` Multiple claims can be defined at the pod level and selectively consumed by different service containers: ```yaml services: resourceClaims: - name: gpu-claim resourceClaimName: my-gpu - name: fpga-claim resourceClaimName: my-fpga onlineStore: server: resources: claims: - name: gpu-claim # online store uses GPU offlineStore: server: resources: claims: - name: fpga-claim # offline store uses FPGA registry: local: server: {} # registry uses neither ``` > **Note**: DRA requires a DRA driver to be installed on the cluster and the appropriate `DeviceClass` and `ResourceSlice` objects to be available. On OpenShift, this is configured at the cluster level for GPU and accelerator resources. *** ## TLS All servers support TLS termination. Provide a Kubernetes Secret containing the TLS certificate and key, and reference it from `tls`: ```yaml services: onlineStore: server: tls: secretRef: name: feast-tls # Secret with keys tls.crt and tls.key ``` For mTLS (mutual TLS), also set a CA certificate ConfigMap: ```yaml tls: secretRef: name: feast-tls caCertConfigMapRef: name: cluster-ca certKeyName: tls.crt # default ``` *** ## Prometheus Metrics When metrics are enabled the feature server starts an HTTP server on port **8000** for Prometheus scraping. The operator automatically adds a `containerPort`, a Kubernetes `Service` port, and a `ServiceMonitor` for Prometheus discovery. ### Two paths — use either or both **Path 1: CLI flag** (existing, simple) ```yaml services: onlineStore: server: metrics: true # injects --metrics into feast serve; exposes port 8000 ``` **Path 2: YAML config** (new, granular) ```yaml services: onlineStore: serving: metrics: enabled: true categories: resource: true # CPU / memory gauges request: true # per-endpoint latency and request counters online_features: true # feature retrieval metrics push: true # push request counters materialization: true # materialization counters and histograms freshness: false # feature freshness (can be expensive at scale) ``` Both paths expose port 8000 and create a `ServiceMonitor`. When `serving.metrics.enabled` is true the Python server reads it from `feature_store.yaml` directly; no `--metrics` flag is injected. When `server.metrics: true` is used, the `--metrics` flag is injected. > **SDK note**: `MetricsConfig` uses `extra="forbid"` in Pydantic. Only use category keys that are recognized by your Feast SDK version. **Verify monitoring is wired:** ```sh kubectl get servicemonitor -n kubectl port-forward svc/-online 8000:8000 -n curl http://localhost:8000/metrics ``` *** ## Offline Push Batching When features are pushed to the online store via `/push`, each request also triggers a synchronous offline store write. At high push throughput this causes OOM. Push batching groups these writes into fixed-size batches flushed on a timer. ```yaml services: onlineStore: serving: offlinePushBatching: enabled: true batchSize: 1000 # max rows per batch batchIntervalSeconds: 10 # flush interval in seconds ``` | Field | Type | Description | | ---------------------- | ---- | ------------------------------------------ | | `enabled` | bool | Activates batching | | `batchSize` | int | Max rows per batch; flush when reached | | `batchIntervalSeconds` | int | Flush interval even when batch is not full | *** ## MCP (Model Context Protocol) MCP mounts LLM-agent-compatible tool endpoints alongside the existing REST API on port 6566. The REST API is not replaced — MCP is additive. The operator writes `feature_server.type: mcp` into `feature_store.yaml` only when `serving.mcp.enabled: true`. Setting `enabled: false` reverts to `type: local`. ```yaml services: onlineStore: server: {} serving: mcp: enabled: true serverName: feast-mcp-server serverVersion: "1.0.0" transport: sse # "sse" (default) or "http" ``` | Field | Type | Default | Description | | --------------- | ------ | ------------------ | --------------------------------------------- | | `enabled` | bool | — | Must be `true`; `false` keeps `type: local` | | `serverName` | string | `feast-mcp-server` | Name advertised to MCP clients | | `serverVersion` | string | `1.0.0` | Version advertised to MCP clients | | `transport` | string | `sse` | `sse` (SSE-based) or `http` (Streamable HTTP) | MCP is mounted at `/mcp` on port 6566 — no additional Kubernetes Service is created. > **Dependency**: the feature server image must include `feast[mcp]` (`fastapi-mcp`). Without it the server starts normally but MCP routes are not registered. ### Registry MCP MCP can also be enabled on the **registry REST server**, exposing registry metadata (entities, feature views, feature services) as MCP tool endpoints. This is configured under `registry.local.server.mcp` and requires `restAPI: true`. ```yaml services: registry: local: server: restAPI: true mcp: enabled: true persistence: store: type: sql secretRef: name: feast-data-stores ``` The operator writes `registry.mcp.enabled: true` into `feature_store.yaml` when this field is set. A CEL validation rule enforces that `restAPI` must be `true` when MCP is enabled. > **Note**: Registry MCP uses only the `enabled` field — `serverName`, `serverVersion`, and `transport` are not applicable to the registry server. *** ## `serving` vs `server` — summary | Capability | `server.*` block | `serving.*` block | | ---------------------- | ---------------------------- | ----------------------------- | | Enable the server | `server: {}` | — | | Log level | `server.logLevel` | — | | Workers / gunicorn | `server.workerConfigs` | — | | TLS | `server.tls` | — | | Env vars / image | `server.env`, `server.image` | — | | Metrics (simple) | `server.metrics: true` | — | | Metrics (per-category) | — | `serving.metrics.categories` | | Offline push batching | — | `serving.offlinePushBatching` | | MCP | — | `serving.mcp` | *** ## See also * [API reference — `ServerConfigs`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#serverconfigs) * [API reference — `ServingConfig`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#servingconfig) * [Sample: serving + metrics + offline push batching](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_serving.yaml) * [Sample: MCP](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_mcp.yaml) * [Sample: log levels](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_services_loglevel.yaml) * [Feast SDK — Python Feature Server](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/feature-servers/python-feature-server.md) * [Online Server Performance Tuning](/master/how-to-guides/online-server-performance-tuning) # 4 — Registry Topology The Feast registry stores metadata about feature views, entities, and feature services. The operator supports three topology options: | Topology | Use case | | ---------------------- | ------------------------------------------------------------------------- | | **Local** (file or DB) | Self-contained deployment; registry runs alongside other services | | **Local + server** | Local registry exposed as a gRPC / REST server for remote clients | | **Remote** | Multiple `FeatureStore` CRs share a single registry managed by another CR | *** ## Local registry ### File-backed (default) ```yaml services: registry: local: persistence: file: path: registry.db pvc: create: {} mountPath: /data/registry ``` ### DB-backed (SQL / Snowflake) ```yaml apiVersion: v1 kind: Secret metadata: name: feast-data-stores stringData: sql: | path: postgresql+psycopg://feast:feast@postgres:5432/feast #pragma: allowlist secret cache_ttl_seconds: 60 --- services: registry: local: persistence: store: type: sql secretRef: name: feast-data-stores # reads key "sql" ``` ### Exposing the registry as a server Add `server: {}` under `registry.local` to expose it on port **6570**: ```yaml services: registry: local: server: {} # exposes gRPC on 6570 persistence: store: type: sql secretRef: name: feast-data-stores ``` The registry server also supports REST and gRPC independently: ```yaml registry: local: server: restAPI: true # enable REST API (default: true when server is set) grpc: true # enable gRPC (default: true when server is set) ``` ### MCP on the registry server When the REST API is enabled, you can additionally expose registry metadata as MCP (Model Context Protocol) tool endpoints for LLM agents: ```yaml services: registry: local: server: restAPI: true mcp: enabled: true persistence: store: type: sql secretRef: name: feast-data-stores ``` The operator writes `registry.mcp.enabled: true` into `feature_store.yaml`. A validation rule enforces that `restAPI` must be `true` when `mcp.enabled` is `true`. See [Guide 3 — Serving & Observability](/master/how-to-guides/feast-on-kubernetes/feast-operator/03-serving-and-observability#registry-mcp) for more details and the full MCP configuration reference. *** ## Remote registry A remote registry lets multiple `FeatureStore` CRs (e.g. in different namespaces or teams) share a single registry. One CR owns the registry; the others point at it. ### Using a hostname Point at any existing Feast registry server endpoint: ```yaml services: registry: remote: hostname: feast-registry.feast-system.svc.cluster.local:6570 ``` ### Using `feastRef` (recommended for operator-managed registries) `feastRef` lets one `FeatureStore` CR reference another CR's registry without hard-coding hostnames. The operator resolves the Service name automatically: ```yaml # CR that owns the registry (in namespace "feast-system") apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: registry-owner namespace: feast-system spec: feastProject: shared_project services: registry: local: server: {} persistence: store: type: sql secretRef: name: feast-data-stores ``` ```yaml # CR that consumes the shared registry (in namespace "team-a") apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: team-a-store namespace: team-a spec: feastProject: shared_project services: registry: remote: feastRef: name: registry-owner namespace: feast-system # omit if same namespace ``` ### Remote registry with TLS If the remote registry server uses TLS, provide the CA certificate so clients can verify it: ```yaml services: registry: remote: feastRef: name: registry-owner namespace: feast-system tls: configMapRef: name: registry-ca-cert # ConfigMap containing the CA cert certKeyName: ca.crt # key inside the ConfigMap (default: ca.crt) ``` *** ## All-remote topology For teams that deploy services independently, all services can use remote endpoints: ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: client-store spec: feastProject: my_project services: offlineStore: remote: hostname: feast-offline.feast-system.svc.cluster.local:8815 onlineStore: remote: hostname: feast-online.feast-system.svc.cluster.local:6566 registry: remote: feastRef: name: central-registry namespace: feast-system ``` *** ## See also * [API reference — `RegistryConfig`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#registryconfig) * [API reference — `LocalRegistryConfig`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#localregistryconfig) * [API reference — `RemoteRegistryConfig`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#remoteregistryconfig) * [Sample: all remote servers](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_all_remote_servers.yaml) * [Sample: DB persistence](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_db_persistence.yaml) * [Sample: MCP](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_mcp.yaml) * [Feast SDK — Registries](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/registries/README.md) # 5 — Security The operator supports two authorization models via `spec.authz`, plus TLS for all servers. Authorization is optional — omitting `authz` deploys Feast with no access control. *** ## Kubernetes RBAC authorization Kubernetes RBAC authorization uses ServiceAccount tokens. The operator creates `ClusterRole`s for each named role you declare and binds them to ServiceAccounts. Feast servers enforce these roles on every API call. ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: sample-rbac spec: feastProject: feast_rbac authz: kubernetes: roles: - feast-writer # created as a ClusterRole - feast-reader services: offlineStore: server: {} onlineStore: server: {} registry: local: server: {} ``` The operator creates `ClusterRole` resources named after each entry in `roles`. Bind them to subjects using standard Kubernetes `ClusterRoleBinding` or `RoleBinding` resources. > Kubernetes auth requires all services to be exposed as servers (the controller rejects partial configurations where some services are local while RBAC is enabled). **SDK docs**: [Feast RBAC](/master/getting-started/architecture/rbac) *** ## OIDC authorization OIDC authorization validates Bearer tokens against an OIDC provider (Keycloak, Dex, etc.). ### Secret format Create a Secret with the OIDC client credentials: ```yaml apiVersion: v1 kind: Secret metadata: name: oidc-secret stringData: client_id: auth_discovery_url: https://keycloak.example.com/realms/feast/.well-known/openid-configuration client_secret: username: # used for client-credentials flow password: audience: # optional: reject tokens whose aud claim differs issuer: # optional: reject tokens whose iss claim differs ``` The optional `audience` and `issuer` keys enable audience and issuer claim verification on the standard OIDC/JWKS validation path; when omitted, the `aud` and `iss` claims are not checked. Set them to the values your IdP puts in the token itself, which are not always the ones in the discovery document (see [OIDC Authorization](/master/getting-started/components/authz_manager#oidc-authorization)). The Secret key `issuer` is distinct from the CR's `issuerUrl`, which selects the discovery endpoint and plays no part in claim verification. Kubernetes ServiceAccount tokens (validated via TokenReview) and intra-server communication follow separate paths and are not subject to these checks. {% hint style="warning" %} Before enabling these, three operational caveats: * **Existing Secret keys take effect on operator upgrade.** Keys named `audience` or `issuer` already present in the referenced Secret were previously ignored; after upgrading they are forwarded to every Feast pod. * **Your IdP must mint matching tokens for Feast's own clients.** Feast's client-credentials flow requests no audience, so in multi-service topologies (e.g. a remote registry) and for the UI's browser tokens, the IdP must be configured to issue tokens carrying the expected claims (e.g. a Keycloak audience mapper), or inter-service calls will be rejected. * **Secret edits are not watched.** Changes to these keys apply on the next reconcile or pod restart, not immediately. {% endhint %} Reference the Secret from the CR: ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: sample-oidc spec: feastProject: my_project authz: oidc: secretRef: name: oidc-secret ``` ### Advanced OIDC options {% hint style="warning" %} Every option in this section requires `apiVersion: feast.dev/v1`. Under the deprecated `feast.dev/v1alpha1`, `authz.oidc` accepts only `secretRef`. The CRD has no conversion webhook, so a resource submitted as v1alpha1 is validated against the v1alpha1 schema and any other field is pruned without error rather than rejected. Applying the example below as v1alpha1 therefore leaves OIDC configured by Secret alone, with none of these settings taking effect and nothing in the output to say so. Use v1, which is the storage version. {% endhint %} ```yaml authz: oidc: secretRef: name: oidc-secret secretKeyName: client_id # override the default Secret key name tokenEnvVar: FEAST_TOKEN # env var from which servers read the Bearer token verifySSL: false # disable SSL verification (dev only) caCertConfigMap: # ConfigMap with CA cert for SSL verification name: oidc-ca-cert jwksCacheLifespanSeconds: 300 # how long servers reuse the fetched JWK set jwksRequestTimeoutSeconds: 10 # network timeout for the JWKS fetch ``` `jwksCacheLifespanSeconds` is not only a performance setting: it also bounds how long a key the provider has **revoked** continues to validate tokens. Lower it if your provider rotates or revokes aggressively, at the cost of proportionally more JWKS fetches. Key rotations that introduce a new key id are picked up immediately regardless, because an unknown key id forces a refetch. `jwksRequestTimeoutSeconds` bounds how long an unresponsive provider can block request serving. Both must be at least 1. When unset, neither key is written to the generated configuration and the feature server applies its own defaults (300 and 10 seconds respectively). {% hint style="warning" %} These two options require a feature server image that recognizes them. The operator deploys a matching image by default, so this only applies if you pin an older one explicitly, through a container `image` override or the operator's `RELATED_IMAGE_FEATURE_SERVER` setting. An image that predates these options rejects its configuration at startup, so leave them unset until the pinned image is updated. {% endhint %} **SDK docs**: [Feast OIDC Auth](/master/getting-started/components/authz_manager#oidc-authorization) *** ## TLS for servers Each server accepts a `tls` block pointing to a Kubernetes Secret that holds the TLS certificate and key. ### Creating a TLS Secret ```sh kubectl create secret tls feast-tls \ --cert=path/to/tls.crt \ --key=path/to/tls.key \ -n ``` ### Applying TLS to servers ```yaml services: onlineStore: server: tls: secretRef: name: feast-tls offlineStore: server: tls: secretRef: name: feast-tls registry: local: server: tls: secretRef: name: feast-tls ``` Each service can use different TLS Secrets. ### Custom certificate key names By default the operator looks for keys `tls.crt` and `tls.key`. Override with: ```yaml tls: secretRef: name: feast-tls certKeyName: server.crt # default: tls.crt ``` ### mTLS — providing a CA certificate For mutual TLS (client certificate verification), supply a CA cert via a ConfigMap: ```yaml tls: secretRef: name: feast-tls caCertConfigMapRef: name: client-ca-cert certKeyName: tls.crt ``` *** ## OpenShift non-TLS mode On OpenShift, services are typically accessed via Routes with TLS termination at the edge. In this case it is common to run the Feast servers without internal TLS: ```yaml # See config/samples/v1_featurestore_all_openshift_non_tls.yaml services: onlineStore: server: {} # no tls block offlineStore: server: {} registry: local: server: {} ``` *** ## See also * [API reference — `AuthzConfig`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#authzconfig) * [API reference — `TlsConfigs`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#tlsconfigs) * [Sample: Kubernetes auth](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_kubernetes_auth.yaml) * [Sample: OIDC auth](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_oidc_auth.yaml) * [Sample: Postgres with TLS volumes](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_postgres_db_volumes_tls.yaml) * [Sample: OpenShift non-TLS](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_all_openshift_non_tls.yaml) * [Feast SDK — Auth Overview](https://github.com/feast-dev/feast/tree/master/docs/how-to-guides/reference/auth/README.md) # 6 — Batch & Jobs This guide covers two related top-level spec fields: * **`spec.batchEngine`** — override the compute engine used for materialization * **`spec.cronJob`** — schedule periodic `feast materialize-incremental` (or any command) as a Kubernetes `CronJob` *** ## Batch Engine (`spec.batchEngine`) By default, Feast runs materialization using the **local** batch engine (in-process Python). For large feature sets you can point the operator at a Spark, Ray, or other supported engine via a Kubernetes ConfigMap. ### ConfigMap format Create a ConfigMap whose value is a YAML snippet identical to the `batch_engine` section of `feature_store.yaml`. Include the `type:` key and all engine-specific options: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: feast-batch-engine data: config: | type: spark spark_conf: spark.master: k8s://https://kubernetes.default.svc spark.kubernetes.namespace: feast spark.kubernetes.container.image: ghcr.io/feast-dev/feast-spark:latest spark.executor.instances: "2" spark.executor.memory: 4g spark.driver.memory: 2g ``` Reference the ConfigMap from the CR: ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: sample-spark spec: feastProject: my_project batchEngine: configMapRef: name: feast-batch-engine # ConfigMap name configMapKey: config # key inside the ConfigMap (default: "config") ``` ### SparkApplication batch engine (optional) For Bring Your Own Spark on Kubernetes, use `spark_application` instead of in-process Spark. The Feast Operator auto-creates RBAC for this type. See [SparkApplication](https://github.com/feast-dev/feast/tree/master/docs/how-to-guides/reference/compute-engine/spark_application.md) for the full config reference. Build an image from the reference [Dockerfile](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/compute_engines/spark_application/Dockerfile) (or equivalent): ```yaml apiVersion: v1 kind: ConfigMap metadata: name: feast-spark-application-engine data: config: | type: spark_application image: my-registry.example.com/feast-spark-driver:latest namespace: feast executor_instances: 2 driver_memory: "2g" executor_memory: "2g" ``` ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: sample-spark-application spec: feastProject: my_project batchEngine: configMapRef: name: feast-spark-application-engine configMapKey: config # Optional: use the Spark driver image for feast-apply / init containers # services: # initImage: my-registry.example.com/feast-spark-driver:latest ``` ### Engine types | `type` | Notes | | ------------------- | ------------------------------------------------------------------------------------------------------------------ | | `local` | Default; in-process Python, no extra infra | | `spark` | Apache Spark; requires a Spark operator or standalone cluster | | `spark_application` | Kubeflow Spark Operator `SparkApplication` CRs; requires Spark Operator + custom image; operator auto-creates RBAC | | `ray` | Ray cluster; requires a Ray operator | | `bytewax` | Bytewax streaming engine | | `snowflake.engine` | Snowflake Snowpark compute | > For engine-specific YAML options (Spark conf, Ray address, etc.) see the [Feast SDK — Compute Engine](https://github.com/feast-dev/feast/tree/master/docs/how-to-guides/reference/compute-engine/README.md) docs. *** ## Scheduled Materialization (`spec.cronJob`) The operator can deploy a Kubernetes `CronJob` that runs `feast materialize-incremental` (or any custom command) on a schedule. This is the recommended way to keep your online store fresh without managing an external job scheduler. ### CronJob image resolution The CronJob container image is resolved through the following priority chain: 1. **`cronJob.containerConfigs.image` in the CR** — per-CronJob override 2. **`RELATED_IMAGE_CRON_JOB` env var on the operator pod** — cluster-wide default set by OLM/platform (default: `quay.io/openshift/origin-cli:4.17`) ```sh # Override cluster-wide for all CronJobs kubectl set env deployment/feast-operator-controller-manager \ RELATED_IMAGE_CRON_JOB=my-registry.example.com/tools/cli:latest \ -n feast-operator-system ``` ### Minimal example — nightly materialization ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: feast-production spec: feastProject: my_project cronJob: schedule: "0 2 * * *" # every day at 02:00 UTC containerConfigs: image: quay.io/feastdev/feature-server:0.62.0 ``` The CronJob runs the default `feast materialize-incremental` command using the same `feature_store.yaml` that the operator generated for this `FeatureStore`. ### Custom command Override the container command to run any Feast CLI command: ```yaml cronJob: schedule: "*/30 * * * *" containerConfigs: image: quay.io/feastdev/feature-server:0.62.0 commands: - feast - materialize-incremental - "2099-01-01T00:00:00" # materialize up to a fixed end time ``` Or run a Python script instead: ```yaml containerConfigs: commands: - python - /app/materialize.py ``` ### Time zone ```yaml cronJob: schedule: "0 3 * * *" timeZone: "Asia/Kolkata" # defaults to the kube-controller-manager time zone ``` ### Concurrency policy ```yaml cronJob: schedule: "0 * * * *" concurrencyPolicy: Forbid # Allow | Forbid | Replace startingDeadlineSeconds: 300 # skip if missed by > 5 minutes ``` ### Job history retention ```yaml cronJob: schedule: "0 2 * * *" successfulJobsHistoryLimit: 3 # keep last 3 successful runs failedJobsHistoryLimit: 5 # keep last 5 failed runs ``` ### Suspend a CronJob To temporarily pause scheduled runs without deleting the `CronJob`: ```yaml cronJob: schedule: "0 2 * * *" suspend: true ``` ### Resource requests for the job pod ```yaml cronJob: schedule: "0 2 * * *" containerConfigs: image: quay.io/feastdev/feature-server:0.62.0 resources: requests: cpu: "1" memory: "2Gi" limits: cpu: "4" memory: "8Gi" ``` ### Environment variables and secrets in the job pod ```yaml cronJob: schedule: "0 2 * * *" containerConfigs: image: quay.io/feastdev/feature-server:0.62.0 envFrom: - secretRef: name: feast-data-stores env: - name: FEAST_USAGE value: "false" ``` ### Advanced job spec ```yaml cronJob: schedule: "0 2 * * *" jobSpec: parallelism: 1 completions: 1 activeDeadlineSeconds: 3600 # abort if job takes more than 1 hour backoffLimit: 2 # retry up to 2 times on failure podTemplateAnnotations: prometheus.io/scrape: "false" containerConfigs: image: quay.io/feastdev/feature-server:0.62.0 ``` ### Full `cronJob` field reference | Field | Type | Default | Description | | ---------------------------------- | -------------------- | ------------------------------- | ------------------------------------------ | | `schedule` | string | — | Cron expression (required) | | `timeZone` | string | kube-controller-manager TZ | IANA time zone name | | `concurrencyPolicy` | string | `Allow` | `Allow` / `Forbid` / `Replace` | | `suspend` | bool | `false` | Suspend future runs | | `startingDeadlineSeconds` | int64 | — | Abort if start missed by this many seconds | | `successfulJobsHistoryLimit` | int32 | — | Successful job history to keep | | `failedJobsHistoryLimit` | int32 | — | Failed job history to keep | | `annotations` | map | — | CronJob metadata annotations | | `jobSpec.parallelism` | int32 | 1 | Job pod parallelism | | `jobSpec.completions` | int32 | 1 | Required completions | | `jobSpec.activeDeadlineSeconds` | int64 | — | Max job duration | | `jobSpec.backoffLimit` | int32 | — | Retry limit | | `containerConfigs.image` | string | operator default | Feature server image | | `containerConfigs.commands` | \[]string | `feast materialize-incremental` | Override container command | | `containerConfigs.resources` | ResourceRequirements | — | CPU/memory requests and limits | | `containerConfigs.env` / `envFrom` | — | — | Environment variables | *** ## Combining batchEngine + cronJob Use both together to run scheduled Spark-based materialization: ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: feast-spark spec: feastProject: my_project batchEngine: configMapRef: name: feast-spark-engine cronJob: schedule: "0 1 * * *" concurrencyPolicy: Forbid containerConfigs: image: quay.io/feastdev/feature-server:0.62.0 resources: requests: memory: "4Gi" ``` *** ## See also * [API reference — `BatchEngineConfig`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#batchengineconfig) * [API reference — `FeastCronJob`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#feastcronjob) * [Feast SDK — Compute Engine](https://github.com/feast-dev/feast/tree/master/docs/how-to-guides/reference/compute-engine/README.md) # 7 — OpenLineage & Materialization Both `spec.openlineage` and `spec.materialization` are written into `feature_store.yaml` for **all** service pods — they apply to the online server, offline server, registry, and materialization jobs alike. *** ## OpenLineage Data Lineage (`spec.openlineage`) OpenLineage emits data lineage events during `feast apply` (registry changes) and materialization. Events go outbound from Feast pods to your OpenLineage-compatible backend ([Marquez](https://marquezproject.ai), any OpenLineage HTTP endpoint, Kafka, or a file). No inbound ports or additional Kubernetes Services are required. > **Dependency**: the Feast image must include `feast[openlineage]` (`openlineage-python`). ### HTTP transport (Marquez) ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: sample-openlineage spec: feastProject: my_project openlineage: enabled: true transportType: http transportUrl: "http://marquez.feast.svc.cluster.local:5000" transportEndpoint: "api/v1/lineage" extraConfig: namespace: "my-feast-project" producer: "feast-operator" emit_on_apply: "true" emit_on_materialize: "true" ``` ### HTTP with API key authentication ```yaml apiVersion: v1 kind: Secret metadata: name: openlineage-secret namespace: feast stringData: api_key: "" --- apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: sample-openlineage-auth namespace: feast spec: feastProject: my_project openlineage: enabled: true transportType: http transportUrl: "https://marquez.example.com" transportEndpoint: "api/v1/lineage" apiKeySecretRef: name: openlineage-secret # Secret must contain key "api_key" extraConfig: namespace: "my-feast-project" emit_on_apply: "true" emit_on_materialize: "true" ``` The operator reads the `api_key` value from the Secret and writes it into `feature_store.yaml`. The Secret must be in the same namespace as the `FeatureStore`. ### Kafka transport ```yaml openlineage: enabled: true transportType: kafka extraConfig: namespace: "my-feast-project" emit_on_apply: "true" emit_on_materialize: "true" bootstrap_servers: "kafka.svc:9092" topic: "openlineage" sasl_mechanism: "PLAIN" ``` ### Console transport (development) Events are printed to stdout — useful for verifying integration without a backend: ```yaml openlineage: enabled: true transportType: console extraConfig: emit_on_apply: "true" emit_on_materialize: "true" ``` ### Field reference | Field | Type | Description | | ---------------------- | ------------------ | ---------------------------------------------------------------------------- | | `enabled` | bool | Activates OpenLineage. Must be `true` | | `transportType` | string | `http` / `console` / `file` / `kafka` (omit to use OpenLineage SDK defaults) | | `transportUrl` | string | Base URL for HTTP transport | | `transportEndpoint` | string | API path appended to `transportUrl` | | `apiKeySecretRef.name` | string | Name of a Secret containing key `api_key` | | `extraConfig` | map\[string]string | Additional settings (see below) | ### `extraConfig` keys Values that are `"true"` or `"false"` are automatically coerced to native YAML booleans so that Feast's Pydantic `StrictBool` validators accept them. | Key | Type | Description | | --------------------- | ----------- | ----------------------------------------------------- | | `namespace` | string | OpenLineage namespace for emitted events | | `producer` | string | Producer identifier in emitted events | | `emit_on_apply` | bool string | Emit events on `feast apply` | | `emit_on_materialize` | bool string | Emit events on materialization | | `bootstrap_servers` | string | Kafka: comma-separated broker addresses | | `topic` | string | Kafka: target topic name | | `sasl_mechanism` | string | Kafka: SASL mechanism (e.g. `PLAIN`, `SCRAM-SHA-256`) | | `file_path` | string | File transport: path to write lineage events | *** ## Materialization (`spec.materialization`) Controls how features are written to the online store during materialization jobs. Settings are written into `feature_store.yaml` for all pods. ```yaml spec: materialization: onlineWriteBatchSize: 10000 extraConfig: pull_latest_features: "false" ``` ### `onlineWriteBatchSize` Limits the number of rows written per batch during materialization. Without this, all rows for a feature view are written in a single batch — this can cause OOM for large feature views. * Supported engines: **local**, **Spark**, **Ray** * Minimum value: **1** (enforced by CRD validation) ```yaml materialization: onlineWriteBatchSize: 10000 # 10 k rows per batch ``` ### `extraConfig` Passes additional `MaterializationConfig` settings inline. Boolean strings (`"true"` / `"false"`) are coerced to native YAML booleans. ```yaml materialization: extraConfig: pull_latest_features: "false" # only materialize the latest value per entity ``` | Key | Type | Description | | ---------------------- | ----------- | ---------------------------------------------------------------------------------------------------- | | `pull_latest_features` | bool string | When `"true"`, only the latest feature value per entity is materialized. Default is engine-dependent | *** ## Full example ```yaml apiVersion: v1 kind: Secret metadata: name: openlineage-secret namespace: feast stringData: api_key: "" --- apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: feast-production namespace: feast spec: feastProject: my_project materialization: onlineWriteBatchSize: 10000 openlineage: enabled: true transportType: http transportUrl: "http://marquez.feast.svc.cluster.local:5000" transportEndpoint: "api/v1/lineage" apiKeySecretRef: name: openlineage-secret extraConfig: namespace: "my-feast-project" producer: "feast-operator" emit_on_apply: "true" emit_on_materialize: "true" services: onlineStore: server: {} ``` *** ## See also * [API reference — `OpenLineageConfig`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#openlineageconfig) * [API reference — `MaterializationConfig`](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/docs/api/markdown/ref.md#materializationconfig) * [Sample: materialization + openlineage](https://github.com/feast-dev/feast/blob/stable/infra/feast-operator/config/samples/v1_featurestore_materialization_openlineage.yaml) * [Feast SDK — OpenLineage](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/reference/openlineage.md) * [Guide 6 — Batch Engine & Scheduled Jobs](/master/how-to-guides/feast-on-kubernetes/feast-operator/06-batch-and-jobs) # Feast Production Deployment Topologies ## Table of Contents * [Overview](#overview) * [1. Minimal Production](#1-minimal-production) * [2. Standard Production (Recommended)](#2-standard-production-recommended) * [3. Enterprise Production](#3-enterprise-production) * [Isolated Registries (per namespace)](#architecture--isolated-registries-per-namespace) * [Shared Registry (cross-namespace)](#architecture--shared-registry-cross-namespace) * [Reliability & Disaster Recovery](#reliability--disaster-recovery) * [Feast Permissions and RBAC](#feast-permissions-and-rbac) * [Actions](#actions) * [Policy types](#policy-types) * [Examples](#example-role-based-permissions) * [Authorization configuration](#authorization-configuration) * [Infrastructure-Specific Recommendations](#infrastructure-specific-recommendations) * [AWS / EKS / ROSA](#aws--eks--rosa) * [GCP / GKE](#gcp--gke) * [On-Premise / OpenShift](#on-premise--openshift--self-managed-kubernetes) * [Air-Gapped / Disconnected Environments](#air-gapped--disconnected-environment-deployments) * [Hybrid Store Configuration](#hybrid-store-configuration) * [Hybrid online store](#hybrid-online-store) * [Hybrid offline store](#hybrid-offline-store) * [Performance Considerations](#performance-considerations) * [Server sizing](#online-feature-server-sizing) * [Store latency](#online-store-latency-guidelines) * [Connection pooling](#connection-pooling-for-remote-online-store) * [Registry cache tuning at scale](#registry-cache-tuning-at-scale) * [Materialization](#materialization-performance) * [Redis sizing](#redis-sizing-guidelines) * [Design Principles](#design-principles) * [Control plane vs data plane](#control-plane-vs-data-plane) * [Stateless vs stateful](#stateless-vs-stateful-components) * [Scalability](#scalability-guidelines) * [Topology Comparison](#topology-comparison) * [Next Steps](#next-steps) *** ## Overview This guide defines three production-ready deployment topologies for Feast on Kubernetes using the [Feast Operator](/master/how-to-guides/feast-on-kubernetes). Each topology addresses a different stage of organizational maturity, from getting started safely to running large-scale multi-tenant deployments. | Topology | Target audience | Key traits | | --------------------------------------------------------- | -------------------------------------- | ------------------------------------------------------- | | [Minimal Production](#1-minimal-production) | Small teams, POCs moving to production | Single namespace, no HA, simple setup | | [Standard Production](#2-standard-production-recommended) | Most production workloads | HA registry, autoscaling, TLS, RBAC | | [Enterprise Production](#3-enterprise-production) | Large orgs, multi-tenant | Namespace isolation, managed stores, full observability | Beyond the core topologies, this guide also covers: * [Feast Permissions and RBAC](#feast-permissions-and-rbac) — securing resources with policies * [Infrastructure-Specific Recommendations](#infrastructure-specific-recommendations) — store choices for AWS, GCP, and on-premise * [Hybrid Store Configuration](#hybrid-store-configuration) — routing to multiple backends from a single deployment * [Performance Considerations](#performance-considerations) — tuning for production workloads {% hint style="info" %} **Prerequisites:** All topologies assume a running Kubernetes cluster with the [Feast Operator installed](/master/how-to-guides/feast-on-kubernetes#deploy-feast-feature-servers-on-kubernetes). Familiarity with Feast [concepts](/master/getting-started/concepts/overview) and [components](/master/getting-started/components/overview) is recommended. {% endhint %} *** ## 1. Minimal Production ### When to use * Small teams with a single ML use case * POCs graduating to production * Low-traffic, non-critical workloads where simplicity is more important than availability ### Architecture ```mermaid graph TD subgraph Kubernetes["Kubernetes (single namespace)"] Operator["Feast Operator"] Registry["Registry Server
(1 replica)"] OnlineServer["Online Feature Server
(1 replica)"] MaterializationJob["Materialization Job
(feast materialize / CronJob)"] NotebookPod(["Notebook / Training Pod
(Feast SDK — remote online + registry)"]) end subgraph BackingStores["Backing Stores (inside or outside Kubernetes)"] Redis["Online Store
(e.g. Redis for prod; SQLite for dev)"] OfflineStore["Offline Store
(e.g. DuckDB/PVC for dev; S3/MinIO for prod)"] end Operator -->|manages| Registry Operator -->|manages| OnlineServer OnlineServer -->|reads/writes| Redis OnlineServer -->|reads metadata| Registry MaterializationJob -->|reads source data| OfflineStore MaterializationJob -->|writes features| Redis NotebookPod -->|online features REST| OnlineServer NotebookPod -->|metadata gRPC| Registry Client(["Client / ML Service"]) -->|REST port 6566| OnlineServer style Kubernetes fill:#f0f4ff,stroke:#3366cc,color:#000 style BackingStores fill:#fafafa,stroke:#999,color:#000 style Operator fill:#e8f5e9,stroke:#388e3c,color:#000 style Registry fill:#fff3e0,stroke:#f57c00,color:#000 style OnlineServer fill:#e3f2fd,stroke:#1976d2,color:#000 style Redis fill:#fce4ec,stroke:#c62828,color:#000 style OfflineStore fill:#f3e5f5,stroke:#7b1fa2,color:#000 style MaterializationJob fill:#fff8e1,stroke:#f9a825,color:#000 style NotebookPod fill:#e8f5e9,stroke:#388e3c,color:#000 ``` ### Components | Component | Configuration | Notes | | ------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | **Feast Operator** | Default install | Manages all Feast CRDs | | **Registry** | REST, 1 replica | Single point of metadata | | **Online Feature Server** | 1 replica, no autoscaling | Serves online features | | **Online Store** | Redis standalone (example) | SQLite is simplest for development; Redis for production. See [supported online stores](/master/reference/online-stores) for all options | | **Offline Store** | File-based or MinIO | DuckDB or file-based for development; MinIO/S3 for production. See [supported offline stores](/master/reference/offline-stores) for all options | | **Compute Engine** | In-process (default) | Suitable for small datasets and development; use Spark, Ray, or Snowflake Engine for larger workloads | ### Sample FeatureStore CR ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: minimal-production spec: feastProject: my_project services: onlineStore: persistence: store: type: redis secretRef: name: feast-online-store server: resources: requests: cpu: 500m memory: 512Mi limits: cpu: "1" memory: 1Gi offlineStore: persistence: file: type: duckdb # Use type: file for generic file-based; swap for S3/MinIO in production pvc: create: storageClassName: standard resources: requests: storage: 10Gi mountPath: /data/offline registry: local: server: restAPI: true resources: requests: cpu: 250m memory: 256Mi limits: cpu: 500m memory: 512Mi ``` ### Limitations {% hint style="warning" %} * **No high availability** — a single replica failure causes downtime * **No automatic failover** — manual intervention required on failure * **Manual scaling** — no HPA configured * **Limited security** — no TLS, no ingress, no RBAC by default {% endhint %} *** ## 2. Standard Production (Recommended) ### When to use * Most production ML workloads * Teams with moderate traffic that need reliability * Environments that require TLS, RBAC, and automated scaling ### Architecture ```mermaid graph TD Client(["Client / ML Service"]) -->|HTTPS| Ingress subgraph Kubernetes["Kubernetes"] Ingress["Ingress Controller
(TLS termination)"] Operator["Feast Operator"] MaterializationJob["Materialization Job
(CronJob / batchEngine)"] NotebookPod(["Notebook / Training Pod
(Feast SDK — remote online, offline, registry)"]) subgraph FeastDeployment["Feast Deployment (HPA autoscaled — all containers scale together)"] Registry["Registry Server"] OnlineServer["Online Feature Server"] OfflineServer["Offline Feature Server"] end Ingress -->|port 6566| OnlineServer Operator -->|manages| Registry Operator -->|manages| OnlineServer Operator -->|manages| OfflineServer OnlineServer -->|reads metadata| Registry OfflineServer -->|reads metadata| Registry MaterializationJob -->|reads metadata| Registry NotebookPod -->|online features REST| OnlineServer NotebookPod -->|historical features Arrow Flight| OfflineServer NotebookPod -->|metadata gRPC| Registry end subgraph BackingStores["Backing Stores (inside or outside Kubernetes)"] RedisCluster["Online Store
(e.g. Redis Cluster, DynamoDB, etc.)"] OfflineStore["Offline Store
(e.g. PostgreSQL, Redshift, BigQuery, etc.)"] end OnlineServer -->|reads/writes| RedisCluster MaterializationJob -->|writes features| RedisCluster OfflineServer -->|historical features| OfflineStore MaterializationJob -->|reads source data| OfflineStore style Kubernetes fill:#f0f4ff,stroke:#3366cc,color:#000 style FeastDeployment fill:#e8f5e9,stroke:#388e3c,color:#000 style BackingStores fill:#fafafa,stroke:#999,color:#000 style Ingress fill:#fff9c4,stroke:#f9a825,color:#000 style Operator fill:#e8f5e9,stroke:#388e3c,color:#000 style Registry fill:#fff3e0,stroke:#f57c00,color:#000 style OnlineServer fill:#e3f2fd,stroke:#1976d2,color:#000 style OfflineServer fill:#ede7f6,stroke:#512da8,color:#000 style RedisCluster fill:#fce4ec,stroke:#c62828,color:#000 style OfflineStore fill:#f3e5f5,stroke:#7b1fa2,color:#000 style MaterializationJob fill:#fff8e1,stroke:#f9a825,color:#000 style NotebookPod fill:#e8f5e9,stroke:#388e3c,color:#000 ``` ### Components **Core** | Component | Configuration | Notes | | -------------------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | **Feast Operator** | Default install | Manages all Feast CRDs | | **Registry** | SQL-backed (PostgreSQL) | Database-backed for consistency and concurrent access | | **Online Feature Server** | HPA (min 2 replicas, max based on peak load) | Separate container — serves online features from the online store | | **Offline Feature Server** | Scales with the same Deployment | Separate container — serves historical features and materialization source reads from the offline store | **Storage** | Component | Configuration | Notes | | ------------------ | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Online Store** | Redis Cluster (example) | Multi-node for availability and low latency; other production stores are also supported — see [supported online stores](/master/reference/online-stores) | | **Offline Store** | PostgreSQL (example) | Platform-agnostic DB-backed store; use Redshift/Athena for AWS, BigQuery for GCP, Spark for S3/MinIO pipelines — see [supported offline stores](/master/reference/offline-stores) for all options | | **Compute Engine** | Spark, Ray (KubeRay), or Snowflake Engine | Distributed compute for materialization and historical retrieval at scale | **Networking & Security** | Component | Configuration | Notes | | ----------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Ingress** | TLS-terminated | Secure external access | | **RBAC** | Kubernetes RBAC | Namespace-scoped permissions | | **Secrets** | Kubernetes Secrets + `${ENV_VAR}` substitution | Store credentials via `secretRef` / `envFrom` in the FeatureStore CR; inject into `feature_store.yaml` with [environment variable syntax](/master/how-to-guides/running-feast-in-production#5-using-environment-variables-in-your-yaml-configuration) | ### Sample FeatureStore CR ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: standard-production spec: feastProject: my_project authz: kubernetes: roles: - feast-admin-role - feast-user-role batchEngine: configMapRef: name: feast-batch-engine services: scaling: autoscaling: minReplicas: 2 maxReplicas: 10 # Set based on your peak load metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 podDisruptionBudgets: maxUnavailable: 1 onlineStore: persistence: store: type: redis secretRef: name: feast-online-store server: resources: requests: cpu: "1" memory: 1Gi limits: cpu: "2" memory: 2Gi offlineStore: persistence: store: type: postgres secretRef: name: feast-offline-store server: resources: requests: cpu: 500m memory: 512Mi limits: cpu: "1" memory: 1Gi registry: local: persistence: store: type: sql secretRef: name: feast-registry-store server: restAPI: true resources: requests: cpu: 500m memory: 512Mi limits: cpu: "1" memory: 1Gi --- apiVersion: v1 kind: ConfigMap metadata: name: feast-batch-engine data: config: | type: ray address: auto # KubeRay cluster address; replace with explicit URL if not using auto-discovery ``` {% hint style="success" %} **Key features:** * **High availability** — multi-replica deployment with auto-injected pod anti-affinity and topology spread constraints * **Scalable serving** — HPA adjusts the shared deployment replicas (all services scale together) based on demand * **Secure external access** — TLS-terminated ingress with RBAC * **Persistent storage** — Online Store (Redis Cluster shown as example; see [supported online stores](/master/reference/online-stores) for all options) + Offline Store (PostgreSQL shown as example; see [supported offline stores](/master/reference/offline-stores) for all options) for durability See [Horizontal Scaling with the Feast Operator](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast#horizontal-scaling-with-the-feast-operator) for full scaling configuration details. {% endhint %} *** ## 3. Enterprise Production ### When to use * Large organizations with multiple ML teams * Multi-tenant environments requiring strict isolation * High-scale deployments with governance, compliance, and SLA requirements ### Architecture — Isolated Registries (per namespace) Each team gets its own registry and online feature server in a dedicated namespace. This provides the strongest isolation but has notable trade-offs: feature discovery is siloed per team (no cross-project visibility), and each registry requires its own [Feast UI](/master/reference/alpha-web-ui) deployment — you cannot view multiple projects in a single UI instance. ```mermaid graph TD Client(["Clients / ML Services"]) -->|HTTPS| Gateway subgraph Kubernetes["Kubernetes"] Gateway["API Gateway / Ingress
(TLS + rate limiting)"] Operator["Feast Operator
(cluster-scoped)"] subgraph NamespaceA["Namespace A — Team A"] subgraph DeployA["Feast Deployment (HPA autoscaled)"] RegistryA["Registry Server"] OnlineServerA["Online Feature Server"] OfflineServerA["Offline Feature Server"] end NotebookPodA(["Notebook — Team A
(Feast SDK)"]) MaterializationJobA["Materialization Job"] end subgraph NamespaceB["Namespace B — Team B"] subgraph DeployB["Feast Deployment (HPA autoscaled)"] RegistryB["Registry Server"] OnlineServerB["Online Feature Server"] OfflineServerB["Offline Feature Server"] end NotebookPodB(["Notebook — Team B
(Feast SDK)"]) MaterializationJobB["Materialization Job"] end Gateway -->|port 6566| OnlineServerA Gateway -->|port 6566| OnlineServerB Operator -->|manages| RegistryA Operator -->|manages| OnlineServerA Operator -->|manages| OfflineServerA Operator -->|manages| RegistryB Operator -->|manages| OnlineServerB Operator -->|manages| OfflineServerB OnlineServerA -->|metadata| RegistryA OnlineServerB -->|metadata| RegistryB OfflineServerA -->|metadata| RegistryA OfflineServerB -->|metadata| RegistryB NotebookPodA -.->|online REST| OnlineServerA NotebookPodA -.->|Arrow Flight| OfflineServerA NotebookPodA -.->|metadata| RegistryA NotebookPodB -.->|online REST| OnlineServerB NotebookPodB -.->|Arrow Flight| OfflineServerB NotebookPodB -.->|metadata| RegistryB MaterializationJobA -->|metadata| RegistryA MaterializationJobB -->|metadata| RegistryB end subgraph BackingStores["Backing Stores (inside or outside Kubernetes)"] RedisA["Online Store — Team A
(e.g. Redis, DynamoDB, etc.)"] RedisB["Online Store — Team B
(e.g. Redis, DynamoDB, etc.)"] OfflineStore["Offline Store — shared instance
(e.g. BigQuery, Redshift, etc.)
isolated via per-team datasets / schemas"] end subgraph Observability["Observability Stack"] OTel["OpenTelemetry
(traces + metrics)"] Prometheus["Prometheus"] Grafana["Grafana"] Jaeger["Jaeger"] end OnlineServerA -->|reads/writes| RedisA OnlineServerB -->|reads/writes| RedisB OfflineServerA -->|Team A dataset| OfflineStore OfflineServerB -->|Team B dataset| OfflineStore MaterializationJobA -->|Team A data| OfflineStore MaterializationJobA -->|writes| RedisA MaterializationJobB -->|Team B data| OfflineStore MaterializationJobB -->|writes| RedisB style Kubernetes fill:#f0f4ff,stroke:#3366cc,color:#000 style NamespaceA fill:#e8f5e9,stroke:#388e3c,color:#000 style NamespaceB fill:#e3f2fd,stroke:#1976d2,color:#000 style DeployA fill:#c8e6c9,stroke:#2e7d32,color:#000 style DeployB fill:#bbdefb,stroke:#1565c0,color:#000 style BackingStores fill:#fafafa,stroke:#999,color:#000 style Observability fill:#fff3e0,stroke:#f57c00,color:#000 style Gateway fill:#fff9c4,stroke:#f9a825,color:#000 style Operator fill:#e8f5e9,stroke:#388e3c,color:#000 ``` ### Architecture — Shared Registry (cross-namespace) Alternatively, a single centralized registry server can serve multiple tenant namespaces. Tenant online feature servers connect to the shared registry via the [Remote Registry](/master/reference/registries/remote) gRPC client. This reduces operational overhead, enables cross-team feature discovery, and allows a single [Feast UI](/master/reference/alpha-web-ui) deployment to browse all projects — while Feast [permissions](#feast-permissions-and-rbac) enforce tenant isolation at the data level. ```mermaid graph TD Client(["Clients / ML Services"]) -->|HTTPS| Gateway subgraph Kubernetes["Kubernetes"] Gateway["API Gateway / Ingress
(TLS + rate limiting)"] subgraph SharedInfra["Shared Infrastructure Namespace"] Operator["Feast Operator
(cluster-scoped)"] subgraph SharedDeploy["Feast Deployment (3 replicas)"] SharedRegistry["Registry Server
(gRPC + REST)"] end SharedDB[("SQL Database
(PostgreSQL)")] SharedRegistry -->|persists| SharedDB end subgraph NamespaceA["Namespace A — Team A"] subgraph DeployA2["Feast Deployment (HPA autoscaled)"] OnlineServerA["Online Feature Server"] OfflineServerA["Offline Feature Server"] end NotebookPodA(["Notebook — Team A
(Feast SDK)"]) MaterializationJobA["Materialization Job"] ConfigA["registry_type: remote
path: shared-registry:6570"] end subgraph NamespaceB["Namespace B — Team B"] subgraph DeployB2["Feast Deployment (HPA autoscaled)"] OnlineServerB["Online Feature Server"] OfflineServerB["Offline Feature Server"] end NotebookPodB(["Notebook — Team B
(Feast SDK)"]) MaterializationJobB["Materialization Job"] ConfigB["registry_type: remote
path: shared-registry:6570"] end Gateway -->|port 6566| OnlineServerA Gateway -->|port 6566| OnlineServerB OnlineServerA -->|gRPC port 6570| SharedRegistry OnlineServerB -->|gRPC port 6570| SharedRegistry OfflineServerA -->|gRPC port 6570| SharedRegistry OfflineServerB -->|gRPC port 6570| SharedRegistry Operator -->|manages| SharedRegistry Operator -->|manages| OnlineServerA Operator -->|manages| OfflineServerA Operator -->|manages| OnlineServerB Operator -->|manages| OfflineServerB NotebookPodA -.->|online REST| OnlineServerA NotebookPodA -.->|Arrow Flight| OfflineServerA NotebookPodA -.->|metadata| SharedRegistry NotebookPodB -.->|online REST| OnlineServerB NotebookPodB -.->|Arrow Flight| OfflineServerB NotebookPodB -.->|metadata| SharedRegistry MaterializationJobA -->|gRPC port 6570| SharedRegistry MaterializationJobB -->|gRPC port 6570| SharedRegistry end subgraph BackingStores["Backing Stores (inside or outside Kubernetes)"] RedisA["Online Store — Team A
(e.g. Redis, DynamoDB, etc.)"] RedisB["Online Store — Team B
(e.g. Redis, DynamoDB, etc.)"] OfflineStore["Offline Store — shared instance
(e.g. BigQuery, Redshift, etc.)
isolated via per-team datasets / schemas"] end OnlineServerA -->|reads/writes| RedisA OnlineServerB -->|reads/writes| RedisB OfflineServerA -->|Team A dataset| OfflineStore OfflineServerB -->|Team B dataset| OfflineStore MaterializationJobA -->|Team A data| OfflineStore MaterializationJobA -->|writes| RedisA MaterializationJobB -->|Team B data| OfflineStore MaterializationJobB -->|writes| RedisB style Kubernetes fill:#f0f4ff,stroke:#3366cc,color:#000 style SharedInfra fill:#fff3e0,stroke:#f57c00,color:#000 style SharedDeploy fill:#ffe0b2,stroke:#e65100,color:#000 style NamespaceA fill:#e8f5e9,stroke:#388e3c,color:#000 style NamespaceB fill:#e3f2fd,stroke:#1976d2,color:#000 style DeployA2 fill:#c8e6c9,stroke:#2e7d32,color:#000 style DeployB2 fill:#bbdefb,stroke:#1565c0,color:#000 style BackingStores fill:#fafafa,stroke:#999,color:#000 style Gateway fill:#fff9c4,stroke:#f9a825,color:#000 style Operator fill:#e8f5e9,stroke:#388e3c,color:#000 style OfflineStore fill:#f3e5f5,stroke:#7b1fa2,color:#000 ``` **Shared registry client configuration** — each tenant's `feature_store.yaml` points to the centralized registry: ```yaml registry: registry_type: remote path: shared-registry.feast-system.svc.cluster.local:6570 ``` {% hint style="info" %} **Shared vs isolated registries:** Use a shared registry when teams need to discover and reuse features across projects, and rely on Feast permissions for access control. Use isolated registries when regulatory or compliance requirements demand physical separation of metadata. {% endhint %} | | Shared Registry | Isolated Registries | | --------------------- | ------------------------------------- | ------------------------------------ | | **Feature discovery** | Cross-team — all projects visible | Siloed — each team sees only its own | | **Feast UI** | Single deployment serves all projects | Separate UI deployment per registry | | **Isolation** | Logical (Feast permissions + tags) | Physical (separate metadata stores) | | **Operational cost** | Lower — one registry to manage | Higher — N registries to maintain | | **Best for** | Feature reuse, shared ML platform | Regulatory/compliance separation | ### Components **Multi-tenancy** | Aspect | Configuration | Notes | | ---------------------- | ------------------------------------------- | ---------------------------------------------------------------------------- | | **Isolation model** | Namespace-per-team | Physical isolation via Kubernetes namespaces | | **Registry strategy** | Shared (remote) or isolated (per-namespace) | See architecture variants above | | **Network boundaries** | NetworkPolicy enforced | Cross-namespace traffic denied by default (allow-listed for shared registry) | **Storage** | Component | Configuration | Notes | | ----------------- | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | **Online Store** | Managed Redis / DynamoDB / Elasticsearch | Cloud-managed, per-tenant instances; see [supported online stores](/master/reference/online-stores) for all options | | **Offline Store** | External data warehouse (Snowflake, BigQuery) | Shared or per-tenant access controls; see [supported offline stores](/master/reference/offline-stores) for all options | **Scaling** | Component | Configuration | Notes | | --------------------------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **FeatureStore Deployment** | HPA + Cluster Autoscaler | All services (Online Feature Server, Registry, Offline Feature Server) scale together per tenant; set `maxReplicas` based on your peak load. Independent scaling across tenants. | | **Cluster** | Multi-zone node pools | Zone-aware scheduling with auto-injected topology spread constraints | **Security** | Component | Configuration | Notes | | ------------------ | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | **Authentication** | OIDC via Keycloak | Centralized identity provider | | **Authorization** | Feast permissions + Kubernetes RBAC | See [Permissions and RBAC](#feast-permissions-and-rbac) below | | **Network** | NetworkPolicies per namespace | Microsegmentation | | **Secrets** | Kubernetes Secrets (`secretRef` / `envFrom`) | Credentials injected via FeatureStore CR; use Kubernetes-native tooling (e.g. External Secrets Operator) to sync from external vaults if needed | **Observability** | Component | Purpose | Notes | | ---------------------------------------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------- | | [**OpenTelemetry**](/master/getting-started/components/open-telemetry) | Traces + metrics export | Built-in Feast integration; emits spans for feature retrieval, materialization, and registry operations | | **Prometheus** | Metrics collection | Collects OpenTelemetry metrics from Online Feature Server + Online Store | | **Grafana** | Dashboards + traces | Per-tenant and aggregate views; can display OpenTelemetry traces via Tempo or Jaeger data source | | **Jaeger** | Distributed tracing | Visualize OpenTelemetry traces for request latency analysis and debugging | **Reliability & Disaster Recovery** | Aspect | Configuration | Notes | | ------------------------ | --------------------------- | --------------------------------------------------------------------- | | **PodDisruptionBudgets** | Configured per deployment | Protects against voluntary disruptions | | **Multi-zone** | Topology spread constraints | Auto-injected by operator when scaling; survives single zone failures | | **Backup / Restore** | See recovery priority below | Strategy depends on component criticality | **Recovery priority guidance** Not all Feast components carry the same recovery urgency. The table below ranks components by restoration priority and provides guidance for **RPO** (Recovery Point Objective — maximum acceptable data loss) and **RTO** (Recovery Time Objective — maximum acceptable downtime). Specific targets depend on your backing store SLAs and organizational requirements. | Priority | Component | RPO guidance | RTO guidance | Rationale | | ------------ | --------------------------------------- | ---------------------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------ | | 1 — Critical | **Registry DB** (PostgreSQL / MySQL) | Minutes (continuous replication or frequent backups) | Minutes (failover to standby) | Contains all feature definitions and metadata; without it, no service can resolve features | | 2 — High | **Online Store** (Redis / DynamoDB) | Reconstructible via materialization | Minutes to hours (depends on data volume) | Can be fully rebuilt by re-running materialization from the offline store; no unique data to lose | | 3 — Medium | **Offline Store** (Redshift / BigQuery) | Per data warehouse SLA | Per data warehouse SLA | Source of truth for historical data; typically managed by the cloud provider with built-in replication | | 4 — Low | **Feast Operator + CRDs** | N/A (declarative, stored in Git) | Minutes (re-apply manifests) | Stateless; redeployable from version-controlled manifests | {% hint style="info" %} **Key insight:** The online store is *reconstructible* — it can always be rebuilt from the offline store by re-running materialization. This means its RPO is effectively zero (no unique data to lose), but RTO depends on how long full materialization takes for your dataset volume. For large datasets, consider maintaining Redis persistence (RDB snapshots or AOF) to reduce recovery time. {% endhint %} **Backup recommendations by topology** | Topology | Registry | Online Store | Offline Store | | -------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | **Minimal** | Manual file backups; accept downtime on failure | Not backed up (re-materialize) | N/A (file-based) | | **Standard** | Automated PostgreSQL backups (daily + WAL archiving) | Redis RDB snapshots or AOF persistence | Per cloud provider SLA | | **Enterprise** | Managed DB replication (multi-AZ); cross-region replicas for DR | Managed Redis with automatic failover (ElastiCache Multi-AZ, Memorystore HA) | Managed warehouse replication (Redshift cross-region, BigQuery cross-region) | ### Sample FeatureStore CR (per tenant) ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: team-a-production namespace: team-a spec: feastProject: team_a authz: oidc: secretRef: name: feast-oidc-secret # Secret keys: client_id, client_secret, auth_discovery_url batchEngine: configMapRef: name: feast-batch-engine services: scaling: autoscaling: minReplicas: 3 maxReplicas: 20 # Set based on your peak load metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 65 podDisruptionBudgets: minAvailable: 2 onlineStore: persistence: store: type: redis secretRef: name: feast-online-store server: resources: requests: cpu: "2" memory: 2Gi limits: cpu: "4" memory: 4Gi offlineStore: persistence: store: type: bigquery # Use snowflake.offline, redshift, etc. as alternatives — see supported offline stores secretRef: name: feast-offline-store server: resources: requests: cpu: "1" memory: 1Gi limits: cpu: "2" memory: 2Gi registry: local: persistence: store: type: sql secretRef: name: feast-registry-store server: restAPI: true resources: requests: cpu: "1" memory: 1Gi limits: cpu: "2" memory: 2Gi --- apiVersion: v1 kind: ConfigMap metadata: name: feast-batch-engine namespace: team-a data: config: | type: spark spark_master: k8s://https://kubernetes.default.svc:443 spark_app_name: feast-materialization ``` *** ## Feast Permissions and RBAC Feast provides a built-in permissions framework that secures resources at the application level, independently of Kubernetes RBAC. Permissions are defined as Python objects in your feature repository and registered via `feast apply`. For full details, see the [Permission concept](/master/getting-started/concepts/permission) and [RBAC architecture](/master/getting-started/architecture/rbac) docs. ### How it works ```mermaid graph LR subgraph Client["Client Request"] Token["Auth Token
(OIDC / K8s SA)"] end subgraph Server["Feast Server"] Extractor["Token Extractor"] Parser["Token Parser"] Enforcer["Policy Enforcer"] end subgraph Registry["Registry"] Permissions["Permission Objects"] end Token --> Extractor Extractor --> Parser Parser -->|user roles/groups/ns| Enforcer Enforcer -->|match resource + action| Permissions Enforcer -->|allow / deny| Response(["Response"]) style Client fill:#e3f2fd,stroke:#1976d2,color:#000 style Server fill:#fff3e0,stroke:#f57c00,color:#000 style Registry fill:#e8f5e9,stroke:#388e3c,color:#000 ``` Permission enforcement happens on the server side (Online Feature Server, Offline Feature Server, Registry Server). There is no enforcement when using the Feast SDK with a local provider. ### Actions Feast defines eight granular actions: | Action | Description | | --------------- | --------------------------- | | `CREATE` | Create a new Feast object | | `DESCRIBE` | Read object metadata/state | | `UPDATE` | Modify an existing object | | `DELETE` | Remove an object | | `READ_ONLINE` | Read from the online store | | `READ_OFFLINE` | Read from the offline store | | `WRITE_ONLINE` | Write to the online store | | `WRITE_OFFLINE` | Write to the offline store | Convenience aliases are provided: | Alias | Includes | | ------------- | ------------------------------------------- | | `ALL_ACTIONS` | All eight actions | | `READ` | `READ_ONLINE` + `READ_OFFLINE` | | `WRITE` | `WRITE_ONLINE` + `WRITE_OFFLINE` | | `CRUD` | `CREATE` + `DESCRIBE` + `UPDATE` + `DELETE` | ### Protected resource types Permissions can be applied to any of these Feast object types: `Project`, `Entity`, `FeatureView`, `OnDemandFeatureView`, `BatchFeatureView`, `StreamFeatureView`, `FeatureService`, `DataSource`, `ValidationReference`, `SavedDataset`, `Permission` The constant `ALL_RESOURCE_TYPES` includes all of the above. `ALL_FEATURE_VIEW_TYPES` includes all feature view subtypes. ### Policy types | Policy | Match criteria | Use case | | -------------------------------------------------------------- | -------------------------------------------------------------- | ------------------------------------ | | `RoleBasedPolicy(roles=[...])` | User must have at least one of the listed roles | Kubernetes RBAC roles, OIDC roles | | `GroupBasedPolicy(groups=[...])` | User must belong to at least one of the listed groups | LDAP/OIDC group membership | | `NamespaceBasedPolicy(namespaces=[...])` | User's service account must be in one of the listed namespaces | Kubernetes namespace-level isolation | | `CombinedGroupNamespacePolicy(groups=[...], namespaces=[...])` | User must match at least one group **or** one namespace | Flexible cross-cutting policies | | `AllowAll` | Always grants access | Development / unsecured resources | ### Example: Role-based permissions This is the most common pattern — separate admin and read-only roles: ```python from feast.feast_object import ALL_RESOURCE_TYPES from feast.permissions.action import READ, ALL_ACTIONS, AuthzedAction from feast.permissions.permission import Permission from feast.permissions.policy import RoleBasedPolicy admin_perm = Permission( name="feast_admin_permission", types=ALL_RESOURCE_TYPES, policy=RoleBasedPolicy(roles=["feast-admin-role"]), actions=ALL_ACTIONS, ) user_perm = Permission( name="feast_user_permission", types=ALL_RESOURCE_TYPES, policy=RoleBasedPolicy(roles=["feast-user-role"]), actions=[AuthzedAction.DESCRIBE] + READ, ) ``` ### Example: Namespace-based isolation for multi-tenant deployments Use `NamespaceBasedPolicy` to restrict access based on the Kubernetes namespace of the calling service account — ideal for the shared-registry enterprise topology. Each team gets two permissions: full access to its own resources (matched by `team` tag), and read-only access to resources any team has explicitly published as shared (matched by `visibility: shared` tag). The two `required_tags` target **different** resources — a feature view tagged `team: team-b, visibility: shared` matches only the second permission for Team A, enabling cross-team discovery without granting write access: ```python from feast.feast_object import ALL_RESOURCE_TYPES from feast.permissions.action import ALL_ACTIONS, READ, AuthzedAction from feast.permissions.permission import Permission from feast.permissions.policy import NamespaceBasedPolicy # Team A: full access to its own resources team_a_own = Permission( name="team_a_full_access", types=ALL_RESOURCE_TYPES, required_tags={"team": "team-a"}, # matches only Team A's resources policy=NamespaceBasedPolicy(namespaces=["team-a"]), actions=ALL_ACTIONS, ) # Team A: read-only access to shared resources published by ANY team # e.g. a Team B feature view tagged {team: team-b, visibility: shared} # satisfies required_tags here but NOT team_a_own above team_a_read_shared = Permission( name="team_a_read_shared", types=ALL_RESOURCE_TYPES, required_tags={"visibility": "shared"}, # matches shared resources from any team policy=NamespaceBasedPolicy(namespaces=["team-a"]), actions=[AuthzedAction.DESCRIBE] + READ, ) # Team B: mirror of the above — full access to its own, read-only to shared team_b_own = Permission( name="team_b_full_access", types=ALL_RESOURCE_TYPES, required_tags={"team": "team-b"}, policy=NamespaceBasedPolicy(namespaces=["team-b"]), actions=ALL_ACTIONS, ) team_b_read_shared = Permission( name="team_b_read_shared", types=ALL_RESOURCE_TYPES, required_tags={"visibility": "shared"}, policy=NamespaceBasedPolicy(namespaces=["team-b"]), actions=[AuthzedAction.DESCRIBE] + READ, ) ``` ### Example: Combined group + namespace policy For organizations that use both OIDC groups and Kubernetes namespaces for identity — ideal when platform engineers lack a dedicated namespace but need cross-team visibility, or when OIDC group membership and namespace ownership should independently grant access: ```python from feast.feast_object import ALL_RESOURCE_TYPES from feast.permissions.action import ALL_ACTIONS, READ, AuthzedAction from feast.permissions.permission import Permission from feast.permissions.policy import CombinedGroupNamespacePolicy # Platform engineers (OIDC group) OR any team namespace can read shared features. # This covers platform engineers who have no dedicated K8s namespace of their own # but need cross-team feature discovery. platform_read_shared = Permission( name="platform_read_shared", types=ALL_RESOURCE_TYPES, required_tags={"visibility": "shared"}, policy=CombinedGroupNamespacePolicy( groups=["ml-platform"], # OIDC group for platform/infra engineers namespaces=["team-a", "team-b"], # team namespaces from enterprise topology ), actions=[AuthzedAction.DESCRIBE] + READ, ) # ML engineers (OIDC) OR team namespace owners have full write access. # Either identity alone is sufficient — useful during namespace migration or # when the same person holds both the OIDC role and the team namespace. ml_engineer_write = Permission( name="ml_engineer_full_access", types=ALL_RESOURCE_TYPES, policy=CombinedGroupNamespacePolicy( groups=["ml-engineers"], namespaces=["team-a", "team-b"], ), actions=ALL_ACTIONS, ) ``` ### Example: Fine-grained resource filtering Permissions support `name_patterns` (regex) and `required_tags` for targeting specific resources: ```python from feast.feature_view import FeatureView from feast.data_source import DataSource from feast.permissions.action import AuthzedAction, READ from feast.permissions.permission import Permission from feast.permissions.policy import RoleBasedPolicy sensitive_fv_perm = Permission( name="sensitive_feature_reader", types=[FeatureView], name_patterns=[".*sensitive.*", ".*pii.*"], policy=RoleBasedPolicy(roles=["trusted-reader"]), actions=[AuthzedAction.READ_OFFLINE], ) high_risk_ds_writer = Permission( name="high_risk_ds_writer", types=[DataSource], required_tags={"risk_level": "high"}, policy=RoleBasedPolicy(roles=["admin", "data_team"]), actions=[AuthzedAction.WRITE_ONLINE, AuthzedAction.WRITE_OFFLINE], ) ``` ### Authorization configuration Enable auth enforcement in `feature_store.yaml`: ```yaml auth: type: kubernetes # or: oidc ``` For OIDC: ```yaml auth: type: oidc client_id: feast-client auth_server_url: https://keycloak.example.com/realms/feast auth_discovery_url: https://keycloak.example.com/realms/feast/.well-known/openid-configuration ``` {% hint style="warning" %} **Permission granting order:** Feast uses an *affirmative* decision strategy — if **any** matching permission grants access, the request is allowed. Access is denied only when **all** matching permissions deny the user. If no permission matches a resource + action combination, access is **denied**. Resources that do not match any configured permission are unsecured. Always define explicit coverage for all critical resources. {% endhint %} ### Recommended RBAC by topology | Topology | Auth type | Policy type | Guidance | | -------------------------------- | ------------------------- | -------------------------------------------------------- | --------------------------------------------------- | | **Minimal** | `no_auth` or `kubernetes` | `RoleBasedPolicy` | Basic admin/reader roles | | **Standard** | `kubernetes` | `RoleBasedPolicy` | K8s service account roles | | **Enterprise (isolated)** | `oidc` or `kubernetes` | `RoleBasedPolicy` + `GroupBasedPolicy` | Per-team OIDC groups | | **Enterprise (shared registry)** | `kubernetes` | `NamespaceBasedPolicy` or `CombinedGroupNamespacePolicy` | Namespace isolation with tag-based resource scoping | *** ## Infrastructure-Specific Recommendations Choosing the right online store, offline store, and registry backend depends on your cloud environment and existing infrastructure. The table below maps common deployment environments to recommended Feast components. ### Recommendation matrix ```mermaid graph TD subgraph AWS["AWS / EKS / ROSA"] A_Online["Online: ElastiCache Redis
or DynamoDB"] A_Offline["Offline: Redshift
or Snowflake or Athena"] A_Registry["Registry: RDS PostgreSQL (SQL)
or S3"] A_Compute["Compute: Snowflake Engine
or Spark on EMR
or Ray (KubeRay)"] end subgraph GCP["GCP / GKE"] G_Online["Online: Memorystore Redis
or Bigtable or Datastore"] G_Offline["Offline: BigQuery
or Snowflake"] G_Registry["Registry: Cloud SQL PostgreSQL
or GCS"] G_Compute["Compute: Snowflake Engine
or Spark on Dataproc
or Ray (KubeRay)"] end subgraph OnPrem["On-Premise / OpenShift"] O_Online["Online: Redis
or PostgreSQL"] O_Offline["Offline: Spark + MinIO
or PostgreSQL or Trino or Oracle"] O_Registry["Registry: PostgreSQL (SQL)"] O_Compute["Compute: Spark
or Ray (KubeRay)"] end style AWS fill:#fff3e0,stroke:#f57c00,color:#000 style GCP fill:#e3f2fd,stroke:#1976d2,color:#000 style OnPrem fill:#e8f5e9,stroke:#388e3c,color:#000 ``` ### AWS / EKS / ROSA | Component | Recommended | Alternative | Notes | | ------------------ | ------------------------ | ------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Online Store** | **Redis** (ElastiCache) | DynamoDB | Redis offers TTL at retrieval, concurrent writes, Java/Go SDK support. DynamoDB is fully managed with zero ops. | | **Offline Store** | **Redshift** | Snowflake, Athena (contrib), Spark | Redshift is the core AWS offline store. Use Snowflake if it's already your warehouse. Athena for S3-native query patterns. | | **Registry** | **SQL** (RDS PostgreSQL) | S3 | SQL registry required for concurrent materialization writers. S3 registry is simpler but limited to single-writer. | | **Compute Engine** | **Snowflake Engine** | Spark on EMR, [Ray (KubeRay)](/master/reference/compute-engine/ray) | Snowflake engine when your offline/online stores are Snowflake. Spark for S3-based pipelines. Ray with KubeRay for Kubernetes-native distributed processing. | {% hint style="info" %} **ROSA (Red Hat OpenShift on AWS):** Same store recommendations as EKS. Use OpenShift Routes instead of Ingress for TLS termination. Leverage OpenShift's built-in OAuth for `auth.type: kubernetes` integration. {% endhint %} ### GCP / GKE | Component | Recommended | Alternative | Notes | | ------------------ | ------------------------------ | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Online Store** | **Redis** (Memorystore) | Bigtable, Datastore | Redis for latency-sensitive workloads. Bigtable for very large-scale feature storage. Datastore is GCP-native and zero-ops. | | **Offline Store** | **BigQuery** | Snowflake, Spark (Dataproc) | BigQuery is the core GCP offline store with full feature support. | | **Registry** | **SQL** (Cloud SQL PostgreSQL) | GCS | SQL for multi-writer. GCS for simple single-writer setups. | | **Compute Engine** | **Snowflake Engine** | Spark on Dataproc, [Ray (KubeRay)](/master/reference/compute-engine/ray) | Use Snowflake engine if your offline store is Snowflake. Spark for BigQuery + GCS pipelines. Ray with KubeRay for Kubernetes-native distributed processing. | ### On-Premise / OpenShift / Self-Managed Kubernetes | Component | Recommended | Alternative | Notes | | ------------------ | ------------------------------------ | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | **Online Store** | **Redis** (self-managed or operator) | PostgreSQL (contrib) | Redis for best performance. PostgreSQL if you want to minimize infrastructure components. | | **Offline Store** | **Spark** + MinIO (contrib) | PostgreSQL (contrib), Trino (contrib), Oracle (contrib), DuckDB | Spark for scale. PostgreSQL for simpler setups. Oracle for enterprise customers with existing Oracle infrastructure. DuckDB for development only. | | **Registry** | **SQL** (PostgreSQL) | — | Always use SQL registry in production on-prem. File-based registries do not support concurrent writers. | | **Compute Engine** | **Spark** | [Ray (KubeRay)](/master/reference/compute-engine/ray) | Run Spark on Kubernetes or standalone. Ray with KubeRay for Kubernetes-native distributed DAG execution. | {% hint style="warning" %} **Multi-replica constraint:** When scaling any Feast service to multiple replicas (via the Feast Operator), you **must** use database-backed persistence for all enabled services. File-based stores (SQLite, DuckDB, `registry.db`) are incompatible with multi-replica deployments. See [Scaling Feast](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast#horizontal-scaling-with-the-feast-operator) for details. {% endhint %} *** ## Air-Gapped / Disconnected Environment Deployments Production environments in regulated industries (finance, government, defense) often have no outbound internet access from the Kubernetes cluster. The Feast Operator supports air-gapped deployments through custom container images, init container controls, and standard Kubernetes image-pull mechanisms. ### Default init container behavior When `feastProjectDir` is set on the FeatureStore CR, the operator creates up to two init containers unless `services.disableInitContainers` is `true`: 1. **`feast-init`** — bootstraps the feature repository by running `git clone`, `feast init`, or copying a repository from `feastProjectDir.packaged.featureRepoPath`. It then writes the operator-generated `feature_store.yaml` into the initialized repository. 2. **`feast-apply`** — runs `feast apply` to register feature definitions in the registry. Controlled by `runFeastApplyOnInit` (defaults to `true`). Skipped when `disableInitContainers` is `true`. In air-gapped environments, use `feastProjectDir.packaged` to identify a feature repository baked into an image. The operator supports two lifecycle modes: * Keep init containers enabled to refresh shared storage from the image, generate configuration from the FeatureStore CR, and optionally run `feast apply`. * Set `services.disableInitContainers: true` to run directly from the baked path and treat its `feature_store.yaml` as authoritative. ### Air-gapped deployment workflow ```mermaid graph TD subgraph BuildEnv["Build Environment (internet access)"] Code["Feature repo source code"] Base["Base Feast image
(feastdev/feature-server)"] Custom["Custom image with
bundled feature repo"] Code --> Custom Base --> Custom end subgraph InternalRegistry["Internal Container Registry"] Mirror["registry.internal.example.com
/feast/feature-server:release"] end subgraph AirGappedCluster["Air-Gapped Kubernetes Cluster"] SA["ServiceAccount
(imagePullSecrets)"] CR["FeatureStore CR
feastProjectDir.packaged
disableInitContainers: true"] Deploy["Feast Deployment
(no init containers)"] SA --> Deploy CR --> Deploy end Custom -->|push| Mirror Mirror -->|pull| Deploy ``` **Steps:** 1. **Build a custom container image** that bundles the feature repository and all Python dependencies into the Feast base image. 2. **Push** the image to your internal container registry. 3. **Configure `feastProjectDir.packaged`** with the image and the canonical absolute path to the bundled repository. Do not use `.`, `..`, repeated separators, or a trailing separator, and keep the path outside operator-mounted locations such as `/feast-data` so it cannot overlap the staged repository. 4. **Choose the lifecycle:** leave init containers enabled for operator-managed configuration and `feast apply`, or set `services.disableInitContainers: true` to use the baked repository and configuration directly. 5. **Set `imagePullPolicy: IfNotPresent`** (or `Never` if images are pre-loaded on nodes). 6. **Configure `imagePullSecrets`** on the namespace's ServiceAccount — the FeatureStore CRD does not expose an `imagePullSecrets` field, so use the standard Kubernetes approach of attaching secrets to the ServiceAccount that the pods run under. ### Sample FeatureStore CR (air-gapped) ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: airgap-production spec: feastProject: my_project feastProjectDir: packaged: image: registry.internal.example.com/feast/feature-server:release featureRepoPath: /opt/feast/feature_repo services: disableInitContainers: true onlineStore: persistence: store: type: redis secretRef: name: feast-online-store server: imagePullPolicy: IfNotPresent resources: requests: cpu: "1" memory: 1Gi limits: cpu: "2" memory: 2Gi registry: local: persistence: store: type: sql secretRef: name: feast-registry-store server: imagePullPolicy: IfNotPresent ``` The packaged image is the default for every Feast service and for the `feast-init` and `feast-apply` init containers. A per-service `image` still takes precedence for that service, and `services.initImage` takes precedence for both init containers. Remove `disableInitContainers: true` to use operator-managed staging and startup apply instead. {% hint style="info" %} **Pre-populating the registry:** With init containers disabled, `feast apply` does not run on pod startup. You can populate the registry by: 1. **Running `feast apply` from your CI/CD pipeline** that has network access to the registry DB. 2. **Using the FeatureStore CR's built-in CronJob** (`spec.cronJob`) — the operator creates a Kubernetes CronJob that runs `feast apply` and `feast materialize-incremental` on a schedule. The CronJob runs inside the cluster (no external access needed) and can use a custom image just like the main deployment. This is the recommended approach for air-gapped environments. 3. **Running `feast apply` manually** from the build environment before deploying the CR. {% endhint %} ### Air-gapped deployment checklist {% hint style="warning" %} **Pre-stage the following artifacts before deploying Feast in an air-gapped environment:** * **Container images** — Feast feature server image (with bundled feature repo) pushed to internal registry * **CRD manifests** — Feast Operator CRDs and operator deployment manifests available locally * **Store credentials** — Kubernetes Secrets for online store, offline store, and registry DB connections created in the target namespace * **Python packages** (if using custom on-demand transforms) — bundled into the custom image or available from an internal PyPI mirror * **ServiceAccount configuration** — `imagePullSecrets` attached to the ServiceAccount used by the Feast deployment {% endhint %} *** ## Hybrid Store Configuration The hybrid store feature allows a single Feast deployment to route feature operations to multiple backends based on tags or data sources. This is useful when different feature views have different latency, cost, or compliance requirements. ### Hybrid online store The `HybridOnlineStore` routes online operations to different backends based on a configurable tag on the `FeatureView`. ```mermaid graph LR FS["Online Feature Server"] --> Router["HybridOnlineStore
(routes by tag)"] Router -->|"tag: dynamodb"| DDB["DynamoDB"] Router -->|"tag: redis"| RD["Redis"] style Router fill:#fff3e0,stroke:#f57c00,color:#000 style DDB fill:#e3f2fd,stroke:#1976d2,color:#000 style RD fill:#fce4ec,stroke:#c62828,color:#000 ``` **`feature_store.yaml` configuration:** ```yaml project: my_feature_repo registry: data/registry.db provider: local online_store: type: hybrid routing_tag: team online_stores: - type: dynamodb conf: region: us-east-1 - type: redis conf: connection_string: "redis-cluster:6379" redis_type: redis_cluster ``` **Feature view with routing tag:** ```python from feast import FeatureView user_features = FeatureView( name="user_features", entities=[user_entity], source=user_source, tags={"team": "dynamodb"}, # Routes to DynamoDB backend ) transaction_features = FeatureView( name="transaction_features", entities=[txn_entity], source=txn_source, tags={"team": "redis"}, # Routes to Redis backend ) ``` The tag value must match the online store `type` name (e.g. `dynamodb`, `redis`, `bigtable`). ### Hybrid offline store The `HybridOfflineStore` routes offline operations to different backends based on the `batch_source` type of each `FeatureView`. ```mermaid graph LR Client["Materialization /
Training Job"] --> Router["HybridOfflineStore
(routes by source type)"] Router -->|"SparkSource"| Spark["Spark"] Router -->|"RedshiftSource"| RS["Redshift"] style Router fill:#fff3e0,stroke:#f57c00,color:#000 style Spark fill:#e8f5e9,stroke:#388e3c,color:#000 style RS fill:#e3f2fd,stroke:#1976d2,color:#000 ``` **`feature_store.yaml` configuration:** ```yaml project: my_feature_repo registry: data/registry.db provider: local offline_store: type: hybrid_offline_store.HybridOfflineStore offline_stores: - type: spark conf: spark_master: local[*] spark_app_name: feast_spark_app - type: redshift conf: cluster_id: my-redshift-cluster region: us-east-1 database: feast_db user: feast_user s3_staging_location: s3://my-bucket/feast-staging iam_role: arn:aws:iam::123456789012:role/FeastRedshiftRole ``` **Feature views with different sources:** ```python from feast import FeatureView, Entity, ValueType from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import SparkSource from feast.infra.offline_stores.redshift_source import RedshiftSource user_features = FeatureView( name="user_features", entities=[user_entity], source=SparkSource(path="s3://bucket/user_features"), # Routes to Spark ) activity_features = FeatureView( name="user_activity", entities=[user_entity], source=RedshiftSource( # Routes to Redshift table="user_activity", event_timestamp_column="event_ts", ), ) ``` {% hint style="warning" %} **Hybrid offline store constraint:** `get_historical_features` requires all requested feature views to share the same `batch_source` type within a single call. You cannot join features across different offline engines in one retrieval request. {% endhint %} *** ## Performance Considerations For detailed server-level tuning (worker counts, timeouts, keep-alive, etc.), see the [Online Server Performance Tuning](/master/how-to-guides/online-server-performance-tuning) guide. ### Online feature server sizing | Traffic tier | Replicas | CPU (per pod) | Memory (per pod) | Notes | | --------------------- | ---------- | ------------- | ---------------- | ---------------------- | | Low (<100 RPS) | 1–2 | 500m–1 | 512Mi–1Gi | Minimal production | | Medium (100–1000 RPS) | 2–5 (HPA) | 1–2 | 1–2Gi | Standard production | | High (>1000 RPS) | 5–20 (HPA) | 2–4 | 2–4Gi | Enterprise, per-tenant | ### Online store latency guidelines | Store | p50 latency | p99 latency | Best for | | ------------------ | ----------- | ----------- | ------------------------------------- | | **Redis** (single) | <1ms | <5ms | Lowest latency, small-medium datasets | | **Redis Cluster** | <2ms | <10ms | High availability + low latency | | **DynamoDB** | <5ms | <20ms | Serverless, variable traffic | | **PostgreSQL** | <5ms | <30ms | On-prem, simplicity | | **Remote (HTTP)** | <10ms | <50ms | Client-server separation | ### Connection pooling for remote online store When using the [Remote Online Store](/master/reference/online-stores/remote) (client-server architecture), connection pooling significantly reduces latency by reusing TCP/TLS connections: ```yaml online_store: type: remote path: http://feast-feature-server:80 connection_pool_size: 50 # Max connections in pool (default: 50) connection_idle_timeout: 300 # Idle timeout in seconds (default: 300) connection_retries: 3 # Retry count with exponential backoff ``` **Tuning by workload:** | Workload | `connection_pool_size` | `connection_idle_timeout` | `connection_retries` | | -------------------------- | ---------------------- | ------------------------- | -------------------- | | High-throughput inference | 100 | 600 | 5 | | Long-running batch service | 50 | 0 (never close) | 3 | | Resource-constrained edge | 10 | 60 | 2 | ### Registry performance * **SQL registry** (PostgreSQL, MySQL) is required for concurrent materialization jobs writing to the registry simultaneously. * **File-based registries** (S3, GCS, local) serialize the entire registry on each write — suitable only for single-writer scenarios. * For read-heavy workloads, scale the Registry Server to multiple replicas (all connecting to the same database). ### Registry cache tuning at scale Each Feast server pod maintains its own in-memory copy of the registry metadata. With multiple Gunicorn workers per pod, the total number of independent registry copies is **replicas x workers**. For example, 5 replicas with 4 workers each means 20 copies of the registry in memory, each refreshing independently. With the default `cache_mode: sync`, the refresh is **synchronous** — when the TTL expires, the next request blocks until the full registry is re-downloaded. At scale, this causes periodic latency spikes across multiple pods simultaneously. **Recommendation:** Use `cache_mode: thread` with a higher TTL in production to avoid refresh storms: ```yaml # In the Operator secret for SQL/DB-backed registries: registry: registry_type: sql path: postgresql://:@:5432/feast cache_mode: thread cache_ttl_seconds: 300 ``` For the server-side refresh interval, set `registryTTLSeconds` on the CR: ```yaml spec: services: onlineStore: server: workerConfigs: registryTTLSeconds: 300 ``` | Scenario | `cache_mode` | `cache_ttl_seconds` | `registryTTLSeconds` | | ------------------------------------ | ---------------- | ------------------- | -------------------- | | Development / iteration | `sync` (default) | 5–10 | 5 | | Production (low-latency) | `thread` | 300 | 300 | | Production (frequent schema changes) | `thread` | 60 | 60 | {% hint style="info" %} `registryTTLSeconds` on the CR controls the **server-side** refresh interval. `cache_ttl_seconds` in the registry secret controls the **SDK client** refresh. In Operator deployments, the CR field is what matters for serving performance. For a deep dive into sync vs thread mode trade-offs, memory impact, and freshness considerations, see the [Registry Cache Tuning](/master/how-to-guides/online-server-performance-tuning#registry-cache-tuning) section in the performance tuning guide. {% endhint %} ### Materialization performance | Data volume | Recommended engine | Notes | | ------------ | -------------------------------------------------------- | ----------------------------------------------------------------- | | <1M rows | In-process (default) | Simple, no external dependencies | | 1M–100M rows | Snowflake Engine, Spark, or Ray | Distributed processing | | >100M rows | Spark on Kubernetes / EMR / Dataproc, or Ray via KubeRay | Full cluster-scale materialization with distributed DAG execution | For detailed engine configuration, see [Scaling Materialization](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast#scaling-materialization). ### Redis sizing guidelines | Metric | Guideline | | ---------------- | ----------------------------------------------------------------------------------------------- | | **Memory** | \~100 bytes per feature value (varies by data type). For 1M entities x 50 features = \~5GB. | | **Connections** | Each online feature server replica opens a connection pool. Plan for `replicas x pool_size`. | | **TTL** | Set `key_ttl_seconds` in `feature_store.yaml` to auto-expire stale data and bound memory usage. | | **Cluster mode** | Use Redis Cluster for >25GB datasets or >10K connections. | *** ## Design Principles Understanding the following principles helps you choose and customize the right topology. ### Control plane vs data plane ```mermaid graph LR subgraph ControlPlane["Control Plane"] Operator["Feast Operator"] Registry["Registry Server"] end subgraph DataPlane["Data Plane"] OnlineServer["Online Feature Server"] OfflineServer["Offline Feature Server"] end subgraph BackingStores["Backing Stores (external)"] OnlineStore["Online Store
(e.g. Redis, DynamoDB, PostgreSQL, etc.)"] OfflineStore["Offline Store
(e.g. Redshift, BigQuery, Spark, etc.)"] RegistryDB["Registry DB
(PostgreSQL / MySQL)"] end ControlPlane -->|configures| DataPlane DataPlane -->|reports status| ControlPlane OnlineServer --> OnlineStore OfflineServer --> OfflineStore Registry --> RegistryDB style ControlPlane fill:#e8f5e9,stroke:#388e3c,color:#000 style DataPlane fill:#e3f2fd,stroke:#1976d2,color:#000 style BackingStores fill:#fce4ec,stroke:#c62828,color:#000 ``` * **Control plane** (Operator + Registry Server) manages feature definitions, metadata, and lifecycle. It changes infrequently and should be highly available. * **Data plane** (Online Feature Server + Offline Feature Server) handles the actual feature reads/writes at request time. It must scale with traffic. * **Backing stores** (databases, object storage) hold the actual data. These are stateful and managed independently. ### Stateless vs stateful components The Feast Operator deploys all Feast services (Online Feature Server, Offline Feature Server, Registry Server) in a **single shared Deployment**. When scaling (`spec.replicas > 1` or HPA autoscaling), all services scale together. {% hint style="warning" %} **Scaling requires DB-backed persistence for all enabled services.** The operator enforces this via CRD validation: * **Online Store** — must use DB persistence (e.g. `type: redis`, `type: dynamodb`, `type: postgres`) * **Offline Store** — if enabled, must use DB persistence (e.g. `type: redshift`, `type: bigquery`, `type: spark`, `type: postgres`) * **Registry** — must use SQL persistence (`type: sql`), a remote registry, or S3/GCS file-backed registry File-based stores (SQLite, DuckDB, `registry.db`) are **rejected** when `replicas > 1` or autoscaling is configured. {% endhint %} | Component | Type | Scaling | DB-backed requirement | | ---------------------------------------- | ---------------------------- | ---------------------------------------------------------- | ---------------------------------------------------------------------------------- | | Online Feature Server | **Stateless** (server) | Scales with the shared Deployment (HPA or `spec.replicas`) | Online store must use DB persistence (e.g. Redis, DynamoDB, PostgreSQL) | | Offline Feature Server | **Stateless** (server) | Scales with the shared Deployment (HPA or `spec.replicas`) | Offline store must use DB persistence (e.g. Redshift, BigQuery, Spark, PostgreSQL) | | Registry Server | **Stateless** (server) | Scales with the shared Deployment (HPA or `spec.replicas`) | Registry must use SQL, remote, or S3/GCS persistence | | Online Store (Redis, DynamoDB, etc.) | **Stateful** (backing store) | Scale via managed service or clustering | Managed independently of Feast services | | Offline Store (Redshift, BigQuery, etc.) | **Stateful** (backing store) | Scale via cloud-managed infrastructure | Managed independently of Feast services | | Registry DB (PostgreSQL, MySQL) | **Stateful** (backing store) | Scale via managed database service | Managed independently of Feast services | ### Scalability guidelines ```mermaid graph TD Read["Read traffic increase"] -->|scale| FS["Online Feature Server replicas (HPA)"] Write["Write / materialization load"] -->|scale| Engine["Compute Engine
(Spark / Ray / Snowflake)"] Storage["Data volume growth"] -->|scale| Store["Online / Offline Store capacity"] FS -.- Independent["Scale independently"] Engine -.- Independent Store -.- Independent style Read fill:#e3f2fd,stroke:#1976d2,color:#000 style Write fill:#fff3e0,stroke:#f57c00,color:#000 style Storage fill:#f3e5f5,stroke:#7b1fa2,color:#000 style FS fill:#e3f2fd,stroke:#1976d2,color:#000 style Engine fill:#fff3e0,stroke:#f57c00,color:#000 style Store fill:#f3e5f5,stroke:#7b1fa2,color:#000 ``` * **Read scaling** — increase Online Feature Server replicas; they are stateless and scale linearly. * **Write scaling** — use a distributed compute engine ([Spark](/master/reference/compute-engine/spark), [Ray/KubeRay](/master/reference/compute-engine/ray), or [Snowflake](/master/reference/compute-engine/snowflake)) for materialization. * **Storage scaling** — scale online and offline stores independently based on data volume and query patterns. For detailed scaling configuration, see [Scaling Feast](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast). *** ## Topology Comparison | Capability | Minimal | Standard | Enterprise | | ------------------------- | :-------: | :-------------: | :-------------------------------------------: | | **High availability** | No | Yes | Yes | | **Autoscaling** | No | HPA | HPA + Cluster Autoscaler | | **TLS / Ingress** | No | Yes | Yes + API Gateway | | **RBAC** | No | Kubernetes RBAC | OIDC + fine-grained RBAC | | **Multi-tenancy** | No | No | Namespace-per-team | | **Shared registry** | N/A | N/A | Optional (remote registry) | | **Hybrid stores** | No | Optional | Recommended for mixed backends | | **Observability** | Logs only | Basic metrics | OpenTelemetry + Prometheus + Grafana + Jaeger | | **Disaster recovery** | No | Partial | Full backup/restore | | **Network policies** | No | Optional | Enforced | | **Recommended team size** | 1–3 | 3–15 | 15+ | *** ## Next Steps * [Feast on Kubernetes](/master/how-to-guides/feast-on-kubernetes) — install the Feast Operator and deploy your first FeatureStore CR * [Scaling Feast](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast) — detailed HPA, registry scaling, and materialization engine configuration * [Online Server Performance Tuning](/master/how-to-guides/online-server-performance-tuning) — worker counts, timeouts, keep-alive, and server-level tuning * [Starting Feast Servers in TLS Mode](/master/how-to-guides/starting-feast-servers-tls-mode) — enable TLS for secure communication * [Running Feast in Production](/master/how-to-guides/running-feast-in-production) — CI/CD, materialization scheduling, and model serving patterns * [Multi-Team Feature Store Setup](/master/how-to-guides/feast-snowflake-gcp-aws/federated-feature-store) — federated feature store for multi-team environments * [Permission Concepts](/master/getting-started/concepts/permission) — full permission model reference * [RBAC Architecture](/master/getting-started/architecture/rbac) — authorization architecture details * [OpenTelemetry Integration](/master/getting-started/components/open-telemetry) — traces and metrics for Feast servers * [Hybrid Online Store](/master/reference/online-stores/hybrid) — hybrid online store configuration reference * [Hybrid Offline Store](/master/reference/offline-stores/hybrid) — hybrid offline store configuration reference # Online Server Performance Tuning This guide covers end-to-end performance tuning for the Feast Python online feature server, with a focus on Kubernetes deployments using the [Feast Operator](/master/how-to-guides/feast-on-kubernetes). It brings together worker configuration, registry caching, online store selection, horizontal scaling, observability, and network optimization into a single operational playbook. ## Request lifecycle Understanding where time is spent during a `get_online_features()` call helps you focus tuning efforts on the right layer: 1. **Registry lookup** — resolve feature references to metadata (cached locally). 2. **Online store read** — fetch feature values from the backing database (network I/O). 3. **On-demand transformation** — execute any ODFVs attached to the request (CPU). 4. **Response serialization** — encode the result as JSON or protobuf. In most production deployments, **step 2 (online store read)** dominates latency. Registry refresh (step 1) can cause periodic latency spikes if not configured for background refresh. Step 3 matters only when ODFVs are present. *** ## Feature view and feature service structuring How you organize features into feature views directly affects online serving latency, because **each distinct feature view in a request produces a separate online store read**. This is a structural decision that no amount of runtime tuning can fix after the fact. ### Fewer feature views = fewer store round-trips When the server processes a `get_online_features()` call, it groups the requested features by their source feature view and issues one `online_read()` per group. For sync stores, these reads are **sequential**; for async stores (DynamoDB, MongoDB), they run **concurrently** via `asyncio.gather()`, but each is still a separate network round-trip. | Request shape | Store reads | Impact | | ------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------- | | 10 features from **1** feature view | 1 read | Minimal latency | | 10 features from **5** feature views | 5 reads | 5× the network round-trips (sequential) or 5 concurrent reads competing for connections (async) | | 10 features from **10** feature views | 10 reads | Latency dominated by store reads, not transformation | **Guideline:** For features that share the same entity key and are frequently requested together, consolidate them into a **single feature view**. This reduces the number of store round-trips per request. Split feature views only when features have different entities, different materialization schedules, or different data source update frequencies. {% hint style="info" %} **Redis exception:** The Redis online store overrides `get_online_features()` to batch all `HMGET` commands across every feature view into a **single pipeline execution**. Because all feature views for the same entity share one Redis hash key, the number of Redis round trips is always **1**, regardless of how many feature views the request touches. This means the "fewer feature views" guideline is less critical for Redis than for other stores — but consolidating feature views still reduces serialization and protobuf overhead at the application layer. {% endhint %} ### Feature services are free (and can be faster) A [Feature Service](/master/getting-started/concepts/feature-retrieval) is a named collection of feature references — it's a convenience grouping, not a separate storage or execution unit. Using a feature service adds only a registry lookup (cached) compared to listing features individually. There is no performance penalty for using feature services, and they are the recommended way to define stable, versioned feature sets for production models. For latency-critical services, feature services also unlock **pre-computed feature vectors** (`precompute_online=True`), which reduce store reads from O(N feature views) to O(1). See the [Pre-computed feature vectors](#pre-computed-feature-vectors) section below for details and benchmarks. ### ODFV overhead is additive Regular feature views incur only **store read** cost. On-demand feature views add **CPU-bound transformation** cost on top: ``` Total request latency ≈ store_read_time + sum(odfv_transformation_times) ``` Each ODFV in the request runs its transformation function after all store reads complete. The overhead depends on: * **Number of ODFVs** in the request — each one adds its transformation time. * **Transformation complexity** — a simple arithmetic operation is microseconds; ML model inference can be milliseconds. * **Transformation mode** — `mode="python"` is 3–10× faster than `mode="pandas"` for online serving (see [ODFV optimization](#on-demand-feature-view-odfv-optimization)). * **Entity count** — transformations run across all entity rows in the request. More entities = more work. ### Hidden store reads from ODFV source dependencies ODFVs declare source feature views. When you request features from an ODFV, the server must also read **all source feature views** that the ODFV depends on, even if you didn't explicitly request features from those views. This can add unexpected store reads: ```python @on_demand_feature_view( sources=[driver_stats_fv, vehicle_stats_fv, request_source], schema=[Field(name="combined_score", dtype=Float64)], mode="python", ) def combined_score(inputs: dict[str, Any]) -> dict[str, Any]: return {"combined_score": [d + v for d, v in zip(inputs["driver_rating"], inputs["vehicle_rating"])]} ``` Requesting just `combined_score` triggers reads from **both** `driver_stats_fv` and `vehicle_stats_fv`. If those are already needed by other features in the same request, there's no extra cost (the server deduplicates). But if they're only needed by the ODFV, these are hidden reads you should account for. ### Structuring recommendations | Practice | Why | | ------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | Group co-accessed features into the same feature view | Reduces store round-trips per request | | Use feature services to define stable production feature sets | No performance cost; improves governance and versioning | | Minimize the number of ODFVs per request | Each ODFV adds CPU-bound transformation latency | | Prefer `write_to_online_store=True` for ODFVs that don't need request-time data | Moves compute from serving to materialization path | | Audit ODFV source dependencies | Avoid pulling in unnecessary store reads via unused source feature views | | Use `track_metrics=True` on ODFVs during profiling | Identifies which transforms are the bottleneck | *** ## Pre-computed feature vectors When a `get_online_features()` request touches multiple feature views, the server issues a separate store read per feature view. For services spanning 5–15+ feature views, this fan-out dominates latency — even with Redis pipeline batching, the protobuf deserialization and response-building overhead grows linearly with the number of views. **Pre-computed feature vectors** solve this by storing all of a feature service's features for each entity as a single serialized blob. At read time, the server fetches one blob per entity instead of N reads per feature view, reducing the operation to O(1). ### How it works 1. **Define** a feature service with `precompute_online=True`: ```python benchmark_service = FeatureService( name="benchmark_customer_service", features=[ customer_demographics_fv, customer_behavioral_profile, transaction_7d_aggregations, transaction_30d_aggregations, transaction_90d_patterns, atm_usage_30d, ], precompute_online=True, ) ``` 2. **Apply** the feature service: `feast apply` 3. **Materialize** as usual — vectors are built automatically: `feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S")` 4. **Read** features as usual — the server automatically uses the pre-computed path: ```python features = store.get_online_features( features=store.get_feature_service("benchmark_customer_service"), entity_rows=[{"customer_id": "CUST_000001"}], full_feature_names=True, ) ``` *** ## Worker and connection tuning The Python feature server uses Gunicorn with async workers. Tuning workers, connections, and timeouts directly impacts throughput and tail latency. ### Feast Operator CR configuration ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: my-feast spec: feastProject: my_project services: onlineStore: server: workerConfigs: workers: 4 workerConnections: 2000 maxRequests: 5000 maxRequestsJitter: 500 keepAliveTimeout: 30 registryTTLSeconds: 300 resources: requests: cpu: "2" memory: 2Gi limits: cpu: "4" memory: 4Gi ``` ### CLI equivalent ```bash feast serve \ --workers 4 \ --worker-connections 2000 \ --max-requests 5000 \ --max-requests-jitter 500 \ --keep-alive-timeout 30 \ --registry_ttl_sec 300 ``` ### Tuning recommendations | Parameter | Default | Recommended starting point | Notes | | -------------------- | ------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | `workers` | 1 | `2 × CPU cores + 1` (use `-1` for auto) | Each worker is a separate process. More workers = more parallel requests, but also more memory. | | `workerConnections` | 1000 | 1000–2000 | Max simultaneous connections **per worker**. Increase for high-concurrency workloads behind a load balancer. | | `maxRequests` | 1000 | 5000–10000 | Recycles a worker after N requests to guard against memory leaks. Higher values reduce recycle overhead. | | `maxRequestsJitter` | 50 | 200–500 | Randomizes recycle timing to avoid a thundering herd of simultaneous worker restarts. | | `keepAliveTimeout` | 30 | 30–60 | Seconds to keep an idle connection open. Match this to your load balancer's idle timeout. | | `registryTTLSeconds` | 60 | 300 | How often the registry cache is refreshed. See [Registry cache tuning](#registry-cache-tuning) below. | {% hint style="info" %} **Sizing rule of thumb:** Start with `workers = 2 × vCPU + 1` and allocate \~256–512 MB memory per worker. Monitor RSS per process and adjust. If p99 latency is high under load, add more workers or scale horizontally rather than increasing connections. {% endhint %} *** ## Registry cache tuning The default registry configuration uses **synchronous** cache refresh. When the TTL expires, the **next** `get_online_features()` call blocks while the full registry is re-downloaded. For large registries this can add tens of milliseconds to a single request. ### The problem With the default `cache_mode: sync` (or unset), the refresh cycle looks like: ``` Request N → cache hit (fast) ...TTL expires... Request N+1 → cache miss → synchronous download → response delayed Request N+2 → cache hit (fast) ``` ### The fix: background thread refresh Configure the registry to refresh in a background thread so that no serving request ever blocks on a download: {% code title="feature\_store.yaml (inside the Operator secret)" %} ```yaml registry: registry_type: sql path: postgresql://:@:5432/feast cache_mode: thread cache_ttl_seconds: 300 ``` {% endcode %} With `cache_mode: thread`: * The cache is populated **eagerly** at startup. * A background thread refreshes every `cache_ttl_seconds`. * Serving requests always read from the **in-memory** cache and are never blocked. When deploying with the Feast Operator using a **file-based** registry, set these fields directly on the CR: ```yaml spec: services: registry: local: persistence: file: cache_mode: thread cache_ttl_seconds: 300 ``` {% hint style="warning" %} For DB/SQL-backed registries deployed via the Operator, `cache_mode` and `cache_ttl_seconds` are set inside the secret's YAML payload rather than on the CR struct. Ensure your secret includes these keys. {% endhint %} ### The trade-off: freshness vs. performance Registry caching is a **freshness vs. performance** trade-off. The registry contains metadata — feature view definitions, entity schemas, data source configs — not the feature values themselves. When someone runs `feast apply` to add or modify a feature view, the change is invisible to the serving pods until the cache refreshes. **Lower TTL** (e.g., 10–30s): * Schema changes propagate faster to serving pods. * But each refresh re-downloads and deserializes the entire registry, consuming CPU and network bandwidth. * With `cache_mode: sync`, a lower TTL means **more frequent latency spikes** — the unlucky request that triggers the refresh is blocked for the full download duration. * With `cache_mode: thread`, no request is directly blocked, but the background thread competes with Gunicorn workers for CPU during deserialization. On CPU-constrained pods, frequent refreshes can **indirectly increase p99 serving latency** by starving workers of CPU cycles. **Higher TTL** (e.g., 300–600s): * Fewer refreshes mean less CPU contention, resulting in **more stable and predictable serving latency**. * But schema changes take longer to propagate. A new feature view added via `feast apply` won't be queryable until the cache refreshes (up to `cache_ttl_seconds` delay). * This is usually acceptable in production where schema changes are infrequent and deployed through CI/CD pipelines, not ad-hoc. **Memory impact:** The full registry is deserialized into memory on each worker process. For registries with hundreds of feature views, this can be tens of megabytes per worker. With 4 workers × 5 replicas, that's 20 copies of the registry in memory. A higher TTL doesn't reduce memory usage (the cache is always held), but it reduces the CPU cost of periodic deserialization — which in turn keeps CPU available for serving requests with lower latency. {% hint style="info" %} The `registryTTLSeconds` field on the Operator CR (or `--registry_ttl_sec` CLI flag) controls how often the **online server** refreshes its cache. The `cache_ttl_seconds` in `feature_store.yaml` controls how often the **SDK client** refreshes. In an Operator deployment, the CR field is what matters for server performance. {% endhint %} ### Recommendations | Scenario | `cache_mode` | `cache_ttl_seconds` | | ------------------------------------ | ---------------- | ------------------- | | Development / iteration | `sync` (default) | 5–10 | | Production (low-latency) | `thread` | 300 | | Production (frequent schema changes) | `thread` | 60 | *** ## Online store selection The online store is the single largest factor in `get_online_features()` latency. Choose based on your latency budget, throughput needs, and existing infrastructure. | Store | Typical p50 latency | Async read | Best for | Key trade-off | | ------------------------ | ------------------- | --------------- | ------------------------------------------------------------------------ | -------------------------------------------------- | | **Redis / Dragonfly** | < 1 ms | No (threadpool) | Ultra-low latency, high throughput; all FV reads batched into 1 pipeline | Requires in-memory capacity for your dataset | | **DynamoDB** | 2–5 ms | Yes | Serverless, auto-scaling on AWS | Pay-per-request cost; batch API limits (100 items) | | **PostgreSQL** | 3–10 ms | No (threadpool) | Teams with existing Postgres infra | Connection pooling needed at scale | | **MongoDB** | 2–5 ms | Yes | Flexible schema, async-native | Requires index tuning for large datasets | | **Aerospike** | < 1 ms | No (threadpool) | Ultra-low latency, hybrid memory (RAM + SSD), large datasets | Namespace must be pre-configured on the cluster | | **Bigtable** | 3–8 ms | No (threadpool) | Large-scale GCP workloads | Row-key design affects read performance | | **Cassandra / ScyllaDB** | 2–5 ms | No (threadpool) | Multi-region, write-heavy | Tunable consistency; requires DC-aware routing | | **Remote** | Varies | No (threadpool) | Centralized feature server architecture | Adds an HTTP hop; tune connection pool | | **SQLite** | N/A | No | Local development only | No concurrent access; not suitable for production | {% hint style="info" %} **General rule:** If your p99 latency budget is under 10 ms, use Redis. If you need serverless scaling on AWS, use DynamoDB. For everything else, PostgreSQL with connection pooling is a strong default. {% endhint %} ### Async vs sync online reads The feature server can read from the online store using either an **async** or **sync** code path. The choice is made automatically based on the store's `async_supported` property — no user configuration is needed. **How it works:** * When a `get_online_features()` request touches multiple feature views, the server must read from each one. * **Async path** (`async_supported.online.read = True`): All feature view reads run concurrently via `asyncio.gather()` in a single event loop — no thread overhead, minimal context switching. * **Sync path** (default fallback): The sync `online_read()` is wrapped in `run_in_threadpool()`, which dispatches to a thread pool. This works correctly but adds thread scheduling overhead, especially when reading from many feature views. **Current async support by store:** | Store | Async read | Async write | Notes | | -------------- | ----------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **DynamoDB** | Yes | Yes | Uses `aiobotocore` for non-blocking I/O | | **MongoDB** | Yes | Yes | Uses `motor` (async MongoDB driver) | | **PostgreSQL** | Implemented | No | Has `online_read_async` but does not yet advertise via `async_supported`; uses sync/threadpool path | | **Redis** | Implemented | **Yes** | `online_read_async` and `online_write_batch_async` both implemented; uses sync/threadpool path for `get_online_features` (overridden with batched single pipeline) | | **Aerospike** | Implemented | No | Async methods wrap the blocking C client via `run_in_executor`; does not yet advertise via `async_supported`, so the server still uses the threadpool path | | All others | No | No | Fall back to sync with `run_in_threadpool()` | **When async matters most:** * Requests that fan out across **many feature views** benefit the most — async gathers all reads concurrently without thread pool contention. * For requests hitting a **single feature view**, the difference is negligible. * If you are choosing between two stores with comparable latency, prefer the one with full async support for better throughput under concurrent load. ### DynamoDB tuning ```yaml online_store: type: dynamodb region: us-west-2 batch_size: 100 max_read_workers: 10 consistent_reads: false warmup_connections: true max_pool_connections: 100 keepalive_timeout: 30.0 connect_timeout: 3 read_timeout: 5 total_max_retry_attempts: 2 retry_mode: adaptive ``` Key knobs: * **`batch_size`**: DynamoDB's `BatchGetItem` accepts up to 100 items per request. For 500 entities, this means 5 batches. Keep at 100 unless hitting the 16 MB response limit. * **`max_read_workers`**: Controls parallelism for batch reads. With 10 workers, those 5 batches run concurrently (\~10 ms) instead of sequentially (\~50 ms). * **`consistent_reads: false`**: Eventually consistent reads are faster and cheaper. Use `true` only if you need read-after-write consistency. * **`warmup_connections: true`**: Pre-warms the DynamoDB connection pool on server startup by making a lightweight call (`describe_limits`). This avoids a cold-start latency penalty (\~20ms) on the very first feature request. * **`max_pool_connections`**: Increase for high-throughput deployments to improve HTTP connection reuse to the DynamoDB endpoint. * **`keepalive_timeout`**: Longer keep-alive reduces TLS handshake overhead on reused connections. * **`connect_timeout` / `read_timeout`**: Lower values fail fast, improving p99. Set aggressively if your retry strategy covers transient failures. * **`total_max_retry_attempts: 2`**: Reduces p99 by capping retry delay. Combine with `retry_mode: adaptive` for backpressure-aware retries. {% hint style="info" %} **VPC Endpoints:** Configure a [VPC endpoint for DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/vpc-endpoints-dynamodb.html) to keep traffic on AWS's private network. This reduces latency by eliminating the public internet hop and can also reduce data transfer costs. {% endhint %} ### PostgreSQL tuning ```yaml online_store: type: postgres host: port: 5432 database: feast db_schema: public user: password: conn_type: pool min_conn: 4 max_conn: 20 keepalives_idle: 30 sslmode: require ``` * **`conn_type: pool`**: Enables connection pooling via `psycopg` pool. Without this, every request opens and closes a TCP connection. * **`min_conn` / `max_conn`**: Set `min_conn` to your expected steady-state concurrency and `max_conn` to your burst limit. Keep `max_conn` below the PostgreSQL `max_connections` divided by your replica count. * **`keepalives_idle`**: TCP keep-alive prevents idle connections from being dropped by firewalls or load balancers. ### Redis tuning ```yaml online_store: type: redis connection_string: "redis-cluster.internal:6379,ssl=true" redis_type: redis_cluster key_ttl_seconds: 604800 skip_dedup: false # set true for initial bulk loads to halve write round trips ``` * Use `redis_cluster` for horizontal partitioning across shards. * Set `key_ttl_seconds` to auto-expire stale feature data, keeping memory usage bounded. This is a **key-level** TTL — it expires the entire entity hash (all feature views for that entity) together. * Ensure the Redis instance is in the **same availability zone** as the feature server pods to minimize network round-trips. #### Batched multi-feature-view reads The Redis online store overrides `get_online_features()` to issue all `HMGET` commands — across every feature view in the request — in a **single pipeline execution**. This reduces Redis round trips from `N` (one per feature view) to `1` regardless of request size. | Feature views | Round trips (other stores) | Round trips (Redis) | | :-----------: | :------------------------: | :-----------------: | | 1 | 1 | 1 | | 5 | 5 | **1** | | 10 | 10 | **1** | | 20 | 20 | **1** | This means the latency cost of adding feature views to a Redis-backed request is primarily **serialization and protobuf overhead** at the application layer, not Redis network latency. #### Write throughput: `skip_dedup` For initial bulk loads or append-only materialization pipelines: ```yaml online_store: type: redis connection_string: "localhost:6379" skip_dedup: true ``` With `skip_dedup: true`, `online_write_batch()` skips the existing-timestamp read pipeline and writes all values directly in a single pass, halving write round trips. Use only when you can guarantee write ordering — under concurrent writers, an older record can overwrite a newer one. ### Cassandra / ScyllaDB tuning Cassandra and ScyllaDB share the same Feast connector. The key read-path knobs are concurrency and data-center-aware routing: ```yaml online_store: type: cassandra hosts: - cassandra-0.internal - cassandra-1.internal keyspace: feast read_concurrency: 100 write_concurrency: 100 request_timeout: 10 load_balancing: load_balancing_policy: DCAwareRoundRobinPolicy local_dc: datacenter1 ``` * **`read_concurrency`** (default: 100): Maximum concurrent async read operations. Increase for high fan-out workloads; decrease if the cluster shows read-timeout pressure. * **`write_concurrency`** (default: 100): Maximum concurrent async write operations during materialization. * **`request_timeout`** (default: driver default): Per-request timeout in seconds. Lower values fail fast and improve p99 at the cost of higher error rates under load. * **`load_balancing.local_dc`**: Set to your local data center name so the driver routes reads to the nearest replicas, avoiding cross-DC latency. ### Bigtable tuning Bigtable's client library uses an internal connection pool that caps concurrency: ```yaml online_store: type: bigtable project_id: my-gcp-project instance: feast-instance ``` * The client library's **connection pool size** is hardcoded at 10 internally. For workloads that need higher concurrent reads, scale horizontally (more feature server replicas) rather than trying to push more concurrency through a single pod. * **`max_versions`** (default: 2): Number of cell versions retained. Lower values reduce storage and read amplification. * Co-locate the feature server in the **same GCP region** as your Bigtable instance. ### MongoDB tuning MongoDB supports async reads natively. Use `client_kwargs` to pass performance options directly to the PyMongo / Motor driver: ```yaml online_store: type: mongodb connection_string: "mongodb://mongo.internal:27017" database_name: feast client_kwargs: maxPoolSize: 100 minPoolSize: 10 maxIdleTimeMS: 30000 connectTimeoutMS: 3000 socketTimeoutMS: 5000 ``` * **`maxPoolSize` / `minPoolSize`**: Controls the driver connection pool. Default `maxPoolSize` is 100 in PyMongo, but explicitly setting it ensures predictability across versions. * **`connectTimeoutMS` / `socketTimeoutMS`**: Tighter timeouts improve p99 by failing fast on slow connections. * MongoDB is one of the stores with **full async support** (read and write), so it benefits from concurrent feature view reads via `asyncio.gather()`. ### Aerospike tuning Aerospike offers sub-millisecond reads thanks to its hybrid-memory architecture (primary index in RAM, data on SSD or RAM). Tune the per-call policies in the Feast config and rely on the Aerospike cluster's own tuning for everything else: ```yaml online_store: type: aerospike hosts: - ["aerospike-1.internal", 3000] - ["aerospike-2.internal", 3000] namespace: feast read_timeout_ms: 150 # hard deadline for a single-record get write_timeout_ms: 300 # hard deadline for a single-record put/operate batch_total_timeout_ms: 500 # hard deadline for online_read / online_write_batch socket_timeout_ms: 50 # per-attempt deadline so max_retries can actually fire max_retries: 2 ttl_seconds: 86400 # record-level TTL; omit to use the namespace default client_kwargs: # escape hatch for any client-config field not surfaced above policies: batch: concurrent_nodes: 0 # 0 = parallel to every node (lowest latency on multi-node clusters) ``` * **`*_timeout_ms` (total)** vs **`socket_timeout_ms` (per-attempt)**: `*_timeout_ms` is the hard deadline for a whole call *including* retries; `socket_timeout_ms` is the per-attempt deadline that allows `max_retries` to actually fire within that budget. Without `socket_timeout_ms`, a single slow attempt can consume the entire total deadline and retries never run. * **`hosts`**: List every seed node. The Aerospike client discovers the rest of the cluster automatically and opens one connection pool per node. * **`ttl_seconds: 0`** means "never expire"; omit the key to inherit the namespace's `default-ttl`. Expiry is enforced by the server's `nsup` thread — nothing to delete on the client side. * Co-locate the feature server in the **same availability zone / rack** as the Aerospike cluster; sub-millisecond reads are bandwidth- and RTT-sensitive. ### Remote online store tuning The Remote online store connects to a Feast feature server over HTTP. Connection pooling is critical: ```yaml online_store: type: remote path: https://feast-server.internal:6566 connection_pool_size: 50 connection_idle_timeout: 300 connection_retries: 3 ``` * **`connection_pool_size`** (default: 50): Number of persistent HTTP connections. Increase for high-throughput deployments to reduce connection setup overhead. * **`connection_idle_timeout`** (default: 300s): Seconds before idle connections are closed. Set to 0 to disable idle cleanup (useful when traffic is bursty). * **`connection_retries`** (default: 3): Number of retries on transient HTTP failures. Lower for tighter p99; higher for better reliability. *** ## Batch size tuning Different online stores have different optimal batch sizes for `get_online_features()` calls. The batch size controls how many entity keys are fetched per round-trip to the store. | Store | Default batch size | Max batch size | Recommendation | | ---------- | ------------------ | --------------- | ---------------------------------------------------------------------------------------------------- | | DynamoDB | 100 | 100 (API limit) | Keep at 100; tune `max_read_workers` for parallelism | | Redis | N/A (pipelined) | N/A | All entity keys **and** all feature views are batched into a single pipeline; no batch tuning needed | | PostgreSQL | N/A (single query) | N/A | Single `SELECT ... WHERE key IN (...)` query; tune connection pool instead | Profile your workload by measuring `feast_feature_server_online_store_read_duration_seconds` (see [Metrics setup](#metrics-setup-prometheus--opentelemetry)) across different entity counts to find the sweet spot. *** ## On-demand feature view (ODFV) optimization If your feature pipeline uses ODFVs, the transformation mode significantly affects serving latency. ### Prefer native Python mode Native Python transformations avoid pandas overhead and are substantially faster for online serving: ```python @on_demand_feature_view( sources=[driver_stats_fv, request_source], schema=[Field(name="trip_rate", dtype=Float64)], mode="python", ) def trip_rate_python(inputs: dict[str, Any]) -> dict[str, Any]: return { "trip_rate": [ trips / max(hours, 1) for trips, hours in zip(inputs["total_trips"], inputs["active_hours"]) ] } ``` | Mode | Relative latency | When to use | | --------------- | ----------------- | ------------------------------------------------------ | | `mode="python"` | **1x** (baseline) | Production serving — minimal overhead | | `mode="pandas"` | **3–10x** | Offline batch retrieval where pandas ergonomics matter | ### Use singleton mode for single-row transforms When your Python ODFV processes one entity at a time (common in online serving), use `singleton=True` to avoid the overhead of wrapping and unwrapping single values in lists: ```python @on_demand_feature_view( sources=[driver_stats_fv, request_source], schema=[Field(name="trip_rate", dtype=Float64)], mode="python", singleton=True, ) def trip_rate_singleton(inputs: dict[str, Any]) -> dict[str, Any]: return {"trip_rate": inputs["total_trips"] / max(inputs["active_hours"], 1)} ``` With `singleton=True`, input values are plain scalars (not lists), and the function returns scalars. This avoids list comprehension and zip overhead for single-entity requests. Singleton mode requires `mode="python"`. ### Write-time transformations For ODFVs that don't depend on request-time data, set `write_to_online_store=True` to compute features during materialization instead of at serving time: ```python @on_demand_feature_view( sources=[driver_stats_fv], schema=[Field(name="is_high_mileage", dtype=Bool)], mode="python", write_to_online_store=True, ) def high_mileage(inputs: dict[str, Any]) -> dict[str, Any]: return {"is_high_mileage": [m > 100000 for m in inputs["total_miles"]]} ``` This moves CPU cost from the serving path to the materialization path, reducing online latency. ### Consistency vs. latency: choosing between read-time and write-time transforms Read-time ODFVs (the default, `write_to_online_store=False`) are **request-blocking** — they execute synchronously during `get_online_features()`. This provides a **strong consistency guarantee**: the output is always computed from the latest source features in the online store and can incorporate request-time data (e.g., the user's current location or the current timestamp). Write-time ODFVs (`write_to_online_store=True`) are pre-computed during materialization and stored. They add zero latency to the serving path, but the result can be **stale** — it reflects the source feature values at the time of the last materialization run, not the current request. Use this decision framework: | Condition | Recommendation | | ------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- | | ODFV uses **request-time data** (e.g., current timestamp, user input) | Must use read-time (default) — request data is not available at materialization | | ODFV depends on features that change **frequently** and freshness matters | Keep read-time for strong consistency | | ODFV is a **pure derivation** of slowly changing features (e.g., age buckets, tier labels) | Safe to move to write-time for lower latency | | ODFV is **computationally expensive** (ML inference, complex aggregations) | Prefer write-time if staleness is acceptable; otherwise use read-time with [static artifacts](#pre-load-heavy-resources-with-static-artifacts) | {% hint style="warning" %} Moving an ODFV to `write_to_online_store=True` is a **performance-consistency trade-off**, not a free optimization. Verify that your use case tolerates stale results bounded by the materialization interval before switching. {% endhint %} ### Pre-load heavy resources with static artifacts If your ODFV uses ML models, lookup tables, or other expensive resources, load them **once** at server startup using [static artifacts loading](/master/reference/alpha-static-artifacts) instead of loading them per request: ```python # static_artifacts.py — loaded once at feature server startup from fastapi import FastAPI def load_artifacts(app: FastAPI): from transformers import pipeline app.state.model = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english") import example_repo example_repo._model = app.state.model ``` ```python # example_repo.py — use pre-loaded artifact in ODFV _model = None @on_demand_feature_view( sources=[text_fv], schema=[Field(name="sentiment_score", dtype=Float64)], mode="python", ) def sentiment(inputs: dict[str, Any]) -> dict[str, Any]: global _model scores = [_model(t)[0]["score"] for t in inputs["text"]] return {"sentiment_score": scores} ``` Without static artifacts, the model would be loaded on every request (or every worker restart), adding seconds of latency. With pre-loading, the ODFV pays only the inference cost. **Using static artifacts with the Feast Operator:** The feature server looks for `static_artifacts.py` in the feature repository directory at startup. Whether this works in an operator deployment depends on how the repo is created: * **`feastProjectDir.git`** — The operator clones your git repository via an init container. Include `static_artifacts.py` in the repo alongside your feature definitions and it will be available at startup. This is the recommended approach for production. ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: my-feast spec: feastProject: my_project feastProjectDir: git: url: https://github.com/my-org/feast-repo.git ref: main ``` * **`feastProjectDir.init`** (default) — The operator runs `feast init` to create a template repo. The generated template does not include `static_artifacts.py`. To use static artifacts in this mode, you would need to build a custom feature server image that bundles the file, or mount it via a ConfigMap/volume. {% hint style="info" %} For production deployments that use ODFVs with heavy resources (ML models, large lookup tables), use `feastProjectDir.git` so your `static_artifacts.py` is version-controlled and deployed automatically alongside your feature definitions. {% endhint %} ### Transformation code best practices Small choices inside transformation functions add up at scale: * **Avoid I/O in the transform function.** Network calls, file reads, or database queries inside an ODFV block the serving thread. Pre-load data via static artifacts or use `write_to_online_store` to compute offline. * **Avoid creating DataFrames.** Even in Python mode, constructing a pandas DataFrame inside the function defeats the purpose of avoiding pandas overhead. * **Minimize allocations.** Reuse structures where possible. List comprehensions are faster than building lists with `.append()` in a loop. * **Profile with `track_metrics=True`.** Enable per-ODFV timing to identify slow transforms, but disable it in production once you've tuned — the timer itself adds minor overhead. ```python @on_demand_feature_view( sources=[driver_stats_fv], schema=[Field(name="score", dtype=Float64)], mode="python", track_metrics=True, # enable during profiling, disable once tuned ) def compute_score(inputs: dict[str, Any]) -> dict[str, Any]: return {"score": [x * 0.5 + y * 0.3 for x, y in zip(inputs["a"], inputs["b"])]} ``` The `feast_feature_server_transformation_duration_seconds` metric (labeled by `odfv_name` and `mode`) shows exactly how much time each ODFV adds to the serving path. *** ## Horizontal scaling When a single pod cannot meet your throughput or latency requirements, scale horizontally using the Feast Operator. Horizontal scaling requires **DB-backed persistence** for all services. File-based stores (SQLite, DuckDB, `registry.db`) do not support concurrent access from multiple pods. The Feast Operator supports both static replicas (`spec.replicas`) and HPA autoscaling (`services.scaling.autoscaling`). See [Scaling Feast](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast) for full configuration examples. ### Expected scaling efficiency | Replicas | Expected throughput | Notes | | -------- | ------------------- | -------------------------------------------------------------------------- | | 1 | Baseline | Single-pod throughput depends on worker count and online store latency | | 2–3 | \~1.8–2.8x | Near-linear scaling; overhead from load balancer routing | | 5–10 | \~4.5–9x | Online store connection pressure becomes the bottleneck; tune pool sizes | | 10+ | Diminishing returns | Online store or network bandwidth may saturate; monitor store-side metrics | {% hint style="info" %} Scaling the feature server is only effective if the online store can handle the aggregate connection and throughput load. Monitor the store's CPU, memory, and connection counts alongside feature server metrics. {% endhint %} ### Coordinating database connections with scaling When you scale horizontally, every replica and every Gunicorn worker within a replica opens its own database connection pool to the online store. The **total connection count** multiplies quickly: ``` Total connections = replicas × workers_per_pod × max_conn_per_worker ``` **Example:** 5 replicas × 4 workers × 20 `max_conn` = **400 connections** to the database. If your PostgreSQL instance has the default `max_connections: 100`, this will fail immediately. This applies to every connection-oriented online store: | Store | Connection setting | Default | What to check on the store side | | -------------- | ---------------------------------- | ------- | ------------------------------------------------------------------------ | | **PostgreSQL** | `max_conn` (per worker pool) | 10 | `max_connections` on the server (typically 100–500) | | **DynamoDB** | `max_pool_connections` (HTTP pool) | 10 | No hard limit, but AWS SDK has per-process pool caps; monitor throttling | | **Redis** | Connection per worker | 1 | `maxclients` on the Redis server (default: 10,000) | | **MongoDB** | `maxPoolSize` (in `client_kwargs`) | 100 | Server's `net.maxIncomingConnections` | | **Aerospike** | Driver manages pool per seed node | Auto | `proto-fd-max` (default 15000) on each Aerospike node | | **Cassandra** | Driver manages pool per node | Auto | `native_transport_max_threads` on each Cassandra node | | **Remote** | `connection_pool_size` (HTTP pool) | 50 | The target feature server's worker capacity | **How to size it:** 1. **Start from the store's limit.** Determine the maximum connections your database can handle (e.g., PostgreSQL `max_connections`, or RDS instance class limit). 2. **Reserve headroom** for other clients (materialization jobs, `feast apply`, monitoring). A common rule is to reserve 20–30% of the store's capacity. 3. **Divide by total workers.** Set `max_conn` (or equivalent) per worker so the total stays within the budget: ``` max_conn_per_worker = (store_max_connections × 0.7) / (replicas × workers_per_pod) ``` **Example calculation for PostgreSQL:** | Parameter | Value | | -------------------------------- | --------------------- | | PostgreSQL `max_connections` | 200 | | Reserved for other clients (30%) | 60 | | Available for feature server | 140 | | Replicas | 5 | | Workers per pod | 4 | | **`max_conn` per worker** | **140 / (5 × 4) = 7** | ```yaml # feature_store.yaml (in the Operator secret) online_store: type: postgres conn_type: pool min_conn: 2 max_conn: 7 ... ``` {% hint style="warning" %} If you use HPA autoscaling, size `max_conn` based on `maxReplicas` (not `minReplicas`) to avoid connection exhaustion during scale-out. Under-provisioning here causes cascading failures: the store rejects connections, requests fail, load increases, HPA scales up further, and even more connections are rejected. {% endhint %} **For DynamoDB and other HTTP-based stores**, the concern is less about hard connection limits and more about SDK-level pool exhaustion and throttling. Monitor `ThrottledRequests` (DynamoDB) or HTTP `429` responses and increase pool sizes or store capacity accordingly. ### High availability When scaling is enabled, the operator automatically injects: * **Soft pod anti-affinity** — prefers spreading pods across nodes. * **Zone topology spread** — distributes pods across availability zones (best-effort). You can also configure a **PodDisruptionBudget** to limit voluntary disruptions: ```yaml spec: services: podDisruptionBudgets: maxUnavailable: 1 ``` See [Scaling Feast](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast) for the full horizontal scaling reference, including KEDA integration. *** ## Metrics setup: Prometheus + OpenTelemetry Observability is essential for tuning. Without metrics, you're guessing. ### Prometheus (built-in) Enable the built-in Prometheus metrics endpoint in the Feast Operator CR: ```yaml spec: services: onlineStore: server: metrics: true ``` This exposes a `/metrics` endpoint on port 8000. If the Prometheus Operator's `ServiceMonitor` CRD is installed, the Feast operator automatically creates a ServiceMonitor for scraping. #### Key metrics for performance tuning | Metric | What to watch | | --------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | | `feast_feature_server_request_latency_seconds` | p50, p95, p99 latency by endpoint. The primary SLI. | | `feast_feature_server_online_store_read_duration_seconds` | Time spent in the online store. If this is close to total latency, tune the store, not the server. | | `feast_feature_server_transformation_duration_seconds` | ODFV execution time (requires `track_metrics=True` on the ODFV). | | `feast_feature_server_cpu_usage` | Per-worker CPU. Sustained > 80% means you need more workers or replicas. | | `feast_feature_server_memory_usage` | Per-worker memory. Growing over time may indicate a memory leak — use `maxRequests` to recycle. | | `feast_feature_server_request_total` | Request throughput and error rates by endpoint and status. | | `feast_feature_freshness_seconds` | Seconds since last materialization per feature view. Alerts you to stale data. | #### Example Prometheus alert rules ```yaml groups: - name: feast-online-server rules: - alert: FeastHighP99Latency expr: histogram_quantile(0.99, rate(feast_feature_server_request_latency_seconds_bucket[5m])) > 0.1 for: 5m labels: severity: warning annotations: summary: "Feast online server p99 latency exceeds 100ms" - alert: FeastStaleFeatures expr: feast_feature_freshness_seconds > 3600 for: 10m labels: severity: critical annotations: summary: "Feature view {{ $labels.feature_view }} has not been materialized in over 1 hour" ``` ### OpenTelemetry For deeper tracing and integration with your existing observability stack, add OpenTelemetry instrumentation. **1. Install the OpenTelemetry Operator** (requires `cert-manager`): ```bash kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml ``` **2. Deploy an OpenTelemetry Collector:** ```yaml apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: feast-otel spec: mode: sidecar config: receivers: otlp: protocols: grpc: {} http: {} processors: batch: {} exporters: prometheus: endpoint: "0.0.0.0:8889" service: pipelines: metrics: receivers: [otlp] processors: [batch] exporters: [prometheus] ``` **3. Add instrumentation** to the FeatureStore pods via the Operator's `env` field: ```yaml spec: services: onlineStore: server: env: - name: OTEL_EXPORTER_OTLP_ENDPOINT value: "http://localhost:4317" - name: OTEL_SERVICE_NAME value: "feast-online-server" ``` See the [OpenTelemetry Integration](/master/getting-started/components/open-telemetry) guide for the complete setup including Instrumentation CRDs and ServiceMonitors. *** ## Network latency optimization Network round-trip time to the online store is the **dominant factor** in `get_online_features()` latency. No amount of server-side tuning can compensate for a slow network path. ### Co-locate feature server and online store * Deploy the feature server pods in the **same AWS region and availability zone** as your online store (DynamoDB, ElastiCache, RDS, etc.). * On Kubernetes or similar platforms, ensure the cluster and the managed database service share the same VPC and subnet. ### Use private network endpoints Public internet round-trips add 5–50 ms of latency compared to private network paths. Use VPC endpoints or private link: | Cloud | Service | Private endpoint | | ----- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | AWS | DynamoDB | [Gateway VPC Endpoint](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/vpc-endpoints-dynamodb.html) | | AWS | ElastiCache (Redis) | Deploy in same VPC; no public endpoint needed | | AWS | RDS (PostgreSQL) | Enable [RDS Private Link](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/vpc-interface-endpoints.html) or same-VPC access | | GCP | Bigtable / Cloud SQL | [Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect) | | Azure | Cosmos DB / Azure Cache | [Private Endpoints](https://learn.microsoft.com/en-us/azure/private-link/private-endpoint-overview) | ### DNS resolution caching Frequent DNS lookups to the store endpoint add latency. In Kubernetes, ensure CoreDNS caching is enabled (default) and consider using IP-based endpoints for critical-path connections. *** ## Client access patterns: REST API vs. SDK with remote store When consuming features from a Feast online server deployed via the Operator, there are three client access patterns with different performance profiles: ### Pattern comparison | Pattern | How it works | Network hops | Client-side overhead | Best for | | ------------------------------- | ----------------------------------------------------------------------- | -------------------------------------- | ----------------------------------------------- | ----------------------------------------------------------------------------- | | **Direct REST API** | HTTP POST to `/get-online-features` from any language | Client → feature server → online store | Minimal (JSON only) | Non-Python clients, lowest client-side latency, microservices in any language | | **Feast SDK with remote store** | `store.get_online_features()` over HTTP via `online_store.type: remote` | Client → feature server → online store | SDK registry resolution + proto/JSON conversion | Python clients that want the SDK API with centralized serving | | **Feast SDK direct** | `store.get_online_features()` directly to the store (no feature server) | Client → online store | SDK registry resolution + store driver | Python clients with direct network access to the store | ### Performance characteristics **Direct REST API** has the lowest client-side overhead. The client sends a JSON payload and receives a JSON response — no SDK, no registry loading, no protobuf conversion. The feature server handles all processing (registry lookup, store read, ODFVs). This is the best option when: * Your inference service is written in Java, Go, C++, or another non-Python language. * You want to minimize client-side CPU and memory usage. * You're already behind a load balancer and want the simplest possible integration. ```bash curl -X POST "http://feast-server:6566/get-online-features" \ -H "Content-Type: application/json" \ -d '{ "features": ["driver_stats:conv_rate", "driver_stats:acc_rate"], "entities": {"driver_id": [1001, 1002]} }' ``` **Feast SDK with remote store** (`online_store.type: remote`) adds client-side overhead: the SDK loads the registry locally to resolve feature references, converts entities to protobuf, serializes to JSON for HTTP transport, and deserializes the response back to protobuf. This means: * The client needs access to the registry (or a remote registry server). * Each request pays for proto → JSON → proto round-trip conversion. * But you get the full SDK API: feature services, type checking, Python-native `OnlineResponse` objects, and built-in connection pooling. ```python store = FeatureStore(repo_path=".") # Internally: registry lookup → proto conversion → HTTP POST → JSON parse → proto conversion features = store.get_online_features( features=["driver_stats:conv_rate", "driver_stats:acc_rate"], entity_rows=[{"driver_id": 1001}, {"driver_id": 1002}], ) ``` **Feast SDK direct** (`online_store.type: postgres/redis/dynamodb/...`) skips the feature server entirely and reads from the online store directly. This eliminates the HTTP hop and JSON serialization, giving the **lowest end-to-end latency** for Python clients. However, it requires: * Direct network access from the client to the online store (often not available in production Kubernetes setups). * Online store credentials on the client side. * Each client manages its own connection pool to the store, which complicates connection budgeting at scale. ### Recommendation for Operator deployments In a typical Feast Operator deployment on Kubernetes: * **Use the direct REST API** for production inference services, especially non-Python ones. It has the least overhead and works with any language. * **Use the SDK with remote store** for Python services that benefit from the SDK ergonomics (feature services, type safety, `OnlineResponse` helpers). * **Avoid SDK direct** in most Operator deployments — the feature server already manages store connections, worker pools, and registry caching. Bypassing it means every client pod opens its own connections to the database, duplicating the connection budget problem. {% hint style="info" %} Regardless of which pattern you use, the **server-side** performance is identical — the feature server processes `/get-online-features` the same way. The performance difference is entirely on the **client side**: how much work the client does before and after the HTTP call. {% endhint %} *** ## Putting it all together Here is a complete Feast Operator CR for a production deployment with all the tuning recommendations applied: ```yaml apiVersion: feast.dev/v1 kind: FeatureStore metadata: name: production-feast spec: feastProject: my_project services: scaling: autoscaling: minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 podDisruptionBudgets: maxUnavailable: 1 onlineStore: server: metrics: true workerConfigs: workers: 4 workerConnections: 2000 maxRequests: 5000 maxRequestsJitter: 500 keepAliveTimeout: 30 registryTTLSeconds: 300 resources: requests: cpu: "2" memory: 2Gi limits: cpu: "4" memory: 4Gi persistence: store: type: postgres secretRef: name: feast-online-store registry: local: persistence: store: type: sql secretRef: name: feast-registry ``` With the online store secret configured for connection pooling and background registry refresh: {% code title="feast-online-store secret (feature\_store.yaml content)" %} ```yaml online_store: type: postgres host: port: 5432 database: feast db_schema: public user: password: conn_type: pool min_conn: 2 max_conn: 7 keepalives_idle: 30 sslmode: require ``` {% endcode %} {% hint style="info" %} The `max_conn: 7` above is sized for the HPA configuration in this example (maxReplicas=10, 4 workers per pod). See [Coordinating database connections with scaling](#coordinating-database-connections-with-scaling) for the calculation. Adjust based on your store's connection limit and replica count. {% endhint %} {% code title="feast-registry secret (feature\_store.yaml content)" %} ```yaml registry: registry_type: sql path: postgresql://:@:5432/feast cache_mode: thread cache_ttl_seconds: 300 ``` {% endcode %} *** ## Materialization write performance Materialization (`feast materialize` / `feast materialize-incremental`) reads features from the offline store and writes them to the online store. For large feature views, two common bottlenecks arise: **memory exhaustion** during proto conversion and **write throughput** to the online store. ### Memory: `online_write_batch_size` By default, Feast converts the entire Arrow table returned by the offline store into Python protobuf objects in a single pass before writing to the online store. For datasets with millions of rows this can consume tens of gigabytes of memory on the materialization worker. Set `online_write_batch_size` in `feature_store.yaml` to break the write into manageable chunks: ```yaml materialization: online_write_batch_size: 10000 # rows per write batch ``` Each chunk is independently converted and written, keeping peak memory proportional to the batch size rather than the full dataset. This is supported by the **local, Spark, and Ray** compute engines. | Dataset size | Without batching | With `online_write_batch_size: 10000` | | ------------------------ | ---------------- | ------------------------------------- | | 1 M rows (100 bytes/row) | \~100 MB peak | \~1 MB peak | | 10 M rows | \~1 GB peak | \~1 MB peak | | 100 M rows | OOM / swap | \~1 MB peak | **Choosing a value:** * **Larger batches** (50 000+): fewer write calls to the online store, lower overhead per row — good when worker memory allows. * **Smaller batches** (1 000–5 000): lower peak memory — necessary for memory-constrained workers or very wide feature views (many features per row). * For **Redis**: pipeline overhead per batch is negligible; a batch size of 10 000–50 000 is a good starting point. * For **DynamoDB**: each batch maps to one or more `BatchWriteItem` calls (max 25 items per call); a larger `online_write_batch_size` amortizes the per-call overhead but doesn't change the 25-item DynamoDB limit. See the [feature-store-yaml reference](/master/reference/feature-repository/feature-store-yaml#online_write_batch_size) for the complete option documentation. ### Throughput: parallel feature view materialization Each feature view in a `feast materialize` call is materialized sequentially by the local engine. To materialize multiple feature views in parallel, use a job orchestrator (Airflow, Kubernetes Jobs) and materialize one feature view per job: ```bash # Airflow / cron: one task per feature view feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S") \ --views driver_stats ``` Alternatively, use the **Spark** or **Ray** compute engines which distribute the work across a cluster. ### Redis: combine with `skip_dedup` for bulk reloads When performing a full historical reload into Redis (not an incremental update), combine `online_write_batch_size` with `skip_dedup` for maximum throughput: ```yaml materialization: online_write_batch_size: 50000 # large chunks — memory is bounded online_store: type: redis connection_string: "redis-cluster.internal:6379" skip_dedup: true # skip per-row timestamp check — halves write round trips ``` `skip_dedup: true` eliminates the timestamp-read pipeline before each write (see [Redis tuning](#redis-tuning)), while `online_write_batch_size` prevents the write worker from converting the entire dataset into memory at once. {% hint style="warning" %} Reset `skip_dedup` to `false` (or remove it) after the bulk reload. Under normal incremental materialization, deduplication prevents older feature values from overwriting newer ones. {% endhint %} *** ## Further reading * [Scaling Feast](/master/how-to-guides/feast-snowflake-gcp-aws/scaling-feast) — Horizontal scaling, HPA, KEDA, and HA in detail * [Python Feature Server](/master/reference/feature-servers/python-feature-server) — CLI flags, metrics reference, and API endpoints * [OpenTelemetry Integration](/master/getting-started/components/open-telemetry) — Full OTEL setup with Prometheus Operator * [DynamoDB Online Store](/master/reference/online-stores/dynamodb) — Store-specific configuration and performance tuning * [PostgreSQL Online Store](/master/reference/online-stores/postgres) — Connection pooling and SSL configuration * [Redis Online Store](/master/reference/online-stores/redis) — Cluster mode, Sentinel, TTL configuration, and batched reads * [On Demand Feature Views](/master/reference/beta-on-demand-feature-view) — Transformation modes and write-time transforms * [Feature Services & `precompute_online`](/master/getting-started/concepts/feature-retrieval#pre-computed-feature-vectors-precompute_online) — Concept docs for pre-computed feature vectors * [feature\_store.yaml reference](/master/reference/feature-repository/feature-store-yaml) — Full configuration reference including `materialization` options # Customizing Feast Feast is highly pluggable and configurable: * One can use existing plugins (offline store, online store, batch materialization engine, providers) and configure those using the built in options. See reference documentation for details. * The other way to customize Feast is to build your own custom components, and then point Feast to delegate to them. Below are some guides on how to add new custom components: {% content-ref url="/pages/8VvmP5jorpNWv8gI8phA" %} [Adding a new offline store](/master/how-to-guides/customizing-feast/adding-a-new-offline-store) {% endcontent-ref %} {% content-ref url="/pages/CDpVISp2tBBQviAkMLBF" %} [Adding a new online store](/master/how-to-guides/customizing-feast/adding-support-for-a-new-online-store) {% endcontent-ref %} {% content-ref url="/pages/PSJZ3RJRwnI4EeQUqA77" %} [Adding a custom compute engine](/master/how-to-guides/customizing-feast/creating-a-custom-compute-engine) {% endcontent-ref %} {% content-ref url="/pages/TkYXRLspj5L464RYRzB2" %} [Adding a custom provider](/master/how-to-guides/customizing-feast/creating-a-custom-provider) {% endcontent-ref %} # Adding a custom compute engine ### Overview Feast batch materialization operations (`materialize` and `materialize-incremental`), and get\_historical\_features are executed through a `ComputeEngine`. Custom batch compute engines allow Feast users to extend Feast to customize the materialization and get\_historical\_features process. Examples include: * Setting up custom materialization-specific infrastructure during `feast apply` (e.g. setting up Spark clusters or Lambda Functions) * Launching custom batch ingestion (materialization) jobs (Spark, Beam, AWS Lambda) * Tearing down custom materialization-specific infrastructure during `feast teardown` (e.g. tearing down Spark clusters, or deleting Lambda Functions) Feast comes with built-in materialization engines, e.g, `LocalComputeEngine`, and an experimental `LambdaComputeEngine`. However, users can develop their own compute engines by creating a class that implements the contract in the [ComputeEngine class](https://github.com/feast-dev/feast/blob/85514edbb181df083e6a0d24672c00f0624dcaa3/sdk/python/feast/infra/compute_engines/base.py#L19). ### Guide The fastest way to add custom logic to Feast is to implement the ComputeEngine. The guide that follows will extend the `LocalProvider` with operations that print text to the console. It is up to you as a developer to add your custom code to the engine methods, but the guide below will provide the necessary scaffolding to get you started. #### Step 1: Define an Engine class The first step is to define a custom compute engine class. We've created the `MyCustomEngine` below. This python file can be placed in your `feature_repo` directory if you're following the Quickstart guide. ```python from typing import List, Sequence, Union from feast.entity import Entity from feast.feature_view import FeatureView from feast.batch_feature_view import BatchFeatureView from feast.stream_feature_view import StreamFeatureView from feast.infra.common.retrieval_task import HistoricalRetrievalTask from feast.infra.compute_engines.local.job import LocalMaterializationJob from feast.infra.compute_engines.base import ComputeEngine from feast.infra.common.materialization_job import MaterializationTask from feast.infra.offline_stores.offline_store import OfflineStore, RetrievalJob from feast.infra.online_stores.online_store import OnlineStore from feast.repo_config import RepoConfig class MyCustomEngine(ComputeEngine): def __init__( self, *, repo_config: RepoConfig, offline_store: OfflineStore, online_store: OnlineStore, **kwargs, ): super().__init__( repo_config=repo_config, offline_store=offline_store, online_store=online_store, **kwargs, ) def update( self, project: str, views_to_delete: Sequence[ Union[BatchFeatureView, StreamFeatureView, FeatureView] ], views_to_keep: Sequence[ Union[BatchFeatureView, StreamFeatureView, FeatureView] ], entities_to_delete: Sequence[Entity], entities_to_keep: Sequence[Entity], ): print("Creating new infrastructure is easy here!") pass def materialize( self, registry, tasks: List[MaterializationTask] ) -> List[LocalMaterializationJob]: print("Launching custom batch jobs or multithreading things is pretty easy...") return [ self._materialize_one( registry, task.feature_view, task.start_time, task.end_time, task.project, task.tqdm_builder, ) for task in tasks ] def get_historical_features(self, task: HistoricalRetrievalTask) -> RetrievalJob: raise NotImplementedError ``` Notice how in the above engine we have only overwritten two of the methods on the `LocalComputeEngine`, namely `update` and `materialize`. These two methods are convenient to replace if you are planning to launch custom batch jobs. If you want to use the compute to execute the get\_historical\_features method, you will need to implement the `get_historical_features` method as well. #### Step 2: Configuring Feast to use the engine Configure your [feature\_store.yaml](/master/reference/feature-repository/feature-store-yaml) file to point to your new engine class: ```yaml project: repo registry: registry.db batch_engine: feast_custom_engine.MyCustomEngine online_store: type: sqlite path: online_store.db offline_store: type: file ``` Notice how the `batch_engine` field above points to the module and class where your engine can be found. #### Step 3: Using the engine Now you should be able to use your engine by running a Feast command: ```bash feast apply ``` ``` Registered entity driver_id Registered feature view driver_hourly_stats Deploying infrastructure for driver_hourly_stats Creating new infrastructure is easy here! ``` It may also be necessary to add the module root path to your `PYTHONPATH` as follows: ```bash PYTHONPATH=$PYTHONPATH:/home/my_user/my_custom_engine feast apply ``` That's it. You should now have a fully functional custom engine! # Adding a new offline store ## Overview Feast makes adding support for a new offline store easy. Developers can simply implement the [OfflineStore](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/offline_stores/offline_store.py#L41) interface to add support for a new store (other than the existing stores like Parquet files, Redshift, and Bigquery). In this guide, we will show you how to extend the existing File offline store and use in a feature repo. While we will be implementing a specific store, this guide should be representative for adding support for any new offline store. The full working code for this guide can be found at [feast-dev/feast-custom-offline-store-demo](https://github.com/feast-dev/feast-custom-offline-store-demo). The process for using a custom offline store consists of 8 steps: 1. Defining an `OfflineStore` class. 2. Defining an `OfflineStoreConfig` class. 3. Defining a `RetrievalJob` class for this offline store. 4. Defining a `DataSource` class for the offline store 5. Referencing the `OfflineStore` in a feature repo's `feature_store.yaml` file. 6. Testing the `OfflineStore` class. 7. Updating dependencies. 8. Adding documentation. ## 1. Defining an OfflineStore class {% hint style="info" %} OfflineStore class names must end with the OfflineStore suffix! {% endhint %} ### Contrib offline stores New offline stores go in `sdk/python/feast/infra/offline_stores/contrib/`. #### What is a contrib plugin? * Not guaranteed to implement all interface methods * Not guaranteed to be stable. * Should have warnings for users to indicate this is a contrib plugin that is not maintained by the maintainers. #### How do I make a contrib plugin an "official" plugin? To move an offline store plugin out of contrib, you need: * GitHub actions (i.e `make test-python-integration`) is setup to run all tests against the offline store and pass. * At least two contributors own the plugin (ideally tracked in our `OWNERS` / `CODEOWNERS` file). #### Define the offline store class The OfflineStore class contains a couple of methods to read features from the offline store. Unlike the OnlineStore class, Feast does not manage any infrastructure for the offline store. To fully implement the interface for the offline store, you will need to implement these methods: * `pull_latest_from_table_or_query` is invoked when running materialization (using the `feast materialize` or `feast materialize-incremental` commands, or the corresponding `FeatureStore.materialize()` method. This method pull data from the offline store, and the `FeatureStore` class takes care of writing this data into the online store. * `get_historical_features` is invoked when reading values from the offline store using the `FeatureStore.get_historical_features()` method. Typically, this method is used to retrieve features when training ML models. * (optional) `offline_write_batch` is a method that supports directly pushing a pyarrow table to a feature view. Given a feature view with a specific schema, this function should write the pyarrow table to the batch source defined. More details about the push api can be found [here](https://github.com/feast-dev/feast/blob/master/docs/how-to-guides/docs/reference/data-sources/push.md). This method only needs implementation if you want to support the push api in your offline store. * (optional) `pull_all_from_table_or_query` is a method that pulls all the data from an offline store from a specified start date to a specified end date. This method is used for **SavedDatasets** and as a fallback compute path for the [Feature Quality Monitoring](/master/how-to-guides/feature-monitoring) system (backends without native SQL push-down). * (optional) `write_logged_features` is a method that takes a pyarrow table or a path that points to a parquet file and writes the data to a defined source defined by `LoggingSource` and `LoggingConfig`. This method is only used internally for **SavedDatasets**. {% code title="feast\_custom\_offline\_store/file.py" %} ```python # Only prints out runtime warnings once. warnings.simplefilter("once", RuntimeWarning) def get_historical_features(self, config: RepoConfig, feature_views: List[FeatureView], feature_refs: List[str], entity_df: Union[pd.DataFrame, str], registry: Registry, project: str, full_feature_names: bool = False) -> RetrievalJob: """ Perform point-in-time correct join of features onto an entity dataframe(entity key and timestamp). More details about how this should work at https://docs.feast.dev/v/v0.6-branch/user-guide/feature-retrieval#3.-historical-feature-retrieval. print("Getting historical features from my offline store").""" warnings.warn( "This offline store is an experimental feature in alpha development. " "Some functionality may still be unstable so functionality can change in the future.", RuntimeWarning, ) # Implementation here. pass def pull_latest_from_table_or_query(self, config: RepoConfig, data_source: DataSource, join_key_columns: List[str], feature_name_columns: List[str], timestamp_field: str, created_timestamp_column: Optional[str], start_date: datetime, end_date: datetime) -> RetrievalJob: """ Pulls data from the offline store for use in materialization.""" print("Pulling latest features from my offline store") warnings.warn( "This offline store is an experimental feature in alpha development. " "Some functionality may still be unstable so functionality can change in the future.", RuntimeWarning, ) # Implementation here. pass def pull_all_from_table_or_query( config: RepoConfig, data_source: DataSource, join_key_columns: List[str], feature_name_columns: List[str], timestamp_field: str, start_date: datetime, end_date: datetime, ) -> RetrievalJob: """ Optional method that returns a Retrieval Job for all join key columns, feature name columns, and the event timestamp columns that occur between the start_date and end_date.""" warnings.warn( "This offline store is an experimental feature in alpha development. " "Some functionality may still be unstable so functionality can change in the future.", RuntimeWarning, ) # Implementation here. pass def write_logged_features( config: RepoConfig, data: Union[pyarrow.Table, Path], source: LoggingSource, logging_config: LoggingConfig, registry: BaseRegistry, ): """ Optional method to have Feast support logging your online features.""" warnings.warn( "This offline store is an experimental feature in alpha development. " "Some functionality may still be unstable so functionality can change in the future.", RuntimeWarning, ) # Implementation here. pass def offline_write_batch( config: RepoConfig, feature_view: FeatureView, table: pyarrow.Table, progress: Optional[Callable[[int], Any]], ): """ Optional method to have Feast support the offline push api for your offline store.""" warnings.warn( "This offline store is an experimental feature in alpha development. " "Some functionality may still be unstable so functionality can change in the future.", RuntimeWarning, ) # Implementation here. pass ``` {% endcode %} ### 1.1 Type Mapping Most offline stores will have to perform some custom mapping of offline store datatypes to feast value types. * The function to implement here are `source_datatype_to_feast_value_type` and `get_column_names_and_types` in your `DataSource` class. * `source_datatype_to_feast_value_type` is used to convert your DataSource's datatypes to feast value types. * `get_column_names_and_types` retrieves the column names and corresponding datasource types. Add any helper functions for type conversion to `sdk/python/feast/type_map.py`. * Be sure to implement correct type mapping so that Feast can process your feature columns without casting incorrectly that can potentially cause loss of information or incorrect data. ## 2. Defining an OfflineStoreConfig class Additional configuration may be needed to allow the OfflineStore to talk to the backing store. For example, Redshift needs configuration information like the connection information for the Redshift instance, credentials for connecting to the database, etc. To facilitate configuration, all OfflineStore implementations are **required** to also define a corresponding OfflineStoreConfig class in the same file. This OfflineStoreConfig class should inherit from the `FeastConfigBaseModel` class, which is defined [here](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/repo_config.py#L44). The `FeastConfigBaseModel` is a [pydantic](https://pydantic-docs.helpmanual.io) class, which parses yaml configuration into python objects. Pydantic also allows the model classes to define validators for the config classes, to make sure that the config classes are correctly defined. This config class **must** container a `type` field, which contains the fully qualified class name of its corresponding OfflineStore class. Additionally, the name of the config class must be the same as the OfflineStore class, with the `Config` suffix. An example of the config class for the custom file offline store : {% code title="feast\_custom\_offline\_store/file.py" %} ```python class CustomFileOfflineStoreConfig(FeastConfigBaseModel): """ Custom offline store config for local (file-based) store """ type: Literal["feast_custom_offline_store.file.CustomFileOfflineStore"] \ = "feast_custom_offline_store.file.CustomFileOfflineStore" uri: str # URI for your offline store(in this case it would be a path) ``` {% endcode %} This configuration can be specified in the `feature_store.yaml` as follows: {% code title="feature\_repo/feature\_store.yaml" %} ```yaml project: my_project registry: data/registry.db provider: local offline_store: type: feast_custom_offline_store.file.CustomFileOfflineStore uri: online_store: path: data/online_store.db ``` {% endcode %} This configuration information is available to the methods of the OfflineStore, via the `config: RepoConfig` parameter which is passed into the methods of the OfflineStore interface, specifically at the `config.offline_store` field of the `config` parameter. This fields in the `feature_store.yaml` should map directly to your `OfflineStoreConfig` class that is detailed above in Section 2. {% code title="feast\_custom\_offline\_store/file.py" %} ```python def get_historical_features(self, config: RepoConfig, feature_views: List[FeatureView], feature_refs: List[str], entity_df: Union[pd.DataFrame, str], registry: Registry, project: str, full_feature_names: bool = False) -> RetrievalJob: warnings.warn( "This offline store is an experimental feature in alpha development. " "Some functionality may still be unstable so functionality can change in the future.", RuntimeWarning, ) offline_store_config = config.offline_store assert isinstance(offline_store_config, CustomFileOfflineStoreConfig) store_type = offline_store_config.type ``` {% endcode %} ## 3. Defining a RetrievalJob class The offline store methods aren't expected to perform their read operations eagerly. Instead, they are expected to execute lazily, and they do so by returning a `RetrievalJob` instance, which represents the execution of the actual query against the underlying store. Custom offline stores may need to implement their own instances of the `RetrievalJob` interface. The `RetrievalJob` interface exposes two methods - `to_df` and `to_arrow`. The expectation is for the retrieval job to be able to return the rows read from the offline store as a parquet DataFrame, or as an Arrow table respectively. Users who want to have their offline store support **scalable batch materialization** for online use cases (detailed in this [RFC](https://docs.google.com/document/d/1J7XdwwgQ9dY_uoV9zkRVGQjK9Sy43WISEW6D5V9qzGo/edit#heading=h.9gaqqtox9jg6)) will also need to implement `to_remote_storage` to distribute the reading and writing of offline store records to blob storage (such as S3). This may be used by a custom [Materialization Engine](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/materialization/batch_materialization_engine.py#L72) to parallelize the materialization of data by processing it in chunks. If this is not implemented, Feast will default to local materialization (pulling all records into memory to materialize). {% code title="feast\_custom\_offline\_store/file.py" %} ```python class CustomFileRetrievalJob(RetrievalJob): def __init__(self, evaluation_function: Callable): """Initialize a lazy historical retrieval job""" # The evaluation function executes a stored procedure to compute a historical retrieval. self.evaluation_function = evaluation_function def to_df(self): # Only execute the evaluation function to build the final historical retrieval dataframe at the last moment. print("Getting a pandas DataFrame from a File is easy!") df = self.evaluation_function() return df def to_arrow(self): # Only execute the evaluation function to build the final historical retrieval dataframe at the last moment. print("Getting a pandas DataFrame from a File is easy!") df = self.evaluation_function() return pyarrow.Table.from_pandas(df) def to_remote_storage(self): # Optional method to write to an offline storage location to support scalable batch materialization. pass ``` {% endcode %} ## 4. Defining a DataSource class for the offline store Before this offline store can be used as the batch source for a feature view in a feature repo, a subclass of the `DataSource` [base class](https://rtd.feast.dev/en/master/index.html?highlight=DataSource#feast.data_source.DataSource) needs to be defined. This class is responsible for holding information needed by specific feature views to support reading historical values from the offline store. For example, a feature view using Redshift as the offline store may need to know which table contains historical feature values. The data source class should implement two methods - `from_proto`, and `to_proto`. For custom offline stores that are not being implemented in the main feature repo, the `custom_options` field should be used to store any configuration needed by the data source. In this case, the implementer is responsible for serializing this configuration into bytes in the `to_proto` method and reading the value back from bytes in the `from_proto` method. {% code title="feast\_custom\_offline\_store/file.py" %} ```python class CustomFileDataSource(FileSource): """Custom data source class for local files""" def __init__( self, timestamp_field: Optional[str] = "", path: Optional[str] = None, field_mapping: Optional[Dict[str, str]] = None, created_timestamp_column: Optional[str] = "", date_partition_column: Optional[str] = "", ): "Some functionality may still be unstable so functionality can change in the future.", RuntimeWarning, ) super(CustomFileDataSource, self).__init__( timestamp_field=timestamp_field, created_timestamp_column, field_mapping, date_partition_column, ) self._path = path @staticmethod def from_proto(data_source: DataSourceProto): custom_source_options = str( data_source.custom_options.configuration, encoding="utf8" ) path = json.loads(custom_source_options)["path"] return CustomFileDataSource( field_mapping=dict(data_source.field_mapping), path=path, timestamp_field=data_source.timestamp_field, created_timestamp_column=data_source.created_timestamp_column, date_partition_column=data_source.date_partition_column, ) def to_proto(self) -> DataSourceProto: config_json = json.dumps({"path": self.path}) data_source_proto = DataSourceProto( type=DataSourceProto.CUSTOM_SOURCE, custom_options=DataSourceProto.CustomSourceOptions( configuration=bytes(config_json, encoding="utf8") ), ) data_source_proto.timestamp_field = self.timestamp_field data_source_proto.created_timestamp_column = self.created_timestamp_column data_source_proto.date_partition_column = self.date_partition_column return data_source_proto ``` {% endcode %} ## 5. Using the custom offline store After implementing these classes, the custom offline store can be used by referencing it in a feature repo's `feature_store.yaml` file, specifically in the `offline_store` field. The value specified should be the fully qualified class name of the OfflineStore. As long as your OfflineStore class is available in your Python environment, it will be imported by Feast dynamically at runtime. To use our custom file offline store, we can use the following `feature_store.yaml`: {% code title="feature\_repo/feature\_store.yaml" %} ```yaml project: test_custom registry: data/registry.db provider: local offline_store: # Make sure to specify the type as the fully qualified path that Feast can import. type: feast_custom_offline_store.file.CustomFileOfflineStore ``` {% endcode %} If additional configuration for the offline store is **not** required, then we can omit the other fields and only specify the `type` of the offline store class as the value for the `offline_store`. {% code title="feature\_repo/feature\_store.yaml" %} ```yaml project: test_custom registry: data/registry.db provider: local offline_store: feast_custom_offline_store.file.CustomFileOfflineStore ``` {% endcode %} Finally, the custom data source class can be use in the feature repo to define a data source, and refer to in a feature view definition. {% code title="feature\_repo/repo.py" %} ```python driver_hourly_stats = CustomFileDataSource( path="feature_repo/data/driver_stats.parquet", timestamp_field="event_timestamp", created_timestamp_column="created", ) driver_hourly_stats_view = FeatureView( source=driver_hourly_stats, ... ) ``` {% endcode %} ## 6. Testing the OfflineStore class ### Integrating with the integration test suite and unit test suite. Even if you have created the `OfflineStore` class in a separate repo, you can still test your implementation against the Feast test suite, as long as you have Feast as a submodule in your repo. 1. In order to test against the test suite, you need to create a custom `DataSourceCreator` that implement our testing infrastructure methods, `create_data_source` and optionally, `created_saved_dataset_destination`. * `create_data_source` should create a datasource based on the dataframe passed in. It may be implemented by uploading the contents of the dataframe into the offline store and returning a datasource object pointing to that location. See `BigQueryDataSourceCreator` for an implementation of a data source creator. * `created_saved_dataset_destination` is invoked when users need to save the dataset for use in data validation. This functionality is still in alpha and is **optional**. 2. Make sure that your offline store doesn't break any unit tests first by running: ``` make test-python-unit ``` 3. Next, set up your offline store to run the universal integration tests. These are integration tests specifically intended to test offline and online stores against Feast API functionality, to ensure that the Feast APIs works with your offline store. * Feast parametrizes integration tests using the `FULL_REPO_CONFIGS` variable defined in `sdk/python/tests/universal/feature_repos/repo_configuration.py` which stores different offline store classes for testing. * To overwrite the default configurations to use your own offline store, you can simply create your own file that contains a `FULL_REPO_CONFIGS` dictionary, and point Feast to that file by setting the environment variable `FULL_REPO_CONFIGS_MODULE` to point to that file. The module should add new `IntegrationTestRepoConfig` classes to the `AVAILABLE_OFFLINE_STORES` by defining an offline store that you would like Feast to test with. A sample `FULL_REPO_CONFIGS_MODULE` looks something like this: ```python # Should go in sdk/python/feast/infra/offline_stores/contrib/postgres_repo_configuration.py from feast.infra.offline_stores.contrib.postgres_offline_store.tests.data_source import ( PostgreSQLDataSourceCreator, ) AVAILABLE_OFFLINE_STORES = [("local", PostgreSQLDataSourceCreator)] ``` 4. You should swap out the `FULL_REPO_CONFIGS` environment variable and run the integration tests against your offline store. In the example repo, the file that overwrites `FULL_REPO_CONFIGS` is `feast_custom_offline_store/feast_tests.py`, so you would run: ```bash export FULL_REPO_CONFIGS_MODULE='feast_custom_offline_store.feast_tests' make test-python-universal ``` If the integration tests fail, this indicates that there is a mistake in the implementation of this offline store! 5. Remember to add your datasource to `repo_config.py` similar to how we added `spark`, `trino`, etc, to the dictionary `OFFLINE_STORE_CLASS_FOR_TYPE`. This will allow Feast to load your class from the `feature_store.yaml`. 6. Finally, add a Makefile target to the Makefile to run your datastore specific tests by setting the `FULL_REPO_CONFIGS_MODULE` and `PYTEST_PLUGINS` environment variable. The `PYTEST_PLUGINS` environment variable allows pytest to load in the `DataSourceCreator` for your datasource. You can remove certain tests that are not relevant or still do not work for your datastore using the `-k` option. {% code title="Makefile" %} ```makefile test-python-universal-spark: PYTHONPATH='.' \ FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.spark_repo_configuration \ PYTEST_PLUGINS=feast.infra.offline_stores.contrib.spark_offline_store.tests \ IS_TEST=True \ python -m pytest -n 8 --integration \ -k "not test_historical_retrieval_fails_on_validation and \ not test_historical_retrieval_with_validation and \ not test_historical_features_persisting and \ not test_historical_retrieval_fails_on_validation and \ not test_universal_cli and \ not test_go_feature_server and \ not test_feature_logging and \ not test_reorder_columns and \ not test_logged_features_validation and \ not test_lambda_materialization_consistency and \ not test_offline_write and \ not test_push_features_to_offline_store.py and \ not gcs_registry and \ not s3_registry and \ not test_universal_types" \ sdk/python/tests ``` {% endcode %} ### 7. Dependencies Add any dependencies for your offline store to `pyproject.toml` under `[project.optional-dependencies]` as a new extra (e.g. ` = ["package1>=1.0", "package2"]`). These packages should be defined as extras so that they are not installed by users by default. You will need to regenerate our requirements lock files: ``` make lock-python-dependencies-all ``` ### 8. Add Documentation Remember to add documentation for your offline store. 1. Add a new markdown file to `docs/reference/offline-stores/` and `docs/reference/data-sources/`. Use these files to document your offline store functionality similar to how the other offline stores are documented. 2. You should also add a reference in `docs/reference/data-sources/README.md` and `docs/SUMMARY.md` to these markdown files. **NOTE**: Be sure to document the following things about your offline store: * How to create the datasource and most what configuration is needed in the `feature_store.yaml` file in order to create the datasource. * Make sure to flag that the datasource is in alpha development. * Add some documentation on what the data model is for the specific offline store for more clarity. * Finally, generate the python code docs by running: ```bash make build-sphinx ``` # Adding a new online store ## Overview Feast makes adding support for a new online store (database) easy. Developers can simply implement the [OnlineStore](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/online_stores/online_store.py#L26) interface to add support for a new store (other than the existing stores like Redis, DynamoDB, SQLite, and Datastore). In this guide, we will show you how to integrate with MySQL as an online store. While we will be implementing a specific store, this guide should be representative for adding support for any new online store. The full working code for this guide can be found at [feast-dev/feast-custom-online-store-demo](https://github.com/feast-dev/feast-custom-online-store-demo). The process of using a custom online store consists of 6 steps: 1. Defining the `OnlineStore` class. 2. Defining the `OnlineStoreConfig` class. 3. Referencing the `OnlineStore` in a feature repo's `feature_store.yaml` file. 4. Testing the `OnlineStore` class. 5. Update dependencies. 6. Add documentation. ## 1. Defining an OnlineStore class {% hint style="info" %} OnlineStore class names must end with the OnlineStore suffix! {% endhint %} ### Contrib online stores New online stores go in `sdk/python/feast/infra/online_stores/`. #### What is a contrib plugin? * Not guaranteed to implement all interface methods * Not guaranteed to be stable. * Should have warnings for users to indicate this is a contrib plugin that is not maintained by the maintainers. #### How do I make a contrib plugin an "official" plugin? To move an online store plugin out of contrib, you need: * GitHub actions (i.e `make test-python-integration`) is setup to run all tests against the online store and pass. * At least two contributors own the plugin (ideally tracked in our `OWNERS` / `CODEOWNERS` file). The OnlineStore class broadly contains two sets of methods * One set deals with managing infrastructure that the online store needed for operations * One set deals with writing data into the store, and reading data from the store. ### 1.1 Infrastructure Methods There are two methods that deal with managing infrastructure for online stores, `update` and `teardown` * `update` is invoked when users run `feast apply` as a CLI command, or the `FeatureStore.apply()` sdk method. The `update` method should be used to perform any operations necessary before data can be written to or read from the store. The `update` method can be used to create MySQL tables in preparation for reads and writes to new feature views. * `teardown` is invoked when users run `feast teardown` or `FeatureStore.teardown()`. The `teardown` method should be used to perform any clean-up operations. `teardown` can be used to drop MySQL indices and tables corresponding to the feature views being deleted. {% code title="feast\_custom\_online\_store/mysql.py" %} ```python # Only prints out runtime warnings once. warnings.simplefilter("once", RuntimeWarning) def update( self, config: RepoConfig, tables_to_delete: Sequence[Union[FeatureTable, FeatureView]], tables_to_keep: Sequence[Union[FeatureTable, FeatureView]], entities_to_delete: Sequence[Entity], entities_to_keep: Sequence[Entity], partial: bool, ): """ An example of creating managing the tables needed for a mysql-backed online store. """ warnings.warn( "This online store is an experimental feature in alpha development. " "Some functionality may still be unstable so functionality can change in the future.", RuntimeWarning, ) conn = self._get_conn(config) cur = conn.cursor(buffered=True) project = config.project for table in tables_to_keep: cur.execute( f"CREATE TABLE IF NOT EXISTS {_table_id(project, table)} (entity_key VARCHAR(512), feature_name VARCHAR(256), value BLOB, event_ts timestamp, created_ts timestamp, PRIMARY KEY(entity_key, feature_name))" ) cur.execute( f"CREATE INDEX {_table_id(project, table)}_ek ON {_table_id(project, table)} (entity_key);" ) for table in tables_to_delete: cur.execute( f"DROP INDEX {_table_id(project, table)}_ek ON {_table_id(project, table)};" ) cur.execute(f"DROP TABLE IF EXISTS {_table_id(project, table)}") def teardown( self, config: RepoConfig, tables: Sequence[Union[FeatureTable, FeatureView]], entities: Sequence[Entity], ): warnings.warn( "This online store is an experimental feature in alpha development. " "Some functionality may still be unstable so functionality can change in the future.", RuntimeWarning, ) conn = self._get_conn(config) cur = conn.cursor(buffered=True) project = config.project for table in tables: cur.execute( f"DROP INDEX {_table_id(project, table)}_ek ON {_table_id(project, table)};" ) cur.execute(f"DROP TABLE IF EXISTS {_table_id(project, table)}") ``` {% endcode %} ### 1.2 Read/Write Methods There are two methods that deal with writing data to and from the online stores.`online_write_batch` and `online_read`. * `online_write_batch` is invoked when running materialization (using the `feast materialize` or `feast materialize-incremental` commands, or the corresponding `FeatureStore.materialize()` method. * `online_read` is invoked when reading values from the online store using the `FeatureStore.get_online_features()` method. {% code title="feast\_custom\_online\_store/mysql.py" %} ```python # Only prints out runtime warnings once. warnings.simplefilter("once", RuntimeWarning) def online_write_batch( self, config: RepoConfig, table: Union[FeatureTable, FeatureView], data: List[ Tuple[EntityKeyProto, Dict[str, ValueProto], datetime, Optional[datetime]] ], progress: Optional[Callable[[int], Any]], ) -> None: warnings.warn( "This online store is an experimental feature in alpha development. " "Some functionality may still be unstable so functionality can change in the future.", RuntimeWarning, ) conn = self._get_conn(config) cur = conn.cursor(buffered=True) project = config.project for entity_key, values, timestamp, created_ts in data: entity_key_bin = serialize_entity_key( entity_key, entity_key_serialization_version=config.entity_key_serialization_version, ).hex() timestamp = _to_naive_utc(timestamp) if created_ts is not None: created_ts = _to_naive_utc(created_ts) for feature_name, val in values.items(): self.write_to_table(created_ts, cur, entity_key_bin, feature_name, project, table, timestamp, val) self._conn.commit() if progress: progress(1) def online_read( self, config: RepoConfig, table: Union[FeatureTable, FeatureView], entity_keys: List[EntityKeyProto], requested_features: Optional[List[str]] = None, ) -> List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]]: warnings.warn( "This online store is an experimental feature in alpha development. " "Some functionality may still be unstable so functionality can change in the future.", RuntimeWarning, ) conn = self._get_conn(config) cur = conn.cursor(buffered=True) result: List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]] = [] project = config.project for entity_key in entity_keys: entity_key_bin = serialize_entity_key( entity_key, entity_key_serialization_version=config.entity_key_serialization_version, ).hex() print(f"entity_key_bin: {entity_key_bin}") cur.execute( f"SELECT feature_name, value, event_ts FROM {_table_id(project, table)} WHERE entity_key = %s", (entity_key_bin,), ) res = {} res_ts = None for feature_name, val_bin, ts in cur.fetchall(): val = ValueProto() val.ParseFromString(val_bin) res[feature_name] = val res_ts = ts if not res: result.append((None, None)) else: result.append((res_ts, res)) return result ``` {% endcode %} ## 2. Defining an OnlineStoreConfig class Additional configuration may be needed to allow the OnlineStore to talk to the backing store. For example, MySQL may need configuration information like the host at which the MySQL instance is running, credentials for connecting to the database, etc. To facilitate configuration, all OnlineStore implementations are **required** to also define a corresponding OnlineStoreConfig class in the same file. This OnlineStoreConfig class should inherit from the `FeastConfigBaseModel` class, which is defined [here](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/repo_config.py#L44). The `FeastConfigBaseModel` is a [pydantic](https://pydantic-docs.helpmanual.io) class, which parses yaml configuration into python objects. Pydantic also allows the model classes to define validators for the config classes, to make sure that the config classes are correctly defined. This config class **must** container a `type` field, which contains the fully qualified class name of its corresponding OnlineStore class. Additionally, the name of the config class must be the same as the OnlineStore class, with the `Config` suffix. An example of the config class for MySQL : {% code title="feast\_custom\_online\_store/mysql.py" %} ```python class MySQLOnlineStoreConfig(FeastConfigBaseModel): type: Literal["feast_custom_online_store.mysql.MySQLOnlineStore"] = "feast_custom_online_store.mysql.MySQLOnlineStore" host: Optional[StrictStr] = None user: Optional[StrictStr] = None password: Optional[StrictStr] = None database: Optional[StrictStr] = None ``` {% endcode %} This configuration can be specified in the `feature_store.yaml` as follows: {% code title="feature\_repo/feature\_store.yaml" %} ```yaml online_store: type: feast_custom_online_store.mysql.MySQLOnlineStore user: foo password: bar ``` {% endcode %} This configuration information is available to the methods of the OnlineStore, via the`config: RepoConfig` parameter which is passed into all the methods of the OnlineStore interface, specifically at the `config.online_store` field of the `config` parameter. {% code title="feast\_custom\_online\_store/mysql.py" %} ```python def online_write_batch( self, config: RepoConfig, table: Union[FeatureTable, FeatureView], data: List[ Tuple[EntityKeyProto, Dict[str, ValueProto], datetime, Optional[datetime]] ], progress: Optional[Callable[[int], Any]], ) -> None: online_store_config = config.online_store assert isinstance(online_store_config, MySQLOnlineStoreConfig) connection = mysql.connector.connect( host=online_store_config.host or "127.0.0.1", user=online_store_config.user or "root", password=online_store_config.password, database=online_store_config.database or "feast", autocommit=True ) ``` {% endcode %} ## 3. Using the custom online store After implementing both these classes, the custom online store can be used by referencing it in a feature repo's `feature_store.yaml` file, specifically in the `online_store` field. The value specified should be the fully qualified class name of the OnlineStore. As long as your OnlineStore class is available in your Python environment, it will be imported by Feast dynamically at runtime. To use our MySQL online store, we can use the following `feature_store.yaml`: {% code title="feature\_repo/feature\_store.yaml" %} ```yaml project: test_custom registry: data/registry.db provider: local online_store: # Make sure to specify the type as the fully qualified path that Feast can import. type: feast_custom_online_store.mysql.MySQLOnlineStore user: foo password: bar ``` {% endcode %} If additional configuration for the online store is \*\*not \*\*required, then we can omit the other fields and only specify the `type` of the online store class as the value for the `online_store`. {% code title="feature\_repo/feature\_store.yaml" %} ```yaml project: test_custom registry: data/registry.db provider: local online_store: feast_custom_online_store.mysql.MySQLOnlineStore ``` {% endcode %} ## 4. Testing the OnlineStore class ### 4.1 Integrating with the integration test suite and unit test suite. Even if you have created the `OnlineStore` class in a separate repo, you can still test your implementation against the Feast test suite, as long as you have Feast as a submodule in your repo. 1. In the Feast submodule, we can run all the unit tests and make sure they pass: ``` make test-python-unit ``` 2. The universal tests, which are integration tests specifically intended to test offline and online stores, should be run against Feast to ensure that the Feast APIs works with your online store. * Feast parametrizes integration tests using the `FULL_REPO_CONFIGS` variable defined in `sdk/python/tests/universal/feature_repos/repo_configuration.py` which stores different online store classes for testing. * To overwrite these configurations, you can simply create your own file that contains a `FULL_REPO_CONFIGS` variable, and point Feast to that file by setting the environment variable `FULL_REPO_CONFIGS_MODULE` to point to that file. A sample `FULL_REPO_CONFIGS_MODULE` looks something like this: {% code title="sdk/python/feast/infra/online\_stores/contrib/postgres\_repo\_configuration.py" %} ```python from feast.infra.offline_stores.contrib.postgres_offline_store.tests.data_source import ( PostgreSQLDataSourceCreator, ) AVAILABLE_ONLINE_STORES = {"postgres": (None, PostgreSQLDataSourceCreator)} ``` {% endcode %} If you are planning to start the online store up locally(e.g spin up a local Redis Instance) for testing, then the dictionary entry should be something like: ```python { "sqlite": ({"type": "sqlite"}, None), # Specifies sqlite as the online store. The `None` object specifies to not use a containerized docker container. } ``` If you are planning instead to use a Dockerized container to run your tests against your online store, you can define a `OnlineStoreCreator` and replace the `None` object above with your `OnlineStoreCreator` class. You should make this class available to pytest through the `PYTEST_PLUGINS` environment variable. If you create a containerized docker image for testing, developers who are trying to test with your online store will not have to spin up their own instance of the online store for testing. An example of an `OnlineStoreCreator` is shown below: {% code title="sdk/python/tests/universal/feature\_repos/universal/online\_store/redis.py" %} ```python class RedisOnlineStoreCreator(OnlineStoreCreator): def __init__(self, project_name: str, **kwargs): super().__init__(project_name) def create_online_store(self) -> Dict[str, str]: self.container.start() log_string_to_wait_for = "Ready to accept connections" wait_for_logs( container=self.container, predicate=log_string_to_wait_for, timeout=10 ) self.container.stop() ``` {% endcode %} 3\. Add a Makefile target to the Makefile to run your datastore specific tests by setting the `FULL_REPO_CONFIGS_MODULE` environment variable. Add `PYTEST_PLUGINS` if pytest is having trouble loading your `DataSourceCreator`. You can remove certain tests that are not relevant or still do not work for your datastore using the `-k` option. {% code title="Makefile" %} ```makefile test-python-universal-cassandra: PYTHONPATH='.' \ FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.online_stores.cassandra_online_store.cassandra_repo_configuration \ PYTEST_PLUGINS=sdk.python.tests.universal.feature_repos.universal.online_store.cassandra \ IS_TEST=True \ python -m pytest -x --integration \ sdk/python/tests ``` {% endcode %} * If there are some tests that fail, this indicates that there is a mistake in the implementation of this online store! ### 5. Add Dependencies Add any dependencies for your online store to `pyproject.toml` under `[project.optional-dependencies]` as a new extra (e.g. ` = ["package1>=1.0", "package2"]`). These packages should be defined as extras so that they are not installed by users by default. * You will need to regenerate our requirements lock files: ``` make lock-python-dependencies-all ``` ### 6. Add Documentation Remember to add the documentation for your online store. 1. Add a new markdown file to `docs/reference/online-stores/`. 2. You should also add a reference in `docs/reference/online-stores/README.md` and `docs/SUMMARY.md`. Add a new markdown document to document your online store functionality similar to how the other online stores are documented. **NOTE**:Be sure to document the following things about your online store: * Be sure to cover how to create the datasource and what configuration is needed in the `feature_store.yaml` file in order to create the datasource. * Make sure to flag that the online store is in alpha development. * Add some documentation on what the data model is for the specific online store for more clarity. * Finally, generate the python code docs by running: ```bash make build-sphinx ``` # Adding a custom provider ### Overview All Feast operations execute through a `provider`. Operations like materializing data from the offline to the online store, updating infrastructure like databases, launching streaming ingestion jobs, building training datasets, and reading features from the online store. Custom providers allow Feast users to extend Feast to execute any custom logic. Examples include: * Launching custom streaming ingestion jobs (Spark, Beam) * Launching custom batch ingestion (materialization) jobs (Spark, Beam) * Adding custom validation to feature repositories during `feast apply` * Adding custom infrastructure setup logic which runs during `feast apply` * Extending Feast commands with in-house metrics, logging, or tracing Feast comes with built-in providers, e.g, `LocalProvider`, `GcpProvider`, and `AwsProvider`. However, users can develop their own providers by creating a class that implements the contract in the [Provider class](https://github.com/feast-dev/feast/blob/745a1b43d20c0169b675b1f28039854205fb8180/sdk/python/feast/infra/provider.py#L22). {% hint style="info" %} This guide also comes with a fully functional [custom provider demo repository](https://github.com/feast-dev/feast-custom-provider-demo). Please have a look at the repository for a representative example of what a custom provider looks like, or fork the repository when creating your own provider. {% endhint %} ### Guide The fastest way to add custom logic to Feast is to extend an existing provider. The most generic provider is the `LocalProvider` which contains no cloud-specific logic. The guide that follows will extend the `LocalProvider` with operations that print text to the console. It is up to you as a developer to add your custom code to the provider methods, but the guide below will provide the necessary scaffolding to get you started. #### Step 1: Define a Provider class The first step is to define a custom provider class. We've created the `MyCustomProvider` below. ```python from datetime import datetime from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Union from feast.entity import Entity from feast.feature_table import FeatureTable from feast.feature_view import FeatureView from feast.infra.local import LocalProvider from feast.infra.offline_stores.offline_store import RetrievalJob from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto from feast.protos.feast.types.Value_pb2 import Value as ValueProto from feast.infra.registry.registry import Registry from feast.repo_config import RepoConfig class MyCustomProvider(LocalProvider): def __init__(self, config: RepoConfig, repo_path): super().__init__(config) # Add your custom init code here. This code runs on every Feast operation. def update_infra( self, project: str, tables_to_delete: Sequence[Union[FeatureTable, FeatureView]], tables_to_keep: Sequence[Union[FeatureTable, FeatureView]], entities_to_delete: Sequence[Entity], entities_to_keep: Sequence[Entity], partial: bool, ): super().update_infra( project, tables_to_delete, tables_to_keep, entities_to_delete, entities_to_keep, partial, ) print("Launching custom streaming jobs is pretty easy...") def materialize_single_feature_view( self, config: RepoConfig, feature_view: FeatureView, start_date: datetime, end_date: datetime, registry: Registry, project: str, tqdm_builder: Callable[[int], tqdm], ) -> None: super().materialize_single_feature_view( config, feature_view, start_date, end_date, registry, project, tqdm_builder ) print("Launching custom batch jobs is pretty easy...") ``` Notice how in the above provider we have only overwritten two of the methods on the `LocalProvider`, namely `update_infra` and `materialize_single_feature_view`. These two methods are convenient to replace if you are planning to launch custom batch or streaming jobs. `update_infra` can be used for launching idempotent streaming jobs, and `materialize_single_feature_view` can be used for launching batch ingestion jobs. It is possible to overwrite all the methods on the provider class. In fact, it isn't even necessary to subclass an existing provider like `LocalProvider`. The only requirement for the provider class is that it follows the [Provider contract](https://github.com/feast-dev/feast/blob/048c837b2fa741b38b0e35b8f8e534761a232561/sdk/python/feast/infra/provider.py#L22). #### Step 2: Configuring Feast to use the provider Configure your [feature\_store.yaml](/master/reference/feature-repository/feature-store-yaml) file to point to your new provider class: ```yaml project: repo registry: registry.db provider: feast_custom_provider.custom_provider.MyCustomProvider online_store: type: sqlite path: online_store.db offline_store: type: file ``` Notice how the `provider` field above points to the module and class where your provider can be found. #### Step 3: Using the provider Now you should be able to use your provider by running a Feast command: ```bash feast apply ``` ``` Registered entity driver_id Registered feature view driver_hourly_stats Deploying infrastructure for driver_hourly_stats Launching custom streaming jobs is pretty easy... ``` It may also be necessary to add the module root path to your `PYTHONPATH` as follows: ```bash PYTHONPATH=$PYTHONPATH:/home/my_user/my_custom_provider feast apply ``` That's it. You should now have a fully functional custom provider! ### Next steps Have a look at the [custom provider demo repository](https://github.com/feast-dev/feast-custom-provider-demo) for a fully functional example of a custom provider. Feel free to fork it when creating your own custom provider! # Adding or reusing tests ## Overview This guide will go over: 1. how Feast tests are setup 2. how to extend the test suite to test new functionality 3. how to use the existing test suite to test a new custom offline / online store ## Test suite overview Unit tests are contained in `sdk/python/tests/unit`. Integration tests are contained in `sdk/python/tests/integration`. Let's inspect the structure of `sdk/python/tests/integration`: ```bash $ tree . ├── e2e │ ├── test_go_feature_server.py │ ├── test_python_feature_server.py │ ├── test_universal_e2e.py │ └── test_validation.py ├── feature_repos │ ├── integration_test_repo_config.py │ ├── repo_configuration.py │ └── universal │ ├── catalog │ ├── data_source_creator.py │ ├── data_sources │ │ ├── __init__.py │ │ ├── bigquery.py │ │ ├── file.py │ │ ├── redshift.py │ │ └── snowflake.py │ ├── entities.py │ ├── feature_views.py │ ├── online_store │ │ ├── __init__.py │ │ ├── datastore.py │ │ ├── dynamodb.py │ │ ├── hbase.py │ │ └── redis.py │ └── online_store_creator.py ├── materialization │ └── test_lambda.py ├── offline_store │ ├── test_feature_logging.py │ ├── test_offline_write.py │ ├── test_push_features_to_offline_store.py │ ├── test_s3_custom_endpoint.py │ └── test_universal_historical_retrieval.py ├── online_store │ ├── test_push_features_to_online_store.py │ └── test_universal_online.py └── registration ├── test_feature_store.py ├── test_inference.py ├── test_registry.py ├── test_universal_cli.py ├── test_universal_odfv_feature_inference.py └── test_universal_types.py ``` * `feature_repos` has setup files for most tests in the test suite. * `conftest.py` (in the parent directory) contains the most common [fixtures](https://docs.pytest.org/en/6.2.x/fixture.html), which are designed as an abstraction on top of specific offline/online stores, so tests do not need to be rewritten for different stores. Individual test files also contain more specific fixtures. * The tests are organized by which Feast component(s) they test. ## Structure of the test suite ### Universal feature repo The universal feature repo refers to a set of fixtures (e.g. `environment` and `universal_data_sources`) that can be parametrized to cover various combinations of offline stores, online stores, and providers. This allows tests to run against all these various combinations without requiring excess code. The universal feature repo is constructed by fixtures in `conftest.py` with help from the various files in `feature_repos`. ### Integration vs. unit tests Tests in Feast are split into integration and unit tests. If a test requires external resources (e.g. cloud resources on GCP or AWS), it is an integration test. If a test can be run purely locally (where locally includes Docker resources), it is a unit test. * Integration tests test non-local Feast behavior. For example, tests that require reading data from BigQuery or materializing data to DynamoDB are integration tests. Integration tests also tend to involve more complex Feast functionality. * Unit tests test local Feast behavior. For example, tests that only require registering feature views are unit tests. Unit tests tend to only involve simple Feast functionality. ### Main types of tests #### Integration tests 1. E2E tests * E2E tests test end-to-end functionality of Feast over the various codepaths (initialize a feature store, apply, and materialize). * The main codepaths include: * basic e2e tests for offline stores * `test_universal_e2e.py` * go feature server * `test_go_feature_server.py` * python http server * `test_python_feature_server.py` * data quality monitoring feature validation * `test_validation.py` 2. Offline and Online Store Tests * Offline and online store tests mainly test for the offline and online retrieval functionality. * The various specific functionalities that are tested include: * push API tests * `test_push_features_to_offline_store.py` * `test_push_features_to_online_store.py` * `test_offline_write.py` * historical retrieval tests * `test_universal_historical_retrieval.py` * online retrieval tests * `test_universal_online.py` * data quality monitoring feature logging tests * `test_feature_logging.py` * online store tests * `test_universal_online.py` 3. Registration Tests * The registration folder contains all of the registry tests and some universal cli tests. This includes: * CLI Apply and Materialize tests tested against on the universal test suite * Data type inference tests * Registry tests 4. Miscellaneous Tests * AWS Lambda Materialization Tests (Currently do not work) * `test_lambda.py` #### Unit tests 1. Registry Diff Tests * These are tests for the infrastructure and registry diff functionality that Feast uses to determine if changes to the registry or infrastructure is needed. 2. Local CLI Tests and Local Feast Tests * These tests test all of the cli commands against the local file offline store. 3. Infrastructure Unit Tests * DynamoDB tests with dynamo mocked out * Repository configuration tests * Schema inference unit tests * Key serialization tests * Basic provider unit tests 4. Feature Store Validation Tests * These test mainly contain class level validation like hashing tests, protobuf and class serialization, and error and warning handling. * Data source unit tests * Feature service unit tests * Feature service, feature view, and feature validation tests * Protobuf/json tests for Feast ValueTypes * Serialization tests * Type mapping * Feast types * Serialization tests due to this [issue](https://github.com/feast-dev/feast/issues/2345) #### Docstring tests Docstring tests are primarily smoke tests to make sure imports and setup functions can be executed without errors. ## Understanding the test suite with an example test ### Example test Let's look at a sample test using the universal repo: {% tabs %} {% tab %} ```python @pytest.mark.integration @pytest.mark.universal_offline_stores @pytest.mark.parametrize("full_feature_names", [True, False], ids=lambda v: f"full:{v}") def test_historical_features(environment, universal_data_sources, full_feature_names): store = environment.feature_store (entities, datasets, data_sources) = universal_data_sources feature_views = construct_universal_feature_views(data_sources) entity_df_with_request_data = datasets.entity_df.copy(deep=True) entity_df_with_request_data["val_to_add"] = [ i for i in range(len(entity_df_with_request_data)) ] entity_df_with_request_data["driver_age"] = [ i + 100 for i in range(len(entity_df_with_request_data)) ] feature_service = FeatureService( name="convrate_plus100", features=[feature_views.driver[["conv_rate"]], feature_views.driver_odfv], ) feature_service_entity_mapping = FeatureService( name="entity_mapping", features=[ feature_views.location.with_name("origin").with_join_key_map( {"location_id": "origin_id"} ), feature_views.location.with_name("destination").with_join_key_map( {"location_id": "destination_id"} ), ], ) store.apply( [ driver(), customer(), location(), feature_service, feature_service_entity_mapping, *feature_views.values(), ] ) # ... more test code job_from_df = store.get_historical_features( entity_df=entity_df_with_request_data, features=[ "driver_stats:conv_rate", "driver_stats:avg_daily_trips", "customer_profile:current_balance", "customer_profile:avg_passenger_count", "customer_profile:lifetime_trip_count", "conv_rate_plus_100:conv_rate_plus_100", "conv_rate_plus_100:conv_rate_plus_100_rounded", "conv_rate_plus_100:conv_rate_plus_val_to_add", "order:order_is_success", "global_stats:num_rides", "global_stats:avg_ride_length", "field_mapping:feature_name", ], full_feature_names=full_feature_names, ) if job_from_df.supports_remote_storage_export(): files = job_from_df.to_remote_storage() print(files) assert len(files) > 0 # This test should be way more detailed start_time = datetime.utcnow() actual_df_from_df_entities = job_from_df.to_df() # ... more test code validate_dataframes( expected_df, table_from_df_entities, sort_by=[event_timestamp, "order_id", "driver_id", "customer_id"], event_timestamp = event_timestamp, ) # ... more test code ``` {% endtab %} {% endtabs %} * The key fixtures are the `environment` and `universal_data_sources` fixtures, which are defined in the `feature_repos` directories and the `conftest.py` file. This by default pulls in a standard dataset with driver and customer entities (that we have pre-defined), certain feature views, and feature values. * The `environment` fixture sets up a feature store, parametrized by the provider and the online/offline store. It allows the test to query against that feature store without needing to worry about the underlying implementation or any setup that may be involved in creating instances of these datastores. * Each fixture creates a different integration test with its own `IntegrationTestRepoConfig` which is used by pytest to generate a unique test testing one of the different environments that require testing. * Feast tests also use a variety of markers: * The `@pytest.mark.integration` marker is used to designate integration tests which will cause the test to be run when you call `make test-python-integration`. * The `@pytest.mark.universal_offline_stores` marker will parametrize the test on all of the universal offline stores including file, redshift, bigquery and snowflake. * The `full_feature_names` parametrization defines whether or not the test should reference features as their full feature name (fully qualified path) or just the feature name itself. ## Writing a new test or reusing existing tests ### To add a new test to an existing test file * Use the same function signatures as an existing test (e.g. use `environment` and `universal_data_sources` as an argument) to include the relevant test fixtures. * If possible, expand an individual test instead of writing a new test, due to the cost of starting up offline / online stores. * Use the `universal_offline_stores` and `universal_online_store` markers to parametrize the test against different offline store and online store combinations. You can also designate specific online and offline stores to test by using the `only` parameter on the marker. ```python @pytest.mark.universal_online_stores(only=["redis"]) ``` ### To test a new offline / online store from a plugin repo * Install Feast in editable mode with `pip install -e`. * The core tests for offline / online store behavior are parametrized by the `FULL_REPO_CONFIGS` variable defined in `feature_repos/repo_configuration.py`. To overwrite this variable without modifying the Feast repo, create your own file that contains a `FULL_REPO_CONFIGS` (which will require adding a new `IntegrationTestRepoConfig` or two) and set the environment variable `FULL_REPO_CONFIGS_MODULE` to point to that file. Then the core offline / online store tests can be run with `make test-python-universal`. * See the [custom offline store demo](https://github.com/feast-dev/feast-custom-offline-store-demo) and the [custom online store demo](https://github.com/feast-dev/feast-custom-online-store-demo) for examples. ### What are some important things to keep in mind when adding a new offline / online store? #### Type mapping/Inference Many problems arise when implementing your data store's type conversion to interface with Feast datatypes. 1. You will need to correctly update `inference.py` so that Feast can infer your datasource schemas 2. You also need to update `type_map.py` so that Feast knows how to convert your datastores types to Feast-recognized types in `feast/types.py`. #### Historical and online retrieval The most important functionality in Feast is historical and online retrieval. Most of the e2e and universal integration test test this functionality in some way. Making sure this functionality works also indirectly asserts that reading and writing from your datastore works as intended. ### To include a new offline / online store in the main Feast repo * Extend `data_source_creator.py` for your offline store. * In `repo_configuration.py` add a new `IntegrationTestRepoConfig` or two (depending on how many online stores you want to test). * Generally, you should only need to test against sqlite. However, if you need to test against a production online store, then you can also test against Redis or dynamodb. * Run the full test suite with `make test-python-integration.` ### Including a new offline / online store in the main Feast repo from external plugins with community maintainers. * This folder is for plugins that are officially maintained with community owners. Place the APIs in `feast/infra/offline_stores/contrib/`. * Extend `data_source_creator.py` for your offline store and implement the required APIs. * In `contrib_repo_configuration.py` add a new `IntegrationTestRepoConfig` (depending on how many online stores you want to test). * Run the test suite on the contrib test suite with `make test-python-contrib-universal`. ### To include a new online store * In `repo_configuration.py` add a new config that maps to a serialized version of configuration you need in `feature_store.yaml` to setup the online store. * In `repo_configuration.py`, add new `IntegrationTestRepoConfig` for online stores you want to test. * Run the full test suite with `make test-python-integration` ### To use custom data in a new test * Check `test_universal_types.py` for an example of how to do this. ```python @pytest.mark.integration def your_test(environment: Environment): df = #...# data_source = environment.data_source_creator.create_data_source( df, destination_name=environment.feature_store.project ) your_fv = driver_feature_view(data_source) entity = driver(value_type=ValueType.UNKNOWN) fs.apply([fv, entity]) # ... run test ``` ### Running your own Redis cluster for testing * Install Redis on your computer. If you are a mac user, you should be able to `brew install redis`. * Running `redis-server --help` and `redis-cli --help` should show corresponding help menus. * * Run `./infra/scripts/redis-cluster.sh start` then `./infra/scripts/redis-cluster.sh create` to start the Redis cluster locally. You should see output that looks like this: ``` Starting 6001 Starting 6002 Starting 6003 Starting 6004 Starting 6005 Starting 6006 ``` * You should be able to run the integration tests and have the Redis cluster tests pass. * If you would like to run your own Redis cluster, you can run the above commands with your own specified ports and connect to the newly configured cluster. * To stop the cluster, run `./infra/scripts/redis-cluster.sh stop` and then `./infra/scripts/redis-cluster.sh clean`. # Starting Feast servers in TLS(SSL) Mode TLS (Transport Layer Security) and SSL (Secure Sockets Layer) are both protocols encrypts communications between a client and server to provide enhanced security.TLS or SSL words used interchangeably. This article is going to show the sample code to start all the feast servers such as online server, offline server, registry server and UI server in TLS mode. Also show examples related to feast clients to communicate with the feast servers started in TLS mode. We assume you have basic understanding of feast terminology before going through this tutorial, if you are new to feast then we would recommend to go through existing [starter tutorials](https://github.com/feast-dev/feast/blob/master/examples/README.md) of feast. ## Obtaining a self-signed TLS certificate and key In development mode we can generate a self-signed certificate for testing. In an actual production environment it is always recommended to get it from a trusted TLS certificate provider. ```shell openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes ``` The above command will generate two files * `key.pem` : certificate private key * `cert.pem`: certificate public key You can use the public or private keys generated from above command in the rest of the sections in this tutorial. ## Create the feast demo repo for the rest of the sections. Create a feast repo and initialize using `feast init` and `feast apply` command and use this repo as a demo for subsequent sections. ```shell feast init feast_repo_ssl_demo #output will be something similar as below Creating a new Feast repository in /Documents/Src/feast/feast_repo_ssl_demo. cd feast_repo_ssl_demo/feature_repo feast apply #output will be something similar as below Applying changes for project feast_repo_ssl_demo Created project feast_repo_ssl_demo Created entity driver Created feature view driver_hourly_stats Created feature view driver_hourly_stats_fresh Created on demand feature view transformed_conv_rate Created on demand feature view transformed_conv_rate_fresh Created feature service driver_activity_v1 Created feature service driver_activity_v3 Created feature service driver_activity_v2 Created sqlite table feast_repo_ssl_demo_driver_hourly_stats_fresh Created sqlite table feast_repo_ssl_demo_driver_hourly_stats ``` You need to execute the feast cli commands from `feast_repo_ssl_demo/feature_repo` directory created from the above `feast init` command. ## Starting feast online server (feature server) in TLS mode To start the feature server in TLS mode, you need to provide the private and public keys using the `--key` and `--cert` arguments with the `feast serve` command. ```shell feast serve --key /path/to/key.pem --cert /path/to/cert.pem ``` You will see the output something similar to as below. Note the server url starts in the `https` mode. ```shell [2024-11-04 15:03:57 -0500] [77989] [INFO] Starting gunicorn 23.0.0 [2024-11-04 15:03:57 -0500] [77989] [INFO] Listening at: https://127.0.0.1:6566 (77989) [2024-11-04 15:03:57 -0500] [77989] [INFO] Using worker: uvicorn_worker.UvicornWorker [2024-11-04 15:03:57 -0500] [77992] [INFO] Booting worker with pid: 77992 [2024-11-04 15:03:57 -0500] [77992] [INFO] Started server process [77992] [2024-11-04 15:03:57 -0500] [77992] [INFO] Waiting for application startup. [2024-11-04 15:03:57 -0500] [77992] [INFO] Application startup complete. ``` ### Feast client connecting to remote online sever started in TLS mode. Sometimes you may need to pass the self-signed public key to connect to the remote online server started in SSL mode if you have not added the public key to the certificate store. feast client example: The registry is pointing to registry of remote feature store. If it is not accessible then should be configured to use remote registry. ```yaml project: feast-project registry: /remote/data/registry.db provider: local online_store: path: http://localhost:6566 type: remote cert: /path/to/cert.pem entity_key_serialization_version: 3 auth: type: no_auth ``` `cert` is an optional configuration to the public certificate path when the online server starts in TLS(SSL) mode. Typically, this file ends with `*.crt`, `*.cer`, or `*.pem`. ## Starting feast Registry server in TLS mode To start the feature server in TLS mode, you need to provide the private and public keys using the `--key` and `--cert` arguments with the `feast serve_registry` command. ```shell feast serve_registry --key /path/to/key.pem --cert /path/to/cert.pem ``` You will see the output something similar to as below. Note the server url starts in the `https` mode. ```shell 11/04/2024 03:10:27 PM feast.registry_server INFO: Starting grpc registry server in TLS(SSL) mode 11/04/2024 03:10:27 PM feast.registry_server INFO: Grpc server started at https://localhost:6570 ``` ### Feast client connecting to remote registry sever started in TLS mode. Sometimes you may need to pass the self-signed public key to connect to the remote registry server started in SSL mode if you have not added the public key to the certificate store. feast client example: ```yaml project: feast-project registry: registry_type: remote path: https://localhost:6570 cert: /path/to/cert.pem provider: local online_store: path: http://localhost:6566 type: remote cert: /path/to/cert.pem entity_key_serialization_version: 3 auth: type: no_auth ``` `cert` is an optional configuration to the public certificate path when the registry server starts in TLS(SSL) mode. Typically, this file ends with `*.crt`, `*.cer`, or `*.pem`. ### Feast client connecting to remote registry server with mTLS If the Registry Server requires mutual TLS (mTLS), the client must present a certificate and private key in addition to trusting the server's CA certificate. Add `client_cert` and `client_key` to the registry configuration: ```yaml project: feast-project registry: registry_type: remote path: feature-registry.example.com:443 cert: /path/to/ca.crt client_cert: /path/to/tls.crt client_key: /path/to/tls.key provider: local online_store: path: http://localhost:6566 type: remote entity_key_serialization_version: 3 auth: type: no_auth ``` * `cert` — CA certificate used to verify the server (or use the `SSL_CERT_FILE` / `REQUESTS_CA_BUNDLE` environment variable). * `client_cert` — Client certificate presented to the server. Must be paired with `client_key`. * `client_key` — Private key for the client certificate. #### Connecting through a tunnel or proxy When connecting through a tunnel (e.g. `gcloud compute start-iap-tunnel`) the client connects to `localhost`, but the server certificate is issued for the real service hostname. Set the `authority` field so that gRPC's TLS hostname verification passes: ```shell # In one terminal — start the tunnel: gcloud compute start-iap-tunnel feature-registry.example.com 443 --local-host-port=localhost:8443 ``` ```yaml registry: registry_type: remote path: localhost:8443 cert: /path/to/ca.crt client_cert: /path/to/tls.crt client_key: /path/to/tls.key authority: feature-registry.example.com ``` Without `authority`, the gRPC client would check the server certificate against `localhost`, which would fail because the certificate's Subject Alternative Name (SAN) is `feature-registry.example.com`. ## Starting feast offline server in TLS mode To start the offline server in TLS mode, you need to provide the private and public keys using the `--key` and `--cert` arguments with the `feast serve_offline` command. ```shell feast serve_offline --key /path/to/key.pem --cert /path/to/cert.pem ``` You will see the output something similar to as below. Note the server url starts in the `https` mode. ```shell 11/07/2024 11:10:01 AM feast.offline_server INFO: Found SSL certificates in the args so going to start offline server in TLS(SSL) mode. 11/07/2024 11:10:01 AM feast.offline_server INFO: Offline store server serving at: grpc+tls://127.0.0.1:8815 11/07/2024 11:10:01 AM feast.offline_server INFO: offline server starting with pid: [11606] ``` ### Feast client connecting to remote offline sever started in TLS mode. Sometimes you may need to pass the self-signed public key to connect to the remote registry server started in SSL mode if you have not added the public key to the certificate store. You have to add `scheme` to `https`. feast client example: ```yaml project: feast-project registry: registry_type: remote path: https://localhost:6570 cert: /path/to/cert.pem provider: local online_store: path: http://localhost:6566 type: remote cert: /path/to/cert.pem entity_key_serialization_version: 3 offline_store: type: remote host: localhost port: 8815 scheme: https cert: /path/to/cert.pem auth: type: no_auth ``` `cert` is an optional configuration to the public certificate path when the registry server starts in TLS(SSL) mode. Typically, this file ends with `*.crt`, `*.cer`, or `*.pem`. `scheme` should be `https`. By default, it will be `http` so you have to explicitly configure to `https` if you are planning to connect to remote offline server which is started in TLS mode. ## Starting feast UI server (react app) in TLS mode To start the feast UI server in TLS mode, you need to provide the private and public keys using the `--key` and `--cert` arguments with the `feast ui` command. ```shell feast ui --key /path/to/key.pem --cert /path/to/cert.pem ``` You will see the output something similar to as below. Note the server url starts in the `https` mode. ```shell INFO: Started server process [78872] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on https://0.0.0.0:8888 (Press CTRL+C to quit) ``` ## Adding public key to CA trust store and configuring the feast to use the trust store. You can pass the public key for SSL verification using the `cert` parameter, however, it is sometimes difficult to maintain individual certificates and pass them individually. The alternative recommendation is to add the public certificate to CA trust store and set the path as an environment variable (e.g., `FEAST_CA_CERT_FILE_PATH`). Feast will use the trust store path in the `FEAST_CA_CERT_FILE_PATH` environment variable. # Importing Features from dbt {% hint style="warning" %} **Alpha Feature**: The dbt integration is currently in early development and subject to change. **Current Limitations**: * Supported data sources: BigQuery, Snowflake, and File-based sources only * Manual entity column specification required Breaking changes may occur in future releases. {% endhint %} This guide explains how to use Feast's dbt integration to automatically import dbt models as Feast FeatureViews. This enables you to leverage your existing dbt transformations as feature definitions without manual duplication. ## Overview [dbt (data build tool)](https://www.getdbt.com/) is a popular tool for transforming data in your warehouse. Many teams already use dbt to create feature tables. Feast's dbt integration allows you to: * **Discover** dbt models tagged for feature engineering * **Import** model metadata (columns, types, descriptions) as Feast objects * **Generate** Python code for Entity, DataSource, and FeatureView definitions This eliminates the need to manually define Feast objects that mirror your dbt models. ## Prerequisites * A dbt project with compiled artifacts (`target/manifest.json`) * Feast installed with dbt support: ```bash pip install 'feast[dbt]' ``` Or install the parser directly: ```bash pip install dbt-artifacts-parser ``` ## Quick Start ### 1. Tag your dbt models In your dbt project, add a `feast` tag to models you want to import: {% code title="models/driver\_features.sql" %} ```sql {{ config( materialized='table', tags=['feast'] ) }} SELECT driver_id, event_timestamp, avg_rating, total_trips, is_active FROM {{ ref('stg_drivers') }} ``` {% endcode %} ### 2. Define column types in schema.yml Feast uses column metadata from your `schema.yml` to determine feature types: {% code title="models/schema.yml" %} ```yaml version: 2 models: - name: driver_features description: "Driver aggregated features for ML models" columns: - name: driver_id description: "Unique driver identifier" data_type: STRING - name: event_timestamp description: "Feature timestamp" data_type: TIMESTAMP - name: avg_rating description: "Average driver rating" data_type: FLOAT64 - name: total_trips description: "Total completed trips" data_type: INT64 - name: is_active description: "Whether driver is currently active" data_type: BOOLEAN ``` {% endcode %} ### 3. Compile your dbt project ```bash cd your_dbt_project dbt compile ``` This generates `target/manifest.json` which Feast will read. ### 4. List available models Use the Feast CLI to discover tagged models: ```bash feast dbt list target/manifest.json --tag-filter feast ``` Output: ``` Found 1 model(s) with tag 'feast': driver_features Description: Driver aggregated features for ML models Columns: driver_id, event_timestamp, avg_rating, total_trips, is_active Tags: feast ``` ### 5. Import models as Feast definitions Generate a Python file with Feast object definitions: ```bash feast dbt import target/manifest.json \ --entity-column driver_id \ --data-source-type bigquery \ --tag-filter feast \ --output features/driver_features.py ``` This generates: {% code title="features/driver\_features.py" %} ```python """ Feast feature definitions generated from dbt models. Source: target/manifest.json Project: my_dbt_project Generated by: feast dbt import """ from datetime import timedelta from feast import Entity, FeatureView, Field from feast.types import Bool, Float64, Int64 from feast.infra.offline_stores.bigquery_source import BigQuerySource # Entities driver_id = Entity( name="driver_id", join_keys=["driver_id"], description="Entity key for dbt models", tags={'source': 'dbt'}, ) # Data Sources driver_features_source = BigQuerySource( name="driver_features_source", table="my_project.my_dataset.driver_features", timestamp_field="event_timestamp", description="Driver aggregated features for ML models", tags={'dbt.model': 'driver_features', 'dbt.tag.feast': 'true'}, ) # Feature Views driver_features_fv = FeatureView( name="driver_features", entities=[driver_id], ttl=timedelta(days=1), schema=[ Field(name="avg_rating", dtype=Float64, description="Average driver rating"), Field(name="total_trips", dtype=Int64, description="Total completed trips"), Field(name="is_active", dtype=Bool, description="Whether driver is currently active"), ], online=True, source=driver_features_source, description="Driver aggregated features for ML models", tags={'dbt.model': 'driver_features', 'dbt.tag.feast': 'true'}, ) ``` {% endcode %} ## Multiple Entity Support The dbt integration supports feature views with multiple entities, useful for modeling relationships involving multiple keys. ### Usage Specify multiple entity columns using repeated `-e` flags: ```bash feast dbt import \ -m target/manifest.json \ -e user_id \ -e merchant_id \ --tag feast \ -o features/transactions.py ``` This creates a FeatureView with both `user_id` and `merchant_id` as entities, useful for: * Transaction features keyed by both user and merchant * Interaction features keyed by multiple parties * Association tables in many-to-many relationships Single entity usage: ```bash feast dbt import -m target/manifest.json -e driver_id --tag feast ``` ### Requirements All specified entity columns must exist in each dbt model being imported. Models missing any entity column will be skipped with a warning. ### Generated Code The `--output` flag generates code like: ```python user_id = Entity(name="user_id", join_keys=["user_id"], ...) merchant_id = Entity(name="merchant_id", join_keys=["merchant_id"], ...) transaction_fv = FeatureView( name="transactions", entities=[user_id, merchant_id], # Multiple entities schema=[...], ... ) ``` ## CLI Reference ### `feast dbt list` Discover dbt models available for import. ```bash feast dbt list [OPTIONS] ``` **Arguments:** * `manifest_path`: Path to dbt's `manifest.json` file **Options:** * `--tag-filter`, `-t`: Filter models by dbt tag (e.g., `feast`) * `--model`, `-m`: Filter to specific model name(s) ### `feast dbt import` Import dbt models as Feast object definitions. ```bash feast dbt import [OPTIONS] ``` **Arguments:** * `manifest_path`: Path to dbt's `manifest.json` file **Options:** | Option | Description | Default | | -------------------------- | ---------------------------------------------------- | ----------------- | | `--entity-column`, `-e` | Entity column name (can be specified multiple times) | (required) | | `--data-source-type`, `-d` | Data source type: `bigquery`, `snowflake`, `file` | `bigquery` | | `--tag-filter`, `-t` | Filter models by dbt tag | None | | `--model`, `-m` | Import specific model(s) only | None | | `--timestamp-field` | Timestamp column name | `event_timestamp` | | `--ttl-days` | Feature TTL in days | `1` | | `--exclude-columns` | Columns to exclude from features | None | | `--no-online` | Disable online serving | `False` | | `--output`, `-o` | Output Python file path | None (stdout) | | `--dry-run` | Preview without generating code | `False` | ## Type Mapping Feast automatically maps dbt/warehouse column types to Feast types: | dbt/SQL Type | Feast Type | | --------------------------- | --------------------------------------- | | `STRING`, `VARCHAR`, `TEXT` | `String` | | `INT`, `INTEGER`, `BIGINT` | `Int64` | | `SMALLINT`, `TINYINT` | `Int32` | | `FLOAT`, `REAL` | `Float32` | | `DOUBLE`, `FLOAT64` | `Float64` | | `BOOLEAN`, `BOOL` | `Bool` | | `TIMESTAMP`, `DATETIME` | `UnixTimestamp` | | `BYTES`, `BINARY` | `Bytes` | | `ARRAY` | `Array(type)` | | `JSON`, `JSONB` | `Map` (or `Json` if declared in schema) | | `VARIANT`, `OBJECT` | `Map` | | `SUPER` | `Map` | | `MAP` | `Map` | | `STRUCT`, `RECORD` | `Struct` (BigQuery) | | `struct<...>` | `Struct` (Spark) | Snowflake `NUMBER(precision, scale)` types are handled specially: * Scale > 0: `Float64` * Precision <= 9: `Int32` * Precision <= 18: `Int64` * Precision > 18: `Float64` ## Data Source Configuration ### BigQuery ```bash feast dbt import manifest.json -e user_id -d bigquery -o features.py ``` Generates `BigQuerySource` with the full table path from dbt metadata: ```python BigQuerySource( table="project.dataset.table_name", ... ) ``` ### Snowflake ```bash feast dbt import manifest.json -e user_id -d snowflake -o features.py ``` Generates `SnowflakeSource` with database, schema, and table: ```python SnowflakeSource( database="MY_DB", schema="MY_SCHEMA", table="TABLE_NAME", ... ) ``` ### File ```bash feast dbt import manifest.json -e user_id -d file -o features.py ``` Generates `FileSource` with a placeholder path: ```python FileSource( path="/data/table_name.parquet", ... ) ``` {% hint style="info" %} For file sources, update the generated path to point to your actual data files. {% endhint %} ## Best Practices ### 1. Use consistent tagging Create a standard tagging convention in your dbt project: ```yaml # dbt_project.yml models: my_project: features: +tags: ['feast'] # All models in features/ get the feast tag ``` ### 2. Document your columns Column descriptions from `schema.yml` are preserved in the generated Feast definitions, making your feature catalog self-documenting. ### 3. Review before committing Use `--dry-run` to preview what will be generated: ```bash feast dbt import manifest.json -e user_id -d bigquery --dry-run ``` ### 4. Version control generated code Commit the generated Python files to your repository. This allows you to: * Track changes to feature definitions over time * Review dbt-to-Feast mapping in pull requests * Customize generated code if needed ### 5. Integrate with CI/CD Add dbt import to your CI pipeline: ```yaml # .github/workflows/features.yml - name: Compile dbt run: dbt compile - name: Generate Feast definitions run: | feast dbt import target/manifest.json \ -e user_id -d bigquery -t feast \ -o feature_repo/features.py - name: Apply Feast changes run: feast apply ``` ## Limitations * **Single entity support**: Currently supports one entity column per import. For multi-entity models, run multiple imports or manually adjust the generated code. * **No incremental updates**: Each import generates a complete file. Use version control to track changes. * **Column types required**: Models without `data_type` in schema.yml default to `String` type. ## Troubleshooting ### "manifest.json not found" Run `dbt compile` or `dbt run` first to generate the manifest file. ### "No models found with tag" Check that your models have the correct tag in their config: ```sql {{ config(tags=['feast']) }} ``` ### "Missing entity column" Ensure your dbt model includes the entity column specified with `--entity-column`. Models missing this column are skipped with a warning. ### "Missing timestamp column" By default, Feast looks for `event_timestamp`. Use `--timestamp-field` to specify a different column name. # Entity Key Serialization (v2 to v3) Entity Key Serialization version 2 is now deprecated. All new and existing Feast deployments should shift to using serialization version 3. However, a challenge arises when existing FeatureViews in online or offline stores have written features with version 2. A version 2 serialized entity key cannot be retrieved using the version 3 deserialization algorithm. ## Reserialize the Feature Views entity Keys to version 3 The solution is to reserialize the entity keys from version 2 to version 3. Follow the following procedures to reserialize the entity key to version 3 in feature View in an offline / online store. In short, you need to iterate through all the feature views in your Feast repository, retrieve their serialized entity keys (if stored in version 2), reserialize them to version 3, and then update the online/offline store or wherever the serialized keys are stored. ### 1. Initialize the Feature Store ``` Load the FeatureStore object to access all feature views in your repository. ``` ### 2. Iterate Through Feature Views ``` Use the list_feature_views() method to retrieve all feature views in the repository. ``` ### 3. Retrieve Serialized Entity Keys ``` For each feature view, retrieve the serialized entity keys stored in the online/offline store or other storage ``` ### 4. Reserialize Entity Keys ``` Use the reserialize_entity_v2_key_to_v3 function to convert the serialized keys from version 2 to version 3. Use [entity key encoding utils](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/infra/key_encoding_utils.py) function `reserialize_entity_v2_key_to_v3`. ``` ### 5. Update the Online/offline Store ``` Write the reserialized keys back to the online/offline store or the appropriate storage ``` # Feature Quality Monitoring ## Overview Feast's data quality monitoring system computes, stores, and serves statistical metrics for every registered feature. It gives you visibility into feature health — distributions, null rates, percentiles, histograms — across batch data and feature serving logs. This guide covers: 1. [Prerequisites](#1-prerequisites) 2. [Auto-baseline on registration](#2-auto-baseline-on-registration) 3. [Scheduled monitoring with the CLI](#3-scheduled-monitoring-with-the-cli) 4. [Monitoring feature serving logs](#4-monitoring-feature-serving-logs) 5. [Reading metrics via REST API](#5-reading-metrics-via-rest-api) 6. [On-demand exploration (transient compute)](#6-on-demand-exploration) 7. [Integrating with orchestrators](#7-integrating-with-orchestrators) 8. [Supported backends](#8-supported-backends) 9. [Monitoring in the Feast UI](#9-monitoring-in-the-feast-ui) ## 1. Prerequisites Monitoring works with any supported offline store backend. No additional infrastructure or configuration is needed — monitoring tables are created automatically on first use. **Minimum setup:** * A Feast project with at least one feature view and a configured offline store * Feast SDK installed (`pip install feast`) **For serving log monitoring:** * At least one feature service with `logging_config` set (see [step 4](#4-monitoring-feature-serving-logs)) ## 2. Auto-baseline on registration When you run `feast apply` to register new features, Feast automatically queues baseline metric computation: ```bash $ feast apply Applying changes... Created feature view 'driver_stats' with 3 features → Queued baseline metrics computation (DQM job: abc-123) Done! ``` The baseline reads all available source data and stores the resulting statistics with `is_baseline=TRUE`. This serves as the reference distribution for future drift detection. Baseline computation is: * **Threaded** — runs in a background thread but completes before `feast apply` exits * **Idempotent** — only features without existing baselines are computed; re-running `feast apply` won't recompute existing baselines ### Enabling auto-baseline To enable automatic baseline computation on `feast apply`, set the DQM config in `feature_store.yaml`: ```yaml data_quality_monitoring: auto_baseline: true ``` When using the Feast operator, set this in the `FeatureStore` CR: ```yaml apiVersion: feast.dev/v1 kind: FeatureStore spec: feastProject: my_project dataQualityMonitoring: autoBaseline: true ``` To disable it, set `auto_baseline: false` (or `autoBaseline: false` in the CR). ## 3. Scheduled monitoring with the CLI ### Auto mode (recommended for production) Schedule a single daily job that computes all granularities automatically: ```bash feast monitor run ``` This detects the latest event timestamp in the source data and computes metrics for 5 time windows: | Granularity | Window | | ----------- | ------------ | | `daily` | Last 1 day | | `weekly` | Last 7 days | | `biweekly` | Last 14 days | | `monthly` | Last 30 days | | `quarterly` | Last 90 days | No date arguments needed. One scheduled job produces all granularities. ### Targeting a specific feature view ```bash feast monitor run --feature-view driver_stats ``` ### Explicit date range and granularity ```bash feast monitor run \ --feature-view driver_stats \ --start-date 2025-01-01 \ --end-date 2025-01-07 \ --granularity weekly ``` ### Setting a manual baseline ```bash feast monitor run \ --feature-view driver_stats \ --start-date 2025-01-01 \ --end-date 2025-03-31 \ --granularity daily \ --set-baseline ``` ### CLI reference ``` Usage: feast monitor run [OPTIONS] Options: -p, --project TEXT Feast project name (defaults to feature_store.yaml) -v, --feature-view TEXT Feature view name (omit for all) -f, --feature-name TEXT Feature name(s), repeatable (omit for all) --start-date TEXT Start date YYYY-MM-DD (omit for auto-detect) --end-date TEXT End date YYYY-MM-DD (omit for auto-detect) -g, --granularity One of: daily, weekly, biweekly, monthly, quarterly --set-baseline Mark this computation as baseline --source-type One of: batch, log, all (default: batch) --help Show this message and exit. ``` ## 4. Monitoring feature serving logs If your feature services have logging configured, you can compute metrics from the actual features served to models in production. ### Setting up feature service logging In your feature definitions: ```python from feast import FeatureService, LoggingConfig from feast.infra.offline_stores.contrib.postgres_offline_store.postgres_source import ( PostgreSQLLoggingDestination, ) driver_service = FeatureService( name="driver_service", features=[driver_stats_fv], logging_config=LoggingConfig( destination=PostgreSQLLoggingDestination(table_name="feast_driver_logs"), sample_rate=1.0, ), ) ``` ### Computing log metrics **Auto mode (all feature services with logging):** ```bash feast monitor run --source-type log ``` **Specific feature service:** ```bash feast monitor run --source-type log --feature-view driver_service ``` **Both batch and log in one run:** ```bash feast monitor run --source-type all ``` Log metrics are stored with `data_source_type="log"` alongside batch metrics in the same monitoring tables. Feature names from the log schema (e.g., `driver_stats__conv_rate`) are automatically normalized back to their original names (`conv_rate`) and associated with the correct feature view — enabling batch-vs-log comparison and drift detection. ### Via REST API ```bash # Compute log metrics POST /monitoring/compute/log { "project": "my_project", "feature_service_name": "driver_service", "granularity": "daily" } # Auto-compute all log metrics POST /monitoring/auto_compute/log { "project": "my_project" } ``` ## 5. Reading metrics via REST API All read endpoints support cascading filters: `project` → `feature_service_name` → `feature_view_name` → `feature_name` → `granularity` → `data_source_type`. ### Per-feature metrics ``` GET /monitoring/metrics/features?project=my_project&feature_view_name=driver_stats&granularity=daily ``` **Response:** ```json [ { "project_id": "my_project", "feature_view_name": "driver_stats", "feature_name": "conv_rate", "feature_type": "numeric", "metric_date": "2025-03-26", "granularity": "daily", "data_source_type": "batch", "row_count": 15000, "null_count": 12, "null_rate": 0.0008, "mean": 0.523, "stddev": 0.189, "min_val": 0.001, "max_val": 0.998, "p50": 0.51, "p75": 0.68, "p90": 0.82, "p95": 0.89, "p99": 0.96, "histogram": { "bins": [0.0, 0.05, 0.1, "..."], "counts": [120, 340, 560, "..."], "bin_width": 0.05 } } ] ``` ### Per-feature-view aggregates ``` GET /monitoring/metrics/feature_views?project=my_project&feature_view_name=driver_stats ``` ### Per-feature-service aggregates ``` GET /monitoring/metrics/feature_services?project=my_project&feature_service_name=driver_service ``` ### Baseline ``` GET /monitoring/metrics/baseline?project=my_project&feature_view_name=driver_stats ``` ### Time-series (for trend charts) ``` GET /monitoring/metrics/timeseries?project=my_project&feature_name=conv_rate&granularity=daily&start_date=2025-01-01&end_date=2025-03-31 ``` ### Filtering batch vs. log metrics Add `data_source_type=batch` or `data_source_type=log` to any read endpoint: ``` GET /monitoring/metrics/features?project=my_project&data_source_type=log ``` ### Full endpoint reference | Method | Endpoint | Description | | ------ | -------------------------------------- | ---------------------------------------- | | `POST` | `/monitoring/compute` | Submit batch DQM job | | `POST` | `/monitoring/auto_compute` | Auto-detect dates, all granularities | | `POST` | `/monitoring/compute/transient` | On-demand compute (not stored) | | `POST` | `/monitoring/compute/log` | Compute from serving logs | | `POST` | `/monitoring/auto_compute/log` | Auto-detect log dates, all granularities | | `GET` | `/monitoring/jobs/{job_id}` | DQM job status | | `GET` | `/monitoring/metrics/features` | Per-feature metrics | | `GET` | `/monitoring/metrics/feature_views` | Per-view aggregates | | `GET` | `/monitoring/metrics/feature_services` | Per-service aggregates | | `GET` | `/monitoring/metrics/baseline` | Baseline metrics | | `GET` | `/monitoring/metrics/timeseries` | Time-series data | ## 6. On-demand exploration When you need metrics for an arbitrary date range (e.g., "show me the distribution for Jan 5 to Jan 20"), use the transient compute endpoint. It reads source data for the exact range, computes fresh statistics, and returns them directly without storing. ```bash POST /monitoring/compute/transient { "project": "my_project", "feature_view_name": "driver_stats", "feature_names": ["conv_rate"], "start_date": "2025-01-05", "end_date": "2025-01-20" } ``` This is necessary because pre-computed histograms from different date ranges have different bin edges and cannot be merged losslessly. ## 7. Integrating with orchestrators ### Airflow ```python from airflow.operators.bash import BashOperator monitor_task = BashOperator( task_id="feast_monitor", bash_command="feast monitor run", cwd="/path/to/feast/repo", ) ``` ### Kubeflow Pipelines (KFP) ```python from kfp import dsl @dsl.component(base_image="feast-image:latest") def monitor_features(): import subprocess subprocess.run(["feast", "monitor", "run"], check=True, cwd="/feast/repo") ``` ### Cron ```cron # Daily at 2:00 AM UTC 0 2 * * * cd /path/to/feast/repo && feast monitor run >> /var/log/feast-monitor.log 2>&1 ``` ### Monitoring both batch and log in one job ```bash feast monitor run --source-type all ``` ## 8. Supported backends Monitoring works natively with all offline stores that serve as compute engines for Feast materialization: | Backend | Compute | Storage | | ---------- | ------------------ | --------------------------- | | PostgreSQL | SQL push-down | `INSERT ON CONFLICT` | | Snowflake | SQL push-down | `MERGE` with `VARIANT` JSON | | BigQuery | SQL push-down | `MERGE` into BQ tables | | Redshift | SQL push-down | `MERGE` via Data API | | Spark | SparkSQL push-down | Parquet tables | | Oracle | SQL via Ibis | `MERGE` from `DUAL` | | DuckDB | In-memory SQL | Parquet files | | Dask | PyArrow compute | Parquet files | Backends not listed above fall back to Python-based computation — the offline store's `pull_all_from_table_or_query()` returns a PyArrow Table, and metrics are computed using `pyarrow.compute` and `numpy`. ## What metrics are computed **Per-feature (full profile):** | Metric | Numeric | Categorical | | ----------------------------------- | :-----------------: | :----------------------: | | row\_count, null\_count, null\_rate | Yes | Yes | | mean, stddev, min, max | Yes | — | | p50, p75, p90, p95, p99 | Yes | — | | histogram (JSONB) | Binned distribution | Top-N values with counts | **Per-feature-view and per-feature-service (aggregate summaries):** | Metric | Description | | -------------------------------- | -------------------------------- | | total\_row\_count | Total rows in the view | | total\_features | Number of features | | features\_with\_nulls | Count of features with any nulls | | avg\_null\_rate, max\_null\_rate | Aggregate null rate statistics | ## RBAC Monitoring respects Feast's existing RBAC: * **Compute operations** (`POST /monitoring/compute`, `/auto_compute`, `/compute/log`, `/auto_compute/log`) require `AuthzedAction.UPDATE` * **Transient compute** (`POST /monitoring/compute/transient`) requires `AuthzedAction.DESCRIBE` * **Read operations** (`GET /monitoring/metrics/*`) require `AuthzedAction.DESCRIBE` ## 9. Monitoring in the Feast UI The Feast web UI includes a built-in monitoring dashboard accessible from the **Monitoring** item in the sidebar navigation. ### What you see The monitoring page has three tabs: | Tab | Shows | | -------------------- | --------------------------------------------------------------------------------- | | **Features** | Per-feature metrics table with null rate, row count, freshness, and health status | | **Feature Views** | Aggregated data quality per feature view | | **Feature Services** | Aggregated metrics per feature service | ### Filters At the top of the monitoring page you can filter by: * **Feature View** — scope to a specific feature view or view all * **Granularity** — select Baseline, Daily, Weekly, Biweekly, Monthly, or Quarterly * **Source** — filter by batch or log data source * **Start/End Date** — filter metrics to a specific date range (disabled for Baseline since baseline uses all data) ### Feature detail page Clicking any feature row navigates to a detail page showing: * **Distribution histogram** — expandable/zoomable chart of the feature's value distribution * **Statistics panel** — null rate, mean, stddev, min/max, percentiles (p50–p99) * **Granularity dropdown** — switch between computed granularities and baseline * **Time Series Analysis** — trend charts for aggregate metrics drift (Mean/P50/P95) and null rate evolution over time ### Computing metrics from the UI Click the **Compute Metrics** button in the page header to trigger an `auto_compute` job. This computes all granularities for all feature views (or the selected feature view if filtered). Results appear after the table refreshes. The **Refresh** button re-fetches already computed metrics from the backend without triggering new computation. ### When no data is available If no metrics have been computed yet, the page shows a prompt: > No monitoring data has been computed for this project. Click "Compute Metrics" to run data quality analysis on your feature views. If the monitoring backend is unreachable, a warning banner appears: > Could not connect to the monitoring API. Make sure the Feast registry server is running with monitoring enabled. ### Enabling monitoring for the UI The monitoring page is always accessible in the sidebar. To see actual data: 1. Add `data_quality_monitoring` to your `feature_store.yaml`: ```yaml data_quality_monitoring: auto_baseline: true ``` Or, when using the Feast operator, set this in the `FeatureStore` CR: ```yaml apiVersion: feast.dev/v1 kind: FeatureStore spec: feastProject: my_project dataQualityMonitoring: autoBaseline: true ``` 2. Run `feast apply` — this computes baseline metrics automatically 3. Schedule `feast monitor run` (or click "Compute Metrics" in the UI) to generate daily/weekly/monthly metrics ## Related: Operational and SOX Metrics Feature Quality Monitoring focuses on **data-level** metrics (distributions, null rates, drift). Feast also provides **operational metrics** for infrastructure observability: * **Prometheus metrics** (`feast_offline_store_*`, `feast_online_store_*`) — latency, throughput, and error rates for offline/online store operations. See [Python Feature Server — Metrics](/master/reference/feature-servers/python-feature-server). * **SOX audit logging** (`feast.audit`) — structured audit events for compliance tracking of feature store operations. * **OpenTelemetry integration** — distributed tracing for feature serving requests. See [OpenTelemetry Integration](/master/getting-started/components/open-telemetry). # Codebase Structure Let's examine the Feast codebase. This analysis is accurate as of Feast 0.23. ``` $ tree -L 1 -d . ├── docs ├── examples ├── go ├── infra ├── java ├── protos ├── sdk └── ui ``` ## Python SDK The Python SDK lives in `sdk/python/feast`. The majority of Feast logic lives in these Python files: * The core Feast objects ([entities](/master/getting-started/concepts/entity), [feature views](/master/getting-started/concepts/feature-view), [data sources](/master/getting-started/concepts/dataset), etc.) are defined in their respective Python files, such as `entity.py`, `feature_view.py`, and `data_source.py`. * The `FeatureStore` class is defined in `feature_store.py` and the associated configuration object (the Python representation of the `feature_store.yaml` file) are defined in `repo_config.py`. * The CLI and other core feature store logic are defined in `cli.py` and `repo_operations.py`. * The type system that is used to manage conversion between Feast types and external typing systems is managed in `type_map.py`. * The Python feature server (the server that is started through the `feast serve` command) is defined in `feature_server.py`. There are also several important submodules: * `infra/` contains all the infrastructure components, such as the provider, offline store, online store, batch materialization engine, and registry. * `dqm/` covers data quality monitoring. See [`monitoring/`](https://github.com/feast-dev/feast/tree/master/sdk/python/feast/monitoring/README.md) for the built-in monitoring system. * `diff/` covers the logic for determining how to apply infrastructure changes upon feature repo changes (e.g. the output of `feast plan` and `feast apply`). * `embedded_go/` covers the Go feature server. * `ui/` contains the embedded Web UI, to be launched on the `feast ui` command. Of these submodules, `infra/` is the most important. It contains the interfaces for the [provider](https://github.com/feast-dev/feast/tree/master/docs/reference/getting-started/components/provider.md), [offline store](https://github.com/feast-dev/feast/tree/master/docs/reference/getting-started/components/offline-store.md), [online store](https://github.com/feast-dev/feast/tree/master/docs/reference/getting-started/components/online-store.md), [compute engine](https://github.com/feast-dev/feast/tree/master/docs/reference/getting-started/components/compute-engine.md), and [registry](https://github.com/feast-dev/feast/tree/master/docs/reference/getting-started/components/registry.md), as well as all of their individual implementations. ``` $ tree --dirsfirst -L 1 infra infra ├── contrib ├── feature_servers ├── materialization ├── offline_stores ├── online_stores ├── registry ├── transformation_servers ├── utils ├── __init__.py ├── aws.py ├── gcp.py ├── infra_object.py ├── key_encoding_utils.py ├── local.py ├── passthrough_provider.py └── provider.py ``` The tests for the Python SDK are contained in `sdk/python/tests`. For more details, see this [overview](/master/how-to-guides/adding-or-reusing-tests#test-suite-overview) of the test suite. ### Example flow: `feast apply` Let's walk through how `feast apply` works by tracking its execution across the codebase. 1. All CLI commands are in `cli.py`. Most of these commands are backed by methods in `repo_operations.py`. The `feast apply` command triggers `apply_total_command`, which then calls `apply_total` in `repo_operations.py`. 2. With a `FeatureStore` object (from `feature_store.py`) that is initialized based on the `feature_store.yaml` in the current working directory, `apply_total` first parses the feature repo with `parse_repo` and then calls either `FeatureStore.apply` or `FeatureStore._apply_diffs` to apply those changes to the feature store. 3. Let's examine `FeatureStore.apply`. It splits the objects based on class (e.g. `Entity`, `FeatureView`, etc.) and then calls the appropriate registry method to apply or delete the object. For example, it might call `self._registry.apply_entity` to apply an entity. If the default file-based registry is used, this logic can be found in `infra/registry/registry.py`. 4. Then the feature store must update its cloud infrastructure (e.g. online store tables) to match the new feature repo, so it calls `Provider.update_infra`, which can be found in `infra/provider.py`. 5. Assuming the provider is a built-in provider (e.g. one of the local, GCP, or AWS providers), it will call `PassthroughProvider.update_infra` in `infra/passthrough_provider.py`. 6. This delegates to the online store and batch materialization engine. For example, if the feature store is configured to use the Redis online store then the `update` method from `infra/online_stores/redis.py` will be called. And if the local materialization engine is configured then the `update` method from `infra/materialization/local_engine.py` will be called. At this point, the `feast apply` command is complete. ### Example flow: `feast materialize` Let's walk through how `feast materialize` works by tracking its execution across the codebase. 1. The `feast materialize` command triggers `materialize_command` in `cli.py`, which then calls `FeatureStore.materialize` from `feature_store.py`. 2. This then calls `Provider.materialize_single_feature_view`, which can be found in `infra/provider.py`. 3. As with `feast apply`, the provider is most likely backed by the passthrough provider, in which case `PassthroughProvider.materialize_single_feature_view` will be called. 4. This delegates to the underlying batch materialization engine. Assuming that the local engine has been configured, `LocalMaterializationEngine.materialize` from `infra/materialization/local_engine.py` will be called. 5. Since materialization involves reading features from the offline store and writing them to the online store, the local engine will delegate to both the offline store and online store. Specifically, it will call `OfflineStore.pull_latest_from_table_or_query` and `OnlineStore.online_write_batch`. These two calls will be routed to the offline store and online store that have been configured. ### Example flow: `get_historical_features` Let's walk through how `get_historical_features` works by tracking its execution across the codebase. 1. We start with `FeatureStore.get_historical_features` in `feature_store.py`. This method does some internal preparation, and then delegates the actual execution to the underlying provider by calling `Provider.get_historical_features`, which can be found in `infra/provider.py`. 2. As with `feast apply`, the provider is most likely backed by the passthrough provider, in which case `PassthroughProvider.get_historical_features` will be called. 3. That call simply delegates to `OfflineStore.get_historical_features`. So if the feature store is configured to use Snowflake as the offline store, `SnowflakeOfflineStore.get_historical_features` will be executed. ## Java SDK The `java/` directory contains the Java serving component. See [here](https://github.com/feast-dev/feast/blob/master/java/CONTRIBUTING.md) for more details on how the repo is structured. ## Go feature server The `go/` directory contains the Go feature server. Most of the files here have logic to help with reading features from the online store. Within `go/`, the `internal/feast/` directory contains most of the core logic: * `onlineserving/` covers the core serving logic. * `model/` contains the implementations of the Feast objects (entity, feature view, etc.). * For example, `entity.go` is the Go equivalent of `entity.py`. It contains a very simple Go implementation of the entity object. * `registry/` covers the registry. * Currently only the file-based registry supported (the sql-based registry is unsupported). Additionally, the file-based registry only supports a file-based registry store, not the GCS or S3 registry stores. * `onlinestore/` covers the online stores (currently only Redis and SQLite are supported). ## Protobufs Feast uses [protobuf](https://github.com/protocolbuffers/protobuf) to store serialized versions of the core Feast objects. The protobuf definitions are stored in `protos/feast`. The [registry](https://github.com/feast-dev/feast/tree/master/docs/getting-started/concepts/registry.md) consists of the serialized representations of the Feast objects. Typically, changes being made to the Feast objects require changes to their corresponding protobuf representations. The usual best practices for making changes to protobufs should be followed ensure backwards and forwards compatibility. ## Web UI The `ui/` directory contains the Web UI. See [here](https://github.com/feast-dev/feast/blob/master/ui/CONTRIBUTING.md) for more details on the structure of the Web UI. # Type System ## Motivation Feast uses an internal type system to provide guarantees on training and serving data. Feast supports primitive types, array types, set types, map types, JSON, and struct types for feature values. Null types are not supported, although the `UNIX_TIMESTAMP` type is nullable. The type system is controlled by [`Value.proto`](https://github.com/feast-dev/feast/blob/master/protos/feast/types/Value.proto) in protobuf and by [`types.py`](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/types.py) in Python. Type conversion logic can be found in [`type_map.py`](https://github.com/feast-dev/feast/blob/master/sdk/python/feast/type_map.py). ## Supported Types Feast supports the following data types: ### Primitive Types | Feast Type | Python Type | Description | | ---------------- | ----------------- | ------------------------------------------------------------- | | `Int32` | `int` | 32-bit signed integer | | `Int64` | `int` | 64-bit signed integer | | `Float32` | `float` | 32-bit floating point | | `Float64` | `float` | 64-bit floating point | | `String` | `str` | String/text value | | `Bytes` | `bytes` | Binary data | | `Bool` | `bool` | Boolean value | | `UnixTimestamp` | `datetime` | Unix timestamp (nullable) | | `ZonedTimestamp` | `datetime` | Timezone-aware datetime preserving its source zone (nullable) | | `Uuid` | `uuid.UUID` | UUID (any version) | | `TimeUuid` | `uuid.UUID` | Time-based UUID (version 1) | | `Decimal` | `decimal.Decimal` | Arbitrary-precision decimal number | ### Domain-Specific Primitive Types These types are semantic aliases over `Bytes` for domain-specific use cases (e.g., RAG pipelines, image processing). They are stored as `bytes` at the proto level. | Feast Type | Python Type | Description | | ------------ | ----------- | ---------------------------------------------------------------------- | | `PdfBytes` | `bytes` | PDF document binary data (used in RAG / document processing pipelines) | | `ImageBytes` | `bytes` | Image binary data (used in image processing / multimodal pipelines) | {% hint style="warning" %} `PdfBytes` and `ImageBytes` are not natively supported by any backend's type inference. You must explicitly declare them in your feature view schema. Backend storage treats them as raw `bytes`. {% endhint %} ### Array Types All primitive types have corresponding array (list) types: | Feast Type | Python Type | Description | | ---------------------- | ----------------------- | ------------------------------------ | | `Array(Int32)` | `List[int]` | List of 32-bit integers | | `Array(Int64)` | `List[int]` | List of 64-bit integers | | `Array(Float32)` | `List[float]` | List of 32-bit floats | | `Array(Float64)` | `List[float]` | List of 64-bit floats | | `Array(String)` | `List[str]` | List of strings | | `Array(Bytes)` | `List[bytes]` | List of binary data | | `Array(Bool)` | `List[bool]` | List of booleans | | `Array(UnixTimestamp)` | `List[datetime]` | List of timestamps | | `Array(Uuid)` | `List[uuid.UUID]` | List of UUIDs | | `Array(TimeUuid)` | `List[uuid.UUID]` | List of time-based UUIDs | | `Array(Decimal)` | `List[decimal.Decimal]` | List of arbitrary-precision decimals | ### Set Types All primitive types (except `Map` and `Json`) have corresponding set types for storing unique values: | Feast Type | Python Type | Description | | -------------------- | ---------------------- | ------------------------------------------ | | `Set(Int32)` | `Set[int]` | Set of unique 32-bit integers | | `Set(Int64)` | `Set[int]` | Set of unique 64-bit integers | | `Set(Float32)` | `Set[float]` | Set of unique 32-bit floats | | `Set(Float64)` | `Set[float]` | Set of unique 64-bit floats | | `Set(String)` | `Set[str]` | Set of unique strings | | `Set(Bytes)` | `Set[bytes]` | Set of unique binary data | | `Set(Bool)` | `Set[bool]` | Set of unique booleans | | `Set(UnixTimestamp)` | `Set[datetime]` | Set of unique timestamps | | `Set(Uuid)` | `Set[uuid.UUID]` | Set of unique UUIDs | | `Set(TimeUuid)` | `Set[uuid.UUID]` | Set of unique time-based UUIDs | | `Set(Decimal)` | `Set[decimal.Decimal]` | Set of unique arbitrary-precision decimals | **Note:** Set types automatically remove duplicate values. When converting from lists or other iterables to sets, duplicates are eliminated. {% hint style="warning" %} **Backend limitations for Set types:** * **No backend infers Set types from schema.** No offline store (BigQuery, Snowflake, Redshift, PostgreSQL, Spark, Athena, MSSQL) maps its native types to Feast Set types. You **must** explicitly declare Set types in your feature view schema. * **No native PyArrow set type.** Feast converts Sets to `pyarrow.list_()` internally, but `feast_value_type_to_pa()` in `type_map.py` does not include Set mappings, which can cause errors in some code paths. * **Online stores** that serialize proto bytes (e.g., SQLite, Redis, DynamoDB) handle Sets correctly. * **Offline stores** may not handle Set types correctly during retrieval. For example, the Ray offline store only special-cases `_LIST` types, not `_SET`. * Set types are best suited for **online serving** use cases where feature values are written as Python sets and retrieved via `get_online_features`. {% endhint %} ### Nested Collection Types Feast supports arbitrarily nested collections using a recursive `VALUE_LIST` / `VALUE_SET` design. The outer container determines the proto enum (`VALUE_LIST` for `Array(…)`, `VALUE_SET` for `Set(…)`), while the full inner type structure is persisted via a mandatory `feast:nested_inner_type` Field tag. | Feast Type | Python Type | ValueType | Description | | ------------------------ | --------------------- | ------------ | --------------- | | `Array(Array(T))` | `List[List[T]]` | `VALUE_LIST` | List of lists | | `Array(Set(T))` | `List[List[T]]` | `VALUE_LIST` | List of sets | | `Set(Array(T))` | `List[List[T]]` | `VALUE_SET` | Set of lists | | `Set(Set(T))` | `List[List[T]]` | `VALUE_SET` | Set of sets | | `Array(Array(Array(T)))` | `List[List[List[T]]]` | `VALUE_LIST` | 3-level nesting | Where `T` is any supported primitive type (Int32, Int64, Float32, Float64, String, Bytes, Bool, UnixTimestamp) or another nested collection type. **Notes:** * Nesting depth is **unlimited**. `Array(Array(Array(T)))`, `Set(Array(Set(T)))`, etc. are all supported. * Inner type information is preserved via Field tags (`feast:nested_inner_type`) and restored during deserialization. This tag is mandatory for nested collection types. * Empty inner collections (`[]`) are stored as empty proto values and round-trip as `None`. For example, `[[1, 2], [], [3]]` becomes `[[1, 2], None, [3]]` after a write-read cycle. ### Map Types Map types allow storing dictionary-like data structures: | Feast Type | Python Type | Description | | ------------ | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | `Map` | `Dict[str, Any]` | Dictionary with string keys and values of any supported Feast type (including nested maps) | | `Array(Map)` | `List[Dict[str, Any]]` | List of dictionaries | | `ScalarMap` | `Dict[Any, Any]` | Dictionary with non-string scalar keys (int, float, bool, UUID, Decimal, bytes, datetime) and values of any supported Feast type | **Note:** `Map` keys must always be strings. `ScalarMap` supports non-string scalar keys — Feast infers `ScalarMap` automatically when the first key of a dict is not a string. Map values can be any supported Feast type, including primitives, arrays, or nested maps at the proto level. However, the PyArrow representation is `map`, which means backends that rely on PyArrow schemas (e.g., during materialization) treat Map as string-to-string. {% hint style="warning" %} `ScalarMap` is **not** inferred from any backend schema. You must declare it explicitly in your feature view schema. It is best suited for online serving use cases where the online store serializes proto bytes directly (e.g., Redis, DynamoDB, SQLite). {% endhint %} **Backend support for Map:** | Backend | Native Type | Notes | | ---------------- | -------------------- | ---------------------------------------------- | | PostgreSQL | `jsonb`, `jsonb[]` | `jsonb` → `Map`, `jsonb[]` → `Array(Map)` | | Snowflake | `VARIANT`, `OBJECT` | Inferred as `Map` | | Redshift | `SUPER` | Inferred as `Map` | | Spark | `map` | `map<>` → `Map`, `array>` → `Array(Map)` | | Athena | `map` | Inferred as `Map` | | MSSQL | `nvarchar(max)` | Serialized as string | | DynamoDB / Redis | Proto bytes | Full proto Map and ScalarMap support | ### JSON Type The `Json` type represents opaque JSON data. Unlike `Map`, which is schema-free key-value storage, `Json` is stored as a string at the proto level but backends use native JSON types where available. | Feast Type | Python Type | Description | | ------------- | -------------------- | ----------------------------------------------- | | `Json` | `str` (JSON-encoded) | JSON data stored as a string at the proto level | | `Array(Json)` | `List[str]` | List of JSON strings | **Backend support for Json:** | Backend | Native Type | | ---------- | ---------------------------------------- | | PostgreSQL | `jsonb` | | Snowflake | `JSON` / `VARIANT` | | Redshift | `json` | | BigQuery | `JSON` | | Spark | Not natively distinguished from `String` | | MSSQL | `nvarchar(max)` | {% hint style="info" %} When a backend's native type is ambiguous (e.g., PostgreSQL `jsonb` could be `Map` or `Json`), **the schema-declared Feast type takes precedence**. The backend-to-Feast mappings are only used during schema inference when no explicit type is provided. {% endhint %} ### Struct Type The `Struct` type represents a schema-aware structured type with named, typed fields. Unlike `Map` (which is schema-free), a `Struct` declares its field names and their types, enabling schema validation. | Feast Type | Python Type | Description | | ------------------------------------- | ---------------------- | ------------------------------ | | `Struct({"field": Type, ...})` | `Dict[str, Any]` | Named fields with typed values | | `Array(Struct({"field": Type, ...}))` | `List[Dict[str, Any]]` | List of structs | **Example:** ```python from feast.types import Struct, String, Int32, Array # Struct with named, typed fields address_type = Struct({"street": String, "city": String, "zip": Int32}) Field(name="address", dtype=address_type) # Array of structs items_type = Array(Struct({"name": String, "quantity": Int32})) Field(name="order_items", dtype=items_type) ``` **Backend support for Struct:** | Backend | Native Type | | ---------------- | ------------------------------------ | | BigQuery | `STRUCT` / `RECORD` | | Spark | `struct<...>` / `array>` | | PostgreSQL | `jsonb` (serialized) | | Snowflake | `VARIANT` (serialized) | | MSSQL | `nvarchar(max)` (serialized) | | DynamoDB / Redis | Proto bytes | ## Complete Feature View Example Below is a complete example showing how to define a feature view with all supported types: ```python from datetime import timedelta from feast import Entity, FeatureView, Field, FileSource from feast.types import ( Int32, Int64, Float32, Float64, String, Bytes, Bool, UnixTimestamp, Uuid, TimeUuid, Decimal, Array, Set, Map, ScalarMap, Json, Struct, ZonedTimestamp ) # Define a data source user_features_source = FileSource( path="data/user_features.parquet", timestamp_field="event_timestamp", ) # Define an entity user = Entity( name="user_id", description="User identifier", ) # Define a feature view with all supported types user_features = FeatureView( name="user_features", entities=[user], ttl=timedelta(days=1), schema=[ # Primitive types Field(name="age", dtype=Int32), Field(name="account_balance", dtype=Int64), Field(name="transaction_amount", dtype=Float32), Field(name="credit_score", dtype=Float64), Field(name="username", dtype=String), Field(name="profile_picture", dtype=Bytes), Field(name="is_active", dtype=Bool), Field(name="last_login", dtype=UnixTimestamp), Field(name="event_time", dtype=ZonedTimestamp), Field(name="session_id", dtype=Uuid), Field(name="event_id", dtype=TimeUuid), Field(name="price", dtype=Decimal), # Array types Field(name="daily_steps", dtype=Array(Int32)), Field(name="transaction_history", dtype=Array(Int64)), Field(name="ratings", dtype=Array(Float32)), Field(name="portfolio_values", dtype=Array(Float64)), Field(name="favorite_items", dtype=Array(String)), Field(name="document_hashes", dtype=Array(Bytes)), Field(name="notification_settings", dtype=Array(Bool)), Field(name="login_timestamps", dtype=Array(UnixTimestamp)), Field(name="related_session_ids", dtype=Array(Uuid)), Field(name="event_chain", dtype=Array(TimeUuid)), Field(name="historical_prices", dtype=Array(Decimal)), # Set types (unique values only — see backend caveats above) Field(name="visited_pages", dtype=Set(String)), Field(name="unique_categories", dtype=Set(Int32)), Field(name="tag_ids", dtype=Set(Int64)), Field(name="preferred_languages", dtype=Set(String)), Field(name="unique_device_ids", dtype=Set(Uuid)), Field(name="unique_event_ids", dtype=Set(TimeUuid)), Field(name="unique_prices", dtype=Set(Decimal)), # Map types Field(name="user_preferences", dtype=Map), Field(name="metadata", dtype=Map), Field(name="activity_log", dtype=Array(Map)), Field(name="event_counts", dtype=ScalarMap), # non-string keys, e.g. {1001: 5, 1002: 12} # Nested collection types Field(name="weekly_scores", dtype=Array(Array(Float64))), Field(name="unique_tags_per_category", dtype=Array(Set(String))), # JSON type Field(name="raw_event", dtype=Json), # Struct type Field(name="address", dtype=Struct({"street": String, "city": String, "zip": Int32})), Field(name="order_items", dtype=Array(Struct({"name": String, "qty": Int32}))), ], source=user_features_source, ) ``` ### Set Type Usage Examples Sets store unique values and automatically remove duplicates: ```python # Simple set visited_pages = {"home", "products", "checkout", "products"} # "products" appears twice # Feast will store this as: {"home", "products", "checkout"} # Integer set unique_categories = {1, 2, 3, 2, 1} # duplicates will be removed # Feast will store this as: {1, 2, 3} # Converting a list with duplicates to a set tag_list = [100, 200, 300, 100, 200] tag_ids = set(tag_list) # {100, 200, 300} ``` ### UUID Type Usage Examples UUID types store universally unique identifiers natively, with support for both random UUIDs and time-based UUIDs: ```python import uuid # Random UUID (version 4) — use Uuid type session_id = uuid.uuid4() # e.g., UUID('a8098c1a-f86e-11da-bd1a-00112444be1e') # Time-based UUID (version 1) — use TimeUuid type event_id = uuid.uuid1() # e.g., UUID('6ba7b810-9dad-11d1-80b4-00c04fd430c8') # UUID values are returned as uuid.UUID objects from get_online_features() response = store.get_online_features( features=["user_features:session_id"], entity_rows=[{"user_id": 1}], ) result = response.to_dict() # result["session_id"][0] is a uuid.UUID object # UUID lists related_sessions = [uuid.uuid4(), uuid.uuid4(), uuid.uuid4()] # UUID sets (unique values) unique_devices = {uuid.uuid4(), uuid.uuid4()} ``` ### Decimal Type Usage Examples The `Decimal` type stores arbitrary-precision decimal numbers using Python's `decimal.Decimal`. Values are stored as strings in the proto to preserve full precision — no floating-point rounding occurs. ```python import decimal # Scalar decimal — e.g., a financial price price = decimal.Decimal("19.99") # High-precision value — all digits preserved tax_rate = decimal.Decimal("0.08750000000000000000") # Decimal values are returned as decimal.Decimal objects from get_online_features() response = store.get_online_features( features=["product_features:price"], entity_rows=[{"product_id": 42}], ) result = response.to_dict() # result["price"][0] is a decimal.Decimal object # Decimal lists — e.g., a history of prices historical_prices = [ decimal.Decimal("18.50"), decimal.Decimal("19.00"), decimal.Decimal("19.99"), ] # Decimal sets — unique price points seen unique_prices = {decimal.Decimal("9.99"), decimal.Decimal("19.99"), decimal.Decimal("29.99")} ``` {% hint style="warning" %} `Decimal` is **not** inferred from any backend schema. You must declare it explicitly in your feature view schema. The pandas dtype for `Decimal` columns is `object` (holding `decimal.Decimal` instances), not a numeric dtype. {% endhint %} ### ZonedTimestamp Type Usage Examples The `ZonedTimestamp` type stores a timezone-aware `datetime` as both the UTC instant and its originating zone, so the original wall-clock zone round-trips losslessly. By contrast, `UnixTimestamp` always decodes to UTC and discards the source zone. ```python from datetime import datetime, timezone from zoneinfo import ZoneInfo # A datetime in a specific zone — both the instant and "America/Los_Angeles" are kept event_time = datetime(2026, 6, 17, 9, 0, 0, tzinfo=ZoneInfo("America/Los_Angeles")) # ZonedTimestamp values are returned as tz-aware datetime objects, in their own zone response = store.get_online_features( features=["event_features:event_time"], entity_rows=[{"user_id": 1001}], ) result = response.to_dict() # result["event_time"][0] == event_time (same instant AND same zone, e.g. 09:00-07:00) # Two values at the same instant but different zones stay distinct la = datetime(2026, 6, 17, 9, 0, 0, tzinfo=ZoneInfo("America/Los_Angeles")) utc = datetime(2026, 6, 17, 16, 0, 0, tzinfo=timezone.utc) # same instant as `la` # A naive (tz-less) datetime is interpreted as UTC naive = datetime(2026, 6, 17, 12, 0, 0) # stored zone is empty, decoded as UTC ``` {% hint style="warning" %} `ZonedTimestamp` is **not** inferred from any backend schema — you must declare it explicitly in your feature view schema. It is not supported as an entity key. The zone is stored as an IANA name (e.g. `America/Los_Angeles`) when available, falling back to a fixed-offset string; offline stores that cannot natively carry a zone may normalize to UTC on that backend. {% endhint %} ### Nested Collection Type Usage Examples ```python # List of lists — e.g., weekly score history per user weekly_scores = [[85.0, 90.5, 78.0], [92.0, 88.5], [95.0, 91.0, 87.5]] # List of sets — e.g., unique tags assigned per category unique_tags_per_category = [["python", "ml"], ["rust", "systems"], ["python", "web"]] # 3-level nesting — e.g., multi-dimensional matrices Field(name="tensor", dtype=Array(Array(Array(Float64)))) # Mixed nesting Field(name="grouped_tags", dtype=Array(Set(Array(String)))) ``` **Limitation:** Empty inner collections round-trip as `None`: ```python # Input: [[1, 2], [], [3]] # Output: [[1, 2], None, [3]] (empty [] becomes None after write-read cycle) ``` ### Map Type Usage Examples Maps can store complex nested data structures: ```python # Simple map (string keys) user_preferences = { "theme": "dark", "language": "en", "notifications_enabled": True, "font_size": 14 } # Nested map metadata = { "profile": { "bio": "Software engineer", "location": "San Francisco" }, "stats": { "followers": 1000, "posts": 250 } } # List of maps activity_log = [ {"action": "login", "timestamp": "2024-01-01T10:00:00", "ip": "192.168.1.1"}, {"action": "purchase", "timestamp": "2024-01-01T11:30:00", "amount": 99.99}, {"action": "logout", "timestamp": "2024-01-01T12:00:00"} ] ``` ### ScalarMap Type Usage Examples `ScalarMap` supports non-string keys. Feast infers it automatically when the first dict key is not a string: ```python import uuid import decimal # Integer keys — e.g., category ID → item count event_counts = {1001: 5, 1002: 12, 1003: 0} # UUID keys — e.g., session ID → score import uuid session_scores = { uuid.UUID("6ba7b810-9dad-11d1-80b4-00c04fd430c8"): 0.95, uuid.UUID("a8098c1a-f86e-11da-bd1a-00112444be1e"): 0.87, } # Decimal keys — e.g., price bucket → product name price_tier = { decimal.Decimal("9.99"): "budget", decimal.Decimal("49.99"): "standard", decimal.Decimal("99.99"): "premium", } # Type inference: Feast automatically picks SCALAR_MAP when the key is non-string from feast.type_map import python_type_to_feast_value_type from feast.value_type import ValueType python_type_to_feast_value_type({1: "a"}) # → ValueType.SCALAR_MAP python_type_to_feast_value_type({"a": 1}) # → ValueType.MAP python_type_to_feast_value_type({}) # → ValueType.MAP (empty dict defaults to MAP) ``` {% hint style="warning" %} `ScalarMap` must be **explicitly declared** in your feature view schema — it is never inferred from backend type schemas. It is best suited for online serving via stores that use proto byte serialization (e.g., Redis, DynamoDB, SQLite). Materialization paths that use PyArrow (e.g., BigQuery, Snowflake, Redshift, Spark) do not have native `ScalarMap` support. {% endhint %} ### JSON Type Usage Examples Feast's `Json` type stores values as JSON strings at the proto level. You can pass either a pre-serialized JSON string or a Python dict/list — Feast will call `json.dumps()` automatically when the value is not already a string: ```python import json # Option 1: pass a Python dict — Feast calls json.dumps() internally during proto conversion raw_event = {"type": "click", "target": "button_1", "metadata": {"page": "home"}} # Option 2: pass an already-serialized JSON string — Feast validates it via json.loads() raw_event = '{"type": "click", "target": "button_1", "metadata": {"page": "home"}}' # When building a DataFrame for store.push(), values must be strings since # Pandas/PyArrow columns expect uniform types: import pandas as pd event_df = pd.DataFrame({ "user_id": ["user_1"], "event_timestamp": [datetime.now()], "raw_event": [json.dumps({"type": "click", "target": "button_1"})], }) store.push("event_push_source", event_df) ``` ### Struct Type Usage Examples ```python # Struct — schema-aware, fields and types are declared from feast.types import Struct, String, Int32 address = Struct({"street": String, "city": String, "zip": Int32}) # Value: {"street": "123 Main St", "city": "Springfield", "zip": 62704} ``` ## Type System in Practice The sections below explain how Feast uses its type system in different contexts. ### Feature inference During `feast apply`, Feast runs schema inference on the data sources underlying feature views. For example, if the `schema` parameter is not specified for a feature view, Feast will examine the schema of the underlying data source to determine the event timestamp column, feature columns, and entity columns. Each of these columns must be associated with a Feast type, which requires conversion from the data source type system to the Feast type system. * The feature inference logic calls `_infer_features_and_entities`. * `_infer_features_and_entities` calls `source_datatype_to_feast_value_type`. * `source_datatype_to_feast_value_type` calls the appropriate method in `type_map.py`. For example, if a `SnowflakeSource` is being examined, `snowflake_python_type_to_feast_value_type` from `type_map.py` will be called. {% hint style="info" %} **Types that cannot be inferred:** `Set`, `Json`, `Struct`, `Decimal`, `ScalarMap`, `PdfBytes`, and `ImageBytes` types are never inferred from backend schemas. If you use these types, you must declare them explicitly in your feature view schema. {% endhint %} ### Materialization Feast serves feature values as [`Value`](https://github.com/feast-dev/feast/blob/master/protos/feast/types/Value.proto) proto objects, which have a type corresponding to Feast types. Thus Feast must materialize feature values into the online store as `Value` proto objects. * The local materialization engine first pulls the latest historical features and converts it to pyarrow. * Then it calls `_convert_arrow_to_proto` to convert the pyarrow table to proto format. * This calls `python_values_to_proto_values` in `type_map.py` to perform the type conversion. ### Historical feature retrieval The Feast type system is typically not necessary when retrieving historical features. A call to `get_historical_features` will return a `RetrievalJob` object, which allows the user to export the results to one of several possible locations: a Pandas dataframe, a pyarrow table, a data lake (e.g. S3 or GCS), or the offline store (e.g. a Snowflake table). In all of these cases, the type conversion is handled natively by the offline store. For example, a BigQuery query exposes a `to_dataframe` method that will automatically convert the result to a dataframe, without requiring any conversions within Feast. ### Feature serving As mentioned above in the section on [materialization](#materialization), Feast persists feature values into the online store as `Value` proto objects. A call to `get_online_features` will return an `OnlineResponse` object, which essentially wraps a bunch of `Value` protos with some metadata. The `OnlineResponse` object can then be converted into a Python dictionary, which calls `feast_value_type_to_python_type` from `type_map.py`, a utility that converts the Feast internal types to Python native types. # Data sources Please see [Data Source](/master/getting-started/concepts/data-ingestion) for a conceptual explanation of data sources. {% content-ref url="/pages/NwhtDvlBAR665qwKx7Ku" %} [Overview](/master/reference/data-sources/overview) {% endcontent-ref %} {% content-ref url="/pages/-McaMirJdDDSO9jykbG-" %} [File](/master/reference/data-sources/file) {% endcontent-ref %} {% content-ref url="/pages/m2jKsSveF5REKuLfhypR" %} [Snowflake](/master/reference/data-sources/snowflake) {% endcontent-ref %} {% content-ref url="/pages/-McaMh1xpATQLbm9thW8" %} [BigQuery](/master/reference/data-sources/bigquery) {% endcontent-ref %} {% content-ref url="/pages/-MgHM6elL1H8qFdLWXHd" %} [Redshift](/master/reference/data-sources/redshift) {% endcontent-ref %} {% content-ref url="/pages/4qY0HS4mK7sQ7HSk5yAW" %} [Push](/master/reference/data-sources/push) {% endcontent-ref %} {% content-ref url="/pages/YGSmcz8tK8lQl5SOV0ap" %} [Kafka](/master/reference/data-sources/kafka) {% endcontent-ref %} {% content-ref url="/pages/Bg9aIshqrVqhA1vGIOCv" %} [Kinesis](/master/reference/data-sources/kinesis) {% endcontent-ref %} {% content-ref url="/pages/l3GYsgYHjd0LXbaKZBp1" %} [Couchbase (contrib)](/master/reference/data-sources/couchbase) {% endcontent-ref %} {% content-ref url="/pages/7K3zzYXbS1uAj7kZtxdy" %} [Spark (contrib)](/master/reference/data-sources/spark) {% endcontent-ref %} {% content-ref url="" %} {% endcontent-ref %} {% content-ref url="/pages/lumVxexGYQOz3vruTmJz" %} [PostgreSQL (contrib)](/master/reference/data-sources/postgres) {% endcontent-ref %} {% content-ref url="/pages/l4o66ASPdeSUDI73ufnp" %} [Trino (contrib)](/master/reference/data-sources/trino) {% endcontent-ref %} {% content-ref url="/pages/wDNRfeqLRZsYYwsIoxMY" %} [Azure Synapse + Azure SQL (contrib)](/master/reference/data-sources/mssql) {% endcontent-ref %} {% content-ref url="/pages/c96xdW5inTBOKd0u1OZg" %} [Clickhouse (contrib)](/master/reference/data-sources/clickhouse) {% endcontent-ref %} {% content-ref url="/pages/J4p1J9OYETGsyDJmYCpw" %} [Athena (contrib)](/master/reference/data-sources/athena) {% endcontent-ref %} {% content-ref url="/pages/zOr6FQVBPHIpAw6EOVpG" %} [Oracle (contrib)](/master/reference/data-sources/oracle) {% endcontent-ref %} {% content-ref url="/pages/UzlCF63V8NvXucik0fxs" %} [Ray (contrib)](/master/reference/data-sources/ray) {% endcontent-ref %} {% content-ref url="/pages/fGI2gMdQItXwHL8UCB3O" %} [MongoDB (contrib)](/master/reference/data-sources/mongodb) {% endcontent-ref %} # Overview ## Functionality In Feast, each batch data source is associated with corresponding offline stores. For example, a `SnowflakeSource` can only be processed by the Snowflake offline store, while a `FileSource` can be processed by both File and DuckDB offline stores. Otherwise, the primary difference between batch data sources is the set of supported types. Feast has an internal type system that supports primitive types (`bytes`, `string`, `int32`, `int64`, `float32`, `float64`, `bool`, `timestamp`), array types, set types, map/JSON types, and struct types. However, not every batch data source supports all of these types. For more details on the Feast type system, see [here](/master/reference/type-system). ## Functionality Matrix There are currently four core batch data source implementations: `FileSource`, `BigQuerySource`, `SnowflakeSource`, and `RedshiftSource`. There are several additional implementations contributed by the Feast community (`PostgreSQLSource`, `SparkSource`, and `TrinoSource`), which are not guaranteed to be stable or to match the functionality of the core implementations. Details for each specific data source can be found [here](/master/reference/data-sources). Below is a matrix indicating which data sources support which types. | | File | BigQuery | Snowflake | Redshift | Postgres | Spark | Trino | Couchbase | | ----------- | ----- | -------- | --------- | -------- | -------- | ----- | ----- | --------- | | `bytes` | yes | yes | yes | yes | yes | yes | yes | yes | | `string` | yes | yes | yes | yes | yes | yes | yes | yes | | `int32` | yes | yes | yes | yes | yes | yes | yes | yes | | `int64` | yes | yes | yes | yes | yes | yes | yes | yes | | `float32` | yes | yes | yes | yes | yes | yes | yes | yes | | `float64` | yes | yes | yes | yes | yes | yes | yes | yes | | `bool` | yes | yes | yes | yes | yes | yes | yes | yes | | `timestamp` | yes | yes | yes | yes | yes | yes | yes | yes | | array types | yes | yes | yes | no | yes | yes | yes | no | | `Map` | yes | no | yes | yes | yes | yes | yes | no | | `Json` | yes | yes | yes | yes | yes | no | no | no | | `Struct` | yes | yes | no | no | yes | yes | no | no | | set types | yes\* | no | no | no | no | no | no | no | \* **Set types** are defined in Feast's proto and Python type system but are **not inferred** by any backend. They must be explicitly declared in the feature view schema and are best suited for online serving use cases. See [Type System](/master/reference/type-system#set-types) for details. # Table formats ## Overview Table formats are metadata and transaction layers built on top of data storage formats (like Parquet). They provide advanced capabilities for managing large-scale data lakes, including ACID transactions, time travel, schema evolution, and efficient data management. Feast supports modern table formats to enable data lakehouse architectures with your feature store. ## Supported Table Formats ### Apache Iceberg [Apache Iceberg](https://iceberg.apache.org/) is an open table format designed for huge analytic datasets. It provides: * **ACID transactions**: Atomic commits with snapshot isolation * **Time travel**: Query data as of any snapshot * **Schema evolution**: Add, drop, rename, or reorder columns safely * **Hidden partitioning**: Partitioning is transparent to users * **Performance**: Advanced pruning and filtering #### Basic Configuration ```python from feast.table_format import IcebergFormat iceberg_format = IcebergFormat( catalog="my_catalog", namespace="my_database" ) ``` #### Configuration Options | Parameter | Type | Description | | ------------ | ----------------- | ------------------------------------------- | | `catalog` | `str` (optional) | Iceberg catalog name | | `namespace` | `str` (optional) | Namespace/schema within the catalog | | `properties` | `dict` (optional) | Additional Iceberg configuration properties | #### Common Properties ```python iceberg_format = IcebergFormat( catalog="spark_catalog", namespace="production", properties={ # Snapshot selection "snapshot-id": "123456789", "as-of-timestamp": "1609459200000", # Unix timestamp in ms # Performance tuning "read.split.target-size": "134217728", # 128 MB splits "read.parquet.vectorization.enabled": "true", # Advanced configuration "io-impl": "org.apache.iceberg.hadoop.HadoopFileIO", "warehouse": "s3://my-bucket/warehouse" } ) ``` #### Time Travel Example ```python # Read from a specific snapshot iceberg_format = IcebergFormat( catalog="spark_catalog", namespace="lakehouse" ) iceberg_format.set_property("snapshot-id", "7896524153287651133") # Or read as of a timestamp iceberg_format.set_property("as-of-timestamp", "1609459200000") ``` ### Delta Lake [Delta Lake](https://delta.io/) is an open-source storage layer that brings ACID transactions to Apache Spark and big data workloads. It provides: * **ACID transactions**: Serializable isolation for reads and writes * **Time travel**: Access and revert to earlier versions * **Schema enforcement**: Prevent bad data from corrupting tables * **Unified batch and streaming**: Process data incrementally * **Audit history**: Full history of all changes #### Basic Configuration ```python from feast.table_format import DeltaFormat delta_format = DeltaFormat() ``` #### Configuration Options | Parameter | Type | Description | | --------------------- | ----------------- | ---------------------------------------------- | | `checkpoint_location` | `str` (optional) | Location for Delta transaction log checkpoints | | `properties` | `dict` (optional) | Additional Delta configuration properties | #### Common Properties ```python delta_format = DeltaFormat( checkpoint_location="s3://my-bucket/checkpoints", properties={ # Time travel "versionAsOf": "5", "timestampAsOf": "2024-01-01 00:00:00", # Performance optimization "delta.autoOptimize.optimizeWrite": "true", "delta.autoOptimize.autoCompact": "true", # Data skipping "delta.dataSkippingNumIndexedCols": "32", # Z-ordering "delta.autoOptimize.zOrderCols": "event_timestamp" } ) ``` #### Time Travel Example ```python # Read from a specific version delta_format = DeltaFormat() delta_format.set_property("versionAsOf", "10") # Or read as of a timestamp delta_format = DeltaFormat() delta_format.set_property("timestampAsOf", "2024-01-15 12:00:00") ``` ### Apache Hudi [Apache Hudi](https://hudi.apache.org/) (Hadoop Upserts Deletes and Incrementals) is a data lake storage framework for simplifying incremental data processing. It provides: * **Upserts and deletes**: Efficient record-level updates * **Incremental queries**: Process only changed data * **Time travel**: Query historical versions * **Multiple table types**: Optimize for read vs. write workloads * **Change data capture**: Track data changes over time #### Basic Configuration ```python from feast.table_format import HudiFormat hudi_format = HudiFormat( table_type="COPY_ON_WRITE", record_key="user_id", precombine_field="updated_at" ) ``` #### Configuration Options | Parameter | Type | Description | | ------------------ | ----------------- | ------------------------------------------ | | `table_type` | `str` (optional) | `COPY_ON_WRITE` or `MERGE_ON_READ` | | `record_key` | `str` (optional) | Field(s) that uniquely identify a record | | `precombine_field` | `str` (optional) | Field used to determine the latest version | | `properties` | `dict` (optional) | Additional Hudi configuration properties | #### Table Types **COPY\_ON\_WRITE (COW)** * Stores data in columnar format (Parquet) * Updates create new file versions * Best for **read-heavy workloads** * Lower query latency ```python hudi_format = HudiFormat( table_type="COPY_ON_WRITE", record_key="id", precombine_field="timestamp" ) ``` **MERGE\_ON\_READ (MOR)** * Uses columnar + row-based formats * Updates written to delta logs * Best for **write-heavy workloads** * Lower write latency ```python hudi_format = HudiFormat( table_type="MERGE_ON_READ", record_key="id", precombine_field="timestamp" ) ``` #### Common Properties ```python hudi_format = HudiFormat( table_type="COPY_ON_WRITE", record_key="user_id", precombine_field="updated_at", properties={ # Query type "hoodie.datasource.query.type": "snapshot", # or "incremental" # Incremental queries "hoodie.datasource.read.begin.instanttime": "20240101000000", "hoodie.datasource.read.end.instanttime": "20240102000000", # Indexing "hoodie.index.type": "BLOOM", # Compaction (for MOR tables) "hoodie.compact.inline": "true", "hoodie.compact.inline.max.delta.commits": "5", # Clustering "hoodie.clustering.inline": "true" } ) ``` #### Incremental Query Example ```python # Process only new/changed data hudi_format = HudiFormat( table_type="COPY_ON_WRITE", record_key="id", precombine_field="timestamp", properties={ "hoodie.datasource.query.type": "incremental", "hoodie.datasource.read.begin.instanttime": "20240101000000", "hoodie.datasource.read.end.instanttime": "20240102000000" } ) ``` ## Table Format vs File Format It's important to understand the distinction: | Aspect | File Format | Table Format | | -------------- | ------------------------- | ---------------------------------- | | **What it is** | Physical encoding of data | Metadata and transaction layer | | **Examples** | Parquet, Avro, ORC, CSV | Iceberg, Delta Lake, Hudi | | **Handles** | Data serialization | ACID, versioning, schema evolution | | **Layer** | Storage layer | Metadata layer | ### Can be used together ```python # Table format (metadata layer) built on top of file format (storage layer) from feast.infra.offline_stores.contrib.spark_offline_store.spark_source import SparkSource from feast.table_format import IcebergFormat iceberg = IcebergFormat(catalog="my_catalog", namespace="db") source = SparkSource( name="features", path="catalog.db.table", file_format="parquet", # Underlying storage format table_format=iceberg, # Table metadata format timestamp_field="event_timestamp" ) ``` ## Benefits of Table Formats ### Reliability * **ACID transactions**: Ensure data consistency across concurrent operations * **Automatic retries**: Handle transient failures gracefully * **Schema validation**: Prevent incompatible schema changes * **Data quality**: Constraints and validation rules ### Performance * **Data skipping**: Read only relevant files based on metadata * **Partition pruning**: Skip entire partitions based on predicates * **Compaction**: Merge small files for better performance * **Columnar pruning**: Read only necessary columns * **Indexing**: Advanced indexing for fast lookups ### Flexibility * **Schema evolution**: Add, remove, or modify columns without rewriting data * **Time travel**: Access historical data states for auditing or debugging * **Incremental processing**: Process only changed data efficiently * **Multiple readers/writers**: Concurrent access without conflicts ## Choosing the Right Table Format | Use Case | Recommended Format | Why | | -------------------------------------------------- | -------------------- | -------------------------------------------------------------- | | Large-scale analytics with frequent schema changes | **Iceberg** | Best schema evolution, hidden partitioning, mature ecosystem | | Streaming + batch workloads | **Delta Lake** | Unified architecture, strong integration with Spark, good docs | | CDC and upsert-heavy workloads | **Hudi** | Efficient record-level updates, incremental queries | | Read-heavy analytics | **Iceberg or Delta** | Excellent query performance | | Write-heavy transactional | **Hudi (MOR)** | Optimized for fast writes | | Multi-engine support | **Iceberg** | Widest engine support (Spark, Flink, Trino, etc.) | ## Best Practices ### 1. Choose Appropriate Partitioning ```python # Iceberg - hidden partitioning iceberg_format.set_property("partition-spec", "days(event_timestamp)") # Delta - explicit partitioning in data source # Hudi - configure via properties hudi_format.set_property("hoodie.datasource.write.partitionpath.field", "date") ``` ### 2. Enable Optimization Features ```python # Delta auto-optimize delta_format.set_property("delta.autoOptimize.optimizeWrite", "true") delta_format.set_property("delta.autoOptimize.autoCompact", "true") # Hudi compaction hudi_format.set_property("hoodie.compact.inline", "true") ``` ### 3. Manage Table History ```python # Regularly clean up old snapshots/versions # For Iceberg: Use expire_snapshots() procedure # For Delta: Use VACUUM command # For Hudi: Configure retention policies ``` ### 4. Monitor Metadata Size * Table formats maintain metadata for all operations * Monitor metadata size and clean up old versions * Configure retention policies based on your needs ### 5. Test Schema Evolution ```python # Always test schema changes in non-production first # Ensure backward compatibility # Use proper migration procedures ``` ## Data Source Support Currently, table formats are supported with: * [Spark data source](/master/reference/data-sources/spark) - Full support for Iceberg, Delta, and Hudi Future support planned for: * BigQuery (Iceberg) * Snowflake (Iceberg) * Other data sources ## See Also * [Spark Data Source](/master/reference/data-sources/spark) * [Apache Iceberg Documentation](https://iceberg.apache.org/docs/latest/) * [Delta Lake Documentation](https://docs.delta.io/latest/index.html) * [Apache Hudi Documentation](https://hudi.apache.org/docs/overview) * [Python API Reference - TableFormat](https://rtd.feast.dev/en/master/#feast.table_format) # File ## Description File data sources are files on disk or on S3. Currently only Parquet and Delta formats are supported. ## Example ```python from feast import FileSource from feast.data_format import ParquetFormat parquet_file_source = FileSource( file_format=ParquetFormat(), path="file:///feast/customer.parquet", ) ``` The full set of configuration options is available [here](https://rtd.feast.dev/en/latest/index.html#feast.infra.offline_stores.file_source.FileSource). ## Supported Types File data sources support all eight primitive types and their corresponding array types. For a comparison against other batch data sources, please see [here](/master/reference/data-sources/overview#functionality-matrix). # Snowflake ## Description Snowflake data sources are Snowflake tables or views. These can be specified either by a table reference or a SQL query. ## Examples Using a table reference: ```python from feast import SnowflakeSource my_snowflake_source = SnowflakeSource( database="FEAST", schema="PUBLIC", table="FEATURE_TABLE", ) ``` Using a query: ```python from feast import SnowflakeSource my_snowflake_source = SnowflakeSource( query=""" SELECT timestamp_column AS "ts", "created", "f1", "f2" FROM `FEAST.PUBLIC.FEATURE_TABLE` """, ) ``` {% hint style="warning" %} Be careful about how Snowflake handles table and column name conventions. In particular, you can read more about quote identifiers [here](https://docs.snowflake.com/en/sql-reference/identifiers-syntax.html). {% endhint %} The full set of configuration options is available [here](https://rtd.feast.dev/en/latest/index.html#feast.infra.offline_stores.snowflake_source.SnowflakeSource). ## Supported Types Snowflake data sources support all eight primitive types. Array types are also supported but not with type inference. For a comparison against other batch data sources, please see [here](/master/reference/data-sources/overview#functionality-matrix). # BigQuery ## Description BigQuery data sources are BigQuery tables or views. These can be specified either by a table reference or a SQL query. However, no performance guarantees can be provided for SQL query-based sources, so table references are recommended. ## Examples Using a table reference: ```python from feast import BigQuerySource my_bigquery_source = BigQuerySource( table_ref="gcp_project:bq_dataset.bq_table", ) ``` Using a query: ```python from feast import BigQuerySource BigQuerySource( query="SELECT timestamp as ts, created, f1, f2 " "FROM `my_project.my_dataset.my_features`", ) ``` The full set of configuration options is available [here](https://rtd.feast.dev/en/latest/index.html#feast.infra.offline_stores.bigquery_source.BigQuerySource). ## Supported Types BigQuery data sources support all eight primitive types and their corresponding array types. For a comparison against other batch data sources, please see [here](/master/reference/data-sources/overview#functionality-matrix). # Redshift ## Description Redshift data sources are Redshift tables or views. These can be specified either by a table reference or a SQL query. However, no performance guarantees can be provided for SQL query-based sources, so table references are recommended. ## Examples Using a table name: ```python from feast import RedshiftSource my_redshift_source = RedshiftSource( table="redshift_table", ) ``` Using a query: ```python from feast import RedshiftSource my_redshift_source = RedshiftSource( query="SELECT timestamp as ts, created, f1, f2 " "FROM redshift_table", ) ``` The full set of configuration options is available [here](https://rtd.feast.dev/en/master/#feast.infra.offline_stores.redshift_source.RedshiftSource). ## Supported Types Redshift data sources support all eight primitive types, but currently do not support array types. For a comparison against other batch data sources, please see [here](/master/reference/data-sources/overview#functionality-matrix). # Push ## Description Push sources allow feature values to be pushed to the online store and offline store in real time. This allows fresh feature values to be made available to applications. Push sources supercede the [FeatureStore.write\_to\_online\_store](https://rtd.feast.dev/en/latest/index.html#feast.feature_store.FeatureStore.write_to_online_store). Push sources can be used by multiple feature views. When data is pushed to a push source, Feast propagates the feature values to all the consuming feature views. Push sources can optionally have a batch\_source specified. If provided, it enables retrieval of historical features and supports materialization from the offline store to the online store. However, if your features are generated post-training or are only needed online (e.g., embeddings), you can omit the batch\_source. When a batch\_source is used, users are responsible for ensuring that data is also pushed to a batch data source, such as a data warehouse. Note that when a push source is used as a stream source in a feature view definition, a batch\_source does not need to be explicitly specified in the feature view itself. ## Stream sources Streaming data sources are important sources of feature values. A typical setup with streaming data looks like: 1. Raw events come in (stream 1) 2. Streaming transformations applied (e.g. generating features like `last_N_purchased_categories`) (stream 2) 3. Write stream 2 values to an offline store as a historical log for training (optional) 4. Write stream 2 values to an online store for low latency feature serving 5. Periodically materialize feature values from the offline store into the online store for decreased training-serving skew and improved model performance Feast allows users to push features previously registered in a feature view to the online store for fresher features. It also allows users to push batches of stream data to the offline store by specifying that the push be directed to the offline store. This will push the data to the offline store declared in the repository configuration used to initialize the feature store. ## Example (basic) ### Defining a push source Note that the push schema needs to also include the entity. ```python from feast import Entity, PushSource, ValueType, BigQuerySource, FeatureView, Feature, Field from feast.types import Int64 push_source = PushSource( name="push_source", batch_source=BigQuerySource(table="test.test"), ) user = Entity(name="user", join_keys=["user_id"]) fv = FeatureView( name="feature view", entities=[user], schema=[Field(name="life_time_value", dtype=Int64)], source=push_source, ) ``` ### Pushing data Note that the `to` parameter is optional and defaults to online but we can specify these options: `PushMode.ONLINE`, `PushMode.OFFLINE`, or `PushMode.ONLINE_AND_OFFLINE`. ```python from feast import FeatureStore import pandas as pd from feast.data_source import PushMode fs = FeatureStore(...) feature_data_frame = pd.DataFrame() fs.push("push_source_name", feature_data_frame, to=PushMode.ONLINE_AND_OFFLINE) ``` See also [Python feature server](/master/reference/feature-servers/python-feature-server) for instructions on how to push data to a deployed feature server. ## Example (Spark Streaming) The default option to write features from a stream is to add the Python SDK into your existing PySpark pipeline. ```python from feast import FeatureStore store = FeatureStore(...) spark = SparkSession.builder.getOrCreate() streamingDF = spark.readStream.format(...).load() def feast_writer(spark_df): pandas_df = spark_df.to_pandas() store.push("driver_hourly_stats", pandas_df) streamingDF.writeStream.foreachBatch(feast_writer).start() ``` This can also be used under the hood by a contrib stream processor (see [Tutorial: Building streaming features](/master/tutorials/building-streaming-features)) # Kafka **Warning**: This is an *experimental* feature. It's intended for early testing and feedback, and could change without warnings in future releases. ## Description Kafka sources allow users to register Kafka streams as data sources. Feast currently does not launch or monitor jobs to ingest data from Kafka. Users are responsible for launching and monitoring their own ingestion jobs, which should write feature values to the online store through [FeatureStore.write\_to\_online\_store](https://rtd.feast.dev/en/latest/index.html#feast.feature_store.FeatureStore.write_to_online_store). An example of how to launch such a job with Spark can be found [here](https://github.com/feast-dev/feast/tree/master/sdk/python/feast/infra/contrib). Feast also provides functionality to write to the offline store using the `write_to_offline_store` functionality. Kafka sources must have a batch source specified. The batch source will be used for retrieving historical features. Thus users are also responsible for writing data from their Kafka streams to a batch data source such as a data warehouse table. When using a Kafka source as a stream source in the definition of a feature view, a batch source doesn't need to be specified in the feature view definition explicitly. ## Stream sources Streaming data sources are important sources of feature values. A typical setup with streaming data looks like: 1. Raw events come in (stream 1) 2. Streaming transformations applied (e.g. generating features like `last_N_purchased_categories`) (stream 2) 3. Write stream 2 values to an offline store as a historical log for training (optional) 4. Write stream 2 values to an online store for low latency feature serving 5. Periodically materialize feature values from the offline store into the online store for decreased training-serving skew and improved model performance ## Example ### Defining a Kafka source Note that the Kafka source has a batch source. ```python from datetime import timedelta from feast import Field, FileSource, KafkaSource, stream_feature_view from feast.data_format import JsonFormat from feast.types import Float32 driver_stats_batch_source = FileSource( name="driver_stats_source", path="data/driver_stats.parquet", timestamp_field="event_timestamp", ) driver_stats_stream_source = KafkaSource( name="driver_stats_stream", kafka_bootstrap_servers="localhost:9092", topic="drivers", timestamp_field="event_timestamp", batch_source=driver_stats_batch_source, message_format=JsonFormat( schema_json="driver_id integer, event_timestamp timestamp, conv_rate double, acc_rate double, created timestamp" ), watermark_delay_threshold=timedelta(minutes=5), ) ``` ### Using the Kafka source in a stream feature view The Kafka source can be used in a stream feature view. ```python @stream_feature_view( entities=[driver], ttl=timedelta(seconds=8640000000), mode="spark", schema=[ Field(name="conv_percentage", dtype=Float32), Field(name="acc_percentage", dtype=Float32), ], timestamp_field="event_timestamp", online=True, source=driver_stats_stream_source, ) def driver_hourly_stats_stream(df: DataFrame): from pyspark.sql.functions import col return ( df.withColumn("conv_percentage", col("conv_rate") * 100.0) .withColumn("acc_percentage", col("acc_rate") * 100.0) .drop("conv_rate", "acc_rate") ) ``` ### Ingesting data See [here](https://github.com/feast-dev/streaming-tutorial) for an example of how to ingest data from a Kafka source into Feast. --- [Next Page](/llms-full.txt/1)