1 of 100

master

Introduction

What is Feast?

Feast (Feature Store) is an open-source feature store that helps teams operate production ML systems at scale by allowing them to define, manage, validate, and serve features for production AI/ML.

Feast's feature store is composed of two foundational components: (1) an offline store for historical feature extraction used in model training and (2) an online store for serving features at low-latency in production systems and applications.

Feast is a configurable operational data system that re-uses existing infrastructure to manage and serve machine learning features to real-time models. For more details, please review our architecture.

Concretely, Feast provides:

A Python SDK for programmatically defining features, entities, sources, and (optionally) transformations
A Python SDK for reading and writing features to configured offline and online data stores
An for reading and writing features (useful for non-python languages)
A for viewing and exploring information about features defined in the project
A for viewing and updating feature information

Feast allows ML platform teams to:

Make features consistently available for training and low-latency serving by managing an offline store (to process historical data for scale-out batch scoring or model training), a low-latency online store (to power real-time prediction), and a battle-tested feature server (to serve pre-computed features online).
Avoid data leakage by generating point-in-time correct feature sets so data scientists can focus on feature engineering rather than debugging error-prone dataset joining logic. This ensures that future feature values do not leak to models during training.

Feast helps ML platform/MLOps teams with DevOps experience productionize real-time models. Feast also helps these teams build a feature platform that improves collaboration between data engineers, software engineers, machine learning engineers, and data scientists.

For Data Scientists: Feast is a tool where you can easily define, store, and retrieve your features for both model development and model deployment. By using Feast, you can focus on what you do best: build features that power your AI/ML models and maximize the value of your data.
For MLOps Engineers: Feast is a library that allows you to connect your existing infrastructure (e.g., online database, application server, microservice, analytical database, and orchestration tooling) that enables your Data Scientists to ship features for their models to production using a friendly SDK without having to be concerned with software engineering challenges that occur from serving real-time production systems. By using Feast, you can focus on maintaining a resilient system, instead of implementing features for Data Scientists.

An / system. Feast is not a general purpose data pipelining system. Users often leverage tools like to manage upstream data transformations. Feast does support some .
A data orchestration tool: Feast does not manage or orchestrate complex workflow DAGs. It relies on upstream data pipelines to produce feature values and integrations with tools like to make features consistently available.
A data warehouse: Feast is not a replacement for your data warehouse or the source of truth for all transformed data in your organization. Rather, Feast is a lightweight downstream layer that can serve data from an existing data warehouse (or other data sources) to models in production.

reproducible model training / model backtesting / experiment management: Feast captures feature and model metadata, but does not version-control datasets / labels or manage train / test splits. Other tools like , , and are better suited for this.
batch feature engineering: Feast supports on-demand and streaming transformations. Feast is also investing in supporting batch transformations.
native streaming feature integration: Feast enables users to push streaming features, but does not pull from streaming sources or manage streaming pipelines.

Many companies have used Feast to power real-world ML use cases such as:

Personalizing online recommendations by leveraging pre-computed historical user or item features.
Online fraud detection, using features that compare against (pre-computed) historical transaction patterns
Churn prediction (an offline model), generating feature values for all users at a fixed cadence in batch

Explore the following resources to get started with Feast:

is the fastest way to get started with Feast
describes all important Feast API concepts
describes Feast's overall architecture.

Blog

Welcome to the Feast blog! Here you'll find articles about feature store development, new features, and community updates.

Featured Posts

https://github.com/feast-dev/feast/blob/master/docs/blog/what-is-a-feature-store.md https://github.com/feast-dev/feast/blob/master/docs/blog/the-future-of-feast.md https://github.com/feast-dev/feast/blob/master/docs/blog/feast-supports-vector-database.md https://github.com/feast-dev/feast/blob/master/docs/blog/rbac-role-based-access-controls.md

Community & getting help

Links & Resources

Come say hi on Slack!
- As a part of the Linux Foundation, we ask community members to adhere to the Linux Foundation Code of Conduct
GitHub Repository: Find the complete Feast codebase on GitHub.
- : See the governance model of Feast, including who the maintainers are and how decisions are made.
: This folder is used as a central repository for all Feast resources. For example:
- Design proposals in the form of Request for Comments (RFC).
- User surveys and meeting minutes.
: Our LFAI wiki page contains links to resources for contributors and maintainers.

GitHub Issues: Found a bug or need a feature? .

Getting started

Architecture

Overview Language Push vs Pull Model Write Patterns Feature Transformation Feature Serving and Model Inference Role-Based Access Control (RBAC)

Overview

Feast's architecture is designed to be flexible and scalable. It is composed of several components that work together to provide a feature store that can be used to serve features for training and inference.

Feast uses a to ingest data from different sources and store feature values in the online store. This allows Feast to serve features in real-time with low latency.
Feast supports for On Demand and Streaming data sources and will support Batch transformations in the future. For Streaming and Batch data sources, Feast requires a separate (in the batch case, this is typically your Offline Store). We are exploring adding a default streaming engine to Feast.

Language

Use Python to serve your features.

Why should you use Python to Serve features for Machine Learning?

Python has emerged as the primary language for machine learning, and this extends to feature serving and there are five main reasons Feast recommends using a microservice written in Python.

1. Python is the language of Machine Learning

You should meet your users where they are. Python’s popularity in the machine learning community is undeniable. Its simplicity and readability make it an ideal language for writing and understanding complex algorithms. Python boasts a rich ecosystem of libraries such as TensorFlow, PyTorch, XGBoost, and scikit-learn, which provide robust support for developing and deploying machine learning models and we want Feast in this ecosystem.

2. Precomputation is The Way

Precomputing features is the recommended optimal path to ensure low latency performance. Reducing feature serving to a lightweight database lookup is the ideal pattern, which means the marginal overhead of Python should be tolerable. Precomputation ensures product experiences for downstream services are also fast. Slow user experiences are bad user experiences. Precompute and persist data as much as you can.

3. Serving features in another language can lead to skew

Ensuring that features used during model training (offline serving) and online serving are available in production to make real-time predictions is critical. When features are initially developed, they are typically written in Python. This is due to the convenience and efficiency provided by Python's data manipulation libraries. However, in a production environment, there is often interest or pressure to rewrite these features in a different language, like Java, Go, or C++, for performance reasons. This reimplementation introduces a significant risk: training and serving skew. Note that there will always be some minor exceptions (e.g., any Time Since Last Event types of features) but this should not be the rule.

Training and serving skew occurs when there are discrepancies between the features used during model training and those used during prediction. This can lead to degraded model performance, unreliable predictions, and reduced velocity in releasing new features and new models. The process of rewriting features in another language is prone to errors and inconsistencies, which exacerbate this issue.

Rewriting features in another language is not only risky but also resource-intensive. It requires significant time and effort from engineers to ensure that the features are correctly translated. This process can introduce bugs and inconsistencies, further increasing the risk of training and serving skew. Additionally, maintaining two versions of the same feature codebase adds unnecessary complexity and overhead. More importantly, the opportunity cost of this work is high and requires twice the amount of resourcing. Reimplementing code should only be done when the performance gains are worth the investment. Features should largely be precomputed so the latency performance gains should not be the highest impact work that your team can accomplish.

Rather than switching languages, it is more efficient to optimize the performance of your feature store while keeping Python as the primary language. Optimization is a two step process.

Use tools like to understand latency bottlenecks in your code. This will help you prioritize the biggest inefficiencies first. When we initially launched Python native transformations in Python, helped us identify that Pandas resulted in a 10x overhead due to type conversion.

As mentioned, precomputation is the recommended path. In some cases, you may want fully synchronous writes from your data producer to your online feature store, in which case you will want your feature computations and writes to be very fast. In this case, we recommend optimizing the feature calculation code first.

You should optimize your code using libraries, tools, and caching. For example, identify whether your feature calculations can be optimized through vectorized calculations in NumPy; explore tools like Numba for faster execution; and cache frequently accessed data using tools like an lru_cache.

Lastly, Feast will continue to optimize serving in Python and making the overall infrastructure more performant. This will better serve the community.

So we recommend focusing on optimizing your feature-specific code, reporting latency bottlenecks to the maintainers, and contributing to help the infrastructure be more performant.

By keeping features in Python and optimizing performance, you can ensure consistency between training and serving, reduce the risk of errors, and focus on launching more product experiences for your customers.

Embrace Python for feature serving, and leverage its strengths to build robust and reliable machine learning systems.

Push vs Pull Model

Feast uses a , i.e., Data Producers push data to the feature store and Feast stores the feature values in the online store, to serve features in real-time.

In a , Feast would pull data from the data producers at request time and store the feature values in the online store before serving them (storing them would actually be unnecessary). This approach would incur additional network latency as Feast would need to orchestrate a request to each data producer, which would mean the latency would be at least as long as your slowest call. So, in order to serve features as fast as possible, we push data to Feast and store the feature values in the online store.

The trade-off with the Push Model is that strong consistency is not guaranteed out of the box. Instead, strong consistency has to be explicitly designed for in orchestrating the updates to Feast and the client usage.

The significant advantage with this approach is that Feast is read-optimized for low-latency feature retrieval.

Implicit in the Push model are decisions about how and when to push feature values to the online store.

Role-Based Access Control (RBAC)

Introduction

Role-Based Access Control (RBAC) is a security mechanism that restricts access to resources based on the roles/groups/namespaces of individual users within an organization. In the context of the Feast, RBAC ensures that only authorized users or groups/namespaces can access or modify specific resources, thereby maintaining data security and operational integrity.

Functional Requirements

The RBAC implementation in Feast is designed to:

Assign Permissions: Allow administrators to assign permissions for various operations and resources to users or groups/namespaces.
Seamless Integration: Integrate smoothly with existing business code without requiring significant modifications.
Backward Compatibility: Maintain support for non-authorized models as the default to ensure backward compatibility.

The primary business goals of implementing RBAC in the Feast are:

Feature Sharing: Enable multiple teams to share the feature store while ensuring controlled access. This allows for collaborative work without compromising data security.
Access Control Management: Prevent unauthorized access to team-specific resources and spaces, governing the operations that each user or group can perform.

Feast operates as a collection of connected services, each enforcing authorization permissions. The architecture is designed as a distributed microservices system with the following key components:

Service Endpoints: These enforce authorization permissions, ensuring that only authorized requests are processed.
Client Integration: Clients authenticate with feature servers by attaching authorization token to each request.
Service-to-Service Communication: This is always granted.

The RBAC system in Feast uses a permission model that defines the following concepts:

Resource: An object within Feast that needs to be secured against unauthorized access.
Action: A logical operation performed on a resource, such as Create, Describe, Update, Delete, Read, or write operations.
Policy: A set of rules that enforce authorization decisions on resources. The polices are based on user roles or groups or namespaces or combined.

The authorization architecture in Feast is built with the following components:

Token Extractor: Extracts the authorization token from the request header.
Token Parser: Parses the token to retrieve user details.
Policy Enforcer: Validates the secured endpoint against the retrieved user details.

Concepts

Overview Project Data ingestion Entity Data types Feature repository Feature view Batch feature view Stream feature view Tiling with Intermediate Representations Feature retrieval Point-in-time joins [Alpha] Saved dataset Label view Permission Tags

Overview

Feast project structure

The top-level namespace within Feast is a project. Users define one or more feature views within a project. Each feature view contains one or more features. These features typically relate to one or more entities. A feature view must always have a data source, which in turn is used during the generation of training datasets and when materializing feature values into the online store. You can read more about Feast projects in the project page.

Data ingestion

For offline use cases that only rely on batch data, Feast does not need to ingest data and can query your existing data (leveraging a compute engine, whether it be a data warehouse or (experimental) Spark / Trino). Feast can help manage pushing streaming features to a batch source to make features available for training.

For online use cases, Feast supports ingesting features from batch sources to make them available online (through a process called materialization), and pushing streaming features to make them available both offline / online. We explore this more in the next concept page (Data ingestion)

Feature registration and retrieval

Features are registered as code in a version controlled repository, and tie to data sources + model versions via the concepts of entities, feature views, and feature services. We explore these concepts more in the upcoming concept pages. These features are then stored in a registry, which can be accessed across users and services. The features can then be retrieved via SDK API methods or via a deployed feature server which exposes endpoints to query for online features (to power real time models).

Feast supports several patterns of feature retrieval.

Use case

Example

API

Project

Projects provide complete isolation of feature stores at the infrastructure level. This is accomplished through resource namespacing, e.g., prefixing table names with the associated project. Each project should be considered a completely separate universe of entities and features. It is not possible to retrieve features from multiple projects in a single request. We recommend having a single feature store and a single project per environment (dev, staging, prod).

Users define one or more within a project. Each feature view contains one or more . These features typically relate to one or more . A feature view must always have a , which in turn is used during the generation of training and when materializing feature values into the online store.

The concept of a "project" provide the following benefits:

Entity

An entity is a collection of semantically related features. Users define entities to map to the domain of their use case. For example, a ride-hailing service could have customers and drivers as their entities, which group related features that correspond to these customers and drivers.

driver = Entity(name='driver', join_keys=['driver_id'])

The entity name is used to uniquely identify the entity (for example to show in the experimental Web UI). The join key is used to identify the physical primary key on which feature values should be joined together to be retrieved during feature retrieval.

Entities are used by Feast in many contexts, as we explore below:

Use case #1: Defining and storing features

Feast's primary object for defining features is a feature view, which is a collection of features. Feature views map to 0 or more entities, since a feature can be associated with:

zero entities (e.g. a global feature like num_daily_global_transactions)
one entity (e.g. a user feature like user_age or last_5_bought_items)
multiple entities, aka a composite key (e.g. a user + merchant category feature like num_user_purchases_in_merchant_category)

Feast refers to this collection of entities for a feature view as an entity key.

Entities should be reused across feature views. This helps with discovery of features, since it enables data scientists understand how other teams build features for the entity they are most interested in.

Feast will use the feature view concept to then define the schema of groups of features in a low-latency online store.

At training time, users control what entities they want to look up, for example corresponding to train / test / validation splits. A user specifies a list of entity keys + timestamps they want to fetch correct features for to generate a training dataset.

At serving time, users specify entity key(s) to fetch the latest feature values which can power real-time model prediction (e.g. a fraud detection model that needs to fetch the latest transaction user's features to make a prediction).

Feature repository

Feast users use Feast to manage two important sets of configuration:

Configuration about how to run Feast on your infrastructure
Feature definitions

With Feast, the above configuration can be written declaratively and stored as code in a central location. This central location is called a feature repository. The feature repository is the declarative source of truth for what the desired state of a feature store should be.

A feature repository is the collection of python files that define entities, feature views and data sources. Feature Repos also have a

Stream feature view

StreamFeatureView is a type of feature view in Feast that allows you to define features that are continuously updated from a streaming source. It is designed to handle real-time data ingestion and feature generation, making it suitable for use cases where features need to be updated frequently as new data arrives.

LocalComputeEngine
SparkComputeEngine

Point-in-time joins

Feature values in Feast are modeled as time-series records. Below is an example of a driver feature view with two feature columns (trips_today, and earnings_today):

The above table can be registered with Feast through the following feature view:

from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Int64
from datetime import timedelta

driver = Entity(name="driver", join_keys=["driver_id"])

driver_stats_fv = FeatureView(
    name="driver_hourly_stats",
    entities=[driver],
    schema=[
        Field(name="trips_today", dtype=Int64),
        Field(name="earnings_today", dtype=Float32),
    ],
    ttl=timedelta(hours=2),
    source=FileSource(
        path="driver_hourly_stats.parquet"
    )
)

Feast is able to join features from one or more feature views onto an entity dataframe in a point-in-time correct way. This means Feast is able to reproduce the state of features at a specific point in the past.

Given the following entity dataframe, imagine a user would like to join the above driver_hourly_stats feature view onto it, while preserving the trip_success column:

The timestamps within the entity dataframe above are the events at which we want to reproduce the state of the world (i.e., what the feature values were at those specific points in time). In order to do a point-in-time join, a user would load the entity dataframe and run historical retrieval:

For each row within the entity dataframe, Feast will query and join the selected features from the appropriate feature view data source. Feast will scan backward in time from the entity dataframe timestamp up to a maximum of the TTL time specified.

Below is the resulting joined training dataframe. It contains both the original entity rows and joined feature values:

Three feature rows were successfully joined to the entity dataframe rows. The first row in the entity dataframe was older than the earliest feature rows in the feature view and could not be joined. The last row in the entity dataframe was outside of the TTL window (the event happened 11 hours after the feature row) and also couldn't be joined.

[Alpha] Saved dataset

Feast datasets allow for conveniently saving dataframes that include both features and entities to be subsequently used for data analysis and model training. Data Quality Monitoring was the original motivation for creating the dataset concept. Note that the Great Expectations-based validation that used saved datasets is now deprecated in favor of Feast's built-in Feature Quality Monitoring system, which does not require saved datasets.

Dataset's metadata is stored in the Feast registry and raw data (features, entities, additional input keys and timestamp) is stored in the offline store.

Dataset can be created from:

Results of historical retrieval
[planned] Logging request (including input for ) and response during feature serving
[planned] Logging features during writing to online store (from batch source or stream)

To create a saved dataset from historical features for later retrieval or analysis, a user needs to call get_historical_features method first and then pass the returned retrieval job to create_saved_dataset method. create_saved_dataset will trigger the provided retrieval job (by calling .persist() on it) to store the data using the specified storage behind the scenes. Storage type must be the same as the globally configured offline store (e.g it's impossible to persist data to a different offline source). create_saved_dataset will also create a SavedDataset object with all of the related metadata and will write this object to the registry.

Saved dataset can be retrieved later using the get_saved_dataset method in the feature store:

Check out our to see how this concept can be applied in a real-world use case.

Components

Registry Offline store Online store Feature server Compute Engine Stream Processor Provider Authorization Manager OpenTelemetry Integration

Overview

Functionality

Create Batch Features: ELT/ETL systems like Spark and SQL are used to transform data in the batch store.
Create Stream Features: Stream features are created from streaming services such as Kafka or Kinesis, and can be pushed directly into Feast via the Push API.
Feast Apply: The user (or CI) publishes versioned controlled feature definitions using feast apply. This CLI command updates infrastructure and persists definitions in the object store registry.
Feast Materialize: The user (or scheduler) executes feast materialize (with timestamps or --disable-event-timestamp to materialize all data with current timestamps) which loads features from the offline store into the online store.
Model Training: A model training pipeline is launched. It uses the Feast Python SDK to retrieve a training dataset that can be used for training models.
Get Historical Features: Feast exports a point-in-time correct training dataset based on the list of features and entity dataframe provided by the model training pipeline.
Deploy Model: The trained model binary (and list of features) are deployed into a model serving system. This step is not executed by Feast.
Prediction: A backend system makes a request for a prediction from the model serving service.
Get Online Features: The model serving service makes a request to the Feast Online Serving service for online features using a Feast SDK.
Feature Retrieval: The online serving service retrieves the latest feature values from the online store and returns them to the model serving service.

A complete Feast deployment contains the following components:

Feast Registry: An object store (GCS, S3) based registry used to persist feature definitions that are registered with the feature store. Systems can discover feature data by interacting with the registry through the Feast SDK.
Feast Python SDK/CLI: The primary user facing SDK. Used to:
- Manage version controlled feature definitions.

Offline store

An offline store is an interface for working with historical time-series feature values that are stored in . The OfflineStore interface has several different implementations, such as the BigQueryOfflineStore, each of which is backed by a different storage and compute engine. For more details on which offline stores are supported, please see .

Offline stores are primarily used for two reasons:

Building training datasets from time-series features.

Online store

Feast uses online stores to serve features at low latency. Feature values are loaded from data sources into the online store through materialization, which can be triggered through the materialize command (either with specific timestamps or using --disable-event-timestamp to materialize all data with current timestamps).

The storage schema of features within the online store mirrors that of the original data source. One key difference is that for each , only the latest feature values are stored. No historical values are stored.

Here is an example batch data source:

Once the above data source is materialized into Feast (using feast materialize

Feature server

The Feature Server is a core architectural component in Feast, designed to provide low-latency feature retrieval and updates for machine learning applications.

It is a REST API server built using FastAPI and exposes a limited set of endpoints to serve features, push data, and support materialization operations. The server is scalable, flexible, and designed to work seamlessly with various deployment environments, including local setups and cloud-based systems.

Motivation

In machine learning workflows, real-time access to feature values is critical for enabling low-latency predictions. The Feature Server simplifies this requirement by:

Serving Features: Allowing clients to retrieve feature values for specific entities in real-time, reducing the complexity of direct interactions with the online store.
Data Integration: Providing endpoints to push feature data directly into the online or offline store, ensuring data freshness and consistency.
Scalability: Supporting horizontal scaling to handle high request volumes efficiently.
Standardized API: Exposing HTTP/JSON endpoints that integrate seamlessly with various programming languages and ML pipelines.
Secure Communication: Supporting TLS (SSL) for secure data transmission in production environments.

The Feature Server operates as a stateless service backed by two key components:

: The primary data store used for low-latency feature retrieval.
: The metadata store that defines feature sets, feature views, and their relationships to entities.

RESTful API: Provides standardized endpoints for feature retrieval and data pushing.
CLI Integration: Easily managed through the Feast CLI with commands like feast serve.
Flexible Deployment: Can be deployed locally, via Docker, or on Kubernetes using Helm charts.

Endpoint

Description

Stream Processor

A Stream Processor is responsible for consuming data from stream sources (such as Kafka, Kinesis, etc.) and loading it directly into the online (and optionally the offline store).

A Stream Processor abstracts over specific technologies or frameworks that are used to materialize data. An experimental Spark Processor for Kafka is available in Feast.

If the built-in processor is not sufficient, you can create your own custom processor. Please see this tutorial for more details.

Provider

A provider is an implementation of a feature store using specific feature store components (e.g. offline store, online store) targeting a specific environment (e.g. GCP stack).

Providers orchestrate various components (offline store, online store, infrastructure, compute) inside an environment. For example, the gcp provider supports as an offline store and as an online store, ensuring that these components can work together seamlessly. Feast has three built-in providers (local, gcp, and aws) with default configurations that make it easy for users to start a feature store in a specific environment. These default configurations can be overridden easily. For instance, you can use the gcp provider but use Redis as the online store instead of Datastore.

Third party integrations

We integrate with a wide set of tools and technologies so you can make Feast work in your existing stack. Many of these integrations are maintained as plugins to the main Feast repo.

See

In order for a plugin integration to be highlighted, it must meet the following requirements:

The plugin must have tests. Ideally it would use the Feast universal tests (see this for an example), but custom tests are fine.

Tutorials

Sample use-case tutorials

These Feast tutorials showcase how to use Feast to simplify end to end model training / serving.

Driver ranking Fraud detection on GCP Real-time credit scoring on AWS Driver stats on Snowflake

Driver ranking

Making a prediction using a linear regression model is a common use case in ML. This model predicts if a driver will complete a trip based on features ingested into Feast.

In this example, you'll learn how to use some of the key functionality in Feast. The tutorial runs in both local mode and on the Google Cloud Platform (GCP). For GCP, you must have access to a GCP project already, including read and write permissions to BigQuery.

This tutorial guides you on how to use Feast with . You will learn how to:

Train a model locally (on your laptop) using data from

Fraud detection on GCP

A common use case in machine learning, this tutorial is an end-to-end, production-ready fraud prediction system. It predicts in real-time whether a transaction made by a user is fraudulent.

Throughout this tutorial, we’ll walk through the creation of a production-ready fraud prediction system. A prediction is made in real-time as the user makes the transaction, so we need to be able to generate a prediction at low latency.

Our end-to-end example will perform the following workflows:

Computing and backfilling feature data from raw data

Real-time credit scoring on AWS

Credit scoring models are used to approve or reject loan applications. In this tutorial we will build a real-time credit scoring system on AWS.

When individuals apply for loans from banks and other credit providers, the decision to approve a loan application is often made through a statistical model. This model uses information about a customer to determine the likelihood that they will repay or default on a loan, in a process called credit scoring.

In this example, we will demonstrate how a real-time credit scoring system can be built using Feast and Scikit-Learn on AWS, using feature data from S3.

This real-time system accepts a loan request from a customer and responds within 100ms with a decision on whether their loan has been approved or rejected.

This end-to-end tutorial will take you through the following steps:

Building streaming features

Feast supports registering streaming feature views and Kafka and Kinesis streaming sources. It also provides an interface for stream processing called the Stream Processor. An example Kafka/Spark StreamProcessor is implemented in the contrib folder. For more details, please see the RFC for more details.

Please see here for a tutorial on how to build a versioned streaming pipeline that registers your transformations, features, and data sources in Feast.

MCP - AI Agent Example

This example demonstrates how to enable MCP (Model Context Protocol) support in Feast, allowing AI agents and applications to interact with your features through standardized MCP interfaces.

Python 3.8+
Feast installed
FastAPI MCP library

Feature Quality Monitoring Quickstart

How-to Guides

Running Feast with Snowflake/GCP/AWS

Install Feast Create a feature repository Deploy a feature store Build a training dataset Load data into the online store Read features from the online store Scaling Feast Structuring Feature Repos

Install Feast

Install Feast using pip:

pip install feast

Install Feast with Snowflake dependencies (required when using Snowflake):

pip install 'feast[snowflake]'

Install Feast with GCP dependencies (required when using BigQuery or Firestore):

pip install 'feast[gcp]'

Install Feast with AWS dependencies (required when using Redshift or DynamoDB):

Install Feast with Redis dependencies (required when using Redis, either through AWS Elasticache or independently):

Create a feature repository

A feature repository is a directory that contains the configuration of the feature store and individual features. This configuration is written as code (Python/YAML) and it's highly recommended that teams track it centrally using git. See Feature Repository for a detailed explanation of feature repositories.

The easiest way to create a new feature repository to use feast init command:

feast init

Creating a new Feast repository in /<...>/tiny_pika.

feast init -t snowflake
Snowflake

feast init -t gcp

Creating a new Feast repository in /<...>/tiny_pika.

feast init -t aws
AWS Region (e.g. us-west-2): ...
Redshift Cluster ID: ...
Redshift Database Name: ...
Redshift User Name: ...
Redshift S3 Staging Location (s3://*): ...
Redshift IAM Role for S3 (arn:aws:iam::*:role/*): ...
Should I upload example data to Redshift (overwriting 'feast_driver_hourly_stats' table)? (Y/n):

Creating a new Feast repository in /<...>/tiny_pika.

The init command creates a Python file with feature definitions, sample data, and a Feast configuration file for local development:

Enter the directory:

You can now use this feature repository for development. You can try the following:

Run feast apply to apply these definitions to Feast.
Edit the example feature definitions in example.py and run feast apply again to change feature definitions.

Deploy a feature store

The Feast CLI can be used to deploy a feature store to your infrastructure, spinning up any necessary persistent resources like buckets or tables in data stores. The deployment target and effects depend on the provider that has been configured in your file, as well as the feature definitions found in your feature repository.

To have Feast deploy your infrastructure, run feast apply from your command line while inside a feature repository:

Depending on whether the feature repository is configured to use a local provider or one of the cloud providers like GCP or AWS, it may take from a couple of seconds to a minute to run to completion.

Build a training dataset

Feast allows users to build a training dataset from time-series feature data that already exists in an offline store. Users are expected to provide a list of features to retrieve (which may span multiple feature views), and a dataframe to join the resulting features onto. Feast will then execute a point-in-time join of multiple feature views onto the provided dataframe, and return the full resulting dataframe.

Retrieving historical features

1. Register your feature views

Please ensure that you have created a feature repository and that you have registered (applied) your feature views with Feast.

Deploy a feature store

2. Define feature references

Start by defining the feature references (e.g., driver_trips:average_daily_rides) for the features that you would like to retrieve from the offline store. These features can come from multiple feature tables. The only requirement is that the feature tables that make up the feature references have the same entity (or composite entity), and that they aren't located in the same offline store.

feature_refs = [
    "driver_trips:average_daily_rides",
    "driver_trips:maximum_daily_rides",
    "driver_trips:rating",
    "driver_trips:rating:trip_completed",
    # Optionally, reference a specific version: "driver_trips@v2:average_daily_rides"
]

3. Create an entity dataframe

An entity dataframe is the target dataframe on which you would like to join feature values. The entity dataframe must contain a timestamp column called event_timestamp and all entities (primary keys) necessary to join feature tables onto. All entities found in feature views that are being joined onto the entity dataframe must be found as column on the entity dataframe.

It is possible to provide entity dataframes as either a Pandas dataframe or a SQL query.

Pandas:

In the example below we create a Pandas based entity dataframe that has a single row with an event_timestamp column and a driver_id entity column. Pandas based entity dataframes may need to be uploaded into an offline store, which may result in longer wait times compared to a SQL based entity dataframe.

SQL (Alternative):

Below is an example of an entity dataframe built from a BigQuery SQL query. It is only possible to use this query when all feature views being queried are available in the same offline store (BigQuery).

4. Launch historical retrieval

Once the feature references and an entity dataframe are defined, it is possible to call get_historical_features(). This method launches a job that executes a point-in-time join of features from the offline store onto the entity dataframe. Once completed, a job reference will be returned. This job reference can then be converted to a Pandas dataframe by calling to_df().

Load data into the online store

Feast allows users to load their feature data into an online store in order to serve the latest features to models for online prediction.

Before proceeding, please ensure that you have applied (registered) the feature views that should be materialized.

The materialize command allows users to materialize features over a specific historical time range into the online store.

The above command will query the batch sources for all feature views over the provided time range, and load the latest feature values into the configured online store.

It is also possible to materialize for specific feature views by using the -v / --views argument.

The materialize command is completely stateless. It requires the user to provide the time ranges that will be loaded into the online store. This command is best used from a scheduler that tracks state, like Airflow.

Read features from the online store

The Feast Python SDK allows users to retrieve feature values from an online store. This API is used to look up feature values at low latency during model serving in order to make online predictions.

Online stores only maintain the current state of features, i.e latest feature values. No historical data is stored or served.

Retrieving online features

1. Ensure that feature values have been loaded into the online store

Please ensure that you have materialized (loaded) your feature values into the online store before starting

Load data into the online store

2. Define feature references

Create a list of features that you would like to retrieve. This list typically comes from the model training step and should accompany the model binary.

3. Read online features

Next, we will create a feature store object and call get_online_features() which reads the relevant feature values directly from the online store.

Customizing Feast

Feast is highly pluggable and configurable:

One can use existing plugins (offline store, online store, batch materialization engine, providers) and configure those using the built in options. See reference documentation for details.
The other way to customize Feast is to build your own custom components, and then point Feast to delegate to them.

Below are some guides on how to add new custom components:

Adding a new offline store Adding a new online store Adding a custom compute engine Adding a custom provider

Entity Key Serialization (v2 to v3)

Entity Key Serialization version 2 is now deprecated. All new and existing Feast deployments should shift to using serialization version 3.

However, a challenge arises when existing FeatureViews in online or offline stores have written features with version 2. A version 2 serialized entity key cannot be retrieved using the version 3 deserialization algorithm.

Reserialize the Feature Views entity Keys to version 3

The solution is to reserialize the entity keys from version 2 to version 3.

Follow the following procedures to reserialize the entity key to version 3 in feature View in an offline / online store.

In short, you need to iterate through all the feature views in your Feast repository, retrieve their serialized entity keys (if stored in version 2), reserialize them to version 3, and then update the online/offline store or wherever the serialized keys are stored.

1. Initialize the Feature Store

2. Iterate Through Feature Views

3. Retrieve Serialized Entity Keys

4. Reserialize Entity Keys

5. Update the Online/offline Store

Reference

File

Description

File data sources are files on disk or on S3. Currently only Parquet and Delta formats are supported.

Example

from feast import FileSource
from feast.data_format import ParquetFormat

parquet_file_source = FileSource(
    file_format=ParquetFormat(),
    path="file:///feast/customer.parquet",
)

The full set of configuration options is available here.

Supported Types

File data sources support all eight primitive types and their corresponding array types. For a comparison against other batch data sources, please see here.

Type System

Motivation

Feast uses an internal type system to provide guarantees on training and serving data. Feast supports primitive types, array types, set types, map types, JSON, and struct types for feature values. Null types are not supported, although the UNIX_TIMESTAMP type is nullable. The type system is controlled by Value.proto in protobuf and by types.py in Python. Type conversion logic can be found in type_map.py.

Supported Types

Feast supports the following data types:

Primitive Types

Feast Type

Python Type

Description

These types are semantic aliases over Bytes for domain-specific use cases (e.g., RAG pipelines, image processing). They are stored as bytes at the proto level.

Feast Type

Python Type

Description

All primitive types have corresponding array (list) types:

Feast Type

Python Type

Description

All primitive types (except Map and Json) have corresponding set types for storing unique values:

Feast Type

Python Type

Description

Note: Set types automatically remove duplicate values. When converting from lists or other iterables to sets, duplicates are eliminated.

Feast supports arbitrarily nested collections using a recursive VALUE_LIST / VALUE_SET design. The outer container determines the proto enum (VALUE_LIST for Array(…), VALUE_SET for Set(…)), while the full inner type structure is persisted via a mandatory feast:nested_inner_type Field tag.

Feast Type

Python Type

ValueType

Description

Where T is any supported primitive type (Int32, Int64, Float32, Float64, String, Bytes, Bool, UnixTimestamp) or another nested collection type.

Notes:

Nesting depth is unlimited. Array(Array(Array(T))), Set(Array(Set(T))), etc. are all supported.
Inner type information is preserved via Field tags (feast:nested_inner_type) and restored during deserialization. This tag is mandatory for nested collection types.

Map types allow storing dictionary-like data structures:

Feast Type

Python Type

Description

Note: Map keys must always be strings. ScalarMap supports non-string scalar keys — Feast infers ScalarMap automatically when the first key of a dict is not a string. Map values can be any supported Feast type, including primitives, arrays, or nested maps at the proto level. However, the PyArrow representation is map<string, string>, which means backends that rely on PyArrow schemas (e.g., during materialization) treat Map as string-to-string.

Backend support for Map:

Backend

Native Type

Notes

The Json type represents opaque JSON data. Unlike Map, which is schema-free key-value storage, Json is stored as a string at the proto level but backends use native JSON types where available.

Feast Type

Python Type

Description

Backend support for Json:

Backend

Native Type

The Struct type represents a schema-aware structured type with named, typed fields. Unlike Map (which is schema-free), a Struct declares its field names and their types, enabling schema validation.

Feast Type

Python Type

Description

Example:

Backend support for Struct:

Backend

Native Type

Below is a complete example showing how to define a feature view with all supported types:

Sets store unique values and automatically remove duplicates:

UUID types store universally unique identifiers natively, with support for both random UUIDs and time-based UUIDs:

The Decimal type stores arbitrary-precision decimal numbers using Python's decimal.Decimal. Values are stored as strings in the proto to preserve full precision — no floating-point rounding occurs.

The ZonedTimestamp type stores a timezone-aware datetime as both the UTC instant and its originating zone, so the original wall-clock zone round-trips losslessly. By contrast, UnixTimestamp always decodes to UTC and discards the source zone.

Limitation: Empty inner collections round-trip as None:

Maps can store complex nested data structures:

ScalarMap supports non-string keys. Feast infers it automatically when the first dict key is not a string:

Feast's Json type stores values as JSON strings at the proto level. You can pass either a pre-serialized JSON string or a Python dict/list — Feast will call json.dumps() automatically when the value is not already a string:

The sections below explain how Feast uses its type system in different contexts.

During feast apply, Feast runs schema inference on the data sources underlying feature views. For example, if the schema parameter is not specified for a feature view, Feast will examine the schema of the underlying data source to determine the event timestamp column, feature columns, and entity columns. Each of these columns must be associated with a Feast type, which requires conversion from the data source type system to the Feast type system.

The feature inference logic calls _infer_features_and_entities.
_infer_features_and_entities calls source_datatype_to_feast_value_type.

Feast serves feature values as proto objects, which have a type corresponding to Feast types. Thus Feast must materialize feature values into the online store as Value proto objects.

The local materialization engine first pulls the latest historical features and converts it to pyarrow.
Then it calls _convert_arrow_to_proto to convert the pyarrow table to proto format.
This calls python_values_to_proto_values in type_map.py

The Feast type system is typically not necessary when retrieving historical features. A call to get_historical_features will return a RetrievalJob object, which allows the user to export the results to one of several possible locations: a Pandas dataframe, a pyarrow table, a data lake (e.g. S3 or GCS), or the offline store (e.g. a Snowflake table). In all of these cases, the type conversion is handled natively by the offline store. For example, a BigQuery query exposes a to_dataframe method that will automatically convert the result to a dataframe, without requiring any conversions within Feast.

As mentioned above in the section on , Feast persists feature values into the online store as Value proto objects. A call to get_online_features will return an OnlineResponse object, which essentially wraps a bunch of Value protos with some metadata. The OnlineResponse object can then be converted into a Python dictionary, which calls feast_value_type_to_python_type from type_map.py, a utility that converts the Feast internal types to Python native types.

Online Server Performance Tuning

This guide covers end-to-end performance tuning for the Feast Python online feature server, with a focus on Kubernetes deployments using the Feast Operator. It brings together worker configuration, registry caching, online store selection, horizontal scaling, observability, and network optimization into a single operational playbook.

Request lifecycle

Understanding where time is spent during a get_online_features() call helps you focus tuning efforts on the right layer:

Registry lookup — resolve feature references to metadata (cached locally).
Online store read — fetch feature values from the backing database (network I/O).
On-demand transformation — execute any ODFVs attached to the request (CPU).
Response serialization — encode the result as JSON or protobuf.

In most production deployments, step 2 (online store read) dominates latency. Registry refresh (step 1) can cause periodic latency spikes if not configured for background refresh. Step 3 matters only when ODFVs are present.

How you organize features into feature views directly affects online serving latency, because each distinct feature view in a request produces a separate online store read. This is a structural decision that no amount of runtime tuning can fix after the fact.

When the server processes a get_online_features() call, it groups the requested features by their source feature view and issues one online_read() per group. For sync stores, these reads are sequential; for async stores (DynamoDB, MongoDB), they run concurrently via asyncio.gather(), but each is still a separate network round-trip.

Request shape

Store reads

Impact

Guideline: For features that share the same entity key and are frequently requested together, consolidate them into a single feature view. This reduces the number of store round-trips per request. Split feature views only when features have different entities, different materialization schedules, or different data source update frequencies.

A is a named collection of feature references — it's a convenience grouping, not a separate storage or execution unit. Using a feature service adds only a registry lookup (cached) compared to listing features individually. There is no performance penalty for using feature services, and they are the recommended way to define stable, versioned feature sets for production models.

For latency-critical services, feature services also unlock pre-computed feature vectors (precompute_online=True), which reduce store reads from O(N feature views) to O(1). See the section below for details and benchmarks.

Regular feature views incur only store read cost. On-demand feature views add CPU-bound transformation cost on top:

Each ODFV in the request runs its transformation function after all store reads complete. The overhead depends on:

Number of ODFVs in the request — each one adds its transformation time.
Transformation complexity — a simple arithmetic operation is microseconds; ML model inference can be milliseconds.
Transformation mode — mode="python" is 3–10× faster than mode="pandas" for online serving (see

ODFVs declare source feature views. When you request features from an ODFV, the server must also read all source feature views that the ODFV depends on, even if you didn't explicitly request features from those views. This can add unexpected store reads:

Requesting just combined_score triggers reads from both driver_stats_fv and vehicle_stats_fv. If those are already needed by other features in the same request, there's no extra cost (the server deduplicates). But if they're only needed by the ODFV, these are hidden reads you should account for.

Practice

Why

When a get_online_features() request touches multiple feature views, the server issues a separate store read per feature view. For services spanning 5–15+ feature views, this fan-out dominates latency — even with Redis pipeline batching, the protobuf deserialization and response-building overhead grows linearly with the number of views.

Pre-computed feature vectors solve this by storing all of a feature service's features for each entity as a single serialized blob. At read time, the server fetches one blob per entity instead of N reads per feature view, reducing the operation to O(1).

Define a feature service with precompute_online=True:

Apply the feature service: feast apply
Materialize as usual — vectors are built automatically: feast materialize-incremental $(date -u +"%Y-%m-%dT%H:%M:%S")
Read features as usual — the server automatically uses the pre-computed path:

The Python feature server uses Gunicorn with async workers. Tuning workers, connections, and timeouts directly impacts throughput and tail latency.

Parameter

Default

Recommended starting point

Notes

The default registry configuration uses synchronous cache refresh. When the TTL expires, the next get_online_features() call blocks while the full registry is re-downloaded. For large registries this can add tens of milliseconds to a single request.

With the default cache_mode: sync (or unset), the refresh cycle looks like:

Configure the registry to refresh in a background thread so that no serving request ever blocks on a download:

With cache_mode: thread:

The cache is populated eagerly at startup.
A background thread refreshes every cache_ttl_seconds.
Serving requests always read from the in-memory cache and are never blocked.

When deploying with the Feast Operator using a file-based registry, set these fields directly on the CR:

Registry caching is a freshness vs. performance trade-off. The registry contains metadata — feature view definitions, entity schemas, data source configs — not the feature values themselves. When someone runs feast apply to add or modify a feature view, the change is invisible to the serving pods until the cache refreshes.

Lower TTL (e.g., 10–30s):

Schema changes propagate faster to serving pods.
But each refresh re-downloads and deserializes the entire registry, consuming CPU and network bandwidth.
With cache_mode: sync, a lower TTL means more frequent latency spikes — the unlucky request that triggers the refresh is blocked for the full download duration.

Higher TTL (e.g., 300–600s):

Fewer refreshes mean less CPU contention, resulting in more stable and predictable serving latency.
But schema changes take longer to propagate. A new feature view added via feast apply won't be queryable until the cache refreshes (up to cache_ttl_seconds delay).
This is usually acceptable in production where schema changes are infrequent and deployed through CI/CD pipelines, not ad-hoc.

Memory impact: The full registry is deserialized into memory on each worker process. For registries with hundreds of feature views, this can be tens of megabytes per worker. With 4 workers × 5 replicas, that's 20 copies of the registry in memory. A higher TTL doesn't reduce memory usage (the cache is always held), but it reduces the CPU cost of periodic deserialization — which in turn keeps CPU available for serving requests with lower latency.

The online store is the single largest factor in get_online_features() latency. Choose based on your latency budget, throughput needs, and existing infrastructure.

Store

Typical p50 latency

Async read

Best for

Key trade-off

The feature server can read from the online store using either an async or sync code path. The choice is made automatically based on the store's async_supported property — no user configuration is needed.

How it works:

When a get_online_features() request touches multiple feature views, the server must read from each one.
Async path (async_supported.online.read = True): All feature view reads run concurrently via asyncio.gather() in a single event loop — no thread overhead, minimal context switching.

Current async support by store:

Store

Async read

Async write

Notes

When async matters most:

Requests that fan out across many feature views benefit the most — async gathers all reads concurrently without thread pool contention.
For requests hitting a single feature view, the difference is negligible.
If you are choosing between two stores with comparable latency, prefer the one with full async support for better throughput under concurrent load.

Key knobs:

batch_size: DynamoDB's BatchGetItem accepts up to 100 items per request. For 500 entities, this means 5 batches. Keep at 100 unless hitting the 16 MB response limit.
max_read_workers: Controls parallelism for batch reads. With 10 workers, those 5 batches run concurrently (~10 ms) instead of sequentially (~50 ms).

conn_type: pool: Enables connection pooling via psycopg pool. Without this, every request opens and closes a TCP connection.
min_conn / max_conn: Set min_conn to your expected steady-state concurrency and max_conn

Use redis_cluster for horizontal partitioning across shards.
Set key_ttl_seconds to auto-expire stale feature data, keeping memory usage bounded. This is a key-level TTL — it expires the entire entity hash (all feature views for that entity) together.
Ensure the Redis instance is in the same availability zone as the feature server pods to minimize network round-trips.

The Redis online store overrides get_online_features() to issue all HMGET commands — across every feature view in the request — in a single pipeline execution. This reduces Redis round trips from N (one per feature view) to 1 regardless of request size.

Feature views

Round trips (other stores)

Round trips (Redis)

This means the latency cost of adding feature views to a Redis-backed request is primarily serialization and protobuf overhead at the application layer, not Redis network latency.

For initial bulk loads or append-only materialization pipelines:

With skip_dedup: true, online_write_batch() skips the existing-timestamp read pipeline and writes all values directly in a single pass, halving write round trips. Use only when you can guarantee write ordering — under concurrent writers, an older record can overwrite a newer one.

Cassandra and ScyllaDB share the same Feast connector. The key read-path knobs are concurrency and data-center-aware routing:

read_concurrency (default: 100): Maximum concurrent async read operations. Increase for high fan-out workloads; decrease if the cluster shows read-timeout pressure.
write_concurrency (default: 100): Maximum concurrent async write operations during materialization.

Bigtable's client library uses an internal connection pool that caps concurrency:

The client library's connection pool size is hardcoded at 10 internally. For workloads that need higher concurrent reads, scale horizontally (more feature server replicas) rather than trying to push more concurrency through a single pod.
max_versions (default: 2): Number of cell versions retained. Lower values reduce storage and read amplification.
Co-locate the feature server in the same GCP region as your Bigtable instance.

MongoDB supports async reads natively. Use client_kwargs to pass performance options directly to the PyMongo / Motor driver:

maxPoolSize / minPoolSize: Controls the driver connection pool. Default maxPoolSize is 100 in PyMongo, but explicitly setting it ensures predictability across versions.
connectTimeoutMS / socketTimeoutMS: Tighter timeouts improve p99 by failing fast on slow connections.

Aerospike offers sub-millisecond reads thanks to its hybrid-memory architecture (primary index in RAM, data on SSD or RAM). Tune the per-call policies in the Feast config and rely on the Aerospike cluster's own tuning for everything else:

*_timeout_ms (total) vs socket_timeout_ms (per-attempt): *_timeout_ms is the hard deadline for a whole call including retries; socket_timeout_ms is the per-attempt deadline that allows max_retries to actually fire within that budget. Without socket_timeout_ms, a single slow attempt can consume the entire total deadline and retries never run.

The Remote online store connects to a Feast feature server over HTTP. Connection pooling is critical:

connection_pool_size (default: 50): Number of persistent HTTP connections. Increase for high-throughput deployments to reduce connection setup overhead.
connection_idle_timeout (default: 300s): Seconds before idle connections are closed. Set to 0 to disable idle cleanup (useful when traffic is bursty).

Different online stores have different optimal batch sizes for get_online_features() calls. The batch size controls how many entity keys are fetched per round-trip to the store.

Store

Default batch size

Max batch size

Recommendation

Profile your workload by measuring feast_feature_server_online_store_read_duration_seconds (see ) across different entity counts to find the sweet spot.

If your feature pipeline uses ODFVs, the transformation mode significantly affects serving latency.

Native Python transformations avoid pandas overhead and are substantially faster for online serving:

Mode

Relative latency

When to use

When your Python ODFV processes one entity at a time (common in online serving), use singleton=True to avoid the overhead of wrapping and unwrapping single values in lists:

With singleton=True, input values are plain scalars (not lists), and the function returns scalars. This avoids list comprehension and zip overhead for single-entity requests. Singleton mode requires mode="python".

For ODFVs that don't depend on request-time data, set write_to_online_store=True to compute features during materialization instead of at serving time:

This moves CPU cost from the serving path to the materialization path, reducing online latency.

Read-time ODFVs (the default, write_to_online_store=False) are request-blocking — they execute synchronously during get_online_features(). This provides a strong consistency guarantee: the output is always computed from the latest source features in the online store and can incorporate request-time data (e.g., the user's current location or the current timestamp).

Write-time ODFVs (write_to_online_store=True) are pre-computed during materialization and stored. They add zero latency to the serving path, but the result can be stale — it reflects the source feature values at the time of the last materialization run, not the current request.

Use this decision framework:

Condition

Recommendation

If your ODFV uses ML models, lookup tables, or other expensive resources, load them once at server startup using instead of loading them per request:

Without static artifacts, the model would be loaded on every request (or every worker restart), adding seconds of latency. With pre-loading, the ODFV pays only the inference cost.

Using static artifacts with the Feast Operator:

The feature server looks for static_artifacts.py in the feature repository directory at startup. Whether this works in an operator deployment depends on how the repo is created:

feastProjectDir.git — The operator clones your git repository via an init container. Include static_artifacts.py in the repo alongside your feature definitions and it will be available at startup. This is the recommended approach for production.

feastProjectDir.init (default) — The operator runs feast init to create a template repo. The generated template does not include static_artifacts.py. To use static artifacts in this mode, you would need to build a custom feature server image that bundles the file, or mount it via a ConfigMap/volume.

Small choices inside transformation functions add up at scale:

Avoid I/O in the transform function. Network calls, file reads, or database queries inside an ODFV block the serving thread. Pre-load data via static artifacts or use write_to_online_store to compute offline.
Avoid creating DataFrames. Even in Python mode, constructing a pandas DataFrame inside the function defeats the purpose of avoiding pandas overhead.
Minimize allocations. Reuse structures where possible. List comprehensions are faster than building lists with .append()

The feast_feature_server_transformation_duration_seconds metric (labeled by odfv_name and mode) shows exactly how much time each ODFV adds to the serving path.

When a single pod cannot meet your throughput or latency requirements, scale horizontally using the Feast Operator.

Horizontal scaling requires DB-backed persistence for all services. File-based stores (SQLite, DuckDB, registry.db) do not support concurrent access from multiple pods. The Feast Operator supports both static replicas (spec.replicas) and HPA autoscaling (services.scaling.autoscaling). See for full configuration examples.

Replicas

Expected throughput

Notes

When you scale horizontally, every replica and every Gunicorn worker within a replica opens its own database connection pool to the online store. The total connection count multiplies quickly:

Example: 5 replicas × 4 workers × 20 max_conn = 400 connections to the database. If your PostgreSQL instance has the default max_connections: 100, this will fail immediately.

This applies to every connection-oriented online store:

Store

Connection setting

Default

What to check on the store side

How to size it:

Start from the store's limit. Determine the maximum connections your database can handle (e.g., PostgreSQL max_connections, or RDS instance class limit).
Reserve headroom for other clients (materialization jobs, feast apply, monitoring). A common rule is to reserve 20–30% of the store's capacity.
Divide by total workers. Set

Example calculation for PostgreSQL:

Parameter

Value

For DynamoDB and other HTTP-based stores, the concern is less about hard connection limits and more about SDK-level pool exhaustion and throttling. Monitor ThrottledRequests (DynamoDB) or HTTP 429 responses and increase pool sizes or store capacity accordingly.

When scaling is enabled, the operator automatically injects:

Soft pod anti-affinity — prefers spreading pods across nodes.
Zone topology spread — distributes pods across availability zones (best-effort).

You can also configure a PodDisruptionBudget to limit voluntary disruptions:

See for the full horizontal scaling reference, including KEDA integration.

Observability is essential for tuning. Without metrics, you're guessing.

Enable the built-in Prometheus metrics endpoint in the Feast Operator CR:

This exposes a /metrics endpoint on port 8000. If the Prometheus Operator's ServiceMonitor CRD is installed, the Feast operator automatically creates a ServiceMonitor for scraping.

Metric

What to watch

For deeper tracing and integration with your existing observability stack, add OpenTelemetry instrumentation.

1. Install the OpenTelemetry Operator (requires cert-manager):

2. Deploy an OpenTelemetry Collector:

3. Add instrumentation to the FeatureStore pods via the Operator's env field:

See the guide for the complete setup including Instrumentation CRDs and ServiceMonitors.

Network round-trip time to the online store is the dominant factor in get_online_features() latency. No amount of server-side tuning can compensate for a slow network path.

Deploy the feature server pods in the same AWS region and availability zone as your online store (DynamoDB, ElastiCache, RDS, etc.).
On Kubernetes or similar platforms, ensure the cluster and the managed database service share the same VPC and subnet.

Public internet round-trips add 5–50 ms of latency compared to private network paths. Use VPC endpoints or private link:

Cloud

Service

Private endpoint

Frequent DNS lookups to the store endpoint add latency. In Kubernetes, ensure CoreDNS caching is enabled (default) and consider using IP-based endpoints for critical-path connections.

When consuming features from a Feast online server deployed via the Operator, there are three client access patterns with different performance profiles:

Pattern

How it works

Network hops

Client-side overhead

Best for

Direct REST API has the lowest client-side overhead. The client sends a JSON payload and receives a JSON response — no SDK, no registry loading, no protobuf conversion. The feature server handles all processing (registry lookup, store read, ODFVs). This is the best option when:

Your inference service is written in Java, Go, C++, or another non-Python language.
You want to minimize client-side CPU and memory usage.
You're already behind a load balancer and want the simplest possible integration.

Feast SDK with remote store (online_store.type: remote) adds client-side overhead: the SDK loads the registry locally to resolve feature references, converts entities to protobuf, serializes to JSON for HTTP transport, and deserializes the response back to protobuf. This means:

The client needs access to the registry (or a remote registry server).
Each request pays for proto → JSON → proto round-trip conversion.
But you get the full SDK API: feature services, type checking, Python-native OnlineResponse objects, and built-in connection pooling.

Feast SDK direct (online_store.type: postgres/redis/dynamodb/...) skips the feature server entirely and reads from the online store directly. This eliminates the HTTP hop and JSON serialization, giving the lowest end-to-end latency for Python clients. However, it requires:

Direct network access from the client to the online store (often not available in production Kubernetes setups).
Online store credentials on the client side.
Each client manages its own connection pool to the store, which complicates connection budgeting at scale.

In a typical Feast Operator deployment on Kubernetes:

Use the direct REST API for production inference services, especially non-Python ones. It has the least overhead and works with any language.
Use the SDK with remote store for Python services that benefit from the SDK ergonomics (feature services, type safety, OnlineResponse helpers).
Avoid SDK direct in most Operator deployments — the feature server already manages store connections, worker pools, and registry caching. Bypassing it means every client pod opens its own connections to the database, duplicating the connection budget problem.

Here is a complete Feast Operator CR for a production deployment with all the tuning recommendations applied:

With the online store secret configured for connection pooling and background registry refresh:

Materialization (feast materialize / feast materialize-incremental) reads features from the offline store and writes them to the online store. For large feature views, two common bottlenecks arise: memory exhaustion during proto conversion and write throughput to the online store.

By default, Feast converts the entire Arrow table returned by the offline store into Python protobuf objects in a single pass before writing to the online store. For datasets with millions of rows this can consume tens of gigabytes of memory on the materialization worker.

Set online_write_batch_size in feature_store.yaml to break the write into manageable chunks:

Each chunk is independently converted and written, keeping peak memory proportional to the batch size rather than the full dataset. This is supported by the local, Spark, and Ray compute engines.

Choosing a value:

Larger batches (50 000+): fewer write calls to the online store, lower overhead per row — good when worker memory allows.
Smaller batches (1 000–5 000): lower peak memory — necessary for memory-constrained workers or very wide feature views (many features per row).
For Redis: pipeline overhead per batch is negligible; a batch size of 10 000–50 000 is a good starting point.

See the for the complete option documentation.

Each feature view in a feast materialize call is materialized sequentially by the local engine. To materialize multiple feature views in parallel, use a job orchestrator (Airflow, Kubernetes Jobs) and materialize one feature view per job:

Alternatively, use the Spark or Ray compute engines which distribute the work across a cluster.

When performing a full historical reload into Redis (not an incremental update), combine online_write_batch_size with skip_dedup for maximum throughput:

skip_dedup: true eliminates the timestamp-read pipeline before each write (see ), while online_write_batch_size prevents the write worker from converting the entire dataset into memory at once.

— Horizontal scaling, HPA, KEDA, and HA in detail
— CLI flags, metrics reference, and API endpoints
— Full OTEL setup with Prometheus Operator

Feast Production Deployment Topologies

Overview
1. Minimal Production
2. Standard Production (Recommended)

This guide defines three production-ready deployment topologies for Feast on Kubernetes using the . Each topology addresses a different stage of organizational maturity, from getting started safely to running large-scale multi-tenant deployments.

Topology

Target audience

Key traits

Beyond the core topologies, this guide also covers:

— securing resources with policies
— store choices for AWS, GCP, and on-premise
— routing to multiple backends from a single deployment

Small teams with a single ML use case
POCs graduating to production
Low-traffic, non-critical workloads where simplicity is more important than availability

Component

Configuration

Notes

Most production ML workloads
Teams with moderate traffic that need reliability
Environments that require TLS, RBAC, and automated scaling

Core

Component

Configuration

Notes

Storage

Component

Configuration

Notes

Networking & Security

Component

Configuration

Notes

Large organizations with multiple ML teams
Multi-tenant environments requiring strict isolation
High-scale deployments with governance, compliance, and SLA requirements

Each team gets its own registry and online feature server in a dedicated namespace. This provides the strongest isolation but has notable trade-offs: feature discovery is siloed per team (no cross-project visibility), and each registry requires its own deployment — you cannot view multiple projects in a single UI instance.

Alternatively, a single centralized registry server can serve multiple tenant namespaces. Tenant online feature servers connect to the shared registry via the gRPC client. This reduces operational overhead, enables cross-team feature discovery, and allows a single deployment to browse all projects — while Feast enforce tenant isolation at the data level.

Shared registry client configuration — each tenant's feature_store.yaml points to the centralized registry:

Shared Registry

Isolated Registries

Multi-tenancy

Aspect

Configuration

Notes

Storage

Component

Configuration

Notes

Scaling

Component

Configuration

Notes

Security

Component

Configuration

Notes

Observability

Component

Purpose

Notes

Reliability & Disaster Recovery

Aspect

Configuration

Notes

Recovery priority guidance

Not all Feast components carry the same recovery urgency. The table below ranks components by restoration priority and provides guidance for RPO (Recovery Point Objective — maximum acceptable data loss) and RTO (Recovery Time Objective — maximum acceptable downtime). Specific targets depend on your backing store SLAs and organizational requirements.

Priority

Component

RPO guidance

RTO guidance

Rationale

Backup recommendations by topology

Topology

Registry

Online Store

Offline Store

Feast provides a built-in permissions framework that secures resources at the application level, independently of Kubernetes RBAC. Permissions are defined as Python objects in your feature repository and registered via feast apply.

For full details, see the and docs.

Permission enforcement happens on the server side (Online Feature Server, Offline Feature Server, Registry Server). There is no enforcement when using the Feast SDK with a local provider.

Feast defines eight granular actions:

Action

Description

Convenience aliases are provided:

Alias

Includes

Permissions can be applied to any of these Feast object types:

Project, Entity, FeatureView, OnDemandFeatureView, BatchFeatureView, StreamFeatureView, FeatureService, DataSource, ValidationReference, SavedDataset, Permission

The constant ALL_RESOURCE_TYPES includes all of the above. ALL_FEATURE_VIEW_TYPES includes all feature view subtypes.

Policy

Match criteria

Use case

This is the most common pattern — separate admin and read-only roles:

Use NamespaceBasedPolicy to restrict access based on the Kubernetes namespace of the calling service account — ideal for the shared-registry enterprise topology.

Each team gets two permissions: full access to its own resources (matched by team tag), and read-only access to resources any team has explicitly published as shared (matched by visibility: shared tag). The two required_tags target different resources — a feature view tagged team: team-b, visibility: shared matches only the second permission for Team A, enabling cross-team discovery without granting write access:

For organizations that use both OIDC groups and Kubernetes namespaces for identity — ideal when platform engineers lack a dedicated namespace but need cross-team visibility, or when OIDC group membership and namespace ownership should independently grant access:

Permissions support name_patterns (regex) and required_tags for targeting specific resources:

Enable auth enforcement in feature_store.yaml:

For OIDC:

Topology

Auth type

Policy type

Guidance

Choosing the right online store, offline store, and registry backend depends on your cloud environment and existing infrastructure. The table below maps common deployment environments to recommended Feast components.

Component

Recommended

Alternative

Notes

Component

Recommended

Alternative

Notes

Component

Recommended

Alternative

Notes

Production environments in regulated industries (finance, government, defense) often have no outbound internet access from the Kubernetes cluster. The Feast Operator supports air-gapped deployments through custom container images, init container controls, and standard Kubernetes image-pull mechanisms.

When feastProjectDir is set on the FeatureStore CR, the operator creates up to two init containers:

feast-init — bootstraps the feature repository by running either git clone (if feastProjectDir.git is set) or feast init (if feastProjectDir.init is set), then writes the generated feature_store.yaml into the repo directory.

In air-gapped environments, git clone will fail because the cluster cannot reach external Git repositories. The solution is to pre-bake the feature repository into a custom container image and disable the init containers entirely.

Steps:

Build a custom container image that bundles the feature repository and all Python dependencies into the Feast base image.
Push the image to your internal container registry.
Set services.disableInitContainers: true on the FeatureStore CR to skip git clone / feast init

The hybrid store feature allows a single Feast deployment to route feature operations to multiple backends based on tags or data sources. This is useful when different feature views have different latency, cost, or compliance requirements.

The HybridOnlineStore routes online operations to different backends based on a configurable tag on the FeatureView.

feature_store.yaml configuration:

Feature view with routing tag:

The tag value must match the online store type name (e.g. dynamodb, redis, bigtable).

The HybridOfflineStore routes offline operations to different backends based on the batch_source type of each FeatureView.

feature_store.yaml configuration:

Feature views with different sources:

For detailed server-level tuning (worker counts, timeouts, keep-alive, etc.), see the guide.

Traffic tier

Replicas

CPU (per pod)

Memory (per pod)

Notes

Store

p50 latency

p99 latency

Best for

When using the (client-server architecture), connection pooling significantly reduces latency by reusing TCP/TLS connections:

Tuning by workload:

SQL registry (PostgreSQL, MySQL) is required for concurrent materialization jobs writing to the registry simultaneously.
File-based registries (S3, GCS, local) serialize the entire registry on each write — suitable only for single-writer scenarios.
For read-heavy workloads, scale the Registry Server to multiple replicas (all connecting to the same database).

Each Feast server pod maintains its own in-memory copy of the registry metadata. With multiple Gunicorn workers per pod, the total number of independent registry copies is replicas x workers. For example, 5 replicas with 4 workers each means 20 copies of the registry in memory, each refreshing independently.

With the default cache_mode: sync, the refresh is synchronous — when the TTL expires, the next request blocks until the full registry is re-downloaded. At scale, this causes periodic latency spikes across multiple pods simultaneously.

Recommendation: Use cache_mode: thread with a higher TTL in production to avoid refresh storms:

For the server-side refresh interval, set registryTTLSeconds on the CR:

Data volume

Recommended engine

Notes

For detailed engine configuration, see .

Metric

Guideline

Understanding the following principles helps you choose and customize the right topology.

Control plane (Operator + Registry Server) manages feature definitions, metadata, and lifecycle. It changes infrequently and should be highly available.
Data plane (Online Feature Server + Offline Feature Server) handles the actual feature reads/writes at request time. It must scale with traffic.
Backing stores (databases, object storage) hold the actual data. These are stateful and managed independently.

The Feast Operator deploys all Feast services (Online Feature Server, Offline Feature Server, Registry Server) in a single shared Deployment. When scaling (spec.replicas > 1 or HPA autoscaling), all services scale together.

Component

Type

Scaling

DB-backed requirement

Read scaling — increase Online Feature Server replicas; they are stateless and scale linearly.
Write scaling — use a distributed compute engine (, , or ) for materialization.
Storage scaling — scale online and offline stores independently based on data volume and query patterns.

For detailed scaling configuration, see .

Capability

Minimal

Standard

Enterprise

— install the Feast Operator and deploy your first FeatureStore CR
— detailed HPA, registry scaling, and materialization engine configuration
— worker counts, timeouts, keep-alive, and server-level tuning