Welcome to the Feast blog! Here you'll find articles about feature store development, new features, and community updates.

Featured Posts

What is Feast?

Feast (Feature Store) is an open-source feature store that helps teams operate production ML systems at scale by allowing them to define, manage, validate, and serve features for production AI/ML.

Feast's feature store is composed of two foundational components: (1) an offline store for historical feature extraction used in model training and an (2) online store for serving features at low-latency in production systems and applications.

Feast is a configurable operational data system that re-uses existing infrastructure to manage and serve machine learning features to realtime models. For more details, please review our architecture.

Concretely, Feast provides:

• A Python SDK for programmatically defining features, entities, sources, and (optionally) transformations

• A Python SDK for reading and writing features to configured offline and online data stores

• An optional feature server for reading and writing features (useful for non-python languages)

• A UI for viewing and exploring information about features defined in the project

• A CLI tool for viewing and updating feature information

Feast allows ML platform teams to:

• Make features consistently available for training and low-latency serving by managing an offline store (to process historical data for scale-out batch scoring or model training), a low-latency online store (to power real-time prediction), and a battle-tested feature server (to serve pre-computed features online).

• Avoid data leakage by generating point-in-time correct feature sets so data scientists can focus on feature engineering rather than debugging error-prone dataset joining logic. This ensures that future feature values do not leak to models during training.

• Decouple ML from data infrastructure by providing a single data access layer that abstracts feature storage from feature retrieval, ensuring models remain portable as you move from training models to serving models, from batch models to real-time models, and from one data infra system to another.

Who is Feast for?

Feast helps ML platform/MLOps teams with DevOps experience productionize real-time models. Feast also helps these teams build a feature platform that improves collaboration between data engineers, software engineers, machine learning engineers, and data scientists.

• For Data Scientists: Feast is a tool where you can easily define, store, and retrieve your features for both model development and model deployment. By using Feast, you can focus on what you do best: build features that power your AI/ML models and maximize the value of your data.

• For MLOps Engineers: Feast is a library that allows you to connect your existing infrastructure (e.g., online database, application server, microservice, analytical database, and orchestration tooling) that enables your Data Scientists to ship features for their models to production using a friendly SDK without having to be concerned with software engineering challenges that occur from serving real-time production systems. By using Feast, you can focus on maintaining a resilient system, instead of implementing features for Data Scientists.

• For Data Engineers: Feast provides a centralized catalog for storing feature definitions, allowing one to maintain a single source of truth for feature data. It provides the abstraction for reading and writing to many different types of offline and online data stores. Using either the provided Python SDK or the feature server service, users can write data to the online and/or offline stores and then read that data out again in either low-latency online scenarios for model inference, or in batch scenarios for model training.

• For AI Engineers: Feast provides a platform designed to scale your AI applications by enabling seamless integration of richer data and facilitating fine-tuning. With Feast, you can optimize the performance of your AI models while ensuring a scalable and efficient data pipeline.

What Feast is not?

Feast is not

• An ETL / ELT system. Feast is not a general purpose data pipelining system. Users often leverage tools like dbt to manage upstream data transformations. Feast does support some transformations.

• A data orchestration tool: Feast does not manage or orchestrate complex workflow DAGs. It relies on upstream data pipelines to produce feature values and integrations with tools like Airflow to make features consistently available.

• A data warehouse: Feast is not a replacement for your data warehouse or the source of truth for all transformed data in your organization. Rather, Feast is a lightweight downstream layer that can serve data from an existing data warehouse (or other data sources) to models in production.

• A database: Feast is not a database, but helps manage data stored in other systems (e.g. BigQuery, Snowflake, DynamoDB, Redis) to make features consistently available at training / serving time

Feast does not fully solve

• reproducible model training / model backtesting / experiment management: Feast captures feature and model metadata, but does not version-control datasets / labels or manage train / test splits. Other tools like DVC, MLflow, and Kubeflow are better suited for this.

• batch feature engineering: Feast supports on-demand and streaming transformations. Feast is also investing in supporting batch transformations.

• native streaming feature integration: Feast enables users to push streaming features, but does not pull from streaming sources or manage streaming pipelines.

• lineage: Feast helps tie feature values to model versions, but is not a complete solution for capturing end-to-end lineage from raw data sources to model versions. Feast also has community contributed plugins with DataHub and Amundsen.

• data quality / drift detection: Feast has experimental integrations with Great Expectations, but is not purpose built to solve data drift / data quality issues. This requires more sophisticated monitoring across data pipelines, served feature values, labels, and model versions.

Example use cases

Many companies have used Feast to power real-world ML use cases such as:

• Personalizing online recommendations by leveraging pre-computed historical user or item features.

• Online fraud detection, using features that compare against (pre-computed) historical transaction patterns

• Churn prediction (an offline model), generating feature values for all users at a fixed cadence in batch

• Credit scoring, using pre-computed historical features to compute the probability of default

How can I get started?

Explore the following resources to get started with Feast:

• Quickstart is the fastest way to get started with Feast

• Concepts describes all important Feast API concepts

• Architecture describes Feast's overall architecture.

• Tutorials shows full examples of using Feast in machine learning applications.

• Running Feast with Snowflake/GCP/AWS provides a more in-depth guide to using Feast.

• Reference contains detailed API and design documents.

• Contributing contains resources for anyone who wants to contribute to Feast.

What is Feast?

Feast (Feature Store) is an open-source feature store designed to facilitate the management and serving of machine learning features in a way that supports both batch and real-time applications.

• For Data Scientists: Feast is a a tool where you can easily define, store, and retrieve your features for both model development and model deployment. By using Feast, you can focus on what you do best: build features that power your AI/ML models and maximize the value of your data.

• For MLOps Engineers: Feast is a library that allows you to connect your existing infrastructure (e.g., online database, application server, microservice, analytical database, and orchestration tooling) that enables your Data Scientists to ship features for their models to production using a friendly SDK without having to be concerned with software engineering challenges that occur from serving real-time production systems. By using Feast, you can focus on maintaining a resilient system, instead of implementing features for Data Scientists.

• For Data Engineers: Feast provides a centralized catalog for storing feature definitions allowing one to maintain a single source of truth for feature data. It provides the abstraction for reading and writing to many different types of offline and online data stores. Using either the provided python SDK or the feature server service, users can write data to the online and/or offline stores and then read that data out again in either low-latency online scenarios for model inference, or in batch scenarios for model training.

• For AI Engineers: Feast provides a platform designed to scale your AI applications by enabling seamless integration of richer data and facilitating fine-tuning. With Feast, you can optimize the performance of your AI models while ensuring a scalable and efficient data pipeline.

For more info refer to Introduction to feast

Prerequisites

• Ensure that you have Python (3.9 or above) installed.

• It is recommended to create and work in a virtual environment:

  Copy

  # create & activate a virtual environment
  python -m venv venv/
  source venv/bin/activate

Overview

In this tutorial we will:

1. Deploy a local feature store with a Parquet file offline store and Sqlite online store.

2. Build a training dataset using our time series features from our Parquet files.

3. Ingest batch features ("materialization") and streaming features (via a Push API) into the online store.

4. Read the latest features from the offline store for batch scoring

5. Read the latest features from the online store for real-time inference.

6. Explore the (experimental) Feast UI

Note - Feast provides a python SDK as well as an optional hosted service for reading and writing feature data to the online and offline data stores. The latter might be useful when non-python languages are required.

For this tutorial, we will be using the python SDK.

In this tutorial, we'll use Feast to generate training data and power online model inference for a ride-sharing driver satisfaction prediction model. Feast solves several common issues in this flow:

1. Training-serving skew and complex data joins: Feature values often exist across multiple tables. Joining these datasets can be complicated, slow, and error-prone.

   • Feast joins these tables with battle-tested logic that ensures point-in-time correctness so future feature values do not leak to models.

2. Online feature availability: At inference time, models often need access to features that aren't readily available and need to be precomputed from other data sources.

   • Feast manages deployment to a variety of online stores (e.g. DynamoDB, Redis, Google Cloud Datastore) and ensures necessary features are consistently available and freshly computed at inference time.

3. Feature and model versioning: Different teams within an organization are often unable to reuse features across projects, resulting in duplicate feature creation logic. Models have data dependencies that need to be versioned, for example when running A/B tests on model versions.

   • Feast enables discovery of and collaboration on previously used features and enables versioning of sets of features (via feature services).

   • (Experimental) Feast enables light-weight feature transformations so users can re-use transformation logic across online / offline use cases and across models.

Step 1: Install Feast

Install the Feast SDK and CLI using pip:

• In this tutorial, we focus on a local deployment. For a more in-depth guide on how to use Feast with Snowflake / GCP / AWS deployments, see Running Feast with Snowflake/GCP/AWS

pip install feast

Step 2: Create a feature repository

Bootstrap a new feature repository using feast init from the command line.

feast init my_project
cd my_project/feature_repo

Creating a new Feast repository in /home/Jovyan/my_project.

Let's take a look at the resulting demo repo itself. It breaks down into

• data/ contains raw demo parquet data

• example_repo.py contains demo feature definitions

• feature_store.yaml contains a demo setup configuring where data sources are

• test_workflow.py showcases how to run all key Feast commands, including defining, retrieving, and pushing features. You can run this with python test_workflow.py.

project: my_project
# By default, the registry is a file (but can be turned into a more scalable SQL-backed registry)
registry: data/registry.db
# The provider primarily specifies default offline / online stores & storing the registry in a given cloud
provider: local
online_store:
  type: sqlite
  path: data/online_store.db
entity_key_serialization_version: 2

# This is an example feature definition file

from datetime import timedelta

import pandas as pd

from feast import (
    Entity,
    FeatureService,
    FeatureView,
    Field,
    FileSource,
    Project,
    PushSource,
    RequestSource,
)
from feast.on_demand_feature_view import on_demand_feature_view
from feast.types import Float32, Float64, Int64

# Define a project for the feature repo
project = Project(name="my_project", description="A project for driver statistics")


# Define an entity for the driver. You can think of an entity as a primary key used to
# fetch features.
driver = Entity(name="driver", join_keys=["driver_id"])

# Read data from parquet files. Parquet is convenient for local development mode. For
# production, you can use your favorite DWH, such as BigQuery. See Feast documentation
# for more info.
driver_stats_source = FileSource(
    name="driver_hourly_stats_source",
    path="%PARQUET_PATH%",
    timestamp_field="event_timestamp",
    created_timestamp_column="created",
)

# Our parquet files contain sample data that includes a driver_id column, timestamps and
# three feature column. Here we define a Feature View that will allow us to serve this
# data to our model online.
driver_stats_fv = FeatureView(
    # The unique name of this feature view. Two feature views in a single
    # project cannot have the same name
    name="driver_hourly_stats",
    entities=[driver],
    ttl=timedelta(days=1),
    # The list of features defined below act as a schema to both define features
    # for both materialization of features into a store, and are used as references
    # during retrieval for building a training dataset or serving features
    schema=[
        Field(name="conv_rate", dtype=Float32),
        Field(name="acc_rate", dtype=Float32),
        Field(name="avg_daily_trips", dtype=Int64, description="Average daily trips"),
    ],
    online=True,
    source=driver_stats_source,
    # Tags are user defined key/value pairs that are attached to each
    # feature view
    tags={"team": "driver_performance"},
)

# Define a request data source which encodes features / information only
# available at request time (e.g. part of the user initiated HTTP request)
input_request = RequestSource(
    name="vals_to_add",
    schema=[
        Field(name="val_to_add", dtype=Int64),
        Field(name="val_to_add_2", dtype=Int64),
    ],
)


# Define an on demand feature view which can generate new features based on
# existing feature views and RequestSource features
@on_demand_feature_view(
    sources=[driver_stats_fv, input_request],
    schema=[
        Field(name="conv_rate_plus_val1", dtype=Float64),
        Field(name="conv_rate_plus_val2", dtype=Float64),
    ],
)
def transformed_conv_rate(inputs: pd.DataFrame) -> pd.DataFrame:
    df = pd.DataFrame()
    df["conv_rate_plus_val1"] = inputs["conv_rate"] + inputs["val_to_add"]
    df["conv_rate_plus_val2"] = inputs["conv_rate"] + inputs["val_to_add_2"]
    return df


# This groups features into a model version
driver_activity_v1 = FeatureService(
    name="driver_activity_v1",
    features=[
        driver_stats_fv[["conv_rate"]],  # Sub-selects a feature from a feature view
        transformed_conv_rate,  # Selects all features from the feature view
    ],
)
driver_activity_v2 = FeatureService(
    name="driver_activity_v2", features=[driver_stats_fv, transformed_conv_rate]
)

# Defines a way to push data (to be available offline, online or both) into Feast.
driver_stats_push_source = PushSource(
    name="driver_stats_push_source",
    batch_source=driver_stats_source,
)

# Defines a slightly modified version of the feature view from above, where the source
# has been changed to the push source. This allows fresh features to be directly pushed
# to the online store for this feature view.
driver_stats_fresh_fv = FeatureView(
    name="driver_hourly_stats_fresh",
    entities=[driver],
    ttl=timedelta(days=1),
    schema=[
        Field(name="conv_rate", dtype=Float32),
        Field(name="acc_rate", dtype=Float32),
        Field(name="avg_daily_trips", dtype=Int64),
    ],
    online=True,
    source=driver_stats_push_source,  # Changed from above
    tags={"team": "driver_performance"},
)


# Define an on demand feature view which can generate new features based on
# existing feature views and RequestSource features
@on_demand_feature_view(
    sources=[driver_stats_fresh_fv, input_request],  # relies on fresh version of FV
    schema=[
        Field(name="conv_rate_plus_val1", dtype=Float64),
        Field(name="conv_rate_plus_val2", dtype=Float64),
    ],
)
def transformed_conv_rate_fresh(inputs: pd.DataFrame) -> pd.DataFrame:
    df = pd.DataFrame()
    df["conv_rate_plus_val1"] = inputs["conv_rate"] + inputs["val_to_add"]
    df["conv_rate_plus_val2"] = inputs["conv_rate"] + inputs["val_to_add_2"]
    return df


driver_activity_v3 = FeatureService(
    name="driver_activity_v3",
    features=[driver_stats_fresh_fv, transformed_conv_rate_fresh],
)

The feature_store.yaml file configures the key overall architecture of the feature store.

The provider value sets default offline and online stores.

• The offline store provides the compute layer to process historical data (for generating training data & feature values for serving).

• The online store is a low latency store of the latest feature values (for powering real-time inference).

Valid values for provider in feature_store.yaml are:

• local: use a SQL registry or local file registry. By default, use a file / Dask based offline store + SQLite online store

• gcp: use a SQL registry or GCS file registry. By default, use BigQuery (offline store) + Google Cloud Datastore (online store)

• aws: use a SQL registry or S3 file registry. By default, use Redshift (offline store) + DynamoDB (online store)

Note that there are many other offline / online stores Feast works with, including Spark, Azure, Hive, Trino, and PostgreSQL via community plugins. See Third party integrations for all supported data sources.

A custom setup can also be made by following Customizing Feast.

Inspecting the raw data

The raw feature data we have in this demo is stored in a local parquet file. The dataset captures hourly stats of a driver in a ride-sharing app.

import pandas as pd
pd.read_parquet("data/driver_stats.parquet")

Step 3: Run sample workflow

There's an included test_workflow.py file which runs through a full sample workflow:

1. Register feature definitions through feast apply

2. Generate a training dataset (using get_historical_features)

3. Generate features for batch scoring (using get_historical_features)

4. Ingest batch features into an online store (using materialize_incremental)

5. Fetch online features to power real time inference (using get_online_features)

6. Ingest streaming features into offline / online stores (using push)

7. Verify online features are updated / fresher

We'll walk through some snippets of code below and explain

Step 4: Register feature definitions and deploy your feature store

The apply command scans python files in the current directory for feature view/entity definitions, registers the objects, and deploys infrastructure. In this example, it reads example_repo.py and sets up SQLite online store tables. Note that we had specified SQLite as the default online store by configuring online_store in feature_store.yaml.

feast apply

Created entity driver
Created feature view driver_hourly_stats
Created feature view driver_hourly_stats_fresh
Created on demand feature view transformed_conv_rate
Created on demand feature view transformed_conv_rate_fresh
Created feature service driver_activity_v3
Created feature service driver_activity_v1
Created feature service driver_activity_v2

Created sqlite table my_project_driver_hourly_stats_fresh
Created sqlite table my_project_driver_hourly_stats

Step 5: Generating training data or powering batch scoring models

To train a model, we need features and labels. Often, this label data is stored separately (e.g. you have one table storing user survey results and another set of tables with feature values). Feast can help generate the features that map to these labels.

Feast needs a list of entities (e.g. driver ids) and timestamps. Feast will intelligently join relevant tables to create the relevant feature vectors. There are two ways to generate this list:

1. The user can query that table of labels with timestamps and pass that into Feast as an entity dataframe for training data generation.

2. The user can also query that table with a SQL query which pulls entities. See the documentation on feature retrieval for details

• Note that we include timestamps because we want the features for the same driver at various timestamps to be used in a model.

Generating training data

from datetime import datetime
import pandas as pd

from feast import FeatureStore

# Note: see https://docs.feast.dev/getting-started/concepts/feature-retrieval for 
# more details on how to retrieve for all entities in the offline store instead
entity_df = pd.DataFrame.from_dict(
    {
        # entity's join key -> entity values
        "driver_id": [1001, 1002, 1003],
        # "event_timestamp" (reserved key) -> timestamps
        "event_timestamp": [
            datetime(2021, 4, 12, 10, 59, 42),
            datetime(2021, 4, 12, 8, 12, 10),
            datetime(2021, 4, 12, 16, 40, 26),
        ],
        # (optional) label name -> label values. Feast does not process these
        "label_driver_reported_satisfaction": [1, 5, 3],
        # values we're using for an on-demand transformation
        "val_to_add": [1, 2, 3],
        "val_to_add_2": [10, 20, 30],
    }
)

store = FeatureStore(repo_path=".")

training_df = store.get_historical_features(
    entity_df=entity_df,
    features=[
        "driver_hourly_stats:conv_rate",
        "driver_hourly_stats:acc_rate",
        "driver_hourly_stats:avg_daily_trips",
        "transformed_conv_rate:conv_rate_plus_val1",
        "transformed_conv_rate:conv_rate_plus_val2",
    ],
).to_df()

print("----- Feature schema -----\n")
print(training_df.info())

print()
print("----- Example features -----\n")
print(training_df.head())

----- Feature schema -----

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3 entries, 0 to 2
Data columns (total 10 columns):
 #   Column                              Non-Null Count  Dtype              
---  ------                              --------------  -----              
 0   driver_id                           3 non-null      int64              
 1   event_timestamp                     3 non-null      datetime64[ns, UTC]
 2   label_driver_reported_satisfaction  3 non-null      int64              
 3   val_to_add                          3 non-null      int64              
 4   val_to_add_2                        3 non-null      int64              
 5   conv_rate                           3 non-null      float32            
 6   acc_rate                            3 non-null      float32            
 7   avg_daily_trips                     3 non-null      int32              
 8   conv_rate_plus_val1                 3 non-null      float64            
 9   conv_rate_plus_val2                 3 non-null      float64            
dtypes: datetime64[ns, UTC](1), float32(2), float64(2), int32(1), int64(4)
memory usage: 336.0 bytes
None

----- Example features -----

   driver_id           event_timestamp  label_driver_reported_satisfaction  \
0       1001 2021-04-12 10:59:42+00:00                                   1   
1       1002 2021-04-12 08:12:10+00:00                                   5   
2       1003 2021-04-12 16:40:26+00:00                                   3   

   val_to_add  val_to_add_2  conv_rate  acc_rate  avg_daily_trips  \
0           1            10   0.800648  0.265174              643   
1           2            20   0.644141  0.996602              765   
2           3            30   0.855432  0.546345              954   

   conv_rate_plus_val1  conv_rate_plus_val2  
0             1.800648            10.800648  
1             2.644141            20.644141  
2             3.855432            30.855432  

Run offline inference (batch scoring)

To power a batch model, we primarily need to generate features with the get_historical_features call, but using the current timestamp

entity_df["event_timestamp"] = pd.to_datetime("now", utc=True)
training_df = store.get_historical_features(
    entity_df=entity_df,
    features=[
        "driver_hourly_stats:conv_rate",
        "driver_hourly_stats:acc_rate",
        "driver_hourly_stats:avg_daily_trips",
        "transformed_conv_rate:conv_rate_plus_val1",
        "transformed_conv_rate:conv_rate_plus_val2",
    ],
).to_df()

print("\n----- Example features -----\n")
print(training_df.head())

----- Example features -----

   driver_id                  event_timestamp  \
0       1001 2024-04-19 14:58:16.452895+00:00   
1       1002 2024-04-19 14:58:16.452895+00:00   
2       1003 2024-04-19 14:58:16.452895+00:00   

   label_driver_reported_satisfaction  val_to_add  val_to_add_2  conv_rate  \
0                                   1           1            10   0.535773   
1                                   5           2            20   0.171976   
2                                   3           3            30   0.275669   

   acc_rate  avg_daily_trips  conv_rate_plus_val1  conv_rate_plus_val2  
0  0.689705              428             1.535773            10.535773  
1  0.737113              369             2.171976            20.171976  
2  0.156630              116             3.275669            30.275669  

Step 6: Ingest batch features into your online store

We now serialize the latest values of features since the beginning of time to prepare for serving. Note, materialize_incremental serializes all new features since the last materialize call, or since the time provided minus the ttl timedelta. In this case, this will be CURRENT_TIME - 1 day (ttl was set on the FeatureView instances in feature_repo/feature_repo/example_repo.py).

CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")

feast materialize-incremental $CURRENT_TIME

Materializing 2 feature views to 2024-04-19 10:59:58-04:00 into the sqlite online store.

driver_hourly_stats from 2024-04-18 15:00:46-04:00 to 2024-04-19 10:59:58-04:00:
100%|████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 370.32it/s]
driver_hourly_stats_fresh from 2024-04-18 15:00:46-04:00 to 2024-04-19 10:59:58-04:00:
100%|███████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1046.64it/s]
Materializing 2 feature views to 2024-04-19 10:59:58-04:00 into the sqlite online store.

Step 7: Fetching feature vectors for inference

At inference time, we need to quickly read the latest feature values for different drivers (which otherwise might have existed only in batch sources) from the online feature store using get_online_features(). These feature vectors can then be fed to the model.

from pprint import pprint
from feast import FeatureStore

store = FeatureStore(repo_path=".")

feature_vector = store.get_online_features(
    features=[
        "driver_hourly_stats:conv_rate",
        "driver_hourly_stats:acc_rate",
        "driver_hourly_stats:avg_daily_trips",
    ],
    entity_rows=[
        # {join_key: entity_value}
        {"driver_id": 1004},
        {"driver_id": 1005},
    ],
).to_dict()

pprint(feature_vector)

{
 'acc_rate': [0.25351759791374207, 0.8949751853942871],
 'avg_daily_trips': [712, 791],
 'conv_rate': [0.5038306713104248, 0.9839504361152649],
 'driver_id': [1004, 1005]
 }

Step 8: Using a feature service to fetch online features instead.

You can also use feature services to manage multiple features, and decouple feature view definitions and the features needed by end applications. The feature store can also be used to fetch either online or historical features using the same API below. More information can be found here.

The driver_activity_v1 feature service pulls all features from the driver_hourly_stats feature view:

from feast import FeatureService
driver_stats_fs = FeatureService(
    name="driver_activity_v1", features=[driver_stats_fv]
)

from pprint import pprint
from feast import FeatureStore
feature_store = FeatureStore('.')  # Initialize the feature store

feature_service = feature_store.get_feature_service("driver_activity_v1")
feature_vector = feature_store.get_online_features(
    features=feature_service,
    entity_rows=[
        # {join_key: entity_value}
        {"driver_id": 1004},
        {"driver_id": 1005},
    ],
).to_dict()
pprint(feature_vector)

{
 'acc_rate': [0.5732735991477966, 0.7828438878059387],
 'avg_daily_trips': [33, 984],
 'conv_rate': [0.15498852729797363, 0.6263588070869446],
 'driver_id': [1004, 1005]
}

Step 9: Browse your features with the Web UI (experimental)

View all registered features, data sources, entities, and feature services with the Web UI.

One of the ways to view this is with the feast ui command.

feast ui

INFO:     Started server process [66664]
08/17/2022 01:25:49 PM uvicorn.error INFO: Started server process [66664]
INFO:     Waiting for application startup.
08/17/2022 01:25:49 PM uvicorn.error INFO: Waiting for application startup.
INFO:     Application startup complete.
08/17/2022 01:25:49 PM uvicorn.error INFO: Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8888 (Press CTRL+C to quit)
08/17/2022 01:25:49 PM uvicorn.error INFO: Uvicorn running on http://0.0.0.0:8888 (Press CTRL+C to quit)

Step 10: Re-examine test_workflow.py

Take a look at test_workflow.py again. It showcases many sample flows on how to interact with Feast. You'll see these show up in the upcoming concepts + architecture + tutorial pages as well.

Next steps

• Read the Concepts page to understand the Feast data model.

• Read the Architecture page.

• Check out our Tutorials section for more examples on how to use Feast.

• Follow our Running Feast with Snowflake/GCP/AWS guide for a more in-depth tutorial on using Feast.

A feature transformation is a function that takes some set of input data and returns some set of output data. Feature transformations can happen on either raw data or derived data.

Feature Transformation Engines

Feature transformations can be executed by three types of "transformation engines":

1. The Feast Feature Server

2. An Offline Store (e.g., Snowflake, BigQuery, DuckDB, Spark, etc.)

3. A Stream processor (e.g., Flink or Spark Streaming)

Importantly, this implies that different feature transformation code may be used under different transformation engines, so understanding the tradeoffs of when to use which transformation engine/communication pattern is extremely critical to the success of your implementation.

In general, we recommend transformation engines and network calls to be chosen by aligning it with what is most appropriate for the data producer, feature/model usage, and overall product.

The list below contains the functionality that contributors are planning to develop for Feast.

• We welcome contribution to all items in the roadmap!

• Natural Language Processing
• Data Sources
• Offline Stores
• Online Stores
• Feature Engineering
• Streaming
• Deployments
• Feature Serving

  • Python Client
• • Data profiling and validation (Great Expectations)

• Feature Discovery and Governance

  • Python SDK for browsing feature registry

  • CLI for browsing feature registry

  • Model-centric feature tracking (feature services)

  • Feast Lineage Explorer

Push vs Pull Model

The trade-off with the Push Model is that strong consistency is not guaranteed out of the box. Instead, strong consistency has to be explicitly designed for in orchestrating the updates to Feast and the client usage.

The significant advantage with this approach is that Feast is read-optimized for low-latency feature retrieval.

How to Push

Implicit in the Push model are decisions about how and when to push feature values to the online store.

From a developer's perspective, there are three ways to push feature values to the online store with different tradeoffs.

This has two important consequences: (1) communication patterns between the Data Producer (i.e., the client) and Feast (i.e,. the server) and (2) feature computation and feature value write patterns to Feast's online store.

Data Producers (i.e., services that generate data) send data to Feast so that Feast can write feature values to the online store. That data can be either raw data where Feast computes and stores the feature values or precomputed feature values.

Communication Patterns

There are two ways a client (or Data Producer) can send data to the online store:

1. Synchronously
2. Asynchronously

Note, in some contexts, developers may "batch" a group of entities together and write them to the online store in a single API call. This is a common pattern when writing data to the online store to reduce write loads but we would not qualify this as a batch job.

Feature Value Write Patterns

Writing feature values to the online store (i.e., the server) can be done in two ways: Precomputing the transformations client-side or Computing the transformations On Demand server-side.

Combining Approaches

In some scenarios, a combination of Precomputed and On Demand transformations may be optimal.

When selecting feature value write patterns, one must consider the specific requirements of your application, the acceptable correctness of the data, the latency tolerance, and the computational resources available. Making deliberate choices can help the performance and reliability of your service.

There are two ways the client can write feature values to the online store:

1. Precomputing transformations

2. Computing transformations On Demand

3. Hybrid (Precomputed + On Demand)

1. Precomputing Transformations

Precomputed transformations can happen outside of Feast (e.g., via some batch job or streaming application) or inside of the Feast feature server when writing to the online store via the push or write-to-online-store api.

2. Computing Transformations On Demand

On Demand transformations can only happen inside of Feast at either (1) the time of the client's request or (2) when the data producer writes to the online store.

3. Hybrid (Precomputed + On Demand)

The hybrid approach allows for precomputed transformations to happen inside or outside of Feast and have the On Demand transformations happen at client request time. This is particularly convenient for "Time Since Last" types of features (e.g., time since purchase).

Tradeoffs

When deciding between synchronous and asynchronous data writes, several tradeoffs should be considered:

• Data Consistency: Asynchronous writes allow Data Producers to send data without waiting for the write operation to complete, which can lead to situations where the data in the online store is stale. This might be acceptable in scenarios where absolute freshness is not critical. However, for critical operations, such as calculating loan amounts in financial applications, stale data can lead to incorrect decisions, making synchronous writes essential.

• Correctness: The risk of data being out-of-date must be weighed against the operational requirements. For instance, in a lending application, having up-to-date feature data can be crucial for correctness (depending upon the features and raw data), thus favoring synchronous writes. In less sensitive contexts, the eventual consistency offered by asynchronous writes might be sufficient.

• Service Coupling: Synchronous writes result in tighter coupling between services. If a write operation fails, it can cause the dependent service operation to fail as well, which might be a significant drawback in systems requiring high reliability and independence between services.

• Application Latency: Asynchronous writes typically reduce the perceived latency from the client's perspective because the client does not wait for the write operation to complete. This can enhance the user experience and efficiency in environments where operations are not critically dependent on immediate data freshness.

The table below can help guide the most appropriate data write and feature computation strategies based on specific application needs and data sensitivity.

Use Python to serve your features.

Why should you use Python to Serve features for Machine Learning?

Python has emerged as the primary language for machine learning, and this extends to feature serving and there are five main reasons Feast recommends using a microservice written in Python.

1. Python is the language of Machine Learning

You should meet your users where they are. Python’s popularity in the machine learning community is undeniable. Its simplicity and readability make it an ideal language for writing and understanding complex algorithms. Python boasts a rich ecosystem of libraries such as TensorFlow, PyTorch, XGBoost, and scikit-learn, which provide robust support for developing and deploying machine learning models and we want Feast in this ecosystem.

2. Precomputation is The Way

Precomputing features is the recommended optimal path to ensure low latency performance. Reducing feature serving to a lightweight database lookup is the ideal pattern, which means the marginal overhead of Python should be tolerable. Precomputation ensures product experiences for downstream services are also fast. Slow user experiences are bad user experiences. Precompute and persist data as much as you can.

3. Serving features in another language can lead to skew

Ensuring that features used during model training (offline serving) and online serving are available in production to make real-time predictions is critical. When features are initially developed, they are typically written in Python. This is due to the convenience and efficiency provided by Python's data manipulation libraries. However, in a production environment, there is often interest or pressure to rewrite these features in a different language, like Java, Go, or C++, for performance reasons. This reimplementation introduces a significant risk: training and serving skew. Note that there will always be some minor exceptions (e.g., any Time Since Last Event types of features) but this should not be the rule.

Training and serving skew occurs when there are discrepancies between the features used during model training and those used during prediction. This can lead to degraded model performance, unreliable predictions, and reduced velocity in releasing new features and new models. The process of rewriting features in another language is prone to errors and inconsistencies, which exacerbate this issue.

4. Reimplementation is Excessive

Rewriting features in another language is not only risky but also resource-intensive. It requires significant time and effort from engineers to ensure that the features are correctly translated. This process can introduce bugs and inconsistencies, further increasing the risk of training and serving skew. Additionally, maintaining two versions of the same feature codebase adds unnecessary complexity and overhead. More importantly, the opportunity cost of this work is high and requires twice the amount of resourcing. Reimplementing code should only be done when the performance gains are worth the investment. Features should largely be precomputed so the latency performance gains should not be the highest impact work that your team can accomplish.

5. Use existing Python Optimizations

Rather than switching languages, it is more efficient to optimize the performance of your feature store while keeping Python as the primary language. Optimization is a two step process.

Step 1: Quantify latency bottlenecks in your feature calculations

Step 2: Optimize your feature calculations

As mentioned, precomputation is the recommended path. In some cases, you may want fully synchronous writes from your data producer to your online feature store, in which case you will want your feature computations and writes to be very fast. In this case, we recommend optimizing the feature calculation code first.

You should optimize your code using libraries, tools, and caching. For example, identify whether your feature calculations can be optimized through vectorized calculations in NumPy; explore tools like Numba for faster execution; and cache frequently accessed data using tools like an lru_cache.

Lastly, Feast will continue to optimize serving in Python and making the overall infrastructure more performant. This will better serve the community.

So we recommend focusing on optimizing your feature-specific code, reporting latency bottlenecks to the maintainers, and contributing to help the infrastructure be more performant.

By keeping features in Python and optimizing performance, you can ensure consistency between training and serving, reduce the risk of errors, and focus on launching more product experiences for your customers.

Embrace Python for feature serving, and leverage its strengths to build robust and reliable machine learning systems.

Feast project structure

The top-level namespace within Feast is a project. Users define one or more feature views within a project. Each feature view contains one or more features. These features typically relate to one or more entities. A feature view must always have a data source, which in turn is used during the generation of training datasets and when materializing feature values into the online store. You can read more about Feast projects in the project page.

Data ingestion

For offline use cases that only rely on batch data, Feast does not need to ingest data and can query your existing data (leveraging a compute engine, whether it be a data warehouse or (experimental) Spark / Trino). Feast can help manage pushing streaming features to a batch source to make features available for training.

For online use cases, Feast supports ingesting features from batch sources to make them available online (through a process called materialization), and pushing streaming features to make them available both offline / online. We explore this more in the next concept page (Data ingestion)

Feature registration and retrieval

Features are registered as code in a version controlled repository, and tie to data sources + model versions via the concepts of entities, feature views, and feature services. We explore these concepts more in the upcoming concept pages. These features are then stored in a registry, which can be accessed across users and services. The features can then be retrieved via SDK API methods or via a deployed feature server which exposes endpoints to query for online features (to power real time models).

Feast supports several patterns of feature retrieval.

Introduction

Role-Based Access Control (RBAC) is a security mechanism that restricts access to resources based on the roles of individual users within an organization. In the context of the Feast, RBAC ensures that only authorized users or groups can access or modify specific resources, thereby maintaining data security and operational integrity.

Functional Requirements

The RBAC implementation in Feast is designed to:

• Assign Permissions: Allow administrators to assign permissions for various operations and resources to users or groups based on their roles.

• Seamless Integration: Integrate smoothly with existing business code without requiring significant modifications.

• Backward Compatibility: Maintain support for non-authorized models as the default to ensure backward compatibility.

Business Goals

The primary business goals of implementing RBAC in the Feast are:

1. Feature Sharing: Enable multiple teams to share the feature store while ensuring controlled access. This allows for collaborative work without compromising data security.

2. Access Control Management: Prevent unauthorized access to team-specific resources and spaces, governing the operations that each user or group can perform.

Reference Architecture

Feast operates as a collection of connected services, each enforcing authorization permissions. The architecture is designed as a distributed microservices system with the following key components:

• Service Endpoints: These enforce authorization permissions, ensuring that only authorized requests are processed.

• Client Integration: Clients authenticate with feature servers by attaching authorization token to each request.

• Service-to-Service Communication: This is always granted.

Permission Model

The RBAC system in Feast uses a permission model that defines the following concepts:

• Resource: An object within Feast that needs to be secured against unauthorized access.

• Action: A logical operation performed on a resource, such as Create, Describe, Update, Delete, Read, or write operations.

• Policy: A set of rules that enforce authorization decisions on resources. The default implementation uses role-based policies.

Authorization Architecture

The authorization architecture in Feast is built with the following components:

• Token Extractor: Extracts the authorization token from the request header.

• Token Parser: Parses the token to retrieve user details.

• Policy Enforcer: Validates the secured endpoint against the retrieved user details.

• Token Injector: Adds the authorization token to each secured request header.

Data source

A data source in Feast refers to raw underlying data that users own (e.g. in a table in BigQuery). Feast does not manage any of the raw underlying data but instead, is in charge of loading this data and performing different operations on the data to retrieve or serve features.

Feast uses a time-series data model to represent data. This data model is used to interpret feature data in data sources in order to build training datasets or materialize features into an online store.

Below is an example data source with a single entity column (driver) and two feature columns (trips_today, and rating).

Feast supports primarily time-stamped tabular data as data sources. There are many kinds of possible data sources:

• Batch data sources: ideally, these live in data warehouses (BigQuery, Snowflake, Redshift), but can be in data lakes (S3, GCS, etc). Feast supports ingesting and querying data across both.

• Stream data sources: Feast does not have native streaming integrations. It does however facilitate making streaming features available in different environments. There are two kinds of sources:

  • Push sources allow users to push features into Feast, and make it available for training / batch scoring ("offline"), for realtime feature serving ("online") or both.

  • [Alpha] Stream sources allow users to register metadata from Kafka or Kinesis sources. The onus is on the user to ingest from these sources, though Feast provides some limited helper methods to ingest directly from Kafka / Kinesis topics.

Batch data ingestion

Ingesting from batch sources is only necessary to power real-time models. This is done through materialization. Under the hood, Feast manages an offline store (to scalably generate training data from batch sources) and an online store (to provide low-latency access to features for real-time models).

A key command to use in Feast is the materialize_incremental command, which fetches the latest values for all entities in the batch source and ingests these values into the online store.

Materialization can be called programmatically or through the CLI:

Batch data schema inference

If the schema parameter is not specified when defining a data source, Feast attempts to infer the schema of the data source during feast apply. The way it does this depends on the implementation of the offline store. For the offline stores that ship with Feast out of the box this inference is performed by inspecting the schema of the table in the cloud data warehouse, or if a query is provided to the source, by running the query with a LIMIT clause and inspecting the result.

Stream data ingestion

Ingesting from stream sources happens either via a Push API or via a contrib processor that leverages an existing Spark context.

Feast datasets allow for conveniently saving dataframes that include both features and entities to be subsequently used for data analysis and model training. Data Quality Monitoring was the primary motivation for creating dataset concept.

Dataset's metadata is stored in the Feast registry and raw data (features, entities, additional input keys and timestamp) is stored in the offline store.

Dataset can be created from:

1. Results of historical retrieval

2. [planned] Logging request (including input for on demand transformation) and response during feature serving

3. [planned] Logging features during writing to online store (from batch source or stream)

Creating a saved dataset from historical retrieval

To create a saved dataset from historical features for later retrieval or analysis, a user needs to call get_historical_features method first and then pass the returned retrieval job to create_saved_dataset method. create_saved_dataset will trigger the provided retrieval job (by calling .persist() on it) to store the data using the specified storage behind the scenes. Storage type must be the same as the globally configured offline store (e.g it's impossible to persist data to a different offline source). create_saved_dataset will also create a SavedDataset object with all of the related metadata and will write this object to the registry.

from feast import FeatureStore
from feast.infra.offline_stores.bigquery_source import SavedDatasetBigQueryStorage

store = FeatureStore()

historical_job = store.get_historical_features(
    features=["driver:avg_trip"],
    entity_df=...,
)

dataset = store.create_saved_dataset(
    from_=historical_job,
    name='my_training_dataset',
    storage=SavedDatasetBigQueryStorage(table_ref='<gcp-project>.<gcp-dataset>.my_training_dataset'),
    tags={'author': 'oleksii'}
)

dataset.to_df()

Saved dataset can be retrieved later using the get_saved_dataset method in the feature store:

dataset = store.get_saved_dataset('my_training_dataset')
dataset.to_df()

Check out our tutorial on validating historical features to see how this concept can be applied in a real-world use case.

Feature values in Feast are modeled as time-series records. Below is an example of a driver feature view with two feature columns (trips_today, and earnings_today):

The above table can be registered with Feast through the following feature view:

from feast import Entity, FeatureView, Field, FileSource
from feast.types import Float32, Int64
from datetime import timedelta

driver = Entity(name="driver", join_keys=["driver_id"])

driver_stats_fv = FeatureView(
    name="driver_hourly_stats",
    entities=[driver],
    schema=[
        Field(name="trips_today", dtype=Int64),
        Field(name="earnings_today", dtype=Float32),
    ],
    ttl=timedelta(hours=2),
    source=FileSource(
        path="driver_hourly_stats.parquet"
    )
)

Feast is able to join features from one or more feature views onto an entity dataframe in a point-in-time correct way. This means Feast is able to reproduce the state of features at a specific point in the past.

Given the following entity dataframe, imagine a user would like to join the above driver_hourly_stats feature view onto it, while preserving the trip_success column:

The timestamps within the entity dataframe above are the events at which we want to reproduce the state of the world (i.e., what the feature values were at those specific points in time). In order to do a point-in-time join, a user would load the entity dataframe and run historical retrieval:

# Read in entity dataframe
entity_df = pd.read_csv("entity_df.csv")

training_df = store.get_historical_features(
    entity_df=entity_df,
    features = [
        'driver_hourly_stats:trips_today',
        'driver_hourly_stats:earnings_today'
    ],
)

For each row within the entity dataframe, Feast will query and join the selected features from the appropriate feature view data source. Feast will scan backward in time from the entity dataframe timestamp up to a maximum of the TTL time specified.

Below is the resulting joined training dataframe. It contains both the original entity rows and joined feature values:

Three feature rows were successfully joined to the entity dataframe rows. The first row in the entity dataframe was older than the earliest feature rows in the feature view and could not be joined. The last row in the entity dataframe was outside of the TTL window (the event happened 11 hours after the feature row) and also couldn't be joined.

Projects provide complete isolation of feature stores at the infrastructure level. This is accomplished through resource namespacing, e.g., prefixing table names with the associated project. Each project should be considered a completely separate universe of entities and features. It is not possible to retrieve features from multiple projects in a single request. We recommend having a single feature store and a single project per environment (dev, staging, prod).

Users define one or more feature views within a project. Each feature view contains one or more features. These features typically relate to one or more entities. A feature view must always have a data source, which in turn is used during the generation of training datasets and when materializing feature values into the online store.

The concept of a "project" provide the following benefits:

Logical Grouping: Projects group related features together, making it easier to manage and track them.

Feature Definitions: Within a project, you can define features, including their metadata, types, and sources. This helps standardize how features are created and consumed.

Isolation: Projects provide a way to isolate different environments, such as development, testing, and production, ensuring that changes in one project do not affect others.

Collaboration: By organizing features within projects, teams can collaborate more effectively, with clear boundaries around the features they are responsible for.

Access Control: Projects can implement permissions, allowing different users or teams to access only the features relevant to their work.

Offline stores are primarily used for two reasons:

1. Building training datasets from time-series features.

2. Materializing (loading) features into an online store to serve those features at low-latency in a production setting.

Only a single offline store can be used at a time. Moreover, offline stores are not compatible with all data sources; for example, the BigQuery offline store cannot be used to query a file-based data source.

The Feast feature registry is a central catalog of all feature definitions and their related metadata. Feast uses the registry to store all applied Feast objects (e.g. Feature views, entities, etc). It allows data scientists to search, discover, and collaborate on new features. The registry exposes methods to apply, list, retrieve and delete these objects, and is an abstraction with multiple implementations.

Updating the registry

Accessing the registry from clients

Users can specify the registry through a feature_store.yaml config file, or programmatically. We often see teams preferring the programmatic approach because it makes notebook driven development very easy:

Option 1: programmatically specifying the registry

Option 2: specifying the registry in the project's feature_store.yaml file

Instantiating a FeatureStore object can then point to this:

An Authorization Manager is an instance of the AuthManager class that is plugged into one of the Feast servers to extract user details from the current request and inject them into the permission framework.

Two authorization managers are supported out-of-the-box:

• One using a configurable OIDC server to extract the user details.

• One using the Kubernetes RBAC resources to extract the user details.

These instances are created when the Feast servers are initialized, according to the authorization configuration defined in their own feature_store.yaml.

Feast servers and clients must have consistent authorization configuration, so that the client proxies can automatically inject the authorization tokens that the server can properly identify and use to enforce permission validations.

Design notes

The server-side implementation of the authorization functionality is defined here. Few of the key models, classes to understand the authorization implementation on the client side can be found here.

Configuring Authorization

The authorization is configured using a dedicated auth section in the feature_store.yaml configuration.

Note: As a consequence, when deploying the Feast servers with the Helm charts, the feature_store_yaml_base64 value must include the auth section to specify the authorization configuration.

No Authorization

This configuration applies the default no_auth authorization:

project: my-project
auth:
  type: no_auth
...

OIDC Authorization

With OIDC authorization, the Feast client proxies retrieve the JWT token from an OIDC server (or Identity Provider) and append it in every request to a Feast server, using an Authorization Bearer Token.

The server, in turn, uses the same OIDC server to validate the token and extract the user roles from the token itself.

Some assumptions are made in the OIDC server configuration:

• The OIDC token refers to a client with roles matching the RBAC roles of the configured Permissions (*)

• The roles are exposed in the access token that is passed to the server

• The JWT token is expected to have a verified signature and not be expired. The Feast OIDC token parser logic validates for verify_signature and verify_exp so make sure that the given OIDC provider is configured to meet these requirements.

• The preferred_username should be part of the JWT token claim.

(*) Please note that the role match is case-sensitive, e.g. the name of the role in the OIDC server and in the Permission configuration must be exactly the same.

For example, the access token for a client app of a user with reader role should have the following resource_access section:

{
  "resource_access": {
    "app": {
      "roles": [
        "reader"
      ]
    }
  }
}

An example of feast OIDC authorization configuration on the server side is the following:

project: my-project
auth:
  type: oidc
  client_id: _CLIENT_ID__
  auth_discovery_url: _OIDC_SERVER_URL_/realms/master/.well-known/openid-configuration
...

In case of client configuration, the following settings username, password and client_secret must be added to specify the current user:

auth:
  type: oidc
  ...
  username: _USERNAME_
  password: _PASSWORD_
  client_secret: _CLIENT_SECRET__

Below is an example of feast full OIDC client auth configuration:

project: my-project
auth:
  type: oidc
  client_id: test_client_id
  client_secret: test_client_secret
  username: test_user_name
  password: test_password
  auth_discovery_url: http://localhost:8080/realms/master/.well-known/openid-configuration

Kubernetes RBAC Authorization

With Kubernetes RBAC Authorization, the client uses the service account token as the authorizarion bearer token, and the server fetches the associated roles from the Kubernetes RBAC resources.

An example of Kubernetes RBAC authorization configuration is the following:

```yaml project: my-project auth: type: kubernetes ... ```

In case the client cannot run on the same cluster as the servers, the client token can be injected using the LOCAL_K8S_TOKEN environment variable on the client side. The value must refer to the token of a service account created on the servers cluster and linked to the desired RBAC roles.

Setting Up Kubernetes RBAC for Feast

To ensure the Kubernetes RBAC environment aligns with Feast's RBAC configuration, follow these guidelines:

• The roles defined in Feast Permission instances must have corresponding Kubernetes RBAC Role names.

• The Kubernetes RBAC Role must reside in the same namespace as the Feast service.

• The client application can run in a different namespace, using its own dedicated ServiceAccount.

• Finally, the RoleBinding that links the client ServiceAccount to the RBAC Role must be defined in the namespace of the Feast service.

If the above rules are satisfied, the Feast service must be granted permissions to fetch RoleBinding instances from the local namespace.

Feature views

A feature view is defined as a collection of features.

• In the online settings, this is a stateful collection of features that are read when the get_online_features method is called.

• In the offline setting, this is a stateless collection of features that are created when the get_historical_features method is called.

Feature views consist of:

• a name to uniquely identify this feature view in the project.

• (optional, but recommended) metadata (for example, description, or other free-form metadata via tags)

• (optional) a TTL, which limits how far back Feast will look when generating historical datasets

Feature views allow Feast to model your existing feature data in a consistent way in both an offline (training) and online (serving) environment. Feature views generally contain features that are properties of a specific object, in which case that object is defined as an entity and included in the feature view.

Feature views are used during

• The generation of training datasets by querying the data source of feature views in order to find historical feature values. A single training dataset may consist of features from multiple feature views.

• Retrieval of features from the online store. Feature views provide the schema definition to Feast in order to look up features from the online store.

Feature views without entities

If a feature view contains features that are not related to a specific entity, the feature view can be defined without entities (only timestamps are needed for this feature view).

Feature inferencing

If the schema parameter is not specified in the creation of the feature view, Feast will infer the features during feast apply by creating a Field for each column in the underlying data source except the columns corresponding to the entities of the feature view or the columns corresponding to the timestamp columns of the feature view's data source. The names and value types of the inferred features will use the names and data types of the columns from which the features were inferred.

Entity aliasing

"Entity aliases" can be specified to join entity_dataframe columns that do not match the column names in the source table of a FeatureView.

This could be used if a user has no control over these column names or if there are multiple entities are a subclass of a more general entity. For example, "spammer" and "reporter" could be aliases of a "user" entity, and "origin" and "destination" could be aliases of a "location" entity as shown below.

It is suggested that you dynamically specify the new FeatureView name using .with_name and join_key_map override using .with_join_key_map instead of needing to register each new copy.

Field

A field or feature is an individual measurable property. It is typically a property observed on a specific entity, but does not have to be associated with an entity. For example, a feature of a customer entity could be the number of transactions they have made on an average month, while a feature that is not observed on a specific entity could be the total number of posts made by all users in the last month. Supported types for fields in Feast can be found in sdk/python/feast/types.py.

Fields are defined as part of feature views. Since Feast does not transform data, a field is essentially a schema that only contains a name and a type:

[Alpha] On demand feature views

On demand feature views allows data scientists to use existing features and request time data (features only available at request time) to transform and create new features. Users define python transformation logic which is executed in both the historical retrieval and online retrieval paths.

Currently, these transformations are executed locally. This is fine for online serving, but does not scale well to offline retrieval.

Why use on demand feature views?

This enables data scientists to easily impact the online feature retrieval path. For example, a data scientist could

1. Call get_historical_features to generate a training dataframe

2. Iterate in notebook on feature engineering in Pandas

3. Copy transformation logic into on demand feature views and commit to a dev branch of the feature repository

4. Verify with get_historical_features (on a small dataset) that the transformation gives expected output over historical data

5. Verify with get_online_features on dev branch that the transformation correctly outputs online features

6. Submit a pull request to the staging / prod branches which impact production traffic

[Alpha] Stream feature views

A stream feature view is an extension of a normal feature view. The primary difference is that stream feature views have both stream and batch data sources, whereas a normal feature view only has a batch data source.

Stream feature views should be used instead of normal feature views when there are stream data sources (e.g. Kafka and Kinesis) available to provide fresh features in an online setting. Here is an example definition of a stream feature view with an attached transformation:

Overview

The Feast permissions model allows to configure granular permission policies to all the resources defined in a feature store.

The configured permissions are stored in the Feast registry and accessible through the CLI and the registry APIs.

The permission authorization enforcement is performed when requests are executed through one of the Feast (Python) servers

• The online feature server (REST)

• The offline feature server (Arrow Flight)

• The registry server (gRPC)

Note that there is no permission enforcement when accessing the Feast API with a local provider.

Concepts

The permission model is based on the following components:

• A resource is a Feast object that we want to secure against unauthorized access.

  • We assume that the resource has a name attribute and optional dictionary of associated key-value tags.

• An action is a logical operation executed on the secured resource, like:

  • create: Create an instance.

  • describe: Access the instance state.

  • update: Update the instance state.

  • delete: Delete an instance.

  • read: Read both online and offline stores.

  • read_online: Read the online store.

  • read_offline: Read the offline store.

  • write: Write on any store.

  • write_online: Write to the online store.

  • write_offline: Write to the offline store.

• A policy identifies the rule for enforcing authorization decisions on secured resources, based on the current user.

  • A default implementation is provided for role-based policies, using the user roles to grant or deny access to the requested actions on the secured resources.

The Permission class identifies a single permission configured on the feature store and is identified by these attributes:

• name: The permission name.

• types: The list of protected resource types. Defaults to all managed types, e.g. the ALL_RESOURCE_TYPES alias. All sub-classes are included in the resource match.

• name_patterns: A list of regex patterns to match resource names. If any regex matches, the Permission policy is applied. Defaults to [], meaning no name filtering is applied.

• required_tags: Dictionary of key-value pairs that must match the resource tags. Defaults to None, meaning that no tags filtering is applied.

• actions: The actions authorized by this permission. Defaults to ALL_VALUES, an alias defined in the action module.

• policy: The policy to be applied to validate a client request.

To simplify configuration, several constants are defined to streamline the permissions setup:

• In module feast.feast_object:

  • ALL_RESOURCE_TYPES is the list of all the FeastObject types.

  • ALL_FEATURE_VIEW_TYPES is the list of all the feature view types, including those not inheriting from FeatureView type like OnDemandFeatureView.

• In module feast.permissions.action:

  • ALL_ACTIONS is the list of all managed actions.

  • READ includes all the read actions for online and offline store.

  • WRITE includes all the write actions for online and offline store.

  • CRUD includes all the state management actions to create, describe, update or delete a Feast resource.

Given the above definitions, the feature store can be configured with granular control over each resource, enabling partitioned access by teams to meet organizational requirements for service and data sharing, and protection of sensitive information.

The feast CLI includes a new permissions command to list the registered permissions, with options to identify the matching resources for each configured permission and the existing resources that are not covered by any permission.

Definition examples

This permission definition grants access to the resource state and the ability to read all of the stores for any feature view or feature service to all users with the role super-reader:

This example grants permission to write on all the data sources with risk_level tag set to high only to users with role admin or data_team:

The following permission grants authorization to read the offline store of all the feature views including risky in the name, to users with role trusted:

Authorization configuration

In order to leverage the permission functionality, the auth section is needed in the feature_store.yaml configuration. Currently, Feast supports OIDC and Kubernetes RBAC authorization protocols.

The default configuration, if you don't specify the auth configuration section, is no_auth, indicating that no permission enforcement is applied.

A provider is an implementation of a feature store using specific feature store components (e.g. offline store, online store) targeting a specific environment (e.g. GCP stack).

The Feature Server is a core architectural component in Feast, designed to provide low-latency feature retrieval and updates for machine learning applications.

It is a REST API server built using FastAPI and exposes a limited set of endpoints to serve features, push data, and support materialization operations. The server is scalable, flexible, and designed to work seamlessly with various deployment environments, including local setups and cloud-based systems.

Motivation

In machine learning workflows, real-time access to feature values is critical for enabling low-latency predictions. The Feature Server simplifies this requirement by:

1. Serving Features: Allowing clients to retrieve feature values for specific entities in real-time, reducing the complexity of direct interactions with the online store.

2. Data Integration: Providing endpoints to push feature data directly into the online or offline store, ensuring data freshness and consistency.

3. Scalability: Supporting horizontal scaling to handle high request volumes efficiently.

4. Standardized API: Exposing HTTP/JSON endpoints that integrate seamlessly with various programming languages and ML pipelines.

5. Secure Communication: Supporting TLS (SSL) for secure data transmission in production environments.

Architecture

The Feature Server operates as a stateless service backed by two key components:

• Online Store: The primary data store used for low-latency feature retrieval.

• Registry: The metadata store that defines feature sets, feature views, and their relationships to entities.

Key Features

1. RESTful API: Provides standardized endpoints for feature retrieval and data pushing.

2. CLI Integration: Easily managed through the Feast CLI with commands like feast serve.

3. Flexible Deployment: Can be deployed locally, via Docker, or on Kubernetes using Helm charts.

4. Scalability: Designed for distributed deployments to handle large-scale workloads.

5. TLS Support: Ensures secure communication in production setups.

Endpoints Overview

The OpenTelemetry integration in Feast provides comprehensive monitoring and observability capabilities for your feature serving infrastructure. This component enables you to track key metrics, traces, and logs from your Feast deployment.

Motivation

Monitoring and observability are critical for production machine learning systems. The OpenTelemetry integration addresses these needs by:

1. Performance Monitoring: Track CPU and memory usage of feature servers

2. Operational Insights: Collect metrics to understand system behavior and performance

3. Troubleshooting: Enable effective debugging through distributed tracing

4. Resource Optimization: Monitor resource utilization to optimize deployments

5. Production Readiness: Provide enterprise-grade observability capabilities

Architecture

The OpenTelemetry integration in Feast consists of several components working together:

• OpenTelemetry Collector: Receives, processes, and exports telemetry data

• Prometheus Integration: Enables metrics collection and monitoring

• Instrumentation: Automatic Python instrumentation for tracking metrics

• Exporters: Components that send telemetry data to monitoring systems

Key Features

1. Automated Instrumentation: Python auto-instrumentation for comprehensive metric collection

2. Metric Collection: Track key performance indicators including:

   • Memory usage

   • CPU utilization

   • Request latencies

   • Feature retrieval statistics

3. Flexible Configuration: Customizable metric collection and export settings

4. Kubernetes Integration: Native support for Kubernetes deployments

5. Prometheus Compatibility: Integration with Prometheus for metrics visualization

Setup and Configuration

To add monitoring to the Feast Feature Server, follow these steps:

1. Deploy Prometheus Operator

Follow the Prometheus Operator documentation to install the operator.

2. Deploy OpenTelemetry Operator

Before installing the OpenTelemetry Operator:

1. Install cert-manager

2. Validate that the pods are running

3. Apply the OpenTelemetry operator:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

For additional installation steps, refer to the OpenTelemetry Operator documentation.

3. Configure OpenTelemetry Collector

Add the OpenTelemetry Collector configuration under the metrics section in your values.yaml file:

metrics:
  enabled: true
  otelCollector:
    endpoint: "otel-collector.default.svc.cluster.local:4317"  # sample
    headers:
      api-key: "your-api-key"

4. Add Instrumentation Configuration

Add the following annotations and environment variables to your deployment.yaml:

template:
  metadata:
    annotations:
      instrumentation.opentelemetry.io/inject-python: "true"

- name: OTEL_EXPORTER_OTLP_ENDPOINT
  value: http://{{ .Values.service.name }}-collector.{{ .Release.namespace }}.svc.cluster.local:{{ .Values.metrics.endpoint.port}}
- name: OTEL_EXPORTER_OTLP_INSECURE
  value: "true"

5. Add Metric Checks

Add metric checks to all manifests and deployment files:

{{ if .Values.metrics.enabled }}
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: feast-instrumentation
spec:
  exporter:
    endpoint: http://{{ .Values.service.name }}-collector.{{ .Release.Namespace }}.svc.cluster.local:4318
  env:
  propagators:
    - tracecontext
    - baggage
  python:
    env:
      - name: OTEL_METRICS_EXPORTER
        value: console,otlp_proto_http
      - name: OTEL_LOGS_EXPORTER
        value: otlp_proto_http
      - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
        value: "true"
{{end}}

6. Add Required Manifests

Add the following components to your chart:

• Instrumentation

• OpenTelemetryCollector

• ServiceMonitors

• Prometheus Instance

• RBAC rules

7. Deploy Feast

Deploy Feast with metrics enabled:

helm install feast-release infra/charts/feast-feature-server --set metric=true --set feature_store_yaml_base64=""

Usage

To enable OpenTelemetry monitoring in your Feast deployment:

1. Set metrics.enabled=true in your Helm values

2. Configure the OpenTelemetry Collector endpoint

3. Deploy with proper annotations and environment variables

Example configuration:

metrics:
  enabled: true
  otelCollector:
    endpoint: "otel-collector.default.svc.cluster.local:4317"

Monitoring

Once configured, you can monitor various metrics including:

• feast_feature_server_memory_usage: Memory utilization of the feature server

• feast_feature_server_cpu_usage: CPU usage statistics

• Additional custom metrics based on your configuration

These metrics can be visualized using Prometheus and other compatible monitoring tools.

Getting started

Which programming language should I use to run Feast in a microservice architecture?

We recommend Python.

Do you have any examples of how Feast should be used?

The quickstart is the easiest way to learn about Feast. For more detailed tutorials, please check out the tutorials page.

Concepts

Do feature views have to include entities?

No, there are feature views without entities.

How does Feast handle model or feature versioning?

Feast expects that each version of a model corresponds to a different feature service.

Feature views once they are used by a feature service are intended to be immutable and not deleted (until a feature service is removed). In the future, feast plan and feast apply will throw errors if it sees this kind of behavior.

What is the difference between data sources and the offline store?

The data source itself defines the underlying data warehouse table in which the features are stored. The offline store interface defines the APIs required to make an arbitrary compute layer work for Feast (e.g. pulling features given a set of feature views from their sources, exporting the data set results to different formats). Please see data sources and offline store for more details.

Is it possible to have offline and online stores from different providers?

Yes, this is possible. For example, you can use BigQuery as an offline store and Redis as an online store.

Functionality

How do I run get_historical_features without providing an entity dataframe?

Feast does not provide a way to do this right now. This is an area we're actively interested in contributions for. See GitHub issue

Does Feast provide security or access control?

Feast currently does not support any access control other than the access control required for the Provider's environment (for example, GCP and AWS permissions).

It is a good idea though to lock down the registry file so only the CI/CD pipeline can modify it. That way data scientists and other users cannot accidentally modify the registry and lose other team's data.

Does Feast support streaming sources?

Yes. In earlier versions of Feast, we used Feast Spark to manage ingestion from stream sources. In the current version of Feast, we support push based ingestion. Feast also defines a stream processor that allows a deeper integration with stream sources.

Does Feast support feature transformation?

There are several kinds of transformations:

• On demand transformations (See docs)

  • These transformations are Pandas transformations run on batch data when you call get_historical_features and at online serving time when you call `get_online_features.

  • Note that if you use push sources to ingest streaming features, these transformations will execute on the fly as well

• Batch transformations (WIP, see RFC)

  • These will include SQL + PySpark based transformations on batch data sources.

• Streaming transformations (RFC in progress)

Does Feast have a Web UI?

Yes. See documentation.

Does Feast support composite keys?

A feature view can be defined with multiple entities. Since each entity has a unique join_key, using multiple entities will achieve the effect of a composite key.

What are the performance/latency characteristics of Feast?

Feast is designed to work at scale and support low latency online serving. See our benchmark blog post for details.

Does Feast support embeddings and list features?

Yes. Specifically:

• Simple lists / dense embeddings:

  • BigQuery supports list types natively

  • Redshift does not support list types, so you'll need to serialize these features into strings (e.g. json or protocol buffers)

  • Feast's implementation of online stores serializes features into Feast protocol buffers and supports list types (see reference)

• Sparse embeddings (e.g. one hot encodings)

  • One way to do this efficiently is to have a protobuf or string representation of https://www.tensorflow.org/guide/sparse_tensor

Does Feast support X storage engine?

The list of supported offline and online stores can be found here and here, respectively. The roadmap indicates the stores for which we are planning to add support. Finally, our Provider abstraction is built to be extensible, so you can plug in your own implementations of offline and online stores. Please see more details about customizing Feast here.

Does Feast support using different clouds for offline vs online stores?

Yes. Using a GCP or AWS provider in feature_store.yaml primarily sets default offline / online stores and configures where the remote registry file can live. You can override the offline and online stores to be in different clouds if you wish.

What is the difference between a data source and an offline store?

The data source and the offline store are closely tied, but separate concepts. The offline store controls how feast talks to a data store for historical feature retrieval, and the data source points to specific table (or query) within a data store. Offline stores are infrastructure-level connectors to data stores like Snowflake.

Additional differences:

• Data sources may be specific to a project (e.g. feed ranking), but offline stores are agnostic and used across projects.

• A feast project may define several data sources that power different feature views, but a feast project has a single offline store.

• Feast users typically need to define data sources when using feast, but only need to use/configure existing offline stores without creating new ones.

How can I add a custom online store?

Please follow the instructions here.

Can the same storage engine be used for both the offline and online store?

Yes. For example, the Postgres connector can be used as both an offline and online store (as well as the registry).

Does Feast support S3 as a data source?

Yes. There are two ways to use S3 in Feast:

• Using Redshift as a data source via Spectrum (AWS tutorial), and then continuing with the Running Feast with Snowflake/GCP/AWS guide. See a presentation we did on this at our apply() meetup.

• Using the s3_endpoint_override in a FileSource data source. This endpoint is more suitable for quick proof of concepts that won't necessarily scale for production use cases.

Is Feast planning on supporting X functionality?

Please see the roadmap.

Project

How do I contribute to Feast?

For more details on contributing to the Feast community, see here and this here.

Feast 0.9 (legacy)

What is the difference between Feast 0.9 and Feast 0.10+?

Feast 0.10+ is much lighter weight and more extensible than Feast 0.9. It is designed to be simple to install and use. Please see this document for more details.

How do I migrate from Feast 0.9 to Feast 0.10+?

Please see this document. If you have any questions or suggestions, feel free to leave a comment on the document!

What are the plans for Feast Core, Feast Serving, and Feast Spark?

Feast Core and Feast Serving were both part of Feast Java. We plan to support Feast Serving. We will not support Feast Core; instead we will support our object store based registry. We will not support Feast Spark. For more details on what we plan on supporting, please see the roadmap.

Overview

Generally, Feast supports several patterns of feature retrieval:

1. Training data generation (via feature_store.get_historical_features(...))

2. Offline feature retrieval for batch scoring (via feature_store.get_historical_features(...))

3. Online feature retrieval for real-time model predictions

   • via the SDK: feature_store.get_online_features(...)

   • via deployed feature server endpoints: requests.post('http://localhost:6566/get-online-features', data=json.dumps(online_request))

Each of these retrieval mechanisms accept:

• some way of specifying entities (to fetch features for)

For code examples of how the below work, inspect the generated repository from feast init -t [YOUR TEMPLATE] (gcp, snowflake, and aws are the most fully fleshed).

Concepts

Before diving into how to retrieve features, we need to understand some high level concepts in Feast.

Feature Services

Feature services are used during

• The generation of training datasets when querying feature views in order to find historical feature values. A single training dataset may consist of features from multiple feature views.

• Retrieval of features for batch scoring from the offline store (e.g. with an entity dataframe where all timestamps are now())

• Retrieval of features from the online store for online inference (with smaller batch sizes). The features retrieved from the online store may also belong to multiple feature views.

Feature services enable referencing all or some features from a feature view.

Retrieving from the online store with a feature service

Retrieving from the offline store with a feature service

Feature References

This mechanism of retrieving features is only recommended as you're experimenting. Once you want to launch experiments or serve models, feature services are recommended.

Feature references uniquely identify feature values in Feast. The structure of a feature reference in string form is as follows: <feature_view>:<feature>

Feature references are used for the retrieval of features from Feast:

It is possible to retrieve features from multiple feature views with a single request, and Feast is able to join features from multiple tables in order to build a training dataset. However, it is not possible to reference (or retrieve) features from multiple projects at the same time.

Event timestamp

The timestamp on which an event occurred, as found in a feature view's data source. The event timestamp describes the event time at which a feature was observed or generated.

Event timestamps are used during point-in-time joins to ensure that the latest feature values are joined from feature views onto entity rows. Event timestamps are also used to ensure that old feature values aren't served to models during online serving.

Dataset

A dataset is a collection of rows that is produced by a historical retrieval from Feast in order to train a model. A dataset is produced by a join from one or more feature views onto an entity dataframe. Therefore, a dataset may consist of features from multiple feature views.

Dataset vs Feature View: Feature views contain the schema of data and a reference to where data can be found (through its data source). Datasets are the actual data manifestation of querying those data sources.

Dataset vs Data Source: Datasets are the output of historical retrieval, whereas data sources are the inputs. One or more data sources can be used in the creation of a dataset.

Retrieving historical features (for training data or batch scoring)

Feast abstracts away point-in-time join complexities with the get_historical_features API.

We go through the major steps, and also show example code. Note that the quickstart templates generally have end-to-end working examples for all these cases.

Step 1: Specifying Features

Feast accepts either:

Example: querying a feature service (recommended)

Example: querying a list of feature references

Step 2: Specifying Entities

Feast accepts either a Pandas dataframe as the entity dataframe (including entity keys and timestamps) or a SQL query to generate the entities.

Both approaches must specify the full entity key needed as well as the timestamps. Feast then joins features onto this dataframe.

Example: entity dataframe for generating training data

Example: entity SQL query for generating training data

You can also pass a SQL string to generate the above dataframe. This is useful for getting all entities in a timeframe from some data source.

Retrieving online features (for model inference)

Feast will ensure the latest feature values for registered features are available. At retrieval time, you need to supply a list of entities and the corresponding features to be retrieved. Similar to get_historical_features, we recommend using feature services as a mechanism for grouping features in a model version.

Note: unlike get_historical_features, the entity_rows do not need timestamps since you only want one feature value per entity key.

There are several options for retrieving online features: through the SDK, or through a feature server

Throughout this tutorial, we’ll walk through the creation of a production-ready fraud prediction system. A prediction is made in real-time as the user makes the transaction, so we need to be able to generate a prediction at low latency.

Our end-to-end example will perform the following workflows:

• Computing and backfilling feature data from raw data

• Building point-in-time correct training datasets from feature data and training a model

• Making online predictions from feature data

Here's a high-level picture of our system architecture on Google Cloud Platform (GCP):

These Feast tutorials showcase how to use Feast to simplify end to end model training / serving.

In this example, you'll learn how to use some of the key functionality in Feast. The tutorial runs in both local mode and on the Google Cloud Platform (GCP). For GCP, you must have access to a GCP project already, including read and write permissions to BigQuery.

Try it and let us know what you think!

In the steps below, we will set up a sample Feast project that leverages Snowflake as an offline store + materialization engine + online store.

Starting with data in a Snowflake table, we will register that table to the feature store and define features associated with the columns in that table. From there, we will generate historical training data based on those feature definitions and then materialize the latest feature values into the online store. Lastly, we will retrieve the materialized feature values.

Our template will generate new data containing driver statistics. From there, we will show you code snippets that will call to the offline store for generating training datasets, and then the code for calling the online store to serve you the latest feature values to serve models in production.

Snowflake Offline Store Example

Install feast-snowflake

Get a Snowflake Trial Account (Optional)

Create a feature repository

The following files will automatically be created in your project folder:

• feature_store.yaml -- This is your main configuration file

• driver_repo.py -- This is your main feature definition file

• test.py -- This is a file to test your feature store configuration

Inspect feature_store.yaml

Here you will see the information that you entered. This template will use Snowflake as the offline store, materialization engine, and the online store. The main thing to remember is by default, Snowflake objects have ALL CAPS names unless lower case was specified.

Run our test python script test.py

What we did in test.py

Initialize our Feature Store

Create a dummy training dataframe, then call our offline store to add additional columns

Materialize the latest feature values into our online store

Retrieve the latest values from our online store based on our entity key

The Feast CLI can be used to deploy a feature store to your infrastructure, spinning up any necessary persistent resources like buckets or tables in data stores. The deployment target and effects depend on the provider that has been configured in your feature_store.yaml file, as well as the feature definitions found in your feature repository.

Deploying

To have Feast deploy your infrastructure, run feast apply from your command line while inside a feature repository:

feast apply

# Processing example.py as example
# Done!

Depending on whether the feature repository is configured to use a local provider or one of the cloud providers like GCP or AWS, it may take from a couple of seconds to a minute to run to completion.

Cleaning up

If you need to clean up the infrastructure created by feast apply, use the teardown command.

feast teardown

****

A common scenario when using Feast in production is to want to test changes to Feast object definitions. For this, we recommend setting up a staging environment for your offline and online stores, which mirrors production (with potentially a smaller data set). Having this separate environment allows users to test changes by first applying them to staging, and then promoting the changes to production after verifying the changes on staging.

Setting up multiple environments

There are three common ways teams approach having separate environments

1. Have separate git branches for each environment

2. Have separate feature_store.yaml files and separate Feast object definitions that correspond to each environment

3. Have separate feature_store.yaml files per environment, but share the Feast object definitions

Different version control branches

To keep a clear separation of the feature repos, teams may choose to have multiple long-lived branches in their version control system, one for each environment. In this approach, with CI/CD setup, changes would first be made to the staging branch, and then copied over manually to the production branch once verified in the staging environment.

Separate feature_store.yaml files and separate Feast object definitions

For this approach, we have created an example repository (Feast Repository Example) which contains two Feast projects, one per environment.

The contents of this repository are shown below:

├── .github
│   └── workflows
│       ├── production.yml
│       └── staging.yml
│
├── staging
│   ├── driver_repo.py
│   └── feature_store.yaml
│
└── production
    ├── driver_repo.py
    └── feature_store.yaml

The repository contains three sub-folders:

• staging/: This folder contains the staging feature_store.yaml and Feast objects. Users that want to make changes to the Feast deployment in the staging environment will commit changes to this directory.

• production/: This folder contains the production feature_store.yaml and Feast objects. Typically users would first test changes in staging before copying the feature definitions into the production folder, before committing the changes.

• .github: This folder is an example of a CI system that applies the changes in either the staging or production repositories using feast apply. This operation saves your feature definitions to a shared registry (for example, on GCS) and configures your infrastructure for serving features.

The feature_store.yaml contains the following:

project: staging
registry: gs://feast-ci-demo-registry/staging/registry.db
provider: gcp

Notice how the registry has been configured to use a Google Cloud Storage bucket. All changes made to infrastructure using feast apply are tracked in the registry.db. This registry will be accessed later by the Feast SDK in your training pipelines or model serving services in order to read features.

If your organization consists of many independent data science teams or a single group is working on several projects that could benefit from sharing features, entities, sources, and transformations, then we encourage you to utilize Python packages inside each environment:

└── production
    ├── common
    │    ├── __init__.py
    │    ├── sources.py
    │    └── entities.py
    ├── ranking
    │    ├── __init__.py
    │    ├── views.py
    │    └── transformations.py
    ├── segmentation
    │    ├── __init__.py
    │    ├── views.py
    │    └── transformations.py
    └── feature_store.yaml

Shared Feast Object definitions with separate feature_store.yaml files

This approach is very similar to the previous approach, but instead of having feast objects duplicated and having to copy over changes, it may be possible to share the same Feast object definitions and have different feature_store.yaml configuration.

An example of how such a repository would be structured is as follows:

├── .github
│   └── workflows
│       ├── production.yml
│       └── staging.yml
├── staging
│   └── feature_store.yaml
├── production
│   └── feature_store.yaml
└── driver_repo.py

Users can then apply the applying them to each environment in this way:

feast -f staging/feature_store.yaml apply

This setup has the advantage that you can share the feature definitions entirely, which may prevent issues with copy-pasting code.

Summary

In summary, once you have set up a Git based repository with CI that runs feast apply on changes, your infrastructure (offline store, online store, and cloud environment) will automatically be updated to support the loading of data into the feature store or retrieval of data.

Feast allows users to build a training dataset from time-series feature data that already exists in an offline store. Users are expected to provide a list of features to retrieve (which may span multiple feature views), and a dataframe to join the resulting features onto. Feast will then execute a point-in-time join of multiple feature views onto the provided dataframe, and return the full resulting dataframe.

Retrieving historical features

1. Register your feature views

Please ensure that you have created a feature repository and that you have registered (applied) your feature views with Feast.

2. Define feature references

Start by defining the feature references (e.g., driver_trips:average_daily_rides) for the features that you would like to retrieve from the offline store. These features can come from multiple feature tables. The only requirement is that the feature tables that make up the feature references have the same entity (or composite entity), and that they aren't located in the same offline store.

feature_refs = [
    "driver_trips:average_daily_rides",
    "driver_trips:maximum_daily_rides",
    "driver_trips:rating",
    "driver_trips:rating:trip_completed",
]