What is Feast?

Feast (Feature Store) is a customizable operational data system that re-uses existing infrastructure to manage and serve machine learning features to realtime models.

Feast allows ML platform teams to:

• Make features consistently available for training and serving by managing an offline store (to process historical data for scale-out batch scoring or model training), a low-latency online store (to power real-time prediction), and a battle-tested feature server (to serve pre-computed features online).

• Avoid data leakage by generating point-in-time correct feature sets so data scientists can focus on feature engineering rather than debugging error-prone dataset joining logic. This ensure that future feature values do not leak to models during training.

• Decouple ML from data infrastructure by providing a single data access layer that abstracts feature storage from feature retrieval, ensuring models remain portable as you move from training models to serving models, from batch models to realtime models, and from one data infra system to another.

Who is Feast for?

Feast helps ML platform teams with DevOps experience productionize real-time models. Feast can also help these teams build towards a feature platform that improves collaboration between engineers and data scientists.

Feast is likely not the right tool if you

• are in an organization that’s just getting started with ML and is not yet sure what the business impact of ML is

• rely primarily on unstructured data

• need very low latency feature retrieval (e.g. p99 feature retrieval << 10ms)

• have a small team to support a large number of use cases

What Feast is not?

Feast is not

• an ETL / ELT system: Feast is not (and does not plan to become) a general purpose data transformation or pipelining system. Users often leverage tools like dbt to manage upstream data transformations.

• a data orchestration tool: Feast does not manage or orchestrate complex workflow DAGs. It relies on upstream data pipelines to produce feature values and integrations with tools like Airflow to make features consistently available.

• a data warehouse: Feast is not a replacement for your data warehouse or the source of truth for all transformed data in your organization. Rather, Feast is a light-weight downstream layer that can serve data from an existing data warehouse (or other data sources) to models in production.

• a database: Feast is not a database, but helps manage data stored in other systems (e.g. BigQuery, Snowflake, DynamoDB, Redis) to make features consistently available at training / serving time

Feast does not fully solve

• reproducible model training / model backtesting / experiment management: Feast captures feature and model metadata, but does not version-control datasets / labels or manage train / test splits. Other tools like DVC, MLflow, and Kubeflow are better suited for this.

• batch + streaming feature engineering: Feast primarily processes already transformed feature values (though it offers experimental light-weight transformations). Users usually integrate Feast with upstream systems (e.g. existing ETL/ELT pipelines). Tecton is a more fully featured feature platform which addresses these needs.

• native streaming feature integration: Feast enables users to push streaming features, but does not pull from streaming sources or manage streaming pipelines. Tecton is a more fully featured feature platform which orchestrates end to end streaming pipelines.

• feature sharing: Feast has experimental functionality to enable discovery and cataloguing of feature metadata with a Feast web UI (alpha). Feast also has community contributed plugins with DataHub and Amundsen. Tecton also more robustly addresses these needs.

• lineage: Feast helps tie feature values to model versions, but is not a complete solution for capturing end-to-end lineage from raw data sources to model versions. Feast also has community contributed plugins with DataHub and Amundsen. Tecton captures more end-to-end lineage by also managing feature transformations.

• data quality / drift detection: Feast has experimental integrations with Great Expectations, but is not purpose built to solve data drift / data quality issues. This requires more sophisticated monitoring across data pipelines, served feature values, labels, and model versions.

Example use cases

Many companies have used Feast to power real-world ML use cases such as:

• Personalizing online recommendations by leveraging pre-computed historical user or item features.

• Online fraud detection, using features that compare against (pre-computed) historical transaction patterns

• Churn prediction (an offline model), generating feature values for all users at a fixed cadence in batch

• Credit scoring, using pre-computed historical features to compute probability of default

How can I get started?

Explore the following resources to get started with Feast:

• Quickstart is the fastest way to get started with Feast

• Concepts describes all important Feast API concepts

• Architecture describes Feast's overall architecture.

• Tutorials shows full examples of using Feast in machine learning applications.

• Running Feast with Snowflake/GCP/AWS provides a more in-depth guide to using Feast.

• Reference contains detailed API and design documents.

• Contributing contains resources for anyone who wants to contribute to Feast.

In this tutorial we will

1. Deploy a local feature store with a Parquet file offline store and Sqlite online store.

2. Build a training dataset using our time series features from our Parquet files.

3. Ingest batch features ("materialization") and streaming features (via a Push API) into the online store.

4. Read the latest features from the offline store for batch scoring

5. Read the latest features from the online store for real-time inference.

6. Explore the (experimental) Feast UI

Overview

In this tutorial, we'll use Feast to generate training data and power online model inference for a ride-sharing driver satisfaction prediction model. Feast solves several common issues in this flow:

1. Training-serving skew and complex data joins: Feature values often exist across multiple tables. Joining these datasets can be complicated, slow, and error-prone.

   • Feast joins these tables with battle-tested logic that ensures point-in-time correctness so future feature values do not leak to models.

2. Online feature availability: At inference time, models often need access to features that aren't readily available and need to be precomputed from other data sources.

   • Feast manages deployment to a variety of online stores (e.g. DynamoDB, Redis, Google Cloud Datastore) and ensures necessary features are consistently available and freshly computed at inference time.

3. Feature and model versioning: Different teams within an organization are often unable to reuse features across projects, resulting in duplicate feature creation logic. Models have data dependencies that need to be versioned, for example when running A/B tests on model versions.

   • Feast enables discovery of and collaboration on previously used features and enables versioning of sets of features (via feature services).

   • (Experimental) Feast enables light-weight feature transformations so users can re-use transformation logic across online / offline use cases and across models.

Step 1: Install Feast

Install the Feast SDK and CLI using pip:

Step 2: Create a feature repository

Bootstrap a new feature repository using feast init from the command line.

Let's take a look at the resulting demo repo itself. It breaks down into

• data/ contains raw demo parquet data

• example_repo.py contains demo feature definitions

• feature_store.yaml contains a demo setup configuring where data sources are

• test_workflow.py showcases how to run all key Feast commands, including defining, retrieving, and pushing features. You can run this with python test_workflow.py.

The feature_store.yaml file configures the key overall architecture of the feature store.

The provider value sets default offline and online stores.

• The offline store provides the compute layer to process historical data (for generating training data & feature values for serving).

• The online store is a low latency store of the latest feature values (for powering real-time inference).

Valid values for provider in feature_store.yaml are:

• local: use a SQL registry or local file registry. By default, use a file / Dask based offline store + SQLite online store

• gcp: use a SQL registry or GCS file registry. By default, use BigQuery (offline store) + Google Cloud Datastore (online store)

• aws: use a SQL registry or S3 file registry. By default, use Redshift (offline store) + DynamoDB (online store)

Inspecting the raw data

The raw feature data we have in this demo is stored in a local parquet file. The dataset captures hourly stats of a driver in a ride-sharing app.

Step 3: Run sample workflow

There's an included test_workflow.py file which runs through a full sample workflow:

1. Register feature definitions through feast apply

2. Generate a training dataset (using get_historical_features)

3. Generate features for batch scoring (using get_historical_features)

4. Ingest batch features into an online store (using materialize_incremental)

5. Fetch online features to power real time inference (using get_online_features)

6. Ingest streaming features into offline / online stores (using push)

7. Verify online features are updated / fresher

We'll walk through some snippets of code below and explain

Step 3a: Register feature definitions and deploy your feature store

The apply command scans python files in the current directory for feature view/entity definitions, registers the objects, and deploys infrastructure. In this example, it reads example_repo.py and sets up SQLite online store tables. Note that we had specified SQLite as the default online store by configuring online_store in feature_store.yaml.

Step 3b: Generating training data or powering batch scoring models

To train a model, we need features and labels. Often, this label data is stored separately (e.g. you have one table storing user survey results and another set of tables with feature values). Feast can help generate the features that map to these labels.

Feast needs a list of entities (e.g. driver ids) and timestamps. Feast will intelligently join relevant tables to create the relevant feature vectors. There are two ways to generate this list:

1. The user can query that table of labels with timestamps and pass that into Feast as an entity dataframe for training data generation.

• Note that we include timestamps because we want the features for the same driver at various timestamps to be used in a model.

Generating training data

Run offline inference (batch scoring)

To power a batch model, we primarily need to generate features with the get_historical_features call, but using the current timestamp

Step 3c: Ingest batch features into your online store

We now serialize the latest values of features since the beginning of time to prepare for serving (note: materialize-incremental serializes all new features since the last materialize call).

Step 3d: Fetching feature vectors for inference

At inference time, we need to quickly read the latest feature values for different drivers (which otherwise might have existed only in batch sources) from the online feature store using get_online_features(). These feature vectors can then be fed to the model.

Step 3e: Using a feature service to fetch online features instead.

The driver_activity_v1 feature service pulls all features from the driver_hourly_stats feature view:

Step 4: Browse your features with the Web UI (experimental)

View all registered features, data sources, entities, and feature services with the Web UI.

One of the ways to view this is with the feast ui command.

Step 5: Re-examine test_workflow.py

Take a look at test_workflow.py again. It showcases many sample flows on how to interact with Feast. You'll see these show up in the upcoming concepts + architecture + tutorial pages as well.

Next steps

Feast project structure

Projects provide complete isolation of feature stores at the infrastructure level. This is accomplished through resource namespacing, e.g., prefixing table names with the associated project. Each project should be considered a completely separate universe of entities and features. It is not possible to retrieve features from multiple projects in a single request. We recommend having a single feature store and a single project per environment (dev, staging, prod).

Data ingestion

For offline use cases that only rely on batch data, Feast does not need to ingest data and can query your existing data (leveraging a compute engine, whether it be a data warehouse or (experimental) Spark / Trino). Feast can help manage pushing streaming features to a batch source to make features available for training.

Feature registration and retrieval

Features are registered as code in a version controlled repository, and tie to data sources + model versions via the concepts of entities, feature views, and feature services. We explore these concepts more in the upcoming concept pages. These features are then stored in a registry, which can be accessed across users and services. The features can then be retrieved via SDK API methods or via a deployed feature server which exposes endpoints to query for online features (to power real time models).

Feast supports several patterns of feature retrieval.

An entity is a collection of semantically related features. Users define entities to map to the domain of their use case. For example, a ride-hailing service could have customers and drivers as their entities, which group related features that correspond to these customers and drivers.

The entity name is used to uniquely identify the entity (for example to show in the experimental Web UI). The join key is used to identify the physical primary key on which feature values should be joined together to be retrieved during feature retrieval.

Entities are used by Feast in many contexts, as we explore below:

Use case #1: Defining and storing features

Feast's primary object for defining features is a feature view, which is a collection of features. Feature views map to 0 or more entities, since a feature can be associated with:

• zero entities (e.g. a global feature like num_daily_global_transactions)

• one entity (e.g. a user feature like user_age or last_5_bought_items)

• multiple entities, aka a composite key (e.g. a user + merchant category feature like num_user_purchases_in_merchant_category)

Feast refers to this collection of entities for a feature view as an entity key.

Entities should be reused across feature views. This helps with discovery of features, since it enables data scientists understand how other teams build features for the entity they are most interested in.

Feast will use the feature view concept to then define the schema of groups of features in a low-latency online store.

Use case #2: Retrieving features

At serving time, users specify entity key(s) to fetch the latest feature values which can power real-time model prediction (e.g. a fraud detection model that needs to fetch the latest transaction user's features to make a prediction).