Functionality

• Create Batch Features: ELT/ETL systems like Spark and SQL are used to transform data in the batch store.

• Feast Apply: The user (or CI) publishes versioned controlled feature definitions using feast apply. This CLI command updates infrastructure and persists definitions in the object store registry.

• Feast Materialize: The user (or scheduler) executes feast materialize which loads features from the offline store into the online store.

• Model Training: A model training pipeline is launched. It uses the Feast Python SDK to retrieve a training dataset and trains a model.

• Get Historical Features: Feast exports a point-in-time correct training dataset based on the list of features and entity dataframe provided by the model training pipeline.

• Deploy Model: The trained model binary (and list of features) are deployed into a model serving system. This step is not executed by Feast.

• Prediction: A backend system makes a request for a prediction from the model serving service.

• Get Online Features: The model serving service makes a request to the Feast Online Serving service for online features using a Feast SDK.

Components

A complete Feast deployment contains the following components:

• Feast Registry: An object store (GCS, S3) based registry used to persist feature definitions that are registered with the feature store. Systems can discover feature data by interacting with the registry through the Feast SDK.

• Feast Python SDK/CLI: The primary user facing SDK. Used to:

  • Manage version controlled feature definitions.

  • Materialize (load) feature values into the online store.

  • Build and retrieve training datasets from the offline store.

  • Retrieve online features.

• Online Store: The online store is a database that stores only the latest feature values for each entity. The online store is populated by materialization jobs and from stream ingestion.

• Offline Store: The offline store persists batch data that has been ingested into Feast. This data is used for producing training datasets. Feast does not manage the offline store directly, but runs queries against it.

The Feast online store is used for low-latency online feature value lookups. Feature values are loaded into the online store from data sources in feature views using the materialize command.

The storage schema of features within the online store mirrors that of the data source used to populate the online store. One key difference between the online store and data sources is that only the latest feature values are stored per entity key. No historical values are stored.

Example batch data source

Once the above data source is materialized into Feast (using feast materialize), the feature values will be stored as follows:

Features can also be written to the online store via push sources

Feast uses offline stores as storage and compute systems. Offline stores store historic time-series feature values. Feast does not generate these features, but instead uses the offline store as the interface for querying existing features in your organization.

Offline stores are used primarily for two reasons

1. Building training datasets from time-series features.

2. Materializing (loading) features from the offline store into an online store in order to serve those features at low latency for prediction.

Offline stores are configured through the feature_store.yaml. When building training datasets or materializing features into an online store, Feast will use the configured offline store along with the data sources you have defined as part of feature views to execute the necessary data operations.

It is not possible to query all data sources from all offline stores, and only a single offline store can be used at a time. For example, it is not possible to query a BigQuery table from a File offline store, nor is it possible for a BigQuery offline store to query files from your local file system.

Please see the Offline Stores reference for more details on configuring offline stores.

Please see the Push Source for reference on how to push features directly to the offline store in your feature store.

Feast users use Feast to manage two important sets of configuration:

• Configuration about how to run Feast on your infrastructure

• Feature definitions

With Feast, the above configuration can be written declaratively and stored as code in a central location. This central location is called a feature repository. The feature repository is the declarative source of truth for what the desired state of a feature store should be.

The Feast CLI uses the feature repository to configure, deploy, and manage your feature store.

An example structure of a feature repository is shown below:

$ tree -a
.
├── data
│   └── driver_stats.parquet
├── driver_features.py
├── feature_store.yaml
└── .feastignore

1 directory, 4 files

For more details, see the Feature repository reference.

The Feast feature registry is a central catalog of all the feature definitions and their related metadata. It allows data scientists to search, discover, and collaborate on new features.

Each Feast deployment has a single feature registry. Feast only supports file-based registries today, but supports three different backends

• Local: Used as a local backend for storing the registry during development

• S3: Used as a centralized backend for storing the registry on AWS

• GCS: Used as a centralized backend for storing the registry on GCP

The feature registry is updated during different operations when using Feast. More specifically, objects within the registry (entities, feature views, feature services) are updated when running apply from the Feast CLI, but metadata about objects can also be updated during operations like materialization.

Users interact with a feature registry through the Feast SDK. Listing all feature views:

fs = FeatureStore("my_feature_repo/")
print(fs.list_feature_views())

Or retrieving a specific feature view:

fs = FeatureStore("my_feature_repo/")
fv = fs.get_feature_view(“my_fv1”)

A provider is an implementation of a feature store using specific feature store components (e.g. offline store, online store) targeting a specific environment (e.g. GCP stack).

Providers orchestrate various components (offline store, online store, infrastructure, compute) inside an environment. For example, the gcp provider supports BigQuery as an offline store and Datastore as an online store, ensuring that these components can work together seamlessly. Feast has three built-in providers (local, gcp, and aws) with default configurations that make it easy for users to start a feature store in a specific environment. These default configurations can be overridden easily. For instance, you can use the gcp provider but use Redis as the online store instead of Datastore.

If the built-in providers are not sufficient, you can create your own custom provider. Please see this guide for more details.

Please see feature_store.yaml for configuring providers.