A provider is an implementation of a feature store using specific feature store components targeting a specific environment. More specifically, a provider is the target environment to which you have configured your feature store to deploy and run.

Providers are built to orchestrate various components (offline store, online store, infrastructure, compute) inside an environment. For example, the gcp provider supports BigQuery as an offline store and Datastore as an online store, ensuring that these components can work together seamlessly.

Providers also come with default configurations which makes it easier for users to start a feature store in a specific environment.

Please see feature_store.yaml for configuring providers.

How Feast SDK usage is measured

The Feast project logs anonymous usage statistics and errors in order to inform our planning. Several client methods are tracked, beginning in Feast 0.9. Users are assigned a UUID which is sent along with the name of the method, the Feast version, the OS (using sys.platform), and the current time.

The source code is available here.

How to disable usage logging

Set the environment variable FEAST_USAGE to False.

Please see Data Source for an explanation of data sources.

Description

BigQuery data sources allow for the retrieval of historical feature values from BigQuery for building training datasets as well as materializing features into an online store.

• Either a table reference or a SQL query can be provided.

• No performance guarantees can be provided over SQL query-based sources. Please use table references where possible.

Examples

Using a table reference

Using a query

Configuration options are available .

The Feast CLI can be used to deploy a feature store to your infrastructure, spinning up any necessary persistent resources like buckets or tables in data stores. The deployment target and effects depend on the provider that has been configured in your feature_store.yaml file, as well as the feature definitions found in your feature repository.

Deploying

To have Feast deploy your infrastructure, run feast apply from your command line while inside a feature repository:

Depending on whether the feature repository is configured to use a local provider or one of the cloud providers like GCP or AWS, it may take from a couple of seconds to a minute to run to completion.

Cleaning up

If you need to clean up the infrastructure created by feast apply, use the teardown command.

****

The Feast online store is used for low-latency online feature value lookups. Feature values are loaded into the online store from data sources in feature views using the materialize command.

The storage schema of features within the online store mirrors that of the data source used to populate the online store. One key difference between the online store and data sources is that only the latest feature values are stored per entity key. No historical values are stored.

Example batch data source

Once the above data source is materialized into Feast (using feast materialize), the feature values will be stored as follows:

Feast uses offline stores as storage and compute systems. Offline stores store historic time-series feature values. Feast does not generate these features, but instead uses the offline store as the interface for querying existing features in your organization.

Offline stores are used primarily for two reasons

1. Building training datasets from time-series features.

2. Materializing (loading) features from the offline store into an online store in order to serve those features at low latency for prediction.

Offline stores are configured through the feature_store.yaml. When building training datasets or materializing features into an online store, Feast will use the configured offline store along with the data sources you have defined as part of feature views to execute the necessary data operations.

It is not possible to query all data sources from all offline stores, and only a single offline store can be used at a time. For example, it is not possible to query a BigQuery table from a File offline store, nor is it possible for a BigQuery offline store to query files from your local file system.

Please see the reference for more details on configuring offline stores.

Install Feast using :

Install Feast with GCP dependencies (required when using BigQuery or Firestore):

In this tutorial we will

1. Deploy a local feature store with a Parquet file offline store and Sqlite online store.

2. Build a training dataset using our time series features from our Parquet files.

3. Materialize feature values from the offline store into the online store.

Please see for an explanation of offline stores.

What is Feast?

Feast (Feature Store) is an operational data system for managing and serving machine learning features to models in production.

Description

The BigQuery offline store provides support for reading .

• BigQuery tables and views are allowed as sources.

Description

The online store provides support for materializing feature values into Redis.

• Both Redis and Redis Cluster are supported

The top-level namespace within Feast is a . Users define one or more within a project. Each feature view contains one or more that relate to a specific . A feature view must always have a , which in turn is used during the generation of training and when materializing feature values into the online store.

Project

Projects provide complete isolation of feature stores at the infrastructure level. This is accomplished through resource namespacing, e.g., prefixing table names with the associated project. Each project should be considered a completely separate universe of entities and features. It is not possible to retrieve features from multiple projects in a single request. We recommend having a single feature store and a single project per environment (

A production deployment of Feast is deployed using Kubernetes.

Kubernetes (with Helm)

This guide installs Feast into an existing Kubernetes cluster using Helm. The installation is not specific to any cloud platform or environment, but requires Kubernetes and Helm.

The Feast Python SDK allows users to retrieve feature values from an online store. This API is used to look up feature values at low latency during model serving in order to make online predictions.

Retrieving online features

Concepts

are objects in an organization like customers, transactions, and drivers, products, etc.

are external sources of data where feature data can be found.

are objects that define logical groupings of features, data sources, and other related metadata.

Description

The online store provides support for materializing feature values into an SQLite database for serving online features.

• All feature values are stored in an on-disk SQLite database

Please see for an explanation of providers.

Description

File data sources allow for the retrieval of historical feature values from files on disk for building training datasets, as well as for materializing features into an online store.

Example

Description

• Offline Store: Uses the File offline store by default. Also supports BigQuery as the offline store.

• Online Store: Uses the Sqlite online store by default. Also supports Datastore as an online store.

Install the using pip:

Connect to an existing Feast Core deployment:

Please see for an explanation of online stores.

A feature repository is a directory that contains the configuration of the feature store and individual features. This configuration is written as code (Python/YAML) and it's highly recommended that teams track it centrally using git. See Feature Repository for a detailed explanation of feature repositories.

The easiest way to create a new feature repository to use feast init command:

feast init

Creating a new Feast repository in /<...>/tiny_pika.

feast init -t gcp

Creating a new Feast repository in /<...>/tiny_pika.

The init command creates a Python file with feature definitions, sample data, and a Feast configuration file for local development:

Enter the directory:

You can now use this feature repository for development. You can try the following:

• Run feast apply to apply these definitions to Feast.

• Edit the example feature definitions in example.py and run feast apply again to change feature definitions.

• Initialize a git repository in the same directory and checking the feature repository into version control.

Feast allows users to load their feature data into an online store in order to serve the latest features to models for online prediction.

Materializing features

1. Register feature views

Before proceeding, please ensure that you have applied (registered) the feature views that should be materialized.

2.a Materialize

The materialize command allows users to materialize features over a specific historical time range into the online store.

The above command will query the batch sources for all feature views over the provided time range, and load the latest feature values into the configured online store.

It is also possible to materialize for specific feature views by using the -v / --views argument.

The materialize command is completely stateless. It requires the user to provide the time ranges that will be loaded into the online store. This command is best used from a scheduler that tracks state, like Airflow.

2.b Materialize Incremental (Alternative)

For simplicity, Feast also provides a materialize command that will only ingest new data that has arrived in the offline store. Unlike materialize, materialize-incremental will track the state of previous ingestion runs inside of the feature registry.

The example command below will load only new data that has arrived for each feature view up to the end date and time (2021-04-08T00:00:00).

The materialize-incremental command functions similarly to materialize in that it loads data over a specific time range for all feature views (or the selected feature views) into the online store.

Unlike materialize, materialize-incremental automatically determines the start time from which to load features from batch sources of each feature view. The first time materialize-incremental is executed it will set the start time to the oldest timestamp of each data source, and the end time as the one provided by the user. For each run of materialize-incremental, the end timestamp will be tracked.

Subsequent runs of materialize-incremental will then set the start time to the end time of the previous run, thus only loading new data that has arrived into the online store. Note that the end time that is tracked for each run is at the feature view level, not globally for all feature views, i.e, different feature views may have different periods that have been materialized into the online store.

Feast allows users to build a training dataset from time-series feature data that already exists in an offline store. Users are expected to provide a list of features to retrieve (which may span multiple feature views), and a dataframe to join the resulting features onto. Feast will then execute a point-in-time join of multiple feature views onto the provided dataframe, and return the full resulting dataframe.

Retrieving historical features

1. Register your feature views

Please ensure that you have created a feature repository and that you have registered (applied) your feature views with Feast.

2. Define feature references

Start by defining the feature references (e.g., driver_trips:average_daily_rides) for the features that you would like to retrieve from the offline store. These features can come from multiple feature tables. The only requirement is that the feature tables that make up the feature references have the same entity (or composite entity), and that they aren't located in the same offline store.

3. Create an entity dataframe

An entity dataframe is the target dataframe on which you would like to join feature values. The entity dataframe must contain a timestamp column called event_timestamp and all entities (primary keys) necessary to join feature tables onto. All entities found in feature views that are being joined onto the entity dataframe must be found as column on the entity dataframe.

It is possible to provide entity dataframes as either a Pandas dataframe or a SQL query.

Pandas:

In the example below we create a Pandas based entity dataframe that has a single row with an event_timestamp column and a driver_id entity column. Pandas based entity dataframes may need to be uploaded into an offline store, which may result in longer wait times compared to a SQL based entity dataframe.

SQL (Alternative):

Below is an example of an entity dataframe built from a BigQuery SQL query. It is only possible to use this query when all feature views being queried are available in the same offline store (BigQuery).

4. Launch historical retrieval

Once the feature references and an entity dataframe are defined, it is possible to call get_historical_features(). This method launches a job that executes a point-in-time join of features from the offline store onto the entity dataframe. Once completed, a job reference will be returned. This job reference can then be converted to a Pandas dataframe by calling to_df().

Description

The Datastore online store provides support for materializing feature values into Cloud Datastore. The data model used to store feature values in Datastore is described in more detail here.

Example

Configuration options are available .

Overview

feature_store.yaml is used to configure a feature store. The file must be located at the root of a feature repository. An example feature_store.yaml is shown below:

project: loyal_spider
registry: data/registry.db
provider: local
online_store:
    type: sqlite
    path: data/online_store.db

Options

The following top-level configuration options exist in the feature_store.yaml file.

• provider — Configures the environment in which Feast will deploy and operate.

• registry — Configures the location of the feature registry.

• online_store — Configures the online store.

Please see the API reference for the full list of configuration options.

Install Feast

If you would like to deploy a new installation of Feast, click on

Connect to Feast

If you would like to connect to an existing Feast deployment, click on

Learn Feast

If you would like to learn more about Feast, click on

Feast Python SDK

The Feast Python SDK is used as a library to interact with a Feast deployment.

• Define, register, and manage entities and features

• Ingest data into Feast

• Build and retrieve training datasets

• Retrieve online features

Feast CLI

The Feast CLI is a command line implementation of the Feast Python SDK.

• Define, register, and manage entities and features from the terminal

• Ingest data into Feast

• Manage ingestion jobs

Online Serving Clients

The following clients can be used to retrieve online feature values:

Description

The File offline store provides support for reading FileSources.

• Only Parquet files are currently supported.

• All data is downloaded and joined using Python and may not scale to production workloads.

Example

Configuration options are available .

Functionality

• Create Batch Features: ELT/ETL systems like Spark and SQL are used to transform data in the batch store.

Links & Resources

Overview

.feastignore is a file that is placed at the root of the . This file contains paths that should be ignored when running feast apply. An example .feastignore is shown below:

.feastignore file is optional. If the file can not be found, every Python in the feature repo directory will be parsed by feast apply

Explore the following resources to learn more about Feast:

• describes all important Feast API concepts.

• provides guidance on completing Feast workflows.

• contains Jupyter notebooks that you can run on your Feast deployment.

Overview

This guide installs Feast on Azure using our .

Overview

This guide installs Feast on an existing Kubernetes cluster, and ensures the following services are running:

• Feast Core

Install the Feast CLI using pip:

Configure the CLI to connect to your Feast Core deployment:

The CLI is a wrapper around the :

Dataset

A dataset is a collection of rows that is produced by a historical retrieval from Feast in order to train a model. A dataset is produced by a join from one or more feature views onto an entity dataframe. Therefore, a dataset may consist of features from multiple feature views.

Dataset vs Feature View: Feature views contain the schema of data and a reference to where data can be found (through its data source). Datasets are the actual data manifestation of querying those data sources.

Dataset vs Data Source: Datasets are the output of historical retrieval, whereas data sources are the inputs. One or more data sources can be used in the creation of a dataset.

Overview

This guide installs Feast on AWS using our reference Terraform configuration.

This Terraform configuration creates the following resources:

• Kubernetes cluster on Amazon EKS (3x r3.large nodes)

• Kafka managed by Amazon MSK (2x kafka.t3.small nodes)

• Postgres database for Feast metadata, using serverless Aurora (min capacity: 2)

• Redis cluster, using Amazon Elasticache (1x cache.t2.micro)

1. Requirements

• Create an AWS account and

• Install > = 0.12 (tested with 0.13.3)

• Install (tested with v3.3.4)

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/aws. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. At a minimum, you need to set name_prefix and an AWS region:

3. Apply

After completing the configuration, initialize Terraform and apply:

Starting may take a minute. A kubectl configuration file is also created in this directory, and the file's name will start with kubeconfig_ and end with a random suffix.

4. Connect to Feast using Jupyter

After all pods are running, connect to the Jupyter Notebook Server running in the cluster.

To connect to the remote Feast server you just created, forward a port from the remote k8s cluster to your local machine. Replace kubeconfig_XXXXXXX below with the kubeconfig file name Terraform generates for you.

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

Overview

This guide installs Feast on GKE using our reference Terraform configuration.

This Terraform configuration creates the following resources:

• GKE cluster

• Feast services running on GKE

• Google Memorystore (Redis) as online store

• Dataproc cluster

1. Requirements

• Install > = 0.12 (tested with 0.13.3)

• Install (tested with v3.3.4)

• GCP and sufficient to create the resources listed above.

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/gcp. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. Sample configurations are provided below:

3. Apply

After completing the configuration, initialize Terraform and apply:

In Feast, a store is a database that is populated with feature data that will ultimately be served to models.

Offline (Historical) Store

The offline store maintains historical copies of feature values. These features are grouped and stored in feature tables. During retrieval of historical data, features are queries from these feature tables in order to produce training datasets.

Online Store

The online store maintains only the latest values for a specific feature.

• Feature values are stored based on their

• Feast currently supports Redis as an online store.

• Online stores are meant for very high throughput writes from ingestion jobs and very low latency access to features during online serving.

Feature View

A feature view is an object that represents a logical group of time-series feature data as it is found in a . Feature views consist of one or more , , and a . Feature views allow Feast to model your existing feature data in a consistent way in both an offline (training) and online (serving) environment.

Feature views are used during

Overview

This guide shows you how to deploy Feast using

Sequence description

1. Log Raw Events: Production backend applications are configured to emit internal state changes as events to a stream.

Description

• Offline Store: Uses the BigQuery offline store by default. Also supports File as the offline store.

• Online Store: Uses the Datastore online store by default. Also supports Sqlite as an online store.

Example

Permissions

Overview

The Feast CLI comes bundled with the Feast Python package. It is immediately available after installing Feast.

Global Options

The Feast CLI provides one global top-level option that can be used with other commands

chdir (-c, --chdir)

This command allows users to run Feast CLI commands in a different folder from the current working directory.

Apply

Creates or updates a feature store deployment

What does Feast apply do?

1. Feast will scan Python files in your feature repository and find all Feast object definitions, such as feature views, entities, and data sources.

2. Feast will validate your feature definitions

3. Feast will sync the metadata about Feast objects to the registry. If a registry does not exist, then it will be instantiated. The standard registry is a simple protobuf binary file that is stored on disk (locally or in an object store).

Entities

List all registered entities

Feature views

List all registered feature views

Init

Creates a new feature repository

It's also possible to use other templates

or to set the name of the new project

Materialize

Load data from feature views into the online store between two dates

Load data for specific feature views into the online store between two dates

Materialize incremental

Load data from feature views into the online store, beginning from either the previous materialize or materialize-incremental end date, or the beginning of time.

Teardown

Tear down deployed feature store infrastructure

Version

Print the current Feast version

Feast manages two important sets of configuration: feature definitions, and configuration about how to run the feature store. With Feast, this configuration can be written declaratively and stored as code in a central location. This central location is called a feature repository, and it's essentially just a directory that contains some code files.

The feature repository is the declarative source of truth for what the desired state of a feature store should be. The Feast CLI uses the feature repository to configure your infrastructure, e.g., migrate tables.

What is a feature repository?

A feature repository consists of:

• A collection of Python files containing feature declarations.

• A feature_store.yaml file containing infrastructural configuration.

• A .feastignore file containing paths in the feature repository to ignore.

Structure of a feature repository

The structure of a feature repository is as follows:

• The root of the repository should contain a feature_store.yaml file and may contain a .feastignore file.

• The repository should contain Python files that contain feature definitions.

• The repository can contain other files as well, including documentation and potentially data files.

An example structure of a feature repository is shown below:

A couple of things to note about the feature repository:

• Feast reads all Python files recursively when feast apply is ran, including subdirectories, even if they don't contain feature definitions.

• It's recommended to add .feastignore and add paths to all imperative scripts if you need to store them inside the feature registry.

The feature_store.yaml configuration file

The configuration for a feature store is stored in a file named feature_store.yaml , which must be located at the root of a feature repository. An example feature_store.yaml file is shown below:

The feature_store.yaml file configures how the feature store should run. See for more details.

The .feastignore file

This file contains paths that should be ignored when running feast apply. An example .feastignore is shown below:

See for more details.

Feature definitions

A feature repository can also contain one or more Python files that contain feature definitions. An example feature definition file is shown below:

To declare new feature definitions, just add code to the feature repository, either in existing files or in a new file. For more information on how to define features, see .

Next steps

• See to get started with an example feature repository.

• See , , or for more information on the configuration files that live in a feature registry.

Overview

An entity is any domain object that can be modeled and about which information can be stored. Entities are usually recognizable concepts, either concrete or abstract, such as persons, places, things, or events.

Examples of entities in the context of ride-hailing and food delivery: customer, order, driver, restaurant, dish, area.

Entities are important in the context of feature stores since features are always properties of a specific entity. For example, we could have a feature total_trips_24h for driver D011234 with a feature value of 11.

Feast uses entities in the following way:

• Entities serve as the keys used to look up features for producing training datasets and online feature values.

• Entities serve as a natural grouping of features in a feature table. A feature table must belong to an entity (which could be a composite entity)

Structure of an Entity

When creating an entity specification, consider the following fields:

• Name: Name of the entity

• Description: Description of the entity

• Value Type: Value type of the entity. Feast will attempt to coerce entity columns in your data sources into this type.

A valid entity specification is shown below:

Working with an Entity

Creating an Entity:

Updating an Entity:

Permitted changes include:

• The entity's description and labels

The following changes are not permitted:

• Project

• Name of an entity

• Type

Overview

Feast Components export metrics that can provide insight into Feast behavior:

See the for documentation on metrics are exported by Feast.

Pushing Ingestion Metrics to StatsD

Feast Ingestion Job

Feast Ingestion Job can be configured to push Ingestion metrics to a StatsD instance. Metrics export to StatsD for Ingestion Job is configured in Job Controller's application.yml under feast.jobs.metrics

Exporting Feast Metrics to Prometheus

Feast Core and Serving

Feast Core and Serving exports metrics to a Prometheus instance via Prometheus scraping its /metrics endpoint. Metrics export to Prometheus for Core and Serving can be configured via their corresponding application.yml

to scrape directly from Core and Serving's /metrics endpoint.

Further Reading

See the for documentation on metrics are exported by Feast.

Please see the following API specific reference documentation:

• Feast Core gRPC API: This is the gRPC API used by Feast Core. This API contains RPCs for creating and managing feature sets, stores, projects, and jobs.

• Feast Serving gRPC API: This is the gRPC API used by Feast Serving. It contains RPCs used for the retrieval of online feature data or historical feature data.

• : These are the gRPC types used by both Feast Core, Feast Serving, and the Go, Java, and Python clients.

• : The Go library used for the retrieval of online features from Feast.

• : The Java library used for the retrieval of online features from Feast.

• : This is the complete reference to the Feast Python SDK. The SDK is used to manage feature sets, features, jobs, projects, and entities. It can also be used to retrieve training datasets or online features from Feast Serving.

Community Contributions

The following community provided SDKs are available:

• : A Node.js SDK written in TypeScript. The SDK can be used to manage feature sets, features, jobs, projects, and entities.

In order to retrieve features for both training and serving, Feast requires data being ingested into its offline and online stores.

Users are expected to already have either a batch or stream source with data stored in it, ready to be ingested into Feast. Once a feature table (with the corresponding sources) has been registered with Feast, it is possible to load data from this source into stores.

The following depicts an example ingestion flow from a data source to the online store.

Batch Source to Online Store

Stream Source to Online Store

Batch Source to Offline Store

Stream Source to Offline Store

Using Feast

Feast development happens through three key workflows:

1. Define and load feature data into Feast

Defining feature tables and ingesting data into Feast

Feature creators model the data within their organization into Feast through the definition of that contain . Feature tables are both a schema and a means of identifying data sources for features, and allow Feast to know how to interpret your data, and where to find it.

After registering a feature table with Feast, users can trigger an ingestion from their data source into Feast. This loads feature values from an upstream data source into Feast stores through ingestion jobs.

Visit to learn more about them.

Retrieving historical features for training

In order to generate a training dataset it is necessary to provide both an and feature references through the to retrieve historical features. For historical serving, Feast requires that you provide the entities and timestamps for the corresponding feature data. Feast produces a point-in-time correct dataset using the requested features. These features can be requested from an unlimited number of feature sets.

Retrieving online features for online serving

Online retrieval uses feature references through the to retrieve online features. Online serving allows for very low latency requests to feature data at very high throughput.

We use RFCs and GitHub issues to communicate development ideas. The simplest way to contribute to Feast is to leave comments in our RFCs in the Feast Google Drive or our GitHub issues. You will need to join our Google Group in order to get access.

We follow a process of lazy consensus. If you believe you know what the project needs then just start development. If you are unsure about which direction to take with development then please communicate your ideas through a GitHub issue or through our Slack Channel before starting development.

Please submit a PR to the master branch of the Feast repository once you are ready to submit your contribution. Code submission to Feast (including submission from project maintainers) require review and approval from maintainers or code owners.

PRs that are submitted by the general public need to be identified as ok-to-test. Once enabled, Prow will run a range of tests to verify the submission, after which community members will help to review the pull request.

Feast API

Overview

This guide installs Feast on Azure Kubernetes cluster (known as AKS), and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Spark

• Kafka

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Requirements

1. Install and configure

2. Install and configure

3. Install

2. Preparation

Create an AKS cluster with Azure CLI. The detailed steps can be found , and a high-level walk through includes:

Add the Feast Helm repository and download the latest charts:

Feast includes a Helm chart that installs all necessary components to run Feast Core, Feast Online Serving, and an example Jupyter notebook.

Feast Core requires Postgres to run, which requires a secret to be set on Kubernetes:

3. Feast installation

Install Feast using Helm. The pods may take a few minutes to initialize.

4. Spark operator installation

Follow the documentation , and Feast documentation to

and ensure the service account used by Feast has permissions to manage Spark Application resources. This depends on your k8s setup, but typically you'd need to configure a Role and a RoleBinding like the one below:

5. Use Jupyter to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

6. Environment variables

If you are running the , you may want to make sure the following environment variables are correctly set:

7. Further Reading

Overview

Sources are descriptions of external feature data and are registered to Feast as part of feature tables. Once registered, Feast can ingest feature data from these sources into stores.

Currently, Feast supports the following source types:

Batch Source

• File (as in Spark): Parquet (only).

• BigQuery

Stream Source

• Kafka

• Kinesis

The following encodings are supported on streams

• Avro

• Protobuf

Structure of a Source

For both batch and stream sources, the following configurations are necessary:

• Event timestamp column: Name of column containing timestamp when event data occurred. Used during point-in-time join of feature values to .

• Created timestamp column: Name of column containing timestamp when data is created. Used to deduplicate data when multiple copies of the same is ingested.

Example data source specifications:

The provides more information about options to specify for the above sources.

Working with a Source

Creating a Source

Sources are defined as part of :

Feast ensures that the source complies with the schema of the feature table. These specified data sources can then be included inside a feature table specification and registered to Feast Core.

Feast provides an API through which online feature values can be retrieved. This allows teams to look up feature values at low latency in production during model serving, in order to make online predictions.

The online store must be populated through ingestion jobs prior to being used for online serving.

Feast Serving provides a gRPC API that is backed by Redis. We have native clients in Python, Go, and Java.

Online Field Statuses

Feast also returns status codes when retrieving features from the Feast Serving API. These status code give useful insight into the quality of data being served.

Overview

This guide installs Feast on an existing IBM Cloud Kubernetes cluster or Red Hat OpenShift on IBM Cloud , and ensures the following services are running:

• Feast Core

Feast provides a historical retrieval interface for exporting feature data in order to train machine learning models. Essentially, users are able to enrich their data with features from any feature tables.

Retrieving historical features

Below is an example of the process required to produce a training dataset:

Custom OnlineStore

Feast allow users to create their own OnlineStore implementations, allowing Feast to read and write feature values to stores other than first-party implementations already in Feast directly. The interface for the is found at , and consists of four methods that need to be implemented.

If at any point in time you cannot resolve a problem, please see the section for reaching out to the Feast community.

Overview

Feature tables are both a schema and a logical means of grouping features, data sources, and other related metadata.

Feature tables serve the following purposes:

• Feature tables are a means for defining the location and properties of data sources.

• Feature tables are used to create within Feast a database-level structure for the storage of feature values.

• The data sources described within feature tables allow Feast to find and ingest feature data into stores within Feast.

• Feature tables ensure data is efficiently stored during by providing a grouping mechanism of features values that occur on the same event timestamp.

Features

A feature is an individual measurable property observed on an entity. For example the amount of transactions (feature) a customer (entity) has completed. Features are used for both model training and scoring (batch, online).

Features are defined as part of feature tables. Since Feast does not apply transformations, a feature is basically a schema that only contains a name and a type:

Visit for the complete feature specification API.

Structure of a Feature Table

Feature tables contain the following fields:

• Name: Name of feature table. This name must be unique within a project.

• Entities: List of to associate with the features defined in this feature table. Entities are used as lookup keys when retrieving features from a feature table.

• Features: List of features within a feature table.

Here is a ride-hailing example of a valid feature table specification:

By default, Feast assumes that features specified in the feature-table specification corresponds one-to-one to the fields found in the sources. All features defined in a feature table should be available in the defined sources.

Field mappings can be used to map features defined in Feast to fields as they occur in data sources.

In the example feature-specification table above, we use field mappings to ensure the feature named rating in the batch source is mapped to the field named driver_rating.

Working with a Feature Table

Creating a Feature Table

Updating a Feature Table

Feast currently supports the following changes to feature tables:

• Adding new features.

• Removing features.

• Updating source, max age, and labels.

Feast currently does not support the following changes to feature tables:

• Changes to the project or name of a feature table.

• Changes to entities related to a feature table.

• Changes to names and types of existing features.

Deleting a Feature Table

Migration from v0.6 to v0.7

Feast Core Validation changes

In v0.7, Feast Core no longer accepts starting with number (0-9) and using dash in names for:

• Project

• Feature Set

• Entities

• Features

Migrate all project, feature sets, entities, feature names:

• with ‘-’ by recreating them with '-' replace with '_'

• recreate any names with a number (0-9) as the first letter to one without.

Feast now prevents feature sets from being applied if no store is subscribed to that Feature Set.

• Ensure that a store is configured to subscribe to the Feature Set before applying the Feature Set.

Feast Core's Job Coordinator is now Feast Job Controller

In v0.7, Feast Core's Job Coordinator has been decoupled from Feast Core and runs as a separate Feast Job Controller application. See its for how to configure Feast Job Controller.

Ingestion Job API

In v0.7, the following changes are made to the Ingestion Job API:

• Changed List Ingestion Job API to return list of FeatureSetReference instead of list of FeatureSet in response.

• Moved ListIngestionJobs, StopIngestionJob, RestartIngestionJob calls from CoreService to JobControllerService.

Users of the Ingestion Job API via gRPC should migrate by:

• Add new client to connect to Job Controller endpoint to call JobControllerService and call ListIngestionJobs, StopIngestionJob, RestartIngestionJob from new client.

• Migrate code to accept feature references instead of feature sets returned in ListIngestionJobs response.

Users of Ingestion Job via Python SDK (ie feast ingest-jobs list or client.stop_ingest_job() etc.) should migrate by:

• ingest_job()methods only: Create a new separate to connect to the job controller and call ingest_job() methods using the new client.

• Configure the Feast Job Controller endpoint url via jobcontroller_url config option.

Configuration Properties Changes

• Rename feast.jobs.consolidate-jobs-per-source property to feast.jobs.controller.consolidate-jobs-per-sources

• Renamefeast.security.authorization.options.subjectClaim to feast.security.authentication.options.subjectClaim

• Rename

Migration from v0.5 to v0.6

Database schema

In Release 0.6 we introduced to handle schema migrations in PostgreSQL. Flyway is integrated into core and for now on all migrations will be run automatically on core start. It uses table flyway_schema_history in the same database (also created automatically) to keep track of already applied migrations. So no specific maintenance should be needed.

If you already have existing deployment of feast 0.5 - Flyway will detect existing tables and omit first baseline migration.

After core started you should have flyway_schema_history look like this

In this release next major schema changes were done:

• Source is not shared between FeatureSets anymore. It's changed to 1:1 relation

  and source's primary key is now auto-incremented number.

• Due to generalization of Source sources.topics & sources.bootstrap_servers columns were deprecated.

  They will be replaced with sources.config. Data migration handled by code when respected Source is used.

New Models (tables):

• feature_statistics

Minor changes:

• FeatureSet has new column version (see for details)

• Connecting table jobs_feature_sets in many-to-many relation between jobs & feature sets

  has now version and delivery_status.

Migration from v0.4 to v0.6

Database

For all versions earlier than 0.5 seamless migration is not feasible due to earlier breaking changes and creation of new database will be required.

Since database will be empty - first (baseline) migration would be applied:

Feast relies on Spark to ingest data from the offline store to the online store, streaming ingestion, and running queries to retrieve historical data from the offline store. Feast supports several Spark deployment options.

Option 1. Use Kubernetes Operator for Apache Spark

To install the Spark on K8s Operator

Currently Feast is tested using v1beta2-1.1.2-2.4.5version of the operator image. To configure Feast to use it, set the following options in Feast config:

Lastly, make sure that the service account used by Feast has permissions to manage Spark Application resources. This depends on your k8s setup, but typically you'd need to configure a Role and a RoleBinding like the one below:

Option 2. Use GCP and Dataproc

If you're running Feast in Google Cloud, you can use Dataproc, a managed Spark platform. To configure Feast to use it, set the following options in Feast config:

See for more configuration options for Dataproc.

Option 3. Use AWS and EMR

If you're running Feast in AWS, you can use EMR, a managed Spark platform. To configure Feast to use it, set at least the following options in Feast config:

See for more configuration options for EMR.

Release process

For Feast maintainers, these are the concrete steps for making a new release.

1. For new major or minor release, create and check out the release branch for the new stream, e.g. v0.6-branch. For a patch version, check out the stream's release branch.