A feature repository is a directory that contains the configuration of the feature store and individual features. This configuration is written as code (Python/YAML) and it's highly recommended that teams track it centrally using git. See Feature Repository for a detailed explanation of feature repositories.

The easiest way to create a new feature repository to use feast init command:

feast init

Creating a new Feast repository in /<...>/tiny_pika.

feast init -t gcp

Creating a new Feast repository in /<...>/tiny_pika.

The init command creates a Python file with feature definitions, sample data, and a Feast configuration file for local development:

$ tree
.
└── tiny_pika
    ├── data
    │   └── driver_stats.parquet
    ├── example.py
    └── feature_store.yaml

1 directory, 3 files

Enter the directory:

# Replace "tiny_pika" with your auto-generated dir name
cd tiny_pika

You can now use this feature repository for development. You can try the following:

• Run feast apply to apply these definitions to Feast.

• Edit the example feature definitions in example.py and run feast apply again to change feature definitions.

• Initialize a git repository in the same directory and checking the feature repository into version control.

In this tutorial we will

1. Deploy a local feature store with a Parquet file offline store and Sqlite online store.

2. Build a training dataset using our time series features from our Parquet files.

3. Materialize feature values from the offline store into the online store.

4. Read the latest features from the online store for inference.

Install Feast

Install the Feast SDK and CLI using pip:

pip install feast

Create a feature repository

Bootstrap a new feature repository using feast init from the command line:

feast init feature_repo
cd feature_repo

Creating a new Feast repository in /home/Jovyan/feature_repo.

Register feature definitions and deploy your feature store

The apply command registers all the objects in your feature repository and deploys a feature store:

feast apply

Registered entity driver_id
Registered feature view driver_hourly_stats
Deploying infrastructure for driver_hourly_stats

Generating training data

The apply command builds a training dataset based on the time-series features defined in the feature repository:

from datetime import datetime

import pandas as pd

from feast import FeatureStore

entity_df = pd.DataFrame.from_dict(
    {
        "driver_id": [1001, 1002, 1003, 1004],
        "event_timestamp": [
            datetime(2021, 4, 12, 10, 59, 42),
            datetime(2021, 4, 12, 8, 12, 10),
            datetime(2021, 4, 12, 16, 40, 26),
            datetime(2021, 4, 12, 15, 1, 12),
        ],
    }
)

store = FeatureStore(repo_path=".")

training_df = store.get_historical_features(
    entity_df=entity_df,
    feature_refs=[
        "driver_hourly_stats:conv_rate",
        "driver_hourly_stats:acc_rate",
        "driver_hourly_stats:avg_daily_trips",
    ],
).to_df()

print(training_df.head())

event_timestamp   driver_id  driver_hourly_stats__conv_rate  driver_hourly_stats__acc_rate  driver_hourly_stats__avg_daily_trips
2021-04-12        1002       0.328245                        0.993218                       329
2021-04-12        1001       0.448272                        0.873785                       767
2021-04-12        1004       0.822571                        0.571790                       673
2021-04-12        1003       0.556326                        0.605357                       335

Load features into your online store

The materialize command loads the latest feature values from your feature views into your online store:

CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")
feast materialize-incremental $CURRENT_TIME

Fetching feature vectors for inference

from pprint import pprint
from feast import FeatureStore

store = FeatureStore(repo_path=".")

feature_vector = store.get_online_features(
    feature_refs=[
        "driver_hourly_stats:conv_rate",
        "driver_hourly_stats:acc_rate",
        "driver_hourly_stats:avg_daily_trips",
    ],
    entity_rows=[{"driver_id": 1001}],
).to_dict()

pprint(feature_vector)

{
    'driver_id': [1001],
    'driver_hourly_stats__conv_rate': [0.49274],
    'driver_hourly_stats__acc_rate': [0.92743],
    'driver_hourly_stats__avg_daily_trips': [72],
}

Next steps

• Follow our Getting Started guide for a hands tutorial in using Feast

• Join other Feast users and contributors in Slack and become part of the community!

Install Feast using pip:

pip install feast

Install Feast with GCP dependencies (required when using BigQuery or Firestore):

pip install 'feast[gcp]'

What is Feast?

Feast (Feature Store) is an operational data system for managing and serving machine learning features to models in production.

Problems Feast Solves

Models need consistent access to data: ML systems built on traditional data infrastructure are often coupled to databases, object stores, streams, and files. A result of this coupling, however, is that any change in data infrastructure may break dependent ML systems. Another challenge is that dual implementations of data retrieval for training and serving can lead to inconsistencies in data, which in turn can lead to training-serving skew.

Feast decouples your models from your data infrastructure by providing a single data access layer that abstracts feature storage from feature retrieval. Feast also provides a consistent means of referencing feature data for retrieval, and therefore ensures that models remain portable when moving from training to serving.

Deploying new features into production is difficult: Many ML teams consist of members with different objectives. Data scientists, for example, aim to deploy features into production as soon as possible, while engineers want to ensure that production systems remain stable. These differing objectives can create an organizational friction that slows time-to-market for new features.

Feast addresses this friction by providing both a centralized registry to which data scientists can publish features, and a battle-hardened serving layer. Together, these enable non-engineering teams to ship features into production with minimal oversight.

Models need point-in-time correct data: ML models in production require a view of data consistent with the one on which they are trained, otherwise the accuracy of these models could be compromised. Despite this need, many data science projects suffer from inconsistencies introduced by future feature values being leaked to models during training.

Feast solves the challenge of data leakage by providing point-in-time correct feature retrieval when exporting feature datasets for model training.

Features aren't reused across projects: Different teams within an organization are often unable to reuse features across projects. The siloed nature of development and the monolithic design of end-to-end ML systems contribute to duplication of feature creation and usage across teams and projects.

Feast addresses this problem by introducing feature reuse through a centralized system (a registry). This registry enables multiple teams working on different projects not only to contribute features, but also to reuse these same features. With Feast, data scientists can start new ML projects by selecting previously engineered features from a centralized registry, and are no longer required to develop new features for each project.

Problems Feast does not yet solve

Feature engineering: We aim for Feast to support light-weight feature engineering as part of our API.

Feature discovery: We also aim for Feast to include a first-class user interface for exploring and discovering entities and features.

‌Feature validation: We additionally aim for Feast to improve support for statistics generation of feature data and subsequent validation of these statistics. Current support is limited.

What Feast is not

ETL or ELT system: Feast is not (and does not plan to become) a general purpose data transformation or pipelining system. Feast plans to include a light-weight feature engineering toolkit, but we encourage teams to integrate Feast with upstream ETL/ELT systems that are specialized in transformation.

Data warehouse: Feast is not a replacement for your data warehouse or the source of truth for all transformed data in your organization. Rather, Feast is a light-weight downstream layer that can serve data from an existing data warehouse (or other data sources) to models in production.

Data catalog: Feast is not a general purpose data catalog for your organization. Feast is purely focused on cataloging features for use in ML pipelines or systems, and only to the extent of facilitating the reuse of features.

How can I get started?

Explore the following resources to get started with Feast:

• Quickstart is the fastest way to get started with Feast

• Getting started provides a step-by-step guide to using Feast.

• Concepts describes all important Feast API concepts.

• Reference contains detailed API and design documents.

• Contributing contains resources for anyone who wants to contribute to Feast.

Feast allows users to load their feature data into an online store in order to serve the latest features to models for online prediction.

Materializing features

1. Register feature views

Before proceeding, please ensure that you have applied (registered) the feature views that should be materialized.

2.a Materialize

The materialize command allows users to materialize features over a specific historical time range into the online store.

feast materialize 2021-04-07T00:00:00 2021-04-08T00:00:00

The above command will query the batch sources for all feature views over the provided time range, and load the latest feature values into the configured online store.

It is also possible to materialize for specific feature views by using the -v / --views argument.

feast materialize 2021-04-07T00:00:00 2021-04-08T00:00:00 \
--views driver_hourly_stats

The materialize command is completely stateless. It requires the user to provide the time ranges that will be loaded into the online store. This command is best used from a scheduler that tracks state, like Airflow.

2.b Materialize Incremental (Alternative)

For simplicity, Feast also provides a materialize command that will only ingest new data that has arrived in the offline store. Unlike materialize, materialize-incremental will track the state of previous ingestion runs inside of the feature registry.

The example command below will load only new data that has arrived for each feature view up to the end date and time (2021-04-08T00:00:00).

feast materialize-incremental 2021-04-08T00:00:00

The materialize-incremental command functions similarly to materialize in that it loads data over a specific time range for all feature views (or the selected feature views) into the online store.

Unlike materialize, materialize-incremental automatically determines the start time from which to load features from batch sources of each feature view. The first time materialize-incremental is executed it will set the start time to the oldest timestamp of each data source, and the end time as the one provided by the user. For each run of materialize-incremental, the end timestamp will be tracked.

Subsequent runs of materialize-incremental will then set the start time to the end time of the previous run, thus only loading new data that has arrived into the online store. Note that the end time that is tracked for each run is at the feature view level, not globally for all feature views, i.e, different feature views may have different periods that have been materialized into the online store.

The Feast Python SDK allows users to retrieve feature values from an online store. This API is used to look up feature values at low latency during model serving in order to make online predictions.

Retrieving online features

1. Ensure that feature values have been loaded into the online store

Please ensure that you have materialized (loaded) your feature values into the online store before starting

2. Define feature references

Create a list of features that you would like to retrieve. This list typically comes from the model training step and should accompany the model binary.

feature_refs = [
    "driver_hourly_stats:conv_rate",
    "driver_hourly_stats:acc_rate"
]

3. Read online features

Next, we will create a feature store object and call get_online_features() which reads the relevant feature values directly from the online store.

fs = FeatureStore(repo_path="path/to/feature/repo")
online_features = fs.get_online_features(
    feature_refs=feature_refs,
    entity_rows=[
        {"driver_id": 1001},
        {"driver_id": 1002}]
).to_dict()

{
   "driver_hourly_stats__acc_rate":[
      0.2897740304470062,
      0.6447265148162842
   ],
   "driver_hourly_stats__conv_rate":[
      0.6508077383041382,
      0.14802511036396027
   ],
   "driver_id":[
      1001,
      1002
   ]
}

The top-level namespace within Feast is a . Users define one or more within a project. Each feature view contains one or more that relate to a specific . A feature view must always have a , which in turn is used during the generation of training and when materializing feature values into the online store.

Project

Projects provide complete isolation of feature stores at the infrastructure level. This is accomplished through resource namespacing, e.g., prefixing table names with the associated project. Each project should be considered a completely separate universe of entities and features. It is not possible to retrieve features from multiple projects in a single request. We recommend having a single feature store and a single project per environment (dev, staging, prod).

Links & Resources

• : Feel free to ask questions or say hello!

• : We have both a user and developer mailing list.

  • Feast users should join group by clicking .

  • Feast developers should join group by clicking .

• : This folder is used as a central repository for all Feast resources. For example:

  • Design proposals in the form of Request for Comments (RFC).

  • User surveys and meeting minutes.

  • Slide decks of conferences our contributors have spoken at.

• : Find the complete Feast codebase on GitHub.

• : Our LFAI wiki page contains links to resources for contributors and maintainers.

How can I get help?

• Slack: Need to speak to a human? Come ask a question in our Slack channel (link above).

• GitHub Issues: Found a bug or need a feature? .

• StackOverflow: Need to ask a question on how to use Feast? We also monitor and respond to .

Community Calls

We have a user and contributor community call every two weeks (Asia & US friendly).

Frequency (alternating times every 2 weeks)

• Tuesday 18:00 pm to 18:30 pm (US, Asia)

• Tuesday 10:00 am to 10:30 am (US, Europe)

Links

Backlog

• Add On-demand transformations support

• Add Data quality monitoring

• Add Snowflake offline store support

• Add Bigtable support

• Add Push/Ingestion API support

Scheduled for development (next 3 months)

• Ensure Feast Serving is compatible with the new Feast

  • Decouple Feast Serving from Feast Core

  • Add FeatureView support to Feast Serving

  • Update Helm Charts (remove Core, Postgres, Job Service, Spark)

• Add Redis support for Feast

• Add direct deployment support to AWS and GCP

• Add Dynamo support

• Add Redshift support

Feast 0.10

New Functionality

1. Full local mode support (Sqlite and Parquet)

2. Provider model for added extensibility

3. Firestore support

4. Native (No-Spark) BigQuery support

5. Added support for object store based registry

6. Add support for FeatureViews

7. Added support for infrastructure configuration through apply

Technical debt, refactoring, or housekeeping

1. Remove dependency on Feast Core

2. Feast Serving made optional

3. Moved Python API documentation to Read The Docs

Feast 0.9

New Functionality

• Added Feast Job Service for management of ingestion and retrieval jobs

Note: Please see discussion thread above for functionality that did not make this release.

Feast 0.8

New Functionality

1. Add support for AWS (data sources and deployment)

2. Add support for local deployment

3. Add support for Spark based ingestion

4. Add support for Spark based historical retrieval

Technical debt, refactoring, or housekeeping

1. Move job management functionality to SDK

2. Remove Apache Beam based ingestion

3. Allow direct ingestion from batch sources that does not pass through stream

4. Remove Feast Historical Serving abstraction to allow direct access from Feast SDK to data sources for retrieval

Feast 0.7

New Functionality

Technical debt, refactoring, or housekeeping

Feast 0.6

New functionality

4. Improved searching and filtering of features and entities

Technical debt, refactoring, or housekeeping

Feast 0.5

New functionality

Technical debt, refactoring, or housekeeping

The Feast online store is used for low-latency online feature value lookups. Feature values are loaded into the online store from data sources in feature views using the materialize command.

The storage schema of features within the online store mirrors that of the data source used to populate the online store. One key difference between the online store and data sources is that only the latest feature values are stored per entity key. No historical values are stored.

Example batch data source

Once the above data source is materialized into Feast (using feast materialize), the feature values will be stored as follows:

Feast uses offline stores as storage and compute systems. Offline stores store historic time-series feature values. Feast does not generate these features, but instead uses the offline store as the interface for querying existing features in your organization.

Offline stores are used primarily for two reasons

1. Building training datasets from time-series features.

2. Materializing (loading) features from the offline store into an online store in order to serve those features at low latency for prediction.

Offline stores are configured through the . When building training datasets or materializing features into an online store, Feast will use the configured offline store along with the data sources you have defined as part of feature views to execute the necessary data operations.

It is not possible to query all data sources from all offline stores, and only a single offline store can be used at a time. For example, it is not possible to query a BigQuery table from a File offline store, nor is it possible for a BigQuery offline store to query files from your local file system.

Please see the reference for more details on configuring offline stores.

Functionality

• Create Batch Features: ELT/ETL systems like Spark and SQL are used to transform data in the batch store.

• Feast Apply: The user (or CI) publishes versioned controlled feature definitions using feast apply. This CLI command updates infrastructure and persists definitions in the object store registry.

• Feast Materialize: The user (or scheduler) executes feast materialize which loads features from the offline store into the online store.

• Model Training: A model training pipeline is launched. It uses the Feast Python SDK to retrieve a training dataset and trains a model.

• Get Historical Features: Feast exports a point-in-time correct training dataset based on the list of features and entity dataframe provided by the model training pipeline.

• Deploy Model: The trained model binary (and list of features) are deployed into a model serving system. This step is not executed by Feast.

• Prediction: A backend system makes a request for a prediction from the model serving service.

• Get Online Features: The model serving service makes a request to the Feast Online Serving service for online features using a Feast SDK.

Components

A complete Feast deployment contains the following components:

• Feast Online Serving: Provides low-latency access to feature values stores in the online store. This component is optional. Teams can also read feature values directly from the online store if necessary.

• Feast Registry: An object store (GCS, S3) based registry used to persist feature definitions that are registered with the feature store. Systems can discover feature data by interacting with the registry through the Feast SDK.

• Feast Python SDK/CLI: The primary user facing SDK. Used to:

  • Manage version controlled feature definitions.

  • Materialize (load) feature values into the online store.

  • Build and retrieve training datasets from the offline store.

  • Retrieve online features.

• Online Store: The online store is a database that stores only the latest feature values for each entity. The online store is populated by materialization jobs.

• Offline Store: The offline store persists batch data that has been ingested into Feast. This data is used for producing training datasets. Feast does not manage the offline store directly, but runs queries against it.

A provider is an implementation of a feature store using specific feature store components targeting a specific environment. More specifically, a provider is the target environment to which you have configured your feature store to deploy and run.

Providers are built to orchestrate various components (offline store, online store, infrastructure, compute) inside an environment. For example, the gcp provider supports as an offline store and as an online store, ensuring that these components can work together seamlessly.

Providers also come with default configurations which makes it easier for users to start a feature store in a specific environment.

Please see for configuring providers.

The Feast CLI can be used to deploy a feature store to your infrastructure, spinning up any necessary persistent resources like buckets or tables in data stores. The deployment target and effects depend on the provider that has been configured in your file, as well as the feature definitions found in your feature repository.

Deploying

To have Feast deploy your infrastructure, run feast apply from your command line while inside a feature repository:

Depending on whether the feature repository is configured to use a local provider or one of the cloud providers like GCP or AWS, it may take from a couple of seconds to a minute to run to completion.

Cleaning up

If you need to clean up the infrastructure created by feast apply, use the teardown command.

****

Feast allows users to build a training dataset from time-series feature data that already exists in an offline store. Users are expected to provide a list of features to retrieve (which may span multiple feature views), and a dataframe to join the resulting features onto. Feast will then execute a point-in-time join of multiple feature views onto the provided dataframe, and return the full resulting dataframe.

Retrieving historical features

1. Register your feature views

Please ensure that you have created a feature repository and that you have registered (applied) your feature views with Feast.

2. Define feature references

Start by defining the feature references (e.g., driver_trips:average_daily_rides) for the features that you would like to retrieve from the offline store. These features can come from multiple feature tables. The only requirement is that the feature tables that make up the feature references have the same entity (or composite entity), and that they aren't located in the same offline store.

3. Create an entity dataframe

An entity dataframe is the target dataframe on which you would like to join feature values. The entity dataframe must contain a timestamp column called event_timestamp and all entities (primary keys) necessary to join feature tables onto. All entities found in feature views that are being joined onto the entity dataframe must be found as column on the entity dataframe.

It is possible to provide entity dataframes as either a Pandas dataframe or a SQL query.

Pandas:

In the example below we create a Pandas based entity dataframe that has a single row with an event_timestamp column and a driver_id entity column. Pandas based entity dataframes may need to be uploaded into an offline store, which may result in longer wait times compared to a SQL based entity dataframe.

SQL (Alternative):

Below is an example of an entity dataframe built from a BigQuery SQL query. It is only possible to use this query when all feature views being queried are available in the same offline store (BigQuery).

4. Launch historical retrieval

Once the feature references and an entity dataframe are defined, it is possible to call get_historical_features(). This method launches a job that executes a point-in-time join of features from the offline store onto the entity dataframe. Once completed, a job reference will be returned. This job reference can then be converted to a Pandas dataframe by calling to_df().

Feature View

A feature view is an object that represents a logical group of time-series feature data as it is found in a . Feature views consist of one or more , , and a . Feature views allow Feast to model your existing feature data in a consistent way in both an offline (training) and online (serving) environment.

Feature views are used during

• The generation of training datasets by querying the data source of feature views in order to find historical feature values. A single training dataset may consist of features from multiple feature views.

• Loading of feature values into an online store. Feature views determine the storage schema in the online store.

• Retrieval of features from the online store. Feature views provide the schema definition to Feast in order to look up features from the online store.

Data Source

Feast uses a time-series data model to represent data. This data model is used to interpret feature data in data sources in order to build training datasets or when materializing features into an online store.

Below is an example data source with a single entity (driver) and two features (trips_today, and rating).

Entity

An entity is a collection of semantically related features. Users define entities to map to the domain of their use case. For example, a ride-hailing service could have customers and drivers as their entities, which group related features that correspond to these customers and drivers.

Entities are defined as part of feature views. Entities are used to identify the primary key on which feature values should be stored and retrieved. These keys are used during the lookup of feature values from the online store and the join process in point-in-time joins. It is possible to define composite entities (more than one entity object) in a feature view.

Entities should be reused across feature views.

Feature

A feature is an individual measurable property observed on an entity. For example, a feature of a customer entity could be the number of transactions they have made on an average month.

Features are defined as part of feature views. Since Feast does not transform data, a feature is essentially a schema that only contains a name and a type:

Dataset

A dataset is a collection of rows that is produced by a historical retrieval from Feast in order to train a model. A dataset is produced by a join from one or more feature views onto an entity dataframe. Therefore, a dataset may consist of features from multiple feature views.

Dataset vs Feature View: Feature views contain the schema of data and a reference to where data can be found (through its data source). Datasets are the actual data manifestation of querying those data sources.

Dataset vs Data Source: Datasets are the output of historical retrieval, whereas data sources are the inputs. One or more data sources can be used in the creation of a dataset.

Feature References

Feature references uniquely identify feature values in Feast. The structure of a feature reference in string form is as follows: <feature_table>:<feature>

Feature references are used for the retrieval of features from Feast:

It is possible to retrieve features from multiple feature views with a single request, and Feast is able to join features from multiple tables in order to build a training dataset. However, It is not possible to reference (or retrieve) features from multiple projects at the same time.

Entity key

Entity keys are one or more entity values that uniquely describe an entity. In the case of an entity (like a driver) that only has a single entity field, the entity is an entity key. However, it is also possible for an entity key to consist of multiple entity values. For example, a feature view with the composite entity of (customer, country) might have an entity key of (1001, 5).

Entity keys act as primary keys. They are used during the lookup of features from the online store, and they are also used to match feature rows across feature views during point-in-time joins.

Event timestamp

The timestamp on which an event occurred, as found in a feature view's data source. The entity timestamp describes the event time at which a feature was observed or generated.

Event timestamps are used during point-in-time joins to ensure that the latest feature values are joined from feature views onto entity rows. Event timestamps are also used to ensure that old feature values aren't served to models during online serving.

Entity row

An entity key at a specific point in time.

Entity dataframe

A collection of entity rows. Entity dataframes are the "left table" that is enriched with feature values when building training datasets. The entity dataframe is provided to Feast by users during historical retrieval:

Example of an entity dataframe with feature values joined to it:

Description

The BigQuery offline store provides support for reading .

• BigQuery tables and views are allowed as sources.

• All joins happen within BigQuery.

• Entity dataframes can be provided as a SQL query or can be provided as a Pandas dataframe. Pandas dataframes will be uploaded to BigQuery in order to complete join operations.

• A is returned when calling get_historical_features().

Example

Configuration options are available .

Please see for an explanation of online stores.

Description

The online store provides support for materializing feature values into an SQLite database for serving online features.

• All feature values are stored in an on-disk SQLite database

• Only the latest feature values are persisted

Example

Configuration options are available .

Description

The File offline store provides support for reading FileSources.

• Only Parquet files are currently supported.

• All data is downloaded and joined using Python and may not scale to production workloads.

Example

project: my_feature_repo
registry: data/registry.db
provider: local
offline_store:
  type: file

Configuration options are available here.

Description

The Redis online store provides support for materializing feature values into Redis.

• Both Redis and Redis Cluster are supported

• The data model used to store feature values in Redis is described in more detail here.

Examples

Connecting to a single Redis instance

project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
  type: redis
  connection_string: "localhost:6379"

Connecting to a Redis Cluster with SSL enabled and password authentication

project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
  type: redis
  redis_type: redis_cluster
  connection_string: "redis1:6379,redis2:6379,ssl=true,password=my_password"

Configuration options are available here.

Description

File data sources allow for the retrieval of historical feature values from files on disk for building training datasets, as well as for materializing features into an online store.

Example

from feast import FileSource
from feast.data_format import ParquetFormat

parquet_file_source = FileSource(
    file_format=ParquetFormat(),
    file_url="file:///feast/customer.parquet",
)

Configuration options are available here.

Description

The Datastore online store provides support for materializing feature values into Cloud Datastore. The data model used to store feature values in Datastore is described in more detail here.

Example

project: my_feature_repo
registry: data/registry.db
provider: gcp
online_store:
  type: datastore
  project_id: my_gcp_project
  namespace: my_datastore_namespace

Configuration options are available here.

Description

BigQuery data sources allow for the retrieval of historical feature values from BigQuery for building training datasets as well as materializing features into an online store.

• Either a table reference or a SQL query can be provided.

• No performance guarantees can be provided over SQL query-based sources. Please use table references where possible.

Examples

Using a table reference

from feast import BigQuerySource

my_bigquery_source = BigQuerySource(
    table_ref="gcp_project:bq_dataset.bq_table",
)

Using a query

from feast import BigQuerySource

BigQuerySource(
    query="SELECT timestamp as ts, created, f1, f2 "
          "FROM `my_project.my_dataset.my_features`",
)

Configuration options are available here.

Overview

The Feast CLI comes bundled with the Feast Python package. It is immediately available after installing Feast.

Usage: feast [OPTIONS] COMMAND [ARGS]...

  Feast CLI

  For more information, see our public docs at https://docs.feast.dev/

  For any questions, you can reach us at https://slack.feast.dev/

Options:
  -c, --chdir TEXT  Switch to a different feature repository directory before
                    executing the given subcommand.

  --help            Show this message and exit.

Commands:
  apply                    Create or update a feature store deployment
  entities                 Access entities
  feature-views            Access feature views
  init                     Create a new Feast repository
  materialize              Run a (non-incremental) materialization job to...
  materialize-incremental  Run an incremental materialization job to ingest...
  registry-dump            Print contents of the metadata registry
  teardown                 Tear down deployed feature store infrastructure
  version                  Display Feast SDK version

Global Options

The Feast CLI provides one global top-level option that can be used with other commands

chdir (-c, --chdir)

This command allows users to run Feast CLI commands in a different folder from the current working directory.

feast -c path/to/my/feature/repo apply

Apply

Creates or updates a feature store deployment

feast apply

What does Feast apply do?

1. Feast will scan Python files in your feature repository and find all Feast object definitions, such as feature views, entities, and data sources.

2. Feast will validate your feature definitions

3. Feast will sync the metadata about Feast objects to the registry. If a registry does not exist, then it will be instantiated. The standard registry is a simple protobuf binary file that is stored on disk (locally or in an object store).

4. Feast CLI will create all necessary feature store infrastructure. The exact infrastructure that is deployed or configured depends on the provider configuration that you have set in feature_store.yaml. For example, setting local as your provider will result in a sqlite online store being created.

Entities

List all registered entities

feast entities list

NAME       DESCRIPTION    TYPE
driver_id  driver id      ValueType.INT64

Feature views

List all registered feature views

feast feature-views list

NAME                 ENTITIES
driver_hourly_stats  ['driver_id']

Init

Creates a new feature repository

feast init my_repo_name

Creating a new Feast repository in /projects/my_repo_name.

.
├── data
│   └── driver_stats.parquet
├── example.py
└── feature_store.yaml

It's also possible to use other templates

feast init -t gcp my_feature_repo

or to set the name of the new project

feast init -t gcp my_feature_repo

Materialize

Load data from feature views into the online store between two dates

feast materialize 2020-01-01T00:00:00 2022-01-01T00:00:00

Load data for specific feature views into the online store between two dates

feast materialize -v driver_hourly_stats 2020-01-01T00:00:00 2022-01-01T00:00:00

Materializing 1 feature views from 2020-01-01 to 2022-01-01

driver_hourly_stats:
100%|██████████████████████████| 5/5 [00:00<00:00, 5949.37it/s]

Materialize incremental

Load data from feature views into the online store, beginning from either the previous materialize or materialize-incremental end date, or the beginning of time.

feast materialize-incremental 2022-01-01T00:00:00

Teardown

Tear down deployed feature store infrastructure

feast teardown

Version

Print the current Feast version

feast version

How Feast SDK usage is measured

The Feast project logs anonymous usage statistics and errors in order to inform our planning. Several client methods are tracked, beginning in Feast 0.9. Users are assigned a UUID which is sent along with the name of the method, the Feast version, the OS (using sys.platform), and the current time.

The source code is available here.

How to disable usage logging

Set the environment variable FEAST_USAGE to False.

Overview

.feastignore is a file that is placed at the root of the Feature Repository. This file contains paths that should be ignored when running feast apply. An example .feastignore is shown below:

# Ignore virtual environment
venv

# Ignore a specific Python file
scripts/foo.py

# Ignore all Python files directly under scripts directory
scripts/*.py

# Ignore all "foo.py" anywhere under scripts directory
scripts/**/foo.py

.feastignore file is optional. If the file can not be found, every Python in the feature repo directory will be parsed by feast apply.

Feast Ignore Patterns

Overview

feature_store.yaml is used to configure a feature store. The file must be located at the root of a feature repository. An example feature_store.yaml is shown below:

project: loyal_spider
registry: data/registry.db
provider: local
online_store:
    type: sqlite
    path: data/online_store.db

Options

The following top-level configuration options exist in the feature_store.yaml file.

• provider — Configures the environment in which Feast will deploy and operate.

• registry — Configures the location of the feature registry.

• online_store — Configures the online store.

• offline_store — Configures the offline store.

• project — Defines a namespace for the entire feature store. Can be used to isolate multiple deployments in a single installation of Feast.

Please see the RepoConfig API reference for the full list of configuration options.

Feast manages two important sets of configuration: feature definitions, and configuration about how to run the feature store. With Feast, this configuration can be written declaratively and stored as code in a central location. This central location is called a feature repository, and it's essentially just a directory that contains some code files.

The feature repository is the declarative source of truth for what the desired state of a feature store should be. The Feast CLI uses the feature repository to configure your infrastructure, e.g., migrate tables.

What is a feature repository?

A feature repository consists of:

• A collection of Python files containing feature declarations.

• A feature_store.yaml file containing infrastructural configuration.

• A .feastignore file containing paths in the feature repository to ignore.

Structure of a feature repository

The structure of a feature repository is as follows:

• The root of the repository should contain a feature_store.yaml file and may contain a .feastignore file.

• The repository should contain Python files that contain feature definitions.

• The repository can contain other files as well, including documentation and potentially data files.

An example structure of a feature repository is shown below:

$ tree -a
.
├── data
│   └── driver_stats.parquet
├── driver_features.py
├── feature_store.yaml
└── .feastignore

1 directory, 4 files

A couple of things to note about the feature repository:

• Feast reads all Python files recursively when feast apply is ran, including subdirectories, even if they don't contain feature definitions.

• It's recommended to add .feastignore and add paths to all imperative scripts if you need to store them inside the feature registry.

The feature_store.yaml configuration file

The configuration for a feature store is stored in a file named feature_store.yaml , which must be located at the root of a feature repository. An example feature_store.yaml file is shown below:

project: my_feature_repo_1
registry: data/metadata.db
provider: local
online_store:
    path: data/online_store.db

The feature_store.yaml file configures how the feature store should run. See feature_store.yaml for more details.

The .feastignore file

This file contains paths that should be ignored when running feast apply. An example .feastignore is shown below:

# Ignore virtual environment
venv

# Ignore a specific Python file
scripts/foo.py

# Ignore all Python files directly under scripts directory
scripts/*.py

# Ignore all "foo.py" anywhere under scripts directory
scripts/**/foo.py

See .feastignore for more details.

Feature definitions

A feature repository can also contain one or more Python files that contain feature definitions. An example feature definition file is shown below:

from datetime import timedelta

from feast import BigQuerySource, Entity, Feature, FeatureView, ValueType

driver_locations_source = BigQuerySource(
    table_ref="rh_prod.ride_hailing_co.drivers",
    event_timestamp_column="event_timestamp",
    created_timestamp_column="created_timestamp",
)

driver = Entity(
    name="driver",
    value_type=ValueType.INT64,
    description="driver id",
)

driver_locations = FeatureView(
    name="driver_locations",
    entities=["driver"],
    ttl=timedelta(days=1),
    features=[
        Feature(name="lat", dtype=ValueType.FLOAT),
        Feature(name="lon", dtype=ValueType.STRING),
    ],
    input=driver_locations_source,
)

To declare new feature definitions, just add code to the feature repository, either in existing files or in a new file. For more information on how to define features, see Feature Views.

Next steps

• See Create a feature repository to get started with an example feature repository.

• See feature_store.yaml, .feastignore, or Feature Views for more information on the configuration files that live in a feature registry.

Description

• Offline Store: Uses the BigQuery offline store by default. Also supports File as the offline store.

• Online Store: Uses the Datastore online store by default. Also supports Sqlite as an online store.

Example

project: my_feature_repo
registry: gs://my-bucket/data/registry.db
provider: gcp

Permissions

Description

• Offline Store: Uses the File offline store by default. Also supports BigQuery as the offline store.

• Online Store: Uses the Sqlite online store by default. Also supports Datastore as an online store.

Example

project: my_feature_repo
registry: data/registry.db
provider: local

Please see Data Source for an explanation of data sources.

Please see Offline Store for an explanation of offline stores.

Please see Provider for an explanation of providers.

Install Feast

If you would like to deploy a new installation of Feast, click on

Connect to Feast

If you would like to connect to an existing Feast deployment, click on

Learn Feast

If you would like to learn more about Feast, click on

A production deployment of Feast is deployed using Kubernetes.

Kubernetes (with Helm)

This guide installs Feast into an existing Kubernetes cluster using Helm. The installation is not specific to any cloud platform or environment, but requires Kubernetes and Helm.

Amazon EKS (with Terraform)

This guide installs Feast into an AWS environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.

Azure AKS (with Helm)

This guide installs Feast into an Azure AKS environment with Helm.

Azure AKS (with Terraform)

This guide installs Feast into an Azure environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.

Google Cloud GKE (with Terraform)

This guide installs Feast into a Google Cloud environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.

IBM Cloud Kubernetes Service (IKS) and Red Hat OpenShift (using Kustomize)

Overview

This guide installs Feast on an existing Kubernetes cluster, and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Requirements

1. Install and configure

2. Install

2. Preparation

Add the Feast Helm repository and download the latest charts:

Feast includes a Helm chart that installs all necessary components to run Feast Core, Feast Online Serving, and an example Jupyter notebook.

Feast Core requires Postgres to run, which requires a secret to be set on Kubernetes:

3. Installation

Install Feast using Helm. The pods may take a few minutes to initialize.

4. Use Jupyter to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

5. Further Reading

Overview

This guide installs Feast on AWS using our .

This Terraform configuration creates the following resources:

• Kubernetes cluster on Amazon EKS (3x r3.large nodes)

• Kafka managed by Amazon MSK (2x kafka.t3.small nodes)

• Postgres database for Feast metadata, using serverless Aurora (min capacity: 2)

• Redis cluster, using Amazon Elasticache (1x cache.t2.micro)

• Amazon EMR cluster to run Spark (3x spot m4.xlarge)

• Staging S3 bucket to store temporary data

1. Requirements

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/aws. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. At a minimum, you need to set name_prefix and an AWS region:

3. Apply

After completing the configuration, initialize Terraform and apply:

Starting may take a minute. A kubectl configuration file is also created in this directory, and the file's name will start with kubeconfig_ and end with a random suffix.

4. Connect to Feast using Jupyter

After all pods are running, connect to the Jupyter Notebook Server running in the cluster.

To connect to the remote Feast server you just created, forward a port from the remote k8s cluster to your local machine. Replace kubeconfig_XXXXXXX below with the kubeconfig file name Terraform generates for you.

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

Feast Python SDK

The Feast Python SDK is used as a library to interact with a Feast deployment.

• Define, register, and manage entities and features

• Ingest data into Feast

• Build and retrieve training datasets

• Retrieve online features

Feast CLI

The Feast CLI is a command line implementation of the Feast Python SDK.

• Define, register, and manage entities and features from the terminal

• Ingest data into Feast

• Manage ingestion jobs

Online Serving Clients

The following clients can be used to retrieve online feature values:

Overview

This guide shows you how to deploy Feast using . Docker Compose allows you to explore the functionality provided by Feast while requiring only minimal infrastructure.

This guide includes the following containerized components:

• • Feast Core with Postgres

  • Feast Online Serving with Redis.

  • Feast Job Service

• A Jupyter Notebook Server with built in Feast example(s). For demo purposes only.

• A Kafka cluster for testing streaming ingestion. For demo purposes only.

Get Feast

Clone the latest stable version of Feast from the :

Create a new configuration file:

Start Feast

Start Feast with Docker Compose:

Wait until all all containers are in a running state:

Try our example(s)

You can now connect to the bundled Jupyter Notebook Server running at localhost:8888 and follow the example Jupyter notebook.

Troubleshooting

Open ports

Please ensure that the following ports are available on your host machine:

• 6565

• 6566

• 8888

• 9094

• 5432

Containers are restarting or unavailable

If some of the containers continue to restart, or you are unable to access a service, inspect the logs using the following command:

Configuration

The Feast Docker Compose setup can be configured by modifying properties in your .env file.

Accessing Google Cloud Storage (GCP)

To access Google Cloud Storage as a data source, the Docker Compose installation requires access to a GCP service account.

• Grant the service account access to your bucket(s).

• Copy the service account to the path you have configured in .env under GCP_SERVICE_ACCOUNT.

• Restart your Docker Compose setup of Feast.

Overview

This guide installs Feast on Azure Kubernetes cluster (known as AKS), and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Spark

• Kafka

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Requirements

1. Install and configure

2. Install and configure

3. Install

2. Preparation

Add the Feast Helm repository and download the latest charts:

Feast includes a Helm chart that installs all necessary components to run Feast Core, Feast Online Serving, and an example Jupyter notebook.

Feast Core requires Postgres to run, which requires a secret to be set on Kubernetes:

3. Feast installation

Install Feast using Helm. The pods may take a few minutes to initialize.

4. Spark operator installation

and ensure the service account used by Feast has permissions to manage Spark Application resources. This depends on your k8s setup, but typically you'd need to configure a Role and a RoleBinding like the one below:

5. Use Jupyter to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

6. Environment variables

7. Further Reading

Overview

This guide installs Feast on an existing IBM Cloud Kubernetes cluster or Red Hat OpenShift on IBM Cloud , and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Kafka (Optional)

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Prerequisites

1. or

2. Install that matches the major.minor versions of your IKS or Install the that matches your local operating system and OpenShift cluster version.

3. Install

4. Install

2. Preparation

IBM Cloud Block Storage Setup (IKS only)

2. Add the IBM Cloud Helm chart repository to the cluster where you want to use the IBM Cloud Block Storage plug-in.
3. Install the IBM Cloud Block Storage plug-in. When you install the plug-in, pre-defined block storage classes are added to your cluster.

   Example output:
4. Verify that all block storage plugin pods are in a "Running" state.
5. Verify that the storage classes for Block Storage were added to your cluster.
6. Set the Block Storage as the default storageclass.

   Example output:

   Security Context Constraint Setup (OpenShift only)

3. Installation

Install Feast using kustomize. The pods may take a few minutes to initialize.

Optional: Enable Feast Jupyter and Kafka

You may optionally enable the Feast Jupyter component which contains code examples to demonstrate Feast. Some examples require Kafka to stream real time features to the Feast online serving. To enable, edit the following properties in the values.yaml under the manifests/contrib/feast folder:

Then regenerate the resource manifests and deploy:

4. Use Feast Jupyter Notebook Server to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

5. Uninstall Feast

6. Troubleshooting

When running the minimal_ride_hailing_example Jupyter Notebook example the following errors may occur:

1. When running job = client.get_historical_features(...):

   or

   Add the following environment variable:
2. When running job.get_status()

   Add the following environment variable:
3. When running job = client.start_stream_to_online_ingestion(...)

   Add the following environment variable:

Overview

This guide installs Feast on GKE using our .

This Terraform configuration creates the following resources:

• GKE cluster

• Feast services running on GKE

• Google Memorystore (Redis) as online store

• Dataproc cluster

• Kafka running on GKE, exposed to the dataproc cluster via internal load balancer

1. Requirements

• Install > = 0.12 (tested with 0.13.3)

• Install (tested with v3.3.4)

• GCP and sufficient to create the resources listed above.

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/gcp. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. Sample configurations are provided below:

3. Apply

After completing the configuration, initialize Terraform and apply:

Install the Feast CLI using pip:

Configure the CLI to connect to your Feast Core deployment:

The CLI is a wrapper around the :

Explore the following resources to learn more about Feast:

• Concepts describes all important Feast API concepts.

• User guide provides guidance on completing Feast workflows.

• Examples contains Jupyter notebooks that you can run on your Feast deployment.

• Advanced contains information about both advanced and operational aspects of Feast.

• Reference contains detailed API and design documents for advanced users.

• Contributing contains resources for anyone who wants to contribute to Feast.

Overview

Sources are descriptions of external feature data and are registered to Feast as part of feature tables. Once registered, Feast can ingest feature data from these sources into stores.

Currently, Feast supports the following source types:

Batch Source

• File (as in Spark): Parquet (only).

• BigQuery

Stream Source

• Kafka

• Kinesis

The following encodings are supported on streams

• Avro

• Protobuf

Structure of a Source

For both batch and stream sources, the following configurations are necessary:

• Event timestamp column: Name of column containing timestamp when event data occurred. Used during point-in-time join of feature values to entity timestamps.

• Created timestamp column: Name of column containing timestamp when data is created. Used to deduplicate data when multiple copies of the same entity key is ingested.

Example data source specifications:

from feast import FileSource
from feast.data_format import ParquetFormat

batch_file_source = FileSource(
    file_format=ParquetFormat(),
    file_url="file:///feast/customer.parquet",
    event_timestamp_column="event_timestamp",
    created_timestamp_column="created_timestamp",
)

from feast import KafkaSource
from feast.data_format import ProtoFormat

stream_kafka_source = KafkaSource(
    bootstrap_servers="localhost:9094",
    message_format=ProtoFormat(class_path="class.path"),
    topic="driver_trips",
    event_timestamp_column="event_timestamp",
    created_timestamp_column="created_timestamp",
)

The Feast Python API documentation provides more information about options to specify for the above sources.

Working with a Source

Creating a Source

Sources are defined as part of feature tables:

batch_bigquery_source = BigQuerySource(
    table_ref="gcp_project:bq_dataset.bq_table",
    event_timestamp_column="event_timestamp",
    created_timestamp_column="created_timestamp",
)

stream_kinesis_source = KinesisSource(
    bootstrap_servers="localhost:9094",
    record_format=ProtoFormat(class_path="class.path"),
    region="us-east-1",
    stream_name="driver_trips",
    event_timestamp_column="event_timestamp",
    created_timestamp_column="created_timestamp",
)

Feast ensures that the source complies with the schema of the feature table. These specified data sources can then be included inside a feature table specification and registered to Feast Core.

Install the Feast Python SDK using pip:

pip install feast==0.9.*

Connect to an existing Feast Core deployment:

from feast import Client

# Connect to an existing Feast Core deployment
client = Client(core_url='feast.example.com:6565')

# Ensure that your client is connected by printing out some feature tables
client.list_feature_tables()

Overview

Feature tables are both a schema and a logical means of grouping features, data sources, and other related metadata.

Feature tables serve the following purposes:

• Feature tables are a means for defining the location and properties of data sources.

• Feature tables are used to create within Feast a database-level structure for the storage of feature values.

• The data sources described within feature tables allow Feast to find and ingest feature data into stores within Feast.

• Feature tables ensure data is efficiently stored during ingestion by providing a grouping mechanism of features values that occur on the same event timestamp.

Features

A feature is an individual measurable property observed on an entity. For example the amount of transactions (feature) a customer (entity) has completed. Features are used for both model training and scoring (batch, online).

Features are defined as part of feature tables. Since Feast does not apply transformations, a feature is basically a schema that only contains a name and a type:

avg_daily_ride = Feature("average_daily_rides", ValueType.FLOAT)

Visit FeatureSpec for the complete feature specification API.

Structure of a Feature Table

Feature tables contain the following fields:

• Name: Name of feature table. This name must be unique within a project.

• Entities: List of entities to associate with the features defined in this feature table. Entities are used as lookup keys when retrieving features from a feature table.

• Features: List of features within a feature table.

• Labels: Labels are arbitrary key-value properties that can be defined by users.

• Max age: Max age affect the retrieval of features from a feature table. Age is measured as the duration of time between the event timestamp of a feature and the lookup time on an entity key used to retrieve the feature. Feature values outside max age will be returned as unset values. Max age allows for eviction of keys from online stores and limits the amount of historical scanning required for historical feature values during retrieval.

• Batch Source: The batch data source from which Feast will ingest feature values into stores. This can either be used to back-fill stores before switching over to a streaming source, or it can be used as the primary source of data for a feature table. Visit Sources to learn more about batch sources.

• Stream Source: The streaming data source from which you can ingest streaming feature values into Feast. Streaming sources must be paired with a batch source containing the same feature values. A streaming source is only used to populate online stores. The batch equivalent source that is paired with a streaming source is used during the generation of historical feature datasets. Visit Sources to learn more about stream sources.

Here is a ride-hailing example of a valid feature table specification:

from feast import BigQuerySource, FeatureTable, Feature, ValueType
from google.protobuf.duration_pb2 import Duration

driver_ft = FeatureTable(
    name="driver_trips",
    entities=["driver_id"],
    features=[
      Feature("average_daily_rides", ValueType.FLOAT),
      Feature("rating", ValueType.FLOAT)
    ],
    max_age=Duration(seconds=3600),
    labels={
      "team": "driver_matching" 
    },
    batch_source=BigQuerySource(
        table_ref="gcp_project:bq_dataset.bq_table",
        event_timestamp_column="datetime",
        created_timestamp_column="timestamp",
        field_mapping={
          "rating": "driver_rating"
        }
    )
)

By default, Feast assumes that features specified in the feature-table specification corresponds one-to-one to the fields found in the sources. All features defined in a feature table should be available in the defined sources.

Field mappings can be used to map features defined in Feast to fields as they occur in data sources.

In the example feature-specification table above, we use field mappings to ensure the feature named rating in the batch source is mapped to the field named driver_rating.

Working with a Feature Table

Creating a Feature Table

driver_ft = FeatureTable(...)
client.apply(driver_ft)

Updating a Feature Table

driver_ft = FeatureTable()

client.apply(driver_ft)

driver_ft.labels = {"team": "marketplace"}

client.apply(driver_ft)

Feast currently supports the following changes to feature tables:

• Adding new features.

• Removing features.

• Updating source, max age, and labels.

Feast currently does not support the following changes to feature tables:

• Changes to the project or name of a feature table.

• Changes to entities related to a feature table.

• Changes to names and types of existing features.

Deleting a Feature Table

Overview

An entity is any domain object that can be modeled and about which information can be stored. Entities are usually recognizable concepts, either concrete or abstract, such as persons, places, things, or events.

Examples of entities in the context of ride-hailing and food delivery: customer, order, driver, restaurant, dish, area.

Entities are important in the context of feature stores since features are always properties of a specific entity. For example, we could have a feature total_trips_24h for driver D011234 with a feature value of 11.

Feast uses entities in the following way:

• Entities serve as the keys used to look up features for producing training datasets and online feature values.

• Entities serve as a natural grouping of features in a feature table. A feature table must belong to an entity (which could be a composite entity)

Structure of an Entity

When creating an entity specification, consider the following fields:

• Name: Name of the entity

• Description: Description of the entity

• Value Type: Value type of the entity. Feast will attempt to coerce entity columns in your data sources into this type.

• Labels: Labels are maps that allow users to attach their own metadata to entities

A valid entity specification is shown below: