The easiest way to create a new feature repository to use feast init command:

feast init

Creating a new Feast repository in /<...>/tiny_pika.

feast init -t gcp

Creating a new Feast repository in /<...>/tiny_pika.

The init command creates a Python file with feature definitions, sample data, and a Feast configuration file for local development:

$ tree
.
└── tiny_pika
    ├── data
    │   └── driver_stats.parquet
    ├── example.py
    └── feature_store.yaml

1 directory, 3 files

Enter the directory:

# Replace "tiny_pika" with your auto-generated dir name
cd tiny_pika

You can now use this feature repository for development. You can try the following:

• Run feast apply to apply these definitions to Feast.

• Edit the example feature definitions in example.py and run feast apply again to change feature definitions.

• Initialize a git repository in the same directory and checking the feature repository into version control.

Deploying

To have Feast deploy your infrastructure, run feast apply from your command line while inside a feature repository:

feast apply

# Processing example.py as example
# Done!

Depending on whether the feature repository is configured to use a local provider or one of the cloud providers like GCP or AWS, it may take from a couple of seconds to a minute to run to completion.

Cleaning up

If you need to clean up the infrastructure created by feast apply, use the teardown command.

feast teardown

****

In this tutorial we will

1. Deploy a local feature store with a Parquet file offline store and Sqlite online store.

2. Build a training dataset using our time series features from our Parquet files.

3. Materialize feature values from the offline store into the online store.

4. Read the latest features from the online store for inference.

Install Feast

Install the Feast SDK and CLI using pip:

pip install feast

Create a feature repository

Bootstrap a new feature repository using feast init from the command line:

feast init feature_repo
cd feature_repo

Creating a new Feast repository in /home/Jovyan/feature_repo.

Register feature definitions and deploy your feature store

The apply command registers all the objects in your feature repository and deploys a feature store:

feast apply

Registered entity driver_id
Registered feature view driver_hourly_stats
Deploying infrastructure for driver_hourly_stats

Generating training data

The apply command builds a training dataset based on the time-series features defined in the feature repository:

from datetime import datetime

import pandas as pd

from feast import FeatureStore

entity_df = pd.DataFrame.from_dict(
    {
        "driver_id": [1001, 1002, 1003, 1004],
        "event_timestamp": [
            datetime(2021, 4, 12, 10, 59, 42),
            datetime(2021, 4, 12, 8, 12, 10),
            datetime(2021, 4, 12, 16, 40, 26),
            datetime(2021, 4, 12, 15, 1, 12),
        ],
    }
)

store = FeatureStore(repo_path=".")

training_df = store.get_historical_features(
    entity_df=entity_df,
    feature_refs=[
        "driver_hourly_stats:conv_rate",
        "driver_hourly_stats:acc_rate",
        "driver_hourly_stats:avg_daily_trips",
    ],
).to_df()

print(training_df.head())

event_timestamp   driver_id  driver_hourly_stats__conv_rate  driver_hourly_stats__acc_rate  driver_hourly_stats__avg_daily_trips
2021-04-12        1002       0.328245                        0.993218                       329
2021-04-12        1001       0.448272                        0.873785                       767
2021-04-12        1004       0.822571                        0.571790                       673
2021-04-12        1003       0.556326                        0.605357                       335

Load features into your online store

The materialize command loads the latest feature values from your feature views into your online store:

CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")
feast materialize-incremental $CURRENT_TIME

Fetching feature vectors for inference

from pprint import pprint
from feast import FeatureStore

store = FeatureStore(repo_path=".")

feature_vector = store.get_online_features(
    feature_refs=[
        "driver_hourly_stats:conv_rate",
        "driver_hourly_stats:acc_rate",
        "driver_hourly_stats:avg_daily_trips",
    ],
    entity_rows=[{"driver_id": 1001}],
).to_dict()

pprint(feature_vector)

{
    'driver_id': [1001],
    'driver_hourly_stats__conv_rate': [0.49274],
    'driver_hourly_stats__acc_rate': [0.92743],
    'driver_hourly_stats__avg_daily_trips': [72],
}

Next steps

The Feast Python SDK allows users to retrieve feature values from an online store. This API is used to look up feature values at low latency during model serving in order to make online predictions.

Retrieving online features

1. Ensure that feature values have been loaded into the online store

Please ensure that you have materialized (loaded) your feature values into the online store before starting

2. Define feature references

Create a list of features that you would like to retrieve. This list typically comes from the model training step and should accompany the model binary.

3. Read online features

Next, we will create a feature store object and call get_online_features() which reads the relevant feature values directly from the online store.

pip install feast

Install Feast with GCP dependencies (required when using BigQuery or Firestore):

pip install 'feast[gcp]'

Feast allows users to build a training dataset from time-series feature data that already exists in an offline store. Users are expected to provide a list of features to retrieve (which may span multiple feature views), and a dataframe to join the resulting features onto. Feast will then execute a point-in-time join of multiple feature views onto the provided dataframe, and return the full resulting dataframe.

Retrieving historical features

1. Register your feature views

Please ensure that you have created a feature repository and that you have registered (applied) your feature views with Feast.

2. Define feature references

Start by defining the feature references (e.g., driver_trips:average_daily_rides) for the features that you would like to retrieve from the offline store. These features can come from multiple feature tables. The only requirement is that the feature tables that make up the feature references have the same entity (or composite entity), and that they aren't located in the same offline store.

feature_refs = [
    "driver_trips:average_daily_rides",
    "driver_trips:maximum_daily_rides",
    "driver_trips:rating",
    "driver_trips:rating:trip_completed",
]

3. Create an entity dataframe

An entity dataframe is the target dataframe on which you would like to join feature values. The entity dataframe must contain a timestamp column called event_timestamp and all entities (primary keys) necessary to join feature tables onto. All entities found in feature views that are being joined onto the entity dataframe must be found as column on the entity dataframe.

It is possible to provide entity dataframes as either a Pandas dataframe or a SQL query.

Pandas:

In the example below we create a Pandas based entity dataframe that has a single row with an event_timestamp column and a driver_id entity column. Pandas based entity dataframes may need to be uploaded into an offline store, which may result in longer wait times compared to a SQL based entity dataframe.

import pandas as pd
from datetime import datetime

entity_df = pd.DataFrame(
    {
        "event_timestamp": [pd.Timestamp(datetime.now(), tz="UTC")],
        "driver_id": [1001]
    }
)

SQL (Alternative):

Below is an example of an entity dataframe built from a BigQuery SQL query. It is only possible to use this query when all feature views being queried are available in the same offline store (BigQuery).

entity_df = "SELECT event_timestamp, driver_id FROM my_gcp_project.table"

4. Launch historical retrieval

from feast import FeatureStore

fs = FeatureStore(repo_path="path/to/your/feature/repo")

training_df = fs.get_historical_features(
    feature_refs=[
        "driver_hourly_stats:conv_rate",
        "driver_hourly_stats:acc_rate"
    ],
    entity_df=entity_df
).to_df()

Once the feature references and an entity dataframe are defined, it is possible to call get_historical_features(). This method launches a job that executes a point-in-time join of features from the offline store onto the entity dataframe. Once completed, a job reference will be returned. This job reference can then be converted to a Pandas dataframe by calling to_df().

Feast allows users to load their feature data into an online store in order to serve the latest features to models for online prediction.

Materializing features

1. Register feature views

Before proceeding, please ensure that you have applied (registered) the feature views that should be materialized.

2.a Materialize

The materialize command allows users to materialize features over a specific historical time range into the online store.

feast materialize 2021-04-07T00:00:00 2021-04-08T00:00:00

The above command will query the batch sources for all feature views over the provided time range, and load the latest feature values into the configured online store.

It is also possible to materialize for specific feature views by using the -v / --views argument.

feast materialize 2021-04-07T00:00:00 2021-04-08T00:00:00 \
--views driver_hourly_stats

The materialize command is completely stateless. It requires the user to provide the time ranges that will be loaded into the online store. This command is best used from a scheduler that tracks state, like Airflow.

2.b Materialize Incremental (Alternative)

For simplicity, Feast also provides a materialize command that will only ingest new data that has arrived in the offline store. Unlike materialize, materialize-incremental will track the state of previous ingestion runs inside of the feature registry.

The example command below will load only new data that has arrived for each feature view up to the end date and time (2021-04-08T00:00:00).

feast materialize-incremental 2021-04-08T00:00:00

The materialize-incremental command functions similarly to materialize in that it loads data over a specific time range for all feature views (or the selected feature views) into the online store.

Unlike materialize, materialize-incremental automatically determines the start time from which to load features from batch sources of each feature view. The first time materialize-incremental is executed it will set the start time to the oldest timestamp of each data source, and the end time as the one provided by the user. For each run of materialize-incremental, the end timestamp will be tracked.

Subsequent runs of materialize-incremental will then set the start time to the end time of the previous run, thus only loading new data that has arrived into the online store. Note that the end time that is tracked for each run is at the feature view level, not globally for all feature views, i.e, different feature views may have different periods that have been materialized into the online store.

Links & Resources

• • Design proposals in the form of Request for Comments (RFC).

  • User surveys and meeting minutes.

  • Slide decks of conferences our contributors have spoken at.

How can I get help?

• Slack: Need to speak to a human? Come ask a question in our Slack channel (link above).

Community Calls

We have a user and contributor community call every two weeks (Asia & US friendly).

Frequency (alternating times every 2 weeks)

• Tuesday 18:00 pm to 18:30 pm (US, Asia)

• Tuesday 10:00 am to 10:30 am (US, Europe)

Links

What is Feast?

Feast (Feature Store) is an operational data system for managing and serving machine learning features to models in production.

Problems Feast Solves

Models need consistent access to data: ML systems built on traditional data infrastructure are often coupled to databases, object stores, streams, and files. A result of this coupling, however, is that any change in data infrastructure may break dependent ML systems. Another challenge is that dual implementations of data retrieval for training and serving can lead to inconsistencies in data, which in turn can lead to training-serving skew.

Feast decouples your models from your data infrastructure by providing a single data access layer that abstracts feature storage from feature retrieval. Feast also provides a consistent means of referencing feature data for retrieval, and therefore ensures that models remain portable when moving from training to serving.

Deploying new features into production is difficult: Many ML teams consist of members with different objectives. Data scientists, for example, aim to deploy features into production as soon as possible, while engineers want to ensure that production systems remain stable. These differing objectives can create an organizational friction that slows time-to-market for new features.

Feast addresses this friction by providing both a centralized registry to which data scientists can publish features, and a battle-hardened serving layer. Together, these enable non-engineering teams to ship features into production with minimal oversight.

Models need point-in-time correct data: ML models in production require a view of data consistent with the one on which they are trained, otherwise the accuracy of these models could be compromised. Despite this need, many data science projects suffer from inconsistencies introduced by future feature values being leaked to models during training.

Feast solves the challenge of data leakage by providing point-in-time correct feature retrieval when exporting feature datasets for model training.

Features aren't reused across projects: Different teams within an organization are often unable to reuse features across projects. The siloed nature of development and the monolithic design of end-to-end ML systems contribute to duplication of feature creation and usage across teams and projects.

Feast addresses this problem by introducing feature reuse through a centralized system (a registry). This registry enables multiple teams working on different projects not only to contribute features, but also to reuse these same features. With Feast, data scientists can start new ML projects by selecting previously engineered features from a centralized registry, and are no longer required to develop new features for each project.

Problems Feast does not yet solve

Feature engineering: We aim for Feast to support light-weight feature engineering as part of our API.

Feature discovery: We also aim for Feast to include a first-class user interface for exploring and discovering entities and features.

‌Feature validation: We additionally aim for Feast to improve support for statistics generation of feature data and subsequent validation of these statistics. Current support is limited.

What Feast is not

Data warehouse: Feast is not a replacement for your data warehouse or the source of truth for all transformed data in your organization. Rather, Feast is a light-weight downstream layer that can serve data from an existing data warehouse (or other data sources) to models in production.

Data catalog: Feast is not a general purpose data catalog for your organization. Feast is purely focused on cataloging features for use in ML pipelines or systems, and only to the extent of facilitating the reuse of features.

How can I get started?

Explore the following resources to get started with Feast:

Project

Projects provide complete isolation of feature stores at the infrastructure level. This is accomplished through resource namespacing, e.g., prefixing table names with the associated project. Each project should be considered a completely separate universe of entities and features. It is not possible to retrieve features from multiple projects in a single request. We recommend having a single feature store and a single project per environment (dev, staging, prod).

Backlog

• Add On-demand transformations support

• Add Data quality monitoring

• Add Snowflake offline store support

• Add Bigtable support

• Add Push/Ingestion API support

Scheduled for development (next 3 months)

• Ensure Feast Serving is compatible with the new Feast

  • Decouple Feast Serving from Feast Core

  • Add FeatureView support to Feast Serving

  • Update Helm Charts (remove Core, Postgres, Job Service, Spark)

• Add Redis support for Feast

• Add direct deployment support to AWS and GCP

• Add Dynamo support

• Add Redshift support

Feast 0.10

New Functionality

1. Full local mode support (Sqlite and Parquet)

2. Provider model for added extensibility

3. Firestore support

4. Native (No-Spark) BigQuery support

5. Added support for object store based registry

6. Add support for FeatureViews

7. Added support for infrastructure configuration through apply

Technical debt, refactoring, or housekeeping

1. Remove dependency on Feast Core

2. Feast Serving made optional

3. Moved Python API documentation to Read The Docs

Feast 0.9

New Functionality

• Added Feast Job Service for management of ingestion and retrieval jobs

Note: Please see discussion thread above for functionality that did not make this release.

Feast 0.8

New Functionality

1. Add support for AWS (data sources and deployment)

2. Add support for local deployment

3. Add support for Spark based ingestion

4. Add support for Spark based historical retrieval

Technical debt, refactoring, or housekeeping

1. Move job management functionality to SDK

2. Remove Apache Beam based ingestion

3. Allow direct ingestion from batch sources that does not pass through stream

4. Remove Feast Historical Serving abstraction to allow direct access from Feast SDK to data sources for retrieval

Feast 0.7

New Functionality

Technical debt, refactoring, or housekeeping

Feast 0.6

New functionality

4. Improved searching and filtering of features and entities

Technical debt, refactoring, or housekeeping

Feast 0.5

New functionality

Technical debt, refactoring, or housekeeping

Feature View

Feature views are used during

• The generation of training datasets by querying the data source of feature views in order to find historical feature values. A single training dataset may consist of features from multiple feature views.

• Loading of feature values into an online store. Feature views determine the storage schema in the online store.

• Retrieval of features from the online store. Feature views provide the schema definition to Feast in order to look up features from the online store.

Data Source

Feast uses a time-series data model to represent data. This data model is used to interpret feature data in data sources in order to build training datasets or when materializing features into an online store.

Below is an example data source with a single entity (driver) and two features (trips_today, and rating).

Entity

An entity is a collection of semantically related features. Users define entities to map to the domain of their use case. For example, a ride-hailing service could have customers and drivers as their entities, which group related features that correspond to these customers and drivers.

Entities are defined as part of feature views. Entities are used to identify the primary key on which feature values should be stored and retrieved. These keys are used during the lookup of feature values from the online store and the join process in point-in-time joins. It is possible to define composite entities (more than one entity object) in a feature view.

Entities should be reused across feature views.

Feature

A feature is an individual measurable property observed on an entity. For example, a feature of a customer entity could be the number of transactions they have made on an average month.

Features are defined as part of feature views. Since Feast does not transform data, a feature is essentially a schema that only contains a name and a type:

Feast uses offline stores as storage and compute systems. Offline stores store historic time-series feature values. Feast does not generate these features, but instead uses the offline store as the interface for querying existing features in your organization.

Offline stores are used primarily for two reasons

1. Building training datasets from time-series features.

2. Materializing (loading) features from the offline store into an online store in order to serve those features at low latency for prediction.

It is not possible to query all data sources from all offline stores, and only a single offline store can be used at a time. For example, it is not possible to query a BigQuery table from a File offline store, nor is it possible for a BigQuery offline store to query files from your local file system.

Dataset

A dataset is a collection of rows that is produced by a historical retrieval from Feast in order to train a model. A dataset is produced by a join from one or more feature views onto an entity dataframe. Therefore, a dataset may consist of features from multiple feature views.

Dataset vs Feature View: Feature views contain the schema of data and a reference to where data can be found (through its data source). Datasets are the actual data manifestation of querying those data sources.

Dataset vs Data Source: Datasets are the output of historical retrieval, whereas data sources are the inputs. One or more data sources can be used in the creation of a dataset.

Feature References

Feature references uniquely identify feature values in Feast. The structure of a feature reference in string form is as follows: <feature_table>:<feature>

Feature references are used for the retrieval of features from Feast:

It is possible to retrieve features from multiple feature views with a single request, and Feast is able to join features from multiple tables in order to build a training dataset. However, It is not possible to reference (or retrieve) features from multiple projects at the same time.

Entity key

Entity keys are one or more entity values that uniquely describe an entity. In the case of an entity (like a driver) that only has a single entity field, the entity is an entity key. However, it is also possible for an entity key to consist of multiple entity values. For example, a feature view with the composite entity of (customer, country) might have an entity key of (1001, 5).

Entity keys act as primary keys. They are used during the lookup of features from the online store, and they are also used to match feature rows across feature views during point-in-time joins.

Event timestamp

The timestamp on which an event occurred, as found in a feature view's data source. The entity timestamp describes the event time at which a feature was observed or generated.

Event timestamps are used during point-in-time joins to ensure that the latest feature values are joined from feature views onto entity rows. Event timestamps are also used to ensure that old feature values aren't served to models during online serving.

Entity row

An entity key at a specific point in time.

Entity dataframe

A collection of entity rows. Entity dataframes are the "left table" that is enriched with feature values when building training datasets. The entity dataframe is provided to Feast by users during historical retrieval:

Example of an entity dataframe with feature values joined to it:

A provider is an implementation of a feature store using specific feature store components targeting a specific environment. More specifically, a provider is the target environment to which you have configured your feature store to deploy and run.

Providers also come with default configurations which makes it easier for users to start a feature store in a specific environment.

Functionality

• Create Batch Features: ELT/ETL systems like Spark and SQL are used to transform data in the batch store.

• Feast Apply: The user (or CI) publishes versioned controlled feature definitions using feast apply. This CLI command updates infrastructure and persists definitions in the object store registry.

• Feast Materialize: The user (or scheduler) executes feast materialize which loads features from the offline store into the online store.

• Model Training: A model training pipeline is launched. It uses the Feast Python SDK to retrieve a training dataset and trains a model.

• Get Historical Features: Feast exports a point-in-time correct training dataset based on the list of features and entity dataframe provided by the model training pipeline.

• Deploy Model: The trained model binary (and list of features) are deployed into a model serving system. This step is not executed by Feast.

• Prediction: A backend system makes a request for a prediction from the model serving service.

• Get Online Features: The model serving service makes a request to the Feast Online Serving service for online features using a Feast SDK.

Components

A complete Feast deployment contains the following components:

• Feast Online Serving: Provides low-latency access to feature values stores in the online store. This component is optional. Teams can also read feature values directly from the online store if necessary.

• Feast Registry: An object store (GCS, S3) based registry used to persist feature definitions that are registered with the feature store. Systems can discover feature data by interacting with the registry through the Feast SDK.

• Feast Python SDK/CLI: The primary user facing SDK. Used to:

  • Manage version controlled feature definitions.

  • Materialize (load) feature values into the online store.

  • Build and retrieve training datasets from the offline store.

  • Retrieve online features.

• Online Store: The online store is a database that stores only the latest feature values for each entity. The online store is populated by materialization jobs.

• Offline Store: The offline store persists batch data that has been ingested into Feast. This data is used for producing training datasets. Feast does not manage the offline store directly, but runs queries against it.

The Feast online store is used for low-latency online feature value lookups. Feature values are loaded into the online store from data sources in feature views using the materialize command.

The storage schema of features within the online store mirrors that of the data source used to populate the online store. One key difference between the online store and data sources is that only the latest feature values are stored per entity key. No historical values are stored.

Example batch data source

Once the above data source is materialized into Feast (using feast materialize), the feature values will be stored as follows:

Description

BigQuery data sources allow for the retrieval of historical feature values from BigQuery for building training datasets as well as materializing features into an online store.

• Either a table reference or a SQL query can be provided.

• No performance guarantees can be provided over SQL query-based sources. Please use table references where possible.

Examples

Using a table reference

from feast import BigQuerySource

my_bigquery_source = BigQuerySource(
    table_ref="gcp_project:bq_dataset.bq_table",
)

Using a query

from feast import BigQuerySource

BigQuerySource(
    query="SELECT timestamp as ts, created, f1, f2 "
          "FROM `my_project.my_dataset.my_features`",
)

Description

File data sources allow for the retrieval of historical feature values from files on disk for building training datasets, as well as for materializing features into an online store.

Example

from feast import FileSource
from feast.data_format import ParquetFormat

parquet_file_source = FileSource(
    file_format=ParquetFormat(),
    file_url="file:///feast/customer.parquet",
)

Description

• Only Parquet files are currently supported.

• All data is downloaded and joined using Python and may not scale to production workloads.

Example

project: my_feature_repo
registry: data/registry.db
provider: local
offline_store:
  type: file

Description

• BigQuery tables and views are allowed as sources.

• All joins happen within BigQuery.

• Entity dataframes can be provided as a SQL query or can be provided as a Pandas dataframe. Pandas dataframes will be uploaded to BigQuery in order to complete join operations.

Example

project: my_feature_repo
registry: gs://my-bucket/data/registry.db
provider: gcp
offline_store:
  type: bigquery
  dataset: feast_bq_dataset

Description

Example

project: my_feature_repo
registry: data/registry.db
provider: gcp
online_store:
  type: datastore
  project_id: my_gcp_project
  namespace: my_datastore_namespace

Description

• All feature values are stored in an on-disk SQLite database

• Only the latest feature values are persisted

Example

project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
  type: sqlite
  path: data/online_store.db

Description

• Offline Store: Uses the File offline store by default. Also supports BigQuery as the offline store.

• Online Store: Uses the Sqlite online store by default. Also supports Datastore as an online store.

Example

project: my_feature_repo
registry: data/registry.db
provider: local

Overview

project: loyal_spider
registry: data/registry.db
provider: local
online_store:
    type: sqlite
    path: data/online_store.db

Options

The following top-level configuration options exist in the feature_store.yaml file.

• provider — Configures the environment in which Feast will deploy and operate.

• registry — Configures the location of the feature registry.

• online_store — Configures the online store.

• offline_store — Configures the offline store.

• project — Defines a namespace for the entire feature store. Can be used to isolate multiple deployments in a single installation of Feast.

Description

• Offline Store: Uses the BigQuery offline store by default. Also supports File as the offline store.

• Online Store: Uses the Datastore online store by default. Also supports Sqlite as an online store.

Example

project: my_feature_repo
registry: gs://my-bucket/data/registry.db
provider: gcp

Permissions

Description

• Both Redis and Redis Cluster are supported

Examples

Connecting to a single Redis instance

Connecting to a Redis Cluster with SSL enabled and password authentication

Feast manages two important sets of configuration: feature definitions, and configuration about how to run the feature store. With Feast, this configuration can be written declaratively and stored as code in a central location. This central location is called a feature repository, and it's essentially just a directory that contains some code files.

The feature repository is the declarative source of truth for what the desired state of a feature store should be. The Feast CLI uses the feature repository to configure your infrastructure, e.g., migrate tables.

What is a feature repository?

A feature repository consists of:

• A collection of Python files containing feature declarations.

• A feature_store.yaml file containing infrastructural configuration.

• A .feastignore file containing paths in the feature repository to ignore.

Structure of a feature repository

The structure of a feature repository is as follows:

• The root of the repository should contain a feature_store.yaml file and may contain a .feastignore file.

• The repository should contain Python files that contain feature definitions.

• The repository can contain other files as well, including documentation and potentially data files.

An example structure of a feature repository is shown below:

$ tree -a
.
├── data
│   └── driver_stats.parquet
├── driver_features.py
├── feature_store.yaml
└── .feastignore

1 directory, 4 files

A couple of things to note about the feature repository:

• Feast reads all Python files recursively when feast apply is ran, including subdirectories, even if they don't contain feature definitions.

• It's recommended to add .feastignore and add paths to all imperative scripts if you need to store them inside the feature registry.

The feature_store.yaml configuration file

The configuration for a feature store is stored in a file named feature_store.yaml , which must be located at the root of a feature repository. An example feature_store.yaml file is shown below:

project: my_feature_repo_1
registry: data/metadata.db
provider: local
online_store:
    path: data/online_store.db

The .feastignore file

This file contains paths that should be ignored when running feast apply. An example .feastignore is shown below:

# Ignore virtual environment
venv

# Ignore a specific Python file
scripts/foo.py

# Ignore all Python files directly under scripts directory
scripts/*.py

# Ignore all "foo.py" anywhere under scripts directory
scripts/**/foo.py

Feature definitions

A feature repository can also contain one or more Python files that contain feature definitions. An example feature definition file is shown below:

from datetime import timedelta

from feast import BigQuerySource, Entity, Feature, FeatureView, ValueType

driver_locations_source = BigQuerySource(
    table_ref="rh_prod.ride_hailing_co.drivers",
    event_timestamp_column="event_timestamp",
    created_timestamp_column="created_timestamp",
)

driver = Entity(
    name="driver",
    value_type=ValueType.INT64,
    description="driver id",
)

driver_locations = FeatureView(
    name="driver_locations",
    entities=["driver"],
    ttl=timedelta(days=1),
    features=[
        Feature(name="lat", dtype=ValueType.FLOAT),
        Feature(name="lon", dtype=ValueType.STRING),
    ],
    input=driver_locations_source,
)

Next steps

Overview

Usage: feast [OPTIONS] COMMAND [ARGS]...

  Feast CLI

  For more information, see our public docs at https://docs.feast.dev/

  For any questions, you can reach us at https://slack.feast.dev/

Options:
  -c, --chdir TEXT  Switch to a different feature repository directory before
                    executing the given subcommand.

  --help            Show this message and exit.

Commands:
  apply                    Create or update a feature store deployment
  entities                 Access entities
  feature-views            Access feature views
  init                     Create a new Feast repository
  materialize              Run a (non-incremental) materialization job to...
  materialize-incremental  Run an incremental materialization job to ingest...
  registry-dump            Print contents of the metadata registry
  teardown                 Tear down deployed feature store infrastructure
  version                  Display Feast SDK version

Global Options

The Feast CLI provides one global top-level option that can be used with other commands

chdir (-c, --chdir)

This command allows users to run Feast CLI commands in a different folder from the current working directory.

feast -c path/to/my/feature/repo apply

Apply

Creates or updates a feature store deployment

feast apply

What does Feast apply do?

1. Feast will scan Python files in your feature repository and find all Feast object definitions, such as feature views, entities, and data sources.

2. Feast will validate your feature definitions

3. Feast will sync the metadata about Feast objects to the registry. If a registry does not exist, then it will be instantiated. The standard registry is a simple protobuf binary file that is stored on disk (locally or in an object store).

4. Feast CLI will create all necessary feature store infrastructure. The exact infrastructure that is deployed or configured depends on the provider configuration that you have set in feature_store.yaml. For example, setting local as your provider will result in a sqlite online store being created.

Entities

List all registered entities

feast entities list

NAME       DESCRIPTION    TYPE
driver_id  driver id      ValueType.INT64

Feature views

List all registered feature views

feast feature-views list

NAME                 ENTITIES
driver_hourly_stats  ['driver_id']

Init

Creates a new feature repository

feast init my_repo_name

Creating a new Feast repository in /projects/my_repo_name.

.
├── data
│   └── driver_stats.parquet
├── example.py
└── feature_store.yaml

It's also possible to use other templates

feast init -t gcp my_feature_repo

or to set the name of the new project

feast init -t gcp my_feature_repo

Materialize

Load data from feature views into the online store between two dates

feast materialize 2020-01-01T00:00:00 2022-01-01T00:00:00

Load data for specific feature views into the online store between two dates

feast materialize -v driver_hourly_stats 2020-01-01T00:00:00 2022-01-01T00:00:00

Materializing 1 feature views from 2020-01-01 to 2022-01-01

driver_hourly_stats:
100%|██████████████████████████| 5/5 [00:00<00:00, 5949.37it/s]

Materialize incremental

Load data from feature views into the online store, beginning from either the previous materialize or materialize-incremental end date, or the beginning of time.

feast materialize-incremental 2022-01-01T00:00:00

Teardown

Tear down deployed feature store infrastructure

feast teardown

Version

Print the current Feast version

feast version

Overview

# Ignore virtual environment
venv

# Ignore a specific Python file
scripts/foo.py

# Ignore all Python files directly under scripts directory
scripts/*.py

# Ignore all "foo.py" anywhere under scripts directory
scripts/**/foo.py

.feastignore file is optional. If the file can not be found, every Python in the feature repo directory will be parsed by feast apply.

Feast Ignore Patterns

How Feast SDK usage is measured

The Feast project logs anonymous usage statistics and errors in order to inform our planning. Several client methods are tracked, beginning in Feast 0.9. Users are assigned a UUID which is sent along with the name of the method, the Feast version, the OS (using sys.platform), and the current time.

How to disable usage logging

Set the environment variable FEAST_USAGE to False.

Install Feast

Connect to Feast

Learn Feast

Overview

This guide includes the following containerized components:

• • Feast Core with Postgres

  • Feast Online Serving with Redis.

  • Feast Job Service

• A Jupyter Notebook Server with built in Feast example(s). For demo purposes only.

• A Kafka cluster for testing streaming ingestion. For demo purposes only.

Get Feast

git clone https://github.com/feast-dev/feast.git
cd feast/infra/docker-compose

Create a new configuration file:

cp .env.sample .env

Start Feast

Start Feast with Docker Compose:

docker-compose pull && docker-compose up -d

Wait until all all containers are in a running state:

docker-compose ps

Try our example(s)

You can now connect to the bundled Jupyter Notebook Server running at localhost:8888 and follow the example Jupyter notebook.

Troubleshooting

Open ports

Please ensure that the following ports are available on your host machine:

• 6565

• 6566

• 8888

• 9094

• 5432

Containers are restarting or unavailable

If some of the containers continue to restart, or you are unable to access a service, inspect the logs using the following command:

docker-compose logs -f -t

Configuration

The Feast Docker Compose setup can be configured by modifying properties in your .env file.

Accessing Google Cloud Storage (GCP)

To access Google Cloud Storage as a data source, the Docker Compose installation requires access to a GCP service account.

• Grant the service account access to your bucket(s).

• Copy the service account to the path you have configured in .env under GCP_SERVICE_ACCOUNT.

• Restart your Docker Compose setup of Feast.

A production deployment of Feast is deployed using Kubernetes.

Kubernetes (with Helm)

This guide installs Feast into an existing Kubernetes cluster using Helm. The installation is not specific to any cloud platform or environment, but requires Kubernetes and Helm.

Amazon EKS (with Terraform)

This guide installs Feast into an AWS environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.

Azure AKS (with Helm)

This guide installs Feast into an Azure AKS environment with Helm.

Azure AKS (with Terraform)

This guide installs Feast into an Azure environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.

Google Cloud GKE (with Terraform)

This guide installs Feast into a Google Cloud environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.

IBM Cloud Kubernetes Service (IKS) and Red Hat OpenShift (using Kustomize)

Overview

This guide installs Feast on an existing Kubernetes cluster, and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Requirements

2. Preparation

Add the Feast Helm repository and download the latest charts:

helm repo add feast-charts https://feast-helm-charts.storage.googleapis.com
helm repo update

Feast includes a Helm chart that installs all necessary components to run Feast Core, Feast Online Serving, and an example Jupyter notebook.

Feast Core requires Postgres to run, which requires a secret to be set on Kubernetes:

kubectl create secret generic feast-postgresql --from-literal=postgresql-password=password

3. Installation

Install Feast using Helm. The pods may take a few minutes to initialize.

helm install feast-release feast-charts/feast

4. Use Jupyter to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

kubectl port-forward \
$(kubectl get pod -l app=feast-jupyter -o custom-columns=:metadata.name) 8888:8888

Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

5. Further Reading

Overview

This Terraform configuration creates the following resources:

• Kubernetes cluster on Amazon EKS (3x r3.large nodes)

• Kafka managed by Amazon MSK (2x kafka.t3.small nodes)

• Postgres database for Feast metadata, using serverless Aurora (min capacity: 2)

• Redis cluster, using Amazon Elasticache (1x cache.t2.micro)

• Amazon EMR cluster to run Spark (3x spot m4.xlarge)

• Staging S3 bucket to store temporary data

1. Requirements

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/aws. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. At a minimum, you need to set name_prefix and an AWS region:

3. Apply

After completing the configuration, initialize Terraform and apply:

Starting may take a minute. A kubectl configuration file is also created in this directory, and the file's name will start with kubeconfig_ and end with a random suffix.

4. Connect to Feast using Jupyter

After all pods are running, connect to the Jupyter Notebook Server running in the cluster.

To connect to the remote Feast server you just created, forward a port from the remote k8s cluster to your local machine. Replace kubeconfig_XXXXXXX below with the kubeconfig file name Terraform generates for you.

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

Overview

This guide installs Feast on Azure Kubernetes cluster (known as AKS), and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Spark

• Kafka

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Requirements

2. Preparation

az group create --name myResourceGroup  --location eastus
az acr create --resource-group myResourceGroup  --name feast-AKS-ACR --sku Basic
az aks create -g myResourceGroup  -n feast-AKS --location eastus --attach-acr feast-AKS-ACR --generate-ssh-keys

az aks install-cli
az aks get-credentials --resource-group myResourceGroup  --name  feast-AKS

Add the Feast Helm repository and download the latest charts:

helm version # make sure you have the latest Helm installed
helm repo add feast-charts https://feast-helm-charts.storage.googleapis.com
helm repo update

Feast includes a Helm chart that installs all necessary components to run Feast Core, Feast Online Serving, and an example Jupyter notebook.

Feast Core requires Postgres to run, which requires a secret to be set on Kubernetes:

kubectl create secret generic feast-postgresql --from-literal=postgresql-password=password

3. Feast installation

Install Feast using Helm. The pods may take a few minutes to initialize.

helm install feast-release feast-charts/feast

4. Spark operator installation

helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator 
helm install my-release spark-operator/spark-operator  --set serviceAccounts.spark.name=spark --set image.tag=v1beta2-1.1.2-2.4.5

and ensure the service account used by Feast has permissions to manage Spark Application resources. This depends on your k8s setup, but typically you'd need to configure a Role and a RoleBinding like the one below:

cat <<EOF | kubectl apply -f -
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: use-spark-operator
  namespace: <REPLACE ME>
rules:
- apiGroups: ["sparkoperator.k8s.io"]
  resources: ["sparkapplications"]
  verbs: ["create", "delete", "deletecollection", "get", "list", "update", "watch", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: use-spark-operator
  namespace: <REPLACE ME>
roleRef:
  kind: Role
  name: use-spark-operator
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: default
EOF

5. Use Jupyter to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

kubectl port-forward \
$(kubectl get pod -o custom-columns=:metadata.name | grep jupyter) 8888:8888

Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

6. Environment variables

demo_data_location = "wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/"
os.environ["FEAST_AZURE_BLOB_ACCOUNT_NAME"] = "<storage_account_name>"
os.environ["FEAST_AZURE_BLOB_ACCOUNT_ACCESS_KEY"] = <Insert your key here>
os.environ["FEAST_HISTORICAL_FEATURE_OUTPUT_LOCATION"] = "wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/out/"
os.environ["FEAST_SPARK_STAGING_LOCATION"] = "wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/artifacts/"
os.environ["FEAST_SPARK_LAUNCHER"] = "k8s"
os.environ["FEAST_SPARK_K8S_NAMESPACE"] = "default"
os.environ["FEAST_HISTORICAL_FEATURE_OUTPUT_FORMAT"] = "parquet"
os.environ["FEAST_REDIS_HOST"] = "feast-release-redis-master.default.svc.cluster.local"
os.environ["DEMO_KAFKA_BROKERS"] = "feast-release-kafka.default.svc.cluster.local:9092"

7. Further Reading

Overview

This Terraform configuration creates the following resources:

• GKE cluster

• Feast services running on GKE

• Google Memorystore (Redis) as online store

• Dataproc cluster

• Kafka running on GKE, exposed to the dataproc cluster via internal load balancer

1. Requirements

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/gcp. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. Sample configurations are provided below:

gcp_project_name        = "kf-feast"
name_prefix             = "feast-0-8"
region                  = "asia-east1"
gke_machine_type        = "n1-standard-2"
network                 = "default"
subnetwork              = "default"
dataproc_staging_bucket = "feast-dataproc"

3. Apply

After completing the configuration, initialize Terraform and apply:

$ cd feast/infra/terraform/gcp
$ terraform init
$ terraform apply -var-file=my_feast.tfvars

Overview

This Terraform configuration creates the following resources:

• Kubernetes cluster on Azure AKS

• Kafka managed by HDInsight

• Postgres database for Feast metadata, running as a pod on AKS

• Redis cluster, using Azure Cache for Redis

• Staging Azure blob storage container to store temporary data

1. Requirements

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/azure. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. At a minimum, you need to set name_prefix and resource_group:

name_prefix = "feast"
resource_group = "Feast" # pre-existing resource group

3. Apply

After completing the configuration, initialize Terraform and apply:

$ cd feast/infra/terraform/azure
$ terraform init
$ terraform apply -var-file=my_feast.tfvars

4. Connect to Feast using Jupyter

After all pods are running, connect to the Jupyter Notebook Server running in the cluster.

To connect to the remote Feast server you just created, forward a port from the remote k8s cluster to your local machine.

kubectl port-forward $(kubectl get pod -o custom-columns=:metadata.name | grep jupyter) 8888:8888

Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

Overview

This guide installs Feast on an existing IBM Cloud Kubernetes cluster or Red Hat OpenShift on IBM Cloud , and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Kafka (Optional)

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Prerequisites

2. Preparation

IBM Cloud Block Storage Setup (IKS only)

2. Add the IBM Cloud Helm chart repository to the cluster where you want to use the IBM Cloud Block Storage plug-in.
3. Install the IBM Cloud Block Storage plug-in. When you install the plug-in, pre-defined block storage classes are added to your cluster.

   Example output:
4. Verify that all block storage plugin pods are in a "Running" state.
5. Verify that the storage classes for Block Storage were added to your cluster.
6. Set the Block Storage as the default storageclass.

   Example output:

   Security Context Constraint Setup (OpenShift only)

3. Installation

Install Feast using kustomize. The pods may take a few minutes to initialize.

Optional: Enable Feast Jupyter and Kafka

You may optionally enable the Feast Jupyter component which contains code examples to demonstrate Feast. Some examples require Kafka to stream real time features to the Feast online serving. To enable, edit the following properties in the values.yaml under the manifests/contrib/feast folder:

Then regenerate the resource manifests and deploy:

4. Use Feast Jupyter Notebook Server to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

5. Uninstall Feast

6. Troubleshooting

When running the minimal_ride_hailing_example Jupyter Notebook example the following errors may occur:

1. When running job = client.get_historical_features(...):

   or

   Add the following environment variable:
2. When running job.get_status()

   Add the following environment variable:
3. When running job = client.start_stream_to_online_ingestion(...)

   Add the following environment variable:

pip install feast==0.9.*

Connect to an existing Feast Core deployment:

from feast import Client

# Connect to an existing Feast Core deployment
client = Client(core_url='feast.example.com:6565')

# Ensure that your client is connected by printing out some feature tables
client.list_feature_tables()

Feast Python SDK

The Feast Python SDK is used as a library to interact with a Feast deployment.

• Define, register, and manage entities and features

• Ingest data into Feast

• Build and retrieve training datasets

• Retrieve online features

Feast CLI

The Feast CLI is a command line implementation of the Feast Python SDK.

• Define, register, and manage entities and features from the terminal

• Ingest data into Feast

• Manage ingestion jobs

Online Serving Clients

The following clients can be used to retrieve online feature values:

Install the Feast CLI using pip:

pip install feast==0.9.*

Configure the CLI to connect to your Feast Core deployment:

feast config set core_url your.feast.deployment

$ feast

Usage: feast [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  config          View and edit Feast properties
  entities        Create and manage entities    
  feature-tables  Create and manage feature tables
  jobs            Create and manage jobs
  projects        Create and manage projects
  version         Displays version and connectivity information

Explore the following resources to learn more about Feast:

Concepts

Concept Hierarchy

Feast contains the following core concepts:

• Projects: Serve as a top level namespace for all Feast resources. Each project is a completely independent environment in Feast. Users can only work in a single project at a time.

• Entities: Entities are the objects in an organization on which features occur. They map to your business domain (users, products, transactions, locations).

• Feature Tables: Defines a group of features that occur on a specific entity.

• Features: Individual feature within a feature table.

Sequence description

1. Log Raw Events: Production backend applications are configured to emit internal state changes as events to a stream.

2. Create Stream Features: Stream processing systems like Flink, Spark, and Beam are used to transform and refine events and to produce features that are logged back to the stream.

3. Log Streaming Features: Both raw and refined events are logged into a data lake or batch storage location.

4. Create Batch Features: ELT/ETL systems like Spark and SQL are used to transform data in the batch store.

6. Poll Feature Definitions: The Feast Job Service polls for new or changed feature definitions.

7. Start Ingestion Jobs: Every new feature table definition results in a new ingestion job being provisioned (see limitations).

8. Batch Ingestion: Batch ingestion jobs are short-lived jobs that load data from batch sources into either an offline or online store (see limitations).

9. Stream Ingestion: Streaming ingestion jobs are long-lived jobs that load data from stream sources into online stores. A stream source and batch source on a feature table must have the same features/fields.

10. Model Training: A model training pipeline is launched. It uses the Feast Python SDK to retrieve a training dataset and trains a model.

11. Get Historical Features: Feast exports a point-in-time correct training dataset based on the list of features and entity DataFrame provided by the model training pipeline.

12. Deploy Model: The trained model binary (and list of features) are deployed into a model serving system.

13. Get Prediction: A backend system makes a request for a prediction from the model serving service.

14. Retrieve Online Features: The model serving service makes a request to the Feast Online Serving service for online features using a Feast SDK.

15. Return Prediction: The model serving service makes a prediction using the returned features and returns the outcome.

Components:

A complete Feast deployment contains the following components:

• Feast Core: Acts as the central registry for feature and entity definitions in Feast.

• Feast Job Service: Manages data processing jobs that load data from sources into stores, and jobs that export training datasets.

• Feast Serving: Provides low-latency access to feature values in an online store.

• Feast Python SDK CLI: The primary user facing SDK. Used to:

  • Manage feature definitions with Feast Core.

  • Launch jobs through the Feast Job Service.

  • Retrieve training datasets.

  • Retrieve online features.

• Online Store: The online store is a database that stores only the latest feature values for each entity. The online store can be populated by either batch ingestion jobs (in the case the user has no streaming source), or can be populated by a streaming ingestion job from a streaming source. Feast Online Serving looks up feature values from the online store.

• Offline Store: The offline store persists batch data that has been ingested into Feast. This data is used for producing training datasets.

• Feast Spark SDK: A Spark specific Feast SDK. Allows teams to use Spark for loading features into an online store and for building training datasets over offline sources.

Overview

An entity is any domain object that can be modeled and about which information can be stored. Entities are usually recognizable concepts, either concrete or abstract, such as persons, places, things, or events.

Examples of entities in the context of ride-hailing and food delivery: customer, order, driver, restaurant, dish, area.

Entities are important in the context of feature stores since features are always properties of a specific entity. For example, we could have a feature total_trips_24h for driver D011234 with a feature value of 11.

Feast uses entities in the following way:

• Entities serve as the keys used to look up features for producing training datasets and online feature values.

• Entities serve as a natural grouping of features in a feature table. A feature table must belong to an entity (which could be a composite entity)

Structure of an Entity

When creating an entity specification, consider the following fields:

• Name: Name of the entity

• Description: Description of the entity

• Value Type: Value type of the entity. Feast will attempt to coerce entity columns in your data sources into this type.

• Labels: Labels are maps that allow users to attach their own metadata to entities

A valid entity specification is shown below:

Working with an Entity

Creating an Entity:

Updating an Entity:

Permitted changes include:

• The entity's description and labels

The following changes are not permitted:

• Project

• Name of an entity

• Type

Overview

Currently, Feast supports the following source types:

Batch Source

• File (as in Spark): Parquet (only).

• BigQuery

Stream Source

• Kafka

• Kinesis

The following encodings are supported on streams

• Avro

• Protobuf

Structure of a Source

For both batch and stream sources, the following configurations are necessary:

Example data source specifications:

Working with a Source

Creating a Source

Feast ensures that the source complies with the schema of the feature table. These specified data sources can then be included inside a feature table specification and registered to Feast Core.

Overview

Feature tables serve the following purposes:

• Feature tables are used to create within Feast a database-level structure for the storage of feature values.

• The data sources described within feature tables allow Feast to find and ingest feature data into stores within Feast.

Features

A feature is an individual measurable property observed on an entity. For example the amount of transactions (feature) a customer (entity) has completed. Features are used for both model training and scoring (batch, online).

Features are defined as part of feature tables. Since Feast does not apply transformations, a feature is basically a schema that only contains a name and a type:

Structure of a Feature Table

Feature tables contain the following fields:

• Name: Name of feature table. This name must be unique within a project.

• Features: List of features within a feature table.

• Labels: Labels are arbitrary key-value properties that can be defined by users.

Here is a ride-hailing example of a valid feature table specification:

By default, Feast assumes that features specified in the feature-table specification corresponds one-to-one to the fields found in the sources. All features defined in a feature table should be available in the defined sources.

Field mappings can be used to map features defined in Feast to fields as they occur in data sources.

In the example feature-specification table above, we use field mappings to ensure the feature named rating in the batch source is mapped to the field named driver_rating.

Working with a Feature Table

Creating a Feature Table

Updating a Feature Table

Feast currently supports the following changes to feature tables:

• Adding new features.

• Removing features.

• Updating source, max age, and labels.

Feast currently does not support the following changes to feature tables:

• Changes to the project or name of a feature table.

• Changes to entities related to a feature table.

• Changes to names and types of existing features.

Deleting a Feature Table