LogoLogo
v0.49-branch
v0.49-branch
  • Introduction
  • Blog
  • Community & getting help
  • Roadmap
  • Changelog
  • Getting started
    • Quickstart
    • Architecture
      • Overview
      • Language
      • Push vs Pull Model
      • Write Patterns
      • Feature Transformation
      • Feature Serving and Model Inference
      • Role-Based Access Control (RBAC)
    • Concepts
      • Overview
      • Project
      • Data ingestion
      • Entity
      • Feature view
      • Feature retrieval
      • Point-in-time joins
      • [Alpha] Saved dataset
      • Permission
      • Tags
    • Use Cases
    • Components
      • Overview
      • Registry
      • Offline store
      • Online store
      • Feature server
      • Batch Materialization Engine
      • Provider
      • Authorization Manager
      • OpenTelemetry Integration
    • Third party integrations
    • FAQ
  • Tutorials
    • Sample use-case tutorials
      • Driver ranking
      • Fraud detection on GCP
      • Real-time credit scoring on AWS
      • Driver stats on Snowflake
    • Validating historical features with Great Expectations
    • Building streaming features
    • Retrieval Augmented Generation (RAG) with Feast
  • How-to Guides
    • Running Feast with Snowflake/GCP/AWS
      • Install Feast
      • Create a feature repository
      • Deploy a feature store
      • Build a training dataset
      • Load data into the online store
      • Read features from the online store
      • Scaling Feast
      • Structuring Feature Repos
    • Running Feast in production (e.g. on Kubernetes)
    • Customizing Feast
      • Adding a custom batch materialization engine
      • Adding a new offline store
      • Adding a new online store
      • Adding a custom provider
    • Adding or reusing tests
    • Starting Feast servers in TLS(SSL) Mode
  • Reference
    • Codebase Structure
    • Type System
    • Data sources
      • Overview
      • File
      • Snowflake
      • BigQuery
      • Redshift
      • Push
      • Kafka
      • Kinesis
      • Spark (contrib)
      • PostgreSQL (contrib)
      • Trino (contrib)
      • Azure Synapse + Azure SQL (contrib)
      • Couchbase (contrib)
    • Offline stores
      • Overview
      • Dask
      • Snowflake
      • BigQuery
      • Redshift
      • DuckDB
      • Couchbase Columnar (contrib)
      • Spark (contrib)
      • PostgreSQL (contrib)
      • Trino (contrib)
      • Azure Synapse + Azure SQL (contrib)
      • Clickhouse (contrib)
      • Remote Offline
    • Online stores
      • Overview
      • SQLite
      • Snowflake
      • Redis
      • Dragonfly
      • IKV
      • Datastore
      • DynamoDB
      • Bigtable
      • Remote
      • PostgreSQL
      • Cassandra + Astra DB
      • Couchbase
      • MySQL
      • Hazelcast
      • ScyllaDB
      • SingleStore
      • Milvus
    • Registries
      • Local
      • S3
      • GCS
      • SQL
      • Snowflake
    • Providers
      • Local
      • Google Cloud Platform
      • Amazon Web Services
      • Azure
    • Batch Materialization Engines
      • Snowflake
      • AWS Lambda (alpha)
      • Spark (contrib)
    • Feature repository
      • feature_store.yaml
      • .feastignore
    • Feature servers
      • Python feature server
      • [Alpha] Go feature server
      • Offline Feature Server
      • Registry server
    • [Beta] Web UI
    • [Beta] On demand feature view
    • [Alpha] Vector Database
    • [Alpha] Data quality monitoring
    • [Alpha] Streaming feature computation with Denormalized
    • Feast CLI reference
    • Python API reference
    • Usage
  • Project
    • Contribution process
    • Development guide
    • Backwards Compatibility Policy
      • Maintainer Docs
    • Versioning policy
    • Release process
    • Feast 0.9 vs Feast 0.10+
Powered by GitBook
On this page
  • Overview
  • 1. Automatically deploying changes to your feature definitions
  • 1.1 Setting up a feature repository
  • 1.2 Setting up a database-backed registry
  • 1.3 Setting up CI/CD to automatically update the registry
  • 1.4 Setting up multiple environments
  • 2. How to load data into your online store and keep it up to date
  • 2.1 Scalable Materialization
  • 2.2 Scheduled materialization with Airflow
  • 2.3 Stream feature ingestion
  • 2.4 Scheduled batch transformations with Airflow + dbt
  • 3. How to use Feast for model training
  • 3.1. Generating training data
  • 3.2 Versioning features that power ML models
  • 4. Retrieving online features for prediction
  • 4.1. Use the Python SDK within an existing Python service
  • 4.2. Deploy Feast feature servers on Kubernetes
  • 5. Using environment variables in your yaml configuration
  • Summary

Was this helpful?

Edit on GitHub
Export as PDF
  1. How-to Guides

Running Feast in production (e.g. on Kubernetes)

PreviousStructuring Feature ReposNextCustomizing Feast

Last updated 9 days ago

Was this helpful?

Overview

After learning about Feast concepts and playing with Feast locally, you're now ready to use Feast in production. This guide aims to help with the transition from a sandbox project to production-grade deployment in the cloud or on-premise (e.g. on Kubernetes).

A typical production architecture looks like:

Important note: Feast is highly customizable and modular.

Most Feast blocks are loosely connected and can be used independently. Hence, you are free to build your own production configuration.

For example, you might not have a stream source and, thus, no need to write features in real-time to an online store. Or you might not need to retrieve online features. Feast also often provides multiple options to achieve the same goal. We discuss tradeoffs below.

In this guide we will show you how to:

  1. Deploy your feature store and keep your infrastructure in sync with your feature repository

  2. Keep the data in your online store up to date (from batch and stream sources)

  3. Use Feast for model training and serving

1. Automatically deploying changes to your feature definitions

1.1 Setting up a feature repository

The first step to setting up a deployment of Feast is to create a Git repository that contains your feature definitions. The recommended way to version and track your feature definitions is by committing them to a repository and tracking changes through commits. If you recall, running feast apply commits feature definitions to a registry, which users can then read elsewhere.

1.2 Setting up a database-backed registry

Note: A SQL-based registry primarily works with a Python feature server. The Java feature server does not understand this registry type yet.

1.3 Setting up CI/CD to automatically update the registry

We recommend typically setting up CI/CD to automatically run feast plan and feast apply when pull requests are opened / merged.

1.4 Setting up multiple environments

A common scenario when using Feast in production is to want to test changes to Feast object definitions. For this, we recommend setting up a staging environment for your offline and online stores, which mirrors production (with potentially a smaller data set).

Having this separate environment allows users to test changes by first applying them to staging, and then promoting the changes to production after verifying the changes on staging.

2. How to load data into your online store and keep it up to date

To keep your online store up to date, you need to run a job that loads feature data from your feature view sources into your online store. In Feast, this loading operation is called materialization.

2.1 Scalable Materialization

Out of the box, Feast's materialization process uses an in-process materialization engine. This engine loads all the data being materialized into memory from the offline store, and writes it into the online store.

2.2 Scheduled materialization with Airflow

It is up to you to orchestrate and schedule runs of materialization.

However, the amount of work can quickly outgrow the resources of a single machine. That happens because the materialization job needs to repackage all rows before writing them to an online store. That leads to high utilization of CPU and memory. In this case, you might want to use a job orchestrator to run multiple jobs in parallel using several workers. Kubernetes Jobs or Airflow are good choices for more comprehensive job orchestration.

from airflow.decorators import task
from feast import RepoConfig, FeatureStore
from feast.infra.online_stores.dynamodb import DynamoDBOnlineStoreConfig
from feast.repo_config import RegistryConfig

# Define Python callable
@task()
def materialize(data_interval_start=None, data_interval_end=None):
  repo_config = RepoConfig(
    registry=RegistryConfig(path="s3://[YOUR BUCKET]/registry.pb"),
    project="feast_demo_aws",
    provider="aws",
    offline_store="file",
    online_store=DynamoDBOnlineStoreConfig(region="us-west-2"),
    entity_key_serialization_version=2
  )
  store = FeatureStore(config=repo_config)
  # Option 1: materialize just one feature view
  # store.materialize_incremental(datetime.datetime.now(), feature_views=["my_fv_name"])
  # Option 2: materialize all feature views incrementally
  # store.materialize_incremental(datetime.datetime.now())
  # Option 3: Let Airflow manage materialization state
  # Add 1 hr overlap to account for late data
  store.materialize(data_interval_start.subtract(hours=1), data_interval_end)

Important note: Airflow worker must have read and write permissions to the registry file on GCS / S3 since it pulls configuration and updates materialization history.

2.3 Stream feature ingestion

This supports pushing feature values into Feast to both online or offline stores.

2.4 Scheduled batch transformations with Airflow + dbt

3. How to use Feast for model training

3.1. Generating training data

After we've defined our features and data sources in the repository, we can generate training datasets. We highly recommend you use a FeatureService to version the features that go into a specific model version.

  1. The first thing we need to do in our training code is to create a FeatureStore object with a path to the registry.

    • One way to ensure your production clients have access to the feature store is to provide a copy of the feature_store.yaml to those pipelines. This feature_store.yaml file will have a reference to the feature store registry, which allows clients to retrieve features from offline or online stores.

      from feast import FeatureStore
      
      fs = FeatureStore(repo_path="production/")
  2. Then, you need to generate an entity dataframe. You have two options

    • Create an entity dataframe manually and pass it in

    • Use a SQL query to dynamically generate lists of entities (e.g. all entities within a time range) and timestamps to pass into Feast

  3. Then, training data can be retrieved as follows:

    training_retrieval_job = fs.get_historical_features(
        entity_df=entity_df_or_sql_string,
        features=fs.get_feature_service("driver_activity_v1"),
    )
    
    # Option 1: In memory model training
    model = ml.fit(training_retrieval_job.to_df())
    
    # Option 2: Unloading to blob storage. Further post-processing can occur before kicking off distributed training.
    training_retrieval_job.to_remote_storage()

3.2 Versioning features that power ML models

The most common way to productionize ML models is by storing and versioning models in a "model store", and then deploying these models into production. When using Feast, it is recommended that the feature service name and the model versions have some established convention.

For example, in MLflow:

import mlflow.pyfunc

# Load model from MLflow
model_name = "my-model"
model_version = 1
model = mlflow.pyfunc.load_model(
    model_uri=f"models:/{model_name}/{model_version}"
)

fs = FeatureStore(repo_path="production/")

# Read online features using the same model name and model version
feature_vector = fs.get_online_features(
    features=fs.get_feature_service(f"{model_name}_v{model_version}"),
    entity_rows=[{"driver_id": 1001}]
).to_dict()

# Make a prediction
prediction = model.predict(feature_vector)

It is important to note that both the training pipeline and model serving service need only read access to the feature registry and associated infrastructure. This prevents clients from accidentally making changes to the feature store.

4. Retrieving online features for prediction

Once you have successfully loaded data from batch / streaming sources into the online store, you can start consuming features for model inference.

4.1. Use the Python SDK within an existing Python service

This approach is the most convenient to keep your infrastructure as minimalistic as possible and avoid deploying extra services. The Feast Python SDK will connect directly to the online store (Redis, Datastore, etc), pull the feature data, and run transformations locally (if required). The obvious drawback is that your service must be written in Python to use the Feast Python SDK. A benefit of using a Python stack is that you can enjoy production-grade services with integrations with many existing data science tools.

To integrate online retrieval into your service use the following code:

from feast import FeatureStore

with open('feature_refs.json', 'r') as f:
    feature_refs = json.loads(f)

fs = FeatureStore(repo_path="production/")

# Read online features
feature_vector = fs.get_online_features(
    features=feature_refs,
    entity_rows=[{"driver_id": 1001}]
).to_dict()

4.2. Deploy Feast feature servers on Kubernetes

Basic steps

  1. Install the Operator

### Install the latest release -
$ kubectl apply -f https://raw.githubusercontent.com/feast-dev/feast/refs/heads/stable/infra/feast-operator/dist/install.yaml

### OR, install a specific version -
# kubectl apply -f https://raw.githubusercontent.com/feast-dev/feast/refs/tags/<version>/infra/feast-operator/dist/install.yaml
  1. Deploy a Feature Store

$ kubectl apply -f https://raw.githubusercontent.com/feast-dev/feast/refs/heads/stable/infra/feast-operator/config/samples/v1alpha1_featurestore.yaml

$ kubectl get feast
NAME     STATUS   AGE
sample   Ready    2m21s
apiVersion: feast.dev/v1alpha1
kind: FeatureStore
metadata:
  name: sample
spec:
  feastProject: my_project

More advanced FeatureStore CR examples can be found in the feast-operator [samples directory](../../infra/feast-operator/config/samples).

Important note: [Scaling a Feature Store Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#scaling-a-deployment) should only be done if the configured data store(s) will support it.

5. Using environment variables in your yaml configuration

You might want to dynamically set parts of your configuration from your environment. For instance to deploy Feast to production and development with the same configuration, but a different server. Or to inject secrets without exposing them in your git repo. To do this, it is possible to use the ${ENV_VAR} syntax in your feature_store.yaml file. For instance:

project: my_project
registry: data/registry.db
provider: local
online_store:
    type: redis
    connection_string: ${REDIS_CONNECTION_STRING}

Summary

In summary, the overall architecture in production may look like:

  • Feast SDK is being triggered by CI (eg, Github Actions). It applies the latest changes from the feature repo to the Feast database-backed registry

  • Data ingestion

    • Batch data: Airflow manages batch transformation jobs + materialization jobs to ingest batch data from DWH to the online store periodically. When working with large datasets to materialize, we recommend using a batch materialization engine

      • If your offline and online workloads are in Snowflake, the Snowflake materialization engine is likely the best option.

      • If your offline and online workloads are not using Snowflake, but using Kubernetes is an option, the Bytewax materialization engine is likely the best option.

      • If none of these engines suite your needs, you may continue using the in-process engine, or write a custom engine (e.g with Spark or Ray).

    • Stream data: The Feast Push API is used within existing Spark / Beam pipelines to push feature values to offline / online stores

  • Online features are served via the Python feature server over HTTP, or consumed using the Feast Python SDK.

  • Feast Python SDK is called locally to generate a training dataset

Additionally, please check the how-to guide for some specific recommendations on .

Out of the box, Feast serializes all of its state into a file-based registry. When running Feast in production, we recommend using the more scalable SQL-based registry that is backed by a database. Details are available .

Different options are presented in the .

This approach may not scale to large amounts of data, which users of Feast may be dealing with in production. In this case, we recommend using one of the more , such as . Users may also need to to work on their existing infrastructure.

See also for code snippets

Feast keeps the history of materialization in its registry so that the choice could be as simple as a . Cron util should be sufficient when you have just a few materialization jobs (it's usually one materialization job per feature view) triggered infrequently.

If you are using Airflow as a scheduler, Feast can be invoked through a after the has been installed into a virtual environment and your feature repo has been synced:

You can see more in an example at .

See more details at , which shows how to ingest streaming features or 3rd party feature data via a push API.

Feast does not orchestrate batch transformation DAGs. For this, you can rely on tools like Airflow + dbt. See for an example and some tips.

For more details, see

To deploy a Feast feature server on Kubernetes, you should use the included .

Install

The above will install a simple like the following. By default, it will run the -

For first-time Operator users, it may be a good exercise to try the . The quickstart will demonstrate some of the Operator's built-in features, e.g. git repos, feast apply jobs, etc.

Please check the how-to guide for some specific recommendations on .

how to scale Feast
how-to guide
unix cron util
PythonOperator
Python SDK
Feast Workshop - Module 1
data ingestion
Feast Workshop - Module 3
feast-operator
kubectl
FeatureStore CR
Online Store feature server
Feast Operator Quickstart
how to scale Feast
Snowflake Materialization Engine
write a custom materialization engine
data ingestion
here
scalable materialization engines
feature retrieval
Overview
From Repository to Production: Feast Production Architecture