Install Feast

If you would like to deploy a new installation of Feast, click on Install Feast

Connect to Feast

If you would like to connect to an existing Feast deployment, click on Connect to Feast

Learn Feast

If you would like to learn more about Feast, click on Learn Feast

Overview

This guide installs Feast on GKE using our reference Terraform configuration.

This Terraform configuration creates the following resources:

• GKE cluster

• Feast services running on GKE

• Google Memorystore (Redis) as online store

• Dataproc cluster

• Kafka running on GKE, exposed to the dataproc cluster via internal load balancer

1. Requirements

• Install Terraform > = 0.12 (tested with 0.13.3)

• Install Helm (tested with v3.3.4)

• GCP authentication and sufficient privilege to create the resources listed above.

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/gcp. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. Sample configurations are provided below:

gcp_project_name        = "kf-feast"
name_prefix             = "feast-0-8"
region                  = "asia-east1"
gke_machine_type        = "n1-standard-2"
network                 = "default"
subnetwork              = "default"
dataproc_staging_bucket = "feast-dataproc"

3. Apply

After completing the configuration, initialize Terraform and apply:

$ cd feast/infra/terraform/gcp
$ terraform init
$ terraform apply -var-file=my_feast.tfvars

Overview

This guide installs Feast on an existing Kubernetes cluster, and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Requirements

1. Install and configure Kubectl

2. Install Helm 3

2. Preparation

Add the Feast Helm repository and download the latest charts:

helm repo add feast-charts https://feast-helm-charts.storage.googleapis.com
helm repo update

Feast includes a Helm chart that installs all necessary components to run Feast Core, Feast Online Serving, and an example Jupyter notebook.

Feast Core requires Postgres to run, which requires a secret to be set on Kubernetes:

kubectl create secret generic feast-postgresql --from-literal=postgresql-password=password

3. Installation

Install Feast using Helm. The pods may take a few minutes to initialize.

helm install feast-release feast-charts/feast

4. Use Jupyter to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

kubectl port-forward \
$(kubectl get pod -l app=feast-jupyter -o custom-columns=:metadata.name) 8888:8888

Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

5. Further Reading

• Feast Concepts

• Feast Examples/Tutorials

• Feast Helm Chart Documentation

• Configuring Feast components

• Feast and Spark

Overview

This guide installs Feast on an existing IBM Cloud Kubernetes cluster or Red Hat OpenShift on IBM Cloud , and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Kafka (Optional)

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Prerequisites

1. IBM Cloud Kubernetes Service or Red Hat OpenShift on IBM Cloud

2. Install Kubectl that matches the major.minor versions of your IKS or Install the OpenShift CLI that matches your local operating system and OpenShift cluster version.

3. Install Helm 3

4. Install Kustomize

2. Preparation

IBM Cloud Block Storage Setup (IKS only)

:warning: If you have Red Hat OpenShift Cluster on IBM Cloud skip to this section.

By default, IBM Cloud Kubernetes cluster uses IBM Cloud File Storage based on NFS as the default storage class, and non-root users do not have write permission on the volume mount path for NFS-backed storage. Some common container images in Feast, such as Redis, Postgres, and Kafka specify a non-root user to access the mount path in the images. When containers are deployed using these images, the containers fail to start due to insufficient permissions of the non-root user creating folders on the mount path.

IBM Cloud Block Storage allows for the creation of raw storage volumes and provides faster performance without the permission restriction of NFS-backed storage

Therefore, to deploy Feast we need to set up IBM Cloud Block Storage as the default storage class so that you can have all the functionalities working and get the best experience from Feast.

1. Follow the instructions to install the Helm version 3 client on your local machine.

2. Add the IBM Cloud Helm chart repository to the cluster where you want to use the IBM Cloud Block Storage plug-in.

   Copy

    helm repo add iks-charts https://icr.io/helm/iks-charts
    helm repo update

3. Install the IBM Cloud Block Storage plug-in. When you install the plug-in, pre-defined block storage classes are added to your cluster.

   Copy

    helm install v2.0.2 iks-charts/ibmcloud-block-storage-plugin -n kube-system

   Example output:

   Copy

   NAME: v2.0.2
   LAST DEPLOYED: Fri Feb  5 12:29:50 2021
   NAMESPACE: kube-system
   STATUS: deployed
   REVISION: 1
   NOTES:
   Thank you for installing: ibmcloud-block-storage-plugin.   Your release is named: v2.0.2
    ...

4. Verify that all block storage plugin pods are in a "Running" state.

   Copy

    kubectl get pods -n kube-system | grep ibmcloud-block-storage

5. Verify that the storage classes for Block Storage were added to your cluster.

   Copy

    kubectl get storageclasses | grep ibmc-block

6. Set the Block Storage as the default storageclass.

   Copy

    kubectl patch storageclass ibmc-block-gold -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
    kubectl patch storageclass ibmc-file-gold -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
   
    # Check the default storageclass is block storage
    kubectl get storageclass | grep \(default\)

   Example output:

   Copy

    ibmc-block-gold (default)   ibm.io/ibmc-block   65s

   Security Context Constraint Setup (OpenShift only)

By default, in OpenShift, all pods or containers will use the Restricted SCC which limits the UIDs pods can run with, causing the Feast installation to fail. To overcome this, you can allow Feast pods to run with any UID by executing the following:

oc adm policy add-scc-to-user anyuid -z default,kf-feast-kafka -n feast

3. Installation

Install Feast using kustomize. The pods may take a few minutes to initialize.

git clone https://github.com/kubeflow/manifests
cd manifests/contrib/feast/
kustomize build feast/base | kubectl apply -n feast -f -

Optional: Enable Feast Jupyter and Kafka

You may optionally enable the Feast Jupyter component which contains code examples to demonstrate Feast. Some examples require Kafka to stream real time features to the Feast online serving. To enable, edit the following properties in the values.yaml under the manifests/contrib/feast folder:

kafka.enabled: true
feast-jupyter.enabled: true

Then regenerate the resource manifests and deploy:

make feast/base
kustomize build feast/base | kubectl apply -n feast -f -

4. Use Feast Jupyter Notebook Server to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

kubectl port-forward \
$(kubectl get pod -l app=feast-jupyter -o custom-columns=:metadata.name) 8888:8888 -n feast

Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

5. Uninstall Feast

kustomize build feast/base | kubectl delete -n feast -f -

6. Troubleshooting

When running the minimal_ride_hailing_example Jupyter Notebook example the following errors may occur:

1. When running job = client.get_historical_features(...):

   Copy

    KeyError: 'historical_feature_output_location'

   or

   Copy

    KeyError: 'spark_staging_location'

   Add the following environment variable:

   Copy

    os.environ["FEAST_HISTORICAL_FEATURE_OUTPUT_LOCATION"] = "file:///home/jovyan/historical_feature_output"
    os.environ["FEAST_SPARK_STAGING_LOCATION"] = "file:///home/jovyan/test_data"

2. When running job.get_status()

   Copy

    <SparkJobStatus.FAILED: 2>

   Add the following environment variable:

   Copy

    os.environ["FEAST_REDIS_HOST"] = "feast-release-redis-master"

3. When running job = client.start_stream_to_online_ingestion(...)

   Copy

    org.apache.kafka.vendor.common.KafkaException: Failed to construct kafka consumer

   Add the following environment variable:

   Copy

    os.environ["DEMO_KAFKA_BROKERS"] = "feast-release-kafka:9092"

Overview

This guide installs Feast on AWS using our reference Terraform configuration.

This Terraform configuration creates the following resources:

• Kubernetes cluster on Amazon EKS (3x r3.large nodes)

• Kafka managed by Amazon MSK (2x kafka.t3.small nodes)

• Postgres database for Feast metadata, using serverless Aurora (min capacity: 2)

• Redis cluster, using Amazon Elasticache (1x cache.t2.micro)

• Amazon EMR cluster to run Spark (3x spot m4.xlarge)

• Staging S3 bucket to store temporary data

1. Requirements

• Create an AWS account and configure credentials locally

• Install Terraform > = 0.12 (tested with 0.13.3)

• Install Helm (tested with v3.3.4)

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/aws. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. At a minimum, you need to set name_prefix and an AWS region:

name_prefix = "my-feast"
region      = "us-east-1"

3. Apply

After completing the configuration, initialize Terraform and apply:

$ cd feast/infra/terraform/aws
$ terraform init
$ terraform apply -var-file=my_feast.tfvars

Starting may take a minute. A kubectl configuration file is also created in this directory, and the file's name will start with kubeconfig_ and end with a random suffix.

4. Connect to Feast using Jupyter

After all pods are running, connect to the Jupyter Notebook Server running in the cluster.

To connect to the remote Feast server you just created, forward a port from the remote k8s cluster to your local machine. Replace kubeconfig_XXXXXXX below with the kubeconfig file name Terraform generates for you.

KUBECONFIG=kubeconfig_XXXXXXX kubectl port-forward \
$(kubectl get pod -o custom-columns=:metadata.name | grep jupyter) 8888:8888

Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

A production deployment of Feast is deployed using Kubernetes.

Kubernetes (with Helm)

This guide installs Feast into an existing Kubernetes cluster using Helm. The installation is not specific to any cloud platform or environment, but requires Kubernetes and Helm.

Amazon EKS (with Terraform)

This guide installs Feast into an AWS environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.

Azure AKS (with Helm)

This guide installs Feast into an Azure AKS environment with Helm.

Azure AKS (with Terraform)

This guide installs Feast into an Azure environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.

Google Cloud GKE (with Terraform)

This guide installs Feast into a Google Cloud environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.

IBM Cloud Kubernetes Service (IKS) and Red Hat OpenShift (using Kustomize)

This guide installs Feast into an existing IBM Cloud Kubernetes Service or Red Hat OpenShift on IBM Cloud using Kustomize.

Feast Python SDK

The Feast Python SDK is used as a library to interact with a Feast deployment.

• Define, register, and manage entities and features

• Ingest data into Feast

• Build and retrieve training datasets

• Retrieve online features

Feast CLI

The Feast CLI is a command line implementation of the Feast Python SDK.

• Define, register, and manage entities and features from the terminal

• Ingest data into Feast

• Manage ingestion jobs

Online Serving Clients

The following clients can be used to retrieve online feature values:

• Feast Python SDK

• Feast Go SDK

• Feast Java SDK

Overview

This guide shows you how to deploy Feast using . Docker Compose allows you to explore the functionality provided by Feast while requiring only minimal infrastructure.

This guide includes the following containerized components:

• • Feast Core with Postgres

  • Feast Online Serving with Redis.

  • Feast Job Service

• A Jupyter Notebook Server with built in Feast example(s). For demo purposes only.

• A Kafka cluster for testing streaming ingestion. For demo purposes only.

Get Feast

Clone the latest stable version of Feast from the :

Create a new configuration file:

Start Feast

Start Feast with Docker Compose:

Wait until all all containers are in a running state:

Try our example(s)

You can now connect to the bundled Jupyter Notebook Server running at localhost:8888 and follow the example Jupyter notebook.

Troubleshooting

Open ports

Please ensure that the following ports are available on your host machine:

• 6565

• 6566

• 8888

• 9094

• 5432

Containers are restarting or unavailable

If some of the containers continue to restart, or you are unable to access a service, inspect the logs using the following command:

Configuration

The Feast Docker Compose setup can be configured by modifying properties in your .env file.

Accessing Google Cloud Storage (GCP)

To access Google Cloud Storage as a data source, the Docker Compose installation requires access to a GCP service account.

• Grant the service account access to your bucket(s).

• Copy the service account to the path you have configured in .env under GCP_SERVICE_ACCOUNT.

• Restart your Docker Compose setup of Feast.

Explore the following resources to learn more about Feast:

• describes all important Feast API concepts.

• provides guidance on completing Feast workflows.

• contains Jupyter notebooks that you can run on your Feast deployment.

• contains information about both advanced and operational aspects of Feast.

• contains detailed API and design documents for advanced users.

• contains resources for anyone who wants to contribute to Feast.

Overview

This guide installs Feast on Azure Kubernetes cluster (known as AKS), and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Spark

• Kafka

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Requirements

1. Install and configure

2. Install and configure

3. Install

2. Preparation

Add the Feast Helm repository and download the latest charts:

Feast includes a Helm chart that installs all necessary components to run Feast Core, Feast Online Serving, and an example Jupyter notebook.

Feast Core requires Postgres to run, which requires a secret to be set on Kubernetes:

3. Feast installation

Install Feast using Helm. The pods may take a few minutes to initialize.

4. Spark operator installation

and ensure the service account used by Feast has permissions to manage Spark Application resources. This depends on your k8s setup, but typically you'd need to configure a Role and a RoleBinding like the one below:

5. Use Jupyter to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

6. Environment variables

7. Further Reading

Overview

This guide installs Feast on Azure using our .

This Terraform configuration creates the following resources:

• Kubernetes cluster on Azure AKS

• Kafka managed by HDInsight

• Postgres database for Feast metadata, running as a pod on AKS

• Redis cluster, using Azure Cache for Redis

• to run Spark

• Staging Azure blob storage container to store temporary data

1. Requirements

• Create an Azure account and

• Install (tested with 0.13.5)

• Install (tested with v3.4.2)

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/azure. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. At a minimum, you need to set name_prefix and resource_group:

3. Apply

After completing the configuration, initialize Terraform and apply:

4. Connect to Feast using Jupyter

After all pods are running, connect to the Jupyter Notebook Server running in the cluster.

To connect to the remote Feast server you just created, forward a port from the remote k8s cluster to your local machine.

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

Install the Feast CLI using pip:

pip install feast==0.9.*

Configure the CLI to connect to your Feast Core deployment:

feast config set core_url your.feast.deployment

The CLI is a wrapper around the Feast Python SDK:

$ feast

Usage: feast [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  config          View and edit Feast properties
  entities        Create and manage entities    
  feature-tables  Create and manage feature tables
  jobs            Create and manage jobs
  projects        Create and manage projects
  version         Displays version and connectivity information

Install the Feast Python SDK using pip:

pip install feast==0.9.*

Connect to an existing Feast Core deployment:

from feast import Client

# Connect to an existing Feast Core deployment
client = Client(core_url='feast.example.com:6565')

# Ensure that your client is connected by printing out some feature tables
client.list_feature_tables()