Overview

This guide installs Feast on AWS using our reference Terraform configuration.

This Terraform configuration creates the following resources:

• Kubernetes cluster on Amazon EKS (3x r3.large nodes)

• Kafka managed by Amazon MSK (2x kafka.t3.small nodes)

• Postgres database for Feast metadata, using serverless Aurora (min capacity: 2)

• Redis cluster, using Amazon Elasticache (1x cache.t2.micro)

• Amazon EMR cluster to run Spark (3x spot m4.xlarge)

• Staging S3 bucket to store temporary data

1. Requirements

• Create an AWS account and configure credentials locally

• Install Terraform > = 0.12 (tested with 0.13.3)

• Install Helm (tested with v3.3.4)

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/aws. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. At a minimum, you need to set name_prefix and an AWS region:

name_prefix = "my-feast"
region      = "us-east-1"

3. Apply

After completing the configuration, initialize Terraform and apply:

$ cd feast/infra/terraform/aws
$ terraform init
$ terraform apply -var-file=my_feast.tfvars

Starting may take a minute. A kubectl configuration file is also created in this directory, and the file's name will start with kubeconfig_ and end with a random suffix.

4. Connect to Feast using Jupyter

After all pods are running, connect to the Jupyter Notebook Server running in the cluster.

To connect to the remote Feast server you just created, forward a port from the remote k8s cluster to your local machine. Replace kubeconfig_XXXXXXX below with the kubeconfig file name Terraform generates for you.

KUBECONFIG=kubeconfig_XXXXXXX kubectl port-forward \
$(kubectl get pod -o custom-columns=:metadata.name | grep jupyter) 8888:8888

Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

Overview

This guide installs Feast on an existing Kubernetes cluster, and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Requirements

1. Install and configure

2. Install

2. Preparation

Add the Feast Helm repository and download the latest charts:

Feast includes a Helm chart that installs all necessary components to run Feast Core, Feast Online Serving, and an example Jupyter notebook.

Feast Core requires Postgres to run, which requires a secret to be set on Kubernetes:

3. Installation

Install Feast using Helm. The pods may take a few minutes to initialize.

4. Use Jupyter to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

5. Further Reading

Overview

This guide installs Feast on Azure using our reference Terraform configuration.

This Terraform configuration creates the following resources:

• Kubernetes cluster on Azure AKS

• Kafka managed by HDInsight

• Postgres database for Feast metadata, running as a pod on AKS

• Redis cluster, using Azure Cache for Redis

• spark-on-k8s-operator to run Spark

• Staging Azure blob storage container to store temporary data

1. Requirements

• Create an Azure account and configure credentials locally

• Install Terraform (tested with 0.13.5)

• Install Helm (tested with v3.4.2)

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/azure. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. At a minimum, you need to set name_prefix and resource_group:

name_prefix = "feast"
resource_group = "Feast" # pre-existing resource group

3. Apply

After completing the configuration, initialize Terraform and apply:

$ cd feast/infra/terraform/azure
$ terraform init
$ terraform apply -var-file=my_feast.tfvars

4. Connect to Feast using Jupyter

After all pods are running, connect to the Jupyter Notebook Server running in the cluster.

To connect to the remote Feast server you just created, forward a port from the remote k8s cluster to your local machine.

kubectl port-forward $(kubectl get pod -o custom-columns=:metadata.name | grep jupyter) 8888:8888

Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

Overview

This guide installs Feast on GKE using our reference Terraform configuration.

This Terraform configuration creates the following resources:

• GKE cluster

• Feast services running on GKE

• Google Memorystore (Redis) as online store

• Dataproc cluster

• Kafka running on GKE, exposed to the dataproc cluster via internal load balancer

1. Requirements

• Install Terraform > = 0.12 (tested with 0.13.3)

• Install Helm (tested with v3.3.4)

• GCP authentication and sufficient privilege to create the resources listed above.

2. Configure Terraform

Create a .tfvars file underfeast/infra/terraform/gcp. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. Sample configurations are provided below:

gcp_project_name        = "kf-feast"
name_prefix             = "feast-0-8"
region                  = "asia-east1"
gke_machine_type        = "n1-standard-2"
network                 = "default"
subnetwork              = "default"
dataproc_staging_bucket = "feast-dataproc"

3. Apply

After completing the configuration, initialize Terraform and apply:

$ cd feast/infra/terraform/gcp
$ terraform init
$ terraform apply -var-file=my_feast.tfvars

Overview

This guide shows you how to deploy Feast using Docker Compose. Docker Compose allows you to explore the functionality provided by Feast while requiring only minimal infrastructure.

This guide includes the following containerized components:

• A complete Feast deployment

  • Feast Core with Postgres

  • Feast Online Serving with Redis.

  • Feast Job Service

• A Jupyter Notebook Server with built in Feast example(s). For demo purposes only.

• A Kafka cluster for testing streaming ingestion. For demo purposes only.

Get Feast

Clone the latest stable version of Feast from the Feast repository:

git clone https://github.com/feast-dev/feast.git
cd feast/infra/docker-compose

Create a new configuration file:

cp .env.sample .env

Start Feast

Start Feast with Docker Compose:

docker-compose pull && docker-compose up -d

Wait until all all containers are in a running state:

docker-compose ps

Try our example(s)

You can now connect to the bundled Jupyter Notebook Server running at localhost:8888 and follow the example Jupyter notebook.

Troubleshooting

Open ports

Please ensure that the following ports are available on your host machine:

• 6565

• 6566

• 8888

• 9094

• 5432

If a port conflict cannot be resolved, you can modify the port mappings in the provided docker-compose.yml file to use different ports on the host.

Containers are restarting or unavailable

If some of the containers continue to restart, or you are unable to access a service, inspect the logs using the following command:

docker-compose logs -f -t

If you are unable to resolve the problem, visit GitHub to create an issue.

Configuration

The Feast Docker Compose setup can be configured by modifying properties in your .env file.

Accessing Google Cloud Storage (GCP)

To access Google Cloud Storage as a data source, the Docker Compose installation requires access to a GCP service account.

• Create a new service account and save a JSON key.

• Grant the service account access to your bucket(s).

• Copy the service account to the path you have configured in .env under GCP_SERVICE_ACCOUNT.

• Restart your Docker Compose setup of Feast.

Overview

This guide installs Feast on Azure Kubernetes cluster (known as AKS), and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Spark

• Kafka

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Requirements

1. Install and configure Azure CLI

2. Install and configure Kubectl

3. Install Helm 3

2. Preparation

Create an AKS cluster with Azure CLI. The detailed steps can be found here, and a high-level walk through includes:

az group create --name myResourceGroup  --location eastus
az acr create --resource-group myResourceGroup  --name feast-AKS-ACR --sku Basic
az aks create -g myResourceGroup  -n feast-AKS --location eastus --attach-acr feast-AKS-ACR --generate-ssh-keys

az aks install-cli
az aks get-credentials --resource-group myResourceGroup  --name  feast-AKS

Add the Feast Helm repository and download the latest charts:

helm version # make sure you have the latest Helm installed
helm repo add feast-charts https://feast-helm-charts.storage.googleapis.com
helm repo update

Feast includes a Helm chart that installs all necessary components to run Feast Core, Feast Online Serving, and an example Jupyter notebook.

Feast Core requires Postgres to run, which requires a secret to be set on Kubernetes:

kubectl create secret generic feast-postgresql --from-literal=postgresql-password=password

3. Feast installation

Install Feast using Helm. The pods may take a few minutes to initialize.

helm install feast-release feast-charts/feast

4. Spark operator installation

Follow the documentation to install Spark operator on Kubernetes , and Feast documentation to configure Spark roles

helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator 
helm install my-release spark-operator/spark-operator  --set serviceAccounts.spark.name=spark --set image.tag=v1beta2-1.1.2-2.4.5

and ensure the service account used by Feast has permissions to manage Spark Application resources. This depends on your k8s setup, but typically you'd need to configure a Role and a RoleBinding like the one below:

cat <<EOF | kubectl apply -f -
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: use-spark-operator
  namespace: <REPLACE ME>
rules:
- apiGroups: ["sparkoperator.k8s.io"]
  resources: ["sparkapplications"]
  verbs: ["create", "delete", "deletecollection", "get", "list", "update", "watch", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: use-spark-operator
  namespace: <REPLACE ME>
roleRef:
  kind: Role
  name: use-spark-operator
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: default
EOF

5. Use Jupyter to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

kubectl port-forward \
$(kubectl get pod -o custom-columns=:metadata.name | grep jupyter) 8888:8888

Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

6. Environment variables

If you are running the Minimal Ride Hailing Example, you may want to make sure the following environment variables are correctly set:

demo_data_location = "wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/"
os.environ["FEAST_AZURE_BLOB_ACCOUNT_NAME"] = "<storage_account_name>"
os.environ["FEAST_AZURE_BLOB_ACCOUNT_ACCESS_KEY"] = <Insert your key here>
os.environ["FEAST_HISTORICAL_FEATURE_OUTPUT_LOCATION"] = "wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/out/"
os.environ["FEAST_SPARK_STAGING_LOCATION"] = "wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/artifacts/"
os.environ["FEAST_SPARK_LAUNCHER"] = "k8s"
os.environ["FEAST_SPARK_K8S_NAMESPACE"] = "default"
os.environ["FEAST_HISTORICAL_FEATURE_OUTPUT_FORMAT"] = "parquet"
os.environ["FEAST_REDIS_HOST"] = "feast-release-redis-master.default.svc.cluster.local"
os.environ["DEMO_KAFKA_BROKERS"] = "feast-release-kafka.default.svc.cluster.local:9092"

7. Further Reading

• Feast Concepts

• Feast Examples/Tutorials

• Feast Helm Chart Documentation

• Configuring Feast components

• Feast and Spark

A production deployment of Feast is deployed using Kubernetes.

Kubernetes (with Helm)

This guide installs Feast into an existing Kubernetes cluster using Helm. The installation is not specific to any cloud platform or environment, but requires Kubernetes and Helm.

Amazon EKS (with Terraform)

This guide installs Feast into an AWS environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.

Azure AKS (with Helm)

This guide installs Feast into an Azure AKS environment with Helm.

Azure AKS (with Terraform)

This guide installs Feast into an Azure environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.

Google Cloud GKE (with Terraform)

This guide installs Feast into a Google Cloud environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.

IBM Cloud Kubernetes Service (IKS) and Red Hat OpenShift (using Kustomize)

This guide installs Feast into an existing IBM Cloud Kubernetes Service or Red Hat OpenShift on IBM Cloud using Kustomize.

Overview

This guide installs Feast on an existing IBM Cloud Kubernetes cluster or Red Hat OpenShift on IBM Cloud , and ensures the following services are running:

• Feast Core

• Feast Online Serving

• Postgres

• Redis

• Kafka (Optional)

• Feast Jupyter (Optional)

• Prometheus (Optional)

1. Prerequisites

1. or

2. Install that matches the major.minor versions of your IKS or Install the that matches your local operating system and OpenShift cluster version.

3. Install

4. Install

2. Preparation

IBM Cloud Block Storage Setup (IKS only)

2. Add the IBM Cloud Helm chart repository to the cluster where you want to use the IBM Cloud Block Storage plug-in.
3. Install the IBM Cloud Block Storage plug-in. When you install the plug-in, pre-defined block storage classes are added to your cluster.

   Example output:
4. Verify that all block storage plugin pods are in a "Running" state.
5. Verify that the storage classes for Block Storage were added to your cluster.
6. Set the Block Storage as the default storageclass.

   Example output:

   Security Context Constraint Setup (OpenShift only)

3. Installation

Install Feast using kustomize. The pods may take a few minutes to initialize.

Optional: Enable Feast Jupyter and Kafka

You may optionally enable the Feast Jupyter component which contains code examples to demonstrate Feast. Some examples require Kafka to stream real time features to the Feast online serving. To enable, edit the following properties in the values.yaml under the manifests/contrib/feast folder:

Then regenerate the resource manifests and deploy:

4. Use Feast Jupyter Notebook Server to connect to Feast

After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:

You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.

5. Uninstall Feast

6. Troubleshooting

When running the minimal_ride_hailing_example Jupyter Notebook example the following errors may occur:

1. When running job = client.get_historical_features(...):

   or

   Add the following environment variable:
2. When running job.get_status()

   Add the following environment variable:
3. When running job = client.start_stream_to_online_ingestion(...)

   Add the following environment variable: