A production deployment of Feast is deployed using Kubernetes.
This guide installs Feast into an existing Kubernetes cluster using Helm. The installation is not specific to any cloud platform or environment, but requires Kubernetes and Helm.
This guide installs Feast into an AWS environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.
This guide installs Feast into an Azure AKS environment with Helm.
This guide installs Feast into an Azure environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.
This guide installs Feast into a Google Cloud environment using Terraform. The Terraform script is opinionated and intended to allow you to start quickly.
This guide installs Feast into an existing or using Kustomize.
This guide is meant for exploratory purposes only. It allows users to run Feast locally using Docker Compose instead of Kubernetes. The goal of this guide is for users to be able to quickly try out the full Feast stack without needing to deploy to Kubernetes. It is not meant for production use.
This guide shows you how to deploy Feast using Docker Compose. Docker Compose allows you to explore the functionality provided by Feast while requiring only minimal infrastructure.
This guide includes the following containerized components:
Feast Core with Postgres
Feast Online Serving with Redis.
Feast Job Service
A Jupyter Notebook Server with built in Feast example(s). For demo purposes only.
A Kafka cluster for testing streaming ingestion. For demo purposes only.
Clone the latest stable version of Feast from the :
Create a new configuration file:
Start Feast with Docker Compose:
Wait until all all containers are in a running state:
You can now connect to the bundled Jupyter Notebook Server running at localhost:8888 and follow the example Jupyter notebook.
Please ensure that the following ports are available on your host machine:
6565
6566
8888
If a port conflict cannot be resolved, you can modify the port mappings in the provided file to use different ports on the host.
If some of the containers continue to restart, or you are unable to access a service, inspect the logs using the following command:
If you are unable to resolve the problem, visit to create an issue.
The Feast Docker Compose setup can be configured by modifying properties in your .env file.
To access Google Cloud Storage as a data source, the Docker Compose installation requires access to a GCP service account.
Create a new and save a JSON key.
Grant the service account access to your bucket(s).
Copy the service account to the path you have configured in .env under GCP_SERVICE_ACCOUNT.
9094
5432
Restart your Docker Compose setup of Feast.
git clone https://github.com/feast-dev/feast.git
cd feast/infra/docker-composecp .env.sample .envdocker-compose pull && docker-compose up -ddocker-compose psdocker-compose logs -f -tThis guide installs Feast on Azure using our reference Terraform configuration.
This Terraform configuration creates the following resources:
Kubernetes cluster on Azure AKS
Kafka managed by HDInsight
Postgres database for Feast metadata, running as a pod on AKS
Redis cluster, using Azure Cache for Redis
to run Spark
Staging Azure blob storage container to store temporary data
Create an Azure account and
Install (tested with 0.13.5)
Install (tested with v3.4.2)
Create a .tfvars file underfeast/infra/terraform/azure. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. At a minimum, you need to set name_prefix and resource_group:
After completing the configuration, initialize Terraform and apply:
After all pods are running, connect to the Jupyter Notebook Server running in the cluster.
To connect to the remote Feast server you just created, forward a port from the remote k8s cluster to your local machine.
You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.
name_prefix = "feast"
resource_group = "Feast" # pre-existing resource group$ cd feast/infra/terraform/azure
$ terraform init
$ terraform apply -var-file=my_feast.tfvarskubectl port-forward $(kubectl get pod -o custom-columns=:metadata.name | grep jupyter) 8888:8888Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888This guide installs Feast on AWS using our reference Terraform configuration.
This Terraform configuration creates the following resources:
Kubernetes cluster on Amazon EKS (3x r3.large nodes)
Kafka managed by Amazon MSK (2x kafka.t3.small nodes)
Postgres database for Feast metadata, using serverless Aurora (min capacity: 2)
Redis cluster, using Amazon Elasticache (1x cache.t2.micro)
Amazon EMR cluster to run Spark (3x spot m4.xlarge)
Staging S3 bucket to store temporary data
Create an AWS account and
Install > = 0.12 (tested with 0.13.3)
Install (tested with v3.3.4)
Create a .tfvars file underfeast/infra/terraform/aws. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. At a minimum, you need to set name_prefix and an AWS region:
After completing the configuration, initialize Terraform and apply:
Starting may take a minute. A kubectl configuration file is also created in this directory, and the file's name will start with kubeconfig_ and end with a random suffix.
After all pods are running, connect to the Jupyter Notebook Server running in the cluster.
To connect to the remote Feast server you just created, forward a port from the remote k8s cluster to your local machine. Replace kubeconfig_XXXXXXX below with the kubeconfig file name Terraform generates for you.
You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.
This guide installs Feast on an existing Kubernetes cluster, and ensures the following services are running:
Feast Core
Feast Online Serving
Postgres
Redis
Feast Jupyter (Optional)
Prometheus (Optional)
Install and configure
Install
Add the Feast Helm repository and download the latest charts:
Feast includes a Helm chart that installs all necessary components to run Feast Core, Feast Online Serving, and an example Jupyter notebook.
Feast Core requires Postgres to run, which requires a secret to be set on Kubernetes:
Install Feast using Helm. The pods may take a few minutes to initialize.
After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:
You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.
helm repo add feast-charts https://feast-helm-charts.storage.googleapis.com
helm repo updatekubectl create secret generic feast-postgresql --from-literal=postgresql-password=passwordhelm install feast-release feast-charts/feastkubectl port-forward \
$(kubectl get pod -l app=feast-jupyter -o custom-columns=:metadata.name) 8888:8888Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888name_prefix = "my-feast"
region = "us-east-1"$ cd feast/infra/terraform/aws
$ terraform init
$ terraform apply -var-file=my_feast.tfvarsKUBECONFIG=kubeconfig_XXXXXXX kubectl port-forward \
$(kubectl get pod -o custom-columns=:metadata.name | grep jupyter) 8888:8888Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888This guide installs Feast on an existing IBM Cloud Kubernetes cluster or Red Hat OpenShift on IBM Cloud , and ensures the following services are running:
Feast Core
Feast Online Serving
Postgres
Redis
Kafka (Optional)
Feast Jupyter (Optional)
Prometheus (Optional)
or
Install that matches the major.minor versions of your IKS or Install the that matches your local operating system and OpenShift cluster version.
Install
:warning: If you have Red Hat OpenShift Cluster on IBM Cloud skip to this .
By default, IBM Cloud Kubernetes cluster uses based on NFS as the default storage class, and non-root users do not have write permission on the volume mount path for NFS-backed storage. Some common container images in Feast, such as Redis, Postgres, and Kafka specify a non-root user to access the mount path in the images. When containers are deployed using these images, the containers fail to start due to insufficient permissions of the non-root user creating folders on the mount path.
allows for the creation of raw storage volumes and provides faster performance without the permission restriction of NFS-backed storage
Therefore, to deploy Feast we need to set up as the default storage class so that you can have all the functionalities working and get the best experience from Feast.
to install the Helm version 3 client on your local machine.
Add the IBM Cloud Helm chart repository to the cluster where you want to use the IBM Cloud Block Storage plug-in.
Install the IBM Cloud Block Storage plug-in. When you install the plug-in, pre-defined block storage classes are added to your cluster.
Example output:
By default, in OpenShift, all pods or containers will use the which limits the UIDs pods can run with, causing the Feast installation to fail. To overcome this, you can allow Feast pods to run with any UID by executing the following:
Install Feast using kustomize. The pods may take a few minutes to initialize.
You may optionally enable the Feast Jupyter component which contains code examples to demonstrate Feast. Some examples require Kafka to stream real time features to the Feast online serving. To enable, edit the following properties in the values.yaml under the manifests/contrib/feast folder:
Then regenerate the resource manifests and deploy:
After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:
You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.
When running the minimal_ride_hailing_example Jupyter Notebook example the following errors may occur:
When running job = client.get_historical_features(...):
or
Add the following environment variable:
When running job.get_status()
Add the following environment variable:
This guide installs Feast on GKE using our .
This Terraform configuration creates the following resources:
GKE cluster
Feast services running on GKE
Verify that all block storage plugin pods are in a "Running" state.
Verify that the storage classes for Block Storage were added to your cluster.
Set the Block Storage as the default storageclass.
Example output:
Security Context Constraint Setup (OpenShift only)
When running job = client.start_stream_to_online_ingestion(...)
Add the following environment variable:
helm repo add iks-charts https://icr.io/helm/iks-charts
helm repo update helm install v2.0.2 iks-charts/ibmcloud-block-storage-plugin -n kube-systemNAME: v2.0.2
LAST DEPLOYED: Fri Feb 5 12:29:50 2021
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing: ibmcloud-block-storage-plugin. Your release is named: v2.0.2
...oc adm policy add-scc-to-user anyuid -z default,kf-feast-kafka -n feastgit clone https://github.com/kubeflow/manifests
cd manifests/contrib/feast/
kustomize build feast/base | kubectl apply -n feast -f -kafka.enabled: true
feast-jupyter.enabled: truemake feast/base
kustomize build feast/base | kubectl apply -n feast -f -kubectl port-forward \
$(kubectl get pod -l app=feast-jupyter -o custom-columns=:metadata.name) 8888:8888 -n feastForwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888kustomize build feast/base | kubectl delete -n feast -f - KeyError: 'historical_feature_output_location' KeyError: 'spark_staging_location' os.environ["FEAST_HISTORICAL_FEATURE_OUTPUT_LOCATION"] = "file:///home/jovyan/historical_feature_output"
os.environ["FEAST_SPARK_STAGING_LOCATION"] = "file:///home/jovyan/test_data" <SparkJobStatus.FAILED: 2> os.environ["FEAST_REDIS_HOST"] = "feast-release-redis-master"Dataproc cluster
Kafka running on GKE, exposed to the dataproc cluster via internal load balancer
Install > = 0.12 (tested with 0.13.3)
Install (tested with v3.3.4)
GCP and sufficient to create the resources listed above.
Create a .tfvars file underfeast/infra/terraform/gcp. Name the file. In our example, we use my_feast.tfvars. You can see the full list of configuration variables in variables.tf. Sample configurations are provided below:
After completing the configuration, initialize Terraform and apply:
gcp_project_name = "kf-feast"
name_prefix = "feast-0-8"
region = "asia-east1"
gke_machine_type = "n1-standard-2"
network = "default"
subnetwork = "default"
dataproc_staging_bucket = "feast-dataproc"$ cd feast/infra/terraform/gcp
$ terraform init
$ terraform apply -var-file=my_feast.tfvars kubectl get pods -n kube-system | grep ibmcloud-block-storage kubectl get storageclasses | grep ibmc-block kubectl patch storageclass ibmc-block-gold -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kubectl patch storageclass ibmc-file-gold -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
# Check the default storageclass is block storage
kubectl get storageclass | grep \(default\) ibmc-block-gold (default) ibm.io/ibmc-block 65s org.apache.kafka.vendor.common.KafkaException: Failed to construct kafka consumer os.environ["DEMO_KAFKA_BROKERS"] = "feast-release-kafka:9092"This guide installs Feast on Azure Kubernetes cluster (known as AKS), and ensures the following services are running:
Feast Core
Feast Online Serving
Postgres
Redis
Spark
Kafka
Feast Jupyter (Optional)
Prometheus (Optional)
Install and configure
Install and configure
Install
Create an AKS cluster with Azure CLI. The detailed steps can be found , and a high-level walk through includes:
Add the Feast Helm repository and download the latest charts:
Feast includes a Helm chart that installs all necessary components to run Feast Core, Feast Online Serving, and an example Jupyter notebook.
Feast Core requires Postgres to run, which requires a secret to be set on Kubernetes:
Install Feast using Helm. The pods may take a few minutes to initialize.
Follow the documentation , and Feast documentation to
and ensure the service account used by Feast has permissions to manage Spark Application resources. This depends on your k8s setup, but typically you'd need to configure a Role and a RoleBinding like the one below:
After all the pods are in a RUNNING state, port-forward to the Jupyter Notebook Server in the cluster:
You can now connect to the bundled Jupyter Notebook Server at localhost:8888 and follow the example Jupyter notebook.
If you are running the , you may want to make sure the following environment variables are correctly set:
az group create --name myResourceGroup --location eastus
az acr create --resource-group myResourceGroup --name feast-AKS-ACR --sku Basic
az aks create -g myResourceGroup -n feast-AKS --location eastus --attach-acr feast-AKS-ACR --generate-ssh-keys
az aks install-cli
az aks get-credentials --resource-group myResourceGroup --name feast-AKShelm version # make sure you have the latest Helm installed
helm repo add feast-charts https://feast-helm-charts.storage.googleapis.com
helm repo updatekubectl create secret generic feast-postgresql --from-literal=postgresql-password=passwordhelm install feast-release feast-charts/feasthelm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator
helm install my-release spark-operator/spark-operator --set serviceAccounts.spark.name=spark --set image.tag=v1beta2-1.1.2-2.4.5cat <<EOF | kubectl apply -f -
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: use-spark-operator
namespace: <REPLACE ME>
rules:
- apiGroups: ["sparkoperator.k8s.io"]
resources: ["sparkapplications"]
verbs: ["create", "delete", "deletecollection", "get", "list", "update", "watch", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: use-spark-operator
namespace: <REPLACE ME>
roleRef:
kind: Role
name: use-spark-operator
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: default
EOFkubectl port-forward \
$(kubectl get pod -o custom-columns=:metadata.name | grep jupyter) 8888:8888Forwarding from 127.0.0.1:8888 -> 8888
Forwarding from [::1]:8888 -> 8888demo_data_location = "wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/"
os.environ["FEAST_AZURE_BLOB_ACCOUNT_NAME"] = "<storage_account_name>"
os.environ["FEAST_AZURE_BLOB_ACCOUNT_ACCESS_KEY"] = <Insert your key here>
os.environ["FEAST_HISTORICAL_FEATURE_OUTPUT_LOCATION"] = "wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/out/"
os.environ["FEAST_SPARK_STAGING_LOCATION"] = "wasbs://<container_name>@<storage_account_name>.blob.core.windows.net/artifacts/"
os.environ["FEAST_SPARK_LAUNCHER"] = "k8s"
os.environ["FEAST_SPARK_K8S_NAMESPACE"] = "default"
os.environ["FEAST_HISTORICAL_FEATURE_OUTPUT_FORMAT"] = "parquet"
os.environ["FEAST_REDIS_HOST"] = "feast-release-redis-master.default.svc.cluster.local"
os.environ["DEMO_KAFKA_BROKERS"] = "feast-release-kafka.default.svc.cluster.local:9092"