This guide installs Feast on AWS using our reference Terraform configuration.
This Terraform configuration creates the following resources:
Kubernetes cluster on Amazon EKS (3x r3.large nodes)
Kafka managed by Amazon MSK (2x kafka.t3.small nodes)
Postgres database for Feast metadata, using serverless Aurora (min capacity: 2)
Redis cluster, using Amazon Elasticache (1x cache.t2.micro)
Amazon EMR cluster to run Spark (3x spot m4.xlarge)
Staging S3 bucket to store temporary data
Create an AWS account and configure credentials locally
Install Terraform > = 0.12 (tested with 0.13.3)
Install Helm (tested with v3.3.4)
.tfvars file under
feast/infra/terraform/aws. Name the file. In our example, we use
my_feast.tfvars. You can see the full list of configuration variables in
variables.tf. At a minimum, you need to set
name_prefix and an AWS region:
my_feast.tfvarsname_prefix = "my-feast"region = "us-east-1"
After completing the configuration, initialize Terraform and apply:
$ cd feast/infra/terraform/aws$ terraform init$ terraform apply -var-file=my_feast.tfvars
Starting may take a minute. A kubectl configuration file is also created in this directory, and the file's name will start with
kubeconfig_ and end with a random suffix.
After all pods are running, connect to the Jupyter Notebook Server running in the cluster.
To connect to the remote Feast server you just created, forward a port from the remote k8s cluster to your local machine. Replace
kubeconfig_XXXXXXX below with the kubeconfig file name Terraform generates for you.
KUBECONFIG=kubeconfig_XXXXXXX kubectl port-forward \$(kubectl get pod -o custom-columns=:metadata.name | grep jupyter) 8888:8888
Forwarding from 127.0.0.1:8888 -> 8888Forwarding from [::1]:8888 -> 8888
You can now connect to the bundled Jupyter Notebook Server at
localhost:8888 and follow the example Jupyter notebook.