This reference describes how to configure Feast components:
Available configuration properties for Feast Core and Feast Online Serving can be referenced from the corresponding application.yml of each component:
Configuration properties for Feast Core and Feast Online Serving are defined depending on Feast is deployed:
Docker Compose deployment - Feast is deployed with Docker Compose.
Kubernetes deployment - Feast is deployed with Kubernetes.
Direct Configuration - Feast is built and run from source code.
For each Feast component deployed using Docker Compose, configuration properties from application.yml can be set at:
The Kubernetes Feast Deployment is configured using values.yaml in the Helm chart included with Feast:
A reference of the sub-chart-specific configuration can found in its values.yml:
Configuration properties can be set via application-override.yaml for each component in values.yaml:
Visit the Helm chart included with Feast to learn more about configuration.
If Feast is built and running from source, configuration properties can be set directly in the Feast component's application.yml:
Configuration options for both the Feast CLI and Feast Python SDK can be defined in the following locations, in order of precedence:
1. Command line arguments or initialized arguments: Passing parameters to the Feast CLI or instantiating the Feast Client object with specific parameters will take precedence above other parameters.
2. Environmental variables: Environmental variables can be set to provide configuration options. They must be prefixed with FEAST_. For example FEAST_CORE_URL.
3. Configuration file: Options with the lowest precedence are configured in the Feast configuration file. Feast looks for or creates this configuration file in ~/.feast/config if it does not already exist. All options must be defined in the [general] section of this file.
Visit the available configuration parameters for Feast Python SDK and Feast CLI to learn more.
The Feast Java SDK and Feast Go SDK are configured via arguments passed when instantiating the respective Clients:
Visit the Feast Go SDK API reference to learn more about available configuration parameters.
Visit the Feast Java SDK API reference to learn more about available configuration parameters.
Component
Configuration Reference
Core
Serving (Online)
Component
Configuration Path
Core
infra/docker-compose/core/core.yml
Online Serving
infra/docker-compose/serving/online-serving.yml
Component
Configuration Path
Core
Serving (Online)
Please see the following API specific reference documentation:
Feast Core gRPC API: This is the gRPC API used by Feast Core. This API contains RPCs for creating and managing feature sets, stores, projects, and jobs.
Feast Serving gRPC API: This is the gRPC API used by Feast Serving. It contains RPCs used for the retrieval of online feature data or historical feature data.
Feast gRPC Types: These are the gRPC types used by both Feast Core, Feast Serving, and the Go, Java, and Python clients.
Go Client SDK: The Go library used for the retrieval of online features from Feast.
Java Client SDK: The Java library used for the retrieval of online features from Feast.
Python SDK: This is the complete reference to the Feast Python SDK. The SDK is used to manage feature sets, features, jobs, projects, and entities. It can also be used to retrieve training datasets or online features from Feast Serving.
The following community provided SDKs are available:
Node.js SDK: A Node.js SDK written in TypeScript. The SDK can be used to manage feature sets, features, jobs, projects, and entities.
This page applies to Feast 0.7. The content may be out of date for Feast 0.8+
Reference of the metrics that each Feast component exports:
For how to configure Feast to export Metrics, see the Metrics user guide.
Exported Metrics
Feast Core exports the following metrics:
Metric Tags
Exported Feast Core metrics may be filtered by the following tags/keys
Exported Metrics
Feast Serving exports the following metrics:
Metric Tags
Exported Feast Serving metrics may be filtered by the following tags/keys
Feast Ingestion computes both metrics an statistics on data ingestion. Make sure you familar with data ingestion concepts before proceeding.
Metrics Namespace
Metrics are computed at two stages of the Feature Row's/Feature Value's life cycle when being processed by the Ingestion Job:
Inflight- Prior to writing data to stores, but after successful validation of data.
WriteToStoreSucess- After a successful store write.
Metrics processed by each staged will be tagged with metrics_namespace to the stage where the metric was computed.
Metrics Bucketing
Metrics with a {BUCKET} are computed on a 60 second window/bucket. Suffix with the following to select the bucket to use:
min - minimum value.
max - maximum value.
mean- mean value.
percentile_90- 90 percentile.
percentile_95- 95 percentile.
percentile_99- 99 percentile.
Exported Metrics
Metric Tags
Exported Feast Ingestion Job metrics may be filtered by the following tags/keys
Configuring Feast to use Spark for ingestion.
Feast relies on Spark to ingest data from the offline store to the online store, streaming ingestion, and running queries to retrieve historical data from the offline store. Feast supports several Spark deployment options.
To install the Spark on K8s Operator
Currently Feast is tested using v1beta2-1.1.2-2.4.5version of the operator image. To configure Feast to use it, set the following options in Feast config:
Lastly, make sure that the service account used by Feast has permissions to manage Spark Application resources. This depends on your k8s setup, but typically you'd need to configure a Role and a RoleBinding like the one below:
If you're running Feast in Google Cloud, you can use Dataproc, a managed Spark platform. To configure Feast to use it, set the following options in Feast config:
See Feast documentation for more configuration options for Dataproc.
If you're running Feast in AWS, you can use EMR, a managed Spark platform. To configure Feast to use it, set at least the following options in Feast config:
See Feast documentation for more configuration options for EMR.
Limitation
Motivation
Features names and entity names cannot overlap in feature table definitions
Features and entities become columns in historical stores which may cause conflicts
The following field names are reserved in feature tables
event_timestamp
datetime
created_timestamp
ingestion_id
job_id
These keywords are used for column names when persisting metadata in historical stores
Limitation
Motivation
Once data has been ingested into Feast, there is currently no way to delete the data without manually going to the database and deleting it. However, during retrieval only the latest rows will be returned for a specific key (event_timestamp, entity) based on its created_timestamp.
This functionality simply doesn't exist yet as a Feast API
Limitation
Motivation
Feast does not support offline storage in Feast 0.8
As part of our re-architecture of Feast, we moved from GCP to cloud-agnostic deployments. Developing offline storage support that is available in all cloud environments is a pending action.
Metrics
Description
Tags
feast_core_request_latency_seconds
Feast Core's latency in serving Requests in Seconds.
service, method, status_code
feast_core_feature_set_total
No. of Feature Sets registered with Feast Core.
None
feast_core_store_total
No. of Stores registered with Feast Core.
None
feast_core_max_memory_bytes
Max amount of memory the Java virtual machine will attempt to use.
None
feast_core_total_memory_bytes
Total amount of memory in the Java virtual machine
None
feast_core_free_memory_bytes
Total amount of free memory in the Java virtual machine.
None
feast_core_gc_collection_seconds
Time spent in a given JVM garbage collector in seconds.
None
Tag
Description
service
Name of the Service that request is made to. Should be set to CoreService
method
Name of the Method that the request is calling. (ie ListFeatureSets)
status_code
Status code returned as a result of handling the requests (ie OK). Can be used to find request failures.
Metric
Description
Tags
feast_serving_request_latency_seconds
Feast Serving's latency in serving Requests in Seconds.
method
feast_serving_request_feature_count
No. of requests retrieving a Feature from Feast Serving.
project, feature_name
feast_serving_not_found_feature_count
No. of requests retrieving a Feature has resulted in a NOT_FOUND field status.
project, feature_name
feast_serving_stale_feature_count
No. of requests retrieving a Feature resulted in a OUTSIDE_MAX_AGE field status.
project, feature_name
feast_serving_grpc_request_count
Total gRPC requests served.
method
Tag
Description
method
Name of the Method that the request is calling. (ie ListFeatureSets)
status_code
Status code returned as a result of handling the requests (ie OK). Can be used to find request failures.
project
Name of the project that the FeatureSet of the Feature retrieved belongs to.
feature_name
Name of the Feature being retrieved.
Metric
Description
Tags
feast_ingestion_feature_row_lag_ms_{BUCKET}
Lag time in milliseconds between succeeding ingested Feature Rows.
feast_store, feast_project_name,feast_featureSet_name,ingestion_job_name,
metrics_namespace
feast_ingestion_feature_value_lag_ms_{BUCKET}
Lag time in milliseconds between succeeding ingested values for each Feature.
feast_store, feast_project_name,feast_featureSet_name,
feast_feature_name,
ingestion_job_name,
metrics_namespace
feast_ingestion_feature_value_{BUCKET}
Last value feature for each Feature.
feast_store, feature_project_name, feast_feature_name,feast_featureSet_name, ingest_job_name, metrics_namepace
feast_ingestion_feature_row_ingested_count
No. of Ingested Feature Rows
feast_store, feast_project_name,feast_featureSet_name,ingestion_job_name,
metrics_namespace
feast_ingestion_feature_value_missing_count
No. of times a ingested Feature values did not provide a value for the Feature.
feast_store, feast_project_name,feast_featureSet_name,
feast_feature_name,
ingestion_job_name,
metrics_namespace
feast_ingestion_deadletter_row_count
No. of Feature Rows that that the Ingestion Job did not successfully write to store.
feast_store, feast_project_name,feast_featureSet_name,ingestion_job_name
Tag
Description
feast_store
Name of the target store the Ingestion Job is writing to.
feast_project_name
Name of the project that the ingested FeatureSet belongs to.
feast_featureSet_name
Name of the Feature Set being ingested.
feast_feature_name
Name of the Feature being ingested.
ingestion_job_name
Name of the Ingestion Job performing data ingestion. Typically this is set to the Id of the Ingestion Job.
metrics_namespace
Stage where metrics where computed. Either Inflight or WriteToStoreSuccess
Feast Setting
Value
SPARK_LAUNCHER
"k8s"
SPARK_STAGING_LOCATION
S3/GCS/Azure Blob Storage URL to use as a staging location, must be readable and writable by Feast. For S3, use s3a:// prefix here. Ex.: s3a://some-bucket/some-prefix/artifacts/
HISTORICAL_FEATURE_OUTPUT_LOCATION
S3/GCS/Azure Blob Storage URL used to store results of historical retrieval queries, must be readable and writable by Feast. For S3, use s3a:// prefix here. Ex.: s3a://some-bucket/some-prefix/out/
SPARK_K8S_NAMESPACE
Only needs to be set if you are customizing the spark-on-k8s-operator. The name of the Kubernetes namespace to run Spark jobs in. This should match the value of sparkJobNamespace set on spark-on-k8s-operator Helm chart. Typically this is also the namespace Feast itself will run in.
SPARK_K8S_JOB_TEMPLATE_PATH
Only needs to be set if you are customizing the Spark job template. Local file path with the template of the SparkApplication resource. No prefix required. Ex.: /home/jovyan/work/sparkapp-template.yaml. An example template is here and the spec is defined in the k8s-operator User Guide.
Feast Setting
Value
SPARK_LAUNCHER
"dataproc"
DATAPROC_CLUSTER_NAME
Dataproc cluster name
DATAPROC_PROJECT
Dataproc project name
SPARK_STAGING_LOCATION
GCS URL to use as a staging location, must be readable and writable by Feast. Ex.: gs://some-bucket/some-prefix
Feast Setting
Value
SPARK_LAUNCHER
"emr"
SPARK_STAGING_LOCATION
S3 URL to use as a staging location, must be readable and writable by Feast. Ex.: s3://some-bucket/some-prefix