Overview

Feature tables are both a schema and a logical means of grouping features, data sources, and other related metadata.

Feature tables serve the following purposes:

• Feature tables are a means for defining the location and properties of data sources.

• Feature tables are used to create within Feast a database-level structure for the storage of feature values.

• The data sources described within feature tables allow Feast to find and ingest feature data into stores within Feast.

• Feature tables ensure data is efficiently stored during ingestion by providing a grouping mechanism of features values that occur on the same event timestamp.

Features

A feature is an individual measurable property observed on an entity. For example the amount of transactions (feature) a customer (entity) has completed. Features are used for both model training and scoring (batch, online).

Features are defined as part of feature tables. Since Feast does not apply transformations, a feature is basically a schema that only contains a name and a type:

avg_daily_ride = Feature("average_daily_rides", ValueType.FLOAT)

Visit FeatureSpec for the complete feature specification API.

Structure of a Feature Table

Feature tables contain the following fields:

• Name: Name of feature table. This name must be unique within a project.

• Entities: List of entities to associate with the features defined in this feature table. Entities are used as lookup keys when retrieving features from a feature table.

• Features: List of features within a feature table.

• Labels: Labels are arbitrary key-value properties that can be defined by users.

• Max age: Max age affect the retrieval of features from a feature table. Age is measured as the duration of time between the event timestamp of a feature and the lookup time on an entity key used to retrieve the feature. Feature values outside max age will be returned as unset values. Max age allows for eviction of keys from online stores and limits the amount of historical scanning required for historical feature values during retrieval.

• Batch Source: The batch data source from which Feast will ingest feature values into stores. This can either be used to back-fill stores before switching over to a streaming source, or it can be used as the primary source of data for a feature table. Visit Sources to learn more about batch sources.

• Stream Source: The streaming data source from which you can ingest streaming feature values into Feast. Streaming sources must be paired with a batch source containing the same feature values. A streaming source is only used to populate online stores. The batch equivalent source that is paired with a streaming source is used during the generation of historical feature datasets. Visit Sources to learn more about stream sources.

Here is a ride-hailing example of a valid feature table specification:

from feast import BigQuerySource, FeatureTable, Feature, ValueType
from google.protobuf.duration_pb2 import Duration

driver_ft = FeatureTable(
    name="driver_trips",
    entities=["driver_id"],
    features=[
      Feature("average_daily_rides", ValueType.FLOAT),
      Feature("rating", ValueType.FLOAT)
    ],
    max_age=Duration(seconds=3600),
    labels={
      "team": "driver_matching" 
    },
    batch_source=BigQuerySource(
        table_ref="gcp_project:bq_dataset.bq_table",
        event_timestamp_column="datetime",
        created_timestamp_column="timestamp",
        field_mapping={
          "rating": "driver_rating"
        }
    )
)

By default, Feast assumes that features specified in the feature-table specification corresponds one-to-one to the fields found in the sources. All features defined in a feature table should be available in the defined sources.

Field mappings can be used to map features defined in Feast to fields as they occur in data sources.

In the example feature-specification table above, we use field mappings to ensure the feature named rating in the batch source is mapped to the field named driver_rating.

Working with a Feature Table

Creating a Feature Table

driver_ft = FeatureTable(...)
client.apply(driver_ft)

Updating a Feature Table

driver_ft = FeatureTable()

client.apply(driver_ft)

driver_ft.labels = {"team": "marketplace"}

client.apply(driver_ft)

Feast currently supports the following changes to feature tables:

• Adding new features.

• Removing features.

• Updating source, max age, and labels.

Feast currently does not support the following changes to feature tables:

• Changes to the project or name of a feature table.

• Changes to entities related to a feature table.

• Changes to names and types of existing features.

Deleting a Feature Table

Overview

An entity is any domain object that can be modeled and about which information can be stored. Entities are usually recognizable concepts, either concrete or abstract, such as persons, places, things, or events.

Examples of entities in the context of ride-hailing and food delivery: customer, order, driver, restaurant, dish, area.

Entities are important in the context of feature stores since features are always properties of a specific entity. For example, we could have a feature total_trips_24h for driver D011234 with a feature value of 11.

Feast uses entities in the following way:

• Entities serve as the keys used to look up features for producing training datasets and online feature values.

• Entities serve as a natural grouping of features in a feature table. A feature table must belong to an entity (which could be a composite entity)

Structure of an Entity

When creating an entity specification, consider the following fields:

• Name: Name of the entity

• Description: Description of the entity

• Value Type: Value type of the entity. Feast will attempt to coerce entity columns in your data sources into this type.

• Labels: Labels are maps that allow users to attach their own metadata to entities

A valid entity specification is shown below:

customer = Entity(
    name="customer_id",
    description="Customer id for ride customer",
    value_type=ValueType.INT64,
    labels={}
)

Working with an Entity

Creating an Entity:

# Create a customer entity
customer_entity = Entity(name="customer_id", description="ID of car customer")
client.apply(customer_entity)

Updating an Entity:

# Update a customer entity
customer_entity = client.get_entity("customer_id")
customer_entity.description = "ID of bike customer"
client.apply(customer_entity)

Permitted changes include:

• The entity's description and labels

The following changes are not permitted:

• Project

• Name of an entity

• Type

Overview

Sources are descriptions of external feature data and are registered to Feast as part of feature tables. Once registered, Feast can ingest feature data from these sources into stores.

Currently, Feast supports the following source types:

Batch Source

• File (as in Spark): Parquet (only).

• BigQuery

Stream Source

• Kafka

• Kinesis

The following encodings are supported on streams

• Avro

• Protobuf

Structure of a Source

For both batch and stream sources, the following configurations are necessary:

• Event timestamp column: Name of column containing timestamp when event data occurred. Used during point-in-time join of feature values to entity timestamps.

• Created timestamp column: Name of column containing timestamp when data is created. Used to deduplicate data when multiple copies of the same entity key is ingested.

Example data source specifications:

from feast import FileSource
from feast.data_format import ParquetFormat

batch_file_source = FileSource(
    file_format=ParquetFormat(),
    file_url="file:///feast/customer.parquet",
    event_timestamp_column="event_timestamp",
    created_timestamp_column="created_timestamp",
)

from feast import KafkaSource
from feast.data_format import ProtoFormat

stream_kafka_source = KafkaSource(
    bootstrap_servers="localhost:9094",
    message_format=ProtoFormat(class_path="class.path"),
    topic="driver_trips",
    event_timestamp_column="event_timestamp",
    created_timestamp_column="created_timestamp",
)

The Feast Python API documentation provides more information about options to specify for the above sources.

Working with a Source

Creating a Source

Sources are defined as part of feature tables:

batch_bigquery_source = BigQuerySource(
    table_ref="gcp_project:bq_dataset.bq_table",
    event_timestamp_column="event_timestamp",
    created_timestamp_column="created_timestamp",
)

stream_kinesis_source = KinesisSource(
    bootstrap_servers="localhost:9094",
    record_format=ProtoFormat(class_path="class.path"),
    region="us-east-1",
    stream_name="driver_trips",
    event_timestamp_column="event_timestamp",
    created_timestamp_column="created_timestamp",
)

Feast ensures that the source complies with the schema of the feature table. These specified data sources can then be included inside a feature table specification and registered to Feast Core.

In Feast, a store is a database that is populated with feature data that will ultimately be served to models.

Offline (Historical) Store

The offline store maintains historical copies of feature values. These features are grouped and stored in feature tables. During retrieval of historical data, features are queries from these feature tables in order to produce training datasets.

Online Store

The online store maintains only the latest values for a specific feature.

• Feature values are stored based on their entity keys

• Feast currently supports Redis as an online store.

• Online stores are meant for very high throughput writes from ingestion jobs and very low latency access to features during online serving.

Concepts

Entities are objects in an organization like customers, transactions, and drivers, products, etc.

Sources are external sources of data where feature data can be found.

Feature Tables are objects that define logical groupings of features, data sources, and other related metadata.

Concept Hierarchy

Feast contains the following core concepts:

• Projects: Serve as a top level namespace for all Feast resources. Each project is a completely independent environment in Feast. Users can only work in a single project at a time.

• Entities: Entities are the objects in an organization on which features occur. They map to your business domain (users, products, transactions, locations).

• Feature Tables: Defines a group of features that occur on a specific entity.

• Features: Individual feature within a feature table.

Sequence description

1. Log Raw Events: Production backend applications are configured to emit internal state changes as events to a stream.

2. Create Stream Features: Stream processing systems like Flink, Spark, and Beam are used to transform and refine events and to produce features that are logged back to the stream.

3. Log Streaming Features: Both raw and refined events are logged into a data lake or batch storage location.

4. Create Batch Features: ELT/ETL systems like Spark and SQL are used to transform data in the batch store.

5. Define and Ingest Features: The Feast user defines feature tables based on the features available in batch and streaming sources and publish these definitions to Feast Core.

6. Poll Feature Definitions: The Feast Job Service polls for new or changed feature definitions.

7. Start Ingestion Jobs: Every new feature table definition results in a new ingestion job being provisioned (see limitations).

8. Batch Ingestion: Batch ingestion jobs are short-lived jobs that load data from batch sources into either an offline or online store (see limitations).

9. Stream Ingestion: Streaming ingestion jobs are long-lived jobs that load data from stream sources into online stores. A stream source and batch source on a feature table must have the same features/fields.

10. Model Training: A model training pipeline is launched. It uses the Feast Python SDK to retrieve a training dataset and trains a model.

11. Get Historical Features: Feast exports a point-in-time correct training dataset based on the list of features and entity DataFrame provided by the model training pipeline.

12. Deploy Model: The trained model binary (and list of features) are deployed into a model serving system.

13. Get Prediction: A backend system makes a request for a prediction from the model serving service.

14. Retrieve Online Features: The model serving service makes a request to the Feast Online Serving service for online features using a Feast SDK.

15. Return Prediction: The model serving service makes a prediction using the returned features and returns the outcome.

Components:

A complete Feast deployment contains the following components:

• Feast Core: Acts as the central registry for feature and entity definitions in Feast.

• Feast Job Service: Manages data processing jobs that load data from sources into stores, and jobs that export training datasets.

• Feast Serving: Provides low-latency access to feature values in an online store.

• Feast Python SDK CLI: The primary user facing SDK. Used to:

  • Manage feature definitions with Feast Core.

  • Launch jobs through the Feast Job Service.

  • Retrieve training datasets.

  • Retrieve online features.

• Online Store: The online store is a database that stores only the latest feature values for each entity. The online store can be populated by either batch ingestion jobs (in the case the user has no streaming source), or can be populated by a streaming ingestion job from a streaming source. Feast Online Serving looks up feature values from the online store.

• Offline Store: The offline store persists batch data that has been ingested into Feast. This data is used for producing training datasets.

• Feast Spark SDK: A Spark specific Feast SDK. Allows teams to use Spark for loading features into an online store and for building training datasets over offline sources.

Please see the configuration reference for more details on configuring these components.