Data source

A data source in Feast refers to raw underlying data that users own (e.g. in a table in BigQuery). Feast does not manage any of the raw underlying data but instead, is in charge of loading this data and performing different operations on the data to retrieve or serve features.

Feast uses a time-series data model to represent data. This data model is used to interpret feature data in data sources in order to build training datasets or materialize features into an online store.

Below is an example data source with a single entity column (driver) and two feature columns (trips_today, and rating).

Feast supports primarily time-stamped tabular data as data sources. There are many kinds of possible data sources:

• Batch data sources: ideally, these live in data warehouses (BigQuery, Snowflake, Redshift), but can be in data lakes (S3, GCS, etc). Feast supports ingesting and querying data across both.

• Stream data sources: Feast does not have native streaming integrations. It does however facilitate making streaming features available in different environments. There are two kinds of sources:

  • Push sources allow users to push features into Feast, and make it available for training / batch scoring ("offline"), for realtime feature serving ("online") or both.

  • [Alpha] Stream sources allow users to register metadata from Kafka or Kinesis sources. The onus is on the user to ingest from these sources, though Feast provides some limited helper methods to ingest directly from Kafka / Kinesis topics.

• (Experimental) Request data sources: This is data that is only available at request time (e.g. from a user action that needs an immediate model prediction response). This is primarily relevant as an input into on-demand feature views, which allow light-weight feature engineering and combining features across sources.

Batch data ingestion

Ingesting from batch sources is only necessary to power real-time models. This is done through materialization. Under the hood, Feast manages an offline store (to scalably generate training data from batch sources) and an online store (to provide low-latency access to features for real-time models).

A key command to use in Feast is the materialize_incremental command, which fetches the latest values for all entities in the batch source and ingests these values into the online store.

Materialization can be called programmatically or through the CLI:

# Define Python callable
def materialize():
  repo_config = RepoConfig(
    registry=RegistryConfig(path="s3://[YOUR BUCKET]/registry.pb"),
    project="feast_demo_aws",
    provider="aws",
    offline_store="file",
    online_store=DynamoDBOnlineStoreConfig(region="us-west-2")
  )
  store = FeatureStore(config=repo_config)
  store.materialize_incremental(datetime.datetime.now())

# (In production) Use Airflow PythonOperator
materialize_python = PythonOperator(
    task_id='materialize_python',
    python_callable=materialize,
)

CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")
feast materialize-incremental $CURRENT_TIME

# Use BashOperator
materialize_bash = BashOperator(
    task_id='materialize',
    bash_command=f'feast materialize-incremental {datetime.datetime.now().replace(microsecond=0).isoformat()}',
)

Batch data schema inference

If the schema parameter is not specified when defining a data source, Feast attempts to infer the schema of the data source during feast apply. The way it does this depends on the implementation of the offline store. For the offline stores that ship with Feast out of the box this inference is performed by inspecting the schema of the table in the cloud data warehouse, or if a query is provided to the source, by running the query with a LIMIT clause and inspecting the result.

Stream data ingestion

Ingesting from stream sources happens either via a Push API or via a contrib processor that leverages an existing Spark context.

• To push data into the offline or online stores: see push sources for details.

• (experimental) To use a contrib Spark processor to ingest from a topic, see Tutorial: Building streaming features

Links & Resources

• GitHub Repository: Find the complete Feast codebase on GitHub.

  • Community Governance Doc: See the governance model of Feast, including who the maintainers are and how decisions are made.

• Google Folder: This folder is used as a central repository for all Feast resources. For example:

  • Design proposals in the form of Request for Comments (RFC).

  • User surveys and meeting minutes.

  • Slide decks of conferences our contributors have spoken at.

• Feast Linux Foundation Wiki: Our LFAI wiki page contains links to resources for contributors and maintainers.

How can I get help?

• GitHub Issues: Found a bug or need a feature? Create an issue on GitHub.

The list below contains the functionality that contributors are planning to develop for Feast.

• We welcome contribution to all items in the roadmap!

• Data Sources

  • Snowflake source

  • Redshift source

  • BigQuery source

  • Parquet file source

  • Azure Synapse + Azure SQL source (contrib plugin)

  • Hive (community plugin)

  • Postgres (contrib plugin)

  • Spark (contrib plugin)

  • Kafka / Kinesis sources (via push support into the online store)

• Offline Stores

  • Snowflake

  • Redshift

  • BigQuery

  • Azure Synapse + Azure SQL (contrib plugin)

  • Hive (community plugin)

  • Postgres (contrib plugin)

  • Trino (contrib plugin)

  • Spark (contrib plugin)

  • In-memory / Pandas

  • Custom offline store support

• Online Stores

  • Snowflake

  • DynamoDB

  • Redis

  • Datastore

  • Bigtable

  • SQLite

  • Dragonfly

  • Azure Cache for Redis (community plugin)

  • Postgres (contrib plugin)

  • Cassandra / AstraDB (contrib plugin)

  • Custom online store support

• Feature Engineering

  • On-demand Transformations (Alpha release. See RFC)

  • Streaming Transformations (Alpha release. See RFC)

  • Batch transformation (In progress. See RFC)

• Streaming

  • Custom streaming ingestion job support

  • Push based streaming data ingestion to online store

  • Push based streaming data ingestion to offline store

• Deployments

  • AWS Lambda (Alpha release. See RFC)

  • Kubernetes (See guide)

• Feature Serving

  • Python Client

  • Python feature server

  • Java feature server (alpha)

  • Go feature server (alpha)

• Data Quality Management (See RFC)

  • Data profiling and validation (Great Expectations)

• Feature Discovery and Governance

  • Python SDK for browsing feature registry

  • CLI for browsing feature registry

  • Model-centric feature tracking (feature services)

  • Amundsen integration (see Feast extractor)

  • DataHub integration (see DataHub Feast docs)

  • Feast Web UI (Beta release. See docs)

An entity is a collection of semantically related features. Users define entities to map to the domain of their use case. For example, a ride-hailing service could have customers and drivers as their entities, which group related features that correspond to these customers and drivers.

driver = Entity(name='driver', join_keys=['driver_id'])

The entity name is used to uniquely identify the entity (for example to show in the experimental Web UI). The join key is used to identify the physical primary key on which feature values should be joined together to be retrieved during feature retrieval.

Entities are used by Feast in many contexts, as we explore below:

Use case #1: Defining and storing features

Feast's primary object for defining features is a feature view, which is a collection of features. Feature views map to 0 or more entities, since a feature can be associated with:

• zero entities (e.g. a global feature like num_daily_global_transactions)

• one entity (e.g. a user feature like user_age or last_5_bought_items)

• multiple entities, aka a composite key (e.g. a user + merchant category feature like num_user_purchases_in_merchant_category)

Feast refers to this collection of entities for a feature view as an entity key.

Entities should be reused across feature views. This helps with discovery of features, since it enables data scientists understand how other teams build features for the entity they are most interested in.

Feast will use the feature view concept to then define the schema of groups of features in a low-latency online store.

Use case #2: Retrieving features

At training time, users control what entities they want to look up, for example corresponding to train / test / validation splits. A user specifies a list of entity keys + timestamps they want to fetch point-in-time correct features for to generate a training dataset.

At serving time, users specify entity key(s) to fetch the latest feature values which can power real-time model prediction (e.g. a fraud detection model that needs to fetch the latest transaction user's features to make a prediction).

Feast project structure

Projects provide complete isolation of feature stores at the infrastructure level. This is accomplished through resource namespacing, e.g., prefixing table names with the associated project. Each project should be considered a completely separate universe of entities and features. It is not possible to retrieve features from multiple projects in a single request. We recommend having a single feature store and a single project per environment (dev, staging, prod).

Data ingestion

For offline use cases that only rely on batch data, Feast does not need to ingest data and can query your existing data (leveraging a compute engine, whether it be a data warehouse or (experimental) Spark / Trino). Feast can help manage pushing streaming features to a batch source to make features available for training.

Feature registration and retrieval

Features are registered as code in a version controlled repository, and tie to data sources + model versions via the concepts of entities, feature views, and feature services. We explore these concepts more in the upcoming concept pages. These features are then stored in a registry, which can be accessed across users and services. The features can then be retrieved via SDK API methods or via a deployed feature server which exposes endpoints to query for online features (to power real time models).

Feast supports several patterns of feature retrieval.