Description

The [Cassandra / Astra DB] online store provides support for materializing feature values into an Apache Cassandra / Astra DB database for online features.

• The whole project is contained within a Cassandra keyspace

• Each feature view is mapped one-to-one to a specific Cassandra table

• This implementation inherits all strengths of Cassandra such as high availability, fault-tolerance, and data distribution

Getting started

In order to use this online store, you'll need to run pip install 'feast[cassandra]'. You can then get started with the command feast init REPO_NAME -t cassandra.

Example (Cassandra)

project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
    type: cassandra
    hosts:
        - 192.168.1.1
        - 192.168.1.2
        - 192.168.1.3
    keyspace: KeyspaceName
    port: 9042                                                              # optional
    username: user                                                          # optional
    password: secret                                                        # optional
    protocol_version: 5                                                     # optional
    load_balancing:                                                         # optional
        local_dc: 'datacenter1'                                             # optional
        load_balancing_policy: 'TokenAwarePolicy(DCAwareRoundRobinPolicy)'  # optional
    read_concurrency: 100                                                   # optional
    write_concurrency: 100                                                  # optional

Example (Astra DB)

project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
    type: cassandra
    secure_bundle_path: /path/to/secure/bundle.zip
    keyspace: KeyspaceName
    username: Client_ID
    password: Client_Secret
    protocol_version: 4                                                     # optional
    load_balancing:                                                         # optional
        local_dc: 'eu-central-1'                                            # optional
        load_balancing_policy: 'TokenAwarePolicy(DCAwareRoundRobinPolicy)'  # optional
    read_concurrency: 100                                                   # optional
    write_concurrency: 100                                                  # optional

The full set of configuration options is available in CassandraOnlineStoreConfig. For a full explanation of configuration options please look at file sdk/python/feast/infra/online_stores/contrib/cassandra_online_store/README.md.

Storage specifications can be found at docs/specs/online_store_format.md.

Functionality Matrix

The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Cassandra online store.

To compare this set of functionality against other online stores, please see the full functionality matrix.

Functionality

Here are the methods exposed by the OnlineStore interface, along with the core functionality supported by the method:

• online_write_batch: write feature values to the online store

• online_read: read feature values from the online store

• update: update infrastructure (e.g. tables) in the online store

• teardown: teardown infrastructure (e.g. tables) in the online store

• plan: generate a plan of infrastructure changes based on feature repo changes

There is also additional functionality not properly captured by these interface methods:

• support for on-demand transforms

• readable by Python SDK

• readable by Java

• readable by Go

• support for entityless feature views

• support for concurrent writing to the same key

• support for ttl (time to live) at retrieval

• support for deleting expired data

Finally, there are multiple data models for storing the features in the online store. For example, features could be:

• collocated by feature view

• collocated by feature service

• collocated by entity key

Functionality Matrix

Below is a matrix indicating which online stores support what functionality.

Please see for an explanation of online stores.

Description

The online store provides support for materializing feature values into Cloud Datastore. The data model used to store feature values in Datastore is described in more detail .

Getting started

In order to use this online store, you'll need to run pip install 'feast[gcp]'. You can then get started with the command feast init REPO_NAME -t gcp.

Example

The full set of configuration options is available in .

Functionality Matrix

Description

is a fully-managed embedded key-value store, primarily designed for storing ML features. Most key-value stores (think Redis or Cassandra) need a remote database cluster, whereas IKV allows you to utilize your existing application infrastructure to store data (cost efficient) and access it without any network calls (better performance). See detailed performance benchmarks and cost comparison with Redis on . IKV can be used as an online-store in Feast, the rest of this guide goes over the setup.

Getting started

Make sure you have Python and pip installed.

Install the Feast SDK and CLI: pip install feast

In order to use this online store, you'll need to install the IKV extra (along with the dependency needed for the offline store of choice). E.g.

• pip install 'feast[gcp, ikv]'

• pip install 'feast[snowflake, ikv]'

• pip install 'feast[aws, ikv]'

• pip install 'feast[azure, ikv]'

You can get started by using any of the other templates (e.g. feast init -t gcp or feast init -t snowflake or feast init -t aws), and then swapping in IKV as the online store as seen below in the examples.

1. Provision an IKV store

Go to or email onboarding[at]inlined.io

2. Configure

Update my_feature_repo/feature_store.yaml with the below contents:

After provisioning an IKV account/store, you should have an account id, passkey and store-name. Additionally you must specify a mount-directory - where IKV will pull/update (maintain) a copy of the index for online reads (IKV is an embedded database). It can be skipped only if you don't plan to read any data from this container. The mount directory path usually points to a location on local/remote disk.

The full set of configuration options is available in IKVOnlineStoreConfig at sdk/python/feast/infra/online_stores/contrib/ikv_online_store/ikv.py

Functionality Matrix

Description

The online store provides support for materializing feature values into a Snowflake Transient Table for serving online features.

• Only the latest feature values are persisted

The data model for using a Snowflake Transient Table as an online store follows a tall format (one row per feature)):

• "entity_feature_key" (BINARY) -- unique key used when reading specific feature_view x entity combination

• "entity_key" (BINARY) -- repeated key currently unused for reading entity_combination

• "feature_name" (VARCHAR)

• "value" (BINARY)

• "event_ts" (TIMESTAMP)

• "created_ts" (TIMESTAMP)

(This model may be subject to change when Snowflake Hybrid Tables are released)

Getting started

In order to use this online store, you'll need to run pip install 'feast[snowflake]'. You can then get started with the command feast init REPO_NAME -t snowflake.

Example

Tags KWARGs Actions:

"snowflake-online-store/online_path": Adding the "snowflake-online-store/online_path" key to a FeatureView tags parameter allows you to choose the online table path for the online serving table (ex. "{database}"."{schema}").

Functionality Matrix

Description

The online store provides support for materializing feature values into Cloud Bigtable. The data model used to store feature values in Bigtable is described in more detail .

Getting started

In order to use this online store, you'll need to run pip install 'feast[gcp]'. You can then get started with the command feast init REPO_NAME -t gcp.

Example

The full set of configuration options is available in .

Functionality Matrix

Description

The PostgreSQL online store provides support for materializing feature values into a PostgreSQL database for serving online features.

• Only the latest feature values are persisted

• sslmode, sslkey_path, sslcert_path, and sslrootcert_path are optional

Getting started

In order to use this online store, you'll need to run pip install 'feast[postgres]'. You can get started by then running feast init -t postgres.

Example

Functionality Matrix

PGVector

The vector_len parameter can be used to specify the length of the vector. The default value is 512.

Please make sure to follow the instructions in the repository, which, as the time of this writing, requires you to run CREATE EXTENSION vector; in the database.

Then you can use retrieve_online_documents to retrieve the top k closest vectors to a query vector. For the Retrieval Augmented Generation (RAG) use-case, you have to embed the query prior to passing the query vector.

Description

In Alpha Development.

The Rockset online store provides support for materializing feature values within a Rockset collection in order to serve features in real-time.

• Each document is uniquely identified by its '_id' value. Repeated inserts into the same document '_id' will result in an upsert.

Rockset indexes all columns allowing for quick per feature look up and also allows for a dynamic typed schema that can change based on any new requirements. API Keys can be found in the Rockset console. You can also find host urls on the same tab by clicking "View Region Endpoint Urls".

Data Model Used Per Doc

{
  "_id": (STRING) Unique Identifier for the feature document.
  <key_name>: (STRING) Feature Values Mapped by Feature Name. Feature
                       values stored as a serialized hex string.
  ....
  "event_ts": (STRING) ISO Stringified Timestamp.
  "created_ts": (STRING) ISO Stringified Timestamp.
}

Example

project: my_feature_app
registry: data/registry.db
provider: local
online_store:
    ## Basic Configs ##

    # If apikey or host is left blank the driver will try to pull
    # these values from environment variables ROCKSET_APIKEY and 
    # ROCKSET_APISERVER respectively.
    type: rockset
    api_key: <your_api_key_here>
    host: <your_region_endpoint_here>
  
    ## Advanced Configs ## 

    # Batch size of records that will be turned per page when
    # paginating a batched read.
    #
    # read_pagination_batch_size: 100

    # The amount of time, in seconds, we will wait for the
    # collection to become visible to the API.
    #
    # collection_created_timeout_secs: 60

    # The amount of time, in seconds, we will wait for the
    # collection to enter READY state.
    #
    # collection_ready_timeout_secs: 1800

    # Whether to wait for all writes to be flushed from log
    # and queryable before returning write as completed. If
    # False, documents that are written may not be seen
    # immediately in subsequent reads.
    #
    # fence_all_writes: True

    # The amount of time we will wait, in seconds, for the
    # write fence to be passed
    #
    # fence_timeout_secs: 600

    # Initial backoff, in seconds, we will wait between
    # requests when polling for a response.
    #
    # initial_request_backoff_secs: 2

    # Initial backoff, in seconds, we will wait between
    # requests when polling for a response.
    # max_request_backoff_secs: 30

    # The max amount of times we will retry a failed request.
    # max_request_attempts: 10000

Description

Dragonfly is a modern in-memory datastore that implements novel algorithms and data structures on top of a multi-threaded, share-nothing architecture. Thanks to its API compatibility, Dragonfly can act as a drop-in replacement for Redis. Due to Dragonfly's hardware efficiency, you can run a single node instance on a small 8GB instance or scale vertically to large 768GB machines with 64 cores. This greatly reduces infrastructure costs as well as architectural complexity.

Similar to Redis, Dragonfly can be used as an online feature store for Feast.

Using Dragonfly as a drop-in Feast online store instead of Redis

Make sure you have Python and pip installed.

Install the Feast SDK and CLI

pip install feast

In order to use Dragonfly as the online store, you'll need to install the redis extra:

pip install 'feast[redis]'

1. Create a feature repository

Bootstrap a new feature repository:

feast init feast_dragonfly
cd feast_dragonfly/feature_repo

Update feature_repo/feature_store.yaml with the below contents:

project: feast_dragonfly
registry: data/registry.db
provider: local
online_store:
type: redis
connection_string: "localhost:6379"

2. Start Dragonfly

There are several options available to get Dragonfly up and running quickly. We will be using Docker for this tutorial.

docker run --network=host --ulimit memlock=-1 docker.dragonflydb.io/dragonflydb/dragonfly

3. Register feature definitions and deploy your feature store

feast apply

The apply command scans python files in the current directory (example_repo.py in this case) for feature view/entity definitions, registers the objects, and deploys infrastructure. You should see the following output:

....
Created entity driver
Created feature view driver_hourly_stats_fresh
Created feature view driver_hourly_stats
Created on demand feature view transformed_conv_rate
Created on demand feature view transformed_conv_rate_fresh
Created feature service driver_activity_v1
Created feature service driver_activity_v3
Created feature service driver_activity_v2

Functionality Matrix

The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Redis online store.

To compare this set of functionality against other online stores, please see the full functionality matrix.

Description

The Hazelcast online store provides support for materializing feature values into a Hazelcast cluster for serving online features in real-time. In order to use Hazelcast as an online store, you need to have a running Hazelcast cluster. See this getting started page for more details.

• Each feature view is mapped one-to-one to a specific Hazelcast IMap

• This implementation inherits all strengths of Hazelcast such as high availability, fault-tolerance, and data distribution.

• Secure TSL/SSL connection is supported by Hazelcast online store.

• You can set TTL (Time-To-Live) setting for your features in Hazelcast cluster.

Each feature view corresponds to an IMap in Hazelcast cluster and the entries in that IMap correspond to features of entities. Each feature value stored separately and can be retrieved individually.

Getting started

In order to use Hazelcast online store, you'll need to run pip install 'feast[hazelcast]'. You can then get started with the command feast init REPO_NAME -t hazelcast.

Examples

project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
  type: hazelcast
  cluster_name: dev
  cluster_members: ["localhost:5701"]
  key_ttl_seconds: 36000

Functionality Matrix

To compare this set of functionality against other online stores, please see the full functionality matrix.

Description

The DynamoDB online store provides support for materializing feature values into AWS DynamoDB.

Getting started

In order to use this online store, you'll need to run pip install 'feast[aws]'. You can then get started with the command feast init REPO_NAME -t aws.

Example

project: my_feature_repo
registry: data/registry.db
provider: aws
online_store:
  type: dynamodb
  region: us-west-2

The full set of configuration options is available in DynamoDBOnlineStoreConfig.

Permissions

Feast requires the following permissions in order to execute commands for DynamoDB online store:

The following inline policy can be used to grant Feast the necessary permissions:

{
    "Statement": [
        {
            "Action": [
                "dynamodb:CreateTable",
                "dynamodb:DescribeTable",
                "dynamodb:DeleteTable",
                "dynamodb:BatchWriteItem",
                "dynamodb:BatchGetItem"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:dynamodb:<region>:<account_id>:table/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

Lastly, this IAM role needs to be associated with the desired Redshift cluster. Please follow the official AWS guide for the necessary steps here.

Functionality Matrix

The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the DynamoDB online store.

To compare this set of functionality against other online stores, please see the full functionality matrix.

Description

The MySQL online store provides support for materializing feature values into a MySQL database for serving online features.

• Only the latest feature values are persisted

Getting started

In order to use this online store, you'll need to run pip install 'feast[mysql]'. You can get started by then running feast init and then setting the feature_store.yaml as described below.

Example

project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
    type: mysql
    host: DB_HOST
    port: DB_PORT
    database: DB_NAME
    user: DB_USERNAME
    password: DB_PASSWORD

The full set of configuration options is available in MySQLOnlineStoreConfig.

Functionality Matrix

The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Mys online store.

To compare this set of functionality against other online stores, please see the full functionality matrix.

Description

The Redis online store provides support for materializing feature values into Redis.

• Both Redis and Redis Cluster are supported.

• The data model used to store feature values in Redis is described in more detail here.

Getting started

In order to use this online store, you'll need to install the redis extra (along with the dependency needed for the offline store of choice). E.g.

• pip install 'feast[gcp, redis]'

• pip install 'feast[snowflake, redis]'

• pip install 'feast[aws, redis]'

• pip install 'feast[azure, redis]'

You can get started by using any of the other templates (e.g. feast init -t gcp or feast init -t snowflake or feast init -t aws), and then swapping in Redis as the online store as seen below in the examples.

Examples

Connecting to a single Redis instance:

project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
  type: redis
  connection_string: "localhost:6379"

Connecting to a Redis Cluster with SSL enabled and password authentication:

project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
  type: redis
  redis_type: redis_cluster
  connection_string: "redis1:6379,redis2:6379,ssl=true,password=my_password"

Connecting to a Redis Sentinel with SSL enabled and password authentication:

project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
  type: redis
  redis_type: redis_sentinel
  sentinel_master: mymaster
  connection_string: "redis1:26379,ssl=true,password=my_password"

Additionally, the redis online store also supports automatically deleting data via a TTL mechanism. The TTL is applied at the entity level, so feature values from any associated feature views for an entity are removed together. This TTL can be set in the feature_store.yaml, using the key_ttl_seconds field in the online store. For example:

project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
  type: redis
  key_ttl_seconds: 604800
  connection_string: "localhost:6379"

The full set of configuration options is available in RedisOnlineStoreConfig.

Functionality Matrix

The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Redis online store.

To compare this set of functionality against other online stores, please see the full functionality matrix.

Description

ScyllaDB is a low-latency and high-performance Cassandra-compatible (uses CQL) database. You can use the existing Cassandra connector to use ScyllaDB as an online store in Feast.

The ScyllaDB online store provides support for materializing feature values into a ScyllaDB or ScyllaDB Cloud cluster for serving online features real-time.

Getting started

Install Feast with Cassandra support:

pip install "feast[cassandra]"

Create a new Feast project:

feast init REPO_NAME -t cassandra

Example (ScyllaDB)

project: scylla_feature_repo
registry: data/registry.db
provider: local
online_store:
    type: cassandra
    hosts:
        - 172.17.0.2
    keyspace: feast
    username: scylla
    password: password

Example (ScyllaDB Cloud)

project: scylla_feature_repo
registry: data/registry.db
provider: local
online_store:
    type: cassandra
    hosts:
        - node-0.aws_us_east_1.xxxxxxxx.clusters.scylla.cloud
        - node-1.aws_us_east_1.xxxxxxxx.clusters.scylla.cloud
        - node-2.aws_us_east_1.xxxxxxxx.clusters.scylla.cloud
    keyspace: feast
    username: scylla
    password: password

The full set of configuration options is available in CassandraOnlineStoreConfig. For a full explanation of configuration options please look at file sdk/python/feast/infra/online_stores/contrib/cassandra_online_store/README.md.

Storage specifications can be found at docs/specs/online_store_format.md.

Functionality Matrix

The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Cassandra plugin.

To compare this set of functionality against other online stores, please see the full functionality matrix.

Resources

• Sample application with ScyllaDB

• ScyllaDB website

• ScyllaDB Cloud documentation

Description

The SQLite online store provides support for materializing feature values into an SQLite database for serving online features.

• All feature values are stored in an on-disk SQLite database

• Only the latest feature values are persisted

Example

project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
  type: sqlite
  path: data/online_store.db

The full set of configuration options is available in SqliteOnlineStoreConfig.

Functionality Matrix

The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Sqlite online store.

To compare this set of functionality against other online stores, please see the full functionality matrix.