All pages
Powered by GitBook
1 of 12

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Online stores

Please see for an explanation of online stores.

Online Store
Overview
SQLite
Snowflake
Redis
Datastore
DynamoDB
Bigtable
PostgreSQL (contrib)
Cassandra + Astra DB (contrib)
MySQL (contrib)
Rockset (contrib)

SQLite

Description

The SQLite online store provides support for materializing feature values into an SQLite database for serving online features.

  • All feature values are stored in an on-disk SQLite database

  • Only the latest feature values are persisted

Example

feature_store.yaml
project: my_feature_repo
registry: data/registry.db
provider: local
online_store:
  type: sqlite
  path: data/online_store.db

The full set of configuration options is available in SqliteOnlineStoreConfig.

Functionality Matrix

The set of functionality supported by online stores is described in detail . Below is a matrix indicating which functionality is supported by the Sqlite online store.

Sqlite

To compare this set of functionality against other online stores, please see the full .

teardown infrastructure (e.g. tables) in the online store

yes

generate a plan of infrastructure changes

yes

support for on-demand transforms

yes

readable by Python SDK

yes

readable by Java

no

readable by Go

yes

support for entityless feature views

yes

support for concurrent writing to the same key

no

support for ttl (time to live) at retrieval

no

support for deleting expired data

no

collocated by feature view

yes

collocated by feature service

no

collocated by entity key

no

write feature values to the online store

yes

read feature values from the online store

yes

update infrastructure (e.g. tables) in the online store

here
functionality matrix

yes

Overview

Functionality

Here are the methods exposed by the OnlineStore interface, along with the core functionality supported by the method:

  • online_write_batch: write feature values to the online store

  • online_read: read feature values from the online store

  • update: update infrastructure (e.g. tables) in the online store

  • teardown: teardown infrastructure (e.g. tables) in the online store

  • plan: generate a plan of infrastructure changes based on feature repo changes

There is also additional functionality not properly captured by these interface methods:

  • support for on-demand transforms

  • readable by Python SDK

  • readable by Java

Finally, there are multiple data models for storing the features in the online store. For example, features could be:

  • collocated by feature view

  • collocated by feature service

  • collocated by entity key

See this for a discussion around the tradeoffs of each of these data models.

There are currently five core online store implementations: SqliteOnlineStore, RedisOnlineStore, DynamoDBOnlineStore, SnowflakeOnlineStore, and DatastoreOnlineStore. There are several additional implementations contributed by the Feast community (PostgreSQLOnlineStore, HbaseOnlineStore, and CassandraOnlineStore), which are not guaranteed to be stable or to match the functionality of the core implementations. Details for each specific online store, such as how to configure it in a feature_store.yaml, can be found .

Below is a matrix indicating which online stores support what functionality.

Sqlite
Redis
DynamoDB
Snowflake
Datastore
Postgres
Hbase
Cassandra
readable by Go
  • support for entityless feature views

  • support for concurrent writing to the same key

  • support for ttl (time to live) at retrieval

  • support for deleting expired data

  • yes

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    read feature values from the online store

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    update infrastructure (e.g. tables) in the online store

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    teardown infrastructure (e.g. tables) in the online store

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    generate a plan of infrastructure changes

    yes

    no

    no

    no

    no

    no

    no

    yes

    support for on-demand transforms

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    readable by Python SDK

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    readable by Java

    no

    yes

    no

    no

    no

    no

    no

    no

    readable by Go

    yes

    yes

    no

    no

    no

    no

    no

    no

    support for entityless feature views

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    yes

    support for concurrent writing to the same key

    no

    yes

    no

    no

    no

    no

    no

    no

    support for ttl (time to live) at retrieval

    no

    yes

    no

    no

    no

    no

    no

    no

    support for deleting expired data

    no

    yes

    no

    no

    no

    no

    no

    no

    collocated by feature view

    yes

    no

    yes

    yes

    yes

    yes

    yes

    yes

    collocated by feature service

    no

    no

    no

    no

    no

    no

    no

    no

    collocated by entity key

    no

    yes

    no

    no

    no

    no

    no

    no

    Functionality Matrix

    issue
    here

    write feature values to the online store

    Rockset (contrib)

    In Alpha Development.

    The online store provides support for materializing feature values within a Rockset collection in order to serve features in real-time.

    • Each document is uniquely identified by its '_id' value. Repeated inserts into the same document '_id' will result in an upsert.

    Rockset indexes all columns allowing for quick per feature look up and also allows for a dynamic typed schema that can change based on any new requirements. API Keys can be found in the Rockset console. You can also find host urls on the same tab by clicking "View Region Endpoint Urls".

    Data Model Used Per Doc

    Bigtable

    The online store provides support for materializing feature values into Cloud Bigtable. The data model used to store feature values in Bigtable is described in more detail .

    In order to use this online store, you'll need to run pip install 'feast[gcp]'. You can then get started with the command feast init REPO_NAME -t gcp.

    The full set of configuration options is available in .

    The set of functionality supported by online stores is described in detail . Below is a matrix indicating which functionality is supported by the Bigtable online store.

    MySQL (contrib)

    The MySQL online store provides support for materializing feature values into a MySQL database for serving online features.

    • Only the latest feature values are persisted

    In order to use this online store, you'll need to run pip install 'feast[mysql]'. You can get started by then running feast init and then setting the feature_store.yaml as described below.

    Description

    Rockset
    {
      "_id": (STRING) Unique Identifier for the feature document.
      <key_name>: (STRING) Feature Values Mapped by Feature Name. Feature
                           values stored as a serialized hex string.
      ....
      "event_ts": (STRING) ISO Stringified Timestamp.
      "created_ts": (STRING) ISO Stringified Timestamp.
    }

    Example

    project: my_feature_app
    registry: data/registry.db
    provider: local
    online_stores
        ## Basic Configs ##
    
        # If apikey or host is left blank the driver will try to pull
        # these values from environment variables ROCKSET_APIKEY and 
        # ROCKSET_APISERVER respectively.
        type: rockset
        apikey: <your_api_key_here>
        host: <your_region_endpoint_here>
      
        ## Advanced Configs ## 
    
        # Batch size of records that will be turned per page when
        # paginating a batched read.
        #
        # read_pagination_batch_size: 100
    
        # The amount of time, in seconds, we will wait for the
        # collection to become visible to the API.
        #
        # collection_created_timeout_secs: 60
    
        # The amount of time, in seconds, we will wait for the
        # collection to enter READY state.
        #
        # collection_ready_timeout_secs: 1800
    
        # Whether to wait for all writes to be flushed from log
        # and queryable before returning write as completed. If
        # False, documents that are written may not be seen
        # immediately in subsequent reads.
        #
        # fence_all_writes: True
    
        # The amount of time we will wait, in seconds, for the
        # write fence to be passed
        #
        # fence_timeout_secs: 600
    
        # Initial backoff, in seconds, we will wait between
        # requests when polling for a response.
        #
        # initial_request_backoff_secs: 2
    
        # Initial backoff, in seconds, we will wait between
        # requests when polling for a response.
        # max_request_backoff_secs: 30
    
        # The max amount of times we will retry a failed request.
        # max_request_attempts: 10000
    Bigtable

    write feature values to the online store

    yes

    read feature values from the online store

    yes

    update infrastructure (e.g. tables) in the online store

    yes

    To compare this set of functionality against other online stores, please see the full functionality matrix.

    feature_store.yaml
    project: my_feature_repo
    registry: data/registry.db
    provider: gcp
    online_store:
      type: bigtable
      project_id: my_gcp_project
      instance: my_bigtable_instance

    Description

    Getting started

    Example

    Functionality Matrix

    Bigtable
    here
    BigtableOnlineStoreConfig
    here
    The full set of configuration options is available in MySQLOnlineStoreConfig.

    The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Mys online store.

    Mys

    write feature values to the online store

    yes

    read feature values from the online store

    yes

    update infrastructure (e.g. tables) in the online store

    To compare this set of functionality against other online stores, please see the full functionality matrix.

    feature_store.yaml
    project: my_feature_repo
    registry: data/registry.db
    provider: local
    online_store:
        type: mysql
        host: DB_HOST
        port: DB_PORT
        database: DB_NAME
        user: DB_USERNAME
        password: DB_PASSWORD

    Description

    Getting started

    Example

    Functionality Matrix

    Snowflake

    Description

    The Snowflake online store provides support for materializing feature values into a Snowflake Transient Table for serving online features.

    • Only the latest feature values are persisted

    The data model for using a Snowflake Transient Table as an online store follows a tall format (one row per feature)):

    • "entity_feature_key" (BINARY) -- unique key used when reading specific feature_view x entity combination

    • "entity_key" (BINARY) -- repeated key currently unused for reading entity_combination

    • "feature_name" (VARCHAR)

    • "value" (BINARY)

    • "event_ts" (TIMESTAMP)

    • "created_ts" (TIMESTAMP)

    (This model may be subject to change when Snowflake Hybrid Tables are released)

    In order to use this online store, you'll need to run pip install 'feast[snowflake]'. You can then get started with the command feast init REPO_NAME -t snowflake.

    "snowflake-online-store/online_path": Adding the "snowflake-online-store/online_path" key to a FeatureView tags parameter allows you to choose the online table path for the online serving table (ex. "{database}"."{schema}").

    The full set of configuration options is available in .

    The set of functionality supported by online stores is described in detail . Below is a matrix indicating which functionality is supported by the Snowflake online store.

    Snowflake

    To compare this set of functionality against other online stores, please see the full .

    teardown infrastructure (e.g. tables) in the online store

    yes

    generate a plan of infrastructure changes

    no

    support for on-demand transforms

    yes

    readable by Python SDK

    yes

    readable by Java

    no

    readable by Go

    no

    support for entityless feature views

    yes

    support for concurrent writing to the same key

    yes

    support for ttl (time to live) at retrieval

    no

    support for deleting expired data

    no

    collocated by feature view

    yes

    collocated by feature service

    no

    collocated by entity key

    yes

    yes

    teardown infrastructure (e.g. tables) in the online store

    yes

    generate a plan of infrastructure changes

    no

    support for on-demand transforms

    yes

    readable by Python SDK

    yes

    readable by Java

    no

    readable by Go

    no

    support for entityless feature views

    yes

    support for concurrent writing to the same key

    no

    support for ttl (time to live) at retrieval

    no

    support for deleting expired data

    no

    collocated by feature view

    yes

    collocated by feature service

    no

    collocated by entity key

    no

    teardown infrastructure (e.g. tables) in the online store

    yes

    generate a plan of infrastructure changes

    no

    support for on-demand transforms

    yes

    readable by Python SDK

    yes

    readable by Java

    no

    readable by Go

    no

    support for entityless feature views

    yes

    support for concurrent writing to the same key

    no

    support for ttl (time to live) at retrieval

    no

    support for deleting expired data

    no

    collocated by feature view

    yes

    collocated by feature service

    no

    collocated by entity key

    no

    write feature values to the online store

    yes

    read feature values from the online store

    yes

    update infrastructure (e.g. tables) in the online store

    Getting started

    Example

    Tags KWARGs Actions:

    Functionality Matrix

    SnowflakeOnlineStoreConfig
    here
    functionality matrix

    yes

    feature_store.yaml
    project: my_feature_repo
    registry: data/registry.db
    provider: local
    online_store:
        type: snowflake.online
        account: SNOWFLAKE_DEPLOYMENT_URL
        user: SNOWFLAKE_USER
        password: SNOWFLAKE_PASSWORD
        role: SNOWFLAKE_ROLE
        warehouse: SNOWFLAKE_WAREHOUSE
        database: SNOWFLAKE_DATABASE
    example_config.py
    driver_stats_fv = FeatureView(
        ...
        tags={"snowflake-online-store/online_path": '"FEAST"."ONLINE"'},
    )

    DynamoDB

    Description

    The DynamoDB online store provides support for materializing feature values into AWS DynamoDB.

    Getting started

    In order to use this online store, you'll need to run pip install 'feast[aws]'. You can then get started with the command feast init REPO_NAME -t aws.

    Example

    feature_store.yaml
    project: my_feature_repo
    registry: data/registry.db
    provider: aws
    online_store:
      type: dynamodb
      region: us-west-2

    The full set of configuration options is available in DynamoDBOnlineStoreConfig.

    Permissions

    Feast requires the following permissions in order to execute commands for DynamoDB online store:

    The following inline policy can be used to grant Feast the necessary permissions:

    Lastly, this IAM role needs to be associated with the desired Redshift cluster. Please follow the official AWS guide for the necessary steps .

    The set of functionality supported by online stores is described in detail . Below is a matrix indicating which functionality is supported by the DynamoDB online store.

    DynamoDB

    To compare this set of functionality against other online stores, please see the full .

    Cassandra + Astra DB (contrib)

    Description

    The [Cassandra / Astra DB] online store provides support for materializing feature values into an Apache Cassandra / Astra DB database for online features.

    • The whole project is contained within a Cassandra keyspace

    • Each feature view is mapped one-to-one to a specific Cassandra table

    • This implementation inherits all strengths of Cassandra such as high availability, fault-tolerance, and data distribution

    In order to use this online store, you'll need to run pip install 'feast[cassandra]'. You can then get started with the command feast init REPO_NAME -t cassandra.

    The full set of configuration options is available in . For a full explanation of configuration options please look at file sdk/python/feast/infra/online_stores/contrib/cassandra_online_store/README.md.

    Storage specifications can be found at docs/specs/online_store_format.md.

    The set of functionality supported by online stores is described in detail . Below is a matrix indicating which functionality is supported by the Cassandra online store.

    Cassandra

    To compare this set of functionality against other online stores, please see the full .

    Redis

    Description

    The Redis online store provides support for materializing feature values into Redis.

    • Both Redis and Redis Cluster are supported.

    • The data model used to store feature values in Redis is described in more detail here.

    Getting started

    In order to use this online store, you'll need to install the redis extra (along with the dependency needed for the offline store of choice). E.g.

    • pip install 'feast[gcp, redis]'

    • pip install 'feast[snowflake, redis]'

    • pip install 'feast[aws, redis]'

    You can get started by using any of the other templates (e.g. feast init -t gcp or feast init -t snowflake or feast init -t aws), and then swapping in Redis as the online store as seen below in the examples.

    Connecting to a single Redis instance:

    Connecting to a Redis Cluster with SSL enabled and password authentication:

    Additionally, the redis online store also supports automatically deleting data via a TTL mechanism. The TTL is applied at the entity level, so feature values from any associated feature views for an entity are removed together. This TTL can be set in the feature_store.yaml, using the key_ttl_seconds field in the online store. For example:

    The full set of configuration options is available in .

    The set of functionality supported by online stores is described in detail . Below is a matrix indicating which functionality is supported by the Redis online store.

    Redis

    To compare this set of functionality against other online stores, please see the full .

    Datastore

    Description

    The Datastore online store provides support for materializing feature values into Cloud Datastore. The data model used to store feature values in Datastore is described in more detail here.

    Getting started

    In order to use this online store, you'll need to run pip install 'feast[gcp]'. You can then get started with the command feast init REPO_NAME -t gcp.

    Example

    feature_store.yaml
    project: my_feature_repo
    registry: data/registry.db
    provider: gcp
    online_store:
      type: datastore
      project_id: my_gcp_project
      namespace: my_datastore_namespace

    The full set of configuration options is available in DatastoreOnlineStoreConfig.

    Functionality Matrix

    The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Datastore online store.

    Datastore

    To compare this set of functionality against other online stores, please see the full .

    PostgreSQL (contrib)

    Description

    The PostgreSQL online store provides support for materializing feature values into a PostgreSQL database for serving online features.

    • Only the latest feature values are persisted

    • sslmode, sslkey_path, sslcert_path, and sslrootcert_path are optional

    Getting started

    In order to use this online store, you'll need to run pip install 'feast[postgres]'. You can get started by then running feast init -t postgres.

    The full set of configuration options is available in .

    The set of functionality supported by online stores is described in detail . Below is a matrix indicating which functionality is supported by the Postgres online store.

    Postgres

    To compare this set of functionality against other online stores, please see the full .

    teardown infrastructure (e.g. tables) in the online store

    yes

    generate a plan of infrastructure changes

    no

    support for on-demand transforms

    yes

    readable by Python SDK

    yes

    readable by Java

    no

    readable by Go

    no

    support for entityless feature views

    yes

    support for concurrent writing to the same key

    no

    support for ttl (time to live) at retrieval

    no

    support for deleting expired data

    no

    collocated by feature view

    yes

    collocated by feature service

    no

    collocated by entity key

    no

    write feature values to the online store

    yes

    read feature values from the online store

    yes

    update infrastructure (e.g. tables) in the online store

    yes

    functionality matrix

    teardown infrastructure (e.g. tables) in the online store

    yes

    generate a plan of infrastructure changes

    no

    support for on-demand transforms

    yes

    readable by Python SDK

    yes

    readable by Java

    no

    readable by Go

    no

    support for entityless feature views

    yes

    support for concurrent writing to the same key

    no

    support for ttl (time to live) at retrieval

    no

    support for deleting expired data

    no

    collocated by feature view

    yes

    collocated by feature service

    no

    collocated by entity key

    no

    Command

    Permissions

    Resources

    Apply

    dynamodb:CreateTable

    dynamodb:DescribeTable

    dynamodb:DeleteTable

    arn:aws:dynamodb:<region>:<account_id>:table/*

    Materialize

    dynamodb.BatchWriteItem

    arn:aws:dynamodb:<region>:<account_id>:table/*

    Get Online Features

    dynamodb.BatchGetItem

    arn:aws:dynamodb:<region>:<account_id>:table/*

    write feature values to the online store

    yes

    read feature values from the online store

    yes

    update infrastructure (e.g. tables) in the online store

    Functionality Matrix

    here
    here
    functionality matrix

    yes

    teardown infrastructure (e.g. tables) in the online store

    yes

    generate a plan of infrastructure changes

    yes

    support for on-demand transforms

    yes

    readable by Python SDK

    yes

    readable by Java

    no

    readable by Go

    no

    support for entityless feature views

    yes

    support for concurrent writing to the same key

    no

    support for ttl (time to live) at retrieval

    no

    support for deleting expired data

    no

    collocated by feature view

    yes

    collocated by feature service

    no

    collocated by entity key

    no

    write feature values to the online store

    yes

    read feature values from the online store

    yes

    update infrastructure (e.g. tables) in the online store

    Getting started

    Example (Cassandra)

    Example (Astra DB)

    Functionality Matrix

    CassandraOnlineStoreConfig
    here
    functionality matrix

    yes

    pip install 'feast[azure, redis]'

    teardown infrastructure (e.g. tables) in the online store

    yes

    generate a plan of infrastructure changes

    no

    support for on-demand transforms

    yes

    readable by Python SDK

    yes

    readable by Java

    yes

    readable by Go

    yes

    support for entityless feature views

    yes

    support for concurrent writing to the same key

    yes

    support for ttl (time to live) at retrieval

    yes

    support for deleting expired data

    yes

    collocated by feature view

    no

    collocated by feature service

    no

    collocated by entity key

    yes

    write feature values to the online store

    yes

    read feature values from the online store

    yes

    update infrastructure (e.g. tables) in the online store

    Examples

    Functionality Matrix

    RedisOnlineStoreConfig
    here
    functionality matrix

    yes

    teardown infrastructure (e.g. tables) in the online store

    yes

    generate a plan of infrastructure changes

    no

    support for on-demand transforms

    yes

    readable by Python SDK

    yes

    readable by Java

    no

    readable by Go

    no

    support for entityless feature views

    yes

    support for concurrent writing to the same key

    no

    support for ttl (time to live) at retrieval

    no

    support for deleting expired data

    no

    collocated by feature view

    yes

    collocated by feature service

    no

    collocated by entity key

    no

    write feature values to the online store

    yes

    read feature values from the online store

    yes

    update infrastructure (e.g. tables) in the online store

    Example

    Functionality Matrix

    PostgreSQLOnlineStoreConfig
    here
    functionality matrix

    yes

    {
        "Statement": [
            {
                "Action": [
                    "dynamodb:CreateTable",
                    "dynamodb:DescribeTable",
                    "dynamodb:DeleteTable",
                    "dynamodb:BatchWriteItem",
                    "dynamodb:BatchGetItem"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws:dynamodb:<region>:<account_id>:table/*"
                ]
            }
        ],
        "Version": "2012-10-17"
    }
    feature_store.yaml
    project: my_feature_repo
    registry: data/registry.db
    provider: local
    online_store:
        type: cassandra
        hosts:
            - 192.168.1.1
            - 192.168.1.2
            - 192.168.1.3
        keyspace: KeyspaceName
        port: 9042                                                              # optional
        username: user                                                          # optional
        password: secret                                                        # optional
        protocol_version: 5                                                     # optional
        load_balancing:                                                         # optional
            local_dc: 'datacenter1'                                             # optional
            load_balancing_policy: 'TokenAwarePolicy(DCAwareRoundRobinPolicy)'  # optional
        read_concurrency: 100                                                   # optional
        write_concurrency: 100                                                  # optional
    feature_store.yaml
    project: my_feature_repo
    registry: data/registry.db
    provider: local
    online_store:
        type: cassandra
        secure_bundle_path: /path/to/secure/bundle.zip
        keyspace: KeyspaceName
        username: Client_ID
        password: Client_Secret
        protocol_version: 4                                                     # optional
        load_balancing:                                                         # optional
            local_dc: 'eu-central-1'                                            # optional
            load_balancing_policy: 'TokenAwarePolicy(DCAwareRoundRobinPolicy)'  # optional
        read_concurrency: 100                                                   # optional
        write_concurrency: 100                                                  # optional
    feature_store.yaml
    project: my_feature_repo
    registry: data/registry.db
    provider: local
    online_store:
      type: redis
      connection_string: "localhost:6379"
    feature_store.yaml
    project: my_feature_repo
    registry: data/registry.db
    provider: local
    online_store:
      type: redis
      redis_type: redis_cluster
      connection_string: "redis1:6379,redis2:6379,ssl=true,password=my_password"
    feature_store.yaml
    project: my_feature_repo
    registry: data/registry.db
    provider: local
    online_store:
      type: redis
      key_ttl_seconds: 604800
      connection_string: "localhost:6379"
    feature_store.yaml
    project: my_feature_repo
    registry: data/registry.db
    provider: local
    online_store:
        type: postgres
        host: DB_HOST
        port: DB_PORT
        database: DB_NAME
        db_schema: DB_SCHEMA
        user: DB_USERNAME
        password: DB_PASSWORD
        sslmode: verify-ca
        sslkey_path: /path/to/client-key.pem
        sslcert_path: /path/to/client-cert.pem
        sslrootcert_path: /path/to/server-ca.pem