Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Here are the methods exposed by the OnlineStore
interface, along with the core functionality supported by the method:
online_write_batch
: write feature values to the online store
online_read
: read feature values from the online store
update
: update infrastructure (e.g. tables) in the online store
teardown
: teardown infrastructure (e.g. tables) in the online store
plan
: generate a plan of infrastructure changes based on feature repo changes
There is also additional functionality not properly captured by these interface methods:
support for on-demand transforms
readable by Python SDK
readable by Java
readable by Go
support for entityless feature views
support for concurrent writing to the same key
support for ttl (time to live) at retrieval
support for deleting expired data
Finally, there are multiple data models for storing the features in the online store. For example, features could be:
collocated by feature view
collocated by feature service
collocated by entity key
See this issue for a discussion around the tradeoffs of each of these data models.
There are currently five core online store implementations: SqliteOnlineStore
, RedisOnlineStore
, DynamoDBOnlineStore
, SnowflakeOnlineStore
, and DatastoreOnlineStore
. There are several additional implementations contributed by the Feast community (PostgreSQLOnlineStore
, HbaseOnlineStore
, and CassandraOnlineStore
), which are not guaranteed to be stable or to match the functionality of the core implementations. Details for each specific online store, such as how to configure it in a feature_store.yaml
, can be found here.
Below is a matrix indicating which online stores support what functionality.
Please see for an explanation of online stores.
The online store provides support for materializing feature values into an SQLite database for serving online features.
The full set of configuration options is available in .
The set of functionality supported by online stores is described in detail . Below is a matrix indicating which functionality is supported by the Sqlite online store.
Sqlite |
---|
To compare this set of functionality against other online stores, please see the full .
Sqlite
Redis
DynamoDB
Snowflake
Datastore
Postgres
Hbase
write feature values to the online store
yes
yes
yes
yes
yes
yes
yes
yes
read feature values from the online store
yes
yes
yes
yes
yes
yes
yes
yes
update infrastructure (e.g. tables) in the online store
yes
yes
yes
yes
yes
yes
yes
yes
teardown infrastructure (e.g. tables) in the online store
yes
yes
yes
yes
yes
yes
yes
yes
generate a plan of infrastructure changes
yes
no
no
no
no
no
no
yes
support for on-demand transforms
yes
yes
yes
yes
yes
yes
yes
yes
readable by Python SDK
yes
yes
yes
yes
yes
yes
yes
yes
readable by Java
no
yes
no
no
no
no
no
no
readable by Go
yes
yes
no
no
no
no
no
no
support for entityless feature views
yes
yes
yes
yes
yes
yes
yes
yes
support for concurrent writing to the same key
no
yes
no
no
no
no
no
no
support for ttl (time to live) at retrieval
no
yes
no
no
no
no
no
no
support for deleting expired data
no
yes
no
no
no
no
no
no
collocated by feature view
yes
no
yes
yes
yes
yes
yes
yes
collocated by feature service
no
no
no
no
no
no
no
no
collocated by entity key
no
yes
no
no
no
no
no
no
write feature values to the online store | yes |
read feature values from the online store | yes |
update infrastructure (e.g. tables) in the online store | yes |
teardown infrastructure (e.g. tables) in the online store | yes |
generate a plan of infrastructure changes | yes |
support for on-demand transforms | yes |
readable by Python SDK | yes |
readable by Java | no |
readable by Go | yes |
support for entityless feature views | yes |
support for concurrent writing to the same key | no |
support for ttl (time to live) at retrieval | no |
support for deleting expired data | no |
collocated by feature view | yes |
collocated by feature service | no |
collocated by entity key | no |
The Snowflake online store provides support for materializing feature values into a Snowflake Transient Table for serving online features.
Only the latest feature values are persisted
The data model for using a Snowflake Transient Table as an online store follows a tall format (one row per feature)):
"entity_feature_key" (BINARY) -- unique key used when reading specific feature_view x entity combination
"entity_key" (BINARY) -- repeated key currently unused for reading entity_combination
"feature_name" (VARCHAR)
"value" (BINARY)
"event_ts" (TIMESTAMP)
"created_ts" (TIMESTAMP)
(This model may be subject to change when Snowflake Hybrid Tables are released)
In order to use this online store, you'll need to run pip install 'feast[snowflake]'
. You can then get started with the command feast init REPO_NAME -t snowflake
.
"snowflake-online-store/online_path": Adding the "snowflake-online-store/online_path" key to a FeatureView tags parameter allows you to choose the online table path for the online serving table (ex. "{database}"."{schema}").
The full set of configuration options is available in SnowflakeOnlineStoreConfig.
The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Snowflake online store.
To compare this set of functionality against other online stores, please see the full functionality matrix.
The Redis online store provides support for materializing feature values into Redis.
Both Redis and Redis Cluster are supported.
The data model used to store feature values in Redis is described in more detail here.
In order to use this online store, you'll need to install the redis extra (along with the dependency needed for the offline store of choice). E.g.
pip install 'feast[gcp, redis]'
pip install 'feast[snowflake, redis]'
pip install 'feast[aws, redis]'
pip install 'feast[azure, redis]'
You can get started by using any of the other templates (e.g. feast init -t gcp
or feast init -t snowflake
or feast init -t aws
), and then swapping in Redis as the online store as seen below in the examples.
Connecting to a single Redis instance:
Connecting to a Redis Cluster with SSL enabled and password authentication:
Additionally, the redis online store also supports automatically deleting data via a TTL mechanism. The TTL is applied at the entity level, so feature values from any associated feature views for an entity are removed together. This TTL can be set in the feature_store.yaml
, using the key_ttl_seconds
field in the online store. For example:
The full set of configuration options is available in RedisOnlineStoreConfig.
The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Redis online store.
To compare this set of functionality against other online stores, please see the full functionality matrix.
Similar to Redis, Dragonfly can be used as an online feature store for Feast.
Make sure you have Python and pip
installed.
Install the Feast SDK and CLI
pip install feast
In order to use Dragonfly as the online store, you'll need to install the redis extra:
pip install 'feast[redis]'
Bootstrap a new feature repository:
Update feature_repo/feature_store.yaml
with the below contents:
There are several options available to get Dragonfly up and running quickly. We will be using Docker for this tutorial.
docker run --network=host --ulimit memlock=-1 docker.dragonflydb.io/dragonflydb/dragonfly
feast apply
The apply
command scans python files in the current directory (example_repo.py
in this case) for feature view/entity definitions, registers the objects, and deploys infrastructure. You should see the following output:
In order to use this online store, you'll need to run pip install 'feast[aws]'
. You can then get started with the command feast init REPO_NAME -t aws
.
Feast requires the following permissions in order to execute commands for DynamoDB online store:
The following inline policy can be used to grant Feast the necessary permissions:
Snowflake | |
---|---|
Redis | |
---|---|
is a modern in-memory datastore that implements novel algorithms and data structures on top of a multi-threaded, share-nothing architecture. Thanks to its API compatibility, Dragonfly can act as a drop-in replacement for Redis. Due to Dragonfly's hardware efficiency, you can run a single node instance on a small 8GB instance or scale vertically to large 768GB machines with 64 cores. This greatly reduces infrastructure costs as well as architectural complexity.
The set of functionality supported by online stores is described in detail . Below is a matrix indicating which functionality is supported by the Redis online store.
Redis |
---|
To compare this set of functionality against other online stores, please see the full .
The online store provides support for materializing feature values into Cloud Datastore. The data model used to store feature values in Datastore is described in more detail .
The full set of configuration options is available in .
The set of functionality supported by online stores is described in detail . Below is a matrix indicating which functionality is supported by the Datastore online store.
Datastore |
---|
To compare this set of functionality against other online stores, please see the full .
The online store provides support for materializing feature values into AWS DynamoDB.
The full set of configuration options is available in .
Lastly, this IAM role needs to be associated with the desired Redshift cluster. Please follow the official AWS guide for the necessary steps .
The set of functionality supported by online stores is described in detail . Below is a matrix indicating which functionality is supported by the DynamoDB online store.
DynamoDB |
---|
To compare this set of functionality against other online stores, please see the full .
write feature values to the online store
yes
read feature values from the online store
yes
update infrastructure (e.g. tables) in the online store
yes
teardown infrastructure (e.g. tables) in the online store
yes
generate a plan of infrastructure changes
no
support for on-demand transforms
yes
readable by Python SDK
yes
readable by Java
no
readable by Go
no
support for entityless feature views
yes
support for concurrent writing to the same key
no
support for ttl (time to live) at retrieval
no
support for deleting expired data
no
collocated by feature view
yes
collocated by feature service
no
collocated by entity key
no
write feature values to the online store
yes
read feature values from the online store
yes
update infrastructure (e.g. tables) in the online store
yes
teardown infrastructure (e.g. tables) in the online store
yes
generate a plan of infrastructure changes
no
support for on-demand transforms
yes
readable by Python SDK
yes
readable by Java
yes
readable by Go
yes
support for entityless feature views
yes
support for concurrent writing to the same key
yes
support for ttl (time to live) at retrieval
yes
support for deleting expired data
yes
collocated by feature view
no
collocated by feature service
no
collocated by entity key
yes
write feature values to the online store | yes |
read feature values from the online store | yes |
update infrastructure (e.g. tables) in the online store | yes |
teardown infrastructure (e.g. tables) in the online store | yes |
generate a plan of infrastructure changes | no |
support for on-demand transforms | yes |
readable by Python SDK | yes |
readable by Java | yes |
readable by Go | yes |
support for entityless feature views | yes |
support for concurrent writing to the same key | yes |
support for ttl (time to live) at retrieval | yes |
support for deleting expired data | yes |
collocated by feature view | no |
collocated by feature service | no |
collocated by entity key | yes |
write feature values to the online store | yes |
read feature values from the online store | yes |
update infrastructure (e.g. tables) in the online store | yes |
teardown infrastructure (e.g. tables) in the online store | yes |
generate a plan of infrastructure changes | no |
support for on-demand transforms | yes |
readable by Python SDK | yes |
readable by Java | no |
readable by Go | no |
support for entityless feature views | yes |
support for concurrent writing to the same key | no |
support for ttl (time to live) at retrieval | no |
support for deleting expired data | no |
collocated by feature view | yes |
collocated by feature service | no |
collocated by entity key | no |
Command | Permissions | Resources |
Apply | dynamodb:CreateTable dynamodb:DescribeTable dynamodb:DeleteTable | arn:aws:dynamodb:<region>:<account_id>:table/* |
Materialize | dynamodb.BatchWriteItem | arn:aws:dynamodb:<region>:<account_id>:table/* |
Get Online Features | dynamodb.BatchGetItem | arn:aws:dynamodb:<region>:<account_id>:table/* |
write feature values to the online store | yes |
read feature values from the online store | yes |
update infrastructure (e.g. tables) in the online store | yes |
teardown infrastructure (e.g. tables) in the online store | yes |
generate a plan of infrastructure changes | no |
support for on-demand transforms | yes |
readable by Python SDK | yes |
readable by Java | no |
readable by Go | no |
support for entityless feature views | yes |
support for concurrent writing to the same key | no |
support for ttl (time to live) at retrieval | no |
support for deleting expired data | no |
collocated by feature view | yes |
collocated by feature service | no |
collocated by entity key | no |
The PostgreSQL online store provides support for materializing feature values into a PostgreSQL database for serving online features.
Only the latest feature values are persisted
sslmode, sslkey_path, sslcert_path, and sslrootcert_path are optional
In order to use this online store, you'll need to run pip install 'feast[postgres]'
. You can get started by then running feast init -t postgres
.
The full set of configuration options is available in PostgreSQLOnlineStoreConfig.
The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Postgres online store.
To compare this set of functionality against other online stores, please see the full functionality matrix.
The Bigtable online store provides support for materializing feature values into Cloud Bigtable. The data model used to store feature values in Bigtable is described in more detail here.
In order to use this online store, you'll need to run pip install 'feast[gcp]'
. You can then get started with the command feast init REPO_NAME -t gcp
.
The full set of configuration options is available in BigtableOnlineStoreConfig.
The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Bigtable online store.
To compare this set of functionality against other online stores, please see the full functionality matrix.
The [Cassandra / Astra DB] online store provides support for materializing feature values into an Apache Cassandra / Astra DB database for online features.
The whole project is contained within a Cassandra keyspace
Each feature view is mapped one-to-one to a specific Cassandra table
This implementation inherits all strengths of Cassandra such as high availability, fault-tolerance, and data distribution
In order to use this online store, you'll need to run pip install 'feast[cassandra]'
. You can then get started with the command feast init REPO_NAME -t cassandra
.
The full set of configuration options is available in CassandraOnlineStoreConfig. For a full explanation of configuration options please look at file sdk/python/feast/infra/online_stores/contrib/cassandra_online_store/README.md
.
Storage specifications can be found at docs/specs/online_store_format.md
.
The set of functionality supported by online stores is described in detail here. Below is a matrix indicating which functionality is supported by the Cassandra online store.
To compare this set of functionality against other online stores, please see the full functionality matrix.
Hazelcast online store is in alpha development.
Each feature view is mapped one-to-one to a specific Hazelcast IMap
This implementation inherits all strengths of Hazelcast such as high availability, fault-tolerance, and data distribution.
Secure TSL/SSL connection is supported by Hazelcast online store.
You can set TTL (Time-To-Live) setting for your features in Hazelcast cluster.
Each feature view corresponds to an IMap in Hazelcast cluster and the entries in that IMap corresponds to features of entities. Each feature value stored separately and can be retrieved individually.
In order to use Hazelcast online store, you'll need to run pip install 'feast[hazelcast]'
. You can then get started with the command feast init REPO_NAME -t hazelcast
.
```yaml project: my_feature_repo registry: data/registry.db provider: local online_store: type: hazelcast cluster_name: dev cluster_members: ["localhost:5701"] key_ttl_seconds: 36000 ```
In Alpha Development.
Each document is uniquely identified by its '_id' value. Repeated inserts into the same document '_id' will result in an upsert.
Rockset indexes all columns allowing for quick per feature look up and also allows for a dynamic typed schema that can change based on any new requirements. API Keys can be found in the Rockset console. You can also find host urls on the same tab by clicking "View Region Endpoint Urls".
Data Model Used Per Doc
The MySQL online store provides support for materializing feature values into a MySQL database for serving online features.
Only the latest feature values are persisted
In order to use this online store, you'll need to run pip install 'feast[mysql]'
. You can get started by then running feast init
and then setting the feature_store.yaml
as described below.
Postgres | |
---|---|
Bigtable | |
---|---|
Cassandra | |
---|---|
The online store provides support for materializing feature values into a Hazelcast cluster for serving online features in real-time. In order to use Hazelcast as online store, you need to have a running Hazelcast cluster. You can create a cluster using Hazelcast Viridian Serverless. See this page for more details.
Hazelcast |
---|
To compare this set of functionality against other online stores, please see the full .
The online store provides support for materializing feature values within a Rockset collection in order to serve features in real-time.
The full set of configuration options is available in .
The set of functionality supported by online stores is described in detail . Below is a matrix indicating which functionality is supported by the Mys online store.
Mys |
---|
To compare this set of functionality against other online stores, please see the full .
write feature values to the online store
yes
read feature values from the online store
yes
update infrastructure (e.g. tables) in the online store
yes
teardown infrastructure (e.g. tables) in the online store
yes
generate a plan of infrastructure changes
no
support for on-demand transforms
yes
readable by Python SDK
yes
readable by Java
no
readable by Go
no
support for entityless feature views
yes
support for concurrent writing to the same key
no
support for ttl (time to live) at retrieval
no
support for deleting expired data
no
collocated by feature view
yes
collocated by feature service
no
collocated by entity key
no
write feature values to the online store
yes
read feature values from the online store
yes
update infrastructure (e.g. tables) in the online store
yes
teardown infrastructure (e.g. tables) in the online store
yes
generate a plan of infrastructure changes
no
support for on-demand transforms
yes
readable by Python SDK
yes
readable by Java
no
readable by Go
no
support for entityless feature views
yes
support for concurrent writing to the same key
yes
support for ttl (time to live) at retrieval
no
support for deleting expired data
no
collocated by feature view
yes
collocated by feature service
no
collocated by entity key
yes
write feature values to the online store
yes
read feature values from the online store
yes
update infrastructure (e.g. tables) in the online store
yes
teardown infrastructure (e.g. tables) in the online store
yes
generate a plan of infrastructure changes
yes
support for on-demand transforms
yes
readable by Python SDK
yes
readable by Java
no
readable by Go
no
support for entityless feature views
yes
support for concurrent writing to the same key
no
support for ttl (time to live) at retrieval
no
support for deleting expired data
no
collocated by feature view
yes
collocated by feature service
no
collocated by entity key
no
write feature values to the online store | yes |
read feature values from the online store | yes |
update infrastructure (e.g. tables) in the online store | yes |
teardown infrastructure (e.g. tables) in the online store | yes |
generate a plan of infrastructure changes | no |
support for on-demand transforms | yes |
readable by Python SDK | yes |
readable by Java | no |
readable by Go | no |
support for entityless feature views | yes |
support for concurrent writing to the same key | yes |
support for ttl (time to live) at retrieval | yes |
support for deleting expired data | yes |
collocated by feature view | no |
collocated by feature service | no |
collocated by entity key | yes |
write feature values to the online store | yes |
read feature values from the online store | yes |
update infrastructure (e.g. tables) in the online store | yes |
teardown infrastructure (e.g. tables) in the online store | yes |
generate a plan of infrastructure changes | no |
support for on-demand transforms | yes |
readable by Python SDK | yes |
readable by Java | no |
readable by Go | no |
support for entityless feature views | yes |
support for concurrent writing to the same key | no |
support for ttl (time to live) at retrieval | no |
support for deleting expired data | no |
collocated by feature view | yes |
collocated by feature service | no |
collocated by entity key | no |