Adding a new online store
Last updated
Was this helpful?
Last updated
Was this helpful?
Feast makes adding support for a new online store (database) easy. Developers can simply implement the interface to add support for a new store (other than the existing stores like Redis, DynamoDB, SQLite, and Datastore).
In this guide, we will show you how to integrate with MySQL as an online store. While we will be implementing a specific store, this guide should be representative for adding support for any new online store.
The full working code for this guide can be found at .
The process of using a custom online store consists of 3 steps:
Defining the OnlineStore
class.
Defining the OnlineStoreConfig
class.
Referencing the OnlineStore
in a feature repo's feature_store.yaml
file.
The OnlineStore class broadly contains two sets of methods
One set deals with managing infrastructure that the online store needed for operations
One set deals with writing data into the store, and reading data from the store.
There are two methods that deal with managing infrastructure for online stores, update
and teardown
update
is invoked when users run feast apply
as a CLI command, or the FeatureStore.apply()
sdk method.
The update
method should be used to perform any operations necessary before data can be written to or read from the store. The update
method can be used to create MySQL tables in preparation for reads and writes to new feature views.
teardown
is invoked when users run feast teardown
or FeatureStore.teardown()
.
The teardown
method should be used to perform any clean-up operations. teardown
can be used to drop MySQL indices and tables corresponding to the feature views being deleted.
There are two methods that deal with writing data to and from the online stores.online_write_batch
and online_read
.
online_write_batch
is invoked when running materialization (using the feast materialize
or feast materialize-incremental
commands, or the corresponding FeatureStore.materialize()
method.
online_read
is invoked when reading values from the online store using the FeatureStore.get_online_features()
method.
Additional configuration may be needed to allow the OnlineStore to talk to the backing store. For example, MySQL may need configuration information like the host at which the MySQL instance is running, credentials for connecting to the database, etc.
This config class must container a type
field, which contains the fully qualified class name of its corresponding OnlineStore class.
Additionally, the name of the config class must be the same as the OnlineStore class, with the Config
suffix.
An example of the config class for MySQL :
This configuration can be specified in the feature_store.yaml
as follows:
This configuration information is available to the methods of the OnlineStore, via theconfig: RepoConfig
parameter which is passed into all the methods of the OnlineStore interface, specifically at the config.online_store
field of the config
parameter.
After implementing both these classes, the custom online store can be used by referencing it in a feature repo's feature_store.yaml
file, specifically in the online_store
field. The value specified should be the fully qualified class name of the OnlineStore.
As long as your OnlineStore class is available in your Python environment, it will be imported by Feast dynamically at runtime.
To use our MySQL online store, we can use the following feature_store.yaml
:
If additional configuration for the online store is not required, then we can omit the other fields and only specify the type
of the online store class as the value for the online_store
.
To facilitate configuration, all OnlineStore implementations are required to also define a corresponding OnlineStoreConfig class in the same file. This OnlineStoreConfig class should inherit from the FeastConfigBaseModel
class, which is defined .
The FeastConfigBaseModel
is a class, which parses yaml configuration into python objects. Pydantic also allows the model classes to define validators for the config classes, to make sure that the config classes are correctly defined.