Feast uses an internal type system to provide guarantees on training and serving data. Feast currently supports eight primitive types -
UNIX_TIMESTAMP- and the corresponding array types. Null types are not supported, although the
UNIX_TIMESTAMPtype is nullable. The type system is controlled by
Value.protoin protobuf and by
types.pyin Python. Type conversion logic can be found in
feast apply, Feast runs schema inference on the data sources underlying feature views. For example, if the
schemaparameter is not specified for a feature view, Feast will examine the schema of the underlying data source to determine the event timestamp column, feature columns, and entity columns. Each of these columns must be associated with a Feast type, which requires conversion from the data source type system to the Feast type system.
- The feature inference logic calls
source_datatype_to_feast_value_typecals the appropriate method in
type_map.py. For example, if a
SnowflakeSourceis being examined,
type_map.pywill be called.
Feast serves feature values as
Valueproto objects, which have a type corresponding to Feast types. Thus Feast must materialize feature values into the online store as
- The local materialization engine first pulls the latest historical features and converts it to pyarrow.
- Then it calls
_convert_arrow_to_prototo convert the pyarrow table to proto format.
- This calls
type_map.pyto perform the type conversion.
The Feast type system is typically not necessary when retrieving historical features. A call to
get_historical_featureswill return a
RetrievalJobobject, which allows the user to export the results to one of several possible locations: a Pandas dataframe, a pyarrow table, a data lake (e.g. S3 or GCS), or the offline store (e.g. a Snowflake table). In all of these cases, the type conversion is handled natively by the offline store. For example, a BigQuery query exposes a
to_dataframemethod that will automatically convert the result to a dataframe, without requiring any conversions within Feast.
As mentioned above in the section on materialization, Feast persists feature values into the online store as
Valueproto objects. A call to
get_online_featureswill return an
OnlineResponseobject, which essentially wraps a bunch of
Valueprotos with some metadata. The
OnlineResponseobject can then be converted into a Python dictionary, which calls
type_map.py, a utility that converts the Feast internal types to Python native types.