The Bytewax batch materialization engine provides an execution engine for batch materializing operations (materialize
and materialize-incremental
).
In order to use the Bytewax materialization engine, you will need a Kubernetes cluster running version 1.22.10 or greater.
The Bytewax materialization engine loads authentication and cluster information from the kubeconfig file. By default, kubectl looks for a file named config
in the $HOME/.kube directory
. You can specify other kubeconfig files by setting the KUBECONFIG
environment variable.
Bytewax jobs can be configured to access Kubernetes secrets as environment variables to access online and offline stores during job runs.
To configure secrets, first create them using kubectl
:
Then configure them in the batch_engine section of feature_store.yaml
:
The Bytewax materialization engine is configured through the The feature_store.yaml
configuration file:
The namespace
configuration directive specifies which Kubernetes namespace jobs, services and configuration maps will be created in.
The image
configuration directive specifies which container image to use when running the materialization job. To create a custom image based on this container, run the following command:
Once that image is built and pushed to a registry, it can be specified as a part of the batch engine configuration:
Please see for an explanation of batch materialization engines.
The batch materialization engine provides a highly scalable and parallel execution engine using a Snowflake Warehouse for batch materializations operations (materialize
and materialize-incremental
) when using a SnowflakeSource
.
The engine requires no additional configuration other than for you to supply Snowflake's standard login and context details. The engine leverages custom (automatically deployed for you) Python UDFs to do the proper serialization of your offline store data to your online serving tables.
When using all three options together, snowflake.offline
, snowflake.engine
, and snowflake.online
, you get the most unique experience of unlimited scale and performance + governance and data security.