The Bytewax materialization engine loads authentication and cluster information from the kubeconfig file. By default, kubectl looks for a file named
$HOME/.kube directory. You can specify other kubeconfig files by setting the
To configure secrets, first create them using
kubectl create secret generic -n bytewax aws-credentials --from-literal=aws-access-key-id='<access key id>' --from-literal=aws-secret-access-key='<secret access key>'
If your Docker registry requires authentication to store/pull containers, you can use this same approach to store your repository access credential and use when running the materialization engine.
Then configure them in the batch_engine section of
- name: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
The Bytewax materialization engine is configured through the The
# example annotation you might include if running on AWS EKS
iam.amazonaws.com/role: arn:aws:iam::<account number>:role/MyBytewaxPlatformRole
image_pull_secretsconfiguration directive specifies the pre-configured secret to use when pulling the image container from your registry.
service_account_namespecifies which Kubernetes service account to run the job under.
include_security_context_capabilitiesflag indicates whether or not
"drop": ["ALL"]are included in the job & pod security context capabilities.
annotationsallows you to include additional Kubernetes annotations to the job. This is particularly useful for IAM roles which grant the running pod access to cloud platform resources (for example).
imageconfiguration directive specifies which container image to use when running the materialization job. To create a custom image based on this container, run the following command:
DOCKER_BUILDKIT=1 docker build . -f ./sdk/python/feast/infra/materialization/contrib/bytewax/Dockerfile -t <image tag>
Once that image is built and pushed to a registry, it can be specified as a part of the batch engine configuration:
image: <image tag>