LogoLogo
v0.26-branch
masterv0.11-branchv0.12-branchv0.13-branchv0.14-branchv0.15-branchv0.16-branchv0.17-branchv0.18-branchv0.19-branchv0.20-branchv0.21-branchv0.22-branchv0.23-branchv0.24-branchv0.25-branchv0.26-branchv0.27-branchv0.28-branchv0.29-branchv0.30-branchv0.31-branchv0.34-branchv0.35-branchv0.36-branchv0.37-branchv0.38-branchv0.39-branchv0.40-branchv0.41-branchv0.42-branchv0.43-branchv0.45-branchv0.45-branchv0.46-branchv0.47-branchv0.48-branch
v0.26-branch
masterv0.11-branchv0.12-branchv0.13-branchv0.14-branchv0.15-branchv0.16-branchv0.17-branchv0.18-branchv0.19-branchv0.20-branchv0.21-branchv0.22-branchv0.23-branchv0.24-branchv0.25-branchv0.26-branchv0.27-branchv0.28-branchv0.29-branchv0.30-branchv0.31-branchv0.34-branchv0.35-branchv0.36-branchv0.37-branchv0.38-branchv0.39-branchv0.40-branchv0.41-branchv0.42-branchv0.43-branchv0.45-branchv0.45-branchv0.46-branchv0.47-branchv0.48-branch
  • Introduction
  • Community & getting help
  • Roadmap
  • Changelog
  • Getting started
    • Quickstart
    • Concepts
      • Overview
      • Data ingestion
      • Entity
      • Feature view
      • Feature retrieval
      • Point-in-time joins
      • Registry
      • [Alpha] Saved dataset
    • Architecture
      • Overview
      • Registry
      • Offline store
      • Online store
      • Batch Materialization Engine
      • Provider
    • Third party integrations
    • FAQ
  • Tutorials
    • Sample use-case tutorials
      • Driver ranking
      • Fraud detection on GCP
      • Real-time credit scoring on AWS
      • Driver stats on Snowflake
    • Validating historical features with Great Expectations
    • Using Scalable Registry
    • Building streaming features
  • How-to Guides
    • Running Feast with Snowflake/GCP/AWS
      • Install Feast
      • Create a feature repository
      • Deploy a feature store
      • Build a training dataset
      • Load data into the online store
      • Read features from the online store
      • Scaling Feast
      • Structuring Feature Repos
    • Running Feast in production (e.g. on Kubernetes)
    • Upgrading for Feast 0.20+
    • Customizing Feast
      • Adding a custom batch materialization engine
      • Adding a new offline store
      • Adding a new online store
      • Adding a custom provider
    • Adding or reusing tests
  • Reference
    • Codebase Structure
    • Type System
    • Data sources
      • Overview
      • File
      • Snowflake
      • BigQuery
      • Redshift
      • Push
      • Kafka
      • Kinesis
      • Spark (contrib)
      • PostgreSQL (contrib)
      • Trino (contrib)
      • Azure Synapse + Azure SQL (contrib)
    • Offline stores
      • Overview
      • File
      • Snowflake
      • BigQuery
      • Redshift
      • Spark (contrib)
      • PostgreSQL (contrib)
      • Trino (contrib)
      • Azure Synapse + Azure SQL (contrib)
    • Online stores
      • Overview
      • SQLite
      • Snowflake
      • Redis
      • Datastore
      • DynamoDB
      • Bigtable
      • PostgreSQL (contrib)
      • Cassandra + Astra DB (contrib)
      • MySQL (contrib)
    • Providers
      • Local
      • Google Cloud Platform
      • Amazon Web Services
      • Azure
    • Batch Materialization Engines
      • Bytewax
      • Snowflake
      • AWS Lambda (alpha)
      • Spark (contrib)
    • Feature repository
      • feature_store.yaml
      • .feastignore
    • Feature servers
      • Python feature server
      • [Alpha] Go feature server
      • [Alpha] AWS Lambda feature server
    • [Beta] Web UI
    • [Alpha] On demand feature view
    • [Alpha] Data quality monitoring
    • Feast CLI reference
    • Python API reference
    • Usage
  • Project
    • Contribution process
    • Development guide
    • Backwards Compatibility Policy
      • Maintainer Docs
    • Versioning policy
    • Release process
    • Feast 0.9 vs Feast 0.10+
Powered by GitBook
On this page
  • Description
  • Example

Was this helpful?

Edit on GitHub
Export as PDF
  1. Reference
  2. Batch Materialization Engines

Spark (contrib)

PreviousAWS Lambda (alpha)NextFeature repository

Last updated 2 years ago

Was this helpful?

Description

The Spark batch materialization engine is considered alpha status. It relies on the offline store to output feature values to S3 via to_remote_storage, and then loads them into the online store.

See for configuration options.

Example

feature_store.yaml
...
offline_store:
  type: snowflake.offline
...
batch_engine:
  type: spark.engine
  partitions: [optional num partitions to use to write to online store]
SparkMaterializationEngine