For the complete documentation index, see llms.txt. This page is also available as Markdown.

2 — Persistence

The offline store, online store, and registry each need a place to store data. The operator supports two persistence patterns for each:

  • File persistence — a path on a volume (ephemeral or PVC-backed)

  • DB/store persistence — an external database or managed store, wired via a Kubernetes Secret

The pattern is the same for all three services; only the store types differ.


Persistence patterns at a glance

Pattern
Best for
Data survives pod restart?

File — emptyDir

Dev / CI

No

File — PVC (ref)

Single-node prod or testing

Yes (if PVC is retained)

File — PVC (create)

Operator-managed storage

Yes

DB store

Production, HA, multi-pod

Yes


File persistence

Ephemeral (emptyDir)

The default when no persistence block is set. Data lives on the pod's local disk and is lost on restart. Suitable for development only.

services:
  onlineStore:
    server: {}          # no persistence block → emptyDir

PVC — reference an existing PVC

When you already have a PVC provisioned (e.g. by your storage team):

PVC — let the operator create one

Omitting storageClassName uses the cluster default StorageClass. Omitting create entirely creates a PVC with the operator's built-in defaults (1 Gi, default StorageClass).


DB / store persistence

For production, point the operator at an external database. The operator reads connection details from a Kubernetes Secret and writes them into feature_store.yaml.

Secret format

The Secret must contain one key per store component. The key name is the store type (e.g. postgres, sql, redis). The value is a YAML snippet identical to what you would write under the corresponding section in feature_store.yaml, minus the type: key (the operator inserts it from persistence.store.type).

Reference the Secret from the CR:

Key lookup rule: the operator looks up the Secret key that matches persistence.store.type (e.g. type: postgres → key postgres). Keep all stores in one Secret or split across multiple — both work.

Injecting DB credentials into the server pod

The Secret key values in the example above hard-code passwords. For production, keep credentials in a separate Secret and inject them as environment variables using envFrom, then reference them with ${VAR} substitution in the data-stores Secret value:


Store types by component

Online store

type

Secret key

SDK docs

sqlite

sqlite

SQLite

redis

redis

Redis

postgres

postgres

Postgres

cassandra

cassandra

Cassandra

hazelcast

hazelcast

Hazelcast

hbase

hbase

HBase

datastore

datastore

Datastore

dynamodb

dynamodb

DynamoDB

bigtable

bigtable

Bigtable

For all store-specific YAML keys (connection options, pool sizes, etc.) see the linked SDK docs — the Secret value accepts the same keys.

Offline store

type (file)

Notes

file

Default pandas-based parquet offline store

dask

Dask-based parallel parquet

duckdb

DuckDB in-process analytical engine

For external DB-backed offline stores (BigQuery, Snowflake, Spark, Trino, etc.), use persistence.store.type and a Secret with the matching key. See Offline Stores in the SDK docs.

Registry

type

Secret key

Notes

file

(file persistence, no Secret)

SQLite-backed file registry

sql

sql

SQLAlchemy URL — supports PostgreSQL, MySQL, SQLite

snowflake.registry

snowflake.registry

Snowflake-backed registry

For sql, the Secret value is a path: (SQLAlchemy URL) plus optional cache_ttl_seconds and sqlalchemy_config_kwargs:


Common patterns

Redis online store

Postgres for both online store and registry


Overriding the Secret key name

By default the operator looks up the Secret key that matches persistence.store.type (e.g. type: postgres → key postgres). To use a different key, set secretKeyName:

This is useful when a single Secret holds configuration for multiple stores of the same type, or when you want a more descriptive key name.


Validation rules

The operator enforces these rules on Secret values at reconciliation time:

  1. The Secret key value must be valid YAML that deserializes to a map.

  2. If the YAML contains a type field, its value must match the CR's persistence.store.type. Otherwise the operator rejects it with an error. Best practice: omit type from the Secret.

  3. If the YAML contains a registry_type field (for registry stores), the same matching rule applies.

  4. The Secret must exist in the same namespace as the FeatureStore CR.

  5. Only one of file or store may be set under each persistence block (enforced by CRD validation).


Complete Secret examples by store type

Below are copy-paste-ready Secret YAML snippets for every operator-supported store type. Each snippet shows the Secret data key and the YAML value the operator expects.

Note: omit the type field from Secret values — the operator injects it from persistence.store.type. Including a matching type value is tolerated but not recommended.

Online store Secrets

Redis

Redis Cluster with SSL:

CR snippet:

SDK reference: Redis


Postgres

With SSL:

CR snippet:

SDK reference: Postgres


Cassandra

CR snippet:

SDK reference: Cassandra


Snowflake (online)

CR snippet:

SDK reference: Snowflake


DynamoDB

CR snippet:

SDK reference: DynamoDB


Bigtable

CR snippet:

SDK reference: Bigtable


Datastore

CR snippet:

SDK reference: Datastore


MySQL

CR snippet:

SDK reference: MySQL


Hazelcast

CR snippet:

SDK reference: Hazelcast


HBase

CR snippet:

SDK reference: HBase


Other supported online store types

The following types also use the same pattern (persistence.store.type + secretRef). Place the driver-specific YAML keys from the SDK docs under the matching Secret key:

type

Secret key

SDK docs

sqlite

sqlite

SQLite

singlestore

singlestore

SingleStore

elasticsearch

elasticsearch

Elasticsearch

qdrant

qdrant

Qdrant

couchbase.online

couchbase.online

Couchbase

milvus

milvus

Milvus

mongodb

mongodb

MongoDB

hybrid

hybrid

Hybrid


Offline store Secrets

Offline DB stores follow the same pattern. The type field tells the operator which store driver to use; the Secret value holds the connection parameters.

Snowflake (offline)

CR snippet:

SDK reference: Snowflake


BigQuery

CR snippet:

SDK reference: BigQuery


Postgres (offline)

CR snippet:

SDK reference: Postgres


Other supported offline store types

type

Secret key

SDK docs

redshift

redshift

Redshift

spark

spark

Spark

trino

trino

Trino

athena

athena

Athena

mssql

mssql

MSSQL

couchbase.offline

couchbase.offline

Couchbase

clickhouse

clickhouse

ClickHouse

ray

ray

Ray

oracle

oracle

Oracle


Registry Secrets

SQL (SQLAlchemy) registry

The most common production registry. Uses a SQLAlchemy URL to connect to PostgreSQL, MySQL, or SQLite:

CR snippet:

SDK reference: SQL Registry


Snowflake registry

CR snippet:

SDK reference: Snowflake Registry


Multi-store Secret (single Secret for all components)

You can combine all store configurations into a single Secret:


ConfigMap usage (batch engine)

The batchEngine is the only operator component that uses a ConfigMap rather than a Secret for its configuration. The ConfigMap must contain a YAML value under key config (default) or the key specified in configMapKey.

Unlike store Secrets, the batch engine ConfigMap value must include the type field:

See Guide 6 — Batch & Jobs for full details.


Troubleshooting

Symptom
Likely cause
Fix

secret key X doesn't exist in secret Y

The Secret key name doesn't match the store type

Either rename the Secret key to match type, or set secretKeyName in the CR

secret X contains invalid value

The Secret value is not valid YAML

Check indentation and quoting in the stringData value

contains tag named type with value X

The Secret includes a type field that doesn't match the CR's persistence.store.type

Remove type from the Secret value, or correct it to match

invalid secret X for offline store

The referenced Secret doesn't exist

Create the Secret in the same namespace as the FeatureStore CR

One selection required between file or store

Both file and store are set under a persistence block

Keep only one — choose either file persistence or store (DB) persistence


See also

Last updated

Was this helpful?