This guide will go over:
how Feast tests are setup
how to extend the test suite to test new functionality
how to use the existing test suite to test a new custom offline / online store
Unit tests are contained in sdk/python/tests/unit
. Integration tests are contained in sdk/python/tests/integration
. Let's inspect the structure of sdk/python/tests/integration
:
feature_repos
has setup files for most tests in the test suite.
conftest.py
(in the parent directory) contains the most common fixtures, which are designed as an abstraction on top of specific offline/online stores, so tests do not need to be rewritten for different stores. Individual test files also contain more specific fixtures.
The tests are organized by which Feast component(s) they test.
The universal feature repo refers to a set of fixtures (e.g. environment
and universal_data_sources
) that can be parametrized to cover various combinations of offline stores, online stores, and providers. This allows tests to run against all these various combinations without requiring excess code. The universal feature repo is constructed by fixtures in conftest.py
with help from the various files in feature_repos
.
Tests in Feast are split into integration and unit tests. If a test requires external resources (e.g. cloud resources on GCP or AWS), it is an integration test. If a test can be run purely locally (where locally includes Docker resources), it is a unit test.
Integration tests test non-local Feast behavior. For example, tests that require reading data from BigQuery or materializing data to DynamoDB are integration tests. Integration tests also tend to involve more complex Feast functionality.
Unit tests test local Feast behavior. For example, tests that only require registering feature views are unit tests. Unit tests tend to only involve simple Feast functionality.
E2E tests
E2E tests test end-to-end functionality of Feast over the various codepaths (initialize a feature store, apply, and materialize).
The main codepaths include:
basic e2e tests for offline stores
test_universal_e2e.py
go feature server
test_go_feature_server.py
python http server
test_python_feature_server.py
data quality monitoring feature validation
test_validation.py
Offline and Online Store Tests
Offline and online store tests mainly test for the offline and online retrieval functionality.
The various specific functionalities that are tested include:
push API tests
test_push_features_to_offline_store.py
test_push_features_to_online_store.py
test_offline_write.py
historical retrieval tests
test_universal_historical_retrieval.py
online retrieval tests
test_universal_online.py
data quality monitoring feature logging tests
test_feature_logging.py
online store tests
test_universal_online.py
Registration Tests
The registration folder contains all of the registry tests and some universal cli tests. This includes:
CLI Apply and Materialize tests tested against on the universal test suite
Data type inference tests
Registry tests
Miscellaneous Tests
AWS Lambda Materialization Tests (Currently do not work)
test_lambda.py
Registry Diff Tests
These are tests for the infrastructure and registry diff functionality that Feast uses to determine if changes to the registry or infrastructure is needed.
Local CLI Tests and Local Feast Tests
These tests test all of the cli commands against the local file offline store.
Infrastructure Unit Tests
DynamoDB tests with dynamo mocked out
Repository configuration tests
Schema inference unit tests
Key serialization tests
Basic provider unit tests
Feature Store Validation Tests
These test mainly contain class level validation like hashing tests, protobuf and class serialization, and error and warning handling.
Data source unit tests
Feature service unit tests
Feature service, feature view, and feature validation tests
Protobuf/json tests for Feast ValueTypes
Serialization tests
Type mapping
Feast types
Serialization tests due to this issue
Docstring tests are primarily smoke tests to make sure imports and setup functions can be executed without errors.
Let's look at a sample test using the universal repo:
The key fixtures are the environment
and universal_data_sources
fixtures, which are defined in the feature_repos
directories and the conftest.py
file. This by default pulls in a standard dataset with driver and customer entities (that we have pre-defined), certain feature views, and feature values.
The environment
fixture sets up a feature store, parametrized by the provider and the online/offline store. It allows the test to query against that feature store without needing to worry about the underlying implementation or any setup that may be involved in creating instances of these datastores.
Each fixture creates a different integration test with its own IntegrationTestRepoConfig
which is used by pytest to generate a unique test testing one of the different environments that require testing.
Feast tests also use a variety of markers:
The @pytest.mark.integration
marker is used to designate integration tests which will cause the test to be run when you call make test-python-integration
.
The @pytest.mark.universal_offline_stores
marker will parametrize the test on all of the universal offline stores including file, redshift, bigquery and snowflake.
The full_feature_names
parametrization defines whether or not the test should reference features as their full feature name (fully qualified path) or just the feature name itself.
Use the same function signatures as an existing test (e.g. use environment
and universal_data_sources
as an argument) to include the relevant test fixtures.
If possible, expand an individual test instead of writing a new test, due to the cost of starting up offline / online stores.
Use the universal_offline_stores
and universal_online_store
markers to parametrize the test against different offline store and online store combinations. You can also designate specific online and offline stores to test by using the only
parameter on the marker.
Install Feast in editable mode with pip install -e
.
The core tests for offline / online store behavior are parametrized by the FULL_REPO_CONFIGS
variable defined in feature_repos/repo_configuration.py
. To overwrite this variable without modifying the Feast repo, create your own file that contains a FULL_REPO_CONFIGS
(which will require adding a new IntegrationTestRepoConfig
or two) and set the environment variable FULL_REPO_CONFIGS_MODULE
to point to that file. Then the core offline / online store tests can be run with make test-python-universal
.
See the custom offline store demo and the custom online store demo for examples.
Many problems arise when implementing your data store's type conversion to interface with Feast datatypes.
You will need to correctly update inference.py
so that Feast can infer your datasource schemas
You also need to update type_map.py
so that Feast knows how to convert your datastores types to Feast-recognized types in feast/types.py
.
The most important functionality in Feast is historical and online retrieval. Most of the e2e and universal integration test test this functionality in some way. Making sure this functionality works also indirectly asserts that reading and writing from your datastore works as intended.
Extend data_source_creator.py
for your offline store.
In repo_configuration.py
add a new IntegrationTestRepoConfig
or two (depending on how many online stores you want to test).
Generally, you should only need to test against sqlite. However, if you need to test against a production online store, then you can also test against Redis or dynamodb.
Run the full test suite with make test-python-integration.
This folder is for plugins that are officially maintained with community owners. Place the APIs in feast/infra/offline_stores/contrib/
.
Extend data_source_creator.py
for your offline store and implement the required APIs.
In contrib_repo_configuration.py
add a new IntegrationTestRepoConfig
(depending on how many online stores you want to test).
Run the test suite on the contrib test suite with make test-python-contrib-universal
.
In repo_configuration.py
add a new config that maps to a serialized version of configuration you need in feature_store.yaml
to setup the online store.
In repo_configuration.py
, add new IntegrationTestRepoConfig
for online stores you want to test.
Run the full test suite with make test-python-integration
Check test_universal_types.py
for an example of how to do this.
Install Redis on your computer. If you are a mac user, you should be able to brew install redis
.
Running redis-server --help
and redis-cli --help
should show corresponding help menus.
Run ./infra/scripts/redis-cluster.sh start
then ./infra/scripts/redis-cluster.sh create
to start the Redis cluster locally. You should see output that looks like this:
You should be able to run the integration tests and have the Redis cluster tests pass.
If you would like to run your own Redis cluster, you can run the above commands with your own specified ports and connect to the newly configured cluster.
To stop the cluster, run ./infra/scripts/redis-cluster.sh stop
and then ./infra/scripts/redis-cluster.sh clean
.