[Alpha] Data quality monitoring

Data Quality Monitoring (DQM) is a Feast module aimed to help users to validate their data with the user-curated set of rules. Validation could be applied during:
Historical retrieval (training dataset generation)
[planned] Writing features into an online store
[planned] Reading features from an online store
Its goal is to address several complex data problems, namely:
Data consistency - new training datasets can be significantly different from previous datasets. This might require a change in model architecture.
Issues/bugs in the upstream pipeline - bugs in upstream pipelines can cause invalid values to overwrite existing valid values in an online store.
Training/serving skew - distribution shift could significantly decrease the performance of the model.
To monitor data quality, we check that the characteristics of the tested dataset (aka the tested dataset's profile) are "equivalent" to the characteristics of the reference dataset. How exactly profile equivalency should be measured is up to the user.
Overview
The validation process consists of the following steps:
1.User prepares reference dataset (currently only saved datasets from historical retrieval are supported).
2.User defines profiler function, which should produce profile by given dataset (currently only profilers based on Great Expectations are allowed).
3.Validation of tested dataset is performed with reference dataset and profiler provided as parameters.
Preparations
Feast with Great Expectations support can be installed via
1
pip install 'feast[ge]'
Copied!
Dataset profile
Currently, Feast supports only Great Expectation's ExpectationSuite as dataset's profile. Hence, the user needs to define a function (profiler) that would receive a dataset and return an ExpectationSuite.
Great Expectations supports automatic profiling as well as manually specifying expectations:
1
from great_expectations.dataset import Dataset
2
from great_expectations.core.expectation_suite import ExpectationSuite
3
​
4
from feast.dqm.profilers.ge_profiler import ge_profiler
5
​
6
@ge_profiler
7
def automatic_profiler(dataset: Dataset) -> ExpectationSuite:
8
    from great_expectations.profile.user_configurable_profiler import UserConfigurableProfiler
9
​
10
    return UserConfigurableProfiler(
11
        profile_dataset=dataset,
12
        ignored_columns=['conv_rate'],
13
        value_set_threshold='few'
14
    ).build_suite()
Copied!
However, from our experience capabilities of automatic profiler are quite limited. So we would recommend crafting your own expectations:
1
@ge_profiler
2
def manual_profiler(dataset: Dataset) -> ExpectationSuite:
3
    dataset.expect_column_max_to_be_between("column", 1, 2)
4
    return dataset.get_expectation_suite()
Copied!
Validating Training Dataset
During retrieval of historical features, validation_reference can be passed as a parameter to methods .to_df(validation_reference=...) or .to_arrow(validation_reference=...) of RetrievalJob. If parameter is provided Feast will run validation once dataset is materialized. In case if validation successful materialized dataset is returned. Otherwise, feast.dqm.errors.ValidationFailed exception would be raised. It will consist of all details for expectations that didn't pass.
1
from feast import FeatureStore
2
​
3
fs = FeatureStore(".")
4
​
5
job = fs.get_historical_features(...)
6
job.to_df(
7
    validation_reference=fs
8
        .get_saved_dataset("my_reference_dataset")
9
        .as_reference(profiler=manual_profiler)
10
)
Copied!

Reference - Previous

[Alpha] Web UI

Next - Reference

[Alpha] On demand feature view

Last modified 7d ago

Export as PDF

Copy link

Edit on GitHub

Contents

Overview

Preparations

Dataset profile

Validating Training Dataset