Migration from v0.6 to v0.7

Feast Core Validation changes

In v0.7, Feast Core no longer accepts starting with number (0-9) and using dash in names for:

• Project

• Feature Set

• Entities

• Features

Migrate all project, feature sets, entities, feature names:

• with ‘-’ by recreating them with '-' replace with '_'

• recreate any names with a number (0-9) as the first letter to one without.

Feast now prevents feature sets from being applied if no store is subscribed to that Feature Set.

• Ensure that a store is configured to subscribe to the Feature Set before applying the Feature Set.

Feast Core's Job Coordinator is now Feast Job Controller

In v0.7, Feast Core's Job Coordinator has been decoupled from Feast Core and runs as a separate Feast Job Controller application. See its Configuration reference for how to configure Feast Job Controller.

Ingestion Job API

In v0.7, the following changes are made to the Ingestion Job API:

• Changed List Ingestion Job API to return list of FeatureSetReference instead of list of FeatureSet in response.

• Moved ListIngestionJobs, StopIngestionJob, RestartIngestionJob calls from CoreService to JobControllerService.

• Python SDK/CLI: Added new Job Controller client and jobcontroller_url config option.

Users of the Ingestion Job API via gRPC should migrate by:

• Add new client to connect to Job Controller endpoint to call JobControllerService and call ListIngestionJobs, StopIngestionJob, RestartIngestionJob from new client.

• Migrate code to accept feature references instead of feature sets returned in ListIngestionJobs response.

Users of Ingestion Job via Python SDK (ie feast ingest-jobs list or client.stop_ingest_job() etc.) should migrate by:

• ingest_job()methods only: Create a new separate Job Controller client to connect to the job controller and call ingest_job() methods using the new client.

• Configure the Feast Job Controller endpoint url via jobcontroller_url config option.

Configuration Properties Changes

• Rename feast.jobs.consolidate-jobs-per-source property to feast.jobs.controller.consolidate-jobs-per-sources

• Renamefeast.security.authorization.options.subjectClaim to feast.security.authentication.options.subjectClaim

• Rename feast.logging.audit.messageLoggingEnabled to feast.audit.messageLogging.enabled

Migration from v0.5 to v0.6

Database schema

In Release 0.6 we introduced Flyway to handle schema migrations in PostgreSQL. Flyway is integrated into core and for now on all migrations will be run automatically on core start. It uses table flyway_schema_history in the same database (also created automatically) to keep track of already applied migrations. So no specific maintenance should be needed.

If you already have existing deployment of feast 0.5 - Flyway will detect existing tables and omit first baseline migration.

After core started you should have flyway_schema_history look like this

>> select version, description, script, checksum from flyway_schema_history

version |              description                |                          script         |  checksum
--------+-----------------------------------------+-----------------------------------------+------------
 1       | << Flyway Baseline >>                   | << Flyway Baseline >>                   | 
 2       | RELEASE 0.6 Generalizing Source AND ... | V2__RELEASE_0.6_Generalizing_Source_... | 1537500232

In this release next major schema changes were done:

• Source is not shared between FeatureSets anymore. It's changed to 1:1 relation

  and source's primary key is now auto-incremented number.

• Due to generalization of Source sources.topics & sources.bootstrap_servers columns were deprecated.

  They will be replaced with sources.config. Data migration handled by code when respected Source is used.

  topics and bootstrap_servers will be deleted in the next release.

• Job (table jobs) is no longer connected to Source (table sources) since it uses consolidated source for optimization purposes.

  All data required by Job would be embedded in its table.

New Models (tables):

• feature_statistics

Minor changes:

• FeatureSet has new column version (see proto for details)

• Connecting table jobs_feature_sets in many-to-many relation between jobs & feature sets

  has now version and delivery_status.

Migration from v0.4 to v0.6

Database

For all versions earlier than 0.5 seamless migration is not feasible due to earlier breaking changes and creation of new database will be required.

Since database will be empty - first (baseline) migration would be applied:

>> select version, description, script, checksum from flyway_schema_history

version |              description                |                          script         |  checksum
--------+-----------------------------------------+-----------------------------------------+------------
 1       | Baseline                                | V1__Baseline.sql                        | 1091472110
 2       | RELEASE 0.6 Generalizing Source AND ... | V2__RELEASE_0.6_Generalizing_Source_... | 1537500232

Overview

Feast Components export metrics that can provide insight into Feast behavior:

• Feast Ingestion Jobs can be configured to push metrics into StatsD

• Prometheus can be configured to scrape metrics from Feast Core and Serving.

See the Metrics Reference for documentation on metrics are exported by Feast.

Pushing Ingestion Metrics to StatsD

Feast Ingestion Job

Feast Ingestion Job can be configured to push Ingestion metrics to a StatsD instance. Metrics export to StatsD for Ingestion Job is configured in Job Controller's application.yml under feast.jobs.metrics

 feast:
   jobs:
    metrics:
      # Enables Statd metrics export if true.
      enabled: true
      type: statsd
      # Host and port of the StatsD instance to export to.
      host: localhost
      port: 9125

Exporting Feast Metrics to Prometheus

Feast Core and Serving

Feast Core and Serving exports metrics to a Prometheus instance via Prometheus scraping its /metrics endpoint. Metrics export to Prometheus for Core and Serving can be configured via their corresponding application.yml

server:
  # Configures the port where metrics are exposed via /metrics for Prometheus to scrape.
  port: 8081

Direct Prometheus to scrape directly from Core and Serving's /metrics endpoint.

Further Reading

See the Metrics Reference for documentation on metrics are exported by Feast.

Overview

Feast supports the following security methods:

• SSL/TLS on messaging between Feast Core, Feast Online Serving and Feast SDKs.

• Authentication to Feast Core and Serving based on Open ID Connect ID tokens.

• Authorization based on project membership and delegating authorization grants to external Authorization Server.

Important considerations when integrating Authentication/Authorization.

SSL/TLS

Feast supports SSL/TLS encrypted inter-service communication among Feast Core, Feast Online Serving, and Feast SDKs.

Configuring SSL/TLS on Feast Core and Feast Serving

The following properties configure SSL/TLS. These properties are located in their corresponding application.ymlfiles:

Read more on enabling SSL/TLS in the gRPC starter docs.

Configuring SSL/TLS on Python SDK/CLI

To enable SSL/TLS in the Feast Python SDK or Feast CLI, set the config options via feast config:

Configuring SSL/TLS on Go SDK

Configure SSL/TLS on the Go SDK by passing configuration via SecurityConfig:

cli, err := feast.NewSecureGrpcClient("localhost", 6566, feast.SecurityConfig{
    EnableTLS: true,
         TLSCertPath: "/path/to/cert.pem",
})Option

Configuring SSL/TLS on Java SDK

Configure SSL/TLS on the Feast Java SDK by passing configuration via SecurityConfig:

FeastClient client = FeastClient.createSecure("localhost", 6566, 
    SecurityConfig.newBuilder()
      .setTLSEnabled(true)
      .setCertificatePath(Optional.of("/path/to/cert.pem"))
      .build());

Authentication

Authentication can be implemented to identify and validate client requests to Feast Core and Feast Online Serving. Currently, Feast uses Open ID Connect (OIDC) ID tokens (i.e. Google Open ID Connect) to authenticate client requests.

Configuring Authentication in Feast Core and Feast Online Serving

Authentication can be configured for Feast Core and Feast Online Serving via properties in their corresponding application.yml files:

Behind the scenes, Feast Core and Feast Online Serving authenticate by:

• Extracting the OIDC ID token TOKENfrom gRPC metadata submitted with request:

('authorization', 'Bearer: TOKEN')

• Validates token's authenticity using the JWK retrieved from the jwkEndpointURI

Authenticating Serving with Feast Core

Feast Online Serving communicates with Feast Core during normal operation. When both authentication and authorization are enabled on Feast Core, Feast Online Serving is forced to authenticate its requests to Feast Core. Otherwise, Feast Online Serving produces an Authentication failure error when connecting to Feast Core.

Properties used to configure Serving authentication via application.yml:

Enabling Authentication in Python SDK/CLI

Configure the Feast Python SDK and Feast CLI to use authentication via feast config:

$ feast config set enable_auth true

$ gcloud auth application-default login

$ export GOOGLE_APPLICATION_CREDENTIALS="path/to/key.json"

Enabling Authentication in Go SDK

Configure the Feast Java SDK to use authentication by specifying the credential via SecurityConfig:

// error handling omitted.
// Use Google Credential as provider.
cred, _ := feast.NewGoogleCredential("localhost:6566")
cli, _ := feast.NewSecureGrpcClient("localhost", 6566, feast.SecurityConfig{
  // Specify the credential to provide tokens for Feast Authentication.  
    Credential: cred, 
})

$ export GOOGLE_APPLICATION_CREDENTIALS="path/to/key.json"

cred, _ := feast.NewGoogleCredential("localhost:6566")

cred := feast.NewOAuthCredential("localhost:6566", "client_id", "secret", "https://oauth.endpoint/auth")

Enabling Authentication in Java SDK

Configure the Feast Java SDK to use authentication by setting credentials via SecurityConfig:

// Use GoogleAuthCredential as provider.
CallCredentials credentials = new GoogleAuthCredentials(
    Map.of("audience", "localhost:6566"));

FeastClient client = FeastClient.createSecure("localhost", 6566, 
    SecurityConfig.newBuilder()
      // Specify the credentials to provide tokens for Feast Authentication.  
      .setCredentials(Optional.of(creds))
      .build());

$ export GOOGLE_APPLICATION_CREDENTIALS="path/to/key.json"

CallCredentials credentials = new GoogleAuthCredentials(
    Map.of("audience", "localhost:6566"));

CallCredentials credentials = new OAuthCredentials(Map.of(
  "audience": "localhost:6566",
  "grant_type", "client_credentials",
  "client_id", "some_id",
  "client_id", "secret",
  "oauth_url", "https://oauth.endpoint/auth",
  "jwkEndpointURI", "https://jwk.endpoint/jwk"));

Authorization

Authorization provides access control to FeatureTables and/or Features based on project membership. Users who are members of a project are authorized to:

• Create and/or Update a Feature Table in the Project.

• Retrieve Feature Values for Features in that Project.

Authorization API/Server

Feast delegates Authorization grants to an external Authorization Server that implements the Authorization Open API specification.

• Feast checks whether a user is authorized to make a request by making a checkAccessRequest to the Authorization Server.

• The Authorization Server should return a AuthorizationResult with whether the user is allowed to make the request.

Authorization can be configured for Feast Core and Feast Online Serving via properties in their corresponding application.yml

Authentication & Authorization

When using Authentication & Authorization, consider:

• Enabling Authentication without Authorization makes authentication optional. You can still send unauthenticated requests.

• Enabling Authorization forces all requests to be authenticated. Requests that are not authenticated are dropped.

If at any point in time you cannot resolve a problem, please see the section for reaching out to the Feast community.

How can I verify that all services are operational?

Docker Compose

The containers should be in an up state:

Google Kubernetes Engine

All services should either be in a RUNNING state or COMPLETEDstate:

How can I verify that I can connect to all services?

First locate the the host and port of the Feast Services.

Docker Compose (from inside the docker network)

You will probably need to connect using the hostnames of services and standard Feast ports:

Docker Compose (from outside the docker network)

You will probably need to connect using localhost and standard ports:

Google Kubernetes Engine (GKE)

You will need to find the external IP of one of the nodes as well as the NodePorts. Please make sure that your firewall is open for these ports:

Testing Connectivity From Feast Services:

Use grpc_cli to test connetivity by listing the gRPC methods exposed by Feast services:

How can I print logs from the Feast Services?

Feast will typically have three services that you need to monitor if something goes wrong.

• Feast Core

• Feast Job Controller

• Feast Serving (Online)

• Feast Serving (Batch)

In order to print the logs from these services, please run the commands below.

Docker Compose

Use docker-compose logs to obtain Feast component logs:

Google Kubernetes Engine

Use kubectl logs to obtain Feast component logs:

Introduction

Feast provides audit logging functionality in order to debug problems and to trace the lineage of events.

Audit Log Types

Audit Logs produced by Feast come in three favors:

Configuration

JSON Format

Audit Logs produced by Feast are written to the console similar to normal logs but in a structured, machine parsable JSON. Example of a Message Audit Log JSON entry produced:

Log Entry Schema

Fields common to all Audit Log Types:

Fields in Message Audit Log Type

Fields in Action Audit Log Type

Fields in Transition Audit Log Type

Log Forwarder

Feast currently only supports forwarding Request/Response (Message Audit Log Type) logs to an external fluentD service with feast.** Fluentd tag.

Request/Response Log Example

Configuration

The Fluentd Log Forwarder configured with the with the following configuration options in application.yml:

When using Fluentd as the Log forwarder, a Feast release_name can be logged instead of the IP address (eg. IP of Kubernetes pod deployment), by setting an environment variable RELEASE_NAME when deploying Feast.