[Alpha] AWS Lambda feature server

Search…
Warning: This is an experimental feature. It's intended for early testing and feedback, and could change without warnings in future releases.
To enable this feature, run feast alpha enable aws_lambda_feature_server
Overview
The AWS Lambda feature server is an HTTP endpoint that serves features with JSON I/O, deployed as a Docker image through AWS Lambda and AWS API Gateway. This enables users to get features from Feast using any programming language that can make HTTP requests. A local feature server is also available. A remote feature server on GCP Cloud Run is currently being developed.
Deployment
The AWS Lambda feature server is only available to projects using the AwsProvider with registries on S3. It is disabled by default. To enable it, feature_store.yaml must be modified; specifically, the enable flag must be on and an execution_role_name must be specified. For example, after running feast init -t aws, changing the registry to be on S3, and enabling the feature server, the contents of feature_store.yaml should look similar to the following:
1
project: dev
2
registry: s3://feast/registries/dev
3
provider: aws
4
online_store:
5
  region: us-west-2
6
offline_store:
7
  cluster_id: feast
8
  region: us-west-2
9
  user: admin
10
  database: feast
11
  s3_staging_location: s3://feast/redshift/tests/staging_location
12
  iam_role: arn:aws:iam::{aws_account}:role/redshift_s3_access_role
13
flags:
14
  alpha_features: true
15
  aws_lambda_feature_server: true
16
feature_server:
17
  enabled: True
18
  execution_role_name: arn:aws:iam::{aws_account}:role/lambda_execution_role
Copied!
If enabled, the feature server will be deployed during feast apply. After it is deployed, the feast endpoint CLI command will indicate the server's endpoint.
Permissions
Feast requires the following permissions in order to deploy and teardown AWS Lambda feature server:
Permissions
Resources
lambda:CreateFunction
lambda:GetFunction
lambda:DeleteFunction
lambda:AddPermission
lambda:UpdateFunctionConfiguration
arn:aws:lambda:<region>:<account_id>:function:feast-*
ecr:CreateRepository
ecr:DescribeRepositories
ecr:DeleteRepository
ecr:PutImage
ecr:DescribeImages
ecr:BatchDeleteImage
ecr:CompleteLayerUpload
ecr:UploadLayerPart
ecr:InitiateLayerUpload
ecr:BatchCheckLayerAvailability
ecr:GetDownloadUrlForLayer
ecr:GetRepositoryPolicy
ecr:SetRepositoryPolicy
ecr:GetAuthorizationToken
*
iam:PassRole
arn:aws:iam::<account_id>:role/
apigateway:*
arn:aws:apigateway:::/apis//routes//routeresponses
arn:aws:apigateway:::/apis//routes//routeresponses/
arn:aws:apigateway:::/apis//routes/
arn:aws:apigateway:::/apis//routes
arn:aws:apigateway:::/apis//integrations
arn:aws:apigateway:::/apis//stages//routesettings/
arn:aws:apigateway:::/apis/
arn:aws:apigateway:*::/apis
The following inline policy can be used to grant Feast the necessary permissions:
1
{
2
    "Statement": [
3
        {
4
        Action = [
5
          "lambda:CreateFunction",
6
          "lambda:GetFunction",
7
          "lambda:DeleteFunction",
8
          "lambda:AddPermission",
9
          "lambda:UpdateFunctionConfiguration",
10
        ]
11
        Effect = "Allow"
12
        Resource = "arn:aws:lambda:<region>:<account_id>:function:feast-*"
13
      },
14
      {
15
        Action = [
16
            "ecr:CreateRepository",
17
            "ecr:DescribeRepositories",
18
            "ecr:DeleteRepository",
19
            "ecr:PutImage",
20
            "ecr:DescribeImages",
21
            "ecr:BatchDeleteImage",
22
            "ecr:CompleteLayerUpload",
23
            "ecr:UploadLayerPart",
24
            "ecr:InitiateLayerUpload",
25
            "ecr:BatchCheckLayerAvailability",
26
            "ecr:GetDownloadUrlForLayer",
27
            "ecr:GetRepositoryPolicy",
28
            "ecr:SetRepositoryPolicy",
29
            "ecr:GetAuthorizationToken"
30
        ]
31
        Effect = "Allow"
32
        Resource = "*"
33
      },
34
      {
35
        Action = "iam:PassRole"
36
        Effect = "Allow"
37
        Resource = "arn:aws:iam::<account_id>:role/<lambda-execution-role-name>"
38
      },
39
      {
40
        Effect = "Allow"
41
        Action = "apigateway:*"
42
        Resource = [
43
            "arn:aws:apigateway:*::/apis/*/routes/*/routeresponses",
44
            "arn:aws:apigateway:*::/apis/*/routes/*/routeresponses/*",
45
            "arn:aws:apigateway:*::/apis/*/routes/*",
46
            "arn:aws:apigateway:*::/apis/*/routes",
47
            "arn:aws:apigateway:*::/apis/*/integrations",
48
            "arn:aws:apigateway:*::/apis/*/stages/*/routesettings/*",
49
            "arn:aws:apigateway:*::/apis/*",
50
            "arn:aws:apigateway:*::/apis",
51
        ]
52
      },
53
    ],
54
    "Version": "2012-10-17"
55
}
Copied!
Example
After feature_store.yaml has been modified as described in the previous section, it can be deployed as follows:
1
$ feast apply
2
10/07/2021 03:57:26 PM INFO:Pulling remote image feastdev/feature-server:aws:
3
10/07/2021 03:57:28 PM INFO:Creating remote ECR repository feast-python-server-key_shark-0_13_1_dev23_gb3c08320:
4
10/07/2021 03:57:29 PM INFO:Pushing local image to remote 402087665549.dkr.ecr.us-west-2.amazonaws.com/feast-python-server-key_shark-0_13_1_dev23_gb3c08320:0_13_1_dev23_gb3c08320:
5
10/07/2021 03:58:44 PM INFO:Deploying feature server...
6
10/07/2021 03:58:45 PM INFO:  Creating AWS Lambda...
7
10/07/2021 03:58:46 PM INFO:  Creating AWS API Gateway...
8
Registered entity driver_id
9
Registered feature view driver_hourly_stats
10
Deploying infrastructure for driver_hourly_stats
11
​
12
$ feast endpoint
13
10/07/2021 03:59:01 PM INFO:Feature server endpoint: https://hkosgmz4m2.execute-api.us-west-2.amazonaws.com
14
​
15
$ feast materialize-incremental $(date +%Y-%m-%d)
16
Materializing 1 feature views to 2021-10-06 17:00:00-07:00 into the dynamodb online store.
17
​
18
driver_hourly_stats from 2020-10-08 23:01:34-07:00 to 2021-10-06 17:00:00-07:00:
19
100%|█████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 16.89it/s]
Copied!
After the feature server starts, we can execute cURL commands against it:
1
$ curl -X POST \                                 
2
    "https://hkosgmz4m2.execute-api.us-west-2.amazonaws.com/get-online-features" \
3
    -H "Content-type: application/json" \
4
    -H "Accept: application/json" \
5
    -d '{
6
        "features": [
7
            "driver_hourly_stats:conv_rate",
8
            "driver_hourly_stats:acc_rate",
9
            "driver_hourly_stats:avg_daily_trips"
10
        ],
11
        "entities": {
12
            "driver_id": [1001, 1002, 1003]
13
        },
14
        "full_feature_names": true
15
    }' | jq
16
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
17
                                 Dload  Upload   Total   Spent    Left  Speed
18
100  1346  100  1055  100   291   3436    947 --:--:-- --:--:-- --:--:--  4370
19
{
20
  "field_values": [
21
    {
22
      "fields": {
23
        "driver_id": 1001,
24
        "driver_hourly_stats__conv_rate": 0.025330161675810814,
25
        "driver_hourly_stats__avg_daily_trips": 785,
26
        "driver_hourly_stats__acc_rate": 0.835975170135498
27
      },
28
      "statuses": {
29
        "driver_hourly_stats__avg_daily_trips": "PRESENT",
30
        "driver_id": "PRESENT",
31
        "driver_hourly_stats__conv_rate": "PRESENT",
32
        "driver_hourly_stats__acc_rate": "PRESENT"
33
      }
34
    },
35
    {
36
      "fields": {
37
        "driver_hourly_stats__conv_rate": 0.7595187425613403,
38
        "driver_hourly_stats__acc_rate": 0.1740121990442276,
39
        "driver_id": 1002,
40
        "driver_hourly_stats__avg_daily_trips": 875
41
      },
42
      "statuses": {
43
        "driver_hourly_stats__acc_rate": "PRESENT",
44
        "driver_id": "PRESENT",
45
        "driver_hourly_stats__avg_daily_trips": "PRESENT",
46
        "driver_hourly_stats__conv_rate": "PRESENT"
47
      }
48
    },
49
    {
50
      "fields": {
51
        "driver_hourly_stats__acc_rate": 0.7785481214523315,
52
        "driver_hourly_stats__conv_rate": 0.33832859992980957,
53
        "driver_hourly_stats__avg_daily_trips": 846,
54
        "driver_id": 1003
55
      },
56
      "statuses": {
57
        "driver_id": "PRESENT",
58
        "driver_hourly_stats__conv_rate": "PRESENT",
59
        "driver_hourly_stats__acc_rate": "PRESENT",
60
        "driver_hourly_stats__avg_daily_trips": "PRESENT"
61
      }
62
    }
63
  ]
64
}
Copied!
Reference - Previous
[Alpha] Local feature server
Next - Reference
Feast CLI reference
Last modified 5d ago
Export as PDF
Copy link
Contents