Introduction
Managing secrets securely in AWS Lambda functions is crucial for maintaining the integrity and confidentiality of your applications. AWS provides services like AWS Secrets Manager and AWS Systems Manager Parameter Store to manage secrets. However, frequent retrieval of secrets can introduce latency and additional costs. To optimize this, we can cache secrets using a Lambda Extension.
In this article, we will demonstrate how to use a pre-existing Lambda Extension to cache secrets for a PHP Lambda function using the Bref layer and AWS CDK for deployment.
On a high-level, these are the components involved:
Using the AWS Parameter and Secrets Lambda extension to cache parameters and secrets
The new AWS Parameters and Secrets Lambda extension provides a managed parameters and secrets cache for Lambda functions. The extension is distributed as a Lambda layer that provides an in-memory cache for parameters and secrets. It allows functions to persist values through the Lambda execution lifecycle, and provides a configurable time-to-live (TTL) setting.
When you request a parameter or secret in your Lambda function code, the extension retrieves the data from the local in-memory cache, if it is available. If the data is not in the cache or it is stale, the extension fetches the requested parameter or secret from the respective service. This helps to reduce external API calls, which can improve application performance and reduce cost.
Prerequisites
- AWS Account
- AWS CLI configured
- AWS CDK installed
- PHP installed
- Composer installed
If you have Docker, all requirements are being installed by it.
Repository Overview
The code for this project is available in the following GitHub repository: rafaelbernard/serverless-patterns. The relevant files are located in the lambda-extension-ssm-secrets-cdk-php folder.
Step-by-Step Guide
1. Cloning the Repository
First, clone the repository and navigate to the relevant directory:
git clone --branch rafaelbernard-feature-lambda-extension-ssm-secrets-cdk-php https://github.com/rafaelbernard/serverless-patterns.git
cd serverless-patterns/lambda-extension-ssm-secrets-cdk-php2. Project Structure
The project structure is as follows:
.
├── assets
│ └── lambda
│ └── lambda.php
├── bin
│ └── cdk.ts
├── cdk
│ └── cdk-stack.ts
├── cdk.json
├── docker-compose.yml
├── Dockerfile
├── example-pattern.json
├── Makefile
├── package.json
├── package-lock.json
├── php
│ ├── composer.json
│ ├── composer.lock
│ └── handlers
│ └── lambda.php
├── README.md
├── run-docker.sh
└── tsconfig.json3. Setting Up the Lambda Function
The main logic for fetching and caching secrets is in php/handlers/lambda.php:
<?php
use Bref\Context\Context;
use Bref\Event\Http\HttpResponse;
use GuzzleHttp\Client;
use Symfony\Component\HttpFoundation\JsonResponse;
// Responsibilities are simplified into one file for demonstration purposes
// We would have have those methods in a Service class
function getParam(string $parameterPath): string
{
// Set `withDecryption=true if you also want to retrieve SecureString SSMs
$url = "http://localhost:2773/systemsmanager/parameters/get?name={$parameterPath}&withDecryption=true";
try {
$client = new Client();
$response = $client->get($url, [
'headers' => [
'X-Aws-Parameters-Secrets-Token' => getenv('AWS_SESSION_TOKEN'),
]
]);
$data = json_decode($response->getBody());
return $data->Parameter->Value;
} catch (\Exception $e) {
error_log('Error getting parameter => ' . print_r($e, true));
}
}
function getSecret(string $secretName): stdClass
{
$url = "http://localhost:2773/secretsmanager/get?secretId={$secretName}";
try {
$client = new Client();
$response = $client->get($url, [
'headers' => [
'X-Aws-Parameters-Secrets-Token' => getenv('AWS_SESSION_TOKEN'),
]
]);
$data = json_decode($response->getBody());
return json_decode($data->SecretString);
} catch (\Exception $e) {
error_log('Error getting secretsmanager => ' . print_r($e, true));
}
}
return function ($request, Context $context) {
$secret = getSecret(getenv('THE_SECRET_NAME'));
$response = new JsonResponse([
'status' => 'OK',
getenv('THE_SSM_PARAM_PATH') => getParam(getenv('THE_SSM_PARAM_PATH')),
getenv('THE_SECRET_NAME') => [
'password' => $secret->password,
'username' => $secret->username,
],
]);
return (new HttpResponse($response->getContent(), $response->headers->all()))->toApiGatewayFormatV2();
};4. Setting Up AWS CDK Stack
The AWS CDK stack is defined in cdk/cdk-stack.ts:
import { CfnOutput, CfnParameter, Stack, StackProps } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import { join } from "path";
import { packagePhpCode, PhpFunction } from "@bref.sh/constructs";
import { FunctionUrlAuthType, LayerVersion, Runtime } from "aws-cdk-lib/aws-lambda";
import { StringParameter } from "aws-cdk-lib/aws-ssm";
import { Policy, PolicyStatement } from 'aws-cdk-lib/aws-iam';
import { Secret } from 'aws-cdk-lib/aws-secretsmanager';
export class CdkStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const stackPrefix = id;
// May be set as parameter new CfnParameter(this, 'parameterStoreExtensionArn', { type: 'String' });
const parameterStoreExtensionArn = 'arn:aws:lambda:us-east-1:177933569100:layer:AWS-Parameters-and-Secrets-Lambda-Extension:11';
const parameterStoreExtension = new CfnParameter(this, 'parameterStoreExtensionArn', { type: 'String', default: parameterStoreExtensionArn });
const paramTheSsmParam = new StringParameter(this, `${stackPrefix}-TheSsmParam`, {
parameterName: `/${stackPrefix.toLowerCase()}/ssm/param`,
stringValue: 'the-value-here',
});
// CDK cannot create SecureString
// You would create the SecureString out of CDK and use the param name here
// const paramAnSsmSecureStringParam = StringParameter.fromSecureStringParameterAttributes(this, `${stackPrefix}-AnSsmSecureStringParam`, {
// parameterName: `/${stackPrefix.toLowerCase()}/ssm/secure-string/params`,
// });
const templatedSecret = new Secret(this, 'TemplatedSecret', {
generateSecretString: {
secretStringTemplate: JSON.stringify({ username: 'postgres' }),
generateStringKey: 'password',
excludeCharacters: '/@"',
},
});
// The param path that will be used to retrieve value by the lambda
const lambdaEnvironment = {
THE_SSM_PARAM_PATH: paramTheSsmParam.parameterName,
THE_SECRET_NAME: templatedSecret.secretName,
// If you create the SecureString
// THE_SECURE_SSMPARAM_PATH: paramAnSsmSecureStringParam.parameterName,
};
const functionName = `${id}-lambda`;
const theLambda = new PhpFunction(this, `${stackPrefix}${functionName}`, {
handler: 'lambda.php',
phpVersion: '8.3',
runtime: Runtime.PROVIDED_AL2,
code: packagePhpCode(join(__dirname, `../assets/lambda`)),
functionName,
environment: lambdaEnvironment,
});
// Add extension layer
theLambda.addLayers(
LayerVersion.fromLayerVersionArn(this, 'ParameterStoreExtension', parameterStoreExtension.valueAsString)
);
// Set additional permissions for parameter store
theLambda.role?.attachInlinePolicy(
new Policy(this, 'additionalPermissionsForParameterStore', {
statements: [
new PolicyStatement({
actions: ['ssm:GetParameter'],
resources: [
paramTheSsmParam.parameterArn,
// If you create the SecureString
// paramAnSsmSecureStringParam.parameterArn,
],
}),
],
}),
)
templatedSecret.grantRead(theLambda);
const fnUrl = theLambda.addFunctionUrl({ authType: FunctionUrlAuthType.NONE });
new CfnOutput(this, 'LambdaUrl', { value: fnUrl.url });
}
}5. Deploying with AWS CDK
Make sure you have already AWS variables set and run below command to install required dependancies:
# Using docker -- check run-docker.sh
make upor
# Using local
npm ci
cd php && composer install --no-scripts && cd -After that, you will have all dependencies installed. Deploy it executing:
# Using docker
make deployor
# Using local
npm run deploy6. Testing the Lambda Function
The CDK output will have the Lambda function URL, which you can use to test and retrieve the values:
Outputs:
LambdaExtensionSsmSecretsCdkPhpStack.LambdaUrl = https://keamdws766oqzr6dbiindaix3a0fdojb.lambda-url.us-east-1.on.aws/You should see the secret value and parameter value returned by the Lambda function. Subsequent invocations should retrieve the values from the cache, reducing latency and cost.
{
"status": "OK",
"/lambdaextensionssmsecretscdkphpstack/ssm/param": "the-value-here",
"TemplatedSecret3D98B577-4jOWSbUMCHmF": {
"password": "!o9GpBzpa>dYdo.Gx3J2!<zd(s-Fg;ev",
"username": "postgres"
}
}Performance benefits
A similar example application written in Python performed three tests, reducing API calls ~98%. I am quoting their findings, as the benefits are the same for this PHP Lambda:
To evaluate the performance benefits of the Lambda extension cache, three tests were run using the open source tool Artillery to load test the Lambda function.
config: target: "https://lambda.us-east-1.amazonaws.com" phases: - duration: 60 arrivalRate: 10 rampTo: 40Test 1: The extension cache is disabled by setting the TTL environment variable to 0. This results in 1650 GetParameter API calls to Parameter Store over 60 seconds.Test 2: The extension cache is enabled with a TTL of 1 second. This results in 106 GetParameter API calls over 60 seconds.
Test 3: The extension is enabled with a TTL value of 300 seconds. This results in only 18 GetParameter API calls over 60 seconds.In test 3, the TTL value is longer than the test duration. The 18 GetParameter calls correspond to the number of Lambda execution environments created by Lambda to run requests in parallel. Each execution environment has its own in-memory cache and so each one needs to make the GetParameter API call.
In this test, using the extension has reduced API calls by ~98%. Reduced API calls results in reduced function execution time, and therefore reduced cost.
7. Clean up
To delete the stack, run:
make bash
npm run destroyConclusion
In this article, we demonstrated how to use a pre-existing Lambda Extension to cache secrets for a PHP Lambda function using the Bref layer and AWS CDK for deployment. By caching secrets, we can improve the performance and reduce the cost of our serverless applications. The approach detailed here can be adapted to various use cases, enhancing the efficiency of your AWS Lambda functions.
For more information on the Parameter Store, Secrets Manager, and Lambda extensions, refer to:
- Using Parameter Store parameters in AWS Lambda functions
- Use AWS Secrets Manager secrets in AWS Lambda functions
- Introducing AWS Lambda Extensions
- Caching data and configuration settings with AWS Lambda extensions
- AWS blog on using Lambda Extensions to cache secrets
For more serverless learning resources, visit Serverless Land.