Prerequisites

To complete the tutorial, you’ll need the following things:

An AWS account.
Basic knowledge of Hashicorp Terraform.
AWS credentials configured, so you can deploy the resources described in the main.tf file.
We won’t use an actual database. But you’re welcome to implement this tutorial on any service that you use.
Some basic idea of what is AWS KMS is.

⚠️ Your operations might acrue costs in AWS. Use this tutorial wisely. Don’t forget to terraform destroy in the end.

(Some Kind of) Introduction

As described here and here, you can enable security in transit of Lambda environment variables via the AWS Console with just a few clicks.

So, I wondered to myself how this can be done with Terraform.

When researching this question, I stumbled upon this and this comments, but none offered a clear-cut solution.

To recap, the examples above use AWS KMS to encrypt the secret in question and then the secret is decrypted on the fly by your application.

As I didn’t find a concise answer, I decided to set out on this journey myself.

So, without further ado, let’s get to the receipe.

Full source code is available here.

Show me the magic sauce

We’ll need the following ingredients in our cookbok.

Your AWS account Id & the secret

Edit your bash (~/.bashrc or ~/.zshrc on macOS) profile by running the following commands:

echo "export TF_VAR_aws_account_id=<your aws account id>" >> ~/.bashrc
echo "export TF_VAR_secret=<your secret>" >> ~/.bashrc

Note that the environment variables are prefixed with TF_VAR_. This is how Terraform can read inputs from local environment variables. We provide the inputs for Terraform in a secure way, so we won’t have to expose clear text or type it in the CLI.

💡 Note that after running Terraform, all the secrets will be discoverable on your device through the terraform.tfstate file. To protect it, I chose to exclude it from the repo.

Let’s declare the following components in our main.tf file:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.1.0"
    }
  }
  required_version = "~> 1.0"
}

provider "aws" {
  region = "eu-west-1"
}

//read from the environment variables
variable "secret" {}
variable "aws_account_id" {}

First statements are needed to initialize the woring directory in Terraform.

The consequent variable statements would be read from the environment variables.

IAM role for Lambda

Every Lambda declared in Terraform must be linked to an IAM role. As discussed here, IAM role is a kind of identity that does not have any secrets/passwords associated to it. The access tokens are generated on the fly by a special AWS service, called AWS Security Token Service.

An IAM role can be assumed by various types of identities, such as AWS services (EC2 instances, Lambdas, etc.), users and other IAM roles.

Our Lambda will assume its IAM role behind the scenes (without our intervention 🎉).

If you need some more reference to IAM roles, you can also refer my previous article in the series.

Let’s declare that IAM role:

resource "aws_iam_role" "kms_access_for_lambda" {
  name = "kms_access_for_lambda"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Sid    = ""
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

In this instance, we want only Lambdas to assume that role.

The KMS Key

resource "aws_kms_key" "symmetric_key" {
  description             = "Symmetric key to encrypt and decrypt the secrets"
  deletion_window_in_days = 7
}

In this step, we declared a KMS customer managed key. Namely, this is the key that we (i.e. a customer) manage.

For the sake of simplicity we used the default symmetric key option. By using this type of encryption, we can easily encrypt and decrypt the secret.

As for the deletion_window_in_days argument, note that the keys in KMS aren’t immediately deleted, but scheduled for deletion. The least amount of time the key is obliged to reside in AWS is 7 days after the deletion operation itself.

As for the reasons why it’s so, you can refer this documentation, as we won’t delve into the reasons for delayed keys deletion in this cookbook.

As stated in the AWS official documentation, once we create a key through API, it’s assigned a default policy, which automatically allows the account owner (i.e. the root user ) to encrypt, decrypt, manage the keys and grant access to them.

To make the receipe more concise, we’ll resort to the default policy, which means we’ll omit its creation.

If you want to refer additional documentation, you can go here.

The Cipher Text

resource "aws_kms_ciphertext" "secret" {
  key_id    = aws_kms_key.symmetric_key.key_id
  plaintext = var.secret
  context   = {
    LambdaFunctionName = "tutorial_lambda"
  }
}

The following Terraform resource won’t be stored anywhere in AWS. It’s the actual encryption operation performed by the AWS KMS key we created earlier.

While asking AWS to perform this operation, we provide also an encryption context. This can serve as an additional layer of authentication.

💡 Note that the ecnryption context demonstrated above is the context implemented by AWS behind the scenes when you’re encrypting the Lambda’s environment variables in the Console.

Of course, you can add additional key-value pairs to ehnance the security or alter the context described here completely.

📕 Bear in mind that the same context should be used upon the key’s decryption.

We’ll use this operation’s output later in our main.tf file.

The Grant

resource "aws_kms_grant" "tutorial_lambda_grant" {
  name              = "tutorial_iam_lambda_grant"
  key_id            = aws_kms_key.symmetric_key.key_id
  grantee_principal = aws_iam_role.kms_access_for_lambda.arn
  operations        = ["Decrypt"]
}

To give access to the key, we can either use a key policy or a grant.

There are several differences between KMS key policies and grants. We’ll list only the ones of interest to us:

Only one key policy can be associated with a key. Should you want to edit the key policy, you should replace it. Conversely, several grants can be associated to a key. Also, grants can be easily revoked or retired when required.
Grants only allow access to a single key. Policies can be associated with several keys.

The above makes the grants a good solution for temporarily providing access to the resources. For example, if our whole infrastructure has changed and the grant is not needed anymore, we can easily remove it.

Another reason for why a grant might be more secure is that we limit access to only a single key when using it.

In the case above, we’re granting access to the key for the IAM role we had declared previously, so only Lambda assuming that IAM role can access it.

The Lambda

Let’s declare the Lambda:

resource "aws_lambda_function" "tutorial_lambda" {
  filename         = "package.zip"
  function_name    = "tutorial_lambda"
  role             = aws_iam_role.kms_access_for_lambda.arn
  handler          = "lambda_function.handler"
  source_code_hash = filebase64sha256("package.zip")
  runtime          = "python3.9"
  environment {
    variables = {
        SECRET = aws_kms_ciphertext.secret.ciphertext_blob
    }
  }
}

We’ll run Python code in this tutorial. Hence, the Lambda’s runtime would be Python 3.9. The code will be deployed via the package.zip file that we’d create later.

Note that here we declared an environment variable, but instead of putting the secret itself in clear text, we transmitted only the cipher text blob.

So, to answer the question what AWS does behind the scenes to encrypt the secrets within Lambda’s environment variables with a key, this is what happens:

When providing the secret, the Lambda’s name is added as the secret’s encryption context.
The secret is being encrypted and its cipher text blob is replacing the actual environment variable.

The Lambda URL

resource "aws_lambda_function_url" "tutorial_lambda_url" {
  function_name      = aws_lambda_function.tutorial_lambda.arn
  authorization_type = "NONE"
}

output "function_url" {
  description = "Function URL."
  value       = aws_lambda_function_url.tutorial_lambda_url.function_url
}

Note that Lambda’s URL is publicly accessible here. You can implement additional means to secure the Lambda’s URL as described in my previous post.

The code

We’ll run our application in Python. But in order for the code to run, we need to install its dependencies first.

Note that the dependencies should be deployed along with the code.

The dependencies

Let’s create the requirements.txt file to list the packages our code depends on:

boto3

Yes, we’ll use the boto3 AWS official Python SDK to decrypt the key.

The lambda hander

Let’s create the lambda_function.py file and populate it wit the following code:

import boto3
import botocore
import os
import base64

SECRET = 'SECRET'
AWS_LAMBDA_FUNCTION_NAME = 'AWS_LAMBDA_FUNCTION_NAME'
LAMBDA_FUNCTION_NAME = "LambdaFunctionName"

def decrypt_key(os_env_key_name, kms_client):
    raw_cipher_text = os.environ.get(os_env_key_name)
    aws_function_name = os.environ.get(AWS_LAMBDA_FUNCTION_NAME)
    if raw_cipher_text is None:
        return "error", "Could not get the environment variable"
    try:
        ciphertext=base64.b64decode(raw_cipher_text.encode()) # encode to UTF-8
        response = kms_client.decrypt(CiphertextBlob=ciphertext,
                                           EncryptionContext = {
                                             LAMBDA_FUNCTION_NAME:aws_function_name
                                           })
        if 'Plaintext' in response:
            plain_text = response['Plaintext']
            return "OK", plain_text
        return "error", "There's no plain text in the response."
    except botocore.exceptions.ClientError as error:
        return "error", str(error)

def handler(event, context):
    response_body = 'the key was decrypted successfully'
    response_status_code = 200
    kms_client = boto3.client('kms')
    decryption_result, decryption_response = decrypt_key(SECRET, kms_client)
    if decryption_result == "error":
        response_body = decryption_response
        response_status_code = 400
    return {
        'body': response_body,
        'headers': {'Content-Type': 'application/json'},
        'statusCode': response_status_code,
    }

Note the below flow when decrypting the secret:

Get the raw cipher text encrypted as a string from the Lambda’s environment variables.
Encode the raw cipher text into bytes, hence the raw_cipher_text.encode() statement.
Decode the Base64 bytes into the cipher text. Hence the base64.b64decode() function.
Use this cipher text to decrypt the secret along with the decryption context.

The deployment script

Let’s create the deployment script by creating the build.sh script:

#!/bin/sh
terraform init
echo "Deleting files from previous builds"
rm -f package.zip
echo "Installing the Python packages [dependencies] into the package directory"
pip3 install -r requirements.txt --target ./package
echo "Copying the Python files into the package directory"
cp *.py package/
echo "Clean up caches in the package directory"
pushd package
rm -rf bin
rm -rf __pycache__
zip ../package.zip *
popd
echo "Zip the files for deployment [package.zip]"
echo "The code is ready for Lambda deployment [package.zip]"
echo "Redeploying the application code"
source ~/.bahrc && terraform apply # replace with ~/.zshrc in macOS
echo "****The function has been deployed****"

To deploy the full solution, just run the following command: ./build.sh

Conclusion

By implementing the following receipe we’ve learned what AWS does behind the scenes to encrypt the environment variables in your Lambdas using the KMS.

This practice is more secure, since now you can encrypt the environment variables not only at rest but also in transit.

So, starting from now you can enhance the security of your applications.

But, when using the service don’t forget to factor in its costs.

To recap the whole tutorial, this is what we did:

Created a default Lambda’s IAM role, so it can be used upon Lambda creation
Created a KMS symmetric key to be able to easily encrypt & decrypt the secret
Encrypted a secret using the KMS key above. Along with the secret above we provided some metadata, and later used the operation’s output in the Lambda function we created later.
Granted access to that key to the Lambda’s IAM role by using a grant
Created a Lambda that uses the secret’s cipher text blob as its environment variable

Later, we studied how to implement the secret decryption on the fly in your application.

Hope the above is helpful.

Have additional questions? You can reach me out using the links on my site.