Add Logging to AWS Lambda Without Redeploying Using a Simple Extension

The Challenge of Debugging AWS Lambda in Production

Debugging Lambda functions in production can be challenging. Without sufficient logging, it’s often difficult to diagnose issues in the system. You could use some observability tools like Lumigo. You could also use Lambda Live Debugger with Observability mode. In most cases, you are mostly left to the mercy of the logs you have. However, excessive logging in high-traffic serverless applications can lead to high CloudWatch costs.

The ideal solution is temporarily enabling logging without redeploying your Lambda function. But what if you didn’t build logging in a way that would support that? Is there a way to introduce this additional logging?

Fortunately, there is. Using this Lambda extension, you can add it to any Lambda function, even in production environments, with minimal impact on your system. It supports Lambda based on Nodej.js.

The extension was originally developed as part of Lambda Live Debugger and a ServerlessSpy tool. Lambda Live Debugger is a tool that enables remote debugging of Lambda functions. It also speeds up development because you can avoid constant and timely redeploys. ServerlessSpy is a CDK-based library for writing elegant integration tests on AWS serverless architecture. It has an additional web console to monitor events in real-time.

Building the Extension

There are two types of Lambda extensions: external and internal. For logging requests and responses, we need an internal extension that wraps the handler function.

Building an internal extension, particularly for Node.js, isn’t well-documented, but the principle is straightforward: the extension wraps the handler function, intercepts, and log the data passing through. It’s important to note that this method is runtime-specific, meaning the solution provided here is tailored for Node.js and cannot be directly applied to other runtimes like Python, Java, or .NET.

You can found source code here.

1. Add a Bash Script to Replace the Lambda Handler

The first step is to include a bash script that replace the original Lambda handler with the one provided by the extension.

#!/bin/bash
export ORIGINAL_HANDLER=$_HANDLER
export _HANDLER="logger.handler"
exec "$@"

2. Set the AWS Lambda Execution Wrapper

On the Lambda itself you need to set the AWS_LAMBDA_EXEC_WRAPPER environment variable to point to our bash script.

AWS_LAMBDA_EXEC_WRAPPER = /opt/logger

3. Create the New Handler

In the JavaScript/TypeScript code, the new handler logs the input event, calls the original handler, logs the response, and then returns it. A key challenge is obtaining the original handler, but AWS provides a utility that can do that.

// @ts-ignore
import { load } from "./aws/UserFunction";

const ORIGINAL_HANDLER_KEY = "ORIGINAL_HANDLER";
let originalHandler: any;

export async function handler(event: any, context: any) {
  console.log("INPUT_EVENT: ", JSON.stringify(event, null, 2));

  if (!originalHandler) {
    originalHandler = await getOriginalHandler();
  }

  const response = await originalHandler(event, context);

  console.log("RESPONSE: ", JSON.stringify(response, null, 2));
  return response;
}

async function getOriginalHandler(): Promise<any> {
  if (process.env[ORIGINAL_HANDLER_KEY] === undefined)
    throw Error("Missing original handler name");
  const originalHandler = load(
    process.env.LAMBDA_TASK_ROOT!,
    process.env[ORIGINAL_HANDLER_KEY]
  ) as Promise<any>;

  if (!originalHandler) {
    throw Error("Original handler not found");
  }

  return originalHandler;
}

How to Use the Extension

This extension is publicly available and can be attached to any already-deployed Lambda function.

1. Pick the Lambda Layer ARN that match your region

• ap-south-1: arn:aws:lambda:ap-south-1:076585967075:layer:lambda-logger-extension:1
• eu-north-1: arn:aws:lambda:eu-north-1:076585967075:layer:lambda-logger-extension:1
• eu-west-3: arn:aws:lambda:eu-west-3:076585967075:layer:lambda-logger-extension:1
• eu-west-2: arn:aws:lambda:eu-west-2:076585967075:layer:lambda-logger-extension:1
• eu-west-1: arn:aws:lambda:eu-west-1:076585967075:layer:lambda-logger-extension:22
• ap-northeast-3: arn:aws:lambda:ap-northeast-3:076585967075:layer:lambda-logger-extension:1
• ap-northeast-2: arn:aws:lambda:ap-northeast-2:076585967075:layer:lambda-logger-extension:1
• ap-northeast-1: arn:aws:lambda:ap-northeast-1:076585967075:layer:lambda-logger-extension:1
• ca-central-1: arn:aws:lambda:ca-central-1:076585967075:layer:lambda-logger-extension:1
• sa-east-1: arn:aws:lambda:sa-east-1:076585967075:layer:lambda-logger-extension:1
• ap-southeast-1: arn:aws:lambda:ap-southeast-1:076585967075:layer:lambda-logger-extension:1
• ap-southeast-2: arn:aws:lambda:ap-southeast-2:076585967075:layer:lambda-logger-extension:1
• eu-central-1: arn:aws:lambda:eu-central-1:076585967075:layer:lambda-logger-extension:2
• us-east-1: arn:aws:lambda:us-east-1:076585967075:layer:lambda-logger-extension:1
• us-east-2: arn:aws:lambda:us-east-2:076585967075:layer:lambda-logger-extension:1
• us-west-1: arn:aws:lambda:us-west-1:076585967075:layer:lambda-logger-extension:1
• us-west-2: arn:aws:lambda:us-west-2:076585967075:layer:lambda-logger-extension:1

2. Add Layer to Lambda

Open your Lambda in your AWS console, switch to the "Code" tab and scroll down to "Layers".

Click "Add a layer".

Select "Specify an ARN", enter the Layer ARN and click Add.

3. Add environment variable

Go to "Configuration" tab and select "Environment variables"

Click Edit and add environment variable.

AWS_LAMBDA_EXEC_WRAPPER = /opt/logger

Important Considerations

This extension will not work if you are using other internal extensions, such as Lumigo.

Use the extension at your own risk.

Conclusion

Using this internal Lambda extension for logging allows you to add critical observability without redeploying your function or use other more complex or expensive tools. This approach gives you the flexibility to log requests and responses dynamically, making it easier to troubleshoot production issues. It is a valuable tool in any developer's toolkit.