CDK Escape Hatches + How to Export CDK to CloudFormation

This article is part of a series that looks at solutions for improving CloudWatch Alarms for Lambda errors:

1. Improving CloudWatch Alarms for Lambda Errors
2. Tips and Tricks for Developing CDK Construct using Projen
3. CDK Escape Hatches + How to Export CDK to CloudFormation (here)

You can find a full solution here.

In the previous article, we built a CDK construct. However, we also want the solution to be functional on its own as a CloudFormation template. This way, you do not have to incorporate the solution into your stack. You deploy a new one and point to the existing SNS topic you use for CloudWatch Alarm notification. We do not want to write CloudFormation from scratch but to reuse the CloudFormation that CDK produces. In this article, we will look at how to convert CDK to clean CloudFormation without CDK metadata and the weird naming that CDK produces. Along the way, we will learn some really useful CDK tricks.

Why convert to CloudFormation and ship independently of CDK?

There are multiple reasons why people may prefer exported CloudFormation over CDK.

• Reuse already-built constructs/solutions. That is our reason.
• CDK has some obvious downsides, and having an escape hatch of converting CloudFormation can be useful.
• Culture war with (Dev)Ops guys. The real developers, some would say, write every line of CloudFormation by themselves. Anything else is “unworthy of the title developer.”
• The requirement is to produce CloudFormation, but the speed of the development with CDK is unmatched.

I highly discourage having a default solution that converts CDK to CloudFormation. That should be the last escape hatch. CDK brings a lot of benefits and features, and a lot of them cannot be replicated by just converting to CloudFormation. If the “culture war” is the cause of this approach, it would be better to sit down with the opposite side and do an unbiased evaluation of each approach. The end goal is to build a good product with the least resources and not to win culture wars.

We can export to CloudFormation like this:

cdk synth > my_cloudformation.yaml

You do not need anything else. But this approach has several issues:

• CloudFormation has a lot of CDK metadata.
• Logical names have random characters in the postfix.
• Assets are not dropped to any appropriate S3.

In our case, we also want to extend CloudFormation with CloudFormation parameters for the ARN of the SNS topic.

CDK Escape Hatches

Here is a list of CDK escape hatches. We will use most of them in our solution.

Setting Logical ID

CDK generates Logical ID. We do not want that because it adds random characters as postfix to make them unique. There are other reasons you will want to alter Logical ID. The most common reason is that you want to prevent the renaming of the logical ID when you organize the CDK files. If you move the element into subconstruct it will have a different Logical ID, but that also means that the CloudFormation will recreate the resources:

Changing Logical ID:

myLambda.overrideLogicalId(`myLambda`);

Modifying the construct

If an L2 construct is missing a feature or you're trying to work around an issue, you can modify the L1 construct that's encapsulated by the L2 construct. L1 constructs are just raw CloudFormation resources. Their names start with “Cfn”. You can find the main resource of an L2 construct in a defaultChild property. It returns the child construct that has the id Default or Resource.

// Get the CloudFormation resource
const cfnBucket = bucket.node.defaultChild as s3.CfnBucket;

// Change its properties
cfnBucket.analyticsConfiguration = [
  { 
    id: 'Config',
    // ...        
  } 
];

Reusing L1 constructs (“An unescape hatch”)

You can create a new L2 construct that wrap the L1 construct:

b1 = new s3.CfnBucket(this, "buck09", { ... });
b2 = s3.Bucket.fromCfnBucket(b1);

Finding a node

We would like to find a node created by some other part of the CDK, possibly by another CDK construct.

First, we want to decide on which node we want to start our search from.

From the root:

const node = this.node;

or from some existing element/construct:

const node = lambdaErrorSnsSender.node;

We can search by name:

const snsErrorFunc = node.findChild('lambdaSnsError') as lambda.Function;

The method goes only one level deep in the tree. So you might want to use the method findAll, which goes through the whole tree, and you can filter elements by any property or type:

const subscriptions = lambdaErrorSnsSender.node
  .findAll()
  .filter((s) => s instanceof sns.CfnSubscription) as sns.CfnSubscription[];

Once we find the correct node, we need to modify it. You cannot modify an L2 construct but you can modify an L1 construct, so you will probably want to find an L1 construct. The simplest way to get an L1 construct from an L2 construct is by using the already mentioned property defaultChild.

Modifying the policy

L1 constructs do not have many constraints. For example, the CfnPolicy has policyDocument property defined as any and can accept any JavaScript object as long as it has the right structure:

lambdaPolicy.policyDocument = {
  Statement: [{
    Effect: 'Allow',
    Action: [...],
    Resource: [...],
  }],
};

Raw overrides

When we want to use brute force, we can use the addOverride method that can change any property (source):

// Get the CloudFormation resource
const cfnBucket = bucket.node.defaultChild as s3.CfnBucket;

// Use dot notation to address inside the resource template fragment
cfnBucket.addOverride('Properties.VersioningConfiguration.Status', 'NewStatus');
cfnBucket.addDeletionOverride('Properties.VersioningConfiguration.Status');

// use index (0 here) to address an element of a list
cfnBucket.addOverride('Properties.Tags.0.Value', 'NewValue');
cfnBucket.addDeletionOverride('Properties.Tags.0');

// addPropertyOverride is a convenience function for paths starting with "Properties."
cfnBucket.addPropertyOverride('VersioningConfiguration.Status', 'NewStatus');
cfnBucket.addPropertyDeletionOverride('VersioningConfiguration.Status');
cfnBucket.addPropertyOverride('Tags.0.Value', 'NewValue');
cfnBucket.addPropertyDeletionOverride('Tags.0');

Putting It All Together

Removing the metadata

You can prevent the creation of metadata with the parameters:

cdk synth --no-asset-metadata --no-path-metadata > my_cloudformation.yaml

Or better, set them in cdk.json:

{
...
  "versionReporting": false,
  "assetMetadata": false,
...
}

Setting Nice Logical Names (Logical ID)

We do not want a random character in logical names:

lambdaErrorSnsSenderlambdaSnsErrorD136C4C4:
  Type: AWS::Lambda::Function

We want to have something like this:

lambdaSnsError:
  Type: AWS::Lambda::Function

Those characters that CDK uses are there for a reason, so it can ensure uniqueness. But they are not that useful when using pure CloudFormation, plus they are ugly and confusing.

To resolve this, we will go through all of CDK node three and set the Logical ID to id the set in the CDK. For this Lambda: new lambda.Function(this, 'LambdaSnsError', …). We want the Logical ID to be "LambdaSnsError".

setNodeNames(this.node.children);
...

function setNodeNames(constructs: IConstruct[], parentName?: string) {
  let i = 0;
  for (const construct of constructs) {
    let newParentName = parentName;

    if (construct.node.defaultChild instanceof cdk.CfnElement) {
      newParentName = parentName
        ? `${parentName}${construct.node.id}`
        : construct.node.id;
      construct.node.defaultChild.overrideLogicalId(newParentName);
    }

    if (construct instanceof lambda.CfnPermission) {
      construct.overrideLogicalId(`${newParentName}Permission${i}`);
      i++;
    }

    setNodeNames(construct.node.children, newParentName);
  }
}

The Lambda permissions need special handling. There may be other cases that you need to take care of. The solution is not bulletproof but is simple and will work in most cases.

Handling Assets

We want to copy the CloudFormation to S3 alongside all the assets. I created a script for that.

The script does the following:

1. Goes through the assets.json in cdk.out folder.
2. Zips all assets and uploads them to S3.
3. Amends the CloudFormation with the correct references of the assets.
4. Uploads the CloudFormation to S3.

async function export() {
  const jsonFile = fs.readFileSync(
    'cdk.out/lambda-error-sns-sender-cf.assets.json',
    'utf-8',
  );
  const assets: Assets = JSON.parse(jsonFile);
  const assetsFiles: string[] = [];
  let cloudFromationTemplateFile: string | undefined;

  for (const key of Object.keys(assets.files)) {
    const file = assets.files[key];
    const packaging = file.source.packaging;
    const destination = file.destinations['current_account-current_region'];
    const objectKey = destination.objectKey;
    const sourcePath = file.source.path;

    const fullPath = path.join('cdk.out', sourcePath);

    if (packaging === 'zip') {
      await uploadZipToS3(fullPath, objectKey);
      assetsFiles.push(objectKey);
    } else if (sourcePath === 'lambda-error-sns-sender-cf.template.json') {
      // our CloudFormation template
      cloudFromationTemplateFile = fullPath;
    } else {
      await uploadFileToS3(fullPath, objectKey);
    }
  }

  if (!cloudFromationTemplateFile) {
    throw new Error('cloudFromationTemplateFile not found');
  }

  await convertToYamlAndUploadZipToS3(
    cloudFromationTemplateFile,
    cloudFromationOutputYamlFileName,
    assetsFiles,
  );
}

async function convertToYamlAndUploadZipToS3(
  fullPath: string,
  objectKey: string,
  assetsFiles: string[],
) {
  console.log(`Converting to yaml and uploading ${fullPath} to ${objectKey}`);

  // read JSON
  const jsonString = fs.readFileSync(fullPath, 'utf-8');
  const json = JSON.parse(jsonString);

  for (const key in json.Resources) {
    const resource = json.Resources[key];
    const s3Key = resource.Properties?.Code?.S3Key;

    if (s3Key && assetsFiles.includes(s3Key)) {
      console.log(
        `Replacing ${JSON.stringify(
          json.Resources[key].Properties.Code.S3Bucket,
        )} with ${bucketName} for resource ${s3Key}`,
      );
      json.Resources[key].Properties.Code.S3Bucket = bucketName;
    }
  }

  // convert to YAML
  const yaml = YAML.stringify(json);

  await uploadToS3(yaml, objectKey);
}

The solution can be easily adapted to your case. The code is written in TypeScript. You can run it with esbuild-runner like this:

npx esr scripts/copyToS3.ts

Adding CloudFormation Parameters and Conditions

The last part is a little more complex. Our stack needs ARN of SNS topics as the CloudFormation Parameter, and if the parameter is not set, we need to disable subscription to that topic and also appropriately adapt the policy. Normally, you wouldn't use CloudFormation Parameters in CDK because CDK provides a better way to parameterize your stack. But our goal is extracting raw CloudFormation, and CloudFormation Parameters are the only way to do that. (Note, the code is abbreviated.)

Create Parameter:

const snsTopicArnParam = new CfnParameter(this, `snsTopicArn${i}`, {
  type: 'String',
  description: 'SNS Topic ARN receive alarms and send Lambda errors to.',
});

Get the SNS topic based on the CloudFormation Parameter that the user has entered when deploying CloudFormation:

const snsTopic = sns.Topic.fromTopicArn(
  this,
  `snsTopic${i}`,
  snsTopicArnParam.valueAsString,
);

Create Condition:

const condition = new cdk.CfnCondition(this, `conditionSnsTopic${i}`, {
  expression: cdk.Fn.conditionNot(
    cdk.Fn.conditionEquals('', snsTopic.topicArn),
  ),
});

conditionsDict[snsTopicArnParam.valueAsString] = condition;

Add condition to Subscription

First, we need to find subscriptions:

const subscriptions = lambdaErrorSnsSender.node
  .findAll()
  .filter((s) => s instanceof sns.CfnSubscription) as sns.CfnSubscription[];

Applying conditions to subscriptions:

for (const subscription of subscriptions) {
  if (conditionsDict[subscription.topicArn]) {
    subscription.cfnOptions.condition = conditionsDict[subscription.topicArn];
  }
}

Add condition to Policy:

Find the Policy:

const lambdaPolicy = snsErrorFunc.node
  .findAll()
  .find((c) => c instanceof iam.CfnPolicy) as iam.CfnPolicy;

Go through all the Statements of Policy and modify the right one:

const newPolicyStatements = [];
for (const statement of lambdaPolicy.policyDocument?.statements) {
  // logic for finding the right Statement 
  ...

  if (addCondition && condition) {
    // the Statement can be just a regular JSON in the L1 construct.
    const newStatement = {
      'Fn::If': [
        condition.logicalId,
        statement.toJSON(),
        ,
        { Ref: 'AWS::NoValue' },
      ],
    };

    newPolicyStatements.push(newStatement);
  } else {
    newPolicyStatements.push(statement.toJSON());
  }
}

lambdaPolicy.policyDocument = {  
  Statement: newPolicyStatements,
};

Conclusion

We went through a lot of CDK escape hatches in this article. I use CDK in most/all of my projects and rarely use these escape hatches. But they could be useful in some cases.

CDK is a great IaC tool, especially if you use one of the available constructs that packages the whole solution with all the best practices (check Construct Hub).

If you do not use CDK, you miss a lot. By following the described method for exporting CDK to independent CloudFormation, you can benefit from both worlds: a raw CloudFormation and powerful features of CDK. But that should be the last escape hatch. The solution does not cover all the edge cases that could appear when exporting CDK to CloudFormation, but the approach can be extremely useful in some cases.