Load Testing Serverless Application using k6

One of the main selling points of serverless is that it can scale to almost infinitive. However, building highly scalable systems is still a challenge. Serverless is complex, and it is quite challenging to get the architecture right. It is also very easy to hit service quotes (build in limits) when services stop scaling. They are there for two reasons: for you to not massively overspend in case of a bug or a huge load and that it helps AWS to maintain a level of service availability for all their customers. Some quotas are “hard,” but most important ones are “soft,” meaning you can open a request to AWS to ask for higher limits. You can see the quotes and request a quota increase in the Service Quotas console.

If you need to build a highly scalable serverless system, it is not enough to study all quotas and architecture in detail because you will definitely miss something. You need to do load testing. The best kind of tool/service for load testing is k6. The winning feature is it supports writing tests in JavaScript and even TypeScript. If your system is written in one of these languages, it is handy to also use it to write load tests. The benefit of using a real language for load testing, in opposite to just sending a static payload, like in other similar tools, is that you can generate test data on the fly and easily simulate a real user. I find k6 an essential tool for my serverless development, and I use it for all systems that handle any significant load.

You can execute tests locally, or you can use their cloud service. There are many benefits of cloud service. The most important is that you can pick regions where the tests are executed. The service also produces nice graphical reports. But for most cases, when you want to find the scaling limits and when the system will hit AWS service quotas, the local testing is good enough.

Our Project

I will describe how to write a simple test using TypeScript. I am a big advocate for TypeScript, but writing tests for k6 is not that indispensable because tests are usually simple. I still use TypeScript because all of the systems I build are built with TypeScript, and it also allows me to reuse some of the code, like the one to generate sample data, that I can reuse for load, unit, and integration tests.

For writing k6 tests in TypeScript, there is already a well-written sample. We will go much further and expand it with:

• Create a monorepo with an application built with the SST framework. We will execute our test on this application.
• Use esbuild to bundle the code (although webpack is the recommended one by k6).
• Authenticate user using Cognito.
• Use of faker-js to generate some sample data. This part of the code is shared between integration and load test.
• Use the SST Config feature for sharing parameters via the SSM Parameter Store. You can use this approach for regular integration tests and also in-load tests.

You can find the solution here. Before using it, install k6. k6 is not an npm module, it requires a separate install.

JavaScript Engine

JavaScript, which is running in the k6 engine, is ES2015(ES6)+ compatible, but you can expect some other incompatibilities with never engines. Until recently, it did not support async/await operations. Still, some npm modules will not work.

Bundling

k6 lacks support for the node module resolution algorithm, so npm modules must first be transformed and bundled with tools such as Webpack, Parcel, Rollup, or Browserify. Webpack is the preferred one by k6.

You might also want to use a bundler if you use newer JavaScript features. The bundling is not necessary if you do not use npm modules, if you only use ES2015 features and write code in JavaScript, not TypeScript. But even if you use a bundler, note that many npm modules will not be usable in k6.

AWS SDK

One of the npm modules that will not work with k6 is AWS SDK. But hope is not lost. They built a special library k6-jslib-aws, for the most essential AWS operations:

• S3Client: allows listing buckets and bucket's objects, as well as uploading, downloading, and deletion of objects.
• SecretsManager: allows to list, get, create, update, and delete secrets from the AWS Secrets Manager service.
• SQS: allows to list queues and send messages from AWS SQS.
• KMS: allows to list KMS keys and generates a unique symmetric data key for use outside of AWS KMS
• SSM: allows retrieving a parameter from AWS Systems Manager
• V4 signature: allows to sign requests to Amazon AWS services
• KinesisClient: allows all APIs for Kinesis available by AWS.

Extensions

You can find over 50 various extensions here. They enable you to support different clients, protocols, improve performance, and so on. Some are official, and some are community-driven. Please note that some community ones are not that well maintained.

One of the more interesting official extensions is xk6-sql, which enables you to load test RDBMSs (PostgreSQL, MySQL, MS SQL, and SQLite3).

A Simple Test

Writing a test is ultra-simple, and that's why we love k6:

import http from 'k6/http';

export let options = {
  vus: 50,
  duration: '10s'
};

export default function () {
  http.get('<http://test.k6.io>');
  sleep(1); // simulate user pause before next click
}

The options part defines how the test is executed. This one will run for 10 seconds, and 50 virtual users will execute the test in the loop. There are many more options that you can configure.

For sending REST, you must use k6 internal module k6/http, which does the monitoring. Note the calls are synchronous. That is not the only k6 module you might use. For example, you will also need a module to generate UUID.

Test Lifecycle

You can also run code before and after the main function.

// 1. init code
export function setup() {
  // 2. setup code
}

export default function (data) {
  // 3. VU code
}

export function teardown(data) {
  // 4. teardown code
}

The following lifecycle functions are supported:

• Init - executed once per virtual user
  You write that in the root of your file, not in function. It is used for loading local files and importing modules.
• Setup - executed only once, not for each virtual user
  Used for set up data for processing. Data is shared among virtual users.
• Default - executed in a loop by virtual users
  Here you put the main test, calling http endpoints, gathering metrics, etc.
• Teardown - executed only once, not for each virtual user
  Process result, stop test environment.

Putting It All Together

Here, you can find the whole project on GitHub. It is monorepo with an already mentioned application built with the SST framework we will use to load tests.

Key folders:

• stacks - stacks written in CDK/SST
• packages - application, Lambda code
• test/integration - integration test
• test/load - load test written in k6

Each part is its own project in a monorepo. Each has its own scripts in package.json that you should use.

Sharing Parameters via SST Config feature:

We declare the SSM parameter in our stack:

new Config.Parameter(stack, "API_URL", {
  value: api.url,
});

If we want to use it in our integration test, we must use the sst bind command that sends those parameters as environment variables:

sst bind vitest

Vitest is the library used for integration tests.

Then we can use it in the integration test:

Config.API_URL

We can use sst bind also when running k6 test:

sst bind "k6 run dist/customer.load.test.js"

We use parameters like this:

__ENV.SST_Parameter_value_API_URL

Reading Parameters Directly from SSM

We could read parameters directly from SSM using k6-jslib-aws. In the code sample, you can find commented out an example of how to use it.

import { AWSConfig, SystemsManagerClient } from "https://jslib.k6.io/aws/0.9.0/ssm.js";

...
const awsConfig = new AWSConfig({
  region: __ENV.AWS_REGION,
  accessKeyId: __ENV.AWS_ACCESS_KEY_ID,
  secretAccessKey: __ENV.AWS_SECRET_ACCESS_KEY,
  sessionToken: __ENV.AWS_SESSION_TOKEN,
});
const secretsManager = new SystemsManagerClient(awsConfig);
const secretName = `/sst/${__ENV.SST_APP}/${__ENV.SST_STAGE}/Parameter/COGNITO_USER_POOL_ID/value`;
const userPoolIdFromSSM = await secretsManager.getParameter(secretName);

Sharing Files and using Faker-js

In the file test/sampleDataGenerator.ts you would find code that is shared between integration and load test. It uses the @faker-js/faker library to generate sample data. Unfortunately, the latest version, 8, does not work since the k6 JavaScript engine is a bit old. But the older version 6 of faker-js works. It is worth mentioning that having a huge library bundled into the test is not great. Our bundled JavaScript file now weighs 3.6MB. The tests should be light so the k6 can easily simulate thousands of parallel users.

Using esbuild

Webpack is the preferred bundler by k6, but esbuild is newer and faster. You might find that some packages are not correctly transpiled, so try to adjust the target property, which configures target JavaScript version (eg. --target=es2017).

esbuild src/customer.load.test.ts --bundle --outfile=dist/customer.load.test.js --platform=node --external:k6 --external:https\*

The Test

Before running a test, you need to create a test user. You can do it by running script create_guest_user in test/load/package.json.

...
export function setup() {
  const username = "guest";
  const password = "paSSword1!";
  const userPoolId = __ENV.SST_Parameter_value_COGNITO_USER_POOL_ID;
  const clientId = __ENV.SST_Parameter_value_COGNITO_USER_POOL_CLIENT_ID;
  const region = __ENV.AWS_REGION;

  const poolData = {
    UserPoolId: userPoolId,
    ClientId: clientId,
  };

  const postData = {
    AuthFlow: "USER_PASSWORD_AUTH",
    ClientId: poolData.ClientId,
    UserPoolId: poolData.UserPoolId,
    AuthParameters: {
      USERNAME: username,
      PASSWORD: password,
    },
  };

  const hostname = `cognito-idp.${region}.amazonaws.com`;

  const res = http.post(
    `https://${hostname}/${userPoolId}`,
    JSON.stringify(postData),
    {
      headers: {
        "Content-Type": "application/x-amz-json-1.1",
        "X-Amz-Target": "AWSCognitoIdentityProviderService.InitiateAuth",
      },
    }
  );

  const body = JSON.parse(res.body as string);

  const accessToken = body.AuthenticationResult.AccessToken;

  return {
    accessToken,
  };
}

export default async function ({ accessToken }: { accessToken: string }) {
  const url = __ENV.SST_Parameter_value_API_URL;

  const customer = SampleDataGenerator.generateCustomer();

  const res = http.post(url, JSON.stringify(customer), {
    headers: {
      Authorization: `Bearer ${accessToken}`,
      "Content-Type": "application/json",
    },
  });
  ...
}

Other Interesting Features of k6

Although k6 is simple to use, it is a beast packed with loads of features that you might use for your load testing:

Supported Protocols

Out of the box, it supports protocols HTTP, WebSocket, and gRPC. Since HTTP is supported, GraphQL and SOAP are. But there are no special utilities for that. However, there are examples for both GraphQL and SOAP.

Parsing HTML

k6 has tools to easily parse HTML:

const res = http.get('<https://k6.io>');
const doc = res.html();
const pageTitle = doc.find('head title').text();
const langAttr = doc.find('html').attr('lang');

This is especially useful for parsing dynamic data that you would need to actually make your test work, such as CSRF tokens, ViewState, wp_nonce, and so on. You need to include them in subsequent requests. More info here.

Submitting a Form

A nice tool to simplify submitting a form:

let res = http.get('https://httpbin.test.k6.io/forms/post');

res = res.submitForm({
  formSelector: 'form',
  fields: { custname: 'test', extradata: 'test2' },
});

Note that it does this without running in the browser.

Improving Reports

You can use tags and groups to visualize, sort, and filter your test results. You use tags to categorize your checks, thresholds, custom metrics, and requests for in-depth filtering. Groups are used to organize a load script by functions.

Besides the built-in metrics, you can create custom metrics.

Autogenerated Tests based on OpenAPI

There is an interesting tool called OpenAPI-generator. It generates client code based on OpenAPI schema for multiple languages and environments. Among those, is k6. It also automatically generates tests. It might be useful for some scenarios and also to quickly boot up your testing solution.

Create Tests from Recordings

Writing tests by hand can be tedious, cumbersome, and pointless since the front-end application already creates all the requests that you want to send.

k6 provides two tools that can directly convert a recording into k6 script:

• Browser recorder generates a k6 script from a browser session.
• HAR converter generates a k6 script from the requests included in a HAR file.

This will also help create tests that are realistic, with pauses between requests and a more natural flow.

In most cases, you will still have to:

• organize generated code
• manual handle CSRF tokens, ViewState, wp_nonce, etc., to correlate requests
• configure load testing options such as the number of virtual users
• remove requests that are not meant for your service

Test Builder

The k6 Test Builder is a GUI tool to generate a k6 test code. In most cases, you would still write tests yourself or get some help from the recordings mentioned above, but this tool might help you quickly boot up your tests and simplify collaboration with other people that are not developers.

Browser Module

k6 is meant to send requests directly to your endpoints. But the new Chromium-based browser module enables you to do performance tests on the browser level, like test page load time, and also to execute end-to-end tests. The module aims to provide rough compatibility with the Playwright API, so you don’t need to learn a completely new API. 7