Load testing

Put an API scenario under load from up to three regions, and read every number on the Load tests page, from p95 and error rate to what “Thresholds breached” means.

Load testing

A load test sends many simulated users at your API at the same time, for a set length of time, and measures how fast and how reliably it answers. Qodex builds the load test from an API scenario you already have, runs it on k6 from up to three regions, and tells you whether your API held up.

This page explains how to start a run, what every number and label on the Load tests page means, and what to do when a run does not pass.

Load tests appears under Test in the sidebar only when load testing is switched on for your project. If you do not see it, ask your Qodex contact to turn it on.

The short version

A run passes when, in every region it ran from, all three of these held:

Threshold	Plain meaning
p95 < 1000ms	95 out of every 100 requests got a response in under one second.
errors < 1%	Fewer than 1 in every 100 requests failed.
checks > 99%	More than 99 in every 100 of your scenario’s checks passed, such as “the status is 200”.

If any one of them fails in any region, the run is marked Thresholds breached. That is not a crash or a Qodex error. It means the load test ran to the end and your API was slower, or failed more often, than the limit allows under that much traffic.

Before you start

A load run needs three things. When you have none yet, the page lists them with Done or To do beside each.

1. An API scenario. Load tests replay API scenarios only, not UI scenarios. Save one from chat or on the Scenarios page.
2. An environment. The run sends its traffic to the base URL and variables of the environment you pick. Add one on the Environments page.
3. A run left in your allowance. The line under the page title shows your plan and how many load runs you have used. See Your load run allowance.

A load test sends real traffic to the environment you choose. Point it at a staging or test environment you are allowed to load, not at production, unless you mean to.

Start a run

Click Start new run, then fill in:

Field	What it means
Scenario	The API scenario every simulated user repeats.
Environment	Where the traffic goes.
Concurrent users	How many simulated users run the scenario at the same time, in each region. 1 to 100.
Duration (sec)	The hold time: how long the full number of users keeps going. 10 to 1800 seconds.
Regions	Where the traffic comes from: US East (N. Virginia), UK, EU West (London), and SG, Asia Pacific (Singapore). Pick one or more.

The box under the fields shows the thresholds the run will be judged on (Passes when p95 < 1000ms, errors < 1%, checks > 99%), how long the run will take, and how much of your allowance it uses. Click Run in N regions to start.

Run again on a past run starts a new one with the same scenario, environment, users, duration, regions and thresholds.

How a run works

Every run has three phases:

1. Ramp up (30 s). Users are added gradually, from zero up to your Concurrent users number, so your API is not hit by everyone at once.
2. Hold. All the users keep running for the Duration you set.
3. Ramp down (30 s). Users are removed gradually back to zero.

So a run with a 120-second duration takes about 3 minutes in total: 30 s up, 120 s hold, 30 s down.

Each simulated user (k6 calls this a virtual user) works in a loop. It runs every step of your scenario in order, checks each response, waits half a second, then starts the scenario again from the first step. Values a step captures, such as a token or an id, carry to later steps in the same pass through the scenario.

Each region runs the full number of users on its own. Fifty users across three regions is 150 users hitting your API at once.

What the numbers mean

Response time: p50, p90, p95, p99

Every request takes a certain time to get its response back. Qodex lines up all of a region’s response times from fastest to slowest and reports these points along that line:

Label	Meaning
min	The fastest request.
p50	Half the requests were faster than this. Also called the median. It shows a typical request.
p90	90% of requests were faster than this.
p95	95% of requests were faster than this. Only 1 in 20 was slower.
p99	99% of requests were faster than this. Only 1 in 100 was slower.
max	The slowest request.

Why p95 and not the average? An average hides slow requests. If 19 requests take 100ms and one takes 5 seconds, the average is about 350ms, which looks fine, yet one in twenty of your users waited 5 seconds. The p95 shows that slow edge, which is what real users notice. That is why the pass rule is written on p95.

Example. A p95 of 840ms means 95% of requests came back in under 840ms and 5% took longer. With a threshold of 1000ms, this region held its p95 threshold.

Times are written in milliseconds (412ms) or, from one second up, in seconds (1.40s).

Requests and RPS

Label	Meaning
Requests	How many HTTP requests the region sent during the whole run, ramps included. A scenario with 3 steps sends 3 requests each time a user goes through it.
RPS	Requests per second: Requests divided by the run’s full length, ramps included. It is lower than your peak rate, because the ramps send less traffic than the hold.

Error rate

The Error rate is the share of requests that failed. A request counts as failed when:

• it got no response at all (the host could not be reached, the connection was refused, or the secure connection could not be set up), or
• it got a response with a status of 400 or higher (4xx or 5xx).

Shown as a percentage: 0.40% means 4 out of every 1,000 requests failed.

A status of 400 or higher counts as a failed request even when your scenario expects it. A step that checks “the status is 404” passes its check but still adds to the error rate. If such steps make up much of your scenario, the errors < 1% threshold will not hold. Load test the steps that should succeed.

Checks rate

Every step of your scenario has checks: its expected status code plus any assertions you added, such as “the body contains an id”. The Checks rate is the share of all those checks, across every request in the run, that passed.

A request can get a fast 200 and still fail a check, for example when the body is missing a field under load. A falling checks rate is a sign your API starts returning wrong answers when it is busy, not just slow ones.

Status codes

The Status codes row counts the responses by class:

Class	Meaning
2xx	Success.
3xx	Redirect.
4xx	The request was refused or not found: for example 401 (not signed in), 404 (not found), or 429 (too many requests, often a rate limit).
5xx	The server failed: for example 500 (internal error), 502 or 503 (the server or a proxy in front of it is overloaded or down), or 504 (timed out upstream).

Many 429s usually mean a rate limit in front of your API is turning away the load test’s traffic. Many 5xx under load usually mean the API or its database ran out of capacity.

Thresholds

The three thresholds are the pass rule. Each one is checked separately in each region, and the result is shown on the run’s page as a chip:

Chip	Meaning
✓ (green)	The threshold held.
✗ (amber)	The threshold was breached: the value went past the limit.
not measured, no responses (grey)	No request got a response, so there was no response time to compare. This is neither a pass nor a breach.

The chips use k6’s own names:

Chip text	Threshold	Breached when
http_req_duration: p(95)<1000	p95 < 1000ms	The region’s p95 was 1000ms or more.
http_req_failed: rate<0.01	errors < 1%	1% or more of the region’s requests failed.
checks: rate>0.99	checks > 99%	1% or more of the region’s checks failed.

Runs started from the page use these defaults. Run again keeps the thresholds of the run it repeats.

Run statuses

A run has one status for the whole run, worked out from its regions.

Status	Meaning
Queued	Waiting to start.
Running	Sending traffic.
Stopping	A region was asked to stop and has not confirmed it yet. It may still be sending traffic.
Passed (on a region) / All passed (on the run)	Every region held all three thresholds.
Thresholds breached	The run finished, and every region breached at least one threshold. Your API handled the traffic, but too slowly or with too many failures.
Mixed results	The regions finished differently: for example one passed and another breached, or one breached and another errored. Open the run to see each region.
Error	The run could not produce a result. No numbers were measured, or not enough to judge. Hover the pill, or open the run, to see the reason.
Cancelled	Someone stopped the run before it finished. Hover the pill to see when.

Thresholds breached is about your API. Error is about the run. A breached run measured your API and found it over a limit. An errored run never got a result, for example because the scenario could not be turned into a load test or the region could not start, so there is nothing to judge.

The Load tests page

The usage line

Under the page title, for example Startup · 1 of 1 load run this month · resets Nov 1. It shows your plan, how many load runs you have used in the current window, and when the next one becomes available. Hover it for which runs count.

The five tiles

Tile	What it shows
Runs used	Load runs used out of your allowance in the current window.
Passed	How many finished runs passed, out of the finished runs on the list, for example 3 of 5. Cancelled, errored and unfinished runs are left out, and the tile says how many errored runs it left out.
Latest p95	The p95 of the newest finished run, taken from its slowest region, with how much it changed against the previous run of the same scenario (for example ↑ 120ms vs its previous run).
Error rate	The error rate of the newest finished run, from the region with the highest one, next to the threshold.
Peak load	The most simulated users any run on the list sent at once: users per region × regions that sent requests, for example 50 × 3 regions = 150 users.

The tiles count the recent runs on the list, not every run the project ever made.

p95 across recent runs

A bar for each recent finished run, oldest on the left. Each bar is that run’s slowest-region p95. A grey bar held the run’s p95 threshold; an amber bar went over it. A rising line of bars means your API is getting slower from run to run.

Scenarios under load

One row per scenario with its newest run: the Last result, its p95, how many Runs of it are on the list, and Run again.

Recent runs

Column	Meaning
Started	When the run started. Hover for the exact time.
Scenario	The scenario under load.
Regions	The regions it ran from. Hover a region for its own status.
Users	Concurrent users per region.
Duration	The hold time.
Requests	Requests sent, all regions added together.
Worst p95	The p95 of the slowest region. The run is judged on it, because a run passes only when every region holds.
Worst error rate	The highest error rate of any region.
Status	The run’s status. A Not charged label beside it means the run did not use your allowance.

A cell that reads No responses means requests were sent and none got an answer, so there is no response time to show. See No responses.

The run page

Click any run to open it. The top line shows the users, the hold and ramp, the number of regions, when it started and how long it took. Cancel stops a run in progress; Run again repeats a finished one.

Below that is one panel per region with:

• the region’s own status;
• Requests, RPS, p95, p99, Error rate and Checks rate;
• Latency: bars for min, p50, p90, p95, p99 and max, so you can see how far the slow end stretches;
• Status codes;
• Thresholds: a chip for each of the three;
• Error: the reason, when the region errored;
• Full report: k6’s own detailed report for that region, opened in a new tab.

Under the panels, Recent runs of this scenario lists earlier runs so you can compare them.

No responses

No responses means the run sent requests and not one of them got an HTTP answer. That almost always means the region could not reach your API: the base URL is wrong, the host is private or behind a firewall or VPN, or it only allows traffic from certain IP addresses.

In that case there is no response time at all, so Qodex shows No responses instead of a misleading 0ms, and the p95 threshold reads not measured, no responses rather than passed.

To fix it, check the environment’s base URL, and make sure the API can be reached from the public internet in the regions you picked. The Full report shows the exact connection errors.

Your load run allowance

Your plan includes a number of load runs per window (for example per month). The usage line and the Runs used tile show where you are.

• A run counts once, however many regions it uses.
• A run that never started, or that was cancelled or errored within its first minute without sending any measured load, does not count. It shows Not charged beside its status.
• A run cancelled or errored after that does count, because the load was already sent.
• When the allowance is used up, Start new run is turned off and says when the next run is available.

If your plan includes no load runs, the page says so and offers Request upgrade.

Common questions

My run says Thresholds breached. Is something broken in Qodex?

No. The test ran to the end and measured your API. At least one region went past a limit: p95 at 1000ms or more, 1% or more of requests failed, or 1% or more of checks failed. Open the run and look at the Thresholds chips in each region: the amber ✗ shows which limit was crossed.

Every request returned 200, so why did the run fail?

The p95 threshold is about speed, not success. If 5% or more of requests took a second or longer, the p95 threshold is breached even when every response was a 200. Look at the Latency bars: p95 at or above 1.00s is the cause.

The scenario passes when I run it on its own, but the load test shows a high error rate.

Three common causes. Under load your API may return 429 (rate limited) or 5xx (overloaded): check Status codes. A step that expects a 4xx status still counts as a failed request (see Error rate). Or the run cannot reach the host at all: see No responses.

Why is the p95 different in each region?

Each region measures from its own location. A region far from your servers adds network distance to every request, so its p95 is higher. The page judges the run on its slowest region, because your users in that part of the world see that time.

What is the difference between Thresholds breached and Error?

Thresholds breached means the run measured your API and it was over a limit. Error means the run could not produce a result at all, so nothing was judged. Hover the Error pill, or open the run, to see why.

What does Mixed results mean?

The regions finished differently, for example one passed and another breached, or one breached and another errored. Open the run to see each region’s panel.

Why is RPS lower than I expected?

RPS is averaged over the whole run, including the 30-second ramps when fewer users are active. Each user also waits half a second before going through the scenario again, and a slow API means each user gets through fewer requests.

Can I change the thresholds?

Runs started from the page use the defaults: p95 < 1000ms, errors < 1%, checks > 99%. Run again keeps the thresholds of the run it repeats.

Related

API scenarios

Build the API scenario a load test replays.

Run tests

Run scenarios once, on a schedule, or from CI.