WOMMWorks On My Machine · onmymachineitworks.comSample · fictional data

Test report · Performance and load

Acme Ledger API · Load, soak, spike and stress

Meets SLOs at expected peak
Environment
Performance env sized as production (2 API pods × 4 vCPU; Postgres 8 vCPU)
Build
4.2.0-rc3
Tool
k6 v0.54 · scripts in acme/qa-perf · 2 load generators (eu-central)
Executed
2026-10-21 – 2026-10-22
Traffic model
Mix from 7 days of production logs; 24 endpoints, think time 3–8 s
SLOs
p95 < 500 ms for API calls, p95 < 2 s for PDFs and reports, errors < 0.5%
Prepared by
Tomasz Kordiak (QA lead)

Summary

Ready for expected peak, with 1.5× headroom

At the expected peak (300 concurrent users, 236 req/s) all SLOs are met: p95 447 ms, errors 0.04%. The system stays within SLOs up to about 450 users.

Beyond ~550 users the database connection pool is exhausted and errors climb to 6.8%. The invoice PDF endpoint is the first to degrade: 1.98 s at expected peak against a 2 s SLO.

Recommendation: release 4.2 as planned; fix ACME-525 and ACME-526 before the January billing peak, which marketing expects at roughly 2× today's traffic.

Key numbers

236 req/sAt expected peak
447 msp95 at peak
0.04%Errors at peak
~450Users within SLO
~550Breaking point
16/20Thresholds passed

Latency as load increases

020004000600080000225450675900PDF SLO 2 sConcurrent virtual usersp95 latency (ms)
p95 · all API callsp95 · invoice PDF

Stress ramp, 0 → 900 users over 25 minutes. PDF requests (6% of traffic) time out at 8 s above 650 users.

Scenarios

ScenarioProfileReq/sp95 (API)ErrorsSLO
Baseline50 VUs · 10 min41290 ms0.00%Met
Load (expected peak)ramp to 300 VUs · 30 min236447 ms0.04%Met
Soak200 VUs · 2 h158402 ms0.02%Met
Spike50 → 600 VUs in 30 s · 5 min4101,840 ms1.90%Missed briefly
Stressramp to 900 VUs · 25 min5124,870 ms6.80%Missed (by design)

Stress and spike are meant to go past the limits; they show where and how the system fails.

Defects 6 raised

KeySummarySeverityStatusComponentFound byOwnerCreated
ACME-525DB connection pool exhausted above ~550 concurrent users (503s)
Planned before the January peak.		HighOpenAPI platformStressPlatform22 Oct
ACME-526Invoice PDF rendered on every request; p95 1.98 s at peakMediumOpenPDFLoadBackend22 Oct
ACME-527Unhandled timeout returns 500 instead of 503 under stressMediumOpenAPI platformStressPlatform22 Oct
ACME-528Search p95 doubles with tenant size; no index on invoices.numberMediumOpenSearchLoadBackend22 Oct
ACME-529Revenue report scans all invoices of the tenantLowOpenReportsLoadBackend22 Oct
ACME-530Spike: autoscaling adds pods after 90 s, too late for 30 s burstsMediumOpenInfrastructureSpikePlatform22 Oct

Observations

IDAreaObservationImpactSuggested action
OBS-01MemoryMemory grew 16 MB in 2 hours (0.3%/h) and flattened after GC; no leak.Positive–
OBS-02DatabaseDB CPU stays below 50% even under stress; the database is not the bottleneck.InfoDon't scale the database; fix the pool (ACME-525).
OBS-03CachingGET /products and /users/me are good cache candidates (rarely change, 8% of traffic).LowAdd HTTP caching headers.
OBS-04Rate limitingPer-client rate limit (100/min) never triggered: load came from many simulated clients.InfoTest one heavy API client separately if a customer integrates in bulk.
OBS-05LogsUnder stress, request logging produced 3 GB per hour.CostSample 2xx logs at 10% above 300 rps.
OBS-06Front endNot in scope: browser rendering and Core Web Vitals.CoverageLighthouse CI on key pages (separate task).

Thresholds 16 of 20 passed

k6 thresholdScenarioMeasuredResult
http_req_duration p(95) < 500 ms (API)Load447 msPass
http_req_duration p(99) < 1000 ms (API)Load812 msPass
http_req_failed rate < 0.5%Load0.04%Pass
checks rate > 99.5%Load99.96%Pass
pdf p(95) < 2000 msLoad1,978 msPass
reports p(95) < 2000 msLoad1,620 msPass
login p(95) < 500 msLoad210 msPass
iteration_duration p(95) < 25 sLoad19.4 sPass
http_req_failed rate < 0.5%Soak0.02%Pass
memory growth < 5% per hourSoak0.3%Pass
http_req_duration p(95) < 500 ms (API)Soak402 msPass
db connections < 80% of poolSoak65%Pass
http_req_failed rate < 2% during spikeSpike1.9%Pass
recovery to baseline p95 < 60 sSpike38 sPass
http_req_duration p(95) < 1500 ms during spikeSpike1,840 msFail
no 5xx after spike endsSpike0Pass
http_req_failed rate < 1%Stress6.8%Fail
http_req_duration p(95) < 2000 msStress4,870 msFail
graceful degradation (429/503, no 500)Stress312 × 500Fail
service recovers within 2 min after stressStress70 sPass

Ramp steps

VUsReq/sp50p95ErrorsAPI CPUDB CPUDB conns / pod
5041118 ms290 ms0%18%22%6 / 20
10082124 ms318 ms0%27%24%9 / 20
150120131 ms344 ms0%36%27%11 / 20
200158141 ms371 ms0%45%30%13 / 20
250197152 ms409 ms0.01%55%33%16 / 20
300236164 ms447 ms0.04%63%35%18 / 20
350272178 ms490 ms0.06%70%37%19 / 20
400306196 ms538 ms0.09%76%39%20 / 20
450338221 ms610 ms0.18%81%41%20 / 20
500366262 ms760 ms0.6%84%42%20 / 20
550384340 ms1,180 ms1.9%85%43%20 / 20
600392520 ms2,260 ms3.8%85%44%20 / 20
650398760 ms3,720 ms5.1%86%45%20 / 20
700401905 ms4,410 ms5.9%86%45%20 / 20
8004051,080 ms4,790 ms6.4%86%46%20 / 20
9004081,210 ms4,870 ms6.8%87%46%20 / 20

Throughput flattens at ~400 req/s once all 20 connections per pod are busy, while DB CPU stays under 50%.

By endpoint and scenario 120 results

EndpointShareSLOBaselineLoadSoakSpikeStress
GET /invoices19%500 ms2103363116382,083
GET /invoices/{id}12%500 ms95143133271884
POST /invoices5%500 ms2604684268892,902
PATCH /invoices/{id}2%500 ms2303913597432,424
POST /invoices/{id}/send2%500 ms3004804449122,976
GET /invoices/{id}/pdf6%2,000 ms8601,9781,7543,75812,264
GET /customers8%500 ms1402242074261,389
GET /customers/{id}5%500 ms80112106213694
POST /customers1%500 ms1903042815781,885
GET /products4%500 ms90117112222725
GET /payments4%500 ms1502402224561,488
POST /payments4%500 ms2203963617522,455
POST /payments/{id}/refund1%500 ms2804764379042,951
GET /credit-notes2%500 ms1402101963991,302
GET /recurring2%500 ms1201681583191,042
GET /reports/revenue3%2,000 ms6401,1201,0242,1286,944
GET /reports/aged-receivables1%2,000 ms7201,2961,1812,4628,035
GET /reports/revenue.csv1%2,000 ms9001,7101,5483,24910,602
GET /dashboard/summary6%500 ms2604424068402,740
GET /users/me4%500 ms607874148484
POST /oauth/token4%500 ms1402101963991,302
GET /webhooks1%500 ms90117112222725
POST /files1%500 ms1802882665471,786
GET /search2%500 ms2304834329182,995

p95 in ms. 120 endpoint × scenario results: green within 90% of the SLO, amber close to it, red over it.

Errors

Scenario500502/504 (timeouts)503 (pool)k6 timeoutsTotal
Baseline–––––
Load (expected peak)–61––61
Soak–––228228
Spike–1,1042,876–3,980
Stress3124,87028,14022,41055,732

500s under stress come from unhandled timeouts (ACME-527); they should be 503 with Retry-After.

Resource usage

ScenarioAPI CPU (max)API memoryDB CPU (max)DB connections / podQueue depth
Baseline18%410 MB11%6 / 200
Load (expected peak)63%520 MB35%18 / 200
Soak46%505 → 521 MB30%13 / 200
Spike85%560 MB41%20 / 201,400 peak
Stress87%590 MB46%20 / 209,800 peak

Soak trend

WindowReq/sp95ErrorsAPI memory
0:00–0:15155398 ms0.00%505 MB
0:15–0:30158401 ms0.02%507 MB
0:30–0:45158399 ms0.01%509 MB
0:45–1:00159404 ms0.03%511 MB
1:00–1:15158402 ms0.02%513 MB
1:15–1:30157405 ms0.02%516 MB
1:30–1:45158403 ms0.03%518 MB
1:45–2:00158406 ms0.02%521 MB

Stable for 2 hours at 200 users: no latency drift, memory flat after garbage collection.

Bottlenecks

High PERF-01 · ACME-525 Database connection pool exhausted above ~550 users

Observed
From 550 users, 503 and timeout acquiring connection; errors 6.8% at 900 users. DB CPU stays at 46%.
Evidence
Pool metrics: 20/20 connections busy for 11 minutes; API pod logs; Grafana snapshot attached.
Recommendation
Pool 40 per pod (DB allows 200) and PgBouncer; re-run stress. Expected breaking point ~900 users.

Medium PERF-02 · ACME-526 Invoice PDF is rendered on every request

Observed
p95 1.98 s at expected peak (SLO 2 s), 4.3 s at 500 users; API CPU 85% when the PDF share rises.
Recommendation
Cache the rendered PDF per invoice version, or render on invoice creation.

Medium PERF-03 · ACME-530 Autoscaling reacts too slowly to spikes

Observed
New pods ready after 90 s; a 30-second burst to 600 users pushed p95 to 1.84 s and errors to 1.9%.
Recommendation
Scale on queue depth, pre-scale before known peaks (month-end, campaigns).

Recommendations

ChangeDefectExpected gainEffort
Raise DB pool to 40 per pod, add PgBouncerACME-525Breaking point ~550 → ~900 users1–2 days
Cache rendered PDFs per invoice versionACME-526PDF p95 1.98 s → < 300 ms (cached)2–3 days
Return 503 with Retry-After on timeoutsACME-527No 500s under overload; clients retry correctly0.5 day
Index invoices(tenant_id, number) for searchACME-528Search p95 −60% on large tenants0.5 day
Pre-scale before known peaks; scale on queue depthACME-530Spike p95 within 1.5 s1 day
Composite index on (tenant_id, issued_at)ACME-529Report p95 1.6 s → ~0.6 s0.5 day

Environment and method

Test data40 tenants, 210,000 invoices, 9,000 customers from the seed script; no production data
Traffic modelEndpoint shares and think times from 7 days of anonymised access logs
Load generators2 × c6i.2xlarge in eu-central-1, same region as the environment
MonitoringGrafana (API, DB, pool metrics), k6 output to Prometheus
Not coveredBrowser rendering and Core Web Vitals, third-party payment latency (stubbed), multi-region traffic
What you keepk6 scripts, thresholds and the Grafana dashboard in your repository; a 50-VU load test runs nightly as a regression check

Sample report. Acme Ledger, its data, people and defects are fictional. Your report shows your product, your tickets and your tools.
More sample reports · Book a 15-minute call