> ## Documentation Index
> Fetch the complete documentation index at: https://www.activepieces.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Benchmark
> Warm webhook throughput of the worker-is-the-sandbox model on GKE as the fleet scales at the recommended 1:10 app-to-worker ratio
This benchmark answers one question for the recommended production shape: **at the 1:10 app-to-worker ratio, how does throughput scale as you grow the fleet from 40 to 160 workers?** It runs the [worker-is-the-sandbox](/docs/install/architecture/workers) model on a real GKE cluster, against a same-region object store with signed URLs and official piece tarballs served from the CDN.
*The shape under test: app tier, Redis job queue, Postgres, S3, and a one-flow-per-worker execution tier.*
## What's measured
A 4-node **synchronous** webhook flow that holds the HTTP connection open until the flow returns:
Catches the request on a `/sync` URL and holds the connection until the flow finishes.
Adds `2 + 3`.
Runs `return inputs.sum + 1` inside an `isolated-vm` context.
Returns the result, closing the held connection.
The compute is sub-millisecond by design — everything measured below is orchestration (queueing, callbacks, sandbox boot), which is what actually shapes production latency.
Each fleet size is held at the recommended **1:10** ratio (1 app per 10 workers) and run **warm** (`AP_REUSE_SANDBOX=true`) — the engine process is reused between jobs.
## Results
Peak warm throughput — 16 apps, 160 workers.
Per worker, held flat from 40 to 160 workers — throughput scales linearly with the fleet.
Each worker is one sandbox at concurrency 1, hard-capped at 0.5 vCPU / 1 GB. Apps are 1 vCPU / 1 GB. Load concurrency is matched to the worker count so requests don't queue behind the concurrency-1 workers.
| Apps · Workers | Ratio | Warm req/s | Warm req/s **per worker** |
| -------------------- | ----- | ---------- | ------------------------- |
| 4 app · 40 workers | 1:10 | 185.3 | 4.6 |
| 8 app · 80 workers | 1:10 | 409.5 | 5.1 |
| 12 app · 120 workers | 1:10 | 553.0 | 4.6 |
| 16 app · 160 workers | 1:10 | 686.3 | 4.3 |
### What each tier ran — and what it was actually doing
Only the **app** and **worker** counts scale (1:10). **Postgres and Redis are a single fixed-size pod each — the same for every row below.** CPU is the average across three warm load tests; the singletons' figures are the whole pod, app/worker are per pod.
| Apps · Workers | Warm req/s | Postgres `used / cap` | Redis `used / cap` | App `used / cap` (per pod) | Worker `used / cap` (per pod) |
| -------------- | ---------- | --------------------- | ------------------ | -------------------------- | ----------------------------- |
| 4 · 40 | 185 | 522m / 3000m | 134m / 2000m | 782m / 1000m | 102m / 500m |
| 8 · 80 | 410 | 640m / 3000m | 150m / 2000m | 518m / 1000m | 72m / 500m |
| 12 · 120 | 553 | 546m / 3000m | 132m / 2000m | 311m / 1000m | 50m / 500m |
| 16 · 160 | 686 | 396m / 3000m | 169m / 2000m | 205m / 1000m | 37m / 500m |
Postgres never crosses \~0.65 of a core and Redis never crosses \~0.17, both far below their caps and **flat as the fleet quadruples** — they are not absorbing a growing share of anything. Workers sit at ≤0.1 of their 0.5-core cap. No tier approaches saturation, which is exactly why each added worker keeps adding throughput. (The singletons are sized this large on purpose — see [Test environment](#test-environment) — so they provably stay off the critical path; the default Postgres `max_connections=100` *would* cap the fleet at \~10 apps, which is the artifact behind the earlier "120 cliff".)
## How throughput scales
* **Warm scales linearly with the fleet.** Per-worker throughput stays flat at \~4.5 req/s from 40 to 160 workers, so total throughput tracks the worker count (185 → 410 → 553 → 686; 3.7× for 4× the fleet). The shared Postgres and Redis singletons are **not** the wall — they sit near-idle at every fleet size (Postgres under 0.6 of a core, Redis under 0.2, both far below their caps), and raising their resources several-fold does not move the curve. The ceiling is the **concurrency-1 worker model**: each worker is busy for the *whole* per-flow time — engine run plus the end-of-run run-log persistence it finishes before taking the next job — so fleet throughput is `workers ÷ per-flow-time`, which is linear in the fleet. (The synchronous response reaches the client sooner than that — it is sent at the response step, before the worker wraps up the log write — so client-perceived latency is lower than the worker-busy time that sets throughput.) Per-flow time carries run-to-run variance (the object-store log-write tail), which is why a single run's curve looks bumpy; the invariant that the per-worker rate holds constant is what shows the scaling is linear.
**Why Production Setup recommends 1:10.** Apps at 1 vCPU are cheap relative to the worker fleet, and 1:10 is the warm-headroom margin that keeps the app tier from becoming the wall during bursts. See [Production Setup](/docs/install/configure-operate/production-setup).
## Latency anatomy
Where the worker's milliseconds go — warm at peak (16 app · 160 w):
| Layer | Warm |
| ----------------------------------------------------------------- | ------------ |
| Provision (flow bundle + piece + engine, mostly disk-cache hits) | \~10 ms |
| Sandbox boot (engine process reused) | \~5 ms |
| Flow run (4 steps: engine→app callbacks + end-of-run log persist) | \~203 ms |
| **Worker-busy avg per job** | **\~218 ms** |
This is the time the **worker** is occupied per job — and at concurrency 1 it is what sets throughput (`workers ÷ worker-busy-time`). The synchronous client sees less: the response is published at the flow's response step, before the worker finishes persisting the run log, so client-perceived latency runs below the worker-busy figure.
## Test environment
* **Cluster**: GKE `n2-standard-16` × 10 nodes, `europe-west1-b`
* **Worker**: 0.5 vCPU / 1 GB, concurrency 1, `SANDBOX_CODE_ONLY` (Node fork + `isolated-vm`)
* **App**: 1 vCPU / 1 GB
* **Object store**: same-region GCS bucket (`europe-west1`) over the S3-interop endpoint, path-style SigV4 presigned URLs (`AP_S3_USE_SIGNED_URLS=true`)
* **Piece bundles**: official tarballs served from the Activepieces CDN (`AP_USE_CDN_FOR_BUNDLES=true`)
* **Postgres + Redis**: in-cluster singletons, deliberately over-provisioned so they stay off the critical path — Postgres at 3 vCPU / 3 GB with `max_connections=2000` (the default 100 would starve the app pools past \~10 apps), durability off, and its data dir on tmpfs; Redis at 2 vCPU / 2 GB with `io-threads`. Under load both stay near-idle (Postgres `<0.6` core, Redis `<0.2`), confirming the worker tier, not the singletons, is the ceiling.
* **Load**: `hey`, concurrency matched to worker count (40/80/120/160) so requests don't queue behind the concurrency-1 workers — latency reflects real service time, not backlog
## How to reproduce
```bash theme={null}
benchmark/run-gke.sh [total_requests] [concurrency]
```
The script mints a worker token, deploys `benchmark/k8s-sandbox.yaml` to the cluster, runs the load test against the app LoadBalancer, and reports warm throughput and the per-run breakdown from worker-pod logs. Set `APP_REPLICAS` and `WORKER_REPLICAS` (keeping the 1:10 ratio) to reproduce any row in the results table.
This benchmark runs in `SANDBOX_CODE_ONLY` mode. It does **not** represent the performance of Activepieces Cloud, which uses a different sandboxing mechanism for multi-tenancy. See [Sandboxing](/docs/install/architecture/sandboxing).
## Benchmark your own installation
Load-test and **diagnose** your own deployment — no cluster scripts required — with the CLI. It publishes a synchronous flow (webhook trigger → data mapper → return response), fires load at its sync webhook endpoint with [autocannon](https://github.com/mcollina/autocannon), and returns one self-contained diagnostic bundle you can hand to support.
```bash theme={null}
AP_API_KEY= npx @activepieces/cli@latest benchmark --url https://your-instance.example.com
```
The CLI provisions a **throwaway project** for the run (with a high concurrency cap so a project rate limiter can't queue-throttle the numbers) and deletes it when finished — nothing is left behind, and it never touches your real projects. The API key must be a platform-admin key.
**Why the numbers are trustworthy.** The CLI runs from a different region than your servers, so its client-side latency is polluted by network distance. The numbers that matter are measured server-side instead: the QUEUE/PROVISION/BOOT/RUN split is timed *inside the worker*, and DB/Redis/S3 round-trips are measured *in-region* by an admin diagnostics endpoint. Client numbers are shown but marked observational.
**Concurrency defaults to your execution slots** (Σ `AP_WORKER_CONCURRENCY` across connected workers) so requests don't queue and you read real service time, not backlog. Comparing two deployments? Match concurrency to each one's own slots — never a fixed number, which makes the smaller one queue.
| Option | Default | Description |
| --------------- | ----------------------- | ---------------------------------------------------------- |
| `--url` | `http://localhost:3000` | Base URL of your instance |
| `--api-key` | `AP_API_KEY` env var | Platform-admin API key (Bearer) — **required** |
| `--concurrency` | auto = execution slots | Concurrent connections |
| `--requests` | `40 × concurrency` | Total requests to fire |
| `--body` | `{"test":true}` | JSON body sent to the webhook |
| `--json` | | Emit the full machine-readable bundle (share with support) |
### Reference numbers
A **real run against the recommended GKE deployment** from the [results table](#results): 4 workers @ 0.5 vCPU / 1 GB, concurrency 1, `SANDBOX_CODE_ONLY`, `AP_REUSE_SANDBOX=true`, same-region GCS with signed URLs, warm, load = `concurrency 4 (= slots) × 200 requests`.
Run the CLI against your own deployment and compare **tier by tier**. A number several times larger localizes the problem: `RUN` ≫ 200 ms means a heavier flow or a CPU-starved worker; `storage` ≫ 240 ms means a mis-regioned or throttled object store; a large `QUEUE` with climbing `queue depth` means you drove more concurrency than you have slots.
```text theme={null}
Version & health
app version : 0.86.2 (latest available: 0.86.2) — all workers match app version
health : app-cpu=ok app-ram=ok disk=ok worker-cpu=ok worker-ram=ok db=ok
Infra round-trip (server-measured, in-region — authoritative, not reachable from the CLI)
database : 21 ms
redis : 17 ms
storage : 237 ms (S3/GCS write+read round-trip; same cost is inside every RUN as the end-of-run log backup)
config : execution=SANDBOX_CODE_ONLY storage=S3 signedUrls=true sandboxMemKB=1048576 s3=https://storage.googleapis.com/europe-west1
workers : 4 connected
- ONLINE worker-z | 0.5 core | cpu 100.0% | ram 13.8%
- ONLINE worker-N | 0.5 core | cpu 63.9% | ram 15.2%
- ONLINE worker-u | 0.5 core | cpu 100.0% | ram 15.5%
- ONLINE worker-e | 0.5 core | cpu 100.0% | ram 15.5%
Config flags (server-reported)
EDITION "ce" DEFAULT_CONCURRENT_JOBS_LIMIT 1000 PROJECT_RATE_LIMITER_ENABLED false
FLOW_RUN_MEMORY_LIMIT_KB 1048576 WEBHOOK_TIMEOUT_SECONDS 30 FLOW_RUN_TIME_SECONDS 600 ...
Setup
workers online : 4, execution slots : 4
[PASS] sandbox mode / reuse sandbox / worker concurrency / worker CPU / worker RAM (all match the recommended shape)
Network (CLI -> server, cross-region)
RTT min / p50 : 43.0 / 46.9 ms over 20 probes
conc 4 (= slots)
throughput : 10.0 req/s (200 reqs in 20.0s)
run outcomes : 200 SUCCEEDED (server-truth: 200 2xx, 0 non-2xx, 0 errors, 0 timeouts)
queue depth : max waiting 3, max active 4, avg waiting 1 (sampled server-side during load)
worker-measured latency (authoritative — each phase timed inside the worker, 200 runs):
QUEUE p50 100 ms — wait for a free execution slot (±app↔worker clock skew; cross-check the queue-depth)
PROVISION p50 1 ms — piece install / cache provision
BOOT p50 0 ms — engine fork + Node boot + isolate + socket connect
RUN p50/p90 201 / 244 ms — engine executes the flow, incl. end-of-run S3 log backup
=> queue-wait p50 102 ms vs service p50 201 ms => verdict: service-bound
observational (CLI-side, cross-region — NOT authoritative): client latency p50/p90/p99 256/311/5638 ms
Storage (log persistence)
50/50 sampled runs have a persisted log — the worker->storage write path is healthy
```
Each section answers one question:
| Section | What it measures | Where it's measured |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| **Version & health** | App release, whether every connected worker matches it (a version-skewed worker is silently withheld jobs), and app/worker/DB CPU-RAM-disk health | `GET /v1/health/system` |
| **Infra round-trip** | Authoritative **in-region** DB / Redis / **S3** write+read latency, the effective execution/storage config, the **app tier** (every app replica's CPU / RAM / disk / event-loop — apps self-register into a diagnostics cache on their metrics tick), and the worker fleet with live per-worker CPU. The S3 round-trip is the same cost folded into every run's end-of-run log backup — a slow object store shows up here **and** in RUN. Self-hosted only. | `GET /v1/health/diagnostics` |
| **Config flags** | The limits and throttles that silently cap throughput — a `PROJECT_RATE_LIMITER_ENABLED` or a low `DEFAULT_CONCURRENT_JOBS_LIMIT` re-queues jobs (latency, not errors); the memory/timeout/log ceilings that turn into `MEMORY_LIMIT_EXCEEDED` / `TIMEOUT` / `LOG_SIZE_EXCEEDED` statuses | `GET /v1/flags` |
| **Setup** | Execution slots (Σ `AP_WORKER_CONCURRENCY`) and a PASS/WARN check of every worker's specs, sandbox mode, reuse, and concurrency against the recommended shape | `GET /v1/worker-machines` |
| **Network** | CLI→server round-trip, so cross-region distance is quantified and subtracted rather than blamed on the server | timed `GET /v1/flags` |
| **Load + latency split** | Throughput, run-status outcomes (server truth, not just HTTP codes), live **queue depth** during load, and the per-run latency split into **QUEUE / PROVISION / BOOT / RUN** — measured inside the worker, so it's the same on any deployment regardless of where the CLI runs. The verdict says whether latency is **queue-bound** (too much concurrency for the slot count) or **service-bound** (real engine time) | `FlowRun.timeline` + `GET /v1/worker-machines/queue-metrics` |
| **Storage** | Fraction of runs whose logs were persisted — proves the worker→storage write path works end to end | `FlowRun.logsFileId` |
Here the read is unambiguous: workers pegged at \~100% of their 0.5-core limit and RUN ≈ 200 ms dominate, while QUEUE (\~100 ms at concurrency = slots) and the infra round-trips are small — the deployment is **service-bound on worker CPU**, so the lever is more/bigger workers, not a code change.
* Exits non-zero if any request fails — usable as a CI gate.
* The throwaway project (and its benchmark flow) is deleted automatically when the run finishes.
If your hardware or sandbox mode differs from the recommended shape, the absolute numbers shift — but the **shape** holds: match concurrency to your own slot count so nothing queues, then read whether you are queue-bound or service-bound and which tier dominates.