# Dispatch APM Server

Dispatch APM · server setup

What it needs, and how to set it up: on one machine of your own, on one EC2 instance, or across availability zones on AWS. Dispatch Studio alone needs none of this; a team watching staging and production does.

11 September 2026 · Linux · Java 21 · Kafka 4

## What you are installing

Agents report to ingest; ingest puts each report on Kafka; the server reads Kafka and answers Dispatch Studio. One HTTPS name serves both kinds of caller.

*Diagram: Agents in your JVMs and Dispatch Studio both reach the APM host over HTTPS through one TLS proxy. The proxy sends agents' reports to ingest, which puts them on Kafka, and everything else to the server, which reads Kafka.*

The proxy sends `/api/agent/*` and `POST /api/profile` to ingest, and everything else to the server. Nothing listens inside your applications.

| Piece | What it does | Artifact |
| --- | --- | --- |
| Ingest | The only thing agents talk to. It checks the application key, puts the report on Kafka and answers. It keeps no data of its own, so you can run as many as you like. | `dispatch-apm-ingest.jar` |
| Kafka | The buffer between ingest and the server. If the server stops, reports wait here for 24 hours by default. | Apache Kafka 4, or Amazon MSK |
| Server | Reads the reports, keeps the numbers, evaluates alerts and answers Studio. | `dispatch-apm-server-all.jar` |
| TLS proxy | One HTTPS name for agents and Studio, each sent to the right service. | Caddy, or an Application Load Balancer |

The server runs as **one instance**: size it for the whole installation.

[**One machine of your own** Docker Compose or systemd, Caddy for the certificate](https://studio-dispatch.com/docs/apm-server.html#one-machine) [**One EC2 instance** The same, with images in ECR and the token in Secrets Manager](https://studio-dispatch.com/docs/apm-server.html#ec2) [**Multi-AZ on AWS** Load balancer, ingest in an Auto Scaling group, Amazon MSK](https://studio-dispatch.com/docs/apm-server.html#multi-az)

## What it needs

One monitored JVM at the default agent settings sends about **0.2 MB a minute**, in about **one request every two seconds**. That was measured on the sample service; a service that logs heavily sends more.

### Machines

| Monitored JVMs | Reports a second | One machine for everything |
| --- | --- | --- |
| Up to 50 | ~25 | 2 vCPU, 8 GB RAM, 100 GB SSD |
| Up to 100 | ~50 | 4 vCPU, 16 GB RAM, 250 GB SSD |
| More than 100 | — | Split as in [multi-AZ](https://studio-dispatch.com/docs/apm-server.html#multi-az), and measure before relying on one server at that size |

On the 16 GB machine, the memory goes to Kafka (a 1 GB heap, plus the operating system's page cache, which is where Kafka gets its speed), ingest (512 MB) and the server (4–6 GB). Kafka's disk is roughly inbound MB/s × 86,400 × days kept × copies, plus 30%. For 100 JVMs kept 24 hours, that is about 30 GB.

### Software

- Operating system — Linux, x86-64 or ARM64: Ubuntu 22.04 or 24.04 LTS, Amazon Linux 2023, RHEL 9. Windows will run it for a trial, not for keeps.
- Java — 21 for ingest and the server (Amazon Corretto 21 or Eclipse Temurin 21). Kafka 4 needs 17 or later. The container images bring their own.
- Containers — Docker Engine 24 or later, with the Compose plugin, if you use them
- Clock — NTP or chrony on every machine, because profiling windows are wall-clock times

### Network

| From | To | Port | Why |
| --- | --- | --- | --- |
| Monitored JVMs | APM host | 443 | Agents' reports, outbound from the JVM |
| Studio users | APM host | 443 | Studio's reads and changes |
| Internet | APM host | 80, 443 | Only while Caddy obtains its Let's Encrypt certificate |
| Server | Your services | as registered | Only for applications registered with an address: the server reads their Actuator metrics |
| Ingest, server | Kafka | 9092 · 9096 | Kept inside the host or VPC; 9096 is MSK with SASL |

### A certificate your JVMs already trust

The agent reports over the JVM's own HTTPS, with its default trust store. The APM host therefore needs a certificate that store already trusts. That can be Let's Encrypt (Caddy obtains one by itself), AWS Certificate Manager behind a load balancer, or your company's CA if your JVMs already trust it. Anything else means changing every monitored application's trust store.

## On one machine of your own

Docker Compose runs Kafka, ingest, the server and Caddy. The files are in `deploy/docker/` in the repository.

1. **The machine.** Linux, sized from the table above, with a DNS name (here `apm.example.com`) pointing at it. Ports 80 and 443 need to be reachable from your JVMs and your Studio users, and from the internet for the certificate.
2. **Docker**, on Ubuntu from Docker's own repository:

```
sudo apt-get update && sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```

3. **The two images.** On a build machine with JDK 21, Maven and Docker, from the repository root:

```
mvn -pl dispatch-apm-ingest,dispatch-apm-server -am package -DskipTests
docker build -f deploy/docker/ingest.Dockerfile -t dispatch-apm-ingest:0.1.0 .
docker build -f deploy/docker/server.Dockerfile -t dispatch-apm-server:0.1.0 .
```

  Then carry them to the APM host, through your own registry or directly:

```
docker save dispatch-apm-ingest:0.1.0 dispatch-apm-server:0.1.0 | gzip | ssh apm-host 'gunzip | docker load'
```

4. **The configuration**, on the APM host:

```
sudo mkdir -p /opt/dispatch-apm && cd /opt/dispatch-apm
# copy compose.yaml, Caddyfile and env.example here from deploy/docker/
sudo cp env.example .env && sudo chmod 600 .env
openssl rand -hex 32        # the operator token
sudo nano .env              # DISPATCH_APM_HOST and DISPATCH_APM_SERVER_TOKEN
```

5. **Start it.**

```
sudo docker compose up -d
sudo docker compose ps
```

  The `topics` service should show *exited (0)*: it creates the eight Kafka topics and stops. The rest should be running, with ingest *healthy*. In `sudo docker compose logs ingest`, look for `report topics ready` and `tables loaded`.
6. **Check it from outside.**

```
TOKEN=…   # the operator token
curl -s -H "X-Dispatch-Token: $TOKEN" https://apm.example.com/api/apps
```

  `[]` means the server is up, the certificate works and the token is right.
7. **Register an application.** Each application gets a key, and its agents present it.

```
curl -s -X POST https://apm.example.com/api/apps \
  -H "X-Dispatch-Token: $TOKEN" -H 'Content-Type: application/json' \
  -d '{"name": "Orders", "baseUrl": ""}'
```

  The answer carries `"key": "dsk_…"`. Ingest learns the new key within a second. Put the service's own address in `baseUrl` if the server can reach its Actuator endpoint; otherwise leave it empty.
8. **Attach the agent** to each JVM of that application:

```
java -javaagent:/opt/dispatch-agent.jar=\
dispatch.apm.agent.report.url=https://apm.example.com,\
dispatch.apm.agent.report.key=dsk_… \
  -jar orders-service.jar
```

  The agent's other settings, and attaching to a JVM that is already running, are in `docs/SETUP.md` §6.
9. **Point Studio at it** with the address `https://apm.example.com` and the operator token, as in `docs/SETUP.md` §7.

### Day to day

- Logs — `sudo docker compose logs -f ingest server`
- Stop, start — `sudo docker compose stop` · `sudo docker compose start`
- Upgrade — Load the new images, set `DISPATCH_APM_VERSION` in `.env`, then run `sudo docker compose up -d`. While ingest restarts, agents get 503 for a few seconds, then send their next report as usual.
- Back up — The `server-data` volume holds applications and their keys, alert rules and history: `sudo docker run --rm -v dispatch-apm_server-data:/data -v "$PWD":/backup ubuntu tar czf /backup/server-data-$(date +%F).tgz -C /data .` Kafka's volume is only a buffer and needs no backup.
- Ingest's numbers — `/metrics` on port 8091 inside the host: requests by route and status, bytes, time to Kafka's acknowledgement, and keys known

### Without containers

The same services as systemd units, from `deploy/systemd/`:

```
# Java 21, and users for the services
sudo apt-get install -y openjdk-21-jre-headless
sudo useradd --system --home-dir /var/lib/dispatch-apm --create-home --shell /usr/sbin/nologin dispatch
sudo useradd --system --no-create-home --shell /usr/sbin/nologin kafka

# Kafka, one node
curl -fLO https://downloads.apache.org/kafka/4.3.1/kafka_2.13-4.3.1.tgz
curl -fLO https://downloads.apache.org/kafka/4.3.1/kafka_2.13-4.3.1.tgz.sha512
sha512sum kafka_2.13-4.3.1.tgz && cat kafka_2.13-4.3.1.tgz.sha512    # the two must match
sudo tar -xzf kafka_2.13-4.3.1.tgz -C /opt && sudo ln -s /opt/kafka_2.13-4.3.1 /opt/kafka
sudo mkdir -p /etc/kafka /var/lib/kafka /var/log/kafka
sudo cp /opt/kafka/config/server.properties /etc/kafka/server.properties
#   then, in /etc/kafka/server.properties:
#     log.dirs=/var/lib/kafka
#     auto.create.topics.enable=false
#     message.max.bytes=8454144
#     listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
#     advertised.listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
sudo /opt/kafka/bin/kafka-storage.sh format --standalone \
     -t "$(/opt/kafka/bin/kafka-storage.sh random-uuid)" -c /etc/kafka/server.properties
sudo chown -R kafka:kafka /var/lib/kafka /var/log/kafka
sudo cp deploy/systemd/kafka.service /etc/systemd/system/

# Ingest and the server
sudo mkdir -p /opt/dispatch-apm /etc/dispatch-apm
sudo cp dispatch-apm-ingest/target/dispatch-apm-ingest.jar /opt/dispatch-apm/
sudo cp dispatch-apm-server/target/dispatch-apm-server-all.jar /opt/dispatch-apm/dispatch-apm-server.jar
printf 'DISPATCH_APM_INGEST_BOOTSTRAP_SERVERS=localhost:9092\nDISPATCH_APM_INGEST_HOST=127.0.0.1\n' \
  | sudo tee /etc/dispatch-apm/ingest.env
printf 'DISPATCH_APM_SERVER_TOKEN=%s\nDISPATCH_APM_SERVER_HOST=127.0.0.1\nDISPATCH_APM_SERVER_KAFKA_BOOTSTRAP_SERVERS=localhost:9092\n' \
  "$(openssl rand -hex 32)" | sudo tee /etc/dispatch-apm/server.env
sudo chmod 600 /etc/dispatch-apm/*.env
sudo cp deploy/systemd/dispatch-apm-*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now kafka
DISPATCH_APM_INGEST_BOOTSTRAP_SERVERS=localhost:9092 \
  java -jar /opt/dispatch-apm/dispatch-apm-ingest.jar create-topics --partitions 6 --replication 1
sudo systemctl enable --now dispatch-apm-ingest dispatch-apm-server
```

Then install Caddy from its own package repository and use `deploy/docker/Caddyfile`, with `ingest:8091` and `server:8090` changed to `localhost:8091` and `localhost:8090`. Steps 6 to 9 above are the same.

## On one EC2 instance

The same Compose setup, in your own AWS account.

1. **Launch the instance.**
- AMI — Ubuntu Server 24.04 LTS, or Amazon Linux 2023
- Type — `m7i.xlarge` (4 vCPU, 16 GiB) for up to 100 JVMs; Graviton's `m7g.xlarge` works too
- Storage — 250 GiB gp3, encrypted
- Subnet — Private, when your JVMs and Studio users reach the VPC over a VPN or Direct Connect. Public with an Elastic IP, when they come over the internet.
- IAM role — `AmazonSSMManagedInstanceCore` for Session Manager, `AmazonEC2ContainerRegistryReadOnly` to pull the images, `CloudWatchAgentServerPolicy` for metrics and logs
2. **The security group.** Allow inbound 443 from your JVMs' subnets or addresses and from your Studio users. Allow 80 from anywhere only if you use Let's Encrypt (option A in step 7). Leave SSH closed and connect with `aws ssm start-session --target i-…`. Kafka, ingest and the server listen only on the instance's internal Docker network.
3. **A name.** A Route 53 record for `apm.example.com`: the Elastic IP for a public instance, or the private IP, in a private hosted zone, for a private one.
4. **The images, in ECR.** From the build machine, after building them as in step 3 of [one machine](https://studio-dispatch.com/docs/apm-server.html#one-machine):

```
REGISTRY=123456789012.dkr.ecr.ap-south-1.amazonaws.com
aws ecr create-repository --repository-name dispatch-apm-ingest
aws ecr create-repository --repository-name dispatch-apm-server
aws ecr get-login-password | docker login --username AWS --password-stdin $REGISTRY
for image in dispatch-apm-ingest dispatch-apm-server; do
  docker tag $image:0.1.0 $REGISTRY/$image:0.1.0 && docker push $REGISTRY/$image:0.1.0
done
```

5. **The token, in Secrets Manager**, so it is in neither a shell history nor a disk image:

```
aws secretsmanager create-secret --name dispatch-apm/server-token --secret-string "$(openssl rand -hex 32)"
```

  Give the instance role `secretsmanager:GetSecretValue` on that one secret.
6. **On the instance**, in a Session Manager shell: install Docker as in step 2 of [one machine](https://studio-dispatch.com/docs/apm-server.html#one-machine), and the AWS CLI (`sudo snap install aws-cli --classic` on Ubuntu; Amazon Linux already has it). Then:

```
sudo mkdir -p /opt/dispatch-apm && cd /opt/dispatch-apm
# compose.yaml, Caddyfile and env.example from deploy/docker/, e.g. by way of an S3 bucket
sudo cp env.example .env && sudo chmod 600 .env
REGISTRY=123456789012.dkr.ecr.ap-south-1.amazonaws.com
TOKEN=$(aws secretsmanager get-secret-value --secret-id dispatch-apm/server-token --query SecretString --output text)
sudo sed -i "s|^DISPATCH_APM_HOST=.*|DISPATCH_APM_HOST=apm.example.com|; \
             s|^DISPATCH_APM_SERVER_TOKEN=.*|DISPATCH_APM_SERVER_TOKEN=$TOKEN|; \
             s|^DISPATCH_APM_REGISTRY=.*|DISPATCH_APM_REGISTRY=$REGISTRY/|" .env
aws ecr get-login-password | sudo docker login --username AWS --password-stdin $REGISTRY
sudo docker compose up -d
```

7. **The certificate: pick one.**
  **A. Caddy and Let's Encrypt.** Nothing more to do, provided `apm.example.com` resolves publicly to the instance and ports 80 and 443 are open to the internet.
  **B. A load balancer with an ACM certificate.** Use this for a private instance, or when the certificate should not depend on the internet. Request a certificate for `apm.example.com` in ACM, validated by DNS. Put an Application Load Balancer, internal or internet-facing, in front of the instance with two target groups:
  - Ingest on port 8091, health check `/readyz`
  - The server on port 8090, health check `/`

The rules for listener 443:

  - Path `/api/agent/*` goes to ingest.
  - Path `/api/profile` **and** method `POST` goes to ingest.
  - Everything else goes to the server.

Drop Caddy from Compose, publish 8091 and 8090 instead, and allow only the load balancer's security group to reach them.

8. **Keep it running.**
  - An AWS Backup plan that backs up the instance's volume daily.
  - A CloudWatch alarm on the instance's status check, with the *recover* action.
  - The CloudWatch agent, to watch disk space.

Steps 6 to 9 of [one machine](https://studio-dispatch.com/docs/apm-server.html#one-machine) then work the same against `https://apm.example.com`.

## Multi-AZ on AWS

For more JVMs than one machine should carry, or when an instance failing must not stop reports being taken.

| Piece | Setup |
| --- | --- |
| Amazon MSK | Provisioned, with 3 brokers across 3 AZs (`kafka.m7g.large` to start). Turn on encryption in transit and SASL/SCRAM authentication, with the credentials in Secrets Manager. |
| Topics | Create them from any machine that can reach MSK, with the MSK settings below in its environment: `java -jar dispatch-apm-ingest.jar create-topics --partitions 24 --replication 3` |
| Ingest | An Auto Scaling group of at least two instances across the AZs, each running the ingest image. The target group's health check is `/readyz` on 8091. Ingest is stateless, so scale it on CPU. |
| Server | One instance. Give it the recover alarm and AWS Backup plan from [one EC2 instance](https://studio-dispatch.com/docs/apm-server.html#ec2). While it is down, agents are still answered and their reports wait in MSK for up to 24 hours; it catches up when it is back. |
| Load balancer | As in option B of [one EC2 instance](https://studio-dispatch.com/docs/apm-server.html#ec2), with the ingest group and the server as the two targets |
| Security groups | The load balancer accepts 443 from your JVMs and Studio users. Ingest accepts 8091, and the server 8090, from the load balancer only. MSK accepts 9096 from ingest and the server only. |

For ingest to reach MSK with SASL/SCRAM:

```
DISPATCH_APM_INGEST_BOOTSTRAP_SERVERS=b-1.….kafka.ap-south-1.amazonaws.com:9096,b-2.…:9096,b-3.…:9096
DISPATCH_APM_INGEST_KAFKA_SECURITY_PROTOCOL=SASL_SSL
DISPATCH_APM_INGEST_KAFKA_SASL_MECHANISM=SCRAM-SHA-512
DISPATCH_APM_INGEST_KAFKA_SASL_JAAS_CONFIG=org.apache.kafka.common.security.scram.ScramLoginModule required username="dispatch" password="…";
```

The server takes the same four settings, with `DISPATCH_APM_SERVER_KAFKA_` in place of `DISPATCH_APM_INGEST_`.

## Settings

Each setting is an environment variable, or the same name lower-cased with dots as a system property.

### Ingest

| Variable | Default | What it does |
| --- | --- | --- |
| `DISPATCH_APM_INGEST_BOOTSTRAP_SERVERS` | `localhost:9092` | Where Kafka is |
| `DISPATCH_APM_INGEST_PORT` | `8091` |  |
| `DISPATCH_APM_INGEST_HOST` | `0.0.0.0` | `127.0.0.1` when a proxy on the same machine is its only caller |
| `DISPATCH_APM_INGEST_MAX_BODY_BYTES` | `8388608` | Largest report accepted, as sent; bigger is answered 413 |
| `DISPATCH_APM_INGEST_AGENT_TOKEN` | none | Also accept agents configured with a token instead of a key. They report without an application. |
| `DISPATCH_APM_INGEST_TLS_CERTIFICATE`, `…_TLS_KEY` | none | PEM files, with the key in PKCS#8, to terminate TLS in ingest itself |
| `DISPATCH_APM_INGEST_TOPIC_PREFIX` | `dispatch.apm.` |  |
| `DISPATCH_APM_INGEST_THREADS` | CPU count | Netty worker threads |
| `DISPATCH_APM_INGEST_KAFKA_*` |  | Any Kafka client setting |

### Server

| Variable | Default | What it does |
| --- | --- | --- |
| `DISPATCH_APM_SERVER_TOKEN` | required | What Studio presents; at least 16 characters |
| `DISPATCH_APM_SERVER_PORT` | `8090` |  |
| `DISPATCH_APM_SERVER_HOST` | `0.0.0.0` |  |
| `DISPATCH_APM_SERVER_KAFKA_BOOTSTRAP_SERVERS` | none | Read reports from Kafka. Without it, the server takes reports over HTTP only, as on a developer's machine. |
| `DISPATCH_APM_SERVER_KAFKA_*` |  | Any Kafka client setting |
| `DISPATCH_APM_SERVER_GROUP` | `dispatch-apm-core` | Kafka consumer group |

### Kafka topics

`create-topics` makes these, and leaves any that already exist alone.

| Topic | Kept for | Partitions |
| --- | --- | --- |
| `dispatch.apm.metrics` · `.jvm` · `.logs` · `.profile` | 24 hours | as given: 6 on one machine, 24 on MSK |
| `dispatch.apm.liveset` · `.capture` | 7 days | as given |
| `dispatch.apm.keys` · `.directives` | always (compacted) | 1 |

## When something is wrong

| What you see | Where to look |
| --- | --- |
| The agent logs `receiver returned HTTP 401` | The key is wrong, or ingest has not seen it yet. In ingest's `/metrics`, `dispatch_ingest_keys` should count every registered application's key. |
| The agent logs `receiver returned HTTP 503` | Ingest is starting or cannot reach Kafka; its `/readyz` says which. The topics may not exist yet: run `create-topics`. |
| `HTTP 413` | A report was larger than `DISPATCH_APM_INGEST_MAX_BODY_BYTES` |
| A certificate error, or no connection at all | The APM host's certificate is not one the monitored JVM trusts |
| Studio connects, but the tabs stay empty | The server is not reading Kafka. Check `DISPATCH_APM_SERVER_KAFKA_BOOTSTRAP_SERVERS`, then the lag: `kafka-consumer-groups.sh --bootstrap-server … --describe --group dispatch-apm-core` |
| Profiling switched on in Studio never starts | In ingest's `/metrics`, `dispatch_ingest_directives` should be at least 1 |

### If a piece stops

| If this stops | Agents | Studio | Reports |
| --- | --- | --- | --- |
| One ingest instance | The load balancer routes around it | — | — |
| Kafka | Answered 503; each agent logs one line and carries on | Works | Those sent meanwhile are lost |
| The server | Answered as usual | Cannot connect | Kept in Kafka for up to 24 hours, then caught up |