Dispatch APM Server

What you are installing

Agents report to ingest; ingest puts each report on Kafka; the server reads Kafka and answers Dispatch Studio. One HTTPS name serves both kinds of caller.

The server runs as one instance: size it for the whole installation.

What it needs

One monitored JVM at the default agent settings sends about 0.2 MB a minute, in about one request every two seconds. That was measured on the sample service; a service that logs heavily sends more.

Machines

On the 16 GB machine, the memory goes to Kafka (a 1 GB heap, plus the operating system's page cache, which is where Kafka gets its speed), ingest (512 MB) and the server (4–6 GB). Kafka's disk is roughly inbound MB/s × 86,400 × days kept × copies, plus 30%. For 100 JVMs kept 24 hours, that is about 30 GB.

Software

Network

A certificate your JVMs already trust

The agent reports over the JVM's own HTTPS, with its default trust store. The APM host therefore needs a certificate that store already trusts. That can be Let's Encrypt (Caddy obtains one by itself), AWS Certificate Manager behind a load balancer, or your company's CA if your JVMs already trust it. Anything else means changing every monitored application's trust store.

On one machine of your own

Docker Compose runs Kafka, ingest, the server and Caddy. The files are in deploy/docker/ in the repository.

1. The machine. Linux, sized from the table above, with a DNS name (here apm.example.com) pointing at it. Ports 80 and 443 need to be reachable from your JVMs and your Studio users, and from the internet for the certificate.

2. Docker, on Ubuntu from Docker's own repository:

   sudo apt-get update && sudo apt-get install -y ca-certificates curl
   sudo install -m 0755 -d /etc/apt/keyrings
   sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
   echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
     https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
     | sudo tee /etc/apt/sources.list.d/docker.list
   sudo apt-get update
   sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

3. The two images. On a build machine with JDK 21, Maven and Docker, from the repository root:

   mvn -pl dispatch-apm-ingest,dispatch-apm-server -am package -DskipTests
   docker build -f deploy/docker/ingest.Dockerfile -t dispatch-apm-ingest:0.1.0 .
   docker build -f deploy/docker/server.Dockerfile -t dispatch-apm-server:0.1.0 .

   Then carry them to the APM host, through your own registry or directly:

   docker save dispatch-apm-ingest:0.1.0 dispatch-apm-server:0.1.0 | gzip | ssh apm-host 'gunzip | docker load'

4. The configuration, on the APM host:

   sudo mkdir -p /opt/dispatch-apm && cd /opt/dispatch-apm
   # copy compose.yaml, Caddyfile and env.example here from deploy/docker/
   sudo cp env.example .env && sudo chmod 600 .env
   openssl rand -hex 32        # the operator token
   sudo nano .env              # DISPATCH_APM_HOST and DISPATCH_APM_SERVER_TOKEN

5. Start it.

   sudo docker compose up -d
   sudo docker compose ps

   The topics service should show exited (0): it creates the eight Kafka topics and stops. The rest should be running, with ingest healthy. In sudo docker compose logs ingest, look for report topics ready and tables loaded.

6. Check it from outside.

   TOKEN=…   # the operator token
   curl -s -H "X-Dispatch-Token: $TOKEN" https://apm.example.com/api/apps

   [] means the server is up, the certificate works and the token is right.

7. Register an application. Each application gets a key, and its agents present it.

   curl -s -X POST https://apm.example.com/api/apps \
     -H "X-Dispatch-Token: $TOKEN" -H 'Content-Type: application/json' \
     -d '{"name": "Orders", "baseUrl": ""}'

   The answer carries "key": "dsk_…". Ingest learns the new key within a second. Put the service's own address in baseUrl if the server can reach its Actuator endpoint; otherwise leave it empty.

8. Attach the agent to each JVM of that application:

   java -javaagent:/opt/dispatch-agent.jar=\
   dispatch.apm.agent.report.url=https://apm.example.com,\
   dispatch.apm.agent.report.key=dsk_… \
     -jar orders-service.jar

   The agent's other settings, and attaching to a JVM that is already running, are in docs/SETUP.md §6.

9. Point Studio at it with the address https://apm.example.com and the operator token, as in docs/SETUP.md §7.

Day to day

Without containers

The same services as systemd units, from deploy/systemd/:

# Java 21, and users for the services
sudo apt-get install -y openjdk-21-jre-headless
sudo useradd --system --home-dir /var/lib/dispatch-apm --create-home --shell /usr/sbin/nologin dispatch
sudo useradd --system --no-create-home --shell /usr/sbin/nologin kafka

# Kafka, one node
curl -fLO https://downloads.apache.org/kafka/4.3.1/kafka_2.13-4.3.1.tgz
curl -fLO https://downloads.apache.org/kafka/4.3.1/kafka_2.13-4.3.1.tgz.sha512
sha512sum kafka_2.13-4.3.1.tgz && cat kafka_2.13-4.3.1.tgz.sha512    # the two must match
sudo tar -xzf kafka_2.13-4.3.1.tgz -C /opt && sudo ln -s /opt/kafka_2.13-4.3.1 /opt/kafka
sudo mkdir -p /etc/kafka /var/lib/kafka /var/log/kafka
sudo cp /opt/kafka/config/server.properties /etc/kafka/server.properties
#   then, in /etc/kafka/server.properties:
#     log.dirs=/var/lib/kafka
#     auto.create.topics.enable=false
#     message.max.bytes=8454144
#     listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
#     advertised.listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
sudo /opt/kafka/bin/kafka-storage.sh format --standalone \
     -t "$(/opt/kafka/bin/kafka-storage.sh random-uuid)" -c /etc/kafka/server.properties
sudo chown -R kafka:kafka /var/lib/kafka /var/log/kafka
sudo cp deploy/systemd/kafka.service /etc/systemd/system/

# Ingest and the server
sudo mkdir -p /opt/dispatch-apm /etc/dispatch-apm
sudo cp dispatch-apm-ingest/target/dispatch-apm-ingest.jar /opt/dispatch-apm/
sudo cp dispatch-apm-server/target/dispatch-apm-server-all.jar /opt/dispatch-apm/dispatch-apm-server.jar
printf 'DISPATCH_APM_INGEST_BOOTSTRAP_SERVERS=localhost:9092\nDISPATCH_APM_INGEST_HOST=127.0.0.1\n' \
  | sudo tee /etc/dispatch-apm/ingest.env
printf 'DISPATCH_APM_SERVER_TOKEN=%s\nDISPATCH_APM_SERVER_HOST=127.0.0.1\nDISPATCH_APM_SERVER_KAFKA_BOOTSTRAP_SERVERS=localhost:9092\n' \
  "$(openssl rand -hex 32)" | sudo tee /etc/dispatch-apm/server.env
sudo chmod 600 /etc/dispatch-apm/*.env
sudo cp deploy/systemd/dispatch-apm-*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now kafka
DISPATCH_APM_INGEST_BOOTSTRAP_SERVERS=localhost:9092 \
  java -jar /opt/dispatch-apm/dispatch-apm-ingest.jar create-topics --partitions 6 --replication 1
sudo systemctl enable --now dispatch-apm-ingest dispatch-apm-server

Then install Caddy from its own package repository and use deploy/docker/Caddyfile, with ingest:8091 and server:8090 changed to localhost:8091 and localhost:8090. Steps 6 to 9 above are the same.

On one EC2 instance

The same Compose setup, in your own AWS account.

1. Launch the instance.

   AMI	Ubuntu Server 24.04 LTS, or Amazon Linux 2023
   Type	m7i.xlarge (4 vCPU, 16 GiB) for up to 100 JVMs; Graviton's m7g.xlarge works too
   Storage	250 GiB gp3, encrypted
   Subnet	Private, when your JVMs and Studio users reach the VPC over a VPN or Direct Connect. Public with an Elastic IP, when they come over the internet.
   IAM role	AmazonSSMManagedInstanceCore for Session Manager, AmazonEC2ContainerRegistryReadOnly to pull the images, CloudWatchAgentServerPolicy for metrics and logs

2. The security group. Allow inbound 443 from your JVMs' subnets or addresses and from your Studio users. Allow 80 from anywhere only if you use Let's Encrypt (option A in step 7). Leave SSH closed and connect with aws ssm start-session --target i-…. Kafka, ingest and the server listen only on the instance's internal Docker network.

3. A name. A Route 53 record for apm.example.com: the Elastic IP for a public instance, or the private IP, in a private hosted zone, for a private one.

4. The images, in ECR. From the build machine, after building them as in step 3 of one machine:

   REGISTRY=123456789012.dkr.ecr.ap-south-1.amazonaws.com
   aws ecr create-repository --repository-name dispatch-apm-ingest
   aws ecr create-repository --repository-name dispatch-apm-server
   aws ecr get-login-password | docker login --username AWS --password-stdin $REGISTRY
   for image in dispatch-apm-ingest dispatch-apm-server; do
     docker tag $image:0.1.0 $REGISTRY/$image:0.1.0 && docker push $REGISTRY/$image:0.1.0
   done

5. The token, in Secrets Manager, so it is in neither a shell history nor a disk image:

   aws secretsmanager create-secret --name dispatch-apm/server-token --secret-string "$(openssl rand -hex 32)"

   Give the instance role secretsmanager:GetSecretValue on that one secret.

6. On the instance, in a Session Manager shell: install Docker as in step 2 of one machine, and the AWS CLI (sudo snap install aws-cli --classic on Ubuntu; Amazon Linux already has it). Then:

   sudo mkdir -p /opt/dispatch-apm && cd /opt/dispatch-apm
   # compose.yaml, Caddyfile and env.example from deploy/docker/, e.g. by way of an S3 bucket
   sudo cp env.example .env && sudo chmod 600 .env
   REGISTRY=123456789012.dkr.ecr.ap-south-1.amazonaws.com
   TOKEN=$(aws secretsmanager get-secret-value --secret-id dispatch-apm/server-token --query SecretString --output text)
   sudo sed -i "s|^DISPATCH_APM_HOST=.*|DISPATCH_APM_HOST=apm.example.com|; \
                s|^DISPATCH_APM_SERVER_TOKEN=.*|DISPATCH_APM_SERVER_TOKEN=$TOKEN|; \
                s|^DISPATCH_APM_REGISTRY=.*|DISPATCH_APM_REGISTRY=$REGISTRY/|" .env
   aws ecr get-login-password | sudo docker login --username AWS --password-stdin $REGISTRY
   sudo docker compose up -d

7. The certificate: pick one.

   A. Caddy and Let's Encrypt. Nothing more to do, provided apm.example.com resolves publicly to the instance and ports 80 and 443 are open to the internet.

   B. A load balancer with an ACM certificate. Use this for a private instance, or when the certificate should not depend on the internet. Request a certificate for apm.example.com in ACM, validated by DNS. Put an Application Load Balancer, internal or internet-facing, in front of the instance with two target groups:

   • Ingest on port 8091, health check /readyz
   • The server on port 8090, health check /

   The rules for listener 443:

   • Path /api/agent/* goes to ingest.
   • Path /api/profile and method POST goes to ingest.
   • Everything else goes to the server.

   Drop Caddy from Compose, publish 8091 and 8090 instead, and allow only the load balancer's security group to reach them.

8. Keep it running.

   • An AWS Backup plan that backs up the instance's volume daily.
   • A CloudWatch alarm on the instance's status check, with the recover action.
   • The CloudWatch agent, to watch disk space.

   Steps 6 to 9 of one machine then work the same against https://apm.example.com.

Multi-AZ on AWS

For more JVMs than one machine should carry, or when an instance failing must not stop reports being taken.

For ingest to reach MSK with SASL/SCRAM:

DISPATCH_APM_INGEST_BOOTSTRAP_SERVERS=b-1.….kafka.ap-south-1.amazonaws.com:9096,b-2.…:9096,b-3.…:9096
DISPATCH_APM_INGEST_KAFKA_SECURITY_PROTOCOL=SASL_SSL
DISPATCH_APM_INGEST_KAFKA_SASL_MECHANISM=SCRAM-SHA-512
DISPATCH_APM_INGEST_KAFKA_SASL_JAAS_CONFIG=org.apache.kafka.common.security.scram.ScramLoginModule required username="dispatch" password="…";

The server takes the same four settings, with DISPATCH_APM_SERVER_KAFKA_ in place of DISPATCH_APM_INGEST_.

Settings

Each setting is an environment variable, or the same name lower-cased with dots as a system property.

Ingest

Server

Kafka topics

create-topics makes these, and leaves any that already exist alone.

When something is wrong

If a piece stops