Deploy AtherOps on Kubernetes

This guide covers deploying the full AtherOps platform on a Kubernetes cluster using central-helm, the GitOps source of truth for the prod cluster. ArgoCD watches the repo and reconciles the cluster to match it.

The stack includes:

Longhorn (v1.11.2) -- replicated block storage for the three datastore volumes
api (Go, :8081) -- identity, auth, fleet control
backend (Go, :8080) -- ingest and query
alerting (Go, :8082) -- alert rule management
ui (nginx SPA, :80) -- the web interface
Postgres -- identity and fleet state (20 GiB Longhorn PVC)
VictoriaMetrics -- metrics time-series (20 GiB Longhorn PVC)
ClickHouse -- logs and traces (50 GiB Longhorn PVC)

All traffic enters through Gateway eg (Envoy Gateway, MetalLB LB IP 192.168.1.245). There is no Ingress controller; routing uses the Gateway API with HTTPRoute.

How it works

ArgoCD uses an app-of-apps pattern: a root Application watches apps/*/application.yaml in the repo and creates a child Application per directory. Each Application is an umbrella Helm chart.

Sync-wave annotations control order. Longhorn (wave -5) must be healthy before atherops (wave 0) can bind its PVCs.

Secrets are encrypted in-place with SOPS + age and decrypted inside the ArgoCD repo-server at render time. No plaintext secret is ever committed to git.

Runtime configuration

All images are environment-portable: service URLs are injected by env at container start. No addresses are baked into the images, so the same image works in compose, minikube, and prod K8s.

UI runtime config (`window.ENV`)

The UI SPA reads service URLs from window.__ENV__ before making any API call. These values are written to /usr/share/nginx/html/config.js at container startup by the nginx entrypoint -- not compiled into the JavaScript bundle.

`overrides.yaml` key	Default	What it controls
`ui.apiURL`	`/api`	Base prefix for all `/api/*` calls
`ui.backendURL`	`` (empty)	Prefix for `/query/` and `/ingest/` calls; empty means same-origin
`ui.alertingURL`	`/alerting`	Base prefix for `/alerting/*` calls

With same-origin routing (the standard setup described in this guide), leave all three keys empty or omit them. The relative-path defaults apply and routing works via the Gateway HTTPRoute.

API inter-service URLs

When operators install the k8s otel agent via the UI, the api service returns a helm install command that includes the platform's public URLs. Set these to the external address of your Gateway:

# apps/atherops/overrides.yaml
api:
  apiBaseURL: "https://atherops.your-domain.com"
  ingestBaseURL: "https://atherops.your-domain.com"

Request routing

All requests arrive at a single hostname. The Gateway routes them to the right service by path prefix. The routing contract is identical in the K8s HTTPRoute and the compose nginx configuration.

Path prefix	Service	Port	Prefix stripped?	Note
`/api/*`	api	8081	No	HMAC agent auth requires the full path
`/auth/*`	api	8081	No	JWT login
`/query/*`	backend	8080	No	Metrics, logs, traces query
`/ingest/*`	backend	8080	No	Metrics, logs, traces ingest; agent heartbeat
`/alerting/*`	alerting	8082	Yes -- `/alerting/rules` becomes `/rules`	K8s: `URLRewrite ReplacePrefixMatch: /`
`/` (catch-all)	ui	80	No	nginx SPA; client-side routing via `try_files`

Prerequisites

Cluster

Kubernetes 1.27 or newer
ArgoCD already bootstrapped via argocd/install.sh (see the central-helm README)
MetalLB providing LoadBalancer IPs (pool 192.168.1.245-192.168.1.250)
Envoy Gateway installed (apps/envoy-gateway + apps/gateway from the app-of-apps)

Every K8s node: open-iscsi

Longhorn attaches volumes over iSCSI. Every node in the cluster must have the open-iscsi package installed and the iscsid daemon running before the first Longhorn sync.

# Run on each node (control plane and workers)
sudo apt-get install -y open-iscsi nfs-common
sudo systemctl enable --now iscsid

# Verify
systemctl is-active iscsid
# Expected output: active

If iscsid is not running when Longhorn tries to attach a volume, the volume fails to mount and the StatefulSet pod stays in ContainerCreating indefinitely.

CLI tools

# Install on your workstation
# helm >= 3.14, kubectl, sops, age, argocd CLI (optional)
bash scripts/prereqs.sh   # installs/updates all of them

One-time setup

Work through these steps in order. Each must complete before the next.

1. Confirm Longhorn node prereqs

SSH to each node and verify iscsid is active (see Prerequisites above). Missing iscsid is the most common cause of Longhorn volume attachment failures.

2. Re-encrypt SOPS secrets to your cluster's age key

The shipped apps/atherops/secrets.sops.yaml is encrypted to the developer's age key. ArgoCD on your cluster cannot decrypt it. You must replace the ciphertexts before merging.

# From the central-helm repo root
bash scripts/sops-encrypt.sh edit apps/atherops/secrets.sops.yaml

Generate each field with the commands below and paste the values into the editor:

Field	How to generate
`postgresPassword`	`openssl rand -hex 32`
`postgresConnectionKey`	`postgresql://atherops:<password>@atherops-postgres:5432/atherops?sslmode=disable`
`jwtSecret`	`openssl rand -hex 32`
`apikeyEncryptionKey`	`openssl rand -hex 32` (same value in api and backend)
`clickhousePassword`	`openssl rand -hex 32`
`harborDockerKey`	base64-encoded docker config JSON (see step 3)

3. Create a Harbor pull robot and build the pull credential

The chart pulls images from harbor.atherops.com/observability/. Create a pull-only robot account in Harbor, then build the credential:

# Replace <robot-name> and <robot-token> with your Harbor robot credentials
ROBOT_AUTH=$(printf '%s' "robot\$<robot-name>:<robot-token>" | base64 -w0)
printf '{"auths":{"harbor.atherops.com":{"auth":"%s"}}}\n' "$ROBOT_AUTH" | base64 -w0

Paste the output as harborDockerKey when editing secrets.sops.yaml.

4. Push images to Harbor

The chart pins image tags in apps/atherops/overrides.yaml. Push them before merging:

# From the platform/ repo root
make docker/push/api TAG=0.1.0
make docker/push/backend TAG=0.1.0
make docker/push/alerting TAG=0.1.0
make docker/push/ui TAG=0.1.1

5. Set the HTTPRoute hostname and public-facing URLs

Edit apps/atherops/overrides.yaml and replace the placeholders with your actual DNS name and the public-facing URLs the agents will use:

httproute:
  hostname: "atherops.your-domain.com"

# Public URLs returned to k8s agents in the helm install command.
# Both point at the Gateway hostname so agents reach the right service via path routing.
api:
  apiBaseURL: "https://atherops.your-domain.com"
  ingestBaseURL: "https://atherops.your-domain.com"

Point DNS for this name at the Gateway LB IP (192.168.1.245 by default).

6. Merge to main

ArgoCD watches targetRevision: main. Only content on main is synced. Merge the current branch (feat/sprint14-routing, PR #3) to main to trigger the sync.

Verify the deployment

After ArgoCD syncs, confirm the full stack is healthy:

# Both apps Synced and Healthy
kubectl get application longhorn atherops -n argocd
# NAME      SYNC STATUS   HEALTH STATUS
# longhorn  Synced        Healthy
# atherops  Synced        Healthy

# Longhorn StorageClass available
kubectl get storageclass longhorn
# NAME       PROVISIONER          RECLAIMPOLICY
# longhorn   driver.longhorn.io   Retain

# Three PVCs bound
kubectl get pvc -n atherops
# NAME                              STATUS   STORAGECLASS   CAPACITY
# data-atherops-postgres-0          Bound    longhorn       20Gi
# data-atherops-victoria-metrics-0  Bound    longhorn       20Gi
# data-atherops-clickhouse-0        Bound    longhorn       50Gi

# Nine pods running (eight services + alerting)
kubectl get pods -n atherops
# NAME                               READY   STATUS
# atherops-alerting-<hash>           1/1     Running
# atherops-api-<hash>                2/2     Running  (x2)
# atherops-backend-<hash>            2/2     Running  (x2)
# atherops-ui-<hash>                 1/1     Running
# atherops-postgres-0                1/1     Running
# atherops-victoria-metrics-0        1/1     Running
# atherops-clickhouse-0              1/1     Running

# Health endpoints
kubectl port-forward -n atherops svc/atherops-api 8081:8081 &
sleep 2
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8081/healthz   # 200
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8081/readyz    # 200
kill %1

# Check the UI runtime config was written correctly
kubectl exec -n atherops deploy/atherops-ui -- cat /usr/share/nginx/html/config.js
# window.__ENV__ = { API_URL: "/api", BACKEND_URL: "", ALERTING_URL: "/alerting" };

Releasing a new version

To roll out a new build:

Build and push the new images to Harbor.

Bump the image tags in apps/atherops/overrides.yaml:

api:
  image:
    tag: "0.2.0"
backend:
  image:
    tag: "0.2.0"
alerting:
  image:
    tag: "0.2.0"
ui:
  image:
    tag: "0.2.0"

Open a PR, get it reviewed, merge to main. ArgoCD reconciles within ~3 minutes.

To roll back, revert the tag change, open a PR, and merge.

Longhorn storage

Longhorn is the default StorageClass (longhorn) for all three datastores. It replicates each volume across 2 nodes by default, so a single-node failure does not lose data. When the StatefulSet pod reschedules on a surviving node, Longhorn re-attaches the volume (typically within 30-90 seconds).

The reclaim policy is Retain: deleting a PVC does not delete the underlying volume. This prevents accidental data loss during a chart update.

To access the Longhorn management UI:

kubectl port-forward -n longhorn-system svc/longhorn-frontend 8080:80
# Open: http://localhost:8080

Manual snapshot before a disruptive change

Take a snapshot before any chart update that touches a StatefulSet or PVC:

# Snapshot the Postgres volume
kubectl -n longhorn-system exec -it \
  $(kubectl -n longhorn-system get pods -l app=longhorn-manager -o name | head -1) -- \
  longhorn-manager snapshot create --volume-name data-atherops-postgres-0

Or use the Longhorn UI: Volumes → select the volume → Create Snapshot.

Troubleshooting

PVCs stuck in Pending

Cause: Longhorn is not yet Healthy, or storageClassName: longhorn in overrides.yaml does not match an existing StorageClass.

Check:

kubectl get application longhorn -n argocd      # must be Synced + Healthy
kubectl get storageclass longhorn               # must exist
kubectl describe pvc <name> -n atherops        # look for the exact error event

If Longhorn is still progressing (DaemonSet rolling out), wait 2-3 minutes.

Longhorn volumes fail to attach after pod reschedule

Cause: iscsid is not running on the node where the pod was rescheduled.

Fix:

# SSH to the node and start iscsid
sudo systemctl enable --now iscsid
systemctl is-active iscsid   # must print: active

Then delete the stuck pod to force a reschedule:

kubectl -n atherops delete pod <statefulset-pod-name>

ImagePullBackOff on api, backend, or ui pods

Cause: The Harbor pull credential is wrong, the robot token is expired, or the image tag does not exist in Harbor.

Check:

kubectl describe pod <pod-name> -n atherops
# Look for: Failed to pull image "harbor.atherops.com/observability/api:..."

Fix:

Verify the image exists: docker pull harbor.atherops.com/observability/api:<tag>
If the robot token is expired: regenerate it in Harbor, re-encrypt in secrets.sops.yaml, commit, merge. ArgoCD updates the pull Secret on next sync.

ArgoCD ComparisonError on first sync

Cause: secrets.sops.yaml is still encrypted to the developer's age key (not your cluster's key). ArgoCD cannot decrypt it.

Fix: Re-run step 2 (re-encrypt with sops-encrypt.sh edit) and push the updated file.

How it works​

Runtime configuration​

UI runtime config (window.__ENV__)​

API inter-service URLs​

Request routing​

Prerequisites​

Cluster​

Every K8s node: open-iscsi​

CLI tools​

One-time setup​

1. Confirm Longhorn node prereqs​

2. Re-encrypt SOPS secrets to your cluster's age key​

3. Create a Harbor pull robot and build the pull credential​

4. Push images to Harbor​

5. Set the HTTPRoute hostname and public-facing URLs​

6. Merge to main​

Verify the deployment​

Releasing a new version​

Longhorn storage​

Manual snapshot before a disruptive change​

Troubleshooting​

PVCs stuck in Pending​

Longhorn volumes fail to attach after pod reschedule​

ImagePullBackOff on api, backend, or ui pods​

ArgoCD ComparisonError on first sync​

See also​