Skip to main content

Troubleshooting

Each entry below is symptom → cause → action. Start by tailing the agent logs:

journalctl -u atherops-otel-agent -f

The service starts but immediately dies (no api_key available)​

Symptom. The service flaps; logs show:

agent fatal error err="no api_key available — ensure INSTALL_TOKEN is set for first run or credentials.json exists"

Cause. The agent has no credentials. The most common reason is running the one-liner without the install token, or with the token in the wrong place so sudo stripped it.

Action.

  1. Make sure you wrote the token after sudo:

    curl -sSL https://releases.atherops.com/otel-agent/install.sh | sudo INSTALL_TOKEN=<token> bash

    Writing INSTALL_TOKEN=<token> sudo … bash does not work. sudo drops the variable before the script reads it.

  2. Generate a fresh token (Settings → Hosts → Install Token) and re-run.

On a host with no existing credentials, the installer detects a missing INSTALL_TOKEN and exits 1 with a clear message before installing anything, so you should normally see the guidance up front rather than a dead service. On a host that already has valid credentials.json, the installer treats a token-less run as a re-install and proceeds.

"Required command not found" during install​

Symptom. The installer stops early:

[atherops] ERROR  Required command not found: jq

Cause. The installer needs curl, sha256sum, tar, and jq. Minimal images (Alpine, Amazon Linux minimal) often ship without jq.

Action. Install the missing command, then re-run the one-liner. For example:

# Debian/Ubuntu
sudo apt-get update && sudo apt-get install -y jq
# Alpine
sudo apk add jq curl tar coreutils

sha256 mismatch during install​

Symptom. The installer prints a red block and exits with code 2:

[atherops] ERROR  sha256 MISMATCH — otelcol-contrib-linux_amd64
          expected : e3b0c44298fc1c149afb...
          got      : 9f86d081884c7d659a2f...

  INSTALL ABORTED. The downloaded file does not match the manifest checksum.
  The file has been removed. No binary was installed and no service was started.

Cause. The downloaded artifact does not match the manifest checksum: a corrupt/partial download, or tampering in transit (MITM / CDN compromise).

Action.

  1. Retry the install command; a transient/partial download usually clears.
  2. If it persists, treat it as a possible security incident: do not run the binary manually, and contact security@atherops.com.

Nothing was installed and no service was started, so the host is in a safe state. Background: Supply-chain trust.

Collector crash-loop​

Symptom. Repeated errors, then a pause:

level=error msg="collector exited unexpectedly" exit_code=1 restarts=3
level=info  msg="restarting collector" delay=2s restarts=3
level=error msg="collector crash-looping (5 restarts in 60s) — pausing restart for 5m"

Cause. The collector keeps exiting, usually due to a config it can't accept. A common case is journald: true on a host without systemd.

Action.

  1. Look at the collector's own output and the agent logs:

    journalctl -u atherops-otel-agent -f
    cat /var/run/atherops/otelcol.log    # if present

  2. Check your config.yaml. On a non-systemd host set collection.logs.journald: false (see Configure log sources).

  3. The agent auto-retries after the 5-minute pause. To reset the counter immediately after a fix:

    sudo systemctl restart atherops-otel-agent

Backend unreachable / host shows offline​

Symptom.

level=warn msg="backend unreachable" url=https://api.atherops.com err="connection refused" retry_in=30s

The host eventually shows offline on the Hosts page (after missing heartbeats).

Cause. The host can't reach AtherOps. The agent does not crash; collection continues locally, but heartbeats and remote-config polls fail.

Action.

  1. Confirm outbound HTTPS to AtherOps on port 443:

    curl -sS -o /dev/null -w '%{http_code}\n' https://api.atherops.com

  2. Check firewalls/proxies. If the host is air-gapped, ensure api.atherops.com:443 is reachable.

  3. Recovery is automatic. The agent retries with backoff (up to ~5 minutes) and the host returns to online once connectivity is restored.

401 / key revoked​

Symptom.

level=error msg="authentication failed (401) — key may be revoked" action="halting; delete /etc/atherops/credentials.json to re-register"

The agent stops forwarding data and the host goes offline.

Cause. The backend rejected the host's API key (revoked, rotated, or otherwise invalid). The agent halts deliberately and does not auto-delete its credentials.

Action. Re-register the host:

sudo rm /etc/atherops/credentials.json
# generate a new token: Settings → Hosts → Install Token
curl -sSL https://releases.atherops.com/otel-agent/install.sh | sudo INSTALL_TOKEN=<token> bash

Full steps: Re-register after a 401.

--version shows unknown for the collector​

Symptom.

atherops-otel-agent 1.2.0
otelcol-contrib     unknown

Cause. The collector hasn't been resolved yet. The current symlink under /var/lib/atherops/otelcol/ doesn't exist. This is normal on a brand-new host before the agent has run (the installer downloads the collector, but the current symlink is established by the install flow / first run).

Action.

  1. Start the agent and let it run once:

    sudo systemctl restart atherops-otel-agent
    journalctl -u atherops-otel-agent -f   # wait for "collector started"

  2. Re-check:

    atherops-otel-agent --version
    ls -l /var/lib/atherops/otelcol/current

    If current still doesn't resolve to a version directory after a successful start, the collector install failed. Check the agent logs for a collector install failed warning and check connectivity to the manifest URL.

Traces: how do I confirm spans are being received?​

Symptom. Apps send spans and there's no obvious place to see them.

Cause / expectation. This release collects and stores traces but does not include an in-product trace explorer UI yet (see scope). On day 1 the absence of a UI is normal, not a failure. You confirm receipt via the query API instead.

Action.

  1. Confirm collection.traces.enabled: true in config.yaml.

  2. Confirm your app sends OTLP to localhost:4317 (gRPC) or localhost:4318 (HTTP). See Enable traces.

  3. Confirm the collector is healthy (no crash-loop lines in the agent log).

  4. Positively confirm receipt by querying your stored spans with your session JWT. A non-empty result proves spans reached AtherOps:

    curl -s -H "Authorization: Bearer $ATHEROPS_JWT" \
      "https://api.atherops.com/query/traces?limit=20"

    $ATHEROPS_JWT is your session JWT. To get it from the command line, see Get a query token. Full query reference: Query your traces back.

See also​