Skip to main content

Instrument a Go service for traces

The AtherOps agent receives traces over OTLP -- it cannot inject tracing into a compiled Go binary. Go services require manual SDK integration: you import the OpenTelemetry Go SDK, initialize a tracer provider, and instrument the code you want to trace.

If you haven't read Metrics/logs vs APM traces yet, that page explains why detecting a Go process does not automatically produce traces. It also covers an important limitation: the agent detects the Go toolchain (the go process from go run or go build), not compiled Go binaries, which use application-specific names.

Prerequisites​

  • The AtherOps agent is installed and running on the host.
  • collection.traces.enabled: true is set in /etc/atherops/config.yaml (this is the default).
  • Go 1.21 or later (the OTel Go SDK supports 1.21+).

Step 1: Add the SDK dependencies​

go get go.opentelemetry.io/otel \
       go.opentelemetry.io/otel/sdk/trace \
       go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp

Use otlptracehttp for HTTP/protobuf (port 4318) or otlptracegrpc for gRPC (port 4317). HTTP requires no additional system libraries and is usually simpler.

Step 2: Initialize the tracer provider​

Add a setup function to your application -- typically called from main before serving traffic:

package main

import (
    "context"
    "log"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func initTracer(ctx context.Context) (func(), error) {
    exporter, err := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpoint("localhost:4318"),
        otlptracehttp.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("my-go-service"),
        )),
    )
    otel.SetTracerProvider(tp)

    return func() {
        if err := tp.Shutdown(ctx); err != nil {
            log.Printf("tracer shutdown: %v", err)
        }
    }, nil
}

func main() {
    ctx := context.Background()
    shutdown, err := initTracer(ctx)
    if err != nil {
        log.Fatalf("init tracer: %v", err)
    }
    defer shutdown()

    // your application code here
}

Step 3: Instrument your handlers​

With the tracer provider set, create spans around the work you care about:

import "go.opentelemetry.io/otel"

tracer := otel.Tracer("my-go-service")

func handleRequest(ctx context.Context) {
    ctx, span := tracer.Start(ctx, "handleRequest")
    defer span.End()

    // pass ctx through to downstream calls so child spans attach correctly
    callDatabase(ctx)
}

For HTTP servers, use the OTel HTTP middleware instead of manual spans:

go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

mux := http.NewServeMux()
mux.HandleFunc("/", yourHandler)
http.ListenAndServe(":8080", otelhttp.NewHandler(mux, "http.server"))

Step 4: Verify spans are reaching AtherOps​

Build and run your service, generate some traffic, then query:

curl -s \
  -H "Authorization: Bearer $ATHEROPS_JWT" \
  "https://api.atherops.com/query/traces?service=my-go-service&limit=10"

A non-empty JSON array means spans are flowing. See Enable traces: verify spans reached AtherOps for details on the query parameters and how to obtain the bearer token.

Choosing the OTLP endpoint​

To use gRPC (port 4317) instead of HTTP, replace the exporter:

go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

exporter, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithEndpoint("localhost:4317"),
    otlptracegrpc.WithInsecure(),
)

Troubleshooting​

No spans appear after startup. Check the agent log:

journalctl -u atherops-otel-agent --since "5 min ago" | grep -i error

A forward traces failed line means the agent receives spans but cannot reach AtherOps. No error lines combined with missing spans means the SDK is not initializing or the exporter endpoint is wrong -- add a log.Printf after initTracer to confirm it returns without error, and verify the host and port match what the agent is listening on.

connection refused when the exporter starts. The OTLP receiver in the agent only starts when collection.traces.enabled: true in /etc/atherops/config.yaml and the collector is running. Verify with:

systemctl status atherops-otel-agent
ss -tlnp | grep 4318

Spans from go test or build tools appear unexpectedly. The agent detects the go process name, which matches the Go toolchain. Spans here come from your service, not the toolchain -- the toolchain itself is not instrumented. If you see unexpected service names, check that semconv.ServiceName is set correctly in your resource.NewWithAttributes call.

Further reading​