OpenTelemetry Connection Pool Tracing

This guide is part of Connection Pool Observability. Pool metrics answer “is the pool saturated?” and traces answer “why was this request slow?”, and for years the two lived in different systems with different names for the same thing: hikaricp_connections_pending in Prometheus, pool.wait in one APM vendor, nothing at all in the trace waterfall. OpenTelemetry closes that gap. Its semantic conventions define a common set of db.client.connection.* instruments that the Java agent, the Python and Node.js instrumentation libraries and several drivers now emit, and the same SDK that records them records the request’s spans. A pool wait can be both a histogram bucket on a dashboard and a visible gap in the trace of the request that suffered it.

This page explains what OpenTelemetry measures about a pool, how those measurements map to pool parameters, how to configure the common runtimes and the Collector, and how to use traces to separate a saturated pool from a slow database.

Key operational takeaways:

• The core pool instruments are db.client.connection.count split by state (idle or used), db.client.connection.max, db.client.connection.pending_requests, db.client.connection.timeouts, and three histograms: create_time, wait_time and use_time. Older instrumentation emits the earlier db.client.connections.* names in milliseconds.
• wait_time rising while use_time stays flat means the pool is too small for the load; use_time rising drags wait_time up behind it and means queries or held transactions are slow. Read them together.
• Traces show pool wait only if acquisition is a span or a span event. Most JDBC instrumentation records query spans but not getConnection; enable data-source instrumentation or add a manual span.
• Every instrument carries db.client.connection.pool.name. Give pools stable names, or dashboards will split one pool into a new series at every restart.
• Exemplars link a wait_time histogram bucket to a trace ID, so a p99 spike on a dashboard opens the trace of a request that actually waited.

Foundational mechanics

Metrics are observed, not pushed by the pool. Most pool instrumentation registers asynchronous instruments: a callback that reads the pool’s current idle, used and pending counts whenever the SDK collects metrics, typically every 60 seconds by default or at the interval configured on the periodic reader. Gauges therefore show the pool state at the moment of collection, not the peak since the last collection. A pool that saturates for eight seconds between two collections may never show pending_requests > 0. The histograms and counters — wait_time, use_time, create_time, timeouts — are synchronous: they are recorded on every acquisition, return and creation, so they capture short bursts that gauges miss.

The three histograms bracket a connection’s life. create_time measures how long opening a new physical connection took: TCP, TLS, authentication. wait_time measures how long a caller waited to obtain a connection from the pool, including any time spent creating one on its behalf. use_time measures how long the caller held the connection before returning it. Together they decompose the database portion of a request into waiting for capacity and consuming it.

Spans need an acquisition boundary. A trace shows query spans — SELECT orders with db.system, db.namespace and server.address attributes — as children of the request span. Time spent waiting for a connection appears only as an unexplained gap before the first query span unless something records it. Three approaches exist: the runtime’s instrumentation creates a span around DataSource.getConnection() (the OpenTelemetry Java agent can, when data-source instrumentation is enabled); the application wraps acquisition in a manual span; or pool wait is recorded as a span event with the wait duration as an attribute, which is cheaper when acquisition is usually immediate.

Context flows with the connection, not the pool. The acquisition span is a child of whatever span is active when the application asks for a connection. Pool-internal work — a housekeeping thread closing idle connections, a background task creating connections to satisfy a minimum — happens outside any request and produces no spans, only create_time observations. When create_time spikes without corresponding request spans, it is pool maintenance, typically connections being recycled at maxLifetime.

Names come from the pool. db.client.connection.pool.name is taken from the pool’s configured name where one exists (HikariCP poolName, SQLAlchemy engine URL, pg.Pool settings) or generated. Generated names that include an instance counter or a hash change on restart and split time series.

Operational Boundary: OpenTelemetry describes the client-side pool. It cannot see a proxy’s internal queue or the database’s own sessions; those need the proxy’s and the database’s own telemetry, ideally exported through the same Collector.

Precision sizing & timeout orchestration

Each instrument corresponds to a pool parameter, and the parameter sets the threshold at which the instrument means trouble.

Histogram buckets decide what the dashboards can show. The default explicit bucket boundaries in many SDKs are tuned for request latencies in milliseconds up to seconds; pool wait is usually zero or a few milliseconds, with a long tail to the acquisition timeout. Configure a view with boundaries that resolve both ends:

0, 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30

The upper bounds should include the acquisition timeout so timed-out waits land in their own bucket rather than in +Inf with everything else.

Collection interval is the other sizing decision. With a 60-second interval, gauges miss short saturation entirely. A 15-second interval costs four times the data points for the gauge series but catches most bursts; the histograms and counters capture bursts regardless of interval, which is why alerts should prefer them.

Cardinality stays small if attributes stay small. Pool metrics carry the pool name and, for count, the state. Do not add request-level attributes such as route or user to pool instruments through views; the pool is shared across routes, and per-route attribution belongs in traces.

Production configuration examples

JVM with the OpenTelemetry Java agent. The agent instruments HikariCP, Apache DBCP, c3p0, Tomcat JDBC and Oracle UCP pools without code changes. Name the pool, enable data-source spans for acquisition, and shorten the metric interval:

JAVA_TOOL_OPTIONS="-javaagent:/otel/opentelemetry-javaagent.jar"
OTEL_SERVICE_NAME=orders-api
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_METRIC_EXPORT_INTERVAL=15000
OTEL_INSTRUMENTATION_JDBC_DATASOURCE_ENABLED=true     # span around DataSource.getConnection()
OTEL_SEMCONV_STABILITY_OPT_IN=database                 # emit current db semantic conventions

spring:
  datasource:
    hikari:
      pool-name: orders-primary          # becomes db.client.connection.pool.name
      maximum-pool-size: 16
      connection-timeout: 3000

Python with SQLAlchemy. The SQLAlchemy instrumentation records query spans and connection usage for the engine’s pool:

from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

trace.set_tracer_provider(TracerProvider())
trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
metrics.set_meter_provider(MeterProvider(
    metric_readers=[PeriodicExportingMetricReader(OTLPMetricExporter(), export_interval_millis=15_000)]))

engine = create_engine(DATABASE_URL, pool_size=8, max_overflow=4, pool_timeout=3)
SQLAlchemyInstrumentor().instrument(engine=engine, enable_commenter=False)

Node.js with pg. The pg instrumentation records query spans and, for pg.Pool, connection count and pending request metrics:

// instrumentation.js — loaded with node --import ./instrumentation.js
import { NodeSDK } from "@opentelemetry/sdk-node";
import { PgInstrumentation } from "@opentelemetry/instrumentation-pg";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-grpc";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-grpc";
import { PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";

new NodeSDK({
  serviceName: "catalog-api",
  traceExporter: new OTLPTraceExporter(),
  metricReader: new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter(), exportIntervalMillis: 15_000 }),
  instrumentations: [new PgInstrumentation({ requireParentSpan: true })],
}).start();

Go with database/sql. The otelsql package wraps the driver for spans and registers sql.DBStats-based metrics:

db, err := otelsql.Open("pgx", dsn,
    otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
    otelsql.WithSpanOptions(otelsql.SpanOptions{OmitConnResetSession: true}))
if err != nil { log.Fatal(err) }
db.SetMaxOpenConns(20)
db.SetMaxIdleConns(10)

if err := otelsql.RegisterDBStatsMetrics(db, otelsql.WithAttributes(semconv.DBSystemPostgreSQL)); err != nil {
    log.Fatal(err)
}

Collector. Receive OTLP, protect the Collector’s memory, keep the histogram boundaries, and export metrics to Prometheus with exemplars enabled:

receivers:
  otlp:
    protocols: { grpc: {}, http: {} }

processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 20 }
  batch: { timeout: 5s }

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
    enable_open_metrics: true      # required for exemplars
    resource_to_telemetry_conversion: { enabled: true }
  otlp/traces:
    endpoint: tempo:4317
    tls: { insecure: true }

service:
  pipelines:
    metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheus] }
    traces:  { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlp/traces] }

The Prometheus exporter converts names: db.client.connection.wait_time in seconds becomes db_client_connection_wait_time_seconds_bucket, _sum and _count. The Collector-side details, including renaming the older instrument names, are in Collecting db.client.connection Metrics with the OpenTelemetry Collector.

Diagnostics & telemetry

Saturation versus slow queries. Plot p99 wait_time and p99 use_time for the same pool on one panel:

histogram_quantile(0.99, sum by (le, db_client_connection_pool_name)
  (rate(db_client_connection_wait_time_seconds_bucket[5m])))

histogram_quantile(0.99, sum by (le, db_client_connection_pool_name)
  (rate(db_client_connection_use_time_seconds_bucket[5m])))

If wait rises while use is flat, demand exceeds the pool: more concurrent borrowers than connections, each holding them as briefly as before. If use rises first and wait follows, each borrower holds longer — slow queries, lock waits or application code inside a transaction — and a larger pool will only admit more load to a slower database.

Leaks. A leak shows count{state="used"} climbing to max and staying there through quiet periods, use_time histograms with observations piling into the highest buckets or not being recorded at all (a leaked connection is never returned, so its use time is never observed), and pending_requests rising. Pair it with the database’s view of idle sessions, as described in Detecting Connection Pool Saturation.

Connection churn. A steady rate of create_time observations while traffic is flat means connections are being closed and reopened: maxLifetime too short, minimumIdle below steady demand, or a proxy or firewall killing idle connections. The rate is visible as rate(db_client_connection_create_time_seconds_count[5m]).

From dashboard to trace. With exemplars enabled, a p99 wait_time point on a Grafana panel links to a trace ID. That trace shows the acquisition span, its duration, and — more usefully — its siblings: which other requests were in flight, and which of them held connections long. The step-by-step version is in Tracing Connection Acquisition Latency with OpenTelemetry Spans.

Integration & proxy compatibility

Behind PgBouncer or RDS Proxy, the client pool’s instruments describe connections to the proxy. A proxy that queues clients internally adds latency that the client sees as slow queries — inflated use_time — rather than as pool wait. Export the proxy’s own metrics through the same Collector (the PgBouncer exporter’s pgbouncer_pools_client_waiting_connections, CloudWatch’s DatabaseConnectionsBorrowLatency for RDS Proxy) and put them on the same dashboard as the client pool. A use_time rise that coincides with proxy waiting clients is the proxy’s queue, covered in Diagnosing PgBouncer cl_waiting Spikes.

Trace context into the database. SQL commenter options in several instrumentations append traceparent to each query as a comment. That lets a slow statement in pg_stat_activity or a query log be tied back to a trace, but it defeats prepared statement reuse, creates distinct query texts for pg_stat_statements, and under transaction pooling can interact badly with statement caching. Enable it selectively, not globally.

Service meshes add a hop between pool and database, and the mesh’s connection pool settings (Envoy max_connections, max_pending_requests) create a second queue invisible to the application’s pool metrics. Export the mesh’s upstream metrics for the database cluster alongside the pool instruments.

Semantic convention versions. Instrumentation released before the database conventions stabilised emits db.client.connections.usage with a state attribute and millisecond histograms such as db.client.connections.wait_time. Newer instrumentation emits db.client.connection.count with db.client.connection.state and second-based histograms. During a migration both may exist; dashboards should query one name consistently, with a Collector transform bridging the other.

Common failure patterns & remediation

Rolling out pool telemetry across a fleet

Instrumenting one service is a configuration change; instrumenting fifty is a migration, and the order matters more than any single setting.

Start with names. Before enabling instruments broadly, give every pool a stable, meaningful name and every service a service.name. Retrofitting names after dashboards exist breaks every query and every alert built on the generated ones. A naming convention of <service>-<database>-<role> — orders-api-orders-primary, orders-api-orders-replica — makes cross-service dashboards readable without a lookup table.

Enable metrics before spans. Pool metrics are cheap and immediately useful for saturation alerts. Acquisition spans add volume and are most valuable once the metrics show where contention is. Roll out in this order:

1. Pool instruments on every service, 15-second export interval, histograms with pool-specific buckets.
2. Recording rules and dashboards for utilisation, p99 wait, timeouts and create rate per pool.
3. Alerts on timeouts and sustained wait, routed to the owning team by service.name.
4. Acquisition spans or threshold events on the services whose wait histograms show contention.
5. Exemplars on the wait histograms, once tracing and metrics backends are both in place.

Pin the semantic convention version. Upgrading an instrumentation library can change metric names, units and attribute keys. Set the stability opt-in explicitly on every service during the rollout, and change it deliberately in one step across the fleet, with dashboards updated in the same change.

Budget the telemetry. Estimate series before enabling: roughly 58 series per pool with the default instrument set and 15-bucket histograms, multiplied by pools per process and process count. A fleet of 400 processes with two pools each is about 46,000 active series — modest for most Prometheus deployments, but worth checking against the metrics backend’s limits before the rollout rather than after.

Verify with a known incident. After the rollout, reproduce a pool exhaustion in staging — cap one service’s pool at two connections under load — and walk the whole path: the alert fires, the dashboard shows wait rising with use flat, an exemplar opens a trace, and the trace shows the acquisition span. If any link fails, the telemetry exists but will not help during a real incident.

Operational Boundary: Pool telemetry tells you which pool is saturated and which requests suffered. Deciding whether to resize the pool, fix queries or add capacity still depends on database-side evidence and the connection budget.

Frequently Asked Questions

Step-by-Step Guides

Each guide below applies this topic to one concrete incident or configuration task.

• Collecting db.client.connection Metrics with the OpenTelemetry Collector — route OpenTelemetry pool metrics through the Collector to Prometheus: bridge old and new names, fix temporality and units, control cardinality and keep exemplars.
• Instrumenting HikariCP with the OpenTelemetry Java Agent — get HikariCP pool metrics and JDBC spans from the OpenTelemetry Java agent without code changes: the properties that matter, pool naming, and what the agent cannot see.
• Linking Slow Queries to Traces with SQLCommenter — attach trace context to SQL as a comment so a slow statement in the database points back at the request, without breaking prepared statements or statement statistics.
• Tracing Connection Acquisition Latency with OpenTelemetry Spans — make pool wait visible in request traces with acquisition spans or threshold span events in Java, Python, Node.js and Go, and keep slow traces with tail sampling.

Related

• Connection Pool Observability — the parent overview of pool metrics, pipelines and alerting.
• Tracing Connection Acquisition Latency with OpenTelemetry Spans — making pool wait visible in request traces.
• Collecting db.client.connection Metrics with the OpenTelemetry Collector — the Collector pipeline, naming and exemplars.
• Prometheus and Grafana Pool Metrics — the Prometheus-native route to the same signals.
• Detecting Connection Pool Saturation — which signals indicate saturation and which only correlate with it.