Collecting db.client.connection Metrics with the OpenTelemetry Collector

This guide is part of OpenTelemetry Connection Pool Tracing. Services emit connection pool metrics over OTLP; dashboards and alerts read them from Prometheus. The OpenTelemetry Collector sits between, and most of the problems with pool dashboards come from what happens there rather than in the services. A fleet with instrumentation of different ages emits two naming generations for the same measurement, some SDKs send delta temporality that Prometheus cannot ingest as-is, units differ between milliseconds and seconds, and pod-level resource attributes multiply the series count. The first sign is usually a dashboard that shows half the fleet:

# Prometheus, same service, two deployments of different ages
db_client_connections_usage{pool_name="orders-primary",state="used",service_name="orders-api"}          14
db_client_connection_count{db_client_connection_pool_name="orders-primary",db_client_connection_state="used",service_name="orders-api"} 11

The objective is a Collector pipeline that delivers one consistent set of pool series in seconds, with cumulative temporality, bounded cardinality and exemplars that link back to traces.

Rapid incident diagnosis

List what is actually arriving. The Collector’s debug exporter prints metric names, types and temporality. Enable it briefly on a staging Collector with a filter for pool metrics:

processors:
  filter/pool-only:
    metrics:
      include:
        match_type: regexp
        metric_names: ["db\\.client\\.connections?\\..*"]
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    metrics/debug:
      receivers: [otlp]
      processors: [filter/pool-only]
      exporters: [debug]

In the output, check for each metric: the name generation (db.client.connections.* or db.client.connection.*), the unit (ms or s), the aggregation temporality (Delta or Cumulative) and the attributes.

Check Prometheus for gaps and duplicates.

count by (__name__) ({__name__=~"db_client_connections?_.*"})
count by (service_name) (db_client_connection_max) unless count by (service_name) (db_client_connections_max)

The first query shows both naming generations if they coexist; the second finds services that only emit the new names.

Common findings.

• Two name generations. Older instrumentation emits db.client.connections.usage with state and pool.name; newer emits db.client.connection.count with db.client.connection.state and db.client.connection.pool.name.
• Delta temporality. Some SDKs, notably when configured for backends that prefer delta, send counters and histograms as deltas. The Prometheus exporter drops or mishandles them, which shows as flat zeros or missing _bucket series.
• Unit mismatch. Older histograms such as db.client.connections.wait_time use milliseconds; the newer ones use seconds. Mixing them in one histogram_quantile gives values off by a factor of a thousand.
• Cardinality. Resource attributes such as k8s.pod.name and host.name promoted to labels create a new series set for every pod restart.

Mathematical sizing or parameter formula

Pool metrics are cheap per pool, but histograms multiply by bucket count and the fleet multiplies by pod count. Estimate active series before promoting attributes to labels:

series_per_pool = gauges × states + counters + histograms × (buckets + 2)
active_series   = pods × pools_per_pod × series_per_pool × label_variants

With the current instrument set — count split into two states, max, idle.min, idle.max and pending_requests as gauges (6 series), timeouts as a counter (1), and three histograms with 15 buckets each plus _sum and _count (51) — one pool produces 58 series. A deployment of 40 pods with 2 pools each produces 40 × 2 × 58 = 4,640 active series. That is modest. The multiplier to watch is label_variants: promoting k8s.pod.name is already included in pods, but adding a label that varies per request or per restart — a container ID, a process start time — makes series churn on every deploy, and the total over a retention window grows with deploy count.

Temporality affects correctness, not volume. Prometheus needs cumulative values. A delta counter converted to cumulative in the Collector must see every delta from a given stream, in order, on the same Collector instance; with several Collector replicas behind a load balancer, delta streams must be routed consistently or converted in the SDK instead.

Histogram buckets should be chosen for the instrument, not inherited from request latency defaults. For wait_time, include resolution near zero and a bucket at the acquisition timeout:

wait_time buckets (s):   0, 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30
use_time buckets (s):    0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10, 30, 60, 300

Buckets are set by views in the SDK. The Collector can drop buckets but cannot add resolution the SDK did not record.

Exact remediation & configuration

A Collector configuration that normalises names and units, converts temporality, trims resource labels and exports with exemplars:

receivers:
  otlp:
    protocols: { grpc: {}, http: {} }

processors:
  memory_limiter: { check_interval: 1s, limit_percentage: 80, spike_limit_percentage: 20 }

  # Delta streams from some SDKs → cumulative, as Prometheus expects
  deltatocumulative:
    max_stale: 10m

  # Bridge the older instrument names and units to the current conventions
  transform/pool-names:
    error_mode: ignore
    metric_statements:
      - context: metric
        statements:
          - set(name, "db.client.connection.count") where name == "db.client.connections.usage"
          - set(name, "db.client.connection.max") where name == "db.client.connections.max"
          - set(name, "db.client.connection.pending_requests") where name == "db.client.connections.pending_requests"
          - set(name, "db.client.connection.timeouts") where name == "db.client.connections.timeouts"
      - context: datapoint
        statements:
          - set(attributes["db.client.connection.pool.name"], attributes["pool.name"]) where attributes["pool.name"] != nil
          - set(attributes["db.client.connection.state"], attributes["state"]) where attributes["state"] != nil
          - delete_key(attributes, "pool.name")
          - delete_key(attributes, "state")

  # Keep only the resource attributes dashboards group by
  resource/trim:
    attributes:
      - { key: process.pid, action: delete }
      - { key: process.command_args, action: delete }
      - { key: container.id, action: delete }

  batch: { timeout: 5s, send_batch_size: 2048 }

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
    enable_open_metrics: true               # exemplars require OpenMetrics
    resource_to_telemetry_conversion: { enabled: true }
    metric_expiration: 10m

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, deltatocumulative, transform/pool-names, resource/trim, batch]
      exporters: [prometheus]

The old millisecond histograms (db.client.connections.wait_time, use_time, create_time) cannot be rescaled into second-based buckets reliably, because bucket boundaries do not convert cleanly. Rename them with an explicit unit suffix so they never mix with the new ones, and migrate those services to current instrumentation:

          - set(name, "db.client.connections.wait_time_ms") where name == "db.client.connections.wait_time"

Prometheus must accept exemplars. Scrape the Collector with OpenMetrics negotiation and enable exemplar storage:

# prometheus.yml
scrape_configs:
  - job_name: otel-collector
    scrape_protocols: [OpenMetricsText1.0.0, PrometheusText0.0.4]
    static_configs:
      - targets: ["otel-collector:8889"]

prometheus --enable-feature=exemplar-storage --config.file=prometheus.yml

Recording rules keep dashboard queries cheap and give alerts stable names:

groups:
  - name: pool-recording
    interval: 30s
    rules:
      - record: pool:wait_time_seconds:p99_5m
        expr: histogram_quantile(0.99, sum by (le, service_name, db_client_connection_pool_name)
              (rate(db_client_connection_wait_time_seconds_bucket[5m])))
      - record: pool:utilisation:ratio
        expr: sum by (service_name, db_client_connection_pool_name) (db_client_connection_count{db_client_connection_state="used"})
              / sum by (service_name, db_client_connection_pool_name) (db_client_connection_max)
      - record: pool:timeouts:rate5m
        expr: sum by (service_name, db_client_connection_pool_name) (rate(db_client_connection_timeouts_total[5m]))

Validation & verification

One name per measurement. After deploying the Collector change, the query below must return only current names for every service:

count by (__name__) ({__name__=~"db_client_connections_(usage|max|pending_requests|timeouts).*"})

It should return nothing once the transform is active (the millisecond histograms, renamed with _ms, are the deliberate exception).

Cumulative behaviour. db_client_connection_timeouts_total must be monotonic per series between restarts. A resets() count much higher than the number of pod restarts means delta streams are still arriving unconverted or are split across Collector replicas:

resets(db_client_connection_timeouts_total[1h])

Units. Induce a known wait in staging — a pool of 1 and two concurrent requests that hold connections for 500 ms — and check that pool:wait_time_seconds:p99_5m reports roughly 0.5, not 500.

Exemplars. In Grafana, enable exemplars on the p99 wait panel for the induced-wait window. Points must appear, and each must open a trace containing the acquisition span described in Tracing Connection Acquisition Latency with OpenTelemetry Spans.

Series count. Check the Collector’s exported series against the estimate:

count({__name__=~"db_client_connection_.*"})

A value far above pods × pools × 58 points to a label that should have been trimmed.

Alerting from Collector-exported pool metrics

With names and units normalised, alert rules can be written once for every runtime in the fleet. The rules below use the recording rules from the configuration section and group by service and pool name, so each alert identifies exactly which pool is in trouble.

groups:
  - name: pool-alerts
    rules:
      - alert: PoolAcquisitionTimeouts
        expr: pool:timeouts:rate5m > 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "{{ $labels.service_name }} pool {{ $labels.db_client_connection_pool_name }}: requests timing out waiting for connections"

      - alert: PoolWaitHigh
        expr: pool:wait_time_seconds:p99_5m > 0.25
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "{{ $labels.service_name }} pool {{ $labels.db_client_connection_pool_name }}: p99 acquisition wait above 250 ms"

      - alert: PoolConnectionChurn
        expr: sum by (service_name, db_client_connection_pool_name) (rate(db_client_connection_create_time_seconds_count[10m])) > 1
        for: 30m
        labels: { severity: info }
        annotations:
          summary: "{{ $labels.service_name }} pool {{ $labels.db_client_connection_pool_name }}: more than one new connection per second"

      - alert: PoolTelemetryMissing
        expr: absent_over_time(db_client_connection_max{service_name="orders-api"}[15m])
        labels: { severity: warning }
        annotations:
          summary: "orders-api has stopped reporting pool metrics"

The last rule matters more than it looks. A pool that stops exporting metrics — after an instrumentation upgrade renames instruments, or a Collector pipeline change drops them — makes every other alert silently stop working. Add one absence rule per critical service.

Choose the wait threshold from the acquisition timeout rather than a round number. A p99 wait above 10 % of the timeout means some requests are spending a noticeable share of their allowance in the pool; above 50 %, timeouts are imminent:

Route alerts by service_name to the owning team, and include the pool name in the notification. For shared databases, add a second route by database so the database owner sees saturation across every service that connects to it, which is often the first indication that the problem is capacity rather than one service’s configuration.

Frequently Asked Questions

Related

• OpenTelemetry Connection Pool Tracing — the parent guide to the instruments and what they mean.
• Tracing Connection Acquisition Latency with OpenTelemetry Spans — the trace side that exemplars link to.
• Building a Grafana Dashboard for Connection Pool Metrics — panels built on the recording rules above.
• Exporting Go database/sql Pool Metrics to Prometheus — the Prometheus-native alternative for Go services.
• Instrumenting HikariCP with the OpenTelemetry Java Agent — producing the metrics this pipeline collects, without code changes.