Exporting node-postgres Pool Metrics to Prometheus

This guide is part of Prometheus and Grafana Pool Metrics. pg.Pool exposes exactly three numbers — totalCount, idleCount and waitingCount — and emits four events. That is less than HikariCP or psycopg offer, and it is enough to answer every operational question about the pool, provided the numbers are exported and the events are counted. Without them, a Node.js service in trouble produces a stream of identical errors and no way to tell whether the pool was full, the database was slow, or connections were failing to establish:

Error: timeout exceeded when trying to connect
    at /app/node_modules/pg-pool/index.js:45:11
Error: Connection terminated unexpectedly
    at Connection. (/app/node_modules/pg/lib/client.js:132:73)

The objective is a small, permanent instrumentation that publishes pool state, acquisition timing and failure counts, with alerts on the leading signals rather than on the errors.

Rapid incident diagnosis

Read the three counters. They are synchronous properties, so a debug endpoint costs nothing:

app.get("/debug/pool", (_req, res) => {
  res.json({ total: pool.totalCount, idle: pool.idleCount, waiting: pool.waitingCount, max: pool.options.max });
});

• totalCount at max with idleCount zero and waitingCount above zero: the pool is saturated.
• totalCount below max with waitingCount above zero: connections are being established and callers are waiting for them — establishment is the constraint.
• totalCount at max with idleCount at max: the pool is idle and holding connections; a candidate for a lower idleTimeoutMillis.

Subscribe to the events for a minute. They distinguish creation from checkout:

for (const ev of ["connect", "acquire", "release", "remove", "error"]) {
  pool.on(ev, (arg) => console.log(ev, arg instanceof Error ? arg.message : ""));
}

A stream of connect and remove in equal numbers is churn; error events are connections failing while idle in the pool.

Confirm from the database. With application_name set on the pool:

SELECT state, count(*), max(now() - state_change) AS longest
FROM pg_stat_activity WHERE application_name = 'orders-api' GROUP BY state;

Handle the error event or the process dies. An idle client that errors emits error on the pool; unhandled, Node.js treats it as an uncaught exception:

grep -rn "pool.on(\"error\"\|pool.on('error'" --include=*.js --include=*.ts src/ | head

An empty result is a latent crash, not a metrics gap — the same failure described in Handling node-postgres Pool Errors and Reconnection.

Mathematical sizing or parameter formula

Derive the rates the counters cannot give. Acquisition timing needs to be measured around the call:

acquire_wait   = time from pool.connect() to resolution
checkout_ratio = acquisitions that waited ÷ total acquisitions

Sampling interval for the gauges. As with any sampled gauge, a burst shorter than the interval can be missed:

p(burst observed) ≈ min(1, burst_duration ÷ sample_interval)

A 10-second sample interval catches most bursts that matter; anything shorter costs little in a Node.js process because reading the counters is free.

Series count. The instrumentation below produces about a dozen series per pool, plus histogram buckets:

series ≈ 3 gauges + 4 counters + histogram(buckets + 2)

With 10 buckets, roughly 19 series per pool — negligible.

Alert thresholds from the timeout. The acquisition timeout is the deadline, so warnings belong well below it:

warn  : p99 acquire_wait > 0.1 × connectionTimeoutMillis
page  : any acquisition failures, or waitingCount > 0 sustained for minutes

With a 2,000 ms timeout, warn at 200 ms.

Churn budget. Connection creation should be rare relative to traffic:

expected_creations_per_second ≈ pool_max ÷ (idleTimeoutMillis ÷ 1000)   when idle

With max: 10 and a 120-second idle timeout, at most 0.08 creations per second when traffic stops. A much higher steady rate means connections are dying rather than being reused.

Exact remediation & configuration

A complete instrumentation module:

// db-metrics.js
import { Counter, Gauge, Histogram, register } from "prom-client";

export function instrumentPool(pool, { name = "default", sampleMs = 10_000 } = {}) {
  const labels = { pool: name };

  const total = new Gauge({ name: "pg_pool_total_connections", help: "Connections in the pool", labelNames: ["pool"] });
  const idle = new Gauge({ name: "pg_pool_idle_connections", help: "Idle connections", labelNames: ["pool"] });
  const waiting = new Gauge({ name: "pg_pool_waiting_requests", help: "Requests waiting for a connection", labelNames: ["pool"] });
  const max = new Gauge({ name: "pg_pool_max_connections", help: "Configured maximum", labelNames: ["pool"] });

  const created = new Counter({ name: "pg_pool_connections_created_total", help: "Physical connections opened", labelNames: ["pool"] });
  const removed = new Counter({ name: "pg_pool_connections_removed_total", help: "Connections removed from the pool", labelNames: ["pool"] });
  const acquired = new Counter({ name: "pg_pool_acquisitions_total", help: "Successful acquisitions", labelNames: ["pool"] });
  const failed = new Counter({ name: "pg_pool_acquisition_failures_total", help: "Failed acquisitions", labelNames: ["pool", "reason"] });
  const idleErrors = new Counter({ name: "pg_pool_idle_client_errors_total", help: "Errors on idle clients", labelNames: ["pool"] });

  const waitSeconds = new Histogram({
    name: "pg_pool_acquire_seconds", help: "Time to acquire a connection", labelNames: ["pool"],
    buckets: [0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.25, 0.5, 1, 2, 5],
  });

  pool.on("connect", () => created.inc(labels));
  pool.on("remove", () => removed.inc(labels));
  pool.on("acquire", () => acquired.inc(labels));
  pool.on("error", () => idleErrors.inc(labels));

  const timer = setInterval(() => {
    total.set(labels, pool.totalCount);
    idle.set(labels, pool.idleCount);
    waiting.set(labels, pool.waitingCount);
    max.set(labels, pool.options.max ?? 10);
  }, sampleMs);
  timer.unref();

  // Wrap connect() so every acquisition is timed and failures are attributed
  const originalConnect = pool.connect.bind(pool);
  pool.connect = async (...args) => {
    const end = waitSeconds.startTimer(labels);
    try {
      const client = await originalConnect(...args);
      end();
      return client;
    } catch (err) {
      end();
      const reason = /timeout/i.test(err?.message ?? "") ? "timeout" : "error";
      failed.inc({ ...labels, reason });
      throw err;
    }
  };

  return { register };
}

// server.js
import pg from "pg";
import { instrumentPool } from "./db-metrics.js";

export const pool = new pg.Pool({
  connectionString: process.env.DATABASE_URL,
  max: 10,
  idleTimeoutMillis: 120_000,
  connectionTimeoutMillis: 2_000,
  application_name: "orders-api",
});

pool.on("error", (err) => logger.warn({ err }, "idle client error"));   // required: prevents a crash
instrumentPool(pool, { name: "orders-primary" });

app.get("/metrics", async (_req, res) => {
  res.set("Content-Type", register.contentType).end(await register.metrics());
});

Wrapping pool.connect covers both direct acquisitions and pool.query, because the latter acquires internally through the same method in current versions; where that is not the case, time pool.query as well.

Alert on the sequence:

groups:
  - name: pg-pool
    rules:
      - alert: PgPoolAcquisitionFailures
        expr: rate(pg_pool_acquisition_failures_total[5m]) > 0
        for: 2m
        labels: { severity: critical }
        annotations:
          summary: "{{ $labels.pool }}: acquisitions failing ({{ $labels.reason }})"

      - alert: PgPoolWaitingSustained
        expr: max_over_time(pg_pool_waiting_requests[5m]) > 0
        for: 10m
        labels: { severity: warning }

      - alert: PgPoolAcquireSlow
        expr: histogram_quantile(0.99, sum by (le, pool) (rate(pg_pool_acquire_seconds_bucket[5m]))) > 0.2
        for: 10m
        labels: { severity: warning }

      - alert: PgPoolChurn
        expr: rate(pg_pool_connections_created_total[10m]) > 1
        for: 30m
        labels: { severity: info }
        annotations:
          summary: "{{ $labels.pool }}: more than one new connection per second — check idleTimeoutMillis"

Instrument every pool. A service with a writer and a reader pool, or a Prisma driver adapter, should name each one; the label is what makes a dashboard readable.

Dashboard and comparison with other pools

Four panels, the same as any pool dashboard:

Split by pool and by instance rather than summing: with several Node.js workers per container, one worker’s saturation is invisible in a sum.

Naming consistency across runtimes. A fleet with Java, Python and Node.js services benefits from one naming scheme. Two options work: map everything onto the OpenTelemetry db.client.connection.* conventions, or keep runtime-native names and reconcile in the dashboard. The first is cleaner and is described in OpenTelemetry Connection Pool Tracing; the second is less work when only one service is in Node.js.

What Node.js cannot offer that others do. pg.Pool has no equivalent of HikariCP’s usage timer, so hold time must be derived — by timing from acquisition to release in the wrapper, which the module above can be extended to do:

const useSeconds = new Histogram({ name: "pg_pool_use_seconds", help: "Time a connection is held",
  labelNames: ["pool"], buckets: [0.001, 0.01, 0.05, 0.1, 0.5, 1, 5] });

// inside the connect wrapper, after acquiring:
const endUse = useSeconds.startTimer(labels);
const release = client.release.bind(client);
client.release = (err) => { endUse(); return release(err); };

Hold time is what distinguishes a pool that is too small from one whose connections are held too long, so it is worth the extra few lines.

Validation & verification

Every metric moves. Under load, confirm the gauges track the counters shown by the debug endpoint, and that the histogram has observations.

Saturation produces the expected sequence. Cap the pool at 1 in staging under concurrency: waiting rises, the histogram’s tail extends to the timeout, and failures appear with reason timeout.

The gauges return to baseline. After traffic stops, waiting must be zero and total must fall to zero (or to the configured minimum if one is emulated) within idleTimeoutMillis.

The error handler works. Terminate an idle backend on the server and confirm pg_pool_idle_client_errors_total increments and the process survives.

Hold time is plausible. Compare the p50 of pg_pool_use_seconds with the database’s mean statement time; a much larger value points at application work inside the checkout.

Frequently Asked Questions

Related

• Prometheus and Grafana Pool Metrics — the parent guide to the collection pipeline.
• Exporting Go database/sql Pool Metrics to Prometheus — the same exercise in Go.
• Tuning node-postgres Pool Timeout Settings — the settings these metrics validate.
• Building a Grafana Dashboard for Connection Pool Metrics — the panel conventions used above.