Implementing graceful connection pool shutdown in Express

This guide is part of Express.js Connection Pool Middleware. Unhandled process termination in Express leaves database connections in IDLE IN TRANSACTION or CLOSE_WAIT states. This triggers pool exhaustion and ECONNRESET errors during rolling updates. This guide provides a deterministic shutdown sequence to halt routing, drain active queries, and safely release pool resources. Proper signal interception becomes the primary control for zero-downtime deployments.

Key operational objectives:

  • Intercept SIGTERM/SIGINT signals before process exit
  • Stop the Express HTTP server to reject new TCP handshakes
  • Await pool.end() to flush in-flight queries and transactions
  • Validate connection teardown via OS-level sockets and DB-side metrics

Diagnosing the Shutdown Failure Mode

Improper pool teardown manifests immediately during deployment pipelines. Monitor your orchestration layer for ECONNRESET spikes coinciding with pod termination events. Lingering sockets prevent the database from reclaiming allocated memory.

Run ss -tnp | grep <port> on the host to identify CLOSE_WAIT states. Cross-reference your database max_connections utilization against deployment timestamps. Verify that liveness and readiness probes are not failing due to stalled query execution.

Connection leaks during rolling updates are typically caused by missing process signal handlers. Tracking middleware-bound clients before termination depends on the request-scoped acquisition pattern, and mid-drain backend drops should be handled per Handling node-postgres Pool Errors and Reconnection so the drain does not hang on a dead socket.

Implementing the Graceful Shutdown Sequence

A deterministic drain requires strict ordering. First, invoke server.close() to stop accepting new HTTP requests. Existing sockets complete their current response cycle.

Set a hard timeout (typically 30s) to force exit if the drain stalls. This prevents zombie processes from blocking orchestrator health checks.

Call pool.end() only after the HTTP server fully closes. This guarantees no new queries are dispatched during teardown. Attach a pool.on('error') listener before shutdown to catch mid-drain network drops or abrupt database restarts.

Exact Remediation: Pool Drain Implementation

The following implementation enforces async/await boundaries. The pg library’s pool.end() method waits for all checked-out clients to be released, then destroys all connections. No new connections can be acquired after pool.end() is called — there is no need to manually zero out pool.max. Drain duration is logged for SLO tracking and alert tuning.

const http = require('http');
const { Pool } = require('pg');

const pool = new Pool({ max: 20 });
const app = require('./app');
const server = http.createServer(app);

let isShuttingDown = false;

// Reject new requests during drain
app.use((req, res, next) => {
  if (isShuttingDown) {
    res.set('Connection', 'close');
    return res.status(503).json({ error: 'Server shutting down' });
  }
  next();
});

async function gracefulShutdown(signal) {
  console.log(`${signal} received. Draining connections...`);
  isShuttingDown = true;

  // 1. Stop accepting new TCP connections
  server.close(async () => {
    console.log('HTTP server closed.');

    try {
      // 2. Wait for all checked-out clients to be released, then destroy the pool
      await pool.end();
      console.log('Database pool drained successfully.');
      process.exit(0);
    } catch (err) {
      console.error('Pool drain failed:', err);
      process.exit(1);
    }
  });

  // 3. Hard timeout to prevent deployment pipeline stalls
  setTimeout(() => {
    console.error('Shutdown timeout exceeded. Forcing exit.');
    process.exit(1);
  }, 30000);
}

process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
process.on('SIGINT', () => gracefulShutdown('SIGINT'));

server.listen(3000);

pool.end() resolves once every client has called client.release() and all underlying connections are destroyed. If any client is never released (a leak), the drain will hang until the hard timeout fires — making leak detection a prerequisite for clean shutdowns.

Validation Commands & Post-Teardown Verification

Execute these commands immediately after deployment to confirm zero dangling connections. Validate socket closure at the OS layer before querying database metadata.

# Linux socket verification — check for CLOSE_WAIT on PostgreSQL port 5432
ss -tnp | grep 5432 | awk '{print $1, $6}' | sort | uniq -c

# PostgreSQL active connection check
psql -U admin -d mydb -c "SELECT state, count(*) FROM pg_stat_activity WHERE datname = 'mydb' GROUP BY state;"

# MySQL active connection check
mysql -u root -p -e "SHOW PROCESSLIST;"
Metric Safe Threshold Alert Condition Action
CLOSE_WAIT sockets 0 > 5 Force restart DB proxy, audit signal handlers
pg_stat_activity idle count 0 > pool.min Check for unhandled pool.end() rejections
ECONNRESET rate (APM) 0 > 0.1% Reduce timeout, verify load balancer drain window
pool.totalCount post-shutdown 0 > 0 Verify server.close() executes before pool.end()

Common Mistakes

Issue Operational Impact Remediation
Not setting a hard shutdown timeout Process hangs indefinitely if a connection is never released, stalling rolling deploys Add setTimeout(() => process.exit(1), 30000) immediately after calling server.close()
Terminating process before pool.end() resolves Leaves connections in CLOSE_WAIT state. Exhausts max_connections on subsequent deploys and triggers ECONNREFUSED. Wrap pool.end() in an await block inside the server.close() callback.
Not closing HTTP server before draining pool New requests acquire connections during shutdown. Prevents pool from reaching zero active queries, causing an infinite drain loop. Call server.close() first. Only proceed to pool.end() inside the server.close() callback.

FAQ

How long should I set the shutdown timeout before forcing process.exit()?
Set 30-45 seconds to align with typical load balancer drain timeouts. Force exit if pool.end() hangs to prevent zombie processes and deployment pipeline stalls.
Can I reuse the pool instance after calling pool.end()?
No. pool.end() permanently closes all underlying client connections. Re-instantiate the pool if the process continues running in a long-lived worker scenario.
How do I handle transactions that are still open during shutdown?
The pool’s end() method waits for active clients to be released, but it does not cancel in-progress queries. For long-running transactions, enforce a strict application-level drain window (e.g., 10s) and ensure all route handlers are wrapped in try/finally so connections return to the pool before shutdown completes.