Implementing graceful connection pool shutdown in Express
This guide is part of Express.js Connection Pool Middleware. Unhandled process termination in Express leaves database connections in IDLE IN TRANSACTION or CLOSE_WAIT states. This triggers pool exhaustion and ECONNRESET errors during rolling updates. This guide provides a deterministic shutdown sequence to halt routing, drain active queries, and safely release pool resources. Proper signal interception becomes the primary control for zero-downtime deployments.
Key operational objectives:
- Intercept
SIGTERM/SIGINTsignals before process exit - Stop the Express HTTP server to reject new TCP handshakes
- Await
pool.end()to flush in-flight queries and transactions - Validate connection teardown via OS-level sockets and DB-side metrics
Diagnosing the Shutdown Failure Mode
Improper pool teardown manifests immediately during deployment pipelines. Monitor your orchestration layer for ECONNRESET spikes coinciding with pod termination events. Lingering sockets prevent the database from reclaiming allocated memory.
Run ss -tnp | grep <port> on the host to identify CLOSE_WAIT states. Cross-reference your database max_connections utilization against deployment timestamps. Verify that liveness and readiness probes are not failing due to stalled query execution.
Connection leaks during rolling updates are typically caused by missing process signal handlers. Tracking middleware-bound clients before termination depends on the request-scoped acquisition pattern, and mid-drain backend drops should be handled per Handling node-postgres Pool Errors and Reconnection so the drain does not hang on a dead socket.
Implementing the Graceful Shutdown Sequence
A deterministic drain requires strict ordering. First, invoke server.close() to stop accepting new HTTP requests. Existing sockets complete their current response cycle.
Set a hard timeout (typically 30s) to force exit if the drain stalls. This prevents zombie processes from blocking orchestrator health checks.
Call pool.end() only after the HTTP server fully closes. This guarantees no new queries are dispatched during teardown. Attach a pool.on('error') listener before shutdown to catch mid-drain network drops or abrupt database restarts.
Exact Remediation: Pool Drain Implementation
The following implementation enforces async/await boundaries. The pg library’s pool.end() method waits for all checked-out clients to be released, then destroys all connections. No new connections can be acquired after pool.end() is called — there is no need to manually zero out pool.max. Drain duration is logged for SLO tracking and alert tuning.
const http = require('http');
const { Pool } = require('pg');
const pool = new Pool({ max: 20 });
const app = require('./app');
const server = http.createServer(app);
let isShuttingDown = false;
// Reject new requests during drain
app.use((req, res, next) => {
if (isShuttingDown) {
res.set('Connection', 'close');
return res.status(503).json({ error: 'Server shutting down' });
}
next();
});
async function gracefulShutdown(signal) {
console.log(`${signal} received. Draining connections...`);
isShuttingDown = true;
// 1. Stop accepting new TCP connections
server.close(async () => {
console.log('HTTP server closed.');
try {
// 2. Wait for all checked-out clients to be released, then destroy the pool
await pool.end();
console.log('Database pool drained successfully.');
process.exit(0);
} catch (err) {
console.error('Pool drain failed:', err);
process.exit(1);
}
});
// 3. Hard timeout to prevent deployment pipeline stalls
setTimeout(() => {
console.error('Shutdown timeout exceeded. Forcing exit.');
process.exit(1);
}, 30000);
}
process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
process.on('SIGINT', () => gracefulShutdown('SIGINT'));
server.listen(3000);
pool.end() resolves once every client has called client.release() and all underlying connections are destroyed. If any client is never released (a leak), the drain will hang until the hard timeout fires — making leak detection a prerequisite for clean shutdowns.
Validation Commands & Post-Teardown Verification
Execute these commands immediately after deployment to confirm zero dangling connections. Validate socket closure at the OS layer before querying database metadata.
# Linux socket verification — check for CLOSE_WAIT on PostgreSQL port 5432
ss -tnp | grep 5432 | awk '{print $1, $6}' | sort | uniq -c
# PostgreSQL active connection check
psql -U admin -d mydb -c "SELECT state, count(*) FROM pg_stat_activity WHERE datname = 'mydb' GROUP BY state;"
# MySQL active connection check
mysql -u root -p -e "SHOW PROCESSLIST;"
| Metric | Safe Threshold | Alert Condition | Action |
|---|---|---|---|
CLOSE_WAIT sockets |
0 | > 5 | Force restart DB proxy, audit signal handlers |
pg_stat_activity idle count |
0 | > pool.min |
Check for unhandled pool.end() rejections |
ECONNRESET rate (APM) |
0 | > 0.1% | Reduce timeout, verify load balancer drain window |
pool.totalCount post-shutdown |
0 | > 0 | Verify server.close() executes before pool.end() |
Common Mistakes
| Issue | Operational Impact | Remediation |
|---|---|---|
| Not setting a hard shutdown timeout | Process hangs indefinitely if a connection is never released, stalling rolling deploys | Add setTimeout(() => process.exit(1), 30000) immediately after calling server.close() |
Terminating process before pool.end() resolves |
Leaves connections in CLOSE_WAIT state. Exhausts max_connections on subsequent deploys and triggers ECONNREFUSED. |
Wrap pool.end() in an await block inside the server.close() callback. |
| Not closing HTTP server before draining pool | New requests acquire connections during shutdown. Prevents pool from reaching zero active queries, causing an infinite drain loop. | Call server.close() first. Only proceed to pool.end() inside the server.close() callback. |
FAQ
How long should I set the shutdown timeout before forcing process.exit()?
pool.end() hangs to prevent zombie processes and deployment pipeline stalls.Can I reuse the pool instance after calling pool.end()?
pool.end() permanently closes all underlying client connections. Re-instantiate the pool if the process continues running in a long-lived worker scenario.How do I handle transactions that are still open during shutdown?
end() method waits for active clients to be released, but it does not cancel in-progress queries. For long-running transactions, enforce a strict application-level drain window (e.g., 10s) and ensure all route handlers are wrapped in try/finally so connections return to the pool before shutdown completes.Related
- Express.js Connection Pool Middleware — the parent guide on request-scoped acquisition and pool lifecycle integration.
- Handling node-postgres Pool Errors and Reconnection — recovering from backend drops that can stall a drain sequence.
- Node.js Async Connection Limits — event-loop concurrency bounds that shape pool sizing and drain timing.