Draining HikariCP During Spring Boot Graceful Shutdown

This guide is part of Spring Boot DataSource Configuration. Every rolling deploy stops pods, and a Spring Boot service that stops badly produces a short burst of errors each time: requests that were accepted after the pod was already terminating, background tasks that reach for a connection after the pool has closed, and queries aborted in the middle of a transaction. None of it shows up in load tests, and all of it shows up in error-rate graphs at deploy time:

2026-09-16T08:14:03.912Z  INFO 1 --- [ionShutdownHook] o.s.b.w.e.tomcat.GracefulShutdown : Commencing graceful shutdown. Waiting for active requests to complete
2026-09-16T08:14:04.020Z  INFO 1 --- [ionShutdownHook] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2026-09-16T08:14:04.311Z ERROR 1 --- [   task-3] c.e.orders.OutboxPublisher : publish failed
java.sql.SQLException: HikariDataSource HikariDataSource (HikariPool-1) has been closed.

The objective is a shutdown sequence in which the pod leaves the load balancer first, in-flight requests finish, executors and schedulers drain, and only then does HikariCP close — with timeouts that fit inside the platform’s grace period.

Rapid incident diagnosis

Read the shutdown log in order. Spring Boot logs each stage. Extract the lines from one terminating pod:

kubectl logs orders-api-7f9c6b7d8-x2k4p --previous \
  | grep -E "Commencing graceful shutdown|Graceful shutdown complete|Graceful shutdown aborted|HikariPool-1 - Shutdown|has been closed|Stopping service"

The healthy order is: Commencing graceful shutdown → Graceful shutdown complete → HikariPool-1 - Shutdown initiated → HikariPool-1 - Shutdown completed. Three unhealthy variants are common:

• Graceful shutdown aborted with one or more requests still active — requests took longer than spring.lifecycle.timeout-per-shutdown-phase (30 seconds by default). The pool then closed under them.
• has been closed from executor or scheduler threads — @Async methods, @Scheduled jobs or message listeners were still running when the data source was destroyed. Web graceful shutdown only waits for HTTP requests.
• No shutdown lines at all — the container was killed with SIGKILL because the Kubernetes terminationGracePeriodSeconds expired first.

Check for requests arriving after termination began. If access logs show requests accepted after Commencing graceful shutdown, and they receive 503 or connection resets, the pod was still in the Service endpoints. Kubernetes removes endpoints asynchronously; without a preStop delay, traffic keeps arriving for a few seconds after SIGTERM.

Look at the database side. Aborted connections leave a trace in PostgreSQL logs at deploy times:

grep -E "unexpected EOF on client connection with an open transaction|could not receive data from client" \
  /var/log/postgresql/postgresql-*.log | awk '{print $1, $2}' | cut -c1-16 | uniq -c

Clusters of these lines at deploy timestamps are transactions HikariCP aborted during close().

Mathematical sizing or parameter formula

The platform’s grace period must contain every stage:

terminationGracePeriodSeconds ≥ preStop_sleep + timeout_per_shutdown_phase × phases_that_wait + pool_close + margin

pool_close is bounded by HikariCP itself: HikariDataSource.close() evicts idle connections immediately, then repeatedly aborts connections that are still in use for up to about 10 seconds. If the earlier phases drained correctly there are no active connections and close takes milliseconds; the 10 seconds is the worst case.

Choose timeout_per_shutdown_phase from the longest legitimate unit of work that must finish, not from a default:

timeout_per_shutdown_phase ≥ max( p99_request_duration, longest_async_task, longest_scheduled_job_iteration )

Worked example. p99 request duration is 1.2 seconds, the outbox publisher’s longest batch is 8 seconds, and a scheduled reconciliation job runs for up to 20 seconds. The phase timeout must be at least 20 seconds; use 25. Endpoint removal typically propagates within a few seconds; a 10-second preStop covers slow kube-proxy and ingress updates. Then:

grace ≥ 10 + 25 + 10 + 5 = 50  →  terminationGracePeriodSeconds: 60

The rollout budget matters too. During a rolling update, a terminating pod keeps its pool until close() completes — up to 45 seconds in this example — while its replacement has already opened its own. With maxSurge: 1, the service briefly claims one extra pool; with maxSurge: 25% on 12 replicas, three extra pools. Include that in the database connection budget.

Exact remediation & configuration

Enable graceful shutdown and wait for executors. Recent Spring Boot releases enable web graceful shutdown by default; set it explicitly so the behaviour does not depend on version, and make the task executors wait:

server:
  shutdown: graceful

spring:
  lifecycle:
    timeout-per-shutdown-phase: 25s
  task:
    execution:
      shutdown:
        await-termination: true
        await-termination-period: 20s
    scheduling:
      shutdown:
        await-termination: true
        await-termination-period: 20s
  datasource:
    hikari:
      pool-name: orders-pool
      maximum-pool-size: 16
      connection-timeout: 3000

management:
  endpoint:
    health:
      probes:
        enabled: true      # readiness flips to OUT_OF_SERVICE as shutdown starts

Make long-running components stop before the data source. Beans that loop over work — an outbox publisher, a polling consumer — should implement SmartLifecycle so Spring stops them during the lifecycle phases, before singleton beans such as the DataSource are destroyed:

@Component
public class OutboxPublisher implements SmartLifecycle {
    private final DataSource dataSource;
    private final ExecutorService worker = Executors.newSingleThreadExecutor();
    private volatile boolean running;

    public OutboxPublisher(DataSource dataSource) { this.dataSource = dataSource; }

    @Override public void start() { running = true; worker.submit(this::loop); }

    @Override public void stop(Runnable callback) {
        running = false;                                   // finish the current batch, take no new one
        worker.shutdown();
        CompletableFuture.runAsync(() -> {
            try { worker.awaitTermination(20, TimeUnit.SECONDS); }
            catch (InterruptedException e) { Thread.currentThread().interrupt(); }
            callback.run();
        });
    }

    @Override public void stop() { stop(() -> {}); }
    @Override public boolean isRunning() { return running; }
    @Override public int getPhase() { return Integer.MAX_VALUE - 1000; }   // stops early, after the web server's graceful phase

    private void loop() {
        while (running) {
            try (Connection c = dataSource.getConnection()) {
                publishBatch(c);
            } catch (SQLException e) {
                log.warn("outbox batch failed", e);
            }
        }
    }
}

Because the DataSource is injected into this bean, Spring destroys the bean before the data source, and stop(Runnable) completes before the context starts destroying singletons at all.

Give Kubernetes time for both.

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: orders-api
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 10"]
      readinessProbe:
        httpGet: { path: /actuator/health/readiness, port: 8080 }
        periodSeconds: 5
        failureThreshold: 1

The preStop sleep runs before the container receives SIGTERM, so the JVM keeps serving normally while endpoints are removed; the readiness probe then reports out of service as soon as shutdown begins, which removes the pod from any endpoint the sleep did not already cover.

Validation & verification

Deploy under load and count errors. Run a steady load against staging and trigger a rollout. The error count during the rollout window must be zero:

k6 run --vus 50 --duration 5m steady.js &
sleep 60 && kubectl rollout restart deploy/orders-api && kubectl rollout status deploy/orders-api

Check the shutdown log of a terminated pod for the healthy order, and the absence of aborted and has been closed:

kubectl logs deploy/orders-api --previous | grep -cE "Graceful shutdown aborted|has been closed"

The count must be 0.

Confirm no active connections remain at close. HikariCP logs pool statistics at debug level; enable it for the pool during the test:

logging:
  level:
    com.zaxxer.hikari.pool.HikariPool: DEBUG

The last orders-pool - Before shutdown stats line should show active=0. A non-zero value is the number of connections aborted.

Database side. No unexpected EOF on client connection with an open transaction lines should appear in the PostgreSQL log for the rollout window, and the service’s session count should drop by one pool size per terminated pod rather than lingering until TCP timeouts close sessions.

Startup is the other half of a rollout

A rolling deploy stops old pods and starts new ones, and the start matters as much as the stop. A pod marked ready before its pool can serve traffic receives requests that wait for connections to be created, and a fleet of new pods creating connections at once can briefly exceed the database’s connection rate limits or max_connections.

Warm the pool before readiness. HikariCP creates minimumIdle connections in the background after the data source is initialised. Spring Boot’s readiness state becomes ACCEPTING_TRAFFIC when the application context has started, which can precede those connections. Include the data source in the readiness group so the probe fails until the database is reachable:

management:
  endpoint:
    health:
      group:
        readiness:
          include: readinessState, db

For services where first-request latency matters, validate that the pool actually holds warm connections before reporting ready, using a startup runner that borrows and returns minimumIdle connections:

@Component
public class PoolWarmup implements ApplicationRunner {
    private final HikariDataSource dataSource;
    public PoolWarmup(HikariDataSource dataSource) { this.dataSource = dataSource; }

    @Override
    public void run(ApplicationArguments args) throws SQLException {
        int n = dataSource.getMinimumIdle();
        List<Connection> held = new ArrayList<>(n);
        try {
            for (int i = 0; i < n; i++) held.add(dataSource.getConnection());
        } finally {
            for (Connection c : held) c.close();   // returned to the pool, now warm
        }
    }
}

ApplicationRunner beans run before the readiness state flips to accepting traffic, so the pod is not added to the Service until the warm-up completes.

Pace new connections across the fleet. With maxSurge: 25% on 20 replicas, five pods start together, each opening minimumIdle connections within a second. On managed databases with connection rate limits, or with IAM authentication that calls an external token service, stagger it: keep minimumIdle modest, let the pool grow on demand, and use maxSurge of one or two pods for services with large pools.

A rollout is clean when all four rows are handled: no errors from new pods’ first requests, no connection-creation spikes on the database, no aborted transactions from old pods, and a connection count that never exceeds the budget during the overlap.

Frequently Asked Questions

Related

• Spring Boot DataSource Configuration — the parent guide to HikariCP auto-configuration in Spring Boot.
• Tuning Spring Boot HikariCP for Microservices — pool sizing for the replicas being rolled.
• Implementing Graceful Connection Pool Shutdown in Express — the same sequence in Node.js.
• Sizing Connection Pools for Kubernetes Horizontal Pod Autoscaling — accounting for surge pods during rollouts.