Using the AWS Advanced JDBC Wrapper for Fast Failover

This guide is part of AWS Aurora Connection Scaling. When an Aurora writer fails, the DB cluster promotes a replica within about thirty seconds, but a plain JDBC pool does not find out until its connections fail and DNS has been re-resolved. With a default JVM DNS cache and a pool that keeps connections for half an hour, an application can keep sending writes to a demoted instance for minutes:

org.postgresql.util.PSQLException: ERROR: cannot execute UPDATE in a read-only transaction
com.mysql.cj.jdbc.exceptions.SQLError: The MySQL server is running with the --read-only option
java.sql.SQLException: HikariPool-1 - Connection is not available, request timed out after 30000ms.

The AWS Advanced JDBC Wrapper is a driver-level wrapper that knows the DB cluster’s topology, watches the connection with a background monitor, and switches to the new writer as soon as it detects the change — without waiting for DNS. The objective is a configuration where failover costs seconds and a handful of retryable errors rather than minutes of write failures.

Rapid incident diagnosis

Establish how long recovery actually took. Compare the promotion time in the RDS events with the moment errors stopped:

aws rds describe-events --source-type db-cluster --source-identifier orders-aurora \
  --duration 120 --query 'Events[?contains(Message, `failover`)].[Date,Message]' --output table

Errors that persist for more than a minute after promotion point at DNS caching or a pool that will not recycle.

Check the JVM’s DNS cache. The default with a security manager is to cache forever; without one, 30 seconds — still long enough to matter:

jcmd $(pgrep -f orders-api) VM.system_properties | grep -i "networkaddress\|inetaddr"

Check what the pool holds. Connections older than the failover that are still in the pool are connections to the old writer:

SELECT pg_is_in_recovery();                       -- true on a reader: a write here fails
SELECT now() - backend_start AS age, state FROM pg_stat_activity
WHERE application_name = 'orders-api' ORDER BY age DESC LIMIT 5;

Identify whether the wrapper is in use at all. The JDBC URL says so:

grep -rn "jdbc:aws-wrapper" --include=*.yml --include=*.properties --include=*.java . | head

A URL of jdbc:postgresql://…cluster-…rds.amazonaws.com is the plain driver; jdbc:aws-wrapper:postgresql://… is the wrapper.

Failover detection with and without the wrapper Without the wrapper, recovery waits for query failures, the JVM DNS cache and pool recycling. With the wrapper, topology monitoring detects the new writer within seconds. Time from promotion to recovered writes plain driver query failures + DNS TTL + pool lifetime AWS wrapper seconds: topology query detects the new writer the DB cluster promotes in about the same time in both cases what differs is how long the application keeps using the old endpoint
A plain driver learns about a failover from a failing query and a DNS re-resolution. The wrapper learns from the DB cluster's own topology table.

Mathematical sizing or parameter formula

Recovery time decomposes into three terms:

recovery = promotion_time + detection_time + reconnect_time

Promotion is Aurora’s, typically under 30 seconds. Detection and reconnection are the application’s, and the wrapper addresses both:

detection_plain   ≈ query_timeout + dns_ttl + pool_lifetime_remaining
detection_wrapper ≈ failureDetectionTime + failureDetectionInterval × failureDetectionCount

With the host-monitoring plugin’s defaults — a 30-second grace period, then probes every 5 seconds with 3 failures required — detection is about 45 seconds in the worst case. Tightening them for latency-sensitive services:

failureDetectionTime = 10_000 ms, interval = 1_000 ms, count = 3  → detection ≈ 13 s

The cost of tighter settings is a higher chance of declaring a healthy-but-slow node failed, which triggers an unnecessary failover attempt on the client side.

Reconnect time is bounded by the failover plugin’s own timeout:

failoverTimeoutMs ≥ promotion_time + topology_refresh + connect_time

A value of 60,000 covers a normal promotion with margin; below about 30,000 the plugin gives up before Aurora finishes.

Connection budget during failover. Every connection to the demoted writer is discarded and reopened, so the new writer receives the fleet’s entire writer pool at once:

reconnect_burst = replicas × writer_pool_size

Twelve replicas with a writer pool of 20 is 240 simultaneous connection attempts — which the instance must accept while it is also taking over write traffic. Keeping the writer pool sized from in-flight work rather than generously is a failover consideration as much as a steady-state one.

Detection time against monitoring settings Default host monitoring settings detect a failure in about forty five seconds, while tightened settings detect it in about thirteen, with a higher risk of declaring a slow node failed. Worst-case detection time defaults (30 s + 3 × 5 s) 45 s tightened (10 s + 3 × 1 s) 13 s tighter settings mean more probes and more false positives on a loaded node measure the node's probe latency before choosing
Tighter monitoring settings cut detection time but raise the chance of a false positive on a slow node. The defaults favour stability; latency-sensitive services trade some of it.

Exact remediation & configuration

Switch the driver to the wrapper. It sits in front of the underlying PostgreSQL or MySQL driver, so both remain on the classpath:

<dependency>
  <groupId>software.amazon.jdbc</groupId>
  <artifactId>aws-advanced-jdbc-wrapper</artifactId>
  <version>${aws.jdbc.wrapper.version}</version>
</dependency>
spring:
  datasource:
    driver-class-name: software.amazon.jdbc.Driver
    url: jdbc:aws-wrapper:postgresql://orders-aurora.cluster-abc123.eu-west-1.rds.amazonaws.com:5432/orders
    username: orders_app
    hikari:
      maximum-pool-size: 16
      connection-timeout: 3000
      max-lifetime: 1500000
      keepalive-time: 120000
      exception-override-class-name: software.amazon.jdbc.util.HikariCPSQLException
      data-source-properties:
        wrapperPlugins: auroraConnectionTracker,failover2,efm2
        failoverMode: strict-writer
        failoverTimeoutMs: 60000
        clusterTopologyRefreshRateMs: 30000
        failureDetectionTime: 10000
        failureDetectionInterval: 1000
        failureDetectionCount: 3

Three of these deserve explanation:

  • exception-override-class-name tells HikariCP to treat the wrapper’s failover exceptions as recoverable, so the pool does not evict every connection when one reports a failover.
  • auroraConnectionTracker tracks connections per instance so that, when an instance is demoted, its connections are closed rather than being handed out for writes.
  • efm2 is the enhanced host-monitoring plugin: a background thread probes the connection, so a hung socket is detected without waiting for a TCP timeout.

Handle the failover exception in application code. The wrapper signals a completed failover with a specific SQL state so the application can retry rather than fail:

try (Connection c = dataSource.getConnection()) {
    return work.apply(c);
} catch (SQLException e) {
    if ("08S02".equals(e.getSQLState())) {       // FailoverSuccessSQLException: reconnected to the new writer
        // the transaction was rolled back; retry it on a fresh connection
        try (Connection c2 = dataSource.getConnection()) {
            return work.apply(c2);
        }
    }
    throw e;
}

State 08007 indicates a failover that interrupted an active transaction whose outcome is unknown — those must not be retried blindly unless the operation is idempotent.

Keep reader traffic separate. A second data source on the reader endpoint, with the same plugins and a shorter maxLifetime, spreads reads and rebalances after scaling events, as described in Routing Reads to Aurora Reader Endpoints with Connection Pools.

Do not disable the JVM DNS cache tuning. The wrapper reduces the dependency on DNS but does not remove it — the initial connection still resolves the endpoint:

JAVA_TOOL_OPTIONS="-Dsun.net.inetaddr.ttl=5 -Dsun.net.inetaddr.negative.ttl=1"

Where the wrapper is not an option — a non-JVM stack, or a driver it does not support — the fallback is a short maxLifetime, a short DNS TTL, validation on borrow and application-level retries. Recovery then takes tens of seconds rather than seconds, which is acceptable for many services.

Rehearsing and measuring failover

The settings above are hypotheses until a failover has been triggered deliberately.

aws rds failover-db-cluster --db-cluster-identifier orders-aurora \
  --target-db-instance-identifier orders-aurora-instance-2

Measure four numbers. Run a steady write workload during the rehearsal and record: the time from the promotion event to the last write error, the count of failed writes, the peak connection attempts on the new writer, and whether any write was applied twice.

# during the rehearsal
while true; do
  date -u +%T
  psql -h orders-aurora.cluster-abc123.eu-west-1.rds.amazonaws.com -U ops -Atc \
    "SELECT pg_is_in_recovery(), count(*) FROM pg_stat_activity WHERE application_name='orders-api'"
  sleep 1
done

Expected shape with the wrapper. Errors for a few seconds, all of them carrying 08S02 or a driver connection error; connection attempts on the new writer equal to the fleet’s writer pool; and no duplicate writes, because retries only happen for transactions the wrapper reports as rolled back.

Compare against the plain driver by running the same rehearsal on a canary without the wrapper. The difference — usually a minute or more — is the number that justifies the dependency.

Rehearse the reader path too. A reader instance being replaced should produce no write errors at all and only a brief interruption for reads, provided the reader pool recycles.

Rehearsal check With the wrapper Plain driver
Write error window seconds tens of seconds to minutes
Errors identifiable 08S02 / 08007 SQL states generic connection errors
Stale writer connections closed by the tracker held until maxLifetime
Reconnect burst writer pool per replica the same
Duplicate writes none, when retrying only on 08S02 possible, if retries are naive
What to measure during a failover rehearsal Record the error window, the number of failed writes, the peak connection attempts against the new writer, and whether any write was applied twice. Error window promotion to last error Failed writes count and SQL state Reconnect burst attempts on the new writer Duplicates any write applied twice Four numbers, one rehearsal the last one is the correctness check: retries must only follow a rolled-back transaction
The rehearsal measures four numbers. The error window is the headline, but the reconnect burst is what the new writer has to survive.

Validation & verification

The wrapper is actually loaded. At startup, log the driver class and the URL; software.amazon.jdbc.Driver with an aws-wrapper URL is the confirmation.

Topology is being refreshed. Enable the wrapper’s logging briefly and confirm topology queries appear at the configured interval:

logging.level.software.amazon.jdbc=FINE

Failover recovers in seconds. The rehearsal’s error window should match the detection settings within a factor of two.

The new writer accepts the burst. During the rehearsal, DatabaseConnections on the promoted instance should rise to the fleet’s writer pool total without connection errors.

No writes to a reader. After recovery, search the logs for read-only transaction errors; any occurrence after the error window means connections to the demoted instance were still in use.

Frequently Asked Questions

Does the wrapper replace RDS Proxy?
No — they solve different problems. The wrapper makes a client’s failover handling fast; RDS Proxy reduces connection counts and holds client connections open across a failover. Using both is common: the wrapper points at the proxy endpoint, and each handles its own layer.
What is the overhead of topology monitoring?
One lightweight query per refresh interval per connection pool, plus the host-monitoring probes. At a 30-second refresh and 1-second probes, it is negligible compared with application traffic.
Does it work with MySQL-compatible Aurora?
Yes. The wrapper supports both engines, wrapping the MySQL Connector/J or the PostgreSQL driver, with the same plugin set.
Is failoverMode: strict-writer always right?
It is right for a writer data source: it insists on connecting to a writer after failover. A reader data source should use reader-or-writer so it can keep serving reads even if only the writer is available.