Diagnosing MongoDB Connection Storms After Failover

This guide is part of MongoDB Driver Connection Pools. A replica set election takes a few seconds. The application’s recovery from it often takes much longer, and the reason is in the drivers rather than the cluster: when a server becomes unreachable the driver closes every connection in that server’s pool, and when the topology settles every process reconnects to the new primary at once. On a large fleet the new primary receives thousands of connection attempts in a few seconds, each requiring a TLS handshake and authentication, while it is also replaying its own state:

14:22:41  MongoServerSelectionError: connection  to 10.2.0.8:27017 closed
14:22:41  connectionPoolCleared  address=10.2.0.8:27017
14:22:43  connectionPoolCleared  address=10.2.0.9:27017
14:22:47  MongoWaitQueueTimeoutError: Timed out while checking out a connection from connection pool
14:23:10  connections: { current: 1487, available: 13, totalCreated: 122903 }

The objective is to recognise the storm from the events, contain it with the settings that throttle reconnection, and make the application’s own behaviour — retries and deadlines — survive a few seconds of disruption without amplifying it.

Rapid incident diagnosis

Confirm it was an election. The cluster’s view shows when the primary changed:

rs.status().members.map(m => ({ name: m.name, state: m.stateStr, elected: m.electionTime }));
db.adminCommand({ replSetGetStatus: 1 }).electionCandidateMetrics;

Match the election time against the application’s error window. Errors that begin at the election and end within a few seconds are expected; errors that continue for minutes afterwards are the storm.

Count pool clears and creations per address. These two event streams distinguish a storm from ordinary saturation:

client.on("connectionPoolCleared", (ev) => log.warn({ address: ev.address }, "pool cleared"));
client.on("connectionCreated", (ev) => metrics.created.labels(ev.address).inc());
client.on("connectionCheckOutFailed", (ev) => metrics.failed.labels(ev.address, ev.reason).inc());

A storm looks like: a pool-cleared event on every address in quick succession, then connectionCreated rising steeply on the new primary’s address, with connectionCheckOutFailed reason timeout while the pool refills.

Look at the server’s acceptance rate. totalCreated climbing by thousands within a minute is the storm measured from the other side:

const before = db.adminCommand({ serverStatus: 1 }).connections.totalCreated;
sleep(10000);
const after = db.adminCommand({ serverStatus: 1 }).connections.totalCreated;
print("new connections per second:", (after - before) / 10);

Check whether the fleet is simply too large. If connections.available reaches zero during recovery, the storm is not just slow — it is being refused, and every refusal produces another retry. The fleet arithmetic from Tuning MongoDB maxPoolSize and waitQueueTimeoutMS is the first thing to re-check.

Connection storm after a replica set election At the election, pools are cleared on every member. Each of forty processes then refills its pool against the new primary, producing a spike of connection attempts that the new primary must authenticate while it is also taking over write traffic. Election primary changes Pools cleared every address process 1 refills process 2 refills process 40 refills New primary thousands of TLS and auth handshakes while taking over writes each handshake costs the new primary CPU at the moment it has least to spare maxConnecting throttles each pool, which is what keeps the sum survivable
An election clears every pool, and the fleet then reconnects to one server at once. The storm is the sum of every process's refill, not any single pool's behaviour.

Mathematical sizing or parameter formula

The size of the storm is the fleet’s pool total, arriving over the time it takes each pool to refill:

storm_connections = processes × maxPoolSize          (to the new primary)
refill_time       ≈ maxPoolSize ÷ maxConnecting × handshake_seconds
storm_rate        ≈ processes × maxConnecting ÷ handshake_seconds

With 40 processes, maxPoolSize = 11, maxConnecting = 2 and a 25 ms handshake: each pool refills in 11 ÷ 2 × 0.025 ≈ 0.14 seconds, and the fleet presents 40 × 2 ÷ 0.025 = 3,200 connection attempts per second at the peak. That rate, not the total, is what a recovering primary feels.

Two levers change it. Lowering maxPoolSize reduces the total and therefore the duration; lowering maxConnecting reduces the rate but lengthens recovery. The product is roughly fixed:

recovery_seconds ≈ maxPoolSize × handshake_seconds ÷ maxConnecting

Compare against the cluster’s acceptance capacity. A rough figure comes from measuring how quickly a single client can open connections when the cluster is healthy:

acceptance_rate ≈ measured connections per second before latency rises

If the fleet’s storm rate exceeds it, connection attempts queue in the server’s listen backlog and some fail, which produces retries and extends the storm. Sizing so that processes × maxConnecting ÷ handshake_seconds sits below the measured acceptance rate is the goal; with 3,200 attempts per second against a cluster that accepts 1,500 comfortably, either the fleet is too large, maxConnecting should stay at 2, or connections should be spread by connecting a proportion of processes to secondaries for reads.

Do not confuse the storm with steady state. A fleet that is comfortable in steady state can still be too large for recovery, because steady state opens connections gradually while recovery opens them all at once. Recovery is the sizing case that matters.

Recovery time versus connection attempt rate With maxConnecting 2 a pool of 11 refills in about 0.14 seconds at 3,200 fleet attempts per second, while maxConnecting 5 refills faster but presents 8,000 attempts per second to the recovering primary. Fleet of 40 processes, 25 ms handshake maxConnecting 2 3,200/s, refill 0.14 s maxConnecting 5 8,000/s, refill 0.055 s halving maxPoolSize halves both the total and the refill time which is the only change that improves both sides at once
Recovery time and peak attempt rate trade against each other. The product is set by pool size and handshake cost, so the real lever is the pool size.

Exact remediation & configuration

Keep the pool small and the establishment throttled. These are the settings that bound the storm:

export const client = new MongoClient(process.env.MONGODB_URI, {
  maxPoolSize: 11,         // sized from operations in flight, not left at 100
  minPoolSize: 2,          // small warm floor; refilled quickly after a clear
  maxConnecting: 2,        // default; raise only if recovery is measurably too slow
  maxIdleTimeMS: 60_000,
  serverSelectionTimeoutMS: 5_000,
  timeoutMS: 2_000,
  retryReads: true,
  retryWrites: true,
  appName: "orders-api",
});

Enable retryable reads and writes. They are on by default in current drivers, and they matter here more than anywhere: an operation interrupted by an election is retried once against the new primary, which turns a user-visible error into a small latency bump. Confirm they were not disabled in the connection string (retryWrites=false is a surprisingly common copy-paste).

Make application retries election-aware. The driver retries once; application-level retries on top should back off and cap, or they multiply the storm:

async function withRetry(fn, { attempts = 3, base = 100 } = {}) {
  let lastErr;
  for (let i = 0; i < attempts; i++) {
    try {
      return await fn();
    } catch (err) {
      lastErr = err;
      if (!isTransient(err)) throw err;
      const delay = base * 2 ** i + Math.random() * base;   // jitter prevents synchronised retries
      await new Promise((r) => setTimeout(r, delay));
    }
  }
  throw lastErr;
}

Jitter is not decoration: without it, every process in the fleet retries at the same instants, which recreates the storm after each wave.

Shorten server selection, but not too much. serverSelectionTimeoutMS of 30 seconds means an operation issued during an election waits a long time before failing; 5 seconds is enough to cover a normal election while failing fast when the cluster is genuinely unreachable. Setting it below the election time turns a survivable event into an outage.

Spread reads onto secondaries where correctness allows. Read-heavy services that use secondaryPreferred keep working through an election for reads, and their reconnect load is spread across members rather than concentrated on the new primary.

Stagger deployments away from elections. Rolling restarts during maintenance windows that also include a planned stepdown double the reconnect load. Trigger planned stepdowns when no deploy is in progress.

Testing failover before it happens

Elections are the one failure mode worth rehearsing, because the fix is mostly configuration that can be validated cheaply.

Force a stepdown in staging under load. With production-like pool settings and a load generator running:

// on the current primary
db.adminCommand({ replSetStepDown: 60, secondaryCatchUpPeriodSecs: 10 });

Record four numbers: how long any error is returned to clients, the peak of connections.totalCreated per second, the number of connectionPoolCleared events, and whether any operation exceeded its deadline. A healthy result is errors for a second or two, a totalCreated spike that matches the fleet arithmetic, and no operation exceeding timeoutMS by more than one retry.

Compare configurations in the same test. Run the stepdown twice, once with the sized pool and once with the default of 100, and record the same numbers. The difference is usually stark enough to settle any debate about whether pool sizing matters.

Verify retryable behaviour explicitly. Issue a long stream of single-document writes during the stepdown and count failures. With retryable writes enabled, the expected result is zero failed writes and a brief latency spike; failures mean either retryWrites=false or a write that is not retryable, such as one inside a multi-document transaction, which needs its own retry logic.

Check that monitoring saw it. The metrics and dashboards should show the event clearly after the fact: pool clears, a creation spike, and a return to steady state. If the dashboards cannot tell the story of a rehearsed failover, they will not explain a real one.

Rehearsal check Healthy result If it fails
Client-visible error window 1–3 seconds Shorten serverSelectionTimeoutMS; verify retryable operations
totalCreated spike matches processes × maxPoolSize Reduce maxPoolSize or process count
Operations exceeding deadline none beyond one retry Re-check timeoutMS and application retries
Failed writes zero Confirm retryWrites; handle transactions separately
Dashboards pool clear and refill visible Add the pool events from the monitoring guide
Client errors and connections during a rehearsed stepdown Errors appear for about two seconds around the election while connection creations spike to match the fleet's pool total, then both return to baseline within ten seconds. client errors connections created stepdown refill completes in seconds −10 s +30 s
A rehearsed stepdown produces a short error window and a creation spike that matches the fleet arithmetic. Anything longer is configuration, not the cluster.

Validation & verification

The storm is bounded. During the rehearsal, connection attempts per second must stay below the cluster’s comfortable acceptance rate, and connections.available must never reach zero:

db.adminCommand({ serverStatus: 1 }).connections    // watch available during recovery

Recovery completes in the expected time. Measured refill time should match maxPoolSize ÷ maxConnecting × handshake_seconds within a factor of two. A much longer refill points at authentication cost — LDAP or x.509 with an external certificate check — rather than the driver.

No amplification from retries. Count application-level retries during the event. They should be a small multiple of the operations in flight, not of the request rate; a much larger number means backoff or jitter is missing.

Steady state returns. Ten seconds after the election, connections.current should be back at the fleet’s normal total, and checkout failures should be zero. A total that stays elevated means pools were recreated without the old ones being closed — usually a client created per request somewhere in the code path.

Frequently Asked Questions

Why does the driver close healthy connections when a server is marked unknown?
Because the driver cannot know whether those connections are still usable, and using a stale connection to a server that has just changed role risks sending a write to a demoted primary. Clearing the pool is the safe choice; the generation counter ensures old connections are never reused afterwards.
Does a larger minPoolSize speed up recovery?
Slightly — the pool refills to the minimum eagerly — but it also increases the steady-state connection count across the fleet, and the eager refill is itself part of the storm. A small minimum is the better balance.
Do sharded clusters have the same problem?
Yes, per mongos. A router restart clears its pool on every client, and a fleet reconnects to the remaining routers. The mitigation is the same: modest pools, throttled establishment, retryable operations.
Should connections be pre-warmed after a failover?
There is no supported pre-warm mechanism, and adding one by issuing dummy operations from every process at once recreates the storm deliberately. Let maxConnecting do the pacing.