Propagating Request Deadlines into Connection Acquisition

This guide is part of Connection Acquisition Timeout Strategies. A fixed acquisition timeout assumes every caller has the same patience. Real systems do not work that way: an interactive request may have 300 ms left by the time it reaches the data layer, while a batch job can wait ten seconds. When the pool uses one static number, the saturated case produces the worst of both — the interactive caller waits far past the point where its own client gave up, and the work it eventually performs is thrown away:

gateway  504 after 1,000 ms      (client gone)
service  acquired a connection at 2,400 ms, query completed at 2,460 ms, response discarded
pool     still 40 waiters, all of them already past their callers' deadlines

Wasted work under saturation is what turns a brief overload into a long one. The objective is to carry the caller’s deadline down into acquisition and query execution so that work begins only if it can still be delivered, and the pool queue drains instead of growing.

Rapid incident diagnosis

Compare the time spent waiting with the caller’s timeout. If p99 acquisition wait is close to or above the upstream timeout, the service is performing work nobody is waiting for:

histogram_quantile(0.99, sum by (le) (rate(hikaricp_connections_acquire_seconds_bucket[5m])))
# compare with the gateway's configured request timeout

Count responses that arrive after the client disconnected. Most servers can report it; in Express, the close event on the request fires when the client goes away:

app.use((req, res, next) => {
  req.on("close", () => { if (!res.writableEnded) metrics.abandoned.inc(); });
  next();
});

A rising abandoned count during saturation is direct evidence of wasted work.

Look for a queue that never drains. A pool where the number of waiters stays constant while requests time out is serving a queue of already-dead requests: each new arrival joins the back, each timeout leaves from the front, and the connection released in between goes to a request that is also too late.

Check whether the deadline exists at all. Many services have no notion of remaining time: the HTTP handler calls the data layer, which uses the pool’s static timeout. Grep for the pool’s acquisition setting and see whether any call site can override it:

grep -rn "connectionTimeout\|pool_timeout\|acquire(timeout\|WithTimeout\|CommandTimeout" --include=*.java --include=*.py --include=*.go --include=*.ts .

Mathematical sizing or parameter formula

The budget is what remains of the caller’s deadline after the time already spent:

remaining        = deadline − elapsed
acquire_budget   = remaining × acquire_share − safety_margin
query_budget     = remaining − acquire_time_used − response_margin

A sensible acquire_share is 0.3 to 0.5: if acquisition consumes more than half the remaining budget, the query that follows is unlikely to finish in time anyway. With a 900 ms remaining budget, a 40 ms safety margin and a 0.4 share: acquire_budget = 320 ms, and whatever time is actually spent waiting is subtracted before the query timeout is set.

Queue behaviour improves because arrivals leave earlier. Under saturation, a queue of waiters with per-request budgets drains at the rate the shortest budgets expire, so the effective queue length is bounded by:

effective_waiters ≈ arrival_rate × mean_acquire_budget

At 400 requests per second and a mean budget of 320 ms, about 128 waiters can exist at once, each of which will still be useful if served. With a static three-second timeout, the same arrival rate supports 1,200 waiters, nearly all of which are past their deadlines by the time they are served.

Deadlines must shrink across hops. Each service in a chain should pass a slightly smaller deadline than it received, so the innermost call cannot outlive the outermost:

downstream_deadline = received_deadline − network_margin − local_overhead

With 10 ms of network margin per hop and three hops, the database layer of the innermost service works with about 30 ms less than the client allowed — a small price for never exceeding it.

Exact remediation & configuration

Go has the cleanest model, because context.Context carries the deadline and database/sql honours it in both acquisition and execution:

func (s *Server) handleOrder(w http.ResponseWriter, r *http.Request) {
    // The server sets a deadline for the whole request; derive the database budget from it
    ctx := r.Context()
    if dl, ok := ctx.Deadline(); ok {
        remaining := time.Until(dl)
        budget := time.Duration(float64(remaining) * 0.4)
        var cancel context.CancelFunc
        ctx, cancel = context.WithTimeout(ctx, budget)
        defer cancel()
    }

    // QueryContext waits for a connection only until ctx expires, then returns immediately
    row := s.db.QueryRowContext(ctx, "SELECT status FROM orders WHERE id = $1", chi.URLParam(r, "id"))
    // ...
}

The important property is that database/sql returns context.DeadlineExceeded from the acquisition wait, not only from the query, so an expired request never occupies a connection.

Python with psycopg_pool, where the timeout is a parameter of the borrow:

import time
from psycopg_pool import PoolTimeout

def with_budget(pool, deadline_monotonic, share=0.4, margin=0.04):
    remaining = deadline_monotonic - time.monotonic()
    if remaining <= margin:
        raise DeadlineExceeded()
    budget = max(0.01, remaining * share - margin)
    return pool.connection(timeout=budget)

with with_budget(pool, request_deadline) as conn:
    conn.execute("SET LOCAL statement_timeout = %s", (int((request_deadline - time.monotonic()) * 1000),))
    rows = conn.execute("SELECT id, total FROM invoices WHERE customer_id = %s", (cid,)).fetchall()

Setting statement_timeout from the same budget means the database also stops work that can no longer be delivered.

Java with HikariCP, which has no per-call acquisition timeout, so the deadline is enforced around it:

public <T> T withBudget(Duration remaining, SqlFunction<T> work) throws SQLException {
    long budgetMs = Math.max(10, (long) (remaining.toMillis() * 0.4) - 40);
    // A dedicated DataSource per budget class is the supported way to vary the wait
    DataSource ds = budgetMs < 500 ? fastDataSource : batchDataSource;
    long start = System.nanoTime();
    try (Connection c = ds.getConnection()) {
        long usedMs = (System.nanoTime() - start) / 1_000_000;
        long queryMs = Math.max(10, remaining.toMillis() - usedMs - 40);
        try (Statement s = c.createStatement()) {
            s.execute("SET LOCAL statement_timeout = " + queryMs);
        }
        return work.apply(c);
    }
}

Two data sources — one with a short connectionTimeout for interactive traffic, one with a longer one for batch work — is the practical way to express two budget classes when the pool cannot vary the wait per call.

Node.js, where the deadline is carried explicitly and races the acquisition:

async function withBudget(pool, deadlineMs, fn) {
  const remaining = deadlineMs - Date.now();
  if (remaining <= 40) throw Object.assign(new Error("deadline exceeded"), { status: 504 });
  const budget = Math.max(10, remaining * 0.4 - 40);

  const acquiring = pool.connect();
  const timer = new Promise((_, rej) => setTimeout(() => rej(new PoolBusy()), budget));
  let client;
  try {
    client = await Promise.race([acquiring, timer]);
  } catch (e) {
    acquiring.then((late) => late.release()).catch(() => {});   // never leak a late arrival
    throw e;
  }
  try {
    await client.query(`SET LOCAL statement_timeout = ${Math.max(10, deadlineMs - Date.now() - 40)}`);
    return await fn(client);
  } finally {
    client.release();
  }
}

Carry the deadline between services. Propagate a header — grpc-timeout in gRPC, or a X-Request-Deadline epoch millisecond value over HTTP — and have each service derive its local deadline from it rather than from its own default.

Rolling it out without breaking batch work

Deadline propagation fails badly when applied uniformly: a nightly job with no deadline inherits a short default and starts failing.

Classify traffic first. Interactive requests have a client waiting and a deadline worth honouring. Background jobs, migrations and exports have no client and need generous or unlimited budgets. Queue consumers sit in between: they have no user, but a message that takes too long should be retried rather than blocking the consumer.

BUDGETS = {
    "interactive": {"share": 0.4, "max": 1.0},
    "queue":       {"share": 0.5, "max": 10.0},
    "batch":       {"share": 1.0, "max": 120.0},
}

Default to the old behaviour where no deadline exists. Code paths that do not set a deadline should use the pool’s static timeout, not a zero budget. A missing deadline is a gap in instrumentation, not an instruction to fail fast.

Roll out on read paths first. Reads are usually idempotent, so a request abandoned at the acquisition stage costs nothing. Write paths need the same care as any retry: an operation that may already have reached the database must be idempotent before the caller retries it.

Watch the error mix, not just the error rate. A successful rollout converts slow successes into fast failures; the total error count may rise while user-visible latency improves and wasted work falls. Track delivered responses per second — responses whose client was still connected — as the outcome that matters.

Validation & verification

Expired requests never acquire. In staging, cap the pool at one connection, hold it, and send requests with a 200 ms deadline. Every one must fail in about 200 ms, and the pool’s acquisition counter must not record a successful checkout for them after they expired.

Delivered throughput improves under saturation. Run a saturation test with and without propagation, measuring responses delivered to still-connected clients per second. The propagated run should deliver at least as many, with far lower p99 latency and far fewer abandoned responses.

Downstream deadlines shrink. Log the deadline at each hop and confirm it decreases monotonically along the chain:

gateway  deadline=+1000ms
api      deadline=+980ms   acquire_budget=376ms
worker   deadline=+960ms   acquire_budget=368ms

Batch work is unaffected. Run the nightly job with propagation enabled and confirm it completes with the same runtime as before, which proves that the batch class is not inheriting an interactive budget.

Frequently Asked Questions

Related

• Connection Acquisition Timeout Strategies — the parent guide to deadline ordering and fail-fast behaviour.
• Tuning Connection Acquisition Timeout Under Burst Load — choosing the static timeout for paths without a deadline.
• Understanding Connection Acquisition Timeouts in Go — how contexts cancel a waiting borrow.
• Implementing Circuit Breakers for Connection Acquisition Failures — shedding load when the database is the problem.