Database Connection Pool Admission and Fairness: Global Permits, Deadlines, and Draining

Ten application Pods each set maxOpen=50. After a deployment scales them out, they can theoretically open 500 connections to PostgreSQL. A batch-processing tenant then occupies every pool slot, leaving online requests to wait indefinitely inside the application. When the database finally rejects connections, the retry storm keeps creating sockets and may exhaust even the administrator connections used for diagnostics. A local pool size is not a global capacity contract.

PostgreSQL Connection Pooling and PgBouncer introduces the pooling modes, while Database Connection Architectures compares where proxies can be placed. This article does not repeat a parameter checklist. Instead, it answers who is allowed to queue, when they receive a database execution slot, and when that slot is returned after cancellation or failover.

Derive Permits from the Database Budget First

Publish dbAdmissionPolicy.v1 and bind it to the database cluster/role, policy generation, hard database connection limit, reserved admin/replication capacity, PgBouncer pool mode, global serverSessionPermits, each workload class's executionPermits, tenant weights, maximum queue size, queueing deadline, and transaction time limit. These two quota types must not be mixed: a session permit constrains real backend connections and remains held across idle periods, while a request ticket governs only fair queueing and transactions that are currently executing.

schema: dbAdmissionPolicy.v1
database: orders-primary/write
generation: 27
serverSessionPermits: 180
adminReserve: 12
executionClasses:
  interactive: {permits: 130, maxQueue: 800}
  batch: {permits: 50, maxQueue: 100}
fairness: weighted-deficit-v2

PostgreSQL max_connections includes the connection budget that the server can accept and reserves slots for superusers and similar roles. It is not a target that applications should consume in full. PgBouncer parameters such as max_client_conn and default_pool_size distinguish client connections from backend server connections. Go database/sql's SetMaxOpenConns limits only one DB handle; none of these local controls automatically coordinate multiple processes.

A request ticket follows QUEUED -> EXECUTION_GRANTED -> CHECKING_OUT -> IN_TRANSACTION -> REQUEST_RELEASED. A queue timeout produces QUEUE_EXPIRED, and when cancellation races with a grant, the result can only be CANCELLED or EXECUTION_GRANTED. A physical connection follows a separate path: SESSION_RESERVED -> CONNECTING -> OPEN_IDLE | OPEN_BUSY -> CLOSING -> SESSION_RELEASED; an indeterminate protocol outcome becomes SESSION_UNKNOWN. A ticket is bound to a tenant, class, deadline, and boot ID. A session permit is additionally bound to the proxy/process identity, backend PID, backend epoch, and connection generation.

Grant Is the Sole Capacity Admission Point

The sole allocation entry point, grantDbAdmission, handles both actions against the same canonical ledger. It first changes a request ticket from QUEUED to EXECUTION_GRANTED according to class/tenant fairness. If checkout requires a new physical connection, it must also atomically consume one global session permit and write a SESSION_RESERVED receipt before connecting is allowed. Reusing an existing idle connection reuses its original session permit and does not consume another. No process or PgBouncer instance may bypass this session ledger, and duplicate requests only read the original receipt.

A request ticket is released through CAS after its transaction commits or rolls back, or once it can be proved that the transaction never started. When a connection returns to the idle pool, it still holds its session permit. A session permit can be released only after the physical session closes or the server confirms that its backend PID no longer exists; a duplicate release does not increase the balance. If a process crashes and session termination cannot be confirmed, the session becomes SESSION_UNKNOWN and continues consuming capacity. A failover advances the backend epoch and drains every old session; the combined session permits across the old and new epochs remain subject to the hard budget. Cross-Service Deadlines manage request budgets, but a cancellation signal alone does not prove that the SQL or session has stopped.

Fair scheduling can use weighted deficit round robin: each active tenant receives capacity according to its weight, unused capacity accumulates only up to a limit, and each tenant remains subject to an inflight limit. interactive and batch use separate guaranteed pools while allowing controlled borrowing. Borrowed capacity has a TTL, and new borrowing stops immediately when queues for core traffic rise. Pure FIFO suffers head-of-line blocking from a tenant running a long transaction, while strict priority can starve background maintenance. The policy therefore sets both a maximum wait and an aging limit. Load Balancing Strategies determine which application receives a request; the admission ledger determines whether that request may enter the database.

A queue is not an infinite buffer. When maxQueue is reached, or the remaining deadline is shorter than the estimated checkout + execution budget, reject the request immediately and return an explicit retry classification. Automatic retries must retain the same ticket identity. The load-shedding principles from Multi-Agent Overload Routing can be reused, but a database permit must additionally be bound to a real session and proof that its transaction has terminated. PostgreSQL Performance Tuning can optimize an individual query, but it cannot fix unbounded admission.

Stress the Test Environment with Leaks and Starvation

The failure suite injects a crash after a request grant, loss of the connect ACK after a session reserve, cancellation concurrent with a grant, client timeout after SQL has executed, an idle connection leak, a PgBouncer restart, database failover, a tenant flood, policy generation rollover, and double reclamation by both types of reapers. It separately asserts conservation of request tickets and session permits, verifies that OPEN + CONNECTING + SESSION_UNKNOWN never exceeds the hard backend budget, keeps the admin reserve available at all times, and ensures that an idle session does not disappear from the ledger when a request is released.

The release gate requires zero discrepancies in both ledgers across 100,000 rounds of concurrent testing, zero real server sessions above the safety threshold, zero double releases and unbounded starvation, interactive queueing p99 within budget, and alerts for session leaks within two lease cycles. Rollout begins by shadowing both types of decisions, then takes over 5% of read-only traffic. To roll back, stop issuing tickets and session permits under the new generation, drain and reconcile inflight sessions, and then switch the alias. The old scheduler must also use the global session ledger and cannot revert to independently expanding the connection pool in every Pod.