Database Replication Lag: Causes, Monitoring, and Mitigation

Let's start with a pitfall you've most likely stepped into.

The user clicks "Save" in the console, the front end gets a 200, and immediately jumps to the list page—only to find the new data isn't there. Refresh once: nothing. Refresh again: there it is. You open the logs: the write succeeded, the replica query succeeded, no errors at all. QA files a bug: "List doesn't update after save, intermittent, ~30% probability."

This isn't a caching issue, and it isn't an uncommitted transaction. It's replication lag—the primary has already written the data, the replica hasn't synced yet, and your read request happened to be routed to the replica.

Read/write splitting is the highest-ROI scaling technique for databases, but it carries an unavoidable side effect: as long as replication is asynchronous, there is a time gap between the primary and the replica. This gap can be large or small—as small as milliseconds (imperceptible) or as large as tens of seconds (users are already cursing).

This article thoroughly explains replication lag: where it comes from, how to monitor it, how to mitigate it, and most importantly—how to achieve "the user can always read their own writes" at the application layer.

1. The "Side Effect" of Read/Write Splitting: Why You Can't Read What You Just Wrote

A classic read/write splitting architecture looks like this:

                    ┌─────────────────┐
        Write req ─▶│   Primary        │
                    │   reads+writes   │
                    └────────┬────────┘
                             │ streaming replication (WAL stream)
                  ┌──────────┴──────────┐
                  ▼                     ▼
         ┌────────────────┐    ┌────────────────┐
Read req ─▶│  Replica 1     │    │  Replica 2     │
         │   read-only     │    │   read-only    │
         └────────────────┘    └────────────────┘

The idea is simple: all writes go to the primary, and reads are distributed across multiple replicas. The primary bears the write load; the replicas bear the read load, scaling read throughput horizontally.

But there's an implicit assumption here: the replica's data is "up to date." The problem is that this assumption doesn't hold under asynchronous replication.

The complete "read-after-write" timeline looks like this:

t0  App sends INSERT to primary        ──▶ primary writes WAL
t1  Primary commits the transaction, returns success ──▶ app gets 200
t2  App sends SELECT to replica        ──▶ replica hasn't received this WAL yet
t3  Replica receives WAL, applies it   ──▶ data "appears"

The window between t1 and t3 is replication lag. If t2 falls within this window, the user can't read the data they just wrote. The larger the window, the higher the probability of hitting the pitfall.

Why does everyone use asynchronous instead of synchronous replication? Because synchronous replication requires the primary to wait for the replica to confirm receipt before returning success to the client, which doubles or more the latency of every write, and if the replica hiccups, the primary's writes stall entirely. Asynchronous replication trades consistency for low write latency and high availability—this trade-off is worthwhile in most scenarios, but the price is exactly the lag problem this article discusses.

Remember one conclusion: read/write splitting + asynchronous replication = replication lag is inevitable. What you can do is not eliminate it, but keep it within a range the business can accept, and make it imperceptible to users.

2. Where Does the Lag Come From: Breaking Down Asynchronous Replication Lag Sources

To mitigate lag, you first need to know where the time is spent. PostgreSQL's streaming replication link can be broken into three segments:

Primary              Network             Replica
┌─────┐  WAL gen     ┌─────┐  WAL xfer   ┌─────┐  WAL apply
│ WAL │ ──────────▶  │     │ ──────────▶ │     │ ──────────▶ data visible
│ buf │              │ net │             │recv │
└─────┘              └─────┘             │apply│
                                        └─────┘

Lag can pile up at any segment. Common sources fall into these categories:

1. Network transfer latency

The network between primary and replica is the single largest source of lag. Within a same-city data center it's usually under 1ms, across availability zones 2–5ms, and across regions it can be tens to hundreds of milliseconds. If the replica isn't in the same AZ as the primary, the network round trip alone is enough to cause pain. Worse is network jitter—occasional packet loss and retransmission can make latency spike instantly.

2. The replica's disk I/O

After the replica receives the WAL, it has to persist it to disk and apply it; this process depends on the replica's own disk performance. If the replica's IOPS or throughput can't keep up with the primary (for example, it uses cheaper disks, or heavy queries on the replica saturate the I/O), WAL piles up on the replica side and apply can never catch up. This is replica-side pile-up and has nothing to do with the primary.

3. Large transactions / long transactions

This is the number-one culprit of lag spikes. A DELETE FROM logs WHERE created_at < ... that deletes millions of rows causes the primary to generate an enormous WAL stream, and the replica has to spend the same effort applying it, during which it can't sync subsequent small transactions at all. Even more insidious are long transactions—if a SELECT ... FOR UPDATE or a report query that runs for several minutes runs on the replica, the snapshot it holds prevents the replica from applying WAL that conflicts with it, leaving the apply process waiting indefinitely.

4. Lock conflicts

There's lock contention between read queries on the replica and the apply process. PostgreSQL's hot standby defaults to max_standby_streaming_delay = 30s, meaning that when a query on the replica conflicts with apply, apply waits at most 30 seconds and then cancels that query. Conversely, if configured to wait indefinitely, apply is indefinitely blocked by queries—lag builds up.

5. Primary write bursts

A sudden wave of high-concurrency writes on the primary (batch imports, big promotions, compensation tasks) generates WAL far faster than the replica can apply; the replica is forced to chase, and lag naturally accumulates.

How do you pinpoint which segment? Look at several fields of pg_stat_replication (detailed in the next section), combined with I/O and network monitoring on both sides. A rule of thumb: if sent_lsn and replay_lsn differ greatly, the replica side is slow (apply or I/O); if write_lsn and sent_lsn differ greatly, the network is slow.

3. Monitoring: Seeing Lag Clearly With pg_stat_replication

You can't mitigate what you can't see. Replication lag must have monitoring and alerting, otherwise you're running naked.

PostgreSQL gives you a handy blade: pg_stat_replication. Note: this view must be queried on the primary; each row represents a replica that has connected.

SELECT
    application_name,
    client_addr,
    state,
    sync_state,
    sent_lsn,       -- LSN the primary has sent up to
    write_lsn,      -- LSN the replica has written to disk
    flush_lsn,      -- LSN the replica has flushed
    replay_lsn,     -- LSN the replica has applied (data visible)
    write_lag,      -- lag in the write phase
    flush_lag,      -- lag in the flush phase
    replay_lag      -- lag in the replay phase
FROM pg_stat_replication;

There are two ways to measure lag, and you need to look at both:

Byte difference (most accurate): The primary's current WAL position minus the replica's replay_lsn. This directly reflects "how many bytes behind the replica is":

SELECT
    application_name,
    pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS lag_bytes
FROM pg_stat_replication;

lag_bytes is the real gap at this moment, and monitoring it is the most stable. But byte difference has an issue—1MB behind might mean tens of seconds for a low-write workload, but only 100ms for a high-write workload. So you also need to look at the time difference.

Time difference (replay_lag): Theoretically tells you directly how many seconds behind you are. But it has a pitfall: it's inaccurate under low write load. The reason is that it measures "the apply-time difference of the last WAL record," and if the primary hasn't had new writes for a long time, this value becomes distorted or even shows NULL. So replay_lag is more useful in high-write scenarios; in low-write scenarios, rely on byte difference.

Monitoring in practice: Collect both and draw two charts:

• Replica lag byte difference (gauge, real-time)
• Replica lag time difference (gauge, real-time)

Alert thresholds should be defined per business. My experience:

Don't only look at the primary side. The replica itself must be monitored too; the key two are:

-- Run on the replica: see how far it has applied
SELECT pg_last_wal_receive_lsn(),   -- latest LSN received
       pg_last_wal_replay_lsn(),    -- latest LSN applied
       pg_last_xact_replay_timestamp(); -- apply time of the last transaction

now() - pg_last_xact_replay_timestamp() is exactly the lag in seconds from the replica's perspective. If this value keeps growing, it means the replica's apply can't keep up, and you need to investigate the replica's I/O, long transactions, and lock conflicts.

An easily overlooked monitoring blind spot: replication connection drops. If the replica can't reach the primary (network partition, primary restart), that row in pg_stat_replication simply disappears, and lag appears to be 0 (because there's no data). So monitoring must include a health check for "does the replication connection exist"—don't be fooled by "no data = no lag."

4. Mitigation Strategy 1: Read Critical Paths From the Primary

The simplest and most effective approach: data that was just written is read directly from the primary.

The logic is this: within a short window after a user writes (say 1–2 seconds), they will most likely immediately read that data (jump to the list, view details, echo it back). During this window the replica may not have synced yet, so don't gamble—read the primary directly. Once past this window, the replica has basically caught up, so route reads back to the replica and enjoy the benefits of read/write splitting.

Write occurs ─────[ critical window N seconds ]─────▶ time
              │
              ▼
       Reads within this window → primary
       Reads afterward          → replica

In implementation, there are two common granularities:

Session-level: After the user performs a write, mark the current session as "recently wrote," and all reads from this session within N seconds go to the primary. After N seconds the mark clears and reads revert to the replica. This approach is simple to implement but coarse-grained—all reads in the entire session are "implicated," even if they aren't reading the table that was just written to.

Object-level: Precise down to the table or row. Record "which tables were written within N seconds," and only requests reading those tables go to the primary; reads of other tables go to the replica as usual. Fine-grained, but you need to maintain a "hot tables" table, which is more complex.

How to choose the key parameter N? Look at your P99 replication lag. If monitoring shows the replica catches up within 1 second 99% of the time, set N to 1.5–2 seconds (leave some safety margin). Too large wastes primary compute; too small still risks hitting the pitfall.

The cost of this approach: the primary has to bear some reads. So it only suits "read-immediately-after-write" critical paths and shouldn't be overused. The principle is: only reads sensitive to "user-perceived consistency" go to the primary; reports, statistics, and list pagination that aren't sensitive continue to use the replica.

5. Mitigation Strategy 2: Read/Write Consistency Window

This strategy is a refined version of Strategy 1, with a more systematic approach.

The core idea: the application layer maintains a "write timestamp," and when reading, if the time since the last write is below a threshold, force the read to go to the primary.

Pseudocode:

class ReadWriteConsistency {
  private lastWriteAt: Map<SessionId, number> = new Map();
  private readonly WINDOW_MS = 2000; // consistency window

  afterWrite(sessionId: string) {
    this.lastWriteAt.set(sessionId, Date.now());
  }

  route(sessionId: string): 'primary' | 'replica' {
    const last = this.lastWriteAt.get(sessionId);
    if (!last) return 'replica';
    if (Date.now() - last < this.WINDOW_MS) return 'primary';
    return 'replica';
  }
}

Call afterWrite after each write operation completes, and call route before each read to decide which node to use.

This strategy's advantage over Strategy 1 is: it puts "when to return to the replica" entirely under the control of the time window, making behavior predictable and tunable. A larger window means stronger consistency (but more primary reads); a smaller window means better throughput (but a higher chance of hitting the pitfall). You can configure different windows per table or interface per business needs.

Advanced technique: let the window adjust dynamically based on the replica's actual lag. If monitoring shows the replica's current lag is 500ms, set the window to 1 second; if the replica's lag spikes to 3 seconds, automatically widen the window to 5 seconds. This requires pushing the replica lag metric to the application layer in real time (via the monitoring system's API or a shared cache); it's slightly heavier to implement but works best—neither wasting primary compute nor failing to tighten consistency when the replica hiccups.

Pitfall warning: Date.now() uses the application server's clock; if the application server and the database server's clocks are out of sync, the window judgment will be off. In a cluster, you must run NTP/chrony for clock synchronization—don't let clock drift break your window logic.

6. Mitigation Strategy 3: Synchronous Replication (synchronous_commit)

All the previous strategies "work around" the lag. If you want to truly eliminate lag (at least eliminate the possibility of "can't read it"), use synchronous replication.

PostgreSQL configures this via synchronous_commit and synchronous_standby_names:

-- postgresql.conf (primary)
synchronous_commit = on                  # on by default, but overridable per transaction
synchronous_standby_names = 'FIRST 1 (replica1)'

-- Fine-grained control at the transaction level (more flexible)
BEGIN;
SET LOCAL synchronous_commit = on;       -- this transaction requires synchronous replication
INSERT INTO orders ... ;
COMMIT;                                  -- the primary waits for replica1 to confirm receipt of WAL before returning success

FIRST 1 (replica1) means: when the primary commits a transaction, it must wait for at least 1 specified replica (replica1) to confirm receipt of the WAL before returning success to the client. This way, from the client's perspective, when success is returned the replica already has the data, and subsequent reads from the replica are guaranteed to see it.

What's the cost?

1. Write latency increases. Each commit adds one network round trip between primary and replica. Same data center may add 1–2ms; across data centers it may add tens of milliseconds. Under high-concurrency write scenarios, this latency significantly drags down write throughput.
2. Availability decreases. If replica1 goes down or the network breaks, the primary's writes are stuck (endlessly waiting for confirmation). This is the biggest risk of synchronous replication—trading availability for consistency, a classic CAP trade-off. In production practice, FIRST 1 (replica1, replica2) is often used to configure multiple candidate replicas so that if one fails there are others, reducing the risk.
3. The lag hasn't disappeared; it's been "front-loaded." Synchronous replication doesn't make replication faster; it makes "replication completion" part of the commit—you don't perceive the lag because you've already paid for it in time during the commit.

My advice: don't enable synchronous replication globally. Most businesses don't need it. The correct approach is fine-grained transaction-level control—only transactions that truly require strong consistency (payments, debits, critical state changes) do SET LOCAL synchronous_commit = on; other transactions stay asynchronous. This concentrates the cost where it's genuinely needed.

Payment transaction    ──▶ synchronous_commit = on   (slow, but strongly consistent)
Post transaction       ──▶ synchronous_commit = off  (fast, tolerates second-level lag)
Log writes             ──▶ synchronous_commit = off  (fast, doesn't matter)

7. Mitigation Strategy 4: Causal Sessions and Sticky Sessions

The previous strategies solve "all users can read it," but often what we really care about is weaker: the user can read their own writes (read-your-writes consistency). It's fine if others see it a few seconds late, but if the user can't see their own post right after publishing it, that breaks the experience.

This is a much weaker and much cheaper guarantee than "global strong consistency." The core technique is a causal session: reads within the same session are guaranteed to see the session's prior writes.

The simplest implementation is sticky session: once a session has performed a write, all subsequent reads from that session are routed to the primary (or to the primary until the replica catches up).

User session A:  write ─▶ read ─▶ read ─▶ read    (all primary, guaranteeing reads of own writes)
User session B:  read ─▶ read ─▶ read              (always replica, enjoying read/write splitting)

Sticky session implementation essentials:

1. Session identification. Use cookies, JWT, or session IDs to group the same user's requests into the same logical session.
2. State marking. After a session performs a write, mark it as "sticky" (for example, set a short-lived cookie rw_sticky=1; Max-Age=3).
3. Routing decision. Reads from requests with the sticky mark go to the primary; once the mark expires, revert to the replica.

Compared to Strategy 1 (read critical paths from the primary), sticky session's granularity is the "session," which better matches the semantics of "read your own writes"; compared to synchronous replication, it doesn't sacrifice the primary's write performance—it just keeps some sessions' reads on the primary.

A more refined version: sticky session can also be linked to replica lag—when the primary's write returns, it carries the current WAL position (pg_current_wal_lsn()); before the session's subsequent reads hit the replica, first check whether the replica's replay_lsn has surpassed this position. If it has, read the replica; if not, read the primary. This achieves "sticky just until the replica catches up," wasting not a bit of primary compute. The cost is one extra pg_last_wal_replay_lsn() query before each replica read, which you can trade off per scenario.

8. Application-Layer Approaches: Version Numbers and Timestamp Comparison

The last line of defense sits at the application layer; the idea is to trust no routing and let the data speak for itself.

Core idea: add a version number or update timestamp to critical records, and when reading, compare versions to judge whether the read data is "fresh enough."

Version number approach:

Add a version column to the table, incremented on each update. A write operation returns the new version number to the client; on subsequent reads the client carries this version number, and if the replica's returned data version is less than the version the client holds, the replica is too stale—trigger a fallback to re-read from the primary.

// After writing, the client holds version = 100
const res = await fetchFromReplica(`/api/order/${id}`);
if (res.version < 100) {
  // Replica too stale, fall back to primary
  return await fetchFromPrimary(`/api/order/${id}`);
}

Timestamp approach:

Use an updated_at field with the same logic. The client holds the timestamp from the write; if the read updated_at is earlier than this timestamp, fall back.

The benefit of this approach is that it closes the loop entirely at the application layer and doesn't depend on the database's replication mechanism, so it works for any database (MySQL, PG, MongoDB). The downside is that every read must carry version-judgment logic, and the client must "remember" the version it holds—typically implemented in conjunction with sessions or request context.

An industrial-grade combined approach: stack all the previous strategies to form multi-layer defense.

Request comes in
   │
   ▼
[sticky session?] ──yes──▶ read primary
   │no
   ▼
[within consistency window?] ──yes──▶ read primary
   │no
   ▼
[version comparison] ──replica too stale──▶ read primary
   │replica fresh enough
   ▼
read replica  ✅ enjoy read/write splitting

Each layer is a fallback; any request that makes it to the "read replica" step has been confirmed safe. This is a common practice at large companies—don't expect a single strategy to cover every case; instead, layer defense in depth.

9. Don't Chase Zero Lag: Choose a Strategy Based on Business Tolerance

After all these strategies, a splash of cold water at the end: don't chase zero lag.

Zero lag means fully synchronous replication + all reads from the primary, at the cost of doubled write latency, the primary becoming a single bottleneck, and reduced availability. This configuration is only worth it for systems like financial cores and payment clearing, where "consistency is life." Most businesses simply don't need it.

The right posture is to choose strategies by business tolerance level:

A few judgment dimensions:

• Can the user see this data? If they can't (internal stats, async tasks), lag is whatever; if they can, continue with the questions below.
• Will the user read it immediately? If yes (jump to list after writing), you need read-your-writes guarantees—use sticky session or a consistency window; if no (they'll look later), plain async is fine.
• Will it cause trouble if they can't read it? If yes (checking balance after a debit), strong consistency—synchronous replication or read from primary; if no (missing one comment), tolerate the lag.

A common over-engineering mistake: all reads go to the primary. I've seen quite a few teams, "for convenience and to avoid lag issues," simply give up on read/write splitting and route all reads and writes to the primary. The result is that the primary quickly becomes a bottleneck, and they then have to shard the database or adopt more complex solutions—when read/write splitting alone could have solved it. The value of read/write splitting is real; don't abandon it out of fear of lag—learn to coexist with lag instead.

Finally, replication lag isn't a bug; it's a physical property of distributed systems. Light is only so fast, and cross-data-center WAL transfer takes time. All your engineering effort isn't defying physics; it's reducing lag's negative impact to the point where users don't perceive it, at a cost the business can accept. That's the pragmatic goal.

Summary

1. Read/write splitting + asynchronous replication = lag is inevitable. Replication lag isn't a failure; it's the result of a trade-off—trading consistency for write performance and high availability. What you can do is control it and keep it imperceptible to users, not eliminate it.

2. Lag sources are segmented. Network transfer, replica disk I/O, large/long transactions, lock conflicts, and write bursts all stack lag. When diagnosing, look at the LSN differences in pg_stat_replication to determine whether the network or the replica is slow, and treat the symptom accordingly.

3. Monitoring must cover both ends. On the primary, query pg_stat_replication for each replica's byte and time differences; on the replica, query pg_last_wal_replay_timestamp() to see how far it has applied. Both must alert, and you must also monitor the existence of the replication connection itself—when the connection breaks, lag actually appears to be 0.

4. Mitigation strategies are layered and stacked. Read critical paths from the primary (simplest), read/write consistency window (tunable), synchronous replication (strongest but most expensive), causal session/sticky session (the highest-ROI "read-your-writes" guarantee), version comparison (application-layer fallback). Large-company practice is layered defense, not reliance on a single approach.

5. Choose a strategy based on business tolerance; don't chase zero lag. If finance needs strong consistency, use synchronous replication; if social can tolerate second-level lag, use sticky session + window; if logs and reports are fine, async is fine. Spend the cost where consistency is genuinely needed, rather than spreading strong consistency globally.

Key takeaway: Replication lag is not an enemy to be eliminated, but a metric to be managed. The real expert doesn't push lag to zero; they optimize system throughput and cost while keeping lag imperceptible to the business.