My Profile Photo

Chang Min Park


Senior Software Engineer at Yahoo!



Replication Lag Filed as an Android Bug

Send, pull to refresh, message gone

The repro is cursed in the best way: send a message, watch “Sent,” pull to refresh, and the thread looks like the send never happened. Ten seconds later it is there. Logs on the phone show a 200. QA cannot reproduce on a quiet staging stack. The ticket lands on Android anyway — Compose list lost the item — because that is the surface the reporter can see.

What actually happened is older than Jetpack: the write hit the primary, the refresh read a replica that had not applied the commit yet. Replication lag wore an Android costume.

Why it looks like a client bug

Observation Android hypothesis Systems reality
Intermittent Race in list state Lag window only sometimes hit
“Fixed” by waiting Lifecycle / recomposition Replica caught up
Staging clean “Works in debug” Staging lag ≈ 0
200 + empty Parser / mapper bug Valid stale payload

No exception, no 5xx, no Compose crash. Optimistic UI may even show the outbound message until a refetch replaces local state with a stale server snapshot — at which point the UI “deletes” something the primary already has. That feels exactly like a client state bug.

sequenceDiagram
  participant App
  participant Primary
  participant Replica
  App->>Primary: POST send
  Primary-->>App: 200 + id
  App->>Replica: GET thread
  Note over Replica: WAL not applied yet
  Replica-->>App: thread without message
  App->>App: list looks wrong

Figure 1. Successful write, then a freshness-blind read.

Sticky routing is not enough

Load-balancer stickiness pins a user to an app instance, not to “always read the primary after my write.” Two requests on the same phone can still fan out to different data-access paths. What you want is read-your-writes in the data layer:

  • After a write, mark the session (cookie, header, or server-side flag) for a short freshness window and route those reads to primary
  • Or return a version / LSN-style token on write; subsequent reads carry X-Min-Read-… and wait or redirect until the replica is past that point
  • Size the window against p99 lag under load, not the peaceful median — batch jobs and failovers stretch the gap

I will not quote internal Yahoo replica metrics. Qualitatively: lag that is fine for global inbox browse is not fine for “I just sent this.” Product paths need different freshness classes.

Read class Example Freshness need
Browse Cold inbox open, search Replica OK; seconds of lag often fine
Read-your-writes Thread after send / archive Primary or token-gated until caught up
Cross-user causal Someone else replies in shared thread Harder; often accept brief lag or push-tap

Sticky sessions at the HTTP edge do not encode that table. The mail API (or a gateway aware of recent writes) has to.

Optimistic UI has to survive the stale refetch

On Android, the fix is shared between client and server:

  1. Keep local causality — do not let a GET that is older than the pending/acked send tombstone the optimistic row
  2. Prefer the write response body (new message id, thread version) to update Room before any list refetch
  3. Delay or condition refetch until a freshness token says the read is safe
  4. Instrument “missing my send” with server freshness metadata when available — so the next ticket is not a week of UI archaeology
// why: stale GET must not clobber a newer local send
fun mergeThread(local: Thread, remote: Thread): Thread {
    if (local.version >= remote.version) return local
    return remote
}

How to stop misrouting the bug

When a report is intermittent, timing-sensitive, and staging-clean, ask for primary vs replica before rewriting adapters. Reproduce with artificial lag if you can — inject delay on the read path in a dogfood build and watch optimistic rows disappear after refresh. Add a debug breadcrumb for which freshness class served the read (primary, replica, cache). Teach triage that “Android” is often the messenger for a consistency contract the backend changed when replicas were added.

Compose migrations make the misfile more likely: the list did change recently, so it becomes the suspect. Keep a short checklist in the bug template — write status, immediate GET body, whether waiting helps — so mobile and server engineers start from the same hypothesis set.

Rule of thumb - if waiting fixes it and logs are green, suspect replication lag before recomposition.

Wrap-up

Send-then-refresh missing mail is a classic read-your-writes failure mislabeled as a Compose bug. Sticky app routing will not save you; session-aware primary reads or version tokens will. On the client, optimistic state and versioned merges must outrank a freshness-blind refetch. Fix the consistency contract once, and a whole class of “Android” tickets evaporates.

References