One Deleted Email Broke Our Live Sync for 11 Days

From 23 July to 4 August 2026, our instant Gmail sync failed 686 times in a row. A separate check running every two minutes still collected every message, so nobody lost mail. The visible effect was smaller and easy to miss: some emails arrived in as long as two minutes instead of about six seconds.

Reconstruction from production logs, not a screenshot of the original monitor. The dates and failure count are exact; the latency samples were taken on 5 August 2026.

What actually broke

The fast route stopped. Google kept notifying us that connected Gmail inboxes had changed, but every attempt to process those notifications ended at the same deleted message. That happened 686 times over eleven days.

Mail kept arriving through a second route. Every two minutes, a backup job asked each connected mailbox for anything new. It delivered all of it, and triage still ran. A person watching an inbox would have seen normal messages and labels, just with occasional delays approaching two minutes rather than a few seconds.

Our monitoring saw the same healthy outcome. It checked whether mail arrived, not which route delivered it. Nothing in the dashboards separated a successful push notification from a rescue by the backup job. The system was degraded for eleven days while its top-line health signal stayed green.

How mail reaches you

Gmail gives us a push mechanism for the fast route. We subscribe to changes on a connected mailbox. When something changes, Google sends a small notification and we fetch the new history records. In our five timing samples after the repair, messages using this route appeared within single-digit seconds.

The backup is intentionally boring. Every two minutes it checks for changes whether or not Google sent a notification. Google recommends this kind of periodic polling because push notifications “might be delayed or dropped” in rare cases. We did not invent the fallback after this incident; it was already in production and doing its job.

That distinction matters here. The incident did not stop Gmail ingestion. It stopped the low-latency path and left the slower path carrying the full load. Calling the whole sync “down” would overstate what users experienced. Calling it healthy would ignore a complete failure in a mechanism we promise to keep fast.

The email that jammed it

An email was permanently deleted. It was not sitting in Trash, where Gmail could still return it. The history list nevertheless named its message ID as part of the changes we needed to process. When we fetched that ID, Gmail returned 404 because the message no longer existed.

The fetch error was ordinary. Our response to it was not. We saved the new history cursor only after every item in a run finished without error. That rule was meant to prevent a partial run from skipping work. In this case it also meant the deleted message stopped the cursor from advancing.

The next notification started from the old cursor, replayed the same history window and reached the same dead ID. Each new change made that replay window larger. By the end it contained about 15,000 records, but every run still stopped at the original missing message.

Reconstructed event trace from production logs. Request names and status codes match the Gmail API flow; message and cursor IDs are illustrative.

Why the backup hid it

The backup job removed every symptom severe enough to wake someone up. No customer reported missing mail because no mail was missing. Labels appeared. The delay was short enough to feel like a slow refresh rather than an outage.

Our alerts reinforced that false calm. They asked whether messages were entering the system and whether processing eventually completed. Both answers were yes. We had no alert for the push path itself, no watch on a history cursor that had stopped moving, and no graph showing that 100% of Gmail traffic had shifted to the backup.

This was not a failure of redundancy. Redundancy worked. It was a failure to observe the redundant routes separately. Once the backup took over, it protected users and concealed the broken path from us at the same time. We had measured the outcome and ignored how the system produced it.

What Google’s docs miss

Google documents one 404 in this workflow clearly. If startHistoryId is invalid or too old, history.list usually returns HTTP 404. The reference tells the client to perform a full sync. We already handled that case.

Our 404 came from the next request. A history record usually contains only id and threadId, so the client must call messages.get to retrieve the email. The message can be permanently purged after the history record is written but before that fetch happens. The returned ID is a record of an event, not a promise that the object still exists.

The reference defines messagesDeleted as messages “deleted (not Trashed) from the mailbox,” which points toward the edge case. It does not say that fetching a message just named by history can return 404, or tell clients whether to skip, retry or reset the cursor.

Google Gmail API reference, captured and annotated 17 September 2026. Open the source page.

What we measured after

We repaired the loop on 4 August and sent real mail through both routes the next day. Push notifications reached the product 6, 8 and 6 seconds after the messages arrived. Two messages picked up by the backup took 27 and 110 seconds. The instant path then completed 45 consecutive runs without another failure.

Five messages from one mailbox on one day are not a latency benchmark. They do establish the scale of this incident without pretending the sample is broader than it is. The push path landed in single-digit seconds in those three observations. The backup remained inside its two-minute schedule in the other two.

That is why “no mail was lost” does not make the failure trivial. For eleven days, every connected Gmail inbox depended on the slower route. The product worked, but one of the mechanisms that makes it feel immediate did not.

What we changed

The code change was small. When messages.get returns 404 for an ID from the history list, we now record that item as resolved and continue. The run can finish, save the latest cursor and move on. Errors that might represent unfinished work still stop the commit.

We also changed the monitoring. Three signals now stand on their own: consecutive failures in the push worker, a history cursor that remains unchanged, and the share of messages collected by the backup. The first or second would have caught this incident on day one. The third would have shown the traffic shift even if the error had been different.

There is a cost signal too. Because the cursor stayed fixed, each retry replayed a larger history window until it reached roughly 15,000 records. A stuck cursor gets more expensive with time. That growth now matters in the alert, not just in a postmortem.

For other developers

Treat every message ID from history.list as a reference to an event, not proof that the message is still fetchable. Decide in advance what a 404 from messages.get means. In our case the item is resolved because the object is gone; retrying it cannot recreate the email.

Make the cursor commit depend on resolved work rather than a perfectly clean loop. A run can contain an expected absence and still be complete. Reserve a blocked commit for failures that leave the state genuinely uncertain.

Monitor push and polling as separate products. A healthy fallback can make an unhealthy fast path look fine, especially when both feed the same success counters. Also monitor subscription renewal. Gmail requires the watch to be renewed at least every seven days and recommends doing it daily. That was not our bug this time, but it can produce the same quiet shift onto polling.

Where this leaves us

MorningZero puts several mailboxes behind one triage layer. If one provider starts arriving two minutes late, the product is two minutes late for that provider even when every message eventually appears. We should have detected that distinction ourselves.

The incident is fixed. More importantly, the route that hid it is now visible. We can see whether a message arrived by push or by the backup, whether the cursor moved, and whether failures are repeating. Those were basic questions our old dashboard could not answer.

We left the numbers public because “real-time” is easy to claim and easy to blur after a failure. The earlier account appears in what happens in the first ten seconds of an email’s life and in our Superhuman comparison. This post is the complete record behind those shorter disclosures.

Sources

Google documentation, checked 17 September 2026:

• Method: users.history.list — the documented stale-cursor 404, returned fields and deleted-message definition
• Synchronize clients with Gmail — partial sync, history retention and the full-sync recovery path
• Configure push notifications with the Gmail API — renewal requirements and the polling recommendation

Incident figures come from our production logs and database: 23 July to 4 August 2026, 686 failed push runs, a replay window of about 15,000 records, and five timing samples from 5 August 2026.

Frequently asked

Was any email lost during the 11 days?

No. The instant path failed on every attempt, but a separate check pulled each connected mailbox every two minutes and delivered every message. People could have seen mail arrive up to two minutes later than usual instead of in about six seconds. Nothing was lost, skipped or deleted.

What actually broke?

Gmail's history list named an email that had already been permanently deleted. Fetching that message returned 404. Our code stopped the run before saving the new history cursor, so the next notification replayed the same records and hit the same deleted email. This happened 686 times between 23 July and 4 August 2026.

Why didn't monitoring catch it?

Our alerts checked whether mail arrived, and it did through the two-minute backup. We had no alert for repeated failures on the instant path, a history cursor that had stopped moving, or an unusual share of messages arriving through the backup. Any one of those signals would have exposed the problem on day one.

Can the Gmail API name a message that no longer exists?

Yes. Gmail's history list returns message IDs rather than complete messages, so an email can be permanently purged before the client fetches it. Google documents the 404 caused by an old history cursor and prescribes a full sync. Its sync documentation does not describe this per-message 404 or the recovery behavior a client should use.

How fast is the sync now?

In one production mailbox on 5 August 2026, three notifications on the repaired instant path arrived 6, 8 and 6 seconds behind the mail. Two messages handled by the backup took 27 and 110 seconds. These are five samples from one day, not a general benchmark, but they show why the instant path matters.

What changed after the incident?

A 404 for a message that no longer exists is now treated as resolved work, which lets the run finish and the history cursor advance. We also added alerts for consecutive instant-path failures, a cursor that does not move, and mail arriving through the backup instead of push notifications.