Microsoft has released new information explaining the massive outage that started Thursday morning and affected Microsoft 365, Copilot, and Outlook access in 74 countries, according to Downdetector.
According to recent reports, charles Lamanna, Microsoft’s executive vice president of Copilot, Agents, and Platform, posted early Monday morning that the outage was due to three factors, including a “latent defect” from a July enhancement to Microsoft 365 networking gateway servers. Low server capacity and peak traffic in the US brought the defect out of hiding, despite no prior reported problems.
It has been revealed that “A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable,” Lamanna said, noting that the enhancement was deployed safely worldwide without any issues for over three months.
He said Microsoft didn’t find the glitch in testing due to “the different traffic and SKU mix used in our test labs,” because “our load testing tended to focus on our weaker hardware SKUs with fewer cores.”.
Also: Microsoft fixes over 1,000 Windows flaws – and one is being exploited.
Two other factors made matters worse, and left servers useless in less than two hours.
Industry observers note that “A surge in load increased the likelihood for the networking gateways to encounter the contention,” Lamanna noted, explaining that Microsoft 365 maintains two gateway systems. Before the outage started, Teams failed from one to the other, which “changed its traffic patterns.” Simultaneously, he added, “nearly 14% of server capacity for the networking gateway was out of service due to an unrelated matter from the previous day.”.
Those two conditions collided and “caused the network gateway to encounter the latent bug from July,” Lamanna concluded. Requests eventually overloaded servers, at which point they stopped responding. At the outage’s low point, over 80% of its servers weren’t responding.
Industry observers note that “When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints,” Lamanna remarked. “However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers.”.
Also: AI finds zero-day flaws for just $4 now – even mediocre hackers can cash in.
Before Microsoft issued an explanation, some suspected the outage may have been the result of a cyberattack, which both Lamanna and Microsoft’s status account denied.
In a recent development, it’s unclear how many people the outage affected, though Microsoft’s own numbers from January say Microsoft 365 “exceeds 450 million commercial paid seats,” illustrating the scale of the problem. Microsoft deployed an upgrade to fix the outage and gradually applied it to groups of servers. Lamanna said Microsoft would publish a full post-incident review with more details later this week.
Industry observers note that service might be restored, but customers aren’t happy about having lost nearly two full workdays. One think tank told CNN it estimated the outage — which lasted through close of business on Friday for some — resulted in $7 billion in losses, due to many businesses and organizations being unable to access their everyday systems.
Also: If the November Windows update fails to install on your PC, here’s what to do.
The fact that Microsoft’s testing couldn’t find the July issue ahead of time may leave customers wondering when the next dormant defect will strike, and whether the manufacturer is prepared.
Industry observers note that organizations across Europe have been leaving Microsoft in recent years, opting instead for locally-controlled, open-source alternatives. The former CEO of Github launched a decentralized alternative called Entire earlier this year, citing developers’ need for something more reliable, as Github has similarly been plagued with outages.
For many, Teams was barely offline over the two-day period, or was the first service to come back online.
“Several services like Teams, OneDrive, SharePoint, and Office apps failed out of the faulty network gateway infrastructure. That is why those services were restored earlier, with Teams chats and meetings functioning without issue for most of the outage,” Lamanna explained. Other services like Exchange Online can’t fail out.
He added that fixing the outage was slow because waiting traffic hit servers as soon as they came back, rendering them unresponsive just as quickly.
“Our normally automated recovery processes were not designed to restore service from this broad unresponsive state,” Lamanna said. “Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.”.
On Thursday morning, people began reporting concerns with Outlook, Teams, and Copilot, according to Downdetector. By 1 p.m. ET, the number of reported Outlook outages was nearly 83,000 but had fallen to just under 75,000 by 2:38 p.m. ET. Teams and Copilot reports started dropping significantly by Thursday evening, but many individuals were still reporting issues accessing systems well into Friday morning on X and Reddit. Even Microsoft’s status page was down until around 3 p.m. ET Thursday.
Also: If the November Windows update fails to install on your PC, here’s what to do.
As late as Friday afternoon, the organization didn’t yet have an ETA of when the outage would be fully solved.
