In a recent development, last week saw a truly epic outage to Microsoft 365, with a huge number of customers unable to send and receive emails. The company has now admitted that at the worst point, more than 80% of its servers were unresponsive.

Microsoft has now explained what went wrong, stating that the mistake that led to the failure was actually made all the way back in July without being detected ….

In a recent development, the tech giant said at the time that it was a technical issue and not a cyber attack, and that it was working hard to fix the problem, admitting that it was taking longer than expected. Microsoft also denied that it was related to the thousands of layoffs revealed the previous month.

Customers were understandably unimpressed at the scale of the outage and the time taken to fix it, although meme creators were having fun.

Sorry but you have to be change the name now. Thems the rules pic.twitter.com/5UttZ8nt9f.

It certainly points out that when you base your business on cloud services, you also take the risk that comes with that.

Platform head Charles Lamanna admitted the severity of the problem.

In a recent development, in less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive.

He remarked it resulted from a combination of two issues. The manufacturer maintains two independent network gateway systems, one of which failed, meaning that the remaining gateway took the full load. That wouldn’t have been a problem on its own, Microsoft stating that each one has enough capacity to handle all traffic. However, a mistake made back in July led to a cascading failure.

A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers […].

According to recent reports, the initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no problems were observed. For more than three months, the change was deployed worldwide without impact. Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores.

In other words, the update wasn’t tested under the conditions in which it was most likely to fail.

While the company claims that the incident is unrelated to laying off staff and replacing them with AI systems, I find myself extremely skeptical. It seems inconceivable that a company of Microsoft’s size could make what appears to be a schoolboy error of this magnitude unless there were insufficient engineers working on the enhancement. Please share your own thoughts in the comments.

It has been revealed that FTC: We use income earning auto affiliate links. More.

Check out 9to5Mac on YouTube for more Apple news:.

According to recent reports, breaking news on Apple and the tech industry. Ge….

It has been revealed that ben Lovejoy is a British technology writer and EU Editor for 9to5Mac. He’s known for his op-eds and diary pieces, exploring his experience of Apple products over time, for a more rounded review. He also writes fiction, with two technothriller novels, a couple of SF shorts and a rom-com!

→ Read original article