Massive Microsoft 365 outage caused by faulty ECS deployment

By Sergiu Gatlan, Bleeping Computer

In a preliminary post-incident report, Microsoft has revealed that this week's 5-hour-long Microsoft 365 worldwide outage was triggered by a faulty Enterprise Configuration Service (ECS) deployment that led to cascading failures and availability impact across multiple regions.

ECS is an internal central configuration repository designed to enable Microsoft services to make wide-scope dynamic changes across multiple services and features, as well as targeted ones such as specific configurations per tenant or user.

What initially started like a minor Microsoft Teams outage ended up expanding downstream to multiple Microsoft 365 services with Teams integration that also leverage ECS, including Exchange Online, Windows 365, and Office Online.

As a result, users worldwide began reporting that they could not use Microsoft Teams and multiple Microsoft 365 services or features.

"This issue affected the users' ability to connect to the Microsoft Teams Desktop, Web and Mobile clients," the company explained in its preliminary report.

"Telemetry indicated that approximately 300k calls were impacted by this event. The Asia Pacific (APAC) region was most affected due to business hours coinciding with the impact window. Additionally, Direct Routing and Skype MFA were mostly impacted service."

According to Redmond's report, the incident started on Thursday, July 21, at 1:05 AM UTC, with the company's engineers remediating most of its impact within five hours, by 6:00 AM UTC.

However, there was also some isolated residual impact until 1:14 PM UTC the same day, matching customer reports on social media.

In the end, the incident affected users attempting to utilize one or more of the following Microsoft 365 services and features (all impacted to some degree by the outage):

  • Exchange Online (Delays sending mail)
  • Microsoft 365 admin center (Inability to access)
  • Microsoft Word within multiple services (Inability to load)
  • Microsoft Forms (Inability to use via Teams)
  • Microsoft Graph API (Any service relying on this API may have been affected)
  • Office Online (Microsoft Word access issues)
  • SharePoint Online (Microsoft Word access issues
  • Project Online (Inability to access)
  • PowerPlatform and PowerAutomate (Inability to create an environment with a database)
  • Autopatches within Microsoft Managed Desktop
  • Yammer (Impact to Yammer flighting)
  • Windows 365 (Unable to provision Cloud PCs)


Comments

Post a Comment

Popular posts from this blog

Why remote desktop tools are facing an onslaught of cyber threats

Image
By Solomon Klappholz, IT Pro Hackers are increasingly targeting remote desktop tools in their attacks, new research reveals, prompting warnings for enterprises globally In the era of hybrid work, remote desktop tools have become vital business enablers, but due to their pervasiveness on corporate networks they have become a popular entry point for cyber criminals. If successfully exploited, remote access tools can provide hackers with a direct pathway into a system or network, and once access is gained attackers can move laterally within the network, escalating privileges and maintaining persistence. In an investigation into which remote desktop tools are targeted the most, Jonathan Tanner, senior security researcher at Barracuda Networks, explained that remote desktop software poses a particular challenge to IT teams to secure. “Among the security challenges facing IT teams implementing remote desktop software is that there are many different tools available, each using different and ...
Read more

Ransomware gang starts leaking alleged stolen Change Healthcare data

Image
By Lawrence Abrams,  Bleeping Computer The RansomHub extortion gang has begun leaking what they claim is corporate and patient data stolen from United Health subsidiary Change Healthcare in what has been a long and convoluted extortion process for the company. In February, Change Healthcare suffered a cyberattack that caused massive disruption to the US healthcare system , preventing pharmacies and doctors from billing or sending claims to insurance companies. The attack was ultimately linked to the BlackCat / ALPHV ransomware operation, who later said they stole 6 TB of data during the attack . After facing increased pressure from law enforcement, the BlackCat gang shut down their operation . This occurred amid claims they were pulling an exit scam by stealing a $22 million Change Healthcare ransom payment from the affiliate who conducted the attack. While Change Healthcare has declined to comment on whether it has paid a ransom, the affiliate known as "Notchy" said they ...
Read more

Notepad++ wants your help in "parasite website" shutdown

Image
By Ax Sharma,  Bleeping Computer The Notepad++ project is seeking the public's help in taking down a copycat website that closely impersonates Notepad++ but is not affiliated with the project. Although, at the time of writing, the lookalike website takes visitors to the official Notepad++ downloads page, there is some concern that it could pose security threats—for example, if it starts pushing malicious releases or spam someday either deliberately or as a result of a hijack. The lookalike website appears prominently in search results
Read more