Microsoft.

It appears Microsoft has shot itself, and half the world, in the foot by fouling up a network change.
I was woken up at 05:40 by my good lady who thought the heating had failed. The Hive app on her phone was not working. Mine was, so I stuck the boost on for her as well as the lamp downstairs. Later in the morning, my app stopped working too and a search on the net pointed the finger at a failure of Microsoft systems.
A snippet on the news said that Microsoft was "rolling back a network change" which had caused a worldwide outage of Outlook, Teams, Azure, X-box games and just about anything else run from their Cloud platform.
At no time did it affect the operation of the heating, just being able to control it from our phones.
You just have to love technology.

 

Issue's not reached this corner of Devon as both Teams and Outlook emails are working on our network.

 

Maybe they are starting to get it back up or maybe it doesn't affect wet string. Hive is still down.

Microsoft working to end Outlook and Teams outage around world

 

From the AA: Due to global Microsoft networking issues some of our services may be affected including the online reporting and tracking . We're working to get this fixed ASAP. Our lines are extremely busy due to this, but please stay on hold and we'll answer as soon as we can.

 

Do we not think the media is hyping things up a bit as usual?

Outlook and Teams have both been working as usual for the majority of people. It's not the apocalypse.

Microsoft has data centres all over the world. When you connect to their services, you could get a connection to any of a number of servers in any of a number of data centres. It is unlikely we'll get the full details of what happened, but it is likely that one DNS or switch failed, and their automatic failover took over immediately, but the internet being a massive web of nodes and connects, it takes up to a few hours for the secondary to propagate the new settings globally, so the minority that were routed through the failed node would experience a loss of connectivity for potentially a few hours, while the majority connected through different routes wouldn't even notice anything.

If it was a blunder by a network engineer, which is possible, such things happen, which is exactly why networks have built in redundancy and failover, then in all likelihood, that poor engineer has dropped all their projects for the day to work on two things in this order, 1 fix it then 2 write up a really tedious report describing what happened, how many people were affected, which services were affected, what the resolution was, etc, and importantly, what vulnerabilities were identified and what proactive measures will be needed to reduce the chances of it happening again. They probably won't get home til midnight and because they've had to spend all that time on the fix, the root cause analysis, and the report, they'll be behind on all their other planned work and have to work their behinds off to catch up. The media never tends to cover that bit.

 

That's the beauty of being 5 hours behind you. It all must have been fixed before I put my slippers on.

I use Office 365 which in the beginning was a real pain but haven't had issues since.

 

Either that or you were among the vast majority that was not affected.

 

The Hive system for the UK is still down as is the AA. That is a hell of a lot of people. They use Microsoft Azure.

 

For any whole company to be down as a consequence of the Microsoft outage suggests that either one of two things is likely.

1. A cyber attack. Possible, but unlikely.
2. Those companies have not built themselves a resilient network infrastructure, such that one blip can take their whole system down. This one is far more likely. Such as configuring firewalls to specific IP addresses instead of DHCP and DNS servers or having a single line to the outside world.

If Microsoft was really down in a big way, we'd certainly know about it, because companies all over the world would just stop. Outlook and Teams would only be the tip of the iceberg. So many business applications now depend to varying extents on Azure, which is the Microsoft cloud platform. It's got company databases in it, all their reporting and business intelligence, file sharing, automation scripts, all sorts. Not to mention that office 365 is powered by the web.

Taking a few outages and extrapolating to the conclusion it's massive is like saying there's been a lane closure on the M1, which is affecting all road, rail, air and sea transportation globally.

 

Hive is finally back up, after more than 12 hours.