Tell HN: Azure outage
Update 16:57 UTC:
Azure Portal Access Issues
Starting at approximately 16:00 UTC, we began experiencing Azure Front Door issues resulting in a loss of availability of some services. In addition. customers may experience issues accessing the Azure Portal. Customers can attempt to use programmatic methods (PowerShell, CLI, etc.) to access/utilize resources if they are unable to access the portal directly. We have failed the portal away from Azure Front Door (AFD) to attempt to mitigate the portal access issues and are continuing to assess the situation.
We are actively assessing failover options of internal services from our AFD infrastructure. Our investigation into the contributing factors and additional recovery workstreams continues. More information will be provided within 60 minutes or sooner.
This message was last updated at 16:57 UTC on 29 October 2025
---
Update: 16:35 UTC:
Azure Portal Access Issues
Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. Customers may experience issues accessing the Azure Portal. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions. More information will be provided within 60 minutes or sooner.
This message was last updated at 16:35 UTC on 29 October 2025
---
Azure Portal Access Issues
We are investigating an issue with the Azure Portal where customers may be experiencing issues accessing the portal. More information will be provided shortly.
This message was last updated at 16:18 UTC on 29 October 2025
---
Message from the Azure Status Page: https://azure.status.microsoft/en-gb/status
I noticed that Starbucks mobile ordering was down and thought “welp, I guess I’ll order a bagel and coffee on Grubhub”, then GrubHub was down. My next stop was HN to find the common denominator, and y’all did not disappoint.
It still surprises me how much essential services like public transport are completely reliant on cloud providers, and don't seem to have backups in place.
Here in The Netherlands, almost all trains were first delayed significantly, and then cancelled for a few hours because of this, which had real impact because today is also the day we got to vote for the next parlement (I know some who can't get home in time before the polls close, and they left for work before they opened).
Google cloud run or cloudflare workers it is.
Personally I am thinking more and more about hetzner, yes I know its not an apples to orange comparison. But its honestly so good
Someone had created a video where they showed the underlying hardware etc., I am wondering if there is something like https://vpspricetracker.com/ but with geek-benchmarks as well.
This video was affiliated with scalahosting but still I don't think that there was too much bias of them and they showed at around 3:37 a graph comparison with prices https://www.youtube.com/watch?v=9dvuBH2Pc1g
Now it shows how contabo has better hardware but I am pretty sure that there might be some other issues, and honestly I feel a sense of trust with hetzner I am not sure about others.
Either hetzner or self hosting stuff personally or just having a very cheap vps and going to hetzner if need be but hetzner already is pretty cheap or I might use some free service that I know of are good as well.
For some reason an Azure outage does not faze me in the same way that an AWS outage does.
I have never had much confidence in Azure as a cloud provider. The vertical integration of all the things for a Microsoft shop was initially very compelling. I was ready to fight that battle. But, this fantasy was quickly ruined by poor execution on Microsoft's part. They were able to convince me to move back to AWS by simply making it difficult to provision compute resources. Their quota system & availability issues are a nightmare to deal with compared to EC2.
At this point I'd rather use GCP over Azure and I have zero seconds of experience with it. The number of things Microsoft gets right in 2025 can be counted single-handedly. The things they do get right are quite good, but everything else tends to be extremely awful.
Currently standing in a half closed supermarket because the tills are down and they cant take payments
Ouch, and login.microsoftonline.com too - i.e. SSO using MS accounts. We'd just rolled that out across most (all?) of our internal systems...
And microsoft.com too - that's gotta hurt
I’ve been migrating our services off of Azure slowly for the past couple of years. The last internet facing things remaining are a static assets bucket and an analytics VM running Matomo. Working with Front Door has been an abysmal experience, and today was the push I needed to finally migrate our assets to Cloudflare.
I feel pretty justified in my previous decisions to move away from Azure. Using it feels like building on quicksand…
We’re 100% on Azure but so far there’s no impact for us.
Luckily, we moved off Azure Front Door about a year ago. We’d had three major incidents tied to Front Door and stopped treating it as a reliable CDN.
They weren’t global outages, more like issues triggered by new deployments. In one case, our homepage suddenly showed a huge Microsoft banner about a “post-quantum encryption algorithm” or something along those lines.
Kinda wild that a company that big can be so shaky on a CDN, which should be rock solid.
Can't download VSCode :D
Error: visual-studio-code: Download failed on Cask 'visual-studio-code' with message: Download failed: https://update.code.visualstudio.com/1.105.1/darwin-arm64/st...
So that's why I can't check in for my Alaska Airlines flight... https://news.microsoft.com/source/features/digital-transform...
microsoft.com and some subdomains (answers.microsoft.com) has no A and AAA records. They screwed up big time.
2026: the year of your own metal in a rack
Instead of cyber security awareness month, we should rename it to cloud availability awareness month.
For me the same. It's very confusing that status page [1] is green
Surely more vibecoding will fix this problem. Time to fire more staff
Yikes, http://schemas.xmlsoap.org/soap/encoding/ is running on Azure and it's down. So any SOAP/WSDL api's are dead in the water.
HTTPSConnectionPool(host='schemas.xmlsoap.org', port=443): Max retries exceeded with url: /soap/encoding/ (Caused by SSLError(CertificateError("hostname 'schemas.xmlsoap.org' doesn't match '*.azureedge.net'")))
160k+ results on GitHub: https://github.com/search?q=http%3A%2F%2Fschemas.xmlsoap.org...
Yeah just took down the prod site for one of our clients since we host the front-end out of their CDN. Just got wrapped up panic hosting it somewhere else for the past hour, very quickly reminds you about the pain of cookies...
Surprised to see the situation getting worse, what the hell.
Had some Frontdoor operations timing out, but now I'm straight up denied with "Message: All Changes to Azure Frondoor Configuration are blocked currently."
What a mess.
Pretty much all Azure services seem to be down. Their status page says it's only the portal since 16:00. It would be nice if these mega-companies could update their status page when they take down a large fraction of the Internet and thousands of services that use them.
So much of Belgium runs on Azure… it's honestly baffling how many services are down, there's no resilience built into (even large) companies anymore.
The Internet is supposed to be decentralized. The big three seem to have all the power now (Amazon, Microsoft, and Google) plus Cloudflare/Oracle.
How did we get here? Is it because of scale? Going to market in minutes by using someone else's computers instead of building out your own, like co-location or dedicated servers, like back in the day.
They admit in their update blurb azure front door is having issues but still report azure front door as having no issues on their status page.
And it's very clear from these updates that they're more focused on the portal than the product, their updates haven't even mentioned fixing it yet, just moving off of it, as if it's some third party service that's down.
The paradox of cloud provider crashes is that if the provider goes down and takes the whole world with it, it's actually good advertisement. Because, that means so many things rely on it, it's critically important, and has so many big customers. That might be why Amazon stock went up after AWS crash.
If Azure goes down and nobody feels it, does Azure really matter?
Looks to be affecting our pipelines that rely on Playwright as they download images from Azure e.g. https://playwright.azureedge.net/builds/chromium/1124/chromi... which aren't currently resolving.
Friend of mine at MSFT says it's a Sev-0 outage and they can't even get to the ticket tracking system.
We’ve been experimenting with multi-cluster failover for Kubernetes workloads, and one open-source project that actually works really well is k8gb .
It acts as a GSLB controller inside Kubernetes — doing DNS-level health checks, region awareness, and automatic failover between clusters when one goes down.
It integrates with ExternalDNS and supports multiple DNS providers (Infoblox, Route53, Azure DNS, NS1, etc.), so it can handle failover across both on-prem and cloud clusters.
It’s not a silver bullet for every architecture, but it’s one of the few OSS projects that make multi-region failover actually manageable in practice.
Hetzner, Netcup, OVH, BunnyCDN, ClouDNS, Postmark
You name them. Other good providers you have experience with?
There is no reason for an expensive cloud. Never has been, but decision makers tried to keep their pants dry.
Sorry - my bad. I literally just connected an old XP VM to the internet to activate it.
Seeing users having issues with the "Modern Outlook", specifically empty accounts. Switching back to the "Legacy Outlook" which functions largely without the help of the cloud fixes the issue. How ironic.
https://azure.status.microsoft/en-us/status says everything's fine! Any place I can read more about this outage?
Seems to be down in Norway.
Even the national digital id service is down.
For us, it looks like most services are still working (eastus and eastus2). Our AKS cluster is still running and taking requests. Failures seem limited to management portal.
The outage was really weird. For me, parts of the portal worked, other parts didn't. I had access to a couple of resource groups, but no resources visible in those groups. Azure Devops Pipelines that needed do download from packages.microsoft.com didn't work.
The Microsoft status page mostly referenced the portal outage, but it was more than that.
High availability is touted as a reason for their high prices, but I swear I read about major cloud outages far more than I experience any outages at Hetzner.
This is impacting the Azure CDN at azureedge.net. DNS A records for azureedge.net tenants are taking 2-6 seconds and often return nothing.
Some exec at Microsoft told the Azure guys to ape everything Amazon does and they took it literally.
We all need to move away from these big cloud providers. Two medium size smaller providers is enough.
-Cloudflare for R2 (object storage) and CDN (Fastly+backblaze also available). -Two VPS/Server providers with a decent reputation and mid-size (using a comparison site like https://serversearcher.com or look directly into people like Hetzner or latitude) -PlanetScale or Neon for database if you don't co-locate it, though better to use someone like digital ocean, vultr or latitude who offer databases too)
I still can't log into Azure Gov Cloud with
https://microsoft.com/deviceloginus
Seems like they migrated the non-Gov login but not the Gov one. C'mon Microsoft, I've got a deadline in a few days.
Updated 16:35 UTC
Azure Portal Access Issues
Starting at approximately 16:00 UTC, we began experiencing DNS issues resulting in availability degradation of some services. Customers may experience issues accessing the Azure Portal. We have taken action that is expected to address the portal access issues here shortly. We are actively investigating the underlying issue and additional mitigation actions. More information will be provided within 60 minutes or sooner.
This message was last updated at 16:35 UTC on 29 October 2025
----
Azure Portal Access Issues
We are investigating an issue with the Azure Portal where customers may be experiencing issues accessing the portal. More information will be provided shortly.
This message was last updated at 16:18 UTC on 29 October 2025
-- From the Azure status page
"Microsoft Azure will serve as the backbone of Asda’s digital infrastructure"[0]
Oh, that'll be why Scan & Go was down yesterday evening. I thought it was another instance of an iOS 26 update breaking their crappy code.
[0] https://corporate.asda.com/newsroom/2025/22/09/asda-announce...
The sad thing is - $MSFT isn't even down by 1%. And IIRC, $AMZN actually went up during their previous outage.
So if we look at these companies' bottom lines, all those big wigs are actually doing something right. Sales and lobbying capacity is way more effective than reliability or good engineering (at least in the short term).
UK, and other regions too; our APAC installation in Australia is affected.
The learning modules on https://learn.microsoft.com/ also seem to have a lot of issues properly loading.
The outage impacted GitSocial minor version bump release: https://marketplace.visualstudio.com/items?itemName=GitSocia...
There's no way to tell, and after about 30 minutes, the release process on VS Code Marketplace failed with a cryptic message: "Repository signing for extension file failed.". And there's no way to restart/resume it.
At least MSFT is consistent: https://www.microsoft.com/en-us/ is down as well
I was having issues a few hours ago. I'm now able to access the portal, although I get lots of errors in the browser console, and things are loading slowly. I have services in the US-East region.
I have been having issues with GitHub and the winget tool for updates throughout the day as well. I imagine things are pulling from the same locations on Azure for some of the software I needed to update (NPM dependencies, and some .NET tooling).
Interesting that everybody knows when AWS goes down but Azure needs a "Tell HN" :)
Best of luck to the teams responding to this incident.
Microsoft posted an update on X: https://x.com/AzureSupport/status/1983569891379835372?ref_sr...
"We’re investigating an issue impacting Azure Front Door services. Customers may experience intermittent request failures or latency. Updates will be provided shortly."
i guess folks in azure wanted to show some solidarity with aws brethren
(couldn't resist adding it. i acknowledge this comment adds no value to the discussion)
They suggest to use Traffic Manager to router around failing FrontDoor CDN, but DNS is failing too, making the suggestion another failure.
Wasn't the saying "It's always DNS" floating around somewhere?
Be interesting to understand cause here. Pretty big impact on services we use
Microsoft have started putting customer status pages up on windows.net, so it must be really really bad!
For example when I try to log into our payroll provider Brightpay, it sends me here:
https://bpuk1prod1environment.blob.core.windows.net/host-pro...
Portal and Azure CDN are down here in the SF Bay Area. Tenant azureedge.net DNS A queries are taking 2-6 seconds and most often return nothing. I got a couple successful A response in the last 10 minutes.
Edit: As of 9:19 AM Pacific time, I'm now getting successful A responses but they can take several seconds. The web server at that address is not responding.
And there goes https://www.microsoft.com/
Deglobalization in geopolitics should be followed by deglobalization in cloud providers as well. Viva la local vendors.
"Front Door" has to be the worst product name for a CDN I've ever heard of. I used to work for a CDN too.
It is much more than azure. One of my kids needs a key for their laptop and can't reach that either. Great excuse though, 'Azure ate my homework'. What a ridiculous world we are building. Fuck MS and their account requirements for windows.
We saw all incoming traffic to our app drop to zero at about 15:45. I wonder how long this one will take to fix.
My best guess at the moment is something global like the CDN is having problems affecting things everywhere. I'm able to use a legacy application we have that goes directly to resources in uswest3, but I'm not able to use our more modern application which uses APIM/CDN networks at all.
Service Status: https://status.cloud.microsoft/ and https://azure.status.microsoft/en-us/status
I bet it’s DNS.
Vibe coded internet keeps getting better
Two hours after the initial outage, they have finally updated the Front Door status on their status page.
Any healthcare IT admins care to chime in? A predominantly MS industry with critical workloads.
OpenAI Clip python library fails because the model download is a hardcoded azure cdn url :(
The VS Code website is down: https://code.visualstudio.com/
And so is Microsoft: http://www.microsoft.com/
This brings to mind this -> https://thenewstack.io/github-will-prioritize-migrating-to-a...
We're on Office 365 and so far it's still responding. At least Outlook and Teams is.
Looking forward to the post mortem.
It seems Azure FrontDoor is affected, because our private VM works fine in different regions.
I noticed issues on Azure so I went to the status page. It said everything was fine even though the Azure Portal was down. It took more than 10 minutes for that status page to update.
How can one of the richest companies in the world not offer a better service?
On our end, our VMs are still working, so our gitlab instance is still up. Our services using Azure App Services are available through their provided url. However, Front Door is failing to resolve any domains that it was responsible for.
I was working when I saw the portal page showing only resource groups and lots of items missing. I thought it was a weird browser cache issue.
The actual stuff I was working on (App Insights, Function App) that was still open was operational.
Unable to use Ona's GitPod through VSCode SSH - Unable to download code server from https://update.code.visualstudio.com
All of our sites went down. This is my company’s busiest time of year. Hooray.
With all the recent outages considered, it is time to move off the cloud.
Looks like MyGet is impacted too. Seems like they use Azure:
>What is required to be able to use MyGet? ... MyGet runs its operations from the Microsoft Azure in the West Europe region, near Amsterdam, the Netherlands.
All of my employers things are hosted on Azure and running just fine and didn't go down at all. Portal access has been fixed.
Doesn't seem to be too bad of an outage unless you were relying on Azure Front Door.
Not seeing it. I have VMs in US East and Netherlands and they're up.
Azure portal still insists the issue is jsut with Console.
We had to bypass the Frontdoor
Quite close to the recent AWS outage. Let me take a look if its a major one similar to AWS.
Any guess on what's causing it?
In hindsight, I guess the foresight of some organizations to go multi-cloud was correct after all.
I am having a bunch of issues. It looks like their sites and azure are both affected.
I also got weird notification in VS2022 that my license key was upgraded to Enterprise, but we did not purchase anything.
Reasons to not use hyperscalers, exhibit 654
There's a lot of outages this month!
Yesterday Amazon, today Microsoft. Are Google's cloud services going down tomorrow?
It's the DNS https://dnschecker.org/#A/get.helm.sh is unreachable
Does (should, could) DownDetector also say what customer-facing services are down, when some infrastructure is unworking? Or is that the info that the malefactors are seeking?
Cant access certain banking websites in the UK, I am assuming it because of this.
Part of this outage involves outlook hanging and then blaming random addins. Pretty terrible practice by Microsoft to blame random vendors for their own outage.
I absolutely love the utility aspect of LLMs but part of me is curious if moving faster by using AI is going to make these sorts of failure more and more often.
https://login.microsoftonline.com/ is down, so that's fun
I know how to fix this but this community is too close minded and argumentative egocentric sensitive pedantic threatened angry etc to bother discussing it
Unable to access the portal and any hit to SSO for other corporate accesses is also broken. Seems like there's something wrong in their Identity services.
Even if the cloud providers have much better reliability than most on-prem infra, the failure correlation they induce negates much of the benefit.
Language models aren't perfect; they can still generate similar outputs. Invertibility is a stretch.
Thank you. I was wondering what was going on at a company whose web app I need to access. I just checked with BuiltWith and it seems they are on Azure.
Azure portal currently mostly not working (UK)... Downdetector reporting various Microsoft linked services are out (Minecraft, Microsoft 365, Xbox...)
Based on the delay in resolving the issue, it appears MC attempted to rehire some of the DevOps engineers whom AI had previously replaced.
I'm mid-deployment, but thankfully it seems to be running ok so far. Just the portal is not working so my visibility is not good.
They suggest to use Traffic Manager to route around failing CDNs. But DNS is not working too, making the suggestion another fail.
LinkedIn has been acting funny for an hour or so, and some pages in the learn.microsoft.com domain have been failing for me too...
Oh, well, I'm sure Azure will be given the same pass that AWS got here recently when they had their 12-hour outage...
GitHub runners (specifically the "larger" runner types) are all down for us. These are known to be hosted on Azure.
Alaska Airlines is redircting folks to their slimmed down international site and you can't check in on mobile.
This probably explains why paying for street parking in Cologne by phone/web didn't work (eternal spinner) then
The iron law of uptime: "The mandatory single point of failure in every possible system is configuration."
So that’s why CapitalOne is out today. Even though their (incorrect) status page says all systems operational.
it took a good half hour after we detected the problem to see a notification on the Azure status page. Thanks to those who responded to my question as it validated the issue was global and we contacted our users t right away
Our Azure DevOps site is still functioning and our Azure hosted databases are accessible. Everything else is cooked.
pretty interesting how datadog's uptime tracker (https://updog.ai/) says all the sites are fully available.
if that's true then it's a sign that Azure's control / data plane separation is doing it's job! at least for now
Guess when/who has the next outage!
I remember the saying "It's always DNS". I'm old.
Kind of mindboggling it's still sometimes DNS maybe.
MS website seems to be up but really slow. Think xbox might still be down, Bing works for some reason tho!?
Luckily, no one uses azure and it's fully expected from azure to go down all the time! Keep it up!
Yeah, Azure is a mess today. Can't do anything without the portal.
>Last week AWS, now this.
This is not the first or second time this happened, multiple Hyperscaler failed one by one.
Appears to be an issue in Front Door. Our back end stuff is fine but FD is bouncing everything.
Always in these large provider outages you see people who have forgotten the old ways.
Buy cloud because you're always safe! Until you aren't.
Somewhere, an ex microsoft engineer that where layoff during the last week, is saying to himself “thank god, this shit is not my problem anymore”
Earnings report today. A coincidence?
I can at least login to Azure. But several MS sites are down.
Many (all?) LinkedIn profiles are also down for me. Luckily the frontpage still works. ;-)
Go cloud!
Is it Cosmos DB? If so the symmetry with AWS/Dynamo would be very eerie.
vscode.dev appears to be down. I think this will be my excuse to find an alternative -- I never really liked vscode.dev anyway.
(Coder is currently at the top of the experiment list. Any other suggestions?)
I just tried to check the Xbox services status page and it never even loaded.
Github Codespaces (for the 5 people that use them) are also still down.
downdetector reports coincident cloudflare outage. is microsoft using cloudflare for management plane, or is there common infra? data center problem somewhere, maybe fiber backbone? BGP?
Ahh it got me, Alaska air web site has an Azure outage banner
downdetector reports coincident cloudflare outage. is microsoft using cloudflare for management plane, or is there common infra? data center problem somewhere, maybe fiber backbone? BGP?
Down in Sweden Central as well (all our production systems are down)
Yudkowsky's feared Superintellignece holding Azure hostage
"On Prem" is looking better and better :-).
Wow, they are still down 12 hours later. :/
yep having trouble logging into https://entra.microsoft.com/ as well
Please sort it out, I'll be out of a job tomorrow.
Yep, down from here too (in Israel).
Services too, not just the portal.
Fascinating timing given the APEC summit ;)
Shouldn't regions be completely independent?
Putting all your eggs software in one basket
I could not access MS Clarity the entire day.
on the line with msft, they said 4 hours is what they are thinking. a workaround they are saying is to use traffic manager,
GitHub also seems to be having trouble for me
Took out the archive.ph and .is sites too?
Intune, Azure, Entra down in Switzerland
Github Actions and Codespaces degraded.
The Azure API is still working though.
microsoft.com is back -
edit: it worked once, then died again. So I guess - some resolvers, or FD servers may be working!
Can't get to microsoft.com even.
Anyone have betting odds on when Google will go down next? Are we looking at all 3 providers having outages in the span of 3 weeks?
Down here too (region West Europe)
My bet is on a bad config change.
Looks like AWS is also impacted?
Can't connect to Claude
they recently had an incident with front door reachability, wonder if it's back.
QNBQ-5W8
This cannot be a coincidence
Shut the front door!
looks like MS completed a failover and things are be recovering slowly
Compare the comments and news coverage on this compared to the AWS outage... pretty telling.
AWS, now Azure - wasn't this a plot point in Terminator where SkyNet was causing computer systems to have issues much before it finally become self-aware?
Funnily enough, AI has been training on its own data as generated by users writing AI conversations back to the internet - there's a feedback loop at play.
We're quickly learning who's relying on a single cloud provider.
An important quality of the cloud is that it is always available.
Except that it is not!
Interesting times...
Yup, see it as well.
one day these outages will cause a starvation.
Hello fellow boomers!
I noticed that winget is also down eg.
winget upgrade fabric
Failed in attempting to update the source: winget
An unexpected error occurred while executing the command:
InternetOpenUrl() failed.
0x80072ee7 : unknown errorthings seem to be coming back up now
It's DNS
As of now Azure Status page still shows no incident. It must be manually updated, someone has to actively decide to acknowledge an issue, and they're just... not. It undermines confidence in that status page.
It’s not DNS
There is no way it’s DNS
It was DNS
From Azure status page: "Customers can consider implementing failover strategies with Azure Traffic Manager, to fail over from Azure Front Door to your origins".
What a terrible advise.
So that's why all of our municipality's digital services are down ... utter chaos at the political meeting I attended just now.
auth services are down
What a time to be alive!
Portal is now accessible, bypassing FDN
Wtf happen with US east????
Just another day with microsoft. Honestly pretty tiring as something is always generally broken.
now aws down again?
Meanwhile the layoffs continue https://www.entrepreneur.com/business-news/microsoft-ceo-exp...
[flagged]
[dead]
[dead]
[dead]
[dead]
[dead]
[flagged]
[flagged]
[flagged]
Reports of Azure and AWS down on the same day? Infrastructure terrorism?
According to downtector.com - both AWS and GCP are down as well. Interesting
Preliminary post incident review: https://azure.status.microsoft/en-gb/status/history/
Timeline
15:45 UTC on 29 October 2025 – Customer impact began.
16:04 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered.
16:15 UTC on 29 October 2025 – We began the investigation and started to examine configuration changes within AFD.
16:18 UTC on 29 October 2025 – Initial communication posted to our public status page.
16:20 UTC on 29 October 2025 – Targeted communications to impacted customers sent to Azure Service Health.
17:26 UTC on 29 October 2025 – Azure portal failed away from Azure Front Door.
17:30 UTC on 29 October 2025 – We blocked all new customer configuration changes to prevent further impact.
17:40 UTC on 29 October 2025 – We initiated the deployment of our ‘last known good’ configuration.
18:30 UTC on 29 October 2025 – We started to push the fixed configuration globally.
18:45 UTC on 29 October 2025 – Manual recovery of nodes commenced while gradual routing of traffic to healthy nodes began after the fixed configuration was pushed globally.
23:15 UTC on 29 October 2025 - PowerApps mitigation of dependency, and customers confirm mitigation.
00:05 UTC on 30 October 2025 – AFD impact confirmed mitigated for customers.