Update 6 Sept 2023, 11:50am NEM Time:

Gaps in Hosted MMS databases have been filled.

Initial status:

All services are back online from yesterday’s Microsoft Azure outage. We are in the process of filling gaps in the databases of the affected services. The Trends widget in ez2view, NEMreview online and some Hosted MMS customers may experience missing data in the affected time range until recovery is complete.

On Wednesday evening 30th August 2023 Microsoft Azure experienced significant disruption to their services which cascaded through to cause significant disruption to some of our services provided to clients.

These impacts were not fully resolved until Thursday morning 31st August 2023.

Following the incident we compiled a longer-form record at this dedicated page (linked here).

The backup webDB experienced intermittent connection issues of short periods. At 16:00h connections to the DB failed until fixed at 16:34h. Products affected: deSide web update, NemTicker, deSide Online. The three main deSide databases were not affected.

deSide updater was slow to deliver data for the intervals 13:30-13:40 for 2023-06-06. A configuration change to one of our backend server meant that it was relying solely on the datafeed from nemweb which had a dely on the data for these intervals due to planned maintenance at the AEMO.

An automated database maintenance script led to a failure to process the analysis of FCAS bids for the 26/05/23 trading day. FCAS bids displayed in ez2view for the 26th may have been incomplete or not visible at all in ez2view from 27/05 until resolution Monday morning 29/05. FCAS bid data prior to the start of the 26th of May trading day was unaffected, as is data from the 27/05 trading day onwards. Energy bid processing was also unaffected. We are reviewing our database maintenance to avoid this situation reoccurring and apologise for any inconvenience.