On Wednesday evening 30th August 2023 Microsoft Azure experienced significant disruption to their services which cascaded through to cause significant disruption to some of our services provided to clients.

These impacts were not fully resolved until Thursday morning 31st August 2023.

Following the incident we compiled a longer-form record at this dedicated page (linked here).

The backup webDB experienced intermittent connection issues of short periods. At 16:00h connections to the DB failed until fixed at 16:34h. Products affected: deSide web update, NemTicker, deSide Online. The three main deSide databases were not affected.

deSide updater was slow to deliver data for the intervals 13:30-13:40 for 2023-06-06. A configuration change to one of our backend server meant that it was relying solely on the datafeed from nemweb which had a dely on the data for these intervals due to planned maintenance at the AEMO.

An automated database maintenance script led to a failure to process the analysis of FCAS bids for the 26/05/23 trading day. FCAS bids displayed in ez2view for the 26th may have been incomplete or not visible at all in ez2view from 27/05 until resolution Monday morning 29/05. FCAS bid data prior to the start of the 26th of May trading day was unaffected, as is data from the 27/05 trading day onwards. Energy bid processing was also unaffected. We are reviewing our database maintenance to avoid this situation reoccurring and apologise for any inconvenience.

The upstream provider we use for our email functionality to send alerts is currently experienceing some issues resulting in email notifications failing to be delivered.