For a few hours on Wednesday afternoon our Dashboards (dashboards.global-roam.com) and Downloads (downloads.wattclarity.com.au) were experiencing errors as a result of a code change that we made to resolve an issue with our infrastructure.

The changes were rolled back as shortly after and the applications were back up and running.

Update 6 Sept 2023, 11:50am NEM Time:

Gaps in Hosted MMS databases have been filled.

Initial status:

All services are back online from yesterday’s Microsoft Azure outage. We are in the process of filling gaps in the databases of the affected services. The Trends widget in ez2view, NEMreview online and some Hosted MMS customers may experience missing data in the affected time range until recovery is complete.

On Wednesday evening 30th August 2023 Microsoft Azure experienced significant disruption to their services which cascaded through to cause significant disruption to some of our services provided to clients.

These impacts were not fully resolved until Thursday morning 31st August 2023.

Following the incident we compiled a longer-form record at this dedicated page (linked here).

The backup webDB experienced intermittent connection issues of short periods. At 16:00h connections to the DB failed until fixed at 16:34h. Products affected: deSide web update, NemTicker, deSide Online. The three main deSide databases were not affected.

deSide updater was slow to deliver data for the intervals 13:30-13:40 for 2023-06-06. A configuration change to one of our backend server meant that it was relying solely on the datafeed from nemweb which had a dely on the data for these intervals due to planned maintenance at the AEMO.