From midnight on Saturday Feb 3rd through approx 7:45am NEM time our Trends engine which provides functionality that backs NEMReview online and Trends editor in ez2View (amongst other things) was unavailable.

The issue was caused by an upstream data provider fixing a typo in a single entry, which led to a duplicate data error at our end, which in turn caused the trends engine to fail to start up.

We apologise for the downtime and we are investigating ways to be more resilient to this kind of failure in the future.

Due to a number of compounding issues, the Live Supply Demand widget was showing data 5 minutes behind real time for the whole day starting midnight NEM time.

At around 3:40pm NEM time, the web service host attempted to restart the widget backend, but it was unable to restart successfully. Clients attempting to access Live Supply Demand from this time would have seen either a blank page or a “ 503” error.

The widget was back online at around 4:40pm NEM time.

Some real time SCADA data was not loaded into the full-history hosted MMS database between approx 11:40AM and 12:05PM NEM time. This affected the timely data presentation in some NEMreview trends and generation of up to date Live Supply Demand data. NEMwatch fuel type data may also have been affected during the time period.

The cause was a backlog of older market data files being back-filled into the database at the time, which consequently blocked real-time inserts of SCADA data.

We are reviewing our procedures for back-filling data to avoid a recurrence of this kind of issue.

From around 12:45pm on Monday 15th (NEM time) our Trends engine which provides functionality that backs NEMReview online and Trends editor in ez2View (amongst other things) was unavailable for about 15 minutes.

At this stage the cause of the error is unknown but seems to be related to a problem with our Azure infrastructure that wasn’t initiated by us. We are continuing to monitor the Trends engine and will provide more information as we learn anything further.

EMMS – FTP error

From around 11:55 on 2024-01-10, NEM time Global-Roam services and data feeds experienced delays in association with AEMO INC0124413 (EMMS – FTP error). AEMO’s incident notice reports: