Service Outage

Our database systems are now back to full performance and all other aspects of the services (except stats) are functioning normally. As this power related issue has been resolved we will mark this incident as such and will create a new one for keeping everyone updated on the stats processing issue.

Uploading in Libsyn 5 has now been fixed and is working normally again. DB performance is still slower than normal as the read-only DB nodes are still recovering.

Stats processing is still stopped and there are no additional updates on that at this time.

Uploading is now working properly in Libsyn 4. We're still fixing it in Libsyn 5, if you need to publish immediately please use Libsyn 4.

We're still working on getting the read-only DB nodes back up and running, however the majority of services have been restored at this time with only publishing and stats still being impacted. The publishing issues seem to be related to a problem with uploads that we are actively investigating.

Stats processing is currently stopped and we expect a longer recovery time getting that up and running. This means that stats numbers won't update and newly published episodes will not appear in stats until the processing is running again.

We'll continue to post further updates.

Primary database cluster has recovered and is running normally. We're working to get the read-only DB nodes back up and running until that is completed database performance will be slower than normal. We're continuing to go through things and restore services as we find issues.

The majority of hardware has powered back on successfully and our database cluster is recovering. Once that is completed we'll have a better sense of where things stand with individual services and getting them back online.

Utility power has been restored and hardware is coming back online, we're checking systems now and will continue to keep everyone posted.

The power issue appears related to the ATS (automatic transfer switch) system that is supposed to automatically switch over from utility to generator when an outage occurs. The generators did start and are online but the transfer system did not switch power over to them automatically.

This is now confirmed to be a power issue at the data center facility. While their generators are running power is not making it into the facility and all hardware is powered off.

Initial word from our datacenter provider is pointing to possible full power loss at the facility on both the primary (A) and backup (B) power feeds. We'll continue to post updates as we learn more.

We have been notified of an outage of all system components and our team is now investigating.