Resolved
After 8 hours and 54 minutes

Our database systems are now back to full performance and all other aspects of the services (except stats) are functioning normally. As this power related issue has been resolved we will mark this incident as such and will create a new one for keeping everyone updated on the stats processing issue.

Avatar for
Recovering
After 5 hours and 5 minutes

Uploading in Libsyn 5 has now been fixed and is working normally again. DB performance is still slower than normal as the read-only DB nodes are still recovering.

Stats processing is still stopped and there are no additional updates on that at this time.

Avatar for
Recovering
After 4 hours and 15 minutes

Uploading is now working properly in Libsyn 4. We're still fixing it in Libsyn 5, if you need to publish immediately please use Libsyn 4.

Avatar for
Recovering
After 3 hours and 8 minutes

We're still working on getting the read-only DB nodes back up and running, however the majority of services have been restored at this time with only publishing and stats still being impacted. The publishing issues seem to be related to a problem with uploads that we are actively investigating.

Stats processing is currently stopped and we expect a longer recovery time getting that up and running. This means that stats numbers won't update and newly published episodes will not appear in stats until the processing is running again.

We'll continue to post further updates.

Avatar for
Recovering
After 2 hours and 33 minutes

Primary database cluster has recovered and is running normally. We're working to get the read-only DB nodes back up and running until that is completed database performance will be slower than normal. We're continuing to go through things and restore services as we find issues.

Avatar for
Identified
After 1 hour and 45 minutes

The majority of hardware has powered back on successfully and our database cluster is recovering. Once that is completed we'll have a better sense of where things stand with individual services and getting them back online.

Avatar for
Identified
After 1 hour and 26 minutes

Utility power has been restored and hardware is coming back online, we're checking systems now and will continue to keep everyone posted.

Avatar for
Identified
After 1 hour and 25 minutes

The power issue appears related to the ATS (automatic transfer switch) system that is supposed to automatically switch over from utility to generator when an outage occurs. The generators did start and are online but the transfer system did not switch power over to them automatically.

Avatar for
Identified
After 45 minutes

This is now confirmed to be a power issue at the data center facility. While their generators are running power is not making it into the facility and all hardware is powered off.

Avatar for
Investigating
After 17 minutes

Initial word from our datacenter provider is pointing to possible full power loss at the facility on both the primary (A) and backup (B) power feeds. We'll continue to post updates as we learn more.

Avatar for
Investigating

We have been notified of an outage of all system components and our team is now investigating.

Avatar for
Began at:

Affected components
  • Libsyn Platform
    • Login Page
    • Dashboard & Publishing
  • Podcast Pages & RSS Feeds
    • Podcast Page
    • RSS Feed
  • Media & Asset Delivery
    • Media Files
    • Web Assets
  • Web Player
  • Mobile Apps
  • FTP
  • Stats