Resolved
Resolved

There have been no further issues and the episode releaser system is fully caught up and running normally.

Avatar for
Recovering

At ~6:28AM (EST) we began receiving alert notifications from our monitoring services. An investigation lead to discovery of a failed service on a primary node in our caching layer that didn't properly failover to backup nodes. This triggered high load on backend database systems causing severe performance issues throughout multiple parts of our systems.

The failed service was manually failed over to backup nodes at 6:41AM and we began to see immediate recovery of our systems. By ~6:44AM all services were fully restored and functioning normally.

We are continuing to monitor the situation. The issue did cause the system responsible for releasing episodes to get backed up and it is now in the process of catching up. We'll post an update when it's fully caught back up.

Avatar for
Began at:

Affected components
  • Libsyn Platform
    • Login Page
    • Dashboard & Publishing
  • Media & Asset Delivery
    • Media Files
    • Web Assets
  • Web Player