Back to overview
Downtime

Infrastructure routing incident on managedradio., library. and streaming-edge services

Oct 25 at 03:39am CEST
Affected services
aircheck. frontend
aircheck. graphql API
aircheck. authentication
musiccompanion. frontend
musiccompanion. graphql API
musiccompanion. authentication
library. app
library. graphql API
library. authentication
managedstreaming. edge service
managedradio. playout center
managedradio. backend API
playingnow. APIs

Updated
Oct 25 at 10:46am CEST

All services have been successfully restored. While all platforms are online, some users may still experience degraded performance or occasional slow responses as systems stabilize.

We continue to monitor performance closely and will provide a final update once the incident is fully closed.

Updated
Oct 25 at 09:42am CEST

We were able to mitigate the issue affecting the streaming-edge service. The problem was traced to an incorrect DNS update introduced during earlier mitigation steps. Normal routing has now been restored, and services are stable.

We are continuing to restore front-end access to aircheck. and musiccompanion., as well as backend services supporting these platforms.

Next update by 10:00 CEST or sooner as progress continues.

Updated
Oct 25 at 09:19am CEST

We have identified a regression affecting the streaming-edge service responsible for routing streaming requests. Our team is investigating the cause and working to restore stable operation.

Next update by 10:00 CEST or sooner as new information becomes available.

Updated
Oct 25 at 06:57am CEST

managedsstreaming. services have now fully recovered. Most playingnow. services as well. Some metadata services, aircheck. frontend and musiccompanion. remain in the process of recovery and DNS propagation. Our team continues to monitor and assist with the final restoration steps.

Next update by 08:00 CEST.

Updated
Oct 25 at 06:09am CEST

We are still in the process of recovery. Most services are now available over HTTP (without TLS), and we are actively reapplying certificates to restore full HTTPS availability.

We expect continued improvement as certificates are redeployed. Next update by 07:00 CEST or sooner if recovery completes.

Updated
Oct 25 at 05:09am CEST

DNS updates have propagated for most services. The library and managedradio platforms have recovered and are operating normally again. We are continuing to work on restoring full functionality for the streaming-edge services.

Next update by 06:00 CEST or sooner as progress continues.

Updated
Oct 25 at 04:39am CEST

We have identified the root cause and started applying the necessary mitigation steps. The mitigation involves updates to DNS entries, which may take a few minutes to propagate globally.

We expect services to gradually recover as DNS changes take effect. Next update by 05:00 CEST or sooner if recovery completes.

Updated
Oct 25 at 03:47am CEST

We have identified the issue as an HTTP routing problem affecting our managedradio., library., musiccompanion. and streaming-edge service clusters. Our team is actively working on mitigation steps to restore normal routing.

Next update by 05:00 CEST or sooner as new information becomes available.

Created
Oct 25 at 03:39am CEST

We are investigating an incident on our managedstreaming. edge service where requests cannot be processed at the moment.

We will update this inciden as soon as we know more or latest at 4:30 CEST