We experienced the Actor runtime slowdown similar to the recent incident from Saturday 5th. Unfortunately, the newly implemented alerting mechanism failed and the actors were slowed down at various levels during the past about 10 hours.
The recovery mechanism did work and we quickly recovered. Everything now works at a normal speed.
Affect on users: - Many of the actors running between about 00:00 and 10:15 GMT+1 have been running slower than usual. - This mainly affects the short-running actors as these are likely to time out. - The actual reason for the slowdown is a high error rate resulting in a higher number of API call retries. The high number of retries are caused by network problems.
We finally collected enough data and worked on a long-term solution. We apologize for any inconvenience caused and will update you on the progress shortly.
-------------------------------------------------
Update (2022/08/02 19:07 GMT+1): We are still working on a final solution but have a recovery system in place to beware of this recurrence of this issue .