A fix has been implemented and we are monitoring the results.
Posted Dec 08, 2021 - 01:17 CET
After about 8 hours the whole system is finally recovering as we found the way how to bypass still broken AWS services.
Current state: - All the Apify components are now fully operational including Actors, Scheduler, API, ... - Performance may still be degraded but it's recovering - We are monitoring the situation and will keep you informed in a case of change
Summary of the impact: - Since the start of the incident both actors and API had a higher error rate and degraded performance that escalated to an almost complete outage during the past 1-2 hours - Scheduler was down for the past 1-2 hours - Many operations such as abort did not work for many of the runs - No data loss
Amazon Web Services are finally recovering but some of the services we critically depend on are still not fully operational. Actors and API are still degraded and we experience a complete outage of the scheduler.
Posted Dec 08, 2021 - 00:21 CET
Current state: - There are finally signs of improvement - Actors are partially functional - runs are starting with a delay and finishing but the error rate is still high - API has a higher error rate than usual but performance got back to normal