The original root cause was discovered. We identified the root cause as a database cluster issue and implemented a successful recovery yesterday. Our systems are now stable and functioning optimally.
Impact:
As a result of this issue, some users of pay-per-result Actors were charged more than they should have paid if the run was running during one of the following intervals: - 2024/07/03 15-17 UTC - 2024/07/04 04-06 UTC
We are currently investigating the full scope of this issue and will issue refunds to affected users within the next week. Based on our initial assessment, the number of impacted users is limited.
Future Prevention:
Over the coming week, our team will prioritize implementing more measures to prevent similar incidents in the future. We consider this issue a high-severity matter and are committed to ensuring the continued reliability of our services.
We sincerely apologize for any inconvenience this caused.
Posted Jul 05, 2024 - 11:47 CEST
Update
A long-term fix was deployed to the production environment to mitigate another performance drop. We will keep monitoring the system.
Posted Jul 04, 2024 - 15:25 CEST
Monitoring
We have identified the cause of the issue and implemented a workaround. A proper fix will be deployed later today.
There was no data loss, however the performance of Actors and API operations was degraded between 4 AM and 6 AM UTC 2024-07-04, and some scheduled Actors and tasks might have been triggered twice.
Posted Jul 04, 2024 - 10:09 CEST
Identified
The incident reappeared. Although the previous fix lessened the impact, we're still facing degraded performance of Actor runs and builds.
Posted Jul 04, 2024 - 06:30 CEST
Monitoring
A fix has been implemented and we will keep monitoring the results.
Posted Jul 03, 2024 - 20:33 CEST
Update
The system is operating normally at this point. We are still investigating the issue and will be implementing new measures.
Posted Jul 03, 2024 - 18:26 CEST
Investigating
The performance of Actor runs and builds is degraded. We are experiencing a slowdown in our internal systems and are investigating the issue.
Posted Jul 03, 2024 - 17:46 CEST
This incident affected: Console (console.apify.com), API (api.apify.com), and Actors.