Operations Center performance degradation
Resolved
Sep 12 at 02:34pm EDT
On September 10th, D4H experienced a service degradation and intermittent downtime on Operations Center between 11:23 UTC and 16:49 UTC. Our root cause analysis showed that high CPU from an automatically updated third-party software component caused the degradation.
We have already taken preventative measures to ensure a similar issue does not happen in the future and confirm that all systems are operating normally.
Please let us know at help@d4h.com if you need any assistance and we will work directly with you to resolve it.
Thank you for your understanding and continued trust.
Robin Blandford
Senior Vice President β D4H
Affected services
Updated
Sep 10 at 12:53pm EDT
All systems are now operating normally. We are finalizing the root cause analysis and will share a full report shortly.
Affected services
Updated
Sep 10 at 09:00am EDT
We continue to encounter severe issues on the US region of Operations Center.
Affected services
Updated
Sep 10 at 08:07am EDT
The increased capacity has restored service, but it remains slower than normal. We continue to implement additional measures to restore full performance.
Affected services
Updated
Sep 10 at 08:05am EDT
The recovery is unstable and we are still observing issues. We are increasing the infrastructure's capacity to handle this load and to recover faster.
Affected services
Updated
Sep 10 at 07:44am EDT
The issues encountered earlier appear resolved. We are monitoring the recovery and assessing the causes to implement permanent solutions.
Affected services
Created
Sep 10 at 07:36am EDT
Operations Center in the US is currently encountering high error rates and may be inaccessible for some users. Our engineers are actively investigating the issue.
Affected services