Back to overview
Downtime

Operations Center performance degradation

Sep 10 at 07:36am EDT
Affected services
Operations Center

Resolved
Sep 12 at 02:34pm EDT

On September 10th, D4H experienced a service degradation and intermittent downtime on Operations Center between 11:23 UTC and 16:49 UTC. Our root cause analysis showed that high CPU from an automatically updated third-party software component caused the degradation.

We have already taken preventative measures to ensure a similar issue does not happen in the future and confirm that all systems are operating normally.

Please let us know at help@d4h.com if you need any assistance and we will work directly with you to resolve it.

Thank you for your understanding and continued trust.

Robin Blandford
Senior Vice President – D4H

Updated
Sep 10 at 12:53pm EDT

All systems are now operating normally. We are finalizing the root cause analysis and will share a full report shortly.

Updated
Sep 10 at 09:00am EDT

We continue to encounter severe issues on the US region of Operations Center.

Updated
Sep 10 at 08:07am EDT

The increased capacity has restored service, but it remains slower than normal. We continue to implement additional measures to restore full performance.

Updated
Sep 10 at 08:05am EDT

The recovery is unstable and we are still observing issues. We are increasing the infrastructure's capacity to handle this load and to recover faster.

Updated
Sep 10 at 07:44am EDT

The issues encountered earlier appear resolved. We are monitoring the recovery and assessing the causes to implement permanent solutions.

Created
Sep 10 at 07:36am EDT

Operations Center in the US is currently encountering high error rates and may be inaccessible for some users. Our engineers are actively investigating the issue.