Update
Root Cause and Resolution
At 5:18 AM GMT, during a minor maintenance update, meant to improve run time security, a human error resulted in wrong configuration setting which prevented user operations. This issue was detected immediately and the configuration was corrected to resolve the issue within 29 minutes.
Corrective and Preventive Measures
We are taking the following steps to prevent recurrence.
- Reviewing deployment process to ensure such issue is caught during internal testing.
- Auditing other critical configuration scripts.
- Adding additional input validation checks in scripts.
Apr 16, 07:25 GMT
Fixed - The issue has been fixed and the service is operational.
Apr 16, 05:47 GMT
Identified - We have identified the issue and are currently working on the resolution.
Apr 16, 05:22 GMT
Investigating - We are investigating reports of warnings showing up on CRM pages.
Apr 16, 05:18 GMT
