Hello all,
Thank you for your patience with this situation.
Now that we have a full picture of what occurred and why, here is a summary for RMS HTTPS and RMS API outage during 2022-08-06 and 2022-08-07 weekend.
The issue:
The RMS web interface was inaccessible, RMS API calls could not be used.
The root cause of the issue:
Failure to update HTTPS certificate, which resulted in inaccessible RMS UI and unusable API, since API calls use the same URL and certificate as HTTPS connections.
There were, however, several approximate causes, leading to fail-safes not being triggered, which then resulted in delayed response in bringing the services back and communication with our customers.
Current status:
RMS services have been restored at ~07:40 UTC+3 on Monday, 8th of august.
No user data has been lost or compromised; RMS alerts, generated remote access links, reports, data collection from routers continued working throughout the weekend, as these services were not affected by the HTTPS certificate expiration.
Even after the underlying issue was rectified, some users have still had trouble logging in to the platform, receiving an “Error 401: UNAUTHORIZED” message once logged in. This is part of the same issue - leading to sessions becoming invalid or expired. This issue can be resolved by clearing your browser data up to around the point in time when the issue began (August 6th).
What actions shall we take:
Since early Monday we are conducting a thorough audit of all procedures and redundancies that control and support RMS uptime and its monitoring. Once this audit is finished, we shall take all necessary preventative actions to ensure that similar outages do not repeat in the future.
Once again, thank you for your patience and continued trust throughout this episode.
