FlexIP Solutions - Incident - Outage - (M) – Incident details

Good afternoon,

We sincerely regret and apologize for the service interruption on Monday, October 31, 2022. We do not take service interruptions lightly and understand the impact they can have on your business.

Below is a breakdown of the reason for outage (RFO), including the root cause, and effect it had on services and the solution we implemented.

RFO for 10.31.2022

Root Cause:

We have analyzed the root cause with Microsoft/Metaswitch (the switch manufacturer) and determined that there was a software bug. The switch has a self-protection mechanism built in that can be triggered for various reasons. In a normal scenario, no effects are encountered by end users. Unfortunately, on 10/31, when the switch performed this self-protection function, it did not operate as designed. This resulted in a full restart of the switch.

Effect:

Following the restart of the switch, a large amount of voice traffic simultaneously attempted to re-subscribe. Microsoft/Metaswitch previously put in place restrictions to prevent this type of traffic from overwhelming the switch in the event of a restart. This programming functioned but not entirely as intended.

Solution:

Microsoft/Metaswitch applied a software patch to address the switch protection function that failed. Furthermore, restrictive settings have been put in place to prevent voice traffic from overwhelming the switch in the event of a restart.

Should you have any further questions please contact our support team at support@flexipsolutions.com.

Thank you for your continued business.

The FlexIP Solutions Team

All operations have been restored and continue to be monitored by Microsoft/MetaSwitch Engineers.

An RFO has been requested and will be sent out once received.

The systems are operational. However, there is a backlog of activity that still needs to be processed.

As the activity continues to clear, we see more clients coming back online.

Microsoft/MetaSwitch continue to monitor the progress.

Please standby for more updates as they are available.

Thank you!