Avaya Experience Platform - OUTAGE - North America (NA) Data Centre - Inbound Interactions failing to reach Agents
Postmortem
PRB0048487 - North America (NA) Data Centre - All Inbound interactions failing to reach agents
Incident Start Time: Approximately 19:50 UTC
Incident Duration: 199mins (3 hours 31 minutes)
Summary:
On the 26th of June, between approximately 19:50 and 23:07 UTC, All inbound interactions we failing to reach agents (on Prod-NA cluster).
While investigating a monitoring alert indicating a potential service disruption, our Operations team discovered that omni-channel interactions were failing to reach agents. Subsequently, we received reports from multiple customers confirming the impact
Investigations & Findings:
We identified issues where several services responsible for routing interactions were unable to connect to our core back-end services. Upon further investigation, it was determined that the root cause stemmed from a platform-wide incident declared by one of our cloud vendors specifically affecting US regions. This incident resulted in disruptions across multiple services on the Avaya Experience Platform (AXP), which handles inbound interactions.
Resolution:
Issue was fully resolved once our Cloud Vendor rolled out a fix for their platform-wide incident.
Nest Steps:
Our Operations team is closely collaborating with our Cloud Vendor and will conduct a retrospective review to analyze the sequence of events and contributing factors.
Posted Jun 28, 2024 - 10:33 EDT
Resolved
We are pleased to inform you that this service outage with Inbound engagements affecting all Agent has been resolved by our Vendor.
We apologize for any inconvenience this outage may have caused and appreciate your patience and understanding as we worked to resolve the issue.
If you continue to experience any problems or have any concerns, please contact our support team
Posted Jun 26, 2024 - 19:43 EDT
Identified
We are currently experiencing a service outage affecting all inbound interactions due to a vendor patch pgrade. Our technical team is actively investigating the issue to restore service as quickly as possible.
Posted Jun 26, 2024 - 17:24 EDT
This incident affected: Avaya Experience Platform (AXP) (Voice - North America (NA), Email - North America (NA), Chat - North America (NA), Messaging - North America (NA), Analytics - North America (NA), Social Connections - North America (NA), API Access - North America (NA), Voice - South America (SA), Email - South America (SA), Chat - South America (SA), Messaging - South America (SA), Analytics - South America (SA), Social Connections - South America (SA), API Access - South America (SA), Voice - Europe (EU), Email - Europe (EU), Chat - Europe (EU), Messaging - Europe (EU), Analytics - Europe (EU), Social Connections - Europe (EU), API Access - Europe (EU), Voice - United Kingdom (UK), Email - United Kingdom (UK), Chat - United Kingdom (UK), Messaging - United Kingdom (UK), Analytics - United Kingdom (UK), Social Connections - United Kingdom (UK), API Access - United Kingdom (UK), Voice - AsiaPacific (APAC), Email - AsiaPacific (APAC), Chat - AsiaPacific (APAC), Messaging - AsiaPacific (APAC), Analytics - Asiapacific (APAC), Social Connections - AsiaPacific (APAC), API Access - AsiaPacific (APAC), Voice - Australia (ANZ), Email - Australia (ANZ), Chat - Australia (ANZ), Messaging - Australia (ANZ), Analytics - Australia (ANZ), Social Connections - Australia (ANZ), API Access - Australia (ANZ), Voice - Canada (CA), Email - Canada (CA), Chat - Canada (CA), Messaging - Canada (CA), Analytics - Canada (CA), API Access - Canada (CA), Voice - Japan (JP), Email - Japan (JP), Chat - Japan (JP), Messaging - Japan (JP), Analytics - Japan (JP), API Access - Japan (JP)).