A certificate used by shared internal services passed its validity period, causing secure connections to fail. During replacement, the certificate chain initially could not be validated by some application services. This prevented the Modules page from retrieving the information it needed until the chain was corrected.

The certificate chain was corrected, and the affected service path was successfully validated after recovery. Our certificate monitoring, pre-cutover checks, staged rollout and rollback, customer-journey monitoring, and recovery controls were already in place and operating as expected. We will continue routine service and certificate monitoring. No action is required from customers.

A simultaneous increase in workload across multiple databases caused the combined demand to temporarily exceed the available shared capacity. Individual databases remained within their expected operating ranges.

We are enhancing our proactive capacity management and autoscaling controls to better account for simultaneous increases in overall demand. This will enable earlier scaling and reduce the likelihood of similar performance degradation recurring.

The root cause was an Azure Container Registry service disruption in Southeast Asia affecting image manifest retrieval.
During this period, appsvc-servicediscovery-sea attempted to start/restart using a container image hosted in the affected registry region. Because ACR manifest retrieval was failing or unstable, App Service could not reliably pull or validate the container image. As a result, the container did not become healthy, and Azure App Service front-end returned 503 Site Unavailable.
The issue first impacted servicediscovery-sea. Other services were not necessarily down because of their own container issue, but they were affected because they depend on Service Discovery for service lookup/routing. Once Service Discovery was unavailable, dependent services experienced high response time, failed calls, or degraded behavior.

Immediate recovery:
- Pointed appsvc-servicediscovery-sea to a known working image from an unaffected registry.

The new application build caused large analytics and report requests to create more concurrent database
processing than expected. Several operations then required substantially more compute and processing time than
normal, causing worker contention and queueing in the report processing path. The underlying platform and
storage remained healthy.

- Existing monitoring alerts identified elevated report completion times and processing pressure.
- The investigation correlated the start of the degradation with the newly introduced application build.
- The affected build was reverted immediately, and queued report work was cleared in a controlled manner.
- New report requests were validated after recovery to confirm that completion times had returned to the normal
range.

The new application build was reverted immediately after it was identified as the source of the degradation. Active
work was then allowed to drain in a controlled manner, which reduced contention and restored normal report
generation. Processing health, completion time, and queue behavior were monitored throughout recovery.

One India monolith API VMSS instance cpu went down. Because of this, requests queued and shifted to the remaining healthy instances, increasing load on them. The remaining instances became slow/overloaded, causing request cancellations, healthcheck failures, and HTTP 500 responses. Redis timeout logs were observed, but Redis metrics were healthy, so Redis was a secondary symptom and not the primary cause. The issue was resolved after reimaging the unhealthy instance.

Monitor VMSS instance health and load balancer health probe status more closely. Add alerting for unhealthy backend count and sudden traffic redistribution across instances. Review auto-healing/reimage automation for unhealthy monolith instances. Validate that failed/unhealthy instances are removed from rotation quickly. Continue monitoring Redis timeout logs, but treat them as application-side backlog symptoms unless Redis platform metrics show saturation.

The immediate cause of the incident was a failure in the internal autoscaling automation responsible for scaling regional
Azure SQL elastic pools. The automation service became unhealthy because one of its background maintenance
workers encountered memory exhaustion while processing a large recovery workload. This caused repeated application
process exits and health probe failures in the automation service runtime.
Because the autoscaling service was not healthy during the India incident window, scale-up actions that would normally
add database pool compute capacity were not completed in time. The affected pool therefore operated with insufficient
headroom while production traffic continued, causing elevated compute and worker utilization and resulting in slower
responses for some users.
The issue was not caused by a customer-side change, a specific tenant database, Azure SQL storage saturation,
transaction log saturation, or an Azure platform outage. The trigger was an internal automation service reliability issue
combined with insufficient regional pool headroom while the automation service was unavailable.

Optimize Autoscaler Background Recovery Processing: Avoid memory exhaustion by paging large recovery queries and preventing full materialization of large operation and log datasets. Status: In Progress.
Separate Autoscaling Control Path from Maintenance Workers: Ensure unrelated background maintenance workloads cannot impact the autoscaling function. Status: Planned.
Add High-Severity Alerts for Autoscaler Crash Loops and Failed Health Probes: Detect autoscaler unavailability before scale actions are missed. Status: In Progress.
Add Alerts for Skipped, Delayed, or Stuck Database Pool Scale Executions: Identify when expected scale actions are not completed within the required window. Status: Planned.
Increase Autoscaler Runtime Headroom and Review Replica Strategy: Reduce the risk of CPU and memory saturation in the automation service and improve service resilience. Status: Planned.
Define Regional Manual Scale Runbook and Escalation Path: Ensure rapid manual mitigation when automation is unavailable. Status: Completed.
Review Regional Pool Headroom Thresholds: Reduce the risk of recurrence during peak traffic while autoscaling is unavailable or delayed. Status: In Progress.
Post-Incident Monitoring Window: Continue monitoring the India pool after mitigation to confirm sustained stability. Status: In Progress.

The immediate cause of the incident was a failure in the internal autoscaling automation responsible for scaling regional
Azure SQL elastic pools. The automation service became unhealthy because one of its background maintenance
workers encountered memory exhaustion while processing a large recovery workload. This caused repeated application
process exits and health probe failures in the automation service runtime.
Because the autoscaling service was not healthy during the India incident window, scale-up actions that would normally
add database pool compute capacity were not completed in time. The affected pool therefore operated with insufficient
headroom while production traffic continued, causing elevated compute and worker utilization and resulting in slower
responses for some users.
The issue was not caused by a customer-side change, a specific tenant database, Azure SQL storage saturation,
transaction log saturation, or an Azure platform outage. The trigger was an internal automation service reliability issue
combined with insufficient regional pool headroom while the automation service was unavailable.

Optimize Autoscaler Background Recovery Processing
Action: Avoid memory exhaustion by paging large recovery queries and preventing full materialization of large operation and log datasets.
Status: In Progress
Separate Autoscaling Control Path from Maintenance Workers
Action: Ensure unrelated background maintenance workloads cannot impact the autoscaling function.
Status: Planned
Add High-Severity Alerts for Autoscaler Crash Loops and Failed Health Probes
Action: Detect autoscaler unavailability before scale actions are missed.
Status: In Progress
Add Alerts for Skipped, Delayed, or Stuck Database Pool Scale Executions
Action: Identify when expected scale actions are not completed within the required window.
Status: Planned
Increase Autoscaler Runtime Headroom and Review Replica Strategy
Action: Reduce the risk of CPU and memory saturation in the automation service and improve service resilience.
Status: Planned
Define Regional Manual Scale Runbook and Escalation Path
Action: Ensure rapid manual mitigation when automation is unavailable.
Status: Completed
Review Regional Pool Headroom Thresholds
Action: Reduce the risk of recurrence during peak traffic while autoscaling is unavailable or delayed.
Status: In Progress
Post-Incident Monitoring Window
Action: Continue monitoring the SEA regional pool after mitigation to confirm sustained stability.
Status: In Progress

The root cause was an Azure platform-side availability degradation on the public Azure Load Balancer
alb-monolith-india-api, which fronts the backend service path used by the Application Gateway apipool
configuration. Azure Resource Health classified the event as Unavailable / Downtime, PlatformInitiated,
Unplanned, and Transient.
As the Load Balancer data path degraded, Application Gateway health and routing for the apipool
upstream became unavailable. The gateway therefore returned 502 responses with
ERRORINFO_UPSTREAM_NO_LIVE instead of forwarding those requests to the backend service
instances.
Backend VM availability metrics did not show a regional compute outage as the initiating condition.
VMSS autoscale activity occurred after the Load Balancer health event had already started, and
automatic VM repair activity occurred after service recovery was already underway. This timing
indicates that autoscale and repair activity were response or recovery signals, not the initial root cause.

The India region availability incident is resolved. Monitoring remains active for Application Gateway
failed requests, backend health, Azure Load Balancer availability, VMSS availability, and regional
synthetic probes.

The issue was caused by an Azure-side underlying storage dependency failure, which degraded backend operations and resulted in increased server errors for Azure Container Registry requests.

Azure has mitigated the issue, and we will monitor the service.

The incident was caused by a mismatch between the newly issued TLS certificate served by the affected endpoints and the SSL certificate/public key pinned in some white-labelled mobile applications.

Although the updated valid TLS certificate was deployed successfully, certain mobile apps continued to trust the previously pinned certificate or public key. This resulted in SSL pinning validation failures for those apps, causing mobile connectivity issues until the pinning configuration was updated.

All remediation steps have been completed. The SSL certificate pinning configuration has been updated across the required services, and access has been restored.

The remaining actions are:

App review and release
The updated mobile app build has been submitted for review. Once approved, the app will be released.
Post-release monitoring
Continue monitoring application availability, SSL handshake errors, mobile login/access issues, and customer-reported incidents after the app release.
Client communication
Inform impacted clients once the updated build is approved and released, and request them to update/upload the latest white-labelled app version where applicable.
Certificate rotation checklist
Maintain a checklist of all services and mobile applications using SSL certificate pinning to ensure they are validated during future certificate updates.
Preventive improvement
Review the SSL pinning approach and consider using public key pinning or backup pins where supported, to reduce the risk of similar issues during future certificate renewals or reissues.

The degradation was caused by an unexpected service instability triggered by changes in the runtime environment, which impacted the availability of a critical component in the production region.

1. Introduce additional safeguards and validation checks before runtime changes
2. Improve failover and recovery mechanisms to minimize impact duration
3. Schedule critical changes with enhanced validation and rollback strategies

As part of ongoing efforts to deprecate the legacy SSO integration, a production change related to the older SSO configuration led to service instability for clients still using the legacy authentication mechanism. This resulted in temporary access issues for those clients.

• Affected clients will be advised to migrate from the legacy SSO to the new SSO integration to ensure continued stability and support.

• The legacy SSO will be progressively deprecated in a controlled manner to avoid future disruptions.

• Additional validation and monitoring will be implemented during the deprecation process to minimize client impact.

During the recent release, a configuration issue in the new build resulted in an unintended app service plan change during the swap process.

As a mitigation measure, all pipelines were enhanced to perform app service swaps automatically, reducing the risk of manual configuration errors.