The incident was caused by a mismatch between the newly issued TLS certificate served by the affected endpoints and the SSL certificate/public key pinned in some white-labelled mobile applications.

Although the updated valid TLS certificate was deployed successfully, certain mobile apps continued to trust the previously pinned certificate or public key. This resulted in SSL pinning validation failures for those apps, causing mobile connectivity issues until the pinning configuration was updated.

All remediation steps have been completed. The SSL certificate pinning configuration has been updated across the required services, and access has been restored.

The remaining actions are:

App review and release
The updated mobile app build has been submitted for review. Once approved, the app will be released.
Post-release monitoring
Continue monitoring application availability, SSL handshake errors, mobile login/access issues, and customer-reported incidents after the app release.
Client communication
Inform impacted clients once the updated build is approved and released, and request them to update/upload the latest white-labelled app version where applicable.
Certificate rotation checklist
Maintain a checklist of all services and mobile applications using SSL certificate pinning to ensure they are validated during future certificate updates.
Preventive improvement
Review the SSL pinning approach and consider using public key pinning or backup pins where supported, to reduce the risk of similar issues during future certificate renewals or reissues.

The degradation was caused by an unexpected service instability triggered by changes in the runtime environment, which impacted the availability of a critical component in the production region.

1. Introduce additional safeguards and validation checks before runtime changes
2. Improve failover and recovery mechanisms to minimize impact duration
3. Schedule critical changes with enhanced validation and rollback strategies

As part of ongoing efforts to deprecate the legacy SSO integration, a production change related to the older SSO configuration led to service instability for clients still using the legacy authentication mechanism. This resulted in temporary access issues for those clients.

• Affected clients will be advised to migrate from the legacy SSO to the new SSO integration to ensure continued stability and support.

• The legacy SSO will be progressively deprecated in a controlled manner to avoid future disruptions.

• Additional validation and monitoring will be implemented during the deprecation process to minimize client impact.

During the recent release, a configuration issue in the new build resulted in an unintended app service plan change during the swap process.

As a mitigation measure, all pipelines were enhanced to perform app service swaps automatically, reducing the risk of manual configuration errors.