Platform connectivity issues

Incident Report for Textlocal

Postmortem

Some customers may have experienced issues with connecting to Messenger and other platform services over the last 24 hours especially so on the 17th November 2020. The issues experienced were caused primarily by capacity issue at a server level.

Typical errors encountered would have been:
503 Service Unavailable server error
504 Gateway Timeout Error

With this issue identified, we adjusted the resources available on the morning of 18th Nov and again that afternoon. With these adjustments we observed increased capacity and stability of the platform, which has continued into the 19th November

We will continue to monitor and will react accordingly.
We can only apologies for any inconvenience caused.

Posted Nov 19, 2020 - 14:46 GMT

Resolved

Thank you for your patience on this issue.
We are continuing to closely monitor the platform.

Posted Nov 19, 2020 - 14:44 GMT

Update

We will continuing to monitor the platform and its stability and will do so throughout the remainder of the day - as a result, we will keep this incident open as "monitoring" - Updates to follow later this afternoon.

Posted Nov 19, 2020 - 11:30 GMT

Update

We are continuing to monitor the platform, and don't expect any further issues. We will continue to monitor the platform through the night and into tomorrow - as a result, we will keep this incident open as "monitoring" and only close it once normal traffic levels resume tomorrow morning.

Next Update: An update will be provided by 12 noon tomorrow (Thursday 19th)

Posted Nov 18, 2020 - 20:05 GMT

Update

Additional resources and configuration changes have been applied stabilising the platform. We are continuing to monitor.
Next Update: An update will be provided within 1 hour or at resolution.

Posted Nov 18, 2020 - 17:45 GMT

Update

We are aware that this issue is still persisting for some of our users. We are investigating further working with our data centre team to understand what is causing this.

Posted Nov 18, 2020 - 14:24 GMT

Monitoring

Following the connectivity issues experienced this morning, a number of configuration changes have been applied to the platform, and we are seeing things have stabalised. We are continuing to work with the data centre provider, to apply further permenant fixes to this, but we don't expect to see any further customer issues.

We apologise for any inconvenience caused, and assure customers that we take all issues seriously, and deal with them as quickly as possible.

If you have any further queries, then please email support@textlocal.com

Posted Nov 18, 2020 - 13:40 GMT

Investigating

Unfortunately, some users may be experiencing connectivity issues with our platform. We are working with our data centre team to understand what is causing this.

Impact: Messenger Portal and APIs may be intermittently unavailable.

Next Update: An update will be provided within 1 hour or at resolution.

Posted Nov 18, 2020 - 11:13 GMT

This incident affected: International SMS (International SMS Delivery, International Inbound Messages), UK SMS (UK SMS Delivery, UK Inbound Messages, UK Delivery Receipting), Textlocal API (Send SMS from API, Email2SMS, FTP/sFTP, API Reporting, Delivery Receipting to Remote, API Documentation), UK MMS (UK MMS Delivery, UK MMS Inbound Messages), and Textlocal Messenger (Sending SMS from Messenger, Payment Pages, Sending MMS from Messenger, SMS Reporting).