The Smartsheet application is healthy.
On February 2, at 8:30 AM PST, a back end message queueing system experienced a failure that held connections open to the database as well as other systems. This stopped processing for all requests to the Smartsheet application causing an unplanned system outage.
Smartsheet’s operations and development teams immediately engaged to troubleshoot and remedy the situation. For the next two hours our engineers worked to determine root cause and return functionality to the system. Initially noticing the flood of open connections, a series of partial and full restarts of the front end application servers was attempted with no impact to functionality. Back end batch services were also halted to ease pressure on the system. During this time thread dumps were taken which lead to the discovery of a problem with the message queueing system.
At 10:20 AM PST our message queueing system was turned off in order to perform a rebuild of the queueing system. This allowed the core application to be available at 10:30 AM but updates to systems such as Search, Reporting, Resource Management, Webhooks, Connectors, and Email were delayed. The message queue was brought back online at 11:55 AM PST allowing new requests to these systems and restoring system availability.
Based on lessons learned from yesterday’s outage, Smartsheet will:
Add more monitoring on all core systems.
Further refine our operating and communications playbooks for addressing such issues.
Increase investments in reliability of components.
Add more redundancy to our network.
Feb 3, 12:09 PST
We're currently experiencing some delay issues with the search, reporting, and webhook functionality in Smartsheet. We're aware of the issue and are working to get it fixed right away. Thanks for your patience.
Feb 2, 10:27 PST
The recent technical issue has been resolved, and we are currently monitoring the Smartsheet application to make sure everything is operating normally
Feb 2, 10:27 PST
Our operations team is continuing to work on a resolution. Thank you for your patience.
Feb 2, 09:54 PST
We have identified the problem and are continuing to work to resolve the issue.
Feb 2, 08:58 PST
The Smartsheet Application is currently experiencing some technical issues. We are aware of the issue and are working to get it fixed right away. Thanks for your patience.
Feb 2, 08:29 PST