First we sincerely apologize for the disruption to your operations on 05/02 morning. We know that in retail, every second at the checkout counts, and we regret the frustration this caused your teams and customers.
What happened?
We recently introduced an update designed to make the system even more resilient. It included a "fallback" feature, essentially a safety net, to catch and fix rare errors that occasionally caused a sale to fail.
While this update performed perfectly in our testing environment for several weeks, a very specific and unusual combination of data occurred in the live environment yesterday morning. This specific data "confused" the safety net, causing the service to shut down unexpectedly.
Why did it spread?
Our system is designed to be "self-healing." If one part goes down, another takes over immediately, and the first part restarts itself within a minute or two. However, because the "bad data" was still being processed, it triggered the same crash across all our backup systems one after another. This created a loop where the services were restarting, but being knocked offline again before they could fully recover.
How are we preventing this?
We are committed to providing a stable platform for your business and are treating this as a priority learning moment for our engineering team.