On 25/11 at 09:50, the core Point-of-Sale (POS) and backend API services experienced a severe performance degradation, leading to transactional timeouts and a critical slowdown across retail operations.
The incident was triggered by a corruption of a primary index on the MongoDB collection. This caused queries to bypass the fast index and perform costly full collection scans, which rapidly consumed all available CPU and I/O resources.
The resulting database congestion caused the application to have timeouts.
The issue was resolved by rebuilding the index, restoring full service. Because of the size and load this took more than half an hour.
We sincerely apologize for the disruption and the resulting inability to process transactions efficiently for the duration of the incident. We understand that peak performance is critical to your daily retail operations, and this incident fell short of the reliability standard you expect from our product.
We confirm that customers utilizing the offline functionality were able to continue processing sales locally. All transactions recorded offline during the database degradation period were successfully synced to the central database immediately after the service was restored, ensuring no loss of sales data.
Thank you for your patience during this incident.