Summary
On Monday 18 May 2026 around 13:00 (local time), our production database became unavailable for 26 minutes.
The cluster recovered automatically and was fully back in service by 13:43.
No data was lost.

Background
Our database runs as a three-node cluster: three identical copies of our data, kept in sync across three separate servers.
One node is the primary - it accepts new data. The other two are secondaries - they continuously copy the primary.
At least two of the three nodes must be online for the cluster to keep accepting data. If only one remains, the cluster intentionally stops, to prevent the servers from disagreeing on what the latest data is.

What happened at around 13:05, the primary node started to slow down and stopped responding to the other two. Within seconds, the cluster automatically promoted one of the secondaries to be the new primary,a routine failover that should have been invisible to users. This time the failover did not complete cleanly. The new primary inherited a large backlog of unfinished work from the old primary and became overloaded while trying to clear it.

At 13:14, the new primary hit a built-in safety mechanism : when a node is stuck and cannot complete its role change within 30 seconds, it shuts itself down so the rest of the cluster can move on. About 30 seconds later, at 13:15, the original primary, still trying to step down, hit the exact same safety mechanism and also shut itself down.

With two of the three nodes down at the same time, the cluster fell below the two-node minimum and stopped accepting reads and writes. Recovery automatically restarted both failed nodes. They came back online around 13:40, a new primary was elected, and full service was restored at 13:43, with no manual intervention required.

Next steps
We are working together with our database supplier to identify the underlying root cause, it could be hardware related.
A follow-up will be shared once those findings are confirmed