Proton explains the technical issues that caused the recent ProtonMail outages

Last week was a tough one for Proton users, the company experienced multiple outages spread across 3 days. Yesterday, Proton sent out an email to its users to explain the issues behind the disruption of its services.

I didn't think the issue was a big deal, as I never noticed any downtime, maybe because I'm in a different time zone than the users who had reported about the outage. That's why I didn't write about it here, because I didn't experience the issue first hand. Some users wrote that they couldn't access Proton VPN and ProtonMail, while some could use the VPN.

When I saw new complaints about another outage the next day, I was quite surprised. Though I could still access ProtonMail, I noticed that the site was slower than usual. I thought it was trying to connect but failed, but it worked after several seconds. Interestingly, or worryingly, the issue recurred for a third consecutive day, and I reckoned this could be a major problem. I don't think I've ever seen an email provider experience so many outages in such a short time frame. You can check the logs at Protonstatus.com for more information.

Image via reddit

Many users were concerned about these outages, the biggest complaint was that they were unable to access their mails, receive new messages, or reply to them. There was another question that had been raised, has Proton been hacked or under attack? Would this result in loss of mails? The short answer is no, the company was experiencing some technical difficulties.

What caused the ProtonMail outage?

As you may know, Proton had been working on various improvements for its products. We just noticed the changes on the front-end, but it appears that the company had also been working on upgrading its infrastructure by adding new servers, and new software with better performance and higher storage capacity. Proton says that it upgraded its servers that were running its core database to a new version on July 10th.

The software had passed the quality checks without any hiccups, before the main database was upgraded. But, it turns out that the core database, which has higher activity than others, was overloaded after the upgrade. High user activity isn't something that could be simulated, which might explain the tests could not predict the outcome. The security system, which Proton had in place, noticed the high activity, and misinterpreted it as a threat. So, it activated some protective measures (on July 11th), which in turn caused a roadblock for users who were trying to log in to their Proton accounts, including users who had previously logged in on the web and on their mobile apps.

Proton states that it solved the problem by splitting the load of its database over multiple servers, and slowed down lower priority services like push notifications. The company had to add more servers, and eventually managed to restore the stability and performance of its systems. Proton has promised to publish more details about the outage on its blog this week.

I don't use Proton as my main email provider, so I wasn't really impacted by the outage. Some users may not be satisfied by the company's approach to fix the issues, after all it did take 3 days to sort out the problems completely. If I were a paying customer who relied on ProtonMail, I would have certainly been annoyed by the downtime, and not being able to get to my receipts, documents, etc.

That's why I think it's good it's important that Proton sent out the mail to users as an apology and explain what happened. It shows that they are willing to take responsibility for the issues that they inadvertently created. The transparency would be appreciated by users, but it is also a neat way by the company to reassure users that they weren't hacked, and that their data is safe.

Were you affected by the Proton outage?

Proton explains the technical issues that caused the recent ProtonMail outages