Incident Report: June 2nd

Onlineboard experienced around a 4-hour outage on 2nd of June between 10:30 - 14:00. This happened due to power outage in data center. We are sorry for the inconvenience, and we are going to take measures to prevent this from happening in the future.

Situation

Today (2nd of June) at around 8am a power outage occurred in Prague, where our servers are located. The data center which hosts our servers is connected with two separated power lines, so this did not affect our services at first.

The second problem occurred around 11am when even the datacenter's second power line also lost power. Moreover, the diesel generator that are in data center were unable to start, so the data center completely lost power.

The data center completely recovered their infrastructure at around 13:40.

Plans for future

While our core infrastructure is hosted in a different data center, and was working during the whole outage, users were unable to use Onlineboard as we host our webpage and synchronization server in the Prague data center.

To prevent this from happening in the future we are taking two steps in the near future.

We are going to move hosting of webpage to a distributed static site hosting such as Cloudflare Workers. This should make the webpage practically outage-free. Regarding the synchronization server which is also hosted in Prague data center, we are going to prioritize launching multiple sync servers in different data centers, so if one becomes disconnected from the internet burns down Onlineboard can still continue to run.

To recap this incident started 2 Jun 2022 at 10:47am CEST and was resolved 2 Jun 2022 at 02:04pm CEST (taking about 3 hours and 14 minutes). Once again we are really sorry for this.