Previous incidents

We experienced an incident caused by a migration on the large demographics_data table. The migration held a database lock for more than 10 hours, which caused dashboard widgets and some API calls to function incorrectly for most of the day.

Once the migration had started, stopping it was not safe. Interrupting it could have triggered a lengthy rollback, caused additional downtime, or left the database schema in an inconsistent state.

We have added precautions to prevent future migrations ...

One of our main domains certificate had expired after failing to auto-renew. This caused a downtime of 3 hours.

The issue has since been resolved permanently, and fallbacks have been added so that the incident doesn't re-occur.

Data from sensors will be re-tried until successfully sent to servers, so there should not be a need to do anything to get the historical data into the system.