We experienced an incident caused by a migration on the large demographics_data table. The migration held a database lock for more than 10 hours, which caused dashboard widgets and some API calls to function incorrectly for most of the day.

Once the migration had started, stopping it was not safe. Interrupting it could have triggered a lengthy rollback, caused additional downtime, or left the database schema in an inconsistent state.

We have added precautions to prevent future migrations from running directly on large tables without proper review and preparation.

No data was lost during this time, as sensors resend until a successful delivery has been made.