Incidents and maintenance reported on status page for Vemco Group https://status.vemcogroup.com/ https://d1lppblt9t2x15.cloudfront.net/logos/aadb8d0ee0bd75ab3b1d8c83baf54cb4.webp https://status.vemcogroup.com/ en https://status.vemcogroup.com/maintenance/1074873 Sat, 26 Sep 2026 21:25:00 -0000 https://status.vemcogroup.com/incident/1074873#3e576e4f019d82cf0b51edfde6735e08c8b14f0bf4487e2dc2ed88a260d3baba Maintenance Maintenance completed https://status.vemcogroup.com/maintenance/1074873 Sat, 26 Sep 2026 21:00:00 -0000 https://status.vemcogroup.com/incident/1074873#da10844cf38260e8c340fce313c553ca4bfabf2a03b795f66ce672f2dc3e0dda Maintenance Upgrading databases to latest version and security patches https://status.vemcogroup.com/incident/1037521 Fri, 28 Aug 2026 11:56:00 -0000 https://status.vemcogroup.com/incident/1037521#53b93726da520bddbda537509f8d4f84cc39833287e082962c0919fa6c932b39 Incident We experienced an incident caused by a migration on the large `demographics_data` table. The migration held a database lock for more than 10 hours, which caused dashboard widgets and some API calls to function incorrectly for most of the day. Once the migration had started, stopping it was not safe. Interrupting it could have triggered a lengthy rollback, caused additional downtime, or left the database schema in an inconsistent state. We have added precautions to prevent future migrations from running directly on large tables without proper review and preparation. No data was lost during this time, as sensors resend until a successful delivery has been made. https://status.vemcogroup.com/incident/993668 Mon, 03 Aug 2026 06:49:00 -0000 https://status.vemcogroup.com/incident/993668#cdc04b808bc35a1a60055ac1dc69336e6bef680576d3e0426b4ccba0e5289314 Incident One of our main domains certificate had expired after failing to auto-renew. This caused a downtime of 3 hours. The issue has since been resolved permanently, and fallbacks have been added so that the incident doesn't re-occur. Data from sensors will be re-tried until successfully sent to servers, so there should not be a need to do anything to get the historical data into the system. https://status.vemcogroup.com/incident/760070 Fri, 07 Nov 2025 19:12:00 -0000 https://status.vemcogroup.com/incident/760070#88a68d3d12db1c86e6c13baf44a70899213d45e233f7df7d0a32846fb9706097 Incident We have found that an onld deployment had been pushed which caused issues with our redis server, forcing it to halt. Once we saw that the old image was deployed, we re-deployed the recent one and redis could get back on it's feet again. We will continue to monitor for the next hours, and deploy a fix so that this issue does not repeat itself. https://status.vemcogroup.com/incident/722986 Fri, 12 Sep 2025 06:02:00 -0000 https://status.vemcogroup.com/incident/722986#edc66e6475463a3e0d29579fd7b75556eb7130443d0e868ee7ab5531e4bce017 Incident Redis servers got slowed down making them reach OOM state, resulting in clients not being able to connect to queue to empty the jobs from it. Server has been re-started allowing clients to connect again to empty the queue, no data has been lost, all jobs were processed. https://status.vemcogroup.com/incident/712023 Sat, 23 Aug 2025 11:56:00 -0000 https://status.vemcogroup.com/incident/712023#da73e5392267572cc4268e5ae318930588c82f7ec8ce08a0f254acdc18cc0fab Incident We found that a node (1 out of 8) was noon responsive and causing the blank pages to show, once removed from the scale cluster system went back to normal. https://status.vemcogroup.com/incident/615402 Sun, 06 Jul 2025 07:02:00 -0000 https://status.vemcogroup.com/incident/615402#bc1064ed5048d407543d6f8d926ee3d9cbbab19992f4ed0ad07ce677b5eaea6f Incident SSL had expired and auto-renewal had failed, this has been fixed so that future renewals will not fail again. https://status.vemcogroup.com/maintenance/411524 Fri, 09 Aug 2024 18:00:00 -0000 https://status.vemcogroup.com/incident/411524#35728e5fd745737596b337706e5b84d328fddc6121ec8cda32857cdf1b164a47 Maintenance Applying security updates to DB, Cache servers.