Delayed file processing for all ILM models in app.nanonets.com

Incident Summary

On 28th Jan 03:00 PM UTC, a secondary database migration unintentionally impacted indexes on a small number of core database tables. This led to degraded performance and temporary disruption of processing.

Impact
During this period, some prediction requests were delayed or unable to complete successfully until the database was stabilized.

Root Cause
A database schema change applied as part of a migration caused unintended modifications to critical indexes. While the migration itself completed, the index impact affected normal query performance on core tables because of high traffic volume.

Resolution
Our engineering team promptly identified the issue and initiated recovery actions, including restoring affected indexes and stabilizing database performance. Once recovery was complete, normal processing resumed.

Preventive Measures
To avoid similar incidents in the future, we are implementing the following improvements:

• Database migrations will no longer be executed during peak usage hours for secondary database as well.
• All future migrations will be scheduled during low-traffic windows with defined rollback plans.
• Additional pre-deployment checks and safeguards will be added to detect potential impact on core database structures.

We sincerely apologize for the inconvenience this incident may have caused. We understand the importance of reliability and take full responsibility for the disruption. Please be assured that we are taking concrete steps to ensure safer deployments and uninterrupted service going forward.