We want to provide an update on our effort to restore the level of service that you and your customers had become accustomed to prior to the recent outages affecting OpenSRS Email Cluster B.
As mentioned in the Incident Report for the latest degradation of service, we’ve begun moving customer mailboxes off the troubled storage infrastructure and onto the new hardware (NAS).
Current migration status:
At this point, about 20% of the total customer mailboxes that were on the troubled storage infrastructure have been moved over to the new NAS cluster(s). When you count only customer mailboxes located on the specific head that was failing and causing the degradation of service, that percentage of customer mailboxes moved is about 40%.
We’re focused on first moving all customer mailboxes on the head that was specifically causing all the issues. Once those customer mailboxes are moved, we’ll continue with customer mailboxes located on the other head. It’s worth noting that all new accounts are being provisioned directly onto the new NAS hardware.
Next steps:
We’re continuing to migrate the remaining customer mailboxes off the affected hardware as quickly as possible.
Those migrations are being done in a very measured way to ensure that we maintain the high level of service and availability that we aim to provide. The speed at which we can migrate customer mailboxes is dictated by the load on the Cluster at any given time. We’re throttling the migration process to move customer mailboxes as quickly as possible, with as little risk as possible.
Additional hardware to create two further NAS clusters has now been installed and is ready to enter service. Once the first NAS cluster is at capacity, we’ll begin migrating customer mailboxes to the second NAS cluster, followed by the third, as each NAS cluster is filled to approximately 40% of the rated capacity (what we consider “full”).
It’s important to note that as we move customer mailboxes off the troubled hardware, we also reduce the overall load on it. We expect that the decrease in load on the troubled storage infrastructure will lead to improved reliability for those customer mailboxes still on that storage system.
Migrations of customer mailboxes will continue over the coming weeks. We estimate that all customer mailboxes will be migrated off the older hardware sometime late next week (October 18-20). The remaining customer mailboxes on the other head will be moved off after that and we estimate all customer mailboxes will be off the problematic hardware sometime in early to mid-November.
We’ll provide further updates on our progress until we have all customer mailboxes migrated. Our aim is to keep you in the loop with posts at least every week to ten days and as we reach significant milestones.