Tweet

By: Elliot Noss on October 11th, 2008
Posted in: OpenSRS Services
Tags: Email Service
Comments: 7

Dear Resellers,

I am writing today to speak to you directly about what happened this week with Cluster A of our Email Service. This will not refer to specific elements of the outage, there are other venues for that. The things I most want to communicate are my deep sorrow, why it won’t happen again and what we will do for you.

More than anything one thought keeps going through my mind as I think about this, the future determines the past. I will return to this thought.

First, and most importantly, we are sorry. I am sorry. I have been in this business a long time and do not know if I have ever been more sad about what we have done to you, to your customers and to how people think about us. An email outage in 1995 was different from one in 2000 and even more different from one in 2008. I know what this does to your reputations, to your customers and to your staff – and I and so many people here are just sad about that.

While it seems trite right now, we really define ourselves by how we make it easier for you in your businesses and with your customers and in our deep understanding of those relationships. That means the pain here is that much greater and believe me I know our pain here does not matter, yours does. Just know we are grieving.

Second, what will we do about it and why will this never happen again? I know for some of you that doesn’t matter, you are done with us, but I want to express this for the rest of you. Let me start here with things that were not the problem, old equipment, people, capacity or redundancy. The equipment is new, the people are great, we have plenty of capacity and redundancy. What this will mean for us is clearly the need to take the other elements of the service to a completely new level. Here I mean monitoring, change management, emergency protocols and procedures and operating efficiencies.

We had decided long before this that the most important part of email was reliability, not features, not groupware, not web 2.0 integration but reliability and deliverability. I have been at this a long time and really believe that these people and this service can be the best in the world, better than Google, Yahoo or Microsoft and most importantly the best partner for service providers. We owe you this and will deliver it.

Lastly, what we will do for you as a result of this? Let me start here by saying two things, we will certainly be doing something and that there is nothing we can do that will make up for your loss of reputation in your customers’ eyes. We know that. The people who will participate in that decision are fried right now, as I know even in your anger you can well imagine. I will ask your indulgence that you give us this week to make our plan in this regard.

There is one thing that I can offer now. I would like to make myself personally available to any of you who would like me to either reach out to your customers, or to any specific customer, with a letter, an email or a phone call. I know this will not often matter but perhaps in a few cases it might. My message here would be simple, this was our fault not yours and while you are responsible for the suppliers you pick, you had good reason to pick us and it was us who let you down. This offer stands whether you are leaving or staying.

In closing, the future determines the past. If we move forward and run the most reliable, service-provider focused, email service the world has ever seen this will be remembered as the few days that turned it around, as being a very important event in forging out mutual future. If we have no change in reliability or in service levels this will barely be remembered. It will just be a point on a mediocre line. I will do everything in my power to make it the former not the latter.

Regards,

Elliot Noss

Tweet

By: James Koole on October 1st, 2008
Posted in: OpenSRS Services
Tags: Email Service
Comments: 0

The underlying philosophy behind the statistics functions inside the OpenSRS Email Service is to provide the most flexibility possible. Rather than providing a small subset of stats to all users inside the Mail Administration Center (MAC), it was thought that our users would be better served by providing a full, extensive stats package, allowing you to take that raw stats data and manipulate or analyze it in whatever ways you like.

You’re free to take all that data, downloaded as .csv files and then import it into your own databases or spreadsheets (Microsoft Excel can read these files, for instance) and mash it up in whatever way that you wish. If you want to know what percentage of users prefer webmail, it’s in there. If you want to spot trends in how often users are logging in to check for mail, it’s in there.

You can even have those stats sent to you daily so you can create a system to automatically update and track usage across your email customer/user base.

We think that the stats engine inside OpenSRS Email satisfies the requirements of a wide range of potential users and uses, from support staff who need an easy way to see a particular user’s account activity, to the executive at a large ISP who wants to know how customers are accessing email.

But as I said, we wanted to provide flexibility for all customers and while that type of stats collection and distribution might work well for some of our larger customers, we also realize that some of our customers will want an easy way to look at certain stats without having to do any complex data mining. So we built in some simple, automatic graphing features for some of the most widely used of the stats we collect.

Inside the MAC, depending on your admin level, you can view stats for things like total logins, broken down across the various login types (POP, IMAP and webmail). You can view a range of stats for your entire company, or for a single domain, and even for a single user. You can see visually if a user hasn’t accessed their email in a long time or if there was a sudden spike in outgoing mail from a specific account or domain indicating a possible abuse situation. You can monitor things like quota usage and get a better understanding of how your customers use the email service.

To better explain the stats features within the MAC, we put together a quick screencast that shows some of the functionality that’s available to you:

Tweet

By: Ken Schafer on August 18th, 2008
Posted in: OpenSRS Services
Tags: Email Service
Comments: 2

First off, I’d like to apologize again for the problems that resulted from the problems last week on Cluster A of our email service. Email is a mission-critical service. We know how awful it is to have your personal and business communications disrupted. We are deeply sorry for any problems that resulted from this interruption.

After around-the-clock work last week to restore full service to our impacted resellers, and their end-users on Cluster A, our team took some time today to review what happened with last week’s service degradation.

Last Tuesday, a shelf controller hardware failure meant that 14 disks required a rebuild. This resulted in the degradation of multiple storage volumes. This failure affected 50% of customer mailboxes on OpenSRS Email Service – Cluster A. The restoration process was consecutive for the affected devices and therefore took a number of days to complete. To resolve the issue, we replaced the shelf controller and rebuilt 14 disks. During the service interruption, we made temporary mail stores available to customers. On Friday, once restoration was complete, all mail content (messages and folders) were merged from the temporary volumes to the user’s original mailbox.

As with any service problem of this magnitude, it is essential we take steps to make sure it does not happen again. Before the end of the month we are making storage architecture changes to Cluster A to ensure that we eliminate the chance that a similar event with storage will occur in the future.

Again, let me say that we are incredibly sorry about the impact this undoubtedly had on you and many of your customers.

Tweet

By: Ken Schafer on August 15th, 2008
Posted in: OpenSRS Services
Tags: Email Service
Comments: 0

Update: August 15, 16:00 P.M. ET:

On August 15, 2008: 16:00 P.M. ET, Full service to OpenSRS – Cluster A was restored. No mail or data was lost.  We truly apologize for this service disruption.

Update: August 15, 09:57 A.M.:

Regular updates are available at http://status.opensrs.com/

As of this morning, we can report that about 80% of users on Cluster ‘A’ have full email service. The remaining 20% of customers on Cluster ‘A’ have limited access, via webmail only. Additionally, we have updated our original estimate for restoration of service to all customers.

We now estimate that full service will be restored by Friday, August 15, 4:00PM ET (20:00 UTC).

Tweet

By: Ken Schafer on August 14th, 2008
Posted in: OpenSRS Services
Tags: Email Service
Comments: 0

We continue to provide regular updates on the progress of restoration at http://status.opensrs.com/

We’re pleased to report that more than two thirds of users on Cluster ‘A’ now have full email service meaning they can log in via POP, IMAP or webmail, as usual, to send and receive mail.

The remaining users that are still affected by this outage (about 1/3 of users on Cluster A) have been provided limited access, by webmail only, to their email. They can log into their webmail normally and send and receive email as usual, and view any messages received since the outage began.

We’re continuing to monitor the rebuild of the additional two storage volumes. Our current estimate is that full service to the remaining affected customers will be restored by Saturday, August 16, at 4:00 P.M. ET (20:00 GMT/UTC).

Once again, we deeply apologize to you and your customers that have been affected by this service interruption and we appreciate your patience as we work to restore full service to all users.