OpenSRS: Reseller Friendly since 1999
 

Author Archive

Getting Customers Using Email Could Be Key To Revenue Success

Not only does selling value add services along with hosting increase your revenue, it can also reduce future churn:

For example, let’s look at email. When Parallels analyzed the churn rates among web hosters for shared web hosting services, we find that the churn rates for small business customers without email are significantly higher than those with email, especially in year one!

via Becoming a full service provider – the key to reducing churn | WHIR Web Hosting Blog”.

Slate.com Gets Domains All Wrong

The popular press (do we still call it that?) often misfires when covering domain names. I’ve seen enough articles about the perils of “cybersquatters” and “domain hijackers” that I’ve become somewhat immune to the misrepresentations of our industry.

And yet, as I saw Farhad Manjoo’s article in Slate.com last week (www.thosenewdomainnames.areforsuckers) get traction – including the front page of Techmeme – I couldn’t resist the opportunity for some serious fisking.

Here’s goes…

“Now ICANN, the international body in charge of domain names, says it has a way to rid the Web of cybersquatting. Late last month, the group voted to create Web addresses that end in a much wider variety of letters than .com, .org, .net, and the dozens of country-specific suffixes that are currently available.”

Manjoo seems to be conflating two different issues. ICANN does indeed intend to introduce newTLDs but the goal is increasing diversity in the domain name space rather than eliminating cybersquatting. Some businesses objected, fearing that newTLDs would increase cybersquatting and essentially force trademark holders to preemptively buy all these newTLDs to protect themselves.  The response was the hotly debated IRT/URS process.

“When the proposal goes into effect later this year, businesses, municipalities, and other large organizations will be able to purchase domains of their own creation. The city of New York could buy its own suffix—to get to a city site, you’d type Police.nyc or Fire.nyc, and you’d e-mail Michael Bloomberg at Mayor@cityhall.nyc.”

Yes, it will be possible for companies to go to ICANN to get their own TLDs, but they’ll be owning the ENTIRE namespace for that TLD if they do that. Once they do own the TLD it is up to them to decide how domains within that TLD are apportioned.  Some TLDs will sell domains without restriction at prices similar to a regular domain today while others will set criteria for acceptance and still others will keep the TLD all to themselves.

So some people using new domains will pay a VERY large amount (for the entire TLD) while others will pay the going market rate.

“But ICANN’s plan comes about five years too late—cybersquatting isn’t a problem anymore.”

Not so. People still register domains that are precariously close to other popular sites or that incorporate trademarked terms in confusing ways. In fact the total number of UDRP filings increased last year (as did the number of registrations), so it’s hardly fair to say this isn’t a problem anymore.

“Indeed, ICANN’s plan to sell all these new top-level domains at very high prices—tens of thousands of dollars or more—seems like a scam, because domain names themselves just don’t matter that much nowadays.”

This is my real beef with the article.  Let’s look at the case he builds for this claim…

“Web browsers have gotten a lot smarter since the 1990s, and they’re now pretty good at determining what we want when we type in names that have many possible meanings. If you’re a fan of the Slate private-party venue in New York and visit its site often, you’ve just got to type S-L-A into your browser’s address bar and the site will pop up in a drop-down list. That Slate would be foolish to pay very much to buy Slate.party.”

Well, yes, auto-complete is a great boon to web users everywhere but that has nothing to do with domain names. It’s about getting them there in the first place so that auto-complete will complete to them.

Besides, on subsequent visits the domain STILL matters as the auto-complete is based on the domain anyway.  If “Slate” was at fdfsfsdfsdfsdfd.info then typing “S – L – …” isn’t going to help you.

“What’s more, lots of people now abandon the address bar entirely and rely, instead, on search engines to get around the Web. How do folks get to Match.com? According to Web traffic analysts, people type Match.com into Google and then click the top result. Are these people stupid? No, they’re smart: It takes a lot of work to remember every company’s exact domain name (is General Motors at GM.com or GeneralMotors.com or General-Motors.com?) and it’s much faster to let Google keep track. Chrome, Google’s Web browser, combines the address bar and search bar into a single field, which lets you use search terms as Web addresses. You don’t have to remember Josh Marshall’s long URL—Talkingpointsmemo.com—to get to his blog. Just type in josh marshall, and Chrome displays Google’s top results.”

Well, yes, search is becoming more effective all the time and the number one result in major search engines quite often does have the information we want.  But that’s not the point. People buy domains to be found when they are NOT the top search result. Imagine you’re some OTHER Josh Marshall.  How will people find you?  Being joshmarshall.me or theotherjoshmarshall.com might be the way to go.

And don’t forget that search engines have to figure out what to show as the top results and a relevant domain name matters.

Yes, SEO is important but so is a domain name. It’s not an ‘either or’ kind of thing.

“To be sure, cybersquatters are still plying their trade, and according to trademark experts commissioned by ICANN, domain-name disputes have lately been on the rise.”

Uh, then why did you say it wasn’t a problem anymore?

“At the same time, though, you see Web sites getting much more adventurous in the domain names they pick—look at the Lolcats site Icanhascheezburger.com or the social-bookmarking site Del.icio.us (which later changed its name to Delicious.com). These names suggest a nonchalance about URLs. It no longer matters whether a domain name is really long or has an unconventional spelling; people will be able to find it, anyway.”

Oh dear.

Funky Web 2.0 names have in no way helped these sites. Yes, a few have flourished in spite of their odd names but that doesn’t mean it’s a good idea to follow their lead.  This is a case of “survivorship bias” – thousands (maybe hundreds of thousands) of sites have been launched at sub-optimal domains that have hindered them considerably. The fact that a few made it to the big time is irrelevant.

Interestingly, Manjoo’s parenthetical note that del.icio.us changed it’s name to delicious.com just proves the point.

“And for cybersquatters, there are now other places to play. Social-networking sites are now the Web’s biggest properties, so getting your identity on Facebook or Twitter has become much more important than getting a good domain. Recently Facebook offered its users vanity URLs—e.g., www.facebook.com/farhad.manjoo—on a first-come, first-served basis; the addresses were snapped up at a rate of more than 500 per second.”

Do you want your domain name to market you or Facebook? What happens when you tire of Facebook (like we tired of MySpace, and Friendster, and GeoCities and AOL before that)?  Owning your domain name lets you point at all the interesting places you might be hanging out online.  Does anyone want to print business cards with facebook.com/mynickname on them when they could spend a few dollars and own a name they want forever?

“Twitter, meanwhile, has become a haven for imposters. The site has had to close down accounts impersonating Exxon Mobil, Kanye West, and my colleague Emily Bazelon, among many others. Twitter has vowed to become more vigilant in its fight against poseurs, and surely it will implement a plan to do so. Because Twitter has total control over its names, it can deal with squatters much more quickly than is possible on the domain-name system, which is administered by thousands of registrars across the world.”

But what recourse to you have when Twitter (or any site) arbitrarily shuts down your account and/or gives your name to someone else? None. This to me is the equivalent of saying “why bother with these complicated laws and lawyers, judges and juries, why not just let the King decide”.

But squatters wouldn’t get very far even if Twitter never got its act together. Last year, someone got on Twitter and began tweeting as Shaquille O’Neal. When the real Shaq got wind of the faker, he didn’t offer to pay for his identity; rather, he set up another name—The_Real_Shaq—and set the record straight. Now, it no longer matters that Shaq doesn’t own his Twitter name; when you Google Shaq Twitter, The_Real_Shaq comes up first (he’s got more than 1.5 million followers). We all should follow Shaq’s example—don’t ever pay for a screen name or a domain name again.

.basketball!

Preview the New OpenSRS System Status

We’re excited to show you a preview of our new System Status communications tool.

Our goals in redesigning System Status were:

  • Provide a new, simplified and easy-to-use status webpage
  • Offer multiple subscription methods
  • Deliver enhanced technical details about every service incident

This new System Status will allow us to provide richer technical details for every maintenance window and service status update. The new System Status, built using the open source blogging/CMS platform: WordPress, uses blog-like functionality to provide rolling updates to give you the whole story of an incident from start to finish. At the end of every service incident, we will provide you with an incident summary. We promise to provide you with the detail about what we did to respond to an issue and what we are doing in the future to prevent a similar issue from happening. We’ve added some historical incidents to the new site so that you can see some examples illustrating the change in our communication style.

We’ve been adding content to the new System Status for about a month and we will continue to simultaneously post using both sites until the current status tool is shut down.

This new System Status will be officially launched to Resellers on December 1, 2008. The current tool at http://status.tucows.com will be retired at that time.

Have a look and let us know your thoughts:

http://status.opensrs.com/

Get the Most from OpenSRS System Status

How to Subscribe:

There are three ways to subscribe to System Status (subscriptions entered during the preview period will be carried over when the new service goes live):

How do I subscribe via email?

1. Go to http://status.opensrs.com/
2. Click on the mail envelope
3. Type in the email address in the text field after the question: “What is your email address?” Click continue.
4. Select either all or a combination of services for notifications.
5. Scroll to the bottom of the page and click Save.
6. You will receive an email requesting confirmation of your subscription. You must click the confirmation link within 72 hours or the subscription request will be discarded.
7. If you make changes to your subscriptions at a later date, you will receive an email confirming the changes, but no action is required.

Subscribers via email will receive System Status updates within minutes of the updates being posted.

How do I subscribe via RSS?

1. Go to http://status.opensrs.com/
2. Click on the RSS icon for the service you wish to monitor.
3. Depending on your own RSS subscription preferences, you will be prompted to add the feed to your default RSS reader (eg. Google Reader).

How do I subscribe via Twitter?

1. If you don’t have a Twitter account, go to http://www.twitter.com to sign up.
2. Login to your Twitter account and go to http://twitter.com/opensrsstatus/ and click “Follow”

Where is the Advance schedule for Maintenance windows?

All maintenance windows for OpenSRS, Registries and our vendors will be posted weekly on the Reseller blog. You can search for these using the “maintenance” category. For OpenSRS service maintenance windows, we will continue to send you our regular advance notifications via email.

Summary of Recent Cluster A Email Service Issues

I’d like to provide further details about poor service we provided to many of our resellers on Cluster A of our Email Service last week.

As promised, we’re conducting a detailed post-mortem but I wanted to kick things off by providing you with some high-level analysis of what happened and what action we took.

We have prepared Incident Report #2993 – October 14, 2008 (260K PDF) as the first part of our analysis.

In the coming days, we’ll be addressing some of the deeper issues brought to light by this incident through an even more technical FAQ that is currently in the works.

As our CEO Elliot Noss expressed in his Open Letter, we’re very sorry this happened in the first place and we’re determined to do everything we can to make sure it doesn’t happen again. We want to thank you for the many words of constructive advice you have provided and we can assure you that we’ll be considering every suggestion.

In Elliot’s letter, he mentioned that in addition to dedicating ourselves to reliability, we are committed to taking other elements of our email service to a new level including: monitoring, change management, emergency protocols and procedures. In the coming weeks, we’ll be posting more about our plans. As always, we welcome your feedback.

Comparison of this incident with the August service interruption

Many of you have been asking us why we have had two outages on the same cluster within a period of three months. We wanted to clarify that this was NOT a reoccurrence of the same issue that caused the service interruption in August. I have published the incident report for the August incident below to allow you to compare, but to summarize briefly:

  1. The August outage was the result of a shelf controller hardware failure. After replacing the defective hardware, we had to rebuild the RAID groups. This process had to be completed in a consecutive manner, meaning that we could only bring mailstores back online one volume at a time. After that incident, we made architecture changes that would prevent a similar hardware failure that would cause a rebuild to be triggered. (Incident Report #1991 – August 18, 2008 (344k PDF))
  2. Last week’s degradation in service was caused by two separate issues (one in the underlying Linux kernel and one in the Dovecot mail server software) which caused corruption in the mail server indexes. This led to an abnormally high server load as users trying to connect received timeout messages and then tried to reconnect. The resulting logjam as all login slots were filled led to more timeouts and degraded service for about 40% of users on Cluster A (or about 20% of all Email Service users). It took us longer to diagnose because we had to rule out a hardware problem first. After that was confirmed, further investigations had to be completed at the same time as we were moving mailboxes to new hardware in an attempt to alleviate the high server loads. Once the problems were diagnosed, we were able to work with some of the top contributors from the Linux kernel and the Dovecot mail server open source communities to develop and apply patches as quickly as possible. Unfortunately, the second bug wasn’t discovered until we had completed reindexing the mailboxes after patching the first problem, leading to a longer than anticipated service disruption.

Once again I’d like to personally apologize for the inconvenience to you our Resellers and to your customers.

We’ll include more posts on this issue and our efforts to make sure it doesn’t happen again in the upcoming days and weeks.

Closing notes on the Cluster A Email Service Interruption

First off, I’d like to apologize again for the problems that resulted from the problems last week on Cluster A of our email service. Email is a mission-critical service. We know how awful it is to have your personal and business communications disrupted. We are deeply sorry for any problems that resulted from this interruption.

After around-the-clock work last week to restore full service to our impacted resellers, and their end-users on Cluster A, our team took some time today to review what happened with last week’s service degradation.

Last Tuesday, a shelf controller hardware failure meant that 14 disks required a rebuild. This resulted in the degradation of multiple storage volumes. This failure affected 50% of customer mailboxes on OpenSRS Email Service – Cluster A. The restoration process was consecutive for the affected devices and therefore took a number of days to complete. To resolve the issue, we replaced the shelf controller and rebuilt 14 disks. During the service interruption, we made temporary mail stores available to customers. On Friday, once restoration was complete, all mail content (messages and folders) were merged from the temporary volumes to the user’s original mailbox.

As with any service problem of this magnitude, it is essential we take steps to make sure it does not happen again. Before the end of the month we are making storage architecture changes to Cluster A to ensure that we eliminate the chance that a similar event with storage will occur in the future.

Again, let me say that we are incredibly sorry about the impact this undoubtedly had on you and many of your customers.

RESTORED: Email Outage Update: August 15, 16:00 P.M. ET

Update: August 15, 16:00 P.M. ET:

On August 15, 2008: 16:00 P.M. ET, Full service to OpenSRS – Cluster A was restored. No mail or data was lost.  We truly apologize for this service disruption.

Update: August 15, 09:57 A.M.:

Regular updates are available at http://status.opensrs.com/

As of this morning, we can report that about 80% of users on Cluster ‘A’ have full email service. The remaining 20% of customers on Cluster ‘A’ have limited access, via webmail only. Additionally, we have updated our original estimate for restoration of service to all customers.

We now estimate that full service will be restored by Friday, August 15, 4:00PM ET (20:00 UTC).

Email Outage Update: August 14, 9:30 A.M. ET

We continue to provide regular updates on the progress of restoration at http://status.opensrs.com/

We’re pleased to report that more than two thirds of users on Cluster ‘A’ now have full email service meaning they can log in via POP, IMAP or webmail, as usual, to send and receive mail.

The remaining users that are still affected by this outage (about 1/3 of users on Cluster A) have been provided limited access, by webmail only, to their email. They can log into their webmail normally and send and receive email as usual, and view any messages received since the outage began.

We’re continuing to monitor the rebuild of the additional two storage volumes. Our current estimate is that full service to the remaining affected customers will be restored by Saturday, August 16, at 4:00 P.M. ET (20:00 GMT/UTC).

Once again, we deeply apologize to you and your customers that have been affected by this service interruption and we appreciate your patience as we work to restore full service to all users.

Email Outage Update: 8:00 PM ET

For OpenSRS Email Service Customers on Cluster A, we have the following update:

At this time 50% of mailboxes on Cluster A cannot access email. Current status has not changed significantly since our last update but we do have additional information to share on the scope and timeline for restoration of services.

Due in large part to the nature and severity of the hardware failure we suffered we now estimate that rebuilding of the email databases impacted will take longer than we had anticipated.

Our current estimate is that full service to all customers will be restored on Saturday, August 16, 20:00 UTC.

We are naturally working on reducing the time these customers are offline in any way possible – this is our best estimate on the current impact. If at all possible we will restore access to users earlier than that, so you may find that some portion of your customers will regain normal access to their email earlier than our estimated time for full restoration of services.

We understand that email is critical to you and many of your customers and we are in the process of rolling out limited access to mailboxes via webmail. Over the next few hours your customers who log in via webmail will find that they can send and receive new messages. They will however not see old mail that has been stored on our servers. This mail is stored on the volumes that are currently being rebuilt and it is therefore not possible for us to give them access to it.

We have included a System Status message in each impacted user’s webmail inbox notifying them that they have access to new mail but not to email archived on the system. The text of the message can be found in the message posted to the System Status page.

We will continue to provide updates approximately every two hours until all mailboxes are fully restored.

Once again we deeply apologize to you and your customers that has been impacted by this service interruption.

Update On Cluster A Email Service Issues

I’d like to provide you with an update on an email outage we are experiencing on Cluster A of our Email Service.  Resellers on Cluster B are not impacted by this.

First off, let me say that we are incredibly sorry about the impact this is undoubtedly having on our resellers and many of their customers.  This shouldn’t have happened and while our focus right now is on letting the team get the system back online we will be looking very closely at how this happened to ensure it doesn’t happen again.

Here’s what we know at this time:

About eight hours ago we suffered a major hardware failure in a NetApp file storage system that is an integral part of our Email Service. The system is of course built with all kinds of redundancy but this hardware failure came at the worst possible time.  A hardware issue affecting redundancy had been found earlier and was corrected in Cluster B last week, with a correction for Cluster A awaiting the arrival of the needed hardware.  However, today a separate hardware issue arose when a disk shelf controller in one of our NetApps failed.  It is the confluence of these two scenarios that puts us in this situation.

We have now replaced the faulty hardware and we are close to having the NetApp back online and in service.

Unfortunately, about half of the mailboxes on Cluster A will now be largely offline while we rebuild the parts of the system that were impacted.  Without drilling down too much into the overall system architecture there are a number of “volumes” within each cluster that store and manage mailboxes.  Three of those volumes now need to be rebuilt, in sequence.  As a volume is fully restored we will bring it back online.

The entire Operations Team and our Network Operating Center Team are working with developers throughout the evening and will continue to work on this until it is fully resolved.  Our CEO Elliot and I are in regular contact with all these teams as well as our communications and support teams but, to be honest, we’re trying to stay out of their hair and let them do their jobs.

The big unknown right now is how long it will take to restore each one of these volumes within Cluster A.  Until we let the rebuilding process continue for 6 to 8 hours we are not confident that any estimates we give at this point will have any meaning.

Right now our focus (in order) is a) rebuilding the volumes and putting them online as soon as we can, b) looking for alternative workarounds that may reduce either the scope or length of the outage, and c) creating and refining our estimates on when full service will be restored.

It’s probably worth noting that the distribution of mailboxes within the volumes is largely random and therefore most resellers on Cluster A will find that some but not all of their customers are impacted.

Once we have the service fully restored we will start the process of investigating root causes and determining how we can make sure this doesn’t happen again.

Once again, on behalf of everyone I want to state how deeply sorry we are that this has happened and we truly appreciate your patience and understanding as we work to resolve the issue at hand.

The best place to get updates on our progress is the OpenSRS Status page.  I’ll post further updates here if we need to share more than we normally do in a Status Update.

Ken Schafer
VP, Product Management & Marketing

Rohan Jayasekera Joins The Herd

One of the best parts of my role here at Tucows is being able to bring talented people into the company.

I’m thrilled to announce that Rohan Jayasekera has joined my team as Director of the Tucows Email Service.

At the risk of sounding like a stalker, I’ve had my eye on Rohan since I first met him at a BarCamp event here in Toronto a few years ago. I was impressed by Rohan’s deep insights on the topic our little “unconference” group was talking about (although I can’t for the life of me remember what exactly we were discussing) and when I got home I did a quick Google on Rohan and found a peer with an incredibly interesting background.

Rather than gush about exact how great a fit Rohan is, I’ll just point you to his blog post about joining Tucows so you can hear it from the man.

Are Registries Aiding And Abetting Front Running?

As James mentioned in yesterday’s post here on the Tucows Blog, Network Solutions Inc. caused quite a furor when they confirmed that they are “front running” (registering domains based on domain searches done by potential registrants).

After a contributor to Domain State broke the story, it was covered on TechmemeDiggSlashdotand a host of individual sites and blogsetcand so on. Heck, it even made USA Today.

To be clear, Network Solutions officially denied they were front running:

Although Network Solutions does temporarily register a site a customer searched for, spokeswoman Susan Wade denied there’s anything nefarious afoot. “Network Solutions is not front-running,” she said. Network Solutions holds the domain for up to four days, during which time a customer can register it only from Network Solutions and after which it again becomes generally available if unregistered, Wade said. But that feature, she said, is a “pre-emptive” measure to protect customers–from front-runners. That’s because front-runners can tell when a customer has searched for a domain at Network Solutions, for example because Network Solutions then must check availability at other sites when a customer searches, Wade said.

Respectfully, this is spin. As many of those up in arms about this have pointed out, Network Solutions is effectively saying “we’re pre-emptively front running to help prevent others from front running”.  My guess is most people would say “thanks but no thanks”. I’m concerned however about an aspect of Susan Wade’s statement that others haven’t made much of, namely that registries are involved in Front Running.

“This search data is captured at the various registries. We believe there are registries and/or Internet service providers that may be selling this data to front-runners. So, by holding domains searched on Network Solutions, this pre-empts the search data being captured,” she said.

If Network Solutions has evidence of registries  - or any service provider for that matter – actually being involved in front running, I urge them to share this information with the Internet community so that we can all make sure that these people are called out for the practice and our customers can be told to avoid them in the future.

Become a Reseller

Sign Up Now