By: Rick Yazwinski on December 19th, 2008
Posted in: Inside OpenSRS
Tags:
Comments: 0

I’m going to tip my hand early:  Sun’s new Open Storage platform is really sweet.

Ok, now that that’s out of the way, let me give you some background.

We first heard from Sun about the new Open Storage platform in April of 2008. The concept of mixing solid state drives (SSD) with SATA drives in a variety of configurations sounded really good, but our experience with Sun’s NFS products in the past had been less than stellar on the performance front.  When it comes to our business, especially the email side of the business, storage performance over NFS is critical. So in the past, Sun’s (and StorageTek’s) solutions were always left behind in favour of other providers (read: Netapp).

In October, Lucian Florea (our Director of Technical Operations and Planning) and I were down in the Valley talking to Sun about Tucows and OpenSRS, our challenges, and their solutions. Mike Shapiro and Victor Walker took us through a demo of the platform – it was real!  The web UI for managing the platform, replication, SSD, the low price; it was all there. This was, of course, still pre-launch, but Victor generously offered us an engineering evaluation unit to put through the wringer. I have to tell you, it was difficult to contain my excitement: in Canada, you get accustomed to waiting MONTHS before hardware released in the US is available north of the border, never mind getting a pre-release platform on site for extensive testing!

Since early November we’ve been putting the platform through progressively more demanding tests. Tests that many vendors fail miserably and which cause us to immediately halt further testing. Tests in which previous Sun/StorageTek equipment has not fared particularly well.

We start with Bonnie++, a tried and true disk subsystem benchmark suite. It’s interesting to note here that the Toro (Sun 7410) was very comparable to a Netapp 3040 in performance and beat the 3040 in many of the tests.

We next move on to a suite of tests using tools we’ve built to simulate the email storage subsystems: many threads, many directories, and huge numbers of files in the directories being randomly created, read, stat’d and deleted.  Many vendors have serious problems when directories become heavily populated. The 3040 has a serious performance hit when there are more than 1,000 files in a directory (the exact number is unclear, but somewhere between 1,000 and 5,000 files performance really drops off). I was happily surprised to see that the Toro we tested didn’t suffer a performance penalty until there were over 100,000 files in a directory and even then the hit was pretty minor.

We’re now working on getting the hardware deployed into our dev/qa environment for email so we can run our email platform on a Netapp drive side by side with the Open Storage hardware for some direct application comparison testing. If  that goes well, we’ll progress to our production test environment and then to limited production deployment on test mailboxes. Finally, if all this testing goes well, we’ll have the ability to slowly introduce the Sun platform into production. This is a painstakingly slow process, but it’s necessary to ensure the stability and performance of the OpenSRS platforms.

I’ll close with a reiteration of my opening:  I’m really impressed with this platform.  Sun’s done an amazing job, and the Open Storage platform is going to shake the (largely ridiculously overpriced) enterprise storage market to its core. Keep up the great work!

By: Rick Yazwinski on November 6th, 2008
Posted in: Inside OpenSRS
Tags:
Comments: 0

Here at OpenSRS we use a lot of Sun hardware, mostly multi-core Opteron servers but we have some big SPARC platforms running Oracle as well. Recently, Lucian Florea, our Director of Technical Operations and I went down to the Valley to talk to a small army of senior Sun geeks about the new and noteworthy stuff coming our way and to generally talk about our challenges and how Sun can help us.

Oracle ? MySQL

We’ve been on the path to convert from Oracle to MySQL (or in some cases no database) for a while now. It’s a slow process, but a liberating one. We currently spend quite a lot for Oracle support, which in our experience has always been pretty “hit or miss” in terms of delivering value.

While we were at Sun we spoke with some senior MySQL folks about our challenges in getting from Oracle to MySQL across the whole organization. Today, I’m not 100% sure we’ll ever completely eliminate Oracle at Tucows, but I’m pretty sure we can reduce its overall footprint to one or maybe two services, most notably our finance/sales datacube. It was really great to have the MySQL team there listening to our needs, and working with us on their roadmap.

One of our DBA’s biggest concerns with MySQL is the relative lack of tools. Sun/MySQL have addressed this need with their Enterprise licensing models (which at $40k-100k for our organization’s site license are super-affordable compared to Oracle) with a series of tools to inspect, monitor,  and tune a MySQL database, specifically MySQL Enterprise Monitor. I haven’t used Enterprise Monitor yet, but it looks at first blush like a very useful tool for managing MySQL across your enterprise without having to hand-craft health checks of every sort.

Sun Storage

Today, we’re a big Netapp shop. We came to Netapp from the convoluted and expensive world of SAN and fibre storage infrastructure. I’ll disclose here that I’ve liked the Netapp storage products for a long time, long before I joined Tucows. But we only landed on Netapp here after testing many storage providers out there for their performance profile in our particular environments. We tested Sun’s Thumper, some StorageTek devices, Isilon, Netapp, and even Linux NFS against each other to find the best performers in terms of high availability and raw throughput (especially with respect to billions of small files) and the Netapp products were clear winners in every test.

This may soon change. Netapp has two Achilles’ heels: their astronomical cost and their proprietary and closed nature.

While I was at Sun we talked excitedly about their new OpenStorage products, being announced November 10. I’m not sure how much I can say about this technology yet. There are clues on Sun’s site and in an interview with Mike Shapiro. Sun is positioned to win by attacking both of Netapp’s vulnerabilities by providing a low-cost, high-performance and open storage platform. I can’t wait to get my eval units in to play with!

One of the other really exciting components of the open storage products coming to market is a dtrace-like utility for examining your storage infrastructure  and really drilling down on what’s going on. This is important to us, like many others, as we continue to tune and improve our systems, to help find storage bottlenecks and inefficiencies. By being able to find hotspots, tune our code and retest, we can immediately see if our changes are having the desired results. This is really a very cool tool. Way to go Mike and team!

Of course, while I’m excited about the new Sun open storage offerings, before they unseat the incumbent, they’ll have to prove themselves in our test lab, and then again in a dev/qa environment, and then again in a production test environment, before they’d have the chance to see use in production.

Other Stuff…

While we were at Sun we also talked to some other folks about topics that weren’t as pressing for us, but still pretty cool in their own right.

We had a bit of a technology tour of Sun’s hardware. The SunRay platform, while not new, is still holding on, and has some interesting bells and whistles now. Project Blackbox, the containerized datacentre, is very cool and worth checking out if you’re not aware of it.

Back on the storage front, we spoke with Glenn and Glenn about Project Celeste. Celeste is a project in Sun Labs (and probably a fair way away from being ready for enterprise production-land), that is a “high availability peer-to-peer data store with semantics for file creation, arbitrary read and write, and deletion”. Seems like everyone is hungry for an open source clustered file system. We are definitely watching for one

In Closing

Evaluating solutions and re-evaluating solutions as they evolve to keep our network and services as efficient as possible is one of the best parts of my job. Thanks to Sun for the exciting look at the technology coming down the pipe and for listening to our wants and needs.

By: Rick Yazwinski on October 20th, 2008
Posted in: Inside OpenSRS
Tags:
Comments: 1

Rick Yazwinski is Principal Engineer at OpenSRS. In this role, he is in charge of all technology decisions and strategy for the company.

Here at Tucows we’re strong supporters of the open source movement and community. We have been standardized on Debian Linux for years. We develop in the LAMP stack (Linux, Apache, MySql, Perl), with some Ruby, Python, and Java thrown in for color. We use open source components in our email platform along with homegrown components.

Why?

Experience has brought us here. Let me explain.

We’ve been down the proprietary software path. Our experience, like many others, has been mixed: nothing hurts more than having a vendor shrug off a critical problem saying they can’t reproduce it, point a finger at “the network” or some other component, or tell you that before they’ll help you you have to upgrade your server farm to the newest minor release because “it may fix the problem”.

Open source projects that have been around for a while just plain work, and work well. Further, when they don’t work well, the depth of understanding “out there” is huge AND when required you can get into the code and find any issues and fix or extend the code base. Unlike proprietary software packages, there are many courses of action you can take when you have a problem.

We aren’t alone in our belief in open source. So much of the Internet is built on open source software, it’s hard to contemplate running an Internet property or service without relying on it. Facebook, Google, Yahoo, Flickr, to name just a few; all of them support open source communities and believe in open source products.

In light of our recent issue, we remain committed to open source. It has served us well for many years and will continue to do so. We are now actually closer to many of the best sources for some parts of the stack, putting us in a better position for the future in terms of getting information and recommendations from some of the most knowledgeable people in the world.