Benchmarking and ‘real FC’
Sometimes I think the only people who read technology blogs are people who write other technology blogs. I have no way to figure out if this is true or not, but it is an interesting topic to ponder. Do IT end users actually read technology blogs? If they are reading, they do not seem to comment very frequently. Much more often comments come from other bloggers or competing vendors.
That said, I am going to talk about an issue that some of the storage bloggers seem to be caught up in at the moment. The issue of ‘emulated FC’ vs ‘real FC.’ Let me start off by sharing a few recent posts from other blogs:
Chuck Hollis at EMC writes about the EMC/Dell relationship and takes the opportunity to compare EMC to NetApp. In this case, he is comparing the EMC NX4 to the NetApp FAS2020. The comment in the post that certainly aggravated NetApp is that EMC does “real deal FC that isn’t emulated.” The obvious implications being that EMC FC is not emulated, NetApp FC is emulated, and FC emulation is bad. (This is not a new debate between EMC and NetApp. Look back through the blogs at both companies and you will find plenty of back and forth on the topic.
Kostadis Russos at NetApp has a post explaining why he, not surprisingly, completely disagrees with Chuck.
Stephen Foskett, a storage consultant, posts what I think is an excellent overview of the issues. He cuts through the marketing spin and asks the right questions. His coverage of the topic is so complete, I almost decided not to write about the topic. I will try not to retrace all the issues he covered. I will hit a couple of his high level points in case you have not had a chance to read his post (I highly recommend it though, it is very good.) In summary:
- All enterprise storage arrays “emulate” Fibre Channel drives to one extent or another
- NetApp is emulating Fibre Channel drives
- All modern storage arrays emulate SCSI drives
- Using the wrong tool for the job will always lead to trouble
- Which is more important to you, integration, performance, or features?
So, why am I writing about it? I am writing about it because Chuck posted a very good blog entry about benchmarking a few days later that, to me, contradicts the importance he gave to ‘real FC’ on 12/9. I have never meet Chuck or Stephen, but they both seem to be very technically adept from their postings.
Without trying to put words in his mouth (text on his blog?), the overall theme of Chuck’s post is to make sure you use meaningful tests if you want meaningful results from a storage product benchmark. He is absolutely correct. I could not agree more. How many times have we seen benchmarks performed that were completely irrelevant to the workload the array would see in production?
My question is, if the end result of performance testing with real world applications produces acceptable results, then who cares what is ‘real’ and what is ‘emulated’? The average driver does not worry about how the computer in her car is controlling the variable valve timing. She worries about whether it reliably gets her to work on time.
VMware is selling plenty of virtualization technology that presents devices that are not ‘real.’ I know it is not storage, but why is that any different? Less and less is ‘real’ in storage these days. It is impossible to continue to drive innovation in storage array technology if we are bound by the old ideas of how we configure and manage our storage. With the introduction of technologies that leverage thin provisioning, dependent pointer based copies, compression, and deduplication we need to rethink concepts as fundamental as RAID groups, block placement, and LUN configuration. Or in my opinion, we need to stop thinking about those things. Controlling the location of the bits is not what matters. Features and performance are what matter. Results in the real world matter. Look at the systems available and decide what blend of features fits your organization and workload best.
Full disclosure: My company provides storage consulting on all of the platforms discussed above. We sell NetApp, Sun, and HDS products. We do not sell EMC products.
Jesse,
Thanks for adding to the conversation. I agree with you on the next step beyond the “real versus emulated” idea: Who cares? The thing that matters is that the storage works in the use case intended.
I hadn’t read your blog in the past, but I’m subscribed now. Good stuff here!
Stephen
Stephen,
Thanks for reading. In the long term, I think this is about storage Quality of Service functionality. That seems like the only practical way to get the performance guarantees administrators want with reasonable management overhead.
I am working on a few more posts about benchmarking. I think it is a great topic and one that is quite challenging for the storage administrators.
– Jesse
Jesse,
now all you need to do is point out that benchmarking is about as Pavlovian as you can get for marketing organizations. Get someone to salivate about “benchmarks” and surely their checkbooks will follow.
Happens in the AMD vs. Intel, ATI vs. nVidia, etc. etc. debates all the time. It’s almost akin to determining who’s technology is “better” all the while using the same hardware. It’s about the implementation of technology, not necessarily the underlying hardware components. (caveat: sometimes the hardware makes an extreme difference in results. witness EFD vs. mechanical disk; global cache vs. localized cache; etc. etc.)
cheers,
Dave
Allow me a bit of clarification?
On a strictly technical level, Steve is right — all modern storage arrays do a certain amount of emulation of one sort or another. He’s right in a pedantic sort of way. But I think he might have missed the point I was trying to make.
And, of course, to your point, if the benchmark is good enough — and delivers acceptable results — does it really matter? You’re right as well.
So, let’s try this again.
Certain emulation schemes can behave differently than one would predict if one was using a more physical approach. If you’re aware of those differences, and they don’t matter to you, no problem.
BUT (!) the vendors who offer such schemes make little effort to highlight differences that might be significant in some situations. And that’s where I raise the flag.
Familar with short stroking? Doesn’t work so well on many of these emulated approaches. Want to place data on the outer tracks to maximize bandwidth? Sorry, no can do in most situations.
Or, my personal favorite, dramatically declining performance numbers as the device fills up. Now, in a physical storage world, you wouldn’t expect THAT to happen, would you?
Or perhaps an array that’s scheduled a defrag/reclamation task in the middle of the day? Not something we’d expect a more physical array to do, right?
There’s more to the discussion, but I think you get my point.
Nothing wrong with emulating per se — but I think harm is done when it’s offered as “no differences whatsoever” as opposed to exposing key differences and helping people make informed choices.
Just to be clear, EMC offers an emulated block device (iSCSI on Celerra) as well as native block devices. We’re quite conversant on the key differences between the two.
Hope this helps.
– Chuck
Chuck,
I think we are all right.
Unfortunately, there is no perfect storage device. Customers have different priorities and that drives them to make different purchasing decisions.
Like most things in life, purchasing a storage device is a compromise. For example, it is challenging to bring fine grained data placement control together with a feature such as pointer based dependent copies or deduplication.
You are again correct that emulation or virtualization is not free. Virtualization (for servers, storage, or anything else) inherently introduces overhead, but lots of customers are making the decision that is is worth it for the benefits they get. I am not making a judgment as to which features are better or more important. Customers tell the vendors everyday when they make a purchase. Maybe a little less frequently in 2009.
– Jesse