When Oracle announced the Exadata V2 database appliance late last year, it created quite a stir. The performance numbers for the box are extremely high, and the feature set and capacity are quite large.

Last week we had an executive briefing for folks interested in Exadata V2. My colleagues Kurt Rosenfeld and John Laferrier presented information on business intelligence and the Exadata, as well as the business case and use cases for considering buying one. Joe LaFlamme from Oracle presented some reference customer examples.

I presented the Exadata V2 technical overview, traveling through the architecture details, migration strategies, and component details. Along the way there were a few points I made that seemed a bit surprising to the audience, and that led to a lively discussion. I summarize those points here, as they do not seem to be well known within the industry.

• Existing Oracle licenses are transferable to Exadata (including Oracle DB, RAC, and Partitioning). That can greatly reduce the cost of an Exadata that is being used for database consolidation, for example.

• The Exadata looks to be an excellent consolidation engine. Included with the Exadata software are resource management tools that can, for example, give some databases resource priority over others. These tools also allow the use of the flash storage to be fine tuned, pinning specific tables into flash or letting Oracle use the flash as an extended cache.

• The Exadata V2 is designed to be able to perform OLTP and Data Warehouse transactions concurrently. If a single system can be used both ways, consider the implications compared to stand-alone, separate Data Warehouse solutions. Normally data must be extracted from the OLTP system, copied to the DW system, imported there, and then processed. The extraction and copying are overhead, on both the OLTP and DW systems. And, any reports or queries on the DW system are performed against “stale data” – data from the time the extraction started. Now consider being able to do DW operations against live, current OLTP data. And according to the performance numbers published by Oracle, those operations could run much faster than on most DW systems. That speed could result in completing more complex reports, the allowing of more ad hoc queries, and so on. Such a change could be a fundamental advantage to DW consumers (finance and senior management, for example).

Read more…

Project Crossbow is an innovate, and I think important, new contribution to the OpenSolaris project. Crossbow makes network virtualization and resource management first-class citizens in OpenSolaris. If follows in the footsteps of ZFS by having a simple and easy-to-understand interface, while providing great flexibility and power to the administrator. Crossbow can only be found in OpenSolaris, and is not available in Solaris 10. My February column for ;login: Magazine describes and explores Project Crossbow in detail. You can download it here, but as always I encourage you to become a member of Usenix, thereby gaining access to all of the content of ;login: (along with many other great benefits).

  2010-02-galvin.pdf (678.9 KiB)

The news of Sun integrating an in-line deduplication feature into ZFS has created quite a buzz in storage circles. And our clients have been asking us about how to gain access to this new feature. This blog post describes the steps needed to build an OpenSolaris server, integrate the deduplication feature, and enable it.

For details about the ZFS deduplication feature, what it does, and how it does it, have a look at Jeff Bonwick’s blog post on the topic. He was the lead engineer on the project so you can take his word on it.

Deduplication was integrated into OpenSolaris build 128. That takes a little explanation. Solaris is Sun’s current commercial operating system. OpenSolaris has two flavors – the semiannual support-able release, and the frequently-updated developer release. The current supportable release is called 2009.06 and is available for download here. Also at that location is the “SXCE” latest build. That distribution is more like Solaris 10 – a big ol’ DVD including all the bits of all the packages. OpenSolaris is the acknowledged future of Solaris, including a new package manager (more like Linux) and a live-CD image that can be booted for exploration, and installed as the core release. To that core more packages can be added via the package manager.
Read more…

The OpenSolaris security team has added an interesting proof of concept feature. Immutable Service Containers are designed to make building, configuring, and recreating pre-secured containers easier. The net result, if incorporated into OpenSolaris and eventually a future version of Solaris, should be a set of security best practices managed via a feature-rich framework. Between now and then, there is quite a bit of work for the team to do. My December 2009 column in ;login: Magazine discusses the design goals and current state of Immutable Service Containers. Members of USENIX can read it on-line, while others can download it here:

  2009-12-galvin-login-column.pdf (269.0 KiB)

There is a bug that has been hit by one of our clients and we wanted to post a quick alert before other sites implement the change that causes this problem.

The problem is only of concern to sites running Sun Solaris and using the IP Multipathing facility – using multiple ethernet connections bundled together for availability and performance.

Here are the details of the problem:

There is an issue with IPMP failures (Probe based detection only) due to a kernel patch (141444-09 {SPARC} and 1414450-09 {x86}) found in the latest Solaris 10 Recommended Patch Cluster (Released 10/21/09).

See Patch Cluster ReadMe for additional details on patch contents.

The included kernel patch causes failures with IPMP Probe Based Failure Detection IPMP Groups, which is what we frequently use when deploying best practices standalone systems as well as SunCluster based systems. The problem can be confirmed by snooping the FAILED interface for outgoing ICMP probe packets that should exist but don’t, due to the bug caused by the kernel patch. Instead, the active interface that hasn’t failed will be sending and receiving ICMP probe packets using both configured IPMP group test IP address.

The details of the problem are in this bug document:
http://sunsolve.sun.com/search/document.do?assetkey=1-66-271519-1 <http://sunsolve.sun.com/search/document.do?assetkey=1-66-271519-1>

Sun is recommending that the patch cluster (and the specific patch) not be backed out and remain in place because of security fixes it addresses.

Customers using probe based IPMP groups that require stability (and probe based IPMP failure detection) rather than security are best to avoid this Patch Cluster. Customers needing the security protection due to either operation within a hostile environment or compliancy requirements will need to convert their probe based IPMP groups to link based IPMP groups prior to applying the new Patch Cluster. This will reduce the effectiveness of the IPMP failure detection, but will allow the IPMP groups to remain functional until Sun addresses the issue.

We will continue to monitor this issue until resolution is announced, and will post updated information here.  Thanks to Corporate Technologies’ solution architect Ed Hamilton for detecting this problem and writing up the details.

UPDATE: I have posted an update to this article here: More boot storm details

Measuring the benefit of cache deduplication with a real world workload can be very difficult unless you try it in production. I have written about the theory in the past and I did a lab test here with highly duplicate synthetic data. The results were revealing about how the NetApp deduplication technology impacts both read cache and disk. Based on our findings, we decided to run another test. This time the plan was to test NetApp deduplication with a VMware guest boot storm. We also added the NetApp Performance Accelerator Module (PAM) to the testing.

The test infrastructure consists of 4 dual socket Intel Nehalem servers with 48GB of RAM each. Each server is connected to a 10GbE switch. A FAS3170 is connected to the same 10GbE switch. There are 200 virtual machines: 50 Microsoft Windows 2003, 50 Microsoft Vista, 50 Microsoft Windows 2008, and 50 linux. Each operating system type is installed in a separate NetApp FlexVol for a total of 4 volumes. This was not done to maximize the deduplication results. Instead we did it to allow the VMware systems to use 4 different NFS datastores. Each physical server mounts all 4 NFS datastores and the guests were split evenly across the 4 physical servers.

The test consisted of booting all 200 guests simultaneously. This test was run multiple times with the FAS 3170 cache warm and cold, with deduplication and without, and with PAM and without. Here is a table summarizing the boot timing results. This is the amount of time between starting the boot and the 200th system acquiring an IP address. Here are the results: Read more…

There are many, many choices available when it comes to virtualization technologies. Even within server virtualization, there are many options. Once the choices have been narrowed, it is still a chore to wade through the options and limitations to determine the best fit for a given datacenter environment.

Some frequent decision points include:

• Is your environment large enough to bother virtualizing?
• If you are running VMware, should you consider Microsoft Windows Server 2008 Hyper-V R2?
• Can Hyper-V run other guest operating systems?
• What should a Windows-only shop do?

To help ease the effort, we’ve created a decision flow chart involving the two contenders on the short list at most sites – VMware vSphere 4 and Microsoft Hyper-V R2. This chart starts from your current infrastructure and leads you through the important decisions, and to the conclusions you are likely to reach.

The chart is based on much more detailed information provided in our vSphere vs. Hyper-V whitepaper available for download in this blog posting as well as the associated talk available here.

Hopefully this chart will help you make your server virtualization decisions. Please get in touch if you would like to review the whitepaper or have us evaluate the virtualization options for your datacenter. (Please click on the image for a full-size view.)

I have migrated some data to ZFS filesystems recently and the capacity consumed has surprised me a couple times. In general, it has appeared that the data uses more capacity when stored on the ZFS filesystem. This prompted me to do a little investigating. Is ZFS using more capacity? Is it simply a reporting anomaly? Where is that space going? Does ZFS record size have a major impact? Does enabling compression have a significant impact?

In part, the extra space use is a result of ZFS reporting space utilization differently than other filesystems. When a ZFS filesystem is formatted, almost no capacity is used. A df command will show nearly the entire raw capacity. Many other filesystems take a portion of the raw capacity off the top and reserve it for metadata. This reserve will not show up in df. As data is added to the ZFS filesystem, blocks are allocated for both data and metadata. Both the data and metadata blocks will show up as used capacity. In many other filesystems, at least some of the metadata blocks will be taken from the reserve and only the data blocks will show as consumed capacity. For example, in Solaris, the du command will return the capacity used by the data blocks in a file. In ZFS, that du command returns the total space consumed by the file including metadata and compression. So the question at hand is, when storing a given set of files, does ZFS use more total space than other file systems? That one is difficult to test, given all the variables. But we can test various ZFS configuration options to determine the best settings for minimizing block use.

Read more…

In May, we published a blog entry about Sun’s “T” servers (also known as CMT or coolthreads). Those servers are terrific for some applications but are sometimes an ill-fit for others.  That blog posting was expanded into a full column for ;login: Magazine, which is available to USENIX members. Thanks to USENIX, we are allowed to republish the column to make it available to non-members. You can download the full column here:

  August 2009 Usenix ;login: column (150.9 KiB)

A client invited us to give a presentation at their internal IT conference based on the virtualization whitepaper that we published last month.  The whitepaper is available in this post. Registration is not required, but if you register, we will let you know when the next whitepaper comes out. We have a simple privacy policy and we will not fill your inbox with junk.

The talk went over well, including a lively discussion of the pros and cons of both approaches and how they would fit into the client’s infrastructure.

We are making a .pdf of the talk available today, containing much of the content of the talk.  You can download the talk here: 

  Virtualization Presentation - VMware vSphere vs. Microsoft Hyper-V (659.7 KiB)

Also, if you are in the Northeastern U.S. and are interested in hearing this talk first hand, please get in touch and perhaps we could present the talk at a lunch-and-learn event at your company.