Jesse St. Laurent will no longer be contributing posts here.. If you’d like to follow his activities, have a look at http://jessestlaurent.com.

Introduction

Computer storage has evolved from Directly Attached (DAS) to Storage Area Networks (SAN). Along the way, Sun in 1984 invented NFS, and Network Area Storage (NAS) was born. Since then other NAS protocols have been added, most notably the Windows-based Server Message Block (SMB), aka CIFS. But throughout the history of storage, NAS has been regarded as poorly performing and unreliable compared to SAN and DAS. Certainly Auspex’s creation and NetApp’s advancement of NAS “appliances” helped move NAS from being a science project to a mainstream production solution, but in my opinion NAS is still under-appreciated and under-deployed. Perhaps in light of the new generation of NAS appliances, that should change.

At a more philosophical level, it’s worth asking “what is SAN” and “what is NAS.” Fundamentally, they are storage arrays that make disk space available via varying protocols over varying interconnect media. For the most part, both technologies are available with Fibre Channel (FC), SATA, and SAS disks. Both have disks of varying speeds, capacities, and performance. Traditionally, SANs have been FC connected and NAS appliances connected via Ethernet, but many current products provide both interconnects—block transactions occur via FC or iSCSI and file transactions over Ethernet. A proof point of this merger of NAS and SAN is the FCOE protocol which places Fibre Channel frames over Ethernet networks. Perhaps the most straightforward definition is that “SAN” is block-based storage and “NAS” is file storage, and that a given datacenter should choose which to use for any given application or function. After those decisions are made, it is easier to determine the best products to implement the resulting storage architecture. Now let’s consider the problem with NAS as well as the solutions it can provide.

The “Problem”

Over the years I’ve seen many, many computing infrastructures. Back in the “old days” (say, the 1980s), we had servers and SANs for production, and NAS was pushed to the side. It was typically used for home directories and the storage of utility programs, if at all. In those cases, NAS storage was mounted to all servers as well as all workstations.

That helped NAS gain a reputation for unreliability—probably because any failure caused everyone to notice it, and failures were difficult to recover from (with hard mounts never timing out, for example, taking down all computing until the NAS server could be fixed). Also, many situations called for “cross mounts,” where servers would mount each other’s directories via NFS. If one server then failed, all servers would eventually end up hanging until the failed one recovered. NFS also had quirks like “stale file handles” that left a bad taste in the mouth.

So failures of NFS servers were quite painful to the computing infrastructure. Why did NAS servers fail as often as they did? Well, they were non-clustered, while their SAN brethren typically had more redundant components and automatic recovery from problems. Originally, a “NAS server” was just a general-purpose Sun server running NFS. SAN originally and usually still is a purpose-built storage array. Also, they were and still are network- connected. Back in the day, there was typically one network connection to each workstation (and frequently between servers as well). That one link was used for NAS and non-NAS network traffic. Even if there was a separate network carved out for storage communication between the servers and NAS, it was rarely redundant. Multiple use and single points of failure meant NAS was more prone to failure than SAN. Thus the lingering impression that SAN is more reliable than NAS.
Read more…

When Oracle announced the Exadata V2 database appliance late last year, it created quite a stir. The performance numbers for the box are extremely high, and the feature set and capacity are quite large.

Last week we had an executive briefing for folks interested in Exadata V2. My colleagues Kurt Rosenfeld and John Laferrier presented information on business intelligence and the Exadata, as well as the business case and use cases for considering buying one. Joe LaFlamme from Oracle presented some reference customer examples.

I presented the Exadata V2 technical overview, traveling through the architecture details, migration strategies, and component details. Along the way there were a few points I made that seemed a bit surprising to the audience, and that led to a lively discussion. I summarize those points here, as they do not seem to be well known within the industry.

• Existing Oracle licenses are transferable to Exadata (including Oracle DB, RAC, and Partitioning). That can greatly reduce the cost of an Exadata that is being used for database consolidation, for example.

• The Exadata looks to be an excellent consolidation engine. Included with the Exadata software are resource management tools that can, for example, give some databases resource priority over others. These tools also allow the use of the flash storage to be fine tuned, pinning specific tables into flash or letting Oracle use the flash as an extended cache.

• The Exadata V2 is designed to be able to perform OLTP and Data Warehouse transactions concurrently. If a single system can be used both ways, consider the implications compared to stand-alone, separate Data Warehouse solutions. Normally data must be extracted from the OLTP system, copied to the DW system, imported there, and then processed. The extraction and copying are overhead, on both the OLTP and DW systems. And, any reports or queries on the DW system are performed against “stale data” – data from the time the extraction started. Now consider being able to do DW operations against live, current OLTP data. And according to the performance numbers published by Oracle, those operations could run much faster than on most DW systems. That speed could result in completing more complex reports, the allowing of more ad hoc queries, and so on. Such a change could be a fundamental advantage to DW consumers (finance and senior management, for example).

Read more…

Project Crossbow is an innovate, and I think important, new contribution to the OpenSolaris project. Crossbow makes network virtualization and resource management first-class citizens in OpenSolaris. If follows in the footsteps of ZFS by having a simple and easy-to-understand interface, while providing great flexibility and power to the administrator. Crossbow can only be found in OpenSolaris, and is not available in Solaris 10. My February column for ;login: Magazine describes and explores Project Crossbow in detail. You can download it here, but as always I encourage you to become a member of Usenix, thereby gaining access to all of the content of ;login: (along with many other great benefits).

  2010-02-galvin.pdf (678.9 KiB)

Topic: DTrace Deep Dive and a short talk on LDOM Domains and ZFS

When:
Burlington MA Sun Campus – Feb 2, 2010 6:00PM to 9:00 PM
Boston MA – Boston University – Feb 3, 2010 6:00PM to 9:00 PM
(Note: The same content will be presented twice – once in Burlington and once in Boston. Pick the best location and date as convenient.)

Where:
Feb 2 – Sun Microsystems Burlington Campus; 1 Network Drive, Burlington, MA
Feb 3 – Boston University, Electrical and Computer Engineering Department Photonics Center Building – Room PHO 339 (3rd floor), 8 Saint Mary’s Street Boston, MA 02215
BU Parking: Street parking available on St. Mary’s Street and Bay State Road. Metered parking spots do not require a fee after 6pm.

RSVP: To Linda Wendlandt: lwendlandt@cptech.com

Registration Required! – so we can plan food and drink

Join Jim Mauro and Shannon Sylvia for how-to DTrace, and how to use LDOMs with ZFS.

AGENDA:

6:00-6:20: Registration, Pizza and Beverages

6:20-6:30: Introductions: Peter Galvin, CTO, Corporate Technologies

6:30-8:30: Solaris Dynamic Tracing – DTrace – Jim Mauro, Principle Engineer, Sun Microsystems

8:30-9:00: LDOM Domains and ZFS: An example of creating a ZFS bootable root LDOM domain using jumpstart – Shannon Sylvia, Sysadmin, Northeastern University

9:00 Q&A and Discussion

Also we’ll be giving out official NEOSUG T-Shirts and other trinkets, and copies of the OpenSolaris CD and instruction manual.

For more information see the NEOSUG discussion forum.

The news of Sun integrating an in-line deduplication feature into ZFS has created quite a buzz in storage circles. And our clients have been asking us about how to gain access to this new feature. This blog post describes the steps needed to build an OpenSolaris server, integrate the deduplication feature, and enable it.

For details about the ZFS deduplication feature, what it does, and how it does it, have a look at Jeff Bonwick’s blog post on the topic. He was the lead engineer on the project so you can take his word on it.

Deduplication was integrated into OpenSolaris build 128. That takes a little explanation. Solaris is Sun’s current commercial operating system. OpenSolaris has two flavors – the semiannual support-able release, and the frequently-updated developer release. The current supportable release is called 2009.06 and is available for download here. Also at that location is the “SXCE” latest build. That distribution is more like Solaris 10 – a big ol’ DVD including all the bits of all the packages. OpenSolaris is the acknowledged future of Solaris, including a new package manager (more like Linux) and a live-CD image that can be booted for exploration, and installed as the core release. To that core more packages can be added via the package manager.
Read more…

The OpenSolaris security team has added an interesting proof of concept feature. Immutable Service Containers are designed to make building, configuring, and recreating pre-secured containers easier. The net result, if incorporated into OpenSolaris and eventually a future version of Solaris, should be a set of security best practices managed via a feature-rich framework. Between now and then, there is quite a bit of work for the team to do. My December 2009 column in ;login: Magazine discusses the design goals and current state of Immutable Service Containers. Members of USENIX can read it on-line, while others can download it here:

  2009-12-galvin-login-column.pdf (269.0 KiB)

There is still time to register for the  VMware vs. Hyper-V Hands-on Workshop we are holding on Wednesday, November 18th, at the Hilton Boston/Woburn hotel in Woburn, MA.

The workshop will begin at 8:30 am and includes lunch. During the workshop John Laferriere will present a quick overview of Corporate Technologies. Next I will present a talk based on our VMware vSphere 4 vs. Hyper-V R2 white paper. Next Sean Daly and Joe Gries will do hands-on demonstrations of the two technologies. This will be followed by Q&A and lunch.

We are encouraging attendees to ask us about specific use cases and solution requirements to optimize the value of the workshop.  For more details and to register please see the invitation.

(Sorry for the short notice… hope you can join us)

Topic: An Introduction to OpenSolaris and its Uses

When: November 11, 2009 6:00PM to 9:00 PM

Where: Boston University, Electrical and Computer Engineering Department Photonics Center Building – Room PHO 339 (3rd floor), 8 Saint Mary’s Street Boston, MA 02215 http://www.mapquest.com/maps?&city=boston&state=ma&address=8%20st.%20mary%27s%20street

Parking: Street parking available on St. Mary’s Street and Bay State Road. Metered parking spots do not require a fee after 6pm.

RSVP: To Ayse Kivilcim Coskun (acoskun at bu dot edu)

Registration Required! – so we can plan food and drink

Join Brian Leonard, Bogdan Vasiliu, Ayse Coskun, and Peter Galvin for an overview of OpenSolaris and its uses.

AGENDA:

6:00-6:20: Registration, Pizza and Beverages

6:20-6:30: Introductions: Peter Galvin, CTO, Corporate Technologies

6:30-8:00: What’s So Cool About OpenSolaris Anyway – Brian Leonard, Technology Evangelist, Sun Microsystems

8:00-8:30: OpenSolaris: Clusters and Clouds from your Laptop – Bogdan Vasiliu, Solaris HPC, Sun Microsystems,Inc.

8:00-8:30: Presentation: Solaris containers – Peter Galvin, CTO – IT Architecture, Corporate Technologies

8:30-8:45: OpenSolaris as a Research and Teaching Tool – Ayse K. Coskun, Assistant Professor Electrical and Computer Engineering Department, Boston University

8:45-9:00 Q&A and Discussion

- Also we’ll be giving out official NEOSUG T-Shirts and other trinkets, and copies of the OpenSolaris CD and instruction manual.

For more information please see the full announcement in the NEOSUG forum.

There is a bug that has been hit by one of our clients and we wanted to post a quick alert before other sites implement the change that causes this problem.

The problem is only of concern to sites running Sun Solaris and using the IP Multipathing facility – using multiple ethernet connections bundled together for availability and performance.

Here are the details of the problem:

There is an issue with IPMP failures (Probe based detection only) due to a kernel patch (141444-09 {SPARC} and 1414450-09 {x86}) found in the latest Solaris 10 Recommended Patch Cluster (Released 10/21/09).

See Patch Cluster ReadMe for additional details on patch contents.

The included kernel patch causes failures with IPMP Probe Based Failure Detection IPMP Groups, which is what we frequently use when deploying best practices standalone systems as well as SunCluster based systems. The problem can be confirmed by snooping the FAILED interface for outgoing ICMP probe packets that should exist but don’t, due to the bug caused by the kernel patch. Instead, the active interface that hasn’t failed will be sending and receiving ICMP probe packets using both configured IPMP group test IP address.

The details of the problem are in this bug document:
http://sunsolve.sun.com/search/document.do?assetkey=1-66-271519-1 <http://sunsolve.sun.com/search/document.do?assetkey=1-66-271519-1>

Sun is recommending that the patch cluster (and the specific patch) not be backed out and remain in place because of security fixes it addresses.

Customers using probe based IPMP groups that require stability (and probe based IPMP failure detection) rather than security are best to avoid this Patch Cluster. Customers needing the security protection due to either operation within a hostile environment or compliancy requirements will need to convert their probe based IPMP groups to link based IPMP groups prior to applying the new Patch Cluster. This will reduce the effectiveness of the IPMP failure detection, but will allow the IPMP groups to remain functional until Sun addresses the issue.

We will continue to monitor this issue until resolution is announced, and will post updated information here.  Thanks to Corporate Technologies’ solution architect Ed Hamilton for detecting this problem and writing up the details.