The news of Sun integrating an in-line deduplication feature into ZFS has created quite a buzz in storage circles. And our clients have been asking us about how to gain access to this new feature. This blog post describes the steps needed to build an OpenSolaris server, integrate the deduplication feature, and enable it.

For details about the ZFS deduplication feature, what it does, and how it does it, have a look at Jeff Bonwick’s blog post on the topic. He was the lead engineer on the project so you can take his word on it.

Deduplication was integrated into OpenSolaris build 128. That takes a little explanation. Solaris is Sun’s current commercial operating system. OpenSolaris has two flavors – the semiannual support-able release, and the frequently-updated developer release. The current supportable release is called 2009.06 and is available for download here. Also at that location is the “SXCE” latest build. That distribution is more like Solaris 10 – a big ol’ DVD including all the bits of all the packages. OpenSolaris is the acknowledged future of Solaris, including a new package manager (more like Linux) and a live-CD image that can be booted for exploration, and installed as the core release. To that core more packages can be added via the package manager.
Read more…

The OpenSolaris security team has added an interesting proof of concept feature. Immutable Service Containers are designed to make building, configuring, and recreating pre-secured containers easier. The net result, if incorporated into OpenSolaris and eventually a future version of Solaris, should be a set of security best practices managed via a feature-rich framework. Between now and then, there is quite a bit of work for the team to do. My December 2009 column in ;login: Magazine discusses the design goals and current state of Immutable Service Containers. Members of USENIX can read it on-line, while others can download it here:

  2009-12-galvin-login-column.pdf (269.0 KiB)

I have migrated some data to ZFS filesystems recently and the capacity consumed has surprised me a couple times. In general, it has appeared that the data uses more capacity when stored on the ZFS filesystem. This prompted me to do a little investigating. Is ZFS using more capacity? Is it simply a reporting anomaly? Where is that space going? Does ZFS record size have a major impact? Does enabling compression have a significant impact?

In part, the extra space use is a result of ZFS reporting space utilization differently than other filesystems. When a ZFS filesystem is formatted, almost no capacity is used. A df command will show nearly the entire raw capacity. Many other filesystems take a portion of the raw capacity off the top and reserve it for metadata. This reserve will not show up in df. As data is added to the ZFS filesystem, blocks are allocated for both data and metadata. Both the data and metadata blocks will show up as used capacity. In many other filesystems, at least some of the metadata blocks will be taken from the reserve and only the data blocks will show as consumed capacity. For example, in Solaris, the du command will return the capacity used by the data blocks in a file. In ZFS, that du command returns the total space consumed by the file including metadata and compression. So the question at hand is, when storing a given set of files, does ZFS use more total space than other file systems? That one is difficult to test, given all the variables. But we can test various ZFS configuration options to determine the best settings for minimizing block use.

Read more…

We’ve just announced the next NEOSUG (New England Open Solaris User’s Group) meeting

You Are Invited to the New England Open Solaris Users Group

Topic: All Virtualization all the Time

When: October 6th, 2009 6:00PM to 9:00 PM

Where: Sun Microsystems Burlington Campus; 1 Network Drive, Burlington, MA

RSVP: To Linda Wendlandt: lwendlandt at cptech dot com lwendlandt@cptech.com

Registration Required! – so we can plan food and drink

Join Eric Sharakan, Kais Belgaied, and Peter Galvin in an update on LDOMs, Solaris on Xen, project Crossbow (network virtualization), and Solaris Containers.

AGENDA:

6:00-6:20: Registration, Pizza and Beverages

6:20-6:30: Introductions: Peter Galvin, CTO, Corporate Technologies

6:30-7:30: Presentation: Logical Domains & Solaris on Xen – Eric Sharakan, Senior Staff Engineer, Sun Microsystems

7:30-8:00: Presentation: Crossbow – Kais Belgaied, Senior Staff Engineer, Sun Microsystems

8:00-8:30: Presentation: Solaris containers – Peter Galvin, CTO – IT Architecture, Corporate Technologies

**** Also we’ll be giving out official NEOSUG T-Shirts and other trinkets, and copies of the OpenSolaris CD and instruction manual.

For more details on the talks please visit the NEOSUG web site at http://www.opensolaris.org/os/project/ne-osug/

You are invited to: The New England Open Solaris User Group (NEOSUG) Meeting

When: June 23rd, 2009 6:00PM to 9:00 PM

Where: Sun Microsystems Burlington Campus; 1 Network Drive, Burlington, MA

RSVP: To Linda Wendlandt: lwendlandt at cptech dot com lwendlandt@cptech.com

Registration Required! – so we can plan food and drink

Read more…

As promised previously, we’ve posted the second FAQ. The Solaris System Analysis FAQ is now live. The purpose of this FAQ is to provide details on how to determine the cause of a performance, reliability, or functionality problem of a Solaris system. There is a link to the new FAQ in the menu bar on the blog front page. If any of the data is inaccurate, please email. If there is something missing, send the question along and we will take a look at it.

We’ve rescheduled the sixth NEOSUG meeting for March 11. Dave Miner will talk about the state of OpenSolaris and demo some new functionality, and I’ll talk about what’s new in the Solaris 10 update 6 (11/08) release. Hope to see you there. For full details and registration info have look at the NEOSUG discussion forum.

My October 2008 column has been published in ;login:. This month it’s about Solaris System Analysis – detailed steps to take to determine why a system is “slow” or “busted”. Some ;login: contents is freely available at ;login:, but my column this month is not one of them. I’ve posted the .pdf here for those without a USENIX membership (although I strongly recommend you get one if you are interested in all things Unix).

The wiki that started with my August 2008 column will be expanded (as soon as I get the time) to include this new content. It’s very lonely having a wiki of one, so please consider contributing your thoughts to what I’ve started. It would be a great advance in systems administration if there was a canonocal source of first-step debugging information, and hopefully you will help make this wiki that source: http://wiki.sage.org/bin/view/Main/AllThingsSun

My August 2008 column has been published in ;login:. This month it’s about Solaris System Analysis – a checklist approach to solving a system being “slow” or “busted”.   Some ;login: contents is freely available at ;login: August 2008, but my column this month is not one of them. I’ve posted the .pdf here for those without a USENIX membership (although I strongly recommend you get one if you are interested in all things Unix).

I hope this column will turn into a living wiki about (Solaris) system analysis. I’ve prepopulated a wiki with the contents of the column, so now it’s up to you to add your thoughts to the procedure. It would be a great advance in systems administration if there was a canonocal source of first-step debugging information, and hopefully you will help make this wiki that source: http://wiki.sage.org/bin/view/Main/AllThingsSun