Sun Storage 7000 Analytics Overview
With the release of the Sun Storage 7000 line of storage appliances, Sun has included a new “Analytics” toolkit. These analytics are based on DTrace (http://en.wikipedia.org/wiki/DTrace), but essentially hide the DTrace complexity in a cloak of Ajax-based browser graphics. Through the GUI, a storage administrator can determine which clients are causing which files on the server to be “hot”, or resource use-intensive. Also the administrator can see the latency of each request to the blocks of that file, or how many request of each protocol are being processed, or how many cache hits a file had. In this blog I’ll explore the basics of Analytics.
The analytics component of the Sun Storage 7000 line can provide useful information to a storage administrator who is trying to manage and monitor the appliance and the files and blocks stored there. Just like DTrace, the analytics run in real time, and allow quick progression from hypothesis through data gathering to new hypothesis, data and conclusions. Unlike DTrace, the analytics component has a very complete and useful graphical interface and visualization engine.
Some examples of what Sun Storage 7000 analytics can do should go a long way toward understanding the power and flexibility of the tool. The analytics can show:
- what clients are making CIFS requests
- what NFS files are currently being accessed
- how long NFS operations are taking
- what LUNS are being written to
- what files are being accessed from a specific client
- what is the read/write mix for a specific file, by a specific client
- how long are writes taking to a specific file at a specific offset in that file
All of these metrics are shown and optionally recorded on a per-second basis. Recorded data can be examined historically for event correlation or trend analysis. Generally, instrumentation is done at a level of abstraction above the implementation – at a level of detail that system administrators care about. The system conveys both the load placed on the appliance and how the appliance is reacting to the load. For example, a problem could be too much load or not enough appliance resources, and the details are available to make that determination. The 7000’s analytics allow ad hoc instrumentation, not just pre-canned or pre-determined diagnostics, for site-specific or problem-specific debugging. The standard Unix tables-and-numbers diagnostic output are frequently not easy to interpret and slow for the brain to understand, so the GUI manages visualization as a first class aspect of storage analytics.
Let’s have a look at the appliance management screen. The main “Status” screen gives an overview of the entire appliance, including space used and free, protocols enabled, and basic performance metrics (Figure 1). From there clicking on a metric brings that metric into a “worksheet”" on the Analytics page. An administrator may create many worksheets, store them, and switch between them to quickly look at various custom views of the activity of the appliance. This is done by selecting the “Saved Worksheets” subpage from Analytics. Many performance aspects are constantly sampled and made available for archiving, deletion, or adding to a worksheet from the “Datasets” subpage. Further, the administrator may have many open worksheets, and can “clone” a worksheet to make another copy from the current worksheet.
On a worksheet, the “Add Statistic…” button brings up a menu of “statistics”, or system metrics, that can be added and manipulated. These statistics can in turn be broken down into constituent elements. Adding a statistic creates a new panel containing a graph of that statistic updated in real time. Averages and other “breakdown” details (for example, when a specific sample or time is clicked upon within the graph) are shown to the left of the graph. Likewise selecting an item (or multiples via shift-click) in the breakdown table highlights the corresponding data in the graph. For files and devices, a “show hierarchy” option creates a sub-panel to visualize the details of that item in a pie chart and with file and device names enumerated. Again, selecting any item, in any part of the panel highlights that item in the other parts of the panel. Another type of statistics is a quantized breakdown, displayed as a heat map (color-coded histogram) of the data. This type is used for latency and size offsets, where scalar representation does not make sense. Consider data where a value of zero must be distinguished from no data, such as the response time of a request.
The screenshot above shows a few graphs on one worksheet with Analytics. There are many controls to manage a panel. Some of these are time controls. These include moving backward and forward in time, pausing the display (but not the data capture), zooming in and own (showing more or less time in the chart), and going to minute, hour, day, week, and month views of the data. Other controls manage viewing specific data, such as going to the smallest recorded sample or the largest, or directly comparing samples by showing them as line graphs rather than a stacked graph. If there is a specific time of interest in one graph, pressing the “synchronize” icon changes all graphs in the worksheet to show that time and the same time scale and to stay synchronized.
The “drilldown” icon starts from the current graph, allows selection of a specific attribute, and creates a new graph of just that attribute of the current graph. For example from the “CPU: percent utilization” graph, choosing “drill down” allows selection of “by user name” or “by application name”, among others. The drill down choices are specific to the graph being controlled. For example in the graph “Disk: I/O operations per second broken down by type of operation”, selecting a point in time shows the types and number of each type of operation that occurred at that time. Selecting a type of operation from the list left of the graph and choosing “drill down” allows creation of a new graph based on that operation type, but further broken down by disk, size, latency, or offset of the operation. Shift-clicking on the drilldown icon highlights every breakdown, creating a “rainbow” differentiating every breakdown on the graph.
The next per-graph control saves the current graph to the Datasets collection. On the Datasets page, each graph has its data being constantly collected. Uninteresting datasets can be deleted to save space and increase performance a bit. Interesting new graphs can be saved as a dataset and the pertinent data will then be continually collected for later examination. The last control exports the data shown in the graph to a comma-separated .csv file for importation into a spreadsheet, for example. Additional analytic options are available, and can be enabled by selecting the “Make available advanced analytics statistics” checkbox on the Configuration->Preferences page.
For more details of how analytics work, take a look at the presentation put together by members of the Fishworks team that implemented them, available at http://blogs.sun.com/bmc/resource/cec_analytics.pdf. All things Fishworks-centric (videos, blogs, white papers, and more) are available from http://blogs.sun.com/fishworks/.
If you want to try out the 7000 appliance software, simply download the Sun Unified Storage Simulator, provided by Sun as a VMware image. This image is a fully functional version of the 7000 appliance software (it’s the same bits), so you can try all the features, including Analytics. On Windows or Linux you could use VMware player (https://www.vmware.com/products/player/), for free, to run the virtual machine. On Mac OS X there is no free version of VMware Fusion (https://www.vmware.com/products/fusion/), but it does have a trial period. Also, VirtualBox, the open source virtual machine package now owned by Sun, should be able to play VMware images.
Joerg Moellenkamp has posted a nice blog entry proving a walkthrough of setting up the 7000 software, configuring it to a point that it is ready to be managed by the GUI (http://www.c0t0d0s0.eu/permalink/A-walkthrough-to-the-Sun-Storage-Simulator-Part-1-Initial-Config.html). Once on the network, you can use analytics by accessing the system via a browser. The GUI is access by browsing via https to its IP address at port 215.
