BMC® Performance Assurance® Suite
Technical Bulletin Version 7.3.00 August 1, 2006 Tracking number 540073: Solaris 10 Container support BMC Software is alerting users of the BMC Performance Assurance Suite version 7.3.00 products to a product update.
This technical bulletin announces support for Sun Solaris 10 Containers, also known as zones, on BMC Performance Assurance Suite. To enable this new platform support, complete the steps described in "Downloading and applying the patch".
If you have any questions about the problem or the patch, contact BMC Software Customer Support at 800 537 1813 (United States or Canada) or call your local support center. This patch is appropriate for
BMC Performance Analyzer for Servers, version 7.3.00 (console patches for supported versions and agent patch for Solaris only) Introduction
This patch provides initial support for Sun Solaris Containers in BMC Performance Assurance solution products. The implementation and reporting may change when full support is developed.
Sun's Solaris 10 release provides a new way of virtualization by creating “zones” inside the operating system image. The operating system image that contains the “zones” is called the “global zone.”
A global zone may contain many zones referred to as non-global zones. Each non-global zone appears as an independent node within the network and it maps application resources to platform resources. Applications running within discreet non-global zones are isolated from each other. Global and non-global zones share one copy of the Solaris 10 operating system, running in the global zone.
This patch supports
additional metric data and treatment of non-global zone as a node, with constraints CPU, Paging I/O and Total I/O metrics at the non-global zone level updates to Visualizer system static (SYSTEMS) and dynamic (SYSTEMD) tables Visualizer 4.2 reporting of Container data, with limitations all process data available at the global zone level only Container reporting in BMC Performance Perceiver 7.2.00, planned for the near future This patch does not support
process detail within a zone (all process data is available at the global zone level only) new graphs in BMC Performance Analyzer for Servers or BMC Performance Predictor for Servers Solaris Dynamic System Domains (DSD)
NOTE: If you run Solaris Containers in a DSD environment, the Containers are reported as if they are nodes or partitions, with no information on the global zone environment.
Platform support
Agent support includes
Solaris 10 on SPARC Solaris 10 on x 86 (64 bit only) Separate, non-standard Solaris 10 package that includes patches required for zone detail
NOTE: You must install the agent patch as well as the console patch because some metric groups are not collected if the agent and collector are not updated along with the console.
Console support includes
Solaris SPARC-64 (versions 8, 9, 10) Windows 2003 x86
Visualizer support (no changes required) includes
4.2.00 Downloading and applying the patch
This section provides the information that you need to download and install the patch.
Preparing for installation
BMC Software recommends that you install this patch over a BMC Performance Assurance version 7.3.00 install for Solaris Container support.
Downloading the patch
To determine which patch you need to download, you will need the following information:
the version of the product that you are using the operating system you are using the installation method for the patch—the install_patch.pl script, the BMC Software installation utility, or the Distribution Server
To download the patch
1. Download the patch to a temporary directory from the FTP location appropriate for your platform, product version, and installation method. "Table 1Patches for the install_patch.pl script" lists the locations of patches that are installed using the install_patch.pl script. "Table 2Patches for the BMC Software installation utility or Distribution Server" lists the locations of patches that are installed using the BMC Software installation utility or Distribution Server.
Table 1 Patches for the install_patch.pl scriptComponent
Operating System
FTP location
Agent
Solaris
ftp://ftp.bmc.com/pub/perform/gfc/patches/7.3.00/solaris/pp_7.3.00_agent_patch_Solaris.latest.tar.Z
Console
Console
Windows 2003
ftp://ftp.bmc.com/pub/perform/gfd/patches/7.3.00/pp_7.3.00_console_patch_Windows.latest.zip
Table 2 Patches for the BMC Software installation utility or Distribution ServerComponent
Operating System
FTP location
Agent
Solaris
2. Extract the patch into the temporary directory using one of the following commands:
For install_patch.pl patches, enter: uncompress -c pp_<ver>_agent_patch_<filename>.tar.Z | tar -xvf -
For example:
uncompress -c pp_7.3.00_agent_patch_Solaris.latest.tar.Z | tar -xvf -
For BMC Software installation utility/Distribution Server patches for Unix, enter: tar -xvf <filename>
For example:
tar -xvf prf_ac_solaris_7300_15.tar
For BMC Software installation utility/Distribution Server patches for Microsoft Windows, double-click the patch file. Installing the patch
You can install the patch using either the BMC Software installation utility or the install_patch.pl script.
Using the BMC Software installation utility or the Distribution Server
If you are using the BMC Software installation utility, use the BMC Software installation utility version 7.5.51 at the following location:
ftp://ftp.bmc.com/pub/patrol/patches/COMMON_INSTALL7.5.51/ins_ALL_7551.tar
If you are installing the patch using the BMC Software installation utility or the Distribution server, see the README.Installation_7.3 file for more information. The file can be found in the bmc_products subdirectory once the patch file has been extracted.
Using the install_patch.pl script
Use the install_patch.pl script to install the patch on Windows 2003 consoles.
NOTE: You must use the BMC Software installation utility to install the patch on Solaris 10 x86 64-bit computers.
To install the patch using the install_patch.pl script
1. Ensure that the install_patch.pl script is executable using the following command:
chmod 755 install_patch.pl
2. As root, run the install_patch.pl script:
# install_patch.pl
3. Restart the bgsagent.
Deploying collectors and interpreting data analysis and reporting
You may install the Performance Assurance console, Performance Manager, inside a Solaris zone. However, data collection for the console infrastructure uses the single agent installed in the global zone.
The following sections look at how to deploy collectors and interpret the subsequent analysis and reporting to support Solaris Containers.
How is data collected?
Since the global zone can “see” all the operations within the global and non-global zones, only the global zone needs collectors to be deployed. Each process has a non-global zone ID which initiates the process. Data analysis uses process-based information to produce resource usage break-down by non-global zones. Installing an agent and collect in the global zone minimizes the installation effort and runtime overhead.
How do I enable data analysis for Containers?
By default, zone data analysis is disabled in BMC Performance Analyzer.
To enable analysis
1. Perform the download and install as directed in "Downloading the patch" and "Installing the patch."
2. Assuming you have already downloaded the BMC Performance Assurance for Servers version 7.3.00 patch, edit AnalyzeMenu.xml which is located in $BEST1_HOME/bgs/analyze.
3. Find the Application element in the .xml file where the label attribute has a value of Solaris Container.
4. Change the defaultExecution attribute value from False to True.
Should I populate Container data to my Visualizer production database?
This patch represents preliminary support for Solaris Containers. Some of the features and reporting may change when subsequent releases of BMC Performance Assurance implement full support. BMC Software urges you not to populate Container data into an existing production Visualizer database containing non-zone data.
NOTE: See "Known issues" for information.
What is the impact on BMC Performance Predictor?
BMC Performance Predictor has not been modified for this update, and therefore does not support Solaris Containers. The global zone is analyzed as a standard Solaris system with workloads and transactions. Non-global zones and a node representing global zone overhead (activity associated only with the global zone) appear in the model file as separate nodes without workload and disks definitions. These nodes have 0% utilization in BMC Performance Predictor reports and are not included in the BMC Performance Predictor Visualizer file.
Solaris Container reporting
The following sections look at how Solaris Container data is reported in BMC Performance Analyzer, Visualizer, and BMC Performance Perceiver and how to interpret the reports.
Are there new reports in BMC Performance Analyzer?
This update does not include new BMC Performance Analyzer reports depicting Container data. The only Container-related output from BMC Performance Analyzer is the Visualizer file.
What is the impact on Investigate?
The Zone Configuration drill down provides configuration information for a Solaris Computers with Containers. The global zone as well as all non-global zones are listed in the drill down.
Figure 1 Zone Configuration drill down
The new metrics associated with this new metric group are
Zone ID Zone Name Unique ID – a counter of how many times it has been rebooted Pool ID – resource pool ID with which the zone is associated Init PID – pid of init process CPU Shares Root – path to root file system Pool Configuration
The Pool Configuration drill down provides pool information for configured pools on a Solaris computer.
Figure 2 Investigate Pool Configuration
The new metrics associated with this new metric group, collected when pools are enabled are
Pool Id Pool Name Pool Status (0 = inactive, 1 = active) Pool Default - is this the default pool (0 = no, 1 = yes) Pset Id Pset Name Pset Size – number of processors in the processor set Pset Min – minimum processors in the processor set Pset Max – maximum number of processors in the processor set Pset Default – is this the default processor set (0 = no, 1 = yes) Pset Cpus – all cpus assigned to the processor set (semicolon separated list) Process Statistics drill down
The Process Statistics drill down displays process information for Solaris computers running Containers. This view lets you identify the processes running in the global zone along with each of the non-global zones in the physical system. The drill down below shows process statistics for both the global zone and non-global zone (bmc-zone5).
Figure 3 Process Statistics drill down
The Investigate filter feature lets you view the process statistics for a particular zone. Process Statistics for the remaining global and non-global zones on the computer are not displayed in the drill down. In Figure 4 the drill down would display process statistics for the global zone, which has a Zone ID equal to 0.
Figure 4 Investigate filter
How is Container information presented in Visualizer?
All Visualizer data is stored in tables. When you select a graph from the Visualizer Graphic menu, the tables are queried and the graphs are drawn. There are new entries in the Visualizer system static (SYSTEMS) table for Container information.
There are two global zone entries. One reports the total activity seen by the global zone. The activity includes the resource usage of all the non-global zones and activities initiated by the global zone. This entry is referenced by the global zone system ID. This global zone entry references workloads, processes, disks, and other reporting objects.
The second global zone entry adds G_prefix to the global system ID. This entry reports activity associated with the global zone only. This is global zone overhead.There is an entry in the system static table for each non-global zone by its system ID. It reports activity unique to a particular non-global zone.
How are the physical and logical processor count fields assigned for non-global zones?
The node representing global zone overhead has the same values as the node representing the global zone. You can assign a non-global zone to a resource pool. A global zone can contain more than one resource pool.
If a non-global zone does not belong to a resource pool, its physical and logical processor count have the same values as the global zone. If a non-global zone belongs to a resource pool, its physical processor count is the same as the global zone’s. Its’ logical processor count is set to the number of processors in the resource pool to which it belongs.
What is the ‘correct’ value?
Solaris Container CPU reporting is available through Visualizer. Since Visualizer has not been modified for this update, there are some limitations on its reporting.
Non-global zone CPU utilization is derived by summing up processes belonging to that zone. The global zone overhead CPU utilization is the difference between the measured CPU utilization and the sum of the non-global zones. No workloads are formed inside the zones.
The zones' I/O and paging reporting metrics are handled similarly to workload I/O and paging. Since there are deficiencies in the raw metrics, BMC Performance Analyzer performs the distribution I/O and paging activity according to other metrics and analysis heuristics.
What is the impact on Visualizer?
The nodes representing the global zone overhead (global zone ID with a G_prefix) and global zone total (global zone ID) are listed in the Node selection list. It is important to understand what they represent before drawing them.
Selecting a node from the Data Selection dialog box
There are two entries in the Node Selection list for the global zone. One is used to represent the total activity seen by the global zone. This includes the resource usage of all the non-global zones and activities initiated by the global zone. This entry is referenced by the global zone system ID. The global zone entry references workloads, processes, disks and other reporting objects.
The second entry uses the global system ID with G_ prefix. This entry refers to activity associated with the global zone only. Think of this as global zone overhead. This entry only references to CPU utilization, total I/O and paging I/O activities without workload, processes, disks and other reporting objects.
There is an entry in the Node Selection list for each non-global zone by its system ID. The entry represents activity unique to this particular non-global zone. This entry, similar to global zone overhead entry, only references to CPU utilization, total I/O and paging I/O activities without workload, processes, disks and other reporting objects.
Current Visualizer file feed from BMC Performance Analyzer
The Visualizer database schema is unchanged for this BMC Performance Assurance patch release to support Solaris Containers. Global and non-global zones are presented as nodes in the system static table. Table 3 describes the contents of the fields for each.
The global zone metrics (Paging I/O, Total I/O, and CPU utilization) include activities from all the non-global zones that the global zone contains. There are also activities attributed exclusively to the global zone.
Non-global zone metrics are based on process information collected from the global zone. If there is a discrepancy between system and process information, system measurement information is distributed proportionally across non-global zones. This process is similar to workload and transaction handling.
The global zone is processed like a Solaris computer with workloads and transactions defined. You can display it as a regular system if you want to view it without non-global zone detail breakdown. BMC Performance Analyzer does not support building workloads by zone id at this time.
On/off switch in BMC Performance Analyzer
BMC Performance Analyzer contains an on and off switch to enable or disable non-global zone processing. The switch is off by default. With zone analysis off, the global zone is processed as standard Unix system and non-global zones are not represented in the Visualizer file. See "How do I enable data analysis for Containers?" for more information.
Best practice for deploying initial zone support
BMC Software strongly recommends that you populate Visualizer files generated with Container support enabled to non-production Visualizer databases. Future releases that fully support Containers might mean additional changes to the Visualizer files.
Changes to Visualizer file generation
There are two entries in the Visualizer database static and dynamic tables to represent global zones. One entry represents total activity, similar to what is currently implemented, and the other entry contains activity exclusive to the global zone. Table 4 shows the changes.
CPU diagram
There are two images for each global zone in the CPU diagram. One image shows the global zone as a regular node and one shows the same total activity level broken down by zone. The latter image is labeled by the global zone name, with a GZ_prefix.
Figure 5 shows sunlab20 as a standalone system on the right. The sunlab20 block shows sunlab20 as the full global zone, including all non-global zone activity. The GZsunlab20 block on the left shows all non-global zones inside the GZsunlab20, plus all the activity that belongs solely to GZsunlab20. The usage height of the two blocks is identical.
In the past, you used the CPU diagram to compare physical systems. In this implementation, global zones are mixed with other systems.
Figure 5 Visualizer CPU diagram
CPU Use Hierarchy graph
The CPU Use Hierarchy graph, Figure 6, contains the same information as the CPU diagram, minus the capacity of the physical hardware.
The global zone has two entries. In Figure 6 sunlab20 represents a regular Solaris computer and GZsunlab20 shows activity broken down by non-global zones (bmc_zone4 and bmc_zone5) and global zone overhead (G_sunlab20). The %nPProc field always shows the number of physical processors associated with the selected node which includes
global zone overhead node non-global zone node that represents the node containing the zones (GZsunlab20) The nLProc value of the node representing all the zones (Gzsunlab20) shows the sum of all the logical processors of the zones. This value does not represent the total capacity of the zones and should be ignored.
Figure 6 CPU Use Hierarchy graph
CPU Use/Max graph
In the CPU Use/Max graph, the node representing global zone overhead (with the G_prefix) is listed in the Node selection list. Since it reflects only CPU usage by the global zone, the headroom displayed is overstated. BMC Software does not recommend using this graph.
You should also avoid this graph if a non-global zone is assigned to a resource pool because the resource might be used by other zones as well as the non-global zone. Comparing one zone to its logical (resource pool) or physical processor capacity suggests there is more capacity available than there is.
CPU Use/Max graphs have four display options: SPECint, Utilization, Per Processor, and Per Logical processor. Displaying a non-global zone using SPECint, Utilization, or Per Processor may overstate the available capacity if the non-global zone has access only to some processors as in Figure 7 BMC Software recommends selecting the Per logical processor display option.
Figure 7 CPU Use/Max graph, selecting non-global zone
Known issues
The following issues will be resolved in future implementations of Solaris Container support.
Missing interval in Visualizer file
The Visualizer file might be missing the first interval of the day for non-global zones and their associated global zone when processing Solaris Container data. The problem does not happen all the time, on all nodes.
The problem occurs because of a time skew in the zone configuration metric. UDR data is stored based on a date-time stamp. Typically, data collection Start and End times are within a 24-hour day as indicated by the directory. But, when a time skew happens, some metric groups might be out of the range as in the following example:
Table 6 Time skew exampleMetric group
Start time
Start date
End date
End time
Zone configuration
7/1/2006
00:00:07
7/2/2006
00:00:07
Process statistics
7/1/2006
00:00:07
7/1/2006
23:58:07
Although the Start times of both metrics are identical, the zone configuration metric End time is the following day. The same time skew happens for the days that follow as well.
When BMC Performance Analyzer processes the zone-related data interval 7/2/2006 00:00:00 to 7/2/2006 01:00:00, it determines which date-stamped repository contains data that fits the request. Using the above example, the zone configuration from 7/1/2006 fits since its End time falls in the requested range.
Once BMC Performance Analyzer selects the data from the incorrect repository, it tries to select all other zone-related metrics from the same repository even though process statistics do not fall in the requested range in the selected repository. This causes the Visualizer file to be missing zone data for the interval.
Work around
To avoid this problem create a Manager script to collect data for a range such as 00.00.00 to 23.45.00 (instead of 23.59.59). This prevents BMC Performance Analyzer from selecting the incorrect data repository. The range specification depends on the skew.
Where to get the latest product information
To view the latest BMC Software documents, visit the Customer Support page at http://www.bmc.com/support_home. BMC Software distributes printed copies of flashes, technical bulletins, and release notes with most product shipments, as indicated on your shipping list. In addition, all notices are available on the Customer Support page, including any notices that BMC Software issues after you receive your product shipment. You will not receive new notices by mail. However, by subscribing to proactive notification, you can receive e-mail messages that direct you to those notices. For more information about proactive notification, refer to the Customer Support page.
*63841*