BMC® Performance Assurance® Suite
Technical Bulletin
Version 7.3.00
August 1, 2006
Tracking number 540073: Solaris 10 Container support

BMC Software is alerting users of the BMC Performance Assurance Suite version 7.3.00 products to a product update.

This technical bulletin announces support for Sun Solaris 10 Containers, also known as zones, on BMC Performance Assurance Suite. To enable this new platform support, complete the steps described in "Downloading and applying the patch".


NOTE:
 

The terms Containers and zones are used interchangeably through out this document.


If you have any questions about the problem or the patch, contact BMC Software Customer Support at 800 537 1813 (United States or Canada) or call your local support center. This patch is appropriate for

  • BMC Performance Analyzer for Servers, version 7.3.00 (console patches for supported versions and agent patch for Solaris only)
  • Introduction

    This patch provides initial support for Sun Solaris Containers in BMC Performance Assurance solution products. The implementation and reporting may change when full support is developed.

    Sun's Solaris 10 release provides a new way of virtualization by creating “zones” inside the operating system image. The operating system image that contains the “zones” is called the “global zone.”

    A global zone may contain many zones referred to as non-global zones. Each non-global zone appears as an independent node within the network and it maps application resources to platform resources. Applications running within discreet non-global zones are isolated from each other. Global and non-global zones share one copy of the Solaris 10 operating system, running in the global zone.

    This patch supports

  • additional metric data and treatment of non-global zone as a node, with constraints
  • CPU, Paging I/O and Total I/O metrics at the non-global zone level
  • updates to Visualizer system static (SYSTEMS) and dynamic (SYSTEMD) tables
  • Visualizer 4.2 reporting of Container data, with limitations
  • all process data available at the global zone level only
  • Container reporting in BMC Performance Perceiver 7.2.00, planned for the near future
  • This patch does not support

  • process detail within a zone (all process data is available at the global zone level only)
  • new graphs in BMC Performance Analyzer for Servers or BMC Performance Predictor for Servers
  • Solaris Dynamic System Domains (DSD)

  • NOTE:
     

    If you run Solaris Containers in a DSD environment, the Containers are reported as if they are nodes or partitions, with no information on the global zone environment.


    Platform support

    Agent support includes

  • Solaris 10 on SPARC
  • Solaris 10 on x 86 (64 bit only)
  • Separate, non-standard Solaris 10 package that includes patches required for zone detail

  • NOTE:
     

    You must install the agent patch as well as the console patch because some metric groups are not collected if the agent and collector are not updated along with the console.


    Console support includes

  • Solaris SPARC-64 (versions 8, 9, 10)
  • Windows 2003 x86

  • NOTE:
     

    There is no console support for AIX and HP.


    Visualizer support (no changes required) includes

  • 4.2.00
  • Downloading and applying the patch

    This section provides the information that you need to download and install the patch.

    Preparing for installation

    BMC Software recommends that you install this patch over a BMC Performance Assurance version 7.3.00 install for Solaris Container support.

    Downloading the patch

    To determine which patch you need to download, you will need the following information:

  • the version of the product that you are using
  • the operating system you are using
  • the installation method for the patch—the install_patch.pl script, the BMC Software installation utility, or the Distribution Server

  • NOTE:
     
    • The BMC Software Installation Utility and Distribution Server methods are only available for Solaris Agent and Collector patches.
    • You cannot use install_patch.pl on Solaris 10 on x 86 computers.


    To download the patch

    1. Download the patch to a temporary directory from the FTP location appropriate for your platform, product version, and installation method. "Table 1Patches for the install_patch.pl script" lists the locations of patches that are installed using the install_patch.pl script. "Table 2Patches for the BMC Software installation utility or Distribution Server" lists the locations of patches that are installed using the BMC Software installation utility or Distribution Server.

    Table 1 Patches for the install_patch.pl script

    Component

    Operating System

    FTP location

    Agent

    Solaris

    ftp://ftp.bmc.com/pub/perform/gfc/patches/7.3.00/solaris/pp_7.3.00_agent_patch_Solaris.latest.tar.Z

    Console

    ftp://ftp.bmc.com/pub/perform/gfc/patches/7.3.00/solaris/pp_7.3.00_console_patch_Solaris.latest.tar.Z

    Console

    Windows 2003

    ftp://ftp.bmc.com/pub/perform/gfd/patches/7.3.00/pp_7.3.00_console_patch_Windows.latest.zip

    Table 2 Patches for the BMC Software installation utility or Distribution Server

    Component

    Operating System

    FTP location

    Agent

    Solaris

    2. Extract the patch into the temporary directory using one of the following commands:

  • For install_patch.pl patches, enter:
  • uncompress -c pp_<ver>_agent_patch_<filename>.tar.Z | tar -xvf -

    For example:

    uncompress -c pp_7.3.00_agent_patch_Solaris.latest.tar.Z | tar -xvf -

  • For BMC Software installation utility/Distribution Server patches for Unix, enter:
  • tar -xvf <filename>

    For example:

    tar -xvf prf_ac_solaris_7300_15.tar

  • For BMC Software installation utility/Distribution Server patches for Microsoft Windows, double-click the patch file.
  • Installing the patch

    You can install the patch using either the BMC Software installation utility or the install_patch.pl script.

    Using the BMC Software installation utility or the Distribution Server

    If you are using the BMC Software installation utility, use the BMC Software installation utility version 7.5.51 at the following location:

    ftp://ftp.bmc.com/pub/patrol/patches/COMMON_INSTALL7.5.51/ins_ALL_7551.tar

    If you are installing the patch using the BMC Software installation utility or the Distribution server, see the README.Installation_7.3 file for more information. The file can be found in the bmc_products subdirectory once the patch file has been extracted.

    Using the install_patch.pl script

    Use the install_patch.pl script to install the patch on Windows 2003 consoles.


    NOTE:
     

    You must use the BMC Software installation utility to install the patch on Solaris 10 x86 64-bit computers.


    To install the patch using the install_patch.pl script

    1. Ensure that the install_patch.pl script is executable using the following command:

    chmod 755 install_patch.pl


    NOTE:
     

    The install_patch.pl script is written in perl and uses the perl binary installed, by default, with the product. For the patch installation script to work, the/usr/adm/best1_default/bgs/bin/perl binary must exist and be executable on the host.


    2. As root, run the install_patch.pl script:

    # install_patch.pl

    3. Restart the bgsagent.

    Deploying collectors and interpreting data analysis and reporting

    You may install the Performance Assurance console, Performance Manager, inside a Solaris zone. However, data collection for the console infrastructure uses the single agent installed in the global zone.

    The following sections look at how to deploy collectors and interpret the subsequent analysis and reporting to support Solaris Containers.

    How is data collected?

    Since the global zone can “see” all the operations within the global and non-global zones, only the global zone needs collectors to be deployed. Each process has a non-global zone ID which initiates the process. Data analysis uses process-based information to produce resource usage break-down by non-global zones. Installing an agent and collect in the global zone minimizes the installation effort and runtime overhead.

    How do I enable data analysis for Containers?

    By default, zone data analysis is disabled in BMC Performance Analyzer.

    To enable analysis

    1. Perform the download and install as directed in "Downloading the patch" and "Installing the patch."

    2. Assuming you have already downloaded the BMC Performance Assurance for Servers version 7.3.00 patch, edit AnalyzeMenu.xml which is located in $BEST1_HOME/bgs/analyze.

    3. Find the Application element in the .xml file where the label attribute has a value of Solaris Container.

    4. Change the defaultExecution attribute value from False to True.

    Should I populate Container data to my Visualizer production database?

    This patch represents preliminary support for Solaris Containers. Some of the features and reporting may change when subsequent releases of BMC Performance Assurance implement full support. BMC Software urges you not to populate Container data into an existing production Visualizer database containing non-zone data.


    NOTE:
     

    See "Known issues" for information.


    What is the impact on BMC Performance Predictor?

    BMC Performance Predictor has not been modified for this update, and therefore does not support Solaris Containers. The global zone is analyzed as a standard Solaris system with workloads and transactions. Non-global zones and a node representing global zone overhead (activity associated only with the global zone) appear in the model file as separate nodes without workload and disks definitions. These nodes have 0% utilization in BMC Performance Predictor reports and are not included in the BMC Performance Predictor Visualizer file.

    Solaris Container reporting

    The following sections look at how Solaris Container data is reported in BMC Performance Analyzer, Visualizer, and BMC Performance Perceiver and how to interpret the reports.

    Are there new reports in BMC Performance Analyzer?

    This update does not include new BMC Performance Analyzer reports depicting Container data. The only Container-related output from BMC Performance Analyzer is the Visualizer file.

    What is the impact on Investigate?

    The Zone Configuration drill down provides configuration information for a Solaris Computers with Containers. The global zone as well as all non-global zones are listed in the drill down.

    Figure 1 Zone Configuration drill down

    The new metrics associated with this new metric group are

  • Zone ID
  • Zone Name
  • Unique ID – a counter of how many times it has been rebooted
  • Pool ID – resource pool ID with which the zone is associated
  • Init PID – pid of init process
  • CPU Shares
  • Root – path to root file system
  • Pool Configuration

    The Pool Configuration drill down provides pool information for configured pools on a Solaris computer.

    Figure 2 Investigate Pool Configuration

    The new metrics associated with this new metric group, collected when pools are enabled are

  • Pool Id
  • Pool Name
  • Pool Status (0 = inactive, 1 = active)
  • Pool Default - is this the default pool (0 = no, 1 = yes)
  • Pset Id
  • Pset Name
  • Pset Size – number of processors in the processor set
  • Pset Min – minimum processors in the processor set
  • Pset Max – maximum number of processors in the processor set
  • Pset Default – is this the default processor set (0 = no, 1 = yes)
  • Pset Cpus – all cpus assigned to the processor set (semicolon separated list)
  • Process Statistics drill down

    The Process Statistics drill down displays process information for Solaris computers running Containers. This view lets you identify the processes running in the global zone along with each of the non-global zones in the physical system. The drill down below shows process statistics for both the global zone and non-global zone (bmc-zone5).

    Figure 3 Process Statistics drill down

    The Investigate filter feature lets you view the process statistics for a particular zone. Process Statistics for the remaining global and non-global zones on the computer are not displayed in the drill down. In Figure 4 the drill down would display process statistics for the global zone, which has a Zone ID equal to 0.

    Figure 4 Investigate filter

    How is Container information presented in Visualizer?

    All Visualizer data is stored in tables. When you select a graph from the Visualizer Graphic menu, the tables are queried and the graphs are drawn. There are new entries in the Visualizer system static (SYSTEMS) table for Container information.

  • There are two global zone entries. One reports the total activity seen by the global zone. The activity includes the resource usage of all the non-global zones and activities initiated by the global zone. This entry is referenced by the global zone system ID. This global zone entry references workloads, processes, disks, and other reporting objects.

    The second global zone entry adds G_prefix to the global system ID. This entry reports activity associated with the global zone only. This is global zone overhead.
  • There is an entry in the system static table for each non-global zone by its system ID. It reports activity unique to a particular non-global zone.

  • NOTE:
     

    There are no workload definitions in the global zone (G_prefix) or the non-global zones.


    How are the physical and logical processor count fields assigned for non-global zones?

    The node representing global zone overhead has the same values as the node representing the global zone. You can assign a non-global zone to a resource pool. A global zone can contain more than one resource pool.

    If a non-global zone does not belong to a resource pool, its physical and logical processor count have the same values as the global zone. If a non-global zone belongs to a resource pool, its physical processor count is the same as the global zone’s. Its’ logical processor count is set to the number of processors in the resource pool to which it belongs.

    What is the ‘correct’ value?

    Solaris Container CPU reporting is available through Visualizer. Since Visualizer has not been modified for this update, there are some limitations on its reporting.


    NOTE:
     

    BMC Performance Perceiver version 7.1.23 does not support Unix partition views. BMC Performance Perceiver version 7.2, to be available in the near future, will support AIX and HP Partition views as well as Container reporting.


    Non-global zone CPU utilization is derived by summing up processes belonging to that zone. The global zone overhead CPU utilization is the difference between the measured CPU utilization and the sum of the non-global zones. No workloads are formed inside the zones.

    The zones' I/O and paging reporting metrics are handled similarly to workload I/O and paging. Since there are deficiencies in the raw metrics, BMC Performance Analyzer performs the distribution I/O and paging activity according to other metrics and analysis heuristics.

    What is the impact on Visualizer?

    The nodes representing the global zone overhead (global zone ID with a G_prefix) and global zone total (global zone ID) are listed in the Node selection list. It is important to understand what they represent before drawing them.


    WARNING:
     

    There are known issues with the first implementation of Solaris Containers as they are reported in Visualizer Hierarchy graphs. Visualizer Hierarchy graphs show the incorrect number of CPUs at the global zone level. This is due to a limitation in the current schema, which does not provide for a global zone summary table.


    Selecting a node from the Data Selection dialog box

    There are two entries in the Node Selection list for the global zone. One is used to represent the total activity seen by the global zone. This includes the resource usage of all the non-global zones and activities initiated by the global zone. This entry is referenced by the global zone system ID. The global zone entry references workloads, processes, disks and other reporting objects.

    The second entry uses the global system ID with G_ prefix. This entry refers to activity associated with the global zone only. Think of this as global zone overhead. This entry only references to CPU utilization, total I/O and paging I/O activities without workload, processes, disks and other reporting objects.

    There is an entry in the Node Selection list for each non-global zone by its system ID. The entry represents activity unique to this particular non-global zone. This entry, similar to global zone overhead entry, only references to CPU utilization, total I/O and paging I/O activities without workload, processes, disks and other reporting objects.

    Current Visualizer file feed from BMC Performance Analyzer

    The Visualizer database schema is unchanged for this BMC Performance Assurance patch release to support Solaris Containers. Global and non-global zones are presented as nodes in the system static table. Table 3 describes the contents of the fields for each.

    Table 3 System static Container metrics

     

    Global zone

    Non-global zone

    Node_ID

    Node name

    Node name

    Spec_rating

    Rating of the physical system

    Rating of the physical system

    Num_of_proc

    Number of processors in the physical system

    Number of processors in the physical system

    Host_ID

    If the global zone is in a Dynamic System Domain: (DSD), physical system name; otherwise it is the global zone node name

    Global zone name

    Num_logical_proc

    Number of processor assigned to the global zone

    Number of processors assigned to non-global zone

    Partition_Type

    Blank if standalone; otherwise DSD

    ZONE

    The global zone metrics (Paging I/O, Total I/O, and CPU utilization) include activities from all the non-global zones that the global zone contains. There are also activities attributed exclusively to the global zone.

    Non-global zone metrics are based on process information collected from the global zone. If there is a discrepancy between system and process information, system measurement information is distributed proportionally across non-global zones. This process is similar to workload and transaction handling.

    The global zone is processed like a Solaris computer with workloads and transactions defined. You can display it as a regular system if you want to view it without non-global zone detail breakdown. BMC Performance Analyzer does not support building workloads by zone id at this time.

    On/off switch in BMC Performance Analyzer

    BMC Performance Analyzer contains an on and off switch to enable or disable non-global zone processing. The switch is off by default. With zone analysis off, the global zone is processed as standard Unix system and non-global zones are not represented in the Visualizer file. See "How do I enable data analysis for Containers?" for more information.

    Best practice for deploying initial zone support

    BMC Software strongly recommends that you populate Visualizer files generated with Container support enabled to non-production Visualizer databases. Future releases that fully support Containers might mean additional changes to the Visualizer files.

    Changes to Visualizer file generation

    There are two entries in the Visualizer database static and dynamic tables to represent global zones. One entry represents total activity, similar to what is currently implemented, and the other entry contains activity exclusive to the global zone. Table 4 shows the changes.

    Table 4 Visualizer table changes

     

    Global zone

    Global zone with its own activity

    Non-global zone

    Node_ID

    Node name

    Node name with GZ prefix

    Node name

    Spec_rating

    Rating of the physical system

    Rating of the physical system * global zone proc / physical system proc

    Rating of the physical system

    Num_of_proc

    Number of processors in the physical system

    Number of processors in the global zone

    Number of processors in the global zone

    Host_ID

    If the global zone is in DSD, it is the physical system name; otherwise it is the global zone node name

    Node name with GZ prefix

    Global zone name with GZ prefix

    Num_logical_proc

    Number of processors assigned to global zone

    Number of processors in the global zone

    Number of processors assigned to non-global zone

    Partition_Type

    Blank if standalone, otherwise “DSD”

    G_ZONE

    ZONE

    CPU diagram

    There are two images for each global zone in the CPU diagram. One image shows the global zone as a regular node and one shows the same total activity level broken down by zone. The latter image is labeled by the global zone name, with a GZ_prefix.


    NOTE:
     
    • The two related images might not appear side by side in the graph when there are multiple global zones or systems in the database.
    • If the global zone is on a DSD, the height of the block representing the global zone with local zone breakdowns does not reflect the true capacity of the DSD. The available capacity of the DSD is overstated.
    • The shared resource pool is not represented in the Visualizer database at this time. The CPU diagram does not show how non-global zones are associated with a shared resource pool.


    Figure 5 shows sunlab20 as a standalone system on the right. The sunlab20 block shows sunlab20 as the full global zone, including all non-global zone activity. The GZsunlab20 block on the left shows all non-global zones inside the GZsunlab20, plus all the activity that belongs solely to GZsunlab20. The usage height of the two blocks is identical.

    In the past, you used the CPU diagram to compare physical systems. In this implementation, global zones are mixed with other systems.


    NOTE:
     

    Only the global zone has the workload definition displayed on the side of the diagram.


    Figure 5 Visualizer CPU diagram

    CPU Use Hierarchy graph

    The CPU Use Hierarchy graph, Figure 6, contains the same information as the CPU diagram, minus the capacity of the physical hardware.

    The global zone has two entries. In Figure 6 sunlab20 represents a regular Solaris computer and GZsunlab20 shows activity broken down by non-global zones (bmc_zone4 and bmc_zone5) and global zone overhead (G_sunlab20). The %nPProc field always shows the number of physical processors associated with the selected node which includes

  • global zone overhead node
  • non-global zone
  • node that represents the node containing the zones (GZsunlab20)
  • The nLProc value of the node representing all the zones (Gzsunlab20) shows the sum of all the logical processors of the zones. This value does not represent the total capacity of the zones and should be ignored.

    Figure 6 CPU Use Hierarchy graph

    CPU Use/Max graph

    In the CPU Use/Max graph, the node representing global zone overhead (with the G_prefix) is listed in the Node selection list. Since it reflects only CPU usage by the global zone, the headroom displayed is overstated. BMC Software does not recommend using this graph.

    You should also avoid this graph if a non-global zone is assigned to a resource pool because the resource might be used by other zones as well as the non-global zone. Comparing one zone to its logical (resource pool) or physical processor capacity suggests there is more capacity available than there is.

    CPU Use/Max graphs have four display options: SPECint, Utilization, Per Processor, and Per Logical processor. Displaying a non-global zone using SPECint, Utilization, or Per Processor may overstate the available capacity if the non-global zone has access only to some processors as in Figure 7 BMC Software recommends selecting the Per logical processor display option.

    Figure 7 CPU Use/Max graph, selecting non-global zone

    Known issues

    The following issues will be resolved in future implementations of Solaris Container support.

    Table 5 Known issues

    Vantive

    Description

    543223

    Request for BMC Performance Analyzer to support Solaris Containers running inside DSD, which is a physical server partition.

    543224

    Request for BMC Performance Analyzer to support resource pools for both AIX and Solaris.

    Missing interval in Visualizer file


    NOTE:
     

    This issue applies to the initial product support of Solaris Containers.


    The Visualizer file might be missing the first interval of the day for non-global zones and their associated global zone when processing Solaris Container data. The problem does not happen all the time, on all nodes.

    The problem occurs because of a time skew in the zone configuration metric. UDR data is stored based on a date-time stamp. Typically, data collection Start and End times are within a 24-hour day as indicated by the directory. But, when a time skew happens, some metric groups might be out of the range as in the following example:

    Table 6 Time skew example

    Metric group

    Start time

    Start date

    End date

    End time

    Zone configuration

    7/1/2006

    00:00:07

    7/2/2006

    00:00:07

    Process statistics

    7/1/2006

    00:00:07

    7/1/2006

    23:58:07

    Although the Start times of both metrics are identical, the zone configuration metric End time is the following day. The same time skew happens for the days that follow as well.

    When BMC Performance Analyzer processes the zone-related data interval 7/2/2006 00:00:00 to 7/2/2006 01:00:00, it determines which date-stamped repository contains data that fits the request. Using the above example, the zone configuration from 7/1/2006 fits since its End time falls in the requested range.

    Once BMC Performance Analyzer selects the data from the incorrect repository, it tries to select all other zone-related metrics from the same repository even though process statistics do not fall in the requested range in the selected repository. This causes the Visualizer file to be missing zone data for the interval.

    Work around

    To avoid this problem create a Manager script to collect data for a range such as 00.00.00 to 23.45.00 (instead of 23.59.59). This prevents BMC Performance Analyzer from selecting the incorrect data repository. The range specification depends on the skew.

    Where to get the latest product information

    To view the latest BMC Software documents, visit the Customer Support page at http://www.bmc.com/support_home. BMC Software distributes printed copies of flashes, technical bulletins, and release notes with most product shipments, as indicated on your shipping list. In addition, all notices are available on the Customer Support page, including any notices that BMC Software issues after you receive your product shipment. You will not receive new notices by mail. However, by subscribing to proactive notification, you can receive e-mail messages that direct you to those notices. For more information about proactive notification, refer to the Customer Support page.

    Copyright 2006 BMC Software, Inc., as an unpublished work. All rights reserved.

    BMC Software, the BMC Software logos, and all other BMC Software product or service names are registered trademarks or trademarks of BMC Software, Inc.

    IBM is a registered trademark of International Business Machines Corporation.

    DB2 is a registered trademark of International Business Machines Corporation.

    Oracle is a registered trademark, and the Oracle product names are registered trademarks or trademarks of Oracle Corporation.

    All other trademarks belong to their respective companies.

    BMC Software considers information included in this documentation to be proprietary and confidential. Your use of this information is subject to the terms and conditions of the applicable End User License Agreement for the product and the proprietary and restricted rights notices in the product documentation.

     

    BMC SOFTWARE INC
    2101 CITYWEST BLVD, HOUSTON TX 77042-2827, USA· 713 918 8800
    Customer Support: 800 537 1813 (United States and Canada) or contact your local support center

    *63841*