The PATROL Performance Assurance Solution is an integrated set of products that monitor, measure, evaluate, predict, and report the performance of distributed systems. PATROL Performance Assurance supports the collection, analysis, and modeling of kernel-level system data.

Version 7.2.00 of PATROL Performance Assurance introduces the PATROL Performance Assurance for Virtual Servers product, which is the premiere product for comprehensive performance management of virtualized server environments, specifically VMware ESX servers. It enables the analysis and reporting of performance information for the virtualized systems (Windows and Linux) and a system-level view of the VMware ESX server for server sizing, management, and daily reporting. PATROL Performance Assurance for Virtual Servers provides the ability to collect VMware ESX server configuration and system-wide statistics, analyze this data with the Analyze component, and report on the data using the Investigate and Visualizer components.1

This paper introduces the capabilities of the PATROL Performance Assurance for Virtual Servers product.

VMware has three platform products—Workstation, GSX Server and ESX Server. VMware Workstation and GSX Server require hosting architecture and run on top of a Windows or Linux host operating system. The VMware ESX Server product runs directly on the hardware, offering the greatest performance and finest resource controls. The ESX Server is a scalable architecture that is well suited for enterprise deployments.

The VMware ESX Server product enables physical systems to act as a group of logical computers, or virtual machines (VMs). Operating systems and applications are configured to run inside a VM that resides on a single piece of hardware, which serves as the host server (the ESX server). This approach allows system resources to be dynamically allocated to any of the VMs based on need, providing mainframe-class capacity usage and control of server resources.

Each VM (also referred to as a VM session) can run Microsoft Windows or Linux operating systems as guest operating systems.

The VMware service console in the ESX server environment is based on a modified version of Red Hat Linux 7.2 and is used to configure, start, and administer all of the VMs. It provides all of the management functionality and runs as another guest OS but has access to the VMware kernel to perform management functions.

The VMware ESX Server virtualization layer (the VMkernel) manages system hardware and the VMs running on the server. It runs directly on the system hardware to provide a secure, uniform platform for deploying, managing, and remotely controlling multiple VMs. This layer presents a uniform hardware environment to each of the guest operating systems.

Example VMware Environment

As shown in Figure 1, each VM session has a generic set of virtual CPU, memory, and I/O (both disk and network) resources that appear to each VM to be a dedicated operating environment. By abstracting the hardware interfaces into virtual resources, VMware provides a way to manage mixed workloads and leverage storage, network, and computing resources, while maintaining the isolation of application servers. For example, under-utilized Microsoft Windows servers can be combined to increase the overall usage of the system, thereby reducing the cost of ownership and improving the return on hardware investments.

In the example shown in Figure 1, several Microsoft Windows servers and a Linux server are consolidated onto a two-CPU ESX server, running four VM sessions and a VMware service console. A different Microsoft Windows operating system instance, and one Linux operating system instance, runs in each VM.

The virtualization layer (the VMware kernel) delegates all of the resource requests on behalf of the VMs. The virtualization layer is completely transparent to the operating system on each VM; to each VM, the physical hardware appears to be dedicated to that VM. The service console acts as the overall administrator for the VMs.

How Can PATROL Performance Assurance Be Used in This Environment?

PATROL Performance Assurance can help manage the performance of each VM and report the appropriate performance data for the VMware server.

Performance data collected from within the VM's operating system may be used to display the “relative” resource usage of individual processes. All VMs share system resources, so obtaining accurate measurements of total actual resource usage is necessary to provide a complete picture of system usage.

With PATROL Performance Assurance for Virtual Servers, a PATROL Perform Agent and a data collector run inside the VMware service console, which provides an interface to the virtualization layer for collecting both total and per VM resource usage. This information is used to create system-level views to support analysis of individual VM usage and to assist performance analysts and planners in determining when and how to deploy individual VMs. When used in concert with the supported Agents running inside the Windows and Linux VMs, a complete analysis of the application workload is now possible.

Sample Implementation Process

For example, a VMware consolidation project in which four Microsoft Windows servers are being moved to a VMware ESX server would involve the following steps:

3. Characterize the workloads—determine which workloads will be moved, which ones do not need to be moved, which ones comprise system overhead, and which ones now would be shared in a consolidation. This process relates system processes to business applications or services.

6. Evaluate the result, tuning as necessary. If the processor that you chose is insufficient, you have the following options:

You can profile and size all kinds of servers—mail servers, web servers, directory servers, application servers, database servers - starting with systems that are important but infrequently used and moving to more full time application services.

Note the following requirements when installing PATROL Performance Assurance in a VMware environment:

You collect data from the VM sessions and the VMware service console just as you would from any other computer. The VMs and the VMware service console appear as individual computers in the Perform user interface.

Using Investigate or in your Manager script, you create a policy that includes the VMware service console and then create a second policy that includes the VM sessions.

How Is the Data Collected?

The VMware data collector installed on the service console gathers VMware kernel statistics. The VMware service console is used to set up the VMs and assign resources among them. The console also provides access to VMware kernel statistics through a set of scripting APIs.

The VMware data collector collects the standard Linux statistics of the service console and makes VMware API calls to collect the VMware metrics.

What Data Is Collected from VMware?

In general, the component closest to the hardware resources has the best view of their activity. The virtualization layer (the VMkernel) keeps track of the actual CPU usage and other resources used by the individual VMs and provides APIs to access these values. These metrics are the most reliable statistics for measuring system activity.

The end result of performance is individual workload response time. An application running on an operating system inside the VM may still have acceptable response time, even if it reports high CPU usage. The response time is driven by two factors: service time and usage. In this case, it is the physical CPU usage reported by the VMkernel that drives the response time, not the usage reported by the operating system. Therefore, it is important to collect data from the VMware service console, which is the only VM that has access to the VMkernel.

VMs do not “own” any of the physical hardware resources in the VMware environment. The virtualization layer (the VMware kernel) delegates all of the resource requests among all of the active VMs. However, the virtualization layer provided by VMware is completely transparent to the operating systems. To each VM, it appears that it is the only operating system that is using the physical hardware, and therefore the statistics collected from the VM reflect this solitary viewpoint.

For example, in Figure 1. , there are five VMs, each with one processor in a two-CPU system. Through the virtualization layer, the VMs get time slices of physical processors. When all five VMs are active, only two of them have physical CPU service at any given time. The operating system instance running inside the VM does not have specialized timing support to account for wall-clock time, so it may not accurately report allocation of physical resources to individual processes. In this example, because each VM does not always have access to all of the cycles of a CPU, the operating systems will always report a higher CPU usage than that measured from the VMware service console.

Performance data collected from within the VM's operating system may be used to display the relative resource usage of individual processes. It is important to note that because all VMs share the real, physical system resources, accurate measurements of the total resource usage are not available from inside the VM.

However, using the process data collected from inside the VM, you can perform basic reporting and analysis of specific workloads and users within the constraints of the virtual environment. For example, you could quickly identify performance problems causing contention among groups of users of a certain application in the VM, and could resolve the problem by moving the users to another VM.

In many environments, due to application vendor requirements, operating system limitations, and the time accounting issue, it is recommended to run only one application per VM. In such cases, measurements of the VM can be used to effectively model the resource requirements of the business application. This best practice can be used as a suitable workaround while BMC Software and VMware jointly explore this matter. However, to obtain a total view of the resource usage for an application that runs inside a VM, you should look at both system-level and application-level data.2

You can access performance data about the VM sessions by using any of the following components:

The following sections describe how to display and interpret the data using each of these components.

How Do I Display VMware Data in Investigate?

Because VMware metrics are part of the Linux collection on the VMware server console, you must identify the service console node of the VMware server of interest. Add that service console node to your policy. To display VMware data in real time, using the Investigate component, complete the following steps:

1. Under PATROL-Investigate in the Tree view, either right-click on Policies and choose New Policy to create a policy file, or open an existing policy that includes the VMware service console or the VMs in which you are interested.

3. Right-click on the VMware service console or the VM session, and choose Drill Down.

5. Expand the VMware Metrics item and then the VMware ESX Server Metrics item to display the available metric groups for the drill down.

The following sections show the available VMware drill downs and provide brief descriptions of the related VMware metrics.

The VMware ESX Host System Configuration metrics provide the configuration information for the VMware Linux host system, as shown in Figure 4 and in the following table.