Agent Toolbox

The Agent Toolbox contains a group of utilities that BMC Customer Support can use to analyze proper functionality and troubleshoot issues in the Agent. Use the toolbox only when instructed by BMC Support to run tests on the Agent.

The tools in this toolbox are divided into several groups, according to the following categories:

• CPU Tests: Provides tools for CPU and memory monitoring, as described in Running CPU Tests on the Agent Host.

• System Tests: Provides tools for obtaining and analyzing system-level statistics, including the following system-level information; I/O measurements for files and details of DLLs, OS error details and time zone details, and Information about semaphores and locks, as described in Running System Tests on the Agent Host.

• Analysis of Agent Functionality: Provides tools for analysis of various aspects of Agent functionality, including Analysis of disk space in Agent directories, Verification of WMI connections, and Stress tests for system monitoring, as described in Analyzing Agent Functionality.

• Analysis of the Agent Operating System: Provides tools for analysis of the Agent operating system, including a check for compliance with operating system requirements and file system stress test, as described in Analyzing the Agent Operating System.

• Analysis of Logs: Provides tools that help you scan the Agent logs for issues, as described in Analyzing the Agent Logs.

• Job Tracker Checks: Provides tools that enable you to check the Job Tracker on the Agent host, as described in Checking the Job Tracker on the Agent Host.

• Job Tracking Management Tasks: Provides tools that perform job tracking tasks on the Agent host, as described in Managing Job Tracking on the Agent Host.

Running CPU Tests on the Agent Host

This procedure describes how to run tools for monitoring of CPU and memory usage by the Agent. The Agent Toolbox offers several CPU tests that you can run on the Agent host.

Begin

1. Open a command prompt window where the Agent is installed.

2. Run the toolbox by entering the full path to the toolbox script file, enclosed in quotes:

   • Linux: "<AGENT_HOME>/ctm/toolbox/toolbox.sh"

   • Windows: "<AGENT_HOME>\toolbox\toolbox.bat"

3. In response to the Start message, type Y (yes) and then press ENTER to continue.

4. In the Main Menu, type 1 (CPU tests) and press ENTER.

5. in the CPU Tests menu, type the number of the relevant option:

   • 1 - Get CPU specs

   • 2 - CPU usage

   • 3 - CPU and Memory monitoring

   • 4 - CPU and Memory monitoring during changes of a directory

6. In the menu of the selected CPU test, specify values for all required input parameters and for any of the optional input parameters. For each input parameter, do the following:

   1. Type the number of the option and press ENTER.

   2. Type the value for the input parameter and press ENTER.

7. Type r (Run) and press ENTER.

For more information about each of the tests, including a description of the input parameters and the final output, see CPU Tests: Input and Output.

CPU Tests: Input and Output

The following table lists the tools for CPU tests in the Agent Toolbox, along with details about the input parameters and output of each tool:

Tool	Input parameters	Output
1 - Get CPU specs (LINUX only)	No input parameters.	Basic CPU specifications for the LINUX platform.
2 - CPU usage	1 - Number of samples: Determines the number of times to measure CPU during the test. The default is 10 times. 2 - Sleep between samples: Determines the length of sleep time (in seconds) between samples.	Percentage of CPU usage for each sample and an average of CPU usage from all samples.
3 - CPU and Memory monitoring	1 - Output file: Determines the file path where the output is saved. The default is OutputCpu.txt in the Toolbox directory (within the Agent Home directory). 2 - Number of hours to run: Determines the total number of hours to run the test. The default is 1 hour. To run the test indefinitely, set the number of hours to -1. 3 - How many seconds to sleep between runs: The default is 5 seconds.	(In the output file) Process information and process memory details for each test run.
4 - CPU and Memory monitoring during changes of a directory	1 - Output file: Determines the file path where the output is saved. The default is OutputCpu_MonitoringDir.txt in the Toolbox directory (within the Agent Home directory). 2 - Number of hours to run: Determines the total number of hours to run the test. The default is 1 hour. To run the test indefinitely, set the number of hours to -1. 3 - How many seconds to sleep between runs: The default is 1 second. 4 - Which directory to check: The out-of-the-box default is the exe directory within the Agent Home directory.	(In the output file) Process information and process memory details from the first run and from various subsequent runs, provided that changes were detected in the specified directory.

Running System Tests on the Agent Host

This procedure describes how to run system-level tests on the Agent host using the Agent Toolbox.

Begin

1. Open a command prompt window where the Agent is installed.

2. Run the toolbox by entering the full path to the toolbox script file, enclosed in quotes:

   LINUX: "<AGENT_HOME>/ctm/toolbox/toolbox.sh"

   Windows: "<AGENT_HOME>\toolbox\toolbox.bat"

3. In response to the Start message, type Y (yes) and then press ENTER to continue.

4. In the Main Menu, type 2 (System tests) and press ENTER.

5. In the System Tests menu, type the number of the relevant option:

   • 1 - IO check

   • 2 - Get OS Error description

   • 3 - Check time zone

   • 4 - Semaphore simulator

   • 5 - File lock simulator

   • 6 - Directory lock test

   • 7 - DLL Tester

   • 8 - Check execv

6. In the menu of the selected system test, specify values for all required input parameters and for any of the optional input parameters. For each input parameter, do the following:

   1. Type the number of the option and press ENTER.

   2. Type the value for the input parameter and press ENTER.

7. Type r (Run) and press ENTER.

For more information about each of the tests, including a description of the input parameters and the final output, see System Tests: Input and Output.

System Tests: Input and Output

The following table lists the tools for system tests in the Agent Toolbox, along with details about the input parameters and output of each tool:

Tool	Input parameters	Output
1 - IO check	1 - File path: Determines the file path where you want to perform the Input/Output test. 2 - Seconds to run: Determines the total number of seconds to run the test. The default is 10 seconds.	Results of the IO test: The number of open/write/close actions performed on the file during the test and the average time (in milliseconds) that a write action took.
2 - Get OS Error description	1 - Error number: Defines the number of the system error for which you want details.	Details of the specific system error.
3 - Check time zone	No input parameters.	Time zone specifications from the Agent host.
4 - Semaphore simulator (Linux only)	1 - File path: Determines the file path where you want to perform the semaphore simulation.	Descriptions of the steps during semaphore simulation on the specified file.
5 - File lock simulator (Linux only)	1 - File path: Determines the file path where you want to perform the lock simulation.	Descriptions of the steps during lock simulation on the specified file.
6 - Directory lock test (Windows only)	1 - Type of Service: Determines the type of service to execute during the lock test, either I (information) or T (termination). The default is I. 2 - Directory full path: Determines the file path to the directory where you want to perform the lock test. It the path contains spaces, remember to enclose it in quotes.	Details about the processes that hold a lock on the directory. If you chose T as the type of service, these processes are terminated. NOTE: Before the first time that you run this tool, you must download the Handle utility from the Windows Sysinternals site and place it in the Toolbox directory. In addition, you must run the tool as an Administrator.
7 - DLL Tester (Windows only)	DLL/EXE Directory: In the DLL/EXE Test window, browse and select the directory to analyze.	Details of the DLL files in the specified directory (full path to the file, whether 32-bit or 64-bit, and version of the DLL).
8 - Check execv (Linux only)	1 - Username to check: Specifies a root user to perform an execv system call, which simulates Agent calls. 2 - Command to run: Specifies a Linux command to use in the execv check.	Results of the execv system call, which simulates Agent calls.

Analyzing Agent Functionality

This procedure describes how to run the Agent Toolbox utilities that analyze Agent functionality.

Begin

1. Open a command prompt window where the Agent is installed.

2. Run the toolbox by entering the full path to the toolbox script file, enclosed in quotes:

   LINUX: "<AGENT_HOME>/ctm/toolbox/toolbox.sh"

   Windows: "<AGENT_HOME>\toolbox\toolbox.bat"

3. In response to the Start message, type Y (yes) and then press ENTER to continue.

4. In the Main Menu, type 3 (Agent Help Tools 1) and press ENTER.

5. In the Agent Help Tools 1 menu, type the number of the relevant option:

   • 1 - Analyze disk space in Agent's directories

   • 2 - WMI Verify

   • 3 - System Monitoring

   • 4 - Permission Check Utility

   • 5 - Java Version Matching Check

   • 6 - Agent Image validation

   • 7 - Extract XML from log file

   • 8 - HTTP(S) Connectivity

   • 9 - Extract SaaS Token

6. In the menu of the selected utility, specify values for all required input parameters and for any of the optional input parameters. For each input parameter, do the following:

   1. Type the number of the option and press ENTER.

   2. Type the value for the input parameter and press ENTER.

7. Type r (Run) and press ENTER.

For more information about each of the tools, including a description of the input parameters and the final output, see Agent Analysis Tools: Input and Output.

Agent Analysis Tools: Input and Output

The following table lists the Agent analysis tools in the Agent Toolbox, along with details about the input parameters and output of each tool:

Tool	Input parameters	Output
1 - Analyze disk space in Agent's directories	1- Debug Level: Determines the level of logging for this test, from 0 (no logging) to 4 (most detailed logging). The default is 0.	A list of Agent directories that cannot be modified, and another numbered list of Agent directories that contain files that can be deleted. You can request more cleanup information about each of the directories in the second list.
2 - WMI Verify (for Windows to Windows connections)	1 - WMI Machine 2 - WMI User 3 - WMI User Domain 4 - WMI User Password 5 - Remote Host Local Directory Path: The default is c:\sysout. 6 - Shared Name: The default is SYSOUT. 7 Debugger: Determines whether to provide additional debugging information for this test, either Y (yes) or N (no). The default is N.	Results of various tests to verify that a Windows Management Instrumentation (WMI) connection can be established and all WMI prerequisites are met. If debugging is turned on, detailed information is provided.
3 - System Monitoring	1 - Time to run tests, in minutes: Determines the total amount of time (in minutes) to run all iterations of the stress tests. Note that the total number of iterations is determined by this parameter divided by the next parameter. 2- Interval, in seconds: Determines the frequency of iterations, that is, the amount of time (in seconds) between iterations. In addition, the stress.properties file in the Toolbox directory (within the Agent Home directory) contains various parameters that control the stress tests, including thresholds for warning and alarm ("problem") states. Review the default values in this configuration file and adjust them as necessary.	Stress tests are performed in the following areas of the system: Hard disk CPU Processes File Descriptors (FD) Results from all iterations of these stress tests are summarized in a file named tool_box_measures.csv, in a directory named Measure within the Agent Home directory. Basic statistics are provided for each test iteration, including time measurements and state (OK, Warning, or Problem).
4 - Permission Check Utility	1 - User name: Defines a name of an existing user. 2: Target path: Determines the path file.	Checks permissions for a specific user and a specific file.
5 - Java Version Matching Check	No input required.	Results of a check for matching of the java version on the Agent.
6 - Agent Image validation	No input required.	Results of a check for problems in the Agent image.
7 - Extract XML from log file	1 - input - log file name: Specifies the name of the log file from which you want to extract XML entries. 2 - output - xml file name: Specifies the name of the .xml output file, in which you want to store the XML entries from the specified log file.	Notification that the .xml file was created and contains entries from the specified log file.
8 - HTTP(S) Connectivity	1 - Mandatory run-option: SSLPOKE 2 - URL endpoint with port: Defines the URL for querying, including the port number. Example: https://google.com 443 3 - HTTPS type: 1: HTTP(S) via java 2: HTTP(S) via Apache HTTPS client Default: 2	Results of an HTTPS connectivity check from the Agent machine using either java HTTPS or an Apache HTTPS client.
9 - Extract SaaS Token	1 - Mandatory run-option: TOKEN 2 - SAAS Token used on Agent install: An Agent Token that you generated for use during installation. For more information, see Generating an Agent Token. 3 - Code used: 1: Uses the same Java as used during Agent installation. 0: Uses similar internal code with more log information. Default: 1	Results of a check of the HTTPS that extracts the endpoint URL from an Agent token that is used during Agent installation.

Analyzing the Agent Operating System

This procedure describes how to run the Agent Toolbox utilities that analyze the Agent operating system.

Begin

1. Open a command prompt window where the Agent is installed.

2. Run the toolbox by entering the full path to the toolbox script file, enclosed in quotes:

   LINUX: "<AGENT_HOME>/ctm/toolbox/toolbox.sh"

   Windows: "<AGENT_HOME>\toolbox\toolbox.bat"

3. In response to the Start message, type Y (yes) and then press ENTER to continue.

4. In the Main Menu, type 4 (Agent Help Tools 2) and press ENTER.

5. In the Agent Help Tools 2 menu, type the number of the relevant option:

   • 1 - Prerequisite Check

   • 2 - File System test

   • 3 - "Secret" files permissions check

   • 4 - Extract Datadog Alerts

6. In the menu of the selected utility, specify values for all required input parameters and for any of the optional input parameters. For each input parameter, do the following:

   1. Type the number of the option and press ENTER.

   2. Type the value for the input parameter and press ENTER.

7. Type r (Run) and press ENTER.

For more information about each of the tools, including a description of the input parameters and the final output, see Agent Operating System Tools: Input and Output.

Agent Operating System Tools: Input and Output

The following table lists the analysis tools for the Agent operating system in the Agent Toolbox, along with details about the input parameters and output of each tool:

Tool	Input parameters	Output
1 - Prerequisite Check (Linux only)	No input required.	Results of a check for compliance with the minimal operating system requirements described in Control-M Limits on UNIX/Linux Accounts.
2 - File System test (Windows only)	1 - Number of loops: Determines the number of loops to run in this test, either a number or infinite. The default is infinite. 2 - Number of threads: Determines the number of threads to run in parallel in this test. The default is 10. 3 - Interval delay: Determines the number of seconds between the creation of threads. The default is 60 seconds. 4 - Log file: Defines the full path and name of a log file for this test. The default is C:\Users\<username>\AppData\Local\Temp\WinFsTest.log 5 - Verbose : Determines whether to include extensive details in the log file, either yes or no. The default is no.	Results of a stress test on a Windows operating system. This test performs various operations, repeated in a loop, and uses multiple threads. Operations include the creation of large files with random data, writing to the registry, and renaming and deleting the created files. The output is written to a log file.
3 - "Secret" files permissions check (Linux only)	No input required.	A list of "secret" files, such as SSH keys or encrypted files, with incorrect read permissions. If no problems were detected, a message notifies you that all files are OK.
4 - Extract Datadog Alerts	1 - Mandatory run-option: UNZIP 2 - Datadog zip-files: Defines the Agent folder that contains alert files or metric files that Control-M/Agent sends to Datadog. The following folders are available: alert: Any alerts that were not yet sent to Datadog due to an Uploader problem. alert/outbox: Alerts that were not sent by the Uploader due to connectivity issues. alert/archive: Alerts that were sent successfully to Datadog metric: Performance metrics that were not yet sent to Datadog due to an Uploader problem. metric/outbox: Performance metrics that were not sent by the Uploader due to connectivity issues. metric/archive: Performance metrics that were sent successfully to Datadog. Default: alert 3 - order: Determines the order of alerts or metrics in the list: DESC: Descending order ASC: Ascending order Default: DESC 4 - output: Defines the destination of the output, either the standard output or a file that you specify: stdout <file path> Default: stdout	A list of alerts or performance metrics created by the Agent and sent by the Agent Uploader to Datadog via AWS S3.

Analyzing the Agent Logs

This procedure describes how to run the Agent Toolbox utilities that analyze Agent logs.

Begin

1. Open a command prompt window where the Agent is installed.

2. Run the toolbox by entering the full path to the toolbox script file, enclosed in quotes:

   LINUX: "<AGENT_HOME>/ctm/toolbox/toolbox.sh"

   Windows: "<AGENT_HOME>\toolbox\toolbox.bat"

3. In response to the Start message, type Y (yes) and then press ENTER to continue.

4. In the Main Menu, type 5 (Agent Log Tests) and press ENTER.

5. In the Agent Log Tests menu, type the number of the relevant option:

   • 1 - Time gaps in log

   • 2 - Analyze daily log

   • 3 - Check Memory Leaks

6. In the menu of the selected utility, specify values for all required input parameters and for any of the optional input parameters. For each input parameter, do the following:

   1. Type the number of the option and press ENTER.

   2. Type the value for the input parameter and press ENTER.

7. Type r (Run) and press ENTER.

For more information about each of the tools, including a description of the input parameters and the final output, see Log Analysis Tools: Input and Output.

Log Analysis Tools: Input and Output

The following table lists the log analysis tools in the Agent Toolbox, along with details about the input parameters and output of each tool:

Tool	Input parameters	Output
1 - Time Gaps in Log	1 - Time gap: Determines the minimum length (in milliseconds) of gaps to search for in the log file. The default is 30 milliseconds. 2 - Log file full path: Determines the file path to the Agent proclog file that you want to analyze.	A list of detected gaps in the Agent proclog file.
2 - Analyze Daily Log	1 - Number of logs: Determines the number of previous daily logs to include in the test. To analyze only today's daily log file, type 0. The default is 0. 2 - Alert threshold: Determines the length of time (in seconds) that a job took to submit, beyond which to issue an alert A notification about the status of a job or component that appears in the Alerts window.. The default is 120 seconds.	A list of alerts for jobs that took a long time to submit (exceeding the specified threshold), as detected in each of the daily logs.
3 - Check Memory Leaks	1 - Print Output to Terminal: Determines whether to display the output on-screen, either y (yes) or n (no). The default is n. If you set this parameter to y, ensure that the Output File Name parameter is set to an empty value, enclosed in quotes (""). 2 - Output File Name: Determines the name of an output file to save in the same directory as the analyzed log files. The default is checkmemoryleak_outputfile.txt. 3 - Logs Directory: Determines the path to the directory that holds the log files that you want to analyze for memory leaks. The default is the proclog directory. 4 - Filter for log files: Analyze only log files with a certain naming pattern. The default pattern is "*.log". 5 - Debug level: Determines the level of logging for this test, from 0 (no logging) to 4 (most detailed logging). The default is 0.	To ensure that leaks are detected, set the Agent debug level to 6 (the highest value). Use the following command: Linux: agdbglvl 6 Windows: agdbglvl -agent <instanceName> 6 A list of leaks in threads, where a malloc function was not followed by a free function, as detected through the proclog files. If you set the debug level of this test to a value above 0, a log file is created for this test in the same directory.

Checking the Job Tracker on the Agent Host

This procedure describes how to run Agent Toolbox utilities that check the Job Tracker on the Agent host.

Begin

1. Open a command prompt window where the Agent is installed.

2. Run the toolbox by entering the full path to the toolbox script file, enclosed in quotes:

   LINUX: "<AGENT_HOME>/ctm/toolbox/toolbox.sh"

   Windows: "<AGENT_HOME>\toolbox\toolbox.bat"

3. In response to the Start message, type Y (yes) and then press ENTER to continue.

4. In the Main Menu, type 6 (Tracker tests) and press ENTER.

5. In the Tracker tests menu, type the number of the relevant option:

   • 1 - Starved jobs

   • 2 - Tracker workers list

   • 3 - Job locks

6. In the menu of the selected utility, specify values for all required input parameters and for any of the optional input parameters. For each input parameter, do the following:

   1. Type the number of the option and press ENTER.

   2. Type the value for the input parameter and press ENTER.

7. Type r (Run) and press ENTER.

For more information about each of the tools, including a description of the input parameters and the final output, see Job Tracker Checks: Input and Output.

Job Tracker Checks: Input and Output

The following table lists the Job Tracker Check tools in the Agent Toolbox, along with details about the input parameters and output of each tool:

Tool	Input parameters	Output
1 - Starved jobs	1 - AT: Determines the Agent Tracker process to query. Keep this parameter set to the default AT (Agent Tracker). 2 - Request: STARVATION 3 - Duration: The minimum length of time, in seconds, since a job status file (in the Agent's STATUS directory) was modified, for inclusion in the list. Default: 10	A list of job status files that have not been modified for some time. This report is written to the Tracker log.
2 - Tracker workers list	1 - AT: Determines the Agent Tracker process to query. Keep this parameter set to the default AT (Agent Tracker). 2 - Request: WORKERS_LIST	A list of Tracker workers currently associated with the Job Tracker, with basic details for each one. This report is written to the Tracker log. Tracker workers are created by the Tracker in response to the accumulation of jobs, and they are responsible for performing monitoring actions on the job status files.
3 - Job locks	1- AT: Determines the Agent Tracker process to query. Keep this parameter set to the default AT (Agent Tracker). 2 - Request: TRACK_LOCKS 3 - Duration: The minimum length of time, in seconds, since a lock was placed on a tracked job, for inclusion in the list. Default: 10	A list of existing locks on jobs that are currently being tracked, with an indication of which Tracker worker is responsible for each lock. This report is written to the Tracker log.

Managing Job Tracking on the Agent Host

This procedure describes how to run Agent Toolbox utilities that perform job tracking tasks on the Agent host.

Begin

1. Open a command prompt window where the Agent is installed.

2. Run the toolbox by entering the full path to the toolbox script file, enclosed in quotes:

   LINUX: "<AGENT_HOME>/ctm/toolbox/toolbox.sh"

   Windows: "<AGENT_HOME>\toolbox\toolbox.bat"

3. In response to the Start message, type Y (yes) and then press ENTER to continue.

4. In the Main Menu, type 7 (Tracker workers tests) and press ENTER.

5. In the Tracker workers tests menu, type the number of the relevant option:

   • 1 - Send job track request

   • 2 - Worker job track lists

6. In the menu of the selected utility, specify values for all required input parameters and for any of the optional input parameters. For each input parameter, do the following:

   1. Type the number of the option and press ENTER.

   2. Type the value for the input parameter and press ENTER.

7. Type r (Run) and press ENTER.

For more information about each of the tools, including a description of the input parameters and the final output, see...

Job Tracking Tasks: Input and Output

The following table lists the Agent Toolbox tools that enable you to perform Job Tracking tasks, along with details about the input parameters and output of each tool:

Tool	Input parameters	Output
1 - Send job track request	1 - Tracker worker name: Determines the name of the Tracker worker. Default: ATW000 2 - CM(Application type): Determines the plug-in or application type of the job. Default: WIN (Windows OS job) 3 - Order ID (5 chars): Determines the 5-character job order ID. 4 - Order number (5 chars): Determines the 5-character job run count. 5 - Job process PID (optional): Determines the job process ID. This parameter is optional.	This tool manually initiates a Job Track request for a specific job. A message regarding this action and its results is written to the Tracker log.
2 - Worker job track lists	1 - Tracker worker: Determines the Tracker worker that you want information for. For example, ATW000 or ATW001. To include information from all Tracker workers, use a value with an asterisk: ATW* Default: ATW000 2 - Request: TRACK_LISTS	A list of jobs that a Tracker worker is responsible for tracking, with basic information for each job. This report is written to the Tracker log.