File Watcher (ctmfw)

The ctmfw (Control-M File Watcher) utility, enables you to detect when files are created, deleted, or successfully transferred. You can use this utility before you activate a job or perform a task, such as when you create an eventClosed An conditional entity that creates a sequential relationship between jobs by enabling the successor job to execute after the predecessor job has executed. that depends on an existing file.

The Control-M File Watcher utility does not detect files with a modify time that is older than 50 years.

The ctmfw utility runs as a process on the Agent host, which waits for the creation or deletion of a user-defined file.

The ctmfw utility does the following:

  • File Transfer Activity: Monitors the size of a file after it is detected. When the file reaches a user-defined minimum size and does not increase in size for a user-defined period of time, ctmfw ends with a status of OK or executes a specified If-Action, such as the creation or deletion of an event or a command execution.

  • File Creation: Ignores the size of a file if a wildcard is specified as part of its filename unless the mon_size_wildcard parameter is set to Y.

  • File Deletion: Monitors for the deletion of a file after it is detected.

The ctmfw utility runs from the CLI and can detect one or more files, as described in Watching a Single File and Watching Multiple Files.

There are two usages for this utility:

  • Service: (Windows only) As a service, ctmfw takes its parameters (rules) during startup from the rull.dat file, whose pathname is specified in <Agent>\data\ctmfw.cfg. For more information, see Usage as a Service.

  • Running as a Utility: ctmfw is invoked from the CLI. Rules can be provided in the command line or by a rule file. You can watch for one or more files, as described in Watching a Single File and Watching Multiple Files.

    To allow other users to run ctmfw, they must have write permissions to $CONTROLM, and the CONTROLM environment variable must be set. For more information, see Running Agent Utilities as Other Users on UNIX. For details about ctmfw processing and return codes that are issued after the utility detects the creation or deletion of a file in the specified time frame, see ctmfw Utility Processing and Return Codes.

Usage as a Service

As a service, ctmfw takes its parameters (rules) during startup from the rull.dat file, whose pathname is defined in the following file:

<Agent_Home>\DATA\ctmfw.cfg

To change one or more rules, you must change the contents of the rull.dat file, which is provided with Agent as a sample file. The ctmfw.cfg configuration file pathname must be defined in the following Windows registry key, which is automatically generated by the installation script:

HKEY_LOCAL_MACHINE\SOFTWARE\BMC Software\CONTROL-M/FileWatcher\SYSPRM\File Watcher Configuration File

You must have administrative permissions to update this registry key. It is recommended that the default value not be changed.

The ctmfw.cfg configuration file must also contain the following line:

-input <ruleFileName>

where the <ruleFileName> parameter defines pathname of a rule file that contains the File Watcher rules.

The following example shows a sample rull.dat file:

INTERVAL 5

FROM_TIME 0001

MIN_SIZE 50

MIN_DETECT 5

WAIT_TIME 2

ON_FILEWATCH NONEXIST CREATE 10 3 1

THEN

DO_CMD "BAD FILE WAS CREATED IN 1 MINUTE"

ELSE

DO_CMD "GOOD FILE WAS NOT CREATED IN 1 MINUTE"

END_ON

The FileWatcher service that runs on the local system account cannot detect network resources (files that are located on remote systems). To enable FileWatcher to detect network resources, you must configure the FileWatcher service to run under a regular user account instead of the Local System Account.

When it runs as a service, ctmfw generates an execution log file, which is saved in the Agent proclog directory, as follows:

U_CTMFW_<process_id>.log

The following example shows a Trace file output:

2017/03/10 13:04:24 182 FW:set INTERVAL=3

2017/03/10 13:04:24 182 FW:set MIN_SIZE=4

2017/03/10 13:04:24 182 FW:set FROM_TIME=0909

2017/03/10 13:04:24 182 FW:ctmfw:command line 'ON_FILEWATCH tst CREATE'(arg#=3).

id=1.

2017/03/10 13:04:24 182 FW:ctmfw:command line 'ON_FILEWATCH prd CREATE 0 0 1 NOW

10'(arg#=8)id=2

2017/03/10 13:04:24 182 FW:ctmfw:command line 'ON_FILEWATCH abc DELETE 0 0 1 NOW

10'(arg#=8)id=3

2017/03/10 13:04:24 182 FW:File 'test' exists, its current size is 265.

id=1.

2017/03/10 13:04:24 182 FW:File 'test' has reached the minimum size of 4.

size=265 bytes id=1.

2017/03/10 13:04:24 182 FW:File 'abc' does not exist. id=3.

2017/03/10 13:04:36 182 FW:File transfer was completed. The size of file 'test'

is 265. id=1.

2017/03/10 13:05:09 182 FW:Executing:< dir >

2017/03/10 13:05:27 182 FW:File 'prd' was not CREATED within the time limit.

id=2.

2017/03/10 13:05:27 182 FW:File prd will be scanned at 1315. id=2.

2017/03/10 13:05:27 182 FW:File 'abc' was not DELETED within the time limit.

id=3.

2017/03/10 13:05:27 182 FW:File abc will be scanned at 1315. id=3.

2017/03/10 13:05:30 182 FW:File prd, is out of time window. next time:1315, id=2.

2017/03/10 13:05:30 182 FW:File abc, is out of time window. next time:1315, id=3.

2017/03/10 13:15:01 182 FW:File prd, entered the time window from '1315' for

monitoring, id=2.

2017/03/10 13:15:01 182 FW:File abc, entered the time window from '1315' for

monitoring, id=3.

By default, proclog directory logs are retained for one day. If the maximum days to retain output files parameter is set to a number higher than 1, logs are retained for the number of days specified in that parameter.

Watching a Single File

This procedure describes how to watch a single file with the ctmfw File Watcher utility. Rules can be provided in the command line or by a Rule file.

Begin

  1. Run the following command:

    ctmfw FILE (absolute path)

  2. Enter the required parameters.

    Copy
    < mode (CREATE|DELETE)> Default: CREATE
    < minimum detected size <number>
    [' '|Bytes(B)|Kilo(K)|Mega(M)|Giga(G)] >Default:0
    < interval between file search (seconds) > Default: 60sec
    < interval between filesize comparison iterations (seconds) > Default: 10sec
    < number of iterations while the size is static > Default: 3 iterations
    < time limit for the process (minutes).> Default: 0 (no time limit)
    Effective while the file does not exists or,
    the file size is static and the minimum size
    was not reached >
    < monitor file size , minimal and maximal age, when wildcard is used > Default: N
    < starting time for detecting files (HHMM or YYYYMMDDHHMM > Default: NOW
    < absolute stop time (HHMM or YYYYMMDDHHMM > Default: +0000 ( No stop time )
    < minimal age of file (modified time)
    format:xxxxYxxxxMxxxxDxxxxHxxxxMin > Default: NO_MIN_AGE
    < maximal age of file ( timestamp monitoring )
    format:xxxxYxxxxMxxxxDxxxxHxxxxMin > Default: NO_MAX_AGE

    All parameters must be assigned a value, even if that value is zero. If only six values are specified, the default value for mon_size_wildcard is used. If five parameters are specified, default values for wait_time and mon_size_wildcard are used.

    ctmfw /home/watchedfile.txt CREATE 100 10

    is resolved by using default values for mon_int, min_detect, wait_time, and mon_size_wildcard as follows:s

    ctmfw /home/samplefile.txt CREATE 100 10 10 3 0 N

ctmfw Parameters

The following table lists the ctmfw utility rule file parameters.

Parameter

Description

FILE

Determines the required file pathname to monitor.

Rules:

  • Wildcards: The filename can include the * and ? wildcards, as described in Pattern-Matching Strings.

  • Characters: The pathname and filename cannot exceed 214 characters.

mode

  • CREATE: Monitors file creation. The file size is ignored if the filename parameter contains wildcards, unless the monitor file size when a wildcard is used parameter is set to Y.

    If a mask is specified as the filename and the monitor file size, and a wildcard is used, the parameter can be set as follows:

    • N: Ends OK after detection of the first file that matches the specified mask.

    • Y: Ends OK after detection of the first file that matches the filename and file size.

  • DELETE: Monitors file deletion. ctmfw first looks for files that match the user-defined filename and then checks for the deletion of that file during the user-defined interval.

    If a mask is specified as the filename, ctmfw ends successfully only after all detected files that match the specified mask are deleted.

Default: CREATE

minimum detected size

Determines the minimum file size in bytes. This parameter is ignored if the FILE parameter contains wildcards (unless the monitor file size when wildcard is used parameter is set to Y) or if the mode parameter is set to DELETE.

Default: 0 (any size detected).

interval between file searches

Determines, in seconds, the interval between successive attempts to detect the existence/deletion of a file.

Default: 60

interval between filesize comparison iterations

Determines, in seconds, the interval between attempts to monitor the size of a file after it is detected. This parameter is ignored when using wildcards in FILE or when using DELETE mode.

Default: 10

number of iterations while size is static

Determines the number of attempts at monitoring the file size, where the size remains static and greater than or equal to minimum detected size, (indicating successful creation of the file). This parameter is ignored when using wildcards in FILE or when using DELETE mode.

Default: 3

time limit for the process

Determines, in minutes, the maximum amount of time taken to run the process without detecting the file at its minimum size (CREATE), or detecting its deletion (DELETE). If the file is not detected/deleted in this specified time frame, the process terminates with an error return code, as described in Return codes.

Default: 0 (no time limit).

monitor file size when wildcard is used

Determines whether the file size should be monitored if the filename contains wildcards. This parameter is ignored if the filename does not contain a wildcard.

Valid Values:

  • N: Does not monitor file size.

  • Y: Monitors the file size.

Default: N

If this parameter is set to Y and more than one file matches the specified mask, the ctmfw utility selects the first file that is detected, monitors its file size, and ignores all other matching files.

starting time for detecting files

Determines an absolute time at which the utility starts monitoring the file.

200712061400 means that at 14:00 (02:00 pm) on 6 December 2007, the File Watcher utility starts to watch the file.

Alternatively, you can use the HHMM format to use the current date.

absolute stop time

Determines an absolute time at which the file is no longer watched.

200712061400 means that at 14:00 (02:00 pm) on 6 December 2007, the File Watcher utility starts to watch the file.

Alternatively, you can use the HHMM format to use the current date.

maximal age of file

Determines the maximum amount of time that can pass since the file you want to watch was last modified.

  • 2y3d5h means that after 2 years, 3 days, and 5 hours have passed, the file is no longer watched.

  • 2H10Min, means that after 2 hours and 10 minutes have passed, the file is no longer detected.

This parameter is ignored if the mode parameter is set to DELETE.

Default: 0

If you use the default value, files older than 50 years cannot be detected by the utility. To detect files older than 50 years, you must define the number of years, such as 51y.

minimal age of file

Determines the minimum amount of time that must have passed since the file you want to watch was last modified.

  • 2y3d5h means that after 2 years, 3 days, and 5 hours have passed, the file is no longer watched.

  • 2H10Min, means that after 2 hours and 10 minutes have passed, the file is no longer detected.

This parameter is ignored if the mode parameter is set to DELETE.

Default: 0

Watching Multiple Files

This procedure describes how to watch multiple files with the ctmfw File Watcher utility.

Begin

  • Type the following command to invoke the ctmfw utility for multiple files:

    ctmfw -input <ruleFileName>

    The variable <ruleFileName> is the complete path name of the file containing the definitions for each file to be detected. The .Rule file contains the following sections:

    • Global parameters, whose default values apply to all the files in the rule file. For more information, see ctmfw Rule File Global Parameters.

    • ON_FILEWATCH statements identifying which files to detect, specific criteria for each file, and the action to take upon detection or non-detection. Any number of ON_FILEWATCH statements can appear in a Rule file. For more information about ON_FILEWATCH parameters, see File Watcher (ctmfw).

    If any mandatory parameter is omitted from a Rule file, the default value for that parameter is used. Parameters entered for ON_FILEWATCH statements override the default values.

ctmfw Rule File Global Parameters

The following table lists the ctmfw Rule file global parameters:

Parameter

Description

CYCLIC_INTERVAL

Determines, in minutes, the interval between multiple operations of detecting the file. This interval must be greater than the value for WAIT_TIME. If the CYCLIC_INTERVAL is 0, only one attempt to detect the file will be performed.

Default: 0

FROM_TIME

Determines the starting time for detecting all the files (default FROM_TIME). Used with WAIT_TIME to identify the time frame for detecting and monitoring the files. This parameter is expressed in 24-hour, HHMM format.

Default: 0000 or Now

INTERVAL

Determines, in seconds, the Sleep interval between successive scans for all the files. This parameter replaces individual sleep_int and mon_int parameters for each file.

Default: 10

MAX_AGE

Determines the maximum amount of time that can pass since the file you want to watch was last modified.

  • If MAX_AGE = 0, any change to the file timestamp means that the event is met.

  • IF MAX_AGE = 10 Min and if the amount of time of the watched file that has passed is less than 10 minutes, then the event is met.

This parameter is ignored if the mode parameter is set to DELETE.

Default: 0

MIN_AGE

Determines the minimum amount of time that must have passed since the file you want to watch was last modified.

2y3d5h means that after 2 years, 3 days, and 5 hours have passed, the file is no longer watched.

This parameter is ignored if the mode parameter is set to DELETE.

Default: 0

MIN_SIZE

Determines whether the file size should be monitored if the filename contains wildcards. This parameter is ignored if the filename does not contain a wildcard.

Valid Values:

  • N: Does not monitor file size.

  • Y: Monitors the file size.

Default: N

If this parameter is set to Y and more than one file matches the specified mask, the ctmfw utility randomly selects one matching file, monitors its file size, and ignores all other matching files.

STOP_TIME

Determines the absolute time at which the file is no longer watched. For example, 200702061400, means that at 2 PM on February 6th, 2007 the FileWatcher utility will stop watching the file.

You can also use the HHMM format, which uses the current date, plus the HHMM entered.

Default: 0 (meaning, no stop time)

STOP_TIME can only be used as a global parameter.

WAIT_TIME

Determines, in minutes, the maximum time to run the process without detecting the file at its minimum size (CREATE), or detecting its deletion (DELETE). If the file is not detected/deleted in this specified time frame, the process terminates with an error return code, as described in ctmfw- return codes table below.

Default: 0 (no time limit).

ctmfw Utility Processing

If the file is detected and the file size remains static within the time frame (CREATE) or the file has been deleted (DELETE), the DO commands in the THEN block are executed. If the file is not detected/deleted within the time frame, the statements following the ELSE block are executed.

CTMFW terminates when either all the files in the Rule file have been processed or a DO_OK/DO-NOTOK action is executed. If any ON_FILEWATCH statement contains a cyclic_interval, CTMFW only terminates on a DO_OK/DO-NOTOK action. If the cyclic_interval Global parameter is specified and not 0, CTMFW only terminates on a DO_OK/DO-NOTOK action or when the STOP_TIME was reached.

  • The sleep interval between succeeding scans must be 10 seconds.

  • If the ctmfw utility detects that the datafile.txt file in the /home/controlm directory is created in the specified time interval, then the command interpreter must execute the command to move the contents of the ~<controlm_owner>/ctm_server/datafile.txt file to~<controlm_owner>/ctm_server/workfile.txt.

  • If the ctmfw utility detects that the datafile.txt file in the ~<controlm_owner>/controlm directory is not created in the specified time interval then command that executed is "FILE WAS NOT CREATED".

  • When the ctmfw utility detects that the ~<controlm_owner>/ctm_server/tempfile.txt file is deleted, then the following command is executed "FILE WAS DELETED".

Copy
#******************************************************************
INTERVAL 10
ON_FILEWATCH ~<controlm_owner>\ctm_server\datafile.txt CREATE
THEN
DO_CMD move \ctm\datafile.txt \ctm\workfile.txt
ELSE
DO_CMD "FILE WAS NOT CREATED"
END_ON
ON_FILEWATCH \ctm\tempfile.txt DELETE
THEN
DO_CMD "FILE WAS DELETED"
END_ON
#*******************************************************************

Valid Actions

The following table lists the ctmfw valid actions:

If the file is detected and the size remains static within the time frame (CREATE) or the file has been deleted (DELETE), the DO commands in the THEN block are executed. If the file is not detected or deleted within the time frame, the statements following the ELSE block are executed. ctmfw terminates when all the files in the Rule file have been processed.

If an ON_FILEWATCH statement contains a cyclic_interval parameter, ctmfw will only stop monitoring a file on a DO_OK or DO_NOTOK action.

Action

Description

DO_CMD <command>

Execute a valid command under the command interpreter. Full path names are required for files.

DO_EXIT [exit code]

Terminate ctmfw with the user-defined exit code

DO_NOTOK [exit code]

Terminate an ON_FILEWATCH statement with status NOTOK. Exit code is optional and replaces the standard return code, as described in the ctmfw-return codes table below.

DO_OK

Terminate an ON_FILEWATCH statement with status OK. If there is more than one file in the Rule file, the result displayed is that of an AND algorithm.

Return Codes

The following table describes ctmfw utility return codes that appear after determining whether a file has been is created or deleted in the user-defined time frame.

Return Code

Description

0

The file was successfully created or deleted (it arrived) in the specified time frame, and the file size is above or equal to the minimum user-defined size.

1

One of the following errors has occurred:

  • The utility failed, for example because of a syntax error.

  • A DO_NOTOK statement occurred, but no user-defined exit code was provided for the DO_NOTOK statement.

2

The ctmfw utility request timed out because the file was not detected in the user-defined time frame.