File Transfer Job
The File Transfer job enables you to watch and transfer files from a local host to a remote host, a remote host to a local host, a remote host to another remote host, or between Cloud storage buckets and containers.
A File Transfer job can contain up to five different file transfer definitions, which are executed sequentially. Each definition can transfer one or more files, or an entire directory with the option of including all of its sub-directories.
File Transfer jobs are integrated with other Control-M jobs into a single scheduling environment.
A File Transfer job enables you to do the following:
-
Rerun file transfers from the exact point where the transfer failed, which saves time and resources, see File Transfer Optional Attributes.
-
Determine how to handle files after a transfer, see File Transfer Advanced General Attributes.
-
Encrypt the transferred files with different encryption methods, see File Transfer SFTP Attributes.
-
Configure pre and post transfer and PGP commands, see File Transfer Commands Attributes.
-
Watch specific files until a maximum size is reached or a defined time has passed, and then transfer the files, see File Transfer File Watcher Attributes.
Defining a File Transfer Job
This procedure describes how to define a File Transfer job, which enables you to watch and transfer files from a local host to a remote host, a remote to a local host, or a remote host to another remote host.
Control-M for AFT jobs on Control-M/Agent and Control-M MFT jobs on another Control-M/Agent cannot belong to the same Host group.
Before you begin
Configure a Control-M Managed File Transfer connection profile, as described in Creating a Centralized Connection Profile.
Begin
-
From the General tab, in the Profile Type field, select one of the following:
-
Two Single Endpoints: Shows connection profiles that are defined with one host. Select this option if the source and destination locations are defined in separate connection profiles.
-
Endpoint1 <-> Endpoint2: Shows connection profiles that are defined with two hosts. Select this option if the source and destination locations are defined in the same connection profile.
-
If you are transferring a file to an external user, you must select the connection profile that is connected to the Hub that uses an SFTP protocol.
-
AS2 connection profiles can be paired with Local connection profiles only. FTP and SFTP are not supported.
-
S3 does not support transfers to and from AS2, MVS, OS400, OS2200, OpenVMS, and TANDEM.
-
S3 ,Google Cloud Storage, and Oracle Object Storage do not support appending content to objects. Therefore, Rerun from point of failure and Append destination action are not supported.
-
-
-
In the Connection profile field, click .
The Select Connection Profile dialog box appears.
-
Select the connection profile(s) that you want to use to transfer files, and click OK.
If you are transferring a file to or from S3, Azure Blob, Oracle Cloud Storage, or Google Cloud Storage, in the Bucket or Container field(s), click and select the required bucket or container.
-
In the File Transfers area, do the following:
-
In the left field, browse for the file, directory, or library to transfer. You can also type the path manually.
-
If you are transferring a file to an external user, you must type in the B2B Subfolder home directory (example: /b2bhome) that is defined in MFT Enterprise B2B settings, as described in General Settings .
-
In AS2 you cannot determine the destination file path. You can override the destination filename, subject, and content type.
-
-
In the right field, browse for the file, directory, or library to transfer. You can also type the path manually. If you are transferring a file to multiple hosts, select one of the following:
-
Home Directory: Transfer the file to the home directory on each host.
-
Common Destination Path: Transfers the file to the same location on each host defined here.
-
Different Destination Paths: Transfers the file to different destinations on each host. Click and define each path by browsing to each file in File Selection or modify the locations in one view in Paths List.
You can use variables in the left or right transfer fields, as described in File Transfer Advanced General Attributes.
-
-
Select the transfer option, as described in Transfer Options.
You can transfer up to five files at a time. File transfers within a job are sequential, therefore, if a file transfer fails subsequent file transfer definitions are not performed unless you select the Continue on failure checkbox, as described in File Transfer Advanced General Attributes.
If a transfer has CJK or Latin-1 characters in the filename, the Control-M/EM server and Control-M/Server databases must be compatible with this format.
-
-
Click Advanced.
The File Transfers dialog box appears.
-
Select the required options, as described in File Transfer Advanced General Attributes, and click OK.
-
Select Binary or Ascii from the dropdown list.
EBCDIC format is supported only when transferring files between two MVS FTP hosts.
-
Select optional parameters if required, as described in File Transfer Job.
-
Click the Advanced drop down list.
The Pre-execution Command and Post-execution Command fields are used to define specific commands to execute at the beginning or at the end of a job. These commands can only be executed on a computer where Control-M/Agent is installed, and the output of the commands (pre or post) appears in the job output.
Transfer Options
If you have successfully watched a file using a variable, you can share that variable to an additional transfer in the Source path within the same job using the following variables.
The following table lists the Transfer Options.
Icon |
Description |
---|---|
|
Transfers the file from Host1 to Host2 |
|
Transfers the file from Host2 to Host1 |
|
Watches the file until it reaches a certain size or passes a time limit, and then transfers the file from Host1 to Host2. For more information on file watching, see File Transfer File Watcher Attributes. |
|
Watches the file until it reaches a certain size or passes a time limit, and then transfers the file from Host2 to Host1. |
|
Watches the file until it reaches a certain size or passes a time limit. For more information on file watching, see File Transfer File Watcher Attributes. You can use a wildcard to watch files in a directory but the wildcard cannot be used for a directory path itself. A filewatcher path of /directory/testfiles/* finds any files in the directory, but /directory/*/testfile.txt cannot find any files. |
|
Lists all the files and directories from either the source or destination hosts or both in the job output. This option enables you to get a snapshot of a directory listing to save, archive, or mail it or automate a following process that needs all files in that content. Directory of C:\Users\vagrant\AppData\Roaming\Microsoft\Windows\ServerManager 05/17/2014 06:47 AM 341 ServerList.xml
|
|
(Windows/Linux only) Synchronizes all files from host1 to host2 by comparing all files in the source and destination folders based on the last modification and does the following:
|
|
(Windows/Linux only) Synchronizes all files from host2 to host1 by comparing all files in the source and destination folders based on the last modification and does the following:
|
-
$$WATCH_ALLn$$: Determines the full file name including the extension of the successfully watched file
-
$$WATCH_NAMEn$$: Determines the file name only without the extension of the successfully watched file
-
$$WATCH_EXTn$$: Determines the extension of the successfully watched file
n represents the number of the watch/transfer of the possible 5 in the job.
If you selected the Rerun from point of failure checkbox, you cannot use the above variables.
File Transfer Optional Attributes
Attribute | Description |
---|---|
End job Not OK when “Continue on failure” option is selected |
If the file transfer job fails and the Continue on failure checkbox is selected, the job status changes to Not OK. |
Rerun from point of failure |
Resumes the file transfer from the point that it failed. If you are connecting to an FTP server you must select this checkbox and the Restart from failure checkbox, as described in Creating a Centralized Connection Profile. The transfer can only be resumed from the exact position where the failure occurred if the FTP server supports REST commands. If you HOLD a job and then rerun it, the job automatically reruns from the beginning. If the Append option is selected in the General tab, the Rerun from point of failure option is ignored. If you are transferring a file from one host to a group of hosts, you cannot use this option. |
Number of retries |
Uses the default or a custom number of retries to re-establish a failed connection. If you are using a z/OS platform, this checkbox is disabled. |
File Transfer Advanced General Attributes
The File name field in the Source file action after successful completion, Destination file action after successful completion and Create Empty File fields parameters can contain variables, constants, and the following variables:
-
$$AFTFILE$$: Contains the name of the original file without file extension. (If there was an extension, it is always added to the name of the file.)
-
$$AFTFILE_ALL$$: Contains the full file name including the extension
-
$$AFTFILE_NAME$$: Contains only the name of the file until the last dot (without the extension)
-
$$AFTFILE_EXT$$: Contains only the extension of the file name
If you want to rename a transferred file old_cmd_060101.exe, when it was originally named cmd.exe, the pattern you must specify is old_$$AFTFILE$$_%%DATE.
The following table lists the File Transfer Advanced general attributes.
Attribute |
Description |
---|---|
Source File Action After Successful Completion |
Determines how a source file is handled after a successful transfer, as follows:
If the action fails, Control-M for MFT attempts a retry according to the number of retries Variable Name: %%FTP-SRCOPTx |
File Name |
Determines the path and name of the file. Rules:
|
Fail transfer when action on source file failed |
Determines whether to fail the transfer when a post action on the source file fails after a successful transfer. Variable Name: %%FTP-SRC_ACT_FAILx |
If a file with the exact same name as already exists |
Determines how to manage a file that has the same name as the destination file, as follows:
Variable Name: %%FTP-IF-EXISTx |
Destination file action after successful completion |
Determines how a destination file is handled after a successful transfer, as follows:
Variable Name: %%FTP-DSTOPTx |
Fail transfer when the action on the destination file failed |
Determines whether to fail the transfer when a post action on the destination file fails after a successful transfer. |
Use temporary file prefix |
Prevents overwriting files by attaching a temporary file prefix or suffix, such as FTP_ to the destination file until the transfer completes successfully.
The temporary file name that is created by applying the prefix or suffix must be a valid file name for the operating system of the destination host computer.
|
Create empty file in destination |
Creates an empty file after a transfer on the destination host. If you are transferring a file or multiple files, the empty file is created in the directory where the file(s) are transferred. If you are transferring a directory, the empty file is created in the directory where the transfer directory is created. This feature does not work in Recursive mode. Variable Name: %%FTP-EMPTY_DEST_FILE_NAMEx |
Continue on failure |
Allows successive transfers to run when this specific transfer fails. Variable Name: %%FTP-CONT_EXEx |
Do post AFT command on failure |
Activates the defined post commands when the job fails. Variable Name: %%FTP-POSTCMD_ON_FAILUREx |
Delete destination file on failure |
Deletes the destination file if the transfer failed. If you transferred multiple files, the last file that failed is deleted. Variable Name: %%FTP-DEL_DEST_ON_FAILUREx |
Recursive |
Transfers a file directory with all its sub-directories and all files that match the wildcard pattern in the source file path field under all sub-directories. Files are transferred in the destination with the same directory structure as the source directory structure. |
Exclude Files |
Determines whether to exclude all the files that match the transfer pattern when you define a transfer with a wildcard. |
Do not delete destination directory files during directory synchronization |
Determines whether to delete destination directory files during directory synchronization. Variable Name: %%FTP-SYNC_DIR_NO_DELn |
Incremental Transfer (transfer files based on modified timestamp since the last successful job execution) |
Determines whether to transfer files that were added or modified in the source directory since the last successful job execution. You might want to use this option in a cyclic job to transfer the changes from the source directory. You might want to use this option in a cyclic job to transfer the changes from the source directory. UNIX/Linux Only:
|
Transfer files based on modified timestamp in the last ___ hours on the first job execution |
Determines whether to transfer files that were modified in the specified interval or to transfer all files that match the pattern on the first job execution. |
File Transfer OS400 Attributes
The following table lists the File Transfer OS400 attributes.
Attribute |
Description |
---|---|
Trailing blanks options for OS/400 database record |
Determines how Trailing Blanks are managed:
Variable Name: %%FTP-TRIMx |
Allow Transfer of Files with NULL Fields |
Transfers files to an OS400 platform with NULL fields. Variable Name: %%FTP-NULLFLDSx |
Allow OS400 IFS Case Sensitive Mode |
Supports the transfer of case sensitive file systems. Variable Name: %%FTP-CASEIFSx |
File Transfer z/OS Attributes
The following table lists the File Transfer z/OS attributes in the z/OS tab:
Attribute |
Description |
---|---|
Template Name |
Defines the name of the z/OS template that is defined in z/OS Template Parameters. Rules:
Variable Name: %%FTP-TEMPLATEx |
Record Format |
Determines the value for the record format. Variable Name: %%FTP-TEMPLATEx |
Logical Record Length |
Determines the logical record length values between 0-32760. Variable Name: %%FTP-LRECLx |
Block Size |
Determines the block size values between 0-32760. Variable Name: %%FTP-BLKSIZEx |
Translate Table |
Determines what table is used by the FTP server during transfer for translation (8 characters maximum or select from list). If you want to add or delete values to and from the list, you need to add them as comma separated values, to the XLT.dat file in the following location: Variable Name: %%FTP-TRANSTABx |
DBCS Encoding |
Determines which DBCS data sets is used for the transfer. Variable Name: %%FTP-DBCSx |
Additional Options |
Defines the z/OS FTP Server SITE command sub-parameters and values (214 characters maximum). To work with GDG data sets, type: DCbdsn=model_dataset_name. Variable Name: %%FTP-ADDOPTxy |
Allocation Units |
Determines which allocation unit type is used for the transfer. Variable Name: %%FTP-ALLOCUNITSx |
Volume |
Defines the volume value (6 characters maximum). Variable Name: %%FTP-VOLUMEx |
Unit |
Defines unit value (8 characters maximum) Variable Name: %%FTP-UNITx |
Primary |
Defines the primary allocation amount between 1-16777215. Variable Name: %%FTP-PRIMARYx |
Secondary |
Defines the secondary allocation amount between 0-16777215. |
Transfer Mode |
Determines the mode used to transfer files.
Variable Name: %%FTP-TRANSMODEx |
SMS Management Class |
Determines which SMS Management Class is assigned to a new data set.
Variable Name: %%FTP-MGMTCLASS |
SMS Data Class |
Defines the SMS Data Class provided by your organization for the FTP server.
Variable Name: %%FTP-DATACLASS |
Transfer to Unique File |
Creates a file with a unique name on the remote system instead of overwriting an existing file. The FTP server on the remote system sends the name of the created file back to the File Transfer client. If you select this option, you cannot do any of the following:
Variable Name: %%FTP-UNIQUEx |
File Transfer SFTP Attributes
The following table lists the File Transfer SFTP attributes.
Attribute |
Description |
---|---|
Compression |
Compresses the file before the transfer. Variable Name: %%FTP-COMPRESSIONxy
For example, when performing a file transfer from one z/OS host to another z/OS host, the variable %%FTP-COMPRESSION21 contains the compression value to set for the connection to the z/OS host defined on the right side of the Connection Profile Details dialog box, in the first file transfer. |
Preserve the File Attributes and Timestamps |
Preserves the source’s file permissions and timestamps on the destination host. Variable Name: %%FTP-PRESERVE_ATTRx |
File Transfer Commands Attributes
The following table below lists the File Transfer Commands attributes.
Attribute |
Description |
---|---|
FTP pre command |
Determines the pre-command for each host, as follows:
Variable Names:
Except for the local host, where the default path is the Control-M/Agent's home directory, you must define the full path when you run the chmod, mkdir, rename, rm, and rmdir commands. For z/OS hosts you do not need to define the full path. You can use the home directory path that is defined in the connection profile definition. |
Fail transfer when the command on the source host failed |
Determines whether to fail the transfer when the command on the source host fails before or after a successful transfer. |
FTP post command |
Determines the post command for each host, as follows:
Variable Names:
|
Fail transfer when the command on the destination host failed |
Determines whether to fail the transfer when the command on the destination host fails before or after a successful transfer. |
Enable PGP encryption |
Enables PGP encryption commands for this transfer. Variable Name: %%FTP-PGP_ENABLEx |
PGP operation |
Determines whether to encrypt or decrypt the transfer. Variable Name: %%FTP-PGP_OPERATIONn |
Keep encrypted files |
Saves the PGP encrypted file after encryption with the following name format: Variable Name: %%FTP-PGP_KEEP_FILESx |
Template Name |
Determines the name of the PGP template that is defined in Creating a PGP Template. Variable Name: %%FTP-PGP_TEMPLATE_NAMEx |
Destination Filename Attributes
The following table describes attributes that enable you to change the destination filename for each file that is transferred when the destination path is a directory. The order of the changes to the destination filename is by Name Pattern, Find & Replace, and then Case Handling.
Attribute |
Description |
---|---|
Pattern |
Defines the rename pattern of the destination filenames based on the following attributes.
|
Find & Replace | Renames the destination file according to find and replace. |
Case Handling |
|
Destination Filename Examples
The following table shows examples of typical Destination filename attributes usage.
Source Name |
Pattern |
Find and Replace |
Case |
Destination filename |
---|---|---|---|---|
sample_file.txt | [N3-].[E] |
|
Unchanged | mple-cool.txt |
sample-file.docx | [N].[E2-3] | To uppercase | SAMPLE-FILE.OC | |
Sample-File.txt | [N] ([C3]).[E] |
|
To lowercase |
sale-file (001).txt Only if it is the first file. |
simple.simple.txt | [N]-[D] |
|
simon.simon-20190704 |
File Transfer File Watcher Attributes
The following table below lists the File Transfer File Watcher attributes.
Attribute |
Description |
---|---|
Time Limit Type |
Determines whether the values defined in the Time Limit field are number of minutes to wait or an absolute time. Variable Name: %%FTP-ABSTIMEx |
Time Limit |
Determines the number of minutes to wait, in the Time Limit field, for a file to reach its minimum detected size and remains static, before the watch fails or determines the specific time, in the Time Limit field, that a file has to reach its minimum detected size and remains static, before the watch fails. Variable Name: %%FTP-TIMELIMITx |
Minimum Detected Size |
Defines the minimum number of bytes transferred before Control-M File Transfer checks if the file is static. File size information for remote file systems is not supported. This feature is not supported on OS400, Tandem, MVS, and OS2200 when the connection type is FTP. Variable Name: %%FTP-MINSIZEx |
Minimum File Age |
Defines the minimum number of years, hours, or days that must pass since the watched file was last modified. 2y3d means that after 2 years and 3 days must pass before the file can be watched. Variable Name: %%FTP-FILE_MIN_AGEx |
Maximum file Age |
Defines the maximum number of years, hours, or days that can pass since the watched file was last modified. 2y3d means that after 2 years and 3 days have passed the file is no longer watched. Variable Name: %%FTP-FILE_MAX_AGEx |
Override Default Interval between File Searches |
Determines whether to override the value of the Interval between file searches field in Configuration dialog box in the CCM. |
If Age and Size Criteria Are Not Met, Continue Searching Matching Files |
Determines whether to continue searching the remaining files if the first file that matches the name pattern doesn't match the size and age criteria. |
Transfer All Matching the Name Pattern |
Determines whether to transfer all remaining static files that match the name pattern (regardless of the size and age) after a single file matches all criteria. Variable Name: %%FTP-TRANSFER_ALLx |
Transfer Files Matching Size and Age Criteria |
Determines whether to transfer all remaining static files that match the name, size, and age criteria after a single file matches all criteria. Variable Name: %%FTP-TRANSFER_ALLx |
Variable Containing Detected Filename |
Defines the variable name that contains the detected file name. Variable Name: %%FTP-WATCHNAMEx |
Variable Type |
Determines whether the variable is one of the following:
|
-
The Maximum File Age and Minimum File Age features are supported if the user defined for the watch host has write permissions on the destination path.
-
In the job log, if the OSCOMPSTAT parameter is 7 it indicates that file watch failed.
Transfer Formats
The following table describes the different formats you can use to transfer files, directories or libraries.
Transfer Format |
Description |
---|---|
File to File |
Transfers a single file from a defined source to a defined destination. To transfer Readme.txt from its source location to a defined destination, renaming the file to Writeme.txt, define D:\Temp\Readme.txt in the source transfer field, and C:\Directory1\Writeme.txt in the destination transfer field. If no file name is specified in the destination transfer field, the file retains its original name. However, a destination file name must be specified when MVS is the transfer destination. |
Directory and Contents (OS/400, Microsoft Windows and UNIX only) |
Transfers a directory from a defined source to a defined destination. To transfer the /h/userb/ directory to /p/home2/, define /h/userb/ in the source transfer field, and /p/home2/ in the destination transfer field. The userb directory is created under /p/home2/. The Recursive transfer option is available. If this is selected, all sub-directories are also transferred. |
-
MVS, OS/400, and OS2200 systems support the * wildcard character only when it is used in the latter part of a file name.
-
MVS supports the % wildcard character.
-
If you use a wildcard, and no files match the defined pattern, the job ends successfully.
-
Transfer of file names with spaces are supported only for platforms that support a convention of file names that include blanks.
Control-M for MFT Job Statuses
The following table describes the different statuses in a Control-M for MFT job:
Status | Description |
---|---|
Waiting |
Waiting for execution. |
In Progress |
File is transferring. |
Ended OK |
Transfer completed successfully. |
Failed |
File failed to transfer. |
Killed |
Transfer is canceled by request. |
Abandoned |
File did not transfer due to a previous transfer that failed. |
Skipped |
File transferred successfully in a previous attempt, but it is part of a job that failed. File transfer is skipped when the job returns. |
File Watching |
File is watched until it reaches a certain size or has reached a maximum time limit. |
If you are transferring a file from z/OS, OS/400, or OpenVMS, or transferring more than a single file in a recursive transfer, the size of the transferred files is not known. Therefore, the progress bar and Time Remaining fields are set at zero and updated to 100% at the end of a successful file transfer. Similarly, the Transferred Out of Total and Rate fields are updated every 30 seconds.