File Transfer Job

The File Transfer job, Job:FileTransfer, enables you to transfer files to and from local hosts, remote hosts, and cloud storage buckets and containers. You can define up to five different file transfer definitions, which are executed sequentially. Each definition transfers one or more files, subdirectories, or directories.

The following examples show how to define a File Transfer job:

  • This JSON defines a file transfer from a local file system to an SFTP server.

    Copy 
    {
       "FileTransferFolder": 
       {
          "Type": "Folder",
          "Application": "aft",
          "TransferFromLocalToSFTP": 
          {
             "Type": "Job:FileTransfer",
             "ConnectionProfileSrc": "LocalConn",
             "ConnectionProfileDest": "SftpConn",
             "NumberOfRetries": "3",
             "Host": "AgentHost",
             "FileTransfers": [
             {
                "Src": "/home/controlm/file1",
                "Dest": "/home/controlm/file2",
                "TransferType": "Binary",
                "TransferOption": "SrcToDest",
                "DeleteFileOnDestIfFails": true,
                "ContinueOnFailure": true,
                "FailJobOnSourceCommandFailure": true,
                "FailJobOnDestCommandFailure": true,
                "FailJobOnSourceActionFailure": true,
                "FailJobOnDestActionFailure": true,
                "CreateEmptyFileDest": "emptyFileName",
                "AddTempFilePrefix": "prefix_",
                "AddTempFileSuffix": "_suffix",                
                "PgpEncryption": 
                { 
                   "PgpEnabled": true,
                   "EnforcePgpOperation": "Decrypt",
                   "KeepEncryptedFiles": true,
                   "PgpTemplateName": "myTemplate"
                },
                "IncrementalTransfer": 
                {                      
                   "IncrementalTransferEnabled": true,
                   "MaxModificationAgeForFirstRunEnabled": true,
                   "MaxModificationAgeForFirstRunInHours": "5"
                },
                "SimultaneousTransfer": 
                {                      
                   "TransferMultipleFilesSimultaneously": true,
                   "MaximumConcurrentTransfers": "10"    
                },
                "DestinationFilename": 
                {                      
                   "NamePattern": "[N]",
                   "FindSubString": "aaa",
                   "ReplaceSubString": "bbb",
                   "ModifyCase": "No"
                }
             }, 
             {
                "Src": "/home/controlm/otherFile1",
                "Dest": "/home/controlm/otherFile2",
                "TransferOption": "DestToSrc"
             }]
          }
       }
    }

  • This JSON defines a file transfer from an Amazon S3 storage service to a local file system.

    Copy 
    {
       "MyS3AftFolder": 
       {
          "Type": "Folder",
          "Application": "aft",
          "TransferFromS3toLocal": 
          {
             "Type": "Job:FileTransfer",
             "ConnectionProfileSrc": "amazonConn",
             "ConnectionProfileDest": "LocalConn",
             "NumberOfRetries": "4",
             "S3BucketName": "bucket1",
             "Host": "agentHost",
             "FileTransfers": [
             {
                "Src": "folder/sub_folder/file1",
                "Dest": "folder/sub_folder/file2"
             }]
          }
       }
    }

  • This JSON defines a file transfer from one Amazon S3 storage service to another S3 storage service.

    Copy 
    {
       "MyS3AftFolder": 
       {
          "Type": "Folder",
          "Application": "aft",
          "TransferFromS3toS3": 
          {
             "Type": "Job:FileTransfer",
             "ConnectionProfileSrc": "amazonConn",
             "ConnectionProfileDest": "amazon2Conn",
             "NumberOfRetries": "6",
             "S3BucketNameSrc": "bucket1",
             "S3BucketNameDest": "bucket2",
             "Host": "agentHost",
             "FileTransfers": [
             {
                "Src": "folder/sub_folder/file1",
                "Dest": "folder/sub_folder/file2"
             }]
          }
       }
    }

  • This JSON defines a file transfer from a local file system to an AS2 server.

    File transfers that use the AS2 protocol are supported only in one direction—from a local file system to an AS2 server.

    Copy 
    {
       "MyAs2AftFolder": 
       {
          "Type": "Folder",
          "Application": "AFT",
          "MyAftJob_AS2":
          {
             "Type": "Job:FileTransfer",
             "ConnectionProfileSrc": "localAConn",
             "ConnectionProfileDest": "as2Conn",
             "NumberOfRetries": "Default",
             "Host": "agentHost",
             "FileTransfers": [
             {
                "Src": "/dev",
                "Dest": "/home/controlm/",
                "As2Subject": "Override subject",
                "As2Message": "Override conntent type"
             }]
          }
       }
    }

The following table describes the File Transfer job parameters.

Parameter

Description

Host

Defines the name of the host machine where the job runs.

An Agent must be installed on this host.

ConnectionProfileSrc

Defines the connection profile to use as the source.

ConnectionProfileDest

Defines the connection profile to use as the destination.

ConnectionProfileDualEndpoint

Defines the dual-endpoint connection profile instead of ConnectionProfileSrc and ConnectionProfileDest.

A dual-endpoint connection profile can be used for FTP, SFTP, and local file system transfers. For more information, see ConnectionProfile:FileTransfer:DualEndPoint.

NumberOfRetries

Determines the number of times to attempt to reconnect after a connection failure.

Valid Values:

  • 0–99

  • Default

Default: 5

S3BucketName

Defines the name of the S3 bucket for transfers between a local file system and an Amazon S3 or S3-compatible storage service.

S3BucketNameSrc

Defines the name of the S3 bucket at the source for transfers between two Amazon S3 or S3-compatible storage services.

S3BucketNameDest

Defines the name of the S3 bucket at the destination for transfers between two Amazon S3 or S3-compatible storage services.

AzureContainerNameSrc

Defines the name of the Azure container at the source for transfers from Azure Blob.

AzureContainerNameDest

Defines the name of the Azure container at the destination for transfers to Azure Blob.

OracleBucketNameSrc

Defines the name of the Oracle bucket at the source for transfers from Oracle Cloud Storage.

OracleBucketNameDest

Defines the name of the Oracle bucket at the destination for transfers to Oracle Cloud Storage.

GCSBucketNameSrc

Defines the name of the GCS bucket at the source for transfers from Google Cloud Storage.

GCSBucketNameDest

Defines the name of the GCS bucket at the destination for transfers to Google Cloud Storage.

SharePointSiteNameSrc

Defines the name of the SharePoint site at the source for transfers from SharePoint Online.

SharePointSiteNameDest

Defines the name of the SharePoint site at the destination for transfers to SharePoint Online.

SharePointSiteIdSrc

Defines the SharePoint site ID at the source for transfers from SharePoint Online.

SharePointSiteIdDest

Defines the SharePoint site ID at the destination for transfers to SharePoint Online.

SharePointDocLibNameSrc

Defines the name of the SharePoint document library at the source for transfers from SharePoint Online.

SharePointDocLibNameDest

Defines the name of the SharePoint document library at the destination for transfers to SharePoint Online.

SharePointDocLibIdSrc

Defines the SharePoint document library ID at the source for transfers from SharePoint Online.

SharePointDocLibIdDest

Defines the SharePoint document library ID at the destination for transfers to SharePoint Online.

FileTransfers

Defines the list of transfers that are made.

Src

Defines the source file pathname.

Dest

Defines the destination file pathname.

TransferType

(Optional) Determines one of the following FTP transfer modes:

  • Ascii: Defines a text file whose line endings are converted.

  • Binary: Defines a data file that creates an identical copy, byte for byte, of the transferred file.

Default: Binary

TransferOption

(Optional) Determines one of the following transfer options:

  • SrcToDest: Transfers files from the source to the destination.

  • DestToSrc: Transfers files from the destination to the source.

  • SrcToDestFileWatcher: Watches the file on the source and transfers it to the destination only when all criteria are met.

  • DestToSrcFileWatcher: Watches the file on the destination and transfers it to the source only when all criteria are met.

  • FileWatcher: Watch a file, and if successful, the succeeding job will run.

  • DirectoryListing: Lists the source and destination files.

  • SyncSrcToDest: Scans the source and destination, transfers only new or modified files from the source to the destination, and deletes the destination files that do not exist on the source.

  • SyncDestToSrc: Scans the source and destination, transfers only new or modified files from the destination to the source, and deletes the source files that do not exist on the destination.

Default: SrcToDest

As2Subject

(Optional) Defines the text that overrides the subject of the AS2 message.

As2Message

(Optional) Defines the text that overrides the content in the AS2 message.

DeleteFileOnDestIfFails

Determines whether to delete the destination file if the transfer fails.

If multiple files are transferred and this parameter is set to true, the last file that fails is deleted.

Valid Values:

  • true

  • false

Default: false

ContinueOnFailure

Determines whether to allow successive transfers to execute when this specific transfer fails.

Valid Values:

  • true

  • false

Default: false

FailJobOnSourceCommandFailure

Determines whether to fail the transfer when the command on the source host fails before or after a successful transfer.

Valid Values:

  • true

  • false

Default: false

FailJobOnDestCommandFailure

Determines whether to fail the transfer when the command on the destination host fails before or after a successful transfer.

Valid Values:

  • true

  • false

Default: false

FailJobOnSourceActionFailure

Determines whether to fail the transfer when a post-action on the source file fails after a successful transfer.

Valid Values:

  • true

  • false

Default: false

FailJobOnDestActionFailure

Determines whether to fail the transfer when a post-action on the destination file fails after a successful transfer.

Valid Values:

  • true

  • false

Default: false

CreateEmptyFileDest

(Optional) Defines the name of an empty file that is created in the transfer directory on the destination host after a transfer completes.

AddTempFilePrefix

(Optional) Defines a temporary file prefix to attach to the destination file until the transfer completes successfully. This prevents overwriting files.

The temporary filename that is created by applying the prefix must be a valid filename for the operating system of the destination host computer.

AddTempFileSuffix

(Optional) Defines a temporary file suffix to attach to the destination file until the transfer completes successfully. This prevents files from being overwritten.

The temporary filename that is created must be supported by the destination host computer operating system.

PgpEncryption

Defines PGP encryption commands for this transfer, using the following child parameters:

  • PgpEnabled: Determines whether to enable PGP encryption commands.

    Valid Values:

    • true

    • false

    Default: false

  • EnforcePgpOperation: Determines whether to encrypt or decrypt the transfer.

    Valid Values:

    • encrypt

    • decrypt

  • KeepEncryptedFiles: Determines whether to save the PGP-encrypted file with the following name format:

    <filename>.pgp_aft

    Valid Values:

    • true

    • false

    Default: false

  • PgpTemplateName: Determines the name of the defined PGP template, as described in Creating a PGP Template.

IncrementalTransfer

Defines which transfer files were added or modified in the source directory since the last successful job execution. You might want to use this option in a cyclic job to transfer the changes from the source directory.

Use the following child parameters:

  • IncrementalTransferEnabled: Determines whether to enable incremental file transfer.

    Valid Values:

    • true

    • false

    Default: false

  • MaxModificationAgeForFirstRunEnabled: Determines whether to transfer files that were modified in the specified interval, or to transfer all files that match the pattern of the first job execution.

    Valid Values:

    • true

    • false

    Default: false

  • MaxModificationAgeForFirstRunInHours: Determines the number of hours for the file modification interval that limits file transfer.

SimultaneousTransfer

Defines a folder or multiple files that are simultaneously transferred using a wildcard.

Use the following child parameters:

  • TransferMultipleFilesSimultaneously: Determines whether to transfer a folder or multiple files using a wildcard, as follows:

    Valid Values:

    • true: Transfers multiple files at once.

    • false: Transfers one file at a time.

    Default: false

  • MaximumConcurrentTransfers: Determines the maximum number of simultaneous file transfers.

    Specify Auto as the parameter value if you want Control-M MFT to decide the optimal number of simultaneous transfers, based on system resources.

DestinationFilename

Defines the destination filename for each transferred file when the destination path is a directory.

Use the following child parameters:

  • NamePattern: Defines the destination filename pattern, based on a combination of any of the following attributes.

    • [N]: Filename—use [N#–],[N#–#] for substrings.

    • [E]: Extension—use [E#–],[E#–#] for substrings.

    • [C]: Counter—use [C#] to pad the filename with preceding zeros.

    • [D]: Listing date—[YYYYMMDD] format.

    • [T]: Listing time—[hhmmss] format. The pattern value must include at least one parameter.

  • FindSubString: Defines a string to find within the destination filename and replace with another string that you specify in the next parameter.

  • ReplaceSubString: Defines the string that replaces the one specified by the previous parameter in the destination filename.

  • ModifyCase: Determines whether and how to handle the text case of the destination filename, as follows:

    • No: Makes no changes to the destination filename.

    • Uppercase: Changes the destination filename uppercase.

    • Lowercase: Changes the destination filename to lowercase.

    Default: No.

The following example shows how to define a File Transfer job that uses the File Watcher utility to monitor the transferred files:

Copy 
{
   "FileTransferFolder": 
   {
      "Type" : "Folder",
      "Application" : "aft",
      "TransferFromLocalToSFTPBasedOnEvent": 
      {
         "Type": "Job:FileTransfer",
         "Host": "AgentHost",
         "ConnectionProfileSrc": "LocalConn",
         "ConnectionProfileDest": "SftpConn",
         "NumberOfRetries": "3",
         "FileTransfers": [
         {
            "Src": "/home/sftp/file1",
            "Dest": "/home/sftp/file2",
            "TransferType": "Binary",
            "TransferOption": "SrcToDestFileWatcher",
            "PreCommandDest": 
            {
               "action" : "rm",
               "arg1" : "/home/sftp/file2"
            },
            "PostCommandDest": 
            {
               "action": "chmod",
               "arg1": "700",
               "arg2": "/home/sftp/file2"
            },
            "FileWatcherOptions":
            {
               "TimeLimitPolicy": "WaitUntil",
               "TimeLimitValue": "2000",
               "UnitsOfTimeLimit": "Minutes",
               "MinDetectedSizeInBytes" : "200",
               "MinFileAge": "3Min",
               "MaxFileAge": "10Min",
               "TransferAllMatchingFiles": true,
               "TransferFilesMatchingSizeAndAgeCriteria": true,
               "SkipToNextFileIfCriteriaNotMatch": true,
               "AssignFileNameToVariable": "FileNameEvent",
               "VariableType": "Named Pool",
               "VariablePoolName": "PoolPool"
            }
         }]
      }
   }
}

The following table describes the parameters in a File Transfer job that uses the File Watcher utility.

Parameter

Description

PreCommandSrc

PreCommandDest

PostCommandSrc

PostCommandDest

Defines commands that occur before and after job execution, as follows:

Each command can run only one action at a time.

  • chmod: Changes the file access permissions, as follows:

    • arg1: Mode

    • arg2: Filename

  • mkdir: Creates a new directory, where arg1 defines the directory name.

  • rename: Renames a file or directory, as follows:

    • arg1: Current filename.

    • arg2: New filename.

  • rm: Deletes a file, where arg1 defines the filename.

  • rmdir: Deletes a directory, where arg1 defines the directory name.

FileWatcherOptions

Defines additional File Watcher utility options for the transferred file.

TimeLimitPolicy / TimeLimitValue

Determines the amount of time to watch the file:

TimeLimitPolicy options: WaitUntil or MinutesToWait

TimeLimitValue:

  • If TimeLimitPolicy: WaitUntil, the TimeLimitValue is the specific time to wait.

    04:22: 4:22 AM.

  • If TimeLimitPolicy: MinutesToWait, the TimeLimitValue is the number of time units (next parameter) to wait.

UnitsOfTimeLimit

Determines one of the following time units for the TimeLimitValue when TimeLimitPolicy is set to MinutesToWait:

  • Seconds

  • Minutes

  • Hours

MinDetectedSizeInBytes

Defines the minimum number of bytes to transfer before checking if the file size is static.

MinFileAge

Defines the minimum number of years, months, days, hours, and/or minutes that must have passed since the watched file was last modified

Valid Values: 9999Y9999M9999d9999h9999Min

2y3d7h

MaxFileAge

Defines the maximum number of years, months, days, hours, and/or minutes that can pass since the watched file was last modified

Valid Values: 9999Y9999M9999d9999h9999Min

2y3d7h

TransferAllMatchingFiles

Determines whether to transfer all remaining static files that match the name pattern, regardless of the file size and age, after a single file matches all criteria.

Valid Values:

  • true

  • false

Default: false

TransferFilesMatchingSizeAndAgeCriteria

Determines whether to transfer all remaining static files that match the name, size, and age criteria after a single file matches all criteria.

Valid Values:

  • true

  • false

Default: false

SkipToNextFileIfCriteriaNotMatch

Determines whether to continue searching the remaining files if the first file that matches the name pattern does not match the size and age criteria.

Valid Values:

  • true

  • false

Default: false

AssignFileNameToVariable

Defines the variable name that contains the detected filename

VariableType

Determines the variable type, as follows:

  • Local: Enables you to use the variable in this job only.

  • Named Pool: Enables you to reference the variable in any job in the same Named Pool.

VariablePoolName

Defines the Named Pool that contains the jobs that can reference the variable.