Control-M Automation API enables you to use Deploy Descriptor to change job definition properties in JSON format before building, deploying, or running the JSON file. Supported changes include adding, deleting, or modifying properties or objects in the JSON file.
Using the Deploy Descriptor enables you to streamline your code across different environments (for example, between Development, Test, or Production). Since each Control-M environment has different properties, the Deploy Descriptor enables you to programmatically manipulate these properties.
The following video demonstrates the use of the Deploy Descriptor: https://youtu.be/kbKp2URhP4U
Deploy Descriptor scenarios
The following scenarios are examples of the typical usage of Deploy Descriptors.
Moving from a Test to Production Environment
In the following example, the Property element defines the job property that is going to be modified. The Deploy Descriptor changes the
{
"DeployDescriptor":
[
{
"Property" :"ControlmServer",
"Assign" : "ProdControlM"
},
{
"Property" :"Application",
"Replace" : [ {"T(.*)" : "P$1"} ]
},
{
"Property" :"SubApplication",
"Replace" : [ {"T(.*)" : "P$1"} ]
},
{
"ApplyOn" : {"Type":"Folder"},
"Property" : "@",
"Replace" : [ {"T(.*)" : "P$1"} ]
}
]
}Mapping Jobs to Hosts in Different Environments
Development, Test, and Production environments rarely have the same computer resources, and you might need to distribute the jobs differently between hosts in the different environments.
In this example, we want to change the name of the host on which the job will run to "host1", only if this job is part of the Finance Application.
The Property that we want to change is "Host". The Source is the field to match — in this example, "Application". When deploying the code file from the Development environment to the Test environment, all jobs belonging to the "Finance" application will be replaced to a single host named "host1".
In the Replace syntax, the value on the left applies to the Source, and the value on the right applies to the Property.
{
"DeployDescriptor":
[
{
"Comment": "Change Host of a Job based on application.",
"ApplyOn": {"Type":"Job.*"},
"Property": "Host",
"Source": "Application",
"Replace": [ { "Finance.*":"host1"} ]
}
]
}In the following example, when deploying the code file from the Test environment to the Production environment, each job is mapped to a different host, based on its job name. See Using JSON Path to Locate Attributes for the "@" syntax.
{
"DeployDescriptor":
[
{
"Comment": "Change Host of a Job based on job name.",
"Property": "Host",
"ApplyOn" : {"Type":"Job.*"},
"Source": "@",
"Replace": [
{ "FinanceJ12":"host1"},
{ "FinanceJ23":"host2"},
{ "FinanceJ45":"host3"}
]
}
]
}Moving Back from Production to Development
The following command shows how to retrieve the jobs or folders from the Production environment:
>ctm deploy jobs::get -s "server=*&folder=*" -e ProdEnv > ProdDefinitions.jsonThe ProdDefinitions.json file contains the production job definitions.
In the following example, the Deploy Descriptor modifies the
The following code example appears in the DeployDescriptor.json file:
{
"DeployDescriptor":
[
{
"Comment": "Assign Control-M server of a Job.",
"Property" :"ControlmServer",
"Assign" : "DevControlM"
},
{
"Comment": "Change Application of a Job based on its name.",
"Property" :"Application",
"Replace" : [ {"P(.*)" : "D$1"} ]
},
{
"Comment": "Change SubApplication of a Job based on its name.",
"Property" :"SubApplication",
"Replace" : [ {"P(.*)" : "D$1"} ]
}
]
} The following command enables you to deploy the ProdDefinition.json file to the Development environment using the Deploy Descriptor to change to the development values:
>ctm deploy ProdDefinitions.json DeployDescriptor.json -e developmentConverting a job to a dummy job
In the following example, the Add element adds the RunAsDummy property, set to a value of true, to a database job. This converts the job to a dummy job.
{
"DeployDescriptor":
[
{
"Add" : {
"Path": "$.Folder_2.Databases_Job_3",
"propertyName": "RunAsDummy",
"propertyValue": true
}
}
]
}Converting a local connection profile to a centralized connection profile
The following example demonstrates how to convert a local connection profile to a centralized connection profile. This is achieved by deleting two parameters, TargetAgent and TargetCTM, using the Delete element, and adding the Centralized parameter, set to true, using the Add element.
{
"DeployDescriptor":
[
{
"Delete" : "$.HADOOP_SPARK_CP.TargetAgent"
},
{
"Delete" : "$.HADOOP_SPARK_CP.TargetCTM"
},
{
"Add" : {
"Path": "$.HADOOP_SPARK_CP",
"propertyName": "Centralized",
"propertyValue": true
}
}
]
}Linking two jobs by defining events
The following Folder_1 contains two jobs, OS_Job_2 to OS_Job_3. We want to use the Deploy Descriptor to link these two jobs during deployment.
{
"Folder_1": {
"Type": "Folder",
"ControlmServer": "LocalControlM",
"ActiveRetentionPolicy": "CleanEndedOK",
"CreatedBy": "emuser",
"OS_Job_2": {
"Type": "Job:Command",
"CreatedBy": "emuser",
"RunAs": "conrolm",
"Command": "echo 'test'"
},
"OS_Job_3": {
"Type": "Job:Command",
"CreatedBy": "emuser",
"RunAs": "controlm",
"Command": "echo 'test'"
}
}
}The following Deploy Descriptor content demonstrates how to define the link between the two jobs using Events:
{
"DeployDescriptor":
[
{
"Add": {
"Path": "$.Folder_1.OS_Job_2",
"propertyName": "eventsToAdd",
"propertyValue": {
"Type": "AddEvents",
"Events": [
{
"Event": "OS_Job_2-TO-OS_Job_3"
}
]
}
}
},
{
"Add": {
"Path": "$.Folder_1.OS_Job_3",
"propertyName": "eventsToWaitFor",
"propertyValue": {
"Type": "WaitForEvents",
"Events": [
{
"Event": "OS_Job_2-TO-OS_Job_3"
}
]
}
}
},
{
"Add": {
"Path": "$.Folder_1.OS_Job_3",
"propertyName": "eventsToDelete",
"propertyValue": {
"Type": "DeleteEvents",
"Events": [
{
"Event": "OS_Job_2-TO-OS_Job_3"
}
]
}
}
}
]
}The Deploy Descriptor applies the changes to the definitions of the two jobs:
{
"Folder_1": {
"Type": "Folder",
"ControlmServer": "LocalControlM",
"ActiveRetentionPolicy": "CleanEndedOK",
"CreatedBy": "emuser1",
"OS_Job_2": {
"Type": "Job:Command",
"CreatedBy": "emuser1",
"RunAs": "conrolm",
"Command": "echo 'test'",
"eventsToAdd": {
"Type": "AddEvents",
"Events": [
{
"Event": "OS_Job_2-TO-OS_Job_3"
}
]
}
},
"OS_Job_3": {
"Type": "Job:Command",
"CreatedBy": "emuser1",
"RunAs": "controlm",
"Command": "echo 'test'",
"eventsToWaitFor": {
"Type": "WaitForEvents",
"Events": [
{
"Event": "OS_Job_2-TO-OS_Job_3"
}
]
},
"eventsToDelete": {
"Type": "DeleteEvents",
"Events": [
{
"Event": "OS_Job_2-TO-OS_Job_3"
}
]
}
}
}
} Debugging and Building the Deploy Descriptor
To debug Deploy Descriptor, use deploy transform to return the JSON jobs definition after processing Deploy Descriptor modifications:
>ctm deploy transform Jobs.json DeployDescriptor.jsonTo verify that the Deploy Descriptor is valid, you can use Build Service:
>ctm build DeployDescriptor.jsonUsing JSON Path to Locate Attributes
The code elements can be located using JSON Path syntax and the following extensions:
-
"@" - refers to the name of an element for which you can define a custom name; for example, the name of a Job, Folder, or Subfolder, as well as the name of an Event or any other nested object.
-
.parent() - refers to the name of the parent element in the JSON.
-
"." - enables you to navigate between the elements.
If the element name contains a dash "-", use bracket notation instead of dot notation. For example, to locate a job named JOB-NAME in a folder, use $['JOB-NAME'] instead of $.JOB-NAME.
The following example shows "@" that is used to refer to the job name, @.parent() is used to refer to Folder name, and $.PreCommands.FailJobOnCommandFailure allows you to describe a specific job property value:
{
"DeployDescriptor":
[
{
"ApplyOn" : {
"Type":"Job",
"@" : "JobTQ.*",
"@.parent()" : "FolderTQ.*",
"$.PreCommands.FailJobOnCommandFailure" : "True"
},
"Property":"Application",
"Assign": "App1"
}
]
}The following example demonstrates a more complex scenario for replacing names of elements for deployment of code from a Test environment to a Production environment, with deep changes in folder and subfolder structure:
{
"DeployDescriptor": [
{
"Comment": "Changes the name of all subfolders to match the production standard",
"Property": "@",
"ApplyOn": {
"Type": "SubFolder"
},
"Replace": [{
"(.*)#(.*)": "Prod$1_$2"
}]
},
{
"Comment": "Changes the name of dummy jobs to match the production standard",
"Property": "@",
"ApplyOn": {
"Type": "Job:Dummy"
},
"Replace": [{
"(.*)#(.*)": "JOBDummy_$2"
}]
},
{
"Comment": "Assigns the name of the production ControlmServer to all elements that have this property.",
"Property": "ControlmServer",
"Assign": "LocalControlM"
},
{
"Comment": "Assigns the RunAs property to the correct user for all jobs in the folder and subfolders.",
"Property": "RunAs",
"Assign": "ctmdk"
},
{
"Comment": "Changes all resources to match the production standard ",
"Property": "@",
"ApplyOn": {
"Type": "Resource.*"
},
"Replace": [{
"TST(.*)": "Prod$1"
}]
},
{
"Comment": "Assigns a new description to a specific subfolder.",
"Property": "Description",
"ApplyOn": {
"Type": "SubFolder",
"@": "Folder#3"
},
"Assign": "NewDescription"
}
]
}Advanced Example 1: Manipulating Values in an Array
In this example scenario, we want to change various values of an array-type argument of a job named job1.
{
"folder1": {
"Type": "Folder",
"ControlmServer": "LocalControlM",
"job1": {
"Type": "Job:Script",
"FileName": "scriptname.sh",
"FilePath": "%%ScriptsPath",
"RunAs": "emuser",
"Arguments": [
"--date",
"%%TodayDate",
"%%TodayDate",
"value1",
"ssssss"
]
}
}
}
The following Deploy Descriptor content demonstrates how to change various argument values in the array:
{
"DeployDescriptor": [
{
"Comment": " Change all arguments that equals 'value1' to 'new value1'",
"Property": "@.Arguments[?(@ == 'value1')]",
"Assign": "new value1"
},
{
"Comment": " Change arguments that finished with 'ssss' to 'new sssss value'",
"Property": "@.Arguments[?(@ =~ /.*ssss/i)]",
"Assign": "new sssss value"
},
{
"Comment": " Change all arguments that equals '%%TodayDate' to '%%TodayDate - new value'",
"Property": "@.Arguments[?(@ == '%%TodayDate')]",
"Assign": "%%TodayDate - new value"
},
{
"Comment": " Change arguments that contains '-da' to 'new --date value'",
"Property": "@.Arguments[?(@ =~ /.*-da.*?/)]",
"Assign": "new --date value"
}
]
}
After the Deploy Descriptor applies the changes, the folder and job definitions appear as follows:
{
"folder1": {
"Type": "Folder",
"ControlmServer": "LocalControlM",
"job1": {
"Type": "Job:Script",
"FileName": "scriptname.sh",
"FilePath": "%%ScriptsPath",
"RunAs": "emuser",
"Arguments": [
"new --date value",
"%%TodayDate - new value",
"%%TodayDate - new value",
"new value1",
"new sssss value"
]
}
}
}
Advanced Example 2: Using a Wildcard in the Manipulation of a Property Name
In this example scenario, we will use wildcards in the Deploy Descriptor to add an -stg (staging) suffix to all event names. In the following folder, two jobs are connected using events.
{
"Folder1": {
"Type": "Folder",
"ControlmServer": "ControlmServer",
"ActiveRetentionPolicy": "CleanEndedOK",
"SubApplication": "SubApplication1",
"CreatedBy": "emuser",
"Priority": "Very Low",
"Application": "ABC",
"When": {
"RuleBasedCalendars": {
"Included": [
"All"
],
"All": {
"Type": "Calendar:RuleBased",
"When": {
"DaysRelation": "OR",
"WeekDays": [
"ALL"
],
"MonthDays": [
"ALL"
]
}
}
}
},
"1-day1-Job": {
"Type": "Job:Command",
"CreatedBy": "emuser",
"RunAs": "ASDSAD",
"Command": "ECHO 'ADAS'",
"When": {
"WeekDays": [
"NONE"
],
"MonthDays": [
"NONE"
],
"FromTime": "0800",
"MonthDaysCalendar": "My_MonthDaysCalendar"
},
"eventsToAdd": {
"Type": "AddEvents",
"Events": [
{
"Event": "1-day1-Job-TO-2-day1-Job"
}
]
}
},
"2-day1-Job-stg": {
"Type": "Job:Command",
"CreatedBy": "emuser",
"RunAs": "ASDSAD",
"Command": "ECHO 'ADAS'",
"When": {
"WeekDays": [
"NONE"
],
"MonthDays": [
"NONE"
],
"FromTime": "0800",
"MonthDaysCalendar": "My_MonthDaysCalendar"
},
"eventsToWaitFor": {
"Type": "WaitForEvents",
"Events": [
{
"Event": "1-day1-Job-TO-2-day1-Job"
}
]
},
"eventsToDelete": {
"Type": "DeleteEvents",
"Events": [
{
"Event": "1-day1-Job-TO-2-day1-Job"
}
]
}
}
}
}The following Deploy Descriptor content demonstrates how to use wildcards to manipulate the name of a specific type of element. We will add the -stg (staging) suffix to all event names, regardless of the exact name of the object that contains the event.
{
"DeployDescriptor": [
{
"Comment": "Change Event naming based on the host - only for staging",
"Property": "@[*].Events[*].Event",
"Replace": [
{
"(.*)": "$1-stg"
}
]
}
]
}After the Deploy Descriptor applies the changes, the folder and job definitions appear as follows:
{
"Folder1": {
"Type": "Folder",
"ControlmServer": "ControlmServer",
"ActiveRetentionPolicy": "CleanEndedOK",
"SubApplication": "SubApplication1",
"CreatedBy": "emuser",
"Priority": "Very Low",
"Application": "ABC",
"When": {
"RuleBasedCalendars": {
"Included": [
"All"
],
"All": {
"Type": "Calendar:RuleBased",
"When": {
"DaysRelation": "OR",
"WeekDays": [
"ALL"
],
"MonthDays": [
"ALL"
]
}
}
}
},
"1-day1-Job": {
"Type": "Job:Command",
"CreatedBy": "emuser",
"RunAs": "ASDSAD",
"Command": "ECHO 'ADAS'",
"When": {
"WeekDays": [
"NONE"
],
"MonthDays": [
"NONE"
],
"FromTime": "0800",
"MonthDaysCalendar": "My_MonthDaysCalendar"
},
"eventsToAdd": {
"Type": "AddEvents",
"Events": [
{
"Event": "1-day1-Job-TO-2-day1-Job-stg"
}
]
}
},
"2-day1-Job-stg": {
"Type": "Job:Command",
"CreatedBy": "emuser",
"RunAs": "ASDSAD",
"Command": "ECHO 'ADAS'",
"When": {
"WeekDays": [
"NONE"
],
"MonthDays": [
"NONE"
],
"FromTime": "0800",
"MonthDaysCalendar": "My_MonthDaysCalendar"
},
"eventsToWaitFor": {
"Type": "WaitForEvents",
"Events": [
{
"Event": "1-day1-Job-TO-2-day1-Job-stg"
}
]
},
"eventsToDelete": {
"Type": "DeleteEvents",
"Events": [
{
"Event": "1-day1-Job-TO-2-day1-Job-stg"
}
]
}
}
}
}Using Regular Expression Patterns for Matching and Modifying Values
You can use Regular Expressions (regex) to match values of job properties. The following example modifies the names of all Applications that start with "A" by adding "Was" to the current Application name. When using a regex set, use $Number for the ordered matched elements.
For more information on how to use the ApplyOn filtering, see Filtering.
{
"DeployDescriptor":
[
{
"ApplyOn" : {
"Application" : "A.*"
},
"Property":"Application",
"Replace": [ {"(.*)" : "Was$1"} ]
}
]
}Deploy Descriptor Rules
In the Deploy Descriptor JSON file, "DeployDescriptor" is the root element. Underneath this root element, you can include one or more rules. The rules are applied in the order that they appear in the file.
A rule defines the property or object to change and the change to apply.
Each rule can contain the following main elements:
-
An optional comment, in which you can provide information about the nature of the change.
-
Filtering capabilities that narrow the scope of the change, using the ApplyOn element (optional).
-
Action to perform — Adding, Deleting, or Modifying properties or objects.
Remaining elements enable you to specify the property that you want to manipulate and the path to that property. These elements differ for each type of action.
Commenting
The optional Comment element enables you to add comments in the Deploy Descriptor JSON. For example, information about the nature of the change that the rule will perform on the job definitions JSON file.
The following example shows how to add a comment:
{
"Comment" : "Change the run as property according to the application and the name of the object",
"ApplyOn" : {
"$.Application" : "m.*",
"@" : "F.*1"
},
"Property": "RunAs",
"Assign": "prodUser"
}Filtering
The ApplyOn element filters the JSON objects to which the rule applies.
The following example shows how ApplyOn is used to filter only jobs that are of type Hadoop Spark Python, in which the Application name starts with "a" and the $.PreCommands.FailJobOnCommandFailure is set to true.
The rule appends "Was" to the name of the Application property.
{
"DeployDescriptor":
[
{
"ApplyOn" : {
"Type" : "Job:Hadoop:Spark:Python",
"Application" : "a.*",
"$.PreCommands.FailJobOnCommandFailure" : "True"
},
"Property":"Application",
"Replace": [ { "(.*)" : "Was$1" } ]
}
]
}The ApplyOn can filter on multiple properties at once. A single property filter has the following parts:
|
Part |
Description |
|---|---|
|
Name |
The Using JSON Path to Locate Attributes to locate a property in the definition file. |
|
Value |
The Using Regular Expression Patterns for Matching and Modifying Values for matching the property value. |
Adding
The Add element enables you to add the following elements to job definitions in the JSON file:
-
A new property with its assigned value
-
A new value in an existing array of property values
Such actions involve the following elements under the Add element:
|
Element |
Description |
|---|---|
|
Path |
Depending on the type of Add action, one of the following:
Mandatory. |
|
PropertyName |
The name of the property that you want to add. Relevant only for a new property; NOT for a new value in an array. |
|
PropertyValue |
The value to assign to the property. Mandatory. This can be any type of value, including a string, Boolean value, numerical value, array, or even a complex object with nested JSON elements. |
The following example adds the "Host":"sqa" property to a folder and the "WeekDays":["NONE"] property under the When property in the folder. In addition, "CreatedBy":"emuser3" is added to every job of type Job:Database:EmbeddedQuery.
{
"DeployDescriptor":
[
{
"Add" : {
"Path": "$.Folder_1.Databases_Job_2",
"propertyName": "Host",
"propertyValue":"sqa"
}
},
{
"Add" : {
"Path": "$.Folder_1.Databases_Job_2.When",
"propertyName": "WeekDays",
"propertyValue": ["NONE"]
}
},
{
"ApplyOn": {
"Type": "Job:Database:EmbeddedQuery"
},
"Add" : {
"Path": "@",
"propertyName": "CreatedBy",
"propertyValue": "emuser3"
}
}
]
}The following example adds new values to array-type properties — a value of 6 to the MonthDays property in a specific folder and job, and a value of 8 to all MonthDays properties in Embedded Query jobs.
{
"DeployDescriptor":
[
{
"Add" : {
"Path": "$.Folder_1.Databases_Job_2.When.MonthDays",
"propertyValue": "6"
}
},
{
"ApplyOn": {
"Type": "Job:Database:EmbeddedQuery"
},
"Add" : {
"Path": "@.When.MonthDays",
"propertyValue": "8"
}
}
]
}Deleting
The Delete element enables you to delete a property or a value of an array-type property in the JSON file:
The following example deletes the Host property from a specific job in a specific folder.
{
"Delete" : "$.Folder_1.Databases_Job_2.Host"
}The following example deletes the first value (index 0 item) in the array-type property MonthDays in any job or folder.
{
"Delete" : "@.When.MonthDays[0]"
}Modifying
Rules for modifying job definitions enable you to do either of the following:
-
Assign a new value to an existing property in the job.
-
Replace an existing job property.
The elements involved in modifying job definitions are discussed below.
Property
The Property element specifies the property to update. This element is mandatory.
Property element uses Using JSON Path to Locate Attributesto locate the property in the definition file.
The following example shows you how to modify the Application property.
{
"Property":"Application",
"Replace": [ { "(.*)" : "Was$1" } ]
}Assign
Assign enables you to set a new value to the specified Property. Assign is only used to update an existing property and cannot be used to add a property to an element.
The following example shows how to use Assign, to set SparkScript to "/home/etc/script.py" property in the definitions file.
{
"Property": "SparkScript",
"Assign": "/home/etc/script.py"
}Replace
Replace enables you to modify the name or value of a specified Property.
Replace contains one or more replacement rules. The first rule that matches the property's original value is applied.
{
"Property":"Application",
"Replace": [ { "FNC_JOB15_WIN" : "FNC_JOB15" },
{ "FNC_JOB16_WIN" : "FNC_JOB16" }
]
}The Replace rule that matches the original string, modifies theProperty using a regular expression mechanism.
By default, the original value is taken from theProperty.
If Source is defined, the value is taken from the Source. The Source has the following parts:
|
Part |
Description |
|---|---|
|
Name |
TheUsing Regular Expression Patterns for Matching and Modifying Values to match the original value. |
|
Value |
The Using Regular Expression Patterns for Matching and Modifying Values to format the new value. When using a regex set, you can use $Number for the ordered matched elements. |
For example, a definition file contains two jobs, where the Application property for the first job is "FNC_JOB15_WIN" and "FNC_JOB16_WIN" for the second job.
The following example shows a Replace rule where "FNC_JOB15_WIN" is replaced with "FNC_JOB15" and "FNC_JOB16_WIN" is replaced with "FNC_JOB16":
{
"Property":"Application",
"Replace": [ { "FNC_JOB15_WIN" : "FNC_JOB15" },
{ "FNC_JOB16_WIN" : "FNC_JOB16" }
]
}The following example shows a Replace rule where "FNC_JOB15_WIN" is replaced with "FNC_APP" and "FNC_JOB16_WIN" is replaced with "FNC_APP":
{
"Property":"Application",
"Replace": [ { "FNC_.*" : "FNC_APP" } ]
}The following example shows a Replace rule where "FNC_JOB15_WIN is replaced with "FNC-WIN- JOB15" and "FNC_JOB16_WIN" is replaced with "FNC-WIN-JOB16":
{
"Property":"Application",
"Replace": [ { "FNC_(.*)_(.*)" : "FNC-$2-$1" } ]
}Source
If Source is defined, the Replace original value is taken from the Source. Source value uses Using JSON Path to Locate Attributesto locate the property in the definition file.
The following example shows how to modify the "Host" property, based on the value of the job name. All job names that start with "Spark" are modified to run on "host1", jobs that start with "MyA" are modified to run on "host2" and all other jobs are modified to run on "host3".
{
"Property": "Host",
"Source": "@",
"Replace": [
{ "Spark.*": "host1"},
{ "MyA.*" : "host2"},
{ ".*" : "host3"}
]
}
YouTube
Twitter
Facebook
LinkedIn