Data Integration Jobs

Determines the authorization credentials that are used to connect Control-M to Alteryx Trifacta.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Determines which Trifacta flow the job executes.

Determines whether to allow re-execution of the job in Trifacta with a new idempotency token—for example, when the job execution times out).

Defines a unique ID (idempotency token), which guarantees that the job executes only once. After successful execution, this ID cannot be used again.

To re-execute the job with a new token, replace the default value with the RUN_ID, which can be retrieved from the job output.

Default: Control-M-Idem_%%ORDERID

Determines whether to track job execution status as the job execution progresses and the status changes (for example, from in-progress to failed or to completed).

Defines the RUN_ID number for the job execution to be tracked.

The RUN_ID is unique to each job execution and can be found in the job output.

Determines the authorization credentials that are used to connect Control-M to AWS Glue.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Defines the AWS Glue job name that you want to execute.

A job name is automatically saved when you create an AWS Glue pipeline.

Determines whether to add arguments to the AWS Glue job.

Defines the AWS Glue job execution-time parameters, in JSON format, as shwon in the following example:

{"--myArg1": "myVal1", "--myArg2": "myVal2"}

(Optional) Determines the number of seconds to wait before checking the status of the job between intervals.

Default: 15

Determines the authorization credentials that are used to connect Control-M to AWS Glue DataBrew.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Defines the AWS Glue DataBrew job name.

Determines whether the DataBrew job logs are included in the Control-M output.

Determines the number of seconds to wait before checking the status of the DataBrew job.

Default: 10

Determines the authorization credentials that are used to connect Control-M to Azure Data Factory.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

• Variable Name: %%AZURE-ACCOUNT

Determines the Azure Resource Group that is associated with a specific data factory. A resource group is a container that holds related resources for an Azure solution.

The resource group can include all the resources for the solution, or only those resources that you want to manage as a group.

Determines the name of the Azure Data Factory that contains the pipeline you want to execute.

Determines which data pipeline executes when you execute the Control-M job.

Defines specific parameters, in JSON format, that are passed when the data pipeline executes.

 {"var1":"value1", "var2":"value2"}

Determines the number of seconds to wait before checking the status of the Data Factory job.

Default: 45

Determines the authorization credentials that are used to connect Control-M to Boomi AtomSphere.

Defines the name of a Boomi Atom associated with the Boomi process.

Defines the name of a Boomi process associated with the Boomi Atom.

Determines the number of seconds to wait before checking the status of the job between intervals.

Default: 20 seconds

Determines the authorization credentials that are used to connect Control-M to GCP Data Fusion.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Determines the region where the GCP Data Fusion job executes.

Defines the name of the predefined Google Cloud project that holds your configured APIs, authentication information, billing details, and job resources.

Defines the name of the predefined virtual machine (instance) that executes your job.

Defines the name of the namespace, which contains the job, job data, and metadata.

Valid Characters: A–Z, a–z, 0–9, and _.

Default: default

Defines the name of a predefined ETL service or data integration pipeline in GCP Data Fusion.

Defines the JSON-based body parameters that are passed to the function, in the following format:

"argument" : {\"var1\":\"value1\",\"var2\":\"value2\"}

Determines whether to append the GCP Data Fusion logs to the output A tab in the job properties pane of the Monitoring domain where the job output appears that indicates whether a job ended OK, and is used, for example, with jobs that check file location..

Determines the number of seconds to wait before checking the job status.

Default: 10

Determines the authorization credentials that are used to connect Control-M to GCP Dataplex, as described in GCP Dataplex Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Defines the ID of the predefined Google Cloud project that holds your configured APIs, authentication information, billing details, and job resources.

Determines the region where the GCP Dataplex job executes.

Determines one of the following GCP Dataplex actions to perform:

• Data Quality Task: Executes a predefined data quality task in GCP BigQuery or Google Cloud Storage locations and defines data controls in BigQuery environments.

• Custom Spark Task: Executes a predefined, scheduled Apache Spark task to analyze and process your data.

• Data Profiling Scan: Executes a predefined data scan to identify shared statistical characteristics between BigQuery tables.

• Data Quality Scan: Executes a predefined data quality scan that validates your data and logs alerts when the data fails validation.

(Data Quality Task and Custom Spark Task actions only) Defines the name of the Google Cloud Storage data lakes where the job executes its task.

(Data Quality Task and Custom Spark Task actions only) Defines the name of the predefined task that the job executes.

(Data Profiling Scan and Data Quality actions only) Defines the name of the predefined scan that the job executes.

Determines the number of seconds to wait before checking the status of the job.

Default: 15

Determines the authorization credentials that are used to connect Control-M to GCP Dataprep.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Defines the name of the flow, which is the workspace where you format and prepare your data.

Defines parameters that override the flow or its datasets when the job executes, as shown in the following example:

{
   "schemaDriftOptions": {
      "schemaValidation": "true",
      "stopJobOnErrorsFound": "true" 
   }
}

For more information on parameter types, see the properties of runFlow service in the GCP Dataprep API documentation.

Determines whether to execute the job with an idempotency token.

Defines a unique ID (idempotency token), which guarantees that the job executes only once.

Default: Control-M-Idem-%%ORDERID

Determines the number of seconds to wait before checking the status of the job.

Default: 10

Determines the authorization credentials that are used to connect Control-M to IBM InfoSphere DataStage.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Variable Name: %%DataStage-ACCOUNT

For more information about creating a local connection profile for this job, see Creating a connection profile.

Defines the Control-M for IBM InfoSphere DataStage project name.

Variable Name: %%DataStage-PROJECT

Defines the Control-M for IBM InfoSphere DataStage job name.

Variable Name: %%DataStage-JOB_NAME

Defines the Control-M for IBM InfoSphere DataStage job invocation ID.

Variable Name: %%DataStage-JOB_INVOCATION_ID

Determines from where Control-M retrieves the parameters, with options as follows:

• None: Does not retrieve parameters.

• Server: Retrieves the parameters from the DataStage Job, and displays them in the Parameters list.

• File: Retrieves the parameters from the file that you define in Parameters File.

• Server and File: Retrieves the parameters from the file that you define in Parameters File and from the DataStage Job, and displays those from the job in the Parameters list.

Displays the parameters and their values from the DataStage Job when Server or Server and File is selected from Parameters Type.

Defines the IBM InfoSphere DataStage parameter file.

Variable Name: %%DataStage-PARAMS_FILE

Opens more options.

Defines limits on the job.

Defines the maximum number of rows that the job can contain. Control-M stops the stages after the maximum is reached.

Variable Name: %%DataStage-MAX_ROWS

Defines the maximum number of warnings about the job. Control-M aborts the job after the maximum is reached.

Variable Name: %%DataStage-MAX_WARNINGS

Determines the type of information that goes in the output as follows:

• Info: Writes Info log entries to Control-M for IBM InfoSphere DataStage job Sysout.

  Variable Name: %%DataStage-OUTPUT_INFO

• Warnings: Writes Warning log entries to Control-M for IBM InfoSphere DataStage job Sysout.

  Variable Name: %%DataStage-OUTPUT_WARNING

• Fatal: Writes Fatal log entries to Control-M for IBM InfoSphere DataStage job Sysout.

  Variable Name: %%DataStage-OUTPUT_FATAL

• Reject: Writes Reject log entries to Control-M for IBM InfoSphere DataStage job Sysout.

  Variable Name: %%DataStage-OUTPUT_REJECT

• Started: Writes Started log entries to Control-M for IBM InfoSphere DataStage job Sysout.

  Variable Name: %%DataStage-OUTPUT_STARTED

• Reset: Writes Reset log entries to Control-M for IBM InfoSphere DataStage job Sysout.

  Variable Name: %%DataStage-OUTPUT_RESET

• Batch: Writes Batch log entries to Control-M for IBM InfoSphere DataStage job Sysout.

  Variable Name: %%DataStage-OUTPUT_BATCH

Defines execution options for the job.

Executes the Control-M for IBM InfoSphere DataStage job in restart mode.

Variable Name: %%DataStage-RESTART_SEQUENCE

Determines the authorization credentials that are used to connect Control-M to Informatica.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Variable Name: %%INF-ACCOUNT

Defines the repository folder that contains the workflow that you want to execute.

Variable Name: %%INF-REP_FOLDER

Defines the workflow that you want to execute.

Variable Name: %%INF-WORKFLOW

Defines the specific instance of the workflow that you want to execute.

Variable Name: %%INF-INSTANCE_NAME

Enables you to specify an OS profile when executing or re-executing an Informatica job.

Defines the workflow task hierarchy.

Determines the number of levels within the workflow task hierarchy that are used to select workflow tasks.

Default: 10

Variable Name: %%INF- DEPTH

Determines whether to execute the whole workflow, start from a specific task, or execute a single task as follows:

• Run the Whole Workflow: Executes the whole workflow.

• Start from Task: Starts the workflow from the task that you specify.

  Variable Name: %%INF- START_FROM_TASK

• Run Single Task: Executes the task that you specify.

  Variable Name: %%INF-RUN_SINGLE_TASK

Determines an array of parameters that is passed to the workflow.

Each parameter is comprised of the following:

• Scope: Defines the scope of the parameter in an array definition.

• Name: Defines the parameter name in an array definition.

• Value: Defines the parameter value in an array definition.

Determines whether to include the workflow event log in the job output A tab in the job properties pane of the Monitoring domain where the job output appears that indicates whether a job ended OK, and is used, for example, with jobs that check file location..

Determines whether to include a detailed error log for a workflow that failed.

Determines whether to retrieve session statistics and log messages.

Determines which operation is executed when the workflow is suspended, as follows:

• Recover: Restarts a suspended workflow from the point of failure.

• Force Restart: Restarts a suspended workflow from the beginning.

• Force Restart from a Specific Task: Restarts the suspended workflow from the task that you define.

Variable Name: %%INF- RESTART_FROM_TASK

Determines the authorization credentials that are used to connect Control-M to Informatica Cloud Services.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Variable Name: %%INF-ACCOUNT

Determines one of the following task types to execute on Informatica Cloud:

• Mapping Task: A set of instructions that defines how data is transformed and moved from its source to its target system.

• Masking Task: A data security technique that enables you to protect sensitive data while allowing it to be used for non-production purposes.

• PowerCenter Task: A data integration tool that enables you to extract, transform, and load data from different sources into a unified target system.

• Replication Task: A data replication solution that enables you to replicate and synchronize data across different systems and databases in real-time.

• Synchronization Task: A data integration solution that enables you to synchronize data between different systems and databases, ensuring that data is consistent and up-to-date across all systems.

• Linear Taskflow: A workflow automation feature that enables you to create and automate a sequence of tasks that are executed in a specific order, which helps streamline data integration and processing tasks.

• Taskflow: A workflow automation feature that enables you to create complex workflows that orchestrate and automate data integration and processing tasks across multiple systems and platforms.

Determines whether to identify the task using a Federated Task ID, which is a unique identifier that is used track and manage tasks across distributed environments in a federated environment. This ID is generated by the Informatica domain and is important for monitoring and troubleshooting tasks.

This attribute is not required when you execute a taskflow.

Defines the name of the task that executes on Informatica Cloud.

This attribute is not required when you execute a taskflow or use a Federated Task ID.

Defines the folder path of the task that executes on Informatica Cloud.

This attribute is required if you are using a Federated Task ID.

Defines the service URL of the taskflow that executes on Informatica Cloud.

You can find this URL by clicking in the top, right corner of the TaskFlow main page of Informatica Data Integrator and clicking Properties Detail....

Determines whether to re-execute a suspended taskflow.

Defines input fields for a taskflow, in the following format:

input1=value1&input2=value2&input3=value3

(Optional) Defines a publicly available URL where the job status is posted.

Defines the Run ID to re-execute a suspended taskflow.

The Run ID is unique to each job execution and is available in the job output, next to the variable name RUN-UCM-RUNID.

Determines the authorization credentials that are used to connect Control-M to Talend Data Management.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Determines one of the following options for execution in Talend:

• Execute Task

• Execute Plan

Defines the name of the Talend task or plan to execute, as defined in the Tasks and Plans page in the Talend Management Console.

(Execute Task) Defines specific parameters, in JSON format, to pass when the Talend job executes.

All parameter names must contain the parameter_ prefix, as shown in the following example:

{"parameter_param1":"value1", "parameter_param2":"value2"}

For no parameters, type {}.

(Execute Task) Determines one of the following levels of detail in log messages for the triggered task in the Talend Management Console:

• Information: All logs available.

• Warning: Only warning logs.

• Error: Only Error logs.

• Off: No logs.

(Execute Task) Determines whether to show Talend log messages in the job output.

Default: Unchecked