Data Integration Jobs

Determines the authorization credentials that are used to connect Control-M to Airbyte, as described in Airbyte Connection Profile Parameters.

Defines the Airbyte Connection ID, which identifies which data pipeline to execute.

In Airbyte, a data pipeline copies data from a source to a destination.

Determines one of the following Airbyte actions to perform:

• Sync: Reads from a source and writes to a destination, depending upon the predefined sync mode in the Airbyte platform.

• Reset: Deletes all the records from your destination and then performs a sync.

Determines whether to append the API REST response to the output.

(Optional)Determines the number of seconds to wait before checking the job status.

Default: 60

Determines the authorization credentials that are used to connect Control-M to Alteryx Trifacta, as described in Alteryx Trifacta Connection Profile Parameters.

Determines which Alteryx Trifacta flow the job runs.

Determines whether to allow rerun of the job in Alteryx Trifacta with a new idempotency token (for example, when the job run times out).

Defines a unique ID (idempotency token), which guarantees that the job runs only once. After it runs successfully, this ID cannot be reused.

To re-run the job with a new token, replace the default value with the RUN_ID retrieved from the job output.

Default: Control-M-Idem_%%ORDERID

Determines whether to track job run status as the job execution and status changes, for example—from in-progress to failed or to completed.

Defines the RUN_ID number for the job run to be tracked.

The RUN_ID is unique to each job run and it can be found in the job output.

Determines the authorization credentials that are used to connect Control-M to AWS AppFlow, as described in AWS AppFlow Connection Profile Parameters.

Determines which of the following AWS AppFlow actions are performed:

• Trigger Flow - Key & Secret Auth: Connects to AWS AppFlow with the Key & Secret authentication method, and launches an AppFlow flow. To use this option, you must use a connection profile that is defined with Key & Secret authentication.

• Trigger Flow - IAM Role Auth: Connects to AWS AppFlow with the IAM Role authentication method, and launches an AppFlow flow. To use this option, you must use a connection profile that is defined with AWS IAM Role authentication.

• Trigger Flow - IAM Assume Role Auth: Connects to AWS AppFlow with the IAM Assume Role authentication method, and launches an AppFlow flow. To use this option, you must use a connection profile that is defined with AWS IAM Assume Role authentication.

Determines the AppFlow flow to trigger.

Determines whether to implement retry behavior.

To use this option, you must define a Client Token.

Defines a unique identifier for this instance of the job. The following example shows how the %%ORDERID variable is used to generate a unique string each time the job runs.

(Optional) Determines the number of seconds to wait before checking the job status.

Default: 60

Determines the authorization credentials that are used to connect Control-M to AWS Glue, as described in AWS Glue Connection Profile Parameters.

Determines one of the following actions to perform an AWS Glue activity:

• Run Job

• Run Workflow

• Start Crawler

• Stop Crawler

Defines the AWS Glue job name that you want to run.

A job name is automatically saved when you create an AWS Glue pipeline.

Defines the AWS Glue workflow name that you want to run.

Defines the AWS Glue Crawler name that you want to start or stop.

Determines whether to add arguments to the AWS Glue job.

Defines the AWS Glue job execution-time parameters, in JSON format, as shown in the following example:

{"--myArg1": "myVal1", "--myArg2": "myVal2"}

(Optional) Determines the number of seconds to wait before checking the job status.

Default: 15

Determines the number of times to check the job status before the job ends Not OK.

Default: 2

Determines whether the Cloudwatch logs are printed to the output.

This is only available if you run an AWS Job.

Determines the authorization credentials that are used to connect Control-M to AWS Glue DataBrew, as described in AWS Glue DataBrew Connection Profile Parameters.

Defines the name of the job.

Determines whether to display job logs in the output.

Determines the number of seconds to wait before checking the job status.

Default: 10

Determines the authorization credentials that are used to connect Control-M to Azure Data Factory, as described in Azure Data Factory Connection Profile Parameters.

• Variable Name: %%AZURE-ACCOUNT

Determines the Azure Resource Group that is associated with a specific data factory. A resource group is a container that holds related resources for an Azure solution.

The resource group can include all the resources for the solution, or only those resources that you want to manage as a group.

Determines the name of the Azure Data Factory that contains the pipeline you want to execute.

Determines which data pipeline executes when you run the Control-M job.

Defines specific parameters, in JSON format, that are passed when the data pipeline executes, in the following format:

{"var1":"value1", "var2":"value2"}

For no parameters, type {}.

Determines the number of seconds to wait before checking the job status.

Default: 45

Determines the number of times to check the job status before the job ends Not OK.

Default: 3

Determines the authorization credentials that are used to connect Control-M to Boomi AtomSphere, as described in Boomi AtomSphere Connection Profile Parameters.

Defines the name of a Boomi Atom associated with the Boomi process.

Defines the name of a Boomi process associated with the Boomi Atom.

Determines the number of seconds to wait before checking the job status.

Default: 20 seconds

Determines the authorization credentials that are used to connect Control-M to Fivetran, as described in Fivetran Connection Profile Parameters.

Determines one of the following Fivetran jobs:

• Sync: Retrieves data changes from the source determined by the Connection ID and updates output destinations.

• Re-Sync: Clears previously retrieved data and retrieves a full data set from the source determined by the Connection ID.

• Transformation: Runs a transformation task.

Determines the unique ID of the connection settings defined in Fivetran.

This is only relevant when the Job Type selected is either Sync or Re-Sync.

Determines the name of a specific data connection in Fivetran that represents the data pipeline.

This appears only when the Job Type selected is either Sync or Re-Sync.

Determines the parameter in the Fivetran API that overrides certain default behaviors.

This is only relevant when the Job Type selected is Sync.

Valid Values:

• Yes: If the connection is in the middle of synchronization, it stops and reruns the synchronization.

• No: The connection will only synchronize if it is not in the middle of synchronization

Determines a unique identifier assigned to a Fivetran transformation to execute a specific transformation.

This is only relevant when the Job Type selected is Transformation.

Determines the number of seconds to wait before checking the job status.

Default: 15

Determines the authorization credentials that are used to connect Control-M to GCP Data Fusion, as described in GCP Data Fusion Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Determines the region where the job runs.

Defines the identifier of the GCP project where the job runs.

A project is a set of configuration settings that define the resources the jobs utilize and how they interact with GCP.

Defines the name of the predefined virtual machine (instance) that runs the job.

Defines the name of the namespace, which contains the job, job data, and metadata.

Valid Characters: A–Z, a–z, 0–9, and _

Default: default

Defines the name of a predefined ETL service or data integration pipeline in GCP Data Fusion.

Defines the JSON-based body parameters that are passed to the function, in the following format:

"argument" : {\"var1\":\"value1\",\"var2\":\"value2\"}

Determines whether to append the logs to the output.

Determines the number of seconds to wait before checking the job status.

Default: 10

Determines the authorization credentials that are used to connect Control-M to GCP Dataplex, as described in GCP Dataplex Connection Profile Parameters.

Defines the identifier of the GCP project where the job runs.

A project is a set of configuration settings that define the resources the jobs utilize and how they interact with GCP.

Determines the region where the job runs.

Determines one of the following GCP Dataplex actions to perform:

• Data Quality Task: Executes a predefined data quality task in GCP BigQuery or Google Cloud Storage locations and defines data controls in BigQuery environments.

• Custom Spark Task: Executes a predefined, scheduled Apache Spark task to analyze and process your data.

• Data Profiling Scan: Executes a predefined data scan to identify shared statistical characteristics between BigQuery tables.

• Data Quality Scan: Executes a predefined data quality scan that validates your data and logs alerts when the data fails validation.

(Data Quality Task and Custom Spark Task actions only) Defines the name of the Google Cloud Storage data lakes where the job runs its task.

(Data Quality Task and Custom Spark Task actions only) Defines the name of the predefined task that the job runs.

(Data Profiling Scan and Data Quality actions only) Defines the name of the predefined scan that the job runs.

Determines the number of seconds to wait before checking the job status.

Default: 15

Determines the authorization credentials that are used to connect Control-M to GCP Dataprep, as described in GCP Dataprep Connection Profile Parameters.

Defines the name of the flow, which is the workspace where you format and prepare your data.

Defines parameters that override the flow or its datasets when the job runs, as shown in the following example:

{
   "schemaDriftOptions":
   {
      "schemaValidation": "true",
      "stopJobOnErrorsFound": "true" 
   }
}

For more information on parameter types, see the properties of runFlow service in the GCP Dataprep API documentation.

Determines whether to run the job with an idempotency token.

Defines a unique ID (idempotency token), which guarantees that the job runs only once.

Default: Control-M-Idem-%%ORDERID

Determines the number of seconds to wait before checking the job status.

Default: 10

Determines the authorization credentials that are used to connect Control-M to IBM DataStage, as described in IBM DataStage Connection Profile Parameters.

Determines the IBM DataStage project name.

Determines the IBM DataStage job name.

(Optional) Defines the instance ID of the IBM DataStage job.

Determines where Control-M retrieves job parameters, with options as follows:

• None: No job parameters are retrieved.

• Parameter List: Uses the parameters you define in Parameters.

• Parameter File: Retrieves parameters from the file that you define in Parameter File.

• Parameter List and File: Uses parameters from the file you define in Parameter File and the parameters you define in Parameters.

Defines a list of parameters for the job in the following format:

attribute=value string

Defines the pathname of the parameters file that is located on the IBM DataStage server.

Defines the maximum number of rows that the job can contain.

Control-M stops the stages after the maximum is reached.

Defines the maximum number of warnings about the job.

Control-M aborts the job after the maximum is reached.

Determines whether to create a log file that appears in the Output tab of the job in the Monitoring domain.

Determines whether to resume a job from where it failed.

Determines the authorization credentials that are used to connect Control-M to Informatica, as described in Informatica Connection Profile Parameters.

Variable Name: %%INF-ACCOUNT

Defines the Repository folder that contains the workflow that you want to run.

Variable Name: %%INF-REP_FOLDER

Defines the workflow that you want to run in Control-M for Informatica.

Variable Name: %%INF-WORKFLOW

Defines the specific instance of the workflow that you want to run.

Variable Name: %%INF-INSTANCE_NAME

Defines an OS profile when you run or rerun an Informatica job.

Defines the workflow task hierarchy.

Determines the number of levels within the workflow task hierarchy for the selection of workflow tasks.

Default: 10

Variable Name: %%INF- DEPTH

Determines whether to execute the whole workflow, start from a specific task, or execute a single task as follows:

• Run the Whole Workflow: Executes the whole workflow.

• Start from Task: Starts the workflow from the task that you specify.

  Variable Name: %%INF- START_FROM_TASK

• Run Single Task: Executes the task that you specify.

  Variable Name: %%INF-RUN_SINGLE_TASK

Determines an array of parameters that is passed to the workflow.

Each parameter is comprised of the following:

• Scope: Defines the scope of the parameter in an array definition.

• Name: Defines the parameter name in an array definition.

• Value: Defines the parameter value in an array definition.

Determines whether to include the workflow event log in the job output A tab in the job properties pane of the Monitoring domain where the job output appears that indicates whether a job ended OK, and is used, for example, with jobs that check file location..

Determines whether to include a detailed error log for a workflow that failed.

Determines whether to retrieve session statistics and log messages.

Determines which operation to run when the workflow is suspended, as follows:

• Recover: Restarts a suspended workflow from the point of failure.

• Force Restart: Restarts a suspended workflow from the beginning.

• Force Restart from a Specific Task: Restarts the suspended workflow from the task that you define.

Variable Name: %%INF- RESTART_FROM_TASK

Determines the authorization credentials that are used to connect Control-M to Informatica Cloud Services, as described in Informatica CS Connection Profile Parameters.

Variable Name: %%INF-ACCOUNT

Determines one of the following task types to execute on Informatica Cloud:

• Mapping Task: A set of instructions that defines how data is transformed and moved from its source to its target system.

• Masking Task: A data security technique that enables you to protect sensitive data while it allows it to be used for non-production purposes.

• PowerCenter Task: A data integration tool that enables you to extract, transform, and load data from different sources into a unified target system.

• Replication Task: A data replication solution that enables you to replicate and synchronize data across different systems and databases in real time.

• Synchronization Task: A data integration solution that enables you to synchronize data between different systems and databases, which ensures that data is consistent and up-to-date across all systems.

• Linear Taskflow: A workflow automation feature that enables you to create and automate a sequence of tasks that are executed in a specific order, which helps streamline data integration and processing tasks.

• Taskflow: A workflow automation feature that enables you to create complex workflows that orchestrate and automate data integration and processing tasks across multiple systems and platforms.

Determines whether to identify the task using a Federated Task ID, which is a unique identifier that is used to track and manage tasks across distributed environments in a federated environment. This ID is generated by the Informatica domain and is important for monitoring and troubleshooting tasks.

This attribute is not required when you run a taskflow.

Defines the name of the task that runs on Informatica Cloud.

This attribute is not required when you run a taskflow or use a Federated Task ID.

Defines the folder path of the task that runs on Informatica Cloud.

This attribute is required if you use a Federated Task ID.

Defines the service URL of the taskflow that runs on Informatica Cloud.

You can find this URL if you click in the top, right corner of the TaskFlow main page of Informatica Data Integrator and click Properties Detail....

Determines whether to re-execute a suspended taskflow.

Defines input fields for a taskflow, in the following format:

input1=value1&input2=value2&input3=value3

(Optional) Defines a publicly available URL where the job status is posted.

Defines the Run ID to rerun a suspended taskflow.

The Run ID is unique to each job run and is available in the job output, next to the variable name RUN-UCM-RUNID.

Determines whether to rerun a job (pipeline) from the point of failure.

Default: Off

Determines the authorization credentials that are used to connect Control-M to Matillion, as described in Matillion Connection Profile Parameters.

Determines one of the following types of Matillion service environments where the job runs:

• Matillion Data Productivity Cloud: Launches a pipeline in the Matillion cloud service. To use this option, you must use a connection profile that is defined with a Matilliion Data Productivity Cloud environment.

• Matillion ETL: Launches a pipeline on the Matillion service on a virtual machine in your environment. To use this option, you must use a connection profile that is defined with a Matilliion ETL environment.

Determines the collection of projects that contains the Matillion Orchestration job determined by Job Name.

Determines the Matillion workspace that contains the Matillion Orchestration job determined by Job Name.

Determines the Matillion Orchestration job that runs the ETL workflow.

Determines the number of seconds to wait before checking the job status.

Default: 15

Determines the authorization credentials that are used to connect Control-M to OCI Data Integration Services, as described in OCI Data Integration Connection Profile Parameters.

Variable Name: %%INF-ACCOUNT

Determines one of the following OCI Data Integration actions:

• Start Workspace: Activates the OCI Data Integration workspace with the Workspace OCID.

• Stop Workspace: Deactivates the OCI Data Integration workspace with the Workspace OCID.

• Run Task: Executes the Oracle dataflow task on the active OCI Data Integration workspace and defines the task parameters.

Determines the ID of the OCI Data Integration workspace, which is a logical container for managing pipelines, data flows, and projects.

Defines the Application Key which identifies the project application where the task runs.

Determines the Task Key that is used to run the job.

Defines the unique name for the specific task run.

(Optional) Defines the input parameters for the specific task run, in JSON format.

Determines the number of seconds to wait before checking the job status.

Default: 15

Determines the authorization credentials that are used to connect Control-M to Talend Data Management, as described in Talend Data Management Connection Profile Parameters.

Determines one the following operations to perform:

• Execute Task: Runs a Talend job with a Task Name.

• Execute Plan: Runs a Talend workflow.

• Execute Task with Environment ID: Runs a Talend job with a Task Name and filter capabilities.

• Execute Task with Task ID: Runs a Talend job with a Task ID.

Defines the name of the Talend task that is executed, as defined in the Tasks and Plans page in the Talend Management Console.

Defines the ID of the Talend task that is executed, as defined in the Tasks and Plans page in the Talend Management Console.

Determines a unique identifier for a specific environment within a Talend Cloud instance.

You can filter the jobs that you want to run by Environment ID, Workspace ID, or both.

Determines a unique identifier automatically assigned to each workspace within the Talend platform.

You can filter the jobs that you want to run by Environment ID, Workspace ID, or both.

Defines specific parameters, in JSON format, to pass when the Talend job runs.

All parameter names must contain the parameter_ prefix, as shown in the following example:

{"parameter_param1":"value1", "parameter_param2":"value2"}

For no parameters, type {}.

Determines the amount of log details recorded in the Talend task logs, as follows:

• Information: Records all task execution details in the logs.

• Warning: Only records warnings.

• Error: Only records errors.

• Off: Records no information in the logs.

Determines whether to append Talend log messages to the job output.

Default: Unchecked

Determines the number of seconds to wait before checking the status of the triggered task.

Default: 10

Defines the name of the Talend plan that is executed, which is defined in the Tasks and Plans page in the Talend Management Console.

 Defines the specific parameters, in JSON format, that are passed to Talend when the job runs, as shown in the following example:

{
   "executable": "b91cf8b2-5dd1-4b18-915b-4c447cee5267",
   "rerunOnlyFailedTasks": true,
   "stepId": "09043c9f-02d0-41f6-b3cb-0ea53ffde377"
}

where executable is the Plan ID.

Determines whether to show all logs from Talend Plan in the output.

Determines the authorization credentials that are used to connect Control-M to Talend OAuth, as described in Talend OAuth Connection Profile Parameters.

Determines one of the following actions to perform:

• Execute Task by ID: Runs a Talend task (job) by Task Executable (ID).

• Execute Task by Name: Runs a Talend task (job) by Name.

• Execute Plan: Executes a Talend plan (workflow).

Determines the Environment ID where the task is executed.

Determines the name of the predefined Talend task that is executed, as defined in the Tasks and Plans page in the Talend Management Console.

Determines the ID of the predefined Talend task that is executed, as defined in the Tasks and Plans page in the Talend Management Console.

Defines specific parameters, in JSON format, to pass when the Talend job runs.

{"custom_message":"Talend test", "number_of_message": 4, "sleep_time": 100 }

For no parameters, type {}.

Determines the amount of log details that are recorded in the Talend task logs, as follows:

• Information: Records all task execution details in the logs.

• Warning: Records warnings.

• Error: Records errors.

• Off: Records no information in the logs.

(Optional) Determines the number of minutes to wait before the task is executed.

Determines whether to append the Talend task logs to the job output when the plan fails during the execution.

Defines the Plan executable that you want to execute, as defined in the Tasks and Plans page in the Talend Management Console.

Determines whether to append the failed Talend Plan logs in the job output when the plan fails during the execution.

(Optional) Determines whether to rerun only the failed tasks.

(Optional) Defines the ID for the re-run plan.

(Optional) Defines the step that the task re-runs.

Determines the number of seconds to wait before checking the job status.

Default :10