Application Workflow Jobs

The following topics describe job attributes that work with application workflow platforms and services:

Airflow JobLink copied to clipboard

Airflow enables you to monitor and manage DAG workflows in Control-M. You can monitor DAG executions in the Airflow tab in the Monitoring domain. You can also view the specific details of each task, open the DAG in the Airflow web server user interface, and view XCom variables from the Airflow tab.

To create an Airflow job, see Creating a Job. For more information about this plug-in, see Control-M for Airflow.

The following table describes the Airflow job type attributes.

Attribute

Description

Connection Profile

Determines the authorization credentials that are used to connect Control-M to Airflow, as described in Airflow Connection Profile Parameters .

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Blank spaces.

DAG ID

Defines the unique identifier of a DAG.

  • Airflow returns 100 records by default. To increase this limit, you must change the maximum_page_limit parameter in the airflow.cfg file.

  • The Airflow plug-in requests 300 records to be returned by default.

    To increase this number, you must add or modify the NumberOfRecordsPerRequest parameter in the <Agent_Home>/cm/AFL/data/cm_container_conf.xml file, <NumberOfRecordsPerRequest>400</NumberOfRecordsPerRequest>.

  • You can filter the returned list of DAG IDs by typing one of the following:

    • A partial string such as abcd, which returns all DAG IDs that contain this string.

    • A partial string followed by an asterisk, such as abcd*, which returns all DAG IDs that start with this string.

Configuration JSON

(Optional) Defines the JSON object, which describes additional configuration parameters.

If you add a JSON with configuration parameters and the job hangs during execution, see KA 000414577.

Output Details

Determines whether to include Airflow DAG task logs in the Control-M job output, as follows:

  • No task logs.

  • Include failing task logs.

  • Include all task logs.

You can also view all task logs from the Airflow tab in an Airflow job in the Monitoring domain. In addition, you can view the log of each Airflow task execution represented by the Try Number field.

If a task executed 3 times during the DAG run, the Try Number shows three options.

Apache Airflow JobLink copied to clipboard

Apache Airflow enables you to create, schedule, and monitor complex data processing and analytics pipelines. It provides an environment to define, manage, and execute workflows as Directed Acyclic Graphs (DAGs) to control task dependencies and execution order.

To create an Apache Airflow job, see Creating a Job. For more information about this plug-in, see Control-M for Apache Airflow.

The following table describes the Apache Airflow job attributes.

Attribute

Description

Connection Profile

Determines the authorization credentials that are used to connect Control-M to Apache Airflow, as described in Apache Airflow Connection Profile Parameters.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Blank spaces

Action

Determines whether to run a new DAG or rerun a DAG.

Valid Values:

  • Run DAG

  • Rerun DAG
DAG Name

Defines the logical name of the Directed Acyclic Graph (DAG).
DAG Run ID

(Optional) Defines the specific DAG run (execution) ID in Airflow to track and manage individual workflow executions.If you do not provide a DAG Run ID, the system generates a random Run ID.

Parameters

Defines the parameters for the Apache Airflow job, in JSON format, which enables you to control how the job executes.

Use backslashes to escape quotes.

CopyCopied to clipboard 
{\"parameter1\":\"value1\"}

If you are not adding parameters, type {}.

Only Failed Tasks

Determines whether to rerun a DAG job only with failed tasks or with all tasks.

Valid Values:

  • true

  • false
Status Polling Frequency

Determines the number of seconds to wait before checking the job status.

Default: 60
Failure Tolerance

Determines the number of times to check the job status before the job ends Not OK.

Default: 2

Apache NiFi JobLink copied to clipboard

Apache NiFi is an open-source tool that automates data flow across systems in real time.

To create an Apache NiFi job, see Creating a Job. For more information about this plug-in, see Control-M for Apache NiFi.

The following table describes the Apache NiFi job attributes.

Attribute

Description

Connection Profile

Determines the authorization credentials that are used to connect Control-M to Apache NiFi, as described in Apache NiFi Connection Profile Parameters.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Blank spaces.
Processor Group ID

Defines the ID number of a specific processor group.
Processor ID

Defines the ID number of a specific processor.
Action

Determines the action to perform on Apache NiFi.

Valid Values:

  • Run Processor

  • Stop Processor

  • Update Processor

  • Run Processor Once
Disconnected Node Ack

Determines whether to disconnect the node to allow mutable requests to proceed.
Status Polling Frequency

Determines the number of seconds to wait before checking the job status.

Default: 5
Failure Tolerance

Determines the number of times to check the job status before the job ends Not OK.

Default: 0

Astronomer JobLink copied to clipboard

Astronomer is a workload automation service based on Apache Airflow that enables you to create, schedule, and manage your workflows.

To create an Astronomer job, see Creating a Job. For more information about this plug-in, see Control-M for Astronomer.

The following table describes the Astronomer job attributes.

Attribute

Description

Connection Profile

Determines the authorization credentials that are used to connect Control-M to Astronomer, as described in Astronomer Connection Profile Parameters.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Whitespace

Action

Determines whether to run a new DAG or rerun a DAG.

Valid Values:

  • Run DAG

  • Rerun DAG

DAG Name

Defines the logical name of the Directed Acyclic Graph (DAG).

The DAG Name is defined in the Airflow interface.

DAG Run ID

(Optional) Defines the specific DAG run (execution) ID in Airflow.

Parameters

Defines the JSON-based body parameters to pass when the DAG executes, in the following format:

CopyCopied to clipboard 
"Variable1":"Value1","Variable2":"Value2"

Only Failed Tasks

Determines whether to rerun a DAG job only with failed tasks or with all tasks.

Valid Values:

  • true

  • false

Status Polling Frequency

Determines the number of seconds to wait before checking the job status.

Default: 60

Failure Tolerance

Determines the number of times to check the job status before the job ends Not OK.

Default: 3

Amazon MWAA JobLink copied to clipboard

Amazon Managed Workflows for Apache Airflow (MWAA) is an orchestration service built on Apache Airflow, designed to create, schedule, and monitor data pipelines and workflows.

To create an Amazon MWAA job, see Creating a Job. For more information about this plug-in, see Control-M for Amazon MWAA.

The following table describes the Amazon MWAA job attributes.

Attribute

Description

Connection Profile

Determines the authorization credentials that are used to connect Control-M to Amazon MWAA, as described in Amazon MWAA Connection Profile Parameters.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Blank spaces

Action

Determines whether to run a new DAG or rerun a DAG.

Valid Values:

  • Run DAG

  • Rerun DAG

MWAA Environment Name

Defines the logical name of the MWAA environment.
DAG Name

Defines the logical name of the Directed Acyclic Graph (DAG).
DAG Run ID

(Optional) Defines the unique identifier for a specific DAG run in an orchestration system. The ID helps track and manage individual workflow executions. If you do not provide a DAG Run ID, the system generates a random Run ID.

Parameters

Defines the parameters for the Amazon MWAA job, in JSON format, which enables you to control how the job executes.

Use backslashes to escape quotes.

CopyCopied to clipboard 
{\"parameter1\":\"value1\"}

If you are not adding parameters, type {}.

Only Failed Tasks

Determines whether to rerun a DAG job only with failed tasks or with all tasks.

Valid Values:

  • true

  • false

Status Polling Frequency

Determines the number of seconds to wait before checking the job status.

Default: 60

Failure Tolerance

Determines the number of times to check the job status before the job ends Not OK.

Default: 3

AWS Step Functions JobLink copied to clipboard

AWS Step Functions enables you to create visual workflows that can integrate other AWS services.

To create an AWS Step Functions job, see Creating a Job. For more information about this plug-in, see Control-M for AWS Step Functions.

The following table describes the AWS Step Functions job attributes.

Attribute

Description

Connection Profile

Determines the authorization credentials that are used to connect Control-M to AWS Step Functions, as described in AWS Step Functions Connection Profile Parameters.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Blank spaces.

Execution Name

Defines the name of the Step Function execution.

An execution executes a state machine, which is a workflow.

State Machine ARN

Determines the Step Function state machine to use.

A state machine is a workflow, and an Amazon Resource Name (ARN) is a standardized AWS resource address.

arn:aws:states:us-east-1:155535555553:stateMachine:MyStateMachine

Parameters

Defines the parameters for the Step Function job, in JSON format, which enables you to control how the job runs.

Use backslashes to escape quotes.

CopyCopied to clipboard 
{\"parameter1\":\"value1\"}

If you are not adding parameters, type {}.

Show Execution Logs

Determines whether to add the job log to the output.

Status Polling Frequency

Determines the number of seconds to wait before checking the job status.

Default: 20

Failure Tolerance

Determines the number of times to check the job status before the job ends Not OK.

Default: 2

Azure Logic Apps JobLink copied to clipboard

Azure Logic Apps enables you to design and automate cloud-based workflows and integrations.

To create an Azure Logic Apps job, see Creating a Job. For more information about this plug-in, see Control-M for Azure Logic Apps.

The following table describes the Azure Logic Apps job attributes.

Attribute

Description

Connection Profile

Determines the authorization credentials that are used to connect Control-M to Azure Logic Apps, as described in Azure Logic Apps Connection Profile Parameters.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Blank spaces.

Workflow

Determines which of the Consumption logic app workflows executes from your predefined set of workflows.

This job does not execute Standard logic app workflows.

Parameters

Defines parameters, in JSON format, that enable you to control the presentation of data.

CopyCopied to clipboard 
{ 
  "param1":"value1",
  "param2":"value2"
}

Rules:

  • Characters: 2–4,000

  • For no parameters, type {}.

Get Logs

Determines whether to display the job output when the job ends.

Failure Tolerance

Determines the number of times to check the job status before the job ends Not OK.

Default: 2

Status Polling Frequency

Determines the number of seconds to wait before checking the job status.

Default: 20

GCP Composer JobLink copied to clipboard

Google Cloud (GCP) Composer is a managed workflow orchestration service built on Apache Airflow that enables you to automate workflow tasks.

To create a GCP Composer job, see Creating a Job. For more information about this plug-in, see Control-M for GCP Composer.

The following table describes GCP Composer job attributes.

Attribute

Description

Connection Profile

Determines the authorization credentials that are used to connect Control-M to GCP Composer, as described in Application Workflow Connection Profiles.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Blank spaces.

Action

Determines whether to run a new DAG or rerun a DAG.

Valid Values:

  • Run DAG

  • Rerun DAG

DAG Name

Defines the logical name of the Directed Acyclic Graph (DAG).

The DAG Name is defined in the GCP interface.

DAG Run ID

(Optional) Defines the specific DAG run (execution) ID in GCP Composer.

Parameters JSON Input

Defines the JSON-based body parameters to pass when the DAG executes, in the following format:

CopyCopied to clipboard 
{"Variable1":"Value1","Variable2":"Value2"}

For no parameters, type {}.

Only Failed Tasks

Determines whether to rerun a DAG job only with failed tasks or with all tasks.

Valid Values:

  • true

  • false

Status Polling Frequency

Determines the number of seconds to wait before checking the job status.

Default: 60

Failure Tolerance

Determines the number of times to check the job status before the job ends Not OK.

Default: 2

GCP Workflows JobLink copied to clipboard

GCP Workflows enables you to design and automate cloud-based workflows and integrations.

To create a GCP Workflows job, see Creating a Job. For more information about this plug-in, see Control-M for GCP Workflows.

The following table describes GCP Workflows job attributes.

Attribute

Description

Connection Profile

Determines the authorization credentials that are used to connect Control-M to GCP Workflows, as described in GCP Workflows Connection Profile Parameters.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Blank spaces

Project ID

Defines the identifier of the GCP project where the job runs.

A project is a set of configuration settings that define the resources the jobs utilize and how they interact with GCP.

Location

Determines the region where the job runs.

us-central1

Workflow Name

Determines the predefined GCP Workflow that executes.

Parameters

Defines the JSON-based body parameters that are passed to the function, in the following format:

CopyCopied to clipboard 
{\"var1\":\"value1\",\"var2\":\"value2\"}

Execution Label

Defines a job execution label, which enables you to group similar executions in the GCP Workflows log.

{"labelName": "name"}

Show Workflow Results

Determines whether the GCP Workflow results appear in the job output.

Status Polling Frequency

Determines the number of seconds to wait before checking the job status.

Default: 20

Failure Tolerance

Determines the number of times to check the job status before the job ends Not OK.

Default: 3