Control-M for Apache Airflow

Apache Airflow enables you to create, schedule, and monitor complex data processing and analytics pipelines. It provides an environment to define, manage, and execute workflows as Directed Acyclic Graphs (DAGs) to control task dependencies and execution order.

Control-M for Apache Airflow enables you to do the following:

• Manage Apache Airflow credentials in a secure connection profile.

• Connect to any Apache Airflow endpoint.

• Introduce all Control-M capabilities to Control-M for Apache Airflow, including advanced scheduling criteria, complex dependencies, resource pools, lock resources, and variables.

• Integrate Apache Airflow jobs with other Control-M jobs into a single scheduling environment.

• Monitor the status, results, and output of Apache Airflow jobs.

• Attach an SLA job to the Apache Airflow jobs.

Setting up Control-M for Apache Airflow

This procedure describes how to deploy the Apache Airflow plug-in, create a connection profile, and define an Apache Airflow job in Control-M SaaS and Automation API.

Before You Begin

• Verify that you have Java installed, as described in Control-M External Java Installation.

• Verify that Automation API is installed, as described in Setting Up the API.

• Verify that Agent version 9.0.21.080 or higher is installed.

Begin

1. Do one of the following:

   • Install: Run one of the following provision image commands:

     • Linux: ctm provision image Apache_Airflow_plugin.Linux

     • Windows: ctm provision image Apache_Airflow_plugin.Windows

   • Upgrade: Run the following command:

     ctm provision agent::update

2. Create an Apache Airflow connection profile in Control-M SaaS or Automation API, as follows:

   • Control-M SaaS: Create a Centralized Connection Profile with Apache Airflow Connection Profile Parameters

   • Automation API: ConnectionProfile:Apache Airflow

3. Define an Apache Airflow job in Control-M SaaS or Automation API, as follows:

   • Control-M SaaS: Create a Job with parameters Apache Airflow Job parameters

   • Automation API: Job:Apache Airflow

Change Log

The following table provides details about changes that were introduced in new versions of this plug-in:

Plug-in Version	Details
1.0.02	Added support for bearer token authentication. Added Airflow REST API parameter to select Airflow REST API v2 or v3.
1.0.01	Rerun DAG action added
1.0.00	Initial version