Control-M for Apache Airflow

Apache Airflow enables you to create, schedule, and monitor complex data processing and analytics pipelines. It provides an environment to define, manage, and execute workflows as Directed Acyclic Graphs (DAGs) to control task dependencies and execution order.

Control-M for Apache Airflow enables you to do the following:

  • Manage Apache Airflow credentials in a secure connection profile.

  • Connect to any Apache Airflow endpoint.

  • Introduce all Control-M capabilities to Control-M for Apache Airflow, including advanced scheduling criteria, complex dependencies, resource pools, lock resources, and variables.

  • Integrate Apache Airflow jobs with other Control-M jobs into a single scheduling environment.

  • Monitor the status, results, and output of Apache Airflow jobs.

  • Attach an SLA job to the Apache Airflow jobs.

Control-M for Apache Airflow Compatibility

The following table lists the prerequisites that are required to use the Apache Airflow plug-in, each with its minimum required version.

Component

Version

Control-M/EM

9.0.22.000

Control-M/Agent

9.0.22.000

Control-M Application Integrator

9.0.22.000

Control-M Automation API

9.0.22.000

Minimum Airflow Version

2.5.0

Control-M for Apache Airflow is supported on Control-M Web and Control-M Automation API, but not on the Control-M client. You cannot use Promotion with the plug-in.

To download the installation files for each prerequisite, see Obtaining Control-M Installation Files.

Setting up Control-M for Apache Airflow

This procedure describes how to deploy the Apache Airflow plug-in, create a connection profile, and define an Apache Airflow job in Control-M Web and Automation API.

Integration plug-ins released by BMC require an Application Integrator installation. However, these plug-ins are not editable and you cannot import them into Application Integrator. To deploy these integrations to your Control-M environment, import them directly into Control-M with Control-M Automation API.

Before You Begin

Begin

  1. Create a temporary directory to save the downloaded files.

  2. Download the Apache Airflow plug-in from the Control-M for Apache Airflow download page in the EPD site.

  3. Do the following to install the Apache Airflow plug-in with the Control-M Automation API provision service:

    1. Log in as an administrator to the Control-M/EM server host and save the downloaded ZIP file to the following location:

      • Linux: $HOME/ctm_em/AUTO_DEPLOY

      • Windows: <EM_HOME>\AUTO_DEPLOY

    2. Log in to the account where Control-M/Agent is installed and run the following provision image command:

      • Linux: ctm provision image Apache_Airflow_plugin.Linux

      • Windows: ctm provision image Apache_Airflow_plugin.Windows

  4. Create an Apache Airflow connection profile in Control-M Web or Automation API, as follows:

  5. Define an Apache Airflow job in Control-M Web or Automation API, as follows:

To remove this plug-in from an Agent, see Removing a Plug-in from the Agent. The plug-in ID is AAF112024.

Change Log

The following table provides details about changes that were introduced in new versions of this plug-in:

Plug-in Version

Details

1.0.06

Added Rerun DAG ID parameter to job definition.

1.0.05

  • Changed the default value of the Switch to Rerun and Keep Run ID parameters from Always to Never.

  • Enhanced the field description of the Logical Date parameter.

1.0.04

Added Get Task Output to job definition.

1.0.03

  • Added Switch to Rerun parameter to job definition to determine when the Action parameter reverts from Run DAG to Rerun DAG.

  • Added Keep Run ID parameter to job definition to determine when a job automatically saves a DAG Run ID.

1.0.02

  • Added support for bearer token authentication.

  • Added Airflow REST API parameter to select Airflow REST API v2 or v3.

1.0.01

Rerun DAG action added

1.0.00

Initial version