Control-M for Azure Data Factory

Azure Data Factory is a cloud-based extract, transform, load (ETL) and data integration service that allows you to create data-driven workflows to automate the movement and transformation of data.

Control-M for Azure Data Factory enables you to do the following:

  • Execute Azure Data Factory jobs.

  • Manage Azure Data Factory credentials in a secure connection profile.

  • Connect to any Azure Data Factory endpoint.

  • Introduce all Control-M capabilities to Control-M for Azure Data Factory, including advanced scheduling criteria, complex dependencies, Resource Pools, Lock Resources, and variables.

  • Integrate Azure Data Factory jobs with other Control-M jobs into a single scheduling environment.

  • Monitor the status, results, and output of Azure Data Factory jobs.

  • Attach an SLA job to the Azure Data Factory jobs.

  • Run 50 Azure Data Factory jobs simultaneously per Agent.

Control-M for Azure Data Factory Compatibility

The following table lists the prerequisites that are required to use the Azure Data Factory plug-in version 1.0.06, each with its minimum required version.

Component

Version

Control-M/EM

9.0.21.300

Control-M/Agent

9.0.21.300

Control-M Application Integrator

9.0.21.301

Control-M Automation API

9.0.21.310

Azure Data Factory version 1.0.05 is still available to support sites with Control-M version 9.0.20.200.

The following table lists the prerequisites that are required to use the Azure Data Factory version 1.0.05 plug-in, each with its minimum required version.

Component

Version

Control-M/EM

9.0.20.200

Control-M/Agent

9.0.20.200

Control-M Application Integrator

9.0.20.201

Control-M Automation API

9.0.20.250

Control-M for Azure Data Factory is supported on Control-M Web and Control-M Automation API, but not on the Control-M client.

To download the required installation files for each prerequisite, see Obtaining Control-M Installation Files.

Setting up Control-M for Azure Data Factory

This procedure describes how to deploy the Azure Data Factory plug-in, create a connection profile, and define an Azure Data Factory job in Control-M Web and Automation API.

Integration plug-ins released by BMC require an Application Integrator installation. However, these plug-ins are not editable and you cannot import them into Application Integrator. To deploy these integrations to your Control-M environment, import them directly into Control-M with Control-M Automation API.

Before You Begin

Verify that Automation API is installed, as described in Automation API Installation.

Begin

  1. Create a temporary directory to save the downloaded files.

  2. Download the Azure Data Factory plug-in from the Control-M for Azure Data Factory download page in the EPD site.

  3. Install the Azure Data Factory plug-in via one of the following methods:

    • (9.0.21 or higher) Use the Automation API Provision service, as follows:

      1. Log in to the Control-M/EM Server machine as an Administrator and store the downloaded zip file in the one of the following locations:

        • Linux: $HOME/ctm_em/AUTO_DEPLOY

        • Windows: <EM_HOME>\AUTO_DEPLOY

      2. Log in to the Control-M/Agent machine and run the provision image command, as follows:

        • Linux: ctm provision image ADF_plugin.Linux

        • Windows: ctm provision image ADF_plugin.Windows

    • (9.0.20.200 or lower) Run the Automation API Deploy service, as described in deploy jobtype.

  4. Create an Azure Data Factory connection profile in Control-M Web or Automation API, as follows:

  5. Define an Azure Data Factory job in Control-M Web or Automation API, as follows:

To remove this plug-in from an Agent, see Removing a Plug-in. The plug-in ID is ADF062021.

Change Log

The following table provides details about changes that were introduced in new versions of this plug-in:

Plug-in Version

Details

1.0.06

  • Added the option to rerun a job (pipeline) from the point of failure

  • Added a rerun step per HTTP code feature for execution steps

  • Added new Restart on Rerun parameter to the job definition to rerun a job (pipeline) from the point of failure

  • Added HTTP Codes, Rerun Interval, and Attempt Reruns parameters to the connection profile to rerun an execution step with an HTTP code

1.0.05

Added resolve on rerun for job parameters

1.0.04

  • Added Failure Tolerance parameter

  • Applied connection timeout override to all execution steps

  • Changed Polling Interval default value to 45

1.0.03

Fixed the abort operation for REST API steps

1.0.02

New job icon

1.0.00

Initial version