Control-M for Azure Data Factory

Azure Data Factory is a cloud-based extract, transform, load (ETL) and data integration service that allows you to create data-driven workflows to automate the movement and transformation of data.

Control-M for Azure Data Factory enables you to do the following:

• Execute Azure Data Factory jobs.

• Manage Azure Data Factory credentials in a secure connection profile.

• Connect to any Azure Data Factory endpoint.

• Introduce all Control-M capabilities to Control-M for Azure Data Factory, including advanced scheduling criteria, complex dependencies, resource pools, lock resources, and variables.

• Integrate Azure Data Factory jobs with other Control-M jobs into a single scheduling environment.

• Monitor the status, results, and output of Azure Data Factory jobs.

• Attach an SLA job to the Azure Data Factory jobs.

• Run 50 Azure Data Factory jobs simultaneously per Agent.

Setting up Control-M for Azure Data Factory

This procedure describes how to deploy the Azure Data Factory plug-in, create a connection profile, and define an Azure Data Factory job in Control-M SaaS and Automation API.

Before You Begin

• Verify that Automation API is installed, as described in Setting Up the API.

• Verify that Agent version 9.0.21.080 or higher is installed.

Begin

1. On the Agent host, run one of the following commands to set the Java environment variable:

   • Linux:

     • Bourne shell/bash: export BMC_INST_JAVA_HOME=<java_directory>

     • csh/tcsh: setenv BMC_INST_JAVA_HOME <java_directory>

   • Windows: set BMC_INST_JAVA_HOME="<java_directory>"

2. Do one of the following:

   • Install: Run one of the following provision image commands:

     • Linux: ctm provision image ADF_plugin.Linux

     • Windows: ctm provision image ADF_plugin.Windows

   • Upgrade: Run the following command:

     ctm provision agent::update

3. Create an Azure Data Factory connection profile in Control-M SaaS or Automation API, as follows:

   • Control-M SaaS: Create a Centralized Connection Profile with Azure Data Factory Connection Profile Parameters

   • Automation API: ConnectionProfile:ADF (Azure Data Factory)

4. Define an Azure Data Factory job in Control-M SaaS or Automation API, as follows:

   • Control-M SaaS: Create a Job with Azure Data Factory Job parameters

   • Automation API: Job:ADF (Azure Data Factory)

Change Log

The following table provides details about changes that were introduced in new versions of this plug-in:

Plug-in Version	Details
1.0.06	Added new Restart on Rerun parameter to the job definition to rerun a job (pipeline) from the point of failure Added HTTP Codes, Rerun Interval, and Attempt Reruns parameters to the connection profile to rerun an execution step with an HTTP code
1.0.05	Added resolve on rerun for job parameters
1.0.04	Added Failure Tolerance parameter Applied connection timeout override to all execution steps Changed Polling Interval default value to 45
1.0.03	Fixed the abort operation for REST API steps
1.0.02	New job icon
1.0.00	Initial version