Control-M for Azure Databricks

Azure Databricks is a cloud-based data analytics platform that enables you to process large workloads of data.

Control-M for Azure Databricks enables you to do the following:

• Execute Azure Databricks jobs.

• Manage Azure Databricks credentials in a secure connection profile.

• Connect to any Azure Databricks endpoint.

• Introduce all Control-M capabilities to Control-M for Azure Databricks, including advanced scheduling criteria, complex dependencies, Resource Pools, Lock Resources, and variables.

• Integrate Azure Databricks jobs with other Control-M jobs into a single scheduling environment.

• Monitor the status, results, and output of Azure Databricks jobs.

• Attach an SLA job to the Azure Databricks jobs.

• Run 50 Azure Databricks jobs simultaneously per Agent.

Setting up Control-M for Azure DatabricksLink copied to clipboard

This procedure describes how to deploy the Azure Databricks plug-in, create a connection profile, and define an Azure Databricks job in Control-M SaaS and Automation API.

Before You Begin

• Verify that Automation API is installed, as described in Setting Up the API.

• Verify that Agent version 9.0.21.080 or later is installed.

1. On the Agent host, run one of the following commands to set the Java environment variable:

   • Linux:

     • Bourne shell/bash: export BMC_INST_JAVA_HOME=<java_11_directory>

     • csh/tcsh: setenv BMC_INST_JAVA_HOME <java_11_directory>

   • Windows: set BMC_INST_JAVA_HOME="<java_11_directory>"

2. Run one of the following API commands:

   • To install, type one of the following provision image commands:

     • Linux: ctm provision image ZDX_plugin.Linux

     • Windows: ctm provision image ZDX_plugin.Windows

   • To upgrade, type the following command:

     ctm provision agent::update

3. Create an Azure Databricks connection profile in Control-M SaaS or Automation API, as follows:

   • Control-M SaaS: Create a Centralized Connection Profile with Azure Databricks Connection Profile Parameters

   • Automation API: ConnectionProfile:Azure Databricks

4. Define an Azure Databricks job in Control-M SaaS or Automation API, as follows:

   • Control-M SaaS: Create a Job with Azure Databricks Job parameters

   • Automation API: Job:Azure Databricks

Change LogLink copied to clipboard

The following table provides details about changes that were introduced in new versions of this plug-in:

Plug-in Version	Details
1.0.06	Added Managed Identity authentication added
1.0.05	Added Failure Tolerance job parameter
1.0.04	Removed the Job Name attribute
1.0.03	Added new job icon
1.0.02	Added idempotency enhancements
1.0.01	Added multiple task enhancements
1.0.00	Initial version