Control-M for Databricks

Databricks is a cloud-based data analytics platform that enables you to process and analyze large workloads of data.

Control-M for Databricks enables you to do the following:

• Execute Databricks jobs.

• Manage Databricks credentials in a secure connection profile.

• Connect to any Databricks endpoint.

• Introduce all Control-M capabilities to Control-M for Databricks, including advanced scheduling criteria, complex dependencies, Resource Pools, Lock Resources, and variables.

• Integrate Databricks jobs with other Control-M jobs into a single scheduling environment.

• Monitor the status, results, and output of Databricks jobs.

• Attach an SLA job to the Databricks jobs.

• Run 50 Databricks jobs simultaneously per Agent.

Setting up Control-M for DatabricksLink copied to clipboard

This procedure describes how to deploy the Databricks plug-in, create a connection profile, and define a Databricks job in Control-M SaaS and Automation API.

Before You Begin

• Verify that Automation API is installed, as described in Setting Up the API.

• Verify that Agent version 9.0.21.080 or later is installed.

1. On the Agent host, run one of the following commands to set the Java environment variable:

   • Linux:

     • Bourne shell/bash: export BMC_INST_JAVA_HOME=<java_11_directory>

     • csh/tcsh: setenv BMC_INST_JAVA_HOME <java_11_directory>

   • Windows: set BMC_INST_JAVA_HOME="<java_11_directory>"

2. Run one of the following API commands:

   • To install, type one of the following provision image commands:

     • Linux: ctm provision image DBX_plugin.Linux

     • Windows: ctm provision image DBX_plugin.Windows

   • To upgrade, type the following command:

     ctm provision agent::update

3. Create a Databricks connection profile in Control-M SaaS or Automation API, as follows:

   • Control-M SaaS: Create a Centralized Connection Profile with Databricks Connection Profile Parameters

   • Automation API: ConnectionProfile:Databricks

4. Define a Databricks job in Control-M SaaS or Automation API, as follows:

   • Control-M SaaS: Create a Job with Databricks Job parameters

   • Automation API: Job:Databricks

Change LogLink copied to clipboard

The following table provides details about changes that were introduced in new versions of this plug-in:

Plug-in Version	Details
1.0.06	Added Failure Tolerance job parameter
1.0.05	Added semantic changes
1.0.04	Removed the Job Name attribute
1.0.03	Add new job icon
1.0.02	Added idempotency enhancement
1.0.01	Added multiple task enhancement
1.0.00	Initial version