Control-M for GCP Dataproc

Google Cloud Platform (GCP) Dataproc enables you to perform cloud-based big data processing and machine learning.

Control-M for GCP Dataproc enables you to do the following:

• Execute single or Workflow Template GCP Dataproc jobs.

• Manage GCP Dataproc credentials in a secure connection profile.

• Connect to any GCP Dataproc endpoint.

• Introduce all Control-M capabilities to Control-M for GCP Dataproc, including advanced scheduling criteria, complex dependencies, resource pools, lock resources, and variables.

• Integrate GCP Dataproc jobs with other Control-M jobs into a single scheduling environment.

• Monitor the status, results, and output of GCP Dataproc jobs.

• Attach an SLA job to the GCP Dataproc jobs.

• Run 50 GCP Dataproc jobs simultaneously per Agent.

Setting up Control-M for GCP Dataproc

This procedure describes how to deploy the GCP Dataproc plug-in, create a connection profile, and define a GCP Dataproc job in Control-M SaaS and Automation API.

Before You Begin

• Verify that Java is installed, as described in Control-M External Java Installation.

• Verify that Automation API is installed, as described in Setting Up the API.

• Verify that Agent version 9.0.21.080 or higher is installed.

Begin

1. Do one of the following:

   • Install: Run one of the following provision image commands:

     • Linux: ctm provision image GDP_plugin.Linux

     • Windows: ctm provision image GDP_plugin.Windows

   • Upgrade: Run the following command:

     ctm provision agent::update

2. Create a GCP Dataproc connection profile in Control-M SaaS or Automation API, as follows:

   • Control-M SaaS: Create a Centralized Connection Profile with GCP Dataproc Connection Profile Parameters

   • Automation API: ConnectionProfile:GCP Dataproc

3. Define a GCP Dataproc job in Control-M SaaS or Automation API, as follows:

   • Control-M SaaS: Create a Job with GCP Dataproc Job parameters

   • Automation API: Job:GCP Dataproc

Change Log

The following table provides details about changes that were introduced in new versions of this plug-in:

Plug-in Version	Details
1.0.04	Updated job behavior to mark jobs as Failed in batch processing if their status is Cancelled Fixed the ability to manually abort a step
1.0.03	Added ability to terminate the interactive session resource.
1.0.02	Set the Batch ID and Requested ID parameters to resolve on rerun
1.0.01	Added the ability to trigger batch jobs using Dataproc Serverless for Spark Batch Jobs Added the new Batches option to the Dataproc task type parameter Added the following new parameters: Batch ID Requested ID
1.0.00	Initial version