Data Processing and Analytics Connection Profiles

The following topics describe connection profiles for data processing and analytics platforms and services:

• ConnectionProfile:AWS Athena

• ConnectionProfile:AWS Data Pipeline

• ConnectionProfile:AWS DynamoDB

• ConnectionProfile:AWS EMR

• ConnectionProfile:Azure Databricks

• ConnectionProfile:Azure HDInsight

• ConnectionProfile:Azure Synapse

• ConnectionProfile:Databricks

• ConnectionProfile:DBT

• ConnectionProfile:GCP BigQuery

• ConnectionProfile:GCP DataFlow

• ConnectionProfile:GCP Dataproc

• ConnectionProfile:Hadoop

• ConnectionProfile:OCI Data Flow

• ConnectionProfile:Snowflake

• ConnectionProfile:Snowflake IdP

ConnectionProfile:AWS Athena

AWS Athena enables you to process, analyze, and store your data in the cloud.

The following examples show how to define an AWS Athena connection profile.

• This JSON-based connection profile authenticates with an AWS access key and secret:

  Copy

  "AWS_ATHENA":
  {
     "Type": "ConnectionProfile:AWS Athena",
     "AWS API Base URL": "https://athena.us-east-1.amazonaws.com",
     "AWS Region": "us-east-1",
     "Authentication": "SECRET",
     "AWS Access Key": "ABCDEF",
     "AWS Secret Key": "******",
     "Description": "",
     "Centralized": true
  }

• This JSON-based connection profile authenticates with an AWS IAM role from inside an EC2 instance:

  Copy

  "AWS_ATHENA": 
  {
     "Type": "ConnectionProfile:AWS Athena",
     "AWS API Base URL": "https://athena.us-east-1.amazonaws.com",
     "AWS Region": "us-east-1",
     "Authentication": "NOSECRET",
     "IAM Role": "ATHENAIAMROLE",
     "Description": "",
     "Centralized": true
  }

The following table describes the AWS Athena connection profile parameters.

ConnectionProfile:AWS Data Pipeline

AWS Data Pipeline is a cloud-based extract, transform, load (ETL) service that enables you to automate the transfer, processing, and storage of your data.

The following examples show how to define an AWS Data Pipeline connection profile.

• This JSON-based connection profile authenticates with an AWS access key and secret:

  Copy

  "AWSDATAPIPELINESECRET ": 
  {
     "Type": "ConnectionProfile:AWS Data Pipeline",
     "Authentication": "SECRET",
     "AWS Access Key": "MYAWSACCESSKEY1234",
     "AWS Secret": "myAwsSecret12345",
     "AWS Region": "us-east-1",
     "Data Pipeline URL": "https://datapipeline.{{AWSRegion}}.amazonaws.com",
     "Connection Timeout": "30",
     "Description": "",
     "Centralized": true
  }

• This JSON-based connection profile authenticates with an AWS IAM role from inside an EC2 instance:

  Copy

  "AWSDATAPIPELINEIAM":
  {
     "Type": "ConnectionProfile:AWS Data Pipeline",
     "Authentication": "NOSECRET",
     "IAM Role": "IAMROLENAME",
     "AWS Region": "us-east-1",
     "Data Pipeline URL": "https://datapipeline.{{AWSRegion}}.amazonaws.com",
     "Connection Timeout": "30",
     "Description": "",
     "Centralized": true
  }

The following table describes the AWS Data Pipeline connection profile parameters.

ConnectionProfile:AWS DynamoDB

AWS DynamoDB is a NoSQL database service that enables you to create database tables, execute statements and transactions, export and import data to and from the Amazon S3 storage service.

For more information, see Control-M for AWS DynamoDB.

The following examples show how to define an AWS DynamoDB connection profile.

• This JSON-based connection profile authenticates with AWS Key & Secret:

  Copy

  "AWS_DynamoDB": 
  {
     "Type": "ConnectionProfile:AWS DynamoDB",
     "Authentication": "Secret",
     "AWS Secret": "*****",
     "AWS Region": "us-east-1",
     "AWS Access Key": "ZKIATY7B2LKB2JQ85I6D",
     "AWS Backup URL": "https://dynamodb.{{AWSRegion}}.amazonaws.com",
     "Description": "",
     "Connection Timeout": "100",
     "Centralized": true
  }

• This JSON-based connection profile authenticates with IAM Role:

  Copy

  "AWS_ADY_IAM": 
  {
     "Type": "ConnectionProfile:AWS DynamoDB",
     "Authentication": "NoSecret",
     "AWS Region": "us-east-1",
     "AWS Backup URL": "https://dynamodb.{{AWSRegion}}.amazonaws.com",
     "IAM Role": "arn:aws:iam::122343212345:role/Amazon12SSMRoleForInstancesQuickSetup",
     "Description": "",
     "Connection Timeout": "50",
     "Centralized": true
  }

The following table describes the AWS DynamoDB Connection Profile Parameters.

ConnectionProfile:AWS EMR

AWS EMR is a managed cluster platform that enables you to execute big data frameworks, such as Apache Hadoop and Apache Spark, to process and analyze vast amounts of data.

The following example shows how to define an AWS EMR connection profile:

"AWS_EMR":
{
   "Type": "ConnectionProfile:AWS EMR",
   "AWSRegion": "us-east-1",
   "EMRAccessKey": “ABCDEF",
   "EMRSecretKey": "****",
   "Description": "",
   "Centralized": true
}

The following table describes the AWS EMR connection profile parameters.

ConnectionProfile:Azure Databricks

Azure Databricks is a cloud-based data analytics platform that enables you to process and analyze large workloads of data.

The following examples show how to define a connection profile for an Azure Databricks instances job:

• This JSON-based connection profile authenticates using a service principal:

  Copy

  "ADF_SERVPRINC":
  {
     "Type": "ConnectionProfile:Azure Databricks",
     "Authentication Method": "PRINCIPAL",
     "Tenant ID": "tenantId",
     "Application ID": "4f477fa3-1a1g-4877-ca92-f39bb563f3b1",
     "Client Secret": "*****", 
     "Databricks url": "https://adb-1111211144444680.0.azuredatabricks.net",
     "Azure Login url": "https://login.microsoftonline.com",
     "Connection Timeout": "50",
     "Description": "",
     "Centralized": true
  }

• This JSON-based connection profile authenticates using a managed identity:

  Copy

  "ADF_MANID":
  {
     "Type": "ConnectionProfile:Azure Databricks",
     "Authentication Method": "MANAGEDID",
     "Specify Managed Identity Client ID": "&client_id=",
     "Managed Identity Client ID": "72d454264f0-ac32-45ea-9158-f8s5433e4ee16",
     "Databricks url": "https://adb-1111211144444680.0.azuredatabricks.net",
     "Azure Login url": "https://login.microsoftonline.com",
     "Connection Timeout": "50",
     "Description": "",
     "Centralized": true
  }

The following table describes the Azure Databricks connection profile parameters.

ConnectionProfile:Azure HDInsight

Azure HDInsight enables you to execute an Apache Spark batch job and perform big data analytics.

The following example shows how to define an Azure HDInsight connection profile:

"AZUREHDINSIGHT":
{
   "Type": "ConnectionProfile:Azure HDInsight",     
   "Cluster Name": "hdcluster", 
   "Cluster Username": "admin",  
   "Cluster Password": "*****", 
   "Description": "",
   "Centralized": true
}

The following table describes the Azure HDInsight connection profile parameters.

ConnectionProfile:Azure Synapse

Azure Synapse enables you to perform data integration and big data analytics.

The following examples show how to define an Azure Synapse connection profile.

• This JSON-based connection profile authenticates with an Azure service principal:

  Copy

  "AZURE_SYNAPSE_1":
  {
     "Type": "ConnectionProfile:Azure Synapse",
     "Authentication Method": "PRINCIPAL",
     "Tenant ID": "tenantId",
     "Azure AD url": "https://login.microsoftonline.com",
     "Synapse url": "https://ncu-if-synapse.dev.azuresynapse.net",
     "Synapse Resource": "https://dev.azuresynapse.net/",
     "App ID": "4f477fa3-1a1g-4877-ca92-f39bb563f3b1",
     "Client Secret": "*****",
     "Connection Timeout": "50",
     "Description": "",
     "Centralized": true
  }

• This JSON-based connection profile authenticates with a managed identity:

  Managed identity authentication is based on an Azure token that is valid, by default, for 24 hours. Token lifetime can be extended by Azure.

  Copy

  "AZURE_SYNAPSE_2":
  {
     "Type": "ConnectionProfile:Azure Synapse",
     "Authentication Method": "MANAGEDID",
     "Specify Managed Identity Client ID": "&client_id=",
     "Managed Identity Client ID": "72d448f0-ac32-45ea-9158-f8653e4ee16",
     "Synapse url": "https://ncu-if-synapse.dev.azuresynapse.net",
     "Synapse Resource": "https://dev.azuresynapse.net/",
     "Connection Timeout": "50",
     "Description": "",
     "Centralized": true
  }

The following table describes the Azure Synapse connection profile parameters.

ConnectionProfile:Databricks

Databricks enables the integration of jobs created in the Databricks environment with your existing Control-M workflows.

The following example shows how to define a Databricks connection profile:

"DATABRICKS":
{
   "Type": "ConnectionProfile:Databricks",
   "Databricks workspace url": "https://dbc-7b944b32-faf0.cloud.databricks.com",
   "Databricks personal access token": "*****", 
   "Connection Timeout": "50",
   "Description": "",
   "Centralized": true
}  

The following table describes the Databricks connection profile parameters.

ConnectionProfile:DBT

dbt (Data Build Tool) is a cloud-based computing platform that enables you to develop, test, schedule, document, and analyze data models.

The following example shows how to define a dbt connection profile:

"DBT_CP":
{
   "Type": "ConnectionProfile:DBT",
   "DBT URL": "https://cloud.getdbt.com", 
   "DBT Token": "*****",
   "Account ID": "123456",
   "Connection Timeout": "60",
   "Description": "",
   "Centralized": true
}

The following table describes the dbt connection profile parameters.

ConnectionProfile:GCP BigQuery

GCP BigQuery is a Google Cloud Platform computing service that you can use for data storage, processing, and analysis.

The following example shows how to define a GCP BigQuery connection profile that authenticates with a service account:

"BIGQSA":
{
   "Type": "ConnectionProfile:GCP BigQuery",
   "Identity Type": "service_account",
   "Service Account Key": "*****",
   "BigQuery URL": "https://bigquery.googleapis.com",
   "Description": "",
   "Centralized": true
}

The following table describes the GCP BigQuery connection profile parameters.

ConnectionProfile:GCP DataFlow

Google Cloud Platform (GCP) Dataflow enables you to perform cloud-based data processing for batch and real-time, data-streaming applications.

The following example shows how to define a GCP Dataflow connection profile, based on a service account:

"GCPDATAFLOW":
{ 
   "Type": "ConnectionProfile:GCP DataFlow", 
   "Identity Type": "service_account", 
   "DataFlow URL": "https://dataflow.googleapis.com", 
   "Service Account Key": "{\"type\":\"service_account\",\"project_id\":\"sso-gcp-dba-ctm1-priv-cc30752\",\"private_key_id\":\"5197d05c5b8212bea944985cec74a34d6c1868aa\",\"private_key\":\"-----BEGIN PRIVATE KEY-----\\nprivate-key\\n-----END PRIVATE KEY-----\\n\",\"client_email\":\"bmc-wla-svc-02@sso-gcp-dba-ctm1-priv-cc30752.iam.gserviceaccount.com\",\"client_id\":\"116650586827623521335\",\"auth_uri\":\"https://accounts.google.com/o/oauth2/auth\",\"token_uri\":\"https://oauth2.googleapis.com/token\",  \"auth_provider_x509_cert_url\":\"https://www.googleapis.com/oauth2/v1/certs\",\"client_x509_cert_url\":\"https://www.googleapis.com/robot/v1/metadata/x509/bmc-wla-svc-02%40sso-gcp-dba-ctm1-priv-cc30752.iam.gserviceaccount.com\"}",
   "Description": "", 
   "Centralized": true 
}

The following table describes the GCP Dataflow connection profile parameters.

ConnectionProfile:GCP Dataproc

Google Cloud Platform (GCP) Dataproc enables you to perform cloud-based big data processing and machine learning.

The following example shows how to define a GCP Dataproc connection profile that authenticates with a service account:

"GCPDATAPROC":
{ 
   "Type": "ConnectionProfile:GCP Dataproc", 
   "Identity Type": "service_account", 
   "Dataproc URL": "https://dataproc.googleapis.com", 
   "Service Account Key": "{\"type\":\"service_account\",\"project_id\":\"sso-gcp-dba-ctm1-priv-cc30752\",\"private_key_id\":\"5197d05c5b8212bea944985cec74a34d6c1868aa\",\"private_key\":\"-----BEGIN PRIVATE KEY-----\\nprivate-key\\n-----END PRIVATE KEY-----\\n\",\"client_email\":\"bmc-wla-svc-02@sso-gcp-dba-ctm1-priv-cc30752.iam.gserviceaccount.com\",\"client_id\":\"116650586827623521335\",\"auth_uri\":\"https://accounts.google.com/o/oauth2/auth\",\"token_uri\":\"https://oauth2.googleapis.com/token\",  \"auth_provider_x509_cert_url\":\"https://www.googleapis.com/oauth2/v1/certs\",\"client_x509_cert_url\":\"https://www.googleapis.com/robot/v1/metadata/x509/bmc-wla-svc-02%40sso-gcp-dba-ctm1-priv-cc30752.iam.gserviceaccount.com\"}",
   "Connection timeout": "20",
   "Description": "", 
   "Centralized": true 
} 

The following table describes the GCP Dataproc connection profile parameters.

ConnectionProfile:Hadoop

The Hadoop job connects to the Hadoop framework, which enables you to split up and process large data sets on clusters of commodity servers. You can expand your enterprise business workflows to include tasks that execute in your Big Data Hadoop cluster in Control-M with the different Hadoop-supported tools, including Pig, Hive, HDFS File Watcher, Map Reduce Jobs, and Sqoop.

The following examples show how to define a Hadoop connection profile for various types of Hadoop jobs.

Hadoop (All Types)

The following example defines the parameters required for all Hadoop types:

"HADOOP_CONNECTION_PROFILE":
{
   "Type": "ConnectionProfile:Hadoop",
   "TargetAgent": "edgenode",
   "TargetCTM": "CTMHost"
}

The following table describes the Hadoop connection profile parameters.

The following example defines the optional parameters for defining the user running the Hadoop job types and choosing between a local or centralized connection profile:

"HADOOP_CONNECTION_PROFILE":
{
   "Type" : "ConnectionProfile:Hadoop",
   "TargetAgent" : "edgenode",
   "TargetCTM" : "CTMHost",
   "RunAs": "",
   "KeyTabPath":"",
   "Centralized":true
}

The following table describes the Hadoop connection profile parameters shown in the example above.

The following table describes the parameters that control security in the case of Kerberos security.

Apache Spark

Apache Spark enables you to define access to a Spark server:

The following example defines an Apache Spark connection profile:

"SPARK_CONNECTION_PROFILE" :
{
   "Type": "ConnectionProfile:Hadoop",
   "TargetAgent": "docker-hadoop5",
   "Spark":
   {
      "CustomPath": "/home"
   }
}

The CustomPath parameter is optional.

Apache Oozie

Apache Oozie enables you to access an Oozie server for a job that submits an Oozie workflow.

The following example defines an Apache Oozie connection profile:

"OOZIE_CONNECTION_PROFILE" :
{
   "Type": "ConnectionProfile:Hadoop",
   "TargetAgent": "hdp-ubuntu",
   "Centralized": false,
   "Oozie":
   {
      "SslEnabled": false,
      "Host": "hdp-centos",
      "Port": "11000",
      "ExtractionRules": [
      {
         "RuleName": "rule_name1",
         "WorkFlowName": "work_flow_name1",
         "WorkFlowUserName": "work_flow_user_name1",
         "FolderName": "folder_name1",
         "JobName": "job_name1"
      },
      {
         "RuleName": "rule_name2",
         "WorkFlowName": "work_flow_name2",
         "WorkFlowUserName": "work_flow_user_name2",
         "FolderName": "folder_name2",
         "JobName": "job_name2"
      } ]
   }
}

The following table describes the Apache Oozie connection profile parameters.

Apache Sqoop

Apache Sqoop enables you to run a Sqoop job.

The following example shows a connection profile that defines a Sqoop data source and access credentials:

"SQOOP_CONNECTION_PROFILE" :
{
   "Type": "ConnectionProfile:Hadoop",
   "TargetAgent": "edgenode",
   "TargetCTM": "CTMHost",
   "Centralized": false,
   "Sqoop":
   {
      "User": "username",
      "Password": "userpassword",
      "ConnectionString": "jdbc:mysql://mysql.server/database",
      "DriverClass": "com.mysql.jdbc.Driver"
   }
}

The following table describes the Sqoop connection profile parameters shown in the example above, as well as several additional optional parameters.

Apache Tajo

Tajo is an advanced data warehousing system on top of HDFS.

The following example shows a connection profile that defines access to a Tajo server:

"TAJO_CP" :
{
   "Type": "ConnectionProfile:Hadoop",
   "TargetAgent": "edgenode",
   "Tajo":
   {
      "BinaryPath": "$TAJO_HOME/bin/",
      "DatabaseName": "myTajoDB",
      "MasterServerName" : "myTajoServer",
      "MasterServerPort": "26001"
   }
}

The following table describes the Tajo connection profile parameters.

Apache Hive

Apache Hive enables you to run a Hive BeeLine job.

The following examples show how to define a Hadoop Hive connection profile.

• This JSON defines a connection profile with a Hive BeeLine endpoint and access credentials:

  The parameters in the example translate to the following BeeLine command:

  beeline -u jdbc:hive2://<Host>:<Port>/<DatabaseName>

  Copy

  "HIVE_CONNECTION_PROFILE" :
  {
     "Type": "ConnectionProfile:Hadoop",
     "TargetAgent": "edgenode",
     "TargetCTM": "CTMHost",
     "Hive":
     {
        "Host": "hive_host_name",
        "Port": "10000",
        "DatabaseName": "hive_database",
     }
  }

• This JSON defines a connection profile with optional parameters for a Hadoop Hive type connection profile:

  The parameters in the example translate to the following BeeLine command:

  beeline -u jdbc:hive2://<Host>:<Port>/<DatabaseName>;principal=<Principal> -n <User> -p <Password>

  Copy

  "HIVE_CONNECTION_PROFILE1":
  {
     "Type": "ConnectionProfile:Hadoop",
     "TargetAgent": "edgenode",
     "TargetCTM": "CTMHost",
     "Centralized": true,
     "Hive":
     {
        "Host": "hive_host_name",
        "Port": "10000",
        "DatabaseName": "hive_database",
        "User": "user_name",
        "Password": "user_password",
        "Principal": "Server_Principal_of_HiveServer2@Realm"
     }
  }

To update an existing connection profile and keep the current password, type five *, as follows:

*****

ConnectionProfile:OCI Data Flow

Oracle Cloud Infrastructure (OCI) Data Flow is a fully managed Apache Spark service that performs processing tasks on extremely large datasets.

The following examples show how to define an OCI Data Flow connection profile.

• This JSON-based connection profile authenticates with defined parameters:

  Copy

  "OCI_DATAFLOW": 
  {
     "Type": "ConnectionProfile:OCI Data Flow",
     "OCI Data Flow URL": "https://dataflow.region.oci.oraclecloud.com",
     "OCI Region": "us-phoenix-1",
     "Authentication": "DefineParameters",
     "User OCID": "ocid1.user.oc1..aaaaaaaasxjplkxcnplaxxxxyutitixxxxxxxxxxxxxxxxxxxxxxxxxxxx",
     "Tenancy OCID": "ocid1.tenancy.oc1..aaaaaaaak4xxxhtutyuyxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",   
     "Fingerprint": "dc:50:2d:e1:ax:af:x7:x6:xe:x9:ax:cx:cb:x3:x3:6x",
     "Private Key": " ----BEGIN PRIVATE KEY---- XXXXXXXXXX ----END PRIVATE KEY---- ",
     "Connection Timeout": "30",
     "Description": "",
     "Centralized": true
  }

• This JSON-based connection profile authenticates with a configuration file:

  Copy

  "OCI_ DATAFLOW_CF": 
  {
     "Type": " ConnectionProfile:OCI Data Flow",
     "OCI Data Flow URL": "https://dataflow.region.oci.oraclecloud.com",
     "Authentication": "ConfigurationFile",
     "Config File Path": "\home\dbauser\config.example", 
     "Profile": "Default",
     "Connection Timeout": "30",
     "Description": "",
     "Centralized": true
  }

The following table describes the OCI Data Flow connection profile parameters.

ConnectionProfile:Snowflake

Snowflake is a cloud-computing platform that enables you to process, analyze, and store your data.

The following example shows how to define a Snowflake connection profile:

"SNOWFLAKE_CONNECTION_PROFILE":
{ 
   "Type": "ConnectionProfile:Snowflake", 
   "Account Identifier": "{Account_ID}",  
   "Region": "us-east-1",
   "Client ID": "DuHj****************",
   "Client Secret": "*****",
   "Refresh Token": "ver%******************",   
   "Redirect URI": "https%****************” 
   "Description": "", 
   "Centralized": true 
}

The following table describes the Snowflake connection profile parameters.

ConnectionProfile:Snowflake IdP

Snowflake is a cloud-computing platform that enables you to process, analyze, and store your data, with authentication based on an Identity Provider (IdP).

The following example shows how to define a Snowflake connection profile with authentication based on an Identity Provider (IdP):

"SNOWFLAKE_IDP_CONNECTION_PROFILE":
{
   "Type": "ConnectionProfile:Snowflake IdP",
   "Account Identifier": "{Account_ID}",
   "Region": "us-east-1", 
   "Client ID": "DuHj****************", 
   "Client Secret": "*****",
   "IDP URL": "https://****************",
   "Scope": "session:role:<custom_role>", 
   "Description": "",
   "Centralized": true
}

The following table describes the Snowflake connection profile parameters.