Data Processing and Analytics Jobs

Defines the ConnectionProfile:AWS Athena name that connects Control-M to AWS Athena.

Defines a unique ID (idempotency token), which guarantees that the job executes only once.

Default: aws-athena-client-request-token-%%ORDERID-%%TIME

Defines the name of the group of databases (catalog) that the query references.

Defines the name of the database that the query references.

Determines which of the following queries executes:

• Query: Executes the query that you enter in the Query attribute.

• Run Prepared Query: Executes a predefined query that is stored in the AWS Athena platform.

• Query and Create Table: Executes the query that you enter in the Query attribute and saves the results to a new table.

• Unload: Executes the query that you enter in the Query attribute and saves the results to a file in an Amazon S3 bucket.

Defines the SQL-based query that executes.

Defines the name of the predefined query that is stored in the AWS Athena platform.

Defines the name of the table that is created, which is populated by the results of a query in AWS Athena.

Determines the file format that the query results are saved in, as follows:

• JSON

• CSV

• ORC

• Parquet

• Avro

• Text File

Defines the AWS S3 bucket path where the file is saved, as follows.

s3://<path>

Defines the workgroup for this job.

Workgroups can consist of users, teams, applications, or workloads, and can set limits on the data that each query or group processes.

Determines whether to add additional job definitions.

Valid Values:

• checked

• unchecked

Default: unchecked

Defines the Amazon S3 canned access control list (ACL), which is a predefined set of grantees and permissions assigned to your stored query results.

BUCKET_OWNER_FULL_CONTROL is the only canned ACL that is currently supported in AWS Athena. This setting gives you and the bucket owner full control of the query results.

Determines one of the following ways to encrypt the query results:

• SSE_S3: Encrypts the data in the Amazon S3 with Server-Side Encryption (SSE) and Amazon S3-managed encryption keys.

• SSE_KMS: Encrypts the data in the Amazon S3 with SSE and the AWS Key Management Service (KMS), which enables you to manage the encryption keys.

• CSE_KMS: Encrypts the data in the Amazon S3 object storage with SSE and enables you to provide your own encryption keys.

(SSE_KMS and CSE_KMS only) Defines the Amazon Resource Name (ARN) of the KMS key.

An ARN is a standardized AWS resource address.

Defines the AWS account ID of the Amazon S3 bucket owner.

Determines whether to show the full JSON API response in the job output.

Valid Values:

• checked

• unchecked

Default: unchecked

Determines the number of seconds to wait before checking the status of the job.

Default: 10

Defines the ConnectionProfile:AWS Data Pipeline name that connects Control-M to AWS Data Pipeline.

Determines one of the following AWS Data Pipeline actions:

• Trigger Pipeline: Runs an existing AWS Data Pipeline.

• Create Pipeline: Creates a new AWS Data Pipeline.

(Create Pipeline) Defines the name of the new AWS Data Pipeline.

(Create Pipeline) Defines the unique ID (idempotency key) that guarantees the pipeline is created only once. After successful execution, this ID cannot be used again.

Valid Values: Any alphanumeric characters.

(Create Pipeline) Defines the parameter objects, which define the variables, for your AWS Data Pipeline in JSON format.

For more information about the available parameter objects, see the descriptions of the PutPipelineDefinition and GetPipelineDefinition actions in the AWS Data Pipeline API Reference.

(Create Pipeline) Determines whether to run (trigger) the newly created AWS Data Pipeline.

Valid Values:

• checked

• unchecked

This parameter is relevant only for a creation action. For a trigger action, set it to unchecked.

(Trigger Pipeline) Determines which pipeline to run (trigger).

Determines the number of seconds to wait before checking the status of the Data Pipeline job.

Default: 20

Defines the ConnectionProfile:AWS DynamoDB name that connects Control-M to AWS DynamoDB.

Determines one of the following AWS DynamoDB actions to perform:

• Execute Statement

• Execute Transaction

• Export Job to S3 Bucket

• Import Job from S3 Bucket

Determines whether to execute the statement with your own JSON parameters.

Valid Values:

• checked

• unchecked

Default: unchecked

Defines one or more the PartiQL statement that are supported by AWS DynamoDB.

Defines the parameters for the AWS DynamoDB job, in JSON format, that enable you to control how the job executes, as appears in the following example:

[{\"N\": \"20\"},{\"S\":\"Stas30\"}]

Defines one or more PartiQL transaction statements, as appears in the following example:

[%4E{%4E \"Parameters\": [{\"N\": \"20\"},{\"S\":\"Stas30\"}],%4E\"Statement\": \"Select * From IFteam where Id=17\"%4E },%4E{%4E \"Parameters\": [{\"N\": \"20\"},{\"S\":\"Stas30\"}],%4E\"Statement\": \"Select * From IFteam where Id=18\"%4E]

Defines the unique ID (idempotency token) that guarantees the job is executed only once. After successful execution, this ID cannot be used again.

Determines one of the following formats to export data:

• DYNAMODB JSON

• ION

Determines one of the following formats of the source data:

• CSV

• DYNAMODB JSON

• ION

Defines the Amazon S3 bucket name to export and import to and from the table.

Defines the Amazon S3 bucket prefix to use as the filename and path of the table.

Defines the ID of the AWS account that owns the bucket.

Defines the Amazon Resource Name (ARN) associated with the table to export.

Determines one of the following compression types to compress the data from the imported table:

• GZIP

• ZSTD

• No Compression

Defines the name of the new table where the data is imported, as appears in the following example:

"Attribute Definitions": [
{
   "AttributeName": "Id".
   "AttributeType": "N"
}]
"KeySchema": [
{
   "AttributeName": "Id".
   "KeyType": "HASH"
}]
"BillingMode": "PROVISIONED",
"ProvisionedThroughput":
{
   "RealCapacityUnits": 1,
   "WriteCapacityUnits": 1
}

Defines the name of the new table where the data is imported.

Determines the number of seconds to wait before checking the status of the job.

Default: 20

Defines the ConnectionProfile:AWS EMR name that connects Control-M to AWS EMR.

Defines the name of the AWS EMR cluster that connects to the Notebook.

In the EMR API, the cluster ID is also known as the Execution Engine ID.

Determines which Notebook ID executes the script.

In the EMR API, the Notebook ID is also known as the Editor ID.

Defines the full directory path and filename of the script in the Notebook.

Defines the job execution name.

Defines the service role that connects to the Notebook.

Determines whether to provide Notebook execution information through JSON code.

Valid Values:

• checked

• unchecked

Default: unchecked

If you set this parameter to checked, the JSON Body parameter replaces several other parameters discussed above (Cluster ID, Notebook ID, Relative Path, Notebook Execution Name, and Service Role).

Defines the ConnectionProfile:Azure Databricks name that connects Control-M to Azure Databricks.

Defines the job ID created in your Databricks workspace.

Defines task parameters to override when the job runs, according to the Databricks convention. The list of parameters must begin with the name of the parameter type.

For more information about the parameter types, review the properties of RunParameters in the OpenAPI specification provided through the Azure Databricks documentation.

For no parameters, specify a value of "params": <>.

(Optional) Defines a token to use to rerun job runs that timed out in Databricks.

Valid Values:

• Control-M-Idem_%%ORDERID: With this token, upon rerun, Control-M invokes the monitoring of the existing job run in Databricks. Default.

• Any other value: Replaces the Control-M idempotency token. When you rerun a job using a different token, Databricks creates a new job run with a new unique run ID.

Defines the ConnectionProfile:Azure HDInsight name that connects Control-M to Azure HDInsight.

Defines parameters to be passed on the Apache Spark application during job execution, in JSON format (name:value pairs).

This JSON must include the file and className elements.

For more information about common parameters, see Batch Job in the Azure HDInsight documentation.

Defines the number of seconds to wait before verification of the Apache Spark batch job.

Default: 10

Defines the ConnectionProfile:Azure Synapse name that connects Control-M to Azure Synapse.

Defines the name of a pipeline that you defined in your Azure Synapse workspace.

Defines pipeline parameters to override when the job runs, defined in JSON format as pairs of name and value, as follows.

<\"param1\":\"val1\", \"param2\":\"val2\">

For no parameters, specify <>.

Defines the ConnectionProfile:Databricks name that connects Control-M to Databricks.

Determines the job ID created in your Databricks workspace.

Defines task parameters to override when the job runs, according to the Databricks convention. The list of parameters must begin with the name of the parameter type.

For more information about the parameter types, review RunParameters properties in the OpenAPI specification provided through the Azure Databricks documentation.

For no parameters, specify a value of "params": {}.

(Optional) Defines a token to use to rerun job runs that timed out in Databricks.

Valid Values:

• Control-M-Idem_%%ORDERID: With this token, upon rerun, Control-M invokes the monitoring of the existing job run in Databricks.

• <Any Other Value>:  Replaces the Control-M idempotency token. When you rerun a job using a different token, Databricks creates a new job run with a new unique run ID.

Default: Control-M-Idem_%%ORDERID

Defines the ConnectionProfile:DBT name that connects Control-M to dbt.

Defines the ID of the preexisting job in the dbt platform that you want to run.

Defines a free-text description of the job.

Determines whether to override the predefined dbt job commands.

Valid Values:

• checked

• unchecked

Default: unchecked

Defines the new dbt job commands as variable pairs, as follows:

"UCM-DefineCommands-Nnnn-element": "command string"

where nnn is a counter for the sequential position of each command.

Determines the number of seconds to wait before checking the status of the job.

Default: 10

Defines the ConnectionProfile:GCP BigQuery name that connects Control-M to GCP BigQuery.

Determines one of the following GCP BigQuery actions to perform:

• Query:  Runs one or more SQL statements that are supported by GCP BigQuery.

• Copy: Creates a copy of an existing table.

• Load: Loads source data into an existing table.

• Extract: Exports data from an existing table into Google Cloud Storage.

• Routine: Runs a stored procedure, table function, or previously defined function.

Determines the project that the job uses.

Determines the database that the job uses.

(Optional) Determines whether to paste the results of a SELECT statement into a new table.

Defines the new table name.

Defines one or more  SQL statements supported by GCP BigQuery.

Rule: It must be written in a single line, with character strings separated by one space only.

Defines the query parameters, which enables you to control the presentation of the data.

"name": "IFteam",
"paramterType":
{ 
   "type": "STRING"
},
"parameterValue":
{
   "value": "BMC"
}

Determines one of the following copy operations:

• Clone: Creates a copy of a base table that has write access.

• Snapshot: Creates a read-only copy of a base table.

• Copy: Creates a copy of a snapshot.

• Restore:  Creates a writable table from a snapshot.

Defines the properties of the table that is cloned, backed up, or copied, in JSON format.

You can copy or back up one or more tables at a time.

   {
      "datasetId": "Test1", 
      "projectId": "SomeProj1",
      "tableId": "IFteam1"
   }
   {
      "datasetId": "Test2", 
      "projectId": "SomepProj2", 
      "tableId": "IFteam2"
   }

Defines the properties of a new table, in JSON format.

{ 
   "datasetId": "Test3", 
   "projectId": "SomeProj3", 
   "tableId": "IFteam3" 
}

Defines the source or destination data URI for the table that you are loading or extracting.

You can load or extract multiple tables.

Rule: Separate elements with ,.

Determines whether to add more fields to a table that you are loading.

Defines additional fields for the table that you are loading.

"schema": 
{
   "fields": [
   {
      "name": "name1",
      "type": "STRING1"
   }
   {
      "name": "name2",
      "type": "STRING2"
   }
   {
      "name": "name3",
      "type": "STRING3"
   } ]
}

Determines one of the following file formats to export the data to:

• CSV

• JSON

Defines a routine and the values that it must run.

Determines the maximum number of milliseconds to run the GCP BigQuery job.

Determines the number of seconds to wait before the job ends Not OK.

Default: 10

Defines the ConnectionProfile:GCP DataFlow name that connects Control-M to GCP DataFlow.

Defines the project ID for your Google Cloud project.

Defines the Google Compute Engine region to create the job.

Defines one of the following types of GCP Dataflow templates:

• Classic Template: Developers run the pipeline and create a template. The Apache Beam SDK stages files in Cloud Storage, creates a template file (similar to job request), and saves the template file in Cloud Storage.

• Flex Template: Developers package the pipeline into a Docker image and then use the Google Cloud CLI to build and save the Flex Template spec file in Cloud Storage.

Defines the path for temporary files. This must be a valid Google Cloud Storage URL that begins with gs://.

The default pipeline option tempLocation is used if it has been set in the GCP Dataflow service.

Defines input parameters to be passed on to job execution, in JSON format (name:value pairs).

This JSON must include the jobname and parameters elements.

(Optional) Determines the number of seconds to wait before checking the status of the job.

Default: 10

Determines one of the following levels of details to retrieve from the GCP outputs in the case of job failure:

• TRACE

• DEBUG

• INFO

• WARN

• ERROR

Defines the ConnectionProfile:GCP Dataproc name that connects Control-M to GCP Dataproc.

Defines the project ID for your Google Cloud project.

Defines the Google Compute Engine region to create the job.

Defines one of the following Dataproc task types to execute:

• Workflow Template: A reusable workflow configuration that defines a graph of jobs with information on where to run those jobs.

• Job: A single Dataproc job.

(Workflow Template) Defines the ID of a Workflow Template.

(Job) Defines input parameters to be passed on to job execution, in JSON format.

You retrieve this JSON content from the GCP Dataproc UI, using the EQUIVALENT REST option in job settings.

(Optional) Determines the number of seconds to wait before checking the status of the job.

Default: 20

Defines the ConnectionProfile:Hadoop name that connects Control-M to Apache Spark.

Defines the HDFS commands to perform before and after running the job. For example, you can use them for preparation and cleanup.

Defines the ConnectionProfile:Hadoop name that connects Control-M to Apache Spark.

Defines HDFS commands to perform before and after running the job. For example, you can use them for preparation and cleanup.

Defines the ConnectionProfile:Hadoop name that connects Control-M to Apache Pig.

Defines the HDFS commands to perform before and after running the job. For example, you can use them for preparation and cleanup.

Defines the ConnectionProfile:Hadoop name that connects Control-M to Apache Sqoop.

Defines the HDFS commands to perform before and after running the job. For example, you can use them for preparation and cleanup.

Determines whether to ignore failure in the pre- or post- commands.

PreCommands: Defaults to true, which ends the job Not OK if a pre-command fails.

PostCommands: Defaults to false, which ends the job OK, even when a post-command fails.

Defines the parameters to pass as the specific sqoop tool args.

Determines the location of the Hadoop archives.

Defines the ConnectionProfile:Hadoop name that connects Control-M to Apache Hive.

Defines the HDFS commands to perform before and after running the job. For example, you can use them for preparation and cleanup.

Determines whether to ignore failure in the pre- or post- commands.

PreCommands: Defaults to true, which ends the job Not OK if a pre-command fails.

PostCommands: Defaults to false, which ends the job OK, even when a post-command fails.

Defines the additional Hadoop command options to pass to beeline as hivevar “name”=”value”.

Defines the additional Hadoop command options to pass to beeline as hiveconf “key”=”value”.

Defines the additional Hadoop command options to pass to beeline as hiveconf mapred.cache.archives=”value”.

Defines the ConnectionProfile:Hadoop name that connects Control-M to Apache Distributed Copy.

Defines the ConnectionProfile:Hadoop name that connects Control-M to Apache HDFS Commands.

Defines the ConnectionProfile:Hadoop name that connects Control-M to Apache HDFS FileWatcher.

Defines the name of the host machine where the job runs. An Agent must be installed on this host. Optionally, you can define a host group instead of a host machine.

Defines the full path of the file being watched.

Defines the minimum file size in bytes to meet the criteria and finish the job as OK. If the file arrives, but the size is not met, the job continues to watch the file.

Defines the ConnectionProfile:Hadoop name that connects Control-M to Apache Oozie.

Defines the name of the host machine where the job runs. An Agent must be installed on this host. Optionally, you can define a host group instead of a host machine.

Defines the HDFS commands to perform before and after running the job. For example, you can use them for preparation and cleanup.

Determines whether to ignore failure in the pre- or post- commands.

PreCommands: Defaults to true, which ends the job Not OK if a pre-command fails.

PostCommands: Defaults to false, which ends the job OK, even when a post-command fails.

Defines the ConnectionProfile:Hadoop name that connects Control-M to Apache MapReduce.

Defines the HDFS commands to perform before and after running the job. For example, you can use them for preparation and cleanup.

Defines the ConnectionProfile:Hadoop name that connects Control-M to Apache MapReduce Streaming.

Defines the HDFS commands to perform before and after running the job. For example, you can use them for preparation and cleanup.

Determines whether to ignore failure in the pre- or post- commands.

PreCommands: Defaults to true, which ends the job Not OK if a pre-command fails.

PostCommands: Defaults to false, which ends the job OK, even when a post-command fails.

Defines the ConnectionProfile:Hadoop for name that name that connects Control-M to Apache Tajo.

Defines the name of the host machine where the job runs. An Agent must be installed on this host. Optionally, you can define a host group instead of a host machine.

Defines the ConnectionProfile:OCI Data Flow name that connects Control-M to OCI Data Flow.

Defines the name of a new Run.

Defines the compartment Oracle Cloud Identifier (OCID) which is a unique identifier assigned to each compartment that is created within the Oracle Data Flow Infrastructure.

Defines the application Oracle Cloud Identifier (OCID) which is a unique identifier assigned to each application that is created within the Oracle Data Flow Infrastructure.

(Optional) Determines whether to add more parameters to the new job run.

Valid Values:

• No

• Yes

Default: No

(Optional) Defines specific parameters, in JSON format, that are passed when you create a new Run.

For more information about the run parameters, see CreateRunDetails Reference 20200129 in the Oracle Cloud Infrastructure Documentation.

{
   "displayName": "<run_name>",
   "applicationId": "<application_ocid>",
   "compartmentId": "<compartment_ocid>",
   "driverShape": "VM.Standard.E4.Flex",
   "executorShape": "VM.Standard.E4.Flex",
   "numExecutors": 1,
   "arguments": [],
   "parameters": [],
   "configuration": {}
}

Determines the number of seconds to wait before checking the job status.

Default:60

Defines one of the following connection profile types that connects Control-M to Snowflake:

• ConnectionProfile:Snowflake

• ConnectionProfile:Snowflake IdP

Determines the database that the job uses.

Determines the schema that the job uses.

A schema is an organizational model that describes layout and definition of the fields and tables, and their relationships to each other, in a database.

Determines one of the following Snowflake actions to perform:

• SQL Statement: Runs any number of Snowflake-supported SQL commands, such as queries, calling or creating procedures, database maintenance tasks, and creating and editing tables.

• Copy from Query: Copies a queried database and schema into an existing or new file in cloud storage.

• Copy from Table: Copies from an existing table.

• Create Table and Query: Creates a table, populated by a query, in the specified database and schema.

• Copy into Table: Copies data from a cloud storage location into an existing table in Snowflake.

• Start or Pause Snowpipe: Starts or pauses an existing Snowpipe.

• Stored Procedure: Calls an existing procedure and its arguments.

• Snowpipe Load Status: Monitors the status of a Snowpipe for a set period of time.

• Run SQL File: Uploads a file that contains Snowflake-supported SQL commands.

Determines one or more Snowflake-supported SQL commands.

Rule: Must be written in a single line, with strings separated by one space only.

Defines the cloud storage location.

Defines the query used for copying the data.

Defines the storage integration object, which stores an Identity and Access Management (IAM) entity and an optional set of blocked cloud storage locations.

Determines whether to overwrite an existing file in the cloud storage, as follows:

• Yes

• No

Determines one of the following file formats for the saved file:

• JSON

• CSV

Determines where the JSON or CSV file is saved.

You can save to Amazon Web Services, Google Cloud Platform, or Microsoft Azure.

Defines the name of the copied table.

Defines the name of the new or existing table where the data is queried.

Defines the query used for the copied data.

Defines the name of the Snowpipe.

A Snowpipe loads data from files when they are ready or staged.

Defines the name of the table that the data is copied into.

Defines the cloud storage location from where the data is copied, in CSV or JSON format.

Determines whether to start or pause the Snowpipe, as follows:

• Start Snowpipe

• Pause Snowpipe

Defines the name of the stored procedure.

Defines the value of the argument in the stored procedure.

Defines the table that is monitored when loaded by the Snowpipe.

Defines the cloud storage location.

A stage is a pointer that indicates where data is stored, or staged.

Determines the number of days to monitor the Snowpipe load status.

Defines the cloud storage location where a CSV file log is created.

The CSV file log details the load status for each Snowpipe.

Defines the Snowflake configuration for the cloud storage location (as defined in the previous parameter, Status File Cloud Location Path).

Defines the full path to the file that contains Snowflake-supported SQL commands.

Determines the maximum number of seconds to run the job in Snowflake.

Determines whether the following job-defining attributes are displayed:

• Parameters

• Role

• Bindings

• Warehouse

Defines Snowflake-provided parameters that let you control how data is presented, as follows.

<"param1":"value1", "param2":"value2">

Determines the Snowflake role used for this Snowflake job.

A role is an entity that can be assigned privileges on secure objects. You can be assigned one or more roles from a limited selection.

Defines the values to bind to the variables used in the Snowflake job, in JSON format. For more information about bindings, see the Snowflake documentation.

"1":
{ 
   "type": "FIXED", 
   "value": "123" 
} 
"2":
{ 
   "type": "TEXT", 
   "value": "String" 
}

Determines the warehouse used in the Snowflake job.

A warehouse is a cluster of virtual machines that processes a Snowflake job.

Determines whether to show a full JSON response in the log output.

Valid Values:

• checked

• unchecked

Default: unchecked