Data Processing and Analytics Jobs

Determines the authorization credentials that are used to connect Control-M to Amazon Athena, as described in Amazon Athena Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Defines a unique ID (idempotency token), which guarantees that the job executes only once.

Default: aws-athena-client-request-token-%%ORDERID-%%TIME

Defines the name of the group of databases (catalog) that the query references.

Defines the name of the database that the query references.

Determines which of the following queries executes:

• Query: Executes the query that you enter in the Query attribute.

• Run Prepared Query: Executes a predefined query that is stored in the Amazon Athena platform.

• Query and Create Table: Executes the query that you enter in the Query attribute and saves the results to a new table.

• Unload: Executes the query that you enter in the Query attribute and saves the results to a file in an Amazon S3 bucket.

Defines the SQL-based query that executes.

Defines the name of the predefined query that is stored in the Amazon Athena platform.

Defines the name of the table that is created, which is populated by the results of a query in Amazon Athena.

Determines which file format the query results are saved in, as follows:

• *.json

• *.csv

• *.orc

• *.parquet

• *.avro

• *.txt

Defines the AWS S3 bucket path where the file is saved, in the following format:

s3://<path>

Defines the workgroup for this job.

Workgroups can consist of users, teams, applications, or workloads, and they can set limits on the data that each query or group processes.

Determines whether to add additional job definitions.

Valid Values:

• Yes

• No

Defines the Amazon S3 canned access control list (ACL), which is a predefined set of grantees and permissions assigned to your stored query results.

BUCKET_OWNER_FULL_CONTROL is the only canned ACL that is currently supported in Amazon Athena. This setting gives you and the bucket owner full control of the query results.

Determines one of the following ways to encrypt the query results:

• SSE_S3: Encrypts the data in the Amazon S3 with Server-Side Encryption (SSE) and Amazon S3-managed encryption keys.

• SSE_KMS: Encrypts the data in the Amazon S3 with SSE and the AWS Key Management Service (KMS), which enables you to manage the encryption keys.

• CSE_KMS: Encrypts the data in the Amazon S3 object storage with SSE and enables you to provide your own encryption keys.

(SSE_KMS and CSE_KMS only) Defines the Amazon Resource Name (ARN) of the KMS key.

An ARN is a standardized AWS resource address.

Defines the AWS account ID of the Amazon S3 bucket owner.

Determines whether to show the full JSON API response in the job output.

Determines the number of seconds to wait before checking the job status.

Default: 10

Determines the authorization credentials that are used to connect Control-M to AWS Data Pipeline, as described in AWS Data Pipeline Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Determines one of the following AWS Data Pipeline actions:

• Trigger Pipeline: Runs an existing AWS Data Pipeline.

• Create Pipeline: Creates a new AWS Data Pipeline.

Defines the name of the new AWS Data Pipeline.

Defines the unique ID (idempotency key) that guarantees the pipeline is created only once. After successful execution, this ID cannot be reused.

Valid Characters: Any alphanumeric characters.

Defines the parameter objects, in JSON format, which define the variables for your AWS Data Pipeline, as shown in the following example:

"parameterObjects": [ 
{
   "attributes": [
   {
      "key":"description",
      "stringValue":"S3outputfolder"
   } ],
   "id": "myS3OutputLoc"
} ],
"parameterValues": [
{
   "id":"myShellCmd",
   "stringValue":"grep -rc \"GET\" ${IN_DIR}/* > ${OUT_DIR}/out.txt"
} ],
"pipelineObjects": [ 
{
   "fields": [
   {
      "key":"input",
      "refValue":"S3InputLocation"
   },
   {
      "key":"stage",
      "stringValue":"true"
   } ],
   "id":"ShellCommandActivityObj",
   "name":"ShellCommandActivityObj"
} ]

For more information about the available parameter objects, see the descriptions of the PutPipelineDefinition and GetPipelineDefinition actions in the AWS Data Pipeline API Reference.

Determines whether to run (trigger) the newly created AWS Data Pipeline.

Determines which pipeline to run (trigger).

Determines the number of seconds to wait before checking the job status.

Default: 20

Determines the authorization credentials that are used to connect Control-M to Amazon DynamoDB, as described in Amazon DynamoDB Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces

Determines one of the following Amazon DynamoDB actions to perform:

• Execute Statement

• Execute Transaction

• Export Job to S3 Bucket

• Import Job from S3 Bucket

Determines whether to execute the statement with your own JSON parameters.

Defines one or more PartiQL statements that are supported by Amazon DynamoDB.

Defines the job parameters, in JSON format, that enable you to control how the job runs, as appears in the following example:

[{
   "N": "20"
},
{
   "S":"BMC30"
}]

Defines one or more PartiQL transaction statements, in JSON format, as appears in the following example:

[{
   "Parameters": [
   {
      "N": "20"
   },
   {
      "S"."Stas30
   }],
   "Statement": "Select*From IFteam where Id=17"
}]

Defines the unique ID (idempotency token) that guarantees the job runs only once.

After it runs successfully, this ID cannot be reused.

Determines one of the following formats to export data:

• DYNAMODB JSON

• ION

Determines the source data file format, as follows:

• *.csv

• *.json

• *.ion

Defines the Amazon S3 bucket name to export and import to and from the table.

Defines the Amazon S3 bucket prefix to use as the filename and path of the table.

Defines the ID of the AWS account that owns the bucket.

Defines the Amazon Resource Name (ARN) associated with the table to export.

Determines one of the following compression types to compress the data from the imported table:

• GZIP

• ZSTD

• No Compression

Defines the name of the new table where the data is imported, as appears in the following example:

"Attribute Definitions": [
{
   "AttributeName": "Id".
   "AttributeType": "N"
}]
"KeySchema": [
{
   "AttributeName": "Id".
   "KeyType": "HASH"
}]
"BillingMode": "PROVISIONED",
"ProvisionedThroughput":
{
   "RealCapacityUnits": 1,
   "WriteCapacityUnits": 1
}

Defines the name of the new table where the data is imported.

Determines the number of seconds to wait before checking the job status.

Default: 20

Determines the authorization credentials that are used to connect Control-M to Amazon EMR, as described in Amazon EMR Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Defines the name of the Amazon EMR cluster to connect to the Notebook.

In the EMR API, this field is called the Execution Engine ID.

Determines which Notebook ID executes the script.

In the EMR API, this field is called the Editor ID.

Defines the pathname of the script file in the Notebook.

Defines the job execution name.

Defines the service role that connects to the Notebook.

Determines whether to provide Notebook execution information through JSON code.

This JSON Body parameter replaces the values of the following parameters (Cluster ID, Notebook ID, Relative Path, Notebook Execution Name, and Service Role).

Determines the authorization credentials that are used to connect Control-M to Amazon Redshift, as described in Amazon Redshift Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Determines one of the following Amazon Redshift actions to perform:

• Redshift SQL Statement: Enables you to run all Redshift commands.

• Unload Data Into S3: Enables you to move data from Redshift to an S3 bucket in CSV or JSON file format.

• Copy Data Into Redshift: Enables you to copy CSV files from an S3 bucket to a Redshift table.

• Run Procedure: Enables you to run a stored procedure.

Defines the workgroup for this job.

Workgroups can consist of users, teams, applications, or workloads, and they can set limits for the data that each query or group processes.

Defines the Amazon Resource Name (ARN) associated with the AWS Secrets Manager, which securely stores and manages the database credentials.

Defines the database in Amazon Redshift.

Defines a query generated in the database as an .sql file.

Determines whether to display the statement results.

Defines the full URI of the S3 bucket that contains the extracted query results.

Determines one of the following file formats of the file placed in the S3 bucket:

• csv

• json

Determines whether to use an IAM Role to access the S3 bucket.

Defines the Amazon Resource Name (ARN) of the AWS IAM Role.

An ARN is a standardized AWS resource address.

The AWS IAM role must be granted read and write privileges to create or update any of the AWS resources that are in the stack.

Defines the name of the new table where the data is imported.

Defines the name of an existing procedure in Amazon Redshift.

Defines the arguments for the procedure that you run.

If you do not add an argument, type ().

(Optional) Determines the number of seconds to wait before checking the job status.

Default: 10

Determines the authorization credentials that are used to connect Control-M to Azure Databricks, as described in Azure Databricks Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

• Variable Name: %%AZURE-ACCOUNT

Defines the job ID created in a Databricks workspace.

Defines task parameters to override when the job runs, according to the Databricks convention. Your list of parameters must begin with the name of the parameter type, as shown in the following example:

"notebook_params": {"param1":"val1", "param2":"val2"}
"jar_params": ["param1", "param2"]

For more information about the parameter types, review the properties of RunParameters in the OpenAPI specification provided in the Azure Databricks documentation.

For no parameters, type the following code:

"params": {}

(Optional) Defines a token to use to rerun job runs that timed out in Databricks.

Values:

• Control-M-Idem_%%ORDERID: With this token, upon re-execution, Control-M invokes the monitoring of the existing job execution in Databricks.

• Any other value: Replaces the Control-M idempotency token. When you re-execute a job using a different token, Databricks creates a new job execution with a new unique run ID.

Default: Control-M-Idem_%%ORDERID

(Optional) Determines the number of seconds to wait before checking the job status.

Default: 30

Determines the authorization credentials that are used to connect Control-M to Azure HDInsight, as described in Azure HDInsight Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Determines which parameters are passed to the Apache Spark Application when the job runs, in JSON format (name:value pairs).

This JSON must include the file and className elements.

Determines the number of seconds to wait before checking the job status.

Default: 10 seconds

Determines the authorization credentials that are used to connect Control-M to Azure Synapse, as described in Azure Synapse Connection Profile Parameters.

Defines the name of a pipeline that you defined in your Azure Synapse workspace.

Defines pipeline parameters, in JSON format, to override when the job runs, as shown in the following example:

{"param1":"value1", "param2":"value2"}

For no parameters, type {}.

Determines the authorization credentials that are used to connect Control-M to Databricks, as described in Databricks Connection Profile Parameters.

Defines the job ID created in a Databricks workspace.

Defines task parameters, in JSON format, to override when the job executes, according to the Databricks convention. Your list of parameters must begin with the name of the parameter type, as shown in the following example:

"notebook_params": {"param1":"val1", "param2":"val2"}
"jar_params": ["param1", "param2"]

For more information about the parameter types, review the properties of RunParameters in the OpenAPI specification provided through the Azure Databricks documentation.

For no parameters, type the following code:

"params": {}

(Optional) Defines a token to use to rerun job runs that timed out in Databricks.

Values:

• Control-M-Idem_%%ORDERID: With this token, upon re-execution, Control-M invokes the monitoring of the existing job execution in Databricks.

• Any other value: Replaces the Control-M idempotency token. When you re-execute a job using a different token, Databricks creates a new job execution with a new unique run ID.

Default: Control-M-Idem_%%ORDERID

(Optional) Determines the number of seconds to wait before checking the job status.

Default: 30

Determines the authorization credentials that are used to connect Control-M to dbt, as described in dbt Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Defines the ID of the preexisting job in the dbt platform that you want to run.

Defines a free-text description of the job.

Determines whether to override the predefined dbt job commands.

Defines the new dbt job commands.

Determines the number of seconds to wait before checking the job status.

Default: 10

Determines the authorization credentials that are used to connect Control-M to GCP BigQuery, as described in GCP BigQuery Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Defines the name of the predefined Google Cloud project with configured APIs, authentication information, billing details, and job resources.

Determines the database that the job uses.

Determines one of the following GCP BigQuery actions to perform:

• Query:  Runs one or more SQL statements that are supported by GCP BigQuery.

• Copy: Creates a copy of an existing table.

• Load: Loads source data into an existing table.

• Extract: Exports data from an existing table into Google Cloud Storage.

• Routine: Runs a stored procedure, table function, or previously defined function.

(Optional) Determines whether to paste the results of a SELECT statement into a new table.

Defines the new table name.

Defines one or more  SQL statements supported by GCP BigQuery.

Rule: It must be written in a single line, with character strings separated by one space only.

Defines the query parameters, in JSON format, which enable you to control the presentation of the data, as shown in the following example:

{
   "name": "IFteam",
   "paramterType": {"type": "STRING"},
   "parameterValue": {"value": "BMC"}
}

Determines one of the following copy operations:

• Clone: Creates a copy of a base table that has write access.

• Snapshot: Creates a read-only copy of a base table.

• Copy: Creates a copy of a snapshot.

• Restore:  Creates a writable table from a snapshot.

Defines the properties of the table, in JSON format, that is cloned, backed up, or copied.

You can copy or back up one or more tables at a time, as shown in the following example:

{ 
    {
        "datasetId": "Test1", 
        "projectId": "SomeProj1",
        "tableId": "IFteam1"
    }
    {
        "datasetId": "Test2", 
        "projectId": "SomepProj2", 
        "tableId": "IFteam2"
    }
}

Defines the properties of a new table, in JSON format, as shown in the following example:

{
   "datasetId": "Test3", 
   "projectId": "SomeProj3", 
   "tableId": "IFteam3" 
}

Defines the source or destination data URI for the table that you are loading or extracting.

You can load or extract multiple tables.

Rule: Separate elements with ,.

Determines whether to add more fields to a table that you are loading.

Defines additional fields, in JSON format, for the table that you are loading, as shown in the following example:

"schema": 
{
   "fields": [
   {
      "name": "name1",
      "type": "STRING1"
   }
   {
      "name": "name2",
      "type": "STRING2"
   }
   {
      "name": "name3",
      "type": "STRING3"
   } ]
}

Determines which file format the data is exported in, as follows:

• *.csv

• *.json

Defines a routine and the values that it must run, as shown in the following example:

Call new_r(‘value1’)

Determines the maximum number of milliseconds to run the GCP BigQuery job.

Default: 30,000 milliseconds (30 seconds)

Determines the number of seconds to wait after Control-M initiates a connection request before a timeout occurs.

Default: 10

Determines the authorization credentials that are used to connect Control-M to GCP Dataflow, as described in GCP Dataflow Connection Profile Parameters.

Defines the identifier of the GCP project where the job runs.

A project is a set of configuration settings that define the resources the jobs utilize and how they interact with GCP.

Defines the Google Compute Engine region to create the job.

Defines one of the following types of GCP Dataflow templates:

• Classic Template: Developers run the pipeline and create a template. The Apache Beam SDK stages files in Cloud Storage, creates a template file (similar to job request), and saves the template file in Cloud Storage.

• Flex Template: Developers package the pipeline into a Docker image and then use the Google Cloud CLI to build and save the Flex Template spec file in Cloud Storage.

Defines the URL on Google Cloud Storage for the file that contains the Template definition, in the following format:

gs://bucketname/filename

Defines input parameters, in JSON format, to be passed on to job execution.

You must include the jobname and parameters elements, as shown in the following example:

{ 
    "jobName": "wordcount", 
    "parameters":
    { 
        "inputFile": "gs://dataflow-samples/shakespeare/kinglear.txt", 
        "output": "gs://controlmbucket/counts"
    } 
} 

(Optional) Determines the number of seconds to wait before checking the job status.

Default: 10

Determines the authorization credentials that are used to connect Control-M to GCP Dataproc, as described in GCP Dataproc Connection Profile Parameters.

Defines the identifier of the GCP project where the job runs.

A project is a set of configuration settings that define the resources the jobs utilize and how they interact with GCP.

Defines the Google Compute Engine region to create the job.

Defines one of the following Dataproc task types to execute:

• Workflow Template: Reusable workflow configuration that defines a graph of jobs with information on where to execute those jobs.

• Job: A single Dataproc job.

• Batches: Dataproc Serverless for Spark Batch.

• Interactive: An interactive job that deletes or terminates the interactive session resource.

(For a Workflow Template task type) Defines the ID of a Workflow Template.

Defines the ID that will become the final component of the batch resource name.

Valid Values:

4-63 characters. The letters a-z, and the numbers 0-9.

(Optional) Defines the unique ID that is used to identify the request. If the service receives two CreateBatchRequests with the same Requesed ID, the second request is ignored and the operation that corresponds to the first Batch which was created and stored in the backend is returned.

Valid Values:

0-40 characters. The letters a-z, A-Z, numbers 0-9, _, and -.

(For a Job task type) Defines input parameters to be passed on to job execution, in JSON format.

You retrieve this JSON content from the GCP Dataproc UI, using the EQUIVALENT REST option in job settings.

(Optional) Determines the number of seconds to wait before checking the job status.

Default: 20

Determines the authorization credentials that are used to connect Control-M to Hadoop, as described in Hadoop Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Variable Name: %%HDP-ACCOUNT

Determines the execution type for Hadoop job execution, as follows:

• DistCp Job

• Distributed Shell Job

• HDFS Commands Job

• HDFS File Watcher Job

• Impala Job

• Hive Job

• Java-Map-Reduce Job

• Oozie Job

• Pig Job

• Spark Job

• Sqoop Job

• Streaming Job

• Tajo Job

Variable Name: %%HDP-EXEC_TYPE

Defines the Pre commands performed before job execution (not for HDFS Commands jobs and Oozie Extractor jobs), and the argument for each command.

Determines whether the entire job fails if any of the Pre commands fail (not for HDFS Commands jobs and Oozie Extractor jobs).

Defines the Post commands performed before job execution (not for HDFS Commands jobs and Oozie Extractor jobs), and the argument for each command.

Defines the absolute destination path.

Variable Name: %%HDP-DISTCP_TARGET_PATH

Defines the source paths.

Variable Name: %%HDP-DISTCP_SOURCE_PATH-Nxxx_ARG

Defines the sets of attributes and values that are added to the command line.

Variable Names:

• Name: %%HDP-DISTCP_OPTION-Nxxx-NAME

• Value: %%HDP- DISTCP_OPTION-Nxxx-VAL

Determines what the Distributed Shell job executes, as follows:

• Command: Executes a shell command entry as defined by Command.

• Script File: Executes a script file as defined by Command, Script Full Path, and Shell Script Arguments.

Variable Name: %%HDP-SHELL_TYPE

Defines the shell command entry to execute for the job execution.

Variable Name: %%HDP-SHELL_COMMAND

Defines the pathname of the script file that is executed. The script file is located in the HDFS.

Variable Name: %%HDP-SHELL_SCRIPT_FULL_PATH

Defines the shell script arguments.

Variable Name: %%HDP-SHELL-Nxxx-ARG

Opens more attributes.

Defines the pathname of the file or archive that is uploaded as a dependency to the HDFS working directory.

Variable Names:

• Type: %%HDP-SHELL_FILE_DEP-Nxxx-TYPE

• Path: %%HDP-SHELL_FILE_DEP -Nxxx-PATH

Defines the additional option (Name and Value) to set when executing the job.

Variable Names:

• Name: %%HDP-SHELL_OPTION -Nxxx-NAME

• Value: %%HDP-SHELL_OPTION -Nxxx-VAL

Defines the environment variables for the shell script/command.

Variable Name: %%HDP-SHELL_ENV_VARIABLE-Nxxx-ARG

Defines the command for the argument to be performed with job execution.

Variable Name: %%HDP-HDFS_CMD_ACTION-Nxxx-CMD

Defines the pathname of the watched file.

Variable Name: %%HDP-HDFS_FILE_PATH

Determines the minimum file size in bytes to meet the criteria and finish the job as OK. If the file arrives, but the size is not met, the job continues to watch the file.

Variable Name: %%HDP-MIN_DETECTED_SIZE

Determines the maximum number of minutes to wait for the file to meet the watching criteria. If criteria are not met (file did not arrive, or minimum size was not reached) the job fails after this maximum number of minutes.

Variable Name: %%HDP-MAX_WAIT_TIME

Determines the source type to execute the queries, as follows:

• Query File: Executes a query file as defined by Query File Full Path.

• Open Query: Executes an open query command as defined by Query.

Variable Name: %%HDP-IMPALA_QUERY_SOURCE

Defines the location of the file used to execute the queries.

Variable Name: %%HDP-IMPALA_QUERY_FILE_PATH

Defines the query command used to execute the queries.

Variable Name: %%HDP-IMPALA_OPEN_QUERY

Defines the pathname of the Hive script on the Hadoop host.

Variable Name: %%HDP-HIVE_SCRIPT_NAME

Defines the list of parameters for the script.

Variable Names:

• Name: %%HDP-HIVE_SCRIPT_PARAM_Nxxx-NAME

• Value: %%HDP-HIVE_SCRIPT_PARAM-Nxxx-VAL

Defines the pathname of the jar file that contains the Map Reduce Java program on the Hadoop host.

Variable Name: %%HDP-JAVA_JAR_NAME

Defines the class that is included in the jar containing a main function and the map reduce implementation.

Variable Name: %%HDP-JAVA_MAIN_CLASS

Defines the argument used by the command.

Variable Name: %%HDP-JAVA_Nxxx_ARG

Defines the job properties pathname.

Variable Name: %%HDP-OOZIE_JOB_PROPERTIES_FILE

Defines the Oozie job properties.

A set of properties is comprised of the following:

• Key: Defines a key name associated with each property.

  Variable Name: %%HDP-OOZIE_PROPERTY-Nxxx-KEY

• Value: Defines a value associated with each property.

  Variable Name: %%HDP-OOZIE_PROPERTY-Nxxx-VAL

You can add new properties or override property values defined in the Job Properties File.

Defines the pathname of the Pig program on the Hadoop host.

Variable Name: %%HDP-PIG_PROG_NAME

Defines the list of program parameters.

Determines whether to add Yarn aggregated logs to the job output.

Defines a list of properties (Name and Value) to be executed with the job.

These properties override the Hadoop defaults.

Defines the location of the Hadoop archives.

Determines the Spark program type, as follows:

• Python Script: As defined by Full Path to Script.

• Java / Scala Application: As defined by Application Jar File and Full Path to Script.

Variable Name: %%HDP-SPARK_PROG_TYPE

Defines the pathname of the Python script that executes.

Variable Name: %%HDP-SPARK_FULL_PATH_TO_PYTHON_SCRIPT

Defines the path to the jar including your application and all the dependencies.

Variable Name: %%HDP-SPARK_APP_JAR_FULL_PATH

Defines the main class of the application.

Variable Name: %%HDP-SPARK_MAIN_CLASS_TO_RUN

Defines the attribute arguments that are added at the end of the Spark command line either after the main class for Java / Scala Applications or after the script of the Python Script.

Variable Name: %%HDP-SPARK_Nxxx_ARG

Defines the sets of attributes and values that are added to the command line.

Variable Names:

• Name: %%HDP-SPARK_OPTION -Nxxx-NAME

• Value: %%HDP-SPARK_OPTION -Nxxx-VAL

Defines any valid Sqoop command necessary for job execution. Sqoop can only be used for job execution if defined in Sqoop connection attributes.

HDP-SQOOP_COMMAND

Determines whether to add Yarn aggregated logs to the job output.

Defines a list of properties (Name and Value) to be executed with the job.

These properties override the Hadoop defaults.

Defines the location of the Hadoop archives.

Defines the input file for the Mapper step.

Variable Name: %%HDP-INPUT_PATH

Defines the HDFS output path for the Reducer step.

Variable Name: %%HDP-OUTPUT_PATH

Defines the command that executes as a mapper.

Variable Name: %%HDP-MAPPER_COMMAND

Defines the command that executes as a reducer.

Variable Name: %%HDP-REDUCER_COMMAND

Defines the sets of attributes (Name and Value) that are added to the end of the Streaming command line.

Variable Names:

• Name: %%HDP-STREAMING_PARAM-Nxxx-NAME

• Value: %%HDP-STREAMING_PARAM-Nxxx-VAL

Defines the sets of attributes (Name and Value) that are added to the Streaming command line.

Variable Names:

• Name: %%HDP-GENERIC_PARAM-Nxxx-NAME

• Value: %%HDP-GENERIC_PARAM-Nxxx-VAL

Determines the source of the Tajo command, as follows:

• Input File: Executes the Tajo command from an input file as defined by the Full File Path.

  Variable Name: %%HDP-TAJO_INPUT_FILE

• Open Query: Executes an open query as the Tajo command, as defined by Open Query.

  Variable Name: %%HDP-TAJO_OPEN_QUERY

Defines the pathname of the input file that executes the Tajo command.

Determines the authorization credentials that are used to connect Control-M to OCI Data Science Services, as described in OCI Data Flow Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Defines the name of a new Run.

Defines the compartment Oracle Cloud Identifier (OCID) which is a unique identifier assigned to each compartment that is created within the Oracle Data Flow Infrastructure.

Defines the application Oracle Cloud Identifier (OCID) which is a unique identifier assigned to each application that is created within the Oracle Data Flow Infrastructure.

(Optional) Determines whether to add more parameters to the new job run.

Valid Values:

• Checked

• Unchecked

Default: Unchecked

(Optional) Defines specific parameters, in JSON format, that are passed when you create a new Run.

For more information about the run parameters, see CreateRunDetails Reference 20200129 in the Oracle Cloud Infrastructure Documentation.

{
   "displayName": "<run_name>",
   "applicationId": "<application_ocid>",
   "compartmentId": "<compartment_ocid>",
   "driverShape": "VM.Standard.E4.Flex",
   "executorShape": "VM.Standard.E4.Flex",
   "numExecutors": 1,
   "arguments": [],
   "parameters": [],
   "configuration": {}
}

Determines the number of seconds to wait before checking the job status.

Default: 60

Determines one of the following types of authorization credentials, which are used to connect Control-M to Snowflake:

• Snowflake: Described in Snowflake Connection Profile Parameters.

• Snowflake IdP: Described in Snowflake IdP Connection Profile Parameters.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Blank spaces.

Determines the database that the job uses.

Determines the schema that the job uses.

A schema is an organizational model that describes the database layout, its table and field definitions, and their relationships to each other.

Determines one of the following Snowflake actions to perform:

• SQL Statement: Runs any number of Snowflake-supported SQL commands, such as queries, calling or creating procedures, database maintenance tasks, and creating and editing tables.

• Copy from Query: Copies a queried database and schema into an existing or new file in cloud storage.

• Copy from Table: Copies from an existing table.

• Create Table and Query: Creates a table, populated by a query, in the specified database and schema.

• Copy into Table: Copies data from a cloud storage location into an existing table in Snowflake.

• Start or Pause Snowpipe: Starts or pauses an existing Snowpipe.

• Stored Procedure: Calls an existing procedure and its arguments.

• Snowpipe Load Status: Monitors the status of a Snowpipe for a set period of time.

• Run SQL File: Uploads a file that contains Snowflake-supported SQL commands.

Determines one or more Snowflake-supported SQL commands.

Rule: Must be written in a single line, with strings separated by one space only.

Defines the pathname of the file that contains Snowflake-supported SQL commands.

Determines the maximum number of seconds to run the job in Snowflake.

Determines whether the following job-defining attributes are displayed:

• Parameters

• Role
• Bindings

• Warehouse

Defines Snowflake-provided parameters, in JSON format, that let you control how data is presented, as shown in the following example:

{ 
  "param1":"value1",
  "param2":"value2"
}

Determines the Snowflake role used for this Snowflake job.

A role is an entity that can be assigned privileges on secure objects. You can be assigned one or more roles from a limited selection.

Defines the values, in JSON format to bind to the variables used in the Snowflake job.

"1": 
    { 
      "type": "FIXED", 
      "value": "123" 
    } 
"2": 
    { 
      "type": "TEXT", 
      "value": "String" 
    }

For more information on bindings, see the Snowflake documentation.

Determines the warehouse used in the Snowflake job.

A warehouse is a cluster of virtual machines that processes a Snowflake job.

Determines whether to show a full JSON response in the log output.

Determines the number of seconds to wait before checking the job status.

Default: 20

Defines the cloud storage location.

Defines the query used to copy the data.

Defines the storage integration object, which stores an Identity and Access Management (IAM) entity and an optional set of blocked cloud storage locations.

Determines whether to overwrite an existing file in the cloud storage, as follows:

• Yes

• No

Determines the file format that the data is saved in, as follows:

• *.csv

• *.json

Defines where the JSON or CSV file is saved.

You can save to Amazon Web Services, Google Cloud Platform, or Microsoft Azure.

Defines the name of the copied table.

Defines the name of the new or existing table where the data is queried.

Defines the query used for the copied data.

Defines the name of the Snowpipe.

A Snowpipe loads data from files when they are ready or staged.

Defines the name of the table that the data is copied into.

Defines cloud storage location where the data is copied, in CSV or JSON format.

Determines whether to start or pause the Snowpipe, as follows:

• Start Snowpipe

• Pause Snowpipe

Defines the name of the stored procedure.

Defines the value of the argument in the stored procedure.

Defines the table that is monitored when loaded by the Snowpipe.

Defines the cloud storage location.

A stage is a pointer that indicates where data is stored or staged.

Determines the number of days to monitor the Snowpipe load status.

Defines the cloud storage location where a CSV file log is created.

The CSV file log details the load status for each Snowpipe.