Machine Learning Jobs

The following topics describe job types for machine learning platforms and services:

Job:AWS Sagemaker

AWS Sagemaker enables you to create, train, and deploy machine learning models on premises, in the cloud, and on edge devices.

To deploy and run an AWS Sagemaker job, ensure that you have installed the AWS Sagemaker plug-in with the provision image command or the provision agent::update command.

The following example shows how to define an AWS Sagemaker job:

Copy
"AWS Sagemaker_Job"
{
   "Type": "Job:AWS Sagemaker",
   "ConnectionProfile": "AWSSAGEMAKER",
   "Pipeline Name": "SageMaker_Pipeline",
   "Idempotency Token": "Token_Control-M_for_SageMaker%%ORDERID",
   "Add Parameters": "checked",
   "Parameters": "{\"Name\":\"string1\", \"value\":\"string2\"}",
   "Retry Pipeline Execution": "checked",
   "Pipeline Execution ARN": "arn:aws:sagemaker:us-east-1:122343283363:pipeline/test-123-p-ixxyfil39d9o/execution/4tl5r9q0ywpw",
   "Status Polling Frequency": "30",
   "Failure Tolerance": "2"
}

The following table describes the AWS Sagemaker job parameters.

Parameter

Description

ConnectionProfile

Defines the ConnectionProfile:AWS Sagemaker name that connects Control-M to AWS SageMaker.

Pipeline Name

Determines the name of the preexisting AWS SageMaker pipeline used in this job.

Idempotency Token

(Optional) Defines a unique ID (idempotency token) that guarantees that the job is executed only once. After successful execution, this ID cannot be used again.

To allow a re-execution of the job with a new token, replace the default value with a unique ID that has not been used before. Use the RUN_ID, which can be retrieved from the job output.

Default: Token_Control-M_for_SageMaker%%ORDERID

Add Parameters

Determines whether to add or change default parameters in the execution of the pipeline.

Values:

  • checked

  • unchecked

Default: unchecked

Parameters

Defines the parameters to add or change, according to the AWS SageMaker convention, in JSON format. The list of parameters must begin with the name of the parameter type.

Retry Pipeline Execution

Determines whether to retry the execution of a pipeline, which you might want to do if a previous execution fails or stops.

Values:

  • checked

  • unchecked

Default: unchecked

Pipeline Execution ARN

Defines the Amazon Resource Name (ARN) of the pipeline, which is required to retry the execution of the pipeline.

An ARN is a standardized AWS resource address.

Status Polling Frequency

(Optional) Determines the number of seconds to wait before checking the status of the SageMaker job.

Default: 30

Failure Tolerance

Determines the number of times to check the job status before ending Not OK.

Default: 2

Job:Azure Machine Learning

Azure Machine Learning enables you to build, train, deploy, and manage machine learning models on premises, in the cloud, and on edge devices.

To deploy and run an Azure Machine Learning job, ensure that you have installed the Azure Machine Learning plug-in with the provision image command or the provision agent::update command.

The following examples show how to define a Azure Machine Learning job.

  • This JSON defines a regular Azure Machine Learning job:

    Copy
    "Azure Machine Learning_Job"
    {
       "Type": "Job:Azure Machine Learning",
       "ConnectionProfile": "AZURE_ML",
       "Workspace Name": "testML2",
       "Resource Group Name": "My_Resource_Group",
       "Action": "Compute Management",
       "Compute Name": "Compute_Name",
       "Compute Action": "Stop",
       "Status polling interval": "20",
       "Failure Tolerance": "2"
    }
  • This JSON defines an Azure Machine Learning job for triggering an endpoint pipeline:

    Copy
    "Azure Machine Learning_Job_2"
    {
       "Type": "Job:Azure Machine Learning",
       "ConnectionProfile": "AZURE_ML",
       "Workspace Name": "testML2",
       "Resource Group Name": "My_Resource_Group",
       "Action": "Trigger Endpoint Pipeline",
       "Pipeline Endpoint Id": "353c4707-fd23-40f6-91e2-83bf7cba764c",
       "Parameters": "{"ExperimentName":"test", "DisplayName":"test1123"}",
       "Status polling interval": "20",
       "Failure Tolerance": "2"
    }

The following table describes the Azure Machine Learning job parameters.

Parameter

Description

ConnectionProfile

Defines the ConnectionProfile:Azure Machine Learning name that connects Control-M to Azure Machine Learning.

Workspace Name

Determines the name of the Azure Machine Learning workspace for the job.

Resource Group Name

Determines the Azure resource group that is associated with a specific Azure Machine Learning workspace.

A resource group is a container that holds related resources for an Azure solution. The resource group can include all the resources for the solution, or only those resources that you want to manage as a group.

Action

Determines one of the following Azure Machine Learning actions to perform:

  • Trigger Endpoint Pipeline: Starts a pipeline.

  • Compute Management: Stops, starts, restarts, or deletes a computer or a cluster.

Pipeline Endpoint Id

Determines the pipeline endpoint ID, which points to a published pipeline in Azure Machine Learning.

Parameters

Defines additional parameters for the pipeline, in JSON format.

For no parameters, specify {}.

Compute Name

Defines the name of the compute function.

Compute Action

Determines one of the following compute actions to perform:

  • Start: Starts a computer or a cluster.

  • Stop: Stops a computer or a cluster.

  • Restart: Restarts a computer or a cluster.

  • Delete: Deletes a computer or a cluster.

Status Polling Interval

Determines the number of seconds to wait before checking the status of the job.

Default: 15

Failure Tolerance

Determines the number of times to check the job status before ending Not OK.

Default: 2

Job:OCI Data Science

OCI Data Science is an Oracle Cloud Infrastructure (OCI) platform, that enables you to build, train, deploy, and manage machine learning (ML) models using Python and open source tools.

To deploy and run an OCI Data Science job, ensure that you have installed the OCI Data Science plug-in with the provision image command or the provision agent::update command.

The following examples show how to define an OCI Data Science job.

  • This JSON defines an OCI Data Science job for the Start Job Run action:

    Copy
    "OCI Data Science_Job"
    {
       "Type": "Job:OCI Data Science",
       "ConnectionProfile": "OCI_DATA_SCIENCE",
       "Action": "Start Job Run",
       "Parameters": "   
       {
          "projectId":"ocid1.datascienceproject.oc1.phx.amaaaaaatdg3y3qa3spw62ticili6ovduagkr6es4j5txq2ehjqq12341234",
          "compartmentId":"ocid1.compartment.oc1..aaaaaaaahjo7g63l5dhmgepb7xfszhpgikuby4rdybd4wywxuz5a23451234",
          "jobId":"ocid1.datasciencejob.oc1.phx.amaaaaaatdg3y3qaax2vzve2s4xpskeildkq36kfshixr65muj123423412",
          "definedTags":{},
          "displayName":"test","freeformTags":{},
          "jobConfigurationOverrideDetails":{"jobType":"DEFAULT"},
          "jobLogConfigurationOverrideDetails":
          {
             "enableAutoLogCreation":true,
             "enableLogging":true,
             "logGroupId":"ocid1.loggroup.oc1.phx.amaaaaaatdg3y3qacy5u3klrnav737vitndpltgdmabrps3sbxl3rw1234123"
          }
       "Status Polling Frequency": "5",
       "Failure Tolerance": "1"
       }
    }
  • This JSON defines an OCI Data Science job for the Delete Notebook Session action:

    Copy
    "OCI Data Science_Job"
    {
       "Type": "Job:OCI Data Science",
       "ConnectionProfile": "OCI_DATA_SCIENCE",
       "Action": "Delete Notebook Session",
       "Status Polling Frequency": "5",
       "Failure Tolerance": "1",
       "Notebook Session ID": "ocid1.datasciencenotebooksession.oc1.phx.amaaaaaa427sdfsd77sdfgyeelvnwzvqi7l57bhkhgxfpiasd341qfd4gze1341234"
    }

The following table describes the OCI Data Science job parameters.

Attribute

Description

ConnectionProfile

Determines the authorization credentials that are used to connect Control-M to OCI Data Science Services, as described in OCI Data Science Connection Profile Parameters.

Rules:

  • Characters: 1−30

  • Case Sensitive: Yes

  • Invalid Characters: Blank spaces.

Variable Name: %%INF-ACCOUNT

Action

Determines one of the following OCI Data Science actions:

  • Start Job Run: Executes the OCI Data Science job.

  • Start Pipeline Run: Activates the OCI Data Science pipeline.

  • Create Model Deployment: Creates a Model Deployment.

  • Create Notebook Session: Creates a Notebook Session.

  • Delete Model Deployment: Deletes a Model Deployment.

  • Delete Notebook Session: Deletes a Notebook Session.

Parameters

Defines the parameters for the following actions, in JSON format, for example:

  • Start Job Run

    Copy
    {
       "projectId":"ocid1.datascienceproject.oc1.phx.amaaaghfyu65876kjh345j35uagkr6es4j5txq2ehjqq4ioyqygq",
       "compartmentId":"ocid1.compartment.oc1..aaaaaaaahjo7g63l5dhmgepb7xjk34kj5k3j5dybd4wywxuz5aziyqpkvq",
       "jobId":"ocid1.datasciencejob.oc1.phx.amaaaaaatdg3y3qakj345kj34ed6v5kfuovupwaqeao3js2mcmcrk673w3fysq",
       "definedTags":{},
       "displayName":"test234",
       "freeformTags":{},
       "jobConfigurationOverrideDetails":{"jobType":"DEFAULT"},
       "jobLogConfigurationOverrideDetails":
       {
         "enableAutoLogCreation":false,
         "enableLogging":false
       }
    }
  • Start Pipeline Run

    Copy
    {
       "pipelineId": "ocid1.datasciencepipeline.oc1.phx.amaaaaaatdg3y3qaqevmdwhe7kjhk345j345h6xiinyyj6rdkvj2l2lkzdvmwoq",
       "compartmentId": "ocid1.compartment.oc1..aaaaaaaahjo7g63l5dhmgepb7ljh3453uby4rdybd4wywxuz5aziyqpkvq"
    }
  • Create Model Deployment

    Copy
    {
       "displayName": "Control_M",
       "projectId": "ocid1.datascienceproject.oc1.phx.amaaaahi34h5kj43h562ticili6ovduagkr6es4j5txq2ehjqq4ioyqygq",    
       "compartmentId": "ocid1.compartment.oc1..aaaaaaaahjo7g63l5dhmgepkjl5kj34lk5j34ikuby4rdybd4wywxuz5aziyqpkvq",
       "modelDeploymentConfigurationDetails"
       {
          "deploymentType": "SINGLE_MODEL",
          "modelConfigurationDetails"
          {
             "modelId": "ocid1.datasciencemodel.oc1.phx.amaaaaaatdg3y3qabio6hdklkj34l534kyfbllazy7n462yqidp3tjwqc7ga",
             "instanceConfiguration"
             {
                "instanceShapeName": "VM.Standard.E4.Flex",
                "modelDeploymentInstanceShapeConfigDetails":
                {
                   "ocpus": 1.0,
                   "memoryInGBs": 16.0,
                   "cpuBaseline": null
                },
                "subnetId": null
             },            
          "scalingPolicy"
          {                
             "policyType": "FIXED_SIZE",
             "instanceCount": 1
          },
          "bandwidthMbps": 10,
          "maximumBandwidthMbps": null
          }
       }
    }
  • Create Notebook Session

    Copy
    {
       "displayName": "Demo",
       "projectId": "ocid1.datascienceproject.oc1.phx.amaaaaaathfgdh6rdejfghjpw62ticili6ovduagkr6es4j5txq2ehjqq4ioyqygq",
       "compartmentId": "ocid1.compartment.oc1..aaaaaaaahjo7g63l5dhmgepb7xfszhpgik6456fhfghywxuz5aziyqpkvq",
       "notebookSessionConfigDetails":
       {
          "shape": "VM.Standard.E4.Flex",
          "blockStorageSizeInGBs": 100,
          "subnetId": null,
          "privateEndpointId": null,
          "notebookSessionShapeConfigDetails"
          {
             "ocpus": 4.0,
             "memoryInGBs": 64.0
          }
       }
    }

    For more information about the action parameters, see Oracle API documentation.

Model Deployment ID

(Delete Model Deployment) Determines the OCID of the model deployment that is deleted.

Notebook Session ID

(Delete Notebook Session) Determines the OCID of the notebook session that is deleted.

Status Polling Frequency

Determines the number of seconds to wait before checking the job status.

Default: 60

Failure Tolerance

Determines the number of times to check the job status before ending Not OK.

Default: 2