Data Assurance

Control-M Data Assurance is an add-on that enables you to monitor and verify the quality and accuracy of your data across multiple data sources, before, during, and after they are extracted, transformed, and loaded through Control-M workflows or plug-ins.

Data Assurance jobs scan data sources for missing, inconsistent, or faulty data according to one or more validation rules. You can define up to 50 individual rules per job, adjust the scope of the data scanned, and determine how the data is measured and checked for accuracy. If one or more validation rules are unmet, which indicates that data is inadequate or incorrect, the Data Assurance job alerts the user and ends Not OK.

The Data Assurance Dashboard enables you to monitor, at a glance, how all Data Assurance jobs have performed and investigate specific current and past job executions in detail, as described in Data Assurance Dashboard.

Data Assurance consists of the following entities:

• Data Source: A database or directory of CSV or text files. Data Assurance supports PostgreSQL, Snowflake, MSSQL, Oracle, and MySQL databases.

• Dataset: A database table or filtered directory of CSV or text files. Each data source contains one or more datasets.

• Validation Scope: A user-defined range of values in the dataset that the Data Assurance job scans, such as the Customer_Full_Name column in the Customers dataset of an MSSQL database.

• Validation Rule: A user-defined rule that examines a specific data attribute in the dataset for completeness or accuracy. Validation rules are based on the user-defined scope of the dataset that the Data Assurance job scans, the metric that you apply to that data, and the user-defined threshold value the metric measurement must meet to validate the data. If the rule is unmet, the data is considered faulty. For more information, see Creating Validation Rules.

• Metric: A predefined standard of measurement, such as minimum file size, standard deviation, or average, that a Data Assurance rule employs to measure the data in the validation scope. Data Assurance provides a list of predefined metrics that are specific to the type and scope of data in the required dataset, as described in Metric Attributes.

  You can also define your own metrics for text file data sources and integrate them into Data Assurance, as described in User-Defined Metrics.

• Alert: A notification that appears in the Dashboard when a validation rule is unmet and a Data Assurance job ends Not OK. This indicates that one or more inconsistencies or missing values are present in your data.

After you define your Data Assurance jobs with one or more validation rules and corresponding alert levels, you can build these jobs into existing workflows that must have complete and correct data to perform as required. If a predecessor Data Assurance job finds no faulty data, indicating that all rules are met, no alert is sent, the job ends OK, and successor jobs in the workflow can execute with quality-assured data. If the Data Assurance job ends Not OK, an alert is issued and successor jobs do not execute, which enables you to prevent the creation of incorrect results from bad data.

The following diagram shows the logical architecture of a typical Data Assurance add-on:

Setting Up Data Assurance

This procedure describes how to set up and get started with Data Assurance.

Begin

1. Add Data Sources: Create a centralized connection profile for one or more data sources with the Data Processing and Analytics Connection Profiles.

2. Create Data Assurance Jobs: Create one or more Data Assurance jobs, as described in Creating a Job, with Data Assurance Job attributes.

3. Create Validation Rules: Define one or more validation rules, as described in Creating Validation Rules.

4. Arrange, Run, and Monitor Jobs: Integrate your Data Assurance jobs into one or more workflows, run the required jobs, and monitor the status of your Data Assurance jobs via the Data Assurance Dashboard, as described in Data Assurance Dashboard.

Creating Validation Rules

This procedure describes how to create one or more validation rules, which enable you to validate specific data attributes (via metrics and threshold values) across an entire dataset, a single column in the dataset, or a selection of values that you define in a custom SQL query.

Before You Begin

• Verify that you have added at least one data source and created at least one Data Assurance job, as described in Setting Up Data Assurance.

Begin

1. Click a Data Assurance job.

   The Data Assurance job properties appear in the right pane.

2. Do one of the following to determine which files or database table rows (records) in the datasets are scanned:

   • File Directory: From the Files to Validate section in the General tab of the job properties pane, define the file selection attributes, as described in File Selection Attributes.

   • Database: From the Records to Validate section in the General tab of the job properties pane, define the record selection attributes, as described in Record Selection Attributes.

     This option is only available if you select a dataset that contains a column with a date-, time-, or date-and-time-based data type when you define the Dataset job attribute.

3. Do one of the following, according to the data source type:

   • Database: Under the Records to Validate section in the General tab of the job properties pane, click Add Rule.

   • Text File Directory: Under the Files to Validate section in the General tab of the job properties pane, click Add Rule.

   • CSV File Directory: Do the following:

     1. Define the CSV file schema.

        The Records to Validate section appears in the General tab of the job properties pane.

     2. From the Records to Validate section, define the record selection attributes, as described in Record Selection Attributes.

     3. Under the Files to Validate section in the General tab of the job properties pane, click Add Rule.

   The Choose Rule Scope dialog box appears, which enables you to define the scope of the validation rule.

4. Select the required validation scope, which is the range of values in the dataset that the validation rule scans, as follows:

   • Entire Dataset: Scans the CSV files or database records in the dataset, according to the file or record selection attributes that you defined.

   • One Dataset Column: Scans one column in the CSV files or database table dataset, according to the file or record selection attributes that you defined. You determine the column that is scanned when you define the validation rule.

     The CSV file schema enables Data Assurance to scan all the values from a single, user-defined column in every CSV file that matches the file selection attributes.

   • SQL Query: Scans the entire dataset or directory of CSV files in an SQL query that you define.

   • Directory: Scans the text file directory.

   • File: Scans the text files inside the text file directory.

   The Validation Rules dialog box appears.

5. Do the following to define the validation rule:

   1. Define the following validation rule attributes, based on the validation scope:

      • Non-SQL Query Scope:

        • Metric:  A predefined standard of measurement, such as minimum length, standard deviation, or average that is specific to the data source type and validation scope, as described in Metric Attributes.

        • Data Column: The name of the column that is validated, when the scope is a single column in the dataset.

        • Operator: A mathematical function that creates a relationship between the metric and the threshold, such as equal, be greater than, or be between.

          BMC recommends that you use a range operator, such as greater than or equal to, instead of equal to with metrics that return decimal numbers.

        • Threshold: The result that must be met for the validation rule to pass and signify that no inconsistency or inaccuracy is found.

        The validation rule appears in the following format:

        <Metric> ... must <Operator> <Threshold>

        • File count must be greater than or equal to 100

        • Address must have a Missing Values Percent that is equal to 0

      • SQL Query Scope:

        • Metric description: A user-defined description of the metric that you build in the SQL query box.

        • SQL Query Box: A user-defined query, written in SQL query syntax.

          (CSV file directories only) You must define the table name as CSV.

          • The following SQL query enables you to scan the entire Uniform_Coffee dataset in a database data source:

            Copy

            SELECT COUNT(*) AS False_Count
            FROM Uniform_Coffee
            WHERE Current_Subscriber = 'FALSE'

          • The following SQL query enables you to scan all CSV files in a CSV file directory data source:

            Copy

            SELECT COUNT(*) AS False_Count
            FROM CSV
            WHERE Current_Subscriber = 'FALSE'

   2. Determine the alert type and job completion behavior when the validation rule is not met, as follows:

      • Critical: A Critical alert appears on the Data Assurance Dashboard, but the job ends Not OK.

      • Warning: A Warning alert appears on the Data Assurance Dashboard and the job ends OK.

   3. (Optional) Click to test the validation rule.

      A test result message appears.

      

   4. (Optional) Click Add Rule to define another validation rule for this job.

The Validation Rules (<Number_of_Rules_Defined>) section appears. You can click Edit Rules to define up to 50 validation rules.

Tango Coffee, a coffee roaster with a subscription service, creates the following validation rule to verify that no addresses are missing from their customer information database:

• Validation Scope: One Dataset Column

• Data Column: Address

• Metric: Missing Values Percent

• Operator: Equals

• Threshold: 0

• Alert: Critical

Before they ship out their monthly subscriptions, Tango Coffee runs a Data Assurance job with this rule. The job ends Not OK and sends a Critical alert to the Data Assurance Dashboard that 3% of the addresses are missing. This enables the roaster to contact the customers with the missing addresses, add the missing information to the database, and then forward the complete monthly shipping list to their courier service so they can fulfill all customer subscriptions on time.

Defining a CSV File Schema

This procedure describes how to define a CSV file schema, which enables a Data Assurance job that is connected to a CSV file directory data source to read the structure of the files in your CSV file directory. You can either define the schema manually or upload a CSV file with a schema that matches the CSV files in your file directory.

Begin

1. From the Files to Validate section in the General tab of the job properties pane, click Define Schema.

   The Schema (CSV File Format) dialog box appears.

2. Define the following attributes to configure how the Data Assurance job reads the CSV files in the file directory:

   • Tables have headers: Toggle on if the tables in the CSV files contain headers.

   • Delimiter (Column Separator): From the drop-down list, select the delimiter that is used in the CSV files.

   • If the file does not match the schema: Click one of the following options:

     • Skip and proceed to the next one

     • Stop Data Assurance job and set to Ended Not OK

     If all of the CSV files in your CSV file directory do not conform to the same schema, you must create a separate Data Assurance job and define a schema that matches those CSV files to successfully scan them.

3. Do one of the following:

   • Upload a CSV File: Do the following:

     1. In the Define Schema field, click Upload CSV.

        A file browser window opens.

     2. Navigate to the relevant directory and select a CSV file with a schema that matches the CSV files in your directory.

        You can select and upload any of the CSV files from your CSV file directory data source, or a CSV file with a matching schema from a different location.

        Data Assurance extracts the schema from the uploaded file, and the schema appears in the Schema Definition section of the Schema (CSV File Format) dialog box.

     3. Review the schema and then add, edit, or delete the schema fields as required, as described in Schema Attributes.

     4. Click Apply.

   • Manually Define the Schema: Do the following:

     1. In the Define Schema field, click Define Manually.

        The schema appears in the Schema Definition section of the Schema (CSV File Format) dialog box.

     2. Add the schema fields as required, as described in Schema Attributes.

     3. Click Apply.

   The schema is saved to the Data Assurance job.

Schema Attributes

The following table describes the schema attributes that you can define in the Schema (CSV File Format) dialog box, as described in Defining a CSV File Schema.

Metric Attributes

The following table describes the attributes for each Data-Assurance-provided, predefined metric that you can define in a validation rule. You can click the Validation Alerts tab in the bottom pane of the Data Assurance Dashboard to monitor and compare the calculated metric values with the desired threshold values defined in your validation rules.

Regex Symbols

The following table lists common regex (regular expression) symbols that you can include in metrics that search for user-defined regex values.

File Selection Attributes

The following table describes the file selection attributes that enable you to filter your CSV and text file directory scans for specific filename patterns and modification dates.

Record Selection Attributes

The following table describes the record (database or CSV table row) selection attributes that enable you to filter your dataset scans for records that were modified within a specific timeframe.

User-Defined Metrics

You can define your own metrics for text file directory data sources and integrate them into Data Assurance. These enable you to measure your data in ways that the predefined, Data Assurance-provided metrics do not.

The following procedures enable you to create and manage your user-defined metrics:

• Creating a User-Defined Metric

• Listing Metrics

• Updating a User-Defined Metric

• Deleting a User-Defined Metric

Creating a User-Defined Metric

This procedure describes how to define your own metric for text file data sources and add it to Data Assurance. This enables you to measure your data in ways that the predefined, Data Assurance-provided metrics do not.

Begin

1. Log in to the Agent where the Data Assurance plug-in is installed.

2. Do the following: 

   1. Write a new Data Assurance metric in Python, according to the Custom Metric Template instructions and examples in the _metric_template.py file that is located in the following directory on the Agent where the Data Assurance plug-in is installed:

      • UNIX: <Agent_Home>/ctm/cm/WDA/files_metrics/custom/

      • Windows: <Agent_Home>\ctm\cm\WDA\files_metrics\custom\

   2. Save the new metric Python file to the same directory where the _metric_template.py file is located.

3. From the command line, navigate to the following directory:

   • UNIX: <Agent_Home>/ctm/cm/WDA/files_metrics/

   • Windows: <Agent_Home>\ctm\cm\WDA\files_metrics\

4. Run the following command:

   • UNIX: ./manage_metrics.sh add

   • Windows: manage_metrics.cmd add

   The script runs and prompts you to define the parameters for the new metric, as described in User-Defined Metric Parameters.

5. Follow the prompts.

   The script validates the user-defined metric and adds it to Data Assurance.

Listing Metrics

This procedure describes how to list Data Assurance metrics. This enables you to distinguish between the user-defined and predefined, Data Assurance-provided metrics that are currently available in Data Assurance.

Begin

1. Log in to the Agent where the Data Assurance plug-in is installed.

2. From the command line, navigate to the following directory:

   • UNIX: <Agent_Home>/ctm/cm/WDA/files_metrics/

   • Windows: <Agent_Home>\ctm\cm\WDA\files_metrics\

3. Run the following command:

   • UNIX: ./manage_metrics.sh list [all | custom | builtin]

   • Windows: manage_metrics.cmd list [all | custom | builtin]

   Where:

   • all: Lists all available Data Assurance metrics.

   • custom: Lists all user-defined Data Assurance metrics.

   • builtin: Lists all predefined, Data Assurance-provided metrics.

Updating a User-Defined Metric

This procedure describes how to update a user-defined metric in Data Assurance, which eliminates the need to delete the current metric and add an updated version.

Begin

1. Log in to the Agent where the Data Assurance plug-in is installed.

2. Do the following: 

   1. Make a copy of the existing user-defined Data Assurance metric Python file, which is located in the following directory on the Data Assurance Plug-in Agent:

      • UNIX: <Agent_Home>/ctm/cm/WDA/files_metrics/custom/

      • Windows: <Agent_Home>\ctm\cm\WDA\files_metrics\custom\

   2. Update the copied Data Assurance metric in Python, as required and according to the Custom Metric Template instructions and examples in the _metric_template.py file.

   3. Save the updated metric Python file to the same directory where the _metric_template.py file is located.

3. From the command line, navigate to the following directory:

   • UNIX: <Agent_Home>/ctm/cm/WDA/files_metrics/

   • Windows: <Agent_Home>\ctm\cm\WDA\files_metrics\

4. List the current metrics to determine and record the required user-defined metric ID.

5. Run the following command:

   • UNIX: ./manage_metrics.sh update <Metric_ID>

   • Windows: manage_metrics.cmd update <Metric_ID>

   ./manage_metrics.sh update NumericLinesCount

   The script runs and prompts you to define the parameters for the updated metric, as described in User-Defined Metric Parameters.

6. Follow the prompts.

   The script validates the user-defined metric and updates it in Data Assurance.

Deleting a User-Defined Metric

This procedure describes how to delete a user-defined metric that was previously added to Data Assurance. If a user-defined metric is no longer required, this enables you to remove it from the list of possible metrics that users can select when they define a validation rule.

Begin

1. Log in to the Agent where the Data Assurance plug-in is installed.

2. From the command line, navigate to the following directory:

   • UNIX: <Agent_Home>/ctm/cm/WDA/files_metrics/

   • Windows: <Agent_Home>\ctm\cm\WDA\files_metrics\

3. Run the following command:

   • UNIX: ./manage_metrics.sh remove

   • Windows: manage_metrics.cmd remove

   The script runs and prompts you to define the required metric ID, as described in User-Defined Metric Parameters.

4. Type the metric ID name.

   A confirmation message appears.

5. Press y to confirm the user-defined metric deletion.

   The specified user-defined metric is removed from Data Assurance.

User-Defined Metric Parameters

The following table describes the user-defined metric parameters that you must define when you create a new user-defined metric and add it to Data Assurance, as described in Creating a User-Defined Metric.

Data Assurance Dashboard

The Data Assurance Dashboard tab, which you can access from the drop-down list in the Monitoring domain, provides a high-level numeric and graphical analysis of all Data Assurance datasets and job validation statuses, alerts, and validation results, which enables you to identify data quality trends and issues as they occur. You can filter results by data source and job run date from the toolbar, and you can view all the datasets with unmet rules or browse a table of validation alerts in the bottom pane. The right Alert Details pane enables you to view the summary, output, and validation history of individual Data Assurance jobs that have triggered alerts, and you can click the job name in the Summary pane to open the job in a viewpoint to retrieve further job and neighbor information.

Configuring Data Assurance

This procedure describes how to configure Data Assurance in the dataAssurance.properties file, which enables you to change the default plug-in behavior when required.

Begin

1. Log in to the Agent where the Data Assurance Plug-in is installed.

2. Navigate to the following directory:

   • UNIX: <Agent_Home>/ctm/cm/WDA/data/

   • Windows: <Agent_Home>\ctm\cm\WDA\data\

3. Open the dataAssurance.properties file in a text editor.

4. Edit the required parameters, as described in Data Assurance Properties File Parameters.

5. Save and close the file.

6. (Publisher.enable, interval.seconds, and maxEvents parameters only) Run the following command to recycle the Agent:

   • UNIX: <Agent_Home>/cm/WDA/exe/cm_container restart

   • Windows: <Agent_Home>/cm\WDA\exe\cm_container.cmd restart

Data Assurance Properties File Parameters

The following table describes some of the dataAssurance.properties file parameters.

[{"metric_id":"LinesContainingString","value":"","metric_scope":"File","dataset_name":"TEXT_DEMO","file_pattern":"*","total_matching_files":5,"files_evaluated":5,"files_passed":4,"files_failed":1,"files_skipped":0,"failure_details":"The following 1 files failed the validation rule (observed value in parentheses):\n/home/smartool/demo_text_files/paycheck_danial.txt (0)","threshold":"1","operator":"Equals","condition":"LinesContainingString Equals 1 (per-file: 5 evaluated, 4 passed, 1 failed)","metric":"LinesContainingString","metric_parameter":"[{\"MetricParameterName\":\"search_string\",\"MetricParameterValue\":\"Employee ID\"}]","severity":"Critical","return_type":"Numeric","status":"FAILED","timestamp":"2026-09-15T07:00:41.834700922"}]