You can add data validations to your pipeline. Data validations let you add rules that review the source dataset's data quality and integrity as it moves through the pipeline.
Rules summary
The validation node in the pipeline has 16 detailed rules that can be applied to your Data Orchestrator dataset and views by column. The options for applicable rules change based on the column data type.
This table lists a summary of the validation rules you can add to the columns.
| Rules summary | Description |
| Regex pattern matching | Define custom regex pattern strings to validate complex formats (for example ensuring emails and phone numbers). |
| Length validation | Ensure text fields contain exactly the expected number of characters, or fit within minimum or maximum limits. |
| Null checks | Instantly identify null, missing, or empty fields across specified columns. |
| Data type enforcement | Detect mismatching data types instantly (such as boolean values 1/0, TRUE/FALSE, or ensuring financial fields like Revenue are strictly numeric instead of strings). |
| Threshold and range checks | Automatically flag values falling outside of designated boundaries, such as ensuring numbers don't exceed a certain value or validating that dates don't fall before or after a specific time. |
For more information, see Introducing Data Validations in ADO Pipelines .
Add validations
You can add validations as you create a pipeline, or add validations to existing pipelines.
To add a validation:
- Select Data Orchestrator from the top-left navigation menu.
- Choose a dataspace from the list.
- Select Pipelines from the left-side panel.
- Create a new pipeline or edit an existing pipeline.
- Hover over the columns area between the Source and Sink, and then select the add icon that appears.
- Select Validate from the Add new step panel on the right.
- Enter a validation name in the Label field.
- Select Rules and complete these steps in the Add validation rules modal:
- Select the add icon for each column under the Validation section, and then from the right-side panel:
- Select a rule from the Rule dropdown menu.
The rules that display in the dropdown are based on the column type. - Optionally:
- Select Add to add additional rules to a column.
- Select the trash icon if you want to delete a rule.
- Select a rule from the Rule dropdown menu.
- Select from the On failure dropdown what happens if the validation fails:
- Remove failed row
- Stop pipeline
- Log warning
- Select Done.
- Select the add icon for each column under the Validation section, and then from the right-side panel:
- Select the Tolerance limit from the dropdown on the Validate panel:
- None: The pipeline run won't fail if there are validation failures.
- Percentage tolerance limit: Enter the Tolerance limit threshold (%) value in the field. This value is the percentage of validation failures that'll cause the pipeline run to fail.
- Absolute row count limit: Enter the Tolerance limit threshold (rows) value in the field. This value is the number of validation failures that'll cause the pipeline run to fail.
If you want to remove the validation, select Delete step at the bottom of the Validate panel on the right.
