You can add additional steps to your pipeline to process the data as it runs through the pipeline.

When you create or edit a pipeline, you can add these steps:

Pipeline stepDescription
ValidateEnables you to add rules that review the dataset's data quality and integrity as it moves through the pipeline.
Remove duplicatesRemoves duplicate rows in the dataset.

To add additional steps to a pipeline, select the add icon between the Source and Sink.

When you run your pipeline, it runs the selected dataset through the additional step you added to the pipeline.