Remove duplicate rows in a pipeline

This step looks for distinct values (duplicate rows) in selected columns, and removes them from the pipeline.

To remove duplicates:

1. Select Data Orchestrator from the top-left navigation menu.
2. Choose a dataspace from the list.
3. Select Pipelines from the left-side panel.
4. Create a new pipeline or edit an existing pipeline.
5. Select the add icon  that appears between the Source and Sink nodes in the designer view.
6. Select Remove duplicates from the right-side panel.
7. Enter a Label for the remove duplicates node.
8. Choose a column from the Select distinct columns dropdown.
   If no columns are selected, all columns are used to distinct rows.
9. Choose whether to keep the first or last occurrence of duplicate rows from the Keep first/last occurrence dropdown.

When you run the pipeline, the run removes the duplicate rows for the columns you selected.