For Anaplan Data Orchestrator (ADO) pipelines, you can define a row-based filter and apply it directly at the source. By filtering data at the source, only ‌rows that meet the set criteria are passed through the pipeline.

Row-based filters enable you to reduce the amount of data transferred into or from ADO. It also limits data usage against your quota and can improve pipeline performance.

Filters for source steps in pipelines

Row-based filters enable you to filter pipeline data at the source.

You can add row-based filters to pipelines created with these sources: 

  • Anaplan datasets
  • Java Database Connectivity (JDBC) sources:
    • Databricks
    • Google BigQuery
    • Microsoft SQL Server (MSSQL)
    • MySQL
    • Oracle
    • PostgreSQL
    • Snowflake
  • SAP
  • Salesforce

Add row-based filters

You can add row-based filters to the source when you create a new pipeline, or you can add it to an existing pipeline.

Add row-based filters to a new pipeline

When you create a new pipeline, you can add row-based filters after you select a source location for your pipeline.

To add row-based filters to a new pipeline:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Pipelines from the left-side panel.
  4. Select Create pipeline.
  5. Enter a Name for your pipeline and then select Create.
    You are taken to the pipeline designer view.
  6. Select the Source icon.
  7. Select a Connection type from the dropdown based on the pipeline type:
    • For a read pipeline, choose an external source that supports row filtering (such as MSSQL).
    • For a writeback pipeline, choose Anaplan.
  8. Select a connection from the Choose connection dropdown based on the pipeline:
    • For a read pipeline, choose an external source connection. 
    • For a writeback pipeline, select Datasets.
  9. Select Source location > Source, choose a data source, and then select Confirm.
  10. Select Filter at source, and configure the filter:
    1. Select a column from the source, select a condition, and enter a comparison value.
    2. Select the add icon to add additional filters, or select the delete icon to delete filters.
    3. Choose your preferred method to filter the source data when multiple filters are applied:
      • Match all filters: Only source data that meets all the criteria is passed through the pipeline.
      • Match any filters: Source data that meets any of the criteria is passed through the pipeline.
    4. Select Confirm.
  11. Configure the Sink details.
    For more information, see the read or writeback pipeline page for your external source system.
  12. Select Publish, and then select Run to execute the pipeline. 

Add row-based filters to an existing pipeline

To set up row-based filters for an existing pipeline:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Pipelines from the left-side panel.
  4. Open an existing pipeline.
  5. Select the Source icon.
  6. Select Filter at source, and configure the filter:
    1. Select a column from the source, select a condition, and enter a comparison value.
    2. Select the add icon to add additional filters, or select the delete icon to delete filters.
  7. Choose your preferred method to filter the source data when more than one filter is set up:
    • Match all filters: Only source data that meets all the criteria is passed through the pipeline.
    • Match any filters: Source data that meets any of the criteria is passed through the pipeline.
  8. Configure the Sink details.
    For more information, see the read or writeback pipeline page for your external source system.
  9. Select Publish, and then select Run to execute the pipeline.