You can use Anaplan datasets as a source or sink in a pipeline. You can also use Anaplan datasets as both the source and sink within the same pipeline. 

When used as a source, data is exported out of the dataset. When used as a sink, data is imported into the dataset.

You can use source datasets or transformation view datasets (derived datasets). You don't need to create a connection to an Anaplan dataset to use them as a source or sink in the pipeline.

Anaplan datasets as a pipeline source

This section describes how to create a pipeline with an Anaplan dataset as the data source.

To create a pipeline with an Anaplan dataset as a source:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Pipelines from the left-side panel.
  4. Select Create pipeline.
  5. Enter a Name for your pipeline and then select Create.
    You are taken to the pipeline designer view.
  6. Select the Source icon, and then complete these fields in the right-side panel:
    1. Select Anaplan from the Connection type dropdown.
    2. Select Datasets from the Choose connection dropdown.
    3. Enter a new Label to change the name that displays in the pipeline source node.
    4. Select Source location > Source, and choose an ADO dataset.
    5. Select Columns, and choose which columns you want to include.
      You can also optionally choose a cursor column.
    6. Optionally, select Filter at source to add row-based filters.
      For more information, see Row-based filtering at the source for pipelines.
  7. Select the Sink icon, and then complete these steps in the right-side panel:
    1. Select a Connection type from the dropdown.
      You can select an external system (for example S3) or Anaplan.
    2. Select a connection from the Choose Connection dropdown:
      • External system: If you selected an external system as the connection type, select an existing connection.
      • Model: If you selected Anaplan as the connection type and want to use a model, select Model.
      • Dataset: If you selected Anaplan as the connection type and want to use a dataset, select Datasets.
    3. Enter a new Label to change the name that displays in the pipeline sink node.
    4. Complete the remaining fields that display in the Targets panel, based on the connection you selected:
      • External system: See the relevant writeback pipeline page for your external system for more information.
      • Anaplan model: See Anaplan model connections in pipelines for more information.
      • Anaplan dataset: If you choose this, then Anaplan datasets are both the source and sink in this pipeline. For information on configuring the sink, see Step 7 in the Anaplan datasets as a pipeline sink section.
  8. Select Publish, and then select Run to execute the data transfer.

Anaplan datasets as a pipeline sink

This section describes how to create a pipeline with an Anaplan dataset as the sink destination.

To create a pipeline with an Anaplan dataset sink:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Pipelines from the left-side panel.
  4. Select Create pipeline.
  5. Enter a Name for your pipeline and then select Create.
    You are taken to the pipeline designer view.
  6. Select the Source icon, and then complete these fields in the right-side panel:
    1. Select a Connection type from the dropdown.
      You can select an external system (for example S3) or select Anaplan.
    2. Select a connection from the Choose Connection dropdown:
      • External system: If you selected an external system as the connection type, select an existing connection.
      • Model: If you selected Anaplan as the connection type and want to use a model, select Model.
      • Dataset: If you selected Anaplan as the connection type and want to use a dataset, select Datasets.
    3. Enter a new Label to change the name that displays in the pipeline source node.
    4. Complete the remaining fields that display in the Source panel, based on the connection you selected:
      • External system: See the relevant read pipeline page for your external system for more information.
      • Anaplan model: See Anaplan model connections in pipelines for more information.
      • Anaplan dataset: If you choose this, then Anaplan datasets are both the source and sink in this pipeline. For information on configuring the source, see Step 6 in the Anaplan datasets as a pipeline source section.
  7. Select the Sink icon, and then complete these steps in the right-side panel:
    1. Select Anaplan from the Connection type dropdown.
    2. Select Datasets from the Choose connection dropdown.
    3. Enter a new Label to change the name that displays in the pipeline sink node.
    4. Select Target location > Table, choose a dataset destination, select a write option, and review the source data.
      For more information, see the Choose a target dataset destination section below.
  8. Select Publish, and then select Run to execute the data transfer.

Choose a target dataset destination

When you choose the target destination for a pipeline sink, you must either choose an existing dataset or create a new one. 

You must also choose a write option for the dataset:

    • Upsert: Updates the existing rows and adds new rows if needed.
    • Append: Adds new data to the dataset without overwriting existing data.
    • Replace: Replaces all existing data with the new data being extracted, and overwrites any previous data.

Refer to this table for the steps you need to follow.

Choose destinationSteps
Choose existing dataset
  1. Select an existing dataset.
  2. Choose a write option for the dataset: Upsert, Append, or Replace.
  3. Map the source columns from the external source table to the target dataset, and then select Done.
Create new dataset
  1. Enter a Name and Description (optional) for the new dataset.
  2. Choose a write option for the dataset: Upsert, Append, or Replace.
  3. Select the source columns from the external source table that you want to include in the dataset, and then select Done.

The write option you choose only applies to subsequent pipeline runs if you update the table.

Anaplan datasets as a pipeline source and sink

You can create a pipeline that uses Anaplan datasets as both the source and sink.

Here’s an overview of how to configure the source and sink steps in this type of pipeline:

  • Source steps: To use an Anaplan dataset as a source, see Step 6 in the Anaplan datasets as a pipeline source section.
  • Sink steps: To use an Anaplan dataset as a sink, see Step 7 in the Anaplan datasets as a pipeline sink section.