You can create a new pipeline that writes back data from an Anaplan Data Orchestrator dataset to Amazon S3.

Prerequisites

Before you can create a pipeline, you need the prerequisite items listed in this table.

ItemNotes
Amazon Web Services (AWS) S3 bucket You must have active access to an S3 bucket where the exported file will be stored. 
Configured S3 connection 

You must configure a valid S3 connection in Data Orchestrator. This process requires an AWS access key ID and an AWS secret access key associated with an IAM user. 

For more information on how to create S3 connections, see the Create a connection with Amazon S3 section below.

Appropriate permissions The IAM user whose credentials are used in the connection must have the necessary permissions to write to the target S3 bucket. This typically includes the s3:PutObject permission for the specified bucket and object key prefix. 

Create a connection with Amazon S3

Use the S3 connector in Data Orchestrator to create a connection.

You will need your S3 credentials to connect your S3 data with Data Orchestrator. View the S3 documentation for more information about your credentials.

To create a connection:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Connections on the left-side panel.
  4. Select Create connection.
  5. On the Create connections page, select S3 and then select Next.
    If you can't find the connector, enter a search term in the Find... field.
  6. On the Connection details page, enter these details and select Next:
    • Name: Create a name for your connection. The name can contain alphanumeric characters and underscores.
    • Description: Enter a description about your connection.
  7. On the Connection Credentials page, enter your S3 credentials and select Next:
    • AWS Key: The access key ID (for example, AKIAIOSFODNN7EXAMPLE). 
    • AWS Secret Key: The secret access key (for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY).
    • Bucket: The S3 bucket where your data is stored (for example, S3-EXAMPLE-BUCKET).
  8. After the connection test is complete, select Done.

Create a writeback pipeline

When you set up the writeback pipeline, you'll use your connection to export either a source dataset or a transformation view dataset (derived dataset) from Data Orchestrator to S3.

Note that the write option for S3 is automatically full replace.

To create a writeback pipeline:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Pipelines from the left-side panel.
  4. Select Create pipeline.
  5. Enter a Name for your pipeline and then select Create.
    You are taken to the pipeline designer view.
  6. Select the Source icon, and then complete these steps in the right-side panel:
    1. Select Anaplan from the Connection type dropdown. 
    2. Select Datasets from the Choose connection dropdown.
    3. Select Source location to choose a Data Orchestrator dataset, then select Confirm.
      The dataset is used as the source for your pipeline.
    4. Enter a new Label if you want to change the source display name in the designer view.
      By default, the name of the dataset you selected displays as the label name.
  7. Select the Sink icon, and then complete these steps in the right-side panel:
    1. Select S3 from the Connection type dropdown.
    2. Select the S3 connection you created from the Choose connection dropdown.
    3. Enter the file name followed by the .csv extension in the File path field.
      We recommend you include the date in the file name in this format YYMMDD (Y for year, M for month, and D for date). For example: S3filename_260701.csv
    4. Select the Overwrite file checkbox, or leave it blank.
      You can either create a new file, or ‌overwrite an existing file in S3: 
      • Create new file: Leave the Overwrite file checkbox blank, and enter the file name in the field.
      • Overwrite existing file: Overwrite the existing file in S3 by selecting the Overwrite file checkbox.
        If you use a file name that already exists in S3, you must overwrite the existing file. If you don't overwrite the existing file, the pipeline fails and shows you a message that the file already exists.
    5. Select a Column Separator from the dropdown:
      • Comma
      • Tab
      • Pipe
      • Semicolon
    6. Select a Text delimiter from the dropdown:
      • Double quote (")
      • Single quote (')
      • None
  8. Enter a new Label if you want to change the sink display name in the designer view.
    By default, the name of the connection you selected displays as the label name.
  9. Select Publish, and then select Run to execute the data transfer.

Run the pipeline with a Workflow

After you create the pipeline, optionally, you can choose to run the pipeline as part of an Anaplan Workflow. This enables you to automate data transfers based on a schedule, or trigger them manually as part of a larger sequence of tasks.