You can create a pipeline that writes back data from an Anaplan Data Orchestrator dataset to Amazon S3.

You must create a connection to Amazon S3 before you can create a pipeline.

Connectivity prerequisites

Before you can create a connection to Amazon S3, you need the prerequisites listed in this table.

ItemNotes
Amazon Web Services (AWS) S3 bucket 

A S3 bucket hosted on AWS or any S3-compatible cloud service. 

You must have active access to a S3 bucket where the exported file will be stored. 

AWS IAM user with security credentialsAn IAM user with an AWS access key ID and an AWS secret access key.
Appropriate permissions for IAM userThe IAM user whose credentials are used in the connection must have the necessary permissions to write to the target S3 bucket. This typically includes the s3:PutObject permission for the specified bucket and object key prefix. 

Create a connection with Amazon S3

Use the S3 connector in Data Orchestrator to create a connection.

You will need your S3 credentials to connect your S3 data with Data Orchestrator. View the S3 documentation for more information about your credentials.

To create a connection:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Connections on the left-side panel.
  4. Select Create connection.
  5. Select the S3 connector on the Select a connector screen, and then select Next.
    If you can't find the connector, enter a search term in the Find... field.
  6. Enter these details on the Connection details screen, and then select Next:
    • Name: Create a name for your connection. The name can contain alphanumeric characters and underscores.
    • Description: Enter a description about your connection.
  7. Enter your S3 credentials on the Connection Credentials screen, and then select Next:
    • AWS Key: The access key ID (for example, AKIAIOSFODNN7EXAMPLE). 
    • AWS Secret Key: The secret access key (for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY).
    • Bucket: The S3 bucket where your data is stored (for example, S3-EXAMPLE-BUCKET).
  8. After the connection test is complete, select Done.

Create a writeback pipeline

When you set up the writeback pipeline, you'll use your connection to export either a source dataset or a transformation view dataset (derived dataset) from Data Orchestrator to S3.

Note that the write option for S3 is automatically full replace.

To create a writeback pipeline:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Pipelines from the left-side panel.
  4. Select Create pipeline.
  5. Enter a Name for your pipeline and then select Create.
    You are taken to the pipeline designer view.
  6. Select the Source icon, and then complete these steps in the right-side panel:
    1. Select Anaplan from the Connection type dropdown. 
    2. Select Datasets from the Choose connection dropdown.
    3. Select Source location to choose a Data Orchestrator dataset, then select Confirm.
      The dataset is used as the source for your pipeline.
    4. Enter a new Label if you want to change the source display name in the designer view.
      By default, the name of the dataset you selected displays as the label name.
  7. Select the Sink icon, and then complete these steps in the right-side panel:
    1. Select S3 from the Connection type dropdown.
    2. Select the S3 connection you created from the Choose connection dropdown.
    3. Enter the file name followed by the .csv extension in the File path field.
      We recommend you include the date in the file name in this format YYMMDD (Y for year, M for month, and D for date). For example: S3filename_260701.csv
    4. Select the Overwrite file checkbox, or leave it blank.
      You can either create a new file, or ‌overwrite an existing file in S3: 
      • Create new file: Leave the Overwrite file checkbox blank, and enter the file name in the field.
      • Overwrite existing file: Overwrite the existing file in S3 by selecting the Overwrite file checkbox.
        If you use a file name that already exists in S3, you must overwrite the existing file. If you don't overwrite the existing file, the pipeline fails and shows you a message that the file already exists.
    5. Select a Column Separator from the dropdown:
      • Comma
      • Tab
      • Pipe
      • Semicolon
    6. Select a Text delimiter from the dropdown:
      • Double quote (")
      • Single quote (')
      • None
  8. Enter a new Label if you want to change the sink display name in the designer view.
    By default, the name of the connection you selected displays as the label name.
  9. Select Publish, and then select Run to execute the data transfer.

Run the pipeline with a Workflow

After you create the pipeline, optionally, you can choose to run the pipeline as part of an Anaplan Workflow. This enables you to automate data transfers based on a schedule, or trigger them manually as part of a larger sequence of tasks.