You can create a pipeline that writes back data from an Anaplan Data Orchestrator dataset to Amazon S3.

Prerequisites

You need a connection to Amazon S3 to create a pipeline. Make sure you meet the prerequisites in these sections before you create a connection to Amazon S3 and a writeback pipeline.

Create a connection with Amazon S3

Use the S3 connector in Data Orchestrator to create a connection.

You will need your S3 credentials to connect your S3 data with Data Orchestrator. View the S3 documentation for more information about your credentials.

To create a connection:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Connections on the left-side panel.
  4. Select Create connection.
  5. Select the S3 connector on the Select a connector screen, and then select Next.
    If you can't find the connector, enter a search term in the Find... field.
  6. Enter these details on the Connection details screen, and then select Next:
    • Name: Create a name for your connection. The name can contain alphanumeric characters and underscores.
    • Description: Enter a description about your connection.
  7. Enter your S3 credentials on the Connection Credentials screen, and then select Next:
    • AWS Key: The access key ID (for example, AKIAIOSFODNN7EXAMPLE). 
    • AWS Secret Key: The secret access key (for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY).
    • Bucket: The S3 bucket where your data is stored (for example, S3-EXAMPLE-BUCKET).
  8. After the connection test is complete, select Done.

Create a writeback pipeline

When you set up the writeback pipeline, you'll use your connection to export either a source dataset or a transformation view dataset (derived dataset) from Data Orchestrator to Amazon S3.

Note that the write option for S3 is automatically full replace.

To create a writeback pipeline:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Pipelines from the left-side panel.
  4. Select Create pipeline.
  5. Enter a Name for your pipeline and then select Create.
    You are taken to the pipeline designer view.
  6. Select the Source icon, and then complete these steps in the right-side panel:
    1. Select Anaplan from the Connection type dropdown. 
    2. Select Datasets from the Choose connection dropdown.
    3. Enter a new Label to change the source display name in the designer view.
    4. Select Source location > Source to choose a Data Orchestrator dataset, then select Confirm.
      The dataset is used as the source for your pipeline.
  7. Optionally, select the add icon that appears between the Source and Sink nodes.
    You can add steps to your pipeline to process data.
  8. Select the Sink icon, and then complete these steps in the right-side panel:
    1. Select S3 from the Connection type dropdown.
    2. Select the S3 connection you created from the Choose connection dropdown.
    3. Enter a new Label to change the sink name that displays in the designer view.
    4. Enter the file name followed by the .csv extension in the File path field.
      We recommend you include the date in the file name in this format YYMMDD (Y for year, M for month, and D for date). For example: S3filename_260701.csv
    5. Select the Overwrite file checkbox, or leave it blank.
      You can either create a new file, or ‌overwrite an existing file in S3: 
      • Create new file: Leave the Overwrite file checkbox blank, and enter the file name in the field.
      • Overwrite existing file: Overwrite the existing file in S3 by selecting the Overwrite file checkbox.
        If you use a file name that already exists in S3, you must overwrite the existing file. If you don't overwrite the existing file, the pipeline fails and shows you a message that the file already exists.
    6. Select a Column Separator from the dropdown:
      • Comma
      • Tab
      • Pipe
      • Semicolon
    7. Select a Text delimiter from the dropdown:
      • Double quote (")
      • Single quote (')
      • None
  9. Select Publish, and then select Run to execute the data transfer.

Verify the writeback in Amazon S3

Once the pipeline is successfully completed, log in AWS to verify the writeback. Navigate to Amazon S3 and verify the file is written back to your bucket.