You can create a pipeline that reads a CSV file from Amazon S3. The read pipeline imports the data from the CSV file in Amazon S3 to Anaplan Data Orchestrator. It then uses the imported data to create or update a source dataset.

You must create a connection to Amazon S3 before you can create a pipeline.

Connectivity prerequisites

Before you can create a connection to Amazon S3, you need the prerequisites listed in this table.

ItemNotes
Amazon Web Services (AWS) S3 bucket A S3 bucket hosted on AWS or any S3-compatible cloud service.
AWS IAM user with security credentials

An IAM user with an AWS access key ID and an AWS secret access key. 

They also need at least read permissions for ‌the bucket and objects.

CSV file in Amazon S3

For the CSV file, make sure that:

  • You specify the prefix or folder path if you restrict access to a subset of the bucket.
  • CSV files are UTF-8 encoded.

See File requirements for Anaplan Data Orchestrator for more information about CSV file requirements.

Amazon S3 との接続の作成

Data Orchestrator の S3 コネクターを使用して接続を作成します。

S3 データを Data Orchestrator に接続するには、S3 認証情報が必要です。認証情報の詳細については、S3 のドキュメントを参照してください。

接続を作成するには以下を実行します。

  1. 左上のナビゲーション メニューから [Data Orchestrator] を選択します。
  2. リストからデータスペースを選択します。
  3. 左側のパネルで [Connections] を選択します。
  4. [Create connection] を選択します。
  5. [Create connections] ページで [S3] を選択し、[Next] を選択します。
    コネクターが見つからない場合は、[Find...] フィールドに検索ワードを入力します。
  6. [Connection details] ページで次の詳細を入力し、[Next] を選択します。
    • Name (名前):接続の名前を作成します。名前には英数字とアンダースコアを使用できます。
    • Description (説明):接続に関する説明を入力します。
  7. [Connection Credentials] ページで S3 の認証情報を入力し、[Next] を選択します。
    • AWS Key (AWS キー):アクセス キー ID (例: AKIAIOSFODNN7EXAMPLE)。 
    • AWS Secret Key (AWS シークレット キー):シークレット アクセス キー (例: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY)。
    • Bucket (バケット):データが保存されている S3 バケット (例: S3-EXAMPLE-BUCKET)。
  8. 接続テストが完了したら、[Done] を選択します。

Create a read pipeline

Use the Amazon S3 connection you created to create the read pipeline. The read pipeline uses the connection to import data from a CSV file in Amazon S3 to a dataset in Data Orchestrator.

To create a read pipeline:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Pipelines from the left-side panel.
  4. Select Create pipeline.
  5. Enter a Name for your pipeline and then select Create.
    You are taken to the pipeline designer view.
  6. Select the Source icon, and then complete these steps in the right-side panel:
    1. Select S3 from the Connection type dropdown. 
    2. Select the S3 connection you created from the Choose connection dropdown.
    3. Enter a Label that displays as the source name in the designer view.
    4. Enter the Path pattern / name for the S3 CSV file.
      To find the S3 CSV path, open the CSV file in S3, and copy the Key shown under the Properties tab.
    5. Optionally, select the Incremental load checkbox.
    6. Select the Column Separator used in the CSV file from the dropdown.
    7. Select the Text delimiter used in the CSV file from the dropdown.
    8. Optionally, enter the Escape character used in the CSV file.
    9. Select the Contains header row checkbox if the CSV file contains a header row.
    10. Select the Skip first data row checkbox if you want the pipeline to skip the first row in the CSV file.
    11. Select Configure columns, review the column types and aliases, and then select Done.
  7. Optionally, select the add icon that appears between the Source and Sink nodes.
    You can add steps to your pipeline to process data.
  8. Select the Sink icon, and then complete these steps in the right-side panel:
    1. Select Anaplan from the Connection type dropdown.
    2. Select Datasets from the Choose connection dropdown.
    3. Select Target location > Table, and choose a dataset destination:
      • Choose existing dataset: If you choose this option, you will be asked to select an existing source dataset.
      • Create new dataset: If you choose this option, you will be asked to enter a Name and Description for the new dataset.
        When you select a write option for the pipeline, it only applies to later pipeline runs if you update the CSV file in S3.
    4. Choose a write option for the dataset:
      • Upsert: Updates the existing rows and adds new rows if needed.
      • Append: Adds new data to the dataset without overwriting existing data.
      • Replace: Replaces all existing data with the new data being extracted, and overwrites any previous data.
    5. Review the source data from the CSV file that's being imported to the dataset, and then select Done.
  9. Select Publish, and then select Run to execute the data transfer.

The dataset displays in the Source datasets screen in Data Orchestrator. If you don't see the dataset, refresh the screen.

Run the pipeline with a Workflow

After you create the pipeline, optionally, you can choose to run the pipeline as part of an Anaplan Workflow. This enables you to automate data transfers based on a schedule, or trigger them manually as part of a larger sequence of tasks.