You can create a pipeline that reads a CSV file from an SFTP server. The read pipeline imports the data from the CSV file on the SFTP server to Anaplan Data Orchestrator. It then uses the imported data to create or update a source dataset.

Prerequisites

You need a connection to an SFTP server to create a pipeline. Make sure you meet the prerequisites in these sections before you create a connection to the SFTP server and a read pipeline.

Create a connection to SFTP

To create a connection:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Connections from the left-side panel.
  4. Select Create connection.
  5. On the Create connection page, select SFTP, and then select Next.
    If you can't find the connector, enter a search term in the Find... field.
  6. On the Connection details page, enter these details and select Next:
    • Name: Create a name for your connection. The name can contain alphanumeric characters and underscores.
    • Description: Optionally, enter a description about your connection.
  7. On the Connection credentials page, enter your SFTP credentials and select Next:
    • Host: Enter your SFTP server hostname (for example, s-52d4b77da9example.server.transfer.us-east-1.amazonaws.com).
    • Port: Enter the SFTP port (for example, 22).
    • Username: Enter the SFTP server user name (for example, sftp-user).
    • Private Key: Enter the SSH private key (OpenSSH format) linked to the SFTP server.

      You can use the ssh-keygen -t rsa -b 4096 -f <keyfilename> command to generate a key pair. When you run the command, the system prompts you to set a passphrase. Press Enter twice to generate the unencrypted private key without a passphrase.
      This creates a public key (.pub) and a private key. The public key is linked to the SFTP server and the private key is used to make connections.

      Private key example:

-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
... (Your private key content in Base64) ...
... multiple lines ...
-----END OPENSSH PRIVATE KEY-----

  1. After the connection test is complete, select Done.

Create a read pipeline

Use the SFTP connection you created to create the read pipeline. The read pipeline uses the connection to import data from a CSV file on an SFTP server to a dataset in Data Orchestrator.

To create a read pipeline:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Pipelines from the left-side panel.
  4. Select Create pipeline.
  5. Enter a Name for your pipeline and then select Create.
    You are taken to the pipeline designer view.
  6. Select the Source icon, and then complete these steps in the right-side panel:
    1. Select SFTP from the Connection type dropdown. 
    2. Select the SFTP connection you created from the Choose connection dropdown.
    3. Enter a new Label to change the source name that displays in the designer view.
    4. Enter the Path pattern / name for the CSV file in the source location.
    5. Optionally, select the Incremental load checkbox.
    6. Select the Column Separator used in the CSV file from the dropdown.
    7. Select the Text delimiter used in the CSV file from the dropdown.
    8. Enter the Escape character used in the CSV file.
    9. Select the Contains header row checkbox if the CSV file contains a header row.
    10. Select the Skip first data row checkbox if you want the pipeline to skip the first row in the CSV file.
    11. Select Configure columns, review the column types and aliases, and then select Done.
  7. Optionally, select the add icon that appears between the Source and Sink nodes.
    You can add steps to your pipeline to process data.
  8. Select the Sink icon, and then complete these steps in the right-side panel:
    1. Select Anaplan from the Connection type dropdown.
    2. Select Datasets from the Choose connection dropdown.
    3. Enter a new Label to change the sink name that displays in the designer view.
    4. Select Target location > Table, and choose a dataset destination:
      • Choose existing dataset: If you choose this option, you will be asked to select an existing source dataset.
      • Create new dataset: If you choose this option, you will be asked to enter a Name and Description for the new dataset.
    5. Choose a write option for the dataset.
      If you chose Create a new dataset in the previous step, the write option only applies to later pipeline runs if you update the CSV file in the source location.
      • Upsert: Updates the existing rows and adds new rows if needed.
      • Append: Adds new data to the dataset without overwriting existing data.
      • Replace: Replaces all existing data with the new data being extracted, and overwrites any previous data.
    6. Review the source data from the CSV file that's being imported to the dataset, and then select Done.
  9. Select Publish, and then select Run to execute the data transfer.

The dataset displays in the Source datasets screen in Data Orchestrator. If you don't see the dataset, refresh the screen.

Run the pipeline with a Workflow

After you create the pipeline, optionally, you can choose to run the pipeline as part of an Anaplan Workflow. This enables you to automate data transfers based on a schedule, or trigger them manually as part of a larger sequence of tasks.