Create an Azure Blob Storage writeback pipeline

When you set up the writeback pipeline, you'll use your connection to export either a source dataset or a transformation view dataset (derived dataset) from Data Orchestrator to Azure Blob Storage.

Note that the write option for Azure Blob Storage is automatically full replace.

To create a writeback pipeline:

1. Select Data Orchestrator from the top-left navigation menu.
2. Choose a dataspace from the list.
3. Select Pipelines from the left-side panel.
4. Select Create pipeline.
5. Enter a Name for your pipeline and then select Create.
   You are taken to the pipeline designer view.
6. Select the Source icon, and then complete these steps in the right-side panel:
   1. Select Anaplan from the Connection type dropdown. 
   2. Select Datasets from the Choose connection dropdown.
   3. Enter a new Label if you want to change the source display name in the designer view.
      By default, the name of the dataset you selected displays as the label name.
   4. Select Source location to choose a Data Orchestrator dataset, then select Confirm.
      The dataset is used as the source for your pipeline.
   5. Select Columns if you want to exclude columns in the writeback.
       By default, all ‌columns are included. You can also select a Cursor column for your source.
7. Optionally, select the add icon that appears between the Source and Sink nodes.
   You can add steps to your pipeline to process data.
8. Select the Sink icon, and then complete these steps in the right-side panel:
   1. Select Azure Blob from the Connection type dropdown.
   2. Select the Azure connection you created from the Choose connection dropdown.
   3. Enter the file name followed by the .csv extension in the File path field.
      We recommend you include the date in the file name in this format YYMMDD (Y for year, M for month, and D for date). For example: Azurefilename_260901.csv
   4. Select the Overwrite file checkbox if you want to overwrite an existing file in Azure, or leave it blank to create a new file:
      • Create new file: Leave the Overwrite file checkbox blank, and enter the new file name in the File path field.
        Note: If the file already exists, the pipeline fails and displays a message with this reason.
      • Overwrite existing file: Select the Overwrite file checkbox to overwrite an existing file in Azure. Enter the name of the existing file in the File path field.
   5. Select a Column Separator from the dropdown:
      • Comma
      • Tab
      • Pipe
      • Semicolon
   6. Select a Text delimiter from the dropdown:
      • Double quote (")
      • Single quote (')
      • None
9. Enter a new Label to change the sink name that displays in the designer view.
10. Select Publish, and then select Run to execute the data transfer.

Once the pipeline is successfully completed, you can log in to the Microsoft Azure platform to verify the writeback. Navigate to Azure Blob Storage and verify the CSV file is written back to your container.