You can create a pipeline that reads objects in Google BigQuery. The read pipeline imports the data from Google BigQuery to Anaplan Data Orchestrator (ADO).

In a read pipeline, the source data flows through the remaining steps in the pipeline to the final sink (target) step. The sink step must use a connection to an Anaplan object.

Prerequisites

You need a connection to Google BigQuery to create the pipeline. Make sure you meet the prerequisites in these sections before you create a connection to BigQuery and a read pipeline.

Create a connection to Google BigQuery

You will need your Google BigQuery credentials to connect the BigQuery data with Data Orchestrator. View the BigQuery documentation for more information about your credentials.

To create a connection:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Connections from the left-side panel.
  4. Select Create connection.
  5. Select the BigQuery connector and then select Next.
    If you can't find the connector, enter a search term in the Find... field.
  6. Enter these details on the Connection details screen, and then select Next:
    • Name: Create a name for your connection. The name can contain alphanumeric characters and underscores.
    • Description: Enter a description about your connection.
  7. Enter your BigQuery credentials on the Connection credentials screen, and then select Next.
    • Project ID: The identifier for the BigQuery project that contains the dataset (for example, myproject-1234).
    • Credentials Json: The BigQuery JSON code for authentication.
    • Dataset ID: The BigQuery dataset ID.
      (If you use the Google BigQuery connection to extract data, only the views defined under the Dataset ID for the connection definition are visible under Path name.)
  8. After the connection test is complete, select Done.

Create a read pipeline

Use the Google BigQuery connection you created to configure the read pipeline. The read pipeline uses the connection to import data from a table in BigQuery to an object in ADO.

To create a read pipeline:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Pipelines from the left-side panel.
  4. Select Create pipeline.
  5. Enter a Name for your pipeline and then select Create.
    You are taken to the pipeline designer view.
  6. Select the Source icon, and then complete these steps in the right-side panel:
    1. Select BigQuery from the Connection type dropdown. 
    2. Select the BigQuery connection you created from the Choose connection dropdown.
    3. Enter a new Label to change the source name that displays in the designer view.
    4. Select a Source location > Source, and then select a table from BigQuery that you want to connect​ to.
    5. Select Columns, choose which columns from the BigQuery table you want to include, optionally choose a cursor column, and then select Done.
  7. Optionally, select the add icon that appears between the Source and Sink nodes.
    You can add steps to your pipeline to process data.
  8. Select the Sink icon, and then complete these steps in the right-side panel:
    1. Select Anaplan from the Connection type dropdown.
    2. Select Datasets from the Choose connection dropdown.
    3. Enter a new Label to change the sink name that displays in the designer view.
    4. Select Target location > Table, and choose a dataset destination:
      • Choose existing dataset: If you choose this option, you will be asked to select an existing source dataset.
      • Create new dataset: If you choose this option, you will be asked to enter a Name and Description for the new dataset.
    5. Choose a write option for the dataset.
      If you chose Create a new dataset in the previous step, the write option only applies to later pipeline runs if you update the table in BigQuery.
      • Upsert: Updates the existing rows and adds new rows if needed.
      • Append: Adds new data to the dataset without overwriting existing data.
      • Replace: Replaces all existing data with the new data being extracted, and overwrites any previous data.
    6. Review the source data from the BigQuery table that’s being imported to the dataset, and then select Done.
  9. Select Publish, and then select Run to execute the data transfer.

The dataset displays in the Source datasets screen in Data Orchestrator. If you don't see the dataset, refresh the screen.

Run the pipeline with a Workflow

After you create the pipeline, optionally, you can choose to run the pipeline as part of an Anaplan Workflow. This enables you to automate data transfers based on a schedule, or trigger them manually as part of a larger sequence of tasks.