You can create a pipeline that reads a data table in Databricks. The read pipeline imports the data from Databricks to Anaplan Data Orchestrator. It then uses the imported data to create or update a source dataset.

You must create a connection to Databricks before you can create a pipeline.

Connectivity prerequisites

Before you can create a connection to Databricks, you need the connectivity prerequisites listed in this table.

ItemNotes
Databricks workspaceA Databricks workspace hosted on Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP). 
Databricks SQL warehouse

A running Databricks SQL warehouse or compute resource that the service principal can access.

When you configure the Databricks SQL warehouse, make sure to give the service principal the appropriate permissions.

Databricks service principal 

You need a Databricks service principal that's configured with these items:

  • OAuth authentication: Configured for OAuth authentication with a Client ID and Secret. For more information, see the Databricks OAuth documentation.
  • Entitlements: Make sure to grant the service principal the necessary entitlements. 
    • These entitlements are required: Databricks SQL access and Workspace access.
    • For more information, see the Databricks entitlements documentation.
Service principal accessAccess for the service principal to the SQL Warehouse or compute resource used for the connection. 
Unity Catalog permissions

The required Unity Catalog permissions for the service principal to access the target catalog and schema: 

  • USE CATALOG
  • USE SCHEMA 
  • SELECT for data extraction 
  • MODIFY and CREATE TABLE if required for writeback

For example: 

GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal_id>`; 

GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal_id>`; 

GRANT SELECT ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal_id>`; 

GRANT MODIFY, CREATE TABLE ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal_id>`; 

Databricks connection details

You need the following Databricks connection details: 

  • Host 
  • Port 
  • HTTP Path 
  • Catalog 
  • Schema 
  • Client ID 
  • Secret 

Create a connection to Databricks

To create a connection:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Connections from the left-side panel.
  4. Select Create connection.
  5. On the Create connection page, select Databricks, and then select Next.
    If you don't find the connector, enter a search term in the Find... field.
  6. On the Connection details page, enter these details and select Next:
    • Name: Create a name for your connection. The name can contain alphanumeric characters and underscores.
    • Description: Optionally, enter a description about your connection.
  7. On the Connection credentials page, enter your Google credentials and select Next.
    See the table below for information about each field.
Connection credentialsDesctiption
HostThe Databricks workspace URL. You can find the host in your browser's address bar. You don't need to include https:// in the host.
Port

The port number for your Databricks SQL warehouse connections. The port number is typically 443. 

To find the port number in your Databricks workspace:

  1. Select SQL Warehouses in the left-side panel.
  2. Select the specific SQL warehouse you want to connect to.
  3. Select ‌the Connection details tab on the SQL Warehouse page.
    The port number displays in the JDBC URL.
HTTP Path

The HTTP path of your Databricks database.

To find the HTTP path in your Databricks workspace:

  1. Select SQL Warehouses in the left-side panel.
  2. Select the specific SQL warehouse you want to connect to.
  3. Select ‌the Connection details tab on the SQL Warehouse page.
    The HTTP path displays and starts with /sql/1.0/warehouses/.
Catalog

The Databricks database catalog defines the data namespace you want to access.

To find the catalog in your Databricks workspace, select Catalog in the left-side panel. A list of available catalogs displays. Use the catalog name for the Catalog field in your Databricks connection.

Schema

The Databricks database schema defines the data namespace you want to access.

To find the schema in your Databricks workspace, select Catalog in the left-side panel, and then select a catalog. The schemas display within the catalog. Use the schema name for the Schema field in your Databricks connection.

Client ID and Secret

These credentials are used to authenticate your application.

You can get the client ID and secret by creating a service principal in Databricks. This is the recommended and most secure method for headless authentication. To create a service principle, you must be a workspace administrator in Databricks.

To create a service principal and get credentials:

  1. Select ‌your user name in the top-right corner of your Databicks workspace, and select Settings.
  2. Select Identity and access.
  3. Select Manage next to Service principals.
  4. Select Add service principal.
  5. Enter a name for your service principal (for example, NewConnectorServicePrincipal) and select Add.
    The service principal is now created.
  6. Follow these steps to get the client ID:
    1. Select the service principal name to open its details page.
      The Application Id is your client ID. 
    2. Copy the Application Id.
  7. Follow these steps to get the secret:
    1. Select the Secrets tab.
    2. Under OAuth secrets, select Generate secret.
    3. Copy the generated secret.
      This is your secret. You won't be able to see it again after you navigate away from this page.

Important: Remember to grant this new service principal the necessary permissions on your SQL warehouse, catalog, and schema for the connector to be able to access the data. You can do this from the permissions tabs in the respective UI sections.

  1. After the connection test is complete, select Done.

Create a read pipeline

Use the Databricks connection you created to create the read pipeline. The read pipeline uses the connection to import data from a table in Databricks to a source dataset in Data Orchestrator.

To create a read pipeline:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Pipelines from the left-side panel.
  4. Select Create pipeline.
  5. Enter a Name for your pipeline and then select Create.
    You are taken to the pipeline designer view.
  6. Select the Source icon, and then complete these steps in the right-side panel:
    1. Select Databricks from the Connection type dropdown. 
    2. Select the Databricks connection you created from the Choose connection dropdown.
    3. Enter a Label that displays as the source name in the designer view.
    4. Select a Source location, and then select a table in Databricks that you want to connect​ to.
  7. Optionally, select the add icon that appears between the Source and Sink nodes.
    You can add steps to your pipeline to process data.
  8. Select the Sink icon, and then complete these steps in the right-side panel:
    1. Select Anaplan from the Connection type dropdown.
    2. Select Datasets from the Choose connection dropdown.
    3. Select Target location > Table, and choose a dataset destination:
      • Choose existing dataset: If you choose this option, you will be asked to select an existing source dataset.
      • Create new dataset: If you choose this option, you will be asked to enter a Name and Description for the new dataset.
        When you select a write option for the pipeline, it only applies to later pipeline runs if you update the table in Databricks.
    4. Choose a write option for the dataset.
      • Upsert: Updates the existing rows and adds new rows if needed.
      • Append: Adds new data to the dataset without overwriting existing data.
      • Replace: Replaces all existing data with the new data being extracted, and overwrites any previous data.
    5. Review the source data from the Databricks table that's being imported to the dataset, and then select Done.
  9. Select Publish, and then select Run to execute the data transfer.

The dataset displays in the Source datasets screen in Data Orchestrator. If you don't see the dataset, refresh the screen.

Run the pipeline with a Workflow

After you create the pipeline, optionally, you can choose to run the pipeline as part of an Anaplan Workflow. This enables you to automate data transfers based on a schedule, or trigger them manually as part of a larger sequence of tasks.