You can import data from Databricks to create source datasets in Anaplan Data Orchestrator (ADO). To import data, you need a connection to Databricks and a read pipeline in Data Orchestrator.

You must first create a connection to Databricks with the Databricks connector. Then use that connection to set up a read pipeline for data import.

Connectivity prerequisites

Before you create a connection to Databricks, you need to meet these prerequisites.

ItemNotes
Databricks workspaceA Databricks workspace hosted on Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP). 
Databricks SQL warehouse

A running Databricks SQL warehouse or compute resource that the service principal can access.

When you configure the Databricks SQL warehouse, make sure to give the service principal the appropriate permissions.

Databricks service principal 

You need a Databricks service principal that's configured with these items:

  • OAuth authentication: Configured for OAuth authentication with a Client ID and Secret. For more information, see the Databricks OAuth documentation.
  • Entitlements: Make sure to grant the service principal the necessary entitlements. 
    • These entitlements are required: Databricks SQL access and Workspace access.
    • For more information, see the Databricks entitlements documentation.
Service principal accessAccess for the service principal to the SQL Warehouse or compute resource used for the connection. 
Unity Catalog permissions

The required Unity Catalog permissions for the service principal to access the target catalog and schema: 

  • USE CATALOG
  • USE SCHEMA 
  • SELECT for data extraction 
  • MODIFY and CREATE TABLE if required for writeback

For example: 

GRANT USE CATALOG ON CATALOG <catalog_name> TO `<service_principal_id>`; 

GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal_id>`; 

GRANT SELECT ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal_id>`; 

GRANT MODIFY, CREATE TABLE ON SCHEMA <catalog_name>.<schema_name> TO `<service_principal_id>`; 

Databricks connection details

You need the following Databricks connection details: 

  • Host 
  • Port 
  • HTTP Path 
  • Catalog 
  • Schema 
  • Client ID 
  • Secret 

Create a connection to Databricks

From the Connection screen in Data Orchestrator, use the Databricks connector to create a connection to Databricks.

To create a connection:

  1. Select Data Orchestrator from the top-left navigation menu.
  2. Choose a dataspace from the list.
  3. Select Connections from the left-side panel.
  4. Select Create connection.
  5. On the Create connection page, select Databricks, and then select Next.
    If you don't find the connector, enter a search term in the Find... field.
  6. On the Connection details page, enter these details and select Next:
    • Name: Create a name for your connection. The name can contain alphanumeric characters and underscores.
    • Description: Optionally, enter a description about your connection.
  7. On the Connection credentials page, enter your Google credentials and select Next.
    See the table below for information about each field.
Connection credentialsDescription
HostThe Databricks workspace URL. You can find the host in your browser's address bar. You don't need to include https:// in the host.
Port

The port number for your Databricks SQL warehouse connections. The port number is typically 443. 

To find the port number in your Databricks workspace:

  1. Select SQL Warehouses in the left-side panel.
  2. Select the specific SQL warehouse you want to connect to.
  3. Select ‌the Connection details tab on the SQL Warehouse page.
    The port number displays in the JDBC URL.
HTTP Path

The HTTP path of your Databricks database.

To find the HTTP path in your Databricks workspace:

  1. Select SQL Warehouses in the left-side panel.
  2. Select the specific SQL warehouse you want to connect to.
  3. Select ‌the Connection details tab on the SQL Warehouse page.
    The HTTP path displays and starts with /sql/1.0/warehouses/.
Catalog

The Databricks database catalog defines the data namespace you want to access.

To find the catalog in your Databricks workspace, select Catalog in the left-side panel. A list of available catalogs displays. Use the catalog name for the Catalog field in your Databricks connection.

Schema

The Databricks database schema defines the data namespace you want to access.

To find the schema in your Databricks workspace, select Catalog in the left-side panel, and then select a catalog. The schemas display within the catalog. Use the schema name for the Schema field in your Databricks connection.

Client ID and Secret

These credentials are used to authenticate your application.

You can get the client ID and secret by creating a service principal in Databricks. This is the recommended and most secure method for headless authentication. To create a service principle, you must be a workspace administrator in Databricks.

To create a service principal and get credentials:

  1. Select ‌your user name in the top-right corner of your Databricks workspace, and select Settings.
  2. Select Identity and access.
  3. Select Manage next to Service principals.
  4. Select Add service principal.
  5. Enter a name for your service principal (for example, NewConnectorServicePrincipal) and select Add.
    The service principal is now created.
  6. Follow these steps to get the client ID:
    1. Select the service principal name to open its details page.
      The Application Id is your client ID. 
    2. Copy the Application Id.
  7. Follow these steps to get the secret:
    1. Select the Secrets tab.
    2. Under OAuth secrets, select Generate secret.
    3. Copy the generated secret.
      This is your secret. You won't be able to see it again after you navigate away from this page.

Important: Remember to grant this new service principal the necessary permissions on your SQL warehouse, catalog, and schema for the connector to be able to access the data. You can do this from the permissions tabs in the respective UI sections.

  1. After the connection test is complete, select Done.

Import data

To import data from Databricks, you must create a read pipeline with your Databricks connection. 

A read pipeline imports the data from a table in Databricks to Data Orchestrator. It then uses the imported data to create or update a source dataset.

You can create a read pipeline from the Pipelines screen in Data Orchestrator.