You can import data from external source systems into Data Orchestrator to create datasets.

To import data from external source systems: 

  1. Create a connection between Data Orchestrator and the external source system.
  2. Create a data extract that creates source datasets from the connection.

Connections

You can create a connection with these external source systems:

  • Azure Blob Storage
  • BigQuery
  • S3
  • Salesforce
  • SAP
  • Snowflake

Data extracts

When you extract data from connections, it creates source datasets.

You are limited to running a maximum of 100 data extracts and a maximum of 100 model links in any 24-hour period per package. You can add additional packages to your entitlement to increase the total number of data extracts and models links that you can run within a 24-hour period.

Deleted records from Salesforce connections

Data that is marked as deleted isn't automatically filtered out by the connector. This allows it to be used in downstream transformations and spoke models.

Salesforce doesn't immediately purge records, even if you delete them from the recycle bin. As stated in the Salesforce documentation , "When records are purged from the recycle bin, they might still be visible through API until they are completely purged from the organization. Users with View All Data and API access can check for records in this state by using Export All in Data Loader, and filtering on isDeleted = True."

If you delete a row and resynchronize with append mode in Salesforce, the deleted row won't be added to the sync. The final table contains a column labeled isDeleted, which indicates whether the row has been deleted. The isDeleted column can be used in a transformation to filter out deleted rows when required.