Databricks Client

The Databricks Client connects to Databricks Workspaces using the Databricks REST API.

Connection Settings

Workspace URL

The full Databricks Workspace URL in the format: https://<deployment-id>.cloud.databricks.com.

Authentication Type

Specifies the authentication method used to connect to the Databricks instance. Personal Access Token and OAuth are supported.

Personal Access Token (PAT)

Used to authenticate individual users.

Token

The token used for user authentication.

OAuth

Used to authenticate service principals and users.

Client ID

The client ID of the service principal or user.

Client Secret

The client secret of the service principal or user.

Request Timeout

The maximum number of seconds to wait for a response.

Input Settings

Inputs are not supported.

Output Settings

Catalog

The name of the catalog.

Schema

The name of the schema.

Volume

The name of the volume.

Path

The relative path of the file within the volume. Use forward slashes (/) to separate directories. The path should end with the file name.

Source Type

Specifies where the file content comes from.

Option	Description
Payload	The traditional experience. File content comes from the pipeline payload.
Source File Path	File content is streamed directly from a path on the machine running Intelligence Hub, avoiding loading the entire file into memory at the same time.

Payload Reference

When working with complex payloads, this setting uses dynamic outputs to specify the attribute that contains the file payload (e.g. {{this.filePayload}}). Available when Source Type is set to Payload.

Note: If left blank, the entire payload is written out.

Source File Path

The path to the file on the machine running Intelligence Hub. The file is streamed directly to Databricks, avoiding loading the entire file into memory at the same time. Available when Source Type is set to Source File Path.