Databricks Zerobus

The Databricks Zerobus connection supports streaming ingestion of large numbers of files directly into a Databricks SQL table.

Connection Settings

Server Endpoint

The full Databricks server endpoint in the format: https://<workspace-id>.zerobus.<region>.cloud.databricks.com.

Workspace URL

The full Databricks Workspace URL in the format: https://<deployment-id>.cloud.databricks.com.

Connection String

The JDBC connection string for your Databricks SQL Warehouse in the format: jdbc:databricks://<deployment-id>:<port>/<schema>.

HTTP Path

The HTTP path to your Databricks SQL Warehouse in the format /sql/1.0/warehouses/<warehouse_id>.

Authentication Type

The only supported authentication type is OAuth. This will be automatically set

Client ID

The client ID of the service principal or user.

Client Secret

The client secret of the service principal or user.

Input Settings

Inputs are not supported.

Output Settings

Catalog

The name of the catalog.

Schema

The name of the schema.

Volume

The name of the volume.

Create Option

Controls how the table definition is managed. Options are Off, Create, and Create & Update.

  • When Off is used, it’s assumed that the table schema is defined and managed externally. The table must exist. See Table Cache Interval for details on how to respond to table schema changes.
  • When Create is used, it’s assumed the connection manages the table and it’s created once. If the table doesn’t exist it’s created using the schema of the payload.
  • When Create & Update is used, it’s assumed the connection manages the table and updates the table on schema change. For example, if a new attribute is added to the data, the column is added to the table.