Databricks

Databricks connections use a SQL warehouse and a personal access token (PAT). Faraday reads tables as datasets by running SQL in that warehouse and downloading the result. Faraday writes full-replacement or upsert targets with COPY INTO from short-lived Faraday S3 credentials.

Cloud

Faraday supports Databricks on AWS, Azure, and GCP. Create a SQL warehouse (not a classic cluster). In the warehouse's Connection details, copy the Server hostname and HTTP path.

Please allowlist these official Faraday IP addresses:

• 34.86.175.54
• 34.86.252.230
• 34.145.239.81
• 35.245.199.181
• 52.22.91.248
• 52.23.137.21
• 52.204.223.208
• 52.204.228.32
• 52.204.230.227

If the workspace uses IP access lists, add Faraday's addresses to the allowlist before testing the connection.

Setup process

1. Create a SQL warehouse in Databricks. Serverless or pro warehouses both work. Note the warehouse HTTP path, which looks like /sql/1.0/warehouses/<id>.

2. Create a Faraday-only catalog or schema that Faraday can read from and write to. A dedicated schema is enough if you prefer to keep Faraday in an existing catalog.

3. Create a service principal or user for Faraday, then generate a personal access token for it. Store the token only in Faraday; it is a secret.

4. Grant least-privilege access. Replace catalog and schema names with yours.

For sources (reading tables and views):

   GRANT USE CATALOG ON CATALOG your_catalog TO `faraday`;
   GRANT USE SCHEMA ON SCHEMA your_catalog.your_schema TO `faraday`;
   GRANT SELECT ON SCHEMA your_catalog.your_schema TO `faraday`;

For targets (full-replacement writes via CREATE OR REPLACE TABLE and COPY INTO, or upsert via MERGE):

   GRANT USE CATALOG ON CATALOG your_catalog TO `faraday`;
   GRANT USE SCHEMA ON SCHEMA your_catalog.your_schema TO `faraday`;
   GRANT CREATE TABLE ON SCHEMA your_catalog.your_schema TO `faraday`;
   GRANT MODIFY ON SCHEMA your_catalog.your_schema TO `faraday`;
   GRANT SELECT ON SCHEMA your_catalog.your_schema TO `faraday`;

1. Create your Faraday connection with host (the workspace server hostname, no protocol; it must end in cloud.databricks.com, gcp.databricks.com, azuredatabricks.net, or another Databricks workspace suffix), http_path, catalog, schema, and token.

2. Test the connection by using Force Update in Faraday.

Faraday suggests that you use an unguessable string somewhere in the path to your data. This avoids what is called the Confused deputy problem

For example, let's say you were using S3. Instead of naming an S3 bucket s3://faraday-acme/, name it s3://faraday-acme-pwiiprz162ez. This guarantees that malicious actors cannot guess the name and request that Faraday import data from it into their account. The same logic applies to any path that is used to locate data.

Targets

By default, each target run replaces the destination table (CREATE OR REPLACE TABLE then COPY INTO from Faraday's staging bucket). The SQL warehouse must be able to reach AWS S3 in us-east-1. If you enable upsert on a referenced target, Faraday merges into the existing table using the referenced dataset's reference_key_column.

Supported types

Faraday maps Databricks scalar types (booleans, integers, floats, decimals, strings, dates, timestamps including TIMESTAMP_NTZ, binary as base64 text). TIMESTAMP / TIMESTAMP_LTZ values are exported as TIMESTAMP_NTZ so downstream loads do not see a timezone offset. Arrays, maps, and structs are exported with to_json and stored as JSON text. Oversized decimals above precision 18 are rejected.