Updating Terraform to use Databricks Service Principal instead of Username/Password¶
Databricks migrated away from supporting databricks-managed password for username and password authentication with the API on July 10th, 2024. To allow Terraform to authenticate with Databricks as part of the upgrade cycle, follow the instructions below to add a service principal and secret.
Databricks¶
Log into the account admin console https://accounts.cloud.databricks.com/
- Go to User Management -> Service Principals tab -> Add service principal
- Under the Roles tab of the service principal, make it an Account Admin and Marketplace Admin
- For the service principal, generate a new OAuth secret
- Copy the secret value (this will be assigned to Terraform variable databricksClientSecret)
- Copy the client ID (this will be assigned to Terraform variable databricksClientId)
- Go to Workspaces within the Account console and click the workspace associated with DataForge
- Under the Permissions tab, click the Add permissions button and assign the Service Principal to the workspace.
Terraform¶
New List of databricks variables in terraform:
- databricksAccountId
- databricksClientId
- databricksClientSecret
- databricksE2Enabled
databricksClientId and databricksClientSecret come from the service principal secret created above in the Databricks steps.
Remove old variables:
- databricksAccountPassword
- databricksAccountUser
- databricksWorkspaceAdminEmail