9.0 Upgrade Guide¶
Pre-Upgrade Process¶
DataForge¶
All cluster configurations will automatically be upgraded to Databricks runtime 15.4 LTS, including custom notebooks. This is a change from previous upgrade behavior.
DataForge and Databricks¶
Databricks runtime 15.4 LTS no longer supports attaching a custom JAR to any job cluster configuration. Be sure to replace any JAR saved in your DataForge cluster configurations with an applicable Maven library and test your custom notebook. If you are not able to replace a JAR and still want to upgrade, you can downgrade your cluster configuration Spark Version to 14.3 LTS to support the previous JAR and test the custom notebook still runs correctly after this upgrade.
Example maven format for cluster configurations: [{"maven":{"coordinates":"<coordinates>"}}]
Terraform (required)¶
Ensure the Terraform workspace contains the following variable for certificate management. Moving forward, all certificates need to be updated directly in Azure and are no longer managed via Terraform. If the certificate name changes in Azure, this variable needs to be updated to avoid issues during following Terraform runs.
- Variable name:
existing_cert_name - Variable value: Type the existing certificate name in the Azure KeyVault
Terraform (optional)¶
Add a Terraform variable to list Azure object IDs that should not have access policies cleared out of the KeyVault after each Terraform run with the following format:
- Variable name:
developer_object_ids - Variable type: Select HCL
- Variable value:
["55555555-5555-5555-5555-555555555555"]or if there are multiple IDs use comma separated like["55555555-5555-5555-5555-555555555555","65555555-5555-5555-5555-555555555555"]
Upgrade Process¶
After completing the pre-upgrade steps, follow the standard upgrade guide, then proceed to post-upgrade.
Post-Upgrade Process¶
Confirm the environment is alive and working as usual. Confirm custom notebooks still succeed with the updated Databricks runtime. Submit a support request if something is not working as intended.