Skip to content

10.0 Upgrade (Azure)

Pre-Upgrade Process

DataForge

All cluster configurations will automatically be upgraded to Databricks runtime 16.4 LTS, including custom notebooks. This is a change from previous upgrade behavior.

Azure

DataForge is officially transferring support from Hive metastore to Unity Catalog. This requires that your Databricks workspace be on the "Premium" pricing tier in Azure before you are able to upgrade to version 10.0.

For more information on upgrading the pricing tier, visit the official Azure Databricks documentation.

Databricks

The 10.0 deployment container connects to a dataforge catalog in your Databricks workspace. This catalog must be created before upgrading — it will host all refining stages and hub tables after sources are migrated to Unity Catalog. Deployment also creates an ember federated catalog for querying DataForge metadata — do not create this yourself, but you do need to grant permissions for deployment to succeed.

To use a catalog name other than "dataforge", contact DataForge support.

To create the catalog, specify a storage location. Any location works, but using the mnt_datalake storage location from the Terraform Quickstart is simplest.

Follow the steps below to create an empty dataforge catalog and assign all required permissions.

  1. Identify the user or service principal that is used to run jobs from DataForge. An easy way to identify this is to open Databricks, navigate to Jobs & Pipelines, open the Job runs tab, and identify the user/principal listed in the "Run as" column for any DataForge job runs. Make note of this user/principal as you will need to assign them multiple permissions.

  2. Create a new Catalog named "dataforge"

  3. If a Storage location does not already exist, follow these steps before moving attempting to create the catalog:

  4. Follow steps 1-4 to Configure a new managed identity for Unity Catalog: https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/azure-managed-identities#config-managed-id
  5. Create a storage credential that access Azure Data Lake Storage: https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/storage-credentials#-create-a-storage-credential-that-accesses-azure-data-lake-storage
  6. Create an external location using catalog explorer. Recommended to use the Copy from DBFS option to copy from "mnt/datalake" (if you've used the Terraform Quickstart): https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/external-locations#-option-1-create-an-external-location-using-catalog-explorer

  • Use "Standard" catalog type
  • Can use any storage location, but recommended using "mnt_datalake" (if you've used the Terraform Quickstart)

3.Once the catalog is created, grant permissions on the catalog for the DataForge authorized user or service principal of "ALL PRIVELEGES".

  1. Open the Catalog page in Databricks and click the gear icon and select the metastore assigned to your Databricks workspace. Navigate to the Permissions tab and assign the following permissions to your DataForge authorized user or service principal:

  2. MANAGE ALLOWLIST

  3. CREATE CONNECTION
  4. CREATE CATALOG

Upgrade Process

After completing the pre-upgrade steps, follow the standard upgrade guide, then proceed to post-upgrade.

Azure Upgrade Process

Post-Upgrade Process

Confirm the environment is alive and working as usual. Confirm custom notebooks still succeed with the updated Databricks runtime. Submit a support request if something is not working as intended.

Begin migrating sources from hive_metastore to Unity Catalog format.