7.0 notes

Upgrade Process

Upgrade to 6.2.4:

Everyone needs to upgrade to 6.2.4 before moving to 7.0

Before running 6.2.4 TF: Need to add terraform variable of "upgradeTo624" and value of "yes", change manualUpgradeVersion variable to "6.2.4"

Run Terraform and upgrade to 6.2.4

Upgrade to 7.0:

1) Create new pools using 11.3 LTS and same settings otherwise to have ready for after the upgrade

2) Update TF variables: manualUpgradeVersion set to "7.0.0"

3) Update TF variables: whitelist add "dataforgelabs.com"

4) Add TF variable for "environment_id" with value found in the public.environment table in release database

5) Update Terraform Databricks permissions 5a) For clients that only use TF cloud (5a1) Joe/Reese use 7.0 code on their local machine and make TF cloud connection to the client workspace being upgraded by adding this block to the code (replace organization and name values)

terraform {
  cloud {
    organization = "westmonroepartners"
    workspaces {
      name = "AWS-Development"
    }
  }
}

(5a2) run this command in terraform cli: rm .terraform.lock.hcl (5a3) run this command in terraform cli: terraform init -upgrade

(5a4) run this command in terraform cli: terraform state replace-provider databrickslabs/databricks databricks/databricks, enter yes

(5a5) run this command to make sure it worked: terraform init -upgrade

(5a6) verify state is upgraded in TF cloud (5a7) Continue upgrade in TF cloud as normal

(5b1) For client that use local TF (qf, inmoment), they just need to do step (5a1-5) on their own in their local TF env

Add red note on release for https://wmpartners.atlassian.net/browse/DEV-3888

Add red note on release for deprecating Sparky Pool (no longer managed). Any custom clusters created in Databricks attached to sparky pool should be updated at some point because it will stay on 9.1 LTS.

Once terraform is ran once, add a variable called "rotateSparkyKeys" with value set to "no".

Match Terraform Version for chocolatey install to the Terraform version the client is on.

Check Databricks mount_job

Internal

Need to update DataForge Usage connections for any environments that upgraded to 7.0 and change database name in the connection to "metastore"

SaaS Deployment