Releases and Rollback

A release is the promotion of integrated work into production: a reviewed pull request from next into main, followed by a Pull on the production Default project — and, when notebooks changed, an update of the production Databricks Git folder. The developer workflow covers the mechanics; this page covers doing it deliberately — what to check first, who approves, how to undo one, and how to automate the last step.

Before the release

A merge carries only Project configuration. Workspace objects, Variable values, and notebooks all travel outside it, so a release can be perfectly reviewed and still fail on import. Confirm before merging into main:

• Every new workspace object (Connection, Schedule, Compute Configuration) exists in the production workspace, with exactly matching names.
• Every new Variable has a value in the production Default project.
• The production Databricks Git folder is ready to pull any new notebooks.
• Both workspaces are on the same major DataForge version.
• The change was tested in integration — pulled into the Default Dev project and run, not just merged.
• The production project is exported as a restore point — a zip import is the fastest way back if the release must be backed out.

Deploy in order

For a release that includes custom notebooks, order matters, and configuration goes first:

1. DataForge first. Pull main into the production Default project. Confirm the import completed on the Imports tab before continuing.
2. Databricks second. Update the production Databricks Git folder to main, so the notebooks match the configuration that now references them.
3. Verify. Confirm the promoted sources are active and a pipeline run succeeds end to end.

Never reverse the first two. A notebook promoted ahead of its configuration runs against configuration that has not arrived; configuration promoted ahead of its notebook leaves the pipeline calling code that does not exist yet. When only configuration changed, step 2 disappears and the release is a single Pull.

Review and approval

Two gates, sized differently, keep the process light where it can be and careful where it must be:

Pull request	Suggested approvals	What review checks
feature/* into next	One peer	Only the expected objects changed; no surprise deletions; no hardcoded name where a Variable belongs.
next into main	Two — for example the team lead and the platform owner	The pre-release checklist above, plus testing evidence and rollback notes.

Enforce the gates with branch protection on both branches: require a pull request before merging, require the approvals, dismiss stale approvals when new commits arrive, and allow no bypass on main.

A pull request template turns the release questions into a form nobody has to remember. Commit one at .github/pull_request_template.md:

## Summary of changes
<!-- What changed, and why. -->

## Change type
- [ ] DataForge configuration only (`dataforge/`)
- [ ] Notebook code only (`dataforge-sdk/`)
- [ ] Combined — configuration and notebooks promoted together in this PR

## Release fields (required for pull requests into `main`)
- **Testing evidence:** <!-- link to the integration run, test output, or screenshots -->
- **Rollback notes:** <!-- how to undo this change if production breaks -->
- **Change record:** <!-- ID from your change-management system, if your organization uses one -->

## Reviewer checklist
- [ ] Base branch is correct: `next` for integration, `main` for production
- [ ] No unexpected deletions in the diff
- [ ] No hardcoded value that should be a Variable
- [ ] New workspace objects are recorded for creation in production

If your organization runs formal change management (ServiceNow or similar), the change-record field ties each release to its approval, and the reviewers confirm the record is in its approved state before merging.

Rolling back

Write the rollback plan into the pull request while the change is fresh — the rollback notes field above — because the options differ by what shipped.

Configuration. Two paths:

• Revert in Git (preferred). In GitHub, revert the release merge on main — GitHub's Revert button opens a pull request that undoes it. Merge the revert, then Pull main into the production Default project. The branch and the project stay in agreement, and history records both the release and its reversal.
• Restore from the pre-release export. If you saved an export before the release (the checklist above), import the zip on the production project. A completed zip import can also be rolled back later from its row on the Imports tab — an option Git-based imports do not offer, which is exactly why the pre-release export is on the checklist. After restoring, still revert main in Git: until the branch matches, the next Pull would re-import the configuration you just backed out.

Notebooks. Point the production Databricks Git folder at the previous commit, or let the reverted main carry the notebooks back with it.

Data. Restoring configuration does not un-write data a bad release produced. For Delta tables, Databricks time travel (RESTORE TABLE ... TO VERSION AS OF) can return an output table to its pre-release state — use it with care, and note in the rollback that downstream consumers may have read the interim data. Re-run the affected outputs once configuration is restored.

For an urgent forward fix instead of a rollback, use the hotfix procedure.

Automating the release

Everything before the merge is review; everything after it is mechanical — which makes the production Pull the natural first thing to automate. The External API exposes the same Pull the Git tab runs, so a GitHub Actions workflow can deploy on every merge to main.

Setup, once:

1. Ask a DataForge administrator to create a machine user with credentials, per the External API page. Grant it Editor on the production Default project — Git operations are imports, so the project's lock does not block them.
2. Store the generated client_id and client_secret as GitHub Actions secrets.
3. Commit the workflow below to .github/workflows/ — it requests a token, starts the Pull, and polls the import until it passes or fails.

name: Deploy to production

on:
  push:
    branches: [main]

jobs:
  dataforge-pull:
    runs-on: ubuntu-latest
    # Optional: tie this job to a GitHub Environment with required
    # reviewers, so a named person releases each deploy.
    environment: production
    steps:
      - name: Pull main into the production project
        shell: python
        env:
          BASE_URL: https://<your-api-hostname>
          CLIENT_ID: ${{ secrets.DATAFORGE_CLIENT_ID }}
          CLIENT_SECRET: ${{ secrets.DATAFORGE_CLIENT_SECRET }}
          PROJECT_NAME: Default
        run: |
          import os, sys, time, urllib.request, urllib.parse, json

          base = os.environ["BASE_URL"]

          def call(method, path, body=None, token=None):
              req = urllib.request.Request(base + path, method=method)
              req.add_header("Content-Type", "application/json")
              if token:
                  req.add_header("Authorization", f"Bearer {token}")
              data = json.dumps(body).encode() if body is not None else None
              with urllib.request.urlopen(req, data=data, timeout=30) as res:
                  return json.load(res)

          token = call("POST", "/m2m/token", {
              "client_id": os.environ["CLIENT_ID"],
              "client_secret": os.environ["CLIENT_SECRET"],
          })["access_token"]

          project = urllib.parse.quote(os.environ["PROJECT_NAME"])
          start = call("PUT", f"/m2m/git/pull?project_name={project}", token=token)
          import_id = start["import_id"]
          print(f"Started import {import_id}")

          while True:
              status = call("GET", f"/m2m/imports/{import_id}/status", token=token)
              print(f"status: {status['status_code']}")
              if status["status_code"] == "P":
                  sys.exit(0)
              if status["status_code"] == "F":
                  print(status.get("error"), file=sys.stderr)
                  sys.exit(1)
              time.sleep(10)

A failed run points at the same import log the UI shows — open the production project's Imports tab for the detail, fix the cause (most often a missing workspace object or an empty Variable), and restart the import from that row or re-run the workflow.

Automate in the same order you stabilized manually: keep the Pull manual until releases are routine, then adopt the workflow above, then extend it — a smoke test against GET /m2m/sources after the Pull, checks on pull requests that flag deletions or brand-new workspace-object names before a human reviews. Automation makes a settled process faster; it makes an unsettled one fail faster.