DataForge 11.0 Breaking Changes and Upgrade Preparation

Read this guide before scheduling your upgrade from 10.3.1 to 11.0. Some changes require preparation while the environment is still on 10.x. Share the relevant sections with your DataForge administrator, notebook developers, Databricks administrator, and owners of external integrations.

For DataForge Cloud, coordinate preparation and the maintenance window with DataForge Support. For self-hosted environments, include these checks in your upgrade plan. Do not assume an application downgrade will undo configuration or database migrations; agree on backups and a recovery plan before the upgrade.

Before-upgrade checklist

Change	Who is affected	Action before upgrading
Project import/export format 2.2	All environments using project exports or Git promotion	Pull important projects while all environments are on 10.x; preserve exports and Git revisions; coordinate promotion across versions.
Legacy Hive processing removed	Databricks environments with legacy Hive sources	Convert legacy sources to Unity on 10.x, or delete those no longer needed. Remaining legacy sources, including inactive ones, block the upgrade.
DBR 18 and custom notebooks	Databricks rules, notebooks, and custom libraries	No pre-upgrade testing is possible. Let rule, notebook, and library owners know processing moves to DBR 18 and custom notebooks run through Sparky, and plan time to test after the upgrade.
Required SQL warehouse	All Databricks deployments	Prepare an accessible serverless SQL warehouse and configure databricks-warehouse-id.
Hub-view destination checks	Projects with qualified hub-view names or conflicting view destinations	Run the collision check query and resolve duplicate destinations; prepare the destination project's catalog variable for imports.
Metastore and Agent access	External, non-standard PostgreSQL connections and customer-installed Agents	For external metastore connections, verify SCRAM support and credential refresh procedures. Verify Agent auto-update.
MCP discovery changes	Custom MCP clients and scripts	Update callers that depend on the previous table-discovery parameters or removed tool.

Project import/export format 2.2

DataForge 11.0 uses project import/export format 2.2, reflecting the custom-processing and SDK configuration changes. DataForge 10.3.1 uses 2.1. These formats are incompatible: 11.0 rejects standard 2.1 imports, and 10.3.1 cannot import 2.2 exports. Do not push an 11.0 export to a branch that a 10.3.1 environment still needs to import.

Before upgrading:

1. Pull all important Git-backed projects into their intended environments while they are still on 10.x. Resolve pending imports and conflicts.
2. Preserve the Git commit/branch used by each project and a pre-upgrade export. Preserve notebooks and custom libraries separately; a project export is not a notebook or data backup.
3. Coordinate upgrades across development, test, and production. Keep a separate 10.x branch if an environment must remain on 10.3.1.
4. After upgrading, review the migrated configuration and export or push a new 2.2 baseline before resuming promotion between 11.0 environments.

Do not change the format number in meta.yaml to bypass compatibility checks. Changing the label does not convert the configuration.

Legacy Hive processing removed

Databricks only. DataForge 11.0 removes the legacy Hive source-processing path and the Convert to Unity workflow. Deployment stops before database migration if legacy Hive sources remain.

Convert these sources while still on 10.x, then verify processing and downstream consumers before scheduling 11.0. Use the existing 10.0 AWS upgrade guide or 10.0 Azure upgrade guide for platform-specific migration preparation. Contact DataForge Support if any source cannot be converted. If a legacy Hive source is no longer needed, you can delete it instead of converting it to Unity.

Inactive sources count

The deployment check includes inactive sources. Every legacy Hive source, active or inactive, must be converted or deleted, or the deployment will fail.

This removal concerns DataForge's legacy processing/storage model. Delta connections and outputs can still explicitly target hive_metastore through the supported catalog-aware path. Delta table connections require an explicit catalog; the old databricks-db-name setting is removed.

DBR 18 and custom notebook changes

Databricks only. DataForge 11.0 moves processing to Databricks Runtime 18, Spark 4.1, Scala 2.13, and Java 21. The Python SDK requires Python 3.12 or later. Rebuild customer Scala libraries for Scala 2.13 and compatible Spark/JVM dependencies; libraries compiled for Scala 2.12 need review and replacement.

Custom ingestion, parsing, and post-output notebooks now run through Sparky on the configured processing compute, using dbutils.notebook.run, instead of separate custom-notebook jobs. Upgrade migration moves legacy notebook paths and base parameters into the source/output operation settings and converts applicable custom compute configurations. Review the resulting Process Configurations, notebook paths, parameters, and client libraries, especially where post-output steps previously used different compute.

These changes cannot be tested on 10.x. After upgrading, test every important custom ingestion, custom parse, and custom post-output path with the matching 11.0 SDK before resuming schedules. Check SDK session initialization, dependencies, notebook access permissions, and timeout behavior. Use the matching DataForge SDK examples when updating notebook code. The Scala SDK is embedded in Sparky; avoid reattaching an older SDK JAR to processing compute. Independently managed development clusters need matching SDK libraries.

ANSI SQL mode changes unsafe conversions

Spark 4 enables spark.sql.ansi.enabled by default. Rules and custom notebooks that previously tolerated invalid casts or arithmetic overflow can now fail. For example, converting a nonnumeric string to an integer can raise an error instead of returning NULL.

Review these expressions and handle invalid data explicitly, for example with try_cast where a NULL result is intended. As a compatibility workaround, add this entry to the affected cluster's spark_conf:

{
  "spark.sql.ansi.enabled": "false"
}

Merge the entry into the existing configuration and restart or recreate the affected compute for cluster settings to take effect. This restores permissive behavior for that setting; it does not undo other Spark 4 changes. Cluster settings do not configure the separate SQL warehouse used for rule validation and data queries; test those workflows too.

Review the Spark 4.1 SQL migration guide, including the Spark 3.5-to-4.0 section, for custom SQL and JDBC behavior changes beyond ANSI mode.

Code-cache exhaustion can slow execution

Some DBR 18 workloads can exhaust the JVM code cache, reducing JIT compilation and causing severe execution slowdowns. Treat this as workload-specific troubleshooting: slow execution alone does not establish code-cache exhaustion or an incorrect query plan.

If driver logs show code-cache exhaustion or a disabled compiler, increase the driver's reserved code cache. A starting point is 1 GiB, adjusted up or down for the workload and driver memory budget:

{
  "spark.driver.extraJavaOptions": "-XX:ReservedCodeCacheSize=1g"
}

Add this to the affected cluster's spark_conf. Preserve existing Java options by appending the flag to the existing value, replacing any earlier ReservedCodeCacheSize flag rather than adding duplicates. Restart or recreate the compute, rerun the same workload, and compare duration and driver logs. This reserves JVM code-cache memory, not additional Spark heap, and is not a guaranteed remedy for every slow query.

Required Databricks SQL warehouse

All Databricks deployments, including self-hosted AWS and Azure. Rule/expression validation, Source Data View, and Talos data queries require databricks-warehouse-id pointing to an accessible serverless Databricks SQL warehouse.

Before upgrading:

1. Create or select the serverless warehouse in Databricks and copy its warehouse ID. We recommend cluster size 2X-Small with scaling set to Min 1 and Max 1; this configuration has worked well across DataForge customer environments.
2. Grant the DataForge Databricks identity permission to use it and read the required catalogs, schemas, and tables.
3. Set databricks-warehouse-id in Global System Configurations. Coordinate service restarts with the upgrade.
4. After upgrading, verify expression validation, Data View, and a Talos data query with that identity.

The dedicated Data Viewer cluster is removed. DataForge does not provision or select a replacement warehouse automatically and does not fall back to a cluster. A missing, invalid, inaccessible, or non-serverless warehouse produces an error in the affected workflows. Retire automation that depends on the old Data Viewer cluster or its settings, including use-default-cluster-as-data-viewer, databricks-data-viewer-id, and databricks-jdbc-uri.

Snowflake continues to use its configured JDBC connection/system warehouse and does not need the Databricks setting. See Global System Configurations.

Hub-view destination checks

Databricks and Snowflake. DataForge 11.0 rejects conflicting view destinations across projects, including conflicts with virtual outputs. Disabled objects still reserve their destinations. Existing collisions or invalid destinations can block the deployment backfill; have your upgrade operator check and resolve them before the upgrade.

Only qualified hub-view names, schema.view or catalog.schema.view, can collide. An unqualified view name is created in the project's own schema, which DataForge keeps distinct per project. In Databricks, this query against the Ember metadata catalog lists qualified hub-view names used by more than one source. Disabled sources are included because they still reserve their destinations. Replace ember if your metadata catalog has a different name.

SELECT lower(hub_view_name) AS hub_view_name,
       count(*) AS source_count,
       collect_list(concat_ws(' | ', project_id, source_id, source_name)) AS sources
FROM ember.meta.source
WHERE hub_view_name LIKE '%.%'
GROUP BY lower(hub_view_name)
HAVING count(*) > 1
ORDER BY source_count DESC

Each row is a collision to resolve. The query compares names as written, so a schema.view name and a catalog.schema.view name that resolve to the same place are not matched; review those by hand. It covers source hub views only; check virtual-output destinations separately.

When importing sources with qualified hub-view names such as schema.view or catalog.schema.view, the destination project needs the reserved view_catalog_name variable of type Source Hub View Catalog. If it is missing, import creates a blank variable and stops: populate it in Project Variables, then retry. A configured destination value is preserved rather than copied from the exporting project. In Snowflake, the catalog is the database.

Ensure the target catalogs/databases, schemas, and permissions already exist. Review dependent readers before changing destinations: existing projects without the variable retain their previous resolution, virtual-output destinations are not overridden by this variable, and old warehouse views are retained after repointing. Stop and drain affected processing before moving destinations.

Metastore and Agent access

PostgreSQL clients and saved credentials

The 11.0 metastore upgrade targets PostgreSQL 18 and moves authentication to SCRAM-SHA-256. Standard DataForge-managed processes and components are already covered by the upgrade, including their database authentication and credential updates. No separate connection changes are required for these managed components.

The following preparation applies only to external, non-standard connections to the DataForge metastore, such as independently configured database clients, custom monitoring or backup tools, and direct integrations. Owners of these connections should verify that their clients support SCRAM-SHA-256 before the maintenance window.

SCRAM conversion itself can reuse the configured password; version-based password rotation is a separate change. AWS SaaS rotates selected metastore credentials with a new platform version. Private AWS and Azure deployments support rotation as an opt-in setting, disabled by default. If rotation applies, update any saved credentials in your external connections using current credentials from your authorized administrator, then verify reconnection after deployment. DataForge-managed credential updates are handled as part of the upgrade.

Customer-installed Agents

Protocol-2.0 Agents use API-issued bearer tokens in 11.0. Verify that customer-installed 10.3 Agents have auto-update enabled and can download and start the matching 11.0 Agent. Arrange a manual upgrade with your administrator if auto-update is disabled; that setup is outside the automatic upgrade path. After deployment, verify the Agent version, heartbeat, and a representative ingestion. External M2M authentication is unchanged by this Agent-specific change.

MCP table-discovery contract changes

Custom MCP clients and scripts only. The DataForge MCP connector and standard MCP clients load tool definitions from the server each time they connect, so they pick up these changes with no action. Only callers that hard-code tool names, arguments, or response shapes need updating. inspect_connection_tables is removed, and find_connection_tables requires an exact connection_id with typed regex filters such as schema_name_regex, table_name_regex, and column_name_regex. Previous conversational include/exclude/rematch parameters are no longer accepted.

Use list_connections or find_connection to resolve the connection first. Use find_connection_tables for name matching and search_connection_metadata for description-based search only when populated description vectors are available. Update hard-coded arguments and response parsing, then test discovery and source creation before resuming automation.

After upgrading: migrate eligible Delta tables

Recommended for existing managed Databricks sources. Run Migrate Delta Tables after the upgrade to adopt the new layouts and reduce unnecessary automatic clustering. This is a performance and cost recommendation, separate from the mandatory pre-upgrade legacy Hive conversion. Benefits depend on workload and current layout.

1. Find sources whose Delta Migration status is Needs migration.
2. Run Migrate Delta Tables from a source's Inputs tab or the source-list Bulk Update action. Eligible refresh types are full, key, custom refresh, none, timestamp, and sequence. Unmanaged refresh and unmanaged external connections are excluded; already-migrated sources are skipped.
3. Allow existing source work to drain. Migration pauses source initiation; successful completion restores its prior state. If it fails, the source remains paused: investigate and retry before resuming processing.
4. Confirm Migrated status and test processing and downstream readers.

Full-refresh migration physically partitions ORE, MINERAL, and ALLOY by s_input_id. Other supported refresh types reconcile clustering metadata without copying data. Enable Cluster By Auto is an INGOT-only source option, disabled by default; enable Databricks Predictive Optimization before creating or migrating an opted-in INGOT table. Migrate existing eligible tables before disabling Predictive Optimization where automatic clustering is no longer needed.

Post-upgrade acceptance checklist

• Verify ingestion through every Agent and important custom notebook, plus parsing, rules, refresh, output, and cleanup.
• Test representative rules and custom ingestion, parse, and post-output notebooks on DBR 18 with the matching 11.0 SDK. Watch for ANSI-mode failures on invalid casts or overflow, and confirm migrated notebook paths, parameters, libraries, and compute.
• Verify rule validation, Data View, and Talos queries against the configured SQL warehouse.
• Confirm project Git pull/push and import using the new baseline; check destination variables and notebook paths in each project.
• Verify external, non-standard metastore connections after any required client or saved-credential updates, and test MCP automation with refreshed tool definitions.
• Complete eligible Delta migrations in a planned window and confirm paused sources have returned to their intended initiation state.

Resolve failures with your upgrade operator or DataForge Support before resuming normal schedules and project promotion.