DataForge Cloud 10.3.1 Version Features Blog¶

DataForge Cloud 10.3.1 is a reliability release. It makes the Talos MCP server more dependable, keeps scheduled ingestion running on its own, makes project imports faster and safe to restart, and cuts full-refresh cleanup time by more than half.

Several changes in this release alter existing behavior. Each one is tagged [BEHAVIOR CHANGE] in the Full Changelog.

Talos MCP Reliability¶

10.3.1 makes the Talos MCP server more dependable over long working sessions.

Identity and permissions. Project Owners are no longer denied access partway through a session or after reconnecting the MCP connector. list_projects returns an empty list instead of null, and access denial messages now say whether your role is insufficient or your identity could not be resolved.

Accurate process status. wait_for_processing now reports the outcome of the current run only, on both source and output paths. Record validation state is returned separately in record_status_codes, and background data profiling is reported in its own field.

Safer mapping edits. update_output_mapping calls that only change column_mappings no longer alter channel-level settings such as operation_type or include_warn_flag. find_connection_tables now honors connection_id.

Compute configurations. create_cluster_config now creates compute configurations with platform defaults populated and the Databricks job provisioned.

Removed tools. The deprecated upsert_source and upsert_output_row tools and their aliases have been removed. Update any agent scripts or saved prompts that call them.

For more information, visit the Setting Up the DataForge MCP Server documentation.

Scheduling and Agent Reliability¶

Scheduled ingestion. Sources that rely only on their schedule could stop ingesting until a manual Pull Now woke the scheduler. Scheduled ingestion now keeps firing on its own.

Agent restarts. A pending agent restart is no longer lost when an agent upgrade is detected at the same time, so a stalled agent recovers without a manual restart.

Pull Now with ingestion disabled. Pull Now through the MCP on a project with ingestion disabled now completes the same way it does from the UI, instead of leaving the input in Loaded status. Scheduled ingestion stays blocked while ingestion is disabled.

For more information, visit the Schedules documentation.

Import and Git Workflow Hardening¶

Asynchronous imports. File imports, Git imports, and import restarts are now queued and return immediately with HTTP 202, so long imports no longer time out on the request. The UI shows the operation as queued and refreshes its status once it is dispatched. The legacy test_flag request path has been removed, and format_spec='core' now fails with a deprecation error.

Safe restart. Only a Failed or Deleted import can be restarted. A restarted import moves to Queued, then to In Progress when processing begins. Restarting an import that is already running returns "Import is already running or is no longer restartable." instead of running it twice.

Faster comparison. Imports now skip the detailed comparison for objects whose definition has not changed, which shortens imports on large projects.

Git pull and import on locked projects. Locking a project no longer disables Pull, Initial Pull, or Import for Editors. Direct configuration editing is still blocked, and the lock toggle tooltip in Project Settings now explains this.

For more information, visit the Managing Projects with Github and Environments and Workspaces documentation.

MCP Relation Path Discovery¶

When a rule or output mapping references a column on another source, DataForge chooses a relation path and stores it in source_relation_ids. The UI lets you see and change that path, and agents can now do the same through the MCP:

get_rule returns the selected path and its alternatives under paths.path[].selections.
create_rule accepts a relation ID or name for each referenced source, so you can pin the path when the rule is created.
find_output with include_columns=true returns the persisted source_relation_ids and alternative hops for each output column mapping.
Output mappings accept a relation_path by exact relation name or ID.

Path selection follows the UI behavior. The chain is truncated at the hop you change, the new relation is chosen from that hop's alternatives, and the expression is parsed again.

For more information, visit the Rules and Relations documentation.

Cleanup and Metadata Performance¶

Full-refresh cleanup. Cleanup now checks each source for stale s_input_id values in batches of up to 25 sources per Spark action, and skips the delete step for sources with nothing stale. Where stale inputs exist, they are deleted from the mineral, ore, and alloy tables. Results are unchanged.

Measured on 140 full-refresh Unity Catalog sources:

	Before	After
Full-refresh cleanup phase	133.2 seconds	52.0 seconds
Spark jobs for the stale-input check	140	6
Total cleanup process	178 seconds	94 seconds

The Sparky driver log now includes per-batch, per-source, and per-table timings.

Connection metadata refresh. Refreshing metadata on a large connection now returns immediately and no longer causes database load spikes. If collection fails, the refresh reports the driver error and leaves existing metadata intact.

Source status updates. Under heavy database load, DataForge now runs at most four source status updates at a time and slows dispatch until the database recovers.

For more information, visit the Cleanup Configuration and Connections documentation.

SaaS User Provisioning¶

On DataForge SaaS deployments, signing in with a single sign-on identity no longer creates a DataForge user account automatically. Only users an administrator has already provisioned can sign in. Add new users in Users before their first sign-in. Self-hosted deployments are not affected.

For more information, visit the Manage Users documentation.

Stricter Configuration Validation¶

The MCP now enforces the same validation as the UI and names the valid values when it rejects a request.

Initiation type. Sources accept only the scheduled and watcher initiation types. Existing sources set to manual are migrated during the upgrade, and watcher sources created through the MCP no longer inherit the default schedule.

Channel operation type. A channel's operation_type accepts only none (shown as N/A in the UI) and Aggregate.

System attributes in rules. Rules and rule templates that reference s_validation_status_code are rejected on save, since that attribute is only available after validation rules run.

Rule datatype casts. Rules created through the MCP now keep their explicit cast_datatype. Legacy cast aliases such as decimal(p,s), numeric, varchar(n), and bigint are normalized during the upgrade to each platform's canonical types:

Platform	Canonical types
Databricks	`decimal`, `string`, `int`, `long`
Snowflake	`DECIMAL`, `VARCHAR`, `INTEGER`

Unsupported values are left unchanged. A batch of rule and relation changes now triggers one recalculation per affected source.

Duplicate relations. When two relations between the same sources share an identical expression, you can now delete or deactivate one of them. Creating a new duplicate is still rejected.

For more information, visit the Rules documentation.

SQL Server datetime2 Output Support¶

SQL Server table outputs no longer fail bulk copy when a timestamp column is mapped to datetime. datetime2 is also now available as a SQL Server output column datatype for higher fractional-second precision and a wider date range. Existing datetime columns are not changed.

For more information, visit the Output Settings documentation.

Full Changelog¶

Added relation path discovery and pinning for rules and output mappings through the MCP, using get_rule, create_rule, find_output, and output mapping relation_path.
Added datetime2 as a SQL Server output column datatype. Existing datetime columns are not changed. [OPTIONAL CHANGE]
Added the TALOS_LOG_LEVEL setting for Talos, defaulting to INFO, and reduced Talos production log volume. Unrecognized values fall back to INFO. [OPTIONAL CHANGE]
Changed file imports, Git imports, and import restarts to queue asynchronously and return immediately. [BEHAVIOR CHANGE]
Removed the legacy import test_flag request path, and changed format_spec='core' imports to fail with a deprecation error. [BEHAVIOR CHANGE]
Removed the deprecated upsert_source and upsert_output_row MCP tools and their aliases. [BEHAVIOR CHANGE]
Changed the project lock so Editors can still use Pull, Initial Pull, and Import, and updated the lock tooltip to explain this. [BEHAVIOR CHANGE]
Changed SaaS deployments to sign in only users provisioned by an administrator, instead of creating accounts on first sign-in. [BEHAVIOR CHANGE]
Changed MCP wait_for_processing to report the current run's outcome, with record validation state in record_status_codes and background profiling reported separately. [BEHAVIOR CHANGE]
Changed MCP validation to reject initiation_type values other than scheduled and watcher, and migrated existing sources using manual. [BEHAVIOR CHANGE]
Changed MCP validation to reject channel operation_type values other than none and Aggregate. [BEHAVIOR CHANGE]
Changed rule and rule template validation to reject references to s_validation_status_code. [BEHAVIOR CHANGE]
Changed legacy rule cast aliases to normalize to platform canonical datatypes during upgrade. [BEHAVIOR CHANGE]
Changed an agent upgrade to consume a pending watchdog restart request. [BEHAVIOR CHANGE]
Improved full-refresh cleanup performance by batching stale-input checks and skipping deletes for sources with no stale inputs.
Improved project import performance by skipping subsection comparisons for unchanged objects.
Improved connection metadata refresh so large refreshes no longer block API requests or cause database load spikes.
Improved database stability by limiting concurrent source status updates under load.
Improved MCP rule and relation batches to issue one recalculation per affected source.
Fixed MCP sessions denying access to project Owners mid-session and after reauthorization, and clarified MCP access denial messages.
Fixed MCP list_projects returning null instead of an empty list.
Fixed MCP update_output_mapping overwriting channel settings when only column mappings were supplied.
Fixed MCP list_dataforge_tools ignoring filters passed inside a nested args object. The nested shape is now rejected.
Fixed MCP find_connection_tables ignoring connection_id.
Fixed MCP create_cluster_config failing with "Could not populate compute configuration defaults".
Fixed MCP-created rules losing their explicit cast_datatype.
Fixed MCP Pull Now leaving inputs in Loaded status when project ingestion is disabled.
Fixed watcher sources created through the MCP inheriting the platform default schedule.
Fixed duplicate relations being impossible to delete or deactivate.
Fixed import restart rerunning an import that was still running after a connection reset.
Fixed scheduled ingestion stopping until an external event woke the scheduler.
Fixed SQL Server table outputs failing bulk copy for timestamp columns mapped to datetime.
Fixed input delete on custom refresh sources clearing record counts on the remaining inputs.