Environments and Workspaces

An "environment" — a place where a stage of your platform runs, such as development or production — is something you assemble in DataForge from two building blocks. Which assembly to choose is the first DevOps decision a team makes, and this page lays out the two standard options, how to pick, and how to move from one to the other later.

The two building blocks

A workspace is the infrastructure boundary. Each DataForge workspace has its own compute and storage targets, its own users and workspace roles, and its own set of workspace-level objects: Connections, Schedules, Outcrop (the DataForge agent that runs inside your network, shown as Agent in the DataForge UI), and Cleanup, Compute, and Process Configurations. Workspace objects are not part of a Project export — anything you promote into a workspace expects them to already exist there, with matching names. A workspace also upgrades as a unit, and Project configuration only moves between workspaces on the same major DataForge version.

A Project is the configuration boundary. Each Project holds its own sources, outputs, templates, relations, and Variables, publishes its hub views under its own schema name, and connects to its own Git branch. Projects in one workspace share the workspace's infrastructure but cannot see each other's configuration.

Every topology below uses the same branch model: main is production, next is integration, and feature branches hold work in progress. What varies is only which Project, in which workspace, pulls each branch.

Two workspaces (recommended)

The recommended topology separates production into its own workspace:

Environment	Workspace	Project	Branch
Production	Prod workspace	Default project (locked)	main
Integration	Dev workspace	Default project (locked)	next
Development	Dev workspace	One project per developer	feature/<name>

Production gets a hard boundary: its own credentials and Connections, its own compute, its own user list, and no developer activity sharing its resources. A mistake in development — a runaway cluster, a mangled Connection, a deleted Schedule — cannot touch production, because those objects exist separately in each workspace. Access can differ too: many teams grant most developers no Prod workspace access at all.

The price is duplication. Every workspace object must be created in both workspaces (deliberately, with matching names), notebooks need a Databricks Git folder per workspace, and there are two workspaces to administer and upgrade. Setting up the GitHub Workflow walks through this topology end to end.

Note

During version upgrades, upgrade Dev first and test, then upgrade Prod. Promotion pauses while the two workspaces are on different major versions, so plan releases around the upgrade window.

One workspace: Projects as environments

Teams starting out — or running a small platform — often have a single workspace. The same branch model still works, with Projects standing in for workspaces as virtual environments:

Environment	Project	Branch
Production	Default project (locked)	main
Integration	An integration project (locked)	next
Development	One project per developer	feature/<name>

Each Project publishes hub views under its own schema name (set when the Project is created and fixed afterwards), so consumers query production's schema while integration and development write elsewhere. The developer cycle, reviews, and promotion work exactly as in the two-workspace topology — merging into main and pulling it into the locked Default project is the production deployment.

Because every Project shares the workspace's infrastructure, the separation is only as good as the Variables that enforce it. Give every non-production Project its own values for:

• Output Schema Name / Output Table Name / Output Connection — so integration and development never write to production tables.
• Schedule — production runs on the real Schedule; integration and development use an on-demand Schedule and pull data when a person decides.
• Custom Compute Configuration — so test runs use development-sized compute, and so custom notebooks resolve per environment (below).

What a single workspace cannot give you:

• Workspace objects are shared. Editing a Connection or Schedule affects every environment at once — there is no "dev copy" of a Connection to break safely. Treat workspace-object changes with production care, and restrict who holds the Power User role.
• One infrastructure pool. A heavy development run competes with production for the same compute and lands in the same data platform, separated by schema rather than by account.
• One failure domain and one version. Maintenance, upgrades, and outages hit all environments together, and nothing can be tested on a new DataForge version before production is on it.
• Ingestion multiplies. Each live Project ingests its own copy of the data it runs. Keep sandbox and integration Projects on on-demand Schedules, and pull into them only when testing calls for fresh data.

Custom notebooks in one workspace

A Compute Configuration holds one notebook path, and Compute Configurations are workspace objects shared by every Project. To run the main version of a notebook in production while integration runs the next version, create one Databricks Git folder per branch (each with sparse checkout of dataforge-sdk, per Repository Structure), create a Compute Configuration per environment pointing at the matching folder's notebook path, and select between them with a Custom Compute Configuration Variable on each Project.

Setup differences from the two-workspace walkthrough

Follow Setting up the GitHub Workflow with these substitutions:

1. Step 3 — seed main from this workspace's Default project (it is the production Project).
2. Step 4 — instead of a Dev workspace's Default project, create a new integration project, connect it to next, and Initial Pull. Set its isolation Variables before anything runs in it.
3. Step 5 — lock both the Default and the integration projects; protect both branches.
4. Steps 6–7 — unchanged: a project per developer with isolation Variables, and a Databricks Git folder per branch if notebooks are in play.

Choosing between them

	Two workspaces	One workspace
Isolation of production data and credentials	Hard boundary	Schema names and Variables
Blast radius of a development mistake	Contained to Dev	Potentially the whole platform
Workspace objects (Connections, Schedules, compute)	Duplicated per workspace, names matching	Shared — one edit affects every environment
Access control for production	Separate user list and roles	Same user list; Project roles and locking
Upgrades	Staged: Dev first, then Prod	All environments at once
Cost and administration	Two of everything	One of everything
Suits	Established teams, regulated data, production SLAs	Small teams, new implementations, cost-sensitive platforms

Both are legitimate destinations, not just stops on a path: plenty of teams run the single-workspace model permanently. Start with what matches your team's size and risk, and let the process — branches, reviews, promotion — stay identical either way, which is what makes the move between them cheap.

Growing from one workspace to two

Because the repository already holds the whole configuration, standing up a segregated production workspace later is mostly infrastructure work:

1. Deploy the new Prod workspace on the same major DataForge version as the existing one.
2. Recreate the workspace objects production needs — Connections, Schedules, Outcrop, and Cleanup, Compute, and Process Configurations — with exactly the names the configuration references. Point them at production credentials and compute. Consult the existing workspace's objects as the checklist.
3. Grant the DataForge GitHub App access for the new workspace's use of the repository, per Managing Projects with GitHub.
4. Connect the new workspace's Default project to main with path dataforge, populate its Variables with production values, and run Initial Pull.
5. Move the notebooks — create the Prod workspace's Databricks Git folder on main and create the Compute Configurations that point at it.
6. Cut over consumers and retire the old production Project. Once outputs from the new workspace are verified, disable ingestion on the old Default project and repurpose it (it makes a natural integration project — unlock it, connect it to next, and re-point its Variables at development values).

The old workspace becomes the Dev workspace of the two-workspace topology, and nothing about branches, reviews, or the developer loop changes.

More than two environments

Some teams add a stage between integration and production — user acceptance testing, or a performance environment. The pattern extends by adding a long-lived branch per stage and keeping promotion linear:

• Branches: feature/* → next → uat → main, each promotion a reviewed pull request.
• Each added branch is pulled by one more locked Project — in its own workspace for hard isolation, or in an existing workspace as another virtual environment.

Add stages only when a concrete need appears; every stage adds a promotion step, a set of Variables or workspace objects, and another place data must be refreshed.