Repository Structure¶

When you version-control a DataForge platform, a single Git repository holds two kinds of artifact side by side:

DataForge Project configuration — the sources, rules, relations, outputs, and templates that make up a Project.
Custom notebooks — the Databricks (or Snowflake) code that DataForge calls at SDK extension points.

Keeping both in one repository means a change that spans them — a new custom-ingestion notebook and the Source configuration that runs it — travels through a single pull request and deploys as one unit. Split across two repositories, the two halves can merge in either order, and a deployment can land one without the other.

To stand this structure up step by step, see Setting up the GitHub Workflow.

Recommended layout¶

your-repo/
├── dataforge/          DataForge Project export — one YAML file per object
│   ├── meta.yaml
│   ├── defaults.yaml
│   ├── variables.yaml
│   ├── relations.yaml
│   ├── sources/
│   ├── outputs/
│   ├── source_templates/
│   ├── rule_templates/
│   ├── relation_templates/
│   ├── output_templates/
│   ├── groups/
│   └── tokens/
└── dataforge-sdk/      Custom notebooks that DataForge calls
    ├── ingestion/
    ├── parsing/
    └── post-output/

The two top-level folder names are a convention, not a requirement — what matters is that each tool is pointed at its own folder, as described below.

The `dataforge` folder¶

This folder holds a Project export: one YAML file per object, organized into typed subfolders, plus the top-level meta.yaml, defaults.yaml, variables.yaml, and relations.yaml. For the full contents of an export, see Projects Overview → Exporting a Project.

DataForge writes this folder through its GitHub integration. Push exports the Project into the folder as a commit; Pull imports the folder back into the Project. Set the project's Git path to dataforge so DataForge reads and writes only this subfolder and leaves the rest of the repository alone. See Managing Projects with GitHub.

Note

The files under dataforge are generated. Change configuration in the DataForge UI (or the workspace's MCP server) and Push. You can also edit the YAML directly on the branch and Pull it back in — the import validates and cross-checks the files. A regular Push does not discard hand edits either: it merges branch-side commits into the push and imports the result back into the project (only an Initial Push overwrites the branch — see What Push and Pull do). For routine changes, prefer the UI or MCP, which carry more safeguards.

The `dataforge-sdk` folder¶

This folder holds the custom notebooks DataForge runs at its extension points — custom ingestion, custom parsing, and custom post-output — along with any helper code they depend on. People write and edit these notebooks in your data platform.

Note

Custom notebooks are not migrated by Project export/import. See Migrating changes.

DataForge reaches a notebook through a Compute Configuration that holds the notebook path; a Source or Output then references that configuration. Because a Compute Configuration is a workspace object rather than a Project object, the notebook path is specific to a workspace and is never part of a Project export — the two folders stay cleanly separated.

Version-control these notebooks with a Databricks Git folder. Enable sparse checkout — a mode that checks out only the folders you list instead of the whole repository — and set the cone pattern (the include list) to dataforge-sdk, so the Git folder contains only the notebooks and not the DataForge YAML.

Each workspace keeps its own Git folder, pointed at the branch that matches that workspace's DataForge Project.

DataForge reaches a notebook by name. The notebook is a Snowflake object, written and edited in Snowsight, and its name is a setting on the Source (custom ingestion, custom parsing) or on the Output (custom post-output) — see Setting up DataForge with Snowflake Custom Notebook. There is no folder to check out.

Because the name is a Source or Output setting, the name travels in a Project export but the notebook it names does not. Promotion therefore turns on names: create the notebook in each Snowflake account you promote into, under exactly the name the configuration references. When the process runs, DataForge executes the notebook by that name, so a name the account does not have fails at run time.

Keep the notebook's source in dataforge-sdk anyway. DataForge does not sync this folder into Snowflake, but holding the code in the repository gets a notebook change reviewed in the same pull request as the configuration that calls it, and gives every account one copy to work from.

One repository, two sync mechanisms¶

Each tool is scoped to one folder and cannot touch the other's files.

Folder	Synced by	Operations
`dataforge/`	DataForge GitHub integration (Git tab)	Push, Pull
`dataforge-sdk/`	Databricks Git folder (sparse checkout)Your own Git client (not DataForge)	Commit, Push, Pull in DatabricksCommit, Push, Pull outside DataForge

The result is one repository, one branch model, and one review gate, with no tool overwriting another's work. The branching strategy and promotion workflow apply to both folders together.

What is not in the repository¶

Objects that are global to a workspace — Connections, Schedules, Outcrop (the DataForge agent that runs inside your network, shown as Agent in the DataForge UI), and Cleanup, Compute, and Process Configurations — are not part of a Project export and do not appear in the dataforge folder. They must already exist, with matching names, in any workspace you import into. See Projects Overview for the full list and the pre-creation rule.