Skip to content

Managing Projects with GitHub

Connecting a Project to GitHub version-controls its configuration and lets you promote changes between environments through pull requests, with a clear view of exactly what changed. This page covers connecting a project, what the four sync operations do, and a recommended branching workflow. For how the repository itself is laid out, see Repository Structure; for why the process looks this way, see DevOps Concepts.

GitHub Integration

DataForge Cloud and supported private deployments authenticate to GitHub with the DataForge GitHub App. Users no longer enter or maintain a GitHub personal access token in DataForge.

Before connecting a DataForge project to a GitHub repository, install the DataForge GitHub App on the GitHub organization that owns the repository. During installation, grant the app access to the specific repositories that DataForge should manage. GitHub's detailed install steps are available in Installing a GitHub App from a third party.

The person installing the app must have permission to install GitHub Apps for the organization or repository. If GitHub shows Request instead of Install, an organization owner must approve the request.

After the app is installed, open a project and navigate to the Git tab. The connection is configured per project:

Choose the GitHub owner, repository name, branch, and path you want the project to connect to. All four settings are required. The path must identify a dedicated directory within the repository, such as dataforge/default: | Field | What it holds | |---|---| | GitHub Organization/Owner | The organization or user that owns the repository. | | Repository name | The repository this project syncs with. | | Git branch | The branch this project reads and writes. The dropdown lists branches that already exist in the repository — to connect to a new branch, create it in GitHub first. | | Path | The folder that holds the Project export — dataforge in the recommended layout. Required. DataForge reads and writes only this folder and leaves the rest of the repository alone. | | Commit SHA | Read-only. The commit the project last synced with. Blank until the first sync — and while it is blank, only the Initial operations described below are available. |

  • Enter a repository-relative path without a leading slash.
  • Nested directories are supported and recommended when a repository contains multiple DataForge projects.
  • Do not use the repository root (.), an absolute path, or parent-directory segments (..).

DataForge recursively manages YAML files under this path. During a push, YAML files in the directory that are not part of the DataForge project can be deleted. Keep unrelated YAML files outside the configured directory.

If DataForge cannot retrieve branches, confirm that the DataForge GitHub App is installed for the organization and has access to the selected repository.

Utilize features like push and pull to automatically send the latest changes to GitHub for further review and pull request/merging. Options exist for regular and force pushes and pulls. User information is passed in the commit message to GitHub for granular tracking back to the DataForge user. If the branch dropdown stays empty, confirm that the DataForge GitHub App is installed for the organization and has access to the selected repository.

Branch and path are set per project, so every project in a workspace can share one repository while sitting on different branches — a developer's project can track a feature branch while the Default project tracks the integration branch, and neither disturbs the other. Keeping all of a workspace's projects on a single repository is the recommended convention.

Note

Changing the owner or repository clears the branch and the last-synced commit, and changing the branch disconnects the project from its previous branch's history. After either change, re-baseline the project with Initial Pull (or overwrite the branch with Initial Push) before returning to the regular operations.

What Push and Pull do

The Git tab offers four operations. The regular pair reconciles the project and the branch; the Initial pair overwrites one side with the other.

Operation What it does
Push Commits the project's changes to the branch — one commit per Push, with the project and your DataForge user recorded in the commit message. Commits made on the branch since the last sync (hand edits, merged pull requests) are merged in rather than overwritten, and the merged result is imported back into the project, so the project and the branch end the operation identical. A Push with nothing to commit reports the branch up to date and stops.
Pull Imports the branch into the project. If the branch has no new commits since the last sync, Pull does nothing and reports the project up to date.
Initial Push Overwrites the branch's path folder to match the project exactly, deleting files for objects the project does not have. Use it for the first-ever Push on a new connection, and to deliberately force a branch to match the project.
Initial Pull Replaces the entire project with the branch contents, deleting objects the branch does not have. An import that would delete sources pauses with a warning — see Importing a Project. Use it for the first-ever Pull, and to re-baseline after switching the project to a different branch.

A Pull that imports — regular or Initial — is a full replacement of the project by the branch. Changes made in DataForge but never pushed exist nowhere in Git, so the import overwrites them. Pull into a project before you make changes, never after.

Every operation appears as a row on the project's Imports tab, which holds the log when one fails.

Merge conflicts

Push and Pull reconcile the two sides with a real Git merge. If the same object changed on both sides since the last sync, the operation fails and the import log lists the conflicting files. Nothing is lost: the project's changes are preserved on a temporary branch named <branch>-<workspace>-<import id>, and the error message names it. See Troubleshooting for the resolution options.

If you already use Git

The names match git push and git pull, but the mechanics are broader. Push stages, commits, pushes, and merges in one server-side step — there is no local clone, and the commit message is written for you. Pull is not git pull into a working copy: unpushed DataForge changes are not in Git, so there is nothing to merge them with — an importing Pull makes the project match the branch exactly. The Initial variants are deliberate overwrites, closer to a force-push and a hard reset.

Branching Strategy

Projects are not tied to any specific GitHub object, so any workflow is viable. DataForge recommends Gitflow. Starter links:

The setup below is a simplified Gitflow for two environments — a Dev workspace and a Prod workspace, each with its own Default project — plus a project for each developer. This is the two-workspace topology from Environments and Workspaces; the same branch model drives the single-workspace topology, with projects standing in for workspaces.

If you have only used Git solo, the reason for two long-lived branches may not be obvious. main holds what production is running. next holds everyone's finished work, combined and tested together before a release — it exists because two changes can each be correct alone and still conflict when combined. Feature branches hold work in progress, one branch per change.

One idea here is new even to experienced Git users: a branch is not just history, because a live project imports from it. Merging a pull request into next or main is the first half of a deployment — the matching project's Pull is the second half.

Branch Represents Connected project Who writes to it
main Production Default project in Prod Nobody directly — only merges from next
next Integration Default project in Dev Nobody directly — only merges from feature branches
feature/<name> One developer's in-progress work That developer's own project in Dev The developer, via Push

Note

Older repositories may use master as the production branch name instead of main; the roles are identical.

Nobody pushes directly to main or next — changes reach them only through a reviewed pull request. Enforce this with branch protection rules — GitHub settings that block direct pushes to a branch and require an approved pull request to merge — and lock the Default projects (Lock Project on the project) so they change only by Pull, never by hand. Protection guards the branch; locking guards the project. Together they close both doors.

Developer Workflow

With native Push and Pull, the whole cycle happens in the DataForge UI and GitHub — no local Git client or manual zip files required.

  1. Create a feature branch from next in GitHub, point your developer project at it, and Initial Pull to baseline.
  2. Build and test in your project — its isolation Variables keep it from colliding with the Default project.
  3. Push, and open a pull request into next. GitHub defaults the base branch to main — change it.
  4. After the merge, Pull next into the Default Dev project and test the integrated result.
  5. Promote with a pull request from next into main, then Pull main into the Default Prod project.

Three companion pages carry the detail:

  • Setting up the GitHub Workflow — the one-time setup: repository, branches, protection, locked Default projects, developer projects with their Variables, and Databricks Git folders for custom notebooks.
  • GitHub Developer Workflow — each step above with exact UI paths and verification points, plus the hotfix procedure and a troubleshooting table.
  • Releases and Rollback — the release checklist, deployment order, rollback options, and automating the production Pull with the External API.