DevOps Concepts¶
This page explains the why behind DataForge's recommended development process. It assumes no prior Git or DevOps experience — every term is defined where it first appears. If your team has always worked in a single shared environment, or "branching" and "pull requests" are terms you have heard but never used, start here before the setup and workflow pages.
Many strong data engineers have never needed a formal DevOps process — plenty of widely used data tools offer no place to put one. None of what follows requires programming background, and almost all of it happens in a browser.
The problem: one shared Project cannot stay safe¶
DataForge holds all configuration in a Project. A Project is one complete codebase: it promotes as a whole, and there is no way to promote part of it.
That single fact creates three pressures the moment more than one person builds in the same Project:
- Everything travels together. A Project holding one finished change and one half-built experiment moves both on the next promotion. Nothing lets you take the first and leave the second.
- No room to explore. Building examples, testing pipelines, and trying unfamiliar features all leave work in the Project. In a shared Project, that work is on the path to production.
- Testing needs stages. Unit testing belongs in development, integrated testing comes next, and production comes last. Each stage needs its own place to run, and one shared Project cannot provide even two.
The goal of the process is to give every developer the freedom to build, test, and break things — with a controlled path that promotes only finished, reviewed work.
Three ideas carry the whole process¶
DevOps here means a version-controlled, reviewed, staged path to production. Each idea is simple on its own, and together they are the entire system:
- Version control. Every change is recorded: what changed, who changed it, and when. Git is the system that does the recording. GitHub is the website that hosts it for the team.
- Review. No change reaches shared configuration without a second set of eyes. The pull request is the page where a teammate sees exactly what would change and approves it.
- Promotion. Changes move through stages: your own sandbox, a shared integration stage, then production. Each stage catches problems the previous one cannot.
The value: freedom to build without fear, a gate before production, and a record of every change that ever shipped.
Git in plain terms¶
Five words carry the vocabulary. If these five make sense, everything in the workflow pages will too.
| Term | Meaning |
|---|---|
| Repository | A folder whose entire history Git tracks. Yours holds all of the platform's configuration and notebooks. |
| Commit | A saved snapshot of changes, with the author and a message. History is a chain of commits. |
| Branch | An independent line of work. Your branch is your sandbox: nothing you do there touches anyone else. |
| Pull request | A request to merge your branch into a shared one, plus the review page where a teammate approves it. |
| Merge | Combining one branch into another once the review approves. This is the moment work becomes shared. |
That is most of it. With DataForge there is no Git client, no terminal, and no local checkout — DataForge and Databricks each connect to GitHub from their own servers, and every step of the process happens in the browser. Nobody installs Git, and nobody clones anything.
What DataForge already handles — and what Git adds¶
DataForge records who changed what and when on its own: the platform's catalog stores both the configuration and the history of every change, and every object records the import that created or updated it. DataForge also defends against two users overwriting each other's edits.
So the process does not exist for an audit trail, and it does not exist to prevent collisions. It exists for the things a live platform cannot provide by itself:
- Selective promotion. Your finished work moves to the shared Project; your experiments stay behind. Git makes this possible by comparing files instead of moving whole Projects — a pull request lists exactly what differs, and a person decides what becomes shared.
- A review gate. Configuration is code. A second reader catches the deleted output, the hardcoded schema name, and the accidental change nobody intended to ship.
- Environments that mean something. A branch per stage, a Project per branch — so "in development", "in testing", and "in production" are enforced states rather than a team's best recollection.
How the pieces map to DataForge¶
| In Git | In DataForge |
|---|---|
| Your branch | Your own Project, connected to it on the Git tab |
| Commit and push | Push — one click; the commit and its message are created for you |
| Bring a branch's state in | Pull — imports the branch into the connected Project |
| Merging a pull request | The first half of a deployment |
| The connected Project's Pull after the merge | The second half |
The last two rows are the idea that surprises even experienced Git users: a branch here is not just history, because a live Project imports from it. Merging is not the end of the story until the matching Project pulls.
Where to go next¶
Read in this order, and stop when you have what your role needs:
- Environments and Workspaces — choose the topology: two workspaces, or one workspace with Projects as environments.
- Repository Structure — the folder layout that keeps configuration and notebooks in one reviewed repository.
- Managing Projects with GitHub — the Git tab, what Push and Pull actually do, and the branching model.
- Setting up the GitHub Workflow — the one-time setup, from empty repository to protected branches.
- GitHub Developer Workflow — the daily loop, step by step, with troubleshooting.
- Releases and Rollback — promoting to production deliberately, undoing a bad release, and automating the last step.