Skip to content

DataForge Documentation

Spark and Databricks

Initializing search

DataForge Documentation
Get started
Build
Run
Reference

DataForge Documentation

DataForge Documentation
Get started
Get started
- Dataforge basics
  Dataforge basics
- Data integration example
  Data integration example
Build
Build
- Sources
  Sources
  - Sources Overview
  - Configuration
    
    Configuration
    
    Source Settings
    
    Raw Schema
    
    Relations
    
    Rules
  - Operations
    
    Operations
    
    Inputs
    
    Process
    
    Dependencies
    
    Viewing Source Data
  - Advanced
    
    Advanced
    
    Complex Data Types
    
    Sub-Sources
    
    Unmanaged External Source
    
    Custom Refresh
- Connections
  Connections
- Processing
  Processing
- Outputs
  Outputs
- Configuration
  Configuration
  - Cluster configuration
    
    Cluster configuration
    
    Compute Configuration Overview
    
    Compute Configuration for Custom Processing Steps
  - Process configuration
    
    Process configuration
    
    Process Configuration Overview
  - System configuration
    
    System configuration
    
    Global System Configurations
    
    Cleanup Configuration
- Cloning
  Cloning
  - Basic example
    
    Basic example
    
    Groups
    
    Performing a Basic Clone
    
    Source Name Templates
  - With connections
    
    With connections
    
    Cloning with Connections Overview
    
    Creating a Multi Group Connection
    
    Creating a Single Group Connection with Connection Name Templates
    
    Performing the Clone
  - With relations
    
    With relations
    
    Cloning with Relations Overview
    
    Example Setup
    
    Configuring a Group Relation Template
    
    Configuring a Master Data Relation Template
    
    Performing a Clone with Relationships
  - With enrichments
    
    With enrichments
    
    Cloning with Enrichments Overview
    
    Templatizing [This] Source and Master Data Relation Rules
    
    Templatizing Group Relation Rules
  - With outputs
    
    With outputs
    
    Cloning with Outputs Overview
    
    Creating a Multi Group Output
    
    Creating A Single Group Output with Output Name Templates
    
    Performing the Clone
- Templates and tokens
  Templates and tokens
- Schedules
  Schedules
  - Schedules
- Lineage
  Lineage
  - Lineage Graphs
- Projects
  Projects
  - Projects Overview
  - Managing Projects with Github
- Sdk
  Sdk
- Talos ai
  Talos ai
  - Talos: Your AI data assistant
- Agents
  Agents
- Users and access
  Users and access
  - Manage Users
Run
Run
- Deployment
  Deployment
- Upgrades and maintenance
  Upgrades and maintenance
  - Version upgrades
    
    Version upgrades
    
    8.2 Upgrade Guide
    
    9.0 Upgrade Guide
    
    Aws
    Aws
    
    8.0.x Upgrade Guide (AWS)
    
    10.0 Upgrade (AWS)
    
    New Version Upgrade Process (Terraform)
    
    Azure
    Azure
    
    8.0.x Upgrade Guide (Azure)
    
    10.0 Upgrade (Azure)
    
    New Version Upgrade Process (Terraform)
  - Maintenance
    
    Maintenance
    
    Maintenance and Versioning
  - Cleanup
    
    Cleanup
    
    Cleanup Help
- Monitoring and alerting
  Monitoring and alerting
  - Process alerts
    
    Process alerts
    
    Process Email Notifications
  - Metadata monitoring dataset
    
    Metadata monitoring dataset
    
    Accessing the DataForge Metastore
  - Spark and databricks
    
    Spark and databricks
    
    Spark and Databricks Overview
    
    Updating Databricks Access Token for Workspace Authentication
- Cost management
  Cost management
  - DataForge Cost Dashboard
  - Production Workload Compute Configuration Recommendations
- Administration
  Administration
  - Dataforge support access
    
    Dataforge support access
    
    Infrastructure Support Access Roles
  - Data collection and privacy
    
    Data collection and privacy
    
    Data Collection and Privacy Overview
  - Help center terms of service
    
    Help center terms of service
    
    Help Center Terms of Service
  - Open source software list
    
    Open source software list
    
    Open Source Software List Overview
  - Private cloud management
    
    Private cloud management
    
    Deployment & Version Management Overview
Reference
Reference
- Architecture
  Architecture
  - Logical
    
    Logical
    
    Data Processing Engine and Steps
    
    Data Storage
    
    User Interface
  - Physical
    
    Physical
    
    Components
    
    Metadata Model
    
    Standard vs. Private Enterprise Deployment Architecture
- Releases
  Releases
- Pricing and billing
  Pricing and billing
  - Legacy Pricing: Processing Units Overview
- Faq
  Faq

Spark and Databricks¶

Spark and Databricks Overview
Updating Databricks Access Token for Workspace Authentication

Accessing the DataForge Metastore

Spark and Databricks Overview