Skip to content

Setting Up DataForge Compute Configuration with Databricks Custom Notebook

Ingestion, Parse, and Post Output can all be configured to run on your custom created notebooks by using the Compute configuration page.

Finding the Notebook Path

Before creating a Compute Configuration, find the notebook path. In the Databricks UI, open the notebook and hover over its name to reveal the full path. Copy it — include the leading slash.

Creating the Compute Configuration

Navigate to the Compute page, click New +, then set Job Task Type to Custom Notebook. Paste the copied notebook path into the Notebook Path field. For libraries beyond the DataForge SDK, add them under Parameters > Job Configuration > Libraries.

The Compute Configuration points toward the Custom Notebook

Attaching to a Source for Custom Ingestion

  1. Navigate to the Source Settings page where the Custom Ingestion will be used, or create a new Source.
  2. Change the Connection Type to "Custom".
  3. Locate the "Custom Ingest Compute Config" dropdown and select your newly created Compute Configuration.
  4. Click save.

The Custom Ingest Compute Config applied to a Source

The source will now use the notebook for ingestion. Test with Pull Now or by setting a schedule.

Attaching to a Source for Custom Parse

  1. Navigate to the Source Settings page where the Custom Parse will be used, or create a new Source.
  2. Change the Connection Type to "File" (custom parse only runs for file based sources)
  3. Change the Parser option to "Custom"
  4. Locate the "Custom Parse Compute Config" dropdown and select your newly created Compute Configuration.
  5. Click save.

The source will now use the notebook for parsing. Test with Pull Now or by resetting parse for an input.

Attaching to an Output for Custom Post Output

  1. Navigate to the Output Settings page where the Custom Post Output will be used, or create a new Output.
  2. Locate the Post Output Commands option and select "Custom Notebook".
  3. Locate the "Custom Post Output Compute Config" dropdown and select your newly created Compute Configuration.
  4. Click save.

The Custom Ingest Compute Config applied to a Source

The output will now run the notebook after each output process. Test with Reset Output on the Mapping page or by resetting a specific channel.