Skip to content

Compute Configuration for Custom Processing Steps

Custom processing steps (Custom Ingestion, Custom Parse, Custom Post-Output) run your Databricks notebooks through Sparky on standard DataForge compute. They do not need a dedicated Compute Configuration. The notebook itself is set on the Source or Output with Notebook Path and Repository Path. See Setting Up a Databricks Custom Notebook.

Changed in 11.0

The Custom Ingest Compute Config, Custom Parse Compute Config, and Custom Post Output Compute Config selectors and the Custom Notebook job task type have been removed. The 11.0 upgrade converts existing custom notebook compute that has client libraries into standard compute and assigns it through a generated, source-specific Process Configuration.

Default Compute

By default, a custom step runs on the default Compute Configuration, the same compute as other processing.

Choosing Different Compute

To run a custom step on different compute, for example a larger cluster or one with extra libraries:

  1. Open the Process Configuration assigned to the Source, or create one and assign it.
  2. Under Process Overrides, click ADD OVERRIDE and select the custom process type.
  3. Select the Compute Configuration to use, then save.

Add libraries beyond the DataForge SDK to that Compute Configuration under Parameters > Job Configuration > Libraries. The DataForge SDK libraries are managed for you, so do not attach an older SDK version.