Compute Configuration for Custom Processing Steps

Custom processing steps (Custom Ingestion, Custom Parse, Custom Post-Output) require users to select a compute configuration for each custom step. This custom compute configuration defines compute parameters and specifies the custom notebook that will be executed.

---

Configure Compute for Custom Ingestion Source

Sources configured to use Custom Ingestion (Custom connection type) on the Source Settings page require users to select Compute Configuration for the Custom Ingestion step:

After Custom is selected, a Custom Ingest Compute Config dropdown appears. Select an existing configuration, or use the open in new icon to create one in a new tab. Use the edit icon to modify an existing compute. After creating or editing, refresh the dropdown to see the latest configurations.

---

Configure Compute for Custom Parse Source

Sources configured to use Custom Parse on the Source Settings page require user to select a Custom Parse Compute Config for the Custom Parse step:

---

Configure Compute for Custom Post-Output

Sources configured to use Custom Post Output on the Output Settings page require user to select a Custom Post Output Compute Configuration for the Custom Post Output step: