Skip to content

DataForge SDK for Custom Processing

SDK Overview

The DataForge SDK allows you to define your own Python or Scala code in a notebook and attach the notebook for automatic processing in DataForge. The SDK can be used for Custom Ingestions, Custom Parsing, and Custom Post Output processes. Notebooks used for Custom Ingestions and Parsing need to return a Spark DataFrame as a result of the code.

Details of the DataForge SDK can be found in the following documentation.

https://github.com/dataforgelabs/dataforge-sdk

Navigate the repository directories for your respective data platform (Databricks or Snowflake) and your preferred language to visit the Samples and see how you can start using the SDK.

Project Schema Session Parameter

Custom ingestion, custom parsing (including manual parsing resets), and custom post-output sessions expose the owning project's schema in their process parameters:

project_schema = session.process.parameters["project_schema"]

For ingestion and parsing, this is the source's project schema. For post-output, it is the output's project schema, even when an output channel uses a source from another project. DataForge resolves this system parameter when the process is enqueued; configured parameters cannot override it.

When splitting a loopback notebook into separate post-output and ingestion notebooks, the ingestion notebook can read its own project_schema instead of receiving the schema from the post-output session. If the notebooks belong to different projects, each receives its own project's schema.

project_schema is the project schema name, not a fully qualified table prefix. Existing output destination parameters remain unchanged: view_database contains the resolved output schema on Databricks and the database name on Snowflake; an output's view_schema override does not change project_schema.

This parameter requires a database deployment containing this change and a newly enqueued process. Processes queued before that deployment retain their existing parameter snapshot.

Development Approach

Below is the recommended approach to developing a new custom notebook for processing.

Databricks

  1. Create an all-purpose cluster in Databricks and attach the DataForge SDK library
  2. Create a new notebook in Databricks and attach the Databricks cluster
  3. Create a Source or Output in DataForge and set its Notebook Path to the Databricks notebook
  4. If the notebook needs extra libraries or larger compute, assign a Process Configuration override for the custom step
  5. Write your custom code in the Databricks notebook, incorporating any optional SDK parameters
  6. Test the notebook with a new Source data pull (Custom Ingest/Parse) or a new Output (Custom Post Output) to ensure the notebook runs correctly in conjunction with DataForge processes

Snowflake

  1. Create a new notebook in Snowflake
  2. Edit the notebook settings for External Access and enable the DataForge External Access Integration (named DATAFORGE_)
  3. Create a Source or Output in DataForge and specify the Snowflake notebook name (double quote the name if there are spaces")
  4. Write your custom code in the Snowflake notebook, incorporating any optional SDK parameters
  5. Test the notebook with a new Source data pull (Custom Ingest/Parse) or a new Output (Custom Post Output) to ensure the notebook runs correctly in conjunction with DataForge processes.