Connections¶
The connections home page enables users to quickly search and access connections already configured in the DataForge platform. A connection holds the credentials, network locations, and any other parameters required to access data in the location where it is either generated or staged for ingestion by DataForge.
Only Connections marked as Active are shown here, unless the Active Only toggle is set to off.
Connection Settings¶
Configure the parameters needed for DataForge to access the system.
- Name*: A unique name
- Description*: A description
- Active: Allows users to disable the connection without deleting the configuration
- Group: Allows users to include the connection in a group (requires Connection Template selection to be used)
- Connection Template:Allows users to select from a connection template to normalize Name (requires Group selection to be used)
- Connection Direction: Specifies if this connection is used to ingest or output data
- Connection Type: Specifies the format or location style of the source or target data. Depending on the Type selected, the remaining parameters will change
- Uses Outcrop: A visual indicator showing whether or not this connection will use Outcrop, the DataForge agent that runs inside your network (shown as Agent in the DataForge UI) (only available if Source Connection Direction is selected)
- Outcrop*: Where Outcrop is required, used to select the Outcrop instance to be used (shown as Agent in the DataForge UI) (only available if Source Connection Direction is selected).
The Duplicate button near Save will create a copy of the configuration in a new tab with the same settings and a name of "
API Connection Type¶
Options available:
- Salesforce: For more information on setup and use, refer to the Salesforce API Connection documentation.
Custom Connection Type¶
Used in the SDK as part of Custom Ingestion. Parameters are optional — not all custom ingestion notebooks require them. Values must be JSON key-value objects (e.g. {"key1": "value1"}); use {} for connections with no parameters.
- Public Connection Parameters*: Passed as plain-text to the custom ingestion session.
- Private Connection Parameters*: Encrypted on save.
Event Type¶
Options available:
- Kafka: For more information on setup and use, refer to the Kafka Events Connections documentation.
File Connection Type¶
- Storage Technology*: Specifies the type of file storage Outcrop or Compute will attempt to access
- File Path*: The folder/container path for DataForge to access when pulling or generating files
Table Connection Type¶
- Driver*: Which JDBC driver should be used
When using the Generic JDBC driver option, users need to enter the connection string, driver class path, and any sensitive parameters.
The driver library also needs to be entered into the compute configuration parameters libraries setting that will be used for ingestion on any sources.
The Generic JDBC driver option is only offered when Uses Outcrop is set to Yes, so the driver JAR is placed on the Outcrop machine rather than installed on ingestion compute.
For more information and examples, refer to the Generic JDBC Connection documentation.
Parameters¶
The parameters section will change dynamically based on the required selections above, are typically optional to configure, and are used for advanced configuration or specifications. To utilize Connection Metadata for database connections, use the parameters Metadata Refresh and Metadata Schema Pattern. See Connection Metadata below for more detail.
Connection Metadata¶
Connection Metadata shows an optional list of tables, referenced tables, and primary/foreign keys. Configure it using these parameters in Connection Settings:
- Metadata Refresh — controls what is collected:
- Tables, Columns, and Keys — most granular; enables Talos AI to search specific fields. Recommended default.
- Tables and Keys — table names and key column identifiers.
- Tables — table/view names only.
- None — disables metadata collection.
- Metadata Schema Pattern (optional) — LIKE pattern to filter schemas for metadata collection.
Connection Metadata only works with Table/Database connections.
When Metadata Refresh is set to any option except None, you can create Sources directly from the Connection Metadata tab. Check the table(s) you want, then use the triple-dot menu to add referenced tables or referenced tables recursively (includes the full chain of related tables).
Click Create Source(s) from the triple-dot menu above the table list to open a source creation modal. Set a naming pattern for the new sources. If Metadata Refresh is set to Tables and Keys, you can also opt to automatically create Relations between all sources — this only applies when creating sources for the first time or when recreating them with a new naming pattern.
- Initiate Data Pull — starts a new ingestion for each source immediately after creation.
- Set Incremental Refresh — creates sources with filter queries to pull only changed data. Requires Keys to be defined in the table's data dictionary beforehand.
The Sources column in the Connection Metadata tab shows a count and link to any existing sources pulling from each table/view.
The Refresh button reruns the Connection Test compute to retest the connection and rescan tables and views.
Data Dictionaries¶
Data dictionaries define table and column metadata for a connection. Requires Tables, Columns, and Keys enabled in the Metadata Refresh parameter.
Open a connection and click Data Dictionary to import or export definitions. Import and Export options are hidden while a connection test is in progress.
Table and column definition files include different columns, listed below.
Table Definitions¶
| Column | Description |
|---|---|
| table_schema | Name of the schema in the connection |
| table_name | Name of the table in the schema |
| description | Description of the table and what information it contains |
Column Definitions:¶
| Column | Description | Values |
|---|---|---|
| table_schema | Name of the schema in the connection | |
| table_name | Name of the table in the schema | |
| column_name | Name of the column in the table | |
| description | Description of the column and what information it contains | |
| category | Category of the column for use in DataForge configurations | key, dimension, metric, name, modified timestamp |
| key | Indicates whether the column is a primary key or foreign key | pk, fk, pk+fk |
| fk table | Schema and table name of the table relating to the foreign key column | schema_name.table_name |