External API¶

DataForge provides external API routes for initiating tasks within the platform, accessible via a JWT token.

Requesting the JWT token¶

An Auth0 Machine-to-Machine application named <environment>-External-<client> exists in each customer Auth0 tenant. Use its Client ID and Client Secret to request a JWT token.

Python example

import http.client
import json

conn = http.client.HTTPSConnection("dataforgedev.us.auth0.com")

payload = json.dumps({"client_id":dbutils.secrets.get(scope="api_connections", key="client_id"),"client_secret":dbutils.secrets.get(scope="api_connections", key="client_secret"),"audience":"https://stage.api.dataforgedev.com","grant_type":"client_credentials"})
headers = { 'content-type': "application/json" }
conn.request("POST", "/oauth/token", payload, headers)

res = conn.getresponse()
data = res.read()

responseJson = json.loads(data.decode("utf-8"))
token = responseJson["access_token"]

External Routes Available¶

POST /external/source-pull/¶

Triggers a source to do a "pull data now" and begin the Ingestion process

Python example

import requests

url = "https://stage.api.dataforgedev.com/external/source-pull/19643"
headers = {"authorization":"Bearer "+token}

resp = requests.post(url, headers=headers)

POST /external/inputs-filtered¶

Returns a list of inputs filtered by limit and effective status
Requires body: {"source_id": , "limit": , "effective_filter": }

Python example

import requests

url = "https://stage.api.dataforgedev.com/external/inputs-filtered"
headers = {'authorization':"Bearer "+token, "content-type":"application/json"}
body = json.dumps({"source_id": "19709", "limit":"100","effective_filter":"false"})

response = requests.request("POST",url, headers=headers,data=body)
print(response.text)

GET /external/sources¶

Returns unfiltered list of sources in the environment

Python example

import requests

projectId = "1"

url = "https://stage.api.dataforgedev.com/external/sources"
headers = {'authorization':"Bearer "+token, "content-type":"application/json"}
params = {"projectId":projectId}

response = requests.request("GET",url, headers=headers, params=params)
print(response.text)

GET /external/sources-filtered-by-name/¶

Returns a list of sources with filter condition applied to the source name field. The filter parameter should be a section of the source name. For example, if you were looking for all sources that had "demo" in the name, you would call /external/sources-filtered-by-name/demo

Python example

import requests

nameFilter = "demo"
projectId = "1"

url = f"https://stage.api.dataforgedev.com/external/sources-filtered-by-name/{nameFilter}"
headers = {'authorization':"Bearer "+token,"content-type":"text/plane"}
params = {"projectId":projectId}

response = requests.request("GET",url, headers=headers, params=params)
print(response.text)