A unified data governance solution that helps manage, protect, and discover data across your organization
Will Purview show ONE connected lineage map for Oracle Fusion -> ADF -> Data Lake -> Databricks, or does the ADF-trigger limitation break the chain?
Hello Microsoft Purview Community,
Before building a POC, I want to confirm whether a specific architecture will produce ONE connected lineage graph in Purview or two disconnected graphs - based on a limitation documented for a related scenario.
PLANNED ARCHITECTURE:
Oracle Fusion --> ADF Copy Activity --> Azure Data Lake (Landing) --> Databricks Notebook --> Delta Table (Transformed Output)
GOAL:
A single, connected end-to-end lineage map in Purview showing the full journey from the Oracle Fusion source to the final Delta table.
THE CONCERN:
The official "Connect to and manage Azure Databricks Unity Catalog in Microsoft Purview" documentation states:
"If an external service triggers a notebook (for example, Azure Data Factory [ADF] pipeline calling a Databricks job), the lineage in Microsoft Purview doesn't reflect that dependency. In this case, lineage between ADF datasets and Databricks assets isn't shown. Only lineage built within Databricks notebooks is captured."
This suggests that if ADF triggers the Databricks notebook directly, the ADF lineage chain (Oracle -> Data Lake) and the Databricks internal lineage chain (Data Lake table -> Notebook -> Delta table) will appear as two separate, unconnected graphs - even though they share the same physical table.
QUESTIONS:
- Does the ADF-to-Databricks lineage gap apply only when ADF directly triggers the notebook or also when ADF just writes to a table, and Databricks reads it on its own separate schedule (no direct trigger)?
- Is there a supported way to connect ADF's lineage with Databricks' lineage into one graph, e.g., via Databricks Workflows or the Purview REST API? If REST API, has anyone done this successfully?
Environment:
- Microsoft Purview, Standard tier
- Azure Data Factory (Copy Activity)
- Azure Databricks Unity Catalog, scanned with lineage extraction enabled
Thank you!