How to save Purview lineage and prove the graph is complete
The short answer
Save lineage with two Data Map API calls: POST /datamap/api/search/query for every page of assets, following continuationToken, then GET /datamap/api/atlas/v2/lineage/{guid} for each asset, depth 1, direction BOTH. A Python and DuckDB script builds one edge list and runs six checks proving every page and asset was saved, every relation appears in both assets' files, and no entity is a shell or deleted.
Why save lineage before the switch-off
Microsoft announced on November 12, 2026 that Purview Data Governance is retiring: “Support for Purview Data Governance will end Nov. 12, 2027” (Microsoft Security blog, read 2026-11-21). For lineage, Microsoft's transition guide (read 2026-11-21) points to Fabric lineage and impact analysis for supported items, and says to keep another method for cross-system, unsupported-workload or granular lineage.
The lineage the Data Map holds today, from source tables through copy and transform steps to reports, is the record of how your data moved. Microsoft's lineage page (read 2026-11-21) describes that lineage as a graph of source and target entities connected by processes. A graph saved one asset at a time can come back with holes: an asset whose lineage call failed, or a relation one file holds and the neighbor's file does not. Save the graph, and prove the saved graph is whole, while the Data Map can still answer.
What you will have at the end
- The lineage as saved files: the asset search and one lineage answer per asset, in the Data Map API's own format.
- One edge list,
edges.csv: every relation from every file, without repeats, with both ends named; andnodes.csv, every entity in the graph with its type. - Six checks, each one PASS or FAIL, with every finding in
findings.csv.
Step 1. Save the lineage through the Data Map API
The Data Map REST API (api-version 2023-09-01) finds assets with POST /datamap/api/search/query, which answers up to 1,000 results a page and a continuationToken for the next page (Discovery Query reference, read 2026-11-21). It answers each asset's lineage with GET /datamap/api/atlas/v2/lineage/{guid}, where direction is INPUT, OUTPUT or BOTH and depth is the number of hops (Lineage Get reference, read 2026-11-21). The commands below save every search page, then each asset's lineage one hop in both directions, so each file holds an asset's direct neighbors and the files together hold the whole graph.
We did not run these commands. We have no tenant to run them against, so they are written from Microsoft's Data Map REST reference, and the runs further down use generated records in the same shape. Set ENDPOINT to your Purview account's endpoint and sign in with the Azure CLI first. The search sends no keywords; add a filter from Microsoft's Discovery Query examples if you need to narrow the assets.
# Not run by us: written from Microsoft's Data Map REST reference, api-version 2023-09-01 (read 2026-11-21).
# Needs the Azure CLI (az login) and an account that can read the Data Map.
ENDPOINT="https://<your-account>.purview.azure.com" # your Purview account's endpoint
OUT="lineage-$(date +%Y%m%d-%H%M%S)"; mkdir -p "$OUT/search" "$OUT/lineage"
TOKEN=$(az account get-access-token --resource https://purview.azure.net --query accessToken -o tsv)
API="$ENDPOINT/datamap/api"
# 1. The assets: every page of the search, following continuationToken. Narrow it with a filter if you need to.
N=1; BODY='{"keywords": null, "limit": 1000}'
while :; do
F="$OUT/search/page-$(printf %03d $N).json"
curl -sf -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
"$API/search/query?api-version=2023-09-01" -d "$BODY" -o "$F" || { echo "FAILED: $F was not saved"; rm -f "$F"; break; }
TOK=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1])).get('continuationToken') or '')" "$F")
[ -z "$TOK" ] && break
BODY=$(python3 -c "import json,sys; print(json.dumps({'keywords': None, 'limit': 1000, 'continuationToken': sys.argv[1]}))" "$TOK")
N=$((N + 1))
done
# 2. Each asset's lineage, one hop in both directions, saved as lineage/<guid>.json.
T0=$SECONDS
NEWTOKEN() { TOKEN=$(az account get-access-token --resource https://purview.azure.net --query accessToken -o tsv); T0=$SECONDS; }
python3 -c "import json,glob,sys; [print(a['id']) for f in sorted(glob.glob(sys.argv[1] + '/search/page-*.json')) for a in json.load(open(f))['value']]" "$OUT" |
while IFS= read -r G; do
[ $((SECONDS - T0)) -ge 240 ] && NEWTOKEN # az hands back a token with at least 5 minutes left: ask again every 4 minutes
F="$OUT/lineage/$G.json"; U="$API/atlas/v2/lineage/$G?depth=1&direction=BOTH"
curl -sf -H "Authorization: Bearer $TOKEN" "$U" -o "$F" \
|| { NEWTOKEN; curl -sf -H "Authorization: Bearer $TOKEN" "$U" -o "$F"; } \
|| { echo "FAILED: lineage of $G was not saved"; rm -f "$F"; }
done
echo "Saved to $OUT"
Step 2. Check the saved graph
Download check_lineage_archive.py, install DuckDB (pip install duckdb) and run the script on the folder the commands wrote:
python3 check_lineage_archive.py lineage-20261121-093000
| Check | What a failure means |
|---|---|
| Every page of the asset search was saved | The last saved page still holds a token for a next page, or a page number is missing, so some assets were never asked for their lineage. |
| Every asset in the search has a saved lineage file | An asset's lineage call failed or was skipped, so the graph has a hole where that asset sits. |
| Every relation's two ends are in the same file's entity map | A file names a relation to an entity the file does not describe, so the edge list cannot say what that end is. |
| Every relation between two saved assets appears in both assets' files | Two files disagree: lineage changed between the two calls, or one answer was cut short. Save both assets again. |
| No entity in the graph is a shell entity | Lineage points to an entity the Data Map holds only as a shell, likely an asset that was never scanned, so that end of the graph has a name and nothing else. |
| No entity in the graph is marked DELETED | The graph still runs through an entity marked deleted; decide whether that path belongs in the record. |
Run 1. A generated lineage graph, clean
The script writes a generated export when you ask for one: three source tables copied to a landing zone, two transforms into a curated zone and a dashboard refresh, the kind of flow Microsoft's lineage page describes, with a fixed seed. That is 15 assets, two pages of search results and one lineage file per asset.
python3 check_lineage_archive.py --make-demo demo
Wrote a generated lineage export to demo: 15 assets in 2 search pages, 15 lineage files, 15 relations (seed 268)
python3 check_lineage_archive.py demo
Python 3.14.8 duckdb 1.5.6 Export: demo Read 15 assets from 2 search pages, 15 lineage files holding 30 relations (a relation between two saved assets is in both files) Graph: 3 adf_copy_activity, 5 azure_datalake_gen2_path, 3 azure_sql_table, 1 powerbi_dashboard, 3 synapse_notebook; 15 relations without repeats Entities with no upstream: 3; entities with no downstream: 1 Checks PASS Every page of the asset search was saved PASS Every asset in the search has a saved lineage file PASS Every relation's two ends are in the same file's entity map PASS Every relation between two saved assets appears in both assets' files PASS No entity in the graph is a shell entity (isIncomplete) PASS No entity in the graph is marked DELETED Wrote nodes.csv (15 rows), edges.csv (15 rows) and findings.csv (0 rows) to . RESULT: 0 findings. The saved graph is whole: every file agrees with its neighbors.
How to read it
The 15 files hold 30 relations because each relation between two saved assets appears in both assets' files; joined without repeats, the graph has 15 relations. All six checks pass, so every file agrees with its neighbors. The three source tables have no upstream and the dashboard has no downstream: the two ends of the flow.
Run 2. The same graph with six planted problems
python3 check_lineage_archive.py --make-demo demo-planted --plant-errors
Wrote a generated lineage export to demo-planted: 15 assets in 2 search pages, 14 lineage files, 16 relations (seed 268) Planted one error for each check: DANGLING_RELATION, DELETED_ENTITY, NO_LINEAGE_FILE, ONE_SIDED_EDGE, PAGES_MISSING, SHELL_ENTITY
python3 check_lineage_archive.py demo-planted --tag planted
Python 3.14.8 duckdb 1.5.6 Export: demo-planted Read 15 assets from 2 search pages, 14 lineage files holding 29 relations (a relation between two saved assets is in both files) Graph: 3 adf_copy_activity, 5 azure_datalake_gen2_path, 4 azure_sql_table, 1 powerbi_dashboard, 3 synapse_notebook; 16 relations without repeats Entities with no upstream: 4; entities with no downstream: 1 Checks FAIL Every page of the asset search was saved (1) FAIL Every asset in the search has a saved lineage file (1) FAIL Every relation's two ends are in the same file's entity map (1) FAIL Every relation between two saved assets appears in both assets' files (1) FAIL No entity in the graph is a shell entity (isIncomplete) (1) FAIL No entity in the graph is marked DELETED (1) PAGES_MISSING search: page-002.json still holds a continuationToken for the next page NO_LINEAGE_FILE asset crm_accounts: azure_sql_table 2cac6404-b9c2-4cf3-9f05-4b70d7b36f79 has no saved lineage file DANGLING_RELATION file of copy_orders: relation copy_orders -> landing/orders: e8e4bb03-d61c-43e5-9349-5ec0cba06c9d is not in the file's entity map ONE_SIDED_EDGE relation build_customer_dim -> curated/customer_dim: in the file of curated/customer_dim, missing from the file of build_customer_dim SHELL_ENTITY entity legacy_stage: azure_sql_table ff0a0615-d143-4bdb-8c73-d50466ffd1d6 is a shell entity (isIncomplete), likely an asset that was never scanned DELETED_ENTITY entity copy_accounts: adf_copy_activity 47f3c2c0-5cb5-4b3e-8986-8b750383f738 is marked DELETED Wrote nodes-planted.csv (16 rows), edges-planted.csv (16 rows) and findings-planted.csv (6 rows) to . RESULT: 6 findings. Save the missing lineage again and re-run.
| Check | The problem planted in run 2 |
|---|---|
| Pages | The last search page still holds a token for a next page that was never saved. |
| Lineage files | The crm_accounts table's lineage file removed. |
| Entity maps | The copy_orders file's entity map without landing/orders, which the file's relation points to. |
| Both files | The relation from build_customer_dim to curated/customer_dim removed from the build_customer_dim file only. |
| Shell entities | A legacy_stage table that feeds copy_orders but was never scanned. |
| Deleted entities | The copy_accounts activity marked DELETED. |
Run the check on your own lineage
Run the commands in step 1 with ENDPOINT set to your Purview account's endpoint. The commands write a folder named lineage- followed by the date and time, for example lineage-20261121-093000, holding a search folder with one file for each page of the asset search and a lineage folder with one file for each asset, named after the asset's guid. A line that starts with FAILED names a page or an asset that was not saved; run the commands again. Then run python3 check_lineage_archive.py followed by the folder's name. The script writes nodes.csv, edges.csv and findings.csv to the script's own folder; add --out followed by a folder name to write the three files somewhere else. You do not edit the script.
We also tested a run on a second generated export, saved the way you would save your own: other asset names and types, upper-case guids, three pages of search results, an asset with no lineage at all, and one planted problem for each check. All six checks failed as planted, and nothing else was flagged.
Download check_lineage_archive.py · Get the full guide (PDF)
Result from run 1: edges.csv.
Get the full guide as a PDF
We'll email you a link to it. Please use your work email.
Thanks, . We're sending the link to now. If it isn't there in a few minutes, check your spam or junk folder, or email info@datameaning.com.
Limits
- The runs on this page use generated records, not a Purview tenant, and the commands in step 1 were not run. The field names come from Microsoft's Data Map REST reference, read on 2026-11-21.
- The search sends no keywords, and Microsoft's reference does not say that such a search returns every asset. Compare the search's
@search.countwith what you expect (Microsoft marks the count as approximate when@search.count.approximateis true), and add a filter from Microsoft's examples if the count is short. - Microsoft's search reference says a result can be an asset or a glossary term (
objectType). If the search returns terms, a term whose lineage call fails is reported as an asset with no saved lineage file; add a filter from Microsoft's examples to search assets only. - The checks prove the saved files agree with each other, not that the Data Map held no more lineage. Lineage of an asset the search did not return is not saved.
- Column-level lineage and process run history are not saved; the commands ask for entity-level lineage only.
- Microsoft's reference offers a next-page call for lineage answers that are paged. The commands make one call per asset without paging; a relation missing from one of two files is the sign to save both assets again.
- Dates are as of 2026-11-21. This guide is about Purview Data Governance, whose support ends on November 12, 2027.
Tested on
Tested on
- Platform
- DuckDB 1.5.6, run from Python 3.14.8
- Edition
- Open-source DuckDB library (
pip install duckdb) - Date
- 2026-11-21
- Records
- Generated: an invented lineage graph of 15 assets, from source tables through copy and transform steps to a dashboard, fixed seed 268, in the shape of the Data Map API's answers (api-version 2023-09-01). Not a run against a tenant.
- Code
- check_lineage_archive.py
- Results
- edges.csv
When this is not enough
The saved graph is the record of how data moved. The paths that still matter then need a new home: Fabric lineage for supported items, and another method for paths across systems Fabric does not cover.
Moving your whole Purview governance catalog, not only the lineage?
Purview to OneLake Migration is in early access. The offering starts with the four-day Catalog Transition Assessment: one fixed fee of $12,000, and no commitment to anything else. You get an inventory of every object in your Purview governance catalog and a decision for each object. AI does the first pass, and your stewards sign every domain.