Guide · Purview to OneLake Migration

How to save Purview lineage and prove the graph is complete

Updated 2026-11-21Python and DuckDBRuns shown on generated records, PDF by email

The short answer

Save lineage with two Data Map API calls: POST /datamap/api/search/query for every page of assets, following continuationToken, then GET /datamap/api/atlas/v2/lineage/{guid} for each asset, depth 1, direction BOTH. A Python and DuckDB script builds one edge list and runs six checks proving every page and asset was saved, every relation appears in both assets' files, and no entity is a shell or deleted.

Why save lineage before the switch-off

Microsoft announced on November 12, 2026 that Purview Data Governance is retiring: “Support for Purview Data Governance will end Nov. 12, 2027” (Microsoft Security blog, read 2026-11-21). For lineage, Microsoft's transition guide (read 2026-11-21) points to Fabric lineage and impact analysis for supported items, and says to keep another method for cross-system, unsupported-workload or granular lineage.

The lineage the Data Map holds today, from source tables through copy and transform steps to reports, is the record of how your data moved. Microsoft's lineage page (read 2026-11-21) describes that lineage as a graph of source and target entities connected by processes. A graph saved one asset at a time can come back with holes: an asset whose lineage call failed, or a relation one file holds and the neighbor's file does not. Save the graph, and prove the saved graph is whole, while the Data Map can still answer.

What you will have at the end

  • The lineage as saved files: the asset search and one lineage answer per asset, in the Data Map API's own format.
  • One edge list, edges.csv: every relation from every file, without repeats, with both ends named; and nodes.csv, every entity in the graph with its type.
  • Six checks, each one PASS or FAIL, with every finding in findings.csv.

Step 1. Save the lineage through the Data Map API

The Data Map REST API (api-version 2023-09-01) finds assets with POST /datamap/api/search/query, which answers up to 1,000 results a page and a continuationToken for the next page (Discovery Query reference, read 2026-11-21). It answers each asset's lineage with GET /datamap/api/atlas/v2/lineage/{guid}, where direction is INPUT, OUTPUT or BOTH and depth is the number of hops (Lineage Get reference, read 2026-11-21). The commands below save every search page, then each asset's lineage one hop in both directions, so each file holds an asset's direct neighbors and the files together hold the whole graph.

We did not run these commands. We have no tenant to run them against, so they are written from Microsoft's Data Map REST reference, and the runs further down use generated records in the same shape. Set ENDPOINT to your Purview account's endpoint and sign in with the Azure CLI first. The search sends no keywords; add a filter from Microsoft's Discovery Query examples if you need to narrow the assets.

# Not run by us: written from Microsoft's Data Map REST reference, api-version 2023-09-01 (read 2026-11-21).
# Needs the Azure CLI (az login) and an account that can read the Data Map.
ENDPOINT="https://<your-account>.purview.azure.com"   # your Purview account's endpoint
OUT="lineage-$(date +%Y%m%d-%H%M%S)"; mkdir -p "$OUT/search" "$OUT/lineage"
TOKEN=$(az account get-access-token --resource https://purview.azure.net --query accessToken -o tsv)
API="$ENDPOINT/datamap/api"
# 1. The assets: every page of the search, following continuationToken. Narrow it with a filter if you need to.
N=1; BODY='{"keywords": null, "limit": 1000}'
while :; do
  F="$OUT/search/page-$(printf %03d $N).json"
  curl -sf -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    "$API/search/query?api-version=2023-09-01" -d "$BODY" -o "$F" || { echo "FAILED: $F was not saved"; rm -f "$F"; break; }
  TOK=$(python3 -c "import json,sys; print(json.load(open(sys.argv[1])).get('continuationToken') or '')" "$F")
  [ -z "$TOK" ] && break
  BODY=$(python3 -c "import json,sys; print(json.dumps({'keywords': None, 'limit': 1000, 'continuationToken': sys.argv[1]}))" "$TOK")
  N=$((N + 1))
done
# 2. Each asset's lineage, one hop in both directions, saved as lineage/<guid>.json.
T0=$SECONDS
NEWTOKEN() { TOKEN=$(az account get-access-token --resource https://purview.azure.net --query accessToken -o tsv); T0=$SECONDS; }
python3 -c "import json,glob,sys; [print(a['id']) for f in sorted(glob.glob(sys.argv[1] + '/search/page-*.json')) for a in json.load(open(f))['value']]" "$OUT" |
while IFS= read -r G; do
  [ $((SECONDS - T0)) -ge 240 ] && NEWTOKEN   # az hands back a token with at least 5 minutes left: ask again every 4 minutes
  F="$OUT/lineage/$G.json"; U="$API/atlas/v2/lineage/$G?depth=1&direction=BOTH"
  curl -sf -H "Authorization: Bearer $TOKEN" "$U" -o "$F" \
    || { NEWTOKEN; curl -sf -H "Authorization: Bearer $TOKEN" "$U" -o "$F"; } \
    || { echo "FAILED: lineage of $G was not saved"; rm -f "$F"; }
done
echo "Saved to $OUT"

Step 2. Check the saved graph

Download check_lineage_archive.py, install DuckDB (pip install duckdb) and run the script on the folder the commands wrote:

python3 check_lineage_archive.py lineage-20261121-093000
CheckWhat a failure means
Every page of the asset search was savedThe last saved page still holds a token for a next page, or a page number is missing, so some assets were never asked for their lineage.
Every asset in the search has a saved lineage fileAn asset's lineage call failed or was skipped, so the graph has a hole where that asset sits.
Every relation's two ends are in the same file's entity mapA file names a relation to an entity the file does not describe, so the edge list cannot say what that end is.
Every relation between two saved assets appears in both assets' filesTwo files disagree: lineage changed between the two calls, or one answer was cut short. Save both assets again.
No entity in the graph is a shell entityLineage points to an entity the Data Map holds only as a shell, likely an asset that was never scanned, so that end of the graph has a name and nothing else.
No entity in the graph is marked DELETEDThe graph still runs through an entity marked deleted; decide whether that path belongs in the record.

Run 1. A generated lineage graph, clean

The script writes a generated export when you ask for one: three source tables copied to a landing zone, two transforms into a curated zone and a dashboard refresh, the kind of flow Microsoft's lineage page describes, with a fixed seed. That is 15 assets, two pages of search results and one lineage file per asset.

python3 check_lineage_archive.py --make-demo demo
Wrote a generated lineage export to demo: 15 assets in 2 search pages, 15 lineage files, 15 relations (seed 268)
python3 check_lineage_archive.py demo
Python 3.14.8 duckdb 1.5.6
Export: demo
Read 15 assets from 2 search pages, 15 lineage files holding 30 relations (a relation between two saved assets is in both files)

Graph: 3 adf_copy_activity, 5 azure_datalake_gen2_path, 3 azure_sql_table, 1 powerbi_dashboard, 3 synapse_notebook; 15 relations without repeats
Entities with no upstream: 3; entities with no downstream: 1

Checks
  PASS  Every page of the asset search was saved
  PASS  Every asset in the search has a saved lineage file
  PASS  Every relation's two ends are in the same file's entity map
  PASS  Every relation between two saved assets appears in both assets' files
  PASS  No entity in the graph is a shell entity (isIncomplete)
  PASS  No entity in the graph is marked DELETED

Wrote nodes.csv (15 rows), edges.csv (15 rows) and findings.csv (0 rows) to .
RESULT: 0 findings. The saved graph is whole: every file agrees with its neighbors.

How to read it

The 15 files hold 30 relations because each relation between two saved assets appears in both assets' files; joined without repeats, the graph has 15 relations. All six checks pass, so every file agrees with its neighbors. The three source tables have no upstream and the dashboard has no downstream: the two ends of the flow.

Run 2. The same graph with six planted problems

python3 check_lineage_archive.py --make-demo demo-planted --plant-errors
Wrote a generated lineage export to demo-planted: 15 assets in 2 search pages, 14 lineage files, 16 relations (seed 268)
Planted one error for each check: DANGLING_RELATION, DELETED_ENTITY, NO_LINEAGE_FILE, ONE_SIDED_EDGE, PAGES_MISSING, SHELL_ENTITY
python3 check_lineage_archive.py demo-planted --tag planted
Python 3.14.8 duckdb 1.5.6
Export: demo-planted
Read 15 assets from 2 search pages, 14 lineage files holding 29 relations (a relation between two saved assets is in both files)

Graph: 3 adf_copy_activity, 5 azure_datalake_gen2_path, 4 azure_sql_table, 1 powerbi_dashboard, 3 synapse_notebook; 16 relations without repeats
Entities with no upstream: 4; entities with no downstream: 1

Checks
  FAIL  Every page of the asset search was saved  (1)
  FAIL  Every asset in the search has a saved lineage file  (1)
  FAIL  Every relation's two ends are in the same file's entity map  (1)
  FAIL  Every relation between two saved assets appears in both assets' files  (1)
  FAIL  No entity in the graph is a shell entity (isIncomplete)  (1)
  FAIL  No entity in the graph is marked DELETED  (1)
  PAGES_MISSING      search: page-002.json still holds a continuationToken for the next page
  NO_LINEAGE_FILE    asset crm_accounts: azure_sql_table 2cac6404-b9c2-4cf3-9f05-4b70d7b36f79 has no saved lineage file
  DANGLING_RELATION  file of copy_orders: relation copy_orders -> landing/orders: e8e4bb03-d61c-43e5-9349-5ec0cba06c9d is not in the file's entity map
  ONE_SIDED_EDGE     relation build_customer_dim -> curated/customer_dim: in the file of curated/customer_dim, missing from the file of build_customer_dim
  SHELL_ENTITY       entity legacy_stage: azure_sql_table ff0a0615-d143-4bdb-8c73-d50466ffd1d6 is a shell entity (isIncomplete), likely an asset that was never scanned
  DELETED_ENTITY     entity copy_accounts: adf_copy_activity 47f3c2c0-5cb5-4b3e-8986-8b750383f738 is marked DELETED

Wrote nodes-planted.csv (16 rows), edges-planted.csv (16 rows) and findings-planted.csv (6 rows) to .
RESULT: 6 findings. Save the missing lineage again and re-run.
CheckThe problem planted in run 2
PagesThe last search page still holds a token for a next page that was never saved.
Lineage filesThe crm_accounts table's lineage file removed.
Entity mapsThe copy_orders file's entity map without landing/orders, which the file's relation points to.
Both filesThe relation from build_customer_dim to curated/customer_dim removed from the build_customer_dim file only.
Shell entitiesA legacy_stage table that feeds copy_orders but was never scanned.
Deleted entitiesThe copy_accounts activity marked DELETED.

Run the check on your own lineage

Run the commands in step 1 with ENDPOINT set to your Purview account's endpoint. The commands write a folder named lineage- followed by the date and time, for example lineage-20261121-093000, holding a search folder with one file for each page of the asset search and a lineage folder with one file for each asset, named after the asset's guid. A line that starts with FAILED names a page or an asset that was not saved; run the commands again. Then run python3 check_lineage_archive.py followed by the folder's name. The script writes nodes.csv, edges.csv and findings.csv to the script's own folder; add --out followed by a folder name to write the three files somewhere else. You do not edit the script.

We also tested a run on a second generated export, saved the way you would save your own: other asset names and types, upper-case guids, three pages of search results, an asset with no lineage at all, and one planted problem for each check. All six checks failed as planted, and nothing else was flagged.

Download check_lineage_archive.py · Get the full guide (PDF)

Result from run 1: edges.csv.

Get the full guide as a PDF

We'll email you a link to it. Please use your work email.

Limits

  • The runs on this page use generated records, not a Purview tenant, and the commands in step 1 were not run. The field names come from Microsoft's Data Map REST reference, read on 2026-11-21.
  • The search sends no keywords, and Microsoft's reference does not say that such a search returns every asset. Compare the search's @search.count with what you expect (Microsoft marks the count as approximate when @search.count.approximate is true), and add a filter from Microsoft's examples if the count is short.
  • Microsoft's search reference says a result can be an asset or a glossary term (objectType). If the search returns terms, a term whose lineage call fails is reported as an asset with no saved lineage file; add a filter from Microsoft's examples to search assets only.
  • The checks prove the saved files agree with each other, not that the Data Map held no more lineage. Lineage of an asset the search did not return is not saved.
  • Column-level lineage and process run history are not saved; the commands ask for entity-level lineage only.
  • Microsoft's reference offers a next-page call for lineage answers that are paged. The commands make one call per asset without paging; a relation missing from one of two files is the sign to save both assets again.
  • Dates are as of 2026-11-21. This guide is about Purview Data Governance, whose support ends on November 12, 2027.

Tested on

Tested on

Platform
DuckDB 1.5.6, run from Python 3.14.8
Edition
Open-source DuckDB library (pip install duckdb)
Date
2026-11-21
Records
Generated: an invented lineage graph of 15 assets, from source tables through copy and transform steps to a dashboard, fixed seed 268, in the shape of the Data Map API's answers (api-version 2023-09-01). Not a run against a tenant.
Code
check_lineage_archive.py
Results
edges.csv

When this is not enough

The saved graph is the record of how data moved. The paths that still matter then need a new home: Fabric lineage for supported items, and another method for paths across systems Fabric does not cover.

Moving your whole Purview governance catalog, not only the lineage?

Purview to OneLake Migration is in early access. The offering starts with the four-day Catalog Transition Assessment: one fixed fee of $12,000, and no commitment to anything else. You get an inventory of every object in your Purview governance catalog and a decision for each object. AI does the first pass, and your stewards sign every domain.

See Purview to OneLake Migration