Link Extracted Data to Document Locations

Find where each extracted value appears in the original document by passing the Extract response and the Parse response to the Ground API. Ground returns, for each extracted field, the blocks the value was quoted from, with each block’s page number and bounding box. Use the locations to highlight values on the page, build a review screen where a person checks each field against its source, or attach citations to extracted data. Ground consumes no credits. A value can appear in more than one place: a name printed in the header and again in the signature block has one entry in its ranges for each place. The script returns one location per range, so each place gets its own box. The tutorial uses the ade-python library and ade-typescript library to:

Parse this PDF: Pay Stub
Extract these fields: Employee Name and Gross Pay
Ground both fields with the Ground API
Save a PNG of each place a field appears, a copy of each page with every location drawn and labeled, and a JSON file with each location’s page number and bounding box

These examples require the Python or TypeScript client library. Before running a script, set your API key and install the library and any required dependencies.

The Python script works with PDF and image files. The TypeScript script is written for PDF files. To export crops from other file types, use it as a reference and adapt the image export logic.

1. Download the Document to Process

Download the Pay Stub and save it to a local directory.

2. Create the Script

Copy the script for your language and save it as grounding.py or grounding.mts in the same directory as the PDF. The TypeScript script needs the .mts extension to run as an ES module, because the mupdf package it uses to render pages is an ES module.

import json
from functools import lru_cache
from pathlib import Path
import pymupdf
from PIL import Image, ImageDraw, ImageOps
from landingai_ade import LandingAIADE

# Initialize client (uses VISION_AGENT_API_KEY environment variable)
client = LandingAIADE()

document_path = Path("pay-stub.pdf")
output_dir = Path("output")

# Define the extraction schema
schema = {
    "type": "object",
    "properties": {
        "employee_name": {
            "description": "The employee's full name",
            "type": "string"
        },
        "gross_pay": {
            "description": "The gross pay amount",
            "type": "number"
        }
    }
}

# 1. Parse the document (save_to keeps the full parse response)
parse_response = client.v2.parse(
    document=document_path,
    model="dpt-3-pro-latest",
    save_to=output_dir
)

# 2. Extract the fields from the parsed Markdown
extract_response = client.v2.extract(
    schema=schema,
    markdown=parse_response.markdown,
    model="extract-latest",
    save_to=output_dir / "pay-stub_extract_output.json"
)

# 3. Confirm the extraction read this parse's Markdown. Ground can't detect a mismatched pair.
if extract_response.metadata.doc_id != parse_response.metadata.job_id:
    raise ValueError("The extraction and the structure come from different parses")

# 4. Map each extracted field to the blocks it was quoted from
ground_response = client.v2.ground(
    extraction_metadata=extract_response.extraction_metadata,
    structure=parse_response.structure
)
(output_dir / "pay-stub_ground_output.json").write_text(ground_response.to_json(), encoding="utf-8")


def iter_fields(metadata, grounding, path=""):
    """Walk extraction_metadata and the grounding tree together, one entry per extracted value."""
    if isinstance(metadata, dict) and "ranges" in metadata:
        yield path, metadata, grounding
    elif isinstance(metadata, dict):
        for key, child in metadata.items():
            yield from iter_fields(child, grounding[key], f"{path}.{key}" if path else key)
    elif isinstance(metadata, list):
        for index, child in enumerate(metadata):
            yield from iter_fields(child, grounding[index], f"{path}[{index}]")


def overlaps(entry, source_range):
    """Whether a grounding entry covers any character of a [start, end) range of the Markdown."""
    return entry["range"]["start"] < source_range["end"] and entry["range"]["end"] > source_range["start"]


def locate(blocks, source_range):
    """Return one location for one range of a field: its page, the tightest box, and its blocks."""
    # Ground returns the blocks for all of a field's ranges: keep the ones this range overlaps
    in_range = [block for block in blocks if overlaps(block["grounding"], source_range)]

    # A value in a table matches both the table and its cell: keep the cell
    parent_ids = {block.get("parent_id") for block in in_range}
    in_range = [block for block in in_range if block["block_id"] not in parent_ids]
    if not in_range:
        return None

    entries = []
    for block in in_range:
        # Prefer the lines or words this range overlaps; fall back to the whole block
        matches = [entry for entry in block.get("atomic_grounding") or [] if overlaps(entry, source_range)]
        entries.extend(matches or [block["grounding"]])

    # Merge the boxes on the first page the range appears on
    page = entries[0]["page"]
    boxes = [entry["box"] for entry in entries if entry["page"] == page]
    return {
        "page": page,
        "box": {
            "xmin": min(box["xmin"] for box in boxes),
            "ymin": min(box["ymin"] for box in boxes),
            "xmax": max(box["xmax"] for box in boxes),
            "ymax": max(box["ymax"] for box in boxes)
        },
        "block_ids": [block["block_id"] for block in in_range if block["grounding"]["page"] == page]
    }


@lru_cache
def render_page(page_number):
    """Render a 1-indexed page of the document as an image."""
    if document_path.suffix.lower() == ".pdf":
        with pymupdf.open(document_path) as pdf:
            return pdf[page_number - 1].get_pixmap(dpi=150).pil_image()
    # Boxes match the upright image, so apply any EXIF orientation first
    return ImageOps.exif_transpose(Image.open(document_path)).convert("RGB")


def to_pixels(box, image):
    """Convert normalized box coordinates to pixel coordinates."""
    width, height = image.size
    return (int(box["xmin"] * width), int(box["ymin"] * height),
            int(box["xmax"] * width), int(box["ymax"] * height))


# 5. Save a crop of each location, and its page and box
grounding_results = {}
labels_by_page = {}
for field, metadata, blocks in iter_fields(extract_response.extraction_metadata, ground_response.grounding):
    locations = []
    if blocks:
        # One location per place the value was quoted from
        for source_range in metadata["ranges"]:
            location = locate(blocks, source_range)
            if location:
                locations.append(location)
    grounding_results[field] = {"value": metadata["value"], "locations": locations}

    if not locations:
        # None: nothing in the document was quoted for this value.
        # []: no block matched, which usually means the extraction and structure come from different parses.
        print(f"{field}: no source location")
        continue

    for number, location in enumerate(locations, start=1):
        image = render_page(location["page"])
        image.crop(to_pixels(location["box"], image)).save(output_dir / f"{field}_{number}.png")
        labels_by_page.setdefault(location["page"], []).append((f"{field} #{number}", location["box"]))
        print(f"{field} #{number}: {metadata['value']!r} on page {location['page']}")

with open(output_dir / "pay-stub_grounding_output.json", "w") as f:
    json.dump(grounding_results, f, indent=2)

# 6. Draw every location on its page, labeled with the field name and occurrence
for page, labels in labels_by_page.items():
    annotated = render_page(page).copy()
    draw = ImageDraw.Draw(annotated)
    for label, box in labels:
        x1, y1, x2, y2 = to_pixels(box, annotated)
        draw.rectangle([x1, y1, x2, y2], outline=(220, 53, 69), width=3)
        label_y = max(0, y1 - 20)
        draw.rectangle([x1, label_y, x1 + len(label) * 8, y1], fill=(220, 53, 69))
        draw.text((x1 + 2, label_y + 2), label, fill=(255, 255, 255))
    annotated.save(output_dir / f"page_{page}_grounding.png")

import * as mupdf from "mupdf";
import LandingAIADE from "landingai-ade";
import fs from "fs";
import { createCanvas, loadImage } from "canvas";

// Initialize client (uses VISION_AGENT_API_KEY environment variable)
const client = new LandingAIADE();

const documentPath = "pay-stub.pdf";
const outputDir = "output";

type Box = { xmin: number; ymin: number; xmax: number; ymax: number };
type Range = { start: number; end: number };
type Entry = { page: number; range: Range; box: Box };
type GroundedBlock = {
  block_id: string;
  type: string;
  parent_id?: string;
  grounding: Entry;
  atomic_grounding?: Entry[];
};
type Leaf = { value: unknown; ranges: Range[] | null };
type Location = { page: number; box: Box; block_ids: string[] };

// Define the extraction schema
const schema = {
  type: "object",
  properties: {
    employee_name: {
      description: "The employee's full name",
      type: "string"
    },
    gross_pay: {
      description: "The gross pay amount",
      type: "number"
    }
  }
};

// 1. Parse the document (saveTo keeps the full parse response)
const parseResponse = await client.v2.parse({
  document: fs.createReadStream(documentPath),
  model: "dpt-3-pro-latest",
  saveTo: outputDir
});

// 2. Extract the fields from the parsed Markdown
const extractResponse = await client.v2.extract({
  schema: schema,
  markdown: parseResponse.markdown,
  model: "extract-latest",
  saveTo: `${outputDir}/pay-stub_extract_output.json`
});

// 3. Confirm the extraction read this parse's Markdown. Ground can't detect a mismatched pair.
if (extractResponse.metadata.doc_id !== parseResponse.metadata.job_id) {
  throw new Error("The extraction and the structure come from different parses");
}

// 4. Map each extracted field to the blocks it was quoted from
// (the library types Ground's structure input as a plain object, so cast the parse structure)
const groundResponse = await client.v2.ground({
  extraction_metadata: extractResponse.extraction_metadata,
  structure: parseResponse.structure as unknown as Record<string, unknown>
});
fs.writeFileSync(`${outputDir}/pay-stub_ground_output.json`, JSON.stringify(groundResponse, null, 2));

// Walk extraction_metadata and the grounding tree together, one entry per extracted value
function* iterFields(metadata: any, grounding: any, path = ""): Generator<[string, Leaf, GroundedBlock[] | null]> {
  if (Array.isArray(metadata)) {
    for (let i = 0; i < metadata.length; i++) {
      yield* iterFields(metadata[i], grounding[i], `${path}[${i}]`);
    }
  } else if (metadata !== null && typeof metadata === "object" && "ranges" in metadata) {
    yield [path, metadata, grounding];
  } else if (metadata !== null && typeof metadata === "object") {
    for (const [key, child] of Object.entries(metadata)) {
      yield* iterFields(child, grounding[key], path ? `${path}.${key}` : key);
    }
  }
}

// Whether a grounding entry covers any character of a [start, end) range of the Markdown
function overlaps(entry: Entry, sourceRange: Range): boolean {
  return entry.range.start < sourceRange.end && entry.range.end > sourceRange.start;
}

// Return one location for one range of a field: its page, the tightest box, and its blocks
function locate(blocks: GroundedBlock[], sourceRange: Range): Location | null {
  // Ground returns the blocks for all of a field's ranges: keep the ones this range overlaps
  const overlapping = blocks.filter((block) => overlaps(block.grounding, sourceRange));

  // A value in a table matches both the table and its cell: keep the cell
  const parentIds = new Set(overlapping.map((block) => block.parent_id));
  const inRange = overlapping.filter((block) => !parentIds.has(block.block_id));
  if (inRange.length === 0) return null;

  // Prefer the lines or words this range overlaps; fall back to the whole block
  const entries = inRange.flatMap((block) => {
    const matches = (block.atomic_grounding ?? []).filter((entry) => overlaps(entry, sourceRange));
    return matches.length ? matches : [block.grounding];
  });

  // Merge the boxes on the first page the range appears on
  const page = entries[0].page;
  const boxes = entries.filter((entry) => entry.page === page).map((entry) => entry.box);
  return {
    page,
    box: {
      xmin: Math.min(...boxes.map((box) => box.xmin)),
      ymin: Math.min(...boxes.map((box) => box.ymin)),
      xmax: Math.max(...boxes.map((box) => box.xmax)),
      ymax: Math.max(...boxes.map((box) => box.ymax))
    },
    block_ids: inRange.filter((block) => block.grounding.page === page).map((block) => block.block_id)
  };
}

// Render a region of a 1-indexed PDF page at 150 DPI and return it as a PNG
const FULL_PAGE: Box = { xmin: 0, ymin: 0, xmax: 1, ymax: 1 };
const pdfDocument = mupdf.Document.openDocument(fs.readFileSync(documentPath), "application/pdf");
function renderPng(pageNumber: number, box: Box): Uint8Array {
  const scaleFactor = 150 / 72;
  const page = pdfDocument.loadPage(pageNumber - 1);
  const bounds = page.getBounds();
  const fullWidth = Math.round((bounds[2] - bounds[0]) * scaleFactor);
  const fullHeight = Math.round((bounds[3] - bounds[1]) * scaleFactor);
  const pixmap = new mupdf.Pixmap(
    mupdf.ColorSpace.DeviceRGB,
    [
      Math.round(box.xmin * fullWidth),
      Math.round(box.ymin * fullHeight),
      Math.round(box.xmax * fullWidth),
      Math.round(box.ymax * fullHeight)
    ],
    false
  );
  pixmap.clear(255);
  const device = new mupdf.DrawDevice(mupdf.Matrix.scale(scaleFactor, scaleFactor), pixmap);
  page.run(device, mupdf.Matrix.identity);
  device.close();
  return pixmap.asPNG();
}

// Draw each location on a copy of its page, labeled with the field name and occurrence
async function drawLocations(pageNumber: number, labels: { label: string; box: Box }[]): Promise<Buffer> {
  const image = await loadImage(Buffer.from(renderPng(pageNumber, FULL_PAGE)));
  const canvas = createCanvas(image.width, image.height);
  const ctx = canvas.getContext("2d");
  ctx.drawImage(image, 0, 0);
  ctx.font = "12px sans-serif";

  for (const { label, box } of labels) {
    const x1 = Math.floor(box.xmin * image.width);
    const y1 = Math.floor(box.ymin * image.height);
    const x2 = Math.floor(box.xmax * image.width);
    const y2 = Math.floor(box.ymax * image.height);
    ctx.strokeStyle = "rgb(220, 53, 69)";
    ctx.lineWidth = 3;
    ctx.strokeRect(x1, y1, x2 - x1, y2 - y1);
    const labelY = Math.max(0, y1 - 20);
    ctx.fillStyle = "rgb(220, 53, 69)";
    ctx.fillRect(x1, labelY, label.length * 8, 20);
    ctx.fillStyle = "white";
    ctx.fillText(label, x1 + 2, labelY + 14);
  }

  return canvas.toBuffer("image/png");
}

// 5. Save a crop of each location, and its page and box
const groundingResults: Record<string, { value: unknown; locations: Location[] }> = {};
const labelsByPage = new Map<number, { label: string; box: Box }[]>();
for (const [field, leaf, blocks] of iterFields(extractResponse.extraction_metadata, groundResponse.grounding)) {
  const locations: Location[] = [];
  if (blocks && blocks.length > 0) {
    // One location per place the value was quoted from
    for (const sourceRange of leaf.ranges ?? []) {
      const location = locate(blocks, sourceRange);
      if (location) locations.push(location);
    }
  }
  groundingResults[field] = { value: leaf.value, locations };

  if (locations.length === 0) {
    // null: nothing in the document was quoted for this value.
    // []: no block matched, which usually means the extraction and structure come from different parses.
    console.log(`${field}: no source location`);
    continue;
  }

  locations.forEach((location, i) => {
    const label = `${field} #${i + 1}`;
    fs.writeFileSync(`${outputDir}/${field}_${i + 1}.png`, renderPng(location.page, location.box));
    labelsByPage.set(location.page, [...(labelsByPage.get(location.page) ?? []), { label, box: location.box }]);
    console.log(`${label}: ${JSON.stringify(leaf.value)} on page ${location.page}`);
  });
}

fs.writeFileSync(`${outputDir}/pay-stub_grounding_output.json`, JSON.stringify(groundingResults, null, 2));

// 6. Draw every location on its page, labeled with the field name and occurrence
for (const [page, labels] of labelsByPage) {
  fs.writeFileSync(`${outputDir}/page_${page}_grounding.png`, await drawLocations(page, labels));
}

3. Run the Script

Run the script from the same directory:

python grounding.py

npx tsx grounding.mts

4. View Output

The script saves the following files to the output folder:

File	Description
`pay-stub_parse_output.json`	Full parse response, including the `structure` tree of pages and blocks.
`pay-stub_extract_output.json`	Full extract response, including the extracted values and the `ranges` behind each one.
`pay-stub_ground_output.json`	Full Ground response: for each field, the blocks it was quoted from. See The Grounding Tree.
`pay-stub_grounding_output.json`	The extracted value of each field and the page number and bounding box of every place it appears.
`employee_name_1.png`, `employee_name_2.png`	Cropped images of the two places the employee name appears, numbered in the order of the field’s `ranges`.
`gross_pay_1.png`	Cropped image of where the gross pay appears.
`page_1_grounding.png`	Page 1 with a box around every location, labeled with the field name and occurrence, such as `employee_name #1`. A value in a table is boxed and labeled at its cell.

Field Locations

Each entry in pay-stub_grounding_output.json is keyed by the field’s path in your schema (a nested field reads issuer.name, an array item items[0].price) and has these fields:

Field	Description
`value`	The extracted value.
`locations`	One location for each entry in the field’s `ranges`, in the same order. Location 1 is saved as `<field>_1.png` and labeled `<field> #1` on the page. Empty when the value has no source location.
`locations[].page`	The 1-indexed page number the location is on.
`locations[].box`	The bounding box around the location, as `xmin`, `ymin`, `xmax`, and `ymax` fractions of the page width and height, from 0 to 1.
`locations[].block_ids`	The IDs of the blocks the location was quoted from, such as `text-4` or `table_cell-7`. Look them up in the `structure` of `pay-stub_parse_output.json`.

On the pay stub, the employee name appears twice and the gross pay once, in a table cell. A run on the pay stub saved this file. Your boxes and block IDs can differ slightly, because they come from your own parse:

pay-stub_grounding_output.json

{
  "employee_name": {
    "value": "JANE HARPER",
    "locations": [
      {
        "page": 1,
        "box": { "xmin": 0.57331, "ymin": 0.17874, "xmax": 0.69302, "ymax": 0.19093 },
        "block_ids": ["text-3"]
      },
      {
        "page": 1,
        "box": { "xmin": 0.0791, "ymin": 0.71835, "xmax": 0.92599, "ymax": 0.88537 },
        "block_ids": ["text-7", "text-8", "text-9"]
      }
    ]
  },
  "gross_pay": {
    "value": 452.43,
    "locations": [
      {
        "page": 1,
        "box": { "xmin": 0.30561, "ymin": 0.33065, "xmax": 0.43804, "ymax": 0.34493 },
        "block_ids": ["table_cell-28"]
      }
    ]
  }
}

How the Script Finds Each Location

The Ground response’s grounding field has the same shape as extraction_metadata. The script walks both together, so each extracted value is paired with its grounding, and handles the three values a field’s grounding can take:

Grounding value	What the script does
A list of blocks	Builds one location for each entry in the field’s `ranges`, crops each one, and draws it on the page.
`null`	Records the value with an empty `locations` list: nothing in the document was quoted for it, because the model synthesized the value or did not find one.
`[]`	Records the value with an empty `locations` list. An empty list usually means the extraction and the structure come from different parses, which the `doc_id` check in step 3 of the script prevents. See Pair the Extraction with Its Own Parse.

A value has one range for each place it was quoted from (see Grounding with Ranges), and Ground returns the blocks for all of a field’s ranges in one list (see Matching Blocks). For each range, the script:

Keeps the blocks this range overlaps. Every returned block and every atomic_grounding entry carries its own range in the Markdown. The script keeps the blocks whose grounding.range overlaps the field’s range, the same overlap Ground uses: the start is inclusive and the end is exclusive.
Keeps the cell, not the table. A value inside a table matches both the table block and the table_cell that contains it. The cell names the table in parent_id, so the script drops the table.
Uses the matching lines or words. Each returned block carries the atomic_grounding entries the field’s ranges overlap: lines with DPT-3 Pro, words with DPT-3 Verity. The script keeps the entries this range overlaps, because their boxes are tighter than the block’s. When a block has no matching entries (a table cell parsed with DPT-3 Pro), the script uses the block’s own box. See Matches Inside a Block.
Merges the boxes on the range’s first page. The script combines the remaining boxes on the first page the range appears on into one box, and crops that region from the page rendered at 150 DPI. Page numbers are 1-indexed, so the script subtracts 1 when it opens a PDF page.
Numbers and labels the location. Locations are numbered in the order of the field’s ranges, starting at 1. The script saves the crop as <field>_<number>.png and draws the box on a copy of the page with the label <field> #<number>, table cells included.

Next Steps

Now that you have a working script, you can:

Replace pay-stub.pdf with another PDF you want to parse, extract from, and ground. The Python script also accepts image files.
Modify the schema to extract different fields. For guidance, see Extraction Schema (JSON).
Draw every block the Parse API found, not only the ones a field was quoted from. See Visualize Parsed Blocks.
Read every field of the Ground response. See Ground Extracted Fields.

Overview

Parse Only

Parse & Extract

Link Extracted Data to Document Locations

1. Download the Document to Process

2. Create the Script

3. Run the Script

4. View Output

Field Locations

How the Script Finds Each Location

Next Steps

​1. Download the Document to Process

​2. Create the Script

​3. Run the Script

​4. View Output

​Field Locations

​How the Script Finds Each Location

​Next Steps

1. Download the Document to Process

2. Create the Script

3. Run the Script

4. View Output

Field Locations

How the Script Finds Each Location

Next Steps