Async Parse: Processing Multiple Documents Concurrently

Overview

Process multiple documents concurrently to significantly reduce total processing time compared to sequential requests.

These examples require the Python or TypeScript client library. Before running a script, set your API key and install the library and any required dependencies.

Python

Use AsyncLandingAIADE for async document processing

TypeScript

Use concurrent parsing with Promise.all() or p-limit

Python

Use AsyncLandingAIADE when you need to process many lightweight documents (such as invoices, receipts, or forms) efficiently. This async client allows you to send multiple parse requests concurrently using Python’s asyncio, which significantly reduces total processing time compared to sequential requests. The async approach lets you send multiple requests in parallel. While one document is being processed, another request can be sent. The API server handles the actual document processing in the background. To avoid exceeding the pages per hour limits and receiving 429 errors, use a client-side rate limiter like aiolimiter to control concurrency.

import asyncio
from pathlib import Path
from landingai_ade import AsyncLandingAIADE

# Initialize client (uses the API key from the VISION_AGENT_API_KEY environment variable)
client = AsyncLandingAIADE()

async def main() -> None:
    response = await client.parse(
        document=Path("path/to/file"),
        model="dpt-2-latest"
    )
    print(response.chunks)

    # Save markdown output (useful if you plan to run extract on the markdown)
    with open("output.md", "w", encoding="utf-8") as f:
        f.write(response.markdown)

asyncio.run(main())

TypeScript

Use concurrent parsing when you need to process many lightweight documents (such as invoices, receipts, or forms) efficiently. The TypeScript library’s methods are already asynchronous, allowing you to send multiple parse requests concurrently using JavaScript’s Promise.all() or Promise.allSettled(). This significantly reduces total processing time compared to sequential requests. The concurrent approach lets you send multiple requests in parallel. While one document is being processed, another request can be sent. The API server handles the actual document processing in the background. To avoid exceeding the pages per hour limits and receiving 429 errors, use a concurrency control library like p-limit to limit the number of simultaneous requests.

Basic Concurrent Parsing

import LandingAIADE from "landingai-ade";
import fs from "fs";
import path from "path";

// Initialize client (uses the API key from the VISION_AGENT_API_KEY environment variable)
const client = new LandingAIADE();

async function parseMultipleDocuments() {
  // Replace with your folder path containing documents to parse
  const dataFolder = "data/";
  const files = fs.readdirSync(dataFolder);

  // Filter for file types (adjust as needed for your use case)
  const documentFiles = files.filter(filename => {
    const fileExt = path.extname(filename).toLowerCase();
    return ['.pdf', '.png', '.jpg', '.jpeg'].includes(fileExt);
  });

  // Parse all documents concurrently
  const parsePromises = documentFiles.map(filename => {
    const filepath = path.join(dataFolder, filename);
    return client.parse({
      document: fs.createReadStream(filepath),
      model: "dpt-2-latest"
    }).then(response => ({
      filename,
      response
    })).catch(error => ({
      filename,
      error: error.message
    }));
  });

  // Wait for all parsing to complete
  const results = await Promise.all(parsePromises);

  // Process results
  for (const result of results) {
    if ('error' in result) {
      console.error(`Failed to parse ${result.filename}: ${result.error}`);
    } else {
      console.log(`Successfully parsed ${result.filename}`);

      // Save Markdown output
      const outputMd = path.basename(result.filename, path.extname(result.filename)) + ".md";
      fs.writeFileSync(outputMd, result.response.markdown, "utf-8");
    }
  }
}

parseMultipleDocuments();

Concurrent Parsing with Rate Limiting

To control concurrency and avoid rate limits, use the p-limit library:

npm install p-limit

import LandingAIADE from "landingai-ade";
import fs from "fs";
import path from "path";
import pLimit from "p-limit";

// Initialize client (uses the API key from the VISION_AGENT_API_KEY environment variable)
const client = new LandingAIADE();

// Adjust concurrency limit as needed to avoid rate limits
const limit = pLimit(5);

async function parseMultipleDocumentsWithRateLimit() {
  // Replace with your folder path containing documents to parse
  const dataFolder = "data/";
  const files = fs.readdirSync(dataFolder);

  // Filter for file types (adjust as needed for your use case)
  const documentFiles = files.filter(filename => {
    const fileExt = path.extname(filename).toLowerCase();
    return ['.pdf', '.png', '.jpg', '.jpeg'].includes(fileExt);
  });

  // Parse all documents with concurrency control
  const parsePromises = documentFiles.map(filename => {
    // Wrap each parse call with the rate limiter
    return limit(async () => {
      const filepath = path.join(dataFolder, filename);
      console.log(`Parsing ${filename}...`);

      try {
        const response = await client.parse({
          document: fs.createReadStream(filepath),
          model: "dpt-2-latest"
        });

        // Save Markdown output
        const outputMd = path.basename(filename, path.extname(filename)) + ".md";
        fs.writeFileSync(outputMd, response.markdown, "utf-8");

        console.log(`Successfully parsed ${filename}`);
        return { filename, success: true };
      } catch (error) {
        console.error(`Failed to parse ${filename}: ${error.message}`);
        return { filename, success: false, error: error.message };
      }
    });
  });

  // Wait for all parsing to complete
  const results = await Promise.all(parsePromises);

  // Summary
  const successful = results.filter(r => r.success).length;
  const failed = results.filter(r => !r.success).length;
  console.log(`\nCompleted: ${successful} successful, ${failed} failed`);
}

parseMultipleDocumentsWithRateLimit();

Overview

Parse Only

Parse & Extract

Async Parse: Processing Multiple Documents Concurrently

Overview

Python

TypeScript

Python

TypeScript

Basic Concurrent Parsing

Concurrent Parsing with Rate Limiting

​Overview

Python

TypeScript

​Python

​TypeScript

​Basic Concurrent Parsing

​Concurrent Parsing with Rate Limiting

Overview

Python

TypeScript

Basic Concurrent Parsing

Concurrent Parsing with Rate Limiting