Imagine the back-office of a medium-sized distribution company, receiving dozens or hundreds of invoices daily from various suppliers worldwide. Each supplier has their own document format. Some send clean digital PDFs generated directly from their ERP, while others send scanned paper documents crookedly photographed with a mobile phone, where a physical stamp overlaps the total amount including tax.
In the traditional IT world, this problem was addressed by deploying Optical Character Recognition (OCR) systems. However, these tools operate on rigid templates or predefined coordinate rules. If a supplier changes the layout of invoice elements by just a few pixels, for example shifting the item table downward or moving the corporate tax ID to a different corner, the template fails. The system either reads nothing or incorrectly maps the reference number to the total amount. That creates constant manual maintenance of hundreds of templates, accounting staff frustration, high error rates, and unnecessary license fees for software that still requires human oversight.
At the technology studio nolimeo, we design and implement templateless data extraction systems for our clients, built on modern Vision-Language Models (VLMs). These models do not view a document as a set of fixed coordinates; instead, they "see" and interpret it semantically, much like a human would. They understand the meaning of the supplier name, due date, or the itemized table even when the layout shifts from one document to another.
In this technical guide, we will show you how to build a production-grade, asynchronous invoice extraction pipeline in TypeScript, how to validate data using the Zod library, and how to handle poor document quality using Dead Letter Queues (DLQ) and human verification.
1. Why Traditional OCR Fails and How Vision-Language AI Differs
Template-based systems (such as older versions of ABBYY, Tesseract, or legacy enterprise OCR tools) rely on zonal coordinate analysis. A developer must configure zones for each supplier: “Look for the supplier name at coordinates X:100, Y:200.”
This approach has three fatal limitations:
- Extreme Rigidity: If a supplier adds a single line of text, the entire page layout shifts downward, and the zone scanner reads incorrect fields.
- Lack of Semantic Understanding: Traditional OCR merely transcribes pixels to text. It cannot deduce whether the number
20251120represents a due date, a reference code, or an order number. - High Maintenance Costs: If a company has 500 suppliers, managing and updating 500 different templates becomes an ongoing nightmare for internal IT departments.
Vision-Language Models (VLMs) (such as GPT-4o, Claude 3.5 Sonnet) merge visual perception with natural language processing. They analyze an invoice as a cohesive image while reading text with a deep understanding of context. When instructed to “Extract the total amount excluding tax,” the model knows to look for labels like “Total Net,” “Subtotal,” “Net Amount,” or “Value before tax,” evaluates their proximity to numerical fields, and returns the correct value, wherever it is placed.
2. Architecture of a Robust AI Extraction Pipeline
When dealing with financial documents, we cannot allow the AI to generate unconstrained free text. Large language models can return poorly formatted JSON or hallucinate missing data in edge cases.
To keep the integration reliable with accounting platforms (such as QuickBooks, Xero, NetSuite, or SAP), we build our pipeline on three main pillars:
- Visual Document Analysis (OpenAI Vision API): We convert the PDF or image (PNG/JPEG) into a base64 string and transmit it to the model alongside structured instructions.
- Enforced JSON Format (Structured Outputs): We leverage API-level JSON schemas (such as OpenAI's
response_format), forcing the model to return data matching our exact database types. - Type Safety and Runtime Validation (Zod): We validate the returned JSON using TypeScript's Zod library. Zod validates data types, normalizes dates to ISO formats, recalculates sums to verify mathematical consistency, and catches errors before any database write occurs.
[ Scanned Invoice ] ──► [ Convert PDF to Base64 ] ──► [ OpenAI GPT-4o Vision ]
│
(Structured JSON Output)
▼
[ Write to ERP / DB ] ◄── [ Tax & Total Check ] ◄── [ Zod Schema Validation ]
3. Technical Implementation in TypeScript and Node.js
Below is a complete backend implementation in TypeScript. The code covers Zod schemas, document processing via OpenAI, and cross-validation to verify that itemized totals match the overall invoice amount.
// src/services/invoice-processor.ts
import { OpenAI } from "openai";
import { z } from "zod";
// 1. Definition of a strict Zod schema for an invoice item
const InvoiceItemSchema = z.object({
description: z.string().describe("Name of the product or service"),
quantity: z.number().positive().describe("Quantity"),
unitPrice: z.number().nonnegative().describe("Unit price excluding tax in USD"),
vatRate: z.number().nonnegative().describe("Tax rate percentage (e.g., 20)"),
totalPriceWithoutVat: z.number().nonnegative().describe("Total line price excluding tax"),
});
// 2. Definition of a schema for the entire invoice
const InvoiceDataSchema = z.object({
invoiceNumber: z.string().describe("Invoice or tax document identification number"),
variableSymbol: z.string().nullable().describe("Payment reference, memo, or identifier number"),
issueDate: z.string().regex(/^\d{4}-\d{2}-\d{2}$/).describe("Date of issue in YYYY-MM-DD format"),
dueDate: z.string().regex(/^\d{4}-\d{2}-\d{2}$/).describe("Due date in YYYY-MM-DD format"),
supplierName: z.string().describe("Supplier business name"),
supplierIco: z.string().describe("Supplier registration / corporate ID number"),
supplierDic: z.string().describe("Supplier tax ID / VAT number"),
supplierIban: z.string().describe("Supplier bank account IBAN or routing details"),
customerName: z.string().describe("Customer business name"),
customerIco: z.string().describe("Customer registration / corporate ID number"),
items: z.array(InvoiceItemSchema).describe("List of invoiced items"),
totalAmountWithoutVat: z.number().nonnegative().describe("Total invoice amount excluding tax"),
totalVatAmount: z.number().nonnegative().describe("Total tax/VAT amount in USD"),
totalAmountWithVat: z.number().nonnegative().describe("Total amount due including tax"),
currency: z.string().default("USD").describe("Invoice currency code (e.g., USD, GBP, EUR)"),
});
export type InvoiceData = z.infer<typeof InvoiceDataSchema>;
export class AIInvoiceProcessor {
private openai: OpenAI;
constructor() {
const apiKey = process.env.OPENAI_API_KEY;
if (!apiKey) {
throw new Error("Missing OPENAI_API_KEY in environment variables.");
}
this.openai = new OpenAI({ apiKey });
}
/**
* Accepts invoice image as a base64 string (supports PNG, JPEG, WEBP) and extracts structured data
*/
public async processInvoiceImage(base64Image: string, mimeType: string = "image/jpeg"): Promise<InvoiceData> {
try {
const prompt = `Analyze the attached invoice image and extract all relevant information in JSON format.
Ensure that:
1. All monetary values are converted to decimal numbers.
2. Dates are strictly formatted as YYYY-MM-DD.
3. If a payment reference or memo is not explicitly stated, use the invoice number.
4. Clean up any whitespace in registration numbers.`;
// Call OpenAI API using structured outputs
const response = await this.openai.chat.completions.create({
model: "gpt-4o",
temperature: 0.0, // Zero temperature for maximum accuracy
response_format: { type: "json_object" }, // Enforced JSON format
messages: [
{
role: "user",
content: [
{ type: "text", text: prompt },
{
type: "image_url",
image_url: {
url: `data:${mimeType};base64,${base64Image}`,
},
},
],
},
],
});
const rawJson = response.choices[0]?.message?.content;
if (!rawJson) {
throw new Error("Model failed to return content.");
}
const parsedData = JSON.parse(rawJson);
// 3. Run runtime validation via Zod schema
const validatedData = InvoiceDataSchema.parse(parsedData);
// 4. Mathematical self-check
this.validateCalculations(validatedData);
return validatedData;
} catch (error) {
if (error instanceof z.ZodError) {
console.error("Zod schema validation failed for invoice data:", error.errors);
throw new Error(`Data structure validation error: ${JSON.stringify(error.errors)}`);
}
console.error("System error during invoice processing:", error);
throw error;
}
}
/**
* Performs mathematical cross-checks on extracted values to prevent LLM hallucinations
*/
private validateCalculations(data: InvoiceData): void {
const TOLERANCE = 0.05; // 5-cent tolerance for rounding variations
// Sum of individual items excluding tax
const calculatedSubtotal = data.items.reduce((sum, item) => sum + item.totalPriceWithoutVat, 0);
const subtotalDiff = Math.abs(calculatedSubtotal - data.totalAmountWithoutVat);
if (subtotalDiff > TOLERANCE) {
throw new Error(
`Mathematical inconsistency: Sum of items (${calculatedSubtotal.toFixed(2)} ${data.currency}) ` +
`does not match the total amount excluding tax (${data.totalAmountWithoutVat.toFixed(2)} ${data.currency}).`
);
}
// Verification of total amount including tax
const expectedTotalWithVat = data.totalAmountWithoutVat + data.totalVatAmount;
const totalDiff = Math.abs(expectedTotalWithVat - data.totalAmountWithVat);
if (totalDiff > TOLERANCE) {
throw new Error(
`Mathematical inconsistency: Sum of subtotal and tax (${expectedTotalWithVat.toFixed(2)} ${data.currency}) ` +
`does not match the extracted total amount including tax (${data.totalAmountWithVat.toFixed(2)} ${data.currency}).`
);
}
console.log(`Mathematical validation for invoice ${data.invoiceNumber} completed successfully.`);
}
}
4. Handling Edge Cases and Failures in Production
Experienced developers know that in production, everything that can go wrong will go wrong. When processing financial data, we must anticipate failures and design systems to ensure no document is lost and no invalid records enter the accounting ledgers.
4.1. Blurred or Low-Quality Scans
If document scans are blurry, the LLM may return partial information, leading to Zod validation failures (such as a missing bank details or an invalid date format).
- Our Solution: The system catches the
ZodErrorinstead of throwing an unhandled504or500crash response. We save the raw JSON draft, label the database entry asREQUIRES_HUMAN_REVIEW, and present it in our custom back-office dashboard. The review interface displays the original document directly alongside the partially extracted fields, allowing staff to correct the fields with a single click.
4.2. Dead Letter Queues (DLQ) for Asynchronous Processing
We process incoming documents asynchronously using message queues (e.g., BullMQ in Node.js). If processing fails due to temporary OpenAI API limits (e.g. Rate Limit 429), we execute an Exponential Backoff Retry strategy (retrying after 5s, 30s, 5m, etc.).
If a job fails 5 times consecutively due to a corrupted file or permanent API error, the task is moved to a Dead Letter Queue (DLQ). This prevents the document from blocking the pipeline for other incoming invoices and triggers an automated Slack alert to notify our development team.
[ New Invoice ] ──► [ Process in BullMQ ]
│
(Error: Rate Limit API) ──► Automatic Retry (x5)
│
(Error: Corrupted File)
▼
[ Dead Letter Queue (DLQ) ] ──► Slack Alert / Manual Review
4.3. Multi-Page and High-Volume Documents
Large PDF files containing dozens of pages (such as monthly utility invoices) consume unnecessary tokens and increase execution costs.
- Our Solution: Our pipeline analyzes the PDF structure first. If it is a multi-page document, we split it using libraries like
pdf-libor helper utilities likegraphicsmagick, extracting only the first and last page (where supplier metadata and payment totals are located). This reduces token usage and keeps the API workload lower for multi-page documents.
5. Integrating Extracted Data into Accounting ERPs
Once data extraction and cross-validation succeed, the data must be securely posted to the accounting system. Many legacy regional ERPs do not offer REST APIs or charge high integration fees.
We solve this using two patterns:
- Standardized XML/ISDOC Generation: Our backend transforms the validated JSON payload into XML schemas aligned with the accounting software imports (such as QuickBooks, Sage, or Xero XML import formats). These files are uploaded to secure SFTP or S3 directories where the accounting software automatically synchronizes them.
- Dedicated REST API Middleware: For modern cloud-native accounting platforms, we develop dedicated Node.js middleware that converts validated JSON structures into live API payloads, recording transaction logs for auditing.
Conclusion: Turn Manual Tasks into Automated Workflows
Deploying Vision-Language AI to automate invoice entry reduces tedious manual work that wastes the time of skilled finance professionals. A properly designed pipeline can handle document extraction quickly with a strong validation layer, while legacy coordinate-based OCR engines often struggle as layouts change.
However, when implementing enterprise AI systems, it is vital to avoid third-party integrations that route your sensitive financial documents through unverified channels. Financial records represent your most critical data, and their extraction must run inside secure, encrypted, and type-safe architectures.
At nolimeo, we are a specialized software engineering studio. We do not sell pre-packaged, generic SaaS licenses, nor do we build fragile "no-code" automations. We write robust, custom TypeScript code, deploy high-performance database architectures, and configure secure API pipelines under the supervision of our senior engineers.
Ready to automate invoice entry inside your organization, integrate data extraction with your ERP, and reduce manual processing errors? Contact us and we’ll review your documents, ERP flow, and the safest technical direction.