Intelligent document processing (IDP) explained, from OCR and templates to LLMs

Intelligent document processing (IDP) is software that reads incoming documents such as invoices, delivery notes, forms, IDs or contracts, pulls out the data you need, checks it and passes it into your systems. It combines several techniques: OCR to turn images into text, rules or templates to find fields in known layouts, machine learning or large language models (LLMs) for varied layouts, and human review for anything the system is unsure about.

No single technique is best. The right mix depends on how many different layouts you receive, how clean the scans are, how costly an error is, and what data protection rules apply. This article explains each layer, when it fits, and how to measure whether it works.

Two directions: reading documents versus generating them

Document automation runs in two directions, and it helps to keep them apart.

• Generation (output side): you have structured data and produce a document from it, such as a contract, offer or certificate. We cover this in how we cut contract drafting from 45 minutes to 90 seconds and in our technical guide to contract generation with Python.
• Processing (input side): you receive documents and need structured data out of them. That is IDP.

Generation is deterministic: the same input always produces the same document. Processing is probabilistic: the system makes a best guess and attaches some measure of confidence. That difference shapes everything about how IDP systems are designed, tested and operated.

The IDP pipeline, step by step

Most IDP systems follow the same stages, whatever tools they use.

1. Ingestion. Documents arrive by email, upload, scanner or API and are stored with an ID and a timestamp.
2. Pre-processing. Pages are split, rotated, de-skewed and cleaned up. Native PDFs that already contain text can skip OCR entirely.
3. Classification. The system decides what each document is: invoice, credit note, delivery note, something else.
4. Extraction. The fields are pulled out: supplier, invoice number, dates, totals, line items.
5. Validation. Extracted values are checked against rules and other systems.
6. Human review. Low-confidence or failed items go to a person.
7. Export. Approved data is written to the ERP, accounting system or database, with a link back to the source document.

The technique choices below mostly concern steps 2 to 4. Steps 5 and 6 are where accuracy is actually won.

OCR: turning pixels into text

Optical character recognition converts an image of text into machine-readable characters. It is only needed for scans, photos and image-only PDFs. Open-source engines such as Tesseract, which recognises more than 100 languages out of the box, are a solid baseline for printed text. Commercial cloud OCR services often add layout and table detection and handwriting support. It is worth testing both on a sample of your own worst scans rather than relying on general claims.

OCR on its own does not give you an invoice number. It gives you text and coordinates. Something else has to decide which piece of text is the invoice number. OCR quality also sets a ceiling for everything downstream.

Fits when: you receive scans or photos. Skip it for documents that are already digital text.

Template and zonal extraction

Template (or zonal) extraction defines, for each known layout, where each field sits: "the invoice number is in this box, top right" or "the value after the label Invoice No.". It is fast, cheap to run, explainable and very accurate on the layouts it knows.

Its weakness is variety: every new supplier or form version needs a new template, and a layout change silently breaks extraction until someone notices.

Fits when: few layouts, high volume per layout, layouts that rarely change.

Machine learning and LLM extraction

Trained extraction models

Machine learning models trained on labelled examples learn to find fields across many layouts without a template per layout. Many commercial IDP products and cloud document services include pre-trained models for common document types such as invoices and receipts. They generalise far better than templates, but need representative training or fine-tuning data for unusual document types.

Large language models

LLMs, including multimodal models that read page images directly, can extract fields from documents they have never seen, given a clear instruction and a target schema (for example a JSON structure with the fields you expect). This makes them attractive for long-tail layouts and for free-text documents such as contracts or letters, where you need to extract meaning, not just values from boxes.

The trade-offs:

• Plausible errors. An LLM can return a well-formatted but wrong value, or fill a field that is not in the document. Validation is not optional.
• Confidence is weaker. Template and trained extraction can produce a usable confidence per field. With LLMs you usually need to derive confidence yourself, for example by cross-checking values against rules or source text.
• Cost and latency per page are higher than with templates.
• Data protection questions are sharper when documents leave your infrastructure. More on this below.

Fits when: many varied layouts, low volume per layout, or unstructured text. In our experience a combination works best: templates or trained models for the high-volume layouts, an LLM for the long tail, and the same validation layer for all of them.

Human-in-the-loop and confidence thresholds

Every IDP system makes mistakes. The design question is which ones reach your systems, and the answer is a review queue governed by thresholds.

A practical setup:

• Each extracted field gets a confidence score from the extractor, adjusted by validation results.
• Above a high threshold, and passing all validation rules, the document goes straight through.
• Below it, or failing any rule, the document goes to a reviewer who sees the source page next to the extracted values and corrects them.
• Reviewer corrections are logged and used to improve templates, prompts or models.

Validation rules catch more errors than better models do. Typical rules: line items sum to the subtotal, subtotal plus tax equals the total, the VAT number has the right format, the purchase order exists in the ERP, the supplier's bank details match the master data, the invoice number has not been seen before.

Set thresholds from data, not intuition. Run a sample of real documents, compare extraction with a hand-checked answer key, and choose the threshold where the error rate of straight-through documents is acceptable for your business.

Measuring accuracy properly

Vendors often quote a single accuracy figure. Ask how it was measured. The numbers that matter, and how to calculate them on your own sample:

• Field-level accuracy: correct fields divided by total fields, per field type. Totals and IBANs matter more than a postal code.
• Document-level accuracy: documents with every required field correct, divided by all documents.
• Straight-through rate: documents that needed no human touch, divided by all documents.
• Error escape rate: wrong values that passed into your systems without review, divided by all documents. This is the one that costs money.

Example, with hypothetical numbers: on a test set of 500 invoices, 410 go straight through and 4 of those contain a wrong field. The straight-through rate is 82% and the error escape rate is 0.8%. Whether that is acceptable depends on what one escaped error costs you.

Build the test set from your own documents, including bad scans, and rerun it whenever templates, prompts or models change.

Data protection and GDPR considerations

Incoming documents are often full of personal data: names, addresses, ID numbers, bank details, sometimes health information. Under the GDPR, a few points deserve attention from the start.

• Processors need a contract. If an OCR, IDP or LLM provider processes the documents for you, Article 28 GDPR requires a binding agreement with that processor and sufficient guarantees about its security measures. Check where data is processed and whether it is retained or used to train models.
• Minimise and delete. Article 5 requires personal data to be limited to what is necessary and kept no longer than needed. Extract only the fields you need and set retention periods for source images and intermediate outputs, including logs and review queues.
• Automated decisions. If extracted data feeds a decision with legal or similarly significant effect on a person, such as rejecting an application, Article 22 restricts decisions based solely on automated processing. Human review is a design requirement there, not just a quality measure.
• Keep sensitive workloads local where needed. Self-hosted OCR and models keep documents in your own infrastructure, at the cost of more operational work.

This is general guidance, not legal advice. Involve your data protection officer early.

A decision checklist

Before choosing tools, answer these:

1. How many distinct layouts do you receive, and how many documents per layout per month?
2. Are documents born-digital PDFs, scans, photos or a mix?
3. Which fields do you need, and which are critical?
4. What does one wrong value cost if it reaches your systems?
5. Which validation data do you already have, such as purchase orders, supplier master data or customer records?
6. Can documents leave your infrastructure, and under which agreements?
7. Who will review exceptions, and how many per day can they handle?

If you also need to match the extracted data against other systems, our article on invoice reconciliation automation, build vs buy covers that next step.

Frequently asked questions

Is OCR the same as intelligent document processing?

No. OCR converts images into text. IDP is the full process around it: classification, field extraction, validation, human review and export into your systems. Many IDP projects involve born-digital PDFs and need no OCR at all.

Can an LLM replace templates entirely?

It can handle many layouts without templates, but it is not automatically the best option. For a few high-volume, stable layouts, templates or trained models are usually cheaper per page, faster and easier to verify. LLMs earn their place on the long tail and on unstructured text.

How accurate can IDP get?

It depends heavily on document quality, layout variety and the validation data available, so be wary of a single headline figure. Measure field-level accuracy and error escape rate on a sample of your own documents. With good validation rules and a review queue, the goal is that wrong values do not reach your systems unchecked.

Do we still need people?

Yes, but for different work. People review exceptions, handle documents the system cannot classify, and improve rules over time. The aim is that they look at the uncertain few percent rather than retyping everything.

Silicode builds both sides of document automation, generation and processing. See our document automation service for what that covers. If you want to test IDP on a sample of your own documents, book a call or request a quote.