Custom extraction is useful when an application needs documents to become structured business data rather than only searchable text. Azure AI Document Intelligence can train custom models from labeled examples and return fields as typed values that downstream systems can validate and process. The engineering challenge is not simply getting a model to recognize a field. It is defining a stable schema, preparing representative training data, and deciding what the application should do when confidence is low.
Microsoft’s current Document Intelligence v4.0 guidance distinguishes custom neural and custom template approaches. Neural models are generally the preferred starting point for mixed or variable document layouts, while template models fit highly consistent visual forms. A small set of examples can start training, but production quality depends much more on coverage and labeling discipline than on meeting the minimum.
Within Microsoft AI agents, custom extraction is most valuable when an agent or workflow needs reliable fields before reasoning continues. It turns an unstructured file into a contract that software can inspect.
Choose extraction only when the downstream system needs fields
Not every document problem needs a custom model. If the application needs reading order, paragraphs, tables, and figures for search or summarization, Azure AI Document Intelligence Layout may provide enough structure without a labeled schema. Custom extraction adds value when downstream code expects named fields such as invoice number, policy date, claimant name, total amount, or a repeated table.
The distinction helps avoid overengineering. A retrieval system can often work from layout-aware chunks and metadata. A claims workflow that must populate a system of record needs stronger field semantics, normalization, and validation. The model choice should follow the data contract required by the next component.
The planned multimodal document extraction article expands this decision for files where vision and generated interpretation are part of the pipeline rather than only deterministic field extraction.
Start with custom neural unless a stable template gives you a reason not to
Custom neural models are designed for structured, semi-structured, and unstructured documents and can adapt to broader layout variation. Microsoft recommends starting with neural models when the scenario supports them. Custom template models rely more heavily on consistent visual placement and are appropriate when the blank form remains effectively the same from one document to the next.
This choice should be tested against real variation. A document family that looks consistent to a human may contain different page breaks, optional sections, scanned versions, signatures, or alternate vendor templates. Training one template model across incompatible layouts can reduce accuracy even if all of the documents represent the same business type.
Where multiple layouts are legitimate, it can be better to train separate models and compose them, or classify the document type first and route it to the appropriate extractor. The extraction boundary should reflect the real document population rather than force every variant into one model.
Training data quality dominates the minimum sample count
Document Intelligence can start a custom extraction model with as few as five labeled examples of the same document type. That minimum is useful for experimentation, not a guarantee of production robustness. Training data should represent the actual variation the system will see: different values, page counts, scan quality, fonts, optional fields, table lengths, and document versions.
Microsoft recommends text-based PDFs when possible, complete examples with fields populated, different values across fields, and larger data sets when image quality is poor. These suggestions matter because a model can appear accurate on near-duplicate training forms while failing on the first unusual real document.
A useful data split includes documents that are never used for training. Those holdout samples provide a more honest estimate of field-level performance and help the team detect whether the model learned the document family or merely the training examples.
Label a business schema, not every visible string
Custom extraction works best when the labeled fields correspond to stable business concepts. Labeling every visible value can create a large schema that is expensive to maintain and difficult to validate. Instead, define the fields the application actually needs and give them names and types that match downstream logic.
Tables deserve special thought. A repeated line-item region may be better modeled as a structured table than as dozens of individually named fields. Optional fields should be explicit so the application can distinguish “not present” from “extraction failed.” Overlapping fields and signatures have support in current custom neural capabilities, but the application still needs rules for what to do when multiple candidates exist.
The schema should also survive document revisions. If a vendor changes the label from “Customer ID” to “Account ID” but the business meaning is the same, the extracted field should ideally remain stable so downstream systems do not need a release for every formatting change.
Confidence should drive workflow decisions, not cosmetic dashboards
Field confidence is useful only when it changes behavior. High-confidence fields may flow straight through. Medium-confidence fields may require additional validation against business rules. Low-confidence fields may need human review or a second extraction strategy. A single global threshold is often too crude because some fields are riskier than others.
For example, a wrong document date may be inconvenient while a wrong bank account number may be unacceptable. The workflow should define field-specific acceptance rules where risk differs. Cross-field validation can also help: totals can be checked against line items, dates can be checked for plausible ranges, and identifiers can be validated against known formats.
This is an important difference between extraction and generation. The application should not assume that a plausible-looking value is correct merely because the model returned it cleanly.
Classification and extraction can be separate stages
Mixed inbound document streams often contain several document types. A custom classifier can identify the document class before the extraction model runs. This reduces pressure on one extractor to understand unrelated layouts and allows each model to specialize in a coherent family.
Classification has its own uncertainty. If the production stream contains document types that were not part of training, the system should define what happens below the classification threshold or include a representative “other” class. Routing an unknown document confidently into the wrong extractor can produce structured but meaningless data.
For agent workflows, this staged design is valuable because the agent can reason from a typed result and explicit document class instead of guessing what kind of form it received from raw OCR text.
Production monitoring should watch document drift
Document populations change. Vendors redesign forms, internal templates move fields, scanners change, and new edge cases appear. A model that was strong at launch can degrade gradually without any code deployment. Monitoring should therefore track field confidence, human correction rates, failure categories, and document versions over time.
When drift appears, the fix may be new training examples, a separate model for the new layout, improved scanning, or a revised schema. The training set should be versioned so the team can explain which documents and labels produced a deployed model.
Custom extraction is successful when it becomes boring infrastructure: known document families go in, validated business fields come out, low-confidence cases take a controlled exception path, and the system notices when the document world changes.
Evaluation should be field-specific and business-aware
An overall extraction accuracy number can hide the fields that matter most. A model may perform extremely well on names and dates while struggling with a rare field that drives a high-value downstream decision. Evaluation should report precision, recall, confidence, and correction rate by field or table element where the business risk differs.
Holdout documents should include difficult examples deliberately: low-quality scans, optional sections, unusual page counts, handwriting where supported, alternate templates, and edge values. The point is not to make the benchmark easy to pass; it is to estimate how the production exception queue will behave.
Business rules can supplement model evaluation. A tax identifier can be checked for format, a total can be reconciled with line items, and a date can be compared with the document period. Those checks do not improve the model directly, but they make the overall extraction system more trustworthy.
Human review should produce training evidence, not disappear into email
Low-confidence or failed extractions often require a person. The review interface should capture the corrected value, the source location, and the reason for the exception so the team can decide whether the model, document quality, or business rule needs improvement. Corrections that live only in an email thread cannot improve the next model version.
Over time, reviewed examples can become candidate training data, but they should be curated. A rushed human correction can be wrong, and a rare one-off document may not belong in the main training distribution. Versioning the accepted training set makes model changes explainable.
This creates a productive feedback loop: production exceptions become labeled evidence, evidence improves the model or routing, and the exception rate should decline for the document patterns the system has learned.
Model promotion should also be reversible. Keep the previous production model available long enough to compare a new version under shadow or limited traffic, and record which model version produced each extraction. If a retrained model improves most fields but regresses one critical field, operators need a fast rollback path instead of rebuilding the entire project from memory.
