Microsoft AI-103: Multimodal Document Extraction
Multimodal document extraction is not simply OCR with a larger model. Business documents combine text, page layout, tables, selection marks, figures, charts, handwriting, headers, and spatial relationships that change meaning when flattened into a plain string. For Microsoft AI agents, an extraction pipeline should preserve document structure before adding generative reasoning so fields can be verified against the source. This makes outputs easier to validate and reduces the risk of asking a model to reconstruct information that a document parser already knows exactly.
For Microsoft AI-103, a production extraction design needs a clear analysis boundary, typed schema, evidence controls, and review path. A model that can “read a document” is not enough when the application needs typed fields and traceable evidence.
Microsoft’s current Document Intelligence Layout API can return structured elements including pages, paragraphs, tables, figures, sections, selection marks, and Markdown-formatted content. Figure extraction can also produce cropped figure images, while newer Content Understanding capabilities can represent images with descriptions or structured figure analysis. These services create a useful boundary between deterministic document parsing and higher-level semantic interpretation.
Preserve the document hierarchy before extracting business fields
Page number, section, heading level, table membership, and figure relationships are evidence. If text is flattened too early, a value in a footnote can become indistinguishable from a main-body statement or a table header can be separated from its rows. Store structural metadata alongside text so later extraction can reason about where information came from.
Reading order deserves explicit testing because visually complex documents can contain sidebars, columns, footnotes, and callouts that are not meant to be consumed linearly. Preserve spans and structural references so the generative layer can reconstruct local context without assuming that every neighboring text block is semantically adjacent. This becomes especially important when documents are later chunked for search or agent use.
For multimodal AI under real constraints, retain the strongest available structural representation for each modality before using a model to interpret meaning. A model should not infer layout from a degraded text dump when the parsing service can provide explicit page and span relationships.
Use Markdown as an interchange format, not as the final truth
Document Intelligence can emit Markdown that preserves headings, paragraphs, HTML tables, figures, and other elements. That format is convenient for LLM context and semantic chunking because it retains more structure than raw OCR text. The application should still retain the underlying JSON response when exact coordinates, spans, confidence-related fields, or figure identifiers matter.
Markdown is convenient because it can carry headings, tables, and figures through text-oriented pipelines, but it remains a serialization. Do not discard the original bounding regions or element IDs after producing Markdown. A reviewer may need to highlight the exact source cell or figure, and a downstream system may need coordinates that cannot be recovered from the rendered text alone.
For enterprise RAG chunking, preserve section identity, intact table rows, and figure captions as part of the retrieval contract. Each chunk should carry a traceable source reference and enough local context to support a precise answer without merging unrelated sections.
Handle tables as structured data before asking a model to summarize them
Tables often contain the most operationally important content in invoices, reports, schedules, and forms. Preserve row and column structure, merged cells, captions, and multi-page continuation logic. Use deterministic normalization where possible, then ask a model to interpret business meaning after the table has been converted into a stable representation.
Multi-page tables deserve dedicated logic because repeated headers, page breaks, and subtotals can look like new records. Preserve page provenance while joining rows, and keep the unmerged representation available for audit. Automated reconciliation against totals or known record counts can catch structural errors before an LLM is asked to interpret them. This is especially important in financial and operational documents where one duplicated row can change a downstream decision.
Data quality and observability should include extraction checks such as unexpected column counts, missing headers, duplicate rows, or sudden changes in table shape. A fluent model summary cannot compensate for a table that was parsed incorrectly at the ingestion layer.
Keep figures and charts connected to their surrounding text
Figures may communicate trends or architecture that surrounding paragraphs merely describe. Document Intelligence exposes figure objects and can generate cropped figure images when figure output is requested. Content Understanding can additionally provide image descriptions and structured figure analysis in supported scenarios. The extraction pipeline should keep figure IDs, captions, page positions, and related text together.
Use Azure computer vision when preprocessing or validating page imagery improves figure extraction before generative reasoning. A chart-reading model should not be asked to interpret a low-quality crop if the original page image can be re-rendered at better resolution or if the needed values already exist in accompanying text.
Define a typed extraction schema for the business outcome
After layout parsing, the generative stage should extract into a schema with explicit field names, types, allowed values, optionality, and evidence references. “Return the important information” is not testable. A schema such as invoice number, supplier, date, currency, line items, subtotal, tax, and total creates a contract that can be validated before data reaches downstream systems.
Content Understanding analyzers provide another way to design that contract. For current production-oriented workflows, Microsoft’s Content Understanding documentation identifies API version 2025-11-01 as generally available, with reusable prebuilt or custom analyzers for supported documents, images, audio, and video. For GA API use and full custom-analyzer authoring, Content Understanding Studio is the supported authoring surface; the Content Understanding playground in Foundry (new) currently supports the 2026-06-01-preview API and a narrower set of authoring functions. A document-analysis workflow should choose an appropriate analyzer, map the desired fields to a typed output, check returned evidence and confidence, and validate totals or cross-field relationships in application code. A successful field extraction does not prove the business record is correct until the evidence has been reviewed against the original file.
Do not silently assume that newer agentic analyzer behavior is part of that GA contract. Microsoft lists the 2026-06-01-preview Content Understanding API as a separate public preview, with optional agentic document understanding, new classification and segmentation features, and a different processing-cost profile. The preview’s agentic mode can reason over evidence but has its own input limits and lacks a production service-level agreement. Choose that mode only for a real task need, record the version explicitly, and compare a representative validation set with the GA alternative instead of promising preview-specific behavior for every analyzer.
The AI-103 computer-vision blueprint also uses the older name pro mode. This referred to the now-retired 2025-05-01-preview Content Understanding API, where standard single-file analysis and pro-mode multi-document reasoning differed. Microsoft now provides an agentic workflow in 2026-06-01-preview, configured with config.workflow; it is not a drop-in replacement and initially supports one input file per analysis request. Keep GA extraction, retired pro mode and current preview agentic analysis distinct rather than instructing readers to enable pro mode in a GA analyzer.
Typed extraction results strengthen API security by supporting server-side format and authorization checks before changing business records. Treat extracted content as untrusted input even when it came from a model. Validate formats, ranges, identifiers, and business rules before using the result to trigger payments, create accounts, or update records.
Separate extraction confidence from generative confidence
A pipeline has multiple uncertainty layers. The parser can misread a character, the layout model can assign text to the wrong structure, and the LLM can map correctly extracted evidence to the wrong business field. Record which stage produced each value and what evidence supports it. A single final confidence score hides where remediation should happen.
The evaluation corpus should deliberately include difficult scans, rotated pages, handwritten annotations, charts, dense tables, and several source formats. Template-perfect PDFs produce reassuring scores but do not reflect a real intake queue. Version every test document so OCR, layout, or prompt changes can be compared against the same evidence, and preserve expected page references so failures can be localized quickly.
Configure generative AI evaluation pipelines to assess field accuracy, source attribution, table integrity, and semantic interpretation separately. This allows teams to tell whether a regression came from document parsing, prompt changes, model changes, or a new document template.
Build human review around exception classes instead of random sampling alone
Human review is most valuable when triggered by specific risk signals: missing required fields, conflicting totals, low-quality scans, unsupported language, ambiguous table structure, high-value transactions, or disagreement between deterministic and generative extraction. Random sampling still helps measure unseen error rates, but exception-driven review protects operational decisions more directly.
Review tools should display the original page beside the proposed structured fields and highlight supporting regions when available. Reviewers should correct the field and choose an error category rather than retyping the entire record. That feedback creates labeled evidence for future evaluation and prevents the review queue from becoming a manual data-entry system that hides which extraction stage actually failed.
At document intake, AI guardrails and content safety should detect untrusted instructions before extracted material influences agent tools. Documents can contain hostile instructions or unsafe content, and an extraction agent that later calls tools must not treat embedded text as trusted operational commands.
Respect file-format differences and preprocessing limits
Different document types expose different capabilities. Current Layout documentation treats PDFs and images as page-based inputs, while Office documents and HTML are processed with format-specific constraints; embedded images in some Office formats are not handled the same way as figures in PDFs. A production system should normalize source files deliberately rather than assuming every format preserves identical visual evidence.
Privacy design should decide how long source documents, cropped figures, and extracted text are retained. Temporary image derivatives can contain the same sensitive content as the original file even when their filenames look harmless. Storage, logs, review queues, and test fixtures all need retention and access rules that follow the underlying document classification. Redaction should happen before content is copied into systems that do not need the original detail.
Production GenAI observability should segment extraction quality by file type, page count, duration, tables and figures, exceptions, and review rates. These dimensions often explain accuracy changes better than aggregate model metrics.
Retain provenance from source page to final field
Every important extracted value should be traceable back to the document page, span, table cell, or figure that supports it. Provenance makes correction possible, lets reviewers verify high-risk fields quickly, and enables downstream systems to show why a value was accepted. It also makes regression tests more diagnostic because the expected evidence can be compared as well as the expected value.
Provenance also makes selective reprocessing possible. If a new parser improves table handling, re-run only documents or pages whose previous outputs depended on the affected table structures rather than replaying every document in the archive. Store extraction version, source checksum, page identifiers, and field evidence so improvements can be applied surgically. This reduces cost and preserves audit history by showing exactly which transformation produced each version of the structured record.
Effective responsible AI at runtime depends on tracing every extracted field back to verifiable source evidence. Microsoft offers strong document parsing and multimodal building blocks, but reliable extraction comes from preserving structure, using typed schemas, validating every stage, and maintaining provenance. The model should interpret a well-formed document representation, not compensate for an ingestion pipeline that discarded the evidence it needed.