A stage-by-stage ADE pipeline guide for enterprise teams: ingestion, parse path selection, field extraction, confidence-based routing, and output delivery.
A document extraction pipeline has five stages. At each stage, the architecture decision is the same question: what does the platform handle, and what does the calling application own? ADE is designed so the platform handles the hard parts -- document complexity, large file splitting, retry logic, compliance constraints -- and the application owns routing decisions and output destinations.
Stage 1: Document Ingestion
Documents enter the pipeline from a source -- a storage bucket (S3, Azure Blob, GCS), an internal system, or an inbound file upload -- and ADE accepts them either as a direct file upload or by URL reference; URL reference is required when Zero Data Retention is enabled. See supported file types for the full list of accepted formats.
Stage 2: Parse Path Selection
Every document is routed to one of two parse paths based on its size and processing urgency:
| Condition | Path | API |
|---|---|---|
| Real-time, latency-sensitive, single document | Synchronous | Parse API |
| Batch, large file, high concurrency | Asynchronous | Parse Jobs API |
The Parse Jobs API accepts documents up to 1 GB or 6,000 pages and processes them asynchronously, returning a job_id immediately. Status is polled via the Get Parse Jobs endpoint and results are retrieved when complete. The Python library handles large-file splitting automatically for documents over 1,000 pages, processing chunks in parallel without any code in the calling application.
Stage 3: Parsing
ADE's parsing layer converts the document into layout-aware Markdown and hierarchical JSON with page and coordinate grounding for every block (text, tables, headers, figures, and form fields) -- the output the extraction stage operates on. Two things happen automatically here that would otherwise require custom code:
- Retry handling. The Python library implements exponential backoff with randomized jitter on transient errors (408, 429, 502, 503, 504), retrying without surfacing failures to the calling application.
- ZDR output routing. When Zero Data Retention is enabled on Parse Jobs, parsed results are written directly to a customer-provided pre-signed URL in customer-controlled storage, not returned through LandingAI's infrastructure. The pipeline retrieves results from that storage location rather than the API response.
Rate limits apply at the organization level and scale with plan tier; Enterprise plans carry customizable limits. See rate limits documentation.
Stage 4: Field Extraction
The parsed Markdown is passed to the Extract API along with a JSON schema that defines which fields to extract and their expected types. The schema is the stable contract between the document and the downstream system: it specifies what data to pull out, not where to find it in any particular layout, so it does not need to change when source documents change format.
The Extract API returns a structured JSON object matching the schema, with a confidence score and bounding-box citation for every extracted field. The extraction model version should be pinned explicitly in the API call -- the current version is extract-20251024 -- so that platform model updates do not silently change extraction behavior in production. See extraction model versions for the changelog.
Stage 5: Confidence Routing and Output Delivery
Extraction output is routed based on the per-field confidence scores before it reaches any downstream system:
- High confidence. Fields above the threshold are written directly to the destination system -- a database, data warehouse, or downstream API -- without review.
- Low confidence. Fields below the threshold are sent to a human review queue. The bounding-box citation returned with each field links the reviewer directly to the source location in the original document, so review is targeted rather than full-document.
- Null returns. Fields that returned null (document did not contain that field) are handled separately from low-confidence fields, allowing the pipeline to distinguish missing data from uncertain extraction.
Confidence thresholds are pipeline configuration, not ADE configuration, and should be calibrated against a representative sample of production documents before go-live.
FAQ
Does the pipeline architecture change when Zero Data Retention is required? Yes, at two stages. At ingestion, documents must be referenced by URL rather than uploaded directly. At the parse stage, a pre-signed output URL pointing to customer-controlled cloud storage must be included in Parse Jobs requests, and the pipeline retrieves results from that location rather than the API response. The extraction stage and confidence routing stage are unchanged. See ZDR documentation for the full requirements.
What does the application need to build versus what does ADE provide? ADE provides: document parsing with layout preservation, large-file splitting, transient error retry logic, per-field confidence scores, and bounding-box citations. The calling application owns: parse path routing logic, the durable job registry for async job_id tracking, the extraction schema definition, confidence threshold values, and output destination writes.
How should the extraction schema be managed across pipeline deployments? The schema should be stored and version-controlled alongside application code, not constructed at runtime. Changes that add required fields or modify field types can break downstream consumers if deployed without coordination; the safe migration pattern is to add new fields as nullable first, deploy, confirm downstream consumers handle them, then make them required.