Extracting Structured Data From Financial PDFs

TL;DR

Most investors spread their money across several accounts, from different brokerages to retirement plans and taxable accounts. Statements, tax forms, and balance sheets pile up as PDFs, and the numbers inside them stay locked in layouts that software struggles to read. Answering a simple question like "what do I hold across every account" still means opening files by hand.

PortfoliMosaic tackles exactly that problem. You upload financial PDFs, the system extracts the structured data, and you ask questions in plain language. The build leans on LandingAI Agentic Document Extraction for document parsing and schema extraction, a FAISS vector index for retrieval, and a four-tool agent built with LangChain that routes each question to the right data source.

Under the hood, Agentic Document Extraction runs a two-step pipeline: /parse converts each PDF to clean markdown, then /extract pulls structured JSON against a fixed PORTFOLIO_SCHEMA. Holdings land in SQLite, document text is chunked and embedded with all-MiniLM-L6-v2 into a single FAISS index, and a LangChain create_agent (running gpt-4o-mini) decides between retrieval, two SQL lookups, and a price API per question.

This is a working reference implementation, not a product. It has no authentication, isolates users only by a browser session ID, and runs on a single shared vector index. The patterns below are reusable, and the limitations are stated plainly where they matter.

The Premise

A typical investor holds positions at several brokerages, a retirement account or two, and a stack of tax paperwork each spring. Stock ownership is mainstream: 62% of Americans reported owning stock in 2025 (Gallup, 2025). Each account produces its own documents in its own format, and almost none of it arrives as clean, queryable data.

The usual answer is template-based OCR. It works when every statement looks the same and breaks the moment a brokerage changes its layout or a new document type shows up. Rules pile on top of rules, and a single redesigned PDF can quietly corrupt an entire pipeline.

Two shifts changed what's possible. Document AI now reads varied layouts and returns structured fields without per-template rules. Large language models can answer questions over that data conversationally. Put them together and a pile of PDFs becomes something you can interrogate. The rest of this article walks through how we wired those pieces into a working build, what it proved, and where it falls short.

Why financial documents resist automation

Most of the information lives in formats software can't query directly. By 2025, an estimated 80% of the global datasphere is unstructured, sitting in documents, images, and free text rather than tidy database rows (IDC, via VentureBeat). Financial services carries one of the highest unstructured shares of any sector, and statements are a prime example.

Source: IDC, via VentureBeat, 2025

Keying that data in by hand is slow and error-prone, a problem documented in peer-reviewed research on data-entry methods (Computers in Human Behavior, 2011). On a portfolio statement, a misread cost basis or a transposed share count doesn't just look wrong. It changes the answer to every downstream question about gains, allocation, and tax exposure.

That combination is the core problem for portfolio documents: the data exists, but it isn't reliably machine-readable, and keying it in by hand is both slow and error-prone.

What we built: PortfoliMosaic at a glance

PortfoliMosaic is a two-tier app: a React front end and a FastAPI back end. The flow is deliberately simple. You upload a financial PDF, the back end processes it asynchronously and reports status as the front end polls, and once it's ready you ask questions in a chat box. Each browser session gets its own session_id, and every document and query is scoped to it.

The architecture has four moving parts: ADE for extraction, SQLite for structured holdings and account data, a FAISS index for document text, and a LangChain agent that ties them together at query time.

Each box maps to a concrete job. ADE owns document intelligence, turning messy PDFs into both markdown and structured fields. SQLite holds the clean rows. FAISS holds the searchable text. The agent is the only part that touches all of them, and it does so per request.

How LandingAI turns PDFs into structured data

Agentic Document Extraction runs as two distinct API calls, not one black box. Splitting the work means the same parsed markdown feeds both the structured extraction and the retrieval index:

1. /parse sends the PDF (model dpt-2-latest) and returns clean markdown that preserves tables and structure.
2. /extract takes that markdown plus a JSON schema and returns structured fields.
3. The pipeline then stores holdings in SQLite and chunks the markdown into the FAISS index for retrieval.

The schema is the contract. PORTFOLIO_SCHEMA defines what we want pulled out: a DocumentType, report dates, company and ticker fields, account details, and a Holdings array with symbol, quantity, price, cost basis, and gain or loss per position. ADE fills that shape from whatever layout the document happens to use, which is the whole point. We don't write a template per brokerage.

Document classification rides along with extraction. The pipeline trusts ADE's own DocumentType first, and only falls back to a keyword classifier when ADE returns nothing usable. That ordering matters: the model sees the full document context, while keyword matching sees only surface text.

The market is moving the same direction, though estimates vary by research firm. Fortune Business Insights valued the intelligent document processing market at $10.57 billion in 2025, projecting $14.16 billion in 2026 at a 26.2% CAGR, with finance and accounting as the largest function at about 45.57% of the 2026 market (Fortune Business Insights, 2025). IDC, tracking the same software category, reports the IDP market continues to grow rapidly on the back of GenAI and agentic AI adoption (IDC, 2025).

Source: Fortune Business Insights, 2025 · 26.2% CAGR

Finance and accounting is projected as the single largest slice of that market, which is the tell for a portfolio tool: extraction is the layer everyone is building on, and it's where document intelligence earns its keep.

Why RAG is only one of four tools

Retrieval-augmented generation alone is the wrong default for portfolio questions. RAG is excellent for "what does my statement say about margin fees," because that answer lives in document text. It's a poor fit for "list every holding across all my accounts," because that's a structured aggregation, not a passage to retrieve. So in PortfoliMosaic, RAG is one of four tools, and the agent picks per question.

The four tools each own a job. search_documents runs semantic search over the FAISS index, embedding the query with all-MiniLM-L6-v2 (a 384-dimension model) and returning the top matching chunks. query_holdings_and_accounts and check_specific_holding hit SQLite directly for structured answers like account counts or a single ticker. get_stock_price calls a price API for live quotes, and falls back to mock values when no API key is set.

A LangChain create_agent running gpt-4o-mini is the router. It reads the question, picks a tool, and composes the answer from what comes back. The agent is rebuilt on every request with the current database session and session_id baked in, so each tool only ever sees that session's data.

Grounding the model in retrieved data measurably cuts fabrication. In an industry study presented at NAACL 2024, hallucination in structured outputs ran as high as 21% without retrieval, dropping to under 7.5% for steps and under 4.5% for tables once a retriever was added (arXiv, 2024).

Source: arXiv 2404.08189, NAACL 2024

For a financial assistant, that gap is the difference between a number you can act on and one you have to double-check by hand.

What the build proves, and what it doesn't

The build demonstrates a few things clearly. ADE handles layout variety without per-document templates, returning both markdown and structured holdings from the same parse. A multi-tool agent answers structured questions ("how many accounts") and document questions ("what fees were charged") in one conversation. Session scoping keeps one user's uploads out of another's results.

The limits are just as real, and worth naming honestly. The FAISS index is a single shared IndexFlatIP, and session isolation happens in Python: the code over-fetches results, then filters by session_id after the fact. That holds up at single-user scale but will not scale to many users or large corpora. There's no authentication and no multi-tenancy. The agent runs on gpt-4o-mini. If an ADE call fails, the document is marked failed with no local-parser fallback, and stock prices are mock data unless an Alpha Vantage key is configured.

None of these are dead ends. A production version would partition vectors per tenant, add auth, and harden the failure paths. The goal was to prove the shape of the system, and the build does.

What this means for fintech teams

The reusable lesson is architectural: treat document intelligence as a dedicated ingestion and extraction layer, then build retrieval and agents on top of clean output. ADE's /parse and /extract split maps neatly onto that separation, and the schema-driven extraction is what frees you from template maintenance.

For teams evaluating this pattern, the components are swappable but the layering holds. Keep extraction, structured storage, and retrieval as distinct stages, and let an agent route across them at query time. Treat the specific stack as one working example rather than a production blueprint. The boundary between "this runs locally" and "this is ready for customer data" is exactly the limitations listed above.

Conclusion

Portfolio data is fragmented by default, trapped in unstructured PDFs across multiple accounts. PortfoliMosaic shows one way to make it answerable.

• ADE's two-step parse and extract pipeline turns varied financial PDFs into both markdown and schema-bound structured data, without per-template rules.
• A four-tool agent routes each question to retrieval, SQL, or a price API, so structured and document questions both get good answers.
• Grounding the model in retrieved data measurably reduces fabrication, which matters most when the output is a financial figure.
• Scope to keep in mind: session-only isolation, a single shared vector index, no auth. The patterns scale; this build does not yet.

PortfoliMosaic was built by Siva Sundharam, who writes about production-grade agent design in his 12-Factor Agent Architecture series on Medium. The full source is on GitHub, and a video walkthrough is available. Watch the walkthrough video to see the upload-to-answer flow end to end, then read the code to see exactly where each component fits.

Frequently Asked Questions

What is PortfoliMosaic?

PortfoliMosaic is an AI financial document assistant. You upload financial PDFs like brokerage statements, 1099 forms, and balance sheets, and ask questions in plain language. It extracts structured data with LandingAI ADE and answers using a retrieval-and-SQL agent. It is a reference implementation, not a production product.

How does LandingAI ADE extract data from financial PDFs?

ADE runs two steps. The /parse endpoint converts a PDF to clean markdown using the dpt-2-latest model. The /extract endpoint then takes that markdown plus a JSON schema and returns structured fields. PortfoliMosaic uses a fixed PORTFOLIO_SCHEMA covering document type, holdings, and account details, so no per-brokerage templates are needed.

Why use an agent instead of plain RAG?

Retrieval is great for document-text questions but weak for structured aggregations like "list all holdings." PortfoliMosaic gives a LangChain agent four tools: semantic search over FAISS, two SQL lookups, and a price API. The agent picks the right tool per question. Grounding answers in retrieved data also cuts hallucination, which fell from as high as 21% to under 7.5% for steps (and under 4.5% for tables) in one structured-output study (arXiv, 2024).

Is PortfoliMosaic ready for production use?

No. It isolates users only by a browser session ID, runs on a single shared FAISS index with filtering done in Python, and has no authentication or multi-tenancy. Stock prices are mock data without an API key. A production build would need per-tenant vector partitioning, auth, and hardened failure handling before touching real customer data.