knowhere: Open Source Machine Learning Infrastructure

What is knowhere?

Knowhere is an Apache-2.0 Python document parsing and retrieval system that turns complex, dirty files into persistent, navigable memory for AI agents and RAG pipelines, built for developers and teams assembling Agentic RAG, vector-based RAG, or other LLM workflows over local and offline document collections.

What it is

Knowhere ingests unstructured documents and produces persistent, navigable memory through a single pipeline: parsing, hierarchy reconstruction, multi-modal structuring, and graph construction. Every result stays connected to its document, section, source pages, and related assets, so the output can be consumed directly by retrieval engines or agents. Version 2.0 runs two complementary tracks. The Text Track preserves precise extracted text and native structure where those are reliable; the Vision Track lets frontier vision models understand a page or slide as a whole. Both tracks converge into the same memory schema, hierarchy, retrieval engine, and citation model.

The concrete problem it solves is brittle element-by-element extraction. Traditional OCR and document intelligence pipelines try to recover every element before a model can understand the document, and on dirty PDFs and slide decks, mistakes in reading order, layout, tables, or hidden text layers accumulate into unreliable model context. Knowhere removes perfect extraction as a prerequisite for retrieval: pages can be indexed through summaries, entities, source text, and hierarchy even when OCR or layout extraction cannot reliably recover every component. It lives in the Python AI and machine learning infrastructure ecosystem, alongside agent harnesses, LangChain, and vector databases, and it replaces the ad hoc OCR-plus-chunking stage that teams otherwise hand-build before feeding documents to a model.

Key capabilities

Dual-track Document Parsing 2.0: a Text Track for text-native documents and a Vision Track for complex PDFs and PowerPoint files, both emitting the same chunk and metadata schema.
Format-based routing, where PDF and .pptx uploads through the V2 Jobs API use the Vision Track while other supported formats use the Text Track.
Ultra-long document handling for PDFs of hundreds of pages, including 300, 500, or more, plus atlas-style documents routed through a dedicated layout-aware parser.
Agent-native Retrieval 2.0, the evolution of MapNav into a corpus-native foundation with a unified schema, hierarchy-aware tools, and resolvable evidence references, letting the agent decide how to search, traverse, read, and cite.
An MCP interface so the same retrieval foundation powers built-in retrieval and external agents.
Evidence-linked output, where each result remains tied to its document, section, source pages, linked assets, and cross-document relationships.
Two-step operation: build memory from documents, then let agents retrieve from it.

Who uses it and how

Teams building Agentic RAG or vector-based RAG over local and offline document collections, where documents cannot be shipped to an external parsing service.
Agent harness users, as indicated by the topic list naming Claude Code, Codex, and Cursor, connecting external agents to Knowhere retrieval through MCP.
Organizations with dirty PDFs and slide decks, where accumulated OCR, reading-order, and hidden-text-layer errors make element-perfect extraction an unreliable foundation.
LangChain and vector database pipelines, which consume Knowhere's schema rather than raw extracted text.
Multi-agent setups, where a shared hierarchy, citation model, and evidence references give several agents a common navigable memory.

Getting started

Knowhere Cloud, the managed API at knowhereto.ai, is the fastest path and includes $5 in free credits on registration. To self-host the full platform, deploy from the knowhere-self-hosted repository; contribution guidance is in CONTRIBUTING.md.

How it compares

No list of paid products is provided in the facts, so comparison rests on the similar tools the facts do name. Knowhere is not a replacement for LangChain or a vector database; it produces the chunk schema, hierarchy, and citation model that those tools consume, which is why its output is described as a natural fit for Agentic RAG and vector-based RAG rather than as a competing framework or store.

When to use it — and when not to

Knowhere suits teams that either accept a managed API or are prepared to operate the full self-hosted stack from knowhere-self-hosted. It is a poor choice for anyone who needs a zero-infrastructure, purely local setup with no separate deployment step, or who requires guaranteed element-perfect extraction from unreliable scans rather than recall through summaries, entities, and hierarchy. The facts also carry honest caveats: the README excerpt is truncated and does not enumerate the backing services self-hosting requires, and the repository shows 40 open issues, so the operational footprint should be assessed from the self-hosted repository before committing.

project readme (upstream, from github) — read inline

Prepare unstructured data for AI Agents

🔗 Website | 📄 Docs | 🏠 Self-Host | 🖥️ Dashboard

Overview

Knowhere is a document parsing and retrieval system that turns complex, dirty files into persistent, navigable memory for AI agents—especially across local and offline document collections.

It ingests unstructured documents and produces persistent, navigable memory: parsing, hierarchy reconstruction, multi-modal structuring, and graph construction in a single pipeline. Every result stays connected to its document, section, source pages, and related assets, making the output a natural fit for Agentic RAG, vector-based RAG, or any LLM workflow.

Knowhere 2.0 supports complementary Vision and Text tracks. Text-native documents retain precise extracted structure, while complex PDFs and PowerPoint files can be understood directly as pages by frontier vision models. Both tracks converge into the same memory schema, hierarchy, retrieval engine, and citation model.

[!NOTE] Get started in seconds with Knowhere Cloud. Avoid the complexity of self-deployment. Use our managed API at knowhereto.ai and enjoy $5 in free credits upon registration.

📢 News

September 8, 2026: 🧭 Introducing agent-native Retrieval 2.0. MapNav has evolved from a fixed navigation workflow into a corpus-native foundation for agents. Knowhere provides a unified schema, hierarchy-aware tools, and resolvable evidence references; the agent decides how to search, traverse, read, and cite. The same foundation powers built-in retrieval and external agents through MCP.
September 2026: 👁️ Introducing dual-track Document Parsing 2.0. Vision Page and Text Track now converge into one hierarchy-native memory schema for retrieval, understanding, and citation.
June 1, 2026: 📚 Knowhere now supports ultra-long PDFs and atlas-style documents. The parsing pipeline can process long-form PDFs with hundreds of pages (for example, 300, 500, or more) and route technical atlases or drawing collections through a dedicated layout-aware parser.
May 7, 2026: 🚀 Knowhere is now Open Source! We have open-sourced our entire stack for document ingestion, parsing, and agentic RAG. You can now self-host the full platform using knowhere-self-hosted. Check out our Contribution Guide to get involved!

Vision + Text: Document Parsing 2.0

Traditional OCR and Document Intelligence pipelines try to extract every element before a model can understand the document. On dirty PDFs and slide decks, mistakes in reading order, layout, tables, or hidden text layers can accumulate into unreliable model context.

Knowhere does not make perfect element-by-element extraction a prerequisite for retrieval. The Text Track preserves precise text and native structure where they are reliable. The Vision Track uses frontier vision models to understand a page or slide as a whole, so visually complex content can still be recalled and understood without first reconstructing every element.

Two tracks, one contract: Both parsing paths produce the same chunk and metadata schema, so downstream storage, hierarchy, graph construction, and retrieval remain format-independent.
Recall without brittle reconstruction: Pages can be indexed through summaries, entities, source text, and hierarchy even when OCR or layout extraction cannot reliably recover every component.
One navigable memory: Text sections and vision-understood pages become compatible hierarchy nodes with source evidence, linked assets, and cross-document relationships.

PDF and .pptx uploads through the V2 Jobs API use the Vision Track; other supported formats use the Text Track. The tracks differ in how they understand the source, not in how agents consume the resulting memory.

How it Works

Knowhere runs in two steps: build memory from documents, then let agents retrieve from it.

Step 1: Parse and Build Memory

Route: Select the Vision or Text track according to the document format and API generation.
Understand: Preserve native text structure where it is reliable, or understand complex pages holistically with a vision model.
Normalize: Convert both tracks into the same hierarchy-native chunk and metadata schema.
Build Memory: Store navigation trees, linked assets, citations, and cross-document relationships as agent-ready context.

Step 2: Agentic Retrieval

Knowhere provides the document-memory substrate; the agent decides how to explore it.

Provide the substrate: Knowhere exposes one corpus schema and tools for document outlines, structural filters, exact search, fuzzy recall, full reading, assets, and cross-document relationships.
Let the agent explore: Instead of forcing every query through a fixed navigation pipeline, the agent chooses which tools to call, in what order, and how deeply to traverse.
Stay agent-neutral: The same corpus and evidence contract works across built-in agents, MCP clients, models, and orchestration frameworks. Whichever agent explores the memory, Knowhere resolves its references into traceable documents, sections, pages, and linked assets.

FAQ

Q: What is Knowhere's relationship with MinerU?

A: MinerU remains the default raw PDF extractor for Knowhere's V1 chunk-based pipeline. PDF and PowerPoint uploads through the V2 API use Vision Page instead: Knowhere renders the source pages, combines their visual interpretation with document profiling and TOC structure, and assembles page-grounded hierarchy nodes. MinerU is still useful, but V2 no longer treats parser-generated Markdown as the only source of truth.

Q: What LLM / VLM dependencies does Knowhere have?

A: We recommend deepseek-v4-flash-vision-exp as a unified model for both Text and Vision workloads. It accepts text and image input, so the same model can handle summarization, hierarchy reasoning, page understanding, and asset descriptions. The model is currently experimental, and Knowhere remains model-agnostic: you can use another model—or separate Text and Vision models—from OpenAI, Qwen, GLM, Volcengine, or any compatible provider.

Q: How is Agentic Retrieval different from traditional RAG?

A: Traditional RAG does a flat vector lookup and returns isolated snippets. Knowhere's agents navigate the document's section tree and cross-document graph, drilling into the most relevant regions the way a human reader would, returning traceable, well-contextualized evidence.

Q: Does it handle images and tables?

A: Yes. Knowhere extracts images and tables, runs them through VLM-assisted summarization and feature extraction, and links them back to their source section nodes. Vision Page also retains rendered page citations, so agents can re

Frequently asked questions

Is knowhere free to use?

knowhere is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does knowhere do?

Knowhere extracts, parses, and outputs structured chunks ready for AI Agents and RAG.

What is knowhere written in?

knowhere is primarily written in Python. Its source is publicly available at https://github.com/Ontos-AI/knowhere, and it has 3,341 GitHub stars.