Sycamore is an open source, AI-powered document processing engine for ETL, retrieval-augmented generation (RAG), LLM applications, and analytics on unstructured data, aimed at Python data engineers and AI developers who must turn reports, presentations, transcripts, and manuals into clean, chunked, searchable records.
What it is
Sycamore is an Apache-2.0 licensed Python project published by aryn-ai and distributed on PyPI as sycamore-ai. It partitions and enriches a wide range of document types, including reports, presentations, transcripts, and manuals, and it handles complex documents such as PDFs and images with embedded tables, figures, graphs, and other infographics. The framework is built around a scalable abstraction for document processing called a DocSet, which encapsulates the reliable transformation, enrichment, and cleaning of unstructured documents, and it runs on a scalable Ray backend. It lives in the Python data engineering and information-retrieval ecosystem, alongside the vector databases and hybrid search engines it loads.
The concrete problem Sycamore solves is the bespoke partition-and-chunk stage that teams otherwise hand-build before loading data into a vector database or a hybrid search engine. Instead of writing custom parsing, OCR, table-extraction, and chunking scripts, a user describes transforms in Python and lets docsets handle the undifferentiated heavy lifting of reliable loading. For processing, Sycamore can call Aryn DocParse (formerly the Aryn Partitioning Service), a serverless, GPU-powered API that segments and labels documents, performs OCR, and extracts tables and images, returning partitioned output as JSON. That partitioner uses Aryn's open source deformable-detr-DocLayNet DETR vision model trained on more than 80,000 enterprise documents, which the project reports can lead to 6x more accurate data chunking and 2x improved recall on hybrid search or RAG compared with alternate systems.
Key capabilities
- Integrated with Aryn DocParse for segmentation and preservation of document semantic structure, with partitioned output returned as JSON.
- Choice of hosted Aryn DocParse or locally run Aryn Partitioner for the document segmentation step.
DocSet abstraction for scalable, reliable transformation of unstructured documents.- High-quality table extraction, OCR, visual summarization, and LLM-powered UDFs written as Python transforms.
- Vector embedding creation with the user's choice of AI model.
- Connectors that load OpenSearch, Elasticsearch, Pinecone, DuckDB, Qdrant, and Weaviate.
- Automatic data crawlers for Amazon S3 and HTTP, a Jupyter notebook for writing and iterating on jobs, and an OpenSearch hybrid search and RAG engine for testing.
Who uses it and how
- Data engineers building ETL pipelines over unstructured document corpora, using docset transforms in place of custom parsing code.
- RAG and LLM application teams that must chunk complex PDFs and images containing embedded tables, figures, and graphs before retrieval.
- Teams standing up hybrid or semantic search over OpenSearch, Elasticsearch, Pinecone, DuckDB, Qdrant, or Weaviate and needing higher-quality chunks as input.
- Analysts running analytics over unstructured documents, with S3 or HTTP crawlers feeding the pipeline.
- Developers prototyping in Jupyter notebooks and scaling the same job onto the Ray backend.
Getting started
Install from PyPI with pip install sycamore-ai; Sycamore currently runs on Linux and macOS. For the partitioning step, either sign up for the free Aryn DocParse hosted service or choose to run the Aryn Partitioner locally.
How it compares
No comparable tools are named in the facts, so Sycamore stands alone in this registry. It sits in the Python document-processing and retrieval ecosystem, and its closest neighbours are the vector databases and hybrid search engines it loads rather than direct competitors.
When to use it — and when not to
A self-hoster must operate the Ray backend plus a downstream store such as OpenSearch, Elasticsearch, Pinecone, DuckDB, Qdrant, or Weaviate, and must either hold an Aryn DocParse account or deploy the Aryn Partitioner locally. Sycamore runs on Linux and macOS only, so teams standardised on Windows, or teams that want a fully managed service with no infrastructure to run, should not pick it. The repository carries 57 open issues, and the README excerpt ends mid-sentence in its connectors section, so users should read the published documentation at sycamore.readthedocs.io before committing to a pipeline.
project readme (upstream, from github) — read inline
Sycamore is an open source, AI-powered document processing engine for ETL, RAG, LLM-based applications, and analytics on unstructured data. Sycamore can partition and enrich a wide range of document types including reports, presentations, transcripts, manuals, and more. It can analyze and chunk complex documents such as PDFs and images with embedded tables, figures, graphs, and other infographics. Check out an example notebook.
For processing documents, Sycamore leverages Aryn DocParse (formerly known as the Aryn Partitioning Service), a serverless, GPU-powered API for segmenting and labeling documents, doing OCR, extracting tables and images, and more. It leverages Aryn's state-of-the-art, open source deep learning DETR AI model trained on 80k+ enterprise documents, and it can lead to 6x more accurate data chunking and 2x improved recall on hybrid search or RAG when compared to alternate systems. You can sign-up for free here, or choose to run the Aryn Partitioner locally.
Aryn DocParse takes documents and returns the partitioned output in JSON, and you can use Sycamore for additional data extraction, enrichment, transforms, cleaning, and loading into downstream databases. You can choose the LLMs to use with these transforms.
Sycamore reliably loads your vector databases and hybrid search engines, including as OpenSearch, ElasticSearch, Pinecone, DuckDB, Qdrant and Weaviate, with higher quality data.
The Sycamore framework is built around a scalable and robust abstraction for document processing called a DocSet, and includes powerful high-level transformations in Python for data processing, enrichment, and cleaning. DocSets also encapsulate scalable data processing techniques removing the undifferentiated heavy lifting of reliably loading chunks. DocSets' functional programming approach allows you to rapidly customize and experiment with your chunking for better quality RAG results.
Features
- Integrated with Aryn DocParse, using a state-of-the art vision AI model for segmentation and preserving the semantic structure of documents
- DocSet abstraction to scalably and reliably transform and manipulate unstructured documents
- High-quality table extraction, OCR, visual summarization, LLM-powered UDFs, and other performant Python data transforms
- Quickly create vector embeddings using your choice of AI model
- Helpful features like automatic data crawlers (Amazon S3 and HTTP), Jupyter notebook for writing and iterating on jobs, and an OpenSearch hybrid search and RAG engine for testing
- Scalable Ray backend
Demo
Introduction to Aryn DocParse (formerly known as the Aryn Partitioning Service)
Get Started
Sycamore currently runs on Linux and Mac OS. To install , run:
pip install sycamore-ai
Sycamore provides connectors to vector databases via Python extras. To install a connector, include it as an extra with your pip install. For example,
pip install sycamore-ai[duckdb]
Supported connectors include duckdb, elasticsearch, opensearch, pinecone, qdrant, and weaviate.
To use Aryn DocParse, sign-up for free here and use the API key.
Resources
Contributing
Check out our Contributing Guide for more information about how to contribute to Sycamore and set up your environment for development.
Contributors
Star History
↑ Back to Top ↑