Running Benchmarks
Prerequisites
Section titled “Prerequisites”- Python 3.10+
- uv (recommended) or pip
- EdgeParse CLI installed
Quick Start
Section titled “Quick Start”cd benchmarkuv sync # install dependenciesuv run python run.py --engine edgeparseRunning a Single Tool
Section titled “Running a Single Tool”uv run python run.py --engine edgeparseAvailable tools include edgeparse, docling, marker, opendataloader, pymupdf4llm, markitdown, and hybrid adapters such as opendataloader_hybrid_docling_fast.
Running Grouped Comparisons
Section titled “Running Grouped Comparisons”uv run python compare_all.py --group non-ocruv run python compare_all.py --group hybriduv run python compare_all.py --group ocr --installThis keeps fast local parsers, hybrid backends, and OCR-heavy engines in separate reports. Use uv run python compare_all.py --all only when you want a single combined report.
Custom PDFs
Section titled “Custom PDFs”Place your PDF files in benchmark/pdfs/ and matching ground-truth Markdown in benchmark/ground-truth/markdown/.
# Run with custom corpusuv run python run.py --engine edgeparse --input-dir ./my-pdfsViewing Reports
Section titled “Viewing Reports”Reports are generated as HTML files:
open reports/benchmark-latest.htmlThresholds
Section titled “Thresholds”The thresholds.json file defines minimum acceptable scores:
{ "nid": 0.85, "teds": 0.70, "mhs": 0.75, "overall": 0.80}CI will fail if EdgeParse scores drop below these thresholds.