Skip to content
EdgeParse EdgeParse EdgeParse

Running Benchmarks

Prerequisites

Section titled “Prerequisites”
  • Python 3.10+
  • uv (recommended) or pip
  • EdgeParse CLI installed

Quick Start

Section titled “Quick Start”
Terminal window
cd benchmark
uv sync          # install dependencies
uv run python run.py --tool edgeparse

Running a Single Tool

Section titled “Running a Single Tool”
Terminal window
uv run python run.py --tool edgeparse

Available tools: edgeparse, docling, marker, edgequake, opendataloader, pymupdf4llm, markitdown.

Running All Tools

Section titled “Running All Tools”
Terminal window
uv run python compare_all.py

This compares all tools against the ground-truth reference set and generates an HTML report in reports/.

Custom PDFs

Section titled “Custom PDFs”

Place your PDF files in benchmark/pdfs/ and matching ground-truth Markdown in benchmark/ground-truth/markdown/.

Terminal window
# Run with custom corpus
uv run python run.py --tool edgeparse --pdf-dir ./my-pdfs

Viewing Reports

Section titled “Viewing Reports”

Reports are generated as HTML files:

Terminal window
open reports/benchmark-latest.html

Thresholds

Section titled “Thresholds”

The thresholds.json file defines minimum acceptable scores:

{
  "nid": 0.85,
  "teds": 0.70,
  "mhs": 0.75,
  "overall": 0.80
}

CI will fail if EdgeParse scores drop below these thresholds.