Skip to content

fix: partition multi-section DOCX files in linear time - #4471

Merged
cragwolfe merged 5 commits into
Unstructured-IO:mainfrom
linhongyu510:linhongyu510/linear-docx-sections
Sep 10, 2026
Merged

cragwolfe merged 5 commits into
Unstructured-IO:mainfrom
linhongyu510:linhongyu510/linear-docx-sections

Conversation

@linhongyu510

@linhongyu510 linhongyu510 commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

This pull request fixes #3592 by replacing per-section Section.iter_inner_content() calls with a single document-body traversal. Section boundaries are detected at paragraph-level sectPr elements so page breaks, headers, body content, and footers retain their existing order.

On the current main branch with python-docx 1.2.0, traversing synthetic documents with 20 paragraphs per section took:

  • 100 sections: 0.43s
  • 200 sections: 3.61s
  • 400 sections: 40.37s

With this change, the isolated traversal path took 0.008s, 0.010s, and 0.024s respectively. The full partition pipeline still includes text classification and metadata work; these numbers intentionally isolate the repeated section-prefix scan fixed here.

Tests

  • uv run --no-sync pytest -q test_unstructured/partition/test_docx.py (79 passed)
  • uv run --no-sync ruff check .
  • uv run --no-sync ruff format --check .
  • git diff --check
  • Regression test verified against the previous implementation: it fails because Section.iter_inner_content() is called.

make test-extra-docx additionally ran 96 tests successfully, while 6 legacy .doc conversion tests failed because the local LibreOffice process did not produce the requested .docx files. The complete DOCX test module passes independently.

Review in cubic

@cragwolfe

Copy link
Copy Markdown
Contributor

One correctness issue:

  • High — a document whose last section ends in a paragraph-level sectPr now fails instead of completing. In unstructured/partition/docx.py:425, the boundary transition unconditionally calls next(sections). A body-level w:sectPr is optional, and python-docx exposes the paragraph-level section properties that exist without synthesizing a terminal body section. For that accepted shape, the old for section in document.sections loop completed normally; the new generator advances past the final section and surfaces RuntimeError: generator raised StopIteration, causing partition_docx() to fail the whole document. Please use next(sections, None) and return when there is no next section, immediately after emitting the completed section's footer. Returning there is important so the footer is not emitted again after the body loop. A focused save/reload regression should remove the body-level sectPr while retaining a paragraph-level one and assert the full header/body/footer sequence exactly once. The new 25-section test does not cover this because Document.add_section() retains the terminal body sentinel.

(authored by codex)

@linhongyu510
linhongyu510 force-pushed the linhongyu510/linear-docx-sections branch from feb5016 to 819e758 Compare September 9, 2026 16:09
@linhongyu510

Copy link
Copy Markdown
Contributor Author

Addressed the paragraph-level terminal sectPr case in 819e7588 and rebased the branch onto current main (resolving the changelog/version conflict). The new save/reload regression moves the final body sectPr onto the last paragraph and asserts the exact Header, Body, Footer sequence once. It fails on the previous head with RuntimeError: generator raised StopIteration; the fix uses next(sections, None) and returns immediately after emitting the final footer when no next section exists. Verification on the rebased head: test_unstructured/partition/test_docx.py 85 passed; Ruff check and format check passed; git diff --check passed.

@cragwolfe

Copy link
Copy Markdown
Contributor

One production-readiness issue remains:

  • Medium — release metadata is still attached to an already committed version, so required lint CI cannot pass. CHANGELOG.md:5 adds this fix under 0.27.6, while unstructured/__version__.py:1 remains 0.27.6. make check fails on this exact head with Error: there is already a commit associated with version 0.27.6. Move the changelog entry under the next version heading and update unstructured/__version__.py to the same version, following the repository’s current versioning convention.

(authored by codex)

@linhongyu510

Copy link
Copy Markdown
Contributor Author

Moved this fix to the next release version in f31f962: CHANGELOG.md and unstructured/__version__.py now both use 0.27.7. Verification on the new head: test_unstructured/partition/test_docx.py 85 passed; Ruff check and format check passed; version metadata agrees and is newer than origin/main (0.27.6); git diff --check passed. The full local make check now gets past the previous committed-version error and stops only at the script’s macOS GNU sed requirement (sed version must be >= 4.3), which is environment-specific.

@cragwolfe
cragwolfe merged commit 9e87666 into Unstructured-IO:main Sep 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(docx): O(N^2) time on large section+paragraph count

2 participants