-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Permalink
Choose a base ref
{{ refName }}
default
Choose a head ref
{{ refName }}
default
Comparing changes
Choose two branches to see what’s changed or to start a new pull request.
If you need to, you can also or
learn more about diff comparisons.
Open a pull request
Create a new pull request by comparing changes across two branches. If you need to, you can also .
Learn more about diff comparisons here.
base repository: Unstructured-IO/unstructured
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 0.27.1
Could not load branches
Nothing to show
Loading
Could not load tags
Nothing to show
{{ refName }}
default
Loading
...
head repository: Unstructured-IO/unstructured
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 0.27.5
Could not load branches
Nothing to show
Loading
Could not load tags
Nothing to show
{{ refName }}
default
Loading
- 5 commits
- 22 files changed
- 6 contributors
Commits on Aug 24, 2026
-
chore(deps): bump claude-code-action to v1 (#4448)
Bumps `anthropics/claude-code-action` from `@beta` to `@v1`. <!-- This is an auto-generated description by cubic. --> <a href="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4448?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4448?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
Configuration menu - View commit details
-
Copy full SHA for 740f5b4 - Browse repository at this point
Copy the full SHA 740f5b4View commit details
Commits on Aug 27, 2026
-
Configuration menu - View commit details
-
Copy full SHA for 593e54a - Browse repository at this point
Copy the full SHA 593e54aView commit details -
fix: stop treating every hyphen as a bullet delimiter in text partiti…
…oning (#4458) ## Summary `-` (U+002D) and `–` (U+2013) are bullet glyphs *and* ordinary punctuation — an intra-word hyphen, an intra-number separator, a minus sign. `UNICODE_BULLETS` was reused as an **unanchored split delimiter**, so `partition_text` split at every hyphen in a qualifying line *and consumed the hyphen itself*: ```python partition_text(text="Phone: 555-123-4567\n\nEnd") # ['Phone: 555', '123', '4567', 'End'] partition_text(text="Contact: Jean-Luc Picard\n\nEnd") # ['Contact: Jean', 'Luc Picard', 'End'] ``` The only thing limiting the blast radius was the `all_lines_short` guard, which counts **space-separated tokens** — so long English prose escaped by accident of word count, while short `Field: value` lines, log lines, YAML and all CJK text (no inter-word spaces, so the count is always 1) did not. This happened at partition time, upstream of every transform, on a job that reported success. Nothing downstream recovered it: both chunkers rejoin elements with a blank line, so the fragments never became contiguous again — a regex for a phone number or an SSN can never match across an element boundary. Scope: `FileType.TXT` and its 18 extensions (`.txt .text .c .cc .cpp .cs .cxx .go .java .js .log .php .py .rb .swift .ts .yaml .yml`) plus `.eml`/`.msg`, which delegate to `partition_text`. PDF/DOCX/PPTX/HTML are unaffected — verified by call graph and by round-tripping the same content through DOCX and HTML. ## The fix Two conditions now have to hold before an ambiguous glyph is read as a bullet, and **three** call sites needed changing, not one. **1. Start of line.** `PARAGRAPH_PATTERN_RE` splits only on the unambiguous glyphs. Dash bullets at line start were already handled by the `\n` branch, so nothing is lost. **2. Followed by whitespace** (or end of text). A bullet is separated from the item it introduces; a sign is not. Without this, `-123.45` reads as a bullet and is rewritten to `123.45` — **silently flipping the value's sign**. `E_BULLET_PATTERN` in the same file already required `(?=\s)` for exactly this reason, so this brings dash handling in line with existing precedent rather than inventing a convention. Applied to: - `PARAGRAPH_PATTERN_RE` — the paragraph split. - `BULLET_SPLIT_RE_0W` (new) — used by `group_bullet_paragraph`, which splits on `UNICODE_BULLETS_RE_0W`. This site is reached for any paragraph that *starts* with a bullet, i.e. exactly a dash bullet list, so without it a genuine dash list whose items contain hyphens was still shredded. - `UNICODE_BULLETS_RE` — backs `clean_bullets`, `is_bulleted_text` and `_is_empty_bullet`. Fixing only the split sites preserves the line boundary but lets classification strip the sign anyway. `-` and `–` stay in `UNICODE_BULLETS`; the qualification is scoped to the ambiguous glyphs via a separate alternation, so `•item` and friends need no separator and are unchanged in every respect. ### Before / after ``` 'Phone: 555-123-4567' ['Phone: 555','123','4567'] -> ['Phone: 555-123-4567'] 'Contact: Jean-Luc Picard' ['Contact: Jean','Luc Picard'] -> ['Contact: Jean-Luc Picard'] 'trace_id: 550e8400-e29b…' 5 fragments -> 1 element 'Content-Type: image/jpeg' ['Content','Type: image/jpeg'] -> ['Content-Type: image/jpeg'] '-123.45' ListItem('123.45') # sign flipped -> Text('-123.45') '- readings\n-5\n-10 °C' 3 sign-stripped items -> ListItem('readings -5 °C -10 °C') ``` ## Behavior changes Both are declared in the changelog. 1. An inline dash bullet list on a single line (`- one - two - three`) no longer splits into separate elements. Ambiguous input either way, and a far better trade than destroying every hyphenated identifier. 2. A dash with no separating whitespace (`-item`) is no longer a bullet: it keeps its leading character and is no longer classified as a `ListItem`. Unchanged: `- item` at line start is still a bullet, a lone `-` is still an empty bullet, `----` is still not a bullet, and inline `•`-style bullets still split. ## Verification **1828 tests pass** — cleaners, nlp, text, email, msg, text_type, partition/common, chunking, auto, html, docx, md, documents. Nothing broke. **Corpus audit** over the 66 `.txt`/`.eml` files in `example-docs/`: | | elements | dash chars recovered | docs with text loss | | -- | -- | -- | -- | | before | 26,857 | — | — | | after | 26,524 | **+430** | **0** | The recovered characters are the point: the old split *deleted* the delimiter. Among them, wrapped URL slugs in `norwich-city.txt` were losing their leading dash (`-can-get-hucks-firing-1-648688`). **Adversarial cases** checked against the bullet regex: `-Xmx512m`, `--flag`, `-o`, `-.5`, `-1e9`, `-5`, `-item`, `----` (none are bullets); `- item`, `– item`, `-\titem`, `-`, `-\nnext`, `•item`, `○x`, `* item` (all are). **New regression tests** cover hyphenated values through `partition_text` and a new `.eml` fixture (Japanese phone, US SSN, ISO date, card, IBAN, UUID, hyphenated personal name, YAML value, log line with hostname); dash/en-dash/indented bullet lists; dash list items containing hyphens; inline `•` bullets; the Apache-License short-line case cited in `group_broken_paragraphs`; signs (`-123.45`, `–10 °C`, `-5`); and both declared behavior changes, pinned with a comment saying they are intentional. ## Notes for review Two related defects found during this work, deliberately **not** in scope: 1. `layout_list_to_list_items` (`partition/common/common.py`) has the same unanchored split on `UNICODE_BULLETS_RE`. Currently unreachable — the live `pdf.py` hi_res path passes `infer_list_items=False` and the `ocr_only` path never emits `ElementType.LIST` — but latent. 2. `all_lines_short` in `group_broken_paragraphs` counts space-separated tokens and is structurally blind to CJK, where the count is always 1. This PR makes it moot for dashes; it stays wrong for any other ambiguous delimiter. `paragraph_grouper=False` was the only workaround, and it is not reachable from the platform — the parameter is absent from the partition node's settings schema and from `unstructured_client`'s `PartitionParameters`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated description by cubic. --> <a href="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4458?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4458?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com> Co-authored-by: badGarnet <badGarnet@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for e2732cf - Browse repository at this point
Copy the full SHA e2732cfView commit details
Commits on Aug 28, 2026
-
fix: preserve attachment elements' own filetype in auto.partition() (#…
…4460) ## Summary When a document with attachments is partitioned, every element in the output was labelled with the **containing** file's MIME type. Content extracted from an email's attached PDF reported `filetype: message/rfc822`, so downstream consumers could not filter or route by document type. Not format-specific to Outlook — it reproduces for any email with attachments, on both the `.eml` and `.msg` paths. ## Root cause The `augment_metadata` closure in `partition()` (`unstructured/partition/auto.py`) re-stamped `metadata.filetype` on every element it was handed, unconditionally. Attachments are partitioned by a nested `partition()` call, so each attachment's elements are stamped with their correct filetype at their own recursion depth, then flattened into the containing document's element list. The outermost call then ran its own `augment_metadata` over the whole flattened list and overwrote all of them. ## Fix Skip the filetype re-stamp for elements carrying `attached_to_filename`. This matches guards the codebase already applies for exactly this case in `partition/common/metadata.py` and in two places in `file_utils/filetype.py` — so the convention is established, it just was not used by this hand-rolled closure. ### One deliberate narrowing `url` and `data_source` are still applied to attachment elements; only the filetype block is guarded. Skipping the element outright would have been the broader change, but the nested `partition()` call receives neither of those values, so attachment elements would come back with `data_source = None` — silently dropping source lineage (record locators, source URLs) for attachment-derived content. The defect here is filetype-only, so the guard is too. ## Verification An `.eml` built with a PDF and a DOCX attachment: | elements | before | after | |---|---|---| | email body (1) | `message/rfc822` | `message/rfc822` | | PDF attachment (25) | `message/rfc822` | `application/pdf` | | DOCX attachment (8) | `message/rfc822` | `…wordprocessingml.document` | ## Tests Adds `test_auto_partition_preserves_the_filetype_of_attachment_elements`, parametrized over the `.eml` and `.msg` paths using existing fixtures (no new binaries). Both cases were confirmed to fail without the fix and pass with it. `test_auto.py`, `test_email.py`, `test_msg.py` and `common/test_metadata.py`: 327 passed, 1 xfailed. Lint and format clean. ## Note The CHANGELOG diff includes a one-character trailing-whitespace removal on the pre-existing 0.27.3 entry, applied by the repo's own `trailing-whitespace` pre-commit hook. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated description by cubic. --> <a href="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4460?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4460?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Configuration menu - View commit details
-
Copy full SHA for 4fe4097 - Browse repository at this point
Copy the full SHA 4fe4097View commit details -
<!-- This is an auto-generated description by cubic. --> <a href="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4461?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4461?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
Configuration menu - View commit details
-
Copy full SHA for d68ab0f - Browse repository at this point
Copy the full SHA d68ab0fView commit details
Loading
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff 0.27.1...0.27.5