Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: Unstructured-IO/unstructured
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 0.27.1
Choose a base ref
...
head repository: Unstructured-IO/unstructured
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 0.27.5
Choose a head ref
  • 5 commits
  • 22 files changed
  • 6 contributors

Commits on Aug 24, 2026

  1. chore(deps): bump claude-code-action to v1 (#4448)

    Bumps `anthropics/claude-code-action` from `@beta` to `@v1`.
    
    <!-- This is an auto-generated description by cubic. -->
    <a
    href="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4448?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4448?utm_source=github"
    target="_blank" rel="noopener noreferrer"
    data-no-image-dialog="true"><picture><source
    media="(prefers-color-scheme: dark)"
    srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
    media="(prefers-color-scheme: light)"
    srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
    alt="Review in cubic"
    src="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
    <!-- End of auto-generated description by cubic. -->
    william-u10d authored Aug 24, 2026
    Configuration menu
    Copy the full SHA
    740f5b4 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2026

  1. Configuration menu
    Copy the full SHA
    593e54a View commit details
    Browse the repository at this point in the history
  2. fix: stop treating every hyphen as a bullet delimiter in text partiti…

    …oning (#4458)
    
    ## Summary
    
    `-` (U+002D) and `–` (U+2013) are bullet glyphs *and* ordinary
    punctuation — an intra-word hyphen, an intra-number separator, a minus
    sign. `UNICODE_BULLETS` was reused as an **unanchored split delimiter**,
    so `partition_text` split at every hyphen in a qualifying line *and
    consumed the hyphen itself*:
    
    ```python
    partition_text(text="Phone: 555-123-4567\n\nEnd")
    # ['Phone: 555', '123', '4567', 'End']
    
    partition_text(text="Contact: Jean-Luc Picard\n\nEnd")
    # ['Contact: Jean', 'Luc Picard', 'End']
    ```
    
    The only thing limiting the blast radius was the `all_lines_short`
    guard, which counts **space-separated tokens** — so long English prose
    escaped by accident of word count, while short `Field: value` lines, log
    lines, YAML and all CJK text (no inter-word spaces, so the count is
    always 1) did not.
    
    This happened at partition time, upstream of every transform, on a job
    that reported success. Nothing downstream recovered it: both chunkers
    rejoin elements with a blank line, so the fragments never became
    contiguous again — a regex for a phone number or an SSN can never match
    across an element boundary.
    
    Scope: `FileType.TXT` and its 18 extensions (`.txt .text .c .cc .cpp .cs
    .cxx .go .java .js .log .php .py .rb .swift .ts .yaml .yml`) plus
    `.eml`/`.msg`, which delegate to `partition_text`. PDF/DOCX/PPTX/HTML
    are unaffected — verified by call graph and by round-tripping the same
    content through DOCX and HTML.
    
    ## The fix
    
    Two conditions now have to hold before an ambiguous glyph is read as a
    bullet, and **three** call sites needed changing, not one.
    
    **1. Start of line.** `PARAGRAPH_PATTERN_RE` splits only on the
    unambiguous glyphs. Dash bullets at line start were already handled by
    the `\n` branch, so nothing is lost.
    
    **2. Followed by whitespace** (or end of text). A bullet is separated
    from the item it introduces; a sign is not. Without this, `-123.45`
    reads as a bullet and is rewritten to `123.45` — **silently flipping the
    value's sign**. `E_BULLET_PATTERN` in the same file already required
    `(?=\s)` for exactly this reason, so this brings dash handling in line
    with existing precedent rather than inventing a convention.
    
    Applied to:
    
    - `PARAGRAPH_PATTERN_RE` — the paragraph split.
    - `BULLET_SPLIT_RE_0W` (new) — used by `group_bullet_paragraph`, which
    splits on `UNICODE_BULLETS_RE_0W`. This site is reached for any
    paragraph that *starts* with a bullet, i.e. exactly a dash bullet list,
    so without it a genuine dash list whose items contain hyphens was still
    shredded.
    - `UNICODE_BULLETS_RE` — backs `clean_bullets`, `is_bulleted_text` and
    `_is_empty_bullet`. Fixing only the split sites preserves the line
    boundary but lets classification strip the sign anyway.
    
    `-` and `–` stay in `UNICODE_BULLETS`; the qualification is scoped to
    the ambiguous glyphs via a separate alternation, so `•item` and friends
    need no separator and are unchanged in every respect.
    
    ### Before / after
    
    ```
    'Phone: 555-123-4567'      ['Phone: 555','123','4567']            ->  ['Phone: 555-123-4567']
    'Contact: Jean-Luc Picard' ['Contact: Jean','Luc Picard']         ->  ['Contact: Jean-Luc Picard']
    'trace_id: 550e8400-e29b…' 5 fragments                            ->  1 element
    'Content-Type: image/jpeg' ['Content','Type: image/jpeg']         ->  ['Content-Type: image/jpeg']
    '-123.45'                  ListItem('123.45')   # sign flipped    ->  Text('-123.45')
    '- readings\n-5\n-10 °C'   3 sign-stripped items                  ->  ListItem('readings -5 °C -10 °C')
    ```
    
    ## Behavior changes
    
    Both are declared in the changelog.
    
    1. An inline dash bullet list on a single line (`- one - two - three`)
    no longer splits into separate elements. Ambiguous input either way, and
    a far better trade than destroying every hyphenated identifier.
    2. A dash with no separating whitespace (`-item`) is no longer a bullet:
    it keeps its leading character and is no longer classified as a
    `ListItem`.
    
    Unchanged: `- item` at line start is still a bullet, a lone `-` is still
    an empty bullet, `----` is still not a bullet, and inline `•`-style
    bullets still split.
    
    ## Verification
    
    **1828 tests pass** — cleaners, nlp, text, email, msg, text_type,
    partition/common, chunking, auto, html, docx, md, documents. Nothing
    broke.
    
    **Corpus audit** over the 66 `.txt`/`.eml` files in `example-docs/`:
    
    | | elements | dash chars recovered | docs with text loss |
    | -- | -- | -- | -- |
    | before | 26,857 | — | — |
    | after | 26,524 | **+430** | **0** |
    
    The recovered characters are the point: the old split *deleted* the
    delimiter. Among them, wrapped URL slugs in `norwich-city.txt` were
    losing their leading dash (`-can-get-hucks-firing-1-648688`).
    
    **Adversarial cases** checked against the bullet regex: `-Xmx512m`,
    `--flag`, `-o`, `-.5`, `-1e9`, `-5`, `-item`, `----` (none are bullets);
    `- item`, `– item`, `-\titem`, `-`, `-\nnext`, `•item`, `○x`, `* item`
    (all are).
    
    **New regression tests** cover hyphenated values through
    `partition_text` and a new `.eml` fixture (Japanese phone, US SSN, ISO
    date, card, IBAN, UUID, hyphenated personal name, YAML value, log line
    with hostname); dash/en-dash/indented bullet lists; dash list items
    containing hyphens; inline `•` bullets; the Apache-License short-line
    case cited in `group_broken_paragraphs`; signs (`-123.45`, `–10 °C`,
    `-5`); and both declared behavior changes, pinned with a comment saying
    they are intentional.
    
    ## Notes for review
    
    Two related defects found during this work, deliberately **not** in
    scope:
    
    1. `layout_list_to_list_items` (`partition/common/common.py`) has the
    same unanchored split on `UNICODE_BULLETS_RE`. Currently unreachable —
    the live `pdf.py` hi_res path passes `infer_list_items=False` and the
    `ocr_only` path never emits `ElementType.LIST` — but latent.
    2. `all_lines_short` in `group_broken_paragraphs` counts space-separated
    tokens and is structurally blind to CJK, where the count is always 1.
    This PR makes it moot for dashes; it stays wrong for any other ambiguous
    delimiter.
    
    `paragraph_grouper=False` was the only workaround, and it is not
    reachable from the platform — the parameter is absent from the partition
    node's settings schema and from `unstructured_client`'s
    `PartitionParameters`.
    
    🤖 Generated with [Claude Code](https://claude.com/claude-code)
    
    
    <!-- This is an auto-generated description by cubic. -->
    <a
    href="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4458?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4458?utm_source=github"
    target="_blank" rel="noopener noreferrer"
    data-no-image-dialog="true"><picture><source
    media="(prefers-color-scheme: dark)"
    srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
    media="(prefers-color-scheme: light)"
    srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
    alt="Review in cubic"
    src="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
    <!-- End of auto-generated description by cubic. -->
    
    ---------
    
    Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
    Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com>
    Co-authored-by: badGarnet <badGarnet@users.noreply.github.com>
    4 people authored Aug 27, 2026
    Configuration menu
    Copy the full SHA
    e2732cf View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2026

  1. fix: preserve attachment elements' own filetype in auto.partition() (#…

    …4460)
    
    ## Summary
    
    When a document with attachments is partitioned, every element in the
    output was labelled with the **containing** file's MIME type. Content
    extracted from an email's attached PDF reported `filetype:
    message/rfc822`, so downstream consumers could not filter or route by
    document type.
    
    Not format-specific to Outlook — it reproduces for any email with
    attachments, on both the `.eml` and `.msg` paths.
    
    ## Root cause
    
    The `augment_metadata` closure in `partition()`
    (`unstructured/partition/auto.py`) re-stamped `metadata.filetype` on
    every element it was handed, unconditionally.
    
    Attachments are partitioned by a nested `partition()` call, so each
    attachment's elements are stamped with their correct filetype at their
    own recursion depth, then flattened into the containing document's
    element list. The outermost call then ran its own `augment_metadata`
    over the whole flattened list and overwrote all of them.
    
    ## Fix
    
    Skip the filetype re-stamp for elements carrying `attached_to_filename`.
    This matches guards the codebase already applies for exactly this case
    in `partition/common/metadata.py` and in two places in
    `file_utils/filetype.py` — so the convention is established, it just was
    not used by this hand-rolled closure.
    
    ### One deliberate narrowing
    
    `url` and `data_source` are still applied to attachment elements; only
    the filetype block is guarded. Skipping the element outright would have
    been the broader change, but the nested `partition()` call receives
    neither of those values, so attachment elements would come back with
    `data_source = None` — silently dropping source lineage (record
    locators, source URLs) for attachment-derived content. The defect here
    is filetype-only, so the guard is too.
    
    ## Verification
    
    An `.eml` built with a PDF and a DOCX attachment:
    
    | elements | before | after |
    |---|---|---|
    | email body (1) | `message/rfc822` | `message/rfc822` |
    | PDF attachment (25) | `message/rfc822` | `application/pdf` |
    | DOCX attachment (8) | `message/rfc822` | `…wordprocessingml.document`
    |
    
    ## Tests
    
    Adds
    `test_auto_partition_preserves_the_filetype_of_attachment_elements`,
    parametrized over the `.eml` and `.msg` paths using existing fixtures
    (no new binaries). Both cases were confirmed to fail without the fix and
    pass with it.
    
    `test_auto.py`, `test_email.py`, `test_msg.py` and
    `common/test_metadata.py`: 327 passed, 1 xfailed. Lint and format clean.
    
    ## Note
    
    The CHANGELOG diff includes a one-character trailing-whitespace removal
    on the pre-existing 0.27.3 entry, applied by the repo's own
    `trailing-whitespace` pre-commit hook.
    
    🤖 Generated with [Claude Code](https://claude.com/claude-code)
    
    <!-- This is an auto-generated description by cubic. -->
    <a
    href="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4460?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4460?utm_source=github"
    target="_blank" rel="noopener noreferrer"
    data-no-image-dialog="true"><picture><source
    media="(prefers-color-scheme: dark)"
    srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
    media="(prefers-color-scheme: light)"
    srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
    alt="Review in cubic"
    src="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
    <!-- End of auto-generated description by cubic. -->
    
    ---------
    
    Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
    badGarnet and claude authored Aug 28, 2026
    Configuration menu
    Copy the full SHA
    4fe4097 View commit details
    Browse the repository at this point in the history
  2. chore: make a release (#4461)

    <!-- This is an auto-generated description by cubic. -->
    <a
    href="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4461?utm_source=github" rel="nofollow">https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4461?utm_source=github"
    target="_blank" rel="noopener noreferrer"
    data-no-image-dialog="true"><picture><source
    media="(prefers-color-scheme: dark)"
    srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
    media="(prefers-color-scheme: light)"
    srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
    alt="Review in cubic"
    src="proxy.php?url=https%3A%2F%2Fgithub.com%2FUnstructured-IO%2Funstructured%2Fcompare%2F%3Ca+href%3D"https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a" rel="nofollow">https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>
    <!-- End of auto-generated description by cubic. -->
    badGarnet authored Aug 28, 2026
    Configuration menu
    Copy the full SHA
    d68ab0f View commit details
    Browse the repository at this point in the history
Loading