fix: honor encoding= in partition_json and partition_ndjson by linhongyu510 · Pull Request #4483 · Unstructured-IO/unstructured

linhongyu510 · 2026-09-10T19:04:00Z

Problem

partition_json() and partition_ndjson() read the source with a hard-coded
UTF-8, and neither accepts an encoding parameter:

if filename is not None:
    with open(filename, encoding="utf8") as f:      # hard-coded
        file_text = f.read()

elif file is not None:
    file_content = file.read()
    file_text = file_content if isinstance(file_content, str) else file_content.decode()  # UTF-8

Because both signatures end in **kwargs: Any, an encoding= passed by the
caller — directly, or forwarded by partition() — is swallowed silently. On
main (ee2b3a35):

>>> partition_json(filename="utf16.json", encoding="utf-16")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

The same call succeeds for the sibling text formats, which is where the
inconsistency shows:

partitioner	takes `encoding`	reads via
`partition_text`	yes	`read_txt_file()`
`partition_csv`	yes	passed to the reader
`partition_json`	no	`open(..., encoding="utf8")`
`partition_ndjson`	no	`open(..., encoding="utf8")`

UTF-16 JSON is not exotic — it is what several Windows tools emit by default,
and json.dump to a file opened in a non-UTF-8 encoding produces it too.

Fix

Add encoding to both signatures and read through the existing
read_txt_file() helper, exactly as partition_text() does:

if filename is not None:
    _, file_text = read_txt_file(filename=filename, encoding=encoding)

elif file is not None:
    _, file_text = read_txt_file(file=file, encoding=encoding)
    file.seek(0)

read_txt_file() uses the given encoding when one is supplied and falls back to
detect_file_encoding() when it is not, so this also makes a non-UTF-8 document
readable without naming its encoding. The text= path is untouched, and
file.seek(0) is preserved so the detect-then-partition sequence over one handle
still works.

Source change is +11/-8 across the two files; no new helper, no behaviour change
for UTF-8 input.

Verification

$ pytest test_unstructured/partition/test_json.py test_unstructured/partition/test_ndjson.py
35 failed, 75 passed

The 35 failures are pre-existing on main in this environment — they come from
unstructured/nlp/tokenize.py failing to import an optional dependency
(ModuleNotFoundError: No module named 'installer'), unrelated to these files.
I compared the failure sets with git stash: identical before and after, and
the pass count goes 70 → 75, which is exactly the five tests added here.

Five regression tests, following the existing naming style in both files:

explicit encoding="utf-16" for filename= (both partitioners)
explicit encoding="utf-16" for file= (both partitioners)
no encoding given, UTF-16 detected (json)

Rollback proof. Restoring just the two read blocks to the hard-coded form —
leaving the new parameter in place — fails exactly those five and nothing else:

FAILED test_json.py::it_honors_an_explicit_encoding_for_a_non_utf8_file
FAILED test_json.py::it_honors_an_explicit_encoding_for_a_non_utf8_file_like_object
FAILED test_json.py::it_detects_the_encoding_of_a_non_utf8_file_when_none_is_specified
FAILED test_ndjson.py::it_honors_an_explicit_encoding_for_a_non_utf8_file
FAILED test_ndjson.py::it_honors_an_explicit_encoding_for_a_non_utf8_file_like_object
5 failed, 105 deselected

each with UnicodeDecodeError. ruff check on the four changed files is clean,
and a CHANGELOG entry is added under the current 0.27.8-dev0 section.

Notes

This is the parsing side. There are open PRs (#4397, #4398, #4447) addressing
non-UTF-8 crashes during file-type detection in file_utils/filetype.py; I
checked all currently open PRs and none touches partition/json.py or
partition/ndjson.py, so this does not overlap with them.

I kept the scope to honoring the parameter. I did not change what happens when a
document really is undecodable, nor touch the text= path.

cragwolfe

Requesting changes for four production blockers:

[P1] Preserve valid UTF-8 when no encoding is supplied (unstructured/partition/json.py:87,90, unstructured/partition/ndjson.py:84,87). The old paths decoded bytes strictly as UTF-8. The new paths call read_txt_file(..., encoding=None), which accepts a sufficiently confident charset_normalizer.detect() guess before attempting UTF-8. With the locked charset-normalizer==3.4.7, valid UTF-8 JSON containing a literal zero-width space can be selected as CP932 and still parse successfully, changing the element text (for example, RAG becomes R窶帰G). This is silent content corruption. Please make explicit encodings authoritative and, when encoding is omitted, preserve valid UTF-8 before falling back to non-UTF-8 detection. Add assertions on exact Unicode content for JSON and NDJSON through both filenames and binary streams; the current ASCII assertions do not cover this regression.
[P1] Preserve the existing file-like contract (unstructured/partition/json.py:90-91, unstructured/partition/ndjson.py:87-88). The removed implementation accepted any readable object and preserved already-decoded str values. The new no-encoding path routes the live object through convert_to_bytes(), which only accepts a few concrete types, may reopen a stream by .name, and uses BytesIO.getvalue() rather than reading from the current cursor. Exact-head CI already demonstrates the regression: both test_unit (3.13) and test_dockerfile fail in test_formskeysvalues_reads_saves because partition_json(file=io.StringIO(...)) now raises ValueError: Invalid file-like object type. Read the supplied stream once from its current position, pass str through, and decode buffered bytes without reopening or taking ownership of the caller's stream. Cover text streams, generic seekable binary streams, temporary-file wrappers, and nonzero cursor positions for both formats.
[P1] Forward encoding through the public dispatcher (unstructured/partition/auto.py:266-269). partition() binds encoding as a named parameter and uses it for detection, but its JSON/NDJSON special case calls the selected partitioner with only filename, file, and **kwargs. Because the bound value is not in kwargs, partition(..., encoding=...) still discards the caller's requested codec before parsing, leaving the stated top-level behavior unfixed and potentially producing different text or error behavior from a direct partition_json()/partition_ndjson() call. Pass encoding=encoding explicitly and add dispatcher-level tests for both formats using bytes that have different meanings or validity under competing codecs; a BOM-bearing UTF-16 success alone can pass through autodetection while the argument is still dropped.
[P2] Keep the existing fourth positional argument compatible (unstructured/partition/json.py:36-42, unstructured/partition/ndjson.py:36-42). Before this change, metadata_last_modified was the fourth positional parameter. Inserting encoding ahead of it silently rebinds existing calls: binary input attempts to use the timestamp as a codec and raises LookupError, while text= input can succeed but lose the requested timestamp. Preserve the original parameter order and place the new parameter afterward (or otherwise make only the new parameter keyword-only), with regression coverage for four-positional-argument calls in both partitioners.

The branch also currently conflicts with main in CHANGELOG.md; that is operational follow-up after the content blockers above are addressed.

(authored by codex)

Both partitioners read the source with a hard-coded UTF-8: with open(filename, encoding="utf8") as f: file_text = f.read() and, for a file-like object, a bare .decode() (also UTF-8). Neither accepted an `encoding` parameter, so an `encoding=` passed by the caller -- directly or via partition() -- landed in **kwargs and was discarded without a warning. A UTF-16 document raised UnicodeDecodeError even when its encoding was stated explicitly: partition_json(filename="utf16.json", encoding="utf-16") UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0 partition_text() and partition_csv() both take `encoding` and read through read_txt_file(), which uses the given encoding when present and detects it otherwise. Route the two JSON partitioners through the same helper and give them the same parameter. Reading a non-UTF-8 document without stating its encoding now works too, since read_txt_file() falls back to detection.

…rward encoding Four fixes from review of Unstructured-IO#4483: 1. Valid UTF-8 is preserved when no encoding is supplied. Routing through `read_txt_file(..., encoding=None)` made charset detection authoritative, and `charset_normalizer` can report high confidence in a legacy codec for bytes that are also valid UTF-8 -- a run of zero-width spaces is detected as CP932 at 0.907 and still parses as JSON, so the text was silently rewritten. Decoding now tries UTF-8 first and falls back to detection only for bytes that are not valid UTF-8. An explicit `encoding` stays authoritative and is used as given. 2. The file-like contract is restored. `convert_to_bytes()` accepts only a few concrete types, may reopen a stream by `.name`, and reads via `BytesIO.getvalue()` rather than from the cursor, which broke `partition_json(file=io.StringIO(...))`. The stream is again read once from its current position, an already-decoded `str` passes through, and the object is never reopened. 3. `partition()` forwards `encoding` to the JSON/NDJSON partitioners. It is bound as a named parameter, so it never reached them through `**kwargs` and the dispatcher silently dropped the caller's codec. 4. `metadata_last_modified` is the fourth positional parameter again; `encoding` follows it, so existing four-positional-argument calls keep binding as before. Both partitioners now share one `read_json_text()` helper so the two paths cannot drift apart.

linhongyu510 · 2026-09-16T04:37:45Z

Thanks — all four are addressed, and the branch is rebased onto current main so the CHANGELOG conflict is gone. Both entries are kept, with the text rewritten to describe what the code now does.

I reproduced each one before changing anything, against charset-normalizer==3.4.7 as you specified.

1. Valid UTF-8 is preserved when no encoding is supplied. Confirmed, and worth recording how it reproduces: my first attempt did not show corruption because json.dumps() defaults to ensure_ascii=True, which escapes the zero-width spaces and leaves the payload pure ASCII. With ensure_ascii=False the real bytes appear and detection picks CP932 at confidence 0.907 — A\u200bB\u200bC… decodes to A窶毅窶気窶汽… and still parses as JSON. main is unaffected because it decodes strictly as UTF-8, so this was a regression I introduced.

Decoding now tries UTF-8 first and only falls back to detection for bytes that are not valid UTF-8. An explicit encoding is used as given, so a genuine mismatch surfaces as an error rather than mojibake.

2. The file-like contract is restored. Reproduced exactly as CI showed — partition_json(file=io.StringIO(...)) raised ValueError: Invalid file-like object type. The stream is again read once from its current position, an already-decoded str passes through untouched, and the object is never reopened or taken ownership of. Covered for text streams, binary streams, TemporaryFile wrappers and a nonzero cursor, in both formats.

3. encoding is forwarded through the dispatcher. Confirmed: it is bound as a named parameter of partition(), so it never reached the partitioner through **kwargs. Now passed explicitly. The dispatcher tests use BOM-less UTF-16-LE precisely because autodetection cannot recover it — a BOM-bearing payload would pass even with the argument still dropped.

4. Positional order is preserved. metadata_last_modified is the fourth positional parameter again, with encoding after it; regression tests call both partitioners with four positional arguments and assert the timestamp lands.

Both partitioners now share one read_json_text() helper, so the two paths cannot drift apart again.

Verification

119 pass across test_json.py and test_ndjson.py. The 3 remaining failures are identical to main on this machine and unrelated to this change (…too_deep_to_pretty_print ×2 and it_rehydrates_on_a_detect_then_partition_sequence_over_the_same_file_handle); main shows 102 passed / the same 3 failed.
Rollback counter-proof: reverting only the UTF-8-first decode fails exactly the 8 new preservation/stream tests; reverting only the dispatcher forwarding makes both BOM-less UTF-16-LE cases fail with Not a valid json / Not a valid ndjson. Each group of tests pins one of the fixes.
ruff check and ruff format --check clean on all 7 changed files.

One note on test placement: the dispatcher tests live in test_auto.py alongside the other partition() routing tests, but I could not execute that file locally — it imports pdf2image/PIL at module scope and the PDF stack is not installable in this environment. I verified the same two scenarios through a standalone script instead (including the rollback check above), so CI will be the first full run of those two tests.

linhongyu510 · 2026-09-20T02:34:11Z

@cragwolfe Requesting a re-review when you have a moment — all four P1s from your 09-15 review were addressed in the 09-16 push (head fb8f5bb7), and the PR has been sitting on the now-outdated changes_requested since.

Not re-litigating the detail here since it is in my 09-16 reply, just the state: CI on this head is green across the board, and there are no unresolved review threads left. The only thing holding mergeable_state at BLOCKED is the review itself.

If anything in the responses did not land the way you intended, I would rather hear that than have it sit — happy to keep iterating.

cragwolfe requested changes Sep 15, 2026

View reviewed changes

linhongyu510 added 2 commits September 16, 2026 12:35

linhongyu510 force-pushed the fix/json-ndjson-honor-encoding branch from 21ca66c to fb8f5bb Compare September 16, 2026 04:37

This was referenced Sep 17, 2026

fix: honor encoding= in partition_md #4488

Open

fix: send the content_type partition_via_api accepts #4489

Open

Navigation Menu

fix: honor encoding= in partition_json and partition_ndjson - #4483

fix: honor encoding= in partition_json and partition_ndjson#4483
linhongyu510 wants to merge 2 commits into
Unstructured-IO:mainfrom
linhongyu510:fix/json-ndjson-honor-encoding

linhongyu510 commented Sep 10, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

cragwolfe left a comment

Uh oh!

linhongyu510 commented Sep 16, 2026

Uh oh!

linhongyu510 commented Sep 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

linhongyu510 commented Sep 10, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Verification

Notes

Uh oh!

cragwolfe left a comment

Choose a reason for hiding this comment

Uh oh!

linhongyu510 commented Sep 16, 2026

Verification

Uh oh!

linhongyu510 commented Sep 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

linhongyu510 commented Sep 10, 2026 •

edited by cubic-dev-ai Bot

Loading