Skip to content

gh-85287: Change codecs to raise precise UnicodeEncodeError and UnicodeDecodeError #113674

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
methane merged 30 commits into from
Mar 17, 2024
Merged

gh-85287: Change codecs to raise precise UnicodeEncodeError and UnicodeDecodeError #113674

Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
7339989
fix issue gh-85287
jjsloboda Jan 3, 2024
e92d414
add news blurb
jjsloboda Jan 3, 2024
10e7cd0
add more lenient unicode error handling within the except blocks
jjsloboda Jan 3, 2024
0122f90
fix IDNA-specific length issue
jjsloboda Jan 3, 2024
a310dd2
Merge branch 'main' into unicode-errors-fix-85287
jjsloboda Jan 3, 2024
63948d2
fix two issues
jjsloboda Jan 3, 2024
4479ab2
Merge branch 'main' into unicode-errors-fix-85287
jjsloboda Jan 3, 2024
81310e3
use plain UnicodeError for problems outside the en/decoded string
jjsloboda Jan 6, 2024
367de4e
split label empty vs too long
jjsloboda Jan 6, 2024
9f57515
use labels input for finding error offset, not output result
jjsloboda Jan 6, 2024
389122d
update test for undefined encoding
jjsloboda Jan 6, 2024
fe47caa
fixed linebreaks on some of the longer exceptions
jjsloboda Jan 6, 2024
a4098fa
Merge branch 'main' into unicode-errors-fix-85287
jjsloboda Jan 6, 2024
95cb5bb
add tests for unicode error offsets, and tighten up the logic for cal…
jjsloboda Jan 7, 2024
10d092f
Merge branch 'main' into unicode-errors-fix-85287
jjsloboda Jan 7, 2024
9ac979f
Merge branch 'main' into unicode-errors-fix-85287
jjsloboda Feb 16, 2024
aefd7c2
reduce scope of exception object, and fail gracefully if it cannot be…
jjsloboda Feb 16, 2024
f73ccfe
use object formatting on inbuf directly in exc
jjsloboda Feb 16, 2024
e0747b4
reduce scope of exception object, and fail gracefully if it cannot be…
jjsloboda Feb 16, 2024
93e99ae
update MultibyteIncrementalEncoder.getstate()
methane Feb 21, 2024
87e1f99
fixup
methane Feb 21, 2024
0728a43
change buffer size issue error back to UnicodeError
jjsloboda Feb 22, 2024
0f80786
Merge branch 'main' into unicode-errors-fix-85287
jjsloboda Feb 22, 2024
5c8c59e
Merge branch 'main' into unicode-errors-fix-85287
jjsloboda Feb 22, 2024
1cc911d
update test to match changed exception
jjsloboda Feb 22, 2024
9594bae
Update Modules/cjkcodecs/multibytecodec.c
methane Feb 23, 2024
ea3ff8a
improve idna codec errors
methane Feb 23, 2024
8a2bc50
improve punycode.decode()
methane Feb 23, 2024
a63e17a
improve punycode_decode again
methane Feb 23, 2024
4c329e4
Merge branch 'main' into unicode-errors-fix-85287
methane Mar 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update Modules/cjkcodecs/multibytecodec.c
  • Loading branch information
@methane
methane authored Feb 23, 2024
commit 9594baef5d4d0b4e55693699e75dc24142eb2a9f
2 changes: 1 addition & 1 deletion Modules/cjkcodecs/multibytecodec.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1006,7 +1006,7 @@ _multibytecodec_MultibyteIncrementalEncoder_setstate_impl(MultibyteIncrementalEn

if (statebytes[0] > MAXENCPENDING*4) {
PyErr_SetString(PyExc_UnicodeError, "pending buffer too large");
goto errorexit;
return NULL;
}

pending = PyUnicode_DecodeUTF8((const char *)statebytes+1,
Expand Down