Skip to content

GH-90344: replace io.IncrementalNewlineDecoder with non incremental newline decoders #30276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
brettcannon merged 7 commits into from
Nov 15, 2025
+5 −10
Merged

GH-90344: replace io.IncrementalNewlineDecoder with non incremental newline decoders #30276

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ffb090f
replace `io.IncrementalNewlineDecoder` with non incremental newline d…
guoci Dec 27, 2021
65e8149
📜🤖 Added by blurb_it.
blurb-it[bot] Oct 31, 2025
99ea6c4
Update Misc/NEWS.d/next/Core_and_Builtins/2025-10-31-14-03-42.gh-issu…
guoci Oct 31, 2025
c2087fa
Merge branch 'main' into replace_IncrementalNewlineDecoder
brettcannon Oct 31, 2025
bd47e96
Fix a lint issue with the news entry
brettcannon Oct 31, 2025
0f04700
fix news item
guoci Nov 9, 2025
ac77424
Merge branch 'main' into replace_IncrementalNewlineDecoder
brettcannon Nov 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 2 additions & 7 deletions Lib/doctest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ def _test():
import traceback
import types
import unittest
from io import StringIO, IncrementalNewlineDecoder
from io import StringIO, TextIOWrapper, BytesIO
from collections import namedtuple
import _colorize # Used in doctests
from _colorize import ANSIColors, can_colorize
Expand Down Expand Up @@ -237,10 +237,6 @@ def _normalize_module(module, depth=2):
else:
raise TypeError("Expected a module, string, or None")

def _newline_convert(data):
# The IO module provides a handy decoder for universal newline conversion
return IncrementalNewlineDecoder(None, True).decode(data, True)

def _load_testfile(filename, package, module_relative, encoding):
if module_relative:
package = _normalize_module(package, 3)
Expand All @@ -252,10 +248,9 @@ def _load_testfile(filename, package, module_relative, encoding):
pass
if hasattr(loader, 'get_data'):
file_contents = loader.get_data(filename)
file_contents = file_contents.decode(encoding)
# get_data() opens files as 'rb', so one must do the equivalent
# conversion as universal newlines would do.
return _newline_convert(file_contents), filename
return TextIOWrapper(BytesIO(file_contents), encoding=encoding, newline=None).read(), filename
with open(filename, encoding=encoding) as f:
return f.read(), filename

Expand Down
3 changes: 1 addition & 2 deletions Lib/importlib/_bootstrap_external.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -552,8 +552,7 @@ def decode_source(source_bytes):
import tokenize # To avoid bootstrap issues.
source_bytes_readline = _io.BytesIO(source_bytes).readline
encoding = tokenize.detect_encoding(source_bytes_readline)
newline_decoder = _io.IncrementalNewlineDecoder(None, True)
return newline_decoder.decode(source_bytes.decode(encoding[0]))
return _io.TextIOWrapper(_io.BytesIO(source_bytes), encoding=encoding[0], newline=None).read()


# Module specifications #######################################################
Expand Down
2 changes: 1 addition & 1 deletion Lib/test/test_importlib/test_abc.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -904,7 +904,7 @@ def test_universal_newlines(self):
mock = self.SourceOnlyLoaderMock('mod.file')
source = "x = 42\r\ny = -13\r\n"
mock.source = source.encode('utf-8')
expect = io.IncrementalNewlineDecoder(None, True).decode(source)
expect = io.StringIO(source, newline=None).getvalue()
self.assertEqual(mock.get_source(name), expect)


Expand Down
1 change: 1 addition & 0 deletions Misc/NEWS.d/next/Core_and_Builtins/2025-10-31-14-03-42.gh-issue-90344.gvZigO.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Replace :class:`io.IncrementalNewlineDecoder` with non incremental newline decoders in codebase where :meth:`!io.IncrementalNewlineDecoder.decode` was being called once.
Loading