Skip to content

gh-125660: Python implementation of json.loads() accepts invalid unicode escapes#125683

Merged
serhiy-storchaka merged 7 commits into
python:mainfrom
nineteendo:patch-1
Oct 18, 2024
Merged

gh-125660: Python implementation of json.loads() accepts invalid unicode escapes#125683
serhiy-storchaka merged 7 commits into
python:mainfrom
nineteendo:patch-1

Conversation

@nineteendo

@nineteendo nineteendo commented Oct 18, 2024 •
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Contributor

This condition in json.decoder is insufficient:

cpython/Lib/json/decoder.py

Line 61 in f203d1c

if len(esc) == 4 and esc[1] not in 'xX': 

>>> import sys
>>> sys.modules["_json"] = None
>>> import json
>>> json.loads(r'["\u 000", "\u-000", "\u+000", "\u0_00"]')
['\x00', '\x00', '\x00', '\x00']

@bedevere-app bedevere-app Bot mentioned this pull request Oct 18, 2024
Closed
@nineteendo nineteendo marked this pull request as ready for review October 18, 2024 09:10
ZeroIntensity
ZeroIntensity approved these changes Oct 18, 2024
View reviewed changes

@ZeroIntensity ZeroIntensity left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For other reviewers, see also: #125687

I think this is fine, but per @taleinat, this shouldn't get backported--it's technically a very minor breaking change.

@taleinat

taleinat commented Oct 18, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

I think this is fine, but per @taleinat, this shouldn't get backported--it's technically a very minor breaking change.

After further consideration, I do consider this to be a clear bug fix, which should be backported to versions in "bugfix" status (i.e. 3.12).

@ZeroIntensity

ZeroIntensity commented Oct 18, 2024

Copy link
Copy Markdown
Member

Are you sure? I'm not certain what policy PyPy, IronPython, Jython, etc. have for changes that we backport, but point of pure-Python modules was almost exclusively for other Python implementations--I don't want to accidentally cause any surprises.

@ZeroIntensity ZeroIntensity mentioned this pull request Oct 18, 2024
Merged
@taleinat

taleinat commented Oct 18, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Are you sure?

No, I am not sure, I've been away from Python development for several years. It would be good to get another core dev's perspective.

@taleinat

taleinat commented Oct 18, 2024

Copy link
Copy Markdown
Contributor

Upon further consideration, yes, the chance of breaking something with a backport is large relative to the likely pain caused by the current, buggy, state. Let's avoid backporting.

serhiy-storchaka
serhiy-storchaka approved these changes Oct 18, 2024
View reviewed changes

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. 👍

@serhiy-storchaka serhiy-storchaka added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Oct 18, 2024
@serhiy-storchaka

serhiy-storchaka commented Oct 18, 2024

Copy link
Copy Markdown
Member

This is clearly a bug. json.loads() can be a part of validation. If it accepts an invalid JSON, it can fail later, in other program with more strict JSON parser. I do not think that anybody depends on such buggy behavior.

@serhiy-storchaka serhiy-storchaka merged commit df75136 into python:main Oct 18, 2024
@miss-islington-app

miss-islington-app Bot commented Oct 18, 2024

Copy link
Copy Markdown

Thanks @nineteendo for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 18, 2024
@nineteendo @miss-islington
pythongh-125660: Reject invalid unicode escapes for Python implementa…
025b9e8 
…tion of JSON decoder (pythonGH-125683)

(cherry picked from commit df75136)

Co-authored-by: Nice Zombies <nineteendo19d0@gmail.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Oct 18, 2024
@nineteendo @miss-islington
pythongh-125660: Reject invalid unicode escapes for Python implementa…
49905fb 
…tion of JSON decoder (pythonGH-125683)

(cherry picked from commit df75136)

Co-authored-by: Nice Zombies <nineteendo19d0@gmail.com>
@bedevere-app

bedevere-app Bot commented Oct 18, 2024

Copy link
Copy Markdown

GH-125694 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.13 bugs and security fixes label Oct 18, 2024
@bedevere-app

bedevere-app Bot commented Oct 18, 2024

Copy link
Copy Markdown

GH-125695 is a backport of this pull request to the 3.12 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.12 only security fixes label Oct 18, 2024
@nineteendo nineteendo deleted the patch-1 branch October 18, 2024 12:30
serhiy-storchaka pushed a commit that referenced this pull request Oct 21, 2024
@miss-islington @nineteendo
[3.13] gh-125660: Reject invalid unicode escapes for Python implement…
d9dafc7 
…ation of JSON decoder (GH-125683) (GH-125694)

(cherry picked from commit df75136)

Co-authored-by: Nice Zombies <nineteendo19d0@gmail.com>
serhiy-storchaka pushed a commit that referenced this pull request Oct 21, 2024
@miss-islington @nineteendo
[3.12] gh-125660: Reject invalid unicode escapes for Python implement…
2746ec4 
…ation of JSON decoder (GH-125683) (GH-125695)

(cherry picked from commit df75136)

Co-authored-by: Nice Zombies <nineteendo19d0@gmail.com>
ebonnal pushed a commit to ebonnal/cpython that referenced this pull request Jan 12, 2025
@nineteendo @ebonnal
pythongh-125660: Reject invalid unicode escapes for Python implementa…
a05b129 
…tion of JSON decoder (pythonGH-125683)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes
@ZeroIntensity ZeroIntensity ZeroIntensity approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@nineteendo @taleinat @ZeroIntensity @serhiy-storchaka