GH-78319: Stop sending the UTF8 marker when appending messages - #107290
GH-78319: Stop sending the UTF8 marker when appending messages#107290arnt wants to merge 1 commit into
Conversation
…to a mailbox. The UTF8 marker is defined in RFC 6855 and tells the server that the message being appended contains UTF8 addresses, an unencoded UTF8 subject, etc. However, if a client appends a message containing UTF8 addresses but without that marker, the bytes can only be parsed as UTF8 because that's the only RFC-compliant way to parse those bytes. RFC 6855 says clients MUST send the UTF8 marker. Due to an accidental discrepancy, RFC 9051 (IMAP4rev2) does not contain that marker. IMAP4rev2 was intended to be upwardly compatible with RFC 6855, but this problem broke that. This has no ill effects, since the marker does not change the message's meaning. While investigating the problem, I noticed that Python uses the marker incorrectly: Python uses it to mark ALL messages if UTF8=ACCEPT support has been enabled, not just ones that contain UTF8 addresses. The best way forward appear to be using the syntax defined in RFC 9051 and publishing a revision to RFC 6855, so this change modifies imaplib to match RFC 9051. FWIW JMAP is like IMAP4rev2 in this case; UTF8 is just there, without any marker. Also, neither UTF8=ACCEPT, IMAP4rev2 or JMAP provide any way to learn whether a message was stored with or without the marker. This quasi-accidentally solves python#78319 by removing the case that broke.
|
Most changes to Python require a NEWS entry. Please add it using the blurb_it web app or the blurb command-line tool. |
|
This seems small enough to not require a NEWS entry, I'd say. |
|
I submitted the internet-draft to supersede RFC 6855 now and expect to get it to RFC fairly quickly. |
|
This PR is stale because it has been open for 30 days with no activity. |
|
Fixed by #9436. |
|
@serhiy-storchaka FYI #9436 doesn't fix this. #9436 makes python send "UTF8" even for messages that only use ASCII. That's a bug. The code in this PR is correct, and conforms to the APPEND rules in both RFC 9051 and RFC 9755. |
|
You are right, I closed this too quickly. #9436 implements the RFC 6855 syntax that RFC 9755 removed. But imaplib cannot tell the two kinds of server apart: RFC 6855 requires the marker for UTF-8 headers, RFC 9755 removes it, and both advertise What do deployed RFC 6855 servers do with an APPEND whose The RFC 6855 syntax is already released in 3.13.8+ and 3.14.1+, so removing it needs backports. I can reopen this PR; it needs a rebase and a NEWS entry. cc @svarshavchik, who reported gh-78319 and implements RFC 6855 in Courier-IMAP. |
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
|
Courier was coded for RFC-6855, and it rejects messages with UTF-8 headers if the APPEND command did not send a UTF-8 marker. This mess is yet another prize-winner from the IETF. I cannot think of a single good reason for the change in RFC 9755. What a clusterfark. |
The UTF8 marker is defined in RFC 6855 and tells the server that the message being appended contains UTF8 addresses, an unencoded UTF8 subject, etc. However, if a client appends a message containing UTF8 addresses but without that marker, the bytes can only be parsed as UTF8 because that's the only RFC-compliant way to parse those bytes.
RFC 6855 says clients MUST send the UTF8 marker.
Due to an accidental discrepancy, RFC 9051 (IMAP4rev2) does not contain that marker. IMAP4rev2 was intended to be upwardly compatible with RFC 6855, but this problem broke that. This has no ill effects, since the marker does not change the message's meaning.
While investigating the problem, I noticed that Python uses the marker incorrectly: Python uses it to mark ALL messages if UTF8=ACCEPT support has been enabled, not just ones that contain UTF8 addresses.
The best way forward appear to be using the syntax defined in RFC 9051 and publishing a revision to RFC 6855, so this change modifies imaplib to match RFC 9051.
FWIW JMAP is like IMAP4rev2 in this case; UTF8 is just there, without any marker. Also, neither UTF8=ACCEPT, IMAP4rev2 or JMAP provide any way to learn whether a message was stored with or without the marker.
This quasi-accidentally solves #78319 by removing the case that broke.