gh-84353: Preserve non-UTF-8 filenames when appending to ZipFile - #150091

…le (GH-150091) (GH-150529)

Preserve non-UTF-8 filenames when appending to a ZipFile.

---------
(cherry picked from commit 24c6bbc)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>

…le (GH-150091) (GH-150528)

Preserve non-UTF-8 filenames when appending to a ZipFile.

---------
(cherry picked from commit 24c6bbc)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>

…le (GH-150091) (GH-150527)

Preserve non-UTF-8 filenames when appending to a ZipFile.

---------
(cherry picked from commit 24c6bbc)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
Co-authored-by: Gregory P. Smith <greg@krypto.org>

Fix a regression introduced by pythongh-84353/pythongh-150091 where the EFS flag
was dropped or omitted when a file with an ASCII filename and a UTF-8
comment was written to an archive. This affected both newly added files
and existing files rewritten to the central directory in append mode,
causing an unexpected metadata change and leading to comment
mis-decoding.

Introduce an internal `_metadata_encoding` attribute for `ZipInfo` to
ensure that files read from an archive preserve their original encoding
and EFS flags, while newly added files now properly enforce EFS if they
contain a non-ASCII filename or comment.

Fix a regression introduced by pythongh-84353/pythongh-150091 where the EFS flag
was dropped or omitted when a file with an ASCII filename and a UTF-8
comment was written to an archive. This affected both newly added files
and existing files rewritten to the central directory in append mode,
causing an unexpected metadata change and leading to comment
mis-decoding.

Introduce an internal `_metadata_encoding` attribute for `ZipInfo` to
ensure that files read from an archive preserve their original encoding
and EFS flags, while newly added files now properly enforce EFS if they
contain a non-ASCII filename or comment.

Fix a regression introduced by pythongh-84353/pythongh-150091 where the EFS flag
was dropped or omitted when a file with an ASCII filename and a UTF-8
comment was written to an archive. This affected both newly added files
and existing files rewritten to the central directory in append mode,
causing an unexpected metadata change and leading to comment
mis-decoding.

Introduce an internal `_metadata_encoding` attribute for `ZipInfo` to
ensure that files read from an archive preserve their original encoding
and EFS flags, while newly added files now properly enforce EFS if they
contain a non-ASCII filename or comment.

python/cpython#150091 changed the default non-utf8 encoding from 'ascii'
to 'cp437' for encoding zip filenames. This means that characters that
could be encoded in cp437 but not ascii would be encoded in the zip data
using cp437 vs. previously they woud've used utf-8.

Note that non-ascii characters that are valid cp437 characters will
always take up a single byte when encoded as cp437, but multiple when
encoded using utf-8.

Because `len(zipstream)` assumes that filenames are encoded using only
ascii or utf-8, this resulted in the calculated length being off by (at
least) 1 byte for every cp437, non-ascii character in all the filenames.

The fix in this commit is to override `ZipInfo._encodeFilenameFlags` to
revert Python's change and force filenames to only use ascii/utf-8.
The change was made to support modifying existing zips with
cp437-encoded filenames which isn't relevant to this project. And while
technically cp437 is the standard encoding for zipfile names, using the
more strict ascii subset is less likely to be misinterpreted by
unarchivers since they won't have to guess at what extended ascii
encoding was used - it'll always be strict ascii or utf-8.

python/cpython#150091 changed the default non-utf8 encoding from 'ascii'
to 'cp437' for encoding zip filenames. This means that characters that
could be encoded in cp437 but not ascii would be encoded in the zip data
using cp437 vs. previously they woud've used utf-8.

Note that non-ascii characters that are valid cp437 characters will
always take up a single byte when encoded as cp437, but multiple when
encoded using utf-8.

Because `len(zipstream)` assumes that filenames are encoded using only
ascii or utf-8, this resulted in the calculated length being off by (at
least) 1 byte for every cp437, non-ascii character in all the filenames.

The fix in this commit is to override `ZipInfo._encodeFilenameFlags` to
revert Python's change and force filenames to only use ascii/utf-8.
The change was made to support modifying existing zips with
cp437-encoded filenames which isn't relevant to this project. And while
technically cp437 is the standard encoding for zipfile names, using the
more strict ascii subset is less likely to be misinterpreted by
unarchivers since they won't have to guess at what extended ascii
encoding was used - it'll always be strict ascii or utf-8.