Start-Transcript uses lossy ASCII character encoding instead of BOM-less UTF-8 if the target file happens to exist #13677

Note:

• Windows PowerShell is not affected - UTF-8 with BOM is used consistently there.
• A fix for the closely related Start-Transcript -Append uses lossy ASCII character encoding when appending to a BOM-less target file #13678 could probably encompass a fix for this issue as well.

When the target file happens to exist and doesn't have a BOM - even without -Append - Start-Transcript mistakenly uses ASCII encoding to write the file.

Without -Append, Start-Transcript shouldn't even look at the existing file - it should simply replace it, and use the default encoding (BOM-less UTF-8). See #13678 for a closely related bug with -Append.

Instead, Start-Transcript apparently looks for a BOM at the start of an existing file and uses ASCII encoding if it doesn't find one.

Steps to reproduce

# To surface the bug, make sure that the target file exists and doesn't have a BOM.
$null > temp:/$PID.txt

$null = Start-Transcript temp:/$PID.txt
'ü' # output a string with a non-ASCII-range character
$null = Stop-Transcript

Select-String -Quiet 'ü' temp:/$PID.txt | Should -BeTrue

Remove-Item temp:/$PID.txt

Expected behavior

The test should succeed.

Actual behavior

The test fails, because ü was transliterated to verbatim ?, suggesting that ASCII encoding was used to write the file.

Environment data

PowerShell Core 7.1.0-preview.7