Web Server returning utf8 (not utf-8) for ContentType header throws an error in ReadBufferAsString method

[kilasuit]

Description

Whilst wrapping some code around a developer API in PowerShell, I was getting an error thrown about the incorrect use of utf8 as returned by in the ContentType Header by that server.

I have a workaround in a private build of PowerShell & a draft PR that I've just closed as wasn't really the best approach.

I feel it makes more sense to fix this in the System.Net.Http dotnet package so that other users including PowerShell could get the benefit of the fix & not come across this issue.

I think I have a interim fix, (not yet built & tested locally that's on the list) which is relatively small that I think makes sense for as part of inclusion in the System.Net.Http dotnet package. This would negate my own draft PR in the PowerShell repo that is currently failing as well as my need for keeping a private build.

Reproduction Steps

set up a webserver that throws a contentType header of application/json; charset=utf8

in PowerShell 7.6.5 that sets the ContentType Header to utf8

function Start-TestWebServer {
Add-Type -AssemblyName System.Net

$listener = [System.Net.HttpListener]::new()
$listener.Prefixes.Add("http://localhost:8088/")
$listener.Start()

Write-Host "Listening on http://localhost:8088/ (Ctrl+C to stop)"

$stop = $false

# Ctrl+C handler
$null = Register-EngineEvent -SourceIdentifier ConsoleCancelEvent -Action {
    Write-Host "Ctrl+C detected, stopping..."
    $stop = $true
}
while (-not $stop -and $listener.IsListening) {
            try {
                $context = $listener.GetContext()   # blocking, but safe inside Task
                $response = $context.Response

                # Intentionally broken charset
                $response.ContentType = "application/json; charset=utf8"

                $body = "Hello from a broken charset server"
                $bytes = [System.Text.Encoding]::UTF8.GetBytes($body)

                $response.ContentLength64 = $bytes.Length
                $response.OutputStream.Write($bytes, 0, $bytes.Length)
                $response.OutputStream.Close()
            }
            catch {
                break
            }
        }
# Main loop waits for Ctrl+C
try {
    while (-not $stop) {
        Start-Sleep -Milliseconds 200
    }
}
finally {
    Write-Host "Ctrl+ C pressed - Stopping listener..."
    $listener.Stop()
    $listener.Close()
    $listener.Dispose()
    Write-Host "Listener stopped & disposed."
}
}
Start-TestWebServer

Then run from another PowerShell 7.6.5 window

Invoke-WebRequest http://localhost:8088 | Select -expand Headers 

This returns showing the broken header

Invoke-RestMethod http://localhost:8088 -Verbose -Debug

This returns exception showing that the utf8 not being supported

 System.ArgumentException: 'utf8' is not a supported encoding name. For information on defining a custom encoding, see the documentation for the Encoding.RegisterProvider method. (Parameter 'name')
   at System.Text.EncodingTable.InternalGetCodePageFromName(String name)
   at System.Text.EncodingTable.GetCodePageFromName(String name)
   at System.Text.Encoding.GetEncoding(String name)
   at System.Net.Http.HttpContent.ReadBufferAsString(LimitArrayPoolWriteStream stream, HttpContentHeaders headers)

Expected behavior

Ideally a graceful conversion that is either contained within this method or pulled in from another package that does a map nicely from utf8 to utf-8 or other encodings

Actual behavior

Errors as shown in the repro steps

Regression?

Unknown

Known Workarounds

In the private build of PowerShell I have & subsequent failing draft PR I got around this with a catch all approach defaulting the Encoding to UTF8 which is a hacky hack & isn't really a sensible fix.

Configuration

Name Value

---

PSVersion 7.6.5
PSEdition Core
GitCommitId 7.6.5
OS Microsoft Windows 10.0.26200
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.4
SerializationVersion 1.1.0.1
WSManStackVersion 3.0

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @karelz, @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[MihaZupan]

Are you able to ask the server authors to fix their logic?

In C# this is rather easy to workaround as you can overwrite the header yourself.
I'm not convinced the default HttpClient logic should be trying to catch such invalid inputs. If this were very common, it seems like something Encoding.GetEncoding would want to consider rather than consuming components.

For PowerShell, the recommendation on the PR to reuse PowerShell's existing mapping support (PowerShell/PowerShell#27789 (comment)) seems like a reasonable way to solve this.

[dotnet-policy-service]

This issue has been marked needs-author-action and may be missing some important information.

[kilasuit]

Yes I intend to do so, & whilst it doesn't feel like it's a common thing, I get the feeling that we shouldn't really need to adapt downstream (where possible, like in via the recommendation to use what we have in PowerShell which may well end up being what I look at doing there if we don't fix it here) when people, or in future 'AI' tooling, may decide to strip out - etc. I know we always strive to get people to "accept & adopt the standards" as they currently are.

In C# this is rather easy to workaround as you can overwrite the header yourself.

Yeah, same in PowerShell or F# though shouldn't need to workaround downstream when it's a case of being like 90% of what's expected.

it seems like something Encoding.GetEncoding would want to consider rather than consuming components.

That may well be a better place to surface a change like this than the attempt I've made via #133322 but this has at least given me the opportunity to understand a bit more about the current codebase, try and build it locally and then propose a "fix" even if that fix isn't what ends up being what gets taken.