PAGE exporter writes with system codepage; crashes on U+FFFD

[CrazyCrud]

What happened?

HI,

when exporting PAGE-XML I get:
UnicodeEncodeError: 'charmap' codec can't encode character '\ufffd' ...

I only run the PAGE exporter (no TXT). This suggests the exporter opens the XML file without specifying encoding, so Python uses the system codepage. U+FFFD is valid in UTF-8 but not f.ex. in cp1252.

Environment: Windows (cp1252), model: fhswf/TrOCR_german_handwritten (but likely model-agnostic).

I think the problem can be resolved by setting the encoding to "utf-8" on this line: https://github.com/AI-Riksarkivet/htrflow/blob/main/src/htrflow/serialization/serialization.py#L369

I tried this and the export worked:

with io.open(filename, "w", encoding="utf-8", newline="") as f:
            f.write(doc)

Thank you in advance!

Best regards,
Constantin

Steps To Reproduce

Use the model "fhswf/TrOCR_german_handwritten" for TextRecognition.

Relevant log output

UnicodeEncodeError: 'charmap' codec can't encode character '\ufffd' in position 10785: character maps to <undefined>

[Borg93]

Thank you for opening up issuess! We will look at them this week :)

[Borg93]

Will try add this to v0.2.7

[CrazyCrud]

@Borg93 thank you!