macOS: borg mount problem with unicode normalisation

[aljungberg]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

BUG

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.1.10

Operating system (distribution) and version.

macos 10.14.6

Hardware / network configuration, and filesystems used.

MBP, Mac OS Extended

How much data is handled by borg?

1.5 TB

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg init -e none test.borg
mkdir test
filename=`printf 'la\xcc\x88.txt'`
echo "hello" >test/$filename
borg create test.borg::a test
mkdir test2
borg mount test.borg::a test2
ls test
# lä.txt
ls test2/test
# lä.txt
cat test/lä.txt
# hello
cat test2/test/lä.txt
# cat: test2/test/lä.txt: No such file or directory

So there is a file we can list but not open. I have the same problem with a real archive.

Thoughts

So the problem here is likely related to https://code.google.com/archive/p/macfuse/issues/139#c2: a mixup between precomposed and decomposed UTF encodings.

In fuse.py, in lookup, we have this:

inode = self.contents[parent_inode].get(name)

Here we get name encoded as b'l\xc3\xa4.txt' (precomposed). But in the archive, in self.contents[parent_inode] we have {b'la\xcc\x88.txt': 1000041} (decomposed). Both forms are technically equivalent:

>>> os.fsdecode(b'l\xc3\xa4.txt')
'lä.txt'
>>> os.fsdecode(b'la\xcc\x88.txt')
'lä.txt'
>>> unicodedata.normalize("NFD", os.fsdecode(b'l\xc3\xa4.txt')) == unicodedata.normalize("NFD", os.fsdecode(b'la\xcc\x88.txt'))
True

I would have liked to submit a patch but I honestly don't know the correct way to deal with this. If I add name = os.fsencode(unicodedata.normalize("NFD", os.fsdecode(name))) in lookup, it works for this particular error. But it would break if the archive used NFC instead.

We could also normalize the encodings before writing them to self.contents. This would work in both cases. But what encoding is supposed to be used to begin with?

[ThomasWaldmann]

Collecting some facts / questions (correct me in case I am wrong):

• on macOS HFS+, file names are NFD-normalized
• on Linux filesystems, it could be anything, but NFC is the usual thing
• borg currently just archives whatever byte representation it gets from the fs and does not do any normalization
• when typing some special char on macOS terminal it is NFC or NFC normalized or can it be both?
• when typing some special char on Linux terminal it is NFC or NFC normalized or can it be both?

[ThomasWaldmann]

Hmm, guess on macOS, typing stuff into terminal creates NFC and in the archive we have NFD, so there is no match.

But if one would copy and paste the filename from ls output to the cat command, it should work, right? @aljungberg

Also, using some gui filemanager which just uses the same filename for opening as it shows in a directory listing should also work, I guess.

[ThomasWaldmann]

Considering unicode not normalized vs. 2 different forms NFC / NFD of normalization:

If we:

• don't know what we have in the archive (could have been made on macOS, on Linux, on ...)
• don't know what we get (e.g. due to being typed in, on macOS, on Linux, on ...)

Then I only see 2 ways:

• just expect / demand same byte representation (== not borg's fault if you do it wrong)

• do a triple lookup:

  • first try: use whatever byte representation we get
  • miss: normalize to NFC, lookup again,
  • miss: normalize to NFD, lookup again.

The triple lookup would only work easily for FUSE mount. For borg extract and pattern matching, I guess it would be quite a pain.

[aljungberg]

But if one would copy and paste the filename from ls output to the cat command, it should work, right?

Correct. And like you say you can view the file in Finder too. But it doesn't work with autocomplete in the shell funnily enough, so it's by no means universal. Some tools are broken by the difference.

just expect / demand same byte representation (== not borg's fault if you do it wrong)

As I understand it on the Mac, HFS+ requires NFD normalisation. So if you extract a borg archive on a Mac and then archive it again, it'll all turn into NFD even if you wanted to standardise on NFC. So it's not within the user's control exactly.

I created the original archive upon which I discovered this error on a Mac as well, but it was with attic, I think. No idea how it ended up with some NFC files. But it does seem possible to get this on a single platform if you're a little unlucky.

I think the easiest way to resolve this is to pick one normalisation scheme and then use it everywhere. Ideally even when creating the archive, but since that train has left the station now I guess, we could just do it in the FUSE layer.

I don't see any immediate problem with your solution. Looks reasonable.

But you could also just normalise ahead of time.

So let's say we pick NFC. Then we can just do the equivalent of self.contents[parent_inode] = {unicodedata.normalize("NFC", k): v for k, v in self.contents[parent_inode].items() (except we'd never do that, just do it right away when building the dict). Now when we want to look up an inode we call self.contents[parent_inode].get(unicodedata.normalize("NFC", name)).

This will work except if you have two files with the same name in the same folder, differing only in their encoding. But that situation is caused by not normalising the names to begin with when creating the archive and it's too late to try to fix it when accessing the files. It's a bit of a pathological situation, not sure what would even happen if you tried to extract such an archive on a Mac.

[pgerber]

I'd be very careful changing the encoding in any way. I've seen mixed encoding (some using NFC and some NFD) when the files where created on Windows in particular. I'm not sure why exactly, could be that some tools just encoding one way or the other. I'm sure changing the encoding while storing or restoring the backups would break some of our systems. In particular, we have some customer-supplied HTML files containing things like <a href="ä.html"> encoded either as NFC or NFD. Changing the file name will break this reference. Also, I'd not be surprised if some other applications could no longer find files after the encoding changed.