borg2: which commands shall do a complete rebuild of the ChunkIndex?

[ThomasWaldmann]

Note: updated for situation WITH packs.

For high-latency stores (sftp: and a lot of what's available via rclone:) a full index rebuild needs to list all objects in all packs in the store and there are potentially a lot of objects. This is slow as it needs to scan over each pack file - only borg check --repair-index shall do that.

Most commands will use the existing chunks index. Most commands only add objects, so they will just write an additional partial index to index/sha256(partial_index) with just the new index entries.

For borg create:

• it's not a big problem if the index does not have all objects that exist in the repo. If that happens, borg create will just store something to the repo that's already there.
• it would be a severe problem though if the index would falsely say "we have that object" and borg would not store it to the repo. the archive would then reference a non-existing object.
• similarly it is fatal if the index points to a wrong pack_id

[ThomasWaldmann]

borg compact emits some stats and for that it needs the stored sizes of the repository objects.

With packs, we have pack_id, obj_offset and obj_size in the chunks index - that is enough to do stats for compact.

As borg compact moves object to other pack ids, it needs to write the complete updated index and get rid of old index files.

[ThomasWaldmann]

borg check could just try to work with the existing chunks index and not rebuild a new one.

repository check: hash all index files, hash all pack files - if it does not find issues we assume all is fine on the repository level.

If borg check finds index issues it would already terminate at that step.
Otherwise it would continue with hashing the packs and report all corrupted pack ids.

borg check --repair-index would rebuild the index by scanning packs (slow). it could either read the full pack (and then also hash it) or iterate only over the object headers.

borg check --repair-pack <pack_ids> could repair just the given pack ids without doing a full scan over all packs. repairing a pack means determining defective objects in the pack (by reading and decrypting them, if AEAD decryption fails, the object is corrupted). all non-corrupt objects will be written to a new pack, index fixed with the new pack_id.