Retry HDF5 opens on transient file-lock failures (fixes zoomify --balance -p N>1 on NFS)

[treznicek]

Problem

cooler zoomify --balance -p N (N > 1) intermittently fails on NFS-mounted
filesystems with BlockingIOError: [Errno 11] ... unable to lock file, raised
from h5py.File(cool_path, 'r+') in cli/balance.py. -p 1 avoids it.

Diagnosis

The issue suggests multiple zoom levels are balanced concurrently. That doesn't
appear to be what's happening — the relevant loops are all strictly sequential:

• zoomify() runs zoomify_cooler() to completion, and only then calls
  invoke_balance() (cli/zoomify.py).
• invoke_balance() balances resolutions one at a time, calling
  balance_cmd.main() in-process and waiting for each to finish.
• zoomify_cooler() likewise coarsens resolutions in a plain sequential loop.

So two zoom levels are never in flight at once, and the only writer is always
the parent process.

What -p N > 1 does introduce is multi-process access to the same file:
coarsen_cooler() forks an mp.Pool whose workers open the file read-only,
while the parent periodically opens it 'r+' to write. With -p 1 no pool is
created at all (builtin map is used), so no second process ever touches the
file — which is why the workaround works.

This is aggravated by write_pixels() opening and closing the output file once
per chunk, i.e. one lock acquire/release cycle per chunk. The finest resolution
has the most chunks, which matches the report that it fails on the largest
resolution level. On NFS, where flock() semantics are frequently unreliable,
that lock acquisition can fail transiently even though cooler's own
multiprocess.Lock correctly prevents any actual read/write race.

Fix

Add open_hdf5_with_retry() in cooler/util.py: a drop-in h5py.File
replacement that retries with exponential backoff on BlockingIOError and
re-raises once attempts are exhausted. Applied to the write-path opens
exercised by zoomify --balance:

• create/_create.py — write_pixels() and all opens in create()
• cli/balance.py — all three opens, including the one in the reported traceback

Also documents HDF5_USE_FILE_LOCKING=FALSE in the zoomify/balance help
text as an immediate workaround.

Deliberately not included

This is a targeted mitigation, not a fix for the underlying lock churn. Holding
a single file handle open across write_pixels()'s whole write loop, instead of
reopening per chunk, would reduce lock acquisitions at the source — but that
touches a performance-sensitive path and felt out of scope here. It's noted as
follow-up work in a code comment and in CHANGES.md.

Testing

• New unit tests for the retry helper (eventual success after transient
  failures; re-raise after exhaustion).
• Full suite passes locally.
• Note: this cannot be reproduced in CI, which doesn't run on NFS. Verification
  on a real NFS mount would be valuable.

Fixes #486

Generated with Claude Code