Skip to content

Apply the bit-vs-byte suffix rule to the binary size units - #4197

Open
vineethsaivs wants to merge 1 commit into
huggingface:mainfrom
vineethsaivs:binary-unit-bit-sizes
Open

Apply the bit-vs-byte suffix rule to the binary size units#4197
vineethsaivs wants to merge 1 commit into
huggingface:mainfrom
vineethsaivs:binary-unit-bit-sizes

Conversation

@vineethsaivs

Copy link
Copy Markdown
Contributor

Symptom

convert_file_size_to_int() treats a lowercase trailing b as bits and divides by 8, but only for the decimal units. The binary units ignore the suffix:

>>> from accelerate.utils import convert_file_size_to_int
>>> convert_file_size_to_int("1Gb")    # gigabit, correct
125000000
>>> convert_file_size_to_int("1Gib")   # gibibit, should be 2**30 // 8
1073741824
>>> convert_file_size_to_int("1GiB")   # gibibyte
1073741824

"1Gib" and "1GiB" return the same number, so a size given in gibibits is read as gibibytes and comes out 8x too large. The same holds for Kib and Mib.

Root cause

Six sibling branches in the same function, and only three of them apply the rule:

elif size.upper().endswith("GIB"):
    mem_size = int(float(size[:-3]) * (2**30))     # no bit handling
...
elif size.upper().endswith("GB"):
    int_size = int(float(size[:-2]) * (10**9))
    mem_size = int_size // 8 if size.endswith("b") else int_size

This is not a question of whether bit units should be supported: the decimal branches already say they are, and give the exact two-line form. The *iB branches sit above them and were never updated.

Fix

Apply the same two lines to the three binary branches so all six units agree on what a trailing b means. An uppercase B takes the identical path it took before, and the float parsing added in #1799 plus the mem_size < 0 guard from #2507 are untouched.

Why it matters

This is what max_memory entries are parsed with, in get_max_memory, get_balanced_memory, infer_auto_device_map and load_checkpoint_in_model. An 8x overshoot there is silent: it plans a device map against memory that is not there, rather than raising.

Test

Extended the existing ModelingUtilsTester::test_convert_file_size with the six lowercase-b spellings:

# this branch
$ pytest tests/test_modeling_utils.py -k test_convert_file_size -q
1 passed, 43 deselected

# against main
$ pytest tests/test_modeling_utils.py -k test_convert_file_size -q
1 failed, 43 deselected
E   AssertionError: 512Kib
E   assert 524288 == 65536
E    +  where 524288 = convert_file_size_to_int('512Kib')

The three decimal cases in that loop (100Kb, 100Mb, 2Gb) pass on both sides, so they are controls rather than tests of the new code.

ruff check reports the same 41 findings on these two files before and after the change (my ruff is newer than the pinned one), and ruff format --check is clean. Three test_infer_auto_device_map_*_buffer* tests fail when the whole file is run, both with and without this change, and pass in isolation either way, so they are unrelated ordering flakes.

convert_file_size_to_int() reads a lowercase trailing "b" as bits and divides by
8, but only on the decimal units. The three binary branches (KiB, MiB, GiB) take
the plain multiplier, so "1Gib" is read as one gibibyte instead of one gibibit
and comes out 8x too large. "1Gib" and "1GiB" currently return the same number.

Apply the same two-line form the GB/MB/KB branches already use, so all six units
agree on what a trailing "b" means. An uppercase "B" takes the identical path it
took before.

The value is what max_memory entries are parsed with, in get_max_memory,
get_balanced_memory, infer_auto_device_map and load_checkpoint_in_model, so an
8x overshoot is silent: it plans a device map against memory that is not there.

test_convert_file_size gains the six lowercase-b spellings; the three decimal
ones already pass and are there as controls.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant