Keep failed LoRA safe merges from changing base weights - #3611
Open
Robertboy18 wants to merge 1 commit into
Open
Keep failed LoRA safe merges from changing base weights#3611Robertboy18 wants to merge 1 commit into
Robertboy18 wants to merge 1 commit into
Conversation
BenjaminBossan
requested changes
Aug 31, 2026
BenjaminBossan
left a comment
Member
There was a problem hiding this comment.
Thanks for this fix, it looks good. Could you please merge with the latest main branch and ensure that make style passes?
A small ask, as this is something I noticed while reviewing the PR: For lora.Embedding, lora_bias is not supported, but this is checked in __init__. It would be better to check this in update_layer instead. No test is necessary for this. Thanks.
Robertboy18
force-pushed
the
fix-lora-safe-merge-bias-atomicity
branch
from
September 3, 2026 18:55
8a48d44 to
a85d736
Compare
Author
|
Thanks! Rebased onto current main, moved the Embedding |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I was looking at the
safe_mergepath withlora_bias=Trueand noticed that the base weight was assigned before the bias candidate was validated. If bias validation then raises, the weight stays changed even though the adapter was never marked as merged, sounmerge()cannot restore it.This checks the weight and bias first and only commits them once both are valid, for linear and convolutional LoRA layers. I also added CPU regression coverage for non-finite adapter bias and biasless target layers.
Tests:
pytest -q tests/test_custom_models.py -k "lora_safe_merge_does_not_mutate_base_layer_on_non_finite_bias or merge_with_lora_bias_when_base_layer_has_no_bias_warns_and_raises or glora_safe_merge_does_not_corrupt_base_layer_on_non_finite_adapter"safe_mergecases with LoRA bias for Linear, Conv2d, and Conv3dI also have a small TorchLean formalization of the failed-merge atomicity property and can attach it if useful :)