feat(nano): support LoRA finetuning of the Qwen3 LLM - #3456
Open
KamitobiHaru wants to merge 1 commit into
Open
Conversation
The FunASR-Nano model config already declares llm_conf.use_lora and llm_conf.lora_conf (r/lora_alpha/lora_dropout/target_modules), but nothing consumed them: use_lora was silently ignored and training fell back to the full-parameter LLM path. Wire it up: - When llm_conf.use_lora is true, replace the LLM target Linear layers (default q_proj/v_proj) with lora.Linear adapters, sharing the frozen base weight and adding trainable lora_A/lora_B. The adapter params are created in the base weight's dtype (bf16), so the LoRA path does not depend on autocast. - Pairs with the existing lora_only / mark_only_lora_as_trainable flow in train.py / train_ds.py: lora_only=true freezes every non-LoRA parameter for pure-LoRA training; lora_only=false + unfrozen encoder/adaptor conf keeps them trainable while LoRA-tweaking only the LLM. - Checkpoints save the full state dict (unchanged base + adapter params), so a LoRA run can be resumed or decoded with the same use_lora=true config; for a standalone deployment checkpoint, fold with W' = W + alpha/r * B @ A. - Add lora_finetune.sh and document the recipe in finetune.md / finetune_zh.md. Validated end-to-end with the real model: 56 adapters injected (28 Qwen3 layers x q_proj+v_proj), optimizer updates only the 112 LoRA params, and a 2-step smoke training run through funasr-train-ds completes and saves a checkpoint whose lora keys are bf16 and match the base dtype. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The FunASR-Nano model config already ships
llm_conf.use_lora/llm_conf.lora_conf(r, lora_alpha, lora_dropout, target_modules), but nothing consumed them —
use_lorawas silently ignored and training fell back to the full-parameter LLM path. This PR wires LoRA up for the Qwen3-0.6B LLM insideFunASRNano.Changes
funasr/models/fun_asr_nano/model.py: whenllm_conf.use_lorais true, replace the LLM targetnn.Linearlayers (defaultq_proj/v_proj) withlora.Linearadapters (reusing the existingfunasr/models/lora/module used by Paraformer). The pretrained base weight is shared and frozen; trainablelora_A/lora_Bare added. Adapter params are created in the base weight's dtype (bf16), so the LoRA path works without relying onautocast to reconcile dtypes.
examples/.../fun_asr_nano/lora_finetune.sh+docs/finetune.md/finetune_zh.md: LoRA recipe with freeze semantics and deployment (fold) notes.Design notes
lora_only/mark_only_lora_as_trainableflow already intrain.py/train_ds.py:lora_only=true(default in the example) freezes every non-LoRA parameter → pure-LoRA.lora_only=false+ unfreeze encoder/adaptor conf keeps them trainable while LoRA-tweaking only the LLM (the recipe used for the medical ASR deliverable).use_lora=trueconfig. No new save/load code.W' = W + (alpha/r) * B @ Aand drop the lora keys.Validation
LoRA trainable.
funasr-train-dscompletes; saved checkpoint'slora_A/lora_Bare bf16 matching the base dtype; merge folds all 56 pairs cleanly.eval()/train()auto merge/unmerge verified (matches the upstreamlora.Linearconvention used by Paraformer).