Skip to content

feat(nano): support LoRA finetuning of the Qwen3 LLM - #3456

Open
KamitobiHaru wants to merge 1 commit into
modelscope:mainfrom
KamitobiHaru:feat/nano-llm-lora
Open

feat(nano): support LoRA finetuning of the Qwen3 LLM#3456
KamitobiHaru wants to merge 1 commit into
modelscope:mainfrom
KamitobiHaru:feat/nano-llm-lora

Conversation

@KamitobiHaru

Copy link
Copy Markdown

Summary

The FunASR-Nano model config already ships llm_conf.use_lora / llm_conf.lora_conf
(r, lora_alpha, lora_dropout, target_modules), but nothing consumed themuse_lora was silently ignored and training fell back to the full-parameter LLM path. This PR wires LoRA up for the Qwen3-0.6B LLM inside FunASRNano.

Changes

  • funasr/models/fun_asr_nano/model.py: when llm_conf.use_lora is true, replace the LLM target nn.Linear layers (default q_proj/v_proj) with lora.Linear adapters (reusing the existing funasr/models/lora/ module used by Paraformer). The pretrained base weight is shared and frozen; trainable lora_A/lora_B are added. Adapter params are created in the base weight's dtype (bf16), so the LoRA path works without relying on
    autocast to reconcile dtypes.
  • examples/.../fun_asr_nano/lora_finetune.sh + docs/finetune.md / finetune_zh.md: LoRA recipe with freeze semantics and deployment (fold) notes.

Design notes

  • Pairs with the existing lora_only / mark_only_lora_as_trainable flow already in
    train.py / train_ds.py:
  • lora_only=true (default in the example) freezes every non-LoRA parameter → pure-LoRA.
  • lora_only=false + unfreeze encoder/adaptor conf keeps them trainable while LoRA-tweaking only the LLM (the recipe used for the medical ASR deliverable).
  • Checkpoints save the full state dict (unchanged base + adapter params), so a LoRA run can be resumed / decoded with the same use_lora=true config. No new save/load code.
  • Inference needs no merge step (forward adds the adapter output); for a standalone deployment checkpoint, fold with W' = W + (alpha/r) * B @ A and drop the lora keys.

Validation

  • Real model build: 56 adapters injected (28 Qwen3 layers × q_proj+v_proj), base frozen,
    LoRA trainable.
  • Optimizer step updates only the 112 LoRA params (0 non-LoRA change).
  • 2-step smoke training via funasr-train-ds completes; saved checkpoint's lora_A/lora_B are bf16 matching the base dtype; merge folds all 56 pairs cleanly.
  • eval()/train() auto merge/unmerge verified (matches the upstream lora.Linear
    convention used by Paraformer).

The FunASR-Nano model config already declares llm_conf.use_lora and
llm_conf.lora_conf (r/lora_alpha/lora_dropout/target_modules), but nothing
consumed them: use_lora was silently ignored and training fell back to the
full-parameter LLM path. Wire it up:

- When llm_conf.use_lora is true, replace the LLM target Linear layers
  (default q_proj/v_proj) with lora.Linear adapters, sharing the frozen base
  weight and adding trainable lora_A/lora_B. The adapter params are created in
  the base weight's dtype (bf16), so the LoRA path does not depend on autocast.
- Pairs with the existing lora_only / mark_only_lora_as_trainable flow in
  train.py / train_ds.py: lora_only=true freezes every non-LoRA parameter for
  pure-LoRA training; lora_only=false + unfrozen encoder/adaptor conf keeps
  them trainable while LoRA-tweaking only the LLM.
- Checkpoints save the full state dict (unchanged base + adapter params), so a
  LoRA run can be resumed or decoded with the same use_lora=true config; for a
  standalone deployment checkpoint, fold with W' = W + alpha/r * B @ A.
- Add lora_finetune.sh and document the recipe in finetune.md / finetune_zh.md.

Validated end-to-end with the real model: 56 adapters injected (28 Qwen3
layers x q_proj+v_proj), optimizer updates only the 112 LoRA params, and a
2-step smoke training run through funasr-train-ds completes and saves a
checkpoint whose lora keys are bf16 and match the base dtype.

Co-Authored-By: Claude <noreply@anthropic.com>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant