Issues

• When I use FSDPtrainer, I met a Bus error

  eric-mitchell/direct-preference-optimization#96 · Filter by author Red-Scarff opened on Jul 14, 2025

• Who can help?When I was training with DPOTrainer.train(), the following error occurred

  eric-mitchell/direct-preference-optimization#94 · Filter by author mmmaxyoung opened on Mar 24, 2025

• May I change the reward function by adding the chosen score and the rejection score inside.

  eric-mitchell/direct-preference-optimization#93 · Filter by author harrysyz99 opened on Dec 18, 2024

• When trying to reproduce the complete example, "NotImplementedError: offload_to_cpu=True and NO_SHARD is not supported yet" is thrown

  eric-mitchell/direct-preference-optimization#91 · Filter by author ZSvedic opened on Nov 12, 2024
  1

• ValueError when using peft on FSDPTrainer

  eric-mitchell/direct-preference-optimization#90 · Filter by author AragornHorse opened on Nov 5, 2024

• In DPO training, I got this ‘train stats after 160768 examples: {'rewards_train/chosen': 'nan', 'rewards_train/rejected': 'nan', 'rewards_train/accuracies': '0', 'rewards_train/margins': 'nan', 'l -ogps_train/rejected': 'nan', 'logps_train/chosen': 'nan', 'loss/train': 'nan', 'examples_per_second': '5.4876', 'grad_norm': 'nan', 'counters/examples': 160768, 'counters/up -dates': 5024}’

  eric-mitchell/direct-preference-optimization#89 · Filter by author Alan-D-Chen opened on Sep 23, 2024
  2

• GPT4 prompt when evaluating DPO

  eric-mitchell/direct-preference-optimization#88 · Filter by author kygguo opened on Sep 5, 2024

• How to gurantee the output.logits.shape[:-1] == labels.shape

  eric-mitchell/direct-preference-optimization#87 · Filter by author foreverhell opened on Aug 13, 2024

• How are evals done on trained models?

  eric-mitchell/direct-preference-optimization#83 · Filter by author lesnikow opened on May 22, 2024

• where is config document of ipo?

  eric-mitchell/direct-preference-optimization#81 · Filter by author 3244we opened on May 7, 2024
  1

• Using Mistral 7B with transformers v4.38.1 on MATH dataset, and facing memory leaks

  eric-mitchell/direct-preference-optimization#80 · Filter by author Jayant1234 opened on May 4, 2024

• Weird logits and model starts degeneration while training DPO

  eric-mitchell/direct-preference-optimization#77 · Filter by author DungNasSa10 opened on Apr 9, 2024
  2

• Was it your intention to recreate wandb tables in iterator?

  eric-mitchell/direct-preference-optimization#76 · Filter by author huskydoge opened on Apr 4, 2024

• The number of training steps in the SHP dataset

  eric-mitchell/direct-preference-optimization#73 · Filter by author bonin147 opened on Mar 16, 2024

• Computing faster lopgs

  eric-mitchell/direct-preference-optimization#72 · Filter by author alexvishnevskiy opened on Mar 9, 2024
  3

• Implementation for Plackett-Luce rank model

  eric-mitchell/direct-preference-optimization#71 · Filter by author rohan598 opened on Mar 4, 2024
  1

• What's the reference policy of Preferred-FT in Figure 2?

  eric-mitchell/direct-preference-optimization#70 · Filter by author zetian1025 opened on Mar 4, 2024

• My Code to Reproduce IMDB

  eric-mitchell/direct-preference-optimization#69 · Filter by author QiyaoWei opened on Feb 26, 2024

• Why does SFT sum the cross-entropy loss within each sequence?

  eric-mitchell/direct-preference-optimization#68 · Filter by author yunjae-won opened on Feb 17, 2024
  3

• Using cross entropy loss to calculate DPO?

  eric-mitchell/direct-preference-optimization#67 · Filter by author zachares opened on Feb 14, 2024
  4

• Unable to Run SFT

  eric-mitchell/direct-preference-optimization#66 · Filter by author Rui-Yuan91 opened on Feb 13, 2024
  4

• Question bout IPO loss vs DPO loss

  eric-mitchell/direct-preference-optimization#64 · Filter by author MoonBlvd opened on Jan 30, 2024
  1

• Reproducing Win Rate inference for TL;DR

  eric-mitchell/direct-preference-optimization#62 · Filter by author jdchang1 opened on Jan 9, 2024
  1

• DPO did not achieve the expected experimental effect

  eric-mitchell/direct-preference-optimization#56 · Filter by author Vance0124 opened on Dec 7, 2023
  2

• How to re-implement the result of IMDB sentiment generation.

  eric-mitchell/direct-preference-optimization#54 · Filter by author junkangwu opened on Nov 14, 2023