Self-Rewarding Language Models (2024)

1 Introduction

Aligning Large Language Models (LLMs)using human preference data can vastly improve the instruction followingperformance of pretrained models (Ouyang etal., 2022; Bai etal., 2022a). The standard approach of Reinforcement Learning from Human Feedback (RLHF)learns a reward model from these human preferences. The reward model is then frozen and used to train the LLM using RL, e.g., via PPO (Schulman etal., 2017). A recent alternative is to avoid training the reward model at all, and directly use human preferences to train the LLM, as in Direct Preference Optimization (DPO; Rafailov etal., 2023).In both cases, the approach is bottlenecked by the size and quality of the human preference data, and in the case of RLHF the quality of the frozen reward model trained from them as well.

In this work, we insteadpropose to train a self-improving reward modelthat, rather than being frozen, is continually updating during LLM alignment, in order to avoid this bottleneck.The key to such an approach is to develop an agent that possesses all the abilities desired during training, rather than separating them out into distinct models such as a reward model and a language model.In the same way that pretraining andmultitasking training of instruction following tasksallow task transfer by training on many tasks at once(Collobert and Weston, 2008; Radford etal., 2019; Ouyang etal., 2022), incorporating the reward model into that same system allows task transfer between the reward modeling task and the instruction following tasks.

We thus introduce Self-Rewarding Language Models, that both (i) act as instruction following models generating responses for given prompts; and (ii) can generate and evaluate new instruction following examples to add to their own training set. We train these models using an Iterative DPO framework similar to that recently introduced in Xu etal. (2023). Starting from a seed model, in each iteration there is a process of Self-Instruction creation whereby candidate responses are generated by the model for newly created prompts, and are then assigned rewards by that same model. The latter is implemented via LLM-as-a-Judge prompting, which can also be seen as an instruction following task. A preference dataset is built from the generated data, and the next iteration of the model is trained via DPO, seeFigure1.

In our experiments, we start with a Llama 2 70B (Touvron etal., 2023) seed model fine-tuned on Open Assistant (Köpf etal., 2023), and then perform the above training scheme.We find that not only does the instruction following performance improve from Self-Rewarding LLM alignment compared to the baseline seed model, but importantlythe reward modeling ability, which is no longer fixed, improves as well.This means that the model during iterative training is able, at a given iteration, to provide a higher quality preference dataset to itself than in the previous iteration.While this effect likely saturates in real-world settings, it provides the intriguing possibility of obtaining reward models (and hence LLMs) that are superior to ones that could have been trained from the original human-authored seed data alone.

2 Self-Rewarding Language Models

These skills are used so that the model can perform self-alignment, i.e., they are the components used to iteratively train itself using AI Feedback (AIF).

Self-instruction creation consists of generating candidate responses and then the model itself judging their quality, i.e., it acts as its own reward model, replacing the need for an external one. This is implemented via the LLM-as-a-Judge mechanism (Zheng etal., 2023b), i.e.,by formulating the evaluation of responses as an instruction following task.This self-created AIF preference data is used as a training set.

Our overall self-alignment procedure is an iterative one, which proceeds by building a series of such models, with the aim that each improves over the last.Importantly, because the model can both improve its generation ability, and act as its own reward model through the same generation mechanism, this means the reward model itself can improve through these iterations, deviating from standard practices where the reward model is fixed (Ouyang etal., 2022).We believe this can increase the ceiling of the potential for self-improvement ofthese learning models going forward, removing a constraining bottleneck.

We describe these steps in more detail below. An overview of the approach is illustrated in Figure 1.

3 Experiments

3.2 Results

3.2.1 Instruction Following Ability

Head to head performance results are provided in Figure3.

See Also

Washington DC’s 500 Most Influential People of 2024

EFT+IFT seed training performs similarly to IFT alone

We find that adding the Evaluation Fine-Tuning (EFT) task to training does not impact instruction following performance compared to using Instruction Fine-Tuning (IFT) data alonewith an almost equal head to head (30.5% wins vs. 30.9% wins).This is a positive result because it means the increased capability of a model to self-reward does not affect its other skills. We can thus use IFT+EFT training as Iteration 1 ($M_{1}$) of our Self-Rewarding model, and then run further iterations.

Iteration 2 ($M_{2}$) improves over Iteration 1 ($M_{1}$) and SFT Baseline

Iteration 2 of Self-Rewarding training ($M_{2}$) provides superior instruction following to Iteration 1 ($M_{1}$) with 55.5% wins for $M_{2}$ compared to only 11.7% for $M_{1}$ in a head to head evaluation.It provides similar gains over the SFT Baseline as well (49.2% wins vs. 14.5% wins). Clearly, there is a large jump in performance from $M_{1}$ to $M_{2}$ by using the preference data AIFT($M_{1}$) provided by the reward model from Iteration 1.

Iteration 3 ($M_{3}$) improves over Iteration 2 ($M_{2}$)

We see a further gain in Iteration 3 over Iteration 2, with 47.7% wins for $M_{3}$ compared to only 12.5% for $M_{2}$ in a head to head evaluation. Similarly, the win rate over the SFT Baseline for $M_{3}$ increases to 62.5% wins vs. 9.8%, i.e., winning more often than the $M_{2}$ model did. Overall, we see large gains from $M_{2}$ to $M_{3}$ through training using the preference data AIFT($M_{2}$) provided by the reward model from Iteration 2.

Self-Rewarding models perform well on AlpacaEval 2 leaderboard

We evaluate our models on the AlpacaEval 2.0 leaderboard format, with results given inTable1. We observe the same findings as in the head-to-head evaluations, that training iterations yield improved win rates, in this case over GPT4-Turbo, from 9.94% in Iteration 1, to 15.38% in Iteration 2, to 20.44% in Iteration 3. Our Iteration 3 model outperforms many existing models in this metric, including Claude 2, Gemini Pro, and GPT4 0613. We show some selected models from the leaderboard in the table. We note that many of those competing models contain either proprietary alignment data (which is typically large, e.g., over 1M annotations in Touvron etal. (2023)) or use targets that are distilled from stronger models. In contrast, our Self-Rewarding model starts from a small set of seed data from Open Assistant, and then generates targets and rewards from the model itself for further iterations of training.

		Alignment Targets
Model	Win Rate	Distilled	Proprietary
Self-Rewarding 70B
Iteration 1 ($M_{1}$)	9.94%
Iteration 2 ($M_{2}$)	15.38%
Iteration 3 ($M_{3}$)	20.44%
Selected models from the leaderboard
GPT-4 0314	22.07%		✓
Mistral Medium	21.86%		✓
Claude 2	17.19%		✓
Gemini Pro	16.85%		✓
GPT-4 0613	15.76%		✓
GPT 3.5 Turbo 0613	14.13%		✓
LLaMA2 Chat 70B	13.87%		✓
Vicuna 33B v1.3	12.71%	✓
Humpback LLaMa2 70B	10.12%
Guanaco 65B	6.86%
Davinci001	2.76%		✓
Alpaca 7B	2.59%	✓

Fine-grained analysis

As described earlier, the overall performance of the model in AlpacaEval improves with each iteration of training. It would be interesting to break down the overall performance improvement to see exactly what type of tasks these improvements come from. Therefore, we cluster the instructions in AlpacaEval test set into different groups based on three perspectives: (1) instruction category (2) instruction complexity (3) expected response length. We achieve this by using GPT-4. The detailed statistical information of the breakdown and the prompting techniques we used for getting this breakdown can be found in AppendixA.6. Results for the instruction category are given in Figure4, and the other two in Appendix Figure11. From the results we can conclude that (i) Self-Rewarding models can substantially improve the win rate in most categories, but there are some tasks for which this approach does not improve, such as mathematics and logical reasoning, indicating that our current training approach mainly allows the models to better utilize their existing knowledge.(ii) Through Self-Rewarding model training, the model’s win rate increases on almost all tasks of different complexity, and especially on slightly more difficult tasks (complexity of 5, 6, 7 out of 10). (iii) The models also show a steady increase in the win rate on tasks with instructions with different expected response lengths.

Data distribution analysis

We perform a t-SNE (Vander Maaten and Hinton, 2008) visualization of the IFT, EFT and AIFT($M_{1}$) data, shown in AppendixA.1. We find good overlap between the IFT and AIFT($M_{1}$) examples, which is desired, while the EFT examples lie in a different part of the embedding space, which can help explain why they would not affect IFT performance.We observe that generations from $M_{1}$ on AlpacaEval have an average length of 1092, for $M_{2}$ they are1552, and for $M_{3}$ they are 2552, so the model is learning to generate longer responses, which we note may be a factor in relative performance.

	Overall	Math, Code	Humanities, Extraction,
	Score	& Reasoning	STEM, Roleplay & Writing
SFT Baseline	6.85	3.93	8.60
$M_{1}$	6.78	3.83	8.55
$M_{2}$	7.01	4.05	8.79
$M_{3}$	7.25	4.17	9.10

	ARC $(\uparrow)$ challenge	HellaSwag $(\uparrow)$	GSM8K $(\uparrow)$	MMLU $(\uparrow)$	NQ $(\uparrow)$
Llama 2	57.40	85.30	56.80	68.90	25.30
SFT Baseline	55.97	85.17	50.72	69.76	34.35
$M_{1}$	57.51	84.99	60.27	69.34	35.48
$M_{2}$	54.51	84.27	59.29	69.31	33.07
$M_{3}$	53.13	83.29	57.70	69.37	31.86

Human evaluation

To examine whether human judgments align with automatic evaluation results, we conduct human evaluations that compare SFT baseline generations with the generations from each iteration of Self-Rewarding training, i.e., models $M_{1}$, $M_{2}$, and $M_{3}$. Specifically, we randomly select 50 instructions from the IFT test set. Each instruction corresponds to three pairs of generations (i.e., baseline vs. $M_{1}$, baseline vs. $M_{2}$, baseline vs. $M_{3}$). For each pair of generations, we assign them to three different annotators (blind evaluation performed by the authors) to make a pairwise judgment, and take a majority vote to decide which generation is better. The human evaluation results are shown in Figure5. We find that Self-Rewarding models from later iterations show a larger advantage over the SFT baseline model, which is consistent with GPT-4’s judgments, and demonstrates the effectiveness of our iterative training procedure.

MT-Bench performance further validates these results

We report performance on MT-Bench in Table2 for the SFT baseline and iterations of the Self-Rewarding model. We again see improvements across the iterations of training from $M_{1}$ to $M_{3}$, from 6.78 (out of 10) up to 7.25, with larger relative gains in the humanities, STEM, roleplay, writing and extraction categories, and smaller gains in the math, code and reasoning categories.We expect that the latter is due to the seed prompts we use from Open Assistant tending to underemphasize the reasoning-based tasks.We note also that these improvements are in spite of our method using and constructing prompts that only involve a single turn, given the MT-Bench benchmark itself is a multi-turn evaluation.

Self-rewarding models did not lose ability on NLP Benchmarks

As shown in Table3, the performance of most NLP benchmark tasks evaluated are roughly similar to the baselines, with further detailed results on more datasets given in Appendix Table9 that follow the same pattern.We hypothesize that given that our training data (seed data and synthetically generated data) are based on the Open Assistant prompts which may not be especially relevant to skills needed in the Table3 tasks, it is expected that the task performance stays roughly similar, or may even drop.For example, in InstructGPT training (Ouyang etal., 2022)they found that “during RLHF fine-tuning, we observe performance regressions comparedto GPT-3 on certain public NLP datasets” which they refer to as an “alignment tax.”A clear future direction is to extend the self-rewarding paradigm to these types of tasks, by relying not only on seed prompts from Open Assistant, but also on seed prompts found in a larger variety of datasets.

3.2.2 Reward Modeling Ability

Reward modeling evaluation results are provided inTable4.

EFT augmentation improves over SFT baseline

Firstly, we find that adding Evaluation Fine-Tuning (EFT) data into training, which gives examples to the model of how to act as an LLM-as-a-Judge, naturally improves its performance compared to training with Instruction Fine-Tuning (IFT) data alone. IFT data covers a wide range of general instruction tasks, and so does endow the SFT Baseline with the ability to evaluate responses; however, EFT data gives more examples of this specific task. We find improvements across all five metrics measured when using IFT+EFT vs. IFT alone, e.g.,the pairwise accuracy agreement with humans increases from 65.1% to 78.7%.

		Self-Rewarding Models
Model	SFT Baseline	Iter 1 ($M_{1}$)	Iter 2 ($M_{2}$)	Iter 3 ($M_{3}$)
Training data	IFT	IFT+EFT	IFT+EFT	IFT+EFT+AIFT($M_{1}$)
			+AIFT($M_{1}$)	+AIFT($M_{2}$)
Pairwise acc. $(\uparrow)$	65.1%	78.7%	80.4%	81.7%
5-best % $(\uparrow)$	39.6%	41.5%	44.3%	43.2%
Exact Match % $(\uparrow)$	10.1%	13.1%	14.3%	14.3%
Spearman corr. $(\uparrow)$	0.253	0.279	0.331	0.349
Kendall $\tau$ corr. $(\uparrow)$	0.233	0.253	0.315	0.324

Reward Modeling ability improves with Self-Training

We find that performing a round of self-reward training improves the ability of the model at providing self-rewards for the next iteration, in addition to its improved instruction following ability. Model $M_{2}$ (Iteration 2) is trained using the reward model from $M_{1}$ (Iteration 1), but provides improved performance on all five metrics compared to $M_{1}$.For example, pairwise accuracy improves from 78.7% to 80.4%.Iteration 3 ($M_{3}$) improves several of these metrics further compared to $M_{2}$, for example pairwise accuracy increases from 80.4% to 81.7%.This performance gain is achieved despite there being no additionalEFT data provided, and the examples created during the Self-Instruction creation loop do not tend to look like LLM-as-a-Judge training examples. We hypothesize that because the model is becoming better at general instruction following, it nevertheless also improves atthe LLM-as-a-Judge task.

Importance of the LLM-as-a-Judge Prompt

In these experiments we used the LLM-as-a-Judge prompt format shown in Figure2.In preliminary experiments we also tried various other prompts to decide the most effective one to use. For example, we tried the prompt proposed in Li etal. (2024) which also proposes a 5-point scale, but describes the options as multiple choice in a range of quality buckets, see Appendix Figure7.In contrast, our prompt describes the points as additive, covering various aspects of quality. We find a large difference between these two prompts when using the SFT Baseline, e.g. 65.1% pairwise accuracy for ours, and only 26.6% pairwise accuracy for theirs. See AppendixA.2 for further details.

4 Related Work

Automatically improving or self-correcting large language models is becominga major focus of research. A recent survey from Pan etal. (2023)attempts to summarize the topic. However, this is a rapidly moving area, andthere are already promising new works not covered there.

5 Conclusion

We have introduced Self-Rewarding Language Models, models capable of self-alignment via judging and training on their own generations. The method learns in an iterative manner, where in each iteration the model creates its own preference-based instruction training data. This is done by assigning rewards to its own generations via LLM-as-a-Judge prompting, and using Iterative DPO to train on the preferences. We showed that this training both improves the instruction following capability of the model, as well as its reward-modeling ability across the iterations. While there are many avenues left unexplored, we believe this is exciting becausethis means the model is better able to assign rewards in future iterations for improving instruction following – a kind of virtuous circle.While this improvement likely saturates in realistic scenarios,it still allows for the possibility of continual improvement beyond the human preferences that are typically used to build reward models and instruction following models today.

6 Limitations

While we have obtained promising experimental results, we currently consider them preliminary because there are many avenues yet to explore, among them the topics of further evaluation, including safety evaluation, and understanding the limits of iterative training.

We showed that the iterations of training improve both instruction following and reward modeling ability, but only ran three iterations in a single setting. A clear line of further research is to understand the “scaling laws” of this effect both for more iterations, and with different language models with more or less capabilities in different settings.

We observed an increase in length in model generations, and there is a known correlation between length and estimated quality, which is a topic that should be understood more deeply in general, and in our results in particular as well. It would also be good to understand if so-called “reward-hacking” can happen within our framework, and in what circ*mstances. As we are using both a language model as the training reward, and a language model for final evaluation (GPT-4) in some of our benchmarks, even if they are different models, this may require a deeper analysis than we have provided. While the human evaluation we conducted did provide validation of the automatic results, further study could bring more insights.

Another clear further avenue of study is to conduct safety evaluations – and to explore safety training within our framework.Reward models have been built exclusively for safety in existing systems (Touvron etal., 2023), and a promising avenue here would be to use the LLM-as-a-Judge procedure to evaluate for safety specifically in our self-rewarding training process.Given that we have shown that reward modeling ability improves over training iterations, this could mean that the safety of the model could potentially improve over time as well, with later iterations being able to catch and mitigate more challenging safety situations that earlier iterations cannot.

References

• Achiam etal. (2023)Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, FlorenciaLeoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, etal.GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023.
• Adolphs etal. (2023)Leonard Adolphs, Tianyu Gao, Jing Xu, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston.The CRINGE loss: Learning what language not to model.In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8854–8874, Toronto, Canada, July 2023. Association for Computational Linguistics.doi: 10.18653/v1/2023.acl-long.493.URL https://aclanthology.org/2023.acl-long.493.
• Anthropic (2023)Anthropic.Claude 2.https://www.anthropic.com/index/claude-2, 2023.
• Bai etal. (2022a)Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, etal.Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022a.
• Bai etal. (2022b)Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, etal.Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073, 2022b.
• Bai etal. (2023)Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou.Benchmarking foundation models with language-model-as-an-examiner.In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.URL https://openreview.net/forum?id=IiRHQ7gvnq.
• Bisk etal. (2020)Yonatan Bisk, Rowan Zellers, RonanLe Bras, Jianfeng Gao, and Yejin Choi.Piqa: Reasoning about physical commonsense in natural language.In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
• Chen etal. (2024a)Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, etal.AlpaGasus: Training a better alpaca with fewer data.In The Twelfth International Conference on Learning Representations, 2024a.URL https://openreview.net/forum?id=FdVXgSJhvz.
• Chen etal. (2024b)Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu.Self-play fine-tuning converts weak language models to strong language models.arXiv preprint arXiv:2401.01335, 2024b.
• Clark etal. (2018)Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.Think you have solved question answering? Try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018.
• Cobbe etal. (2021)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021.
• Collobert and Weston (2008)Ronan Collobert and Jason Weston.A unified architecture for natural language processing: Deep neural networks with multitask learning.In Proceedings of the 25th International Conference on Machine Learning, pages 160–167, 2008.
• Dubois etal. (2023)Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and TatsunoriB Hashimoto.Alpacafarm: A simulation framework for methods that learn from human feedback.arXiv preprint arXiv:2305.14387, 2023.
• Fernandes etal. (2023)Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat.The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation.In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation, pages 1066–1083, Singapore, December 2023. Association for Computational Linguistics.doi: 10.18653/v1/2023.wmt-1.100.URL https://aclanthology.org/2023.wmt-1.100.
• Gulcehre etal. (2023)Caglar Gulcehre, TomLe Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, etal.Reinforced self-training (rest) for language modeling.arXiv preprint arXiv:2308.08998, 2023.
• Hendrycks etal. (2021)Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.Measuring massive multitask language understanding.In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.URL https://openreview.net/forum?id=d7KBjmI3GmQ.
• Honovich etal. (2023)OrHonovich, Thomas Scialom, Omer Levy, and Timo Schick.Unnatural instructions: Tuning language models with (almost) no human labor.In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428, Toronto, Canada, July 2023. Association for Computational Linguistics.doi: 10.18653/v1/2023.acl-long.806.URL https://aclanthology.org/2023.acl-long.806.
• Kim etal. (2023)Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, etal.Prometheus: Inducing fine-grained evaluation capability in language models.arXiv preprint arXiv:2310.08491, 2023.
• Köpf etal. (2023)Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, NguyenMinh Duc, Oliver Stanley, Richárd Nagyfi, etal.OpenAssistant conversations–democratizing large language model alignment.arXiv preprint arXiv:2304.07327, 2023.
• Kwiatkowski etal. (2019)Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, KristinaN. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.Natural questions: a benchmark for question answering research.Transactions of the Association of Computational Linguistics, 2019.
• Lee etal. (2023)Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi.RLAIF: Scaling reinforcement learning from human feedback with ai feedback.arXiv preprint arXiv:2309.00267, 2023.
• Li etal. (2024)Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Luke Zettlemoyer, Omer Levy, Jason Weston, and Mike Lewis.Self-alignment with instruction backtranslation.In The Twelfth International Conference on Learning Representations, 2024.URL https://openreview.net/forum?id=1oijHJBRsT.
• Li etal. (2023)Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and TatsunoriB. Hashimoto.Alpacaeval: An automatic evaluator of instruction-following models.https://github.com/tatsu-lab/alpaca_eval, 2023.
• Lin (2004)Chin-Yew Lin.ROUGE: A package for automatic evaluation of summaries.In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.URL https://aclanthology.org/W04-1013.
• Mihaylov etal. (2018)Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.Can a suit of armor conduct electricity? a new dataset for open book question answering.In EMNLP, 2018.
• Ouyang etal. (2022)Long Ouyang, Jeffrey Wu, XuJiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, etal.Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
• Pan etal. (2023)Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and WilliamYang Wang.Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies.arXiv preprint arXiv:2308.03188, 2023.
• Radford etal. (2019)Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, etal.Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019.
• Rafailov etal. (2023)Rafael Rafailov, Archit Sharma, Eric Mitchell, ChristopherD Manning, Stefano Ermon, and Chelsea Finn.Direct preference optimization: Your language model is secretly a reward model.In Thirty-seventh Conference on Neural Information Processing Systems, 2023.URL https://openreview.net/forum?id=HPuSIXJaa9.
• Saha etal. (2023)Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li.Branch-solve-merge improves large language model evaluation and generation.arXiv preprint arXiv:2310.15123, 2023.
• Sap etal. (2019)Maarten Sap, Hannah Rashkin, Derek Chen, RonanLe Bras, and Yejin Choi.Socialiqa: Commonsense reasoning about social interactions.CoRR, abs/1904.09728, 2019.URL http://arxiv.org/abs/1904.09728.
• Schulman etal. (2017)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017.
• Stiennon etal. (2020)Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and PaulF Christiano.Learning to summarize with human feedback.Advances in Neural Information Processing Systems, 33:3008–3021, 2020.
• Taori etal. (2023)Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and TatsunoriB. Hashimoto.Stanford alpaca: An instruction-following llama model.https://github.com/tatsu-lab/stanford_alpaca, 2023.
• Touvron etal. (2023)Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, etal.Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023.
• Vander Maaten and Hinton (2008)Laurens Vander Maaten and Geoffrey Hinton.Visualizing data using t-SNE.Journal of machine learning research, 9(11), 2008.
• Wang etal. (2023)Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, NoahA. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.Self-instruct: Aligning language models with self-generated instructions.In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada, July 2023. Association for Computational Linguistics.doi: 10.18653/v1/2023.acl-long.754.URL https://aclanthology.org/2023.acl-long.754.
• Xiong etal. (2023)Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang.Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf.arXiv preprint arXiv:2312.11456, 2023.
• Xu etal. (2023)Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston.Some things are more cringe than others: Preference optimization with the pairwise cringe loss.arXiv preprint arXiv:2312.16682, 2023.
• Yuan etal. (2023)Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang.RRHF: Rank responses to align language models with human feedback.In Thirty-seventh Conference on Neural Information Processing Systems, 2023.URL https://openreview.net/forum?id=EdIGMCHk4l.
• Zellers etal. (2019)Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.Hellaswag: Can a machine really finish your sentence?In Anna Korhonen, DavidR. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791–4800. Association for Computational Linguistics, 2019.doi: 10.18653/V1/P19-1472.URL https://doi.org/10.18653/v1/p19-1472.
• Zhao etal. (2023)Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and PeterJ Liu.SLiC-HF: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023.
• Zheng etal. (2023a)Chujie Zheng, Pei Ke, Zheng Zhang, and Minlie Huang.Click: Controllable text generation with sequence likelihood contrastive learning.In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 1022–1040, Toronto, Canada, July 2023a. Association for Computational Linguistics.doi: 10.18653/v1/2023.findings-acl.65.URL https://aclanthology.org/2023.findings-acl.65.
• Zheng etal. (2023b)Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, ZiLin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, JosephE. Gonzalez, and Ion Stoica.Judging LLM-as-a-judge with MT-bench and chatbot arena.In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023b.URL https://openreview.net/forum?id=uccHPGDlao.
• Ziegler etal. (2019)DanielM Ziegler, Nisan Stiennon, Jeffrey Wu, TomB Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019.

Appendix A Appendix

A.1 Distributions of IFT, EFT and AIFT data

We have plotted the distribution of instructions for IFT, EFT and AIFT($M_{1}$) data, and the distribution of responses for IFT, EFT and AIFT($M_{1}$) data in Figure6. It is clear that the IFT data and EFT data come from very different distributions while the IFT and AIFT($M_{1}$) data come from similar distributions.

A.3 Self-rewarding Models Using IFT Data Only

To demonstrate the importance of the EFT data, we also trained a series of models starting with the model trained only on the IFT data. The following is the model sequence.

• $M_{0}$

  : Base pretrained LLM with no fine-tuning.

• $M_{1}^{\prime}$

  : Initialized with $M_{0}$, then fine-tuned on the IFT seed data only using SFT.

• $M_{2}^{\prime}$

  : Initialized with $M_{1}^{\prime}$, then trained with AIFT($M_{1}^{\prime}$) data using DPO.

• $M_{3}^{\prime}$

  : Initialized with $M_{2}^{\prime}$, then trained with AIFT($M_{2}^{\prime}$) data using DPO.

Since we did not use EFT data to train the series of models, they were not always able to score the responses according to the format and even when they did, the scores given typically converged to 4. Therefore, even when starting from the same number of generated new prompts, we could only collect a very small number of valid training samples for DPO.In total, we collected 541 pairs to form the AIFT($M_{1}^{\prime}$) dataset used to train $M_{2}^{\prime}$ via DPO, and 429 pairs to form AIFT($M_{2}^{\prime}$) used to train $M_{3}^{\prime}$. The win rates are shown in Figure8. From the figure we can conclude that EFT data helps to get better performance in the same number of iterations and the gap in performance between the model trained with EFT data and the model trained without EFT data widens in the later iterations.

Category	Number	Percentage
Science / Technology / Engineering	134	16.65%
Professional / Business / Marketing	77	9.57%
Social Interaction / Relationships / Human Behavior	68	8.45%
Miscellaneous / Other	61	7.58%
Mathematics / Logical Reasoning	52	6.46%
Cooking / Recipes	48	5.96%
Software Development / Coding / Algorithms	44	5.47%
Travel / Geography / Exploration	41	5.09%
Literature / Writing / Communication	39	4.84%
History / Social Studies	38	4.72%
Entertainment / Media Analysis	34	4.22%
Language Learning / Linguistics	32	3.98%
Music / Audio / Arts	30	3.73%
DIY Projects / Hobbies	24	2.98%
Technology / Gadgets / Consumer Products	20	2.48%
Gaming / Game Development	18	2.24%
Exercise / Health / Wellness	16	1.99%
Philosophy / Ethics / Ideology	15	1.86%
Sports / Athletics / Physical Activity	12	1.49%
Strategy / Problem-Solving / Critical Thinking	2	0.24%

Complexity	Number	Percentage
3	238	29.57%
2	206	25.59%
4	122	15.16%
6	79	9.81%
5	68	8.45%
7	41	5.09%
1	34	4.22%
8	14	1.74%
9	3	0.37%

Expected Length	Number	Percentage
1-3 sentences	361	44.84%
1 paragraph	269	33.42%
1 sentence	143	17.76%
2 paragraphs	31	3.85%
3 or more paragraphs	1	0.13%