Naturally Occurring Feedback Is Common, Extractable and Useful

Shachar Don-Yahiya et al.

Audio by Paper2Audio.

Abstract

Human feedback data is a critical component in developing language models. However, collecting this feedback is costly and ultimately not scalable. Inspired by the way human interlocutors provide spontaneous unsolicited feedback to each other, we propose to extract feedback that users naturally include when interacting with chat models. We manually annotated conversations to confirm the presence of naturally occurring feedback in a standard corpus, finding that as much as ∼30% of the chats include explicit feedback. Comparing to older datasets, we find that naturally occurring feedback is more prevalent in recent conversation datasets, suggesting that more than ever, naturally occurring feedback can serve as a valuable resource for feedback data. We propose a method for automatically extracting this feedback, and apply it to over 1M conversations to obtain hundreds of thousands of feedback samples. The extracted feedback shows promise: training with it improves over baseline models and enhances model alignment to human preferences.

1 Introduction

Human feedback is a valuable resource for large language model (LLM) development. The current standard model training process includes a pretraining phase, followed by an alignment phase, where the model is usually fine-tuned and trained with reinforcement learning on human preference data. The more data at hand, the better the model. However, collecting such data usually requires costly human labor, limiting its scalability.

Humans nevertheless do not need commentators to know that their conversation partner is satisfied. Rather, they infer it from the communication itself. We suggest employing a similar rationale with LLMs, and consider conversations as a source for natural human feedback (see Fig. 1) . As naturalistic feedback comes in diverse forms and even implicit forms (e.g., the user continues to the next question feeling satisfied/displeased with the model's response), in this work we focus on explicit cues, such as when the user directly refers to the quality of the model's response (e.g., "thank you!" , or "that's wrong" ) or rephrases and asks the same question again (cf. §3.1) .

Figure 1 summary: The figure illustrates an example of naturally occurring feedback within a conversation between a user and a chatbot. The user initially asks the chatbot for assistance with plotting numbers on a line plot using the Seaborn library. After receiving an initial response, the user seeks clarification on how to plot the actual numbers above the line. Following a subsequent response, the user expresses satisfaction, indicating that the solution works, and then proceeds to ask another question. The user's responses contain feedback, such as asking for clarification and expressing positive feedback.

One advantage of this approach is that this form of feedback is potentially closer to the feedback given by two human interlocutors, possibly containing genuine information for better alignment.

With the introduction of general assistant models like ChatGPT and OpenAsistant, human-model interactions have become very prominent, not only among machine learning experts but also among the general public. Thus, huge amounts of conversation data are potentially available, and those can be viewed as potential feedback data.

We manually annotate and show that naturally occurring feedback is indeed prevalent in conversation data, finding 101 feedback cases within the 300 conversations we examined. Furthermore, we find that naturally occurring feedback is more common in recently collected data than in older data, possibly due to users raising their expectations and being able to conduct a more "humanlike" conversation with the model. This further underscores the importance of ever-growing data resources, over static datasets. Models keep improving and therefore the data used to align them should evolve too.

We introduce a method to automatically extract the naturally occurring feedback from humanmodel interactions ( §4) . We validate our method, both quantitatively and qualitatively, finding that it correctly extracts the feedback to a reasonable degree. We use our extraction method to obtain over 170k feedback samples from 1M non-annotated conversations. We release it as a dataset ( §4.3) . 1 To demonstrate the usefulness of the data, we use it to train a model to better align with human preferences. The resulting model demonstrates superior performance, outperforming the pretrained model in up to 79% of the test cases ( §5) .

2 Background

To compile a preference dataset, human annotators are asked to rank/score the generated responses of LLMs at the time of the interaction, or in retrospect. To save this costly human effort, sometimes other models are doing the ranking at the expense of introducing noise and biases. For example, it was shown that LLMs tend to prefer longer responses regardless of their quality. Although automatically extracted, naturally occurring feedback differs from model as a judge methods as it is anchored in the human response, and therefore is less prone to "hallucinations" and biases, and easier to explain and verify.

Another line of work collects data samples online during the interaction, by eliciting free-text feedback from the user. This feedback is then used in various ways for training. Hancock et al. suggested estimating user satisfaction and only if it is low, to elicit feedback from the users.

We focus on naturally occurring feedback, i.e., spontaneous unsolicited feedback. When two humans talk, they do not score each other's responses nor explicitly ask for feedback (at least not often). Rather, the interlocutors actively signal their understanding and agreement through the use of verbal and visual responses, such as "hmm", "yeah" or facial expressions, head nods, etc.

We show that also in a human-model textual conversation, such feedback signals exist. Finding them, ideally automatically, will allow us to extract freely annotated training examples. Extracting feedback from an endless stream of conversations has the potential to grow unboundedly, becoming a valuable complementary to other human feedback resource.

3 The Discovery of Natural Feedback

We begin by defining a taxonomy for naturally occurring feedback. We then manually annotate conversations to account for the statistics of such feedback types in conversations.

Throughout our discussion when we consider feedback we refer to (a part of) a human response that refers to (a part of) the last model's response.

3.1 Feedback Taxonomy

We define the following categories, split into four negative feedback categories and one positive: 1. Repeat or Rephrase (rephrase): The user repeats or rephrases their last response, explaining again what they wants. 2. Make Aware with Correction (aware + correct): The user points to the model that it was wrong, and provides information regarding the error/how to fix it. E.g., No, I wanted... 3. Make Aware Without Correction (aware correct): The user points to the model that it was wrong, without providing any additional information. E.g., That's incorrect 4. Ask for Clarification (clarify): The user asks for additional resolution that was expected to be in the the previous response, but was missing. E.g., Was it like that? 5. Positive Feedback (positive): The user confirms that the model did a good job, possibly thanking it. E.g., Thank you!

We now turn to motivating this set of categories. The two main design features are simplicity and text-anchoredness, i.e., the feedback should be directly and explicitly derived from the text, without requiring complex subjective interpretation.

Following this line, the feedback type that appears the most explicitly in the text is "Positive Feedback". Although we found it to be less common (see §4.3) , positive feedback can usually be recognized at the vocabulary level. The user thanks the model (e.g., ty ), says it did a good job (e.g., great! ) or that it was right (e.g., that's correct ). The negative feedback cases, on the other hand, are much more diverse. There are vocabulary-level feedback cases ( that's wrong ), but also more semantically complex instances ( actually, I was asking about... ). Thus, using the feedback patterns from Citation 2023, we break the negative feedback cases into finer categories to avoid too general a definition. Also, more detailed categories provide additional information that can be used later for better training/inference.

We found the "Ask for Clarification" category to be somewhat in the middle in terms of sentiment and feedback nature between the Positive Feedback and the rest of the negative categories. The user asks for more information or confirmation, indicating that the model's response was in the right direction, so not entirely wrong, but still provides some subtle feedback. This category is very common (see §4) , and we expect these cases to be even more frequent as models improve.

Another distinction we found useful is between "Make Aware with Correction" and "Make Aware without Correction". The first holds clear potential for training/inference, as the user provides information regarding the required fix. The latter is less useful, but still can be used as a strictly negative example (in contrast to Ask for Clarification). "Repeat or Rephrase" is unique compared to the other feedback forms as it does not leverage the model's ability to process multi-turn interactions. Instead, the user ignores the previous response and rephrases again what they want, as if it was the beginning of the conversation. Assuming that the following model's response would be better, the two one-turn user-model interactions can be used as a preference pair for training. However, it is important to note that the context of the full conversation is crucial to recognize this feedback form.

In their taxonomy, Petrak et al. also have an "Ignore and Continue" category, where the user ignores an error. We leave it out as it does not contain feedback, but rather implies that none was left despite an error. It is only meaningful when accompanied by an annotated error in the previous model response, which we do not have in our setting.

3.2 Natural Feedback Commonly Occurs

To get an initial impression of the distribution of categories in this taxonomy, one of the authors manually annotated the first 300 multi-turn conversations from the LMSYS-Chat-1M dataset (see Fig. 2). After filtering out non-English conversations and offensive/unsettling conversations, we were left with 223 conversations. We find 77 conversations with a total of 101 feedback cases: 37 Repeat or Rephrase, 18 Make Aware with Correction, 13 Make Aware without Correction, 24 Ask for Clarification and 9 Positive Feedback (see Fig. 2). The fact that ∼ 30% of conversations include feedback is an encouraging result. As the percentage is so high it is likely that simple methods would already suffice to extract notable amounts of feedback data.

Figure 2 summary: The figure is a bar chart. It shows the distribution of different feedback categories in a dataset of conversations. The categories include "Repeat or Rephrase", "Make Aware with Correction", "Make Aware without Correction", "Ask for Clarification", and "Positive Feedback". The "Repeat or Rephrase" category appears more frequently than the others, followed by "Ask for Clarification". "Positive Feedback" is the least frequent category.

To validate our manual annotation, we ask an in-house annotator to re-annotate the first 100 conversations, of which 68 pass the filtering. We get a Cohen's kappa of 0.65 for the binary task of feedback recognition (see §4.2) . Of the feedback cases that both annotators agreed upon, they also agreed on the category in 0.79 of the cases.

3.3 Prevalence Increases with Newer Models

In the previous section we saw that current conversations contain a decent amount of feedback cases. Here, by comparing to older datasets, we try to asses what to expect in the future.

The state-of-the-art of LLMs advances rapidly. In the interim, as models get better, users expect more; Users use the models for new scenarios that were not possible before and do so in a more natural way (except in extreme cases). We expect that with more fluent and diverse conversations, we will see more feedback.

We measure that empirically by comparing our annotation of current models to the annotation efforts of earlier models. Out of the six datasets that were annotated by Petrak et al., only the Self-Feeding Chatbot dataset is both human-model and open domain, and thus comparable. The Self-Feeding dataset was created in 2019, and so is the model that was used to generate it. Only 11 feedback instances were found within a random sample of 100 conversations. This is less than half the feedback frequency found in the newer LMSYS-Chat-1M dataset (omitting the positive feedback category as it was introduced by us). We note that there are 48 annotated errors in the 100 Self-Feeding dataset sample, and hence it is unlikely that it was a lack of errors that caused the users to give less feedback.

Our findings suggest that more than ever, naturally occurring feedback can serve as a valuable resource for feedback data. We believe in the future not only would models be better, continuing the above trend, but natural feedback itself may become a known resource, one which users expect the models to use (see §8) .

4 Automatically Extracting Feedback

Given that natural feedback is already present in current ongoing human-model conversations, we propose a method to automatically extract it.

Based on the five feedback categories ( §3.1) , we instruct an LLM to recognize spans – part of the human responses that contain feedback in a given conversation and classify them. We then use a Python script to parse the generated response and extract all the feedback instances. We discuss the implementation details next. Repeat or Rephrase (UR1) - The user repeats or rephrases their concern, e.g., Actually, I wanted… Make Aware with Correction (UR2) - The user makes the system aware of the error and provides information to address what is missing or wrong in its utterance, e.g., No. I wanted you to… Make Aware without Correction (UR3) - The user makes the system aware of the error without providing additional information, e.g., You're wrong. Ask for Clarification (UR4) - The user asks for clarification, e.g., Are you sure? Is it really that… Positive Feedback (UR5) - The user confirms that the assistant did a good job by directly saying so or thanking it, e.g., Thank you There are five different patterns in user responses subsequent to errors in assistant utterances: Given these guidelines, please recognize such user responses in the following dialogue. Please use the format: { "User Response Pattern": [Insert User Response Pattern], "User Response Text": [Insert User Response Text] } If there is no feedback, use the following format: { "User Response Pattern": "No Feedback", "User Response Text": "" }

4.1 Extraction Implementation Details

Data. We use the LMSYS-Chat-1M dataset, a collection of real-world conversations with 25 state-of-the-art LLMs. We select this dataset for its size and variety of models and conversation topics. We filter out conversations with less than two turns, as there is no human feedback in a one turn conversation (one user query followed by one model response).

Model. We use Mixtral-8x7B-Instruct-v0.1 with 4-bit quantization (see App. §A). During development, we also experimented with Yi-34B-Chat and GPT-3.5, but found that Mixtral surpasses them. Prompt. After experimenting with a couple of versions, we found the prompt in App. 4 to perform best. One key point is asking the model to provide its output in JSON format, to ease the parsing. Using few-shot examples seems to confuse the model, probably due to the conversations length and the difficulty to separate different conversations. Parsing. If the generated text contains the prompt, we delete the prompt. We then extract all JSON objects and confirm they contain the "User Response Pattern" and "User Response Text" fields. For each of the JSON objects we verify that the "User Response Text" is indeed contained in one of the user responses and that the category is valid (one of the 5 possibilities). If any of these do not hold, we discard the example.

4.2 Extraction Evaluation

To evaluate model performance in correctly identifying and classifying the naturally occurring feedback, we use the 300 manually annotated conversations from §3.2. We analyze the two abilities separately; extraction of feedback span and classifying it into the correct category. A feedback span is considered correct if it is a sub-string of a manually annotated feedback span and is at least half as long. We report both precision and recall. We define the span-precision as the percentage of the correctly identified feedbacks out of the total number of identified feedbacks. Correspondingly, the span-recall is the ratio of those correctly identified feedbacks, but divided by the total number of manually annotated feedbacks. We define the category precision/recall as the number of feedbacks that were both identified and classified correctly to the right category, divided by the total number of identified/manually annotated feedbacks respectively.

While our manual annotation found 101 feedback cases, our automatic method found 134 feedback cases, out of which 58 are correct. Employing bootstrap with 1000 repetitions, this results to 0.43 plus or minus 0.05 precision and 0.58 plus or minus 0.06 recall. Taking the categories into account, we get 0.28 plus or minus 0.04 precision and 0.38 plus or minus 0.06 recall. Fig 4 presents the confusion matrix for the categories. We can see that "Repeat and Rephrase" and "Make Aware with Correction" are the most frequent categories for both manual annotation and the automatic method, and that the automatic method did not predict the "Make Aware without Correction" category at all. If we reduce it to binary categories, i.e., positive slash negative, we can see that there is no confusion between the positive category and the negative categories.

Figure 4 summary: The figure is a confusion matrix. It shows the performance of an automatic method for extracting and classifying feedback. The automatic method identifies a portion of the manually annotated feedback cases, with a smaller number correctly classified into categories. There is no confusion between positive feedback and other categories, indicating the method's ability to distinguish positive feedback from other types of feedback.

Examining the false positives, we see that many of them are debatable. For example, the user started by telling the model "I am interested to know how you work" . The model then responded with "I am an AI language model that uses machine learning algorithms to understand and generate human-like text. I am trained on a large dataset of text... If you have any specific questions about how I work, feel free to ask!" . The user then asked "Can I make you on my own computer?" and our extraction method marked this text as a "Ask for Clarification" feedback case. We did not annotate this response as feedback as we considered it to be a new request and not a clarification of the previous one. However, we do see why it could be pointing to missing information in the original question.

Given that also the false positives seem to encode a relevant signal, we hypothesize that although not perfectly retrieved, the extracted feedback would be beneficial for training. We validate it in §5.

4.3 The Natural Feedback Dataset

Using the manual annotation as a test set, and our extraction method to acquire more feedback, we create a large Natural Feedback Dataset. We run the described extraction method on all 1000000 conversations of the LMSYS-Chat-1000000 dataset. After filtering out two turn conversations (see §4.1) , we are left with 334, 319 conversations. We apply our method and end up with 173, 859 feedback examples from 115, 312 different conversations. See Fig. 5 for the category distribution. In terms of positive/negative examples, we have about 15 times more negative examples, similar to the ratio we had in the manual annotation ( §3.2) . Note that this ratio is not surprising, as correcting a model is potentially beneficial for the user (helping the model to help me), while thanking it is less practical. For more statistics and running the extraction method on another conversation dataset, see App. §B.

Figure 5 summary: The figure is a bar chart. It shows the distribution of automatically extracted feedback across different categories. The categories include rephrase, aware+correct, aware-correct, clarify, and positive. The chart indicates that "Ask For Clarification" is the most common feedback category, followed by "Repeat or Rephrase". "Make Aware without Correction" and "Positive Feedback" are the least common categories.

5 Validating the Feedback Usefulness

To demonstrate the usefulness of the extracted data, we use it to train LLMs and show the improvement.

Our data contains both positive and negative examples. We start by using the positive examples only, to finetune the models. We then present some initial results for preference training, with both positive and negative examples.

5.1 Training Details

We randomly split the positive examples to 80/20% for train/val data, remaining with 8448 training examples. We use three models in increasing sizes: EleutherAI/pythia-1.4b , pythia-2.8b , and mistralai/Mistral-7B-v0.1. For more details see App. §1.

5.2 Model Performance Evaluation

To measure the improvement of the models given the new data, we use the validation split of the OpenAssistant dataset. We generate the last response with both our trained models and the corresponding pretrained models. See App. §A. Human Evaluation. We perform human evaluation of the model outputs to acquire a reliable evaluation. An in-house human annotator was asked to rate not consistently ordered pairs of 100 model responses for each of the models, without knowing what model created which response (the pretrained baseline or the finetuned version). Our trained mod- els won 69% / 81.5% / 77% over their corresponding pretrained versions.

Evaluation by Open Models. In addition to manual evaluation, we perform automatic evaluation, which allows more flexibility in the analysis. To prioritize replicable science, we explore the usability of open models as evaluators in our scenario. The RewardBench leaderboard evaluates the capabilities of models in the task of rating model responses, and its top models outperform some closed models that are frequently used as judges. Based on the leaderboard, we take openbmb/Eurus-RM-7b and sfairXC/FsfairX-LLaMA3-RM-v0.1 (leaders of the leaderboard, when conducting these experiments), and compare the pretrained to the finetuned models responses. However, we were disappointed to find that these models do a poor job comparing the outputs of the smaller models. Eurus-RM-7b reported 31% and 38% wins for the 1.4B and 2.8B models, and FsfairX-LLaMA3-RM-v0.1 reported 48% and 60%. We assume this is due to the distribution of the data they were trained on, which only represents stronger models. The results for the 7B model on the other hand are comparable to those we got in the human and GPT evaluation. Eurus-RM-7b and FsfairX-LLaMA3-RM-v0.1 reported 70% and 72% wins for the trained model respectively.

GPT as a Judge. To complete the picture, we run a GPT-4 as a judge evaluation. We use the RewardBench implementation to instruct GPT-4 to compare response pairs. Our trained models won 65% / 74% / 78% over their corresponding pretrained versions.

Conducting a binominal test on these results, we find that all above reported results are significant with p less than 10 times e to the power of negative 9. We conclude that our automatically extracted training data is indeed beneficial. Training on about 8k positive examples yields a significant improvement for all our tested model sizes, more as the model is larger. See App. Section 5 for the aggregated results table.

5.3 Random Chats Baseline

As an additional baseline, we replace our extracted positive examples with a random sample of chat examples from the LMSYS-Chat-1M dataset of the same size. These examples are not necessarily positive, but they are in a chat format and of relatively well-performing models and therefore might be useful for knowledge distillation nonetheless. We want to test whether training on our extracted data has any advantage over this randomly sampled data. We finetune the 7B model on them, and evaluate their performance. Eurus-RM-7b reports 64% wins, FsfairX-LLaMA3-RMv0.1 reports 68% wins, and GPT-4 reports 75% wins, all outperformed by the model we trained on the extracted positive examples. This strengthens our conclusion that our extracted data is beneficial.

5.4 Preference Training

So far, we have shown promising results for finetuning. While finetuning is a performant way to use feedback, it only trains on positive examples (see §5.1) . To test the benefits of the negative examples we try other training methods.

Experimental Setup. As mentioned in §4.3, there are many more negative examples than positive ones. To balance this, we use only the "Make Aware with Correction" and "Make Aware without Correction" categories, and on top of that we down-sample. We chose these categories as we assume their 'negative' signal is the strongest. For the hyperparameters, see App. §A. The positive and negative examples are not given in pairs that share a prompt and have varying quality degrees. We hence use KTO, which improves over DPO objective, while also handling non-paired preference data.

We run this experiment with the 7B model only, as preference training is not beneficial for smaller models. We start from the previously finetuned model. Results. All tuned models score overwhelmingly better than the pretrained. Eurus-RM-7b reports 74% wins, FsfairX-LLaMA3-RM-v0.1 reports 75% wins, and GPT-4 reports 79% wins. Those are also somewhat better than those we had for the finetuned model (1-3 points improvement). We conclude that our negative data (or at least some of the categories) is indeed useful.

6 Ablation Experiments

We analyze different aspects of our extraction method, including our choice of taxonomy, and other frequently used prompting techniques.

6.1 Taxonomy Effect on the Extraction

Here we examine the effect of the feedback taxonomy on the success of the model in accurately extracting the feedback spans from the conversations. We examine several taxonomy alternatives. We evaluate each by calculating the precision and recall relative to the 300 manually annotated conversations from §3.2.

6.1.1 No Categories

Our taxonomy introduces 5 different feedback categories (see §3.1) . Here we examine whether there is even a need for any taxonomy at all.

We change our prompt such that it will not contain any category definition. We instruct the model to recognize spans of text that are informitive as to the satisfaction of the user, and rate them on a scale of 1-5. See App. §4 for the prompt.

Running the model in this setting, the model found 693(!) text spans, while not even one of them matches the manually annotated feedback examples. Manually looking at a few of them, it seems that the model fails miserably at identifying relevant text spans. For example, it often suggests seeing the original user's requests as an indication of user satisfaction (e.g., "Show me how to imple ment a toy version of a relational database. Begin by writing a toy query planner that convert SQL..." ), which is of course not a valid user feedback as it precedes the model response.

We conclude that an overly general extraction prompt is harder for the model to handle, and that a detailed taxonomy helps automatic extraction.

6.1.2 Limited Categorization

We examine the effect of using fewer feedback categories on the extraction process. This followed the hypothesis that focusing on a smaller set of categories would allow for better precision.

We limit ourselves to the "Repeat and Rephrase" and "Positive Feedback" categories, as we recognize that they are both relatively easier for the model to distinguish from the other categories (see 4) . We instruct the model to extract these two types for feedback only. See App. §4 for the prompt.

For the "Positive Feedback" category, the model manages to achieve 0.5 precision for both span and category precision, i.e., all positive cases that the model found were classified correctly.

For the "Repeat or Rephrase" category, the model managed to achieve 0.43 text-span precision and 0.17 category precision. Given that there are only two possible categories, there is a relatively large gap between the span and category precision. Looking at the extracted examples, we see that the model tends to invent new categories, for example "Asking for Assistance" or "Ask for Examples".

We conclude that focusing on fewer feedback categories is not necessarily easier for the model.

6.2 Confidence Level

We examine the usefulness of asking the model to generate a "confidence level" value, to better filter the extracted feedback samples such that we will get a higher precision score. To do so, in addition to the "User Response Pattern" and "User Response Text" fields, we instruct the model to provide a "Confidence Level (1-5)" field. See App. §D.

Looking at the distribution of confidence scores the model assigned, we find that over 96% of the feedback cases received a 5 score. The other 4% are mostly "No Feedback" or hallucinations that are automatically removed at the parsing stage (see §4.1) . We conclude that this method is ineffective.

7 Related Work

In addition to the LMSYS-Chat-1M dataset which we used due to its size and inclusion of multiple models, there are other recent datasets such as Collective Cognition, PRISM, and WildChat. We use the last to replicate the feedback statistics, see App. 2.

Petrak et al. investigated the types of errors and user responses in 6 different datasets. Although we adopt and modify their feedback taxonomy, we take two steps further. We focus on user responses and extract them automatically, and we show the importance of using up-to-date conversation data ( §3.3), contrary to their conclusion.

We opted for KTO to train a model on our data, but there are many more options for training on nonpositive examples. Ouyang et al. and Shi et al. suggested to create possible corrections for negative examples and train on them. Other methods use pairs of positive and negative examples and train on them both to predict their scores. Peng et al. and Wu et al. showed the possible gain of fine-grained feedback.

We want to emphasize the difference between works that use natural language feedback and ours. In their settings, the user is asked to provide free-text feedback (unlike binary feedback or other close forms). We take another step further and look for feedback that spontaneously occurs in the conversations, without explicitly asking the user to provide it.

Perhaps the most closely related work to ours is Citation 2024. However, our approach differs in several key aspects: focusing on feedback retrieval over binary classification, using extrinsic rather than intrinsic evaluation, and employing an open model instead of a closed one.

Another set of related works are ones creating synthetic data from datasets or augmenting user feedback. We note their similarity in better utilizing human effort for creating data samples for training. Those, however, differ from our work in the problem they address. Such efforts rely on boosting existing training signals, whether found in the human explicit annotations, the model, or both. In contrast, our approach aims to identify signals in a scalable fashion. In fact, the output of our method can be used as their input.

8 Discussion and Future Work

This paper advocates the use of naturally occurring feedback and introduces a method to extract it. We find that naturally occurring feedback is common in human-model chats. We use our method to extract over 170k feedback samples and train models on them, demonstrating their usefulness.

We observed in §3.3 that newer conversation data tends to contain more naturally occurring feedback. Buschmeier and Kopp showed the importance of "listener feedback" (subtle verbal signals, head gestures, and facial expressions) for communication. They showed that this feedback encourages humans interacting with a model to provide more feedback by themselves, and to rate the conversation as more helpful. Therefore, we expect voice assistant data to contain even more feedback.

Another interesting line of future work is the incorporation of feedback into chats in real-time, with interactive reinforcement learning for example, or at least in a manner that would directly affect future user conversations, making giving feedback more beneficial for the user.

Limitations

Although the original LMSYS-Chat-1M dataset contains some non-English conversations, we filter those out during evaluation, as our annotators are not familiar with these languages.

Our automatic extraction method can be improved further to achieve better precision and recall (e.g., with better models, prompts, or a more sophisticated extraction algorithm). We believe that the fact that even the current relatively low precision data managed to achieve good training results underscores the importance and potential of naturally occurring feedback. With the abundance of data, future work might seek better precision or keep high recall depending on their goals. One could train on a lot of low quality data, focus on specific subsets of interest (e.g., a domain) or focus on quality annotation throwing a lot and still no ending up wanting, each requiring different precision-recall tradeoffs.

We used GPT as a judge and other models for evaluating the trained model. This approach is both costly and known to have biases. Therefore, we use it only to complement the human evaluation and the open-models evaluation. We would like to emphasize that our extraction method itself does not use GPT or any proprietary models.

Another limitation of our evaluation method is that it does not allow absolute results, but instead compares the model at hand to another model. This raises the question of which model should be used for the comparison. Here, as in previous work, we selected one reference model for all our experiments, to allow comparison between the different settings. We used the pretrained model, as it was the best fit for the main experiment (pretrain vs. pretrain + finetune on our positive data).

Ethics Statement

This work has been approved by the IRB of our institution. We abide by the terms and conditions of the LMSYS-Chat-1M dataset (see the license here. As mentioned by the LMSYS-Chat-1M dataset authors, the LMSYS-Chat-1M dataset contains unsafe conversations that may be perceived as offensive or unsettling. The provided OpenAI moderation API tag can be used to filter it. We informed our annotators of this and instructed them to skip these conversations.

You have reached the end of the main paper. Additional summarized content follows

Table 1 summary: The table compares different training methods and model sizes based on human evaluation, open model performance, and GPT-4 evaluation. Finetuned models show better human evaluation scores than random chat models. Finetuned with preference training yields higher scores in open model and GPT-4 evaluations compared to other methods.