
==== Front
Data Brief
Data Brief
Data in Brief
2352-3409
Elsevier

S2352-3409(24)00833-3
10.1016/j.dib.2024.110869
110869
Data Article
The plausibility machine commonsense (PMC) dataset: A massively crowdsourced human-annotated dataset for studying plausibility in large language models
Nananukul Navapat
Shen Ke
Kejriwal Mayank kejriwal@isi.edu
⁎
University of Southern California, 4676 Admiralty Way, Suite 1001 Marina del Rey, CA 90292, USA
⁎ Corresponding author. kejriwal@isi.edu
24 8 2024
12 2024
24 8 2024
57 11086917 7 2024
11 8 2024
19 8 2024
© 2024 The Author(s)
2024
https://creativecommons.org/licenses/by-nc/4.0/ This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
Commonsense reasoning has emerged as a challenging problem in Artificial Intelligence (AI). However, one area of commonsense reasoning that has not received nearly as much attention in the AI research community is plausibility assessment, which focuses on determining the likelihood of commonsense statements. Human-annotated benchmarks are essential for advancing research in this nascent area, as they enable researchers to develop and evaluate AI models effectively. Because plausibility is a subjective concept, it is important to obtain nuanced annotations, rather than a binary label of ‘plausible’ or ‘implausible’. Furthermore, it is also important to obtain multiple human annotations for a given statement, to ensure validity of the labels.

In this data article, we describe the process of re-annotating an existing commonsense plausibility dataset (SemEval-2020 Task 4) using large-scale crowdsourcing on the Amazon Mechanical Turk platform. We obtain 10,000 unique annotations on a corpus of 2000 sentences (five independent annotations per sentence). Based on these labels, each was labelled as plausible, implausible, or ambiguous. Next, we prompted the GPT-3.5 and GPT-4 models developed by OpenAI. Sentences from the human-annotated files were fed into the models using custom prompt templates, and the models’ generated labels were used to determine if they were aligned with those output by humans.

The PMC-Dataset is meant to serve as a rich resource for analysing and comparing human and machine commonsense reasoning capabilities, specifically on plausibility. Researchers can utilise this dataset to train, fine-tune, and evaluate AI models on plausibility. Applications include: determining the likelihood of everyday events, assessing the realism of hypothetical scenarios, and distinguishing between plausible and implausible statements in commonsense text. Ultimately, we intend for the dataset to support ongoing AI research by offering a robust foundation for developing models that are better aligned with human commonsense reasoning.

Keywords

Commonsense benchmark
Large language models
Machine annotation
Human annotation
==== Body
pmcSpecifications TableSubject	Artificial Intelligence	
Specific subject area	Plausibility in commonsense reasoning is a critical research area in the emerging field of human-AI alignment.	
Type of data	Table.	
Data collection	The dataset was sourced from the SemEval-2020 Task 4 plausibility assessment dataset, which categorises statements as either plausible or implausible based on commonsense reasoning. We extend the dataset with two types of annotations: human and machine. For the human annotations, we utilise the Amazon Mechanical Turk (AMT) platform, acquiring five distinct AMT-worker annotations for each statement in SemEval-2020 Task 4. This approach ensures a diverse range of opinions by collecting five independent labels per sentence. For obtaining plausibility predictions from machines, we employ the GPT-3.5 and GPT-4 models developed by OpenAI. A prompt template is designed to elicit responses from these models, and the generated responses were incorporated as machine-generated annotations.	
Data source location	University of Southern California
Information Sciences Institute
4676 Admiralty Way
Marina Del Rey, CA, USA 90,292.	
Data accessibility	Repository name: A Human and Machine-annotated Dataset for Plausibility Commonsense Analysis [1]
Data identification number: 10.5281/zenodo.12742226
Direct URL to data: https://zenodo.org/records/12742226	
Related research article	None	

1 Value of the Data

• This dataset combines human and machine annotations to provide a comprehensive benchmark of thousands of data points for plausibility assessment tasks in commonsense reasoning across various scenarios. By integrating both types of annotations, it offers a unique perspective on how different sources assess plausibility.

• Including individual labels from both human and machine annotations allows for a detailed analysis of commonsense reasoning capabilities. Researchers can study the consistency, variability, and noise in human judgments and compare them with machine-generated annotations, offering insights into the strengths and weaknesses of both approaches.

• Researchers can utilise this dataset to train, fine-tune, and evaluate AI models on various plausibility tasks. These tasks include determining the likelihood of everyday events, assessing the realism of hypothetical scenarios, and distinguishing between plausible and implausible statements. The dataset's diverse scenarios may prove useful in multiple research applications.

2 Background

Plausibility assessment is a critical area of research within the larger research agenda of commonsense reasoning in Artificial Intelligence (AI). It involves determining whether a given scenario is likely or unlikely based on common human experience [2]. This capability is essential for AI systems to make realistic and human-like judgments and decisions. In natural language processing, assessing the plausibility of statements helps ensure that generated text, translations, and dialogues are coherent and contextually appropriate [3]. For example, an AI system capable of plausibility assessment can distinguish between realistic and implausible statements, improving the quality of its responses and interactions.

Human annotators bring significant benefits to plausibility assessment, as commonsense reasoning derives from human understanding of world scenarios [4]. They capture a more realistic and nuanced perspective of plausibility, ensuring that the dataset accurately reflects how people assess plausibility in various contexts. However, datasets of sufficient scale for exploring commonsense plausibility, both from a human behavioural perspective, and for evaluating AI-based commonsense reasoning systems, have been lacking. This data article presents a comprehensive dataset for assessing plausibility in commonsense reasoning and describes a process for adding annotations from both human and machine sources to an existing dataset.

3 Data Description

The proposed PMC dataset is categorised into three spreadsheet files. Each file contains plausibility assessment sentences, ground truth, and both human and machine annotations. The categorization is based on the levels of agreement and disagreement among human annotators. Detailed explanations and the rationale behind this categorization are provided in the subsequent section. Users can choose to utilise any combination of these files, ranging from one to all three. Additionally, we provide the raw responses from both human and machine annotators with ground truth in JavaScript Object Notation (JSON) format. The complete list of files is detailed in Table 1 where researchers can access the PMC-Dataset through the public repository A Human and Machine-annotated Dataset for Plausibility Commonsense Analysis. The dataset is available at https://zenodo.org/records/12742226, and is distributed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.Table 1 Description of PMC-Dataset and Label Data. This table contains description files provided in PMC dataset. The files are available in CSV and JSON formats.

Table 1:File	Description	
AgreeAnnotation.csv
NeutralAnnotation.csv
DisagreeAnnotation.csv	Contain all annotated sentences from both human and GPT models where each sentence has a unique identifier (ID). Each file is categorised based on the definition explained in the experimental section. File structure: {“ID”: string, “annotator1”: int, “annotator2”: int, “annotator3”: int, “annotator4”: int, “annotator5”: int, “sentence”: string, “gpt”: int}	
Response.json	Contain raw responses from GPT-3.5 and GPT-4. File structure: {“ID”: string, “content”: string, “annotation”: int, “job-name”: string, “confidence”: float}	
Response_groundtruth.json	Contain raw responses from GPT-3.5 and GPT-4 with ground truth answer. File structure: {“ID”: string, “content”: string, “annotation”: int, “job-name”: string, “confidence”: float, “gt”: int}	
Request.json	Contain raw response from Amazon Mechanical turkers. File structure: {“ID” string, “content”: string, “annotation”: [“workerId”: string, “annotation”: string]	

4 Experimental Design, Materials and Methods

The process begins with an existing dataset from SemEval-2020 Task 4, which focuses on binary plausibility assessment, and with only one human annotation provided per sentence. This dataset contains 2000 sentences with similar natural language wording, where each pair consists of one plausible statement and one implausible statement. An example pair is:• S1: John put a turkey into the fridge (plausible)

• S2: John put an elephant into the fridge (implausible)

Each sentence in these pairs come with a ground truth label indicating its plausibility. PMC significantly expands this dataset by using human annotators from Amazon Mechanical Turk and further testing LLMs’ abilities to annotate this dataset by prompting the GPT-3.5 and GPT-4 models from OpenAI [5].

4.1 Human annotation

We used five Amazon Mechanical Turkers (AMTs) to annotate the plausibility of each sentence. The annotation process began with uploading each of the 2000 sentences to the Amazon Mechanical Turk platform. In the annotation task, five Turkers were presented with randomised individual sentences and asked to label them as either plausible or implausible. The complete set of collected responses from AMTs is consolidated into Request.json.

Based on the collected annotations, sentences were classified into three files: Agree-annotation.csv, Neutral-annotation.csv, and Disagree-annotation.csv. A sentence is included in Agree-annotation when all five AMTs assigned to that sentence unanimously labelled it as either plausible or implausible. For Neutral-annotation, four out of five AMTs agreed on one label, with one disagreeing. Lastly, in Disagree-annotation, sentences received mixed labels, with two out of five Turkers disagreeing with the majority.

We validated the agreement classification by performing Fleiss' kappa statistical test for all three files in our dataset. The results align with the levels of agreement we classified. Table 2 shows the Fleiss' kappa scores for each classification where the “Agree” category shows perfect agreement among annotators, while the “Neutral” and “Disagree” categories show neutral and negative consensus, highlighting the different levels of difficulty in assessing plausibility.Table 2 This table includes Fleiss’ kappa scores of three classifications (files) in PMC dataset.

Table 2:File	# of human Annotation	Fleiss’ Kappa	Level of Agreement	
AgreeAnnotation.csv	6245	1.0	Perfect agreement	
NeutralAnnotation.csv	2485	0.072	Slight agreement	
DisagreeAnnotation.csv	1270	−0.203	Poor agreement	

This classification helps differentiate sentences with varying consensus levels among human annotators, highlighting the different levels of difficulty in assessing plausibility. Understanding these consensus levels provides valuable context when comparing human labels with results from large language models, enabling a more nuanced evaluation of model performance. We acknowledge the potential for cultural and linguistic biases in the dataset, particularly in the NeutralAnnotation.csv and DisagreeAnnotation.csv files, where differing perspectives among annotators may influence plausibility assessments. However, this variability is valuable for commonsense research, as it reflects the natural diversity in human judgement. These biases actually capture a more realistic range of human plausibility reasoning.

4.2 Machine annotation

Following the human annotation process, we perform machine annotation using advanced AI models, specifically GPT-3.5 and GPT-4 developed by OpenAI. The machine annotation process involves several steps:

First, we develop a Python script to prompt GPT-3.5 and GPT-4 using the OpenAI API. This script automates the process of feeding each sentence from the dataset into the models and retrieving the generated labels. Next, we design custom prompt templates tailored to elicit plausibility assessments from the AI models. These templates are carefully crafted to ensure that the models receive clear and consistent instructions on how to evaluate each sentence. Table 3 shows an example of a prompt template with a response to a real sentence from the dataset.Table 3 This table includes a prompt template with examples of sentences and responses from GPT-3.5 and GPT-4.

Table 3:Prompt template	
Is the following sentence “{Sentence}” plausible or implausible?
Please choose one. You do not need to provide any reasoning	
{Sentence}	Response	
He just won 100 dollars	Plausible	
The cat jumped onto a skyscraper	Implausible	

Next, each sentence from Agree-annotation.csv, Neutral-annotation.csv, and Disagree-annotation.csv is processed by the Python script, which uses the prompt templates to query both GPT-3.5 and GPT-4. The models generate labels for each sentence, categorising them as plausible, implausible, or ambiguous. The raw responses are saved as response.json.

Third, given the potential for inconsistency in the responses from GPT-3.5 and GPT-4, a response post-processing step is implemented. During this step, all responses are standardised: labels such as “plausible,” “yes,” and “1” are converted to a uniform label of “1” (plausible), while labels such as “implausible,” “no” and “0” are converted to “0” (implausible). The post-processed responses are saved and combined with the original ground truth as response_groundtruth.json.

Lastly, we update Agree-annotation.csv, Neutral-annotation.csv, and Disagree-annotation.csv with the new labels from GPT-3.5 and GPT-4. Each file now contains all labels for the 2000 sentences: five labels from human and two labels from the machine annotators per sentence, totalling 14,000 annotations.

We acknowledge that GPT-3.5 and GPT-4 are advanced and continually evolving models, with the machine annotations in this dataset reflecting their performance as of July 2024. While these annotations provide a valuable baseline for evaluating future large language models (LLMs), they do not offer a complete view on their own. We recommend using machine annotations alongside human annotations to obtain a comprehensive set of labels for each plausibility sentence. This combination offers researchers the benefit of both human judgement and GPT models' insights as of July 2024, enabling them to more effectively evaluate their own models or research.

Limitations

Not applicable.

Ethics Statement

All human annotators involved in this work have consented to the release of the produced individual label data. In addition, the annotators have been anonymized in the released datasets.

CRediT authorship contribution statement

Navapat Nananukul: Methodology, Software, Writing – original draft. Ke Shen: Conceptualization, Methodology, Software. Mayank Kejriwal: Conceptualization, Methodology, Writing – review & editing, Supervision, Project administration.

Data Availability

A Human and Machine-annotated Dataset for Plausibility Commonsense Analysis (Original data) (Zenodo)

Acknowledgements

This work was funded under the DARPA Machine Common Sense (MCS) program, award number N660011924033.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
==== Refs
References

1 N. Nananukul, K. Shen, M. Kejriwal. A human and machine-annotated dataset for plausibility commonsense analysis (2024), doi:10.5281/zenodo.12742226
2 C. Wang, S. Liang, Y. Jin, Y. Wang, X. Zhu, Y. Zhang, 2020, SemEval-2020 Task 4: commonsense Validation and Explanation, preprint
3 Sap M. Shwartz V. Bosselut A. Choi Y. Roth D. Commonsense reasoning for natural language processing Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts 2020 27 33
4 Kejriwal M. Santos H. Shen K. Mulvehill A.M. McGuinness D.L. A noise audit of human-labeled benchmarks for machine commonsense reasoning Sci. Rep. 14 1 2024 8609 38615039
5 Achiam J. Adler S. Agarwal S. Ahmad L. Akkaya I. Aleman F.L. Almeida D. Altenschmidt J. Altman S. Anadkat S. Avila R. Gpt-4 technical report 2023
arXiv preprint arXiv:2303.08774, doi:10.48550/arXiv.2303.08774.
