ParaGPT, a new dataset for paraphrase generation, has been introduced in the “Expert Systems, EarlyView” article titled “Comparative analysis of paraphrasing performance of ChatGPT, GPT‐3, and T5 language models using a new ChatGPT generated dataset: ParaGPT.” This dataset, comprising 81,000 machine-generated sentence pairs, seeks to enhance natural language processing (NLP) by ensuring semantic similarity while introducing syntactic and lexical diversity. The creation of such a dataset is significant as it addresses the shortage of high-quality paraphrase datasets, particularly those generated by machine learning models.
Dataset Composition
The ParaGPT dataset features 27,000 reference sentences generated by ChatGPT, along with 81,000 paraphrases produced using three large language models (LLMs): ChatGPT, GPT-3, and T5. The reference sentences span a wide array of topics and structures, providing diverse inputs that enable comprehensive model evaluation. The primary goal is to generate well-formed and coherent paraphrases that maintain meaningful connections to the original sentences.
Evaluation Metrics
Various automatic evaluation metrics were employed to assess the quality of the generated paraphrases. These metrics highlight ChatGPT’s notable performance, particularly in preserving semantic similarity. High semantic similarity scores indicate that the paraphrased sentences closely match the original content’s meaning. On the contrary, ChatGPT exhibited relatively lower syntactic diversity scores, reflecting a broader range of sentence structures in the paraphrased outputs.
A comparative analysis of the three LLMs—ChatGPT, GPT-3, and T5—revealed distinct strengths and weaknesses in their paraphrase generation capabilities. ChatGPT’s higher semantic similarity scores suggest it excels in preserving the original sentence’s meaning, while its syntactic diversity scores indicate a greater variety of sentence structures in its paraphrases. These insights are invaluable for researchers focusing on NLP tasks such as paraphrasing, text simplification, and text generation. The dataset has been made publicly accessible, marking it as the first paraphrase dataset generated using ChatGPT.
Examining past publications on paraphrase generation, earlier datasets have often faced limitations in diversity and quality due to the constraints of earlier models and less comprehensive reference sentences. Previous datasets primarily relied on human-generated paraphrases, which, while high in quality, lacked the scalability offered by machine-generated paraphrases. The introduction of ParaGPT addresses these limitations by leveraging advanced LLMs to generate a vast and diverse set of paraphrases.
Additionally, earlier research often emphasized syntactic transformations without fully considering semantic integrity, leading to paraphrases that, while structurally varied, sometimes drifted from the original meaning. ParaGPT’s balanced approach, emphasizing both semantic similarity and syntactic diversity, represents an advancement in creating paraphrase datasets. This balance is pivotal for developing NLP applications that require nuanced language understanding and generation.
ParaGPT’s public availability provides a valuable resource for the research community. This dataset not only facilitates the development of better paraphrasing models but also offers a benchmark for comparing different LLMs’ performance. Researchers can leverage ParaGPT to fine-tune models for specific applications, enhancing the quality and coherence of generated text in various NLP contexts. As the first dataset of its kind created using ChatGPT, ParaGPT sets a new standard for future paraphrase generation research, potentially leading to significant advancements in the field.
