An Analysis of Research Trends on Language Model Using BERTopic

Woojin Kang, Yumi Kim, Heesop Kim, Jongwook Lee

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Although language models have played a crucial role in various natural language processing tasks, there has been little research that focuses on systematic analysis and review of research topic trends in these models. In this paper, we conducted a comprehensive analysis of 31 years of research trends in the field of language models, using publications from Scopus, an internationally renowned academic database, to identify research topics related to language models. We adopted BERTopic, a state-of-the-art topic modeling technique, on the 13,754 research articles about language models. The research on language models has gradually increased since 1991, and there is a sudden increase in the number of publications with the emergence of BERT and GPT in 2018. We assigned 14 main topics with meaningful keywords clustered by BERTopic model. Among 14 topics, research related to speech recognition, statistical language models, and pre-trained language models demonstrated the most vigorous research fields. Our results demonstrate a more systematic and comprehensive trend in language model research, which is expected to provide an important foundation for future research directions.

Original languageEnglish
Title of host publicationProceedings - 2023 Congress in Computer Science, Computer Engineering, and Applied Computing, CSCE 2023
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages168-172
Number of pages5
ISBN (Electronic)9798350327595
DOIs
StatePublished - 2023
Event2023 Congress in Computer Science, Computer Engineering, and Applied Computing, CSCE 2023 - Las Vegas, United States
Duration: 24 Jul 202327 Jul 2023

Publication series

NameProceedings - 2023 Congress in Computer Science, Computer Engineering, and Applied Computing, CSCE 2023

Conference

Conference2023 Congress in Computer Science, Computer Engineering, and Applied Computing, CSCE 2023
Country/TerritoryUnited States
CityLas Vegas
Period24/07/2327/07/23

Keywords

  • BERT
  • language models
  • re-search trends
  • Short Research Paper
  • topic modeling

Fingerprint

Dive into the research topics of 'An Analysis of Research Trends on Language Model Using BERTopic'. Together they form a unique fingerprint.

Cite this