Evaluating LLMs Capabilities Towards Understanding Social Dynamics
Document Type
Article
Publication Date
1-24-2025
Publication Title
Social Networks Analysis and Mining
Volume
15213
Pages
230–244
Publisher Name
Springer Nature Switzerland AG
Publisher Location
Cham, Switzerland
Abstract
Social media discourse involves people from different backgrounds, beliefs, and motives. Thus, often such discourse can devolve into toxic interactions. Generative Models, such as Llama and ChatGPT, have recently exploded in popularity due to their capabilities in zero-shot question-answering. Because these models are increasingly being used to ask questions of social significance, a crucial research question is whether they can understand social media dynamics. This work provides a critical analysis regarding generative LLM’s ability to understand language and dynamics in social contexts, particularly considering cyberbullying and anti-cyberbullying (posts aimed at reducing cyberbullying) interactions. Specifically, we compare and contrast the capabilities of different large language models (LLMs) to understand three key aspects of social dynamics: language, directionality, and the occurrence of bullying/anti-bullying messages. We found that while fine-tuned LLMs exhibit promising results in some social media understanding tasks (understanding directionality), they presented mixed results in others (proper paraphrasing and bullying/anti-bullying detection). We also found that fine-tuning and prompt engineering mechanisms can have positive effects in some tasks. We believe that a understanding of LLM’s capabilities is crucial to design future models that can be effectively used in social applications.
Recommended Citation
Tahir, A., Cheng, L., Sandoval, M., Silva, Y.N., Hall, D.L., & Liu, H. (2025). Evaluating LLMs Capabilities Towards Understanding Social Dynamics. In L.M. Aiello, T. Chakraborty, & S. Gaito (Eds.), Social Networks Analysis and Mining (ASONAM 2024), Lecture Notes in Computer Science, vol. 15213 (pp. 230–244). Springer, Cham. https://doi.org/10.1007/978-3-031-78548-1_18
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 License.
Copyright Statement
© Springer Nature Switzerland AG, 2025
Comments
Author Posting © The Author(s), 2025. This article is posted here by permission of Springer Nature Switzerland AG for personal use and non-commercial redistribution. This article was published open access in Social Networks Analysis and Mining, Vol. 15213 (January 2025), https://doi.org/10.1007/978-3-031-78548-1_18.
Published in the ASONAM 2024 conference proceedings as part of the Lecture Notes in Computer Science (LNCS) series.