On the authenticity of generated OSN Content: Advances in Agent-based LLM prompting for persuasive posts

Simon Münker, M.Sc.

14 February 2024 C429, 4:15–5:15pm

Generated content (bot posts) has become an integral part of social networks, and identifying those posts is more difficult for humans and machines alike. Beyond collecting this content, analyzing and finding patterns is an ongoing challenge in contemporary communications science research. Enabling the generation of personalized posts for multiple networks and languages allows research centered around A/B testing and controlled manipulation with human users.

In this work, we present an Agent-based LLM framework, reachable through a web API to generate content and also to react to given discourses. Our work allows using state-of-the-art models provided by external API (OpenAI, Hugging Face) or local resources (CL-Trier high-performance GPU-Server) with a high degree of configuration options. Our main feature is the selection of different personas describing political ideologies and social media archetypes to align the generated content with the research indent. In combination with configurable platform types, languages, and optional agent histories, we can generate highly individualized synthetic content.

For our first experiments – presented at Etmaal 2024, a communication science conference – we generated and manually evaluated 1000 posts with varying political personas across five topics, three languages, two models, and two platforms. The preliminary results suggest that modern LLMs can generate individualized and authentic content on a large scale. However, when analyzed on a quantitative level, we observe research-backed limitations like a built-in liberal/left-leaning bias in GPT‑3.5 and an American-centrism perspective on underrepresented languages like Dutch.

In the scope of the presentation, we will take the audience through a journey of the rapidly evolving landscape of LLMs and their tendency to omit evaluation with domain experts, introduce the “science” of prompting, and provide an in-depth insight into patterns of generated content. We hope to complement this by discussing the possible application in language and communications science as a straightforward tool for researchers. The curious mind can gaze at the open repository:
https://github.com/smnmnkr/TWON-Agents/tree/master