Investigating and Designing for Behavioral Improvement in Online Community Moderation


Sponsoring Agency
National Science Foundation


This is a study of human implications of online moderation systems that deal with disruptive online behaviors, such as offensive language and hate speech, by issuing penalties such as content removal or account suspension to users they determine to be disruptive. These moderation systems usually fail to provide punished users enough support in terms of explaining why they are punished and suggesting how they can improve. Such severe limitations in fairness, accountability, and transparency lead to enormous challenges to online moderation and community wellbeing. Punished users may not understand the rationale behind penalties, and risk becoming repeat offenders. This is even more challenging when newcomers are punished for violating community norms which they were previously not aware of.

The study site is a high-population online community, where the research will document and describe human-punishment interaction (HPI) in terms of how users experience punishment, what are users' post-penalty actions, and what support resources users use for a better understanding of community behavioral standards and behavioral improvement. The research has two goals. First, empirical methods such as interview and survey will be used to investigate and theorize major sociotechnical dimensions of HPI. This will extend the existing moderation literature by articulating interactions and experiences associated with punishment and punished users. Second, more concentrated empirical methods such as narrative interview and focus group will be used to identify user-initiated ways of understanding community norms and behavioral improvement, with a focus on existing support resources that users have drawn from. Cognitive and social theories of behavior change will be used to understand existing, user-initiated ways of behavior improvement. This prioritizes a dynamic, evolving view of disruptive behavior, tracing the temporal development of user online behavior and studying and theorizing critical moments where users internalize community norms and improve behavior.