Kenneth Huang to present lecture Feb. 20 at Cornell

Ting-Hao ‘Kenneth’ Huang, associate professor in the College of IST’s Department of Human-Centered Computing and Social Informatics, will talk about annotation, large language models and humans at Cornell’s Human-Centered Design Department on Friday, Feb. 20.

Huang will present “What Roles Will Humans Play in the Future of Data Annotation?”

Abstract:

“Large language models (LLMs) have rapidly and impressively taken over many tasks once handled by human annotators in constructing text datasets. But does this mean we no longer need humans in the annotation loop? What roles should humans play in future data annotation pipelines?

“In this talk, I will present two recent studies that explore the evolving role of humans in the landscape of text data annotation. First, we ask whether a well-designed and carefully executed traditional crowdsourcing pipeline can still outperform LLMs in labeling quality. Our study offers an in-depth, holistic comparison of human and LLM annotation performance.

“Second, we turn to a future where LLMs increasingly replace manual annotation labor. In this scenario, the human role shifts toward instructing the models—often through prompting. But how effective are humans at prompting LLMs for annotation tasks, especially when working without access to gold-standard labels? We investigate this growing practice, which we call "prompting in the dark," and assess its implications for the quality and reliability of LLM-generated annotations."

>>>Read more at Cornell University College of Human Ecology