Picture for Jeffrey T. Hancock

Jeffrey T. Hancock

Jury Learning: Integrating Dissenting Voices into Machine Learning Models

Add code
Feb 07, 2022
Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeffrey T. Hancock, Tatsunori Hashimoto, Michael S. Bernstein

Figure 1 for Jury Learning: Integrating Dissenting Voices into Machine Learning Models
Figure 2 for Jury Learning: Integrating Dissenting Voices into Machine Learning Models
Figure 3 for Jury Learning: Integrating Dissenting Voices into Machine Learning Models
Figure 4 for Jury Learning: Integrating Dissenting Voices into Machine Learning Models

Whose labels should a machine learning (ML) algorithm learn to emulate? For ML tasks ranging from online comment toxicity to misinformation detection to medical diagnosis, different groups in society may have irreconcilable disagreements about ground truth labels. Supervised ML today resolves these label disagreements implicitly using majority vote, which overrides minority groups' labels. We introduce jury learning, a supervised ML approach that resolves these disagreements explicitly through the metaphor of a jury: defining which people or groups, in what proportion, determine the classifier's prediction. For example, a jury learning model for online toxicity might centrally feature women and Black jurors, who are commonly targets of online harassment. To enable jury learning, we contribute a deep learning architecture that models every annotator in a dataset, samples from annotators' models to populate the jury, then runs inference to classify. Our architecture enables juries that dynamically adapt their composition, explore counterfactuals, and visualize dissent.

* To appear at CHI 2022 
Viaarxiv icon
Github IconAccess Paper or Ask Questions
Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon

Finding Deceptive Opinion Spam by Any Stretch of the Imagination

Add code
Jul 22, 2011
Myle Ott, Yejin Choi, Claire Cardie, Jeffrey T. Hancock

Figure 1 for Finding Deceptive Opinion Spam by Any Stretch of the Imagination
Figure 2 for Finding Deceptive Opinion Spam by Any Stretch of the Imagination
Figure 3 for Finding Deceptive Opinion Spam by Any Stretch of the Imagination
Figure 4 for Finding Deceptive Opinion Spam by Any Stretch of the Imagination

Consumers increasingly rate, review and research products online. Consequently, websites containing consumer reviews are becoming targets of opinion spam. While recent work has focused primarily on manually identifiable instances of opinion spam, in this work we study deceptive opinion spam---fictitious opinions that have been deliberately written to sound authentic. Integrating work from psychology and computational linguistics, we develop and compare three approaches to detecting deceptive opinion spam, and ultimately develop a classifier that is nearly 90% accurate on our gold-standard opinion spam dataset. Based on feature analysis of our learned models, we additionally make several theoretical contributions, including revealing a relationship between deceptive opinions and imaginative writing.

* Proceedings of ACL 2011: HLT, pp. 309-319  
* 11 pages, 5 tables, data available at: http://www.cs.cornell.edu/~myleott 
Viaarxiv icon
Github IconAccess Paper or Ask Questions
Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon