(NSF Project) SoCS: Studying the Computability of Emotions by Harnessing Massive Online Social Data

Papers: [ACII 2017] [IEEE TAC 2018] [Psychiatry Research 2018]
Probabilistic Multigraph Modeling for Improving the Quality of Crowdsourced Affective Data

Jianbo Ye, Jia Li, Michelle G. Newman, Reginald B. Adams, Jr., James Z. Wang
The Pennsylvania State University, USA

Abstract:

We proposed a probabilistic approach to joint modeling of participants' reliability and humans' regularity in crowdsourced affective studies. Reliability measures how likely a subject will respond to a question seriously; and regularity measures how often a human will agree with other seriously-entered responses coming from a targeted population. Crowdsourcing-based studies or experiments, which rely on human self-reported affect, pose additional challenges as compared with typical crowdsourcing studies that attempt to acquire concrete non-affective labels of objects. The reliability of participants has been massively pursued for typical non-affective crowdsourcing studies, whereas the regularity of humans in an affective experiment in its own right has not been thoroughly considered. It has been often observed that different individuals exhibit different feelings on the same test question, which does not have a sole correct response in the first place. High reliability of responses from one individual thus cannot conclusively result in high consensus across individuals. Instead, globally testing consensus of a population is of interest to investigators. Built upon the agreement multigraph among tasks and workers, our probabilistic model differentiates subject regularity from population reliability. We demonstrate the method's effectiveness for in-depth robust analysis of large-scale crowdsourced affective data, including emotion and aesthetic assessments collected by presenting visual stimuli to human subjects.


Full Paper
(PDF, 9MB) (R Code)

Dataset: 90% confident high/low valence images from Amazon Mechanical Turk
amt_validated_valence_90.zip (524MB, 4415 images; due to copyright please contact Dr. James Wang to obtain for research; jwang at ist.psu.edu)

Dataset: 95% confident high/low valence images from Amazon Mechanical Turk
amt_validated_valence_95.zip (11MB, 94 images; due to copyright please contact Dr. James Wang to obtain for research; jwang at ist.psu.edu)

Citation: Jianbo Ye, Jia Li, Michelle G. Newman, Reginald B. Adams, Jr. and James Z. Wang, ``Probabilistic Multigraph Modeling for Improving the Quality of Crowdsourced Affective Data,'' IEEE Transactions on Affective Computing, vol. 8, 14 pages, 2018.

© 2017 All rights reserved.