Toward Universal Creativity Assessment by Untrained Judges

Abstract

The purpose of a creativity session being first to generate many ideas, how to fast and reliably assess each one’s creativity, even with non-creativity experts? Creative people appearing not only to be good at generating ideas but also at implicitly evaluating them, improving people’s assessment would further improve their own creativity. This paper investigates canonical components of originality to identify universal creative performance subscales assessable by untrained judges. Three originality criteria emerged from theoretical research: (1) Feelings, emotions; (2) Imagination, fantasy; (3) Logical complexity. One criterion was used to assess appropriateness: (4) Consistency as defined by four rules. A corpus of 100 ideas were distributed for assessments by untrained judges (n=41). All ideas were assessed by 4 expert judges on a single criterion: creativity. Surprisingly the criterion (4), which had four clear rules to define it, showed insufficient reliability and had to be disregarded for the analysis. The three other criteria proved consistent with one another and were hence reduced to an assessment of originality. They showed poor consistency with expert judges’ ratings. These results are discussed and call for further research to better understand creativity.