Comparing Scales for the Assessment of Visual Aesthetics of Mobile GUIs Through Human Judgments

Visual aesthetics is a success criterion for mobile apps. Despite considerable research on graphical user interface (GUI) assessments, there is a lack of studies investigating the reliability and validity of scale types on visual aesthetics as a unidimensional construct. In this study, 208 subjects were divided into four groups, each using a different rating scale and the VisAWI-S questionnaire as the golden standard, to assess the visual aesthetics of nine mobile GUIs. As a result, all scales showed excellent inter-rater reliability and good agreement. Seven-point scales resulted in slightly higher intra-rater reliability than those with five points, but agreement was lower using five-point Likert scales. All scales have shown to be valid compared with VisAWI-S and presented strong correlations pairwise. Results indicate that any of these scales are suitable to assess mobile GUI visual aesthetics reliably and validly as long as response quality is analyzed. This work supports the adoption of single-item questionnaires reducing effort and time, especially in large-scale assessment designs.