Assessing and Reconciling Between-Group Differences in Job Analysis Ratings

Abstract

Job analysis is an important tool underlying several organizational functions including performance appraisals, selection, and training. Typical job analyses are conducted with a random, stratified sample of subject matter experts (SMEs). These individuals are often job incumbents who are asked to rate and make judgments concerning the frequency and importance of job tasks. It is presumed that incumbents who perform a job should evaluate it similarly. Accordingly, it may be concerning if there is significant variance in employees’ job analysis ratings depending on what is causing the differences. As such, job analysts must determine if rating differences are legitimate (“real”) or due to error. This review provides an overview of various statistical techniques that can be used to assess the significance of between-group differences in job analysis ratings and how they can be used by job analysis experts to determine the extent to which SME ratings are consistent. In addition to detecting rating differences among SME groups, we also provide practical recommendations for reconciling those differences, if necessary.