Rob Tett and colleagues published a recent paper in the Journal of Occupational and Organizational Psychology that looked at personality test norms. Examining a large set of data from the Hogan Personality Inventory, they took the norms derived from several large samples of individuals employed in sales, trucking, clerical work, finance, and management. No big surprise, there were some rather significant differences in average scale elevation between groups (e.g., the truckers were substantially less sociable than the sales group). However, a bit unexpected were the differences in mean scores between samples within the same job family. As an illustration, some samples of sales personnel had much higher scores on scales measuring ambition and sociability than did others. Indeed, there were cases where the differences within job family were nearly as great as those between job families.

There are a handful of takeaway points to consider. First, relying solely on job type to select a normative sample can be problematic. Rather, factors such as organizational culture of the work group where the samples were obtained or whether the data were derived from job applicants vs. incumbents need to considered. (The latter is important because applicants tend to obtain higher scores than incumbents, something I will blog about in the near future.)  Secondly, this article also analyzed the impact of sample size on the reliability of normative samples. As a saving grace, they found that a sample of 100 is usually quite sufficient to develop a good set of local norms.

Finally, it should be noted that the importance of norms varies greatly depending upon how a test is being used. For example, if I want to know how the conscientiousness of the  maintenance personnel  in my company compares to that of folks who work in maintenance for other organizations, the normative sample chosen is obviously critical. However, if I am only interested in ranking maintenance job candidates on conscientiousness or have already conducted a validation study to determine cut scores, the use of norms isn’t particularly relevant or useful as we could use untransformed scores for these purposes.