pan blog - Assessment

Assessing the Value of Assessments

Reid Klion — Mon, 14 Jan 2013 19:34:38 GMT

Charles Handler recently blogged on a topic that I have written about several times as well—the challenges involved with gathering the data needed to explore the degree to which assessment programs brings value to an organization. These include problems associated with measuring job performance (subjective ratings tend to be quite unreliable so doing this properly requires an investment of time and resources), small samples (which create difficulties for statistical analysis), and translating this information into terms the rest of the organization understands (that is, dollars and cents).

Charles makes a number of suggestions that I think are worth considering. The first is to target “low-hanging” fruit. Here, the approach is to find those positions for which there is already decent job performance data and start your efforts here. If successful, you will have built up some good will that may convince the organization to invest the resources to tackle those job positions which will require more effort to obtain good outcome measures. Another is to consider using supervisory ratings. While sometimes suspect, they can be used as effectively if supervisors are trained in their use and supported in their efforts. A third suggestion is for organizations to look to vendors in the pre-employment selection space who have substantial experience in implementing local validation studies.

While there are no silver bullets, it is important for organizations that use assessments to seriously consider making the effort to collect the type of job performance data that will enable to make a reasonable decision about the value they obtain from their assessment programs. Otherwise, decisions are often made upon beliefs that may or may not be consistent with the actual situation.

Reid Klion

Assessing the Value of Assessments

Reid Klion — Mon, 14 Jan 2013 19:34:35 GMT

Charles Handler recently blogged on a topic that I have written about several times as well—the challenges associated with gathering the data needed to explore the degree to which assessment programs brings value to an organization. These include problems associated with measuring job performance (subjective ratings tend to be quite unreliable so doing this properly requires an investment of time and resources), small samples (which create difficulties for statistical analysis), and translating this information into terms the rest of the organization understands (that is, dollars and cents).

Charles makes a number of suggestions that I think are worth considering. The first is to target “low-hanging” fruit. Here, the approach is to find those positions for which there is already decent job performance data and start your efforts here. If successful, you will have built up some good will that may convince the organization to invest the resources to tackle those job positions which will require more effort to obtain good outcome measures. Another is to consider using supervisory ratings. While sometimes suspect, they can be used as effectively if supervisors are trained in their use and supported in their efforts. A third suggestion is for organizations to look to vendors in the pre-employment selection space who have substantial experience in implementing local validation studies.

While there are no silver bullets, it is important for organizations that use assessments to seriously consider making the effort to collect the type of job performance data that will enable to make a reasonable decision about the value they obtain from their assessment programs.

Reid Klion

Predicting Elections: Intuition vs. Statistics

Reid Klion — Tue, 27 Nov 2012 19:47:50 GMT

The recently concluded presidential election provided an interesting example of the arguments between those who advocate using an empirically-based approach to predicting behavior as opposed to those who are more comfortable with using their intuition and other less structured approaches. This is a topic I’ve blogged about previously because it has implications for how hiring decisions are often made. Research has found that organizations often place too much weight upon impressions gathered from non-structured interviews when deciding whom to hire and put little stock in empirical assessment data, even though the latter tend to be much more predictive of job performance.

The current controversy centered around Nate Silver, a statistician who first gained notoriety by developing a system for predicting the performance of major league baseball players and now writes a blog, FiveThirtyEight, that focuses on political polling data. For the current presidential election, he aggregated and weighted data from a large number of polls in order to predict the eventual outcome (which he did quite accurately on 2008). When he offered his predictions for 2012 (which turned about to be quite accurate again, getting all 50 states right and the overall vote percentage won by 0.3%), the objections in the press were quite similar to those often heard about the use of psychometric assessments in making hiring decisions. For example, one writer complained that polling data only offered a “fuzzy snapshot” and not predictive of the future, another based her disagreement on the “intensity and joy” of the crowds she observed at various political rallies as well as the fact that the eventual winner seemed “tired,” and a third rejected the notion that a specific probability could even be calculated for the outcome.

When it comes to predicting the future, whether it relates to elections or job performance, it is often worthwhile to set intuitions aside and at least consider the alternate perspective that can be provided by statistical analyses.

Reid Klion

Part 2 - Busting Talent Assessment Myths: A Risk Assessment of Talent Assessments

Sara Camden — Mon, 26 Nov 2012 17:07:15 GMT

If you have any uncertainties about when and how to leverage talent assessments, you are not alone. There are many misconceptions when it comes to developing, implementing, measuring and refining an assessment program that achieves results. This is precisely why the talent measurement experts at pan > have developed a 4-part guide for HR professionals wanting to separate fact from fiction, and set up their talent assessment initiatives for success. Having worked with all types of testing tools and publishers over the course of 10+ years, we’ve decided to share our lessons learned when it comes to the most common myths about assessments.

In this second installment of our series on busting talent assessment myths, we explore the experience of administering a talent assessment program as well as how candidates and employees view the test taking process. This segment will expand on the myths related to selecting testing content that we discussed during part 1 of our series titled ‘Testing the Test’, and will focus on the talent assessment experience, both from the administrator’s perspective as well as the test taker’s view. In the last 2 segments of our series, we’ll cover the technical and logistical aspects of assessments and close with talent measurement.

A Risk Assessment of Talent Assessments

With the war for talent picking up steam, best-in-class organizations continue to find creative and effective ways in which assessments enhance their talent strategy goals and gain a competitive edge, as reported in the Aberdeen Group’s annual research report. There are a number of right and wrong answers when it comes to implementing a talent assessment program. Once you’ve developed a sound program with scientifically validated tools and measureable HR business objectives, it’s now time to see if all this groundwork you’ve laid achieves the outcomes you seek. Since pan > has helped facilitate countless assessment programs for organizations of all sizes and industries, we are prepared to share our expert advice about the most common uncertainties related to talent assessment programs in practice.

MYTH: Assessments are too subjective

To address this myth, one must understand the intention behind an assessment program and particular test. Many tests for development purposes are indeed subjective, and not meant to be utilized as a screening tool. Tests designed and utilized for hiring purposes, however, actually show a very strong correlation to job performance.

The American Psychological Association performed a meta-analysis of 85 years of research findings regarding the validity and utility of selection methods, and found that talent assessments were more predictive of job performance than any other screening measure examined. The test types showing strong correlations to performance were skills, cognitive and integrity assessments which are all designed to collect objective data. Other traditional screening tools such as unstructured interviews, reference checks and years of experience showed a much lower correlation to real-world job performance.

MYTH: Employees and Candidates Hate Taking Assessments

It is true that some people experience anxiety about taking tests, but ambitious candidates and employees look at assessments as a way to differentiate themselves from the pack. pan > operates a network of 600+ remote testing center facilities that serve 100’s of thousands of assessments to candidates annually. Each test taker is required to take an exit survey about their experience, and the feedback is overwhelmingly positive at a rate of 99.5%+.

pan > is getting more and more requests from job seekers desiring to proactively take assessments with the intention of leveraging their results as a selling tool for advancing their careers. With the job market becoming increasingly competitive, and the HR function becoming more strategic, the use of assessments is only going to continue to explode as it has for the past 10 years.

MYTH: Assessments Expose my Organization to Legal Risk

Concerns about legal compliance issues represent one of the most common misconceptions about talent assessments that our team encounters, and it’s no wonder considering the dire consequences an EEOC/OFCCP employment discrimination charge can have. Many hiring organizations are surprised to discover that talent assessments in actuality can help mitigate an employer’s risk of discriminatory charges, serving as solid, quantifiable evidence of an objective hiring process that’s consistent and proven. The keys to guarding your organization against any legal risk when it comes to talent assessments are to prove job-relatedness, and implement your program on a consistent basis.

Kate Williams, a doctoral candidate at Clemson University, conducted an in-depth analysis of all discriminatory legal cases related to selection tools over the past 12 years. There are several key takeaways nicely summarized by talent assessment industry thought leader Dr. Charles Handler for ERE.net. At an average of less than 20 cases per year, the volume is very small in comparison to the number of hiring organizations in the USA meaning that even if you are doing everything wrong in your assessment program, the risk is still incredibly low. For all of the charges analyzed in Williams’ study, any of those citing talent assessment instruments were related to cognitive and psychomotor tests, and whether or not they were job-related in nature. In one particular example (EEOC vs Nations Bank of Tennessee (2001)), an employer was found to be discriminating against Spanish speaking employment candidates because they were using an English language cognitive test when speaking English was unrelated to the job’s requirements.

When a strategic talent assessment program is well-designed and implemented correctly and consistently, it actually enhances the objectivity of hiring processes, providing solid evidence of fairness and relevance. This translates to an assessment program where valid instruments are used, job-relatedness is proven, the program is implemented with consistency and ongoing due diligence is practiced to monitor for any adverse impact on protected demographic groups. pan has served as an expert witness for clients over the years in employment discrimination cases, and 100% of these cases have turned out favorably. One such case involved EEOC charges against a state government agency in Texas. Because they conducted a thorough job analysis and competency mapping to prove their talent assessment program’s job-relatedness before implementation, and then conducted ongoing monitoring for adverse impact, they were able to prove that the charges were entirely unwarranted, and won the case.

To summarize, here are some key steps to ensure your assessment program is not only legally compliant, but also built to serve as protective evidence of fair hiring practices:

1.) Select scientifically valid testing instruments based on thorough job analyses

2.) Perform competency mapping and prove validity for your unique use case

3.) Implement the program consistently

4.) Monitor for adverse impact throughout the testing program’s lifecycle

CLOSING

Now that we’ve armed you with the facts related to legal compliance and employee engagement risks surrounding talent assessments, we hope you’ve gained reassurance that talent assessments in practice are not as scary as they might seem. Be on the lookout for part 3 in our series in which we’ll share insights on the technical implementation of an assessment program. If you have any questions in the meantime, you can always reach out to the team at pan for complimentary advice about the talent assessment topics most relevant to you.

>> REGISTER FOR FREE pan ACCOUNT

Personality Factors and College Success

Reid Klion — Thu, 08 Nov 2012 22:11:08 GMT
A colleague recently shared some papers on the relationship between personality and success at college, a topic which has been researched fairly extensively. Overall, it does appear that personality differences can add predictive validity beyond cognitive measures such as high school GPA and SAT scores. (The underlying explanation is based upon the "can do" v. "will do" principle--cognitive factors predict an individual's maximal ability while personality is more predictive of typical, day-to-day behavior.)

The personality trait that has been found to be most linked to academic performance is conscientiousness. Not unlike the workplace, it predicts an individual’s inclination to follow rules and motivation to perform well so its connection to academics is fairly intuitive. Openness to experience also predicts GPA to a lesser degree. To my mind, the most interesting finding (which also demonstrates that more of a characteristic is not necessarily better) is that extraversion has a slight but negative correlation with academic performance. This may indicate that students with higher levels of sociability tend to pursue this at the expense of academic work. Here is a recent meta-analysis if you are interested in learning more.

Reid Klion

The Achilles Heel of Using "Big Data" for Selection

Reid Klion — Fri, 02 Nov 2012 16:19:02 GMT
Getting caught up on my reading, I just saw an interesting post by Charles Handler on hiring systems based upon complex analyses of “big data.” Charles makes a number of good points worth consideration. However, one that he highlights which has been of concern to me for some time whenever these discussions come up centers on job performance metrics. Simply put, for any type of selection system to be effective (be it based upon “big data” or more traditional methods), we need to have this information. Otherwise, we have little idea about what we are really trying to predict.

A topic of a post last year, workforce psychologists have faced this challenge since the early days of the field. Essentially, until organizations implement performance metrics that accurately capture how individuals function on the job (and standard performance measurement systems are usually not up to the task), the ability to predict job performance is inherently limited. For many jobs, there are effective methods available. The challenge for organizations often lies in making the investment (mostly in terms of time) to implement and use them. It needs to be said that for some jobs (call centers being a prime example), it has become relatively easy to capture meaningful work metrics. However, the greater the extent that organizations seek to accurately measure the performance of all workers, the further down the road they will be toward being able to select applicants who will function most effectively on the job.

Reid Klion

School Kids Rating Their Teachers

Reid Klion — Tue, 30 Oct 2012 18:00:09 GMT
There has been quite a bit of discussion in the press over the evaluation of school teachers—what methods should be used (e.g., principal observations vs. standardized test scores) and how the results should be used (if at all) to determine compensation and even continued employment. Unfortunately, it appears that much of the discussion around this is agenda-driven and often seems to miss the point. It is also an area that would likely benefit from the involvement of industrial-organizational psychologists who have specific experience in developing work performance systems.

That being said, one innovative method that has been found to be surprisingly effective in assessing teacher effectiveness is just asking the kids. At first glance, this wouldn’t seem a terribly fruitful approach—the students might be too young to provide useful information or favor teachers who seen as “easy” as opposed to those who were stringent graders or strict disciplinarians. However, in research sponsored by the Bill and Melinda Gates Foundation focused on developing ways of measuring effective teaching, this was not found to be the case at all.

Rather than asking about whether students “like” their teacher or not, questions include items such as “Our class stays busy and doesn’t waste time” and “the teacher in this class encourages me to do my best." And, despite of obvious messiness and inherent measurement error associated with collecting survey data from schoolchildren as young as five years old, several years of data indicate these student survey responses are surprisingly effective in predicting a classroom’s standardized test scores. More importantly (and not dissimilar to the use of multi-rater feedback tools in the workplace), these student data can also be used to help teacher’s to better understand their strengths and weaknesses in the classroom.

Reid Klion

Part 1: Busting Talent Assessment Myths - Testing the Test

Sara Camden — Thu, 04 Oct 2012 21:26:53 GMT

The talent assessment industry has exploded over the last 10 years as paper and pencil tests have migrated online. With hundreds of test vendors and thousands of tests at your fingertips, selecting the right assessment tools for achieving the best possible results for your organization is no small feat. Let’s take a moment to separate fact from fiction when it comes to the testing tools available.

MYTH: XYZ Test Works for Everything

Once you’ve determined what the goals of your assessment program are, it’s no easy task unearthing which testing tools are right for the job. Although many assessment tools can be very effectively leveraged across a number of applications, not every test works for every purpose. There are 100’s of respected test publishers who have developed 1000’s of scientifically sound instruments, giving you lots of options to find the exact tools suited for your program’s unique goals. From skills tests to personality tests, cognitive tests, job-specific tools and more, the options are limitless when it comes to selecting a tool that addresses exactly what you’re looking to accomplish.

Since pan > is an assessment marketplace working with a diverse cross-section of the most respected test publishers on the market today, we have vast exposure to how the most popular testing tools are being leveraged. This unique setup equips us with the ability to provide neutral advice about which assessment tools would suit your unique program best. In many cases, we find that it’s a complementary mix of tests from different sources that facilitates the most impactful assessment programs. Another great source for unbiased advice on choosing the right assessments is Charles Handler’s Rocket-Hire Buyer’s Guide to Screening and Assessment Systems.

MYTH: Online Assessment Options are too Restrictive and Narrow

This myth might have you scratching your head after just reading about the wealth of assessment tools available. When we encounter this feedback, it typically comes from organizations that are talking with or utilizing a particular assessment publisher. Working with a single assessment vendor can present limitations, particularly if you’d like to consider using a battery of tests to accomplish your assessment program goals. Some questions to ask yourself when selecting an assessment vendor:

Does this vendor offer the best of everything I need to accomplish my assessment goals, or would it be beneficial to cherry pick assessment tools from multiple sources?

Does my organization have any custom, proprietary tests, forms or surveys we’d like to incorporate in an assessment program?

What level of administration is involved, and how are results reported?

Will these assessment results integrate with my ATS or other technology systems?

If these questions raise any red flags for you, the team at pan > is happy to provide friendly advice on how to come up with a custom solution that addresses your needs in each of these key areas.

MYTH: Developing a Custom Assessment is More Effective Than Using an Off-the-shelf Tool

To get to the bottom of this myth, it’s important to have a deep understanding of your assessment program’s distinct purposes and goals. pan has found that for the vast majority of hiring and development initiatives, there are a wide range of off-the-shelf assessment tools that can be deployed immediately and leveraged very successfully. There’s no doubt that custom assessments have value in unique or highly proprietary scenarios, but take a lot of time and money to develop, validate and refine.

Before pursuing the development of such a tool, consult with an unbiased talent measurement expert source such as pan. Our technology platform hosts 1000’s of custom assessments for world-class public, private and government organizations, and we are happy to offer our insights on when and how to utilize such tests either solely or in conjunction with off-the-shelf tools.

Stay tuned for Part 2 of our mythbusting series

Next time, we will explore the experience of a talent assessment program in action both from the test taker’s point of view, as well as the organization’s vantage point when considering best practices and legal compliance. In the meantime, you can always reach out to the team at pan for complimentary advice about the talent assessment topics most relevant to you.

>> REGISTER FOR FREE pan ACCOUNT

What's New in the World of IQ?

Sara Camden — Mon, 24 Sep 2012 21:06:59 GMT
The topic of intelligence has always been a challenging one for psychologists. On the one hand, it is considered controversial by some due to arguments about the role of nature vs. nurture, sub-group differences, and its historical linkage to now disavowed notions such as eugenics. On the other, it has been studied quite extensively since the early days of psychology and it is also recognized as one of the best predictors of both academic and workplace success.

A good review of recent research in the area was published recently in the American Psychologist. Fortunately, unlike many review articles, this one is not overly technical and likely to be of interest to the public.

There were several points I found to be of interest. While it is broadly recognized that IQ is at least partially determined by genetics (like virtually everything as recent research has borne out), virtually no progress has been made in identifying the specific genes that are responsible. It also should be noted that the heritability of IQ may vary as a function of socioeconomic status. More specifically, if children are raised in poverty, they are unlikely to develop to their full potential. Interestingly, though, children who are adopted tend to have IQs that correlate more highly with those of the family who adopted them as opposed to their birth siblings.

The paper also provides some updated research on the Flynn effect. This is the well-documented finding that IQ scores go up 3 points every decade once a culture begins the process of modernization and industrialization. Some of this is obviously due to improvements in nutrition and sanitation. However, since these effects have continued in the US and western Europe long-after the conditions have been well-established, it is also likely that improved schooling and the prevalence of more cognitively demanding occupations have contributed to this trend. And when we look at scales that measure specific skills, all have gone up substantially, especially those that tap abstract reasoning. The only exception is arithmetic (which has improved only marginally), probably due to the fact these skills are no longer emphasized. Obviously, there is no way to summarize all the interesting findings, and the article is definitely worth a look.

Reid Klion

But Isn't More Better?

Sara Camden — Mon, 06 Aug 2012 19:37:03 GMT
Before an assessment program is implemented, the first thing to we need to understand is which competencies contribute to success in the position at hand. A second, but sometimes more subtle issue, is what levels of these characteristics are required. Here, there are situations where enough is good but more may not necessarily be better.

Consider the use of a physical ability test. If a job requires frequent lifting of 40 pound boxes, using a pre-employment assessment that parallels this task is likely to be effective. However, requiring job candidates to be able to lift a 120 pound box is likely to be problematic. The issue of adverse impact aside, using such a stringent test will almost inevitably eliminate a number of candidates who would likely do quite well in the position because physical strength is not the only factor that determines job success.

Another example is cognitive ability. On the one hand, we know that if we only have one piece of information about a job applicant, this is likely to be the most predictive factor. However, if we set our cognitive scores at a level beyond that which is required for job success, we likely remove from consideration a number of candidates who may do quite well on the job and needlessly reduce our pool of qualified applicants (not to mention increasing the risk of adverse impact with little added benefit).

A slightly more subtle situation occurs with individual attitudes and preferences. For example, a moderate level of achievement orientation is often predictive of job success. However, when it comes to positions that involve a considerable degree of routine repetitive work (say, for example, some call center positions or long haul trucking), employees with a significant need for achievement tend to become dissatisfied with this type of work rather quickly and leave when other job opportunities avail themselves.

Assessments are a powerful way to build a better workforce. However, like any tool, the more wisely we use them, the better the chance of our obtaining the results we want.

Reid Klion

On Not Being Able to Find Qualfied Employees

Sara Camden — Tue, 31 Jul 2012 20:25:12 GMT
Peter Cappelli just released a book that was based on his Fall 2011 Wall Street Journal article “Why Companies Aren’t Getting the Employees They Need.” Cappelli tackles the complaint that employers can’t find skilled workers despite unemployment still hovering near 8%. While poor schooling and even worse work attitudes on the part of job candidates are often cited, he argues that the major culprit is employers and outlines a number of factors.

Primarily, employers often want people who already have several years experience in similar work rather than identifying candidates who are more than capable of learning to do the job, if given an opportunity. He argues that it would behoove employers to be more flexible when it comes to experience-based qualifications. Doing so would also open the market to recent graduates (who almost by definition have spent very little time on the job) or people with experiences in other fields. The situation is made worse by the fact that employers have often cut back (or even eliminated) their training budgets in the interest of cost savings. To address these issues, Cappelli makes several suggestions including creating alliances with technical schools as well as providing opportunities for on-the-job job training through informal apprenticeship and extended internship programs. In thinking about the situation, it also seems logical that if employers are interested in measuring the ability of job candidates to learn a job before they are hired, assessments provide an excellent source of predictive information. Reviews of the new book have piqued my interest and I'll blog about this again after I have a chance to read it. (Skimming the book, he also sees applicant tracking systems contributing to these problems, so this should definitely be an interesting read.)

Reid Klion

Automated Essay Scoring Competition

Sara Camden — Tue, 10 Jul 2012 15:27:54 GMT
A topic that has garnered a fair amount of attention in the assessment world, especially in education, is the use of computers to score essays written by students. While researchers have been working on this since mid-1990s, there has been a major surge of interest due to the widespread growth of standardized testing in schools spurred by No Child Left Behind and other related legislation.

Essentially, these systems are first calibrated by having them analyze a large sample of essays (perhaps up to 1000) that have already been graded by human raters. After the systems develop a "model" of both good and bad essays, they are then able to determine how well a new set of essays correspond with the calibration essays. A number of studies have compared the accuracy of computer vs. human-scored essays. In most instances, computers have been found to be more accurate than humans, in large part because computers are perfectly reliable in that they will score the same essay the exact same way each time whereas human scorers are prone to inconsistency due to fatigue, boredom (imagine scoring 30 essays an hour for eight hours a day), and human error. That being said, computers do a much better job of scoring for grammar and sentence structure than content. As a result, an essay that is well-constructed from a grammar perspective but that makes no sense may receive a high score from a computer, but would be immediately recognized as nonsensical by a person. As a result, the typical process when automated essay scoring is used is to have each essay scored by both a computer and a person.

As a testament to the growth of interest in this area, the William and Flora Hewlett Foundation recently sponsored a contest to compare the results of nine automated essay scoring engines. Based upon a sample of over 22,000 writing samples produced by students in six states as part of their high-stakes writing assessments, all of the scoring systems performed remarkably well, as well as quite similarly to each other. While computers will not replace people when it comes to evaluating writing samples, they can definitely play a supporting role in increasing the consistency of scoring.

Reid Klion

Video on Gao Kao

Sara Camden — Mon, 02 Jul 2012 13:51:09 GMT
A few days ago, the New York Times posted a video on the gao kao, or Chinese university entrance exam with some interesting interviews with students as well as their families. It discusses how some students spend up to 12 hours a day every day studying and also highlighted a family that rented a second apartment that was closer to their son's school so that he could spend more time studying as opposed to traveling back and forth to school.

Reid Klion

Everything is Big in China

Reid Klion — Tue, 26 Jun 2012 15:56:40 GMT
The first week in June marked the annual administration of China’s gao kao, or college entrance exam. While participation is down some from past years due to increasing numbers of China’s students seeking international university placements, over 9.1 million took the assessment in pursuit of 6.85 million spots in Chinese universities. Unlike US universities where admissions decisions are based upon a wide variety of factors including extracurricular accomplishments, the gao kao is the primary decision making tool. As a result, students spend months in preparation with some going as far as obtaining IV drips of amino acids and supplemental oxygen before the exam in an attempt to enhance performance. In order to reduce distractions, authorities seek to re-route flights and funeral processions from areas near testing centers. Unfortunately, but not unexpectedly, there are major challenges due to cheating. Most testing centers have video surveillance and there was a series of 1500 arrests of individuals who were suspected of selling equipment to facilitate cheating (such as clear-plastic earphones) immediately before the exam.

Reid Klion

LSAT Administrations Down Nearly 25%

Sara Camden — Tue, 20 Mar 2012 19:12:41 GMT
Over the past few years, the employment market for newly minted lawyers has plummeted. From a 2009-2010 high of 171,514 administrations, it went to 155,150 last year and now to 129,925 this season. While data can be hard to come by, and there have been multiple reports (and lawsuits) over law schools exaggerating both employment rates as well as starting salaries, recent credible reports indicate there are approximately twice as many law grads who pass the bar as there are legal jobs for them.

While some of this is reflective of general economic trends, it is also a testament to how legal practice has changed due to a number of factors including clients who are no longer willing to pay high hourly rates to newly minted lawyers, increased use of technology, as well as offshoring of legal work. As a result, the job market has contracted and will likely not return to previous levels nor to similar levels of compensation. Perhaps even more so during poor economic times, law schools often serve as a temporary destination for recent college graduates who couldn’t find work or were unsure of what career to pursue. Now, with the recognition of the substantial loan burden that law school often entails (sometimes well over $100,000) and the realization that there are limited employment opportunities for young lawyers upon graduation, it seems the attraction of law school has been significantly diminished.

Reid Klion

Nominations Being Accepted for IPAC Innovations in Assessment Award

Reid Klion — Fri, 16 Mar 2012 17:22:41 GMT
The International Personnel Assessment Council (IPAC) is currently accepting nominations for its annual Innovations in Assessment Award. The award recognizes a person or work group for the development and application of an innovative personnel assessment tool or procedure which has resulted in improved effectiveness, efficiencies or cost savings. Self-nominations are also accepted. The award will be presented at the 2012 IPAC Conference to be held July 17-20 at July 22-25 in Las Vegas. More here.

Reid Klion

Collecting Outcome Data: Practical Considerations

Sara Camden — Fri, 09 Mar 2012 16:15:48 GMT
A topic that often receives less attention than it should in the HR community is the collection of employee performance data. On the one hand, we need it in order to evaluate the effectiveness of selection tools and other measures that might be used to improve job performance. On the other, it is often difficult to obtain useful job performance measures that are truly effective because the data we tend to collect in the course of our day-to-day work, such as performance management system is often not useful for this process. (Here is an earlier post on this issue.)

With this in mind, I attended a presentation by Eugene Burke who is Chief Science and Analytics Officer at pan’s content partner, SHL, at the recent Association of Test Publishers meeting where he spoke about some of the findings in his recent paper. Reflecting on some of the findings in the paper (and other research indicating that many organizations do not collect employee performance data in a systematic way), I asked him how he managed to induce organizations to collect the performance data he reported. His response, which I thought was rather pragmatic and effective, is to encourage clients to first identify the job metrics they want to impact within the organization (e.g., reaching sales quota, early turnover, absenteeism) and then collect data internally which tap into this. By identifying organizationally, goals which are of specific interest, it is often easier to garner internal support and resources to collect them. Additionally, these can be used as outcome measures to assess the effectiveness of selection programs and other interventions.

Reid Klion

Difficulties Hiring Skilled Factory Workers

Reid Klion — Mon, 20 Feb 2012 19:06:49 GMT
While the unemployment rate continues to trend down slowly, it is still above 8% (8.3% nationally as of January 2012). However, there are also increasing reports of businesses having difficulty hiring skilled workers, specifically in the manufacturing trades. While the US has lost several million factory jobs over the past decade, there is an increasing need for individuals who can program and operate the automated manufacturing systems that are being used on an ever broadening scale The challenge for employers is finding people with the aptitude and skills who can do this work. A number of factors contribute to this scarcity including the dynamic that many traditionally trained tool and die makers don’t have the skills to run CNC machines, demographic shifts in younger workers away from factory work, and the significant level of math and technical skills this work now requires. Since this new approach to manufacturing reduces the number of people required for production, it reduces the need for workers with lower skills (who are often relatively easy to hire) while simultaneously increasing the need for those with higher level skills who are difficult to find. Some proposed solutions include re-introducing industrial arts programs in high schools (which may induce students to pursue CNC training post high school), broadening training opportunities in technical colleges for skilled manufacturing, and employers using testing to identify inexperienced potential employees who may have the aptitude for post-hire higher level training.

Reid Klion

Call Centers and Selection

Sara Camden — Thu, 02 Feb 2012 18:38:16 GMT
Assessments are frequently used to select call center personnel. There are several reasons for this. One is the sheer number of call center positions with some estimating that over 4 million people do this work in the US alone, not to mention millions more in offshore locations. Another contributing factor is the nature of call center work. On the one hand, a high level of customer service and accuracy is critical in addition to computer skills, attention to detail, problem solving, and multi-tasking. Complicating things, call center representatives are often asked to help confused, angry, or upset callers in a professional manner yet do so as quickly as possible because a frequent performance metric is number of calls handled. However, since the work is often routine and repetitive, it can become boring fairly quickly and lead to absenteeism and turnover unless the right employees are selected.

Fortunately, there are a number of assessments that have been developed specifically for call centers (and pan is fortunate to distribute a number of them). Some look at personality factors such as conscientiousness, customer service attitude, and reliability. Others, which are designed to test an individual’s call center skills, actually place the individual in the simulated role of a call center representative.

A dynamic to consider when selecting call center personnel is that given the substantial cost in reputation that negative client experiences can bring coupled with the large number of people employed in these positions is that even relatively minor improvements in the hiring process can have a substantial impact upon the bottom line. A final consideration is that unlike many other jobs, call centers can often readily provide accurate and meaningful metrics about call center performance with indicators such a number of calls handled per shift, sales, and caller satisfaction surveys. As a result, it often can become apparent quite readily if a selection system is working and how it might be adjusted to further optimize the process.

Reid Klion

So Is This Test Valid?

Reid Klion — Fri, 27 Jan 2012 19:51:42 GMT
This is something we are often asked. In answering it, the critical issue to bear in mind is that validity has to do with the conclusions, inferences, or decisions we want to make based upon the assessment, not the assessment itself. Consider a typing test. If it is reasonably constructed, it is pretty likely that it measures typing speed and accuracy. However, is it valid? Without knowing what the test is going to be used for, we don't know. If we are using it to select administrative assistants, we are likely on the right track if we assume keyboarding is important for job success in this role. However, by the same token, it is likely to be invalid for selecting groundskeepers.

Commercially available assessments have technical documentation that details how the test measures what it is designed to measure. And this is certainly something that enters into the validity equation. However, the key question about validity has to do with the intended use of the assessment. The underlying question really isn't whether a test itself is valid but rather Valid for What?

Reid Klion

Talent Selection and the Super Bowl

Reid Klion — Wed, 25 Jan 2012 16:35:37 GMT
The Super Bowl, coming up in about a week and a half, will be a re-match of the 2008 game with the New England Patriots facing the New York Giants. An interesting fact is how different the rosters of the two teams are today from four years ago. While the quarterbacks and coaches are the same, the Patriots only have 5 starters (of 24, counting the punter and kicker) from the 2008 team while the Giants have 9 remaining starters. (Going a bit further, there are only 7 Patriots and 16 Giants on the team roster in total today who were there 4 years ago.). While this is in part testament to the relatively short tenure of most players in the NFL (around 3.5 years on average) and, admittedly, two very good quarterbacks, it is even stronger evidence about the importance of selecting and training new talent. So while many of the individual players are different, the strength of the talent management skills of both these organizations is a constant thread.

Reid Klion

Let's Profile the High Performers!

Reid Klion — Tue, 24 Jan 2012 18:57:09 GMT
When we help a client to implement an assessment program, we are sometimes asked to use a “profiling” approach which involves giving the assessment to handful of the best performing employees, look at their scores, and then use this to create a benchmark for hiring new employees. While this model is intuitive, quick, and easy to execute, it has a number of drawbacks. From a measurement perspective, the first challenge is that it is virtually impossible to create norms that are stable and meaningful when they are only based on a handful of cases. Secondly, since most assessments employ a large number of scales and sub-scales, it is rather difficult to decide which scales are actually predictive of performance without additional analyses of the job at hand and what it requires. (Alternately, it is possible that the selected assessment many not measure what actually differentiates high and low performing employees unless when have a sense of what the job requires when we initially select the assessment.) Another factor to consider is that an employee’s skills may change and improve as a result of on the job experience. That is, successful employees may look significantly different after several months on the job than they did when they were hired. Finally, unless we also make a systematic job task analysis, we may lose sight of the likelihood that there are several ways for someone to be successful in the job.

The take-away lesson is that while “profiling” may give us a sense of the competencies of some of our most successful employees, it is not a sufficient foundation upon which to base an assessment program. Unless we take additional steps to fully understand the requirements of a position, we are likely to implement a solution that is less than fully effective. In the final analysis, our task is to match individuals to a job, not individuals to each other.

Reid Klion

Evaluating Teachers

Reid Klion — Wed, 18 Jan 2012 16:26:13 GMT
I just saw another round of discussions in the press as to whether student scores on standardized tests should be used to evaluate teachers. The debates often devolve into all-or-nothing propositions such as “yes, we need to fire teachers whose kids do poorly on tests” vs. “only a teacher can evaluate another teacher and test scores don’t mean much of anything.” This also takes place in a context where some schools devote very substantial amounts of time “teaching the test” and searching for ways to improve test scores (which in a few unfortunate cases involve cheating). While some of this reflects the irrationality that often seems to emerge whenever public education policy is discussed, it also highlights the larger issue of what is known as the “criterion” problem in I/O psychology.

A topic I have written about previously, we face a major challenge (which is often ignored) in measuring work performance in a meaningful way, especially for higher level positions. We often lack good performance metrics and even these can be contaminated by a number of external factors. For example, sales revenue often reflects a number of things such as territory, market served, product mix, and tenure in addition to sales ability. (Also, consider the complexity of using student test scores when it is not uncommon in some urban districts for up to a quarter of children to change schools at least once during the course of a year.) As a result, we often need to use performance ratings which are prone to “halo effects” (e.g., if someone is liked, they are rated high on all attributes) and a lack of differentiation amongst employees (that is, other than a few exceptions, pretty much everyone tends to be highly rated).

In the case of education, it certainly seems reasonable that student test scores be considered as one factor in teacher evaluation but also indicators such as peer ratings, student input, and repeated observation by supervisors also taken into account. Just as we often advise that a single test score should not be used to ultimately determine whether someone should be hired or not, similar logic should apply the evaluation of job performance.

Reid Klion

ATP Program Posted

Reid Klion — Thu, 12 Jan 2012 15:22:44 GMT
The Association of Test Publishers (ATP) will hold its annual Innovations in Assessment Conference on February 26 to 29 in Palm Springs, CA. The program is now posted and registration information available.

Reid Klion

Requiring HS Diploma May Violate ADA

Reid Klion — Mon, 09 Jan 2012 20:43:18 GMT
In an informal discussion letter issued late last year, the EEOC outlined how requiring a high school diploma may lead to issues with the Americans with Disabilities Act. The EEOC points out that if an employer requires a high school diploma and that requirement removes from consideration someone

who is unable to graduate because of a learning disability that meets the ADA’s definition of "disability," the employer may not apply the standard unless it can demonstrate that the diploma requirement is job related and consistent with business necessity. The employer will not be able to make this showing, for example, if the functions in question can easily be performed by someone who does not have a diploma.

This clearly follows the long held principle that selection criteria must be based upon a “business necessity.”

Additional, since the ADA is involved, employers also need to consider the principle of “reasonable accommodation.” Here, the EEOC goes on to say:

Even if the diploma requirement is job related and consistent with business necessity, the employer may still have to determine whether a particular applicant whose learning disability prevents him from meeting it can perform the essential functions of the job, with or without a reasonable accommodation. It may do so, for example, by considering relevant work history and/or by allowing the applicant to demonstrate an ability to do the job’s essential functions during the application process. If the individual can perform the job’s essential functions, with or without a reasonable accommodation, despite the inability to meet the standard, the employer may not use the high school diploma requirement to exclude the applicant.

None of these are new principles or concepts. However, it is important remember that whenever screen out criteria are used for job selection purposes, it is critical that they be job-related and care must be taken that they do not take actions that unduly discriminate against individuals with disabilities without consideration of reasonable accommodations.

Reid Klion

Happy 2012!

Reid Klion — Wed, 04 Jan 2012 20:31:36 GMT
Best wishes to everyone for the New Year. December 31 is a common deadline for college applications. While the process of applying to college has gone on for many decades, it has also evolved. A much reviled part of the process is completing the typical standardized tests, primarily the SAT and ACT. There have been lengthy debates about the benefits and drawbacks of standardized tests to guide admissions decisions. However, virtually all colleges and universities use them because they provide an objective and common ground for evaluating potential students whereas letters of recommendation, GPA, and access to advanced classes can vary quite dramatically from school to school and community to community.

Before the advent of standardized testing, universities wrote their own admissions tests. A colleague recently sent me a link to Harvard’s 1869 admissions exam which included heavy doses of Greek and Latin as well as history, geography, and math. Questions included specifying the source of the Danube, Volga, Ganges, and Amazon rivers as well as proving that “the perpendicular from the center of a circle upon a chord bisects the chord and the arc subtended by the chord.” While the current regimen of ACTs and SATs is admittedly less than pleasant, it is very likely a major improvement from the very content specific exams that were used before their advent.

Reid Klion

IPAC Call for Papers Now Open

Reid Klion — Mon, 19 Dec 2011 20:43:03 GMT
The IPAC Call for Proposals has been posted and will be open until the end of January 2012 for its annual conference to be held July 22-25, 2012 in Las Vegas.

IPAC is an organization of applied personnel selection and assessment professionals to which a number of us at pan belong. IPAC focuses it efforts on increasing the awareness and application of empirically-based methods to HR practice. In addition to an annual conference, IPAC offers webinars as well as white papers on topics related to assessment and selection.

Reid Klion

Predicting Strategic Thinking in Executives

Reid Klion — Tue, 29 Nov 2011 21:18:30 GMT
There was a recent study published in Personnel Psychology looking at the development of executive leaders. The research question examined the relative roles of cognitive ability, personality factors, and work experience in predicting strategic thinking (which was measured by a series of five assessment center exercises). Based upon a sample of over 700 executives at an international consulting firm, it was found that cognitive ability (incidentally assessed by the Watson- Glaser Critical Thinking Appraisal and the Wesman Personnel Classification Tests, both published by Pearson and available on pan) was by far the best predictor of strategic thinking. Second most important was work experience. More specifically, it was found the broader the experience, both in terms of range of work activities and level of leadership, the more beneficial it was. The personality factors, openness to experience and extraversion, also made statistically significant contributions but to a much lesser degree. The takeaway lesson from this study is that if you are looking to develop individuals who can become strategic leaders, it is most important to find the brightest people and then provide them with a broad array of diverse work experiences.

Reid Klion

"Statistics Don't Always Work Out"

Reid Klion — Tue, 15 Nov 2011 18:07:01 GMT
Last week, I blogged about how people are prone to rely upon intuitive rather than more empirical decision making strategies. Another related issue has to do with the difficulty most people have in understanding statistics and what they mean. For example, we tend to be more biased by salient events and discount the actual underlying probabilities. A classic illustration is parents who are hesitant to fly on the same commercial flight for fear they might both perish in a crash and subsequently orphan their children but not think twice about riding in the same car on a daily basis. (For those interested in the math, there were no deaths on any U.S airline carriers in 2010 while there were nearly 33,000 automotive fatalities.)

This point was highlighted this morning when I was listening to sports talk radio show on the way to work. The hosts were debating a controversial decision by an NFL coach that past weekend that led to a loss. An expert on football statistics had written that while the coach’s decision was considered risky by traditionalists, it actually had a greater chance of success than the alternative course of action based upon actual game data (47% vs. 43%). One of the hosts (a former NFL quarterback and graduate of a Big Ten university) denigrated this thinking and complained that this was “only statistics and statistics don’t always work out.” He then went on to support this with his experience in playing blackjack in that even when he made the statistically correct decisions, he would sometimes lose several consecutive hands.

Reid Klion

The Challenge of Relying Upon Intuition

Reid Klion — Tue, 08 Nov 2011 20:37:13 GMT
One of the things that things that I-O psychologists stress is the importance of using consistent decision rules when making hiring decisions. While this message is sometimes well-received, a counter-argument often heard from hiring managers who feel most comfortable relying upon their “gut feel” about a candidate (almost invariably based upon an unstructured interview). These sorts of discussion about the relative merits of reliance upon intuition as opposed to structure decision rules has gone for decades in virtually every field that involves decision making, be it in medical settings or the HR office.

Something to be aware of is the growing body of literature that indicates that our intuitive decision making can often be prone to biases we aren’t aware of. For example, medical school applicants tend to be rated lower when interviewed on rainy days and judges are found to be stricter in their sentencing right before lunchtime. In both cases, these trends were statistically significant but neither the interviewers nor judges were aware of this bias.

Daniel Kahneman, a psychologist and Nobel Prize Winner, has written extensively on this. One of his many findings is that we tend to rely upon intuition out of habit and that it requires additional deliberate mental effort to not do so. As a result, we tend to rely on intuition by “default” and rely upon this automatic decision-making unless we make a conscious effort to not do so. As a result, it is critical for us as HR practitioners to collect good objective outcome data on the hiring decisions that we make. If we don’t, not only fail to learn from our mistakes but may not even be aware of the mistakes we have actually made.

Reid Klion

On Training 360 Raters...

Reid Klion — Tue, 18 Oct 2011 15:26:36 GMT
A few postings back, I made wrote about David Bracken and Dale Rose’s article on how 360s can be best used to facilitate behavior change. One of their suggestions was to train raters on how to use the rating scales. In a follow-up piece entitled “What is a ‘5’?,” Bracken further develops this theme. His concern is that most raters use scales as they please which tends to strip them of much consistent meaning. He writes:

I have used a rating scale where “5” is defined as “role model, in the top 5-10%” that attempts to create a frame of reference for raters (and ratees) that does help reduce leniency effects. What if we defined “5” as “can’t get any better” or something equivalent to that? …. Some raters will undoubtedly ignore those directions, but rater training can help drill in the need for everyone to reconfigure their conceptualization of what optimal behavior is.

Another related approach that I have seen used successfully is educating both raters and subjects about how the results will be used. For example, I saw the entire room relax when it was stressed that the multi-rater feedback project at hand was being used exclusively for developmental purposes, that only a few individuals are expected to receive the highest ratings, and that it is highly unlikely that anyone will be exceptional in all areas. Of course, the message needs to be seen as credible for raters to accept it. However, unless there is a reasonable level of trust within the organization, it is unlikely that a developmentally-based 360s project will be of much value for any number of reasons beyond inconsistent use of rating scales.

Reid Klion

Professional Sports and Succession Planning

Reid Klion — Wed, 12 Oct 2011 15:05:38 GMT
I was listening to sports talk radio this morning and one of the hosts (a former NFL player) observed that professional sports is one of the few businesses where incumbents are expected to help train the individuals selected to become their potential successors. The typical scenario is that when new players are drafted or otherwise acquired by a team, the veterans will share advice and help them to get up to speed with the hopes of their eventually becoming contributing members of the organization. However, it is likely that if a rookie does well, he will take the job of one of the people who helped train him.

Certainly there are business contexts where incumbents will help groom their successors. However, this is usually a planned process where there is some degree of acquiescence on the part of the incumbent. In sports, however, it is rare for a player to leave an organization voluntarily. Rather, the standard situation is that someone comes along whom the team believes can do a better job and the incumbent is either relegated to the bench or simply released from the team. An interesting bit of food for thought…

Reid Klion

Job Applicants and E-mail Addresses

Reid Klion — Thu, 06 Oct 2011 18:17:03 GMT
For the past several years, the use of e-mail addresses essentially has become the default way of contacting someone. However, unlike your family name or street address, it is quite easy to obtain an e-mail address to your own liking or even have as many as you would like. In this light, I have often been struck when job applicants opt to use e-mail addresses that might be suitable for use with their friends but likely to cast them in an unfavorable light by a potential employer (e.g., luv2partyhard@mail.com or PowerSlacker@mail.com).

There was a recent study in SIOP’s journal The Industrial-Organizational Psychologist that looked at the relationship between the use of questionable e-mail addresses when applying for jobs and test data. Based upon large sample job applicants, it was found that applicants who used inappropriate e-mail addresses scored lower on measures of conscientiousness, professionalism, and work-related experience but not cognitive ability. It should be noted that while the differences on the first three measures were statistically significant, the effect sizes were not. Thus, it certainly is not suggested that candidates be screened out solely on their use of a questionable e-mail address. However, as the authors point out, it might be appropriate to view this as something as a “yellow flag” in the selection process as a potential indicator of a candidate’s potential lack of awareness of social norms and appropriate workplace behavior.

Reid Klion

Using 360s to Change Behavior

Reid Klion — Thu, 18 Aug 2011 19:59:07 GMT
I recently read a great article by David Bracken and Dale Rose (principal of the 3D Group, a longtime pan content partner) on how 360-degree feedback tools can be used to create behavior change. The entire article is worth reading but here are some key points that the authors stress are often critical in increasing the effectiveness of a multi-rater feedback project in affecting behavior change.

Item content needs to be meaningful and relevant to both raters and subject. As a result, the assessment to be used should be carefully selected and/or customized.

To influence behavior, the feedback data need to be credible. This includes using an adequate number of raters who have had a sufficient level of contact with the subject to be able to provide an accurate and meaningful picture. The authors also suggest that rater credibility is enhanced when they are selected by the subject. Additionally, it is often quite valuable to train raters to avoid the pitfalls common in these exercises—rating too leniently and rating an individual at the same level on every item.

Accountability also helps to create behavior change. They stress the importance of subjects following up with both coaches and the raters to help facilitate development. It is also important for subjects to involve their bosses when creating an individual development plan.

As before, the article is definitely a worthwhile read because it is based upon both the authors’ extensive experience and reference to the research literature on the topic. As a final endorsement, the paper also provides a good bibliography of the work that has been carried out in the area.

Reid Klion

Not Hiring the Unemployed?

Reid Klion — Fri, 12 Aug 2011 14:46:27 GMT
Last week, I had the pleasure of speaking at the SHRM Missouri Conference on the use of assessments in HR practice (and would be pleased to share the slides if you are interested; just drop me an e-mail). One of the topics that came up was the policy of some organizations refusing to consider candidates who are currently unemployed.

The logic apparently underlying this practice is that the unemployed are unqualified because their current lack of work is reflective of some personal limitation such as poor work motivation or general incompetency. This is problematic on several grounds. First, given the current economic climate where the unemployment is greater than 9%, it would seem logical that there are huge numbers of people who are unemployed due to no fault of their own. Secondly, if even if we do assume that unemployment can reflect an individual’s poor work potential (which indeed may be the case in some situations), it makes no sense to use current employment as an absolute minimal qualification. Not only is this unfair, it is also quite likely to remove otherwise highly qualified persons from the applicant pool which impairs the organization’s ability to find qualified workers. Certainly, it is reasonable to ask unemployed interviewees how they have maintained (or improved) their work skills since their last job but it seems quite misguided to dismiss them unilaterally from further consideration.

As might be expected, this practice has generated quite a bit of press, most of it being negative. Indeed, a bill was recently introduced in the U.S. House of Representatives that forbids employers and employment agencies from screening out job applicants solely because they are out of work.

Reid Klion

Multiple Mini-Interview and Medical School Admissions

Reid Klion — Wed, 13 Jul 2011 18:35:23 GMT
Based upon work started in Canadian medical schools over 10 years ago, there is an increasing use of serial brief structured interviews for admissions purposes. Called the multiple mini-interview (MME), each candidate rotates through a series of 8 interview rooms. Prior to entering the room, candidates have 2 minutes to review a non-medical scenario posted on the door that taps into non-clinical skills like critical thinking and ethics (e.g., should a doctor endorse herbal remedies if her patients fervently believe in them even though she knows they are ineffective?) and then discuss it for 8 minutes. Each interviewer is trained in the use of a standardized rating scale.

There are several motivations behind the MME. First, it has been well-documented that unstructured interviews are poor predictors of work performance and are prone to all sorts of biases, including bad weather on the day of the interview. Secondly, it has become broadly recognized that non-cognitive skills—such as empathy and the ability to work in teams—are critical success factors in a wide variety of jobs, including medicine.

Initially, it was hoped that the measures derived the MME would be independent of GMAT scores. However, subsequent research has found some shared variance between the two (which really shouldn’t be surprising given the broader I-O literature indicating that cognitive ability is able to partially predict virtually every aspect of job performance). Regardless, the MME does appear to bring value to the medical schools that use it, and its practice has spread to other medically-related fields such as dentistry, physical therapy, and veterinary science. This is also another interesting example of how a structured interviewing can be incorporated into a selection process.

Reid Klion

Hiring Overqualified Workers

Reid Klion — Wed, 22 Jun 2011 15:43:15 GMT
An interesting question that often comes up is whether an organization should hire an individual who seems to be overqualified for a position. This perhaps has come up more frequently as of late as a result of the economic downturn where people are applying for a much broader range of jobs than they might have otherwise (though it should be noted that a late 1990s lawsuit ruled in favor of a Connecticut police department that declined to hire an applicant whose scores on a cognitive exam were deemed to be “too high.”)

This issue was discussed in a recent series of articles in the journal Industrial and Organizational Psychology where a number of well-taken points were raised. First, it should be noted that "overqualification" should be viewed as something to be understood from the applicant/employee perspective because there are substantial numbers of workers who consider themselves to be “overqualified” for their current positions even though an objective review of their background would not indicate this to be the fact. Additionally, even if someone’s overall educational attainment is greater than might be required for a job, they may not be overqualified in terms of specific skills (e.g., consider the individual with an advanced degree in English seeking work as an apprentice electrician). Another factor to consider is an individual’s motivation for seeking the position in question. Given the growing number of older individuals who are employed, there is also a subset of overqualified individuals who specifically seek such positions because they want a less demanding and more flexible work role that will permit them to pursue non-vocational interests and activities.

Looking to the research literature, there aren’t much data to suggest that hiring overqualified workers is necessarily detrimental to an organization. To the contrary, it is likely that hiring such individuals may actually bring benefits. Even while overqualified workers may not have long tenure in a position (which is often cited as a fear in hiring them even though it is not well-supported by research data), they may bring outside perspectives to an organization that are quite valuable. Further, overqualified workers may provide a pool of workers who are well-suited for promotion. Finally, there are data to suggest that these overqualified individuals may do better on the job than their peers.

Reid Klion

IPAC 2011 Program Posted

Reid Klion — Fri, 20 May 2011 14:27:55 GMT
The program for IPAC’s 2011 Conference (promotional video) to be held in July 17 to 20 in Washington, DC has been posted. Presentations will cover a wide variety of topics related to assessment and selection including developments in testing methods, technological innovations, legal compliance, and leadership development. Of special note, pan’s Jocelyn Hays will speak on using assessments to identify high potential employees. As in the past, pan is pleased to support IPAC 2011 and is a Gold Sponsor of this year’s meeting.

Reid Klion

Allowing Job Candidates to Re-Test

Reid Klion — Tue, 17 May 2011 17:56:45 GMT
A question that often comes up when implementing an assessment program is whether job candidates should be allowed to re-test. In most situations, it is good policy to permit it. For example, an individual may not have been feeling well, was excessively anxious, or for some other reason wasn’t able to show the best of his or her abilities on the day of the first test. Additionally, it is often important to allow re-testing for the sake of an organization's reputation because prohibiting it will often cast it in a bad light. Indeed, re-testing is also encouraged by professional standards as long as it is done in a reasonable way that protects the integrity of the assessment program.

That being established, the question then becomes what impact does this have upon test scores? Deidra Schliecher and associates recently published a paper in the Journal of Applied Psychology that analyzed testing data from over 2000 candidates who applied for a professional-level government service position in the US. Looking across the battery of tests that were administered, scores increased by .1 to .4 of a standard deviation depending upon the test. (This is consistent with other research indicating that scores do tend to increase somewhat upon re-testing.) Complicating matters a bit, there were some sub-group differences in the rate of improvement. For example, whites improved more than African-American and Hispanic candidates (though African-Americans improved more than other groups on the structured interview.) Also, younger candidates improved more than older candidates while women improved more than men.

The end result is that re-testing (while considered a reasonable policy) can slightly increase the level of adverse impact for some groups though this will vary as a function of factors such as the selected cut score and differential levels of re-testing between groups. Of course, this study is limited by the fact that this is only for one position, and these sorts of findings can vary significantly based upon the specific population at hand. However, the takeaway message is that all assessment programs need to be monitored on an ongoing basis, not simply for compliance purposes but to ensure they continue to meet the needs of the organization.

Reid Klion

Employee Performance Metrics

Reid Klion — Wed, 27 Apr 2011 17:07:40 GMT
Collecting solid outcome data is a critical step in evaluating the effectiveness of an assessment program. Unless we have good data about employee performance, we cannot reliably or consistently differentiate between employees who are doing their jobs well from those who are not. Unfortunately, rather than using formal performance metrics, many organizations rely upon global impressions of employee performance. A typical response is "I have been running this unit for 7 years. I know who is doing good just by seeing their daily work." Unfortunately, decades of research indicate that such informal global assessments are prone to "halo" effects--meaning that if an employee is well-liked, they tend to receive good reviews while those who are not favored are judged more harshly. This becomes a problem when we need to evaluate the effectiveness of an employee selection program. Simply stated, if there are not good data on actual employee performance, we can’t determine who well the system works in selecting employees.

Indeed a recent survey by pan content partner, SHL, indicates that only half of all organizations actually collect formal metrics to support the value of their HR investments. Not surprisingly, over a third simply rely upon the belief of hiring managers that their programs are effective. Here, we see the same reliance upon global impressionistic data to evaluate the effectiveness of selection processes that is unfortunately often used to evaluate employees themselves. When we consult with clients to implement an assessment program, we also encourage them to create good metrics to evaluate their employees if they don’t already have a system in place. This not only provides critical data that will enable us to actually see how a selection process is working (and adjust it if need be), but it is also sound HR practice.

Reid Klion

Detecting Cheating

Reid Klion — Fri, 01 Apr 2011 18:33:31 GMT
The issue of cheating on tests is a rather frequent topic in the news. Unfortunately, these are often superficial accounts and don’t provide much detail. In contrast, USA Today had a recent investigative piece that looked at cheating on statewide school-based tests and the methods used to detect it.

There appear to be two basic detection approaches, and both look for test taking behavior that deviates significantly from the norm. One is simply based upon changes in scores from year to year. Specifically, when the scores for an entire grade go up very substantially from one year to the next concerns are raised. (Here, the typical criterion for a “very substantial increase” is when scores go up by 3 or more standard deviations, which is a huge difference). This is not to say that something untoward has necessarily occurred (it is certainly possible that there are some very good teachers in that grade) but it does raise concerns. This is even more the case if the grades drop substantially the following year.

Another metric is the average number of erasures. (Recall that most school-based testing is still administered with paper and pencil.) Certainly, mistakes happen when kids put an answer in the wrong row, and all the following responses are out of sequence and need to be “fixed” and realigned. As a result, the focus is on whether entire groups of students have more erasures than others, especially if most of them result in correct answers.

This is certainly not to say that cheating is widespread on school-based testing. However, it would only seem to be human nature that there may be times when teachers do things that unfairly increase student test scores, especially when these assessments are used, in my opinion often unfairly, as a primary measure of an educator’s effectiveness. Regardless, this report provides some interesting insights into how cheating behavior (which has been documented for over a millennium) is detected is school settings.

Reid Klion

IPAC 2011 Conference Registration

Reid Klion — Wed, 30 Mar 2011 14:54:48 GMT
Registration is now open for the 2011 Conference of the International Personnel Assessment Council. To be held July 17-20 in Washington, DC, the meeting promises to be of great value to anyone working in the area of selection and assessment. Guest speakers include Elaine Pulakos (PDRI), Wayne Camara (College Board) and Eric Dunleavy, Mike Aamodt, and David Cohen (all from the Center for Corporate Equality). There are also a number of pre-conference workshops on topics including how to design an assessment strategy, best practices in questionnaire design, strategies for scoring job simulations, and job task analysis. pan is again proud to be a sponsor of the conference and that Jocelyn Courtney-Hays will be speaking.

Reid Klion

Unproctored Testing: Some Data

Reid Klion — Tue, 15 Mar 2011 18:08:36 GMT
There have been seemingly countless discussions about the potential risks associated with unproctored assessment. Facilitated by the development of internet-based testing, the benefits of having job candidates complete assessments without supervision are rather clear and include increased convenience as well as cost and time savings compared with proctored assessment. As might be expected, risks associated with non-proctored assessment center around concerns about whether candidates are receiving unauthorized help in completing the test (either by consulting with others or using various tools and resources available on the Internet or elsewhere) as well as the fact that candidates are completing assessments non-standardized in settings of their choice. Unfortunately, these discussions have been almost exclusively theoretical in nature as there has been little empirical evidence to serve as a guide.

A recent study may shed some light on this question. The paper looked at the very question of how the validity of non-cognitive assessments (personality and biodata-based tools) may change when they are moved from proctored to non-proctored modes of administration. In general, it was found that these assessments largely retained their ability to predict job performance when candidates completed them without the benefit of proctoring. Of course, this is a single study (albeit a large one with one analysis involving over 1200 candidates and the other about 3300) that only involved a handful of tests. It also should be noted that personality and biodata assessments are not objectively scored and makes “cheating” on them more difficult than knowledge-based assessments. However, these findings do support the notion that, at least in some settings, test validity may not be significantly compromised when assessments are delivered in a non-proctored fashion.

Reid Klion

Test Candidate Bill of Rights

Reid Klion — Wed, 16 Feb 2011 15:39:09 GMT
Charles Handler, an I-O psychologist, assessment industry commentator, and friend of pan, recently wrote an interesting piece on “A Pre-Employment Assessment Candidate Bill of Rights” where he outlines a number of “rights” to which he believes test takers are entitled. For the sake of discussion, I will leave aside debate as to whether what are described are actually “rights” as might thought of in the same terms as the Bill of Rights or more accurately “best practices” or “good ideas.” That being said, he raises a number of provocative points, and several are worthy of comment.

Candidates have the right to an assessment experience that is of a reasonable length.

True but this is often a balancing act between collecting enough information to make a reasonable decision about an applicant’s viability as a potential employee and not overtaxing a candidate’s time Here, there can be a tension between recruiters who favor a briefer experience and I-O psychologists, tending to be a bit obsessive by nature, who will almost always opt for collecting more, rather than less, information when given a choice.

Candidates have the right to know where the assessment fits within the overall hiring process, and what they can expect next.

I couldn’t agree more that we should be transparent about the assessment process and “next steps.” This also seems important for recruitment and PR purposes since job candidates may be clients of the organizations where they are seeking work,

Candidates have the right to a good user experience.
Candidates have the right to technical support, no matter when they are applying.

Here, too, I would completely concur that assessments should employ good user interface design, be easy-to-use, and as foolproof as possible. Given the advances in web-based technologies over the past 10 years, there is little reason to not accomplish this nor to provide competent technical support if problems arise.

Candidates have the right to assessment content that appears job related.

Well, I am not necessarily convinced of this in all cases. Part of the issue is that what doesn’t appear to be job-related to candidates may very well be critical to job performance to those who understand the job. For example, someone may think that a test of cognitive ability may not be relevant to a position when it is well-known that cognitive assessments are often the best predictors of future job performance. Also, biodata assessments are often quite predictive of job performance even though the items may not necessarily appear to be job-related to an external observer.

Candidates have the right to an enjoyable assessment experience.

Another “right” I am not terribly convinced of. Our business is assessment, not entertainment. The primary goal of a testing program is to provide a fair determination of each candidate’s ability to succeed on a job. While we certainly may want to design as assessment process that is engaging, our primary goal is to provide a valid assessment of the candidate’s ability, not necessarily to provide a fun experience.

As always, Charles is a provocative writer. This article is certainly not an exception and definitely worth a look.

Reid Klion

Everything You Wanted to Know About O*NET

Reid Klion — Tue, 25 Jan 2011 22:02:08 GMT
First published in 1998 by the Department of Labor (DOL), O*NET (Occupational Information Network) is a framework and database of job characteristics for 956 occupations. It is used by a wide variety of professionals in domains including vocational guidance, education, human resources, and the determination of work-related disabilities. It is most typically used in the HR community as a data source in carrying out job analysis and writing job descriptions. (It is also worth noting that the O*NET website itself offers a broad array of information and resources including tools for matching job titles with O*NET occupations, user guides, and a variety of career exploration assessments

Given its broad use, the DOL requested in 2008 that the National Academies of Science convene an expert panel to review the O*NET and consider its future directions. The result was a document of more than 200 pages that is available as a free download. The report documents the committee’s findings and also provides a good overview of the O*NET’s background. Some the conclusions drawn are that despite its limitations, O*NET is used by a very broad array of individuals and organizations and continues to be seen as valuable resource. One major suggestion is that the DOL focus its O*NET-related resources specifically on the continued development of the occupational database itself and defer to others the development of tools and other ancillary related products. To help maintain this focus, it was further suggested that an advisory board be created to guide DOL in these efforts. While lengthy, the report provides a good overview of where O*NET has been and a vision for its future.

Reid Klion

Joint Standards Being Revised

Reid Klion — Tue, 18 Jan 2011 20:33:16 GMT
The Standards for Educational and Psychological Testing, known as the “Joint Standards” because they are a collaborative effort of the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, were last revised in 1999. Since that time, much has changed in the world of testing to include the proliferation of web-based assessment, frequent high profile educational testing in schools, and a growing awareness of the complexities associated with fairly assessing individuals with disabilities and for whom English is not a first language.

Over the past few years, a number of committees representing a broad range of constituencies were convened to develop proposed revisions to the Standards. Their work was recently posted and comments are now being solicited from both testing professionals and the public. Providing input and review are critical because the Standards (along with the SIOP Principles and EEOC's Uniform Guidelines) are considered the foundational documents that define accepted practices in world of assessment. Since the Joint Standards cover all forms of psychological assessment, they are somewhat broader in scope than the other two sets of documents, but are still critical in defining professional standards for human capital assessment.

Reid Klion

Credit Data, Hiring, and the EEOC

Reid Klion — Wed, 05 Jan 2011 18:55:12 GMT
A few months ago, the EEOC held hearings on the use of credit checks in the hiring process. It is estimated that 60% of employers use credit checks for at least some positions. This is an important topic given the increased attention that the EEOC is now paying to this issue and a recent suit filed by the EEOC against Kaplan Higher Education in this regard. It is also worth noting that the EEOC has reported an increase in the number of charges filed based upon the use of credit checks in pre-employment selection.

The testimony highlights a number of important issues:

The stipulations of the Fair Credit Reporting Act (FCRA) apply when credit data are used in hiring. The most important in this context is that if candidates are not being hired because of their credit report, the employer must inform them of this and provide a copy of the report. This often will lead to a discussion between the organization and candidate about any mitigating factors and also provides the opportunity for the consumer to correct the record with the provider of the data if there are inaccuracies. The FTC recently created a summary of the implications of the FCRA for job candidates.

The credit report obtained by employers contains applicant credit history but not his or her credit score. As a result, there is not a single summary score for employers to use. Rather, they need to draw their own conclusions from the provided data.

Some protected classes (e.g., women, Latinos, African Americans) tend to have lower credit scores than other groups. As a result, credit data will tend to have adverse impact when used for selection purposes.

As always, though, a selection method that has adverse impact can be used defensibly if meets a “business necessity.” For example, the federal government often uses credit checks in the hiring process for certain positions of trust. However, when an employer opts to use a process that has adverse impact, it is much easier to defend if there are not alternative ways to obtain the same result with a lesser degree of disparate impact.

Michael Aamodt, an I-O psychologist (who will be an invited speaker at the 2011 IPAC meeting), provided some very useful data about the validity of credit data in pre-employment selection. Unfortunately the research base is limited, and he could only identify 10 studies in the literature (with five using self-reported financial distress v. credit data and three of these involving law enforcement positions which may not generalize well to other jobs). Based upon his meta-analysis, he found an uncorrected correlation of .14 between credit history and counter-productive work behaviors, .07 with performance ratings, and .20 with attendance. These correlations are lower or at best equal to those found when personality assessments are used. However, the literature is quite limited on the question.

To my way of understanding the situation, credit data can definitely play a useful role in the hiring process for certain positions but, like any selection “test” (and this would be considered from a legal perspective to be a test just like a psychometric or physical ability assessment), it needs to be job-related and meet a business necessity. If a position requires an individual to have good creditworthiness (for duties such as handling money or being in a position of trust), it would seem quite reasonable to consider the use of credit data. However, if the credit data are being used to make other inferences about the applicants (such as whether there are conscientious or will be good workplace citizens), personality-based assessments are either as or more effective in doing so, have a track record of not having adverse impact, and are likely the best option to first consider in such a situation.

Reid Klion

IPAC Call for Papers

Reid Klion — Tue, 04 Jan 2011 14:34:19 GMT
The Call for Proposals for IPAC’s 2011 Conference on Personnel Assessment (to be held July 17-20 in Washington, DC) is now open and will be until the end of the month. The program committee is open to proposals for symposia, panel discussions, tutorials, and workshops on topics related to talent management. More information on the conference is also available. Invited speakers include Wayne Camara (College Board), Elaine Pulakos (PDRI) and Eric Dunleavy, Mike Aamodt, and David Cohen (all of the Center for Corporate Equality which was blogged about earlier). A number of us at pan have been active in IPAC for the past several years and the conferences are notable for their conceptual rigor, practical applicability, and relaxed atmosphere.

Reid Klion

Civil Service Exam in China

Reid Klion — Fri, 17 Dec 2010 19:51:14 GMT
The Chinese Civil Service exam was administered to nearly over a million job candidates in early December and the topic of post last year. The National Public Servant Exam is required for hiring into many government jobs. Civil services examinations in China have a very long history and have always been the focus of much attention going back several centuries. Given the recent difficulties that young Chinese college graduates are having in finding work, it is a bit surprising that some 40,000 fewer people took the exam this year than last. Regardless, 1.41 million people applied for the exam, and 1.03 million actually took it. Since there are only 16,000 government jobs reported to be available at present, that means the selection ratio is 1:64.

Reid Klion

"Elective" Use of Testing

Reid Klion — Wed, 01 Dec 2010 20:29:00 GMT
In a rather unique application of testing, political candidates in Australia are being asked to complete a formal testing process. Apparently due to the large number of candidates interested in running for parliament, the Queensland Liberal National Party is requiring its hopefuls to complete an assessment processes. As in more typical situations, the party’s goal is to select the best possible candidates from a large group of individuals who are interested in holding office. While the specific tests to be used are not specified, news reports indicate they are assessments typically used in corporate hiring situations and look at competencies such as problem solving and negotiation. At this point, only individuals who are newly running for office, but not incumbents, are required to complete the assessment.

Reid Klion

To Re-Test or Not: That is the Question

Reid Klion — Thu, 04 Nov 2010 20:27:36 GMT
A question that is often heard in regard to assessment is whether a candidate should be allowed to re-test. A recent article by John Hausknecht may shed some light on the issue. The paper looks at an analysis of a large sample of job applicants in the hospitality industry who took a personality assessment as part of the selection process. First, it should be noted is that candidates who re-test are those individuals who were not hired initially but opted to persist in attempting to get the job. Also, a large proportion of the persons in the study sample who re-tested were internal candidates, meaning they were already employed by the organization; as a result, they may have had some insights into the purposes of the testing process.

Regardless, the re-testers were able to increase their scores to a moderate degree, especially on the scales which were initially quite low. (This is a somewhat different from other studies that looked at individuals who took the same assessment when hired and then later for developmental purposes; in these situations, scores tend to stay fairly stable.) The takeaway message from this study is that scores on personality assessments can increase if job candidates are permitted to re-test. Accordingly, organizations should take this into account when crafting re-test policies and consider using assessment models that may help to limit a candidate’s ability to increase his or her re-test score by implementing assessments with multiple forms or considering the use of assessments that control for socially desirable responding.

Reid Klion

Using Assessments to Verify Job Candidate Qualifications

Reid Klion — Thu, 21 Oct 2010 16:46:17 GMT
One of the major benefits of using an assessment is to corroborate an individual’s qualifications for a position. As noted in a prior posting, it is unfortunately the case that job candidates may exaggerate their academic or professional experiences or present (or even purchase) credentials from schools that are less than reputable. The result is that assessment results (if the program is appropriately administered) may provide one of the few solid sources of information that a potential employer has about an applicant.

The importance of verifying an individual’s qualifications was highlighted in a recent article about the proliferation in China of fake credentials within the professional ranks and widespread plagiarism in scientific journals (which is important in academic settings where importance placed upon publication record). For example, there are individuals who claim to have medical degrees (who only took a correspondence course) and the recent case of a Chinese airline that discovered over 100 of its pilots had embellished their flying histories. A complicating factor in China is widespread cheating on tests which only highlights the importance of using good testing practices in high-stakes examination programs. While there are no easy answers, the use a good assessment program can play a critical role in evaluating job applicants by providing reliable and verifiable information about a potential employee.

Reid Klion

Presentation on Job Analysis

Reid Klion — Mon, 18 Oct 2010 17:44:55 GMT
A number of us at pan are active in IPAC--the International Personnel Assessment Council. In addition to sponsoring an annual conference and supporting a newsletter and listserve, IPAC also sponsors webinars on topics related to selection and assessment. Julia Bayless, IPAC’s President Elect and Director of Talent Development at Sodexo, recently gave a great presentation on job analysis. She did a particularly nice job of covering both the conceptual issues involved in job analysis as well as the practicalities and challenges often associated with actually getting one done in the field. For a limited time, both a recording of the presentation and the slides are available to the public.

Reid Klion

College Students, Cheating, and the Role of Personality

Reid Klion — Tue, 12 Oct 2010 18:37:27 GMT
Cheating on tests seems to be fairly widespread with some estimates indicating a lifetime prevalence of 80%. While an interesting statistic, it is important to realize this is based the admission of having been academically dishonest at least once during a school career. Given how virtually everyone is in school for at least a decade (and some, almost two), I am not sure this is terribly surprising. As a result, it has often been thought that it would be difficult to identify the personality correlates of a behavior with such a high base rate (and is also most typically based upon self-report).

In an attempt to clarify the issue, a recent study from the University of British Columbia examined the personality correlates of individuals who engaged in plagiarism on a writing assignment. Using a commercial tool employed by professors to detect cheating on essays, they were able to identify those essays (about 15%) which had a high degree of overlap with other essays that had been submitted or materials that can be found on the internet. (It should be noted that plagiarism may differ from other forms of cheating because it is premeditated as opposed to copying on a multiple-choice test which may be unplanned and impulsive.) They found that the authors of plagiarized essays had higher scores on a measure of what they call the “Dark Triad” (Machievellianism, narcissism, and psychopathy) than did other students.

Reid Klion

Guidance on Adverse Impact Analysis

Reid Klion — Thu, 23 Sep 2010 18:35:47 GMT
Whenever an assessment program is implemented, it is critical to understand the extent to which it may have adverse impact. (To review, adverse impact is defined as a “substantially different rate of selection in hiring, promotion, or other employment decision which works to the disadvantage of members of a race, sex, or ethnic group.” For example, women tend to have lower scores on tests of physical strength.) The basic rule is that if the selection rate for a protected group is less than 80% of that for the majority group, adverse impact is seen as being present.

However, the practicalities of actually calculating these statistics requires a number of decisions upon which there is little guidance but can have a substantive impact upon the final results. In order to address this issue, the Center of Corporate Equality created a technical advisory report based upon the input of 45 experts in the field. Some of best practices outlined include reinforcing the difference between a job seeker and a job applicant (because seekers are not included in EEOC analyses), not “guessing” the race or gender of an applicant who declines to self-identify, not counting as applicants those job seekers who either withdraw or fail to follow-up, and that EEOC statistics should be calculated for each sub-group individually (with the practice of combining them all into a “total minorities” group not being recommended). There is also a lengthy section on the statistical methods that can be used in adverse impact analyses.

The document is long and technical in spots (to be expected given its very nature) but it provides a good reference and source of guidance in an area where it is much needed.

Reid Klion

Rainy Days and Monday (and Getting into Medical School)

Reid Klion — Fri, 03 Sep 2010 15:40:46 GMT
The interview has a very well-established role in virtually every selection situation—be it for a job or getting into school. This is despite the fact that it has been well-established that unstructured interviews tend to have very limited validity. (However, in the interest of full disclosure, I would not be comfortable in hiring someone without interviewing him or her first.)

One of the factors that contributes to the limited efficacy of most forms of interviews is that there can be a number of subjective factors that influence how interviewees are rated that have no relationship to their actual qualifications. A good illustration can be found in a study looking at the impact of weather upon medical school admissions interviews. By looking at the records of nearly 3000 candidates at a major Canadian medical school, it was found that interview ratings were lower on rainy as opposed to sunny days. While the rating difference was only 1%, this still had an impact upon admission decisions because a substantial number of candidates obtain scores that cluster near the cut score. The result is that the chances of getting into medical school decrease for borderline candidates if they happen to interview on a bad weather day.

Reid Klion

Throwing the Baby Out With the Bathwater

Reid Klion — Mon, 30 Aug 2010 15:07:38 GMT
Chicago has a long history of difficulty with its public safety testing programs. For example, there was a Supreme Court ruling last year that allowed litigation to continue in relation to a fire department assessment that was administered in 1995. Now, there are reports that Chicago may do away with its police testing program to save money, avoid the potential for litigation, and boost minority hiring.

It is difficult to understand how this is a good business decision. Not using a test certainly does not ensure against litigation because any process used to evaluate a job candidate is legally considered to be a “test” and subject to the same legal considerations as a formal psychometric assessment program. Additionally, other cities have managed to develop public safety recruitment and selection programs that have yielded a diverse police force. Finally, and perhaps most importantly, a well-designed assessment program is the fairest and most effective means of selecting job candidates. Removing the objective data from the decision making process and simply depending upon the subjective review of job applications is likely to result in more difficulties and expense in the long run than would developing a validated testing program when the costs of a poorly selected police force are considered. Additionally, not using a formal testing program would make Chicago unique amongst major city police departments.

Reid Klion

Reconisdering Vocational Interest in Selection

Reid Klion — Fri, 20 Aug 2010 18:15:40 GMT
I heard a presentation the other day by Dan Putka of HumRRO about the role vocational interest in job selection. To this point, the received wisdom has been that vocational interest has little value in predicting job performance after the Big Five personality factors and cognitive ability are taken into account. While counter-intuitive, this finding has been replicated repeatedly in the literature and has become quite well-accepted.   Putka and his colleagues hypothesized that part of this may be due to the fact that most measures of vocational preference were developed to measure occupational interest (e.g., what are the jobs that match my interests) and not designed for selection (e.g., what jobs do I want to do). For example, while some may enjoy fixing things around the house, they may not necessarily be interested in an occupation that involves this.

Based upon this assumption, they created a job interest inventory for selection that targets the sort of work an individual would be interested in having. Using in a very large scale study with the military, they found that job interest was more predictive of both intent to stay in a position and job knowledge than were either personality factors or cognitive ability. They hypothesize that while initial performance may be better predicted by other factors, whether an individual will have a long-term emotional investment in a job (as evidenced by the desire to stay in the job and learn more about it over time) are better predicted by interest. The article is currently in press at the Journal of Applied Psychology.

Reid Klion

Pass the Test, Get a Loan

Reid Klion — Tue, 10 Aug 2010 17:27:18 GMT

There was a recent article in Businessweek about the use of assessments to help banks in developing countries to identify those individuals who are likely to be good candidates for loans. The assessment looks at traits such as integrity, motivation, and intelligence. It is reported to work as well as (or better than) traditional methods of assessing creditworthiness with claims it can reduce default rates up to 40%.

I have heard of other organizations that have developed assessments for similar purposes. There is little reason to think such an approach would not have some measure of efficacy, especially in developing nations where most individuals have little to no exposure to formalized banking which makes credit history difficult to obtain. As such, psychometric assessment data may be one of the few pieces of objective information available upon which to base a loan decision. While testing loan applicants in the U.S. would likely run afoul of a number of a legal, business, and cultural barriers, it is still an interesting concept to ponder as well as a rather unique example of the innovative use of assessments.

Reid Klion

A New Perspective on "Faking" and Personality Assessments

Reid Klion — Tue, 03 Aug 2010 21:30:02 GMT
The issue of “faking” or impression management on personality inventories is a fascinating topic. It was recognized as a potential concern in the 1930s, and has both led to nearly 1000 publications and (in the words of Robert Hogan) “consumed the best minds of the discipline.” Despite this attention, the field has not reached any definitive conclusions other than the recognition that personality assessments can be highly effective tools for personnel selection when used in the right context even though some candidates very likely exaggerate their positive attributes while minimizing the negative.

A recent paper by Bernd Marcus, a German social psychologist, recently cast this in a new light. Marcus argues that we need to understand the applicant-employer dynamic from the applicant’s perspective. Some of the points he makes are that both the applicant and employer are engaged in goal-directed behavior. The applicant wants to induce the employer to hire him or her while the employer’s task is two-fold: to evaluate the candidate but also encourage the applicant to take the job if offered. Essentially, both parties are in the process of negotiating a long term relationship (perhaps not dissimilar to dating). As a result, it is no surprise both will be on their best behavior. In this vein, just as applicants are unlikely to disclose much negative about themselves, this is also the case for the employer. For example, how often does a potential employer openly share things like the reason why the position is being filled, concerns expressed by incumbents about the heavy workload associated with the job, or disclose other negative factors about the organization or position in question?

Reid Klion

Needing to Base Hiring Decisions on Something

Reid Klion — Tue, 27 Jul 2010 16:49:43 GMT
While those of us who work with assessments are well-aware of their limitations, research data have consistent indicated that well-validated personality and cognitive assessments are almost always the best predictors we have of future occupational success. This has been established repeatedly in the professional literature for the past 15 years.

While assessments are obviously not foolproof, the critical underlying issue is that we need to make hiring decisions based upon something. And in most cases, that best something is an assessment. In making this statement, we need to consider what the alternatives are. This issue was recently highlighted for me in a recent series of articles about GPAs which employers often consider when hiring recent graduates.

For example, there are now high schools that name up to 30 valedictorians. Traditionally reserved for the student with the highest GPA, it is now seen an honor to be shared by all “top” students. Apparently, many schools give extra credit for Advanced Placement classes so a 4.0 might be posted as a 4.5. Also, since some schools will cap GPAs at an arbitrary level, say 4.3, the result can be several students who all appear to have achieved a perfect record. All that being said, while there may be sound reasons for naming multiple valedictorians, it obviously no longer represents the academic distinction it once did.

In a related development, some law schools are now increasing their students’ GPAs retroactively. For example, a B- may become a B overnight, even though the student may have graduated some time ago. The rationale is that some law schools grade on a curve (e.g., only permitting a fixed percentage of 4.0s to be assigned) whereas others do not or do so less stringently. The more stringent schools, being concerned that potential employers may not be aware of this, have taken the step of increasing all their students' GPA by .33--even if they graduated five years ago. The point is not to argue the propriety of a school’s adopting this policy. Rather, it highlights the fact that when compared to other data that we have available, a well-validated assessment program often provides the fairest source of data about a potential employee’s suitability for a particular position within an organization.

Reid Klion

Colleges and Cheating

Reid Klion — Wed, 07 Jul 2010 14:48:26 GMT
While most of us tend to focus on testing in HR settings, we sometimes forget that the people who probably administer the most tests work in education, both at the K-12 and university level. While likely not a surprise, cheating can be a major concern in these contexts. For example, the University of Central Florida developed a specialized testing center to reduce the incidence of cheating. Amongst the measures used is an array of cameras monitoring student behavior and video monitors recessed into desks so that any efforts to take pictures of the test will become evident. Even gum chewing is prohibited for fear that it could be used to disguise a student’s efforts to speak into a recording device or phone.

While certainly interesting from a test security perspective, one also wonders about the message this sends to student about how they are viewed by the university. This point was made salient to me last week when I was on a college tour with my daughter at a couple of institutions which have honor codes. For example, students at Harvey Mudd College take their exams without supervision in the setting of their choice. As the walking-backwards-while talking student tour guide explained, professors simply distribute exams to students with instructions about how long they have to take the test and what reference materials are permitted. While the setting prevented my asking many probing questions, the faculty is apparently content with this approach as it has been in place for some time. “Take-home” testing is also the norm at other institutions such as CalTech which uses a similar honor code model.

Reid Klion

Lie Detection and Pre-Employment Selection

Reid Klion — Wed, 16 Jun 2010 15:01:06 GMT
The National Security Administration (NSA) recently posted a video attempting to quell applicant anxiety about taking a polygraph test as part of its pre-employment selection process. Whether this has the desired impact or not remains to be seen. However, the use of lie detectors for job selection has a long and interesting history.

The "father of polygraph" was a Harvard psychologist named William Marston (who later went on to fame and fortune as creator of the “Wonderwoman” comic book character). It was thought that the truth of a candidate’s responses could be determined by measuring changes in blood pressure, breathing, and other physiological indices. Unfortunately, data supporting the validity of lie detectors has never been strong, and a federal law passed in 1988 largely prevented the use of polygraphs for pre-employment screening other than by the government in the case of national security, defense or law enforcement. (It is also interesting to note that the development of “integrity” or work attitude tests in the late 1980s was spurred by this law and the desire to find more effective ways to select good employees.) While the polygraphs’ proponents acknowledge its limitation, they support its use continued as an interrogation aid in encouraging individuals to reveal information that they may not have otherwise.

Reid Klion

Multi-Tasking: A Myth?

Reid Klion — Mon, 14 Jun 2010 17:37:35 GMT
Our days at work are often filled with a series of tasks that complete for our attention including e-mail, phone calls, cell phones, and instant messaging. Not surprisingly, recent data indicate that people working at computers change tpiawindows or check their e-mail almost 37 times an hour. Indeed, there are growing concerns about how this may impact brain functioning as well as increase work stress levels.

A related issue is whether this is a productive way to work. Fortunately, psychologists have started turning their attention to this question. Researchers at Stanford University compared individuals who were heavy vs. light multi-taskers. From the outset, my expectations were that people who multi-task often would become quite adept at working in this fashion. To the contrary, it was found that frequent multi-taskers were more easily distracted by irrelevant stimuli and actually did worse at multi-tasking than were their peers who did it less often.

If you are curious about your own abilities to attend and multi-task, here are samples of the tools used in the study to assess attention and ability to shift between tasks.

Reid Klion

One Word Essay Question Prompts

Reid Klion — Tue, 08 Jun 2010 14:28:19 GMT
When it comes to human capital assessment, the vast majority of the tests that we use employ some variant of a multiple choice format where candidates are asked to select from response options that we provide. However, there is another type of test, called constructed response, which is primarily used in educational settings where the test taker is asked to provide a written response to a prompt. This approach has even come to be used in many of the standardized tests that have become quite popular in K-12 settings with the advent of the No Child Left Behind Act.

Apparently, All Souls College at Oxford University in the UK had been using a similar approach for admissions testing. Here, candidates were provided with a single word prompt (like “water,” “novelty,” or “death”) and then asked to write about it for 3 hours. It was decided recently, though, that the data obtained from the test were not as useful as previously thought, and this particular test format was dropped as a result. However, in case there are concerns that All Souls is lowering its standards, candidates still need to write essay questions on topics which include:
•   Why is a leather jacket more acceptable than a fur coat?
•   Can computers think?
•   How European was Chaucer?

Reid Klion

Non-Proctored Testing Poorly Done

Reid Klion — Tue, 25 May 2010 18:02:19 GMT
I have blogged and presented several times on unproctored internet testing (UIT). As most are familiar, it can be a powerful tool but it also needs to be done in a way that mitigates the risks inherent in having an individual complete a test without supervision. Here is a situation where Transport Canada wanted to ensure that powerboat operators were aware of the basics of watercraft safety. Apparently, anyone wanting to obtain the required “proof of competency” card needed to complete an unproctored online test. To facilitate access, a number of companies were sanctioned to provide the test but many didn’t follow prescribed guidelines in doing so. Unfortunately, there were also major security issues and answers to the questions were widely available on the internet. Additionally, while verification can always be an issue with UIT, matters were made worse from an enforcement perspective because photographs do not appear on the competency cards, and there is not a central registry of individuals who passed. As a result, a number of individuals have criticized the program as having little value. While officials do acknowledge the need for improvement, they also believe that the program has increased overall safety knowledge amongst boaters.

Reid Klion

Backlash Against Kindergarten Admissions Tests?

Reid Klion — Wed, 12 May 2010 21:14:49 GMT
A while ago, I blogged about the growth of “boot camps” for pre-school admissions tests but there now seems a growing backlash against the use of standardized testing programs for the kindergarten set. While the agency which sponsors the exam indicates that test scores have stayed stable, there are concerns about the validity of the test due to the growth in popularity of prep classes. Also, some schools have become concerned about the importance which has been placed on the testing program itself. However, it seems that these issues will continue for some time given the competition for placement in prestigious elementary schools in Manhattan and the overall social and cultural atmosphere. As one mother says: “We’re in a competitive city,” she said. “People want their kids to excel, and this is a differentiating factor. You can help.”

Reid Klion

Emotional Intelligence and Work Functioning

Reid Klion — Thu, 22 Apr 2010 21:25:29 GMT
Emotional intelligence (EI) is a topic that has garnered substantial interest over the past several years, especially in the business literature. It should be noted that EI has not been without its critics due to concerns about how the concept itself has not been well-defined and the fact that several measures which purport to measure it are not necessarily well-correlated with each other.

A recent study looked at how well measures of EI might be able to predict work functioning. Some of the interesting findings were that personality and performance-based measures of EI are not highly correlated with each other, that EI is more highly correlated with measures of personality than overall intellectual functioning (which argues against the proposition that EI is actually a form of intelligence), and that measures of EI contribute little new information in terms of predicting job performance after personality and intelligence are accounted for. This is not necessarily to say that measures of EI are not useful, especially in coaching or in a developmental context. However, since EI tends to have such a substantial overlap with measures of personality, it often adds little new information in selection settings. Additionally, since the pattern of personality traits which predict job success typically will vary from position to position, measures of EI, which are comprised of an agglomeration of personality traits, may not be particularly useful when applied across a diverse set of jobs simply because the requirements for those jobs may differ rather substantially. Thus, a high level of EI may predict performance in one job type but not another.

Reid Klion

Reasons to Consider Using Assessments

Reid Klion — Fri, 02 Apr 2010 15:17:36 GMT
Charles Handler, president of Rocket-Hire and a friend of pan, almost always has something interesting to say about the world of assessment. He wrote a recent piece about the factors that should lead an organization to consider using psychometric assessments that is worth a read.

One he identifies is that if the organization relies heavily on unstructured interviews. While many are very much wedded to them, the research data are fairly clear that unstructured interviews tend to have little if any predictive validity. As a result, using a psychometric assessment that is only modestly predictive of occupational outcome will yield better results than the typical interview. Other factors he mentions include organizations with either high turnover or high volume hiring situations. In both these instances, a well-validated assessment will almost inevitably increase the efficacy of the hiring process because its impact will be readily apparent given the current state of affairs. Finally, one issue that he also notes is a topic I blogged about a few days ago. That is, regardless of how an organization selects employees, its selection processes are legally considered to be a test. As a result, if an unstructured selection process is being used that isn’t based upon a job analysis, a properly implemented psychometric testing program will almost inevitably decrease an organization’s legal exposure (as well as very likely improve the quality of its hiring decisions).

Reid Klion

ATP Model Guidelines for Preemployment Integrity Testing

Reid Klion — Fri, 26 Mar 2010 13:47:14 GMT
Association of Test Publishers recently released the third edition of its Model Guidelines for Preemployment Integrity Testing. Integrity tests are assessments that have been specifically designed to predict counter-productive workplace behavior such as theft, poor productivity, and substance abuse and are typically used for screening candidates for entry level positions. While the Guidelines obviously address issues around integrity tests, they also provide a good overview of best practices in pre-employment assessment use as well as review of the empirical work on the validity of integrity tests.

Copies are available from the Association of Test Publishers; Lauren Schieb should be contacted for details.

Reid Klion

Hiring, Testing, and Legal Compliance

Reid Klion — Mon, 22 Mar 2010 15:15:54 GMT
A question we often hear about assessments has to do with the legal issues which surround their use. A critical point that sometimes gets ignored in these discussions is the fact the same legal principles pertain when any method is used to select employees, whether testing is part of it or not. A good example is a recent EEOC suit that Walmart settled for nearly $12 million due to gender discrimination in hiring distribution center personnel for a facility in Kentucky. Essentially, based upon the assumption that the positions were not suitable for women, Walmart only selected men for these jobs.

In contrast to most testing cases, this did not involve adverse impact which occurs when the differences in selection ratios between groups are due to inadvertent factors. (For example, if Walmart had used a physical ability test with a cut score that was higher than necessary given the demands of the job, this would have been a case of adverse impact because it would have eliminated a large proportion of female applicants.) Rather, this was a case of gender discrimination where women simply weren’t considered. However, the critical point is that the same EEOC stipulations apply to how employees are selected and hired--whether tests are used are not.

Reid Klion

Is There Score Inflation When Cognitive Assessments Aren’t Proctored?

Reid Klion — Thu, 18 Mar 2010 13:46:20 GMT
Winfred Arthur and colleagues recently published a paper that looked at the extent to which job candidates may try to cheat on cognitive assessments. Since cognitive assessments are objectively scored as either right or wrong, job candidates can try to improve their performance by doing in things such as seeking the assistance of others in responding to items or accessing items from those who have taken the test previously. In order to explore this, the authors obtained test data from several thousand job candidates who had completed a cognitive assessment as part of a hiring process (a high stakes situation where it is assumed there would be motivation to optimize one’s performance) and then asked them to repeat the test a second time as part of a research study (where the stakes are assumed to be much lower). If cheating were widespread, we would expect these scores to go down when the test were repeated when the stakes were lower and there was not motivation to obtain a higher score. In contrast, they found that scores actually went up in the low stakes setting when the candidates repeated the test. (It is likely that this increase was due to a practice effect.)

In terms of drawing conclusions, it should be noted that this study used a timed assessment which likely decreases the opportunity for job candidates to seek assistance during a test session as well as the fact that a relatively small sample of subjects agreed to complete the assessment a second time. However, this finding is quite consistent with other research which indicates that the validity of most assessments is not significantly compromised when they are administered in unproctored settings.

Reid Klion

More on Credit Checks and Hiring

Reid Klion — Wed, 03 Mar 2010 19:11:48 GMT
As a follow-up to the posting a few weeks about credit checks, there are at least 5 states that are challenging the use of employer credit checks for selection. The legislators involved claim that their actions are spurred by the perception that employers are using credit checks inappropriately. If any of these laws are passed, each one would need to be examined carefully to understand its implications, and this would need to be done on a state-by-state basis.

However, it is important to remember that credit checks are a legally defensible selection method if creditworthiness is indeed a job requirement. For example, applicants for high risk positions, such as intelligence work, are often not hired if they have substantial levels of debt because this is seen as increasing their risk for bribery. However, employers do need to take a second look when they use credit data as an indirect predictor of other behavior such as trustworthiness. While there are a few studies that indicate some relationship between these two factors, there are several well-developed psychometric assessments that have been designed to measure integrity and trustworthiness directly. Additionally, unlike credit data which is often prone to adverse impact, there is a substantial body of literature indicating that integrity tests are not. Here, like in any assessment situation, it is important to first have a clear understanding of what the requirements for a position are and then finding the assessment strategy that is most likely to meet the organization’s goals which limiting unnecessary risks.

Reid Klion

Certification Body Wins Lawsuit Against Cheater

Reid Klion — Mon, 22 Feb 2010 16:23:19 GMT
National Council of Examiners for Engineering and Surveying (NCEES) won a civil lawsuit against a civil engineering candidate who attempted to take one of the organization’s primary exams with wireless a/v transmitter, video camera, receiver, pocket video recorder, and battery packs all on her person (see picture) with the intent to steal the content of the exam. She apparently had done this previously in 2005 but was caught by an alert proctor in the most recent incident. The federal court awarded NCEES a judgment of over a million dollars for loss of intellectual property. This figure was determined by estimating the cost that is involved in developing the items that had to be discarded. This individual was previously convicted of fraud in a criminal court for charges related to the incident.

I suspect that judgments of this type and size are rare but they do highlight the costs that can be involved in the development of sophisticated testing programs.

Reid Klion

Credit and Background Checks in Hiring

Reid Klion — Mon, 15 Feb 2010 22:22:58 GMT
Here is an update on the EEOC’s E-RACE (Eradicating Racism and Colorism from Employment) Program. Ongoing for the past several years, E-RACE focuses on race and color discrimination in the workplace with a particular focus on employment screening methods (such as background and credit checks) that are prone to have adverse impact. Here, it is important to recall that adverse impact occurs whenever a seemingly neutral “test” unintentionally results significantly lower selection rates for protected groups of persons. (A classic example is that if the physical ability requirements are extreme, many women will be disqualified from certain labor jobs.)

Since credit and background checks are quite susceptible to having adverse impact, businesses must be sure that their use is job related (just as they must for any assessment method that has adverse impact). Not only is this consistent with the law, it also prevents the needless disqualification of job applicants who otherwise may become excellent employees.

Reid Klion

IPAC Call for Papers

Reid Klion — Wed, 03 Feb 2010 21:45:28 GMT
The call for proposals for the 2010 IPAC Conference is now open. Formerly known IPMAAC, IPAC will meet on July 18-21 in Newport Beach, California at the Hyatt Regency. Session formats include symposia, panel discussions, paper presentations, tutorials, and workshops. Presentation proposals will be accepted until March 5th.

pan has long been active in IPMAAC and now IPAC and is proud to have been an conference sponsor for many years. IPAC is an organization of applied HR selection and assessment professionals, with members who work for both public and private-sector organizations. If you aren’t familiar with the organization, I might suggest you take a look a IPAC website and consider attending or even submitting to the conference.

Reid Klion

ATP Conference Coming Up

Reid Klion — Mon, 25 Jan 2010 16:13:37 GMT
The Association of Test Publishers Conference is coming up on February 7-10 in Orlando. The ATP Conference is a good opportunity to see what is new in the world of testing and meet many of people who are active in the industry. The conference program has tracks on promoting state-of-the-art assessment, integrating technology into assessment, and innovations in test security. pan will also have a booth in the exhibit hall, and a number of us will be attending including Doug Cole, Jocelyn Courtney (who recently became Secretary of the ATP I-O Division), and Jenni Ginsburg. If you will be there and would like to arrange a time to meet with one of us, please drop me a note. We look forward to seeing you in Orlando in a couple of weeks!

Reid Klion

On Writing Items...

Reid Klion — Tue, 19 Jan 2010 16:58:27 GMT
Writing items for tests and surveys is often more difficult than it appears. There are professional guidelines for item writing but it is also an art that might be best learned through experience and observing how items perform when they are actually used.

How an item is written can have a substantial impact upon how subjects respond to it and inadvertently skew the results. As an example, I received what was labeled a survey in the mail the other day. Here are a couple of items:

-Do you believe that Barack Obama's nominees for federal courts should be immediately and unquestionably approved for their lifetime appointments by the U.S. Senate?

-Do you believe the best way to increase the quality and effectiveness of public education in the U.S. is to rapidly expand federal funding while eliminating performance standards and accountability?

To be honest, I suspect that the underling intent of the survey is to motivate political supporters (as opposed to actually being an opinion poll) because it was accompanied by a fund raising request. And regardless of one’s political leanings, I suspect that it would be difficult to find anyone who would endorse either of these statements as written. However, it illustrates how an item is constructed can impact the results. In this case, I think that it is fair to conclude this was the intent. Problems do occur, though, when the goal is gather information in an unbiased way but poor item writing gets in the way.

Reid Klion

Trust but Verify

Reid Klion — Thu, 14 Jan 2010 16:16:00 GMT
One of the major reasons to use an assessment is to ascertain if a candidate is actually capable of doing the job. While there other ways to accomplish this, such as reviewing references and resumes, these approaches can fall short. Unfortunately, employers often do a poor job of actually checking references and many of us don’t verify a candidate’s educational history. (There haven been some well-documented cases of people in high profile positions who don’t have the background they claimed. For example, there is the former director of admissions at MIT didn’t have any of the degrees listed on her resume and George O’Leary who was the head football coach at Notre Dame for 5 days until it was discovered that both his claims to have played college football and having a graduate degree were false.)

Another challenge in credentials verification is the apparent growth of degree granting universities that may not have particularly stringent academic standards. There is growing concern about U.S. military personnel getting degrees (almost wholly paid for by the government) from propriety, for profit institutions. While the students who enroll in these programs do so with good intentions, they often discover their degrees are not well-regarded by employers because of the nature of the schools. In a similar vein, there is a physics professor at the University of Illinois who has essentially been on a crusade against diploma mills, businesses often based outside the United States where one can essentially purchase a degree without fulfilling any requirements at all. When it comes to vetting a job candidate’s credentials, it is important the employers have a way of verifying that an individual actually has the qualifications to do the job well. Whether it be with looking into a candidate’s credentials or using an assessment, this is critical information to have before making a hiring decision.

Reid Klion

IRS to Test Tax Preparers

Reid Klion — Mon, 11 Jan 2010 15:17:46 GMT
The Internal Revenue Service recently announced that it intends to develop a testing program for tax preparers. While CPAs and enrolled agents (non-attorneys who may represent clients before the IRS) have had testing and licensure programs for a number of years, the estimated 900,000 to 1.2 million individuals who prepare tax returns for a fee can do so without any sort of credential. The logic behind this new IRS initiative is to protect the public from incompetent or unscrupulous practitioners. Apparently, this move is supported by some of the large tax preparations firms who view this as a way to force ill-prepared practitioners out of the field.

There are some who argue that licensure programs should not exist because they represent a restraint of trade and that the market should be the sole determinant of who can offer professional services. Indeed, there may be situations where the development of a licensure program may be a bit misguided. (For example, there is a battle in Virginia over the licensure of yoga schools.) That being said, tax preparation very much seems a situation where some form of regulation seems reasonable to protect the public. Similarly, until the recent mortgage crisis, many states did not have a licensure program for mortgage brokers. To that end, pan was pleased to assist the Indiana Secretary of State in implementing a testing program for loan brokerage professionals.

Reid Klion

Personality and Medical School Success

Reid Klion — Thu, 17 Dec 2009 19:50:17 GMT
A recent article by Filip Lievens, Deniz Ones, and Stephan Dilchert looked at the relationship between personality and medial school success. This follows the path of other researchers who are looking at non-cognitive factors in predicting academic outcomes.

They followed 600 Belgian medical students longitudinally throughout their academic careers. (Differing from the US model where medical school is essentially a 4-year graduate program, European students typically enter medical school directly from high school in what is a 7-year program.) They found that personality differentially predicted medical school outcome based upon the changing demands of medical school. (Especially in the European model, the first few years are comprised almost exclusively of basic science while latter years have an increased focus on clinical work and patient contact.) For example, openness and extraversion had increased validity in predicting performance over the 7-year period. Similarly, they found that while conscientiousness predicted grades throughout a medical student’s career, its validity increased from .18 to .45.

These specific findings aside, the article also is a good reminder that it can be important to study the validity of a predictor over time, especially in situations where task demands are dynamic and change predictably over time.

Reid Klion

Problems with CAT in India

Reid Klion — Fri, 11 Dec 2009 15:49:39 GMT
The Indian Institutes of Management decided to move the Common Admission Test (required for admission to Indian MBA programs and similar to the GMAT in North America) to a computer-based test delivery model available over a ten day period. (In previous years, the test was administered in paper/pencil format on only one day.) This year, nearly 200,000 candidates applied to take the exam. Unfortunately, due to a number of factors including widespread hardware issues and a Cornflicker virus infection, there were major problems in delivering the test. A number of testing centers had to be shut down with reports indicating that some 10%-18% of candidates had to be re-scheduled (though it appears all who were not successful in doing so). Given the scale of the program, it has garnered substantial coverage in the press as well as in the Indian Parliament and brought forth public interest from number of other test delivery organizations that manage large-scale test delivery projects in India and would like to take on the project next year.

Reid Klion

Chinese Civil Service Exam 2009

Reid Klion — Thu, 10 Dec 2009 16:01:38 GMT
In the past, I have blogged about testing in China and Korea. While we often thinking of standardized testing as a Western phenomenon, it really has its roots over 2000 years ago in the Chinese imperial exam system with the concept later being adopted in the East in the 19th century.

The scale of some of these Asian programs is quite impressive (as well as is the potential for high profile problems which I will blog about tomorrow in regard to recent issues in India with delivery of their equivalent of the GMAT). At the end last month, nearly a million candidates took the Chinese Civil Service Examination. Attributed to the economic slowdown, the number of test takers (photos) has increased by a factor of 15 since 2003. Given there are only 15,000 openings for a million test takers, the stakes are impressively high. These government positions are often coveted for their stability and are sometimes referred to as the "Iron Rice Bowl."

Reid Klion

Boot Camps for Pre-School Admissions Tests

Reid Klion — Mon, 30 Nov 2009 21:16:51 GMT
It sometimes concerns me when I read an article in the mainstream press and have to check if haven’t accidentally ended up on The Onion. I had one of those experiences the other day when I came across an article about how parents in New York City are paying up to $1000 to have their 3- and 4-year-olds coached on how to take the tests used to qualify students for gifted and talented kindergarten public school programs. Apparently motivated by the desire to avoid the cost of private schooling, these parents describe themselves as seeking the opportunity “to have a choice.” While I admit to not fully understanding all the dynamics around raising a child in Manhattan, I find the whole thing rather ridiculous and resonate to a recent piece in Time about over-involved parents.

At the other end of the spectrum in terms of over-parenting, I was intrigued by a high school graduation coaching program in the Indianapolis Public Schools. Unfortunately, not unlike many large urban school districts, IPS struggles with high school graduation rates under 50% and students and families who often face a myriad of challenges outside the classroom. Financed by the local Chamber of Commerce, the intent of the program is to help students to complete high school. Here, it is good to see the business community lend a hand where resources of this type are scarce. In the long run, it is hard to see how a program like this won't benefit all involved--students, their families, and the community at large.

Reid Klion

International Test Commission Call for Papers

Reid Klion — Wed, 18 Nov 2009 22:15:15 GMT
Just a reminder that the Call for Papers for the International Testing Commission Conference (to be held July 19-21, 2010 in Hong Kong) closes on December 1. I had the opportunity to attend the ITC Conference in 2007 and would suggest it to anyone interested in increasing their understanding of issues related to the globalization of assessment. There are multiple considerations that emerge when a testing program is implemented internationally with challenges ranging from the practical to the psychometric to the legal. As the need for assessment programs that be used in multinational contexts grows, so does the necessity for assessment professionals to understand the complexities that come along with the territory.

Reid Klion

Testing Times for Korean High School Seniors

Reid Klion — Fri, 13 Nov 2009 21:21:38 GMT
Yesterday was the day for almost 700,000 Korean high school students to take the nearly 9-hour long College Scholastic Ability Test. Given only once a year, it is critical to gaining admission to a Korean university. Coupled with the fact that more than 80% of Korean high school seniors attend university, this is a critical event. High school students spend months preparing, and their families often go to churches and temples to pray for a good outcome. To help manage the logistics, work started an hour late yesterday to minimize traffic jams, and aircraft were diverted from flying near test sites to minimize noise disruptions. In past years, the power company has even placed crews on standby in case the power grid would somehow become disrupted and impact testing.

While we often think of testing as a Western phenomenon, standardized tests have been used in Asia for over 2000 years. Indeed, the concept of using tests to hire civil servants was first implemented in the West in the late 1800s when the British began following testing practices they first observed in China.

Reid Klion

Resource on Job Simulations

Reid Klion — Thu, 12 Nov 2009 22:07:38 GMT
The Merit Systems Protection Board, a federal agency charged with ascertaining that other federal agencies carry out personnel-related activities in compliance with the law, recently released a report on the use of job simulations in assessing job candidates. While written specifically for the federal context, it is a good source of information to anyone interested in the topic.

For those who may not be familiar, a job simulation presents a realistic work situation and assesses how an applicant responds to it. When you think about it, the set of tasks that might be considered job simulations is rather broad to include things like having welders demonstrate their skills on standard tasks, having pilots demonstrate their skills in a flight simulator, or a store manager asking applicants how they might respond to specific customer inquiries. While they may differ in terms of fidelity to actual work tasks, job simulations often provide good information about a job candidate’s skill level with a high degree of face validity. The MSPB report discusses the different types job simulations, their relative benefits and weakness, as well as how to create and conduct them. The document also goes on to detail how simulations might be more broadly used by federal government. Some very useful information at the right price...

Reid Klion

Problems with SAT Proctors

Reid Klion — Mon, 02 Nov 2009 22:31:16 GMT
While virtually all the proctored testing that pan does is in the HR space, there is a huge amount of proctored testing that takes place within an education context--think PSAT, SAT, MCAT, GRE, LSAT and the like. A recent NYT piece focuses on the problem of proctors who fail to do their jobs or even manage to be disruptive during SAT testing sessions. The article indicates that SAT proctors are not specifically trained for the task (which surprised me a bit since our policy is to train all proctors on each program they supervise). And reviewing some of the reported issues that occur during paper and pencil testing, it is clear that computer-based test administration (as we use in our testing centers) can ensure a much more standardized experience in terms of test timing, the right test forms going to the right candidate, and the like.

One bit of advice provided by an SAT test coach (yes, this is apparently a whole industry in and of itself) is that if a candidate notices a problem during test administration, he or she should either voice a complaint immediately and ask that it be addressed or “forget about it” (as opposed to simply grumbling about it after the fact). Here, it is important that test takers assume some responsibility in advocating for themselves by pointing out problems if they become apparent. The coach also goes on to say that “you have to expect that there will be distractions. If you’re seated next to a kid who is muttering under his breath, or sneezing, or grunting, just deal with it. Move on.”

Reid Klion

The Opposite of "Faking Good" is...

Reid Klion — Thu, 29 Oct 2009 15:39:46 GMT
We are often concerned about the role of self-presentation during hiring situations. For example, much attention has been devoted to understanding hown “faking good” may impact personality assessments, the potential role of cheating on exams, and the fact that people tend to exaggerate their qualifications during interview situations. That being the case, at least the folks who engage in these behaviors are making an effort to present themselves in a positive light (and, as have some like Bob Hogan argue, the ability and willingness to present oneself positively is often a critical job qualification in and of itself).

In contrast, consider those individuals who fail to put their best foot forward when applying for a job (and the reaction this tends to elicit from hiring personnel). For example, virtually all of us can recount instances of job candidates who have multiple misspellings on their résumés, manifest a complete lack of phone etiquette, and the like. To this end, a recent CNN article took an informal survey of hiring managers on some of the grossly inappropriate behaviors they have seen during interviews. These include job candidates answering cell phones during an interview, acting bizarrely, and being completely uninformed about the company or what it does. While it has been well-established that non-structured interviews are poor predictors of future job performance, it is also clear that certain interview behaviors are too salient and extreme to be ignored even by the most heavily research-based interviewer.

Reid Klion

Innovation and Creativity: Creating the New Untouchables

Reid Klion — Fri, 23 Oct 2009 19:34:56 GMT
Thomas Friedman, well-known author and columnist, recently wrote about the workers he labels the “untouchables.” He argues that those individuals who have the imagination and drive to “invent smarter ways to do old jobs, energy-saving ways to provide new services, new ways to attract old customers or new ways to combine existing technologies” are the people who are critical to organizations that want to be successful in today’s economic environment. As a result, these employees will always be of great value. In contrast, average workers who can only carry out routine work are becoming increasingly endangered because many of these tasks can be automated or outsourced. Friedman further points out how our educational system needs to find ways to increase the number of students with high level problem solving and analytic skills and have a spirit of creativity, innovation, and entrepreneurship. However, until that day (and if it ever comes), it is incumbent upon employers to carefully select and build a workforce comprised of individuals who can truly create value for their organization.

Reid Klion

ATP Conference Program Posted

Reid Klion — Tue, 20 Oct 2009 13:42:35 GMT
The program for the Association of Test Publisher’s Innovations in Testing Conference (to be held February 7-10, 2010 in Orlando) is now available. As usual, there appear to be a large number of interesting presentations on the docket. Some that initially caught my eye focused on topics including a survey about the future of testing, detecting cheating, and ATP’s new revised guidelines on pre-employment integrity testing. Also, registration is open with a $75 discount to folks who do so before November 16.

Reid Klion

Cheating: An International Perspective

Reid Klion — Fri, 16 Oct 2009 17:33:50 GMT
I receive regular updates from ATA which is the largest testing company in China (and also happens to trade on the NASDAQ as ATAI). Last month, they administered the National Judicial Examination in China to more than 350,00 candidates. An individual must pass this exam to work in Chinese legal system and is a requirement for lawyers, prosecutors, and judges. Due to the high stakes nature of this exam, there were major concerns about cheating. (Cheating on exams in China has been an issue for over 2000 years with reports dating back to at least the Han Dynasty. There are a number of cultural factors which contribute to this which I hope to find the time to blog about fairly soon.)

In addition to taking candidates’ temperatures to make sure no one was ill, many test centers were also outfitted with handheld devices to detect the presence of wireless earphones well as equipment to block mobile phone signals. Additionally, 8 fixed monitoring stations and 2 mobile signal detection vehicles (labeled “Signal Detection”) were used to identify “suspicious” signals in the proximity of testing centers in Beijing. In at least one instance, a test taker was discovered to have a wrist watch which could send and receive data.

Reid Klion

Survey on Usage of Online Assessments

Reid Klion — Tue, 13 Oct 2009 19:20:03 GMT
Back in August, Charles Handler of Rocket-Hire released his annual Online Assessment Usage Survey. Surveying almost 150 persons who work in the areas of recruitment and hiring, he found that approximately 2/3 of respondents use assessments. Of those using assessments, almost a third report using online processes exclusively. The types of assessments that are most broadly used are personality measures, tests of skills and knowledge, and cognitive tools. He has yet to release the full report but will forward a copy when it is complete. Charles is about the only the only independent source of information on online assessment and is certainly one of the foremost experts in the area.

Reid Klion

Deviant Workplace Behavior and the Economy

Reid Klion — Wed, 07 Oct 2009 19:55:20 GMT
Jack Jones, Ph.D., respected colleague and Vice President of Human Capital Sciences at Vangent (a long-time pan content partner) has a recent article in Talent Management on how poor economic conditions may increase the prevalence of problematic behavior at work. While some of the factors cited include the fairly obvious such as increased general stress and financial needs, he also outlines how more subtle dynamics such as decreased levels of supervision due to staffing cuts or even poor maintenance of a facility’s physical plant can also contribute. The latter may be related to the “broken window” concept in urban sociology which argues that if vandalized windows are quickly repaired, it tends to dissuade further acts of minor anti-social behavior in the neighborhood. Similarly, if there is an increased sense of disorder at work, it may influence the increased expression of deviant behavior.

The article goes on to describe the extent to which candidates will admit to specific counterproductive behaviors as assessed by the Reid Report with the most common being fired from prior jobs, having multiple unexcused absences from work, and giving unauthorized discounts. A brief case example of ROI is provided citing a retailer that estimated an annual savings of $6 million by screening out high risk candidates.

Reid Klion

Testing the Test Graders

Reid Klion — Fri, 02 Oct 2009 18:34:45 GMT
There was a recent NYT Op-Ed by a guy who is authoring a book on the standardized testing industry and his exploits as a “test grader.” For those not familiar with this aspect of the testing industry (or who don’t have school-aged kids), the standardized tests administered by schools often include “open ended” questions where students need to provide a written response.

In order to scoring, armies of temporary workers are hired to do the work. The author recounts some of the goings-on as well as his less-than-conscientious behavior in this work. His conclusion is that “scoring should be done only by professionals who have made a commitment to education—rather than people like me.”

This brings a couple of thoughts to mind. The first is that one wonders if a work attitude assessment should be part of the hiring criteria for the essay graders. In this, as in all jobs, we want people who will carry out their work in a responsible and conscientiousness manner. Additionally, it also highlights why there is significant interest in the use of automated essay scoring systems where computers grade the written responses. There are now several years of research on these systems, and they tend to work well. The obvious benefits are that compared to human raters, they are not susceptible to either fatigue or poor work ethic. And unlike humans, they are 100% reliable because they will grade the same essay the same way every time.

Reid Klion

Testing Times

Reid Klion — Tue, 29 Sep 2009 15:44:47 GMT
One of the services that pan provides is a network of over 600 proctored testing centers. We offer these to clients who require the supervised delivery of high-stakes assessments. However, outside of our Operations Team, most of us have little regular contact with the candidates who use our testing centers.

I was jolted from this earlier today when I had to make quick stop at my local bank branch. When I pulled up to the office building where it is located, I saw several people anxiously smoking and pacing in front of the building. I quickly realized that across the lobby from the bank was a proctored testing center operated by another assessment company. When I understood the situation and that these folks were about to take a test, I was brought back to the recognition that taking a high stakes test can be very stressful given the potential outcome—perhaps not qualifying for a coveted job or obtaining the license required to enter a profession.

While testing is something that we do every day, we can sometimes become a bit detached from the actual experience of the candidate. As such, when we ask someone to take a test, we need to be aware of the anxiety that this can create and the importance of using good test administration practices such as providing appropriate information about the test beforehand and maintaining a quiet, comfortable, and professional environment. This is especially important to bear in mind when assessments are not administered by trained staff in a formal proctored testing center but are supervised by individuals who have other duties and are carried out done in a location where other activities are also conducted.

Reid Klion

Intelligence, Environment, Heredity, and Smarter Kids

Reid Klion — Thu, 24 Sep 2009 15:00:03 GMT
I recently finished Richard Nisbett’s fascinating book, Intelligence and How to Get It. Subtitled, Why School and Cultures Count, Nisbett (a prominent cognitive psychologist at the University of Michigan) argues that intelligence is quite malleable and that it has a more limited genetic component than many believe. Arguing against a strong hereditarian model of intelligence (and contrary to Herrnstein and Murray’s controversial 1994 book, The Bell Curve), his stance is that while there are some genetic components to intelligence, environmental and cultural factors have the greatest influence (as well as being the domains that can be influenced and shaped). Overall, he provides a good review of the nature vs. nurture debate on intelligence (and rather clearly comes down on the nurture side).

As one who stresses environmental factors in the development of intelligence, Nisbett also makes some research-based suggestions about how to raise a child’s IQ. Included are:
•   Exercise during pregnancy
•   Talk to children using complex vocabulary and include them in adult conversations.
•   Teach kids to categorize and compare objects in their environment
•   Foster curiosity
•   Encourage regular exercise and activity
•   Limit family stress and instability
•   Teach and reward self-control and delaying gratification
•   Praise children for hard work and efforts which are under their control.
•   Have high expectations and model the behaviors you would like to see in your kids

Reid Klion

Carl Jung's "Hidden" Book

Reid Klion — Tue, 22 Sep 2009 16:00:57 GMT
The modern psychological study of personality can be traced back to Sigmund Freud and his student, Carl Jung. Jung later broke with Freud and developed his own theoretical system which focused upon concepts such as the collective unconscious and the role of archetypes. While the study of personality has obviously evolved very substantially in the 100 or years from the time that Freud and Jung started their work, their influence can be seen today. For example, the Meyers-Briggs Type Indicator and other assessments of psychological type are based upon Jung's work.

It also should be noted while Freud and Jung are largely viewed as historical footnotes for most of us interested in workplace personality, both continue to have very ardent followers in other circles. The New York Times Magazine has a fascinating (but long) article about the efforts of Jung's family (he died in 1961) to prevent the publication of book that he wrote while in the throes of a personal crisis. Filled with mythology, fantasy, calligraphy, and paintings, Jung's family always considered it to be too personal to be made public. The article describes the history of the "Red Book" and how it has recently come to be published.

Reid Klion

IPMA-HR/IPAC Conference

Reid Klion — Fri, 18 Sep 2009 20:45:07 GMT
I had the pleasure of attending the joint IPMA-HR/IPAC meeting this past week in Nashville. The conference was composed of both public sector HR professionals and assessment-oriented psychologists. The mix led to some interesting discussions where the HR folks could hear about the nitty-gritty involved in testing and the psychologists learn more about the day-to-day challenges faced in managing public sector workforces in challenging economic times.

Many of the presentations are posted. As always, Bob Hogan gave an entertaining and scholarly talk on the critical importance of leadership in organizational success. Bryan Baldwin (whose blog HR Tests is certainly worth a look) also gave a great talk on the potential role and challenges associated with using social networking websites in recruiting. (Bryan didn’t post his presentation but his recent article can be found in the Assessment Council News on page 6.)

Reid Klion

E-Seminar Tomorrow: Recession Proofing You Assessment Program

Reid Klion — Wed, 16 Sep 2009 16:08:05 GMT
John Callender (Biodata Online) and Jack Smith (Kingwood Group), two respected testing and assessment experts and long-time pan content partners, will discuss how the current economic climate impacts recruitment and assessment strategies. Some of the topics to be covered include how investing in assessments makes economic sense and the potential benefits and consequences of shifting to a lower cost assessment program. Please join us!

Thursday, September 17 @ 1PM ET     Register

Reid Klion

Predicting College Success

Reid Klion — Thu, 10 Sep 2009 20:24:35 GMT
There was a WSJ article a couple weeks back on the growing trend for universities to consider using personality assessments to assist in admissions decisions. In addition to the ETS Personal Potential Index I blogged about a while back, there are a number of other tools in development. For example, ACT is developing an assessment to identify students who are at high risk for dropping out while the College Board has been working at Michigan State University to develop a biodata predictor for university success.

Interest in these instruments appears to be spurred by the seeming unassailable belief that there is more to predicting academic success than GPA and standardized tests scores. However, there are also concerns about cheating and students learning how to “game” such non-cognitive assessments. It should be noted that colleges have always attempted to discern a potential student’s strengths and weaknesses outside the academic realm through personal statements and admissions essays. Proponents of these new non-cognitive assessments argue they are carrying on this same tradition but doing so in a way that standardizes how these non-academic factors are evaluated.

Reid Klion

On Background: 360 Degree Feedback Assessments

Reid Klion — Thu, 03 Sep 2009 21:31:56 GMT
We have always had a number of clients who are interested in using 360 degree feedback instruments. Also known as multi-rater feedback, these tools gather information from several people who interact with an employee on a regular basis such as managers, peers, or subordinates. Each tool varies slightly in its focus, but all use questions that are behaviorally-based and focus on work-related behaviors and job performance analysis. Jocelyn blogged last week on some tips and suggestions to consider when using 360º feedback tools.

The major benefit of 360 degree feedback instruments is that they gather information from a wide array of individuals who interact with the subject of the assessment. A 360 degree feedback tool can either be used alone or as one component of a broader development or coaching program. While controversial, some organizations also use multi-rater feedback instruments as part of their employee performance review process. However, it is important that you inform raters that their feedback may be used for evaluative purposes (such as for an employee performance review).if you chose to use 360 degree feedback instruments in this way.

Finally, organizations can use a 360 degree feedback tool as a teamwork inventory to provide insight into each team member’s strengths and weaknesses and how they function together as a team. This information can inform the way team members interact with one another as well as help them to pinpoint and understand potential areas of conflict.

IPAC Conference Coming Up

Reid Klion — Wed, 02 Sep 2009 13:13:20 GMT
IPAC (formerly known as IPMAAC) will be holding its annual conference in conjunction with IPMA-HR on September 12-16 in Nashville. IPAC is comprised of individuals who are interested in public sector selection and assessment. As it has for the past several years, pan is pleased to be a conference sponsor. Additionally, Nate Studebaker will present on his work with the Texas Department of Family and Protective Services that helped to streamline their assessment process while I will participate in panel discussion on technology trends in assessment. We hope to see you there!

Reid Klion

The Power of Feedback

Reid Klion — Mon, 31 Aug 2009 21:20:42 GMT
I was recently asked to contribute to two of my coworkers’ professional development efforts by completing 360º feedback instruments. I truly enjoy receiving feedback from others, and so I was more than happy to oblige the request. I completed two different 360º assessments and spent some time reflecting on the act of providing feedback. As a result, I have a few helpful hints to share.

1) Use 360º feedback instruments to gather developmental information.
The people I was asked to rate are extremely helpful and valuable to me at work. While my positive feedback is important, constructive criticism is equally, if not more, useful. Although I may have very little criticism of these individuals, others such as their supervisors and direct reports have a different perspective and can provide unique information and suggestions. 360º tools provide a comprehensive view of the individual’s strengths and weaknesses, rather than the narrower perspective given by self-ratings or the ratings of only a few individuals.

2)   Provide a “Not Applicable” response option.
One of the tools I completed included an N/A option for each rating item; the other did not. Given the breadth of the abilities and characteristics rated across these types of tools, raters are likely to see items that they simply do not have enough information to rate accurately. When asking others to provide feedback, it is important to provide an N/A option so that the rater doesn’t feel obligated to respond to items about which he/she may be uncertain.

3)   Educate raters on common rating errors
Studies consistently show that when providing ratings, people tend to make certain errors. When using any kind of feedback program, encourage the people providing information to use the entire response scale and avoid errors, such as leniency, strictness, and central tendency biases. Inform raters of the similar-to-me/different-than-me and halo/horn effects to encourage accurate ratings.

4)   Separate development from evaluation
Although feedback may be important in both development programs and performance appraisals, it is critical to keep the two systems separate. When collecting developmental information, inform raters that the data will not be used for evaluation and will not affect the ratee in any negative way, as this promotes honest and accurate responding. It is important to note that 360º feedback instruments are not appropriate for evaluation and should only be used in a developmental context. In addition, it is wise to keep peers and direct reports out of the performance appraisal process, so as not to cause conflict or discomfort.

Jocelyn Courtney-Hays

Norms and Personality Tests

Reid Klion — Wed, 26 Aug 2009 16:13:07 GMT
Rob Tett and colleagues published a recent paper in the Journal of Occupational and Organizational Psychology that looked at personality test norms. Examining a large set of data from the Hogan Personality Inventory, they took the norms derived from several large samples of individuals employed in sales, trucking, clerical work, finance, and management. No big surprise, there were some rather significant differences in average scale elevation between groups (e.g., the truckers were substantially less sociable than the sales group). However, a bit unexpected were the differences in mean scores between samples within the same job family. As an illustration, some samples of sales personnel had much higher scores on scales measuring ambition and sociability than did others. Indeed, there were cases where the differences within job family were nearly as great as those between job families.

There are a handful of takeaway points to consider. First, relying solely on job type to select a normative sample can be problematic. Rather, factors such as organizational culture of the work group where the samples were obtained or whether the data were derived from job applicants vs. incumbents need to considered. (The latter is important because applicants tend to obtain higher scores than incumbents, something I will blog about in the near future.) Secondly, this article also analyzed the impact of sample size on the reliability of normative samples. As a saving grace, they found that a sample of 100 is usually quite sufficient to develop a good set of local norms.

Finally, it should be noted that the importance of norms varies greatly depending upon how a test is being used. For example, if I want to know how the conscientiousness of the maintenance personnel in my company compares to that of folks who work in maintenance for other organizations, the normative sample chosen is obviously critical. However, if I am only interested in ranking maintenance job candidates on conscientiousness or have already conducted a validation study to determine cut scores, the use of norms isn’t particularly relevant or useful as we could use untransformed scores for these purposes.

Trying to Measure "Faking Good"

Reid Klion — Thu, 20 Aug 2009 21:40:59 GMT
For the past several years, there has been substantial debate about the impact of “faking good” (or socially desirable responding) on personality tests. While clients sometimes want to know how a particular test adjusts for an individual’s portraying him or herself in an overly positive fashion, there are no easy answers to this seemingly simple question.

First, there is disagreement as to whether socially desirable responding even has a significant impact upon the validity of most assessments when they used to actually select employees. Second, there is the argument that it is expected for job seekers to present themselves in a favorable light (e.g., “putting your best foot forward”) and that one should be concerned about an applicant who isn’t either willing or able to do this. Thirdly, when assessments are developed based on data samples of job seekers, this positive self presentation bias is already “baked into” the norms. Finally, as a practical matter, it has been quite difficult to find accurate measures of “faking good,” let alone ones that are helpful in increasing the validity of a test.

A recent Personnel Psychology article by Nathan Kuncle and Auke Tellegen helps shed some light on this. In their research on a scale designed to measure socially desirable responding, they found that increasingly elevated scores on a favorable personality trait are not necessarily seen as being more desirable. For example, moderate levels of “being organized” are typically seen more favorably than either very low or very high levels of this behavior. Additionally, when subjects are instructed to respond favorably on a scale designed to assess "faking good,” they tend to endorse these items at moderate, not extreme, levels. As a result, the authors argue that measures of socially desirable responding need to consider the desirability of each response alternative and not simply look at the overall score on the scale.

Reid Klion

Controversy Erupts Over MMPI

Reid Klion — Mon, 17 Aug 2009 18:10:36 GMT
While it had been brewing in the professional literature for the past three years or so, the dispute between the authors of the MMPI has become public. (The MMPI is the pre-eminent clinical assessment that has been used for over 70 years. While sometimes discussed mistakenly in relation to pre-employment testing, it is almost never used to select employees because it is a clinical, not a work-related, assessment. About the only times that it is used to hire employees is in occupations where it is critical to rule out significant psychopathology such as police or nuclear plant operators.)

The issue at hand has to do with a “reformulated” version of MMPI-2, the MMPI-2-RF. The “RF” version, which has just been published, is shorter and described as easier to interpret that its predecessor. However, James Butcher, the lead author of the earlier MMPI-2 is highly critical of this revision. The issue has become personal because the authors of the “RF” version (Auke Tellegen and Yossef Ben-Porath) had been long-time colleagues and collaborators with Butcher. The result has been charges of conflict of interest and mismanagement on the part of the University of Minnesota press which owns the MMPI.

Reid Klion

Non-Proctored Testing of Another Sort...

Reid Klion — Mon, 10 Aug 2009 20:36:32 GMT
One topic that has been controversial in the assessment literature is non-proctored assessment—that is, having job candidates complete tests without supervision. I blogged about it previously and a seeming plethora of conference presentations and articles have addressed the topic. A consensus seems to be emerging that unsupervised testing may be appropriate in a particular situation based upon a careful risk-benefit-analysis that involves consideration of a variety of factors including overall business strategy, type of test, how the results will be used, and the candidate population.

A similar sort of cost-benefit analysis seems to have taken place in the federal prison system when transferring inmates from facility to facility. Since the early 1990s, prisoners who are considered low risk are apparently given a one-way bus tickets and sent on their way without supervision. Nearly 90,000 prisoners have been transferred in this fashion since 2006, and fewer than 1 in 500 have failed to make it to their intended destination. Given the expense that would be associated with escorting the prisoners, and the fact that a large proportion of these felons are on their way to a halfway house where they will have substantial freedom to move about the community, federal prison officials seemed pleased with the system. Others, including Greyhound officials and local law enforcement authorities where these prisoners have absconded, take a somewhat differing position.

Reid Klion

On Background: Watson Glaser Critical Thinking Appraisal

Reid Klion — Fri, 07 Aug 2009 20:42:26 GMT
With its original version dating back to 1925, the Watson Glaser is a well-known critical thinking exam. Critical thinking exams are designed to measure an individual’s ability to evaluate the evidence in support of an assertion, to engage in abstract thinking, and to come to logically accurate conclusions.

As might be expected, scores on the Watson Glaser are highly correlated with other assessments of verbal, non-verbal, and numerical reasoning. The Watson Glaser also predicts job performance and leadership ability since critical thinking is closely associated with success in higher level positions.

The Watson Glaser is often used as part of a pre-employment testing process because of its ability to predict work performance in positions which require decision making. However, with the advent of the Watson Glaser II (which will soon be available on pan), there is now a developmental report which will broaden the Watson Glaser’s use beyond pre-employment testing to coaching, training, and other related domains.

A Rorschach Cheat Sheet?

Reid Klion — Tue, 04 Aug 2009 14:39:42 GMT
An uproar has been brewing because the images used for the Rorschach inkblot test been posted on Wikipedia along some of the more common responses to each. . As might be expected, some psychologists are concerned that this may impact the validity of the test, especially in cases where the assessment is used within a legal context (for example, in a child custody evaluation or pre-trial assessment of a sex offender) as opposed to a counseling situation. In contrast, others have pointed out that the Rorschach is almost never used in isolation but in combination with a number of other clinical assessment tools to make decisions and that this same information about the Rorschach is available in training texts on the test as well as from other sources.

However, this event does highlight an important issue about test security and the Internet. While these images have been available to the public previously, their posting on Wikipedia does make them easily and broadly accessible. And even if we accept the argument that posting this information will have little impact upon the Rorschach’s validity, it should be noted that real harm can be done when test security is compromised. Not only is copyright law often being violated and the validity of a test compromised that may have taken years to develop, but the public may be placed at harm. For example, how many of us would be concerned we found out that our child’s pediatrician only passed a licensing exam because he or she had knowledge of the test content before taking the licensing exam?

As it turns out, the individual who posted the images actually is an emergency room doctor from Saskatchewan. When questioned, he sees little harm in what he has done and goes on to state that his elderly father passed a driver’s license eye exam by looking it up on the Internet. I guess our only hope is that his dad can actually see well enough to drive to keep both himself and the rest of us safe.

Reid Klion

On Background: Mechanical Aptitude Tests

Reid Klion — Thu, 30 Jul 2009 19:03:48 GMT
While personality and cognitive tests are often used for hiring and selection, another set of tests focus on an individual’s aptitude, or ability, to learn a set of tasks. Rather than assessing an individual’s knowledge, mechanical aptitude tests predict an individual’s ability to learn a job. Mechanical aptitude tests are most typically used in the hiring selection process to identity the most suitable candidates for training programs or entry level positions.

Probably the best known mechanical aptitude test is Pearson’s Bennett Mechanical Comprehension Test. Here, the applicant is presented with pictures that represent mechanical principles in everyday situations. For example, one question may show a lever and ask where it should be placed to make moving a heavy object easiest. The intent of the items is to assess mechanical reasoning, not specific knowledge. Other similar tests are the Hogrefe PPM for Mechanical Understanding,Pearson DAT for PCA Mechanical Reasoning, and the Ramsay Mechanical Aptitude Test.

Mechanical aptitude tests are powerful tools when used for applicant screening into positions that require these skills. They have been used successfully for decades and are well-suited for hiring selection decisions for jobs that require mechanical ability or will involve training in these skills.

SIOP Resource on Employee Assessment

Reid Klion — Mon, 27 Jul 2009 21:09:46 GMT
I was on the SIOP website the other day and came across the “SIOP FYI on Workplace Topics” section which has some rather extensive information on both employment testing and coaching. While I haven’t had a chance to review the coaching section in any detail, there is a wealth of descriptive information about assessment. Overall, the presentation seems quite balanced with a section on reasons to use (as well as not to use) assessments, types of tests, and item formats.

These pages provide an excellent, even-handed orientation which focuses on the pros and cons of various approaches to assessment. These resources would be quite useful to individuals who want to learn more about using assessments in the workplace. Interestingly, there is a “hit counter” on the bottom of each page. While I don’t know when the pages or counters were created, it appears that some have been viewed fewer than 1000 times. Let’s hope they find the audience that they deserve.

Reid Klion

Playing in Traffic

Reid Klion — Fri, 24 Jul 2009 19:36:41 GMT
While perusing Amazon.com for good books recently, I came across the Penn and Teller tome, How to Play in Traffic. Generally, I wouldn’t mention such mundane details of my life, but I wanted to make note of it here. You see, it has come to my attention that there are some people who make playing in traffic their full-time job.

You may remember from my previous post that I am an Indy 500 fan. However, the excitement here in town is that NASCAR will be at the Indianapolis Motor Speedway this weekend for the Brickyard 400. In almost any form of racing, a key part of any race are the pit stops where tires are changed, and…well, I actually don’t know what happens aside from the tires, but I’m sure it’s all very important and mechanical. While I was at the 500, it occurred to me that the people who work in pit row are dealing with cars moving in and out of the pits at high speeds. I have read stories of injuries, from the minor to the severe, occurring due to pit crew members coming too close to the racecars, and seeing the pit stops in person made me realize just how easily injuries could happen.

There are probably a number of important attributes that make for a good pit crew member. I’m willing to bet that in addition to having a high level of mechanical skill and knowledge, you also need to be diligent, conscientious, and focused on safety – not only for your own well-being but also for that of your driver, fellow pit crew members, and competitors. How many other jobs require this same commitment to safety? I can think of a few-- assembly line employees, truck drivers, healthcare workers. Given the inherent risk in these jobs, it is important to hire people who will be safe at work. While training is an important part of workplace safety, there are also a number of assessments that can help identify job candidates who are (and are not) well-suited for work in dangerous environments.

Jocelyn Courtney-Hays

Importance of Selection and Assessment

Reid Klion — Thu, 23 Jul 2009 20:44:03 GMT
When I was in high school Biology during the early 90s, my teacher spoke of a new emerging technology called the Internet. It had been around for a while but had not really caught on outside of scientific and academic circles as computers were for geeks, and most people didn’t see their benefit over using a typewriter. He spoke of the ability to access information about breakthroughs in science as if you you were down the hall from the actual experiment and having immediate access to information from all over the world. Most people thought this type of technology was for large companies like AT&T and IBM, and the idea a personal computer could retrieve information from some nebulous place on the other side of the world was completely foreign and pretty much inconceivable.

In my naiveté, I did not realize he was really referring to the emergence of the global community. The Internet is something which joins everyone together and enables interaction with someone across the world without ever leaving the comforts of the living room.   Transactions can be conducted by anyone with access to a computer, and what was once an exotic item from a foreign country is now a few clicks away on sites like Amazon and EBay. All that is required is the belief that the person on the other side of the connection is legitimate.

In the past, the beginning of trust was largely based on who people commonly knew. If a friend endorsed someone, that person was probably trustworthy. This concept propelled earning a college degree into being a popular activity, as it was a kind of voucher indicating that a person is diligent and has a strong work ethic. A potential employer could look at an academic record or ask a colleague and, based on the results, know if the company found a good fit for an open position. Along with the person’s interview, most people felt secure in knowing they made the right hiring decision.

Today, this isn’t necessarily true. Attracting and hiring potential candidates has become a global search for the right person. Twenty years ago, a company would place an ad in the local newspaper or engage a staffing agency to seek the ideal employee. Now, organizations post jobs on their web site or with companies like Dice and Monster, and a candidate’s ability to find the perfect job is only limited by a willingness to explore the web. Although this is favorable for the both the employer and the employee, the employer ultimately faces a new hurdle, filtering through a pool of global candidates. Moving to any part of the world has become an almost trivial matter. This, coupled with technological breakthroughs removing the need to enter the office, the probability of working with people who are all from the even the same country is small. As a result, verifying assumptions about person’s character through traditional means is difficult at best. Hiring managers can no longer depend on an employee knowing a potential candidate or having heard of the university he or she attended. Employers are forced into trusting a façade and must ultimately face the truth that it is almost impossible to spot a character flaw in an interview as very few candidates will share firsthand experiences of their own inappropriate behavior.

Companies like Google, Microsoft, and Apple have a decent solution, and it’s termed “When in doubt, no hire.” This is perfect for them, because they literally have thousands of applicants. They look for a very specific type of person and are most likely going to find at least two or three. Their interviews are long and brutal and focused at learning everything possible about a candidate. They can afford to discount someone because that person likes the color blue, and blue is considered the enemy, because someone else of equal caliber will more than likely be available. Most companies don’t have that luxury because they haven’t achieved an exalted status in the eyes of potential employees. They don’t have thousands of people trying to interview simply so they can glimpse inside the secret haven of a celebrity company. As a result, they are often compelled to hire who is available, not who is ideal.

A bad hiring decision can be one of the costliest mistakes an organization can make. Between the lost productivity of other coworkers during the former employee’s tenure and the work required to clean up the mess left behind, removing a bad apple can be a long and painful process. Some dodge the situation by not firing the employee in hopes the situation will correct itself. For others, justice is swift; if a new employee isn’t working out, the company quickly reacts trying to prevent as much damage as possible. Either way, the recovery process is often expensive.

For some organizations, hiring may be little more than an educated guess. Others take a systematic approach and utilize assessments. This way, a candidate’s strengths can be viewed in light of the competencies required for a position. As a result, a number of factors are considered in addition to work history and is based on the realization that the new hire must work well within the established culture to advance the organization’s agenda.

Currently, companies can ponder over candidates and extend the amount of time needed to make a hiring decision. As the economy emerges from the recession, candidates will regain the advantage over employers in terms of finding new jobs, and companies will have to quickly make tough decisions as other companies will attempt to hire the same candidates. If only armed with cursory knowledge an applicants’ background, these decisions will essentially result in flipping a coin with potentially expensive consequences. In the end, organizations will benefit once they realize it is very much to their benefit to carefully understand a candidate's potential strengths and weakenssses before extending an employment offer.

Kevin Miller

Neuroticism and Mental Health

Reid Klion — Wed, 22 Jul 2009 18:33:16 GMT
Benjamin Lahey has a recent American Psychologist article on the public health implications of neuroticism. While often viewed as a clinical construct, neuroticism is reliably found as a stable personality factor and often labeled as “adjustment,” “negativity”, or “emotionality stability.” In the personnel selection realm, high scores on this factor are often negatively correlated with occupational success. Similarly, Lahey outlines how neuroticism predicts both mental and physical health as well as response to treatment for both types of conditions. While it is certainly plausible that illness will increase an individual’s level of negative emotionality, there are also data to suggest that it predicts many health problems even when depression is controlled.

The article goes on to explore both the genetic and environmental components that may contribute to neuroticism as well as the interplay between them. Indeed, it is easy to understand how one’s propensity to respond negatively to environmental stressors can act in a self-perpetuating cycle. For example, if an individual reacts to a negative stressor (say the end of a relationship leads to increased social isolation as opposed to a healthier response of seeking social support), it is easy to see how the cycle can continue. While we often only consider personality traits in the context of personnel selection, it is also interesting to consider how they fit into the larger context.

Reid Klion

A Fortunate Job Loss

Reid Klion — Mon, 20 Jul 2009 18:16:48 GMT
Let me introduce myself, I am Gerry Groe, Managing Director of Business Development at pan. I joined pan on March 16 of this year after restructuring myself out of my position at Denny’s as VP- Human Resources & Organization Development. Denny’s is a fine company, but, like many consumer-oriented businesses, it has had to deal with a very tough, turbulent economy over the last two years. Consequently, in my OD role I worked with several outside consultants and the Denny’s senior team to restructure the company to better facilitate achievement of the company’s business strategy. As part of this new organization design, I actually suggested that my job be eliminated. Why, because HR was “top heavy” and we had the necessary senior HR talent to lead a leaner HR function. Fortunately, even before I left Denny’s I knew where I wanted to go - I wanted to join pan. While I have more than twenty years experience in VP and SVP – level HR positions with major corporations and as an OD consultant, by trade I am an industrial/organizational psychologist and I felt a strong pull to pursue a career with pan focusing on assessments - in my next posting I will explain why.

Gerry Groe

Rating Scales and Graduate School Admissions

Reid Klion — Thu, 16 Jul 2009 20:58:06 GMT
ETS recently rolled out a product called the Personal Potential Index. Intended for use by graduate school applicants, it is a standardized rating form with 24 items pertaining to factors such as creativity, communication skills, and teamwork that is completed by a set of student-nominated “evaluators” (typically professors). The intent is to provide graduate program admission committees with information beyond GPA, GRE scores, and the traditionally glowing letters of recommendation. (Exaggerated letters of recommendation are not a new problem; there is a classic American Psychologist article from 1966 on the topic rather brilliantly entitled “Mine Eyes Have Seen a Host of Angels.”)

The ETS tool is premised upon the recognition that non-cognitive factors likely contribute to success in graduate school (as well as virtually everywhere else in life) and part of a larger ETS project to find ways of assessing them. There are concerns about using self-report inventories in a graduate admissions context due to “faking” (though such instruments have been found to function effectively in pre-employment situations). As a result, attention is being directed to rating scales as well as biodata and situational judgment tests.

The Personal Potential Index appears to be a work in progress. There are plans for developing local norms as well as procedures to adjust for differences in individual rater severity. Additionally, only further research will be able to determine the extent to which this rating approach actually adds any incremental validity (and all of this is predicated upon the ability to collect enough real-life data). A concern that comes to my mind is the degree to which most professors actually have enough observational data to accurately rate their undergraduate students’ “non-academic” behaviors, especially in the case of institutions with large, often anonymous, classes. Additionally, one also wonders about the extent to which raters, knowing the implications of their evaluations, will engage in the same rampant exaggeration that is well-recognized in letters of recommendation.

Reid Klion

Hiring Temporary Employees

Reid Klion — Tue, 14 Jul 2009 13:36:47 GMT
One of my favorite places is the lawn at Verizon Wireless Music Center. Deer Creek, as it is still affectionately known by local residents, is an outdoor amphitheater located just north of Indianapolis. I have spent many nights there--enjoying alternative rock as a misunderstood (or so it seemed) teenager, classic rock as a cool (or so I thought) undergrad, and contemporary rock as a well-rounded (or so I hope) adult.

At a recent live show, I took notice of the hundreds of employees at the venue. They are all temporary or seasonal workers – Indiana weather being what it is, the music center is not open in the winter. I wondered how these people got their jobs. Some of the security guards looked as if their physical stature alone may have secured them a position. But what about the young man selling beverages or the ladies working the souvenir stand?

Temporary employees are a unique segment of the workforce. Selecting them can be a challenge because the jobs hold no promise of longevity or advancement and the most important requirements may be personality traits, such as dependability or a focus on satisfying customers. Staffing agencies and other organizations that hire temporary employees should consider pre-employment assessments as an effective and efficient way to gain information about job candidates without having to spend the time and resources necessary to conduct an interview. For instance, integrity tests can be used to predict counterproductive behaviors such as absenteeism and theft. Personality-based assessments can be used to determine how conscientious and reliable a person is, likely important in most if not all temporary positions. This type of assessment can also provide information on traits like assertiveness and stress tolerance which may be important in specific roles as security guards or gate/ticket checkers. Finally, biodata instruments can be used to assess a person’s fit for a certain position in customer service and sales by examining work experience, attitudes, interests, and values.

Jocelyn Courtney-Hays

Predictors of Long-Term Physical and Mental Health

Reid Klion — Tue, 07 Jul 2009 21:48:19 GMT
Most of us who work in the I-O arena are interested in looking at how personality and other characteristics predict things such as occupational success or leadership ability. However, there are also others who take an approach that is more oriented toward theory or research. A fascinating example is the Harvard Study of Adult Development (part of which is known as the Grant Study as it was originally funded by W.T. Grant, the founder of the now defunct chain of stores). Begun in 1937, it started studying a series of undergraduates who were at Harvard in the late 1930s to early 1940s and continued to track and collect data from them throughout their lives. The study is now in its 72nd year, and it is interesting to note that about half of the original subjects are still alive and now in their late 80s to early 90s.

A recent article in the Atlantic profiles the study as well as George Vaillant, the psychiatrist who has managed the project for the past 42 years. One of the many uses of data collected in the study was to identify the predictors of physical and psychological health. Among them are:

•   Learning to cope with life changes and stresses maturely by with methods such as finding healthy outlets for anger or disappointment
•   Education
•   Not smoking
•   Not abusing alcohol
•   Maintaining some level of exercise
•   Staying at a healthy weight

A study like this can also serve as a lens as to what research methods were considered important at the time. For example, in the early days of the study, subjects completed Rorschach tests, submitted handwriting samples, and had virtually every dimension of their body measured (recall that Sheldon’s concept of somatotype was still seen as having validity during that era). Now, surviving subjects take MRIs, provide DNA samples, and are being asked to donate their brains to the study.   The study, which has amassed over seven decades of data on it subject, will likely be a treasure trove for researchers for another several decades.

Reid Klion

Journal of Applied Testing Technology

Reid Klion — Wed, 01 Jul 2009 21:21:06 GMT
A bit ago, I blogged about the online journal, Practical Assessment, Research, and Evaluation. Another online journal also worthy of notice is the Journal of Applied Testing Technology. Published by the Association of Test Publishers and edited by Chad Buckendahl and William G. Harris, it is a peer reviewed journal that is concerned with issues related to assessment. The most recent issue focuses on the assessment of individuals with disabilities. Some other recent articles of note include a piece by Nathan Thompson that outlines the types of test administration methods that are available and tries to untangle acronyms like CAT, CBT, CCT, LOFT that psychologists are prone to throw around and a contribution by Richard Luecht that outlines some cost-benefit criteria to be used in evaluating computer-based testing models.

As is the case for virtually all the professional journals that I read, there is almost always at least one article in each issue that is of interest or applicable to something that I have been trying to sort out on my own. An added bonus with these two online journals, you can not only access them directly from your computer desktop, it is hard to argue with the cost of the subscription!

Reid Klion

Lemonade Stand Economics

Jenni Ginsburg — Mon, 29 Jun 2009 18:10:05 GMT

I was out with some friends one evening last week and came across a couple of little girls running a lemonade stand. It was hot and they were friendly, so I decided to stop and encourage their entrepreneurial spirit. They had two drink sizes to choose from (at different rates, of course). When I selected the smaller 25 cent size, the older girl took a paper cup, and filled it halfway, and then handed it to me. I gave her the money and caught up with my friends. I laughed when I told them about the half-full cup and joked that the girls were maximizing their resources in a difficult economic climate.

Then I realized that they were doing exactly what most of us are right now – trying to find a way to maximize our resources, minimize expenses, and get the most output. Jocelyn Courtney-Hays gave an e-seminar recently on the same topic: Two for One: Using Assessments for Selection and Leadership Development. Her presentation is an excellent resource to learn how organizations can increase their effectiveness with an assessment for both employee selection and development.

Jenni Ginsburg

On Background: Cognitive Aptitude Tests

Reid Klion — Thu, 25 Jun 2009 19:17:33 GMT
Knowing a job candidate’s cognitive aptitude or ability level is one of the most useful pieces of information in making an informed hiring decision. Cognitive ability is one of the best predictors of job success because it tells us both how quickly an individual will learn a job as well his or her likely level of work performance.

Cognitive aptitude tests are typically comprised of items that measure a job candidate’s ability to learn new material, use logical reasoning, or solve problems. Other cognitive aptitude tests measure more specific topics such as reading ability or math aptitude. The benefit of cognitive aptitude tests is that they are often quite valid in making inferences about job performance. They also tend to be both time and cost effective. However, something to be aware of is that cognitive aptitude assessments may be susceptible to adverse impact. That is, members of protected classes are prone to obtain lower scores on these tests than members of other groups. As a result, it is important that a careful job analysis be carried out before a cognitive aptitude assessment is implemented to ascertain that cognitive ability is critical for occupational success in the position at hand and that the cut score and decisions rules used with these tests are appropriate.

Importance of Test Security

Reid Klion — Wed, 24 Jun 2009 18:29:18 GMT
The press episodically covers incidents of cheating on exams. The high profile situations often seem to be found in university settings where large numbers of students are impacted. However, they occur in other programs as well. While these reports often provide commentators with the opportunity to bemoan the tragedy of “falling moral standards in today’s youth” or some such thing, it is important to realize that there were cheating scandals during the Han dynasty on the Chinese Imperial exams over 2000 years ago.

In a recent episode, our friends at Canada’s Public Service Commission (PSC) suspected that their Second Language Exam may have been compromised because students who attended one language training school did particularly well on the exam. (Being able to speak both of Canada’s official languages--English and Canadian French—can be a qualification for certain positions in the Canadian federal government so these exams can have rather high stakes.) In impressively thorough and transparent fashion, PSC shared the results of their investigation (that the school was providing its students with a practice exams practically identical to the actual exams but it couldn’t determine exactly how the copies were obtained) as well as next steps (retest everyone who attended this particular language school and accelerate development efforts of alternate test forms).

Obviously, test security is a critical issue in what we do, especially with cognitive and knowledge-based tests which are objectively scored with right-wrong answers. While web-based testing systems tend to increase security because they don’t require the distribution of content on paper--which can be notoriously difficult to track, readily misplaced, or copied--it is still incumbent upon us to be aware that test content is sensitive and should always be managed with security in mind.

Reid Klion

Social Networking and Assessment

Reid Klion — Fri, 19 Jun 2009 15:33:50 GMT
While I am almost 20 years removed from my high school days, it somehow became task to help organize a reunion this summer. When I looked at the lists from our 10 year event, I had 90 classmate email addresses on hand. Over a period of about six months and sending “spam” emails, I managed to add the whole of 10 additional addresses to my list. So, for a class of 600, I had 100 addresses and fears of a less than spectacular event. Knowing my high school reputation (or lack thereof) was at stake, I knew that I needed a new plan.

Back in the winter, a friend suggested that I go onto Facebook and see who had registered. To my surprise, it is like a little nostalgic reception where everyone is friends again. I had requests galore from people who wanted me to be their friend, join their cause, or be a part of their group. Over a six month period, I managed to accumulate over 300 friends, 6 groups, and no causes (fear of the unknown) . I made no efforts to actually search for classmates but my list organically grew to over 300 people, a 200% increase—and all without really trying!

My mind started the machinations about how we can we get these social networking sites to benefit folks in the assessment world and bring forums of like-minded groups together? I don’t have the million dollar idea yet, but I am interested in any feedback people have on discussion boards or social networking tools like Facebook, Twitter, LinkedIn, MySpace, and the like. We need to help get those interested in testing and assessment to find each other so that knowledge of best practices and resources are available to all. Like my reunion, it would be no fun if everyone can’t make the party.

Jared Owens

On Background: Personality Assessment Inventories

Reid Klion — Thu, 18 Jun 2009 19:14:24 GMT
Personality assessments focus on measuring work-related personality traits and psychological aptitudes. There are two major groups of personality assessment inventories. The first set measures a broad range of personality traits. These personality surveys are typically based upon the Five Factor model and cover the personality dimensions of openness to experience, conscientiousness, extraversion, agreeableness, and adjustment. These assessments are quite useful for personality profiling because they provide a broad view for understanding a job candidate’s psychological aptitudes. They are particularly useful in positions that involve interpersonal interaction, teamwork, and serving in leadership roles.

A second set of personality assessment inventories focus on more specific traits or attributes such as integrity, customer service attitudes, or sales ability. In contrast with broad band personality inventories, these personality surveys are only concerned with a specific set of behaviors. They are particularly well-suited for identifying job candidates who have the right psychological aptitudes for a specific job with well-defined requirements such as in customer service or sales.

Exploring Majors and Careers

Reid Klion — Tue, 16 Jun 2009 17:32:45 GMT
My sister-in-law recently graduated from college. I can’t tell you how proud my husband was to see his baby sister all grown up and ready to embark on a career and life in the “real world.” As I watched the graduates file across the stage, I wondered how many are truly ready to begin their careers. My sister-in-law spent time in college exploring different majors and used an internship to get a clear picture of life in the business world. However, I wonder how many of her fellow graduates followed a similar path. There’s no judgment here – truth be told, I selected Psychology as an undergraduate major because 1) I thought Psych 101 was interesting, and 2) I had run out of core curriculum classes to take. I was lucky in that after exploring other possible career avenues, I discovered Industrial-Organizational Psychology was truly the best path for me. How many other students aren’t so lucky?

Many college students may be surprised to by the number of resources that are available to help them choose the right major and occupation. Most, if not all, university career centers have assessments that can be taken to identify potential career paths. For example, the Career Exploration Inventory provides information about occupations, leisure activities and education and training opportunities that match an individual’s preferences. The Career Orientation Inventory assesses interests and then generates a list of the jobs and occupations that might be a good fit for the individual. Type-based personality inventories, such as the Golden Personality Type Profiler and the Type Dynamics Inventory can provide information on an individual’s work style and the kinds of environments and organizations in which he or she is likely to thrive. These are just some examples of a number of tools that can assist students in important decisions regarding not only which major they choose but also which extracurricular and professional development activities they may enjoy.

Colleges and universities should advertise the services they offer, including assessments and career counseling, to ensure students are aware of the help that is available. In addition, parents, professors, and other educators should encourage students to utilize their school’s resources and explore different majors and potential careers.   While assessments are an excellent way to start this process, job shadowing and internships, alumni networking, and extracurricular activities, such as on-campus clubs and volunteering, are also helpful tools to facilitate a student’s choice of major and eventual profession.

Jocelyn Courtney-Hays

Middle School Science Lesson - The Scientific Method

Reid Klion — Fri, 12 Jun 2009 20:03:56 GMT

In my spare time I serve as course coordinator for my golf league. We are fortunate to have a great turnout - so good, in fact, that the last group usually has had to play the last hole in the dark. After two consecutive weeks of this, I started wondering what changes I could make that might help avoid this situation.

Without realizing it, I followed the steps of the scientific method that many of us learned in middle school science - ask a question, do background research, form a hypothesis, test the hypothesis by collecting and analyzing some data, and then re-evaluate the validity of the hypothesis. I was able to test my hypothesis this week (that a 2 person scramble format instead of an individual stroke play format would lead to a faster round), but unfortunately it wasn’t supported because my group still didn’t get finished any earlier.

On the drive home this week, I was pondering my next hypothesis (maybe trying fewer players in each group next time), and realized that what I was doing wasn’t much different from the process used in employee selection. When implementing an assessment program, one of the first steps is to figure out what we want to accomplish and identify the skills or competencies required by a candidate. The next step is to research potentially useful assessment tools and then essentially run an experiment by implementing them. After some data are collected, it can be determined whether or not the hypothesized solution meets the organization’s needs.

Hopefully, we obtain the desired results but what if we don’t? Perhaps the job analysis data didn’t fully capture the job’s requirements. Alternately, the appropriate assessment wasn’t used. Another possibility is that the applicant population changed or that the wrong criteria were used to judge the outcome.

Even though my initial hypothesis wasn’t supported, I think that I followed the right process and learned something from my "study." Whether it is on the golf course or in the office, by proceeding systematically, reviewing the results, and then attempting to understand them, the important thing is to keep working to find the solution that achieves your goals.

Jenni Ginsburg

Managing Assessment Data

Reid Klion — Thu, 11 Jun 2009 20:43:56 GMT
Not too long ago, I met with an old friend who had just moved to a new company to become Vice President of HR. In his efforts to understand his new organization, he spent a few hours reviewing a random selection of employee personnel folders where he discovered that the results of each employee’s pre-employment selection assessment report were stored. After a bit more digging, he found out that not only were these test results readily accessible to every member of the HR department who used the employee files, but also that line managers (who were “trained” by the test vendor to interpret the results themselves) later used these test data to make decisions about promotions--sometimes several years after the test was taken!

Realizing something was awry, he asked for some guidelines. Some of my suggestions were:

Create a secure system for managing assessment data so that only appropriate personnel can access them on a strict “need to know” basis.

Remember that assessments are only valid for specific purposes. If an assessment is designed for pre-employment selection, it probably shouldn’t be used for promotional decisions, especially several years after the fact.

Use clear-cut and consistent decision rules for hiring and promotion developed by an assessment professional. Encouraging managers come to their own conclusions about test results based upon their own self-created interpretive rules will only lead to inconsistency, poor personnel decisions, and perhaps even legal consequences.

For those who are interested in a bit more detail, some colleagues and I presented on this topic at the Association of Test Publishers Conference in February 2009.

Reid Klion

Online Journal: Practical Assessment, Research & Evaluation

Reid Klion — Mon, 08 Jun 2009 17:08:11 GMT

I recently came across Practical Assessment, Research & Evaluation, a free online journal supported exclusively by volunteers that has been published 1999. It describes itself as soliciting manuscripts that are “scholarly syntheses of research and ideas about methodological issues and practices” in the area of assessment.

As might be expected, some of the papers that it publishes tend toward the technical (e.g., "Assessing unidimensionality: A comparison of Rasch Modeling, Parallel Analysis, and TETRAD"). However, there are also a number of articles that are likely to be of more general interest. For example, Wallace Judd, a pioneer in the development of performance testing, had a recent piece on the concept of gating items, items which an examinee must pass in order to pass the entire test. An example can be found on the FAA pilot’s test. After demonstrating his or her proficiency on pre-flight inspection, takeoff, navigation, and the like, the student pilot must then be able to land the plane. If potential licensees cannot do this in three attempts, he or she will not pass regardless of performance on prior items.

Some other articles that caught my eye were a discussion on the use of extended time accommodations , whether a hand-written vs. keyboarded essay responses are judged differentially (raters are more lenient with handwritten samples), and a two articles with advice on writing better multiple choice items

What impressed me the most about the journal were the quality of the content, the fact that it has been published for over a decade, and that the entire enterprise is based upon volunteer efforts. It also appears to have a wide readership as a counter on the table of contents indicates that articles from the journal have been accessed over 6,700,000 times since its inception!

Reid Klion

On Background: Pre-Employment Personality Tests

Reid Klion — Fri, 05 Jun 2009 19:08:49 GMT
A wide variety of personality inventories can be used for pre-employment selection. Some of these are broad band personality assessments which provide a comprehensive picture of an individual’s personality functioning and are typically based upon the five factor model of personality. Some of the most commonly used pre-employment personality tests of this nature are the Hogan Personality Inventory and SHL’s OPQ32. The benefits of pre-employment personality inventories of this type are that they provide an in-depth picture of a potential employee’s strengths and weaknesses and can also be used for post-hire development and training. A drawback of using these personality inventories for pre-employment assessment is that because of their breadth and comprehensiveness, they are sometimes more expensive and require more time to complete than other alternatives.

Another type of pre-employment personality test are those that only focus upon one or two aspects of personality, most often integrity or customer service attitudes. Here, rather than providing a broad picture of an individual’s personality, only the specific attributes seen as being most critical for job success are assessed. Some of the more widely used pre employment personality tests of this nature are Vangent’s Reid Report 29th Edition, Helm QWIK-RELY, and GNeil Service Ability Inventory. The benefits of using this class of pre-employment personality inventories is that they can be completed quickly and, given their lower price point, can be used as a screening tool early in the employee selection process. However, by design and, in contrast to broad band pre-employment personality tests, they do not provide a full picture of a potential employee’s strengths and weaknesses.

Monograph on Content Validation

Reid Klion — Thu, 04 Jun 2009 16:14:11 GMT

Earlier this week, Lori blogged about content-oriented approaches to test validation. IPAC (the organization formerly known as IPMAAC) recently published an extensive monograph by Charles Sproule on the rationale and research in support of using content validation models in personnel assessment. If you aren’t a member of IPAC, you can request a free copy of the paper here. (Hat tip to Bryan Baldwin of the blog HRTests for the link.)

While you are on the IPAC website, you can also learn a bit more about the organization if you aren’t already familiar as well as the details regarding its upcoming Conference (September 12-16, 2009 in Nashville, TN to be held concurrently with the IPMA-HR Conference). Several of us at pan have been active with this group for a number of years and have presented regularly at its meetings.

Reid Klion

Buy One, Get One

Reid Klion — Tue, 02 Jun 2009 20:41:19 GMT
I recently enjoyed a day of shopping with my girlfriends. As any woman (and any man who has shopped with one) will tell you, the stars must be perfectly aligned to have a good shopping experience. Luckily, on this trip they were. The babies were quiet and content in their strollers, flattering and stylish things were actually on sale, and no one went home empty handed. During this particular shopping extravaganza, I was also reminded of one of life’s little pleasures – the BOGO (i.e., buy one get one free).

It might be surprising, but the principle of the BOGO applies to employment testing as well.   Too often, I see clients use assessments for screening and selection and then stick the test results in a personnel file, never to see the light of day again. What many organizations fail to realize is that selection assessment data can also be useful for employee on-boarding and development. Even if the primary goal of assessment use is selection, why not take advantage of the BOGO opportunity and use it for development, too?

Of course, not all assessments are well-suited for both selection and development. For example, integrity tests are generally used only for selection, and 360° feedback instruments are designed specifically for development. However, if an organization chooses to use a personality or behaviorally-based assessment for selection, the results may be useful for post-hire purposes as well. For instance, after the assessment is used to identify high potential job candidates, the results may be shared with new hires and their supervisors so that they can start the job with knowledge of their strengths and areas for possible improvement. By immediately engaging the new hire in the development process, the organization is demonstrating its commitment to his/her success. In addition, the assessment data may reveal potential performance derailers that can be addressed to ensure that new hires perform their jobs effectively. After the on-boarding process is complete, the assessment data can be used to determine training needs and contribute to individualized development plans. Finally, the assessment data may be used in succession planning as an indicator of fit for higher level jobs or in different departments and divisions.

In these tough economic times, it is critical to maximize the utility of all HR processes, as well as other programs throughout the organization. Using assessment data pre- and post-hire not only gives an organization greater bang for their assessment buck, but it also enhances the selection system while facilitating the on-boarding and employee development processes.

Jocelyn Courtney-Hays

Reconciling Fact with Intuition

Reid Klion — Mon, 01 Jun 2009 12:43:23 GMT
While some relationships make intuitive sense, they just don’t necessarily play out in real life once you look at the data.

Consider the relationship between household books and children’s academic performance. Intuitively speaking, we would expect a strong relationship between book reading in the home and academic performance. However, according to Steven Levitt’s Freakonomics, reading many books does not correlate as strongly with children’s academic performance as does having many books in the home.

In the context of employment assessments, we would intuitively expect a score on a mechanical reasoning test to be predictive of a mechanical engineer’s on-the-job performance because the test content closely mirrors what he or she does on the job. Similarly, we might expect a mechanical reasoning test to be more predictive of a mechanical engineer’s performance than, say, a verbal reasoning test because the content of the mechanical reasoning test would be more similar to the actual job than the content of the verbal reasoning test.

While many in the assessment world continue to believe that a high degree of overlap between test content and job content results in greater predictive validity, the research does not support this. In fact, there is a preponderance of statistical evidence demonstrating that knowledge, skills, and abilities tests which are strikingly similar in content to the jobs they purport to test for may not be any more predictive of future job performance than tests with content that bears little to no resemblance to the job in question (assuming that these tests are scientifically sound).

Kevin Murphy, a renowned I-O researcher at Penn State University, cautions against making predictive validation claims based upon a content-oriented approach (see Murphy Article on the SIOP website). As he notes, “job-relatedness is neither necessary nor sufficient for demonstrating that people who receive high scores on selection tests will tend to perform well on the job.”

No one, including Murphy, is suggesting that we simply abandon the process of trying to align test content with job content. This process in and of itself not only has a positive impact on the acceptability of testing in terms of organization and applicant reactions, but it also contributes to the legal defensibility of employment tests and creates a more transparent test development and validation process that can be easily understood by clients. Likewise, there are times when content validation is the only validation approach at our disposal, owing in part to factors such as small sample size or lack of performance data.

In the end, although I-O psychologists cannot lose sight of the importance of job content when developing and implementing assessments, they are advised not to become overly reliant on content validation approaches in their quest to support the use of various assessments. Tests that measure job-relevant knowledge, skills, abilities, and attributes will likely be predictive of job performance but this assumption should be statistically supported whenever feasible.

Lori Ferzandi

Simply Un(Bee)Lievable

Reid Klion — Fri, 29 May 2009 14:26:04 GMT

Every spring, top competitors from around the country come together to see who is the best. Through preparation, pure skill, and talent, they put on a show in which only the strongest survive. I’m not talking about Kobe and LeBron and the NBA Playoffs or even Sid the Kid and the Stanley Cup Playoffs. No, I’m talking about something even more ruthless and pressure packed-- the Scripps National Spelling Bee!

Each year, I’m amazed by the talent and sometimes unintentional comedy that the Bee offers. It got me thinking about what an interesting form of assessment the Bee provides. Not only do these kids have to memorize and/or figure out how to spell hundreds of words, they have to show their skills on the national stage without making a single mistake. The sound of the bell that’s rung when a word is misspelled must give them nightmares. The kids (and I emphasize kids) are all obviously very smart, and its interesting to watch their talents on display. I can never figure out how being informed that the definition of a word is “a wooden barrel used to collect water in ancient Mesopotamia ” makes it any easier to spell, but to these kids it apparently does.

However, what really draws me to the Bee is how the competitors respond to the moment. Of course, there’s the infamous case of the girl screaming the letters of the word “euonym” so loudly that the judges almost seemed annoyed by the fact that she won. But then there’s the case of last year’s winner, Sameer Mishra, being so unflappable when given the word “numnah” that the comic relief he provided was a welcome respite from the tension of the competition.

Now that’s what I like to see in a Bee winner! Someone with more smarts than I can ever hope to possess who can remain calm under pressure and even manage to make me laugh. Other cases include the poor kid from a couple years ago who fainted during his turn, but got up, spelled his word correctly, and almost managed to win. There was also a kid from a few years back who finished second but was classy enough to cheer on his competitors on whenever they spelled a word correctly. These kids displayed the smarts and likability that will make them successful in the long run. Anyway, I wish all the Bee competitors luck this year, and I look forward to being amazed again.

Nate Studebaker

To Proctor or Not: Toward Some Clarity of Thought

Reid Klion — Wed, 27 May 2009 15:11:08 GMT
One of the recent hot topics in the assessment world has been debate over the use of non-proctored assessments (NPAs) for pre-employment selection. (For those not familiar, the typical NPA scenario is one in which the employer asks job candidates to complete an unsupervised online assessment as part of the application process.)

While the convenience of NPAs is substantial, a number of potential risks such as cheating, proxy testing taking (e.g., getting my smarter roommate to take the test for me), and item disclosure are also readily apparent. Quite a bit of attention has been directed toward finding ways to control the risks associated with NPA as well as trying to determine the real-life negative impact of non-proctored administration upon test validity (frankly less that I would have expected based upon what has been published and presented).

However, reasonable folks differ quite substantially in their comfort level with NPA and likely will for the foreseeable future. This is a literature I follow pretty closely because pan provides both proctored and non-proctored assessment services. (Here is my presentation at the last SIOP meeting on how technology can help mitigate some of the risks associated with NPA). And given NPA’s rather controversial status, I guess it isn’t surprising that writers haven’t always been particularly nuanced in their positions. Some conclude that NPA is either the best thing since sliced bread while others believe it’s a professional abomination to be avoided at virtually any cost.

Doug Reynolds and colleagues’ recent must-read paper in Industrial and Organizational Psychology does a very nice job of contextualizing the discussion. Their major point is that we need to attend to all the relevant factors when a web-based assessment program is implemented. The issue is not simply whether or not a proctor is present during the test session. Rather, consideration also needs to be paid to factors such as the type of assessment to be implemented, how it is used in the hiring process, the nature of the applicant population, as well as an evaluation of what the alternatives may be. As Reynolds paper concludes, “by positioning unproctored Internet testing as something that professionals should be either for or against, the underlying complexities associated with broader test deployment conditions will remain obscured.”

Reid Klion

Assessing Risk Through Assessment

Reid Klion — Tue, 26 May 2009 14:10:11 GMT

A recent Wall Street Journal article I came across described how the City of Los Angeles is using a new multiple choice test to try and predict the likelihood of a 10-15 year old joining a gang. The primary purpose of this assessment is to provide gang prevention programs where needed in order to reduce the risk these youths face. The assessment is still in early developmental stages and it will take several years for the city to determine its effectiveness.

Assessing risk through the use of behavioral assessments is nothing new in the employment world. Many companies use assessments early on in the selection process to screen out applicants that pose a risk, whether it is for safety, security, dependability, or other potential areas of risk. Test publishers have responded to the need for this type of assessment. We have seen a growth in the development of short, cost effective assessments that are specifically designed to be used early on in the selection process to assess risk.

These assessments have been shown to lead to reductions in safety violations and absenteeism in newly hired workers. Additionally, some of our clients have also seen direct cost savings with reductions in workers compensation claims. Finally, screening high risk applicants early on improves the efficiency of the hiring process by not spending time on unnecessary interviews. While the utility of this type of assessment is not going to be useful for all employment situations, it is certainly worth thinking about for jobs where risk is a concern.

Nate Studebaker

What I Learned from a Concrete Squirrel

Reid Klion — Fri, 22 May 2009 14:10:06 GMT

I take the same route to and from work every day. I’m sure others have had the same experience, but there are days when I get to work and don’t remember much of the drive. I don’t pay very much attention to my surroundings because the buildings I pass and the streets I drive on don’t change.

One afternoon at work, one of my colleagues (and fellow blogger), Reid, asked me if I had noticed the squirrel statue in the roundabout by our building. (If you aren’t familiar with a roundabout, it’s basically a four-way intersection, but it has a circular roadway around a central island). Even though Reid is a psychologist, I thought he had lost his mind. Surely if there was something as strange as a statue of a squirrel in the roundabout that I go through twice a day I would have noticed it. But sure enough, as soon as he pointed it out to me from our office window, I saw the squirrel positioned right in the middle of the roundabout. I couldn’t believe it and wondered how I could have missed something so obvious. Now I wonder what else I’ve missed not only in my surroundings but in my day-to-day activities as well.

It is easy for work processes to become like my commute – so routine that we switch to autopilot and perform our tasks without even noticing them. When was the last time you systematically evaluated your pre-employment hiring process? If you always go through the process in the same way without really looking at what you are doing, how will you make improvements? If you look for candidates without really thinking about what makes a person successful on the job, what are you missing? Perhaps we should take some time to step back and evaluate our processes and determine what it is that we are really trying to accomplish. We might find something important that needs to be altered (or at least something we hadn’t noticed before) that will help us be more successful. Feel free to call or email us so that pan can help you find your squirrel statue.

Jenni Ginsburg

Ladies and Gentleman, Start Your Engines...

Reid Klion — Thu, 21 May 2009 13:09:34 GMT
In Indiana, we are serious about sports– think the tradition of Notre Dame football or the movie Hoosiers. In Central Indiana, however, one sport reigns supreme: IndyCar Racing. I’m not talking about NASCAR where the cars look like street cars with top speeds of 180 MPH. No, I’m talking about open-wheel, 230 MPH racing. I’m talking about sports royalty – three generations of Andrettis, not to mention A.J. Foyt, A.J. Jr., A.J. III, and now A.J. IV who will be driving in this year’s 500. Right now, the local news is full of stories about familiar names – the likeable Helio Castroneves (better known for his Dancing with the Stars trophy and recent legal troubles than his two Indy 500 trophies), the always entertaining Danica Patrick, and others.

However, due to my husband’s penchant for racing, I happen to know a few things beyond these names. (OK, I admit it. Before I posted this, I had it fact-checked by him and his best friend.) Most importantly, I know that there is one determining factor that leads to a driver’s failure or success: the pit crew. No matter how good a driver is, a win is impossible without a good crew behind the pit wall. That means that in order to be successful at Indy, you need to have multiple people working as team, managing their own responsibilities, solving problems on the fly, and communicating effectively with each other.

In many organizations, just as in the world of racing, it is not enough to be a superstar. Of course, you have to be able to do your job well, but you also must function successfully within your work group or team. To ensure that your team is most effective, it is critical to develop each member to be successful both independently as well as with each other

In many organizations, just as in the world of racing, it is not enough to be a superstar. Of course, you have to be able to do your job well, but you also must function successfully within your work group or team. To ensure that your team is most effective, it is critical to develop each member to be successful both independently as well as with each other.

pan offers assessments and custom solutions focused on employee and team development. Let us help you build a winning team.

Jocelyn Courtney-Hays

Something Is Missing Here...

Reid Klion — Tue, 12 May 2009 19:38:49 GMT
Recently, I traveled to St. Louis to participate in a training class.

After a long drive, I arrived at a beautiful, newly remodeled hotel.

As I walked in, I was greeted at the front desk by a pleasant young woman named Heather. Heather gave me my room key, and I proceeded up the elevator to my floor. As I turned the corner, I noticed that my room door was propped open, and I started to think that something was wrong. I pushed open the door and what did I see? Nothing. No bed!

There was an imprint of a bed, but no bed. I called down to the front desk; Heather immediately apologized, said she would bring me a key to a different room (along with another treat), and told me my breakfast would be on the house. When she came with my new room key, she apologized repeatedly and offered to carry my bags to the other room. I declined, but was impressed with her willingness to take care of me.

I lugged my bags to the next room and saw the door propped open again! You guessed it. No bed! I called back down to the front desk and said, “Heather, you are never going to believe this. There is no bed here either!” I had to laugh because Heather was so confused by the situation that she was speechless.

She was incredibly apologetic again and immediately returned to the ninth floor to meet me. She offered me free room service that night, carried my bags to my room, and promised she would call and check on me later that evening.

After an hour, I finally had a bed. I ordered room service and Heather came to my room to make sure I was happy with dinner and the accommodations. I found out later that there was a convention the night before and the hotel had done some rearranging. The lack of beds was due to a communication failure between housekeeping and the front desk.

This could have been a truly miserable experience. Instead it turned into a comical mishap because Heather displayed a “Yes, I will take care of you attitude”. She had a drive for service. She was genuine, sincere, compassionate, and went the extra mile. She took a bad situation, resolved it quickly, and took the opportunity to make a connection with her customer.

Do you wish all of the people you encounter in the hospitality industry were like Heather? Do you wish your employees were more like her?
Are you missing out on candidates like Heather?

pan offers several pre-employment assessments that measure an individuals’ drive for service. In conjunction with one of our content partners, we also conducted an eSeminar to discuss how to select someone like Heather and the value it provides to an organization’s success.

Please feel free to post your comments - we would love to hear your hospitality stories, learn more about your current selection process, or share with you information about pan.

Sarah Lacey

Hiring, Stereotypes, and Susan Boyle

Reid Klion — Mon, 04 May 2009 19:14:15 GMT
While I have tried to avoid the massive media coverage of Susan Boyle (the Scottish singer whose sensibilities and personal style are somewhat less than high fashion), I came across a recent article that does a nice job of reviewing current psychological research on stereotyping.

From an evolutionary perspective, it was often life preserving to be able to tell whether the person approaching us in the forest was likely to be benign or pose an immediate threat to our wellbeing. Since aspects of the stereotyping process seem to be “hard wired,” we need to be aware of this and keep an open mind when meeting people. Indeed, much of the sensation about Ms. Boyle seems to lie in the fact that her musical skills were quite inconsistent with our initial impressions of her.

As might be expected, there is quite a bit of research examining how stereotypes can influence interviewers’ impressions of job candidates as well as how interviews are rather poor in predicting job performance. To this end, I think that one of the often unnoticed benefits of using a selection testing program is that unexpectedly positive test results sometimes force us to reconsider job candidates whose initial impressions (either in terms of looks, training, or background) are inconsistent with our stereotype of how potentially successful job candidates “should” appear. By failing to consider candidates who don’t necessarily fit the usual mold, we may be doing our organizations a significant disservice by missing potential star players.

Reid Klion

Why Assessment is Serious Business

TALX Admin — Wed, 29 Apr 2009 15:43:32 GMT

From a New York Post (April 19, 2009) article entitled BIG 'F' ON EMT ADVANCEMENT EXAM:

It was a massive medical failure for hundreds of FDNY medics who hoped to get promoted, as a measly 8 out of 721 city Emergency Medical Service workers passed the most recent lieutenant exam.

The 1.1 percent pass rate for the 2008 test is about 38 percentage points lower than the last time the exam was given, in 2004, when 1,044 medics took the test and 409, or 39 percent, passed, The Post has learned.

The FDNY was at a loss to explain how so many medics bombed the test. Entire Article

The issue here is that only 1% of the test takers passed the most recent exam! My initial conclusions are that the test was poorly constructed, the job candidates ill-prepared, or some combination of the two.

There are a number of factors which may account for this. It is even suggested later in the article that one factor may have been poor test taker motivation because the pay for lieutenants can be less than that for paramedics when overtime and meal allowances are considered (which still leads me to question why they would have taken the test in the first place).

Regardless, this only goes to highlight that using a test is a serious matter. As a result, if an organization decides to use an assessment, it needs to be sure that the test has been well-constructed, fits the requirements for the position, and is used in an appropriate way.

Reid Klion

Bob Hogan: Personality and the Fate of Organizations

Reid Klion — Tue, 28 Apr 2009 18:47:03 GMT

I just had the opportunity to read Bob Hogan’s recent book Personality and the Fate of Organizations. Always an entertaining writer and speaker (and long-time pan content partner), Hogan outlines how a leader’s personality quite frequently has a clear and direct impact upon an organization’s success or failure. By drawing upon examples from recent business news as well as history, he illustrates his arguments well.
One of the chapters that I found most engaging was “The Psychology of Managerial Incompetence.” Here, Hogan is quite willing to name names and outlines how what he calls the “dark side” of personality (or what clinicians would label a personality disorder) often lies at the root of executive—and organizational—failure. >
Another chapter of note traces what he calls the ‘personality wars,” the period between the mid-1960s and mid-1990s when there was remarkably little interest in either academic or business circles in the role of personality. In a rather fascinating bit of intellectual and cultural history, Hogan links this to a number of factors including academic debates about what personality theorists should study (neuroticism vs. self actualization vs. mathematically-determined traits), a distracting preoccupation with the role of social desirability in personality assessment, as well a burgeoning interest at the time in both humanistic psychology and behaviorism. (For those who are interested, he then goes on to link today’s resurgent interest in personality to factors including development of the five factor model and the pragmatic realization that adverse impact was nearly unavoidable when cognitive assessments were used for personnel selection.)

Hogan is a well-known personality theorist. As might be expected, the book is scientifically grounded. It is written in an engaging and conversational tone that would be accessible to the layperson, but is also sufficiently unique in its tone and approach to be of interest to professionals as well. I would strongly suggest a place for it in your library.

Reid Klion