Assessments are frequently used to select call center personnel. There are several reasons for this. One is the sheer number of call center positions with some estimating that over 4 million people do this work in the US alone, not to mention millions more in offshore locations. Another contributing factor is the nature of call center work. On the one hand, a high level of customer service and accuracy is critical in addition to computer skills, attention to detail, problem solving, and multi-tasking. Complicating things, call center representatives are often asked to help confused, angry, or upset callers in a professional manner yet do so as quickly as possible because a frequent performance metric is number of calls handled.  However, since the work is often routine and repetitive, it can become boring fairly quickly and lead to absenteeism and turnover unless the right employees are selected.

Fortunately, there are a number of assessments that have been developed specifically for call centers (and pan is fortunate to distribute a number of them). Some look at personality factors such as conscientiousness, customer service attitude, and reliability. Others, which are designed to test an individual’s call center skills, actually  place the individual in the simulated  role of a call center representative.  

A dynamic to consider when selecting call center personnel is that given the substantial cost in reputation that negative client experiences can bring coupled with large number of people employed in these positions is that even relatively minor improvements in the hiring process can have a substantial impact upon the bottom line.  A final consideration is that unlike many other jobs, call centers can often readily provide accurate and meaningful metrics about call center performance with indicators such a number of calls handled per shift, sales, and caller satisfaction surveys. As a result, it often can become apparent quite readily if a selection system is working and how it might be adjusted to further optimize the process.

Reid Klion

This is something we are often asked. In answering it, the critical issue to bear in mind is that validity has to do with the conclusions, inferences, or decisions we want to make based upon the assessment, not the assessment itself. Consider a typing test.  If it is reasonably constructed, it is pretty likely that it measures typing speed and accuracy. However, is it valid? Without knowing what the test is going to be used for, we don't know. If we are using it to select administrative assistants, we are likely on the right track if we assume keyboarding is important for job success in this role. However, by the same token, it is likely to be invalid for selecting groundskeepers.

Commercially available assessments have technical documentation that details how the test measures what it is designed to measure.  And this is certainly something that enters into the validity equation. However, the key question about validity has to do with the intended use of the assessment. The underlying question really isn't whether a test itself is valid but rather Valid for What?

Reid Klion

The Super Bowl, coming up in about a week and a half, will be a re-match of the 2008 game with the New England Patriots facing the New York Giants. An interesting fact is  how different the rosters of the two teams are today from four years ago. While the quarterbacks and coaches are the same, the Patriots only have 5 starters (of 24, counting the punter and kicker) from the 2008 team while the Giants have 9 remaining starters. (Going a bit further, there are only 7 Patriots and 16 Giants on the team roster in total today who were there 4 years ago.). While this is in part testament to the relatively short tenure of most players in the NFL (around 3.5 years on average) and, admittedly, two very good quarterbacks, it is even stronger evidence about the importance of selecting and training new talent. So while many of the individual players are different, the strength of the talent management skills of both these organizations is a constant thread.

Reid Klion

When we help a client to implement an assessment program, we are sometimes asked to use a “profiling” approach which involves giving the assessment to handful of the best performing employees, look at their scores,  and then use this to create a benchmark for hiring new employees. While this model is intuitive, quick, and easy to execute, it has a number of drawbacks. From a measurement perspective, the first challenge is that it is virtually impossible to create norms that are stable and meaningful when they are only based on a handful of cases. Secondly, since most assessments employ a large number of scales and sub-scales, it is rather difficult to decide which scales are actually predictive of performance without additional analyses of the job at hand and what it requires. (Alternately, it is possible that the selected assessment many not measure what actually differentiates high and low performing employees unless when have a sense of what the job requires when we initially select the assessment.)  Another factor to consider is that an employee’s skills may change and improve as a result of on the job experience. That is, successful employees may look significantly different after several months on the job than they did when they were hired. Finally, unless we also make a systematic job task analysis, we may lose sight of the likelihood that there are several ways for someone to be successful in the job.

The take-away lesson is that while “profiling” may give us a sense of the competencies of some of our most successful employees, it is not a sufficient foundation upon which to base an assessment program. Unless we take additional steps to fully understand the requirements of a position, we are likely to implement a solution that is less than fully effective. In the final analysis, our task is to match individuals to a job, not individuals to each other.

Reid Klion

In an informal discussion letter issued late last year, the EEOC outlined how requiring a high school diploma may lead to issues with the Americans with Disabilities Act. The EEOC points out that if an employer requires a high school diploma and that requirement removes from consideration someone

who is unable to graduate because of a learning disability that meets the ADA’s definition of "disability," the employer may not apply the standard unless it can demonstrate that the diploma requirement is job related and consistent with business necessity. The employer will not be able to make this showing, for example, if the functions in question can easily be performed by someone who does not have a diploma.

This clearly follows the long held principle that selection criteria must be based upon a “business necessity.”

Additional, since the ADA is involved, employers also need to consider the principle of “reasonable accommodation.”  Here, the EEOC goes on to say:

Even if the diploma requirement is job related and consistent with business necessity, the employer may still have to determine whether a particular applicant whose learning disability prevents him from meeting it can perform the essential functions of the job, with or without a reasonable accommodation. It may do so, for example, by considering relevant work history and/or by allowing the applicant to demonstrate an ability to do the job’s essential functions during the application process. If the individual can perform the job’s essential functions, with or without a reasonable accommodation, despite the inability to meet the standard, the employer may not use the high school diploma requirement to exclude the applicant.

None of these are new principles or concepts. However, it is important remember that whenever screen out criteria are used for job selection purposes, it is critical that they be job-related and care must be taken that they do not  take actions that unduly discriminate against individuals with disabilities without consideration of reasonable accommodations.

Reid Klion

There was a recent study published in Personnel Psychology looking at the development of executive leaders. The research question examined the relative roles of cognitive ability, personality factors, and work experience in predicting strategic thinking (which was measured by a series of five assessment center exercises). Based upon a sample of over 700 executives at an international consulting firm, it was found that cognitive ability (incidentally assessed by the Watson- Glaser Critical Thinking Appraisal and the Wesman Personnel Classification Tests, both published by Pearson and available on pan) was by far the best predictor of strategic thinking. Second most important was work experience. More specifically, it was found the broader the experience, both in terms of range of work activities and level of leadership, the more beneficial it was. The personality factors, openness to experience and extraversion, also made statistically significant contributions but to a much lesser degree. The takeaway lesson from this study is that if you are looking to develop individuals who can become strategic leaders, it is most important to find the brightest people and then provide them with a broad array of diverse work experiences.

Reid Klion

Last week, I blogged about how people are prone to rely upon intuitive rather than more empirical decision making strategies. Another related issue has to do with the difficulty most people have in understanding statistics and what they mean. For example, we tend to be more biased by salient events and discount the actual underlying probabilities. A classic illustration is parents who are hesitant to fly on the same commercial flight for fear they might both perish in a crash and subsequently orphan their children but not think twice about riding in the same car on a daily basis. (For those interested in the math, there were no deaths on any U.S airline carriers in 2010 while there were nearly 33,000 automotive fatalities.)

This point was highlighted this morning when I was listening to sports talk radio show on the way to work. The hosts were debating a controversial decision by an NFL coach that past weekend that led to a loss. An expert on football statistics had written that while the coach’s decision was considered risky by traditionalists, it actually had a greater chance of success than the alternative course of action based upon actual game data (47% vs. 43%). One of the hosts (a former NFL quarterback and graduate of a Big Ten university) denigrated this thinking and complained that this was “only statistics and statistics don’t always work out.” He then went on to support this with his experience in playing blackjack in that even when he made the statistically correct decisions, he would sometimes lose several consecutive hands.

Reid Klion

I was listening to sports talk radio this morning and one of the hosts (a former NFL player) observed that professional sports is one of the few businesses where incumbents are expected to help train the individuals selected to become their potential successors.  The typical scenario is that when new players are drafted or otherwise acquired by a team, the veterans will share advice and help them to get up to speed with the hopes of their eventually becoming contributing members of the organization. However, it is likely that if a rookie does well, he will take the job of one of the people who helped train him.

Certainly there are business contexts where incumbents will help groom their successors. However, this is usually a planned process where there is some degree of acquiescence on the part of the incumbent. In sports, however, it is rare for a player to leave an organization voluntarily.  Rather, the standard situation is that someone comes along whom the team believes can do a better job and the incumbent is either relegated to the bench or simply released from the team.  An interesting bit of food for thought…

Reid Klion

For the past several years, the use of e-mail addresses essentially has become the default way of contacting someone. However, unlike your family name or street address, it is quite easy to obtain an e-mail address to your own liking or even have as many as you would like. In this light,  I have often been struck  when job applicants opt to use e-mail addresses that might be suitable for use with their friends but likely to cast them in an unfavorable light by a potential employer (e.g., luv2partyhard@mail.com or PowerSlacker@mail.com).

There was a recent study in SIOP’s journal The Industrial-Organizational Psychologist that looked at the relationship between the use of questionable e-mail addresses when applying for jobs and test data.  Based upon large sample job applicants, it was found that applicants who used inappropriate e-mail addresses scored lower on measures of conscientiousness, professionalism, and work-related experience but not cognitive ability. It should be noted that while the differences on the first three measures were statistically significant, the effect sizes were not. Thus, it certainly is not suggested that candidates be screened out solely on their use of a questionable e-mail address. However, as the authors point out, it might be appropriate to view this as something as a “yellow flag” in the selection process as a potential indicator of a candidate’s potential lack of awareness of social norms and appropriate workplace behavior.

Reid Klion

Last week, I had the pleasure of speaking at the SHRM Missouri Conference on the use of assessments in HR practice (and would be pleased to share the slides if you are interested; just drop me an e-mail). One of the topics that came up was the policy of some organizations refusing to consider candidates who are currently unemployed.

The logic apparently underlying this practice is that the unemployed are unqualified because their current lack of work is reflective of some personal limitation such as poor work motivation or general incompetency.  This is problematic on several grounds. First, given the current economic climate where the unemployment is greater than 9%, it would seem logical that there are huge numbers of people who are unemployed due to no fault of their own. Secondly, if even if we do assume that unemployment can reflect an individual’s  poor work potential (which indeed may be the case in some situations), it makes no sense to use current employment as an absolute minimal qualification. Not only is this unfair, it is also quite likely to remove otherwise highly qualified persons from the applicant pool which impairs the organization’s ability to find qualified workers. Certainly, it is reasonable to ask unemployed interviewees how they have maintained (or improved) their work skills since their last job but it seems quite misguided to dismiss them unilaterally from further consideration.

As might be expected, this practice has generated quite a bit of press, most of it being negative. Indeed, a bill was recently introduced in the U.S. House of Representatives that forbids employers and employment agencies from screening out job applicants solely because they are out of work.

Reid Klion

Adverse impact occurs when a selection method results in substantially fewer members of a protected class being selected compared to majority groups. In contrast to discrimination, adverse impact pertains to selection methods that appear to be neutral but still result in disparate hiring rates. For example, adverse impact upon women tends to occur when physical ability tests are used and for African Americans when measures of general cognitive ability are employed.

However, adverse impact needs to be understood as a function of the entire selection system, not the test itself. One factor relates to the decision rule or score that an individual must obtain to qualify for the job. For instance, requiring that a job candidate be able to lift a 15 pound box from the floor to a 2 foot shelf 5 times in a minute is unlikely to have adverse impact for women while being able to bench press 200 pounds 5 times very likely would. However, another critical factor is the candidate pool and the necessity of encouraging qualified members of minority groups to apply for the position at hand.

This point was recently highlighted by the outreach efforts of the New York City Fire Commissioner toward potential African American candidates. The New York Fire Department unfortunately has had a long history of discrimination against minority candidates which very likely discourages qualified minority candidates from applying to the department which only exacerbates the issue. However, even in other situations where there has not been a history of discrimination, effective recruitment efforts and the attraction of well-qualified candidates are often key factors in limiting adverse impact as well as increased the quality of the applicant pool. It also emphasizes the point that adverse impact is not simply a matter of the test itself but the entire system in which it is used.

Reid Klion

Based upon work started in Canadian medical schools over 10 years ago, there is an increasing use of serial brief structured interviews for admissions purposes. Called the multiple mini-interview (MME), each candidate rotates through a series of 8 interview rooms. Prior to entering the room, candidates  have 2 minutes to review a non-medical scenario  posted on the door that taps into non-clinical skills like critical thinking and ethics (e.g., should a doctor endorse herbal remedies if her patients fervently believe in them even though she knows they are ineffective?) and then discuss it for 8 minutes.  Each interviewer is trained in the use of a standardized rating scale.

There are several motivations behind the MME. First, it has been well-documented that unstructured interviews are poor predictors of work performance and are prone to all sorts of biases, including bad weather on the day of the interview.  Secondly, it has become broadly recognized that non-cognitive skills—such as empathy and the ability to work in teams—are critical success factors in a wide variety of jobs, including medicine.

Initially, it was hoped that the measures derived the MME would be independent of GMAT scores. However, subsequent research has found some shared variance between the two (which really shouldn’t be surprising given the broader I-O literature indicating that cognitive ability is able to partially predict virtually every aspect of job performance). Regardless, the MME does appear to bring value to the medical schools that use it, and its practice has spread to other medically-related fields such as dentistry, physical therapy, and veterinary science.  This is also another interesting example of how a structured interviewing can be incorporated into a selection process.

Reid Klion

An interesting question that often comes up is whether an organization should hire an individual who seems to be overqualified for a position. This perhaps has come up more frequently as of late as a result of the economic downturn where people are applying for a much broader range of jobs than they might have otherwise (though it should be noted that a late 1990s lawsuit ruled in favor of a Connecticut police department that declined to hire an applicant whose scores on a cognitive exam were deemed to be “too high.”)

This issue was discussed in a recent series of articles in the journal Industrial and Organizational Psychology where a number of well-taken points were raised. First, it should be noted that "overqualification" should be viewed as something to be understood from the applicant/employee perspective because there are substantial numbers of workers who consider themselves to be “overqualified” for their current positions even though an objective review of their background would not indicate this to be the fact. Additionally, even if someone’s overall educational attainment is greater than might be required for a job, they may not be overqualified in terms of specific skills (e.g., consider the individual with an advanced degree in English seeking work as an apprentice electrician). Another factor to consider is an individual’s motivation for seeking the position in question. Given the growing number of older individuals who are employed, there is also a subset of overqualified individuals who specifically seek such positions because they want a less demanding and more flexible work role that will permit them to pursue non-vocational interests and activities.

Looking to the research literature, there aren’t much data to suggest that hiring overqualified workers is necessarily detrimental to an organization. To the contrary, it is likely that hiring such individuals may actually bring benefits. Even while overqualified workers may not have long tenure in a position (which is often cited as a fear in hiring them even though it is not well-supported by research data), they may bring outside perspectives to an organization that are quite valuable. Further, overqualified workers may provide a pool of workers who are well-suited for promotion. Finally, there are data to suggest that these overqualified individuals may do better on the job than their peers.

Reid Klion

The program for IPAC’s 2011 Conference (promotional video) to be held in July 17 to 20 in Washington, DC has been posted. Presentations will cover a wide variety of topics related to assessment and selection including developments in testing methods, technological innovations, legal compliance, and leadership development. Of special note, pan’s Jocelyn Hays will speak on using assessments to identify high potential employees. As in the past, pan is pleased to support IPAC 2011 and is a Gold Sponsor of this year’s meeting.

Reid Klion

A question that often comes up when implementing an assessment program is whether job candidates should be allowed to re-test. In most situations, it is good policy to permit it. For example, an individual may not have been feeling well, was excessively anxious, or for some other reason wasn’t able to show the best of his or her abilities on the day of the first test. Additionally, it is often important to allow re-testing for the sake of an organization's reputation because prohibiting it will often cast it in a bad light. Indeed, re-testing is also encouraged by professional standards as long as it is done in a reasonable way that protects the integrity of the assessment program.

That being established, the question then becomes what impact does this have upon test scores? Deidra Schliecher and associates recently published a paper in the Journal of Applied Psychology that analyzed testing data from over 2000 candidates who applied for a professional-level government service position in the US. Looking across the battery of tests that were administered, scores increased by .1 to .4 of a standard deviation depending upon the test. (This is consistent with other research indicating that scores do tend to increase somewhat upon re-testing.) Complicating matters a bit, there were some sub-group differences in the rate of improvement. For example, whites improved more than African-American and Hispanic candidates (though African-Americans improved more than other groups on the structured interview.) Also, younger candidates improved more than older candidates while women improved more than men.

The end result is that re-testing (while considered a reasonable policy) can slightly increase the level of adverse impact for some groups though this will vary as a function of factors such as the selected cut score and differential levels of re-testing between groups. Of course, this study is limited by the fact that this is only for one position, and these sorts of findings can vary significantly based upon the specific population at hand.  However, the takeaway message is that all assessment programs need to be monitored on an ongoing basis, not simply for compliance purposes but to ensure they continue to meet the needs of the organization.

Reid Klion

Collecting solid outcome data is a critical step in evaluating the effectiveness of an assessment program. Unless we have good data about employee performance, we cannot reliably or consistently differentiate between employees who are doing their jobs well from those who are not. Unfortunately, rather than using formal performance metrics, many organizations rely upon global impressions of employee performance. A typical response is "I have been running this unit for 7 years. I know who is doing good just by seeing their daily work." Unfortunately, decades of research indicate that such informal global assessments are prone to "halo" effects--meaning that if an employee is well-liked, they tend to receive good reviews while those who are not favored are judged more harshly. This becomes a problem when we need to evaluate the effectiveness of an employee selection program.  Simply stated, if there are not good data on actual employee performance,  we can’t determine who well the system works in selecting employees.

Indeed a recent survey by pan content partner, SHL, indicates that only half of all organizations actually collect formal metrics to support the value of their HR investments. Not surprisingly, over a third simply rely upon the belief of hiring managers that their programs are effective. Here, we see the same reliance upon global impressionistic data to evaluate the effectiveness of selection processes that is unfortunately often used to evaluate employees themselves. When we consult with clients to implement an assessment program, we also encourage them to create good metrics to evaluate their employees if they don’t already have a system in place. This not only provides critical data that will enable us to actually see how a selection process is working (and adjust it if need be), but it is also sound HR practice.

Reid Klion

Registration is now open for the 2011 Conference of the International Personnel Assessment Council. To be held July 17-20 in Washington, DC, the meeting promises to be of great value to anyone working in the area of selection and assessment. Guest speakers include Elaine Pulakos (PDRI), Wayne Camara (College Board) and Eric Dunleavy, Mike Aamodt, and David Cohen (all from the Center for Corporate Equality). There are also a number of pre-conference workshops on topics including how to design an assessment strategy, best practices in questionnaire design, strategies for scoring job simulations, and job task analysis. pan is again proud to be a sponsor of the conference and that Jocelyn Courtney-Hays will be speaking.

Reid Klion

There have been seemingly countless discussions about the potential risks associated with unproctored assessment. Facilitated by the development of internet-based testing, the benefits of having job candidates complete assessments without supervision are rather clear and include increased convenience as well as cost and time savings compared with proctored assessment. As might be expected, risks associated with non-proctored assessment center around concerns about whether candidates are receiving unauthorized help in completing the test (either by consulting with others or using various tools and resources available on the Internet or elsewhere) as well as the fact that candidates are completing assessments  non-standardized in settings of their choice. Unfortunately, these discussions have been almost exclusively theoretical in nature as there has been little empirical evidence to serve as a guide.

A recent study may shed some light on this question. The paper  looked at the very question of how the validity of non-cognitive assessments (personality and biodata-based tools) may change when they are moved from  proctored to non-proctored modes of administration.  In general, it was found that these assessments largely retained their ability to predict job performance when candidates completed them without the benefit of proctoring. Of course, this is a single study (albeit a large one with one analysis involving over 1200 candidates and the other about 3300) that only involved  a handful  of tests. It also should be noted that personality and biodata assessments are not objectively scored and makes “cheating” on them more difficult than knowledge-based assessments. However, these findings do  support the notion that, at least in some settings, test validity may not be significantly compromised when assessments are delivered in a non-proctored fashion.

Reid Klion

Charles Handler, an I-O psychologist, assessment industry commentator, and friend of pan, recently wrote an interesting piece on “A Pre-Employment Assessment Candidate Bill of Rights” where he outlines a number of “rights” to which he believes test takers are entitled.  For the sake of discussion, I will leave aside debate as to whether what are described are actually “rights” as might thought of in the same terms as the Bill of Rights or more accurately “best practices” or “good ideas.”  That being said, he raises a number of provocative points, and several are worthy of comment.

Candidates have the right to an assessment experience that is of a reasonable length.

True but this is often a balancing act between collecting enough information to make a reasonable decision about an applicant’s viability as a potential employee and not overtaxing a candidate’s time Here, there can be a tension between recruiters who favor a briefer experience and I-O psychologists, tending to be a bit obsessive by nature, who will almost always opt for collecting more, rather than less, information when given a choice.

Candidates have the right to know where the assessment fits within the overall hiring process, and what they can expect next.

I couldn’t agree more that we should be transparent about the assessment process and “next steps.” This also seems important for recruitment and PR purposes since job candidates may be clients of the organizations where they are seeking work,

Candidates have the right to a good user experience.
Candidates have the right to technical support, no matter when they are applying.

Here, too, I would completely concur that assessments should employ good user interface design, be  easy-to-use,  and as foolproof as possible. Given the advances in web-based technologies over the past 10 years, there is little reason to not accomplish this nor to provide competent technical support if problems arise.

Candidates have the right to assessment content that appears job related.

Well, I am not necessarily convinced of this in all cases. Part of the issue is that what doesn’t appear to be job-related to candidates may very well be critical to job performance to those who understand the job. For example, someone may think that a test of cognitive ability may not be relevant to a position when it is well-known that cognitive assessments are often the best predictors of future job performance. Also, biodata assessments are often quite predictive of job performance even though the items may not necessarily appear to be job-related to an external observer.

Candidates have the right to an enjoyable assessment experience.

Another “right” I am not terribly convinced of. Our business is assessment, not entertainment. The primary goal of a testing program is to provide a fair determination of each candidate’s ability to succeed on a job. While we certainly may want to design as assessment process that is engaging, our primary goal is to provide a valid assessment of the candidate’s ability, not necessarily to provide a fun experience.  

As always, Charles is a provocative writer. This article is certainly not an exception and definitely worth a look.

Reid Klion

A few months ago, the EEOC held hearings on the use of credit checks in the hiring process. It is estimated that 60% of employers use credit checks for at least some positions. This is an important topic given the increased attention that the EEOC is now paying to this issue and a recent suit filed by the EEOC against Kaplan Higher Education in this regard.  It is also worth noting that the EEOC has reported an increase in the number of charges filed based upon the use of credit checks in pre-employment selection.

The testimony highlights a number of important issues:

• The stipulations of the Fair Credit Reporting Act (FCRA) apply when credit data are used in hiring. The most important in this context is that if candidates are  not being hired because of their credit report, the employer must inform them of this and provide a copy of the report. This often will lead to a discussion between the organization and candidate about any mitigating factors and also provides the opportunity for the consumer to correct the record with the provider of the data if there are inaccuracies.  The FTC recently created a summary of the implications of the FCRA for job  candidates.

• The credit report obtained by employers contains applicant credit history but not his or her  credit score. As a result, there is not a single summary score for employers to use. Rather, they need to draw their own conclusions from the provided data. 

• Some protected classes (e.g., women, Latinos, African Americans) tend to have lower credit scores than other groups. As a result, credit data will tend to have adverse impact when used for selection purposes.

• As always, though, a selection method that has adverse impact can be used defensibly if meets a “business necessity.” For example, the federal government often uses credit checks in the hiring process for certain positions of trust. However, when an employer opts to use a process that has adverse impact, it is much easier to defend if there are not alternative ways to obtain the same result with a lesser degree of disparate impact.

• Michael Aamodt, an I-O psychologist (who will be an invited speaker at the 2011 IPAC meeting), provided some very useful data about the validity of credit data in pre-employment selection. Unfortunately the research base is limited, and he could only identify 10 studies in the literature (with five using self-reported financial distress v. credit data and three of these involving law enforcement positions which may not generalize well to other jobs). Based upon his meta-analysis, he found an uncorrected correlation of .14 between credit history and counter-productive work behaviors, .07 with performance ratings, and .20 with attendance. These correlations are lower or at best equal to those found when personality assessments are used. However, the literature is quite limited on the question.

To my way of understanding the situation, credit data can definitely play a useful role in the hiring process for certain positions but, like any selection “test” (and this would be considered from a legal perspective to be a test just like a psychometric or physical ability assessment), it needs to be job-related and meet a business necessity. If a position requires an individual to have good creditworthiness (for duties such as handling money or being in a position of trust), it would seem quite reasonable to consider the use of credit data. However, if the credit data are being used to make other inferences about the applicants (such as whether there are conscientious or will be good workplace citizens), personality-based assessments are either as or more effective in doing so, have a track record of not having adverse impact, and are likely the best option to first consider in such a situation.

Reid Klion

A question that is often heard in regard to assessment is whether a candidate should be allowed to re-test. A recent article by John Hausknecht may shed some light on the issue. The paper looks at an analysis of a large sample of job applicants in the hospitality industry who took a personality assessment as part of the selection process. First, it should be noted is that candidates who re-test are those individuals who were not hired initially but opted to persist in attempting to get the job. Also, a large proportion of the persons in the study sample who re-tested were internal candidates, meaning they were already employed by the organization; as a result, they may have had some insights into the purposes of the testing process.

Regardless, the re-testers were able to increase their scores to a moderate degree, especially on the scales which were initially quite low.  (This is a somewhat different from other studies that looked at individuals who took the same assessment when hired and then later for developmental purposes; in these situations, scores tend to stay fairly stable.) The takeaway message from this study is that scores on personality assessments can increase if job candidates are permitted to re-test. Accordingly, organizations should take this into account when crafting re-test policies and consider using assessment models that may help to limit a candidate’s ability to increase his or her re-test score by implementing assessments with multiple forms or considering the use of assessments that control for socially desirable responding.

Reid Klion

One of the major benefits of using an assessment is to corroborate an individual’s qualifications for a position. As noted in a prior posting, it is unfortunately the case that job candidates may exaggerate their academic or professional experiences or present (or even purchase) credentials from schools that are less than reputable. The result is that assessment results (if the program is appropriately administered) may provide one of the few solid sources of information that a potential employer has about an applicant.

The importance of verifying an individual’s qualifications was highlighted in a recent article about the proliferation in China of fake credentials within the professional ranks and widespread plagiarism in scientific journals (which is important in academic settings where importance placed upon publication record). For example, there are individuals who claim to have medical degrees (who only took a correspondence course) and the recent case of a Chinese airline that discovered over 100 of its pilots had embellished their flying histories. A complicating factor in China is widespread cheating on tests which only highlights the importance of using good testing practices in high-stakes examination programs. While there are no easy answers, the use a good assessment program can play a critical role in evaluating job applicants by providing reliable and verifiable information about a potential employee.

Reid Klion

A number of us at pan are active in IPAC--the International Personnel Assessment Council. In addition to sponsoring an annual conference and supporting a newsletter and listserve, IPAC also sponsors webinars on topics related to selection and assessment. Julia Bayless, IPAC’s President Elect and Director of Talent Development at Sodexo, recently gave a great presentation on job analysis. She did a particularly nice job of covering both the conceptual issues involved in job analysis as well as the practicalities and challenges often associated with actually getting one done in the field.  For a limited time, both a recording of the presentation and the slides are available to the public.

Reid Klion

Whenever an assessment program is implemented, it is critical to understand the extent to which it may have adverse impact. (To review, adverse impact is  defined as a “substantially different rate of selection in hiring, promotion, or other employment decision which works to the disadvantage of members of a race, sex, or ethnic group.”  For example, women tend to have lower scores on tests of physical strength.)  The basic rule is that if the selection rate for a protected group is less than 80% of that for the majority group, adverse impact is seen as being present.

However, the practicalities of actually calculating these statistics requires a number of decisions upon which there is little guidance but can have a substantive impact upon the final results. In order to address this issue, the Center of Corporate Equality created a technical advisory report based upon the input of 45 experts in the field.  Some of best practices outlined include reinforcing the difference between a job seeker and a job applicant (because seekers are not included in EEOC analyses), not “guessing” the race or gender of an applicant who declines to self-identify, not counting as applicants those  job seekers who either withdraw or fail to follow-up, and that EEOC statistics should be calculated for each sub-group individually (with the practice of combining them all into a “total minorities” group not being recommended). There is also a lengthy section on the statistical methods that can be used in adverse impact analyses.

The document is long and technical in spots (to be expected given its very nature) but it provides a good reference and source of guidance in an area where it is much needed.

Reid Klion

The interview has a very well-established role in virtually every selection situation—be it for a job or getting into school. This is despite the fact that it has been well-established that unstructured interviews tend to have very limited validity. (However, in the interest of full disclosure, I would not be comfortable in hiring someone without interviewing him or her first.)

One of the factors that contributes to the limited efficacy of most forms of interviews is that there can be a number of subjective factors that influence how interviewees are rated that have no relationship to their actual qualifications. A good illustration can be found in a study looking at the impact of weather upon medical school admissions interviews. By looking at the records of nearly 3000 candidates at a major Canadian medical school, it was found that interview ratings were lower on rainy as opposed to sunny days.  While the rating difference was only 1%, this still had an impact upon admission decisions because a substantial number of candidates obtain scores that cluster near the cut score.  The result is that the chances of getting into medical school decrease for borderline candidates if they happen to interview on a bad weather day.

Reid Klion

Chicago has a long history of difficulty with its public safety testing programs. For example, there was a Supreme Court ruling last year that allowed litigation to continue in relation to a fire department assessment that was administered in 1995. Now, there are reports that Chicago may do away with its police testing program to save money, avoid the potential for litigation, and boost minority hiring.

It is difficult to understand how this is a good business decision. Not using a test certainly does not ensure against litigation because any process used to evaluate a job candidate is legally considered to be a “test” and subject to the same legal considerations as a formal psychometric assessment program. Additionally, other cities have managed to develop public safety recruitment and selection programs that have yielded a diverse police force. Finally, and perhaps most importantly, a well-designed assessment program is the fairest and most effective means of selecting job candidates.  Removing the objective data from the decision making process and simply depending upon the subjective review of job applications is likely to result in more difficulties and expense in the long run than would developing a validated testing program when the costs of a poorly selected police force are considered. Additionally, not using a formal testing program would make Chicago unique amongst major city police departments.

Reid Klion

I heard a presentation the other day by Dan Putka of HumRRO about the role vocational interest in job selection. To this point, the received wisdom has been that vocational interest has little value in predicting job performance after the Big Five personality factors and cognitive ability are taken into account. While counter-intuitive, this finding has been replicated repeatedly in the literature and has become quite well-accepted.   Putka and his colleagues hypothesized that part of this may be due to the fact that most measures of vocational preference were developed to measure occupational interest (e.g., what are the jobs that match my interests) and not designed for selection (e.g., what jobs do I want to do). For example, while some may enjoy fixing things around the house, they may not necessarily be interested in an occupation that involves this.

Based upon this assumption, they created a job interest inventory for selection that targets the sort of work an individual would be interested in having. Using in a very large scale study with the military, they found that job interest was more predictive of both intent to stay in a position and job knowledge than were either personality factors or cognitive ability. They hypothesize that while initial performance may be better predicted by other factors, whether an individual will have a long-term emotional investment in a job (as evidenced by the desire to stay in the job and learn more about it over time) are better predicted by interest. The article is currently in press at the Journal of Applied Psychology.

Reid Klion

The issue of “faking” or impression management on personality inventories is a fascinating topic. It  was recognized as a potential concern in the 1930s, and has both led to nearly 1000 publications and (in the words of Robert Hogan) “consumed the best minds of the discipline.” Despite this attention, the field has not reached any definitive conclusions other than the recognition that personality assessments can be highly effective tools for personnel selection when used in the right context even though some candidates very likely exaggerate their positive attributes while minimizing the negative.  

A recent paper by Bernd Marcus, a German social psychologist, recently cast this in a new light. Marcus argues that we need to understand the applicant-employer dynamic from the applicant’s perspective. Some of the points he makes are that both the applicant and employer are engaged in goal-directed behavior. The applicant wants to induce the employer to hire him or her while the employer’s task is two-fold: to evaluate the candidate but also encourage the applicant to take the job if offered.  Essentially, both parties are in the process of negotiating a long term relationship (perhaps not dissimilar to dating). As a result, it is no surprise both will be on their best behavior. In this vein, just as applicants are unlikely to disclose much negative about themselves, this is also the case for the employer. For example, how often does a potential employer openly share things like the reason why the position is being filled, concerns expressed by incumbents about the heavy workload associated with the job, or disclose other negative factors about the organization or position in question?

Reid Klion

The EEOC just recently announced that it was suing FAPS, a New Jersey company that processes newly imported cars for sale in the United States. While we won’t know about the truthfulness of the charges until the legal process is completed, the allegations against FAPS provide examples of the sorts of things that employers clearly need to avoid.

FAPS was found to have a statistically smaller percentage of African American employees in entry level positions than would be expected given the demography of the area (northern New Jersey). First, it is alleged that the organization relied almost exclusively on word-of-mouth recruiting which led to limited number of African Americans applying for jobs. This recruitment model would likely be seen as having adverse impact because it in and of itself is not deliberately discriminatory but did result in decreased numbers of African Americans applying for work. (This also illustrates how the concept of adverse impact applies to the entire selection process, not simply assessments). However, it is also further alleged that when African Americans did come to apply, they were told that no jobs were available ever though they were. This would be considered to be discriminatory because the intent is deliberate. Making matters worse, the EEOC also claims that the company asked questions about physical impairments on the application form which is a violation of the Americans with Disabilities Act.

Reid Klion

The National Security Administration (NSA) recently posted a video attempting to quell applicant anxiety about taking a polygraph test as part of its pre-employment selection process. Whether this has the desired impact or not remains to be seen. However, the use of lie detectors for job selection has a long and interesting history.

The "father of polygraph" was a Harvard psychologist named William Marston (who later went on to fame and fortune as creator of the “Wonderwoman” comic book character). It was thought that the truth of a candidate’s responses could be determined by measuring changes in blood pressure, breathing, and other physiological indices. Unfortunately, data supporting the validity of lie detectors has never been strong, and a federal law passed in 1988 largely prevented the use of polygraphs for pre-employment screening other than by the government in the case of national security, defense or law enforcement. (It is also interesting to note that the development of “integrity” or work attitude tests in the late 1980s was spurred by this law and the desire to find more effective ways to select good employees.)  While the polygraphs’ proponents acknowledge its limitation, they support its use continued as an interrogation aid in encouraging individuals to reveal information that they may not have otherwise.

Reid Klion

A while ago, I blogged about the growth of “boot camps” for pre-school admissions tests but there now seems a growing backlash against the use of standardized testing programs for the kindergarten set. While the agency which sponsors the exam indicates that test scores have stayed stable, there are concerns about the validity of the test due to the growth in popularity of prep classes.  Also, some schools have become concerned about the importance which has been placed on the testing program itself. However, it seems that these issues will continue for some time given the competition for placement in prestigious elementary schools in Manhattan and the overall social and cultural atmosphere. As one mother says: “We’re in a competitive city,” she said. “People want their kids to excel, and this is a differentiating factor. You can help.”

Reid Klion

Virtually anyone who has been in the position of interviewing job applicants is aware of the importance of not asking questions that may lead to discrimination on the basis of factors such race, age, and national origin. Additionally, like any sort of pre-employment assessment, interviews must pertain to job requirements and the applicant’s ability to do the job. As such, I was surprised to hear recent  reports about NFL teams asking one potential players if he was gay and another if his mother (who had been arrested previously on drug charges) was a prostitute.

Here is a recent article by a sports attorney on this very issue who makes a number of interesting points. One is that pre-draft interviews technically might not be considered to be pre-employment interviews because drafting a player only gives a team the right to negotiate to hire him. As a result, the rules regarding pre-employment interviews may not pertain. Also, while some states do prohibit discrimination on the basis of sexual orientation, it is not forbidden by federal law. Thus, not hiring someone because of sexual orientation may be illegal in some states but not others.

Reid Klion

Emotional intelligence (EI) is a topic that has garnered substantial interest over the past several years, especially in the business literature. It should be noted that EI has not been without its critics due to concerns about how the concept itself has not been well-defined and the fact that several measures which purport to measure it are not necessarily well-correlated with each other.

A recent study looked at how well measures of EI might be able to predict work functioning. Some of the interesting findings were that personality and performance-based measures of EI are not highly correlated with each other, that EI is more highly correlated with measures of personality than overall intellectual functioning (which argues against the proposition that EI is actually a form of intelligence), and that measures of EI contribute little new information in terms of predicting job performance after personality and intelligence are accounted for. This is not necessarily to say that measures of EI are not useful, especially in coaching or in a developmental context. However, since EI tends to have such a substantial overlap with measures of personality, it often adds little new information in selection settings. Additionally, since the pattern of personality traits which predict job success typically will vary from position to position, measures of EI, which are comprised of an agglomeration of personality traits, may not be particularly useful when applied across a diverse set of jobs simply because the requirements for those jobs may differ rather substantially. Thus, a high level of EI may predict performance in one job type but not another.

Reid Klion

Charles Handler, president of Rocket-Hire and a friend of pan, almost always has something interesting to say about the world of assessment. He wrote a recent piece about the factors that should lead an organization to consider using psychometric assessments that is worth a read.

One he identifies is that if the organization relies heavily on unstructured interviews. While many are very much wedded to them, the research data are fairly clear that unstructured interviews tend to have little if any predictive validity. As a result, using a psychometric assessment that is only modestly predictive of occupational outcome will yield better results than the typical interview. Other factors he mentions include organizations with either high turnover or high volume hiring situations. In both these instances, a well-validated assessment will almost inevitably increase the efficacy of the hiring process because its impact will be readily apparent given the current state of affairs. Finally, one issue that he also notes is a topic I blogged about a few days ago. That is, regardless of how an organization selects employees, its selection processes are legally considered to be a test. As a result, if an unstructured selection process is being used that isn’t based upon a job analysis, a properly implemented psychometric testing program will almost inevitably decrease an organization’s legal exposure (as well as very likely improve the quality of its hiring decisions).

Reid Klion

A question we often hear about assessments has to do with the legal issues which surround their use.  A critical point that sometimes gets ignored in these discussions is the fact the same legal principles pertain when any method is used to select employees, whether testing is part of it or not. A good example is a recent EEOC suit that Walmart settled for nearly $12 million due to gender discrimination in hiring distribution center personnel for a facility in Kentucky. Essentially, based upon the assumption that the positions were not suitable for women, Walmart only selected men for these jobs.

In contrast to most testing cases, this did not involve adverse impact which occurs when the differences in selection ratios between groups are due to inadvertent factors. (For example, if Walmart had used a physical ability test with a cut score that was higher than necessary given the demands of the job, this would have been a case of adverse impact because it would have eliminated a large proportion of female applicants.) Rather, this was a case of gender discrimination where women simply weren’t considered. However, the critical point is that the same EEOC stipulations apply to how employees are selected and hired--whether tests are used are not.

Reid Klion

Winfred Arthur and colleagues recently published a paper that looked at the extent to which job candidates may try to cheat on cognitive assessments. Since cognitive assessments are objectively scored as either right or wrong, job candidates can try to improve their performance by doing in things such as seeking the assistance of others in responding to items or accessing items from those who have taken the test previously.  In order to explore this, the authors obtained test data from several thousand job candidates who had completed a cognitive assessment as part of a hiring process (a high stakes situation where it is assumed there would be motivation to optimize one’s performance) and then asked them to repeat the test a second time as part of a research study (where the stakes are assumed to be much lower). If cheating were widespread, we would expect these scores to go down when the test were repeated when the stakes were lower and there was not motivation to obtain a higher score. In contrast, they found that scores actually went up in the low stakes setting when the candidates repeated the test. (It is likely that this increase was due to a practice effect.)

In terms of drawing conclusions, it should be noted that this study used a timed assessment which likely decreases the opportunity for job candidates to seek assistance during a test session as well as the fact that a relatively small sample of subjects agreed to complete the assessment a second time. However, this finding is quite consistent with other research which indicates that the validity of most assessments is not significantly compromised when they are administered in unproctored settings.

Reid Klion

Here is an update on the EEOC’s E-RACE (Eradicating Racism and Colorism from Employment) Program. Ongoing for the past several years, E-RACE focuses on race and color discrimination in the workplace with a particular focus on employment screening methods (such as background and credit checks) that are prone to have adverse impact. Here, it is important to recall that adverse impact occurs whenever a seemingly neutral “test” unintentionally results significantly lower selection rates for protected groups of persons. (A classic example is that if the physical ability requirements are extreme, many women will be disqualified from certain labor jobs.)

Since credit and background checks are quite susceptible to having adverse impact, businesses must be sure that their use is job related (just as they must for any assessment method that has adverse impact). Not only is this consistent with the law, it also prevents the needless disqualification of job applicants who otherwise may become excellent employees.

Reid Klion

One of the major reasons to use an assessment is to ascertain if a candidate is actually capable of doing the job. While there other ways to accomplish this, such as reviewing references and resumes, these approaches can fall short. Unfortunately, employers often do a poor job of actually checking references and many of us don’t verify a candidate’s educational history. (There haven been some well-documented cases of people in high profile positions who don’t have the background they claimed. For example, there is the former director of admissions at MIT didn’t have any of the degrees listed on her resume and George O’Leary who was the head football coach at Notre Dame for 5 days until it was discovered that both his claims to have played college football and having a graduate degree were false.)

Another challenge in credentials verification is the apparent growth of degree granting universities that may not have particularly stringent academic standards. There is growing concern  about U.S. military personnel getting degrees (almost wholly paid for by the government) from propriety, for profit institutions. While the students who enroll in these programs do so with good intentions, they often discover their degrees are not well-regarded by employers because of the nature of the schools. In a similar vein, there is a physics professor at the University of Illinois who has essentially been on a crusade against diploma mills, businesses often based outside the United States where one can essentially purchase a degree without fulfilling any requirements at all. When it comes to vetting a job candidate’s credentials, it is important the employers have a way of verifying that an individual actually has the qualifications to do the job well. Whether it be with looking into a candidate’s credentials or using an assessment, this is critical information to have before making a hiring decision.

Reid Klion

A recent article by Filip Lievens, Deniz Ones, and Stephan Dilchert looked at the relationship between personality and medial school success. This  follows the path of other researchers who are looking at non-cognitive factors in predicting academic outcomes.

They followed 600 Belgian medical students longitudinally throughout their academic careers. (Differing from the US model where medical school is essentially a 4-year graduate program, European students typically enter medical school directly from high school in what is a 7-year program.) They found that personality differentially predicted medical school outcome based upon the changing demands of medical school. (Especially in the European model, the first few years are comprised almost exclusively of basic science while latter years have an increased focus on clinical work and patient contact.) For example, openness and extraversion had increased validity in predicting performance over the 7-year period. Similarly, they found that while conscientiousness predicted grades throughout a medical student’s career, its validity increased from .18 to .45.

These specific findings aside, the article also is a good reminder that it can be important to study the validity of a predictor over time, especially in situations where task demands are dynamic and change predictably over time.

Reid Klion

The Merit Systems Protection Board, a federal agency charged with ascertaining that other federal agencies carry out personnel-related activities in compliance with the law, recently released a report on the use of job simulations in assessing job candidates.  While written specifically for the federal context, it is a good source of information to anyone interested in the topic.

For those who may not be familiar, a job simulation presents a realistic work situation and assesses how an applicant responds to it. When you think about it, the set of tasks that might be considered job simulations is rather broad to include things like having welders demonstrate their skills on standard tasks, having pilots demonstrate their skills in a flight simulator, or a store manager asking applicants how they might respond to specific customer inquiries. While they may differ in terms of fidelity to actual work tasks, job simulations often provide good information about a job candidate’s skill level with a high degree of face validity.  The MSPB report discusses the different types job simulations, their relative benefits and weakness, as well as how to create and conduct them. The document also goes on to detail how simulations might be more broadly used by federal government. Some very useful information at the right price...

Reid Klion

We are often concerned about the role of self-presentation during hiring situations. For example, much attention has been devoted to understanding hown “faking good” may impact personality assessments, the potential role of cheating on exams, and the fact that people tend to exaggerate their qualifications during interview situations. That being the case, at least the folks who engage in these behaviors are making an effort to present themselves in a positive light (and, as have some like Bob Hogan argue, the ability and willingness to present oneself positively is often a critical job qualification in and of itself).

In contrast, consider those individuals who fail to put their best foot forward when applying for a job (and the reaction this tends to elicit from hiring personnel). For example, virtually all of us can recount instances of job candidates who have multiple misspellings on their résumés, manifest a complete lack of phone etiquette, and the like.  To this end, a recent CNN article took an informal survey of hiring managers on some of the grossly inappropriate behaviors they have seen during interviews. These include job candidates answering cell phones during an interview, acting bizarrely, and being completely uninformed about the company or what it does. While it has been well-established that non-structured interviews are poor predictors of future job performance, it is also clear that certain interview behaviors are too salient and extreme to be ignored even by the most heavily research-based interviewer.

Reid Klion

Jack Jones, Ph.D.,  respected colleague and Vice President of Human Capital Sciences at Vangent (a long-time pan content partner) has a recent article in Talent Management on how poor economic conditions may increase the prevalence of problematic behavior at work. While some of the factors cited include the fairly obvious such as increased general stress and financial needs, he also outlines how more subtle dynamics such as decreased levels of supervision due to staffing cuts or even poor maintenance of a facility’s physical plant can also contribute. The latter may be related to the “broken window” concept in urban sociology which argues that if vandalized windows are quickly repaired, it tends to dissuade further acts of minor anti-social behavior in the neighborhood. Similarly, if there is an increased sense of disorder at work, it may influence the increased expression of deviant behavior.

The article goes on to describe the extent to which candidates will admit to specific counterproductive behaviors as assessed by the Reid Report with the most common  being fired from prior jobs, having multiple unexcused absences from work, and giving unauthorized discounts. A brief case example of ROI is provided citing a retailer that estimated an annual savings of $6 million by screening out high risk candidates.

Reid Klion

John Callender (Biodata Online) and Jack Smith (Kingwood Group), two respected testing and assessment experts and long-time pan content partners, will  discuss how the current economic climate impacts recruitment and assessment strategies.  Some of the topics to be covered include how investing in assessments makes economic sense and the potential benefits and consequences of shifting to a lower cost assessment program. Please join us!

Thursday, September 17 @ 1PM ET     Register

Reid Klion

Rob Tett and colleagues published a recent paper in the Journal of Occupational and Organizational Psychology that looked at personality test norms. Examining a large set of data from the Hogan Personality Inventory, they took the norms derived from several large samples of individuals employed in sales, trucking, clerical work, finance, and management. No big surprise, there were some rather significant differences in average scale elevation between groups (e.g., the truckers were substantially less sociable than the sales group). However, a bit unexpected were the differences in mean scores between samples within the same job family. As an illustration, some samples of sales personnel had much higher scores on scales measuring ambition and sociability than did others. Indeed, there were cases where the differences within job family were nearly as great as those between job families.

There are a handful of takeaway points to consider. First, relying solely on job type to select a normative sample can be problematic. Rather, factors such as organizational culture of the work group where the samples were obtained or whether the data were derived from job applicants vs. incumbents need to considered. (The latter is important because applicants tend to obtain higher scores than incumbents, something I will blog about in the near future.)  Secondly, this article also analyzed the impact of sample size on the reliability of normative samples. As a saving grace, they found that a sample of 100 is usually quite sufficient to develop a good set of local norms.

Finally, it should be noted that the importance of norms varies greatly depending upon how a test is being used. For example, if I want to know how the conscientiousness of the  maintenance personnel  in my company compares to that of folks who work in maintenance for other organizations, the normative sample chosen is obviously critical. However, if I am only interested in ranking maintenance job candidates on conscientiousness or have already conducted a validation study to determine cut scores, the use of norms isn’t particularly relevant or useful as we could use untransformed scores for these purposes.

This week, I had the opportunity to work with a coworker on two different projects. The first was within her area of expertise, and I needed her help to complete my responsibilities. The second was in my area of expertise, and she came to me for assistance.  It occurred to me that in my job, I rarely accomplish goals by myself. I nearly always rely on others to help me get the job done, even if it is just bouncing ideas off of my boss before implementing a solution for a client.

In a previous posting, I wrote about the importance of developing employees to help make them productive team members. However, it is also important to look at an individual’s propensity for teamwork during the selection process. Assuming the job for which you are hiring does, in fact, require teamwork, you may benefit from using selection assessments and behavioral interview questions to identify job candidates’ tendencies towards collaboration and cooperation. Job candidates whose assessment results or interview performance indicate high levels of traits, such as extroversion and agreeableness may be more effective on the job-- simply because they are more likely to work with others to accomplish organizational goals.

Jocelyn Courtney-Hays

While perusing Amazon.com for good books recently, I came across the Penn and Teller tome, How to Play in Traffic.  Generally, I wouldn’t mention such mundane details of my life, but I wanted to make note of it here.  You see, it has come to my attention that there are some people who make playing in traffic their full-time job.

You may remember from my previous post that I am an Indy 500 fan. However, the excitement here in town is that NASCAR will be at the Indianapolis Motor Speedway this weekend for the Brickyard 400. In almost any form of racing, a key part of any race are the pit stops where tires are changed, and…well, I actually don’t know what happens aside from the tires, but I’m sure it’s all very important and mechanical.  While I was at the 500, it occurred to me that the people who work in pit row are dealing with cars moving in and out of the pits at high speeds.  I have read stories of injuries, from the minor to the severe, occurring due to pit crew members coming too close to the racecars, and seeing the pit stops in person made me realize just how easily injuries could happen.

There are probably a number of important attributes that make for a good pit crew member.  I’m willing to bet that in addition to having a high level of mechanical skill and knowledge, you also need to be diligent, conscientious, and focused on safety – not only for your own well-being but also for that of your driver, fellow pit crew members, and competitors.  How many other jobs require this same commitment to safety?  I can think of a few-- assembly line employees, truck drivers, healthcare workers.  Given the inherent risk in these jobs, it is important to hire people who will be safe at work. While training is an important part of workplace safety, there are also a number of assessments that can help identify job candidates who are (and are not) well-suited for work in dangerous environments.

Jocelyn Courtney-Hays

When I was in high school Biology during the early 90s, my teacher spoke of a new emerging technology called the Internet. It had been around for a while but had not really caught on outside of scientific and academic circles as computers were for geeks, and most people didn’t see their benefit over using a typewriter. He spoke of the ability to access information about breakthroughs in science as if you you were down the hall from the actual experiment and having immediate access to information from all over the world. Most people thought this type of technology was for large companies like AT&T and IBM, and the idea a personal computer could retrieve information from some nebulous place on the other side of the world was completely foreign and pretty much inconceivable.

In my naiveté, I did not realize he was really referring to the emergence of the global community. The Internet is something which joins everyone together and enables interaction with someone across the world without ever leaving the comforts of the living room. Transactions can be conducted by anyone with access to a computer, and what was once an exotic item from a foreign country is now a few clicks away on sites like Amazon and EBay. All that is required is the belief that the person on the other side of the connection is legitimate.

In the past, the beginning of trust was largely based on who people commonly knew. If a friend endorsed someone, that person was probably trustworthy.

A few weekends back, I was at home watching a golf tournament on TV.  As I’m sitting on the couch, I see the familiar faces of Tiger Woods, Phil Mickelson, and Anthony Kim making their way around the course.  But during one stretch of action, something caught my eye – player after player wearing Nike apparel.  I thought back to my junior (pre-Tiger) golf days when Nike had a limited golf line with just the basics – shirts, shoes, socks, hats, and the like. Now, though, you can head to your local sporting goods store and pick up Nike golf balls, bags, headcovers, and even clubs.  It is amazing what one endorsement contract has done for that company.

Thinking about what Tiger Woods did for Nike Golf, I wonder if organizations encounter the same situation. Can one superstar employee make a huge difference to a company? Probably so – a story in Inc. cited a case where a programmer at a technology company developed an idea that eventually became a seven-figure revenue generator. In light of stories like that, you have to wonder, What could the one right person do for my company?  And more importantly, how do you find that employee who can help take your business to another level?  

I think there are multiple answers to that question. First, organizations should use the right pre-hire selection criteria. Utilizing valid pre-employment assessments, behavioral interviews, and situational judgment tests can help you to identify high potential job candidates who possess characteristics such as diligence, creativity, and ambition. Once employees are on the job, it is important to foster an organizational culture that encourages idea sharing and open communication. You could host an online discussion forum for idea sharing or sponsor idea generation drives to motivate employees to “think outside the box.” Finally, it is important to invest in employee professional development. Providing opportunities for individuals to grow and learn may lead them to develop ideas they never would have thought of otherwise. In addition, investing in your employees’ development is likely to increase their organizational engagement and may motivate them to contribute at an even higher level to the overall success of the business.

Jenni Ginsburg

One of my favorite places is the lawn at Verizon Wireless Music Center.  Deer Creek, as it is still affectionately known by local residents, is an outdoor amphitheater located just north of Indianapolis.  I have spent many nights there--enjoying alternative rock as a misunderstood (or so it seemed) teenager, classic rock as a cool (or so I thought) undergrad, and contemporary rock as a well-rounded (or so I hope) adult.  

At a recent live show, I took notice of the hundreds of employees at the venue. They are all temporary or seasonal workers – Indiana weather being what it is, the music center is not open in the winter.  I wondered how these people got their jobs.  Some of the security guards looked as if their physical stature alone may have secured them a position.  But what about the young man selling beverages or the ladies working the souvenir stand?  

Temporary employees are a unique segment of the workforce. Selecting them can be a challenge because the jobs hold no promise of longevity or advancement and the most important requirements may be personality traits, such as dependability or a focus on satisfying customers. Staffing agencies and other organizations that hire temporary employees should consider pre-employment assessments as an effective and efficient way to gain information about job candidates without having to spend the time and resources necessary to conduct an interview.  For instance, integrity tests can be used to predict counterproductive behaviors such as absenteeism and theft. Personality-based assessments can be used to determine how conscientious and reliable a person is, likely important in most if not all temporary positions. This type of assessment can also provide information on traits like assertiveness and stress tolerance which may be important in specific roles as security guards or gate/ticket checkers.  Finally, biodata instruments can be used to assess a person’s fit for a certain position in customer service and sales by examining work experience, attitudes, interests, and values.

Jocelyn Courtney-Hays

Being from Northwest Indiana, my husband is wholly dedicated to the Chicago Cubs. Sadly, the Cubs have always struggled. (Their last World Series win was in *sigh* 1908.)  The team’s current problem: poor offense. While the pitching and defensive strategies appear to be working, the Cubbie bats are not swinging like they should be. In mid-June, the Cubs hitting coach fell victim to the slump and lost his job. One Cub player said, “He’s taking the blame for something that’s our fault…” Let’s assume that the coach wasn’t responsible for their poor performance. Yet when results weren’t achieved, he was held responsible. As the team’s offensive leader, the buck stopped with him.

I think the principle of leaders being held accountable for their team’s performance holds true in many organizations. To be effective, managers must ensure their employees are effective. Investing in employees’ professional development not only benefits the employees, it benefits their managers as well. Think about your own organization: Are you dedicating effort to developing your people? Are your superiors committed to your development?

If the answer to either of these questions is no, consider what might be done to change it. Assessments are a great starting point for development because they can provide information about job-related areas where the individual is strong and areas he or she needs to develop. Assessments can be combined with other tools such as interviews, simulations, or role plays to collect comprehensive information about the individual’s strengths and weaknesses. From there, the individual can build a development plan and be coached to improve. Given how much is riding on the effectiveness of your employees, these kinds of development programs can lead to big returns.

Jocelyn Courtney-Hays

Knowing a job candidate’s cognitive aptitude or ability level is one of the most useful pieces of information in making an informed hiring decision. Cognitive ability is one of the best predictors of job success because it tells us both how quickly an individual will learn a job as well his or her likely level of work performance.

Cognitive aptitude tests are typically comprised of items that measure a job candidate’s ability to learn new material, use logical reasoning, or solve problems. Other cognitive aptitude tests measure more specific topics such as reading ability or math aptitude. The benefit of cognitive aptitude tests is that they are often quite valid in making inferences about job performance. They also tend to be both time and cost effective. However, something to be aware of is that cognitive aptitude assessments may be susceptible to adverse impact. That is, members of protected classes are prone to obtain lower scores on these tests than members of other groups. As a result, it is important that a careful job analysis be carried out before a cognitive aptitude assessment is implemented  to ascertain that cognitive ability is critical for occupational success in the position at hand and that the cut score and decisions rules used with these tests are appropriate.

I love watching sports and the Met's recent loss to the Yankees due to Luis Castillo’s fumble stunned everyone!

I’ve been spending a lot of time at the baseball diamonds watching my son play Little League, on a team ironically called the Mets.  One kid on his team who talked about Castillo’s now infamous dropped ball claimed that he really could have caught it himself. I believe him. Here’s why:

Growing up and playing collegiate athletics taught me many life lessons including the importance of routines and how we all can drop the ball if we don’t use proven techniques that lead to success. For example, using two hands to catch a fly ball is a proven method that leads to success in baseball and softball. We learn this at an early age, but we still see Major league players failing to use both hands all the time.

In the world of employee selection, process plays the same important role. Using a selection assessment that measures exactly what it needs to measure is extremely important. If your organization needs to select safety conscious individuals, but your assessment is measuring the ability to work in teams, this method is unlikely to lead to much success in the long run. Dropping the ball in the selection world may mean you are missing out on a great candidate or selecting someone who may not be the right fit.

My grandma used to say, an ounce of prevention is worth a pound of cure. We can all learn from Luis Castillo and make sure we prevent dropping the ball by using a proven process that leads to success.

Sarah Lacey

Last week, my wife and I celebrated our one year anniversary by taking a trip to San Francisco.  We spent a couple of days in the Napa/Sonoma Valley area tasting and learning about wine.  Being wine novices, we were pretty fascinated by all the factors that go into the taste and quality of each of wine.  It turns out there’s more to making wine than just fermenting some grapes.  Not only are there the grapes themselves to consider, but the soil they’re grown in, the amount of moisture where they are grown, and even the elevation.  Another major factor we learned was the importance of the type of barrel used.  It was interesting to learn how a barrel made from an oak tree in France leads to a different taste than a barrel made from an oak tree in Missouri.  This multitude of factors made what seemed to be a fairly straightforward process into something very complex and interesting.

Being the I/Oish person that I am, I found the complexity of factors that go into making a good wine to be not so different from the complexity of factors that go into finding a good job candidate.  Now of course I’m not talking about storing candidates in barrels to see how well they age, but often times searching for a quality job candidate is mistaken as a fairly straightforward and obvious process.  The truth is that most of the time there are a number of factors that go into what it takes to be successful on a job.  As a result, it is important to use appropriate selection tools available to ensure that we’re accounting for these critical factors.  This can include assessments, behavioral interviews, job simulations, and other kinds of job related tools.  The key is to take the due diligence to appreciate the complexity of truly finding quality job candidates.  Without accounting for this complexity we can be left with sour grapes.

Nate Studebaker

Personality assessments focus on measuring work-related personality traits and psychological aptitudes. There are two major groups of personality assessment inventories. The first set measures a broad range of personality traits. These personality surveys are typically based upon the Five Factor model and cover the personality dimensions of openness to experience, conscientiousness, extraversion, agreeableness, and adjustment. These assessments are quite useful for personality profiling because they provide a broad view for understanding a job candidate’s psychological aptitudes.  They are particularly useful in positions that involve interpersonal interaction, teamwork, and serving in leadership roles.

A second set of personality assessment inventories focus on more specific traits or attributes such as integrity, customer service attitudes, or sales ability. In contrast with broad band personality inventories, these personality surveys are only concerned with a specific set of behaviors. They are particularly well-suited for identifying job candidates who have the right psychological aptitudes for a specific job with well-defined requirements such as in customer service or sales.

When I was growing up, the typical procedure for landing a seasonal summer job was to fill out a brief application (i.e., name, age, work experience) and simply keep your fingers crossed.  The challenge of course was convincing the person in charge of hiring that despite not having any formal paid work experience, you would make a great employee at the tender age of 16.  Back then, we typically found ourselves in competition with other equally and minimally qualified teens for entry-level positions in grocery stores, restaurants, and the like.

This summer, teens looking for short-term employment are facing much stiffer competition as they attempt to beat out more experienced and educated adults who are currently unemployed as a result of the ongoing economic crisis.  So what is a hiring manager to do?  

Managers should be encouraged to find the means to compare inexperienced teens with more experienced applicants on a level playing field.  Specifically, hiring managers need to delve into the underlying personality characteristics that have the potential to make teens more viable candidates than their older counterparts.  While many organizations may not have the financial resources to evaluate seasonal candidates with formal personality assessments, they do have the option to utilize behaviorally-based interviews as an alternative, effective and cost-efficient selection procedure.

Thanks in part to a recent initiative rolled out by the Federal Government, the Workforce Investment Act, $1.9 billion in stimulus funds have been set aside for the purpose of training youth for both summer and year-round jobs.  According to Dr. Andrew Sum, Professor of Labor & Economics and Director of the Center for Labor Market Studies at Northeastern University, there are multiple benefits to providing teens with early work experience.  First, teens who work at ages 16 and 17 are able to command higher wages at 19 and 20.  Second, working at an early age helps disadvantaged kids to stay in school.  Finally, employers are more likely to provide subsequent training to teens who have previous work experience.

Organizations must now follow the government’s lead in recognizing the importance of employing teens in the workforce.  As a result of the job opportunities they are afforded, teens help to keep companies in business, particularly when customer activity is substantially increased during summer and holiday seasons.  Separately, teens often use the money earned from seasonal jobs to save for upcoming college tuition and related expenses.  Excluding this segment from consideration for ‘09 summer employment could have long-term ramifications.  As pointed out by Dr. Sum, lost work experience generates longer term losses for young people.

Lori Ferzandi

One of the recent hot topics in the assessment world has been debate over the use of non-proctored assessments (NPAs) for pre-employment selection. (For those not familiar, the typical NPA scenario is one in which the employer asks job candidates to complete an unsupervised online assessment as part of the application process.)

While the convenience of NPAs is substantial, a number of potential risks such as cheating, proxy testing taking (e.g., getting my smarter roommate to take the test for me), and item disclosure are also readily apparent. Quite a bit of attention has been directed toward finding ways to control the risks associated with NPA as well as trying to determine the real-life negative impact of non-proctored administration upon test validity (frankly less that I would have expected based upon what has been published and presented).

However, reasonable folks differ quite substantially in their comfort level with NPA and likely will for the foreseeable future. This is a literature I follow pretty closely because pan provides both proctored and non-proctored assessment services. (Here is my presentation at the last SIOP meeting on how technology can help mitigate some of the risks associated with NPA). And given NPA’s rather controversial status, I guess it isn’t surprising that writers haven’t always been particularly nuanced in their positions. Some conclude that NPA is either the best thing since sliced bread while others believe it’s a professional abomination to be avoided at virtually any cost.

Doug Reynolds and colleagues’ recent must-read paper in Industrial and Organizational Psychology does a very nice job of contextualizing the discussion. Their major point is that we need to attend to all the relevant factors when a web-based assessment program is implemented. The issue is not simply whether or not a proctor is present during the test session. Rather, consideration also needs to be paid to factors such as the type of assessment to be implemented, how it is used in the hiring process, the nature of the applicant population, as well as an evaluation of what the alternatives may be.  As Reynolds paper concludes, “by positioning unproctored Internet testing as something that professionals should be either for or against, the underlying complexities associated with broader test deployment conditions will remain obscured.”

Reid Klion

A recent Wall Street Journal article I came across described how the City of Los Angeles is using a new multiple choice test to try and predict the likelihood of a 10-15 year old joining a gang.  The primary purpose of this assessment is to provide gang prevention programs where needed in order to reduce the risk these youths face.  The assessment is still in early developmental stages and it will take several years for the city to determine its effectiveness. 

Assessing risk through the use of behavioral assessments is nothing new in the employment world.  Many companies use assessments early on in the selection process to screen out applicants that pose a risk, whether it is for safety, security, dependability, or other potential areas of risk.  Test publishers have responded to the need for this type of assessment.  We have seen a growth in the development of short, cost effective assessments that are specifically designed to be used early on in the selection process to assess risk.

These assessments have been shown to lead to reductions in safety violations and absenteeism in newly hired workers.  Additionally, some of our clients have also seen direct cost savings with reductions in workers compensation claims.  Finally, screening high risk applicants early on improves the efficiency of the hiring process by not spending time on unnecessary interviews.  While the utility of this type of assessment is not going to be useful for all employment situations, it is certainly worth thinking about for jobs where risk is a concern.

Nate Studebaker

A pan team member recently sent me an online Pepsi commercial that got me thinking that many organizations likely find themselves facing philosophically similar hiring issues.

The current economic conditions have resulted in increasingly larger candidate pools. In turn, hiring managers have found themselves up against the wall with not enough time to thoroughly evaluate all candidates. Fortunately, there are solutions. pan has assisted numerous organizations in creating more efficient and reliable employee selection processes.

pan’s solutions are scientifically-grounded and are available to all types of organizations, private, public, or public sector. Furthermore, each and every solution is customized to meet the client’s needs, which ultimately ensures that solutions are never hampered by job type, candidate pool size, or other challenging workplace situations.

In a recent pan case study, our client, PDS, was challenged to research and develop an employee selection process that would lower worker compensation claims and costs. pan worked with them to assess the most qualified candidates based on a set of criteria.

Rest assured, you don’t have to settle for leftovers. A pan-based solution will free up valuable time and resources and ensure that only the best and the brightest find their way into your organization.

As always, please feel free to post your comments - we would love to hear your interview stories, learn more about your current selection process, or share with you more information about pan.

Sarah Lacey

Three Indianapolis police officers who took a promotional exam have filed a suit are suing because they claim they were given the wrong study prep book. The story is replete with charges (the omission was politically motivated) and counter-charges (it was the officers’ responsibility to make sure they had the correct materials).

It is certainly the case that public sector assessment is often rife with controversy. And, because of the very nature of these disputes, they become public matters. (A great resource for anyone interested in public sector assessment is IPAC-the International Personnel Assessment Council, an organization formerly known as IPMAAC. I have had the pleasure of being active in the group for a number of years and the invaluable opportunity to learn from some very well-seasoned practitioners.)

I think that the critical take-away lesson here is that we need to ensure that personnel decision-making be carried out in an open and equitable manner that is fair to all candidates. And, while testing is anything but a perfect process, a well-developed assessment program is often the fairest and most valid way of making these decisions.

Reid Klion

While I have tried to avoid the massive media coverage of Susan Boyle (the Scottish singer whose sensibilities and personal style are somewhat less than high fashion), I came across a recent article that does a nice job of reviewing current psychological research on stereotyping.

From an evolutionary perspective, it was often life preserving to be able to tell whether the person approaching us in the forest was likely to be benign or pose an immediate threat to our wellbeing. Since aspects of the stereotyping process seem to be “hard wired,” we need to be aware of this and keep an open mind when meeting people. Indeed, much of the sensation about Ms. Boyle seems to lie in the fact that her musical skills were quite inconsistent with our initial impressions of her.

As might be expected, there is quite a bit of research examining how stereotypes can influence interviewers’ impressions of job candidates as well as how interviews are rather poor in predicting job performance. To this end, I think that one of the often unnoticed benefits of using a selection testing program is that unexpectedly positive test results sometimes force us to reconsider job candidates whose initial impressions (either in terms of looks, training, or background) are inconsistent with our stereotype of how potentially successful job candidates “should” appear. By failing to consider candidates who don’t necessarily fit the usual mold, we may be doing our organizations a significant disservice by missing potential star players.

Reid Klion