As a follow-up to the posting a few weeks about credit checks, there are at least 5 states that are challenging the use of employer credit checks for selection. The legislators involved claim that their actions are spurred by the perception that employers are using credit checks inappropriately. If any of these laws are passed, each one would need to be examined carefully to understand its implications, and this would need to be done on a state-by-state basis.

However, it is important to remember that credit checks are a legally defensible selection method if creditworthiness is indeed a job requirement. For example, applicants for  high risk positions, such as intelligence work, are often not hired if they have substantial levels of debt because this is seen as increasing their risk for bribery. However, employers do need to take a second look when they use credit data as an indirect predictor of other behavior such as trustworthiness. While there are a few studies that indicate some relationship between these two factors, there are several well-developed psychometric assessments that have been designed to measure integrity and trustworthiness directly. Additionally, unlike credit data which is often prone to adverse impact, there is a substantial body of literature indicating that integrity tests are not. Here, like in any assessment situation, it is important to first have a clear understanding of what the requirements for a position are and then finding the assessment strategy that is most likely to meet the organization’s goals which limiting unnecessary risks. 

Reid Klion

National Council of Examiners for Engineering and Surveying (NCEES) won a civil lawsuit against a civil engineering candidate who attempted to take one of the organization’s primary exams with wireless a/v transmitter, video camera, receiver, pocket video recorder, and battery packs all on her person (see picture) with the intent to steal the content of the exam. She apparently had done this previously in 2005 but was caught by an alert proctor in the most recent incident. The federal court awarded NCEES a judgment of over a million dollars for loss of intellectual property. This figure was determined by estimating the cost that is involved in developing the items that had to be discarded. This individual was previously convicted of fraud in a criminal court for charges related to the incident.

I suspect that judgments of this type and size are rare but they do highlight the costs that can be involved in the development of sophisticated testing programs.

Reid Klion

Here is an update on the EEOC’s E-RACE (Eradicating Racism and Colorism from Employment) Program. Ongoing for the past several years, E-RACE focuses on race and color discrimination in the workplace with a particular focus on employment screening methods (such as background and credit checks) that are prone to have adverse impact. Here, it is important to recall that adverse impact occurs whenever a seemingly neutral “test” unintentionally results significantly lower selection rates for protected groups of persons. (A classic example is that if the physical ability requirements are extreme, many women will be disqualified from certain labor jobs.)

Since credit and background checks are quite susceptible to having adverse impact, businesses must be sure that their use is job related (just as they must for any assessment method that has adverse impact). Not only is this consistent with the law, it also prevents the needless disqualification of job applicants who otherwise may become excellent employees.

Reid Klion

The call for proposals for the 2010 IPAC Conference is now open. Formerly known IPMAAC,  IPAC will meet on July 18-21 in Newport Beach, California at the Hyatt Regency.  Session formats include symposia, panel discussions, paper presentations, tutorials, and workshops.  Presentation proposals will be accepted until March 5th.  

pan has long been active in IPMAAC and now IPAC and is proud to have been an conference sponsor for many years. IPAC is an organization of applied HR selection  and assessment professionals, with members who work for both public and private-sector organizations. If you aren’t familiar with the organization, I might suggest you take a look a IPAC website and consider attending or even submitting to the conference.

Reid Klion

The Association of Test Publishers Conference  is coming up on February 7-10 in Orlando. The ATP Conference is a good opportunity to see what is new in the world of testing and meet many of people who are active in the industry. The conference program has tracks on promoting state-of-the-art assessment, integrating technology into assessment, and innovations in test security. pan will also have a booth in the exhibit hall, and a number of us will be attending including Doug Cole, Jocelyn Courtney (who recently became Secretary of the ATP I-O Division), and Jenni Ginsburg. If you will be there and would like to arrange a time to meet with one of us, please drop me a note. We look forward to seeing you in Orlando in a couple of weeks!

Reid Klion

Writing items for tests and surveys is often more difficult than it appears. There are professional guidelines for item writing but it is also an art that might be best learned through experience and observing how items perform when they are actually used.

How an item is written can have a substantial impact upon how subjects respond to it and inadvertently skew the results. As an example, I received what was labeled a survey in the mail the other day. Here are a couple of items:

-Do you believe that Barack Obama's nominees for federal courts should be immediately and unquestionably approved for their lifetime appointments by the U.S. Senate?

-Do you believe the best way to increase the quality and effectiveness of public education in the U.S. is to rapidly expand federal funding while eliminating performance standards and accountability?

To be honest, I suspect that the underling intent of the survey is to motivate political supporters (as opposed to actually being an opinion poll) because it was accompanied by a fund raising request. And regardless of one’s political leanings, I suspect that it would be difficult to find anyone who would endorse either of these statements as written. However, it illustrates how an item is constructed can impact the results. In this case, I think that it is fair to conclude this was the intent. Problems do occur, though, when the goal is gather information in an unbiased way but poor item writing gets in the way.

Reid Klion

One of the major reasons to use an assessment is to ascertain if a candidate is actually capable of doing the job. While there other ways to accomplish this, such as reviewing references and resumes, these approaches can fall short. Unfortunately, employers often do a poor job of actually checking references and many of us don’t verify a candidate’s educational history. (There haven been some well-documented cases of people in high profile positions who don’t have the background they claimed. For example, there is the former director of admissions at MIT didn’t have any of the degrees listed on her resume and George O’Leary who was the head football coach at Notre Dame for 5 days until it was discovered that both his claims to have played college football and having a graduate degree were false.)

Another challenge in credentials verification is the apparent growth of degree granting universities that may not have particularly stringent academic standards. There is growing concern  about U.S. military personnel getting degrees (almost wholly paid for by the government) from propriety, for profit institutions. While the students who enroll in these programs do so with good intentions, they often discover their degrees are not well-regarded by employers because of the nature of the schools. In a similar vein, there is a physics professor at the University of Illinois who has essentially been on a crusade against diploma mills, businesses often based outside the United States where one can essentially purchase a degree without fulfilling any requirements at all. When it comes to vetting a job candidate’s credentials, it is important the employers have a way of verifying that an individual actually has the qualifications to do the job well. Whether it be with looking into a candidate’s credentials or using an assessment, this is critical information to have before making a hiring decision.

Reid Klion

The Internal Revenue Service recently announced that it intends to develop a testing program for tax preparers. While CPAs and enrolled agents (non-attorneys who may represent clients before the IRS) have had testing and licensure programs for a number of years, the estimated 900,000 to 1.2 million individuals who prepare tax returns for a fee can do so without any sort of credential. The logic behind this new IRS initiative is to protect the public from incompetent or unscrupulous practitioners. Apparently, this move is supported by some of the large tax preparations firms who view this as a way to force ill-prepared practitioners out of the field.

There are some who argue that licensure programs should not exist because they represent a restraint of trade and that the market should be the sole determinant of who can offer professional services. Indeed, there may be situations where the development of a licensure program may be a bit misguided. (For example, there is a battle in Virginia over the licensure of yoga schools.) That being said, tax preparation very much seems a situation where some form of regulation seems reasonable to protect the public. Similarly, until the recent mortgage crisis, many states did not have a licensure program for mortgage brokers. To that end, pan was pleased to assist the Indiana Secretary of State in implementing a testing program for loan brokerage professionals.

Reid Klion

A recent article by Filip Lievens, Deniz Ones, and Stephan Dilchert looked at the relationship between personality and medial school success. This  follows the path of other researchers who are looking at non-cognitive factors in predicting academic outcomes.

They followed 600 Belgian medical students longitudinally throughout their academic careers. (Differing from the US model where medical school is essentially a 4-year graduate program, European students typically enter medical school directly from high school in what is a 7-year program.) They found that personality differentially predicted medical school outcome based upon the changing demands of medical school. (Especially in the European model, the first few years are comprised almost exclusively of basic science while latter years have an increased focus on clinical work and patient contact.) For example, openness and extraversion had increased validity in predicting performance over the 7-year period. Similarly, they found that while conscientiousness predicted grades throughout a medical student’s career, its validity increased from .18 to .45.

These specific findings aside, the article also is a good reminder that it can be important to study the validity of a predictor over time, especially in situations where task demands are dynamic and change predictably over time.

Reid Klion

The Indian Institutes of Management decided to move the Common Admission Test (required for admission to Indian MBA programs and similar to the GMAT in North America) to a computer-based test delivery model available over a ten day period. (In previous years, the test was administered in paper/pencil format on only one day.) This year, nearly 200,000 candidates applied to take the exam. Unfortunately, due to a number of factors including widespread hardware issues and a Cornflicker virus infection, there were major problems in delivering the test. A number of testing centers had to be shut down with reports indicating that some 10%-18% of candidates had to be re-scheduled (though it appears all who were not successful in doing so). Given the scale of the program, it has garnered substantial coverage in the press as well as in the Indian Parliament and brought forth public interest from number of other test delivery organizations that manage large-scale test delivery projects in India and would like to take on the project next year.

Reid Klion

In the past, I have blogged about testing in China and Korea.  While we often thinking of standardized testing as a Western phenomenon, it really has its roots over 2000 years ago in the Chinese imperial exam system with the concept later being adopted in the East in the 19th century.

The scale of some of these Asian programs is quite impressive (as well as is the potential for high profile problems which I will blog about tomorrow in regard to recent issues in India with delivery of their equivalent of the GMAT). At the end last month, nearly a million candidates took the Chinese Civil Service Examination. Attributed to the economic slowdown, the number of test takers (photos) has increased by a factor of 15 since 2003. Given there are only 15,000 openings for a million test takers, the stakes are impressively high. These government positions are often coveted for their stability and are sometimes referred to as the "Iron Rice Bowl."

Reid Klion

It sometimes concerns me when I read an article in the mainstream press and have to check if haven’t accidentally ended up on The Onion. I had one of those experiences the other day when I came across an article about how parents in New York City are paying up to $1000 to have their 3- and 4-year-olds  coached on how to take the tests used to qualify students for gifted and talented kindergarten public school programs. Apparently motivated by the desire to avoid the cost of private schooling, these parents describe themselves as seeking the opportunity “to have a choice.” While I admit to not fully understanding all the dynamics around raising a child in Manhattan, I find the whole thing rather ridiculous and resonate to a recent piece in Time about over-involved parents.  

At the other end of the spectrum in terms of over-parenting, I was intrigued by a high school graduation coaching program in the Indianapolis Public Schools. Unfortunately, not unlike many large urban school districts, IPS struggles with high school graduation rates under 50% and students and families who often face a myriad of challenges outside the classroom. Financed by the local Chamber of Commerce, the intent of the program is to help students to complete high school.  Here, it is good to see the business community lend a hand where resources of this type are scarce. In the long run, it is hard to see how a program like this won't benefit all involved--students, their families, and the community at large.

Reid Klion

Just a reminder that the Call for Papers for the International Testing Commission Conference (to be held July 19-21, 2010 in Hong Kong) closes on December 1.  I had the opportunity to attend the ITC Conference in 2007 and would suggest it to anyone interested in increasing their understanding of issues related to the globalization of assessment. There are multiple considerations that emerge when a testing program is implemented internationally with challenges ranging from the practical to the psychometric to the legal. As the need for assessment programs that be used in multinational contexts grows, so does the necessity for assessment professionals to understand the complexities that come along with the territory.

Reid Klion

Yesterday was the day for almost 700,000 Korean high school students to take the nearly 9-hour long College Scholastic Ability Test. Given only once a year, it is critical to gaining admission to a Korean university. Coupled with the fact that more than 80% of Korean high school seniors attend university, this is a critical event. High school students spend months preparing, and their families often go to churches and temples to pray for a good outcome. To help manage the logistics, work started an hour late yesterday to minimize traffic jams, and aircraft were diverted from flying near test sites to minimize noise disruptions. In past years, the power company has even placed crews on standby in case the power grid would somehow become disrupted and impact testing.

While we often think of testing as a Western phenomenon, standardized tests have been used in Asia for over 2000 years. Indeed, the concept of using tests to hire civil servants was first implemented in the West in the late 1800s when the British began following testing practices they first observed in China.

Reid Klion

The Merit Systems Protection Board, a federal agency charged with ascertaining that other federal agencies carry out personnel-related activities in compliance with the law, recently released a report on the use of job simulations in assessing job candidates.  While written specifically for the federal context, it is a good source of information to anyone interested in the topic.

For those who may not be familiar, a job simulation presents a realistic work situation and assesses how an applicant responds to it. When you think about it, the set of tasks that might be considered job simulations is rather broad to include things like having welders demonstrate their skills on standard tasks, having pilots demonstrate their skills in a flight simulator, or a store manager asking applicants how they might respond to specific customer inquiries. While they may differ in terms of fidelity to actual work tasks, job simulations often provide good information about a job candidate’s skill level with a high degree of face validity.  The MSPB report discusses the different types job simulations, their relative benefits and weakness, as well as how to create and conduct them. The document also goes on to detail how simulations might be more broadly used by federal government. Some very useful information at the right price...

Reid Klion

While virtually all the proctored testing that pan does is in the HR space, there is a huge amount of proctored testing that takes place within an education context--think PSAT, SAT,  MCAT, GRE, LSAT and the like. A recent NYT piece focuses on the problem of proctors who fail to do their jobs or even manage to be disruptive during SAT testing sessions. The article indicates that SAT proctors are not specifically trained for the task (which surprised me a bit since our policy is to train all proctors on each program they supervise).  And reviewing some of the reported issues that occur during paper and pencil testing, it is clear that computer-based test administration (as we use in our testing centers) can ensure a much more standardized experience in terms of test timing, the right test forms going to the right candidate, and the like.

One bit of advice provided by an SAT test coach (yes, this is apparently a whole industry in and of itself) is that if a candidate notices a problem during test administration, he or she should either voice a complaint immediately and ask that it be addressed or “forget about it” (as opposed to simply grumbling about it after the fact). Here, it is important that test takers assume some responsibility in advocating for themselves by pointing out problems if they become apparent. The coach also goes on to say that “you have to expect that there will be distractions. If you’re seated next to a kid who is muttering under his breath, or sneezing, or grunting, just deal with it. Move on.”

Reid Klion

We are often concerned about the role of self-presentation during hiring situations. For example, much attention has been devoted to understanding hown “faking good” may impact personality assessments, the potential role of cheating on exams, and the fact that people tend to exaggerate their qualifications during interview situations. That being the case, at least the folks who engage in these behaviors are making an effort to present themselves in a positive light (and, as have some like Bob Hogan argue, the ability and willingness to present oneself positively is often a critical job qualification in and of itself).

In contrast, consider those individuals who fail to put their best foot forward when applying for a job (and the reaction this tends to elicit from hiring personnel). For example, virtually all of us can recount instances of job candidates who have multiple misspellings on their résumés, manifest a complete lack of phone etiquette, and the like.  To this end, a recent CNN article took an informal survey of hiring managers on some of the grossly inappropriate behaviors they have seen during interviews. These include job candidates answering cell phones during an interview, acting bizarrely, and being completely uninformed about the company or what it does. While it has been well-established that non-structured interviews are poor predictors of future job performance, it is also clear that certain interview behaviors are too salient and extreme to be ignored even by the most heavily research-based interviewer.

Reid Klion

Thomas Friedman, well-known author and columnist, recently wrote  about the workers he labels the “untouchables.” He argues that those individuals who have the imagination and drive to “invent smarter ways to do old jobs, energy-saving ways to provide new services, new ways to attract old customers or new ways to combine existing technologies” are the people who are critical to organizations that want to be successful in today’s economic environment. As a result, these employees will always be of great value. In contrast, average workers who can only carry out routine work are becoming increasingly endangered because many of these tasks can be automated or outsourced. Friedman further points out how our educational system needs to find ways to increase the number of students with high level problem solving and analytic skills and have a spirit of creativity, innovation, and entrepreneurship. However, until that day (and if it ever comes), it is incumbent upon employers to carefully select and build a workforce comprised of individuals who can truly create value for their organization.

Reid Klion

The program for the Association of Test Publisher’s Innovations in Testing Conference (to be held February 7-10, 2010 in Orlando) is now available. As usual, there appear to be a large number of interesting presentations on the docket. Some that initially caught my eye focused on topics including a survey about the future of testing, detecting cheating, and ATP’s new revised guidelines on pre-employment integrity testing. Also, registration is open with a $75 discount to folks who do so before November 16.

Reid Klion

I receive regular updates from ATA which is the largest testing company in China (and also happens to trade on the NASDAQ as ATAI). Last month, they administered the National Judicial Examination in China to more than 350,00 candidates. An individual must pass this exam to work in Chinese legal system and is a requirement for lawyers, prosecutors, and judges. Due to the high stakes nature of this exam, there were major concerns about cheating. (Cheating on exams in China has been an issue for over 2000 years with reports dating back to at least the Han Dynasty. There are a number of cultural factors which contribute to this which I hope to find the time to blog about fairly soon.)

In addition to taking candidates’ temperatures to make sure no one was ill, many test centers were also outfitted with handheld devices to detect the presence of wireless earphones well as equipment to block mobile phone signals. Additionally, 8 fixed monitoring stations and 2 mobile signal detection vehicles (labeled “Signal Detection”) were used to identify “suspicious” signals in the proximity of testing centers in  Beijing. In at least one instance, a test taker was discovered to have a wrist watch which could send and receive data. 

Reid Klion

Back in August, Charles Handler of Rocket-Hire released his annual Online Assessment Usage Survey. Surveying almost 150 persons who work in the areas of recruitment and hiring, he found that approximately 2/3 of respondents use assessments. Of those using assessments, almost a third report using online processes exclusively. The types of assessments that are most broadly used are personality measures, tests of skills and knowledge, and cognitive tools. He has yet to release the full report but will forward a copy when it is complete. Charles is about the only the only independent source of information on online assessment  and is certainly one of the foremost experts in the area.

Reid Klion

Jack Jones, Ph.D.,  respected colleague and Vice President of Human Capital Sciences at Vangent (a long-time pan content partner) has a recent article in Talent Management on how poor economic conditions may increase the prevalence of problematic behavior at work. While some of the factors cited include the fairly obvious such as increased general stress and financial needs, he also outlines how more subtle dynamics such as decreased levels of supervision due to staffing cuts or even poor maintenance of a facility’s physical plant can also contribute. The latter may be related to the “broken window” concept in urban sociology which argues that if vandalized windows are quickly repaired, it tends to dissuade further acts of minor anti-social behavior in the neighborhood. Similarly, if there is an increased sense of disorder at work, it may influence the increased expression of deviant behavior.

The article goes on to describe the extent to which candidates will admit to specific counterproductive behaviors as assessed by the Reid Report with the most common  being fired from prior jobs, having multiple unexcused absences from work, and giving unauthorized discounts. A brief case example of ROI is provided citing a retailer that estimated an annual savings of $6 million by screening out high risk candidates.

Reid Klion

There was a recent NYT Op-Ed by a guy who is authoring a book on the standardized testing industry and his exploits as a “test grader.” For those not familiar with this aspect of the testing industry (or who don’t have school-aged kids), the standardized tests administered by schools often include “open ended” questions where students need to provide a written response.

In order to scoring, armies of temporary workers are hired to do the work. The author recounts some of the goings-on as well as his less-than-conscientious behavior in this work. His conclusion is that “scoring should be done only by professionals who have made a commitment to education—rather than people like me.”

This brings a couple of thoughts to mind. The first is that one wonders if a work attitude assessment should be part of the hiring criteria for the essay graders. In this, as in all jobs, we want people who will carry out their work in a responsible and conscientiousness manner. Additionally, it also highlights why there is significant interest in the use of automated essay scoring systems where computers grade the written responses. There are now several years of research on these systems, and they tend to work well. The obvious benefits are that compared to human raters, they are not susceptible to either fatigue or poor work ethic. And unlike humans, they are 100% reliable because they will grade the same essay the same way every time.

Reid Klion

One of the services that pan provides is a network of over 600 proctored testing centers. We offer these to clients who require the supervised delivery of high-stakes assessments. However, outside of our Operations Team, most of us have little regular contact with the candidates who use our testing centers.  

I was jolted from this earlier today when I had to make quick stop at my local bank branch. When I pulled up to the office building where it is located, I saw several people anxiously smoking and pacing in front of the building. I quickly realized that across the lobby from the bank was a proctored testing center operated by another assessment company. When I understood the situation and that these folks were about to take a test, I was brought back to the recognition that taking a high stakes test can be very stressful given the potential outcome—perhaps not qualifying for a coveted job or obtaining the license required to enter a profession.

While testing is something that we do every day, we can sometimes become a bit detached from the actual experience of the candidate. As such, when we ask someone to take a test, we need to be aware of the anxiety that this can create and the importance of using good test administration practices such as providing appropriate information about the test beforehand and maintaining a quiet, comfortable, and professional  environment. This is especially important to bear in mind when assessments are not administered by trained staff in a formal proctored testing center but are supervised by individuals who have other duties and are carried out done in a location where other activities are also conducted.

Reid Klion

I recently finished Richard Nisbett’s fascinating book, Intelligence and How to Get It.  Subtitled, Why School and Cultures Count, Nisbett (a prominent cognitive psychologist at the University of Michigan) argues that intelligence is quite malleable and that it has a more limited genetic component than many believe. Arguing against a strong hereditarian model of intelligence (and contrary to Herrnstein and Murray’s controversial 1994 book, The Bell Curve), his stance is that while there are some genetic components to intelligence, environmental and cultural factors have the greatest influence (as well as being the domains that can be influenced and shaped). Overall, he provides a good review of the nature vs. nurture debate on intelligence (and rather clearly comes down on the nurture side).

As one who stresses environmental factors in the development of intelligence, Nisbett also makes some research-based suggestions about how to raise a child’s IQ. Included are:
•    Exercise during pregnancy
•    Talk to children using complex vocabulary and include them in adult conversations.
•    Teach kids to categorize and compare objects in their environment
•    Foster curiosity
•    Encourage regular exercise and activity
•    Limit family stress and instability
•    Teach and reward self-control and delaying gratification
•    Praise children for hard work and efforts which are under their control.
•    Have high expectations and model the behaviors you would like to see in your kids

Reid Klion

The modern psychological study of personality can be traced back to Sigmund Freud and his student, Carl Jung. Jung later broke with Freud and developed his own theoretical system which focused upon concepts such as the collective unconscious and the role of archetypes. While the study of personality has obviously evolved very substantially in the 100 or years from the time that Freud and Jung started their work, their influence can be seen today. For example, the Meyers-Briggs Type Indicator and other assessments of psychological type are based upon Jung's work.

It also should be noted while Freud and Jung are largely viewed as historical footnotes for most of us interested in workplace personality, both continue to have very ardent followers in other circles. The New York Times Magazine has a fascinating (but long) article about the efforts of Jung's family (he died in 1961) to prevent the publication of book that he wrote while in the throes of a personal crisis. Filled with mythology, fantasy, calligraphy, and paintings, Jung's family always considered it to be too personal to be made public. The article describes the history of the "Red Book" and how it has recently come to be published. 

Reid Klion

I had the pleasure of attending the joint IPMA-HR/IPAC meeting this past week in Nashville. The conference was composed of both public sector HR professionals and assessment-oriented psychologists. The mix led to some interesting discussions where the HR folks could hear about the nitty-gritty involved in testing and the psychologists learn more about the day-to-day challenges faced in managing public sector workforces in challenging economic times.

Many of the presentations are posted. As always, Bob Hogan gave an entertaining and scholarly talk on the critical importance of leadership in organizational success. Bryan Baldwin (whose blog HR Tests is certainly worth a look) also gave a great talk on the potential role and challenges associated with using social networking websites in recruiting.  (Bryan didn’t post his presentation but his recent article can be found in the Assessment Council News on page 6.)

Reid Klion

John Callender (Biodata Online) and Jack Smith (Kingwood Group), two respected testing and assessment experts and long-time pan content partners, will  discuss how the current economic climate impacts recruitment and assessment strategies.  Some of the topics to be covered include how investing in assessments makes economic sense and the potential benefits and consequences of shifting to a lower cost assessment program. Please join us!

Thursday, September 17 @ 1PM ET     Register

Reid Klion

There was a WSJ article a couple weeks back on the growing trend for universities to consider using personality assessments to assist in admissions decisions. In addition to the ETS Personal Potential Index  I blogged about a while back, there are a number of other tools in development. For example, ACT is developing an assessment to identify students who are at high risk for dropping out while the College Board has been working at Michigan State University to develop a biodata predictor for university success.

Interest in these instruments appears to be spurred by the seeming unassailable belief that there is more to predicting academic success than GPA and standardized tests scores. However, there are also concerns about cheating and students learning how to “game” such non-cognitive assessments. It should be noted that colleges have always attempted to discern a potential student’s strengths and weaknesses outside the academic realm through personal statements and admissions essays. Proponents of these new non-cognitive assessments argue they are carrying on this same tradition but doing so in a way that standardizes how these non-academic factors are evaluated.

Reid Klion

We have always had a number of clients who are interested in using 360 degree feedback instruments. Also known as multi-rater feedback, these tools gather information from several people who interact with an employee on a regular basis such as managers, peers, or subordinates. Each tool varies slightly in its focus, but all use questions that are behaviorally-based and focus on work-related behaviors and job performance analysis. Jocelyn blogged last week on some tips and suggestions to consider when using 360º feedback tools.

The major benefit of 360 degree feedback instruments is that they gather information from a wide array of individuals who interact with the subject of the assessment. A 360 degree feedback tool can either be used alone or as one component of a broader development or coaching program. While controversial, some organizations also use multi-rater feedback instruments as part of their employee performance review process. However, it is important that you inform raters that their feedback may be used for evaluative purposes (such as for an employee performance review).if you chose to use 360 degree feedback instruments in this way.

Finally, organizations can use a 360 degree feedback tool as a teamwork inventory to provide insight into each team member’s strengths and weaknesses and how they function together as a team. This information can inform the way team members interact with one another as well as help them to pinpoint and understand potential areas of conflict.

IPAC (formerly known as IPMAAC) will be holding its annual conference in conjunction with IPMA-HR on September 12-16 in Nashville. IPAC is comprised of individuals who are interested in public sector selection and assessment. As it has for the past several years, pan is pleased to be a conference sponsor. Additionally, Nate Studebaker will present on his work with the Texas Department of Family and Protective Services that helped to streamline their assessment process while I will participate in panel discussion on technology trends in assessment. We hope to see you there!

Reid Klion

I was recently asked to contribute to two of my coworkers’ professional development efforts by completing 360º feedback instruments. I truly enjoy receiving feedback from others, and so I was more than happy to oblige the request. I completed two different 360º assessments and spent some time reflecting on the act of providing feedback. As a result, I have a few helpful hints to share.

1)  Use 360º feedback instruments to gather developmental information.
The people I was asked to rate are extremely helpful and valuable to me at work. While my positive feedback is important, constructive criticism is equally, if not more, useful. Although I may have very little criticism of these individuals, others such as their supervisors and direct reports have a different perspective and can provide unique information and suggestions.  360º tools provide a comprehensive view of the individual’s strengths and weaknesses, rather than the narrower perspective given by self-ratings or the ratings of only a few individuals.

2)    Provide a “Not Applicable” response option.
One of the tools I completed included an N/A option for each rating item; the other did not. Given the breadth of the abilities and characteristics rated across these types of tools, raters are likely to see items that they simply do not have enough information to rate accurately. When asking others to provide feedback, it is important to provide an N/A option so that the rater doesn’t feel obligated to respond to items about which he/she may be uncertain.

3)    Educate raters on common rating errors
Studies consistently show that when providing ratings, people tend to make certain errors. When using any kind of feedback program, encourage the people providing information to use the entire response scale and avoid errors, such as leniency, strictness, and central tendency biases. Inform raters of the similar-to-me/different-than-me and halo/horn effects to encourage accurate ratings. 

4)    Separate development from evaluation
Although feedback may be important in both development programs and performance appraisals, it is critical to keep the two systems separate. When collecting developmental information, inform raters that the data will not be used for evaluation and will not affect the ratee in any negative way, as this promotes honest and accurate responding. It is important to note that 360º feedback instruments are not appropriate for evaluation and should only be used in a developmental context. In addition, it is wise to keep peers and direct reports out of the performance appraisal process, so as not to cause conflict or discomfort.

Jocelyn Courtney-Hays

Rob Tett and colleagues published a recent paper in the Journal of Occupational and Organizational Psychology that looked at personality test norms. Examining a large set of data from the Hogan Personality Inventory, they took the norms derived from several large samples of individuals employed in sales, trucking, clerical work, finance, and management. No big surprise, there were some rather significant differences in average scale elevation between groups (e.g., the truckers were substantially less sociable than the sales group). However, a bit unexpected were the differences in mean scores between samples within the same job family. As an illustration, some samples of sales personnel had much higher scores on scales measuring ambition and sociability than did others. Indeed, there were cases where the differences within job family were nearly as great as those between job families.

There are a handful of takeaway points to consider. First, relying solely on job type to select a normative sample can be problematic. Rather, factors such as organizational culture of the work group where the samples were obtained or whether the data were derived from job applicants vs. incumbents need to considered. (The latter is important because applicants tend to obtain higher scores than incumbents, something I will blog about in the near future.)  Secondly, this article also analyzed the impact of sample size on the reliability of normative samples. As a saving grace, they found that a sample of 100 is usually quite sufficient to develop a good set of local norms.

Finally, it should be noted that the importance of norms varies greatly depending upon how a test is being used. For example, if I want to know how the conscientiousness of the  maintenance personnel  in my company compares to that of folks who work in maintenance for other organizations, the normative sample chosen is obviously critical. However, if I am only interested in ranking maintenance job candidates on conscientiousness or have already conducted a validation study to determine cut scores, the use of norms isn’t particularly relevant or useful as we could use untransformed scores for these purposes.

For the past several years, there has been substantial debate about the impact of “faking good” (or socially desirable responding) on personality tests. While clients sometimes want to know how a particular test adjusts for an individual’s portraying him or herself in an overly positive fashion, there are no easy answers to this seemingly simple question.

First, there is disagreement as to whether socially desirable responding even has a significant impact upon the validity of most assessments when they used to actually select employees. Second, there is the argument that it is expected for job seekers to present themselves in a favorable light (e.g., “putting your best foot forward”) and that one should be concerned about an applicant who isn’t either willing or able to do this. Thirdly, when assessments are developed based on data samples of job seekers, this positive self presentation bias is already “baked into” the norms. Finally, as a practical matter, it has been quite difficult to find accurate measures of “faking good,” let alone ones that are helpful in increasing the validity of a test.

A recent Personnel Psychology article by Nathan Kuncle and Auke Tellegen helps shed some light on this. In their research on a scale designed to measure socially desirable responding, they found that increasingly elevated scores on a favorable personality trait are not necessarily seen as being more desirable. For example, moderate levels of “being organized” are typically seen more favorably than either very low or very high levels of this behavior. Additionally, when subjects are instructed to respond favorably on a scale designed to assess "faking good,” they tend to endorse these items at moderate, not extreme, levels.  As a result, the authors argue that measures of socially desirable responding need to consider the desirability of each response alternative and not simply look at the overall score on the scale.

Reid Klion

While it had been brewing in the professional literature for the past three years or so, the dispute between the authors of the MMPI has become public. (The MMPI is the pre-eminent clinical assessment that has been used for over 70 years. While sometimes discussed mistakenly in relation to pre-employment testing, it is almost never used to select employees because it is a clinical, not a work-related, assessment. About the only times that it is used to hire employees is in occupations where it is critical to rule out significant psychopathology such as police or nuclear plant operators.)

The issue at hand has to do with a “reformulated” version of MMPI-2, the MMPI-2-RF. The “RF” version, which has just been published, is shorter and described as easier to interpret that its predecessor. However, James Butcher, the lead author of the earlier MMPI-2 is highly critical of this revision. The issue has become personal because the authors of the “RF” version (Auke Tellegen and Yossef Ben-Porath) had been long-time colleagues and collaborators with Butcher. The result has been charges of conflict of interest and mismanagement on the part of the University of Minnesota press which owns the MMPI.

Reid Klion

One topic that has been controversial in the assessment literature is non-proctored assessment—that is, having job candidates complete tests without supervision. I blogged  about it previously and a seeming plethora of conference presentations and articles have addressed the topic. A consensus seems to be emerging that unsupervised testing may be appropriate in a particular situation based upon a careful risk-benefit-analysis that involves consideration of a variety of factors including overall business strategy, type of test, how the results will be used, and the candidate population.

A similar sort of cost-benefit analysis seems to have taken place in the federal prison system when transferring inmates from facility to facility. Since the early 1990s, prisoners who are considered low risk are apparently given a one-way bus tickets and sent on their way without supervision. Nearly 90,000 prisoners have been transferred in this fashion since 2006, and fewer than 1 in 500 have failed to make it to their intended destination. Given the expense that would be associated with escorting the prisoners, and the fact that a large proportion of these felons are on their way to a halfway house where they will have substantial freedom to move about the community, federal prison officials seemed pleased with the system. Others, including Greyhound officials and local law enforcement authorities where these prisoners have absconded, take a somewhat differing position.

Reid Klion

With its original version dating back to 1925, the Watson Glaser is a well-known critical thinking exam. Critical thinking exams are designed to measure an individual’s ability to evaluate the evidence in support of an assertion, to engage in abstract thinking, and to come to logically accurate conclusions.

As might be expected, scores on the Watson Glaser are highly correlated with other assessments of verbal, non-verbal, and numerical reasoning. The Watson Glaser also predicts job performance and leadership ability since critical thinking is closely associated with success in higher level positions.

The Watson Glaser is often used as part of a pre-employment testing process because of its ability to predict work performance in positions which require decision making. However, with the advent of the Watson Glaser II (which will soon be available on pan), there is now a developmental report which will broaden the Watson Glaser’s  use beyond pre-employment testing to coaching, training, and other related domains.

An uproar has been brewing because the images used for the Rorschach inkblot test been posted on Wikipedia along some of the more common responses to each. . As might be expected, some psychologists are concerned that this may impact the validity of the test, especially in cases where the assessment is used within a legal context (for example, in a child custody evaluation or pre-trial assessment of a sex offender) as opposed to a counseling situation. In contrast, others have pointed out that the Rorschach is almost never used in isolation but in combination with a number of other clinical assessment tools to make decisions and that this same information about the Rorschach is available in training texts on the test as well as from other sources.

However, this event does highlight an important issue about test security and the Internet. While these images have been available to the public previously, their posting on Wikipedia does make  them easily and broadly accessible. And even if we accept the argument that posting this information will have little impact upon the Rorschach’s validity, it should be noted that real harm can be done when test security is compromised. Not only is copyright law often being violated and the validity of a test compromised that may have taken years to develop, but the public may be placed at harm. For example, how many of us would be concerned we found out that our child’s pediatrician only passed a licensing exam because he or she had knowledge of the test content before taking the licensing exam?

As it turns out, the individual who posted the images actually is an emergency room doctor from Saskatchewan. When questioned, he sees little harm in what he has done and goes on to state that his elderly father passed a driver’s license eye exam by looking it up on the Internet. I guess our only hope is that his dad can actually see well enough to drive to keep both himself and the rest of us safe.

Reid Klion

While personality and cognitive tests are often used for hiring and selection, another set of tests focus on an individual’s aptitude, or ability, to learn a set of tasks. Rather than assessing an individual’s knowledge, mechanical aptitude tests predict an individual’s ability to learn a job. Mechanical aptitude tests are most typically used in the hiring selection process to identity the most suitable candidates for training programs or entry level positions.

Probably the best known mechanical aptitude test is Pearson’s Bennett Mechanical Comprehension Test. Here, the applicant is presented with pictures that represent mechanical principles in everyday situations. For example, one question may show a lever and ask where it should be placed to make moving a heavy object easiest. The intent of the items is to assess mechanical reasoning, not specific knowledge. Other similar tests are the Hogrefe PPM for Mechanical Understanding,Pearson DAT for PCA Mechanical Reasoning, and the Ramsay Mechanical Aptitude Test. 

Mechanical aptitude tests are powerful tools when used for applicant screening into positions that require these skills. They have been used successfully for decades and are well-suited for hiring selection decisions for jobs that require mechanical ability or will involve training in these skills.

I was on the SIOP website the other day and came across the “SIOP FYI on Workplace Topics” section which has some rather extensive information on both employment testing  and coaching. While I haven’t had a chance to review the coaching section in any detail, there is a wealth of descriptive information about assessment. Overall, the presentation seems quite balanced with a section on reasons to use  (as well as not to use) assessments, types of tests, and item formats. 

These pages provide an excellent, even-handed orientation which focuses on the pros and cons of various approaches to assessment. These resources would be quite useful to individuals who want to learn more about using assessments in the workplace. Interestingly, there is a “hit counter” on the bottom of each page. While I don’t know when the pages or counters were created, it appears that some have been viewed fewer than 1000 times. Let’s hope they find the audience that they deserve.

Reid Klion

While perusing Amazon.com for good books recently, I came across the Penn and Teller tome, How to Play in Traffic.  Generally, I wouldn’t mention such mundane details of my life, but I wanted to make note of it here.  You see, it has come to my attention that there are some people who make playing in traffic their full-time job.

You may remember from my previous post that I am an Indy 500 fan. However, the excitement here in town is that NASCAR will be at the Indianapolis Motor Speedway this weekend for the Brickyard 400. In almost any form of racing, a key part of any race are the pit stops where tires are changed, and…well, I actually don’t know what happens aside from the tires, but I’m sure it’s all very important and mechanical.  While I was at the 500, it occurred to me that the people who work in pit row are dealing with cars moving in and out of the pits at high speeds.  I have read stories of injuries, from the minor to the severe, occurring due to pit crew members coming too close to the racecars, and seeing the pit stops in person made me realize just how easily injuries could happen.

There are probably a number of important attributes that make for a good pit crew member.  I’m willing to bet that in addition to having a high level of mechanical skill and knowledge, you also need to be diligent, conscientious, and focused on safety – not only for your own well-being but also for that of your driver, fellow pit crew members, and competitors.  How many other jobs require this same commitment to safety?  I can think of a few-- assembly line employees, truck drivers, healthcare workers.  Given the inherent risk in these jobs, it is important to hire people who will be safe at work. While training is an important part of workplace safety, there are also a number of assessments that can help identify job candidates who are (and are not) well-suited for work in dangerous environments.

Jocelyn Courtney-Hays

When I was in high school Biology during the early 90s, my teacher spoke of a new emerging technology called the Internet. It had been around for a while but had not really caught on outside of scientific and academic circles as computers were for geeks, and most people didn’t see their benefit over using a typewriter. He spoke of the ability to access information about breakthroughs in science as if you you were down the hall from the actual experiment and having immediate access to information from all over the world. Most people thought this type of technology was for large companies like AT&T and IBM, and the idea a personal computer could retrieve information from some nebulous place on the other side of the world was completely foreign and pretty much inconceivable.

In my naiveté, I did not realize he was really referring to the emergence of the global community. The Internet is something which joins everyone together and enables interaction with someone across the world without ever leaving the comforts of the living room. Transactions can be conducted by anyone with access to a computer, and what was once an exotic item from a foreign country is now a few clicks away on sites like Amazon and EBay. All that is required is the belief that the person on the other side of the connection is legitimate.

In the past, the beginning of trust was largely based on who people commonly knew. If a friend endorsed someone, that person was probably trustworthy.

Benjamin Lahey has a recent American Psychologist article on the public health implications of neuroticism. While often viewed as a clinical construct, neuroticism is reliably found as a stable personality factor and often labeled as “adjustment,” “negativity”, or “emotionality stability.” In the personnel selection realm, high scores on this factor are often negatively correlated with occupational success. Similarly, Lahey outlines how neuroticism predicts both mental and physical health as well as response to treatment for both types of conditions. While it is certainly plausible that illness will increase an individual’s level of negative emotionality, there are also data to suggest that it predicts many health problems even when depression is controlled.

The article goes on to explore both the genetic and environmental components that may contribute to neuroticism as well as the interplay between them. Indeed, it is easy to understand how one’s propensity to respond negatively to environmental stressors can act in a self-perpetuating cycle. For example, if an individual reacts to a negative stressor (say the end of a relationship leads to increased social isolation as opposed to a healthier response of seeking social support), it is easy to see how the cycle can continue. While we often only consider personality traits in the context of personnel selection, it is also interesting to consider how they fit into the larger context.

Reid Klion

Let me introduce myself, I am Gerry Groe, Managing Director of Business Development at pan.  I joined pan on March 16 of this year after restructuring myself out of my position at Denny’s as VP- Human Resources & Organization Development.  Denny’s is a fine company, but, like many consumer-oriented businesses, it has had to deal with a very tough, turbulent economy over the last two years.  Consequently, in my OD role I worked with several outside consultants and the Denny’s senior team to restructure the company to better facilitate achievement of the company’s business strategy.  As part of this new organization design, I actually suggested that my job be eliminated.  Why, because HR was “top heavy” and we had the necessary senior HR talent to lead a leaner HR function.  Fortunately, even before I left Denny’s I knew where I wanted to go - I wanted to join  pan.  While I have more than twenty years experience in VP and SVP – level HR positions with major corporations and as an OD consultant, by trade I am an industrial/organizational psychologist and I felt a strong pull to pursue a career with pan focusing on assessments - in my next posting I will explain why.

Gerry Groe

ETS recently rolled out a product called the Personal Potential Index. Intended for use by graduate school applicants, it is a standardized rating form with 24 items pertaining to factors such as creativity, communication skills, and teamwork that is completed by a set of student-nominated “evaluators” (typically professors). The intent is to provide graduate program admission committees with information beyond GPA, GRE scores, and the traditionally glowing letters of recommendation. (Exaggerated letters of recommendation are not a new problem; there is a classic American Psychologist article from 1966 on the topic rather brilliantly entitled “Mine Eyes Have Seen a Host of Angels.”)

The ETS tool is premised upon the recognition that non-cognitive factors likely contribute to success in graduate school (as well as virtually everywhere else in life) and part of a larger ETS project to find ways of assessing them. There are concerns about using self-report inventories in a graduate admissions context due to “faking” (though such instruments have been found to function effectively in pre-employment situations). As a result, attention is being directed to rating scales as well as biodata and situational judgment tests.

The Personal Potential Index appears to be a work in progress. There are plans for developing local norms as well as procedures to adjust for differences in individual rater severity.  Additionally, only further research will be able to determine the extent to which this rating approach actually adds any incremental validity (and all of this is predicated upon the ability to collect enough real-life data). A concern that comes to my mind is the degree to which most professors actually have enough observational data to accurately rate their undergraduate students’ “non-academic” behaviors, especially in the case of institutions with large, often anonymous, classes. Additionally, one also wonders about the extent to which raters, knowing the implications of their evaluations, will engage in the same rampant exaggeration that is well-recognized in letters of recommendation.

Reid Klion

One of my favorite places is the lawn at Verizon Wireless Music Center.  Deer Creek, as it is still affectionately known by local residents, is an outdoor amphitheater located just north of Indianapolis.  I have spent many nights there--enjoying alternative rock as a misunderstood (or so it seemed) teenager, classic rock as a cool (or so I thought) undergrad, and contemporary rock as a well-rounded (or so I hope) adult.  

At a recent live show, I took notice of the hundreds of employees at the venue. They are all temporary or seasonal workers – Indiana weather being what it is, the music center is not open in the winter.  I wondered how these people got their jobs.  Some of the security guards looked as if their physical stature alone may have secured them a position.  But what about the young man selling beverages or the ladies working the souvenir stand?  

Temporary employees are a unique segment of the workforce. Selecting them can be a challenge because the jobs hold no promise of longevity or advancement and the most important requirements may be personality traits, such as dependability or a focus on satisfying customers. Staffing agencies and other organizations that hire temporary employees should consider pre-employment assessments as an effective and efficient way to gain information about job candidates without having to spend the time and resources necessary to conduct an interview.  For instance, integrity tests can be used to predict counterproductive behaviors such as absenteeism and theft. Personality-based assessments can be used to determine how conscientious and reliable a person is, likely important in most if not all temporary positions. This type of assessment can also provide information on traits like assertiveness and stress tolerance which may be important in specific roles as security guards or gate/ticket checkers.  Finally, biodata instruments can be used to assess a person’s fit for a certain position in customer service and sales by examining work experience, attitudes, interests, and values.

Jocelyn Courtney-Hays

Most of us who work in the I-O arena are interested in looking at how personality and other characteristics predict things such as occupational success or leadership ability. However, there are also others who take an approach that is more oriented toward theory or research. A fascinating example is the Harvard Study of Adult Development  (part of which is known as the Grant Study as it was originally funded by W.T. Grant, the founder of the now defunct chain of stores). Begun in 1937, it started studying a series of undergraduates who were at Harvard in the late 1930s to early 1940s and continued to track and collect data from them throughout their lives. The study is now in its 72nd year, and it is interesting to note that about half of the original subjects are still alive and now in their late 80s to early 90s.

A recent article in the Atlantic profiles the study as well as George Vaillant, the psychiatrist who has managed the project for the past 42 years. One of the many uses of data collected in the study was to  identify the predictors of physical and psychological health. Among them are:

•    Learning to cope with life changes and stresses maturely by with methods such as finding healthy outlets for anger or disappointment
•    Education
•    Not smoking
•    Not abusing alcohol
•    Maintaining some level of exercise
•    Staying at a healthy weight

A study like this can also serve as a lens as to what research methods were considered important at the time. For example, in the early days of the study, subjects completed Rorschach tests, submitted handwriting samples, and had virtually every dimension of their body measured (recall that Sheldon’s concept of somatotype was still seen as having validity during that era). Now, surviving subjects take MRIs, provide DNA samples, and are being asked to donate their brains to the study.   The study, which has amassed over seven decades of data on it subject, will likely be a treasure trove for researchers for another several decades.

Reid Klion

A bit ago, I blogged about the online journal, Practical Assessment, Research, and Evaluation. Another online journal also worthy of notice is the Journal of Applied Testing Technology. Published by the Association of Test Publishers and edited by Chad Buckendahl and William G. Harris, it is a peer reviewed journal that is concerned with issues related to assessment. The most recent issue focuses on the assessment of individuals with disabilities. Some other recent articles of note include a piece by Nathan Thompson that outlines the types of test administration methods that are available and tries to untangle acronyms like CAT, CBT, CCT, LOFT that psychologists are prone to throw around and a contribution by Richard Luecht  that outlines some cost-benefit criteria to be used in evaluating computer-based testing models.

As is the case for virtually all the professional journals that I read, there is almost always at least one article in each issue that is of interest or applicable to something that I have been trying to sort out on my own. An added bonus with these two online journals, you can not only access them directly from your computer desktop, it is hard to argue with the cost of the subscription!

Reid Klion

I was out with some friends one evening last week and came across a couple of little girls running a lemonade stand.  It was hot and they were friendly, so I decided to stop and encourage their entrepreneurial spirit.  They had two drink sizes to choose from (at different rates, of course). When I selected the smaller 25 cent size, the older girl took a paper cup, and filled it halfway, and then handed it to me. I gave her the money and caught up with my friends. I laughed when I told them about the half-full cup and joked that the girls were maximizing their resources in a difficult economic climate. 

Then I realized that they were doing exactly what most of us are right now – trying to find a way to maximize our resources, minimize expenses, and get the most output.  Jocelyn Courtney-Hays gave an e-seminar recently on the same topic: Two for One: Using Assessments for Selection and Leadership Development.  Her presentation is an excellent resource to learn how organizations can increase their effectiveness with an assessment for both employee selection and development.

Jenni Ginsburg

Knowing a job candidate’s cognitive aptitude or ability level is one of the most useful pieces of information in making an informed hiring decision. Cognitive ability is one of the best predictors of job success because it tells us both how quickly an individual will learn a job as well his or her likely level of work performance.

Cognitive aptitude tests are typically comprised of items that measure a job candidate’s ability to learn new material, use logical reasoning, or solve problems. Other cognitive aptitude tests measure more specific topics such as reading ability or math aptitude. The benefit of cognitive aptitude tests is that they are often quite valid in making inferences about job performance. They also tend to be both time and cost effective. However, something to be aware of is that cognitive aptitude assessments may be susceptible to adverse impact. That is, members of protected classes are prone to obtain lower scores on these tests than members of other groups. As a result, it is important that a careful job analysis be carried out before a cognitive aptitude assessment is implemented  to ascertain that cognitive ability is critical for occupational success in the position at hand and that the cut score and decisions rules used with these tests are appropriate.

The press episodically covers incidents of cheating on exams. The high profile situations often seem to be found in university settings where large numbers of students are impacted. However, they occur in other programs as well. While these reports often provide commentators with the opportunity to bemoan the tragedy of “falling moral standards in today’s youth”  or some such thing, it is important to realize that there were cheating scandals during the Han dynasty on the Chinese Imperial exams over 2000 years ago.

In a recent episode, our friends at Canada’s Public Service Commission (PSC)  suspected that their Second Language Exam may have been compromised because students who attended one language training school did particularly well on the exam. (Being able to speak both of Canada’s official languages--English and Canadian French—can be a qualification for certain positions in the Canadian federal government so these exams can have rather high stakes.) In impressively thorough and transparent fashion, PSC shared the results of their investigation (that the school was providing its students with a practice exams practically identical to the actual exams but it couldn’t determine exactly how the copies were obtained) as well as next steps (retest everyone who attended this particular language school and accelerate development efforts of alternate test forms).

Obviously, test security is a critical issue in what we do, especially with cognitive and knowledge-based tests which are objectively scored with right-wrong answers. While web-based testing systems tend to increase security because they don’t require the distribution of content on paper--which can be notoriously difficult to track, readily misplaced, or copied--it is still incumbent upon us to be aware that test content is sensitive and should always be managed with security in mind.

Reid Klion

While I am almost 20 years removed from my high school days, it somehow became task to help organize a reunion this summer.  When I looked at the lists from our 10 year event, I had 90 classmate email addresses on hand.  Over a period of about six months and sending “spam” emails, I managed to add the whole of 10 additional addresses to my list. So, for a class of 600, I had 100 addresses and fears of a less than spectacular event.  Knowing my high school reputation (or lack thereof) was at stake, I knew that I needed a new plan.

Back in the winter, a friend suggested that I go onto Facebook and see who had registered. To my surprise, it is like a little nostalgic reception where everyone is friends again.  I had requests galore from people who wanted me to be their friend, join their cause, or be a part of their group.  Over a six month period, I managed to accumulate over 300 friends, 6 groups, and no causes (fear of the unknown) .  I made no efforts to actually search for classmates but my list organically grew to over 300 people, a 200% increase—and all without really trying!  

My mind started the machinations about how we can we get these social networking sites to benefit folks in the assessment world and bring forums of like-minded groups together?  I don’t have the million dollar idea yet, but I am interested in any feedback people have on discussion boards or social networking tools like Facebook, Twitter, LinkedIn, MySpace, and the like.  We need to help get those interested in testing and assessment to find each other so that knowledge of best practices and resources are available to all.  Like my reunion, it would be no fun if everyone can’t make the party.

Jared Owens

Personality assessments focus on measuring work-related personality traits and psychological aptitudes. There are two major groups of personality assessment inventories. The first set measures a broad range of personality traits. These personality surveys are typically based upon the Five Factor model and cover the personality dimensions of openness to experience, conscientiousness, extraversion, agreeableness, and adjustment. These assessments are quite useful for personality profiling because they provide a broad view for understanding a job candidate’s psychological aptitudes.  They are particularly useful in positions that involve interpersonal interaction, teamwork, and serving in leadership roles.

A second set of personality assessment inventories focus on more specific traits or attributes such as integrity, customer service attitudes, or sales ability. In contrast with broad band personality inventories, these personality surveys are only concerned with a specific set of behaviors. They are particularly well-suited for identifying job candidates who have the right psychological aptitudes for a specific job with well-defined requirements such as in customer service or sales.

My sister-in-law recently graduated from college.  I can’t tell you how proud my husband was to see his baby sister all grown up and ready to embark on a career and life in the “real world.”  As I watched the graduates file across the stage, I wondered how many are truly ready to begin their careers.  My sister-in-law spent time in college exploring different majors and used an internship to get a clear picture of life in the business world. However, I wonder how many of her fellow graduates followed a similar path. There’s no judgment here – truth be told, I selected Psychology as an undergraduate major because 1) I thought Psych 101 was interesting, and 2) I had run out of core curriculum classes to take. I was lucky in that after exploring other possible career avenues, I discovered Industrial-Organizational Psychology was truly the best path for me. How many other students aren’t so lucky?

Many college students may be surprised to by the number of resources that are available to help them choose the right major and occupation. Most, if not all, university career centers have assessments that can be taken to identify potential career paths.  For example, the Career Exploration Inventory provides information about occupations, leisure activities and education and training opportunities that match an individual’s preferences. The Career Orientation Inventory  assesses interests and then generates a list of the jobs and occupations that might be a good fit for the individual. Type-based personality inventories, such as the Golden Personality Type Profiler and the Type Dynamics Inventory  can provide information on an individual’s work style and the kinds of environments and organizations in which he or she is likely to thrive. These are just some examples of a number of tools that can assist students in important decisions regarding not only which major they choose but also which extracurricular and professional development activities they may enjoy.

Colleges and universities should advertise the services they offer, including assessments and career counseling, to ensure students are aware of the help that is available. In addition, parents, professors, and other educators should encourage students to utilize their school’s resources and explore different majors and potential careers.   While assessments are an excellent way to start this process, job shadowing and internships, alumni networking, and extracurricular activities, such as on-campus clubs and volunteering, are also helpful tools to facilitate a student’s choice of major and eventual profession.

Jocelyn Courtney-Hays

In my spare time I serve as course coordinator for my golf league. We are fortunate to have a great turnout - so good, in fact, that the last group usually has had to play the last hole in the dark. After two consecutive weeks of this, I started wondering what changes I could make that might help avoid this situation. 

Without realizing it, I followed the steps of the scientific method that many of us learned in middle school science - ask a question, do background research, form a hypothesis, test the hypothesis by collecting and analyzing some data, and then re-evaluate the validity of the hypothesis. I was able to test my hypothesis this week (that a 2 person scramble format instead of an individual stroke play format would lead to a faster round), but unfortunately it wasn’t supported because my group still didn’t get finished any earlier.

On the drive home this week, I was pondering my next hypothesis (maybe trying fewer players in each group next time), and realized that what I was doing wasn’t much different from the process used in employee selection. When implementing an assessment program, one of the first steps is to figure out what we want to accomplish and identify the skills or competencies required by a candidate.  The next step is to research potentially useful assessment tools and then essentially run an experiment by implementing them. After some data are collected, it can be determined whether or not the hypothesized solution meets the organization’s needs. 

Hopefully, we obtain the desired results but what if we don’t?  Perhaps the job analysis data didn’t fully capture the job’s requirements. Alternately, the appropriate assessment wasn’t used. Another possibility is that the applicant population changed or that the wrong criteria were used to judge the outcome.

Even though my initial hypothesis wasn’t supported, I think that I followed the right process and learned something from my "study." Whether it is on the golf course or in the office, by proceeding systematically, reviewing the results, and then attempting to understand them, the important thing is to keep working to find the solution that achieves your goals. 

Jenni Ginsburg

Not too long ago, I met with an old friend who had just moved to a new company to become Vice President of HR. In his efforts to understand his new organization, he spent a few hours reviewing a random selection of employee personnel folders where he discovered that the results of each employee’s pre-employment selection assessment report were stored. After a bit more digging, he found out that not only were these test results readily accessible to every member of the HR department who used the employee files, but also that line managers (who were “trained” by the test vendor to interpret the results themselves) later used these test data to make decisions about promotions--sometimes several years after the test was taken!

Realizing something was awry, he asked for some guidelines. Some of my suggestions were:

• Create a secure system for managing assessment data so that only appropriate personnel can access them on a strict “need to know” basis.

• Remember that assessments are only valid for specific purposes. If an assessment is designed for pre-employment selection, it probably shouldn’t be used for promotional decisions, especially several years after the fact.

• Use clear-cut and consistent decision rules for hiring and promotion developed by an assessment professional. Encouraging managers come to their own conclusions about test results based upon their own self-created interpretive rules will only lead to inconsistency, poor personnel decisions, and perhaps even legal consequences.
  

For those who are interested in a bit more detail, some colleagues and I presented on this topic at the Association of Test Publishers Conference in February 2009.

Reid Klion

I recently came across Practical Assessment, Research & Evaluation, a free online journal supported exclusively by volunteers that has been published 1999. It describes itself as soliciting manuscripts that are “scholarly syntheses of research and ideas about methodological issues and practices” in the area of assessment.

As might be expected, some of the papers that it publishes tend toward the technical (e.g., "Assessing unidimensionality: A comparison of Rasch Modeling, Parallel Analysis, and TETRAD"). However, there are also a number of articles that are likely to be of more general interest. For example, Wallace Judd, a pioneer in the development of performance testing, had a recent piece on the concept of gating items, items which an examinee must pass in order to pass the entire test. An example can be found on the FAA pilot’s test. After demonstrating his or her proficiency on pre-flight inspection, takeoff, navigation, and the like, the student pilot must then be able to land the plane. If potential licensees cannot do this in three attempts, he or she will not pass regardless of performance on prior items.

Some other articles that caught my eye were a discussion on the use of extended time accommodations, whether a hand-written vs. keyboarded essay responses are judged differentially (raters are more lenient with handwritten samples), and a two articles with advice on writing better multiple choice items

What impressed me the most about the journal were the quality of the content, the fact that it has been published for over a decade, and that the entire enterprise is based upon volunteer efforts. It also appears to have a wide readership as a counter on the table of contents indicates that articles from the journal have been accessed over 6,700,000 times since its inception!

Reid Klion

A wide variety of personality inventories can be used for pre-employment selection. Some of these are broad band personality assessments which provide a comprehensive picture of an individual’s personality functioning and are typically based upon the five factor model of personality. Some of the most commonly used pre-employment personality tests of this nature are the Hogan Personality Inventory and SHL’s OPQ32. The benefits of pre-employment personality inventories of this type are that they provide an in-depth picture of a potential employee’s strengths and weaknesses and can also be used for post-hire development and training. A drawback of using these personality inventories for pre-employment assessment is that because of their breadth and comprehensiveness, they are sometimes more expensive and require more time to complete than other alternatives.

Another type of pre-employment personality test are those that only focus upon one or two aspects of personality, most often integrity or customer service attitudes. Here, rather than providing a broad picture of an individual’s personality, only the specific attributes seen as being most critical for job success are assessed. Some of the more widely used pre employment personality tests of this nature are Vangent’s Reid Report 29th Edition, Helm QWIK-RELY, and GNeil Service Ability Inventory.  The benefits of using this class of pre-employment personality inventories is that they can be completed quickly and, given their lower price point, can be used as a screening tool early in the employee selection process. However, by design and, in contrast to broad band pre-employment personality tests, they do not provide a full picture of a potential employee’s strengths and weaknesses. 

Earlier this week, Lori blogged about content-oriented approaches to test validation. IPAC (the organization formerly known as IPMAAC) recently published an extensive monograph by Charles Sproule on the rationale and research in support of using content validation models in personnel assessment. If you aren’t a member of IPAC, you can request a free copy of the paper here. (Hat tip to Bryan Baldwin of the blog HRTests for the link.)

While you are on the IPAC website, you can also learn a bit more about the organization if you aren’t already familiar as well as the details regarding its upcoming Conference (September 12-16, 2009  in Nashville, TN to be held concurrently with the IPMA-HR Conference). Several of us at pan have been active with this group for a number of years and have presented regularly at its meetings.

Reid Klion

I recently enjoyed a day of shopping with my girlfriends.  As any woman (and any man who has shopped with one) will tell you, the stars must be perfectly aligned to have a good shopping experience. Luckily, on this trip they were.  The babies were quiet and content in their strollers, flattering and stylish things were actually on sale, and no one went home empty handed.  During this particular shopping extravaganza, I was also reminded of one of life’s little pleasures – the BOGO (i.e., buy one get one free).

It might be surprising, but the principle of the BOGO applies to employment testing as well.   Too often, I see clients use assessments for screening and selection and then stick the test results in a personnel file, never to see the light of day again.  What many organizations fail to realize is that selection assessment data can also be useful for employee on-boarding and development.  Even if the primary goal of assessment use is selection, why not take advantage of the BOGO opportunity and use it for development, too?

Of course, not all assessments are well-suited for both selection and development. For example, integrity tests are generally used only for selection, and 360° feedback instruments are designed specifically for development.  However, if an organization chooses to use a personality or behaviorally-based assessment for selection, the results may be useful for post-hire purposes as well.  For instance, after the assessment is used to identify high potential job candidates, the results may be shared with new hires and their supervisors so that they can start the job with knowledge of their strengths and areas for possible improvement.  By immediately engaging the new hire in the development process, the organization is demonstrating its commitment to his/her success.  In addition, the assessment data may reveal potential performance derailers that can be addressed to ensure that new hires perform their jobs effectively.  After the on-boarding process is complete, the assessment data can be used to determine training needs and contribute to individualized development plans. Finally, the assessment data may be used in succession planning as an indicator of fit for higher level jobs or in different departments and divisions.

In these tough economic times, it is critical to maximize the utility of all HR processes, as well as other programs throughout the organization.  Using assessment data pre- and post-hire not only gives an organization greater bang for their assessment buck, but it also enhances the selection system while facilitating the on-boarding and employee development processes.  

Jocelyn Courtney-Hays

While some relationships make intuitive sense, they just don’t necessarily play out in real life once you look at the data.

Consider the relationship between household books and children’s academic performance.  Intuitively speaking, we would expect a strong relationship between book reading in the home and academic performance.  However, according to Steven Levitt’s Freakonomics, reading many books does not correlate as strongly with children’s academic performance as does having many books in the home.   

In the context of employment assessments, we would intuitively expect a score on a mechanical reasoning test to be predictive of a mechanical engineer’s on-the-job performance because the test content closely mirrors what he or she does on the job.  Similarly, we might expect a mechanical reasoning test to be more predictive of a mechanical engineer’s performance than, say, a verbal reasoning test because the content of the mechanical reasoning test would be more similar to the actual job than the content of the verbal reasoning test.

While many in the assessment world continue to believe that a high degree of overlap between test content and job content results in greater predictive validity, the research does not support this.  In fact, there is a preponderance of statistical evidence demonstrating that knowledge, skills, and abilities tests which are strikingly similar in content to the jobs they purport to test for may not be any more predictive of future job performance than tests with content that bears little to no resemblance to the job in question (assuming that these tests are scientifically sound).

Kevin Murphy, a renowned I-O researcher at Penn State University, cautions against making predictive validation claims based upon a content-oriented approach (see Murphy Article on the SIOP website).  As he notes, “job-relatedness is neither necessary nor sufficient for demonstrating that people who receive high scores on selection tests will tend to perform well on the job.”  

No one, including Murphy, is suggesting that we simply abandon the process of trying to align test content with job content.  This process in and of itself not only has a positive impact on the acceptability of testing in terms of organization and applicant reactions, but it also contributes to the legal defensibility of employment tests and creates a more transparent test development and validation process that can be easily understood by clients.  Likewise, there are times when content validation is the only validation approach at our disposal, owing in part to factors such as small sample size or lack of performance data.

In the end, although I-O psychologists cannot lose sight of the importance of job content when developing and implementing assessments, they are advised not to become overly reliant on content validation approaches in their quest to support the use of various assessments.  Tests that measure job-relevant knowledge, skills, abilities, and attributes will likely be predictive of job performance but this assumption should be statistically supported whenever feasible.  

Lori Ferzandi

Every spring, top competitors from around the country come together to see who is the best. Through preparation,  pure skill, and talent, they put on a show in which only the strongest survive.  I’m not talking about Kobe and LeBron and the NBA Playoffs or even Sid the Kid and the Stanley Cup Playoffs.  No, I’m talking about something even more ruthless and pressure packed-- the Scripps National Spelling Bee!

Each year, I’m amazed by the talent and sometimes unintentional comedy that the Bee offers.  It got me thinking about what an interesting form of assessment the Bee provides.  Not only do these kids have to memorize and/or figure out how to spell hundreds of words, they have to show their skills on the national stage without making a single mistake.  The sound of the bell that’s rung when a word is misspelled must give them nightmares.  The kids (and I emphasize kids) are all obviously very smart, and its interesting to watch their talents on display.  I can never figure out how being informed that the definition of a word is “a wooden barrel used to collect water in ancient Mesopotamia” makes it any easier to spell, but to these kids it apparently does.

However, what really draws me to the Bee is how the competitors respond to the moment.  Of course, there’s the infamous case of the girl screaming the letters of the word “euonym” so loudly that the judges almost seemed annoyed by the fact that she won. But then there’s the case of last year’s winner, Sameer Mishra, being so unflappable when given the word “numnah” that the comic relief he provided was a welcome respite from the tension of the competition.

Now that’s what I like to see in a Bee winner!  Someone with more smarts than I can ever hope to possess who can remain calm under pressure and even manage to make me laugh.  Other cases include the poor kid from a couple years ago who fainted during his turn, but got up, spelled his word correctly, and almost managed to win.  There was also a kid from a few years back who finished second but was classy enough to cheer on his competitors on whenever they spelled a word correctly.  These kids displayed the smarts and likability that will make them successful in the long run.  Anyway, I wish all the Bee competitors luck this year, and I look forward to being amazed again.

Nate Studebaker

One of the recent hot topics in the assessment world has been debate over the use of non-proctored assessments (NPAs) for pre-employment selection. (For those not familiar, the typical NPA scenario is one in which the employer asks job candidates to complete an unsupervised online assessment as part of the application process.)

While the convenience of NPAs is substantial, a number of potential risks such as cheating, proxy testing taking (e.g., getting my smarter roommate to take the test for me), and item disclosure are also readily apparent. Quite a bit of attention has been directed toward finding ways to control the risks associated with NPA as well as trying to determine the real-life negative impact of non-proctored administration upon test validity (frankly less that I would have expected based upon what has been published and presented).

However, reasonable folks differ quite substantially in their comfort level with NPA and likely will for the foreseeable future. This is a literature I follow pretty closely because pan provides both proctored and non-proctored assessment services. (Here is my presentation at the last SIOP meeting on how technology can help mitigate some of the risks associated with NPA). And given NPA’s rather controversial status, I guess it isn’t surprising that writers haven’t always been particularly nuanced in their positions. Some conclude that NPA is either the best thing since sliced bread while others believe it’s a professional abomination to be avoided at virtually any cost.

Doug Reynolds and colleagues’ recent must-read paper in Industrial and Organizational Psychology does a very nice job of contextualizing the discussion. Their major point is that we need to attend to all the relevant factors when a web-based assessment program is implemented. The issue is not simply whether or not a proctor is present during the test session. Rather, consideration also needs to be paid to factors such as the type of assessment to be implemented, how it is used in the hiring process, the nature of the applicant population, as well as an evaluation of what the alternatives may be.  As Reynolds paper concludes, “by positioning unproctored Internet testing as something that professionals should be either for or against, the underlying complexities associated with broader test deployment conditions will remain obscured.”

Reid Klion

A recent Wall Street Journal article I came across described how the City of Los Angeles is using a new multiple choice test to try and predict the likelihood of a 10-15 year old joining a gang.  The primary purpose of this assessment is to provide gang prevention programs where needed in order to reduce the risk these youths face.  The assessment is still in early developmental stages and it will take several years for the city to determine its effectiveness. 

Assessing risk through the use of behavioral assessments is nothing new in the employment world.  Many companies use assessments early on in the selection process to screen out applicants that pose a risk, whether it is for safety, security, dependability, or other potential areas of risk.  Test publishers have responded to the need for this type of assessment.  We have seen a growth in the development of short, cost effective assessments that are specifically designed to be used early on in the selection process to assess risk.

These assessments have been shown to lead to reductions in safety violations and absenteeism in newly hired workers.  Additionally, some of our clients have also seen direct cost savings with reductions in workers compensation claims.  Finally, screening high risk applicants early on improves the efficiency of the hiring process by not spending time on unnecessary interviews.  While the utility of this type of assessment is not going to be useful for all employment situations, it is certainly worth thinking about for jobs where risk is a concern.

Nate Studebaker

I take the same route to and from work every day. I’m sure others have had the same experience, but there are days when I get to work and don’t remember much of the drive. I don’t pay very much attention to my surroundings because the buildings I pass and the streets I drive on don’t change.

One afternoon at work, one of my colleagues (and fellow blogger), Reid, asked me if I had noticed the squirrel statue in the roundabout by our building. (If you aren’t familiar with a roundabout, it’s basically a four-way intersection, but it has a circular roadway around a central island). Even though Reid is a psychologist, I thought he had lost his mind. Surely if there was something as strange as a statue of a squirrel in the roundabout that I go through twice a day I would have noticed it. But sure enough, as soon as he pointed it out to me from our office window, I saw the squirrel positioned right in the middle of the roundabout. I couldn’t believe it and wondered how I could have missed something so obvious. Now I wonder what else I’ve missed not only in my surroundings but in my day-to-day activities as well.

It is easy for work processes to become like my commute – so routine that we switch to autopilot and perform our tasks without even noticing them. When was the last time you systematically evaluated your pre-employment hiring process? If you always go through the process in the same way without really looking at what you are doing, how will you make improvements? If you look for candidates without really thinking about what makes a person successful on the job, what are you missing? Perhaps we should take some time to step back and evaluate our processes and determine what it is that we are really trying to accomplish. We might find something important that needs to be altered (or at least something we hadn’t noticed before) that will help us be more successful. Feel free to call or email us so that pan can help you find your squirrel statue.

Jenni Ginsburg

In Indiana, we are serious about sports– think the tradition of Notre Dame football or the movie Hoosiers.  In Central Indiana, however, one sport reigns supreme: IndyCar Racing.  I’m not talking about NASCAR where the cars look like street cars with top speeds of  180 MPH.  No, I’m talking about open-wheel, 230 MPH racing.  I’m talking about sports royalty – three generations of Andrettis, not to mention A.J. Foyt, A.J. Jr., A.J. III, and now A.J. IV who will be driving in this year’s 500.  Right now, the local news is full of stories about familiar names – the likeable Helio Castroneves (better known for his Dancing with the Stars trophy and recent legal troubles than his two Indy 500 trophies), the always entertaining Danica Patrick, and others.  

However, due to my husband’s penchant for racing, I happen to know a few things beyond these names.  (OK, I admit it.  Before I posted this, I had it fact-checked by him  and his best friend.)  Most importantly, I know that there is one determining factor that leads to a driver’s failure or success: the pit crew.  No matter how good a driver is, a win is impossible without a good crew behind the pit wall.  That means that in order to be successful at Indy, you need to have multiple people working as team, managing their own responsibilities, solving problems on the fly, and communicating effectively with each other.

In many organizations, just as in the world of racing, it is not enough to be a superstar.  Of course, you have to be able to do your job well, but you also must function successfully within your work group or team.  To ensure that your team is most effective, it is critical to develop each member to be successful both independently as well as with each other

In many organizations, just as in the world of racing, it is not enough to be a superstar.  Of course, you have to be able to do your job well, but you also must function successfully within your work group or team.  To ensure that your team is most effective, it is critical to develop each member to be successful both independently as well as with each other.

pan offers assessments and custom solutions focused on employee and team development.  Let us help you build a winning team.

Jocelyn Courtney-Hays

Recently, I traveled to St. Louis to participate in a training class.

After a long drive, I arrived at a beautiful, newly remodeled hotel.

As I walked in, I was greeted at the front desk by a pleasant young woman named Heather. Heather gave me my room key, and I proceeded up the elevator to my floor. As I turned the corner, I noticed that my room door was propped open, and I started to think that something was wrong. I pushed open the door and what did I see? Nothing. No bed!

There was an imprint of a bed, but no bed. I called down to the front desk; Heather immediately apologized, said she would bring me a key to a different room (along with another treat), and told me my breakfast would be on the house. When she came with my new room key, she apologized repeatedly and offered to carry my bags to the other room. I declined, but was impressed with her willingness to take care of me.

I lugged my bags to the next room and saw the door propped open again! You guessed it. No bed! I called back down to the front desk and said, “Heather, you are never going to believe this. There is no bed here either!” I had to laugh because Heather was so confused by the situation that she was speechless.

She was incredibly apologetic again and immediately returned to the ninth floor to meet me. She offered me free room service that night, carried my bags to my room, and promised she would call and check on me later that evening.

After an hour, I finally had a bed. I ordered room service and Heather came to my room to make sure I was happy with dinner and the accommodations. I found out later that there was a convention the night before and the hotel had done some rearranging. The lack of beds was due to a communication failure between housekeeping and the front desk.

This could have been a truly miserable experience. Instead it turned into a comical mishap because Heather displayed a “Yes, I will take care of you attitude”. She had a drive for service. She was genuine, sincere, compassionate, and went the extra mile. She took a bad situation, resolved it quickly, and took the opportunity to make a connection with her customer.

Do you wish all of the people you encounter in the hospitality industry were like Heather? Do you wish your employees were more like her?
Are you missing out on candidates like Heather?

pan offers several pre-employment assessments that measure an individuals’ drive for service. In conjunction with one of our content partners, we also conducted an eSeminar to discuss how to select someone like Heather and the value it provides to an organization’s success.

Please feel free to post your comments - we would love to hear your hospitality stories, learn more about your current selection process, or share with you information about pan.

Sarah Lacey

While I have tried to avoid the massive media coverage of Susan Boyle (the Scottish singer whose sensibilities and personal style are somewhat less than high fashion), I came across a recent article that does a nice job of reviewing current psychological research on stereotyping.

From an evolutionary perspective, it was often life preserving to be able to tell whether the person approaching us in the forest was likely to be benign or pose an immediate threat to our wellbeing. Since aspects of the stereotyping process seem to be “hard wired,” we need to be aware of this and keep an open mind when meeting people. Indeed, much of the sensation about Ms. Boyle seems to lie in the fact that her musical skills were quite inconsistent with our initial impressions of her.

As might be expected, there is quite a bit of research examining how stereotypes can influence interviewers’ impressions of job candidates as well as how interviews are rather poor in predicting job performance. To this end, I think that one of the often unnoticed benefits of using a selection testing program is that unexpectedly positive test results sometimes force us to reconsider job candidates whose initial impressions (either in terms of looks, training, or background) are inconsistent with our stereotype of how potentially successful job candidates “should” appear. By failing to consider candidates who don’t necessarily fit the usual mold, we may be doing our organizations a significant disservice by missing potential star players.

Reid Klion

From a New York Post (April 19, 2009) article entitled BIG 'F' ON EMT ADVANCEMENT EXAM:

It was a massive medical failure for hundreds of FDNY medics who hoped to get promoted, as a measly 8 out of 721 city Emergency Medical Service workers passed the most recent lieutenant exam.

The 1.1 percent pass rate for the 2008 test is about 38 percentage points lower than the last time the exam was given, in 2004, when 1,044 medics took the test and 409, or 39 percent, passed, The Post has learned.

The FDNY was at a loss to explain how so many medics bombed the test. Entire Article

The issue here is that only 1% of the test takers passed the most recent exam! My initial conclusions are that the test was poorly constructed, the job candidates ill-prepared, or some combination of the two.

There are a number of factors which may account for this. It is even suggested later in the article that one factor may have been poor test taker motivation because the pay for lieutenants can be less than that for paramedics when overtime and meal allowances are considered (which still leads me to question why they would have taken the test in the first place).

Regardless, this only goes to highlight that using a test is a serious matter.  As a result, if an organization decides to use an assessment, it needs to be sure that the test has been well-constructed, fits the requirements for the position, and is used in an appropriate way.

Reid Klion

I just had the opportunity to read Bob Hogan’s recent book Personality and the Fate of Organizations. Always an entertaining writer and speaker (and long-time pan content partner), Hogan outlines how a leader’s personality quite frequently has a clear and direct impact upon an organization’s success or failure. By drawing upon examples from recent business news as well as history, he illustrates his arguments well.

One of the chapters that I found most engaging was “The Psychology of Managerial Incompetence.” Here, Hogan is quite willing to name names and outlines how what he calls the “dark side” of personality (or what clinicians would label a personality disorder) often lies at the root of executive—and organizational—failure.

Another chapter of note traces what he calls the ‘personality wars,” the period between the mid-1960s and mid-1990s when there was remarkably little interest in either academic or business circles in the role of personality. In a rather fascinating bit of intellectual and cultural history, Hogan links this to a number of factors including academic debates about what personality theorists should study (neuroticism vs. self actualization vs. mathematically-determined traits), a distracting preoccupation with the role of social desirability in personality assessment, as well a burgeoning interest at the time in both humanistic psychology and behaviorism. (For those who are interested, he then goes on to link today’s resurgent interest in personality to factors including development of the five factor model and the pragmatic realization that adverse impact was nearly unavoidable when cognitive assessments were used for personnel selection.)

Hogan is a well-known personality theorist. As might be expected, the book is scientifically grounded. It is written in an engaging and conversational tone that would be accessible to the layperson, but is also sufficiently unique in its tone and approach to be of interest to professionals as well. I would strongly suggest a place for it in your library.

Reid Klion