The Association of Test Publishers (ATP) will hold its annual Innovations in Assessment Conference on February 26 to 29 in Palm Springs, CA. The program is now posted and registration information available. Reid Klion
The IPAC Call for Proposals has been posted and will be open until the end of January 2012 for its annual conference to be held July 22-25, 2012 in Las Vegas. IPAC is an organization of applied personnel selection and assessment professionals to which a number of us at pan belong. IPAC focuses it efforts on increasing the awareness and application of empirically-based methods to HR practice. In addition to an annual conference, IPAC offers webinars as well as white papers on topics related to assessment and selection. Reid Klion
The program for IPAC’s 2011 Conference ( promotional video) to be held in July 17 to 20 in Washington, DC has been posted. Presentations will cover a wide variety of topics related to assessment and selection including developments in testing methods, technological innovations, legal compliance, and leadership development. Of special note, pan’s Jocelyn Hays will speak on using assessments to identify high potential employees. As in the past, pan is pleased to support IPAC 2011 and is a Gold Sponsor of this year’s meeting. Reid Klion
The issue of cheating on tests is a rather frequent topic in the news. Unfortunately, these are often superficial accounts and don’t provide much detail. In contrast, USA Today had a recent investigative piece that looked at cheating on statewide school-based tests and the methods used to detect it. There appear to be two basic detection approaches, and both look for test taking behavior that deviates significantly from the norm. One is simply based upon changes in scores from year to year. Specifically, when the scores for an entire grade go up very substantially from one year to the next concerns are raised. (Here, the typical criterion for a “very substantial increase” is when scores go up by 3 or more standard deviations, which is a huge difference). This is not to say that something untoward has necessarily occurred (it is certainly possible that there are some very good teachers in that grade) but it does raise concerns. This is even more the case if the grades drop substantially the following year. Another metric is the average number of erasures. (Recall that most school-based testing is still administered with paper and pencil.) Certainly, mistakes happen when kids put an answer in the wrong row, and all the following responses are out of sequence and need to be “fixed” and realigned. As a result, the focus is on whether entire groups of students have more erasures than others, especially if most of them result in correct answers. This is certainly not to say that cheating is widespread on school-based testing. However, it would only seem to be human nature that there may be times when teachers do things that unfairly increase student test scores, especially when these assessments are used, in my opinion often unfairly, as a primary measure of an educator’s effectiveness. Regardless, this report provides some interesting insights into how cheating behavior (which has been documented for over a millennium) is detected is school settings. Reid Klion
Registration is now open for the 2011 Conference of the International Personnel Assessment Council. To be held July 17-20 in Washington, DC, the meeting promises to be of great value to anyone working in the area of selection and assessment. Guest speakers include Elaine Pulakos (PDRI), Wayne Camara (College Board) and Eric Dunleavy, Mike Aamodt, and David Cohen (all from the Center for Corporate Equality). There are also a number of pre-conference workshops on topics including how to design an assessment strategy, best practices in questionnaire design, strategies for scoring job simulations, and job task analysis. pan is again proud to be a sponsor of the conference and that Jocelyn Courtney-Hays will be speaking. Reid Klion
First published in 1998 by the Department of Labor (DOL), O*NET (Occupational Information Network) is a framework and database of job characteristics for 956 occupations. It is used by a wide variety of professionals in domains including vocational guidance, education, human resources, and the determination of work-related disabilities. It is most typically used in the HR community as a data source in carrying out job analysis and writing job descriptions. (It is also worth noting that the O*NET website itself offers a broad array of information and resources including tools for matching job titles with O*NET occupations, user guides, and a variety of career exploration assessments Given its broad use, the DOL requested in 2008 that the National Academies of Science convene an expert panel to review the O*NET and consider its future directions. The result was a document of more than 200 pages that is available as a free download. The report documents the committee’s findings and also provides a good overview of the O*NET’s background. Some the conclusions drawn are that despite its limitations, O*NET is used by a very broad array of individuals and organizations and continues to be seen as valuable resource. One major suggestion is that the DOL focus its O*NET-related resources specifically on the continued development of the occupational database itself and defer to others the development of tools and other ancillary related products. To help maintain this focus, it was further suggested that an advisory board be created to guide DOL in these efforts. While lengthy, the report provides a good overview of where O*NET has been and a vision for its future. Reid Klion
The Standards for Educational and Psychological Testing,
known as the “Joint Standards” because they are a collaborative effort of the
American Educational Research Association, American Psychological Association,
and National Council on Measurement in Education, were last revised in 1999. Since
that time, much has changed in the world of testing to include the proliferation
of web-based assessment, frequent high profile educational testing in schools, and a growing
awareness of the complexities associated with fairly assessing individuals with
disabilities and for whom English is not a first language.
Over the past few years, a number of committees representing
a broad range of constituencies were convened to develop proposed revisions to
the Standards. Their work was recently posted and comments are now being solicited from both testing professionals and the public.
Providing input and review are critical because the Standards (along with the SIOP Principles and EEOC's Uniform Guidelines) are considered the foundational
documents that define accepted practices in world of assessment. Since the
Joint Standards cover all forms of psychological assessment, they are somewhat
broader in scope than the other two sets of documents, but are still critical in
defining professional standards for human capital assessment.
Reid Klion
The Call for Proposals for IPAC’s 2011 Conference on Personnel Assessment (to be held July 17-20 in Washington, DC) is now open and will be until the end of the month. The program committee is open to proposals for symposia, panel discussions, tutorials, and workshops on topics related to talent management. More information on the conference is also available. Invited speakers include Wayne Camara (College Board), Elaine Pulakos (PDRI) and Eric Dunleavy, Mike Aamodt, and David Cohen (all of the Center for Corporate Equality which was blogged about earlier). A number of us at pan have been active in IPAC for the past several years and the conferences are notable for their conceptual rigor, practical applicability, and relaxed atmosphere. Reid Klion
The Chinese Civil Service exam was administered to nearly over a million job candidates in early December and the topic of post last year. The National Public Servant Exam is required for hiring into many government jobs. Civil services examinations in China have a very long history and have always been the focus of much attention going back several centuries. Given the recent difficulties that young Chinese college graduates are having in finding work, it is a bit surprising that some 40,000 fewer people took the exam this year than last. Regardless, 1.41 million people applied for the exam, and 1.03 million actually took it. Since there are only 16,000 government jobs reported to be available at present, that means the selection ratio is 1:64. Reid Klion
The program for the Association of Test Publisher’s Annual Innovations in Assessment Conference, to be held February 27 to March 2, 2011 in Phoenix, has been posted. As usual, the program provides a nice window into the broad use of assessments across the education, certification, clinical, and HR spaces. Highlights include a number of presentations on item security and protecting intellectual property as well as a presentation by Richard Tonowski. Ph.D., of the U.S Equal Employment Opportunity Commission who has kindly agreed to attend and provide an update from his agency. For many years, the EEOC been the primary the “rule making” agency when it comes to acceptable testing practices in the US. Discounted early registration is available until December 1. Reid Klion
The program for the 2010 IPAC Conference has been posted. To be held July 18-21 in Newport Beach, CA, the meeting provides a prime opportunity to learn about the applied uses of personnel assessment tools in a friendly and collegial atmosphere. pan is proud to be a conference sponsor and several of us, including Nate Studebaker, Jocelyn Hays, and me, will be presenting. Conference registration is still open if you are interested in attending. Reid Klion
Speaking of SIOP, the ATP I/O Division will be holding a Meeting and Reception on Friday, April 9th from 4:30pm until 6:00pm in Room 315 of the Hilton Atlanta. It will be hosted by John Weiner (Chair), Julie Carswell (Vice Chair), and pan’s very own Jocelyn Courtney-Hays (Secretary). If you have a chance, please stop by to learn more about ATP and the Division, hear about updates in the legal and policy arenas regarding employment testing, and have the opportunity to network with others interested in employment testing issues. Reid Klion
Association of Test Publishers recently released the third edition of its Model Guidelines for Preemployment Integrity Testing. Integrity tests are assessments that have been specifically designed to predict counter-productive workplace behavior such as theft, poor productivity, and substance abuse and are typically used for screening candidates for entry level positions. While the Guidelines obviously address issues around integrity tests, they also provide a good overview of best practices in pre-employment assessment use as well as review of the empirical work on the validity of integrity tests. Copies are available from the Association of Test Publishers; Lauren Schieb should be contacted for details. Reid Klion
A question we often hear about assessments has to do with the legal issues which surround their use. A critical point that sometimes gets ignored in these discussions is the fact the same legal principles pertain when any method is used to select employees, whether testing is part of it or not. A good example is a recent EEOC suit that Walmart settled for nearly $12 million due to gender discrimination in hiring distribution center personnel for a facility in Kentucky. Essentially, based upon the assumption that the positions were not suitable for women, Walmart only selected men for these jobs. In contrast to most testing cases, this did not involve adverse impact which occurs when the differences in selection ratios between groups are due to inadvertent factors. (For example, if Walmart had used a physical ability test with a cut score that was higher than necessary given the demands of the job, this would have been a case of adverse impact because it would have eliminated a large proportion of female applicants.) Rather, this was a case of gender discrimination where women simply weren’t considered. However, the critical point is that the same EEOC stipulations apply to how employees are selected and hired--whether tests are used are not. Reid Klion
National Council of Examiners for Engineering and Surveying (NCEES) won a civil lawsuit against a civil engineering candidate who attempted to take one of the organization’s primary exams with wireless a/v transmitter, video camera, receiver, pocket video recorder, and battery packs all on her person (see picture) with the intent to steal the content of the exam. She apparently had done this previously in 2005 but was caught by an alert proctor in the most recent incident. The federal court awarded NCEES a judgment of over a million dollars for loss of intellectual property. This figure was determined by estimating the cost that is involved in developing the items that had to be discarded. This individual was previously convicted of fraud in a criminal court for charges related to the incident. I suspect that judgments of this type and size are rare but they do highlight the costs that can be involved in the development of sophisticated testing programs. Reid Klion
The call for proposals for the 2010 IPAC Conference is now open. Formerly known IPMAAC, IPAC will meet on July 18-21 in Newport Beach, California at the Hyatt Regency. Session formats include symposia, panel discussions, paper presentations, tutorials, and workshops. Presentation proposals will be accepted until March 5th.
pan has long been active in IPMAAC and now IPAC and is proud to have been an conference sponsor for many years. IPAC is an organization of applied HR selection and assessment professionals, with members who work for both public and private-sector organizations. If you aren’t familiar with the organization, I might suggest you take a look a IPAC website and consider attending or even submitting to the conference.
Reid Klion
The Association of Test Publishers Conference is coming up on February 7-10 in Orlando. The ATP Conference is a good opportunity to see what is new in the world of testing and meet many of people who are active in the industry. The conference program has tracks on promoting state-of-the-art assessment, integrating technology into assessment, and innovations in test security. pan will also have a booth in the exhibit hall, and a number of us will be attending including Doug Cole, Jocelyn Courtney (who recently became Secretary of the ATP I-O Division), and Jenni Ginsburg. If you will be there and would like to arrange a time to meet with one of us, please drop me a note. We look forward to seeing you in Orlando in a couple of weeks! Reid Klion
The Indian Institutes of Management decided to move the Common Admission Test (required for admission to Indian MBA programs and similar to the GMAT in North America) to a computer-based test delivery model available over a ten day period. (In previous years, the test was administered in paper/pencil format on only one day.) This year, nearly 200,000 candidates applied to take the exam. Unfortunately, due to a number of factors including widespread hardware issues and a Cornflicker virus infection, there were major problems in delivering the test. A number of testing centers had to be shut down with reports indicating that some 10%-18% of candidates had to be re-scheduled (though it appears all who were not successful in doing so). Given the scale of the program, it has garnered substantial coverage in the press as well as in the Indian Parliament and brought forth public interest from number of other test delivery organizations that manage large-scale test delivery projects in India and would like to take on the project next year. Reid Klion
In the past, I have blogged about testing in China and Korea. While we often thinking of standardized testing as a Western phenomenon, it really has its roots over 2000 years ago in the Chinese imperial exam system with the concept later being adopted in the East in the 19th century. The scale of some of these Asian programs is quite impressive (as well as is the potential for high profile problems which I will blog about tomorrow in regard to recent issues in India with delivery of their equivalent of the GMAT). At the end last month, nearly a million candidates took the Chinese Civil Service Examination. Attributed to the economic slowdown, the number of test takers ( photos) has increased by a factor of 15 since 2003. Given there are only 15,000 openings for a million test takers, the stakes are impressively high. These government positions are often coveted for their stability and are sometimes referred to as the "Iron Rice Bowl." Reid Klion
pan recently developed a strategic partnership relationship with Global Psychometrics Services (GPS). Headed by Reed Castle, Ph.D., GPS is a recognized leader in the development of high quality testing programs for certification and licensure organizations. By combining pan’s network of more than 800 proctored testing centers with GPS’s experience in test program development, our partnership can now provide a full-service certification solution ranging from job analysis studies and test content creation to proctored test delivery. More information can be found here. Reid Klion
Just a reminder that the Call for Papers for the International Testing Commission Conference (to be held July 19-21, 2010 in Hong Kong) closes on December 1. I had the opportunity to attend the ITC Conference in 2007 and would suggest it to anyone interested in increasing their understanding of issues related to the globalization of assessment. There are multiple considerations that emerge when a testing program is implemented internationally with challenges ranging from the practical to the psychometric to the legal. As the need for assessment programs that be used in multinational contexts grows, so does the necessity for assessment professionals to understand the complexities that come along with the territory. Reid Klion
While virtually all the proctored testing that pan does is in the HR space, there is a huge amount of proctored testing that takes place within an education context--think PSAT, SAT, MCAT, GRE, LSAT and the like. A recent NYT piece focuses on the problem of proctors who fail to do their jobs or even manage to be disruptive during SAT testing sessions. The article indicates that SAT proctors are not specifically trained for the task (which surprised me a bit since our policy is to train all proctors on each program they supervise). And reviewing some of the reported issues that occur during paper and pencil testing, it is clear that computer-based test administration (as we use in our testing centers) can ensure a much more standardized experience in terms of test timing, the right test forms going to the right candidate, and the like. One bit of advice provided by an SAT test coach (yes, this is apparently a whole industry in and of itself) is that if a candidate notices a problem during test administration, he or she should either voice a complaint immediately and ask that it be addressed or “forget about it” (as opposed to simply grumbling about it after the fact). Here, it is important that test takers assume some responsibility in advocating for themselves by pointing out problems if they become apparent. The coach also goes on to say that “you have to expect that there will be distractions. If you’re seated next to a kid who is muttering under his breath, or sneezing, or grunting, just deal with it. Move on.” Reid Klion
The program for the Association of Test Publisher’s Innovations in Testing Conference (to be held February 7-10, 2010 in Orlando) is now available. As usual, there appear to be a large number of interesting presentations on the docket. Some that initially caught my eye focused on topics including a survey about the future of testing, detecting cheating, and ATP’s new revised guidelines on pre-employment integrity testing. Also, registration is open with a $75 discount to folks who do so before November 16. Reid Klion
I receive regular updates from ATA which is the largest testing company in China (and also happens to trade on the NASDAQ as ATAI). Last month, they administered the National Judicial Examination in China to more than 350,00 candidates. An individual must pass this exam to work in Chinese legal system and is a requirement for lawyers, prosecutors, and judges. Due to the high stakes nature of this exam, there were major concerns about cheating. (Cheating on exams in China has been an issue for over 2000 years with reports dating back to at least the Han Dynasty. There are a number of cultural factors which contribute to this which I hope to find the time to blog about fairly soon.) In addition to taking candidates’ temperatures to make sure no one was ill, many test centers were also outfitted with handheld devices to detect the presence of wireless earphones well as equipment to block mobile phone signals. Additionally, 8 fixed monitoring stations and 2 mobile signal detection vehicles (labeled “Signal Detection”) were used to identify “suspicious” signals in the proximity of testing centers in Beijing. In at least one instance, a test taker was discovered to have a wrist watch which could send and receive data. Reid Klion
There was a recent NYT Op-Ed by a guy who is authoring a book on the standardized testing industry and his exploits as a “test grader.” For those not familiar with this aspect of the testing industry (or who don’t have school-aged kids), the standardized tests administered by schools often include “open ended” questions where students need to provide a written response. In order to scoring, armies of temporary workers are hired to do the work. The author recounts some of the goings-on as well as his less-than-conscientious behavior in this work. His conclusion is that “scoring should be done only by professionals who have made a commitment to education—rather than people like me.” This brings a couple of thoughts to mind. The first is that one wonders if a work attitude assessment should be part of the hiring criteria for the essay graders. In this, as in all jobs, we want people who will carry out their work in a responsible and conscientiousness manner. Additionally, it also highlights why there is significant interest in the use of automated essay scoring systems where computers grade the written responses. There are now several years of research on these systems, and they tend to work well. The obvious benefits are that compared to human raters, they are not susceptible to either fatigue or poor work ethic. And unlike humans, they are 100% reliable because they will grade the same essay the same way every time. Reid Klion
I had the pleasure of attending the joint IPMA-HR/IPAC meeting this past week in Nashville. The conference was composed of both public sector HR professionals and assessment-oriented psychologists. The mix led to some interesting discussions where the HR folks could hear about the nitty-gritty involved in testing and the psychologists learn more about the day-to-day challenges faced in managing public sector workforces in challenging economic times. Many of the presentations are posted. As always, Bob Hogan gave an entertaining and scholarly talk on the critical importance of leadership in organizational success. Bryan Baldwin (whose blog HR Tests is certainly worth a look) also gave a great talk on the potential role and challenges associated with using social networking websites in recruiting. (Bryan didn’t post his presentation but his recent article can be found in the Assessment Council News on page 6.) Reid Klion
IPAC (formerly known as IPMAAC) will be holding its annual conference in conjunction with IPMA-HR on September 12-16 in Nashville. IPAC is comprised of individuals who are interested in public sector selection and assessment. As it has for the past several years, pan is pleased to be a conference sponsor. Additionally, Nate Studebaker will present on his work with the Texas Department of Family and Protective Services that helped to streamline their assessment process while I will participate in panel discussion on technology trends in assessment. We hope to see you there! Reid Klion
While it had been brewing in the professional literature for the past three years or so, the dispute between the authors of the MMPI has become public. (The MMPI is the pre-eminent clinical assessment that has been used for over 70 years. While sometimes discussed mistakenly in relation to pre-employment testing, it is almost never used to select employees because it is a clinical, not a work-related, assessment. About the only times that it is used to hire employees is in occupations where it is critical to rule out significant psychopathology such as police or nuclear plant operators.) The issue at hand has to do with a “reformulated” version of MMPI-2, the MMPI-2-RF. The “RF” version, which has just been published, is shorter and described as easier to interpret that its predecessor. However, James Butcher, the lead author of the earlier MMPI-2 is highly critical of this revision. The issue has become personal because the authors of the “RF” version (Auke Tellegen and Yossef Ben-Porath) had been long-time colleagues and collaborators with Butcher. The result has been charges of conflict of interest and mismanagement on the part of the University of Minnesota press which owns the MMPI. Reid Klion
An uproar has been brewing because the images used for the Rorschach inkblot test been posted on Wikipedia along some of the more common responses to each. . As might be expected, some psychologists are concerned that this may impact the validity of the test, especially in cases where the assessment is used within a legal context (for example, in a child custody evaluation or pre-trial assessment of a sex offender) as opposed to a counseling situation. In contrast, others have pointed out that the Rorschach is almost never used in isolation but in combination with a number of other clinical assessment tools to make decisions and that this same information about the Rorschach is available in training texts on the test as well as from other sources. However, this event does highlight an important issue about test security and the Internet. While these images have been available to the public previously, their posting on Wikipedia does make them easily and broadly accessible. And even if we accept the argument that posting this information will have little impact upon the Rorschach’s validity, it should be noted that real harm can be done when test security is compromised. Not only is copyright law often being violated and the validity of a test compromised that may have taken years to develop, but the public may be placed at harm. For example, how many of us would be concerned we found out that our child’s pediatrician only passed a licensing exam because he or she had knowledge of the test content before taking the licensing exam? As it turns out, the individual who posted the images actually is an emergency room doctor from Saskatchewan. When questioned, he sees little harm in what he has done and goes on to state that his elderly father passed a driver’s license eye exam by looking it up on the Internet. I guess our only hope is that his dad can actually see well enough to drive to keep both himself and the rest of us safe. Reid Klion
The American Psychological Association will be holding its annual convention next week in Toronto on August 5-9. Both Nate Studebaker and I were fortunate enough to be invited to speak. Nate will be chairing a symposium on the implementation of 360º feedback programs in decentralized organizations while I will be chairing a panel discussion on the ethical issues surrounding the management of assessment data.
Two panelists in my session work for long-time pan content partners: Ryan Ross of Hogan and Julie Carswell of Sigma Assessments. If you will be at APA, Nate and I would certainly be pleased to get together to either discuss business, socialize, or both. Please drop a note to either Nate or me so that we can sort out the logistics. We hope to see you in Toronto! Reid Klion
I was on the SIOP website the other day and came across the “ SIOP FYI on Workplace Topics” section which has some rather extensive information on both employment testing and coaching. While I haven’t had a chance to review the coaching section in any detail, there is a wealth of descriptive information about assessment. Overall, the presentation seems quite balanced with a section on reasons to use (as well as not to use) assessments, types of tests, and item formats. These pages provide an excellent, even-handed orientation which focuses on the pros and cons of various approaches to assessment. These resources would be quite useful to individuals who want to learn more about using assessments in the workplace. Interestingly, there is a “hit counter” on the bottom of each page. While I don’t know when the pages or counters were created, it appears that some have been viewed fewer than 1000 times. Let’s hope they find the audience that they deserve. Reid Klion
ETS recently rolled out a product called the Personal Potential Index. Intended for use by graduate school applicants, it is a standardized rating form with 24 items pertaining to factors such as creativity, communication skills, and teamwork that is completed by a set of student-nominated “evaluators” (typically professors). The intent is to provide graduate program admission committees with information beyond GPA, GRE scores, and the traditionally glowing letters of recommendation. (Exaggerated letters of recommendation are not a new problem; there is a classic American Psychologist article from 1966 on the topic rather brilliantly entitled “Mine Eyes Have Seen a Host of Angels.”) The ETS tool is premised upon the recognition that non-cognitive factors likely contribute to success in graduate school (as well as virtually everywhere else in life) and part of a larger ETS project to find ways of assessing them. There are concerns about using self-report inventories in a graduate admissions context due to “faking” (though such instruments have been found to function effectively in pre-employment situations). As a result, attention is being directed to rating scales as well as biodata and situational judgment tests. The Personal Potential Index appears to be a work in progress. There are plans for developing local norms as well as procedures to adjust for differences in individual rater severity. Additionally, only further research will be able to determine the extent to which this rating approach actually adds any incremental validity (and all of this is predicated upon the ability to collect enough real-life data). A concern that comes to my mind is the degree to which most professors actually have enough observational data to accurately rate their undergraduate students’ “non-academic” behaviors, especially in the case of institutions with large, often anonymous, classes. Additionally, one also wonders about the extent to which raters, knowing the implications of their evaluations, will engage in the same rampant exaggeration that is well-recognized in letters of recommendation. Reid Klion
Proposal submissions for the 2010 SIOP Conference to be held April 8-10 in Atlanta are now being accepted. The deadline for submissions in September 9; your 2010 dues must be paid in order to submit. Reid Klion
pan is a member of the Association of Test Publishers (ATP) and a number of us are active in the organization. ATP describes itself as the “intelligent voice for testing” and serves as a trade organization for those in the assessment industry.
ATP will have its annual Innovations in Testing Conference on February 7-10, 2010 in Orlando. While the date is several months off, I recently received notice that the Call for Presentations will open next month on July 7 and close August 8. I'll post a reminder when the date rolls around, but it is probably good to start thinking ahead about possible submissions as the dates are right in the middle of summer vacation and often not the time most of us are thinking about conference presentations. In a change from prior years, presentations are being solicited around four tracks: - Promoting State-of-the-Art Assessment: Risks and Benefits
- Advancing the Image of Testing
- Effective Integration of Technology Into Assessment
- Innovations in Test Security
In other matters related to ATP, registration is now open for 1st Annual European Association of Test Publishers (E-ATP) Conference, which will be held September 30 through October 2, 2009 in Brussels. Also, some online ATP-related resources for your perusal include the E-ATP wiki and the I-O Division blog.
Reid Klion
|