DDI’s Talent Management Intelligence

Incredible, Edible. . . Cognitive Testing?

By Evan Sinar, Ph.D.

In recent years, the humble egg, once viewed as a shell-encased, dietary time bomb, has been increasingly lauded as an ideal high-protein food (so long as you stick to the egg whites). In a similar way, the changed perception of the risks/benefits of eggs mirrors the changing perceptions about cognitive testing.

Cognitive testing long ago fell out of favor due to a variety of concerns including those related to legal defensibility and test security. But as jobs grow in complexity, the need to understand a candidate’s level of intelligence and problem-solving ability in order to project his or her job effectiveness has remained, and, in many cases, even increased. Today, employees have more opportunities than ever to apply cognitive skills on the job in the pursuit of organizational goals. In fact, the move away from cognitive testing has coincided with a confluence of offsetting trends that have made cognitive ability skills—and the job competencies they connect to—increasingly valuable.

Just as what happened with all of the bad press that eggs once received, the naysaying about cognitive testing, ultimately, was overblown. There was no need to avoid eggs; people just needed to be smarter about how they consumed them. The same goes for cognitive testing. Organizations can do cognitive testing, they just need to be smart about it. Fortunately, advances in testing methodology and technology have allayed most of the long-held concerns.

So what changed? Plenty, as it turns out. “Adaptive” cognitive testing instruments are now available that remedy many of the old concerns and offer new advantages.

Here’s how these new instruments address the most common concerns:

They safeguard against cheating. With some of the new adaptive instruments, the test is structured so that each person taking it may see a different set of items, presented in a unique sequence. There are some forms of cheating you simply can’t avoid without having the candidate come in and take the test in person (for example, you can’t prevent someone from having a smart friend sit next to the candidate and tell him or her what the right answers are). In terms of large-scale cheating, however, such as where an answer key is created and widely shared, the new adaptive instruments effectively make that form of cheating extremely difficult, if not impossible.

They are more engaging and perceived as more credible. A breakthrough feature in one or more of the adaptive cognitive tests now available prevents candidates from disengaging from the testing process because they view the test as either too easy or too difficult. With these instruments, after the candidate has input an answer for an item, whether or not the answer is correct will determine the difficulty of the next item presented. In other words, as far as difficulty goes, the questions will always be “just right.” When paired with test content that avoids perceptions of being overly specific and unrelated to the job for which candidates are applying, this item targeting leads candidates to feel as if each test item is truly reflective of their ability.

They are just long enough to be accurate. One of the foundational ideas behind adaptive cognitive tests is that every single item is calibrated to the item that’s best tailored to the person’s ability level, as described above. With every item targeted directly at discerning a candidate’s true ability, less time is needed to come up with an accurate estimate of his or her cognitive skill. That means an organization can get the same precision from a 15-item test as from a traditional test that might include 40 or 50 items—a difference to which candidates are likely to react favorably.

They are more fair. The new adaptive cognitive tests replace the text-based items that predominated in the past with non-text-based items. One major benefit of this is that it eliminates the opportunities for differing interpretations of text-based items that might be grounded in cultural or language differences. Therefore, it mitigates the risk of adverse impact when the test is included as part of a selection system. It also allows for global delivery of the test, as language is no longer a barrier.

As jobs continually change, it’s a good idea to periodically look at your selection systems to make sure the tools you are using meet all of your criteria. Now is also a good time to examine (or reexamine) your systems through the lens of cognitive ability. As your jobs evolve, do they require more advanced reasoning skills? And do the assessments you are currently using help you get at the cognitive data you need? What are the potential consequences of your organization not assessing for cognitive ability?

Answers to questions such as these can help to guide you as you refine your selection system. And unlike as it was in the past, you should feel good about seriously considering adding a cognitive test to your arsenal.

Evan Sinar, Ph.D., is Director of DDI's Center for Analytics and Behavioral Research (CABER) and Chief Scientist.
 
For a more in depth look at how cognitive tests have evolved in recent years, read “Cognitive Testing Has Come Out of Its ‘Shell’” or learn more about DDI’s Adaptive Reasoning Test.

Volkswagen Chattanooga's Innovative Plant Start-up

By: Elizabeth Kabus

Eighty percent of the success of the people you hire is determined by the selection process, according to Volkswagon's staffing experts. So when the world’s third largest auto manufacturer needed to staff a new plant in Chattanooga, Tenn. to build the new Passat, selecting the right talent was crucial to their success.

That undertaking was especially challenging as there were more than 80,000 applications for the 2,000 production, maintenance, managerial, and professional positions.

DDI partnered with Volkswagen Chattanooga to design a multistep selection process for each of the four job families to include sourcing, screening, assessment, and interviewing. Jobs were posted with a number of outlets to reach the most people and “fill the funnel” with a diverse pool of candidates. Volkswagen Chattanooga’s HR team incorporated DDI’s career batteries to narrow the pool of candidates to those who were well-qualified and also appropriately motivated. In addition, DDI’s Targeted Selection behavioral interviewing system was used to check for the knowledge, skills, and motivations related to the critical competencies for each job family.

The four selection processes were defined by the best-practice use of simulation and assessment, with different tools employed for each job family. Among these was a state-of-the-art production simulation, in which applicants progress through three hands-on, experiential exercises. Volkswagen Chattanooga evaluates each individual’s quality orientation, safety orientation, and ability to maintain an appropriate pace of work over time. This simulation provided a realistic preview of the kind of work performed on the production line.

Since hiring began in November 2009, over 85,000 applicants have applied for all positions, and over 13,000 have been tested. Two thousand applicants have accepted offers, enabling Volkswagen Chattanooga to meet its staffing goals with high-quality workers. Candidates rated the process a 4 or higher, with 5 being the most positive, on multiple aspects of the selection process.

Among the comments candidates share on the evaluation forms: “I felt it gave us a clear view on what to expect. I appreciate the opportunity and if given the chance I would make a great employee at Volkswagen. Thank you,” said one. “This was a very unique approach to measuring a person’s capabilities…I enjoyed it!” said another. And someone else said a lot with just a few words: “Ready to start today!”

Click here to view the full Volkswagon Chattanooga story.

Missing Out: Five Ways to Misuse Online Screening Tools

Note: For e-mail subscribers, please click on the title to access the full article

By: Evan Sinar, Ph.D

Part Four

Misuse #4: Generating Legal Risk via Haphazard Screening

The flexibility of online screening systems to enable idiosyncratic filtering of large candidate pools, if not carefully monitored and managed, can be an unrecognized source of legal risk in at least two ways. First, it can increase the potential for differential treatment of candidates with similar backgrounds. If one such candidate is a member of a legally-protected candidate group and is screened-out while another non-protected group candidate is progressed, this can be considered a form of discrimination. Second, using an online screening tool to search or prioritize candidates may automatically classify all individuals searched against as formal applicants. This can substantially impact the necessity to document the job-relatedness of each requirement or search term. Although these legal risks can be managed through careful planning, if left ignored they can quickly threaten the defensibility of the entire selection system.

 

Misuse #5: Neglecting the Untrainables

A final common misuse is factoring a candidate’s past skills, experiences, knowledges, and other “trainables” too prominently into the selection process in general, and within online screening tools in particular. New hire ramp-up time is a critical concern for many employers given lean staffing levels and low budgets available to support extensive training and on-boarding. Hiring an individual who already possesses many of the perceived prerequisites to effective performance is one way to counteract these restrictions.

However, this approach also brings some important caveats. Taken too far it can lead to a situation where “untrainables” – attributes that are difficult or impossible to develop someone in after they have been hired – play, at best, a minor role in progressing candidates to a final interview. These attributes include vital building blocks such as adaptability, integrity, learning adeptness, and innovation, which can be the foundation for an employee expanding his or her role within and beyond the original job description. Hiring processes that fail to early and adequately capture information about these characteristics, using tools designed specifically for this purpose such as cognitive tests and personality inventories, waste the opportunity to understand the “hows” behind a candidate’s background for the sake of knowing only “what” was achieved.

What a candidate brings to the job on day one may indeed help him or her ramp up quickly, but without pairing this knowledge with insights about untrainables, an employer risks bringing on board an employee with a severely restricted ceiling for future growth. Exercise caution to ensure that online screening tools do not overweight trainable characteristics merely because they are more easily searched and screened upon.

 

Summary 

Making the perfect the enemy of the good is a critical misstep when online screening tools are used to target the mythical ideal candidate. But it’s far from the only caution to keep in mind, and other consequences of squandering the potential of these tools can be equally damaging or even more so. Online screening is here to stay and can add immense value and efficiency when deployed appropriately. However, if the recent negative attention directed toward online screening systems is causing you to re-evaluate and seek to improve the way you use these tools – and it should – consider this broader set of potential concerns while redesigning your processes and approaches, and when you’re determining the influence online screening tools will have within your broader hiring process.

How is the role of technology changing your processes for large-scale, geographically-dispersed candidate screening and selection? What do you see as the next evolution of technology-driven hiring
tools?

This article was originally published by Recruiting Trends and is reprinted here with permission.

 

Missing Out: Five Ways to Misuse Online Screening Tools

Note: For e-mail subscribers, please click on the title to access the full article

By: Evan Sinar, Ph.D

Part Three

Misuse #3: Codifying a Narrow Route to Job Success

Problems also emerge when online screening criteria are based solely on conventional or “typical” qualifications. Such qualifications may include a minimum number of years performing a similar job, a particular college major, or demonstrated knowledge of a specific industry. When these are used as hard, uncompromising limits to narrow the candidate pool from the outset, a company loses any chance to consider non-traditional backgrounds. Veterans of the military, in particular, often have highly-relevant skills that aren’t spotted and credited accordingly if a conventional search approach is used – meaning that these valuable individuals can slip through the process entirely unnoticed. If “required” skills and experiences don’t fully account for variations in the experiential opportunities available to those from other cultures and countries, they too can be inadvertently overlooked. Even “cloning” current employees to identify screening factors can be problematic – if a company’s talent status quo is no longer sufficient, modeling new employees after existing ones may only perpetuate current workforce challenges.

The long-term impact of this type of misuse can be wide-ranging given the consequences of narrow employee perspectives for many key talent management issues, including poor global acumen, stifled innovativeness, and proces inefficiency stemming from long-unchallenged “standard operating procedures”. Instead, screening factors should accommodate alternative routes to success whenever possible. This could involve broadening the scope of “acceptable” backgrounds to encompass adjacent fields, industries, and skillsets, appropriately recognizing military service as a valuable source of experience, crediting candidates for their interest in and active learning about emerging areas where extensive formal work experience may not yet be possible, and carefully vetting all criteria to ensure that candidates from different cultures are not mistakenly excluded.

This article was originally published by Recruiting Trends and is reprinted here with
permission.

Missing Out: Five Ways to Misuse Online Screening Tools

Note: For e-mail subscribers, please click on the title to access the full article

By: Evan Sinar, Ph.D

Part Two

Misuse #2: Using the Least Accurate Tools to Eliminate the Most Candidates 

Online tools used for initial candidate screening frequently capture three types of information: years of experience, education, and accomplishments indicating skills, knowledge, and training. While this information can aid in prioritizing candidates at the earliest stages of a funnel-based selection process, their relatively low level of predictive accuracy simply does not justify their use as primary screening methods – that is, using these factors alone to exclude a majority of the candidate pool from further consideration.

When ranked against all major selection tools in their ability to predict job effectiveness, measures of training and experience and those of years of education fall near the bottom, and far below other methods such as cognitive tests, personality tests, and structured interviews. The maximum proportion of candidates to screen out with a single selection tool should roughly match the accuracy with which the tool can gauge job success. In the case of skill and experience requirements often used for automated candidate screening, their accuracy generally supports screening out a relatively small proportion of candidates, in most cases no more than 20-30%.  Be wary about using these factors to screen candidates at substantially higher rates. Rather, supplement and sequence them with other, more predictive, selection tools to strike the right balance between the need to rapidly prioritize a large candidate pool, and the cumulative accuracy for gauging job success of the tools used to reach this goal.

 

This article was originally published by Recruiting Trends and is reprinted here with permission.

Missing Out: Five Ways to Misuse Online Screening Tools

Note: For e-mail subscribers, please click on the title to access the full article

By: Evan Sinar, Ph.D

Part One

Online screening tools have recently come under fire as the focus of a June 2012 book by Peter Cappelli, “Why Good People Can’t Get Jobs: The Skills Gap and What Companies Can Do about It.” These tools are blamed for enabling the “purple squirrel” phenomenon where a hiring manager sets the expectation that a single candidate – the purple squirrel – can be found with an ideal combination of skills, education, and experience. These then become must-have qualifications used to filter a candidate pool, often using an automated screening system. Frequently, no candidates emerge from such a search, spawning employer claims of a pervasive skills gap among the available candidate pool.

Simultaneously, from the candidate viewpoint, the use of the ideal profile as a benchmark has excluded hundreds or even thousands of possibly suitable individuals from consideration.

Beyond these concerning outcomes, poorly-designed or unstructured applications of online screening tools also have other dangerous consequences, especially when these tools are key workhorses within large-scale recruiting processes. For online screening to fulfill the objectives it was put in place to achieve – improving quality of hire, increasing hiring process efficiency, and democratizing candidates’ access to new job opportunities – users must recognize and avoid several common misuses.

Misuse #1: Hiring Through a Straw, Not a Funnel 

Hiring processes work best when designed as a funnel with multiple selection tools ordered in a well-planned sequence in which the full candidate pool enters the mouth of the funnel and candidates move through a series of tools, some online and some in-person, each providing unique insights about likelihood to succeed on the job.  At each stage, information is captured to screen out those less likely to succeed, with remaining candidates progressed onward until a final hiring decision is made based on the full set of data gathered.

Contrast this model with an online screening approach that is more like a straw than a funnel, where these tools are used to screen out all candidates other than the very low proportion meeting a narrowly-defined profile of skills and experiences, with only this group remaining eligible for hire. Although this straw model can be credited for its focus, this minor advantage is often more than outweighed by its questionable accuracy, excessively constricted definition of suitable candidate backgrounds, and exposure to legal risk, each of which will be outlined further in Misuse #2 in the next part of this blog.

 

This article was originally published by Recruiting Trends and is reprinted here with permission.

 

 

With Tests, Measuring Better Beats Looking Better…But Why Choose?

This is the second of a series of blogs from DDI associates presenting at the 26th annual SIOP Conference.

By Evan Sinar, Ph.D.

Employment tests have been used to drive objective, efficient and accurate hiring decisions for decades.  Testing’s relationship with technology is newer and still evolving, however. Technology is changing the way tests are administered while the use of online testing greatly increases the reach and convenience of these tools for companies and their candidates. 

These are important benefits, but it’s not enough: technology can and should do more for employment testing and can dramatically advance testing in two major ways – Looking Better and Measuring Better. Looking Better means test appearance – does technology enable items that are more realistic? More interactive? Measuring Better means test effectiveness – does technology generate more information about an applicant’s skills? Can the test better predict what they’ll excel in and struggle in if hired? 

Obviously, both of these are worthwhile goals – but they can conflict. Better measurement can involve more established or time-consuming item types that don’t incorporate interactivity, graphics, or other technological advancements. On the other hand, exclusively focusing on “looks” can pair, for example, a highly-realistic video with a low-value radio button to indicate a response – neglecting measurement for the sake of appearances. 

To avoid these problems, measurement-focused test design is the key – my recommendation is to start with the measurement targets in mind before building or buying a test. As one example of how to think about this topic, there’s a great site (http://atc21s.org) describing a partnership between a number of companies and educational institutions to define “21^st Century Skills” bridging education and the workplace, and what it takes to measure them using technology-enhanced tests and assessments.

Defining the measurement targets may also mean finding the gaps in existing tests and designing new technology-based measures to fill them. 

None of this precludes tests Looking Better – in fact, designing items around known and focused measurement targets is one of the best ways to get creative when designing interactive items with high visual appeal. But tests that effectively use technology for appearances only will be perpetually limited and will squander their potential. (Although it’s tempting to put appearances first and fill in the measurement later, avoid this direction.)

Technology can provide enormous advantages for Measuring Better – make absolutely sure it’s used this way first; once this is done and done well, Looking Better comes surprisingly naturally.   

Evan Sinar, Ph.D., is the Team Leader of the Assessment Technology Group for Development Dimensions International (DDI).

You can hear Evan at the SIOP Conference at 2 p.m. on Thursday, April 14, 2011 on "Advancing the Research Behind Technological Innovations in Assessment."

Flatter Than You Might Think

By Brad Schneider, Ph.D.

On a recent trip to Buenos Aires, I was reminded just how flat the world really is. Traveling at night, you board a flight in Chicago, close your eyes, and when you wake up in the morning, you're there.  Even the time zone stays constant.

There are, of course, a few notable differences—in climate, language, and culture—but if you adapt to those differences, you can bridge critical gaps, ensure a smooth transition, and even leverage the unique strengths that each region has to offer.

The same strategy holds for organizations looking to expand and strengthen operations across the globe.  As economic climates ebb and flow from one country to the next, having the flexibility to conduct business in a variety of countries is not just a strategy for accelerated growth—it's a hedged bet that ensures stability.

The challenge is that operating in new countries presents new issues—and exacerbates a few old ones.  How do you ensure your brand is consistent across the globe? How do you ensure the caliber of your offerings and services are of the highest quality, regardless of where you do business?

The answer, of course, is ensuring that your internal processes are consistent across the map. For example, if your business differentiates on customer service, then hire global talent based on their ability to provide superior service to your clients. Ensuring a consistent selection process around the world—using employment testing to assess candidates motivations, skills and capabilities, for example—is the means for getting there.

Although cultural differences must be taken into consideration (e.g., client expectations regarding the effectiveness of employee service behaviors differ by country), pre-employment tests that have been translated and culturally adapted for each region of the world will already take these crucial nuances into account - ensuring that new hires in The Windy City and The City of Fair Winds provide the same brand positive experience to your clients.

Brad Schneider, Ph.D., is a Director of Global Testing Solutions with Development Dimensions International (DDI).

Will Opportunity Knock?

By Brad Schneider, Ph.D.

Opportunity passes by your door a thousand times a day……but it never stops to knock
   – Ben Vareen

It was 30 years ago when I saw Ben Vareen on the Tonight Show with Johnny Carson. I can’t tell you if he danced well or how he sang – it’s not what mattered.  What resonated with me – and what I vividly remember three decades later – is what he said to Johnny as they sat on the couch.

Just to drive the point, since 1954 there have been thousands of guests on the show. Some, such as Joey Bishop, have sat on that couch 177 times. I don’t remember any of them.

Fast forward 30 years, and I’m sitting in a hotel lobby in Dallas. As I glance at the counter, the cover story in USA Today reads that high unemployment has resulted in perhaps the most talented candidate pool of all time.

In fact, according to the U.S. Department of Labor, since 1948, unemployment has peaked 11 times, each coinciding with a brief recession.  As we sit on the verge of economic recovery – an S&P 500 that’s above 1000 and 50% up from it’s low back in March, ever decreasing new unemployment claims, and a housing market with new signs of growth, it doesn’t take a rocket scientist – or a great singer and dancer, for that matter – to figure out that unemployment is going to drop. As it does, the cream of the current candidate pool will be hired right off the top….by either you or your competition.

Organizations have a brief window of opportunity to quickly and efficiently identify top performers from among the crowd—using tools like employment tests—and bring them on board before their competitors hire the best of the best.

The only question is whether you’re going to let this opportunity pass by your door.

Brad Schneider, Ph.D., is a Senior Consultant with Development Dimensions International (DDI).

The Supreme Court on Ricci: Boon or bust for employment testing?

By Doug Reynolds, Ph.D.

I’ve been asked by several colleagues now about the implications of the Ricci case for hiring practices in the U.S. This is the one where the City of New Haven chose to discard the results of a promotion test for firefighters because it produced an adverse impact on protected classes who failed the test. The city was sued by a group of firefighters who passed the test, and the Supreme Court found in favor of the firefighters, citing that the city had engaged in disparate treatment by using race as a factor in their decision to throw out results from a validated test.

Some labor attorneys believe this decision will dampen enthusiasm for employment testing. My feeling is that this interpretation is short-sighted. The legal logic is that employers will see the decision as a damned-if-you-do-and-damned-if-you-don’t conclusion, with risk coming from all directions, so you’d might as well give up and not use tests at all (and pick fire lieutenants by lottery, perhaps?). 

For those who design selection procedures, the case means something much different. Employers use valid assessment tools because they add tremendous value and fairness to the process of making decisions about people. Decades of scientific research support this, and organizations that use well-designed tools stand by them even when the lawyers start sniffing around. In fact, the use of assessments has been steadily increasing over the past decade.

A better interpretation of the Supreme Court’s decision is based on the fact that the validity of the test was supported. To be clear, the New Haven test was far from perfect. However, the test designers did take necessary steps to map the test to the body of knowledge required for promotion, and outside experts in testing and in firefighting looked at the test and judged it to be reasonable and relevant. In the end, the court did not take issue with the validity of the test—giving employers who use valid assessments the confidence that selection decisions made based on a well-designed process can be defended.

Had the New Haven test not been validated, most people would never have heard of the Ricci case; the test would have been thrown out with good reason and the case would not have made its way to the Supreme Court. Sophisticated employers will continue to benefit from the value that objective and valid assessments provide, regardless of the shenanigans in New Haven.

Doug Reynolds is vice president of assessment technology at Development Dimensions International (DDI).