New Standard for Getting In

With criticism growing that standardized tests and grades fail to convey the full picture of applicants, the Educational Testing Service is preparing a standardized way for graduate schools to consider students' non-cognitive strengths and weaknesses.

Under the "Personal Potential Index," which was developed at the request of the advisory board for the Graduate Record Exam, three or four professors or supervisors would answer a series of questions about candidates' non-cognitive skills in various areas, as well as a more general set of questions. Applicants would be rated on a scale of 1-5 on questions about their abilities in these six areas: knowledge and creativity, communication skills, team work, resilience, planning and organization, and ethics and integrity. Those filling out the forms would also be able to provide narrative answers on each of those areas, and the applicants' overall suitability for the programs to which they are applying.

For ETS, the development of the index is a significant acknowledgment of the dissatisfaction with traditional admissions measures with which the testing service has long been associated. In particular, ETS officials and some graduate school officials who have encouraged the index project hope that the project will create a standardized tool that gives credit to strengths that many students -- especially minority students -- may have that don't show up on the GRE.

ETS has not publicly announced the new product, but did a pilot of it last year and has had expressions of interest from a few dozen graduate schools to be part of a larger pilot, to start this year or next with 10 or so schools. While the project has been developed for graduate school admissions, ETS also has ambitions for it to be used in some form in professional school settings, and officials have talked about it with some business school leaders. At this time, there are no plans to introduce the index for undergraduate admissions.

The project is also a delicate one for ETS. The original name of the index was the "Standardized Letter of Recommendation," with the idea that it would replace letters of recommendation in their current form, but the name was dropped and the index is now being pitched as a supplement to other materials. And some critics of ETS wonder whether it should be trying to quantify qualities that relate to character and personality in ways that may be unique to individuals and not suited for numbers.

In an interview Thursday, David Payne, associate vice president of the higher education division at ETS, said that the index arose out of a request from the GRE board several years ago to look for ways to measure and evaluate non-cognitive abilities. The initial conclusion of the ETS experts was that all the existing measures operating on small scales were coachable.

GRE board members have been concerned, Payne said, about the gaps among different racial and ethnic groups on tests, and the impact that has on admissions. In addition, he said, all the studies ETS conducted suggested that letters of recommendation -- which in theory provide a way to learn about applicants beyond their test scores and grades -- end up having "no predictive value" of applicants' success in graduate school.

The new index would be filled out online by three or four people recommending a candidate for admission. They would first rank the candidate on a five-point scale (from below average to truly exceptional) on four questions for each of the six non-cognitive factors studied. For example, a team work question asks whether the candidate supports the efforts of others. A resilience question asks whether the candidate can overcome setbacks and challenges. Then there are a series of questions on the candidate's overall suitability for the program. For each of the six categories and the overall evaluation, the people filling out the index would receive a place to offer narrative answers or elaborations.

Colleges would receive average scores for each category and over all, but would also be able to see the complete answers of each person who evaluated the candidate.

Payne said that he suspected most candidates would seek out the same people to evaluate them on the index and to do letters of recommendation. To save time, he said he expected that colleges would eliminate the "check a box" portion of letters of recommendation that typically asks professors to evaluate candidates compared to others they have taught, and just leave the recommendation as a narrative.

Pricing for the new index has not been set, Payne said. The GRE costs $140 and is paid by applicants. Payne said that the pricing model might be similar in that it would be paid by applicants, or it might be paid by institutions, and wrapped into the application fee paid to the given graduate school. He said no price estimates were available under either scenario.

Long term, Payne said that graduate schools would probably use the index in different ways by tracking students' scores and their later performance in graduate school. One school (or a program within a school) might find that team work has the highest correlation with success, while another might find that the key is organization or some other category. Over time, that school might pay more attention to scores in that non-cognitive category than others.

Carol Lynch, the former graduate dean at the University of Colorado at Boulder and now a senior scholar at the Council of Graduate Schools, has followed the development of the index as a member of the GRE board. She characterized the effort as "an attempt to get away from just the reliance on standardized test scores and grades."

Lynch said that the original name of the index was a problem because it put too much emphasis on standardization and not enough on the idea that graduate schools want to know about students' non-cognitive abilities. But she acknowledged that one motivation was dissatisfaction with letters of recommendation. "Some of the things we're trying to get at here appear in some letters, but most do not; some letters are helpful [in making decisions] and some are not," she said.

Relying on letters doesn't work well, she said. "There are some busy faculty members who write the same letters for every student," Lynch said. "And it's amazing how many students are in the top 10 percent" of those taught by those writing letters. By asking very specific questions in the index, the new measure should yield better information, she said.

Lynch said it was hard to predict if the index might replace letters of recommendation. She said that she suspected some graduate schools would want to preserve them as freestanding letters on principle. But she said that if people use the free narrative portions of the index well, they may start to feel that they have provided the necessary information that way and that an additional letter is superfluous.

Ultimately, she predicted, the GRE board would not try to impose an answer. "I don't think anybody at ETS would say that we insist on having letters or not having letters," she said.

The first test of the Personal Potential Index came last year with applicants who are participants in Project 1000, a national program based at Arizona State University to help underrepresented minority students be admitted to graduate school. Michael Sullivan, director of the program, said that he and his colleagues have long been frustrated by the impact that GRE scores and grade point averages have on the success of minority applicants being admitted. "We understood that a fairly high percentage of our students had many qualities that were not well represented on traditional application materials," which he said are better at predicting the success of "traditional, mainstream students."

Project 1000 has a common graduate school application that about 90 graduate schools accept for program participants, and last year, the project included the Personal Potential Index with the application. "We have always wanted to find a better way to identify and represent the qualities that would predict our students' success," he said, so he was happy to test the model for ETS.

At the beginning of the test, applicants were using the index instead of letters of recommendation, but they found that the answers weren't complete enough "to satisfy the readers at the other end," so the index was used only as an addition, along with the regular application and letters. While Sullivan said that it is too early to tell scientifically whether the index helped applicants, he said that his impression is that it did.

"This was telling part of the story that is not well represented," Sullivan said. "My sense is that as this gets wider acceptance and faculty recommenders and readers understand it better, it will become richer and richer."

Robert Schaeffer, public education director of the National Center for Fair and Open Testing, said that the index represents a major shift for ETS. "I think what's happening is that the largest manufacturer of standardized tests has seen the writing on the wall, and that the future [of admissions] is not just in testing," Schaeffer said. "So ETS is establishing a beachhead in non-cognitive factors, which it has ignored in the past."

Schaeffer said that ETS has a range of motivations at play. ETS lost the contract to manage the Graduate Management Admission Test to ACT and Pearson, which started running the test for prospective M.B.A. students last year. A new test that can give ETS more of a presence in graduate education may be seen as valuable, he said.

As one of the more prominent critics of standardized testing, Schaeffer also questioned the approach ETS is using. He said he applauded the kinds of qualities ETS is trying to measure, but asked why the questions couldn't be more open ended with the idea that admissions committees could read what people write, rather than having a score to quickly consider. "It's not the standardization that's so bad," he said, saying that asking detailed, common questions is appropriate to compare students.

"But in typical fashion, they are trying to impose quantification to a degree that's not justified," he said, asking what a 3.8 would mean on resilience or organization. "This is overkill, trying to over-quantify everything to a decimal point," he said.

Still, Schaeffer said that it sounded like the ETS move was "a step forward."