July 6, 2007

New Standard for Getting In

With criticism growing that standardized tests and grades fail to convey the full picture of applicants, the Educational Testing Service is preparing a standardized way for graduate schools to consider students’ non-cognitive strengths and weaknesses.

Looking for a job?

See all 169 new postings

Browse all job listings:

Faculty: 2,694
Administrative: 2,046
Executive: 153
FEATURED EMPLOYERS

Under the “Personal Potential Index,” which was developed at the request of the advisory board for the Graduate Record Exam, three or four professors or supervisors would answer a series of questions about candidates’ non-cognitive skills in various areas, as well as a more general set of questions. Applicants would be rated on a scale of 1-5 on questions about their abilities in these six areas: knowledge and creativity, communication skills, team work, resilience, planning and organization, and ethics and integrity. Those filling out the forms would also be able to provide narrative answers on each of those areas, and the applicants’ overall suitability for the programs to which they are applying.

For ETS, the development of the index is a significant acknowledgment of the dissatisfaction with traditional admissions measures with which the testing service has long been associated. In particular, ETS officials and some graduate school officials who have encouraged the index project hope that the project will create a standardized tool that gives credit to strengths that many students — especially minority students — may have that don’t show up on the GRE.

ETS has not publicly announced the new product, but did a pilot of it last year and has had expressions of interest from a few dozen graduate schools to be part of a larger pilot, to start this year or next with 10 or so schools. While the project has been developed for graduate school admissions, ETS also has ambitions for it to be used in some form in professional school settings, and officials have talked about it with some business school leaders. At this time, there are no plans to introduce the index for undergraduate admissions.

The project is also a delicate one for ETS. The original name of the index was the “Standardized Letter of Recommendation,” with the idea that it would replace letters of recommendation in their current form, but the name was dropped and the index is now being pitched as a supplement to other materials. And some critics of ETS wonder whether it should be trying to quantify qualities that relate to character and personality in ways that may be unique to individuals and not suited for numbers.

In an interview Thursday, David Payne, associate vice president of the higher education division at ETS, said that the index arose out of a request from the GRE board several years ago to look for ways to measure and evaluate non-cognitive abilities. The initial conclusion of the ETS experts was that all the existing measures operating on small scales were coachable.

GRE board members have been concerned, Payne said, about the gaps among different racial and ethnic groups on tests, and the impact that has on admissions. In addition, he said, all the studies ETS conducted suggested that letters of recommendation — which in theory provide a way to learn about applicants beyond their test scores and grades — end up having “no predictive value” of applicants’ success in graduate school.

The new index would be filled out online by three or four people recommending a candidate for admission. They would first rank the candidate on a five-point scale (from below average to truly exceptional) on four questions for each of the six non-cognitive factors studied. For example, a team work question asks whether the candidate supports the efforts of others. A resilience question asks whether the candidate can overcome setbacks and challenges. Then there are a series of questions on the candidate’s overall suitability for the program. For each of the six categories and the overall evaluation, the people filling out the index would receive a place to offer narrative answers or elaborations.

Colleges would receive average scores for each category and over all, but would also be able to see the complete answers of each person who evaluated the candidate.

Payne said that he suspected most candidates would seek out the same people to evaluate them on the index and to do letters of recommendation. To save time, he said he expected that colleges would eliminate the “check a box” portion of letters of recommendation that typically asks professors to evaluate candidates compared to others they have taught, and just leave the recommendation as a narrative.

Pricing for the new index has not been set, Payne said. The GRE costs $140 and is paid by applicants. Payne said that the pricing model might be similar in that it would be paid by applicants, or it might be paid by institutions, and wrapped into the application fee paid to the given graduate school. He said no price estimates were available under either scenario.

Long term, Payne said that graduate schools would probably use the index in different ways by tracking students’ scores and their later performance in graduate school. One school (or a program within a school) might find that team work has the highest correlation with success, while another might find that the key is organization or some other category. Over time, that school might pay more attention to scores in that non-cognitive category than others.

Carol Lynch, the former graduate dean at the University of Colorado at Boulder and now a senior scholar at the Council of Graduate Schools, has followed the development of the index as a member of the GRE board. She characterized the effort as “an attempt to get away from just the reliance on standardized test scores and grades.”

Lynch said that the original name of the index was a problem because it put too much emphasis on standardization and not enough on the idea that graduate schools want to know about students’ non-cognitive abilities. But she acknowledged that one motivation was dissatisfaction with letters of recommendation. “Some of the things we’re trying to get at here appear in some letters, but most do not; some letters are helpful [in making decisions] and some are not,” she said.

Relying on letters doesn’t work well, she said. “There are some busy faculty members who write the same letters for every student,” Lynch said. “And it’s amazing how many students are in the top 10 percent” of those taught by those writing letters. By asking very specific questions in the index, the new measure should yield better information, she said.

Lynch said it was hard to predict if the index might replace letters of recommendation. She said that she suspected some graduate schools would want to preserve them as freestanding letters on principle. But she said that if people use the free narrative portions of the index well, they may start to feel that they have provided the necessary information that way and that an additional letter is superfluous.

Ultimately, she predicted, the GRE board would not try to impose an answer. “I don’t think anybody at ETS would say that we insist on having letters or not having letters,” she said.

The first test of the Personal Potential Index came last year with applicants who are participants in Project 1000, a national program based at Arizona State University to help underrepresented minority students be admitted to graduate school. Michael Sullivan, director of the program, said that he and his colleagues have long been frustrated by the impact that GRE scores and grade point averages have on the success of minority applicants being admitted. “We understood that a fairly high percentage of our students had many qualities that were not well represented on traditional application materials,” which he said are better at predicting the success of “traditional, mainstream students.”

Project 1000 has a common graduate school application that about 90 graduate schools accept for program participants, and last year, the project included the Personal Potential Index with the application. “We have always wanted to find a better way to identify and represent the qualities that would predict our students’ success,” he said, so he was happy to test the model for ETS.

At the beginning of the test, applicants were using the index instead of letters of recommendation, but they found that the answers weren’t complete enough “to satisfy the readers at the other end,” so the index was used only as an addition, along with the regular application and letters. While Sullivan said that it is too early to tell scientifically whether the index helped applicants, he said that his impression is that it did.

“This was telling part of the story that is not well represented,” Sullivan said. “My sense is that as this gets wider acceptance and faculty recommenders and readers understand it better, it will become richer and richer.”

Robert Schaeffer, public education director of the National Center for Fair and Open Testing, said that the index represents a major shift for ETS. “I think what’s happening is that the largest manufacturer of standardized tests has seen the writing on the wall, and that the future [of admissions] is not just in testing,” Schaeffer said. “So ETS is establishing a beachhead in non-cognitive factors, which it has ignored in the past.”

Schaeffer said that ETS has a range of motivations at play. ETS lost the contract to manage the Graduate Management Admission Test to ACT and Pearson, which started running the test for prospective M.B.A. students last year. A new test that can give ETS more of a presence in graduate education may be seen as valuable, he said.

As one of the more prominent critics of standardized testing, Schaeffer also questioned the approach ETS is using. He said he applauded the kinds of qualities ETS is trying to measure, but asked why the questions couldn’t be more open ended with the idea that admissions committees could read what people write, rather than having a score to quickly consider. “It’s not the standardization that’s so bad,” he said, saying that asking detailed, common questions is appropriate to compare students.

“But in typical fashion, they are trying to impose quantification to a degree that’s not justified,” he said, asking what a 3.8 would mean on resilience or organization. “This is overkill, trying to over-quantify everything to a decimal point,” he said.

Still, Schaeffer said that it sounded like the ETS move was “a step forward.”

— Scott Jaschik

Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.

Comments

More money for ETS

Does this sound like a scam to anyone else? My impression from talking to math professors is that for graduate admissions they rely mostly on letters of rec, grades, and subject test GRE scores. The general test already goes practically unused, and an index quantifying your capacity for teamwork doesn’t sound like it’ll do much better. Doesn’t applying to grad school cost enough as it is?

Math Grad, at 5:05 am EDT on July 6, 2007

ETS’s intentions

I don’t think ETS has any bad intentions here.. First, they’re a non-profit organization.. Second, the article said graduate schools may have to pay for it, and no one is forcing graduate schools.. The school can decide not to use these tests.. Third, this is not out of step with a major move in testing for employment that has been going on for years, where non-cognitive (e.g. personality, assessment centers, etc.) measures are used in conjunction with cognitive measures to curb the adverse impact against racial minorities that are almost universally associated with cognitive measures.. As a fellow conspiracy theorist, I understand people’s concerns about ETS’s intentions but I honestly think this not just a “rip off” but a concerted effort to improve their product line and to provide schools with the best possible admissions criteria.. PS — I noticed you’re a math major and that you said your professors don’t use the GRE general.. Well, that’s smart.. See, the only part of the GRE that would really be that relevant would be the Quant section.. BUT everyone applying for math programs should be scoring in the 700’s, right? SO there’s no variance, and therefore no relationship to future performance.. That’s why it makes sense to use the SUBJECT GRE for math grad students (Which, by the way, is also made by ETS)

I/O Psychologist, Bowling Green State, at 7:00 am EDT on July 6, 2007

Cultural bias?

I’m wondering if these new questions won’t just give further advantage to traditional students and those from privileged backgrounds. The information they request will require academic recommenders who (a) know the person fairly deeply (a group that’s easier to populate for students still enrolled in small colleges, and much harder for someone who, say, graduated from a big, public college 15 years ago), and (b) understand what’s expected and learn how to write compelling answers (again, mostly profs who do this a lot and know the applicant well-enough to make the recommendation effective).

It would seem that both qualities are likely to be correlated with the age and life-situation of the applicant. In such circumstances, the very effectiveness of this measure may introduce bias, if it leads graduate schools to weigh more heavily data that systematically reinforce the advantages held by already-privileged and traditional applicants. In effect, today’s “bad” recommendation letters may be better than good ones, if their limitations cause grad schools to pay less attention to the biases they reinforce.

It could be that the benefits of the data outweigh the costs...to the grad school. It might even improve outcomes; after all, traditional students have some advantages that might help them to succeed at a greater rate. But this seems like a scheme to move the costs of an inherently imperfect system from the schools to the applicants—and to improve efficiency at the expense of equality of opportunity.

CJ, at 8:05 am EDT on July 6, 2007

Alternative to another test

So ETS is going to charge students (if the school pays for it, they’ll just raise fees so the students will pay for it anyway) to basically have a graded interview. I can’t wait to see the companion study guides to go with those for the GRE.

I know! Ask students to include examples of their own original work or group projects accompanied by a brief analysis from a prof who has taught the student. We can call it a portfolio and a recommendation letter.

Instead of setting up another hurdle for those who want to pursue an advanced degree, how about actually doing something to encourage an increased pursuit of the STEM fields?

Unfortunately, we’ll probably only get another test for this test crazed society. At least we can comfort ourselves with the fact that grad students have “creative personalities” because they passed a test that said they do.

Ben, MA English, at 8:50 am EDT on July 6, 2007

Non-cognitive?

I am reading this differently, as a way for ETS to get around the “coachable” achievement gap that separates minorities. This problem (social capital of middle and upper classes, as opposed to lower classes) has long been a major issue in sociology (which is probably why psychologists continue to ignore it).

But labeling these six new indices “non-cognitive” is not right either — they are still “cognitive” that is, mind-based. See for example the area of study called social cognition, etc.

Glen McGhee, FHEAP, at 9:20 am EDT on July 6, 2007

One major challenge

The major challenge for this approach will be getting honest responses from the “recommenders". My institution faces a similar situation in teacher education — student teachers are evaluated by their supervisor and their cooperating teacher on a standardized form. These forms are then used when the students apply for teaching jobs. However, knowing that the student will use them in applying for jobs, many cooperating teachers and supervisors intentionally score the student much higher, often giving the student all “above average” marks when the actual performance is much lower.

So I like the idea...but it will be challenging to get honest responses, particularly when faculty members know that their responses might keep their student out of graduate school.

T-bone, at 9:30 am EDT on July 6, 2007

RIGHT ON Ben, MA English!

This is some bull sh$$

hum..., GRE-prep at no sé university, at 10:40 am EDT on July 6, 2007

This “assessment” would be an unfair joke. There’s nothing “standardized” about students picking several biased professors to rate them. Besides, how many professors know their students this well? Students tend to have very little one-on-one contact with their profs. This assessment should be banned.

Marky Marc, at 10:55 am EDT on July 6, 2007

What a ripoff — and where’s my cut??

So let me get this straight.... A student has two ways to get a professor’s recommendation to grad school: (1) Professor writes a recommendation letter for a student, AND completes the detailed form that most grad schools provide these days. This form assesses student on a dozen skills and accurately compares students to their peers using a percentile system. (2) Student pays ETS to do it for them. ETS solicits professors using a simpler five-point scale, and then charges student for the information.

I think you can tell where I’m going with this. I am not convinced that this will improve the information a grad school receives, but I am certainly convinced that ETS will make money on it. It is disingenuous of ETS not to mention that such a form is already in wide use by grad schools — unless, of course, they see such a form as competition.

A word to I/O Psychologist: just because the organization is a nonprofit does not mean it isn’t out to make a profit! ETS, in fact, depends substantially on earned income. It’s only what’s done with the profit that separates NPOs from FPOs.

A final word to Carol Lynch: I don’t buy the rationalization that this tool is needed because all professors rank their students in the top 10 percent of their class. I haven’t written a letter yet where a student deserved a lower ranking. These are the students who go on to grad school, after all.

Hoosier Prof, at 11:05 am EDT on July 6, 2007

“Applicants would be rated on a scale of 1-5 on questions about their abilities in these six areas: knowledge and creativity, communication skills, team work, resilience, planning and organization, and ethics and integrity.”

How are these skills not cognitive (i.e., based on conscious intellectual actitivty)?

JBM, at 11:20 am EDT on July 6, 2007

This adds nothing

What is wrong with traditional recommendations? Most recommendation forms already ask for some quantification of the applicant in the areas mentioned. Furthermore, a good recommendation (one worth writing) will go into detail, and grad school committees should be willing to read those details about the applicant. We make time for things that matter to us, so it is not too much to ask a mentor for a lengthy letter, nor is it too much to ask an admissions committee to read it. This standardized approach defeats the purpose it sets out to achieve, a purpose already achieved by traditional recommendations.

Angelo, Prof, at 12:10 pm EDT on July 6, 2007

Predictive validity

There is scant attention paid to predictive validity in either the article or the comments thus far. Does anyone care if the new system predicts which candidates will succeed? Or is this just a ruse to help change the racial make up of the programs at the expense (not financial but professional and personal)of the students?

I don’t imagine there will be much more variability in faculty recommendations with this system. Like Lake Woebegone on steroids, all students will be above the second standard deviation.

By the way, I remember when the prevailing wisdom was that the GRE general test was favored over the subject test, reasoning that the subject was very “coachable” but that the general score reflected a more enduring basic intelligence. I believe the general test was more predictive of success than the subject test.

Barry Tigay, at 12:10 pm EDT on July 6, 2007

Creating new hurdles?

I look forward to learning more about this initiative, but it seems duplicative to me. Admission should be a holistic process already taking into consideration more than just test scores and grades. Applicants are likely to ask the same people for references for the GRE and for their applications. Many of the same (or similar) rating categories will appear on both ETS forms and already appear on institutional reference forms. This duplication may lead to frustration on the part of reference providers who “already did this” or candidates who are reluctant to ask the same providers for yet another form.

I think it is a worthy goal, but perhaps it would be better to develop a reference protocol that can be used by schools (with customization) as appropriate for the local admission process. Admission processes should be as streamlined (and dare I say “customer friendly") as possible. Pulling what should be a school specific assessment item into a standardized test setting would not in my opinion be efficient. Helping schools that need to develop more thorough reference forms would be a better way to achieve holistic admission decisions relevant to the individual university, school or department.

Dan Chatham, Director of Admissions at School of International Relations and Pacific Studies — UCSD, at 12:50 pm EDT on July 6, 2007

Stop ETS’s money grubbing

This is sadly typical of ETS, where the commercial imperative of expanding product lines and increasing revenue trumps any meaningful contribution to higher education.

The attempt to devise a standardized letter of recommendation acknowledges just how awful the GRE is at predicting future graduate school performance. ETS’s “solution” to this problem, however, is really just another attempt to extract money out of potential graduate students.

Creating a standardized format for graduate school letters of recommendation, at least within broad categories of disciplines, is not a bad idea in itself. Something along the lines of the common application only specifically for letters of recommendation might be a possibility. The problem is letting ETS manage the project and then charging students exorbitantly.

Justin, Grad Student, at 1:20 pm EDT on July 6, 2007

ets is left out

Sounds to me as Universities find that their techniques for evaluating students may be better than the supposed standardized test, ETS is feeling a bit left out. In the coporate structure of the non-profit it seems that if testing goes further in the hole ETS will need a new way to be involved!

Universities wake up... if you can educate the student your great minds should be able to pick them!

Tina Osborne, at 1:35 pm EDT on July 6, 2007

Cognitive is not the problem

What is being ignored here is the well-supported data that show that scores on all of ETS’ standardized tests vary by race, ethnicity, and economic status. Those who do best on the exams are the students who are most like those who write the exams.So the problem isn’t whether “non-cognitive” abilities are tested by the exam, but why any institution would use the scores at all, given that almost every study shows that (a) they maintain the racist and classist status quo and (b) the exam scores have no relationship with success in graduate programs.

Michael, University of Massachusetts, at 3:45 pm EDT on July 6, 2007

More hoops to jump through to file an application

This sounds awfully much like a letter of recommendation. If this is adopted wholesale, will colleges and universities be willing to drop their letters of rec? Isn’t it enough that we have to ask letter writers to fill out this inventory? Would they also then be asked to produce an additional, narrative evaluation? For each institution we apply to?

Erik, Starting PhD program next year, at 6:55 pm EDT on July 6, 2007

“tests in 6 key areas: knowledge and creativity, communication skills, team work, resilience, planning and organization, and ethics and integrity.”

It’s great to see a test for creativity finally being added to the roster, but one thing I think belongs in this list that makes all the difference in the world.. something that can make up for a lesser ability in any of these areas is the persistence of an individual.

Mike, at 5:30 am EDT on July 7, 2007

This is going to fail in a funny way

This is silly, since people recommending students will be expected to give all their students top marks in all categories. In fact, if I was a student, I would expect nothing less. Students will come to expect that professors will simply give them all “above average” marks. The “cool” professors will complete the surveys in front of the students – just like good professors reveal the “confidential” letters of recommendation they give students.

Just like there is nothing “standardized” about a letter of recommendation, there is nothing standardized about this.

I/O Psychologist, Just because an institution is “non profit” doesn’t mean that it can’t have bad motives. Heck, universities, trade associates, and even Mastercard are non-profit.

Larry, at 10:40 am EDT on July 7, 2007

No thank you

Unfortunately, this is yet another form of standardization that further attempts to flatten human uniqueness. If the corporate world was using this to hire it’s higher level positions, academics would scoff. But for assessing potential students? Make our job easier?? Sure!

The admission process should not be further bureaucratized so that simple boxes can be checked. I sincerely hope colleges and universities would value the dignity of individuals enought to find a more humane way to get to know candidates and assess their fit for particular institutions.

praxis, at 10:40 am EDT on July 7, 2007

May I please thank you. Your report, in earnest manner, that there is a proposed non-standardized “standardized test” to be used by the most vocal opponents of “standardized tests” and produced by the biggest vendor of “standardized tests” to be used by those opponents of “standardized tests” who are also the biggest users of “standardized tests” to substitute for “standardized tests". Jabberwocky good! This provides me with a great idea for a book...or two. (Although, as a mathematician, I may have to write under a pseudonym to maintain my reputation.)

Charles Lutwidge Dodgson, at 5:35 am EDT on July 8, 2007

Dodgson, you magnificent bastard! Now that’s the way to write a post.

JBM, at 2:20 pm EDT on July 8, 2007

I’m sick of standardized tests......

Fortunately, SAT/ACT scores have become optional among various colleges such as Harvard, Stanford, and Chatham University in Pittsburgh. This allows students of equal educational levels and who have a greater chance of succeeding to further their interests, especially those whose skills don’t show on standardized tests. Students can express their unique characteristics in a portfolio or similar work profile.

Without the SAT/ACT optional policy, potential candidates would not have been admitted in the last year. Conversely, the cons of this new graduate admission policy may out way the pros, if there are any. I am also skeptical about the costs associated with this policy. Undergraduate school is already unbelievably expensive. We don’t need getting in to graduate school to become even more expensive.

In addition, the various students who are not hard-working, dedicated, or competent can be scored much higher on these tests than they deserve to be. There are “recommenders” who like their students, date their former students, favor certain ones, etc., therefore they can give students much higher of a recommendation than was earned. This can also effect students who are disliked by a “recommender.”

We are a test crazed society and it is becoming more and more sickening. A lot of incompetence surrounds us each day in society in the professional schools/colleges and work-place and I believe it is because schools focus primarily on tests, which doesn’t always correctly reveal one’s ability. Anyone can study, remember, and take a test.

This whole new measurement of graduate qualifications may not have too bright a future. I’m sure a lot of students who are not rated positively will become angry if they aren’t admitted into their desired school. This can create racial tension if it is a minority. Some professors may not be good at discerning their students’ ability or rating them. I don’t know, I’m on the fence with this one.

Stay tuned............

I know I will because I have to apply to graduate school in another year!!! The clinical psychology field is getting more and more competitive. Considering this, the new standardized way of scoring potential condidates might not be so bad.

MiMi, at 8:25 pm EDT on July 8, 2007

Jobs Related to New Standard for Getting In

or search for jobs directly.

Transfer Credit & Articulation Specialist (Transfer & Articulation Specialist)
University of Idaho

Transfer Credit & Articulation Specialist (Transfer & Articulation Specialist) Office of the Registrar Open for Recruitment: ... see job

Admissions Coordinator
New York University

Help change the direction of MBA applicants’ lives by joining a collaborative team of elite professionals in the MBA ... see job

Assistant Director of Admissions
DeVry College of New York

EDUCATIONAL ADVISOR A GREAT JOB IN A GROWING CAMPUS As one of the largest degree-granting higher education systems in North ... see job

Recruiting & Admissions Marketing Coordinator
Monterey Institute of International Studies

Department: Admissions and Recruiting Locations: Monterey, CA Type: Full-time Non-Exempt Ref. No.: 1997 Job Description: ... see job

Transfer Specialist/Developer – Title V
University of La Verne

Transfer Specialist/Developer – Title V Position #2938 The University of La Verne has an opening for a part-time (20 hours ... see job

Admissions and Recruiting Manager
University of Denver

Job Summary: This position is primarily responsible for managing the recruiting, marketing, and ... see job

Registrar/Director of Academic Records
University of Houston — Clear Lake

Registrar/Director of Academic Records position available at the University of Houston-Clear Lake. We have a beautiful campus ... see job

International Application Specialist
University of Colorado at Denver and Health Sciences Center-Anschutz Medical Campus in Aurora

Posting Description: The Office of International Affairs has two positions open as International Application ... see job

Vice President for Enrollment Management and Student Success
Fashion Institute of Technology

FIT—Where Creativity Gets Down to Business! see job

Assistant Director of Executive MBA Programs
Loyola College in Maryland

1. Student Recruitment Assist Director with EMBA and MBA Fellows prospective student lead generation, admission recruitment ... see job