If You Can't Beat 'Em, Join 'Em

For decades, critics of standardized testing -- and especially of the SAT -- have said that these examinations fail to capture important qualities, resulting in admissions systems that favor certain groups over others, while failing to represent test takers' full identities. And generally, these critics have said, the qualities that the SAT is best at identifying are those that wealthy white students are more likely than others to possess.

On Saturday at the annual meeting of the National Association for College Admission Counseling, the College Board -- the creator and defender of the SAT -- said pretty much what critics have been saying all along. The board presented the most detailed results yet of new approaches to standardized tests that would measure non-cognitive qualities and could become what some have called the "SAT III."

Thus far, the board has found that there are specific non-cognitive qualities that relate to college success, and that these qualities can be measured. Further, board research suggests that if the admissions process included these qualities in addition to traditional measures, black and Latino enrollments would increase significantly while white and Asian enrollments would drop -- the latter significantly at the most competitive colleges.

"There are things that matter that are not measured by the SAT or ACT and we know that,” said Wayne Camara, vice president for research at the College Board.

Camara's comments came at a time of unprecedented scrutiny of the SAT, with a special NACAC panel calling on colleges to study more carefully whether they need admissions testing, and suggesting that a thorough study would determine that the tests aren't needed at many colleges. Camara's comments didn't endorse the idea of anyone dropping the SAT. He stressed that the College Board's research was designed to identify admissions procedures that could be used by a college on top of, not instead of, grades and test scores. And he made a strong pitch that efforts to measure non-cognitive qualities be done in nationally standardized systems -- like those the College Board sells.

But he also made no attempt to minimize the reality that the College Board is now endorsing the use of other measures -- a shift from years in which board officials have said that the combination of high school grades and the SAT is the best way to evaluate applicants.

The College Board and other groups have periodically attempted to develop testing systems that would measure non-cognitive skills. In 1999, a few colleges participated in a program in which some students were evaluated in part based on in-depth interviews and their ability to manipulate Legos in certain ways, among other tests. And more recently, institutions such as Tufts University and Oregon State University have reported success in attracting more creative and more diverse students using non-traditional approaches to admissions.

Camara seemed to minimize the significance of such efforts, referring to "one-off experiments," and saying that colleges needed "standardized" measures for these alternatives to have real value. "It's one thing to give some people Legos to build," he said, but what could work for 2.5 million college applicants?

The system that the College Board is testing adds two features in a standardized way to traditional admissions criteria: "biodata" and the "situational judgment inventory." Both are intended to measure 12 qualities that contribute in many cases to the success of some college students. These qualities include artistic and cultural appreciation, multicultural appreciation, leadership, interpersonal skills, career orientation, perseverance and integrity.

For each quality, there are questions for students both for the biodata and situational judgment parts of the program. For leadership, for example, the biodata questions might ask for the number of leadership positions held by the student in high school activities, or whether the student talks his or her friends into activities or is talked into them.

A situational question for leadership would ask what the student would do if, when assigned a group project, all the group members sat down and no one said anything. Choices would include:

• Look at them until someone eventually says something.
• Start the conversation yourself.
• Get to know everyone first and see what they are thinking about the project to make sure the goals are clear to everyone.
• Try to start working on the project by asking everyone's opinion about the nature of the project.
• Take the leadership role by assigning people to do thing or ask questions to get things rolling.

Scores are assigned based on the answers and students would then be identified as being stronger or weaker in leadership, or the other qualities. Camara stressed that the research was not just based on assumptions of what qualities are good, but based on scientific analyses of successful high school juniors and the qualities they had that helped them succeed. Further, studies have examined whether students identified with these approaches do in fact succeed -- as measured by such factors as returning to college, graduating and so forth. The work, largely based on Michigan State University, but now expanded to other colleges as well, is conducted through a Michigan State center called the Group for Research and Assessment of Student Potential, or GRASP.

Camara acknowledged that many colleges believe that they already consider factors such as leadership by judging in-person interviews or examining lists of extracurricular activities. But he said that these approaches are not scientific and that many people are less able to judge character in interviews than they believe. A standardized system, he said, is needed.

Using data from participating colleges, the College Board has determined the impact of changing admissions from a system based on equal weighting of high school grades and the SAT to one in which equal weighting is given to grades, the SAT, the biodata survey and the situational judgment score. Models were produced based on composites for colleges that admit different shares of students. The following shows the impact at a college that admits 15 percent of applicants -- a highly competitive college.

Impact on Enrollment by Race of Adding Factors to Admissions Reviews at College That Admits 15% of Applicants

At a college that is less competitive, the pattern of the shifts is the same, but less dramatic.

Impact on Enrollment by Race of Adding Factors to Admissions Reviews at College That Admits 50% of Applicants

Camara said that the College Board is currently working at expanding the research project, having found the initial results encouraging both for their validity at predicting success and in offering tools that would diversify the student body.

Pamela T. Horne, dean of admissions at Purdue University (and formerly at Michigan State), is among the college officials who have been involved with analyzing the findings from GRASP, and she said the results pointed to the need to broaden criteria considered for admissions. "This is mission-driven," she said, noting that colleges don't define their missions as "enroll students with high SAT scores," but they do prize leadership, artistic vision and various other qualities that might now be measured.

She stressed that the idea wasn't to look for students who have high scores in all 12 GRASP qualities, but to look for balance. "In college admissions, we don’t just care about what students bring to the table, but we are building a class of different talents," she said. The "well rounded class," she said, may be more important than "the well rounded individual." She also said that the data gathered through GRASP might point to ways to improve high schools, or to encourage college students to develop in areas where they are relatively weak.

While the research has been promising, Horne also said that there may be limitations. For instance, she said some of the qualities only have the positive impact on graduation when matched with certain other characteristics. She said that "strong career orientation" -- one of the qualities for which applicants might be rewarded -- is generally a positive indicator of graduation. But with low-income students, she said, a strong career orientation when not matched by appropriate academic preparation does not have the same impact on graduation rates.

She also said that much of the research to date has been at moderately selective institutions, and that more studies are needed with more competitive colleges.

Assuming that the College Board works out those details, there are other dangers down the road, Horne said. If these become "high stakes admissions instruments," she said, it's only a matter of time until the test-prep industry materializes and offers to coach students on how to answer. The whole system is based on students answering questions honestly, she said, and its value would be "quite diminished" if students figured out how to answer to get high scores in various categories.

While Horne feared the commercial influence of the test-prep industry, Robert Schaeffer, public education director for the National Center for Fair and Open Testing, said that the College Board's interest in this form of testing reflects the organization's own commercial interest. The College Board is "a net revenue maximizing corporation, which happens to operate under a non-profit charter," Schaeffer said. So, he added that he wasn't surprised in the board's interest "in possible new products" at a time that it is facing increased competition from the ACT, and more colleges ending SAT requirements.

While he acknowledged a possible oversimplification, Schaeffer compared the College Board's move into non-cognitive tests to the way McDonald's started selling chicken and fish in addition to burgers. "Sure, the new products cannibalize their sales of hamburgers," he said. "But the company's total revenues will be higher and their industry dominance will not be undermined."