Buying statistical gymnastics

The fact that a university president unquestionably buys a set of statistical gymnastics from an unofficial source—and without bothering to check it against official sources with statistical standards and public review—indicates just how much we have to teach our leaders the most basic lessons of common sense empiricism, let alone what is honest, and what isn’t.

NCHEMS claims that “out of every 100 9th graders only 18 will earn a bachelor’s or associate’s degree ten years later.” Sounds like a longitudinal study of 9th graders, right? Except there has never been a longitudinal study of 9th graders—something a university president’s assistant might have told him. So you go to NCHEMS’ technical notes (something nobody reads) and find they were aggregating state data—from the same year—for high school graduates, college entrants, first-to-second year college persistence, and degree awards. Have you ever met anybody who graduated from high school, entered college, got to the 2nd year of college, and graduated from college in the same year? Quite a feat, but not for people who bought this nonsense. The NCHEMS methodology would not be accepted by any official statistics agency in the world. It doesn’t pass the laugh test—and that’s being polite.

Two official federal sources, using completely different methodologies, separately produce estimates that are double the NCHEMS number. From the Census Bureau’s Current Population Survey you get 29% for bachelor’s degrees alone in the 25-29 age group (to which one can add, from the Census “some college, including associate’s degrees” attainment bracket, another 6 percentage points) for a total of 35%. At the same time, the Department of Education’s real longitudinal study of 8th graders—-based on transcript records, which don’t lie—-show 35% earning either a bachelor’s or associate’s degree by age 26. When two official sources coming at the issue from different directions, produce the same number, there has to be something right about it.

But for people determined to tell the worst and most sensational stories possible, the bad number fits the propaganda machine to a tee, and responsible statistics can be ignored. No, we are not doing as well as we could, but 35% puts us within the range of hope, whereas at 18% we throw up our hands, engage in orgies of self-flaggelation, and essentially freeze. That’s one of the principal problems with the propaganda of numbers.

Furthermore, the official accounts, particularly that of a real NCES longitudinal study that started with 8th graders and followed them for 12 years using real records, test scores at different points in time, interviews with parents and teachers as well as with the students themselves, provide a zoom lens and macro for informing us of where the problems lie and where the most profitable points of intervention are to be found. I’ve devoted a lot of my research efforts, using the real longitudinal studies, to identifying these points (in the Toolbox studies) and I’m hardly alone. And organizations all over the country—from Achieve to CalPASS to the College Access Network and others—are using these analyses to address the points of intervention in serious, creative, and level-headed ways.They know better than to accept blind throwaway numbers.

Clifford Adelman, Senior Associate at Institute for Higher Education Policy, at 10:10 am EDT on September 20, 2007

Lack of real access allows the elite to advantage their children

While it is certainly not new for members of the social, educational and political elite in the U.S. to distort science, measurement and technology for the political agenda of blaming “native intelligence” for population differences in educational, economic and other forms of social ascent, that doesn’t make it true. Einsteins are few and far between, and for the other 99.9999% or so in the non-Downs-Syndrome population, it’s all about educational, social and economic advantage. Most of the kids who rise to the top are nothing special (or worse), and they make it because the usual amount of work that everyone puts in from birth onwards occurred for them in the context of plenty of scaffolding and advantages, as opposed to no or poor scaffolding and disadvantages (I know because I’ve been to these elite institutions and I’ve seen it happen). Moreover, many Einsteins are lost to society because of the discrimination against and social/educational/economic exclusion of huge tracts of our population. Another respondent pointed to IQ, native intelligence and the “G” factor as the cause of educational disparities in the country. Outside of the small cadre of researchers devoted to this view despite the realities we are discovering each day in fields as diverse as education (see Claude Steele’s work), evolutionary biology (see Richard Lewontin’s work) and, yes, even genetics (see MJ Meany’s work on hypertension using rat models), this entire world view has been so thoroughly disproven it’s a testament to ideological power that it continues to be championed. Indeed, it does assuage elite consciousness while they sit smugly on their amassed advantages, blind to the many forms of scaffolding, support, and background conditions that lifted them along the way. The view loses a bit of its bite when we consider its history, however. In the 1920s, Lewis Termin gave the IQ test to women for the first time, and they scored...a standard deviation above the men. From their vantage point, this, of course, couldn’t possibly reflect “native intelligence” (though of course the reverse outcome would have been trumpeted loudly as such). What was the response? Deciding that the test must be flawed, the guardians of the “G” factor quietly re-normed the test until they found just the right items to eliminate the difference in scores. Of course, when the biases in the IQ tests lead minorities to score lower on them, this is not deemed a problem with the test but is merely a reflection of “native endowment". Do we smell hypocrisy?

If our access system isn’t broken, someone should tell those NYC parents who are donating millions to their selected kindergartens in order to shoe-horn their kids into them so that, 15 years later, their kids can end up at the ‘right’ college. And if the proponents of native intelligence really think the educational pipeline isn’t broken, they should walk the walk by putting their own kids in the lowest performing schools from nursery school through H.S. (if their kids make it that far).

rc, at 12:20 pm EDT on September 20, 2007

Thanks, Cliff, for energetically dispatching the flaws in statistical reasoning. The literary devices of the pipeline and epidemic also deserve critical scrutiny. Pipelines have beginnings ("P-K") and ends ("16″), hardly a suitable analogy for higher education given today’s realities. Adults will want and need access to college throughout their lifetimes, not just after high school. A major reason that online learning has skyrocketed in recent years is the inaccessibility (read: inflexibility) of traditional higher education delivery systems. Focusing on ‘leakages’ from K-12 and of student skill deficits skirts the more challenging task of changing instructional practices to make them relevant and effective for people who already have real lives, not just for 18-21 year olds who are waiting to start theirs. A good model is found in the Empire State College right there in the author’s back yard. Next, a disease model of drop-out (’epidemic’) suggests something is wrong with the student rather than the system. Perhaps it is time to re-read John Dewey and re-think where we have failed students, not where they have failed us.

Tom Flint, Director of Accreditation at Kaplan University, at 2:10 pm EDT on September 20, 2007

Kgotthardt writes “Since when do we measure potential with an IQ?” It is a good question and merits an answer, which is that IQ and IQ surrogate testing to measure potential has existed for many decades, and has long been in use by the U.S. military. Parts of the Armed Forces Qualification Test, now the Armed Forces Vocational Aptitude Battery (AFVAB), are strongly correlated with IQ despite fervent denials by the government that the AFVAB is a measure of intelligence. I consider it to be an IQ test and believe that Mensa would accept ASVAB results if such use were not strongly opposed by the government.

Kgotthardt also comments “If you think IQ is an excuse to dismiss the worth of any student, you are in the wrong business.” I don’t think that IQ, academic prowess, or any other factor defines student worth, or the worth of any other person. Worth is very much in the eye of the beholder. Anyone who has seen the love and attention given by some mothers to a down’s syndrome child understands that perceived worth has little to do with intelligence.

A real-world test of low IQ adults came when Robert S. McNamara, then Secretary of Defense, ordered the military to accept Category 4 personnel. The subsequent failure of those unfortunates had little to do with lack of specific factual knowledge and much to do with lack of intelligence. An example was their near-total inability to understand military technical manuals. Despite being written at the eighth grade level, comprehending the technical information required synthesis of multiple technical concepts, which proved a bridge too far for the Category 4’s.

Lest these comments be misinterpreted as a crude belief that IQ is everything in determining success, of course I do not. Luck, health, wealth, beauty, height, work ethic, social class, parentage, patronage, etc. play major roles in life success, or failure.

Marvin McConoughey, at 10:25 pm EDT on September 20, 2007

Student view

When was the last time the writer and those posting were in a public high school?

Dreary, boring, mentally stifling, dull, uncreative, and poorly organized and managed are some terms that come to mind.

Intrinsic motivation is one thing. Extrinsic motivation is something that can be managed.

Charters. Please.

Buzz, at 8:25 am EDT on September 21, 2007