Search News


Browse Archives

News

Aggressive Plan for State Data Systems

September 28, 2009

Share This Story

FREE Daily News Alerts

Advertisement

WASHINGTON -- It has become an article of faith among many federal and especially state policy makers that the United States cannot possibly improve the performance of its higher education system without a significantly better way of collecting data about the performance of individual students and colleges. Advocates for better data have abandoned the idea of a federal "unit records" system in the face of vocal opposition from private colleges, privacy advocates and many Republican lawmakers.

But the notion that an alternative could potentially emerge by developing state student data systems and stitching them together into a "national" network has taken hold with widespread backing (and in many cases funding) from the Obama administration, leading foundations, and policy groups -- so much so that many critics have largely resigned themselves to the prospect that they are a done deal.

Student aid legislation that the U.S. Senate could unveil as soon as this week would, like a comparable measure that the House of Representatives passed this month, give states significant financial incentives to develop such data systems. But the approach laid out in a draft section of the Senate legislation obtained by Inside Higher Ed would pursue the goal so broadly and aggressively -- by pushing states to collect an enormous array of information about individual students and link it to reams of other personal data -- that even some supporters of the need for data systems fear that it is politically impossible.

Among other things, the proposal would prod states that wish to tap into hundreds of millions of dollars in federal funds to collect data about students' scores on SAT and ACT exams and on tests, like the Collegiate Learning Assessment, that colleges might give their students to measure how much they've learned in college.

"In a non-political world, this approach makes perfect sense," said Travis Reindl, director of state policy and campaigns at CommunicationWorks, a Washington-based firm. "But given how much opposition there is to this, you could look at this as getting greedy" and giving opponents ammunition to fight it.

The legislative language in question was included in a document described as an early glimpse at the Senate's version of the Student Aid and Fiscal Responsibility Act, which is designed to carry out the Obama administration's proposal to end the guaranteed student loan program and use the savings to ratchet up spending on Pell Grants and a range of other education programs.

One major new aspect of the administration's plan is the proposed establishment of a fund to encourage colleges and states to increase the enrollment and graduation of students, particularly those from low-income backgrounds and others who are underrepresented in higher education. The bill the House passed this month would create a $3 billion College Access and Completion Challenge Fund, and state recipients of the grants can use the money for, among other purposes, expanding data systems designed to track students' academic performance.

The Senate document lays out an alternative version for a $4.25 billion fund for college access, persistence and completion. Several people familiar with the document said they believed that members of the Senate Committee on Health, Education, Labor and Pensions had already distanced themselves from some of the most controversial ideas in the document, including the proposed creation of State Higher Education Planning Councils that would be charged with developing statewide plans to improve college access and completion.

But one aspect of the Senate draft that is unlikely to change significantly, according to several people familiar with it, deals with "expanding statewide longitudinal data systems." Under the plan, states that seek to compete for funds through the program would have to assure that the data systems they create include all public postsecondary institutions in the state (private institutions could be included if they choose, but not required to participate) and that the systems collect information on each individual student's:

  • Secondary school record, including graduation date and scores on college entrance tests (like SAT, ACT, AP or International Baccalaureate exams).
  • Financial status.
  • Entry and exit from colleges.
  • Higher education progress and performance, including remedial course placement, credit completion, time to degree, receipt of degrees and certificates, and "performance on nationally validated assessments of postsecondary learning or value-added measures of postsecondary learning, if available."
  • Job placement, postsecondary earnings, and attainment of industry credentials.

States would also have to be able to disaggregate their data systems by "secondary school, postsecondary educational institution, race, ethnicity, gender, disability status, migrant status, English proficiency, and status as economically disadvantaged."

In addition, they would have to match the information in their postsecondary data systems not only with pre-kindergarten-12th grade data, but with a long list of other state systems, including those for "workforce development, unemployment insurance, child welfare, juvenile justice, military services information, and migrant student records."

Advocates for student-level data systems, like Peter Ewell, vice president at the National Center for Higher Education Management Systems and co-author of a recent paper on "The Ideal State Postsecondary Data System," applauded the general direction of the Senate legislation and said it would encourage states to move their data systems toward the best existing systems, like Florida's.

He said that unlike alternative proposals that Senate leaders had reportedly encouraged, which would have sought to prod states into folding postsecondary education data systems into existing ones for elementary and secondary school students, the current Senate plan "doesn't disturb the integrity of existing postsecondary data systesm by merging them" with K-12 systems.

Critics of student data systems like Barmak Nassirian, associate executive director of the American Association of Collegiate Registrars and Admissions Officers, say they have long since given up hope that the headlong rush toward state data systems can be stopped or even slowed, given that the Obama administration (with hundreds of millions of dollars it has dedicated to the effort in last winter's economic recovery legislation) has joined the Bush administration in championing the idea.

Even so, Nassirian said he was stunned and disturbed by the "grandiosity of the vision and the massive amount of money" contained in Senate proposal, which other critics described as virtually "cradle to grave." "You'd be hard-pressed to find another tracking system that goes into this level of detail about people, with such loosey goosey access rules," he said.

Nassirian said he is most troubled not by how the Senate legislation envisions using the postsecondary education data and the other personal information with which it would be linked, but how future Congresses might. "These will become the data marts of choice for activities that we would not endorse today," he said. "If you build it, the [Federal Bureau of Investigation] will come looking for criminals. The military will come looking to recruit. [Health and Human Services] will come looking for deadbeat dads.

"It's not a question of, who's against research, better alignment and better policy? It's that plugging the entire nation into a data tracking system creates, in the name of research, a consequential tracking system for Americans."

State policy experts like Reindl think that such fears are deeply exaggerated -- but also that the Senate proposal, as currently framed, would give unnecessary ammunition to critics by going much further than it needs to.

"It has taken us more than 15 years to really get the public agenda in this country focused on completing college," he said. "And now that that agenda has finally taken hold, they seem to be shooting the moon. We know how these data discussions go, especially when they're had in Washington. It's likely to create an unnecessary distraction."

See all postings »
Advertisement
Advertisement

Matching Jobs

Comments on Aggressive Plan for State Data Systems

  • Moves Towards A Totalitarian State
  • Posted by Bret on September 28, 2009 at 8:45am EDT
  • The United States is slowly moving towards totalitarianism. This means that all aspects of life are watched and in all essential ways controlled. The Republicans and Democrats both support this program. In terms of education, what this means is that universities will increasingly come under the rules and administrative tests required by K-12. Almost universally educators believe these systems to have already failed, and we are seeing the first students graduating in Texas that entered into elementary school under the model that is now national. The students in most Texas colleges -- even UT-Austin -- cannot actually read most medium or high level texts and actually comprehend the main points. Monitoring and testing does not actually lead to real literacy -- comprehension of ideas -- but rather a rote ability to pass certain tests and skills that can be memorized. I teach students who are going to be teachers and they generally don't care about the subject unless it is mandated by the state, both for their education and for their future students' education. 

    We will also begin passing students with lower socio-economic standing because it is required by the state. We already see this with many universities who draw from poorer areas. The undergraduate college I went to is barely staying alive; cutting students literally means the school might close. As social justice mandates become the basis for justifying education, the only way to pass as many students as politicians want will be to just pass them. Zimbabwe just prints money and it has high inflation. American degrees will also become inflated. In fact, one has to almost have a graduate degree or a degree in a hard science or engineering to actually get a reasonable job anymore because a BA is essentially worthless, even from top schools. Standardizing testing and giving more money to people who pass the most students will not lead to better education; on the contrary, it most likely will lead to a much worse standard.

    How can we get around these problems without merely being a crank? First of all, we need to recognize that testing is merely statistics, which are more often than not political indicators. By striving for uniformity, we do not allow the subjective decisions of teachers or the creativity of students. As such, we cannot actually test anything other than rote skills. We need independence, and that means there will be discrepancy. We cannot have highs without lows. But with broad choices, there will continue to be high levels of success across the country at universities that had independent missions. Governments have been able to enforce a broader educational standard throughout the 20th century through mass education, but it seems likely that the ability to just mass educate people has reached its maximum effect. Now we need to let different schools try to create new models of success and knowledge. We cannot also overlook the fact that career earnings are dependent upon the socio-economic standing of parents more than the educational quality of a school -- we learn from our parents and our neighbors. We cannot expect universities to do all of the job of educating. A failing school also means a failing community most of the time. 

    Of course these are general statements that many might criticize.  But without freedom, flexibility, and the ability to allow independence, we will become more and more rigid and totalitarian in the way we view success and knowledge. Difference is the key to life, and this also means that we cannot seek to equalize the world by making the mandate of all schools the same. To ensure more equality, we need to actually seek the opposite by embracing different models, methods, and types of educational systems, something that government is not likely to be able to achieve. 

  • The Feds in the dubious business of testing? Please!
  • Posted by Cliff Adelman , Senior Associate at Institute for Higher Education Policy on September 28, 2009 at 9:15am EDT
  • First, the draft of this bill is about as sloppy as Congressional staffers can get. Consider: you are "traditionally underrepresented" only if you are, simultaneously, a previously homeless, first generation parent working 25 hours a week or more and nonetheless low-income, who also never graduated from high school and "delayed college entry by 3 or more years" (though if you never graduated from high school, one has to ask, "delayed from WHAT?"). You are either all these things or you are or have been in custody of a child welfare agency (the "are" is interesting here for legal adults). That cuts out a lot of people. Go figure!

    Second, let's give them credit for including education histories of military personnel, which would inevitably include everything from pre-enlistment through active duty and into veterans' status---data that today are fragmented and often unintelligible for a population that is the largest low-hanging apple for degree completion out there, and one that foundations ignore because these folks are not considered "low income" (they have housing allowances, after all---Whoopee!) One could get a lot of this information by fixing DoD and VA records, and linking them, but that doesn’t seem to be in the cards.

    But then comes the disaster: the test factories are rubbing their hands in delight. We're going to stuff "value added tests" (there ain't no such creature) down the throats of everybody. “Value added” is the mindless, reflex response to the desire to know what college students actually learn. Give us a number! Even if nobody understands what the number means. As the UK’s Burgess Scoping Group noted in its report on the issue, there are a number of formulations of “value added, all of which have considerable measurement problems. Consider the 4 they reviewed:

    1. “Comparative learning gain.” That is, take students with the same qualifications at entry to higher education, no matter where they subsequently go to school, and develop a common set of metrics for assessing where they wind up. It’s not like SAT and GRE, rather performance in specific subjects, so SAT II and parallel GRE major field tests, where applicable. This is a very delicate psychometric matter, but take this line and we will spend more time testing than learning.

    2. “Comparative institutional effect.” As the Burgess committee so accurately points out, “not all ‘growth’ is necessarily attributable to the time spent under an institution’s aegis; natural maturation and engagement in a variety of extra-curricular activities will also contribute.” (p. 26) I would add that 60 percent of our undergraduates attend more than one institution, and parceling out the putative “effects” of all the institutions attended would be a statistical nightmare. The arrogance of assuming that an institution of higher education is responsible for everything one learns in the process of becoming an adult (from jobs, family, lovers, sports, entertainment, religious institutions, etc.), whether you are traditional-age or older, simply defies common sense.

    3. “Distance traveled.” Classic pre/post testing for individuals, and using the same test—which is a problem right away. The appropriateness of using the same test (or parallel forms of the same test) is open to question, not that that stops us from doing it all the time. While one might use different assessments “provided that the relationship between [those assessments] is calibrated to enable some interpretation of the gain to be made [italics mine],” the Committee points out that one would never be able to get beyond “some relatively generic aspects of curriculum.” (p. 26) More seriously, students learn a great deal in higher education that is not “captured in formal assessments,” so one winds up measuring only “part of the ‘distance traveled’.” (p. 26) No kidding! But if you want “No Child Left Behind” for higher ed, this is where it leads.

    4. “Wider benefits.” These are “collateral effects,” the value of social, spiritual, and economic experience in an institutional environment beyond the degree or measures of learning. Maybe NSSE captures some of this, but even so, these effects do not necessarily derive from institutional programming and micro-management of students’ lives. However much we all believe in these “wider benefits,” do we think that we can disentangle them? Well, there are no tests here, so Congress isn’t interested anyway.

    The Burgess Group added a very enlightening framework for our judgment of the potential uses of “value added,” namely by asking the identity of the audience for this information. They concluded —and we should follow—that if the principal audience for any information on the outcomes of higher education consists of prospective students, that the information provided should be transparent and useful enough to result in better choices. Test scores do not provide tht information.

    What would provide that information? If we developed our own version of the European Bologna Process’ degree qualification frameworks, public templates for student learning outcomes in the disciplines under the “Tuning” process (and we already have three state systems working on that across 6 disciplines), and benchmarking with discrete statements of what students can expect to learn and be able to do after majoring in accounting or nursing or history or chemical engineering (benchmarking is a distinct UK approach), we would be far along the path to public accountability as well. The only business the Feds have in this is to encourage these developments by higher education faculty, administrators, and state leaders, not to suffocate them with very dubious testing.

  • Real Issues
  • Posted by John Lee , Presdident at JBL Associates on September 28, 2009 at 1:30pm EDT
  • We have been managing the Achieving the Dream data systems for the last few years. The value of getting educators to think about student outcomes is an important goal and longitudinal data will help move the policy community in that direction. Having said that, we should not underestimate the problems. The Achieving the Dream database includes longitudinal records for students in 85 community colleges in several states. We have learned several things during this time.

    1. Longitudinal data is slow to develop and it is backward looking. This makes it difficult to use in a decision making process that is forward looking and immediate. Turning results into information that can be used for decisions takes careful thought.

    2. It is hard to differentiate random variation in measures from trends. We have measures that move up and down between years so it takes several years to see if there is a direction indicated in the results.

    3. Aggregating data about student outcomes across institutions often shows no change. For example, most colleges have developmental education programs, but some of them are done well and others are not. Determining what works in improving student outcomes will require much more specific data than can be collected in a standardized state-wide data system.

    4. It is important to have the analytic capability to work with the resulting data. Many states do not support the necessary research functions at the state level. Many colleges and universities do not have the institutional research staff needed to work with a complex student database. Collecting the data is only part of what needs to be done.

    In short, planners and policy makers should think carefully about what such a data system would be used for, how it will be supported and who will provide the support over the long-run to make sure that the full value of the effort will be realized.

  • Posted by Glen S. McGhee at FHEAP on September 28, 2009 at 1:45pm EDT
  • Peter Ewell overstates the virtues of Florida's "Cradle to Grave" tracking --
    http://www.fldoe.org/fetpip/
    All it does well is -- create more paper!
    But such paper is useless without highly legitimated experts (who are also highly paid, of course) wading hip-deep in the data-swamp, and alchemizing the muck into policy-gold.
    Regarding Adelman's references to the Burgess report (where is the link?) you can even scale further downwards -- to individual teachers' effects, to the classroom effect as well.
    You would think that the identification of "diploma mill" niches within the structures that Adelman cites is what the "quality assurance" function of higher ed accreditation is all about -- well, guess again.
    Here's an idea: while we are building ideal systems, why not explicitly link data-collection and results with accreditation processes?

  • Finding Burgess
  • Posted by Cliff Adelman , Senior Associate at Institute for Higher Education Policy on September 28, 2009 at 3:15pm EDT
  • For Glenn, to help with homework: Universities UK 2004. Measuring and Recording Student Achievement. Not hard to find a link through Google.

  • Posted by Daniel F. Chambliss , Eugene M. Tobin Distinguished Professor of Sociology at Hamilton College on September 28, 2009 at 4:45pm EDT
  • The “data-collection advocates” trumpet that to “drive real change” we need enormous national databases of student test scores, graduation rates and individual student performances over time. It’s an education researcher’s Holy Grail, no doubt. But I see no evidence, practically speaking, that even particular colleges with such tracking systems create more successful graduates than those without; or that the tremendous expense of creating such databases would be offset by producing smarter, better prepared, less debt-burdened graduates; or that American higher education is stronger now in the current age of “assessment and accountability” than it was 25 years ago.

     

    Are colleges better now than then?

     

    If our goal is to improve education, at any level, we could just concentrate on hiring and retaining the best possible teachers and making it easy for students to find them. After all, our goal should be great higher education, not great higher education research.

    Daniel F. Chambliss
    Eugene M. Tobin Distinguished Professor of Sociology
    Director, Mellon Foundation Project for Assessment of Liberal Arts Education at Hamilton
    Hamilton College
    315-859-4291 or dchambli@hamilton.edu

  • Bret is right
  • Posted by DFS on September 28, 2009 at 7:00pm EDT
  • And can we all just admit that the only time the Senate is "aggressive" about anything it must be looking towards something like the Carnivore program?

    We all must just eventually admit also that the logical progression of "assessment" is the final obfuscation of standards.

    It never works, no matter how many future government employees will be created by this fiasco.

  • Reagan had a good line for this....
  • Posted by Chris , Minimual Value Added Prof at HKU on September 29, 2009 at 5:15am EDT
  • I never thought I would agree with Reagan but this is one situation where when you here "I am from the federal government and am here to help" you should just run. Doesn't the federal government have better things to do? I am appalled to see the Dems continue the program initiated by Misspellings (former secretary of ed, Margaret) of bringing No Child Left Behind to college. Just go away and do something productive like reforming health care. Our system of higher education is clearly the best in the world. Leave it alone.