Last spring, I posted about the challenge of calculating a return on investment (ROI) for individual colleges. The problem is with obtaining accurate data on the earnings of the graduates of a given college or university. I described there the problem with a ROI website created by the firm PayScale, which relies on self-reported data from individuals. There are a number of issues with PayScale’s methodology, but the biggest is that there is just no way to know how representative the respondents to their surveys are of the totality of the graduates for any single institution. There is also no way to judge the accuracy of the data reported.

Now there is another report out that uses the PayScale data to calculate ROI measures for community colleges. Last week, a study issued jointly by theNexus Research and Policy Center and the American Institutes for Research (AIR) took on the challenge of calculating the returns to associate’s degrees not just for individuals, but also for taxpayers in the form of the government subsidy provided to community colleges. There are a number of methodological problems with this study, which examined 579 community colleges around the country, but key among them is the study’s reliance on the PayScale data as the outcome measure for the ROI calculations. AIR is also the author of the College Measures website which, as I described in last spring’s post, shares the same flaw in using the PayScale data for some of its analyses.

The Nexus/AIR report is not just an academic exercise; President Obama’s recent proposal for holding colleges and universities accountable for their use of federal funds talks about the use of similar measures, as I wrote about in August. Proposals to tie eligibility for Title IV federal financial funds to how colleges’ graduates perform in labor markets is a serious threat to hold over the institutions.

After the Nexus/AIR report was issued, I was approached by The Chronicle of Higher Education to write an op-ed responding to the report. You can read the full essay, but here is a preview from the last two paragraphs:

To the authors’ credit, the report’s introduction notes the limitations of their study, stating that “the data needed to fully understand what, on average, a degree is worth to a graduate or what the tax benefit of that degree is to taxpayers, is not currently available.”

But then the report goes on to do precisely what the authors say cannot be done – so precisely that they calculate exact dollar amounts (graduates of Colorado Northwestern Community College, for example, are expected to have an average net financial return of $850,903 over their lifetimes). The precision of the numbers reported in this study, based on data of such unknown validity, is careless at best and grossly misleading at worst. And the notion that this study can be used to compare community colleges is preposterous.