One of my favorite phrases when teaching educational policy is “the devil is in the details” (I once wrote an article that used that phrase as the title – it’s chapter 2 of this report). Politicians and policymakers will often issue grand policy proposals that address issues from an altitude of 35,000 feet, and until the details of the new policy are fleshed out it is difficult to determine what the impact will be at ground level.

This is very much the case with President Obama’s announcement last week of a new set of proposals to address the issue of the rising price of college and how Americans pay for it. As I described in my post last week, the president articulated a series of proposals that are very broad in scope and for the most part will require Congressional action in order to implement them. Thus, it is difficult at this point to determine with any degree of certainty what impact the proposals will have on colleges, universities, and students, but I will do my best to analyze them from the information the White House has provided.

The proposals fall into three broad categories:

• Provide incentives for both higher education institutions and students to link financial aid to performance;
• Encourage innovation on the part of colleges and universities to come up with new pathways toward less expensive degrees and provide better information to students and parents; and
• Make loan debt more manageable for those who borrowed to pay for college.

In this post I will tackle the first set of ideas that, in the president’s words, would “pay colleges and students for performance,” with a focus on institutional performance.

In understanding the president’s proposals, it is important to understand that federal financial aid today has, with some very minor exceptions, minimal performance standards for either students or institutions. College students receiving federal grants (such as Pell grants) or loans are required to maintain satisfactory academic progress as defined by their institution. This is typically a cumulative 2.0, or C, average. Some institutions require students to complete a certain percentage of classes in which they enroll, i.e., excessive course failures can affect academic progress (for an example of a satisfactory academic progress policy, you can see Michigan State’s here).

Institutions have a few more hoops they must jump through in order to maintain institutional eligibility for participation in the federal financial aid programs. They must adhere to all regulations related to the provision of Title IV aid, maintain student loan default rates below established standards, and adhere to other federal laws and regulations as a recipient of federal funds. But there is little in the regulations that require institutions to do much beyond these minimal standards.

President Obama has a two-step plan to hold institutions more accountable. The first step would be to create a new college rating system to be created by the Department of Education. He has not proposed the specific measures to be used in the new rating system, but the outline of his plan states that they “will be based upon such measures as … percentage of students receiving Pell grants; affordability, such as average tuition, scholarships and loan debt; and outcomes, such as graduation and transfer rates, graduate earnings, and advanced degrees of college graduates.” This ratings system would be put in place by 2015.

The second step would be to use these same, or similar, measures to determine institutional eligibility for participation in the Title IV federal financial aid programs. This would be the real teeth in the system; while a ratings system focuses on providing more information to consumers, removing the eligibility of an institution to award Pell grants and federal loans to its students could threaten the livelihood of some colleges and universities.

It is this linkage between outcome measures and financial aid eligibility that has attracted much of the attention of the higher education industry. The American Council on Education, the umbrella lobbying organization for public and not-for-profit universities, had a swift response:

“We will be vigilant in working to prevent tying the receipt of aid to metrics, which could have a profoundly negative impact on the very students and families the administration is trying to help.”

Coming up with higher education outcome measures that can be easily used to compare institutions has long been the Holy Grail of accountability measures. The U.S. Department of Education already collects a number of measures that, in theory, can be used to benchmark institutions. Graduation rates is one metric that at first glance would seem like a simple enough way of comparing institutions. But the research on graduation rates demonstrates that they are highly dependent not just on institutional performance, i.e., what policies and practices an institution has and how it applies them, but even more so on the academic and other characteristics of the students attending the institution (see for example the reviews of this literature conducted by Ernie Pascarella and Pat Terenzini in their books How College Affects Students, Volume 1 and Volume 2).

Here is one simple example to illustrate this. Let’s say you wanted to compare Harvard University with Valdosta State University in Georgia. The Department of Education’s College Navigator website provides easy access to graduation rate data. For the cohort of students who began at Harvard in the fall of 2006, the four-year graduation rate is 86 percent. At Valdosta State, the same cohort had a 15 percent graduation rate. Based on these figures, one may conclude that Harvard provides a better education than does Valdosta State.

But imagine now an experiment where you took the entire undergraduate student body from Harvard, 10,564 students last year, and swapped them with the 10,290 students at Valdosta State. I feel confident in stating that if you tracked the four-year graduation rates of these swapped students, you would find that Harvard’s rate would plummet while Valdosta State’s would skyrocket – all without changing a single policy or practice at either institution. So this leaves us with the puzzle of determining how much of an institution’s graduation rate – or other outcome measures that are highly correlated with student characteristics – is due to the institution, and how much is determined by the students themselves.

Another outcome measure the president has suggested using, as I noted above, is “graduate earnings.” As I described in earlier posts here and here, getting good data on the wages earned by the graduates of individual institutions is currently a challenging proposition. But even if you were able to get access to accurate earnings data, what exactly would this tell you?

Let me provide one example that illustrates the challenge of using the earnings of a college’s graduates as a measure for rating institutions or determining their eligibility for federal financial aid. Imagine a graduate of the College of Education here at Michigan State who has been trained as a secondary science teacher. If that student takes a job in a wealthy, suburban school district, one that has no problem attracting qualified, certified teachers, she will earn a certain salary as a first-year teacher. Now imagine her classmate, also trained as a secondary science teacher, and he takes a job in an urban or rural school district in Michigan, one that struggles to attract strong teachers, particularly in STEM fields. This first-year teacher may earn a wage that is perhaps 10 or 15 percent lower than his classmate in the suburban district. By the measure of wages alone, that first student will help my college rate higher. But I would argue that the placement of that second student in the high-need district is a much better outcome for the state of Michigan – and for Michigan State University.

This is just one example, but I believe it is a good one to demonstrate the limitation of using wage data to rate colleges. The president, and the websites I described in my earlier posts on measuring college earnings, want to use earnings data because they are relatively easy to measure, and because more and more, a college education is seen as primarily a form of vocational training. But if we are really concerned with what colleges and universities are doing to help society, and not just individuals, don’t we want broader measures of those contributions? And it is these broader contributions that are much harder to measure.

In my next post, I will turn to the president’s ideas for tying financial aid to student performance.