The U.S. Department of Education has finally released the outline of its college ratings plan, 16 months after the idea was first proposed by President Barack Obama. When he first suggested the plan, he said, “Bottom line is this: We’ve got a crisis in terms of college affordability and student debt. … It is time to stop subsidizing schools that are not producing good results and reward schools that deliver for American students and our future.”

The plan – and it truly is just a plan, not a fully-formed program – focuses on many of the measures that department officials have been talking about over the ensuing period. This includes such metrics as average net price (after taking into account financial aid), the net price paid by students from families of different income levels, proportion of students receiving Pell Grants (the primary federal need-based grant program, an indicator of how many low- and moderate-income students a college enrolls), proportion of first-generation college students, graduation rates, and loan repayment rates.

None of these are metrics are revolutionary; the Department of Education already collects and publishes these measures for colleges, albeit not in the most consumer-friendly fashion. Some third-party websites, such as those from the College Board and the Institute for College Access and Success, present these data in a manner that makes it easier for students and parents to understand.

There is one measure the department has proposed to publish that does represent a new venture: the earnings of graduates of each college. The government does not collect these data for each school now, and any attempts to do so are fraught with challenges and traps that potentially render the effort dead on arrival.

There are numerous problems with using earnings of graduates as a measure of institutional effectiveness, beyond the most obvious one, which is how the department will collect these data. First, as I and others have written in the past, the greatest variation in earnings of college graduates is not across colleges, but within colleges – across the majors in each college.

The important measure, and what is most valuable to consumers making decisions about what college to attend, is not to compare the average earnings of graduates of Michigan State University versus those of Ohio State University. The more relevant and valuable information is to compare the average earnings of political science majors to those of chemical engineering majors, and it is unclear whether the government’s ratings plan will do this.

Using earnings as the primary outcome measure of college also diminishes what we expect for those graduating from college. While preparing students for the world of work is important, two or four years of college also provides them with skills for life that are not well measured through wages alone. This includes such things as civic engagement, volunteerism, improved health status, and lower levels of dependency on public services. There are also many unintended consequences of using earnings data to rate colleges, which I have detailed in earlier blog posts here and here.

As noted above, the federal government has published data about colleges for some time. What is new in this plan is the idea of using the data it collects to provide a rating for colleges. We do not know how the government will do this, as the outline of the plan released so far does not give details about the ratings. But the mere fact that the Department of Education proposes to put its stamp of approval or disapproval on individual institutions has raised the hackles of colleges, their lobbying organizations, and members of Congress.

Even critics of the ratings plan acknowledge that the federal government mandates the disclosure of information across many industries. The Food and Drug Administration (FDA) requires that most commercially sold foods have nutritional labels, and that drugs are packaged with information about proper use and side effects. The Environmental Protection Agency (EPA) requires that automobile manufacturers publish the federally tested fuel economy of cars they sell.

But critics point out that in none of these other domains does the federal government provide a rating for one commercial good over another; it leaves this to the private sector. The EPA does not rate one car better than another, but ConsumerReports uses the fuel economy data when it does rate them. The FDA does not compare Ritz Crackers to Wheat Thins, but many consumer websites weigh in on which one is better.

It is important to acknowledge that investing in an undergraduate degree program – which is what the ratings plan will be focused on – is different from the purchase of most other goods or services. First, for many people, a college degree will be the most expensive thing they purchase in their lifetime outside of a home, so the stakes are very high. Second, most people only purchase an undergraduate degree once, so they do not have the ability to try different ones – as they may with crackers or pain relievers – and settle on which one is right for them. Third, the federal government invests over $150 billion per year in federal loans and grants, so it has a stake in ensuring the money is used wisely. So given the magnitude of the decision, one could argue that perhaps the government has a role in putting its stamp of approval on some colleges over others.

Members of Congress on both sides of the aisle have been raising concerns about the department’s ratings plan ever since the president first proposed it. The recent budget agreement that passed Congress and was signed into law by the president did not include any funding for the Department of Education to implement a ratings plan. And many prominent members–including incoming chair of the Senate Committee on Health, Education, Labor and Pensions, Senator Lamar Alexander of Tennessee–have been very vocal in objecting to any such plan. Republicans in Congress in particular are opposed to the Obama administration’s plan to eventually tie the ratings to institutions’ eligibility for federal student aid funds.

The primary problem with determining whether the Department of Education will be able to rate colleges in any reliable and defensible fashion is that the information they have provided is just too sparse to understand exactly what the plan will look like. To be fair to the department, we cannot yet give it two stars out of five, or a grade of C. The true grade it deserves is I – for incomplete.