Last month I described my testimony at a U.S. House of Representatives hearing on getting better information about college. One of the topics mentioned by a few of the House committee members was the issue of the return on the investment in college as measured by what students are earning today. There has been much interest in getting better data about what graduates of different colleges earn when they enter the labor markets, so that prospective students can get an idea of what the return on their investment in postsecondary education will be.

While measuring the return on investment (ROI) of attending different colleges sounds like a good idea, in reality it is a complex and challenging task. To calculate an ROI, one has to have accurate data on both the value of the investment as well as the value of the return, or the earnings that the investor receives from her investment. While we have lots of data on the cost of attending different colleges, including some data on net prices (as I described in my testimony), we have very little accurate, reliable, and comprehensive data on what the graduates of different colleges earn.

There have been some noble attempts at compiling data on the earnings of college graduates. One of the most visible is the work of a website called PayScale, which compiles a “College Education ROI Rankings,” with salary data from graduates of over 1,000 higher education institutions in the United States. The website uses data from the U.S. Department of Education on the price of attending each college, including both the sticker price as well as the net price after the average amount of grant aid awarded by the institution is subtracted.

The earnings data on PayScale come from individuals who visit the website, and voluntarily enter information about which college they attended, when they graduated, and annual earnings from their job. And this is where the problem lies with the PayScale calculations of ROI for individual colleges: they are based entirely on these voluntary, self-reported data. Here are just a few examples of the problems with the earnings data:

• We don’t know if the data are representative of all graduates of the colleges and universities in the PayScale database, but I would highly doubt that they are. So, for example, when PayScale reports that the “Typical starting salary after graduation” for graduates of Michigan State University is $44,300, there is no way to tell just how representative the MSU graduates who reported their salaries to PayScale are of all MSU grads. Are the ones who reported salary information to PayScale more heavily weighted toward graduates of the business or engineering colleges? If so, then the “typical starting salary” is likely to be higher than the “typical” MSU grad could expect, as business and engineering represent a small proportion of all undergraduate degrees awarded by the university (less than a quarter).
• There is no way to tell if people are reporting their salaries accurately. Do people report their exact salary? Do they round up to the nearest $5,000? $10,000? Do they inflate the number in order to make their alma mater look a little better? Do graduates of different institutions differ in the way they report these data?
• A single median earnings figure for a college has relatively little meaning for someone trying to decide which college to attend. Good data on the earnings of college graduates indicate that the greatest variations in earnings are not among colleges, but within individual colleges – with the differences driven primarily by students’ majors and occupational choices. So even if the PayScale data were representative of a single institution, and accurately reported, what is much more important in determining future earnings is what major a student chooses, not which college he attends. I’ll return to this topic in Part 2 of this post.

One could argue that it should not matter too much if the earnings data are problematic. After all, this is just a website trying to make some money like thousands of others out there, so what difference does it make? The problem is that PayScale is not the only website reporting these data and using them to rank colleges in order to help people distinguish one institution from another. Other websites, which are purporting to be more unbiased, objective providers of data about college are also relying on these data.

The Chronicle of Higher Education, a leading and well-respected publication that focuses exclusively on the higher education industry, recently unveiled its “College Reality Check” website. The Chronicle is a for-profit company, but as a member of the media, it prides itself on being an unbiased, objective source of information. The Chronicle’s College Reality Check website uses similar data from the U.S. Department of Education on college prices as does PayScale, and it too uses the PayScale data on earnings of graduates of specific colleges – thus incorporating into its “reality check” the same problems with the PayScale website.

Another recent entry into the college information landscape is CollegeMeasures.org, a website created in part by the American Institutes for Research. It too provides similar data to PayScale and College Reality Check on college prices, and it also relies on the PayScale earnings data.

As I pointed out in my House testimony, we have lots of “data” about college, but we suffer from a lack of good “information.” Attempts to rank colleges based on their “return on investment” – just one more way in which we as a society rank colleges – that utilize misleading and inaccurate data does nothing to help consumers do a better job in choosing a college.